2024-09-13T12:43:14.493 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T12:43:14.510 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T12:43:14.606 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-13_12:30:34-orch:cephadm-wip-guits-main-2024-09-13-0840-distro-default-smithi/7903583 branch: wip-guits-main-2024-09-13-0840 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} email: gabrioux@ibm.com first_in_suite: false job_id: '7903583' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-13_12:30:34-orch:cephadm-wip-guits-main-2024-09-13-0840-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-main-2024-09-13-0840 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) log-only-match: - CEPHADM_ mon_bind_msgr2: false sha1: b0201b8c79733293453e7f10a10c7fa43119222b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: b0201b8c79733293453e7f10a10c7fa43119222b selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-guits-main-2024-09-13-0840 sha1: b0201b8c79733293453e7f10a10c7fa43119222b owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 4031 sha1: b0201b8c79733293453e7f10a10c7fa43119222b sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-main-2024-09-13-0840 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_b0201b8c79733293453e7f10a10c7fa43119222b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: b0201b8c79733293453e7f10a10c7fa43119222b targets: smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKEo+BbE/ZkdNHh3wxtVMM8cp4i1/wvEFld3t4+VUubqkH6wjg4RRLKwXnuDPYk3Dv7O2URCPiKYRjK7Xksi0Eg= smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIIhXc9Ahow9e3dH60OPk+8eCstekIc46So7MJUDGlHhbVeEB2IcsxBTIRdlRCZb7o/9K87JXg2w3mVANh1GXbU= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_12:30:34 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T12:43:14.606 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_b0201b8c79733293453e7f10a10c7fa43119222b/qa; will attempt to use it 2024-09-13T12:43:14.607 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_b0201b8c79733293453e7f10a10c7fa43119222b/qa/tasks 2024-09-13T12:43:14.607 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T12:43:14.609 INFO:teuthology.task.internal:Checking packages... 2024-09-13T12:43:14.629 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b0201b8c79733293453e7f10a10c7fa43119222b' 2024-09-13T12:43:14.629 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T12:43:14.629 INFO:teuthology.packaging:ref: None 2024-09-13T12:43:14.630 INFO:teuthology.packaging:tag: None 2024-09-13T12:43:14.630 INFO:teuthology.packaging:branch: wip-guits-main-2024-09-13-0840 2024-09-13T12:43:14.630 INFO:teuthology.packaging:sha1: b0201b8c79733293453e7f10a10c7fa43119222b 2024-09-13T12:43:14.630 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-main-2024-09-13-0840 2024-09-13T12:43:14.780 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4917.gb0201b8c 2024-09-13T12:43:14.782 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T12:43:14.790 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T12:43:14.790 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T12:43:14.806 INFO:teuthology.task.internal:Saving configuration 2024-09-13T12:43:14.819 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T12:43:14.828 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T12:43:14.856 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-13_12:30:34-orch:cephadm-wip-guits-main-2024-09-13-0840-distro-default-smithi/7903583', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 12:38:00.660828', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKEo+BbE/ZkdNHh3wxtVMM8cp4i1/wvEFld3t4+VUubqkH6wjg4RRLKwXnuDPYk3Dv7O2URCPiKYRjK7Xksi0Eg='} 2024-09-13T12:43:14.886 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-13_12:30:34-orch:cephadm-wip-guits-main-2024-09-13-0840-distro-default-smithi/7903583', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 12:38:00.659299', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIIhXc9Ahow9e3dH60OPk+8eCstekIc46So7MJUDGlHhbVeEB2IcsxBTIRdlRCZb7o/9K87JXg2w3mVANh1GXbU='} 2024-09-13T12:43:14.886 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T12:43:14.895 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-13T12:43:14.895 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-13T12:43:14.895 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T12:43:14.993 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f370ef9fbe0>, signals=[15]) 2024-09-13T12:43:14.993 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T12:43:15.001 INFO:teuthology.task.internal:Opening connections... 2024-09-13T12:43:15.001 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2024-09-13T12:43:15.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T12:43:15.086 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2024-09-13T12:43:15.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T12:43:15.169 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T12:43:15.176 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2024-09-13T12:43:15.195 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2024-09-13T12:43:15.195 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2024-09-13T12:43:15.253 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2024-09-13T12:43:15.253 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="9" 2024-09-13T12:43:15.253 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2024-09-13T12:43:15.253 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2024-09-13T12:43:15.253 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="9" 2024-09-13T12:43:15.253 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el9" 2024-09-13T12:43:15.253 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T12:43:15.253 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2024-09-13T12:43:15.253 INFO:teuthology.orchestra.run.smithi052.stdout:LOGO="fedora-logo-icon" 2024-09-13T12:43:15.253 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T12:43:15.254 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2024-09-13T12:43:15.254 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T12:43:15.254 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T12:43:15.254 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T12:43:15.254 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2024-09-13T12:43:15.282 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2024-09-13T12:43:15.300 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2024-09-13T12:43:15.301 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2024-09-13T12:43:15.359 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="CentOS Stream" 2024-09-13T12:43:15.359 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="9" 2024-09-13T12:43:15.359 INFO:teuthology.orchestra.run.smithi167.stdout:ID="centos" 2024-09-13T12:43:15.359 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE="rhel fedora" 2024-09-13T12:43:15.359 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="9" 2024-09-13T12:43:15.359 INFO:teuthology.orchestra.run.smithi167.stdout:PLATFORM_ID="platform:el9" 2024-09-13T12:43:15.359 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T12:43:15.359 INFO:teuthology.orchestra.run.smithi167.stdout:ANSI_COLOR="0;31" 2024-09-13T12:43:15.359 INFO:teuthology.orchestra.run.smithi167.stdout:LOGO="fedora-logo-icon" 2024-09-13T12:43:15.360 INFO:teuthology.orchestra.run.smithi167.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T12:43:15.360 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://centos.org/" 2024-09-13T12:43:15.360 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T12:43:15.360 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T12:43:15.360 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T12:43:15.360 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2024-09-13T12:43:15.390 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T12:43:15.402 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T12:43:15.411 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T12:43:15.411 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2024-09-13T12:43:15.414 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2024-09-13T12:43:15.430 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T12:43:15.438 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T12:43:15.438 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2024-09-13T12:43:15.471 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2024-09-13T12:43:15.500 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T12:43:15.574 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T12:43:15.590 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T12:43:15.590 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T12:43:15.590 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T12:43:15.591 DEBUG:teuthology.orchestra.run.smithi167:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T12:43:15.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:43:15.608 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-09-13T12:43:15.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:43:15.609 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-09-13T12:43:15.667 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-503.el9.x86_64 2024-09-13T12:43:15.668 INFO:teuthology.orchestra.run.smithi167.stdout:5.14.0-503.el9.x86_64 2024-09-13T12:43:15.668 INFO:teuthology.task.kernel:Running kernel on smithi052: 5.14.0-503.el9.x86_64 2024-09-13T12:43:15.668 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2024-09-13T12:43:15.668 INFO:teuthology.task.kernel:Running kernel on smithi167: 5.14.0-503.el9.x86_64 2024-09-13T12:43:15.668 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum install -y kernel 2024-09-13T12:43:19.481 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.2 MB 00:02 2024-09-13T12:43:20.348 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.2 MB 00:03 2024-09-13T12:43:22.333 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-13T12:43:24.198 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 20 MB 00:03 2024-09-13T12:43:26.784 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-13T12:43:28.533 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-13T12:43:28.584 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 19 kB 00:00 2024-09-13T12:43:29.813 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-13T12:43:30.183 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 19 kB 00:00 2024-09-13T12:43:31.387 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-13T12:43:35.453 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-13T12:43:36.885 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-13T12:43:37.492 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T12:43:37.492 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T12:43:37.585 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:43:37.596 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:43:37.597 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-09-13T12:43:37.597 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:43:37.597 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:43:37.597 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T12:43:37.597 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-13T12:43:37.597 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T12:43:37.597 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T12:43:37.597 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T12:43:37.597 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:43:37.597 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:43:37.597 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:43:37.597 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2024-09-13T12:43:37.598 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:43:37.598 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 79 M 2024-09-13T12:43:37.598 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 126 M 2024-09-13T12:43:37.598 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:43:37.946 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 228 kB/s | 23 kB 00:00 2024-09-13T12:43:38.321 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-13T12:43:38.704 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 38 MB/s | 29 MB 00:00 2024-09-13T12:43:38.916 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T12:43:38.916 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T12:43:39.009 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:43:39.017 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:43:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-09-13T12:43:39.017 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:43:39.017 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:43:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T12:43:39.017 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-09-13T12:43:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T12:43:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T12:43:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T12:43:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:43:39.018 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:43:39.018 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:43:39.018 INFO:teuthology.orchestra.run.smithi167.stdout:Install 4 Packages 2024-09-13T12:43:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:43:39.019 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 79 M 2024-09-13T12:43:39.019 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 126 M 2024-09-13T12:43:39.019 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:43:39.021 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-09-13T12:43:39.021 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:43:39.021 INFO:teuthology.orchestra.run.smithi052.stdout:Total 55 MB/s | 79 MB 00:01 2024-09-13T12:43:39.611 INFO:teuthology.orchestra.run.smithi167.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 42 kB/s | 23 kB 00:00 2024-09-13T12:43:39.732 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:43:40.126 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:43:40.126 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:43:40.486 INFO:teuthology.orchestra.run.smithi167.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-13T12:43:41.120 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:43:41.121 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:43:41.170 INFO:teuthology.orchestra.run.smithi167.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-13T12:43:41.837 INFO:teuthology.orchestra.run.smithi167.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 13 MB/s | 29 MB 00:02 2024-09-13T12:43:41.837 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:43:41.837 INFO:teuthology.orchestra.run.smithi167.stdout:Total 28 MB/s | 79 MB 00:02 2024-09-13T12:43:42.518 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:43:42.868 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:43:42.869 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:43:43.682 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:43:43.820 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:43:43.820 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:43:44.356 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T12:43:44.438 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T12:43:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:43:46.954 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T12:43:47.049 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T12:43:47.246 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T12:43:47.376 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T12:43:49.843 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T12:43:49.982 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T12:43:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T12:43:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T12:43:54.825 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T12:43:54.915 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T12:43:57.682 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T12:43:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T12:44:36.649 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T12:44:36.690 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T12:44:36.998 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T12:44:36.998 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T12:44:36.999 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T12:44:36.999 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T12:44:37.810 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T12:44:37.818 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T12:44:37.818 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:44:37.819 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:44:37.819 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T12:44:37.819 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T12:44:37.819 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T12:44:37.819 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T12:44:37.819 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:44:37.819 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:44:37.854 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T12:44:38.173 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T12:44:38.249 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T12:44:38.249 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T12:44:38.249 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T12:44:38.249 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T12:44:38.249 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T12:44:38.250 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T12:44:38.250 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T12:44:38.250 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T12:44:38.254 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T12:44:38.254 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T12:44:38.254 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T12:44:38.254 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T12:44:38.308 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T12:44:38.308 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T12:44:38.309 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -qi grub2-tools 2024-09-13T12:44:38.343 INFO:teuthology.orchestra.run.smithi052.stdout:Name : grub2-tools 2024-09-13T12:44:38.343 INFO:teuthology.orchestra.run.smithi052.stdout:Epoch : 1 2024-09-13T12:44:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:Version : 2.06 2024-09-13T12:44:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:Release : 82.el9 2024-09-13T12:44:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: x86_64 2024-09-13T12:44:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T12:44:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:Group : Unspecified 2024-09-13T12:44:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:Size : 8274670 2024-09-13T12:44:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:License : GPLv3+ 2024-09-13T12:44:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T12:44:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T12:44:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T12:44:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T12:44:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:Packager : builder@centos.org 2024-09-13T12:44:38.345 INFO:teuthology.orchestra.run.smithi052.stdout:Vendor : CentOS 2024-09-13T12:44:38.345 INFO:teuthology.orchestra.run.smithi052.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T12:44:38.345 INFO:teuthology.orchestra.run.smithi052.stdout:Summary : Support tools for GRUB. 2024-09-13T12:44:38.345 INFO:teuthology.orchestra.run.smithi052.stdout:Description : 2024-09-13T12:44:38.345 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:44:38.345 INFO:teuthology.orchestra.run.smithi052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T12:44:38.345 INFO:teuthology.orchestra.run.smithi052.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T12:44:38.345 INFO:teuthology.orchestra.run.smithi052.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T12:44:38.345 INFO:teuthology.orchestra.run.smithi052.stdout:hardware devices. 2024-09-13T12:44:38.345 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:44:38.345 INFO:teuthology.orchestra.run.smithi052.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T12:44:38.346 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T12:44:38.346 INFO:teuthology.task.kernel:Updating grub on smithi052 to boot 5.14.0-505.el9.x86_64 2024-09-13T12:44:38.346 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T12:44:38.942 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2024-09-13T12:44:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T12:44:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:44:39.127 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:44:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T12:44:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T12:44:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T12:44:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T12:44:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:44:39.128 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:44:39.476 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T12:44:39.562 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T12:44:39.563 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T12:44:39.563 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T12:44:39.563 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T12:44:39.563 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T12:44:39.563 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T12:44:39.563 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi167.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T12:44:39.563 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T12:44:39.622 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T12:44:39.622 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T12:44:39.622 DEBUG:teuthology.orchestra.run.smithi167:> sudo rpm -qi grub2-tools 2024-09-13T12:44:39.654 INFO:teuthology.orchestra.run.smithi167.stdout:Name : grub2-tools 2024-09-13T12:44:39.654 INFO:teuthology.orchestra.run.smithi167.stdout:Epoch : 1 2024-09-13T12:44:39.654 INFO:teuthology.orchestra.run.smithi167.stdout:Version : 2.06 2024-09-13T12:44:39.654 INFO:teuthology.orchestra.run.smithi167.stdout:Release : 82.el9 2024-09-13T12:44:39.654 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: x86_64 2024-09-13T12:44:39.654 INFO:teuthology.orchestra.run.smithi167.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout:Group : Unspecified 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout:Size : 8274670 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout:License : GPLv3+ 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout:Packager : builder@centos.org 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout:Vendor : CentOS 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout:Summary : Support tools for GRUB. 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout:Description : 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:44:39.655 INFO:teuthology.orchestra.run.smithi167.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T12:44:39.656 INFO:teuthology.orchestra.run.smithi167.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T12:44:39.656 INFO:teuthology.orchestra.run.smithi167.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T12:44:39.656 INFO:teuthology.orchestra.run.smithi167.stdout:hardware devices. 2024-09-13T12:44:39.656 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:44:39.656 INFO:teuthology.orchestra.run.smithi167.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T12:44:39.657 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T12:44:39.657 INFO:teuthology.task.kernel:Updating grub on smithi167 to boot 5.14.0-505.el9.x86_64 2024-09-13T12:44:39.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T12:44:40.260 INFO:teuthology.orchestra.run.smithi167.stderr:Generating grub configuration file ... 2024-09-13T12:44:40.296 INFO:teuthology.orchestra.run.smithi052.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T12:44:40.311 INFO:teuthology.orchestra.run.smithi052.stderr:done 2024-09-13T12:44:40.312 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-09-13T12:44:40.327 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.pUEVhRBfiR 2024-09-13T12:44:40.328 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pUEVhRBfiR 2024-09-13T12:44:40.394 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.pUEVhRBfiR 2024-09-13T12:44:40.544 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.pUEVhRBfiR is 6KB 2024-09-13T12:44:40.593 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.pUEVhRBfiR 2024-09-13T12:44:40.608 DEBUG:teuthology.orchestra.run.smithi052:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T12:44:40.670 INFO:teuthology.orchestra.run.smithi052.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T12:44:40.670 INFO:teuthology.orchestra.run.smithi052.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T12:44:40.670 INFO:teuthology.orchestra.run.smithi052.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T12:44:40.670 INFO:teuthology.orchestra.run.smithi052.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T12:44:40.671 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T12:44:40.838 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2024-09-13T12:44:41.620 INFO:teuthology.orchestra.run.smithi167.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T12:44:41.652 INFO:teuthology.orchestra.run.smithi167.stderr:done 2024-09-13T12:44:41.656 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-09-13T12:44:41.672 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.vka5sSgfnS 2024-09-13T12:44:41.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vka5sSgfnS 2024-09-13T12:44:41.739 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.vka5sSgfnS 2024-09-13T12:44:41.876 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.vka5sSgfnS is 6KB 2024-09-13T12:44:41.925 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.vka5sSgfnS 2024-09-13T12:44:41.940 DEBUG:teuthology.orchestra.run.smithi167:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T12:44:42.002 INFO:teuthology.orchestra.run.smithi167.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T12:44:42.002 INFO:teuthology.orchestra.run.smithi167.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T12:44:42.003 INFO:teuthology.orchestra.run.smithi167.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T12:44:42.003 INFO:teuthology.orchestra.run.smithi167.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T12:44:42.004 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T12:44:42.159 DEBUG:teuthology.orchestra.run.smithi167:> sudo shutdown -r now 2024-09-13T12:45:10.869 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T12:45:10.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-13T12:45:10.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T12:45:12.163 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T12:45:12.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-09-13T12:45:12.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T12:45:29.277 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-09-13T12:45:30.561 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-09-13T12:45:38.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-13T12:45:38.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T12:45:39.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-09-13T12:45:39.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T12:45:41.373 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-09-13T12:45:42.621 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-09-13T12:45:53.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-13T12:45:53.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T12:45:54.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-09-13T12:45:54.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T12:46:11.773 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-09-13T12:46:26.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-13T12:46:26.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T12:46:27.481 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-09-13T12:46:27.948 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-13T12:46:27.949 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T12:46:27.949 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-09-13T12:46:27.966 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-505.el9.x86_64 2024-09-13T12:46:27.966 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T12:46:27.966 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T12:46:27.966 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T12:46:28.967 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T12:46:28.967 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T12:46:29.031 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2024-09-13T12:46:29.062 DEBUG:teuthology.parallel:result is None 2024-09-13T12:46:54.636 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T12:47:09.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-09-13T12:47:09.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T12:47:10.075 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-09-13T12:47:10.457 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-09-13T12:47:10.457 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-13T12:47:10.457 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-09-13T12:47:10.474 INFO:teuthology.orchestra.run.smithi167.stdout:5.14.0-505.el9.x86_64 2024-09-13T12:47:10.474 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T12:47:10.474 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T12:47:10.474 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T12:47:11.476 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T12:47:11.476 DEBUG:teuthology.orchestra.run.smithi167:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T12:47:11.540 INFO:teuthology.orchestra.run.smithi167.stdout:ttyS1 2024-09-13T12:47:11.559 DEBUG:teuthology.parallel:result is None 2024-09-13T12:47:11.560 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T12:47:11.572 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T12:47:11.573 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T12:47:11.575 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T12:47:11.595 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T12:47:11.603 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T12:47:11.659 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T12:47:11.659 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T12:47:11.662 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T12:47:11.696 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T12:47:11.704 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T12:47:11.705 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T12:47:11.732 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T12:47:11.768 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:47:11.770 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:47:11.780 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:47:11.783 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T12:47:11.784 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T12:47:11.793 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T12:47:11.793 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T12:47:11.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T12:47:11.853 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T12:47:11.864 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T12:47:11.864 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T12:47:11.895 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T12:47:11.912 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T12:47:12.008 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T12:47:12.047 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T12:47:12.142 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T12:47:12.176 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:47:12.176 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T12:47:12.244 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T12:47:12.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T12:47:12.340 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T12:47:12.434 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T12:47:12.468 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-09-13T12:47:12.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T12:47:12.535 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2024-09-13T12:47:12.538 DEBUG:teuthology.orchestra.run.smithi167:> sudo service rsyslog restart 2024-09-13T12:47:12.577 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T12:47:12.635 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T12:47:13.102 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T12:47:13.111 INFO:teuthology.task.internal:Starting timer... 2024-09-13T12:47:13.112 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T12:47:13.178 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T12:47:13.190 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-13T12:47:13.190 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2024-09-13T12:47:13.265 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2024-09-13T12:47:13.267 DEBUG:teuthology.orchestra.run.smithi167:> sudo service auditd rotate 2024-09-13T12:47:13.331 INFO:teuthology.orchestra.run.smithi167.stdout:Rotating logs: 2024-09-13T12:47:13.333 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T12:47:13.333 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2024-09-13T12:47:13.373 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2024-09-13T12:47:13.373 DEBUG:teuthology.orchestra.run.smithi167:> /usr/sbin/getenforce 2024-09-13T12:47:13.393 INFO:teuthology.orchestra.run.smithi167.stdout:Permissive 2024-09-13T12:47:13.393 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi052.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi167.front.sepia.ceph.com': 'permissive'} 2024-09-13T12:47:13.393 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T12:47:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:47:13.442 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T12:47:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:47:13.470 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T12:47:13.470 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2024-09-13T12:47:13.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo /usr/sbin/setenforce permissive 2024-09-13T12:47:13.555 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T12:47:13.564 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T12:47:13.577 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T12:47:13.578 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com,smithi167.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T12:51:01.402 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com'), Remote(name='ubuntu@smithi167.front.sepia.ceph.com')] 2024-09-13T12:51:01.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-13T12:51:01.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T12:51:01.531 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-09-13T12:51:01.629 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-13T12:51:01.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-09-13T12:51:01.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T12:51:01.768 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-09-13T12:51:01.855 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-09-13T12:51:01.856 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T12:51:02.002 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T12:51:02.002 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T12:51:02.003 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T12:51:02.006 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T12:51:02.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T12:51:02.039 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T12:51:02.041 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T12:51:02.056 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T12:51:02.058 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T12:51:02.084 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2024-09-13T12:51:02.085 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: ntpd: command not found 2024-09-13T12:51:02.097 INFO:teuthology.orchestra.run.smithi167.stdout:506 Cannot talk to daemon 2024-09-13T12:51:02.097 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2024-09-13T12:51:02.111 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T12:51:02.112 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T12:51:02.126 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T12:51:02.127 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T12:51:02.183 INFO:teuthology.orchestra.run.smithi167.stderr:bash: line 1: ntpq: command not found 2024-09-13T12:51:02.187 INFO:teuthology.orchestra.run.smithi167.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T12:51:02.187 INFO:teuthology.orchestra.run.smithi167.stdout:=============================================================================== 2024-09-13T12:51:02.187 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T12:51:02.187 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T12:51:02.187 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T12:51:02.187 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T12:51:02.189 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-09-13T12:51:02.193 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T12:51:02.193 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-09-13T12:51:02.193 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T12:51:02.194 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T12:51:02.194 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T12:51:02.194 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T12:51:02.195 INFO:teuthology.run_tasks:Running task pexec... 2024-09-13T12:51:02.204 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-13T12:51:02.205 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi052.front.sepia.ceph.com 2024-09-13T12:51:02.205 DEBUG:teuthology.orchestra.run.smithi052:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-13T12:51:02.205 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi167.front.sepia.ceph.com 2024-09-13T12:51:02.205 DEBUG:teuthology.orchestra.run.smithi167:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-13T12:51:02.581 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:51:02.583 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:51:02.583 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-09-13T12:51:02.583 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:51:02.583 INFO:teuthology.orchestra.run.smithi167.stdout:Removing: 2024-09-13T12:51:02.583 INFO:teuthology.orchestra.run.smithi167.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-13T12:51:02.583 INFO:teuthology.orchestra.run.smithi167.stdout:Removing unused dependencies: 2024-09-13T12:51:02.583 INFO:teuthology.orchestra.run.smithi167.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-13T12:51:02.583 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:02.583 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:51:02.584 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:51:02.584 INFO:teuthology.orchestra.run.smithi167.stdout:Remove 2 Packages 2024-09-13T12:51:02.584 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:02.584 INFO:teuthology.orchestra.run.smithi167.stdout:Freed space: 5.5 M 2024-09-13T12:51:02.584 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:51:02.588 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:51:02.588 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:51:02.603 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:51:02.614 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:51:02.614 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-09-13T12:51:02.615 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:51:02.615 INFO:teuthology.orchestra.run.smithi052.stdout:Removing: 2024-09-13T12:51:02.615 INFO:teuthology.orchestra.run.smithi052.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-13T12:51:02.615 INFO:teuthology.orchestra.run.smithi052.stdout:Removing unused dependencies: 2024-09-13T12:51:02.615 INFO:teuthology.orchestra.run.smithi052.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-13T12:51:02.615 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:02.615 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:51:02.615 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:51:02.615 INFO:teuthology.orchestra.run.smithi052.stdout:Remove 2 Packages 2024-09-13T12:51:02.615 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:02.615 INFO:teuthology.orchestra.run.smithi052.stdout:Freed space: 5.5 M 2024-09-13T12:51:02.615 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:51:02.619 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:51:02.619 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:51:02.650 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:51:02.651 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:51:02.683 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:51:02.684 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:51:02.807 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:51:02.824 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:51:02.876 INFO:teuthology.orchestra.run.smithi167.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-13T12:51:02.902 INFO:teuthology.orchestra.run.smithi052.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-13T12:51:02.930 INFO:teuthology.orchestra.run.smithi167.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T12:51:02.957 INFO:teuthology.orchestra.run.smithi052.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T12:51:03.429 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T12:51:03.429 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-13T12:51:03.580 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T12:51:03.580 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-13T12:51:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-13T12:51:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:03.677 INFO:teuthology.orchestra.run.smithi167.stdout:Removed: 2024-09-13T12:51:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T12:51:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:03.677 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:51:03.822 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-13T12:51:03.823 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:03.823 INFO:teuthology.orchestra.run.smithi052.stdout:Removed: 2024-09-13T12:51:03.823 INFO:teuthology.orchestra.run.smithi052.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T12:51:03.823 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:03.823 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:51:04.566 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:50 ago on Fri 13 Sep 2024 12:50:14 PM UTC. 2024-09-13T12:51:04.655 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:46 ago on Fri 13 Sep 2024 12:50:18 PM UTC. 2024-09-13T12:51:04.711 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:51:04.712 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:51:04.712 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-09-13T12:51:04.712 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:51:04.712 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:51:04.712 INFO:teuthology.orchestra.run.smithi167.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-13T12:51:04.713 INFO:teuthology.orchestra.run.smithi167.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-13T12:51:04.713 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-09-13T12:51:04.713 INFO:teuthology.orchestra.run.smithi167.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-13T12:51:04.713 INFO:teuthology.orchestra.run.smithi167.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-13T12:51:04.713 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-13T12:51:04.713 INFO:teuthology.orchestra.run.smithi167.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-13T12:51:04.713 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:04.713 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:51:04.713 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:51:04.713 INFO:teuthology.orchestra.run.smithi167.stdout:Install 6 Packages 2024-09-13T12:51:04.713 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:04.719 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 2.0 M 2024-09-13T12:51:04.720 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 9.2 M 2024-09-13T12:51:04.720 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:51:04.799 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:51:04.800 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:51:04.800 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-09-13T12:51:04.800 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:51:04.801 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:51:04.801 INFO:teuthology.orchestra.run.smithi052.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-13T12:51:04.801 INFO:teuthology.orchestra.run.smithi052.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-13T12:51:04.801 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-13T12:51:04.801 INFO:teuthology.orchestra.run.smithi052.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-13T12:51:04.801 INFO:teuthology.orchestra.run.smithi052.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-13T12:51:04.801 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-13T12:51:04.801 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-13T12:51:04.801 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:04.801 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:51:04.801 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:51:04.801 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-09-13T12:51:04.801 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:04.805 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.0 M 2024-09-13T12:51:04.805 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 9.2 M 2024-09-13T12:51:04.805 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:51:05.085 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 291 kB/s | 44 kB 00:00 2024-09-13T12:51:05.110 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 595 kB/s | 106 kB 00:00 2024-09-13T12:51:05.169 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-13T12:51:05.194 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): python3-configshell-1.1.30-1.el9.noarch. 659 kB/s | 72 kB 00:00 2024-09-13T12:51:05.319 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.3 MB/s | 906 kB 00:00 2024-09-13T12:51:05.386 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 3.8 MB/s | 837 kB 00:00 2024-09-13T12:51:05.386 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:51:05.386 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.4 MB/s | 2.0 MB 00:00 2024-09-13T12:51:05.395 INFO:teuthology.orchestra.run.smithi167.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 71 kB/s | 44 kB 00:00 2024-09-13T12:51:05.465 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:51:05.470 INFO:teuthology.orchestra.run.smithi167.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 151 kB/s | 106 kB 00:00 2024-09-13T12:51:05.479 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:51:05.480 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:51:05.737 INFO:teuthology.orchestra.run.smithi167.stdout:(3/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 936 kB/s | 906 kB 00:00 2024-09-13T12:51:05.757 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:51:05.757 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:51:06.080 INFO:teuthology.orchestra.run.smithi167.stdout:(4/6): python3-configshell-1.1.30-1.el9.noarch. 105 kB/s | 72 kB 00:00 2024-09-13T12:51:06.171 INFO:teuthology.orchestra.run.smithi167.stdout:(5/6): python3-kmod-0.9-32.el9.x86_64.rpm 120 kB/s | 84 kB 00:00 2024-09-13T12:51:06.190 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:51:06.273 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-13T12:51:06.325 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-13T12:51:06.361 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-13T12:51:06.584 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-13T12:51:06.616 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-13T12:51:06.630 INFO:teuthology.orchestra.run.smithi167.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 940 kB/s | 837 kB 00:00 2024-09-13T12:51:06.630 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:51:06.630 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.0 MB/s | 2.0 MB 00:01 2024-09-13T12:51:06.712 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:51:06.727 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:51:06.727 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:51:06.996 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:51:06.997 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:51:07.089 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-13T12:51:07.120 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-13T12:51:07.434 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:51:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-13T12:51:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-13T12:51:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-13T12:51:07.791 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-13T12:51:07.827 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-13T12:51:08.151 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-13T12:51:08.152 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-13T12:51:08.152 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-13T12:51:08.152 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-13T12:51:08.152 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-13T12:51:08.152 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-13T12:51:08.262 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-13T12:51:08.317 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-13T12:51:08.499 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-13T12:51:08.499 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:08.499 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:51:08.499 INFO:teuthology.orchestra.run.smithi052.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T12:51:08.499 INFO:teuthology.orchestra.run.smithi052.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-13T12:51:08.499 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-13T12:51:08.500 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:08.500 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:51:08.813 DEBUG:teuthology.parallel:result is None 2024-09-13T12:51:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-13T12:51:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-13T12:51:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-13T12:51:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-13T12:51:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-13T12:51:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-13T12:51:09.686 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-13T12:51:09.686 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:09.686 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:51:09.687 INFO:teuthology.orchestra.run.smithi167.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T12:51:09.687 INFO:teuthology.orchestra.run.smithi167.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-13T12:51:09.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-13T12:51:09.687 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:09.687 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:51:10.023 DEBUG:teuthology.parallel:result is None 2024-09-13T12:51:10.023 INFO:teuthology.run_tasks:Running task install... 2024-09-13T12:51:10.032 DEBUG:teuthology.task.install:project ceph 2024-09-13T12:51:10.032 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b0201b8c79733293453e7f10a10c7fa43119222b'}} 2024-09-13T12:51:10.032 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'b0201b8c79733293453e7f10a10c7fa43119222b'} 2024-09-13T12:51:10.032 INFO:teuthology.task.install:Using flavor: default 2024-09-13T12:51:10.039 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T12:51:10.039 INFO:teuthology.task.install:extra packages: [] 2024-09-13T12:51:10.039 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b0201b8c79733293453e7f10a10c7fa43119222b', 'tag': None, 'wait_for_package': False} 2024-09-13T12:51:10.040 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b0201b8c79733293453e7f10a10c7fa43119222b 2024-09-13T12:51:10.042 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b0201b8c79733293453e7f10a10c7fa43119222b', 'tag': None, 'wait_for_package': False} 2024-09-13T12:51:10.042 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b0201b8c79733293453e7f10a10c7fa43119222b 2024-09-13T12:51:10.199 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/flavors/default/ 2024-09-13T12:51:10.199 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4917.gb0201b8c 2024-09-13T12:51:10.214 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/flavors/default/ 2024-09-13T12:51:10.215 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4917.gb0201b8c 2024-09-13T12:51:10.344 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T12:51:10.344 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:51:10.344 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T12:51:10.362 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T12:51:10.362 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-09-13T12:51:10.362 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T12:51:10.381 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T12:51:10.382 DEBUG:teuthology.orchestra.run.smithi052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b0201b8c79733293453e7f10a10c7fa43119222b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T12:51:10.392 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T12:51:10.392 DEBUG:teuthology.orchestra.run.smithi167:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b0201b8c79733293453e7f10a10c7fa43119222b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T12:51:10.453 DEBUG:teuthology.orchestra.run.smithi052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T12:51:10.459 DEBUG:teuthology.orchestra.run.smithi167:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T12:51:10.534 DEBUG:teuthology.orchestra.run.smithi052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T12:51:10.535 DEBUG:teuthology.orchestra.run.smithi167:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T12:51:10.601 INFO:teuthology.orchestra.run.smithi167.stdout:check_obsoletes = 1 2024-09-13T12:51:10.602 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum clean all 2024-09-13T12:51:10.603 INFO:teuthology.orchestra.run.smithi052.stdout:check_obsoletes = 1 2024-09-13T12:51:10.604 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum clean all 2024-09-13T12:51:10.836 INFO:teuthology.orchestra.run.smithi167.stdout:45 files removed 2024-09-13T12:51:10.846 INFO:teuthology.orchestra.run.smithi052.stdout:45 files removed 2024-09-13T12:51:10.864 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-radosgw 2024-09-13T12:51:10.873 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw 2024-09-13T12:51:11.420 INFO:teuthology.orchestra.run.smithi167.stdout:ceph packages for x86_64 313 kB/s | 84 kB 00:00 2024-09-13T12:51:11.449 INFO:teuthology.orchestra.run.smithi052.stdout:ceph packages for x86_64 305 kB/s | 84 kB 00:00 2024-09-13T12:51:11.721 INFO:teuthology.orchestra.run.smithi167.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-13T12:51:11.749 INFO:teuthology.orchestra.run.smithi052.stdout:ceph noarch packages 95 kB/s | 24 kB 00:00 2024-09-13T12:51:12.016 INFO:teuthology.orchestra.run.smithi167.stdout:ceph source packages 7.5 kB/s | 1.9 kB 00:00 2024-09-13T12:51:12.041 INFO:teuthology.orchestra.run.smithi052.stdout:ceph source packages 7.6 kB/s | 1.9 kB 00:00 2024-09-13T12:51:12.630 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-13T12:51:12.808 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-13T12:51:16.217 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 20 MB 00:02 2024-09-13T12:51:16.331 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 20 MB 00:02 2024-09-13T12:51:20.624 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-13T12:51:21.235 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.5 MB 00:01 2024-09-13T12:51:22.300 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-13T12:51:22.969 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 19 kB 00:00 2024-09-13T12:51:24.088 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-13T12:51:24.298 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-13T12:51:29.569 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-13T12:51:29.954 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-09-13T12:51:31.911 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:51:31.914 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:51:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-09-13T12:51:31.914 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:51:31.914 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:51:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 11 M 2024-09-13T12:51:31.914 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrading: 2024-09-13T12:51:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: librados2 x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 3.5 M 2024-09-13T12:51:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1 x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 3.2 M 2024-09-13T12:51:31.914 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-09-13T12:51:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T12:51:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 5.5 M 2024-09-13T12:51:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 22 M 2024-09-13T12:51:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 25 k 2024-09-13T12:51:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T12:51:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T12:51:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T12:51:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2 x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 756 k 2024-09-13T12:51:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-13T12:51:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T12:51:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T12:51:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T12:51:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1 x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 508 k 2024-09-13T12:51:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T12:51:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2 x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 5.5 M 2024-09-13T12:51:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T12:51:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T12:51:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T12:51:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 45 k 2024-09-13T12:51:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 143 k 2024-09-13T12:51:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 162 k 2024-09-13T12:51:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T12:51:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 321 k 2024-09-13T12:51:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 305 k 2024-09-13T12:51:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 99 k 2024-09-13T12:51:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T12:51:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T12:51:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T12:51:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T12:51:31.917 INFO:teuthology.orchestra.run.smithi167.stdout:Installing weak dependencies: 2024-09-13T12:51:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T12:51:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:31.917 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:51:31.917 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:51:31.917 INFO:teuthology.orchestra.run.smithi167.stdout:Install 31 Packages 2024-09-13T12:51:31.917 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrade 2 Packages 2024-09-13T12:51:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:31.918 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 62 M 2024-09-13T12:51:31.918 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:51:32.442 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:51:32.445 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:51:32.445 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-13T12:51:32.445 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:51:32.445 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:51:32.445 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 11 M 2024-09-13T12:51:32.445 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrading: 2024-09-13T12:51:32.445 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 3.5 M 2024-09-13T12:51:32.445 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1 x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 3.2 M 2024-09-13T12:51:32.445 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-13T12:51:32.446 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T12:51:32.446 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 5.5 M 2024-09-13T12:51:32.446 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 22 M 2024-09-13T12:51:32.446 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 25 k 2024-09-13T12:51:32.446 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T12:51:32.446 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T12:51:32.446 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T12:51:32.446 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 756 k 2024-09-13T12:51:32.446 INFO:teuthology.orchestra.run.smithi052.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-13T12:51:32.446 INFO:teuthology.orchestra.run.smithi052.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T12:51:32.446 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T12:51:32.446 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 508 k 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2 x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 5.5 M 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 45 k 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 143 k 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 162 k 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 321 k 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 305 k 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 99 k 2024-09-13T12:51:32.447 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T12:51:32.448 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T12:51:32.448 INFO:teuthology.orchestra.run.smithi052.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T12:51:32.448 INFO:teuthology.orchestra.run.smithi052.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T12:51:32.448 INFO:teuthology.orchestra.run.smithi052.stdout:Installing weak dependencies: 2024-09-13T12:51:32.448 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T12:51:32.448 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:32.448 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:51:32.448 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:51:32.448 INFO:teuthology.orchestra.run.smithi052.stdout:Install 31 Packages 2024-09-13T12:51:32.448 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrade 2 Packages 2024-09-13T12:51:32.448 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:32.449 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 62 M 2024-09-13T12:51:32.449 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:51:32.613 INFO:teuthology.orchestra.run.smithi167.stdout:(1/33): ceph-base-19.3.0-4917.gb0201b8c.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-13T12:51:32.679 INFO:teuthology.orchestra.run.smithi167.stdout:(2/33): ceph-selinux-19.3.0-4917.gb0201b8c.el9. 378 kB/s | 25 kB 00:00 2024-09-13T12:51:32.854 INFO:teuthology.orchestra.run.smithi167.stdout:(3/33): ceph-radosgw-19.3.0-4917.gb0201b8c.el9. 14 MB/s | 11 MB 00:00 2024-09-13T12:51:32.888 INFO:teuthology.orchestra.run.smithi167.stdout:(4/33): libcephfs2-19.3.0-4917.gb0201b8c.el9.x8 3.5 MB/s | 756 kB 00:00 2024-09-13T12:51:32.921 INFO:teuthology.orchestra.run.smithi167.stdout:(5/33): libradosstriper1-19.3.0-4917.gb0201b8c. 7.5 MB/s | 508 kB 00:00 2024-09-13T12:51:32.980 INFO:teuthology.orchestra.run.smithi167.stdout:(6/33): python3-ceph-argparse-19.3.0-4917.gb020 777 kB/s | 45 kB 00:00 2024-09-13T12:51:33.080 INFO:teuthology.orchestra.run.smithi167.stdout:(7/33): python3-ceph-common-19.3.0-4917.gb0201b 1.4 MB/s | 143 kB 00:00 2024-09-13T12:51:33.165 INFO:teuthology.orchestra.run.smithi167.stdout:(8/33): librgw2-19.3.0-4917.gb0201b8c.el9.x86_6 20 MB/s | 5.5 MB 00:00 2024-09-13T12:51:33.189 INFO:teuthology.orchestra.run.smithi167.stdout:(9/33): python3-cephfs-19.3.0-4917.gb0201b8c.el 1.5 MB/s | 162 kB 00:00 2024-09-13T12:51:33.214 INFO:teuthology.orchestra.run.smithi167.stdout:(10/33): python3-rados-19.3.0-4917.gb0201b8c.el 6.3 MB/s | 321 kB 00:00 2024-09-13T12:51:33.239 INFO:teuthology.orchestra.run.smithi167.stdout:(11/33): python3-rbd-19.3.0-4917.gb0201b8c.el9. 5.9 MB/s | 305 kB 00:00 2024-09-13T12:51:33.265 INFO:teuthology.orchestra.run.smithi167.stdout:(12/33): python3-rgw-19.3.0-4917.gb0201b8c.el9. 1.9 MB/s | 99 kB 00:00 2024-09-13T12:51:33.474 INFO:teuthology.orchestra.run.smithi167.stdout:(13/33): ceph-common-19.3.0-4917.gb0201b8c.el9. 16 MB/s | 22 MB 00:01 2024-09-13T12:51:33.607 INFO:teuthology.orchestra.run.smithi167.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 291 kB/s | 107 kB 00:00 2024-09-13T12:51:33.657 INFO:teuthology.orchestra.run.smithi167.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 903 kB/s | 45 kB 00:00 2024-09-13T12:51:33.683 INFO:teuthology.orchestra.run.smithi167.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 408 kB/s | 171 kB 00:00 2024-09-13T12:51:33.700 INFO:teuthology.orchestra.run.smithi167.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 710 kB/s | 160 kB 00:00 2024-09-13T12:51:33.750 INFO:teuthology.orchestra.run.smithi167.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 842 kB/s | 42 kB 00:00 2024-09-13T12:51:33.783 INFO:teuthology.orchestra.run.smithi167.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-13T12:51:33.797 INFO:teuthology.orchestra.run.smithi052.stdout:(1/33): ceph-base-19.3.0-4917.gb0201b8c.el9.x86 5.5 MB/s | 5.5 MB 00:00 2024-09-13T12:51:33.817 INFO:teuthology.orchestra.run.smithi167.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-09-13T12:51:33.839 INFO:teuthology.orchestra.run.smithi052.stdout:(2/33): ceph-selinux-19.3.0-4917.gb0201b8c.el9. 606 kB/s | 25 kB 00:00 2024-09-13T12:51:33.842 INFO:teuthology.orchestra.run.smithi167.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 610 kB/s | 36 kB 00:00 2024-09-13T12:51:33.876 INFO:teuthology.orchestra.run.smithi167.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-13T12:51:33.902 INFO:teuthology.orchestra.run.smithi167.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 563 kB/s | 47 kB 00:00 2024-09-13T12:51:33.931 INFO:teuthology.orchestra.run.smithi052.stdout:(3/33): libcephfs2-19.3.0-4917.gb0201b8c.el9.x8 8.1 MB/s | 756 kB 00:00 2024-09-13T12:51:33.960 INFO:teuthology.orchestra.run.smithi167.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-13T12:51:33.985 INFO:teuthology.orchestra.run.smithi167.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 373 kB/s | 26 kB 00:00 2024-09-13T12:51:34.006 INFO:teuthology.orchestra.run.smithi052.stdout:(4/33): libradosstriper1-19.3.0-4917.gb0201b8c. 6.6 MB/s | 508 kB 00:00 2024-09-13T12:51:34.011 INFO:teuthology.orchestra.run.smithi167.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 965 kB/s | 49 kB 00:00 2024-09-13T12:51:34.036 INFO:teuthology.orchestra.run.smithi167.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-13T12:51:34.078 INFO:teuthology.orchestra.run.smithi167.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-13T12:51:34.103 INFO:teuthology.orchestra.run.smithi167.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-13T12:51:34.198 INFO:teuthology.orchestra.run.smithi052.stdout:(5/33): ceph-radosgw-19.3.0-4917.gb0201b8c.el9. 7.9 MB/s | 11 MB 00:01 2024-09-13T12:51:34.229 INFO:teuthology.orchestra.run.smithi167.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-13T12:51:34.240 INFO:teuthology.orchestra.run.smithi052.stdout:(6/33): python3-ceph-argparse-19.3.0-4917.gb020 1.1 MB/s | 45 kB 00:00 2024-09-13T12:51:34.287 INFO:teuthology.orchestra.run.smithi167.stdout:(31/33): librados2-19.3.0-4917.gb0201b8c.el9.x8 19 MB/s | 3.5 MB 00:00 2024-09-13T12:51:34.298 INFO:teuthology.orchestra.run.smithi052.stdout:(7/33): python3-ceph-common-19.3.0-4917.gb0201b 2.4 MB/s | 143 kB 00:00 2024-09-13T12:51:34.357 INFO:teuthology.orchestra.run.smithi052.stdout:(8/33): python3-cephfs-19.3.0-4917.gb0201b8c.el 2.7 MB/s | 162 kB 00:00 2024-09-13T12:51:34.429 INFO:teuthology.orchestra.run.smithi167.stdout:(32/33): libarrow-9.0.0-11.el9.x86_64.rpm 8.0 MB/s | 4.4 MB 00:00 2024-09-13T12:51:34.440 INFO:teuthology.orchestra.run.smithi052.stdout:(9/33): python3-rados-19.3.0-4917.gb0201b8c.el9 3.8 MB/s | 321 kB 00:00 2024-09-13T12:51:34.488 INFO:teuthology.orchestra.run.smithi167.stdout:(33/33): librbd1-19.3.0-4917.gb0201b8c.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-13T12:51:34.493 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:51:34.493 INFO:teuthology.orchestra.run.smithi167.stdout:Total 24 MB/s | 62 MB 00:02 2024-09-13T12:51:34.516 INFO:teuthology.orchestra.run.smithi052.stdout:(10/33): librgw2-19.3.0-4917.gb0201b8c.el9.x86_ 11 MB/s | 5.5 MB 00:00 2024-09-13T12:51:34.549 INFO:teuthology.orchestra.run.smithi052.stdout:(11/33): python3-rbd-19.3.0-4917.gb0201b8c.el9. 2.7 MB/s | 305 kB 00:00 2024-09-13T12:51:34.575 INFO:teuthology.orchestra.run.smithi052.stdout:(12/33): python3-rgw-19.3.0-4917.gb0201b8c.el9. 1.7 MB/s | 99 kB 00:00 2024-09-13T12:51:34.605 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:51:34.642 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:51:34.643 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:51:34.658 INFO:teuthology.orchestra.run.smithi052.stdout:(13/33): boost-program-options-1.75.0-8.el9.x86 985 kB/s | 107 kB 00:00 2024-09-13T12:51:34.700 INFO:teuthology.orchestra.run.smithi052.stdout:(14/33): libnbd-1.20.2-2.el9.x86_64.rpm 1.3 MB/s | 171 kB 00:00 2024-09-13T12:51:34.734 INFO:teuthology.orchestra.run.smithi052.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-09-13T12:51:34.807 INFO:teuthology.orchestra.run.smithi052.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 425 kB/s | 45 kB 00:00 2024-09-13T12:51:34.860 INFO:teuthology.orchestra.run.smithi052.stdout:(17/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-09-13T12:51:35.077 INFO:teuthology.orchestra.run.smithi052.stdout:(18/33): ceph-common-19.3.0-4917.gb0201b8c.el9. 9.6 MB/s | 22 MB 00:02 2024-09-13T12:51:35.102 INFO:teuthology.orchestra.run.smithi052.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 175 kB/s | 42 kB 00:00 2024-09-13T12:51:35.127 INFO:teuthology.orchestra.run.smithi052.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 915 kB/s | 292 kB 00:00 2024-09-13T12:51:35.152 INFO:teuthology.orchestra.run.smithi052.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 717 kB/s | 36 kB 00:00 2024-09-13T12:51:35.171 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:51:35.171 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:51:35.177 INFO:teuthology.orchestra.run.smithi052.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-09-13T12:51:35.219 INFO:teuthology.orchestra.run.smithi052.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-09-13T12:51:35.261 INFO:teuthology.orchestra.run.smithi052.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-13T12:51:35.295 INFO:teuthology.orchestra.run.smithi052.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 355 kB/s | 26 kB 00:00 2024-09-13T12:51:35.320 INFO:teuthology.orchestra.run.smithi052.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 858 kB/s | 49 kB 00:00 2024-09-13T12:51:35.345 INFO:teuthology.orchestra.run.smithi052.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-13T12:51:35.388 INFO:teuthology.orchestra.run.smithi052.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-13T12:51:35.421 INFO:teuthology.orchestra.run.smithi052.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-13T12:51:35.563 INFO:teuthology.orchestra.run.smithi052.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-09-13T12:51:35.655 INFO:teuthology.orchestra.run.smithi052.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2024-09-13T12:51:35.714 INFO:teuthology.orchestra.run.smithi052.stdout:(32/33): librados2-19.3.0-4917.gb0201b8c.el9.x8 12 MB/s | 3.5 MB 00:00 2024-09-13T12:51:35.822 INFO:teuthology.orchestra.run.smithi052.stdout:(33/33): librbd1-19.3.0-4917.gb0201b8c.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-13T12:51:35.828 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:51:35.828 INFO:teuthology.orchestra.run.smithi052.stdout:Total 18 MB/s | 62 MB 00:03 2024-09-13T12:51:35.946 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:51:35.987 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:51:35.988 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:51:36.531 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:51:36.531 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:51:36.637 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:51:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-13T12:51:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-13T12:51:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-13T12:51:36.799 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-13T12:51:36.821 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-13T12:51:37.053 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-13T12:51:37.105 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-13T12:51:37.449 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-13T12:51:37.559 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librados2-2:19.3.0-4917.gb0201b8c.el9.x86_64 7/35 2024-09-13T12:51:37.644 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-2:19.3.0-4917.gb0201b8c.el9.x86_64 7/35 2024-09-13T12:51:37.734 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rados-2:19.3.0-4917.gb0201b8c.el9.x86_64 8/35 2024-09-13T12:51:37.769 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs2-2:19.3.0-4917.gb0201b8c.el9.x86_64 9/35 2024-09-13T12:51:37.833 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs2-2:19.3.0-4917.gb0201b8c.el9.x86_64 9/35 2024-09-13T12:51:37.870 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-13T12:51:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-argparse-2:19.3.0-4917.gb0201b8c.el9. 11/35 2024-09-13T12:51:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cephfs-2:19.3.0-4917.gb0201b8c.el9.x86_64 12/35 2024-09-13T12:51:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libradosstriper1-2:19.3.0-4917.gb0201b8c.el9.x86_6 13/35 2024-09-13T12:51:38.046 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:51:38.086 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4917.gb0201b8c.el9.x86_6 13/35 2024-09-13T12:51:38.121 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-13T12:51:38.133 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-13T12:51:38.160 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-13T12:51:38.186 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-13T12:51:38.208 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-13T12:51:38.232 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-13T12:51:38.233 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-13T12:51:38.280 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-13T12:51:38.322 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-13T12:51:38.353 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-13T12:51:38.383 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-13T12:51:38.416 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-13T12:51:38.429 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-13T12:51:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-13T12:51:38.489 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-13T12:51:38.772 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-13T12:51:38.813 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librbd1-2:19.3.0-4917.gb0201b8c.el9.x86_64 23/35 2024-09-13T12:51:38.847 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-13T12:51:38.904 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-2:19.3.0-4917.gb0201b8c.el9.x86_64 23/35 2024-09-13T12:51:38.935 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 24/35 2024-09-13T12:51:38.953 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librados2-2:19.3.0-4917.gb0201b8c.el9.x86_64 7/35 2024-09-13T12:51:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-13T12:51:39.037 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:19.3.0-4917.gb0201b8c.el9.x86_64 7/35 2024-09-13T12:51:39.128 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-2:19.3.0-4917.gb0201b8c.el9.x86_64 8/35 2024-09-13T12:51:39.154 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-2:19.3.0-4917.gb0201b8c.el9.x86_64 9/35 2024-09-13T12:51:39.209 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-2:19.3.0-4917.gb0201b8c.el9.x86_64 9/35 2024-09-13T12:51:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-13T12:51:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-argparse-2:19.3.0-4917.gb0201b8c.el9. 11/35 2024-09-13T12:51:39.378 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-2:19.3.0-4917.gb0201b8c.el9.x86_64 12/35 2024-09-13T12:51:39.406 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradosstriper1-2:19.3.0-4917.gb0201b8c.el9.x86_6 13/35 2024-09-13T12:51:39.463 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-13T12:51:39.471 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4917.gb0201b8c.el9.x86_6 13/35 2024-09-13T12:51:39.499 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librgw2-2:19.3.0-4917.gb0201b8c.el9.x86_64 27/35 2024-09-13T12:51:39.505 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-13T12:51:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librgw2-2:19.3.0-4917.gb0201b8c.el9.x86_64 27/35 2024-09-13T12:51:39.617 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-13T12:51:39.649 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 28/35 2024-09-13T12:51:39.656 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-13T12:51:39.693 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-common-2:19.3.0-4917.gb0201b8c.el9.x8 29/35 2024-09-13T12:51:39.706 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-13T12:51:39.738 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-13T12:51:39.767 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-13T12:51:39.800 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-13T12:51:39.838 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-13T12:51:40.206 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-13T12:51:40.255 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librbd1-2:19.3.0-4917.gb0201b8c.el9.x86_64 23/35 2024-09-13T12:51:40.339 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:19.3.0-4917.gb0201b8c.el9.x86_64 23/35 2024-09-13T12:51:40.368 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 24/35 2024-09-13T12:51:40.405 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-13T12:51:40.927 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-13T12:51:40.957 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librgw2-2:19.3.0-4917.gb0201b8c.el9.x86_64 27/35 2024-09-13T12:51:41.029 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librgw2-2:19.3.0-4917.gb0201b8c.el9.x86_64 27/35 2024-09-13T12:51:41.111 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 28/35 2024-09-13T12:51:41.143 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-common-2:19.3.0-4917.gb0201b8c.el9.x8 29/35 2024-09-13T12:51:41.972 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:19.3.0-4917.gb0201b8c.el9.x86_64 30/35 2024-09-13T12:51:42.006 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-common-2:19.3.0-4917.gb0201b8c.el9.x86_64 30/35 2024-09-13T12:51:42.580 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:19.3.0-4917.gb0201b8c.el9.x86_64 30/35 2024-09-13T12:51:42.626 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-base-2:19.3.0-4917.gb0201b8c.el9.x86_64 31/35 2024-09-13T12:51:42.709 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-base-2:19.3.0-4917.gb0201b8c.el9.x86_64 31/35 2024-09-13T12:51:42.710 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T12:51:42.710 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T12:51:42.710 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:42.745 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-selinux-2:19.3.0-4917.gb0201b8c.el9.x86_64 32/35 2024-09-13T12:51:43.565 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.3.0-4917.gb0201b8c.el9.x86_64 30/35 2024-09-13T12:51:43.606 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-2:19.3.0-4917.gb0201b8c.el9.x86_64 30/35 2024-09-13T12:51:44.204 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.3.0-4917.gb0201b8c.el9.x86_64 30/35 2024-09-13T12:51:44.242 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-2:19.3.0-4917.gb0201b8c.el9.x86_64 31/35 2024-09-13T12:51:44.337 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-2:19.3.0-4917.gb0201b8c.el9.x86_64 31/35 2024-09-13T12:51:44.337 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T12:51:44.337 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T12:51:44.337 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:44.377 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-selinux-2:19.3.0-4917.gb0201b8c.el9.x86_64 32/35 2024-09-13T12:51:53.078 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4917.gb0201b8c.el9.x86_64 32/35 2024-09-13T12:51:53.078 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /sys 2024-09-13T12:51:53.079 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /proc 2024-09-13T12:51:53.079 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /mnt 2024-09-13T12:51:53.079 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /var/tmp 2024-09-13T12:51:53.079 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /home 2024-09-13T12:51:53.079 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /root 2024-09-13T12:51:53.079 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /tmp 2024-09-13T12:51:53.079 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:54.106 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-radosgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 33/35 2024-09-13T12:51:54.131 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 33/35 2024-09-13T12:51:54.132 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:51:54.132 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T12:51:54.132 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T12:51:54.132 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T12:51:54.132 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:54.133 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-13T12:51:54.217 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-13T12:51:54.217 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-13T12:51:54.912 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4917.gb0201b8c.el9.x86_64 32/35 2024-09-13T12:51:54.912 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /sys 2024-09-13T12:51:54.912 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /proc 2024-09-13T12:51:54.912 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /mnt 2024-09-13T12:51:54.912 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /var/tmp 2024-09-13T12:51:54.912 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /home 2024-09-13T12:51:54.912 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /root 2024-09-13T12:51:54.912 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /tmp 2024-09-13T12:51:54.912 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-13T12:51:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-base-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/35 2024-09-13T12:51:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-common-2:19.3.0-4917.gb0201b8c.el9.x86_64 2/35 2024-09-13T12:51:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-radosgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 3/35 2024-09-13T12:51:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-selinux-2:19.3.0-4917.gb0201b8c.el9.x86_64 4/35 2024-09-13T12:51:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs2-2:19.3.0-4917.gb0201b8c.el9.x86_64 5/35 2024-09-13T12:51:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libradosstriper1-2:19.3.0-4917.gb0201b8c.el9.x86_6 6/35 2024-09-13T12:51:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librgw2-2:19.3.0-4917.gb0201b8c.el9.x86_64 7/35 2024-09-13T12:51:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4917.gb0201b8c.el9. 8/35 2024-09-13T12:51:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-common-2:19.3.0-4917.gb0201b8c.el9.x8 9/35 2024-09-13T12:51:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cephfs-2:19.3.0-4917.gb0201b8c.el9.x86_64 10/35 2024-09-13T12:51:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rados-2:19.3.0-4917.gb0201b8c.el9.x86_64 11/35 2024-09-13T12:51:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 12/35 2024-09-13T12:51:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 13/35 2024-09-13T12:51:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-13T12:51:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-13T12:51:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-13T12:51:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-13T12:51:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-13T12:51:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-13T12:51:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-13T12:51:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-13T12:51:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-13T12:51:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-13T12:51:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-13T12:51:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-13T12:51:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-13T12:51:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-13T12:51:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-13T12:51:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-13T12:51:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-13T12:51:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-13T12:51:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-2:19.3.0-4917.gb0201b8c.el9.x86_64 32/35 2024-09-13T12:51:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-13T12:51:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-2:19.3.0-4917.gb0201b8c.el9.x86_64 34/35 2024-09-13T12:51:55.951 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 33/35 2024-09-13T12:51:55.978 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 33/35 2024-09-13T12:51:55.978 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:51:55.978 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T12:51:55.978 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T12:51:55.979 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T12:51:55.979 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:55.980 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-13T12:51:56.070 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-13T12:51:56.071 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-13T12:51:56.192 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-13T12:51:56.192 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:56.192 INFO:teuthology.orchestra.run.smithi167.stdout:Upgraded: 2024-09-13T12:51:56.192 INFO:teuthology.orchestra.run.smithi167.stdout: librados2-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.192 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.192 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:51:56.192 INFO:teuthology.orchestra.run.smithi167.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T12:51:56.192 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.192 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.192 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.192 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.193 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T12:51:56.193 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T12:51:56.193 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T12:51:56.193 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.193 INFO:teuthology.orchestra.run.smithi167.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-13T12:51:56.193 INFO:teuthology.orchestra.run.smithi167.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T12:51:56.193 INFO:teuthology.orchestra.run.smithi167.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T12:51:56.193 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T12:51:56.193 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.193 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T12:51:56.193 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.193 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T12:51:56.193 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T12:51:56.194 INFO:teuthology.orchestra.run.smithi167.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T12:51:56.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T12:51:56.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:56.194 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T12:51:56.194 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T12:51:56.194 INFO:teuthology.orchestra.run.smithi167.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T12:51:56.194 INFO:teuthology.orchestra.run.smithi167.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T12:51:56.195 INFO:teuthology.orchestra.run.smithi167.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T12:51:56.195 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:56.195 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:51:56.555 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-test 2024-09-13T12:51:57.125 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:28 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:51:57.264 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:51:57.264 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:51:57.264 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-09-13T12:51:57.264 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:51:57.265 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:51:57.265 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 50 M 2024-09-13T12:51:57.265 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-09-13T12:51:57.265 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 161 k 2024-09-13T12:51:57.265 INFO:teuthology.orchestra.run.smithi167.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T12:51:57.265 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T12:51:57.265 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:57.265 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:51:57.265 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:51:57.265 INFO:teuthology.orchestra.run.smithi167.stdout:Install 4 Packages 2024-09-13T12:51:57.265 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:51:57.266 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 51 M 2024-09-13T12:51:57.266 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 217 M 2024-09-13T12:51:57.266 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:51:57.292 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-13T12:51:57.292 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/35 2024-09-13T12:51:57.292 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-2:19.3.0-4917.gb0201b8c.el9.x86_64 2/35 2024-09-13T12:51:57.292 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 3/35 2024-09-13T12:51:57.292 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-selinux-2:19.3.0-4917.gb0201b8c.el9.x86_64 4/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-2:19.3.0-4917.gb0201b8c.el9.x86_64 5/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradosstriper1-2:19.3.0-4917.gb0201b8c.el9.x86_6 6/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librgw2-2:19.3.0-4917.gb0201b8c.el9.x86_64 7/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4917.gb0201b8c.el9. 8/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-common-2:19.3.0-4917.gb0201b8c.el9.x8 9/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-2:19.3.0-4917.gb0201b8c.el9.x86_64 10/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-2:19.3.0-4917.gb0201b8c.el9.x86_64 11/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 12/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 13/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-13T12:51:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-13T12:51:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-13T12:51:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-13T12:51:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-13T12:51:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-13T12:51:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-13T12:51:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-13T12:51:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-13T12:51:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-13T12:51:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-13T12:51:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-13T12:51:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-13T12:51:57.295 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-13T12:51:57.295 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-13T12:51:57.295 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:19.3.0-4917.gb0201b8c.el9.x86_64 32/35 2024-09-13T12:51:57.295 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-13T12:51:57.295 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:19.3.0-4917.gb0201b8c.el9.x86_64 34/35 2024-09-13T12:51:57.417 INFO:teuthology.orchestra.run.smithi167.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.3 MB/s | 304 kB 00:00 2024-09-13T12:51:57.459 INFO:teuthology.orchestra.run.smithi167.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-13T12:51:57.509 INFO:teuthology.orchestra.run.smithi167.stdout:(3/4): libcephsqlite-19.3.0-4917.gb0201b8c.el9. 737 kB/s | 161 kB 00:00 2024-09-13T12:51:58.119 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-13T12:51:58.119 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:58.119 INFO:teuthology.orchestra.run.smithi052.stdout:Upgraded: 2024-09-13T12:51:58.119 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.119 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.119 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:51:58.119 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T12:51:58.119 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.119 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.119 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.119 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.119 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T12:51:58.120 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T12:51:58.120 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T12:51:58.120 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.120 INFO:teuthology.orchestra.run.smithi052.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-13T12:51:58.120 INFO:teuthology.orchestra.run.smithi052.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T12:51:58.120 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T12:51:58.120 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T12:51:58.120 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.120 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T12:51:58.120 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.120 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T12:51:58.120 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T12:51:58.120 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:58.121 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:51:58.493 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test 2024-09-13T12:51:58.852 INFO:teuthology.orchestra.run.smithi167.stdout:(4/4): ceph-test-19.3.0-4917.gb0201b8c.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-13T12:51:58.854 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:51:58.854 INFO:teuthology.orchestra.run.smithi167.stdout:Total 32 MB/s | 51 MB 00:01 2024-09-13T12:51:58.880 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:51:58.901 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:51:58.901 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:51:59.082 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:30 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:51:59.220 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:51:59.220 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:51:59.221 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-13T12:51:59.221 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:51:59.221 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:51:59.221 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 50 M 2024-09-13T12:51:59.221 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-13T12:51:59.221 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 161 k 2024-09-13T12:51:59.221 INFO:teuthology.orchestra.run.smithi052.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T12:51:59.221 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T12:51:59.221 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:59.221 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:51:59.221 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:51:59.221 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2024-09-13T12:51:59.221 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:51:59.222 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 51 M 2024-09-13T12:51:59.222 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 217 M 2024-09-13T12:51:59.222 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:51:59.282 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:51:59.282 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:51:59.443 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-09-13T12:51:59.485 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-13T12:51:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): libcephsqlite-19.3.0-4917.gb0201b8c.el9. 736 kB/s | 161 kB 00:00 2024-09-13T12:51:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:51:59.926 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T12:51:59.971 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T12:51:59.996 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephsqlite-2:19.3.0-4917.gb0201b8c.el9.x86_64 3/4 2024-09-13T12:52:00.877 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): ceph-test-19.3.0-4917.gb0201b8c.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-13T12:52:00.879 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:00.880 INFO:teuthology.orchestra.run.smithi052.stdout:Total 31 MB/s | 51 MB 00:01 2024-09-13T12:52:00.908 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:52:00.931 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:52:00.931 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:52:01.334 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:52:01.334 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:52:01.886 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:52:01.961 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T12:52:02.004 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T12:52:02.029 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephsqlite-2:19.3.0-4917.gb0201b8c.el9.x86_64 3/4 2024-09-13T12:52:04.735 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4917.gb0201b8c.el9.x86_64 3/4 2024-09-13T12:52:04.810 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-test-2:19.3.0-4917.gb0201b8c.el9.x86_64 4/4 2024-09-13T12:52:07.079 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4917.gb0201b8c.el9.x86_64 3/4 2024-09-13T12:52:07.155 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-2:19.3.0-4917.gb0201b8c.el9.x86_64 4/4 2024-09-13T12:52:07.432 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-test-2:19.3.0-4917.gb0201b8c.el9.x86_64 4/4 2024-09-13T12:52:07.433 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-test-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/4 2024-09-13T12:52:07.433 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephsqlite-2:19.3.0-4917.gb0201b8c.el9.x86_64 2/4 2024-09-13T12:52:07.433 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T12:52:07.842 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T12:52:07.842 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:07.842 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:52:07.843 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:07.843 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:07.843 INFO:teuthology.orchestra.run.smithi167.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T12:52:07.843 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T12:52:07.843 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:07.843 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:52:08.120 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph 2024-09-13T12:52:08.688 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:39 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:08.822 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:52:08.824 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:08.824 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-09-13T12:52:08.824 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:08.825 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:52:08.825 INFO:teuthology.orchestra.run.smithi167.stdout: ceph x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 6.6 k 2024-09-13T12:52:08.825 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-09-13T12:52:08.825 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 2.4 M 2024-09-13T12:52:08.825 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 1.5 M 2024-09-13T12:52:08.825 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 291 k 2024-09-13T12:52:08.825 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 4.8 M 2024-09-13T12:52:08.825 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 17 M 2024-09-13T12:52:08.825 INFO:teuthology.orchestra.run.smithi167.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T12:52:08.825 INFO:teuthology.orchestra.run.smithi167.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T12:52:08.825 INFO:teuthology.orchestra.run.smithi167.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T12:52:08.825 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T12:52:08.826 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T12:52:08.826 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T12:52:08.826 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T12:52:08.826 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T12:52:08.826 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T12:52:08.826 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T12:52:08.826 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-collections 2024-09-13T12:52:08.826 INFO:teuthology.orchestra.run.smithi167.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T12:52:08.826 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T12:52:08.826 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T12:52:08.826 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T12:52:08.826 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T12:52:08.826 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T12:52:08.827 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T12:52:08.827 INFO:teuthology.orchestra.run.smithi167.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T12:52:08.827 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T12:52:08.827 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T12:52:08.827 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T12:52:08.827 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T12:52:08.827 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T12:52:08.827 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T12:52:08.827 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T12:52:08.827 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T12:52:08.827 INFO:teuthology.orchestra.run.smithi167.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T12:52:08.827 INFO:teuthology.orchestra.run.smithi167.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T12:52:08.827 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T12:52:08.828 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T12:52:08.828 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T12:52:08.829 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:08.829 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:52:08.829 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:08.829 INFO:teuthology.orchestra.run.smithi167.stdout:Install 37 Packages 2024-09-13T12:52:08.829 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:08.830 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 31 M 2024-09-13T12:52:08.830 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 109 M 2024-09-13T12:52:08.830 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:52:09.345 INFO:teuthology.orchestra.run.smithi167.stdout:(1/37): ceph-19.3.0-4917.gb0201b8c.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-09-13T12:52:09.513 INFO:teuthology.orchestra.run.smithi167.stdout:(2/37): ceph-mgr-19.3.0-4917.gb0201b8c.el9.x86_ 4.5 MB/s | 1.5 MB 00:00 2024-09-13T12:52:09.548 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-2:19.3.0-4917.gb0201b8c.el9.x86_64 4/4 2024-09-13T12:52:09.548 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/4 2024-09-13T12:52:09.548 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephsqlite-2:19.3.0-4917.gb0201b8c.el9.x86_64 2/4 2024-09-13T12:52:09.548 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T12:52:09.972 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T12:52:09.973 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:09.973 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:52:09.973 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:09.973 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:09.973 INFO:teuthology.orchestra.run.smithi052.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T12:52:09.973 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T12:52:09.973 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:09.973 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:52:10.105 INFO:teuthology.orchestra.run.smithi167.stdout:(3/37): ceph-osd-19.3.0-4917.gb0201b8c.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-13T12:52:10.155 INFO:teuthology.orchestra.run.smithi167.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4917.gb020 5.7 MB/s | 291 kB 00:00 2024-09-13T12:52:10.247 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph 2024-09-13T12:52:10.280 INFO:teuthology.orchestra.run.smithi167.stdout:(5/37): ceph-mds-19.3.0-4917.gb0201b8c.el9.x86_ 2.1 MB/s | 2.4 MB 00:01 2024-09-13T12:52:10.364 INFO:teuthology.orchestra.run.smithi167.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-13T12:52:10.406 INFO:teuthology.orchestra.run.smithi167.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-13T12:52:10.448 INFO:teuthology.orchestra.run.smithi167.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-13T12:52:10.481 INFO:teuthology.orchestra.run.smithi167.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 6.1 MB/s | 1.2 MB 00:00 2024-09-13T12:52:10.507 INFO:teuthology.orchestra.run.smithi167.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-09-13T12:52:10.532 INFO:teuthology.orchestra.run.smithi167.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-13T12:52:10.674 INFO:teuthology.orchestra.run.smithi167.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-13T12:52:10.699 INFO:teuthology.orchestra.run.smithi167.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-13T12:52:10.724 INFO:teuthology.orchestra.run.smithi167.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 691 kB/s | 35 kB 00:00 2024-09-13T12:52:10.750 INFO:teuthology.orchestra.run.smithi167.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-13T12:52:10.775 INFO:teuthology.orchestra.run.smithi167.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-13T12:52:10.801 INFO:teuthology.orchestra.run.smithi167.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 825 kB/s | 42 kB 00:00 2024-09-13T12:52:10.830 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:41 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:10.867 INFO:teuthology.orchestra.run.smithi167.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-13T12:52:10.893 INFO:teuthology.orchestra.run.smithi167.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-13T12:52:10.968 INFO:teuthology.orchestra.run.smithi167.stdout:(20/37): ceph-mon-19.3.0-4917.gb0201b8c.el9.x86 3.0 MB/s | 4.8 MB 00:01 2024-09-13T12:52:10.971 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:52:10.974 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:10.974 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-13T12:52:10.974 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:10.974 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:52:10.974 INFO:teuthology.orchestra.run.smithi052.stdout: ceph x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 6.6 k 2024-09-13T12:52:10.975 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-13T12:52:10.975 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 2.4 M 2024-09-13T12:52:10.975 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 1.5 M 2024-09-13T12:52:10.975 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 291 k 2024-09-13T12:52:10.975 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 4.8 M 2024-09-13T12:52:10.975 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 17 M 2024-09-13T12:52:10.975 INFO:teuthology.orchestra.run.smithi052.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T12:52:10.975 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T12:52:10.975 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T12:52:10.975 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T12:52:10.975 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T12:52:10.975 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T12:52:10.976 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T12:52:10.976 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T12:52:10.976 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T12:52:10.976 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T12:52:10.976 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections 2024-09-13T12:52:10.976 INFO:teuthology.orchestra.run.smithi052.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T12:52:10.976 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T12:52:10.976 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T12:52:10.976 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T12:52:10.977 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T12:52:10.977 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T12:52:10.977 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T12:52:10.977 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T12:52:10.977 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T12:52:10.977 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T12:52:10.977 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T12:52:10.977 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T12:52:10.977 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T12:52:10.977 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T12:52:10.977 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T12:52:10.977 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T12:52:10.977 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T12:52:10.978 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T12:52:10.978 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T12:52:10.978 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T12:52:10.978 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T12:52:10.978 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:10.978 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:52:10.978 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:10.978 INFO:teuthology.orchestra.run.smithi052.stdout:Install 37 Packages 2024-09-13T12:52:10.978 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:10.979 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 31 M 2024-09-13T12:52:10.979 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 109 M 2024-09-13T12:52:10.979 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:52:10.994 INFO:teuthology.orchestra.run.smithi167.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 102 kB/s | 22 kB 00:00 2024-09-13T12:52:11.027 INFO:teuthology.orchestra.run.smithi167.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-13T12:52:11.069 INFO:teuthology.orchestra.run.smithi167.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 556 kB/s | 18 kB 00:00 2024-09-13T12:52:11.103 INFO:teuthology.orchestra.run.smithi167.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-09-13T12:52:11.128 INFO:teuthology.orchestra.run.smithi167.stdout:(25/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2024-09-13T12:52:11.153 INFO:teuthology.orchestra.run.smithi167.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 282 kB/s | 23 kB 00:00 2024-09-13T12:52:11.179 INFO:teuthology.orchestra.run.smithi167.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-09-13T12:52:11.204 INFO:teuthology.orchestra.run.smithi167.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-09-13T12:52:11.238 INFO:teuthology.orchestra.run.smithi167.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 550 kB/s | 46 kB 00:00 2024-09-13T12:52:11.263 INFO:teuthology.orchestra.run.smithi167.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 938 kB/s | 79 kB 00:00 2024-09-13T12:52:11.288 INFO:teuthology.orchestra.run.smithi167.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-13T12:52:11.314 INFO:teuthology.orchestra.run.smithi167.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-13T12:52:11.339 INFO:teuthology.orchestra.run.smithi167.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-13T12:52:11.364 INFO:teuthology.orchestra.run.smithi167.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-13T12:52:11.389 INFO:teuthology.orchestra.run.smithi167.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 807 kB/s | 20 kB 00:00 2024-09-13T12:52:11.415 INFO:teuthology.orchestra.run.smithi167.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-13T12:52:11.440 INFO:teuthology.orchestra.run.smithi167.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-13T12:52:11.445 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:11.445 INFO:teuthology.orchestra.run.smithi167.stdout:Total 12 MB/s | 31 MB 00:02 2024-09-13T12:52:11.597 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:52:11.638 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:52:11.638 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:52:11.709 INFO:teuthology.orchestra.run.smithi052.stdout:(1/37): ceph-19.3.0-4917.gb0201b8c.el9.x86_64.r 47 kB/s | 6.6 kB 00:00 2024-09-13T12:52:11.918 INFO:teuthology.orchestra.run.smithi052.stdout:(2/37): ceph-mgr-19.3.0-4917.gb0201b8c.el9.x86_ 4.4 MB/s | 1.5 MB 00:00 2024-09-13T12:52:12.035 INFO:teuthology.orchestra.run.smithi052.stdout:(3/37): ceph-mds-19.3.0-4917.gb0201b8c.el9.x86_ 5.1 MB/s | 2.4 MB 00:00 2024-09-13T12:52:12.085 INFO:teuthology.orchestra.run.smithi052.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4917.gb020 5.7 MB/s | 291 kB 00:00 2024-09-13T12:52:12.094 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:52:12.095 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:52:12.310 INFO:teuthology.orchestra.run.smithi052.stdout:(5/37): ceph-mon-19.3.0-4917.gb0201b8c.el9.x86_ 8.0 MB/s | 4.8 MB 00:00 2024-09-13T12:52:12.377 INFO:teuthology.orchestra.run.smithi052.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 867 kB/s | 253 kB 00:00 2024-09-13T12:52:12.427 INFO:teuthology.orchestra.run.smithi052.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-13T12:52:12.478 INFO:teuthology.orchestra.run.smithi052.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-09-13T12:52:12.528 INFO:teuthology.orchestra.run.smithi052.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-13T12:52:12.603 INFO:teuthology.orchestra.run.smithi052.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.8 MB/s | 215 kB 00:00 2024-09-13T12:52:12.695 INFO:teuthology.orchestra.run.smithi052.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 3.2 MB/s | 1.2 MB 00:00 2024-09-13T12:52:12.720 INFO:teuthology.orchestra.run.smithi052.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-09-13T12:52:12.754 INFO:teuthology.orchestra.run.smithi052.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 1.0 MB/s | 35 kB 00:00 2024-09-13T12:52:12.796 INFO:teuthology.orchestra.run.smithi052.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-13T12:52:12.821 INFO:teuthology.orchestra.run.smithi052.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-13T12:52:12.838 INFO:teuthology.orchestra.run.smithi052.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-13T12:52:12.855 INFO:teuthology.orchestra.run.smithi052.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-09-13T12:52:12.872 INFO:teuthology.orchestra.run.smithi052.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 665 kB/s | 22 kB 00:00 2024-09-13T12:52:12.933 INFO:teuthology.orchestra.run.smithi052.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-13T12:52:13.106 INFO:teuthology.orchestra.run.smithi052.stdout:(20/37): ceph-osd-19.3.0-4917.gb0201b8c.el9.x86 14 MB/s | 17 MB 00:01 2024-09-13T12:52:13.123 INFO:teuthology.orchestra.run.smithi052.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 173 kB/s | 43 kB 00:00 2024-09-13T12:52:13.194 INFO:teuthology.orchestra.run.smithi052.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 658 kB/s | 172 kB 00:00 2024-09-13T12:52:13.265 INFO:teuthology.orchestra.run.smithi052.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 80 kB/s | 11 kB 00:00 2024-09-13T12:52:13.290 INFO:teuthology.orchestra.run.smithi052.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 188 kB/s | 18 kB 00:00 2024-09-13T12:52:13.316 INFO:teuthology.orchestra.run.smithi052.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-09-13T12:52:13.341 INFO:teuthology.orchestra.run.smithi052.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-13T12:52:13.366 INFO:teuthology.orchestra.run.smithi052.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-13T12:52:13.391 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:52:13.391 INFO:teuthology.orchestra.run.smithi052.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-09-13T12:52:13.425 INFO:teuthology.orchestra.run.smithi052.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-09-13T12:52:13.450 INFO:teuthology.orchestra.run.smithi052.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 664 kB/s | 16 kB 00:00 2024-09-13T12:52:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T12:52:13.475 INFO:teuthology.orchestra.run.smithi052.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 972 kB/s | 358 kB 00:00 2024-09-13T12:52:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T12:52:13.500 INFO:teuthology.orchestra.run.smithi052.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-13T12:52:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T12:52:13.526 INFO:teuthology.orchestra.run.smithi052.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-13T12:52:13.551 INFO:teuthology.orchestra.run.smithi052.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-13T12:52:13.577 INFO:teuthology.orchestra.run.smithi052.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 796 kB/s | 20 kB 00:00 2024-09-13T12:52:13.591 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T12:52:13.602 INFO:teuthology.orchestra.run.smithi052.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-13T12:52:13.627 INFO:teuthology.orchestra.run.smithi052.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-13T12:52:13.632 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:13.632 INFO:teuthology.orchestra.run.smithi052.stdout:Total 12 MB/s | 31 MB 00:02 2024-09-13T12:52:13.636 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T12:52:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T12:52:13.779 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T12:52:13.786 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:52:13.831 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:52:13.831 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:52:13.848 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T12:52:13.897 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T12:52:13.932 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T12:52:13.963 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T12:52:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T12:52:14.030 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T12:52:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T12:52:14.114 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T12:52:14.148 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T12:52:14.197 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T12:52:14.237 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T12:52:14.310 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T12:52:14.315 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:52:14.315 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:52:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T12:52:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T12:52:14.582 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T12:52:14.620 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T12:52:14.670 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T12:52:14.726 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T12:52:14.925 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T12:52:14.976 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T12:52:15.054 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T12:52:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T12:52:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T12:52:15.320 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T12:52:15.490 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4917.gb0201b8c.el9. 32/37 2024-09-13T12:52:15.527 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-2:19.3.0-4917.gb0201b8c.el9.x86_64 33/37 2024-09-13T12:52:15.552 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4917.gb0201b8c.el9.x86_64 33/37 2024-09-13T12:52:15.553 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:52:15.553 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T12:52:15.553 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T12:52:15.553 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T12:52:15.553 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:15.622 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:52:15.688 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T12:52:15.732 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T12:52:15.765 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T12:52:15.840 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T12:52:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T12:52:15.922 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T12:52:16.018 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T12:52:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T12:52:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T12:52:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T12:52:16.214 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T12:52:16.273 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T12:52:16.307 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T12:52:16.357 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T12:52:16.399 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T12:52:16.434 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T12:52:16.483 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T12:52:16.525 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T12:52:16.599 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T12:52:16.751 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T12:52:16.833 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T12:52:16.884 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T12:52:16.930 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T12:52:16.989 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T12:52:17.011 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-osd-2:19.3.0-4917.gb0201b8c.el9.x86_64 34/37 2024-09-13T12:52:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-osd-2:19.3.0-4917.gb0201b8c.el9.x86_64 34/37 2024-09-13T12:52:17.039 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:52:17.039 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T12:52:17.039 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T12:52:17.039 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T12:52:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T12:52:17.274 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T12:52:17.319 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T12:52:17.399 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T12:52:17.487 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mon-2:19.3.0-4917.gb0201b8c.el9.x86_64 35/37 2024-09-13T12:52:17.497 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T12:52:17.511 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mon-2:19.3.0-4917.gb0201b8c.el9.x86_64 35/37 2024-09-13T12:52:17.511 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:52:17.511 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T12:52:17.511 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T12:52:17.511 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T12:52:17.512 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:17.534 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T12:52:17.670 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T12:52:17.730 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mds-2:19.3.0-4917.gb0201b8c.el9.x86_64 36/37 2024-09-13T12:52:17.754 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mds-2:19.3.0-4917.gb0201b8c.el9.x86_64 36/37 2024-09-13T12:52:17.754 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:52:17.754 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T12:52:17.754 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T12:52:17.754 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T12:52:17.754 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:17.818 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-2:19.3.0-4917.gb0201b8c.el9.x86_64 37/37 2024-09-13T12:52:17.840 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4917.gb0201b8c.el9. 32/37 2024-09-13T12:52:17.869 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-2:19.3.0-4917.gb0201b8c.el9.x86_64 33/37 2024-09-13T12:52:17.897 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4917.gb0201b8c.el9.x86_64 33/37 2024-09-13T12:52:17.897 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:52:17.897 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T12:52:17.897 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T12:52:17.897 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T12:52:17.898 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:19.462 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-2:19.3.0-4917.gb0201b8c.el9.x86_64 34/37 2024-09-13T12:52:19.490 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-2:19.3.0-4917.gb0201b8c.el9.x86_64 34/37 2024-09-13T12:52:19.491 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:52:19.491 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T12:52:19.491 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T12:52:19.491 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T12:52:19.491 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:19.580 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-2:19.3.0-4917.gb0201b8c.el9.x86_64 37/37 2024-09-13T12:52:19.580 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/37 2024-09-13T12:52:19.581 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mds-2:19.3.0-4917.gb0201b8c.el9.x86_64 2/37 2024-09-13T12:52:19.581 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-2:19.3.0-4917.gb0201b8c.el9.x86_64 3/37 2024-09-13T12:52:19.581 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mon-2:19.3.0-4917.gb0201b8c.el9.x86_64 4/37 2024-09-13T12:52:19.581 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-osd-2:19.3.0-4917.gb0201b8c.el9.x86_64 5/37 2024-09-13T12:52:19.581 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4917.gb0201b8c.el9. 6/37 2024-09-13T12:52:19.581 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T12:52:19.581 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T12:52:19.581 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T12:52:19.581 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T12:52:19.581 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T12:52:19.582 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T12:52:19.582 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T12:52:19.582 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T12:52:19.582 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T12:52:19.582 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T12:52:19.582 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T12:52:19.582 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T12:52:19.582 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T12:52:19.582 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T12:52:19.582 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T12:52:19.583 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T12:52:19.583 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T12:52:19.583 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T12:52:19.583 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T12:52:19.583 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T12:52:19.583 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T12:52:19.583 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T12:52:19.584 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T12:52:19.584 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T12:52:19.584 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T12:52:19.584 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T12:52:19.584 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T12:52:19.584 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T12:52:19.584 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T12:52:19.584 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T12:52:19.963 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-2:19.3.0-4917.gb0201b8c.el9.x86_64 35/37 2024-09-13T12:52:19.987 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-2:19.3.0-4917.gb0201b8c.el9.x86_64 35/37 2024-09-13T12:52:19.988 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:52:19.988 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T12:52:19.988 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T12:52:19.988 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T12:52:19.988 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:20.230 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-2:19.3.0-4917.gb0201b8c.el9.x86_64 36/37 2024-09-13T12:52:20.255 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-2:19.3.0-4917.gb0201b8c.el9.x86_64 36/37 2024-09-13T12:52:20.255 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:52:20.255 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T12:52:20.255 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T12:52:20.255 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T12:52:20.255 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-2:19.3.0-4917.gb0201b8c.el9.x86_64 37/37 2024-09-13T12:52:20.717 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T12:52:20.717 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:20.717 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:52:20.717 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:20.717 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:20.717 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:20.717 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:20.717 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:20.717 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:20.717 INFO:teuthology.orchestra.run.smithi167.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T12:52:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T12:52:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T12:52:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T12:52:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T12:52:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T12:52:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T12:52:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T12:52:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T12:52:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T12:52:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T12:52:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T12:52:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T12:52:20.718 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T12:52:20.719 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T12:52:20.719 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T12:52:20.719 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T12:52:20.719 INFO:teuthology.orchestra.run.smithi167.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T12:52:20.719 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T12:52:20.719 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T12:52:20.719 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T12:52:20.719 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T12:52:20.719 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T12:52:20.719 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T12:52:20.720 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T12:52:20.720 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T12:52:20.720 INFO:teuthology.orchestra.run.smithi167.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T12:52:20.720 INFO:teuthology.orchestra.run.smithi167.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T12:52:20.720 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T12:52:20.720 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T12:52:20.720 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T12:52:20.720 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:20.720 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:52:21.104 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-base 2024-09-13T12:52:21.671 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:52 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:21.761 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-base-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:52:21.803 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:52:21.804 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-09-13T12:52:21.804 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:52:21.863 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install cephadm 2024-09-13T12:52:22.002 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-2:19.3.0-4917.gb0201b8c.el9.x86_64 37/37 2024-09-13T12:52:22.002 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/37 2024-09-13T12:52:22.002 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-2:19.3.0-4917.gb0201b8c.el9.x86_64 2/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-2:19.3.0-4917.gb0201b8c.el9.x86_64 3/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-2:19.3.0-4917.gb0201b8c.el9.x86_64 4/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-2:19.3.0-4917.gb0201b8c.el9.x86_64 5/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4917.gb0201b8c.el9. 6/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T12:52:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T12:52:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T12:52:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T12:52:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T12:52:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T12:52:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T12:52:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T12:52:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T12:52:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T12:52:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T12:52:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T12:52:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T12:52:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T12:52:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T12:52:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T12:52:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T12:52:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T12:52:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T12:52:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T12:52:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T12:52:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T12:52:22.428 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:22.560 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:52:22.560 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:22.560 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-09-13T12:52:22.560 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:22.561 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:52:22.561 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 784 k 2024-09-13T12:52:22.561 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:22.561 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:52:22.561 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:22.561 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-09-13T12:52:22.561 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:22.561 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 784 k 2024-09-13T12:52:22.561 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 791 k 2024-09-13T12:52:22.561 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:52:22.830 INFO:teuthology.orchestra.run.smithi167.stdout:cephadm-19.3.0-4917.gb0201b8c.el9.noarch.rpm 2.9 MB/s | 784 kB 00:00 2024-09-13T12:52:22.831 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:22.832 INFO:teuthology.orchestra.run.smithi167.stdout:Total 2.8 MB/s | 784 kB 00:00 2024-09-13T12:52:22.832 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:52:22.836 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:52:22.837 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:52:22.843 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:52:22.843 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:52:22.919 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:52:22.984 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T12:52:22.984 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:22.984 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:52:22.984 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:22.984 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:22.984 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:22.984 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:22.984 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:22.985 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:22.985 INFO:teuthology.orchestra.run.smithi052.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T12:52:22.985 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T12:52:22.985 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T12:52:22.985 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T12:52:22.985 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T12:52:22.985 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T12:52:22.985 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T12:52:22.985 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T12:52:22.985 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T12:52:22.985 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T12:52:22.986 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T12:52:22.986 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T12:52:22.986 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T12:52:22.986 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T12:52:22.986 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T12:52:22.986 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T12:52:22.986 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T12:52:22.986 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T12:52:22.986 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T12:52:22.986 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T12:52:22.986 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T12:52:22.987 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T12:52:22.987 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T12:52:22.987 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T12:52:22.987 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T12:52:22.987 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T12:52:22.987 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T12:52:22.987 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T12:52:22.987 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T12:52:22.987 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T12:52:22.987 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T12:52:22.987 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:22.987 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:52:23.373 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base 2024-09-13T12:52:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:52:23.484 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:52:23.885 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:52:23.963 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:24.055 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:52:24.100 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:52:24.101 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-13T12:52:24.101 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:52:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:52:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:24.137 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:52:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:24.137 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:52:24.163 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install cephadm 2024-09-13T12:52:24.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T12:52:24.750 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:24.889 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:52:24.890 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:24.890 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-13T12:52:24.890 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:24.890 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:52:24.890 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 784 k 2024-09-13T12:52:24.890 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:24.890 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:52:24.890 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:24.890 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-13T12:52:24.890 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:24.890 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 784 k 2024-09-13T12:52:24.890 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 791 k 2024-09-13T12:52:24.890 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:52:24.923 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:25.060 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:52:25.061 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:25.061 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-09-13T12:52:25.061 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:25.061 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:52:25.061 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 145 k 2024-09-13T12:52:25.061 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:25.061 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:52:25.061 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:25.062 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-09-13T12:52:25.062 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:25.062 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 145 k 2024-09-13T12:52:25.062 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 432 k 2024-09-13T12:52:25.062 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:52:25.179 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm-19.3.0-4917.gb0201b8c.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-13T12:52:25.179 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:25.180 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.6 MB/s | 784 kB 00:00 2024-09-13T12:52:25.180 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:52:25.185 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:52:25.185 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:52:25.192 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:52:25.192 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:52:25.254 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-immutable-object-cache-19.3.0-4917.gb0201b 755 kB/s | 145 kB 00:00 2024-09-13T12:52:25.255 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:25.255 INFO:teuthology.orchestra.run.smithi167.stdout:Total 747 kB/s | 145 kB 00:00 2024-09-13T12:52:25.255 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:52:25.263 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:52:25.263 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:52:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:52:25.321 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:52:25.321 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:52:25.464 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:52:25.522 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4917.gb0201b8c. 1/1 2024-09-13T12:52:25.547 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4917.gb0201b8c. 1/1 2024-09-13T12:52:25.547 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:52:25.547 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T12:52:25.547 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:25.760 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:52:25.851 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:52:26.201 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:52:26.468 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:52:26.468 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:26.468 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:52:26.469 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:26.469 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:26.469 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:52:26.496 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4917.gb0201b8c. 1/1 2024-09-13T12:52:26.496 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:26.496 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:52:26.496 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:26.496 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:26.496 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:52:26.694 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T12:52:26.715 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr 2024-09-13T12:52:27.286 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:27.286 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:27.381 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-mgr-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:52:27.421 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:52:27.422 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-09-13T12:52:27.422 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:52:27.430 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:52:27.430 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:27.431 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-09-13T12:52:27.431 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:27.431 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:52:27.431 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 145 k 2024-09-13T12:52:27.431 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:27.431 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:52:27.431 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:27.431 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-13T12:52:27.431 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:27.432 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 145 k 2024-09-13T12:52:27.432 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 432 k 2024-09-13T12:52:27.432 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:52:27.487 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T12:52:27.650 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-19.3.0-4917.gb0201b 667 kB/s | 145 kB 00:00 2024-09-13T12:52:27.650 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:27.650 INFO:teuthology.orchestra.run.smithi052.stdout:Total 662 kB/s | 145 kB 00:00 2024-09-13T12:52:27.651 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:52:27.659 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:52:27.659 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:52:27.721 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:52:27.721 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:52:27.878 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:52:27.942 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4917.gb0201b8c. 1/1 2024-09-13T12:52:27.969 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4917.gb0201b8c. 1/1 2024-09-13T12:52:27.969 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:52:27.969 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T12:52:27.969 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:28.050 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:28.184 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:52:28.185 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:28.185 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-09-13T12:52:28.185 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:28.185 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:52:28.185 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 14 M 2024-09-13T12:52:28.185 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-09-13T12:52:28.185 INFO:teuthology.orchestra.run.smithi167.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T12:52:28.185 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 34 k 2024-09-13T12:52:28.185 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 16 k 2024-09-13T12:52:28.185 INFO:teuthology.orchestra.run.smithi167.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T12:52:28.185 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T12:52:28.186 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T12:52:28.186 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T12:52:28.186 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T12:52:28.186 INFO:teuthology.orchestra.run.smithi167.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T12:52:28.186 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T12:52:28.186 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T12:52:28.186 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:28.186 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:52:28.186 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:28.186 INFO:teuthology.orchestra.run.smithi167.stdout:Install 12 Packages 2024-09-13T12:52:28.186 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:28.187 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 19 M 2024-09-13T12:52:28.187 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 159 M 2024-09-13T12:52:28.187 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:52:28.890 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4917.gb0201b8c. 1/1 2024-09-13T12:52:28.890 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:28.890 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:52:28.890 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:52:28.890 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:28.890 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:52:28.925 INFO:teuthology.orchestra.run.smithi167.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4917.gb0 249 kB/s | 34 kB 00:00 2024-09-13T12:52:28.951 INFO:teuthology.orchestra.run.smithi167.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4917.gb02 101 kB/s | 16 kB 00:00 2024-09-13T12:52:29.085 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr 2024-09-13T12:52:29.451 INFO:teuthology.orchestra.run.smithi167.stdout:(3/12): ceph-mgr-dashboard-19.3.0-4917.gb0201b8 21 MB/s | 14 MB 00:00 2024-09-13T12:52:29.626 INFO:teuthology.orchestra.run.smithi167.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-09-13T12:52:29.680 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:00 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:29.693 INFO:teuthology.orchestra.run.smithi167.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.0 MB/s | 548 kB 00:00 2024-09-13T12:52:29.718 INFO:teuthology.orchestra.run.smithi167.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 786 kB/s | 19 kB 00:00 2024-09-13T12:52:29.772 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-mgr-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:52:29.817 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:52:29.818 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-13T12:52:29.818 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:52:29.827 INFO:teuthology.orchestra.run.smithi167.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 19 MB/s | 2.0 MB 00:00 2024-09-13T12:52:29.852 INFO:teuthology.orchestra.run.smithi167.stdout:(8/12): python3-protobuf-3.14.0-13.el9.noarch.r 299 kB/s | 269 kB 00:00 2024-09-13T12:52:29.886 INFO:teuthology.orchestra.run.smithi167.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-09-13T12:52:29.891 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T12:52:29.920 INFO:teuthology.orchestra.run.smithi167.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 461 kB/s | 31 kB 00:00 2024-09-13T12:52:29.945 INFO:teuthology.orchestra.run.smithi167.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.3 MB/s | 188 kB 00:00 2024-09-13T12:52:30.045 INFO:teuthology.orchestra.run.smithi167.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 923 kB/s | 1.0 MB 00:01 2024-09-13T12:52:30.049 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:30.050 INFO:teuthology.orchestra.run.smithi167.stdout:Total 10 MB/s | 19 MB 00:01 2024-09-13T12:52:30.099 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:52:30.119 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:52:30.119 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:52:30.399 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:52:30.400 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:52:30.476 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:01 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:30.615 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:52:30.616 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:30.616 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-13T12:52:30.616 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:30.616 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:52:30.616 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 14 M 2024-09-13T12:52:30.617 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-13T12:52:30.617 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T12:52:30.617 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 34 k 2024-09-13T12:52:30.617 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 16 k 2024-09-13T12:52:30.617 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T12:52:30.617 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T12:52:30.617 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T12:52:30.617 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T12:52:30.617 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T12:52:30.617 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T12:52:30.617 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T12:52:30.617 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T12:52:30.617 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:30.618 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:52:30.618 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:30.618 INFO:teuthology.orchestra.run.smithi052.stdout:Install 12 Packages 2024-09-13T12:52:30.618 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:30.618 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 19 M 2024-09-13T12:52:30.618 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 159 M 2024-09-13T12:52:30.619 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:52:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:52:31.003 INFO:teuthology.orchestra.run.smithi052.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4917.gb02 103 kB/s | 16 kB 00:00 2024-09-13T12:52:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4917.gb0 194 kB/s | 34 kB 00:00 2024-09-13T12:52:31.063 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T12:52:31.098 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T12:52:31.144 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T12:52:31.183 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T12:52:31.186 INFO:teuthology.orchestra.run.smithi052.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 269 kB 00:00 2024-09-13T12:52:31.237 INFO:teuthology.orchestra.run.smithi052.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-09-13T12:52:31.284 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T12:52:31.295 INFO:teuthology.orchestra.run.smithi052.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-13T12:52:31.321 INFO:teuthology.orchestra.run.smithi052.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 775 kB/s | 19 kB 00:00 2024-09-13T12:52:31.409 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T12:52:31.471 INFO:teuthology.orchestra.run.smithi052.stdout:(7/12): ceph-mgr-dashboard-19.3.0-4917.gb0201b8 23 MB/s | 14 MB 00:00 2024-09-13T12:52:31.503 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T12:52:31.555 INFO:teuthology.orchestra.run.smithi052.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 8.7 MB/s | 2.0 MB 00:00 2024-09-13T12:52:31.572 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T12:52:31.580 INFO:teuthology.orchestra.run.smithi052.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-09-13T12:52:31.601 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T12:52:31.605 INFO:teuthology.orchestra.run.smithi052.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 614 kB/s | 31 kB 00:00 2024-09-13T12:52:31.633 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4917.gb0201b8c.el9 10/12 2024-09-13T12:52:31.638 INFO:teuthology.orchestra.run.smithi052.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.9 MB/s | 863 kB 00:00 2024-09-13T12:52:31.664 INFO:teuthology.orchestra.run.smithi052.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-09-13T12:52:31.667 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:31.668 INFO:teuthology.orchestra.run.smithi052.stdout:Total 18 MB/s | 19 MB 00:01 2024-09-13T12:52:31.717 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:52:31.737 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:52:31.737 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:52:32.020 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:52:32.020 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:52:32.640 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:52:32.707 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T12:52:32.751 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T12:52:32.798 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T12:52:32.835 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T12:52:32.963 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T12:52:33.092 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T12:52:33.193 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T12:52:33.240 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T12:52:33.268 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T12:52:33.301 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4917.gb0201b8c.el9 10/12 2024-09-13T12:52:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4917.gb0201b8c.el 11/12 2024-09-13T12:52:34.888 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4917.gb0201b8c.el9.noa 12/12 2024-09-13T12:52:36.694 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4917.gb0201b8c.el 11/12 2024-09-13T12:52:36.746 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4917.gb0201b8c.el9.noa 12/12 2024-09-13T12:52:36.913 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4917.gb0201b8c.el9.noa 12/12 2024-09-13T12:52:36.913 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4917.gb0201b8c.el 1/12 2024-09-13T12:52:36.913 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4917.gb0201b8c.el9.noa 2/12 2024-09-13T12:52:36.914 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4917.gb0201b8c.el9 3/12 2024-09-13T12:52:36.914 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T12:52:36.914 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T12:52:36.914 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T12:52:36.914 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T12:52:36.914 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T12:52:36.914 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T12:52:36.914 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T12:52:36.914 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T12:52:37.261 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T12:52:37.261 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:37.261 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:52:37.261 INFO:teuthology.orchestra.run.smithi167.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T12:52:37.261 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:37.262 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:37.262 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:37.262 INFO:teuthology.orchestra.run.smithi167.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T12:52:37.262 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T12:52:37.262 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T12:52:37.262 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T12:52:37.262 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T12:52:37.262 INFO:teuthology.orchestra.run.smithi167.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T12:52:37.262 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T12:52:37.262 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T12:52:37.262 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:37.262 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:52:37.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T12:52:38.062 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:09 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:38.197 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:52:38.198 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================= 2024-09-13T12:52:38.198 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-09-13T12:52:38.198 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================= 2024-09-13T12:52:38.198 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:52:38.198 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 7.4 M 2024-09-13T12:52:38.199 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-09-13T12:52:38.199 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T12:52:38.199 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T12:52:38.199 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T12:52:38.199 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T12:52:38.199 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T12:52:38.199 INFO:teuthology.orchestra.run.smithi167.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T12:52:38.199 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T12:52:38.199 INFO:teuthology.orchestra.run.smithi167.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T12:52:38.199 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T12:52:38.199 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T12:52:38.199 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T12:52:38.199 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:38.200 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:52:38.200 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================= 2024-09-13T12:52:38.200 INFO:teuthology.orchestra.run.smithi167.stdout:Install 12 Packages 2024-09-13T12:52:38.200 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:38.200 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 38 M 2024-09-13T12:52:38.200 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 212 M 2024-09-13T12:52:38.201 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:52:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4917.gb0201b8c.el9.noa 12/12 2024-09-13T12:52:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4917.gb0201b8c.el 1/12 2024-09-13T12:52:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4917.gb0201b8c.el9.noa 2/12 2024-09-13T12:52:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4917.gb0201b8c.el9 3/12 2024-09-13T12:52:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T12:52:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T12:52:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T12:52:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T12:52:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T12:52:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T12:52:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T12:52:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T12:52:38.705 INFO:teuthology.orchestra.run.smithi167.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-13T12:52:38.763 INFO:teuthology.orchestra.run.smithi167.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 562 kB/s | 33 kB 00:00 2024-09-13T12:52:38.797 INFO:teuthology.orchestra.run.smithi167.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.9 MB/s | 801 kB 00:00 2024-09-13T12:52:38.864 INFO:teuthology.orchestra.run.smithi167.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 273 kB/s | 18 kB 00:00 2024-09-13T12:52:38.910 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T12:52:38.910 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:38.910 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:52:38.910 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T12:52:38.910 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:38.910 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:38.910 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:38.910 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T12:52:38.910 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T12:52:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T12:52:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T12:52:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T12:52:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T12:52:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T12:52:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T12:52:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:38.911 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:52:38.989 INFO:teuthology.orchestra.run.smithi167.stdout:(5/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-13T12:52:39.006 INFO:teuthology.orchestra.run.smithi167.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 271 kB/s | 38 kB 00:00 2024-09-13T12:52:39.132 INFO:teuthology.orchestra.run.smithi167.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-13T12:52:39.162 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T12:52:39.448 INFO:teuthology.orchestra.run.smithi167.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2024-09-13T12:52:39.498 INFO:teuthology.orchestra.run.smithi167.stdout:(9/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2024-09-13T12:52:39.524 INFO:teuthology.orchestra.run.smithi167.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 3.2 MB/s | 250 kB 00:00 2024-09-13T12:52:39.748 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:10 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:39.888 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:52:39.889 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2024-09-13T12:52:39.889 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-13T12:52:39.889 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2024-09-13T12:52:39.890 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:52:39.890 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 7.4 M 2024-09-13T12:52:39.890 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-13T12:52:39.890 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T12:52:39.890 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T12:52:39.890 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T12:52:39.890 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T12:52:39.890 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T12:52:39.890 INFO:teuthology.orchestra.run.smithi052.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T12:52:39.890 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T12:52:39.890 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T12:52:39.890 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T12:52:39.891 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T12:52:39.891 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T12:52:39.891 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:39.891 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:52:39.891 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2024-09-13T12:52:39.891 INFO:teuthology.orchestra.run.smithi052.stdout:Install 12 Packages 2024-09-13T12:52:39.891 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:39.892 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 38 M 2024-09-13T12:52:39.893 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 212 M 2024-09-13T12:52:39.893 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:52:40.102 INFO:teuthology.orchestra.run.smithi052.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-13T12:52:40.136 INFO:teuthology.orchestra.run.smithi052.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.9 MB/s | 801 kB 00:00 2024-09-13T12:52:40.332 INFO:teuthology.orchestra.run.smithi167.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.2 MB/s | 5.1 MB 00:01 2024-09-13T12:52:40.411 INFO:teuthology.orchestra.run.smithi052.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-13T12:52:40.761 INFO:teuthology.orchestra.run.smithi052.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 50 kB/s | 33 kB 00:00 2024-09-13T12:52:40.937 INFO:teuthology.orchestra.run.smithi052.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 36 kB/s | 18 kB 00:00 2024-09-13T12:52:41.237 INFO:teuthology.orchestra.run.smithi052.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2024-09-13T12:52:41.371 INFO:teuthology.orchestra.run.smithi052.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 63 kB/s | 38 kB 00:00 2024-09-13T12:52:41.708 INFO:teuthology.orchestra.run.smithi167.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.4 MB/s | 16 MB 00:02 2024-09-13T12:52:41.709 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:41.709 INFO:teuthology.orchestra.run.smithi167.stdout:Total 11 MB/s | 38 MB 00:03 2024-09-13T12:52:42.004 INFO:teuthology.orchestra.run.smithi052.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 320 kB/s | 245 kB 00:00 2024-09-13T12:52:42.066 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:52:42.071 INFO:teuthology.orchestra.run.smithi052.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.4 MB/s | 4.9 MB 00:01 2024-09-13T12:52:42.093 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:52:42.093 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:52:42.534 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:52:42.534 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:52:42.621 INFO:teuthology.orchestra.run.smithi052.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.1 MB/s | 5.1 MB 00:01 2024-09-13T12:52:42.888 INFO:teuthology.orchestra.run.smithi052.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 284 kB/s | 250 kB 00:00 2024-09-13T12:52:43.304 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:52:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T12:52:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T12:52:43.516 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T12:52:43.557 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T12:52:43.680 INFO:teuthology.orchestra.run.smithi052.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-13T12:52:43.681 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:43.681 INFO:teuthology.orchestra.run.smithi052.stdout:Total 10 MB/s | 38 MB 00:03 2024-09-13T12:52:43.742 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T12:52:43.773 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T12:52:43.850 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T12:52:44.055 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:52:44.083 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:52:44.083 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:52:44.512 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T12:52:44.546 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:52:44.547 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:52:44.629 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T12:52:45.339 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:52:45.397 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T12:52:45.437 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T12:52:45.541 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T12:52:45.588 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T12:52:45.780 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T12:52:45.811 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T12:52:45.890 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T12:52:45.915 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T12:52:46.595 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T12:52:46.713 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T12:52:46.982 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T12:52:47.007 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4917.gb0201 12/12 2024-09-13T12:52:48.050 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T12:52:49.132 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T12:52:49.159 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4917.gb0201 12/12 2024-09-13T12:52:49.631 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4917.gb0201 12/12 2024-09-13T12:52:49.631 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4917.gb0201 1/12 2024-09-13T12:52:49.631 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T12:52:49.632 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T12:52:49.632 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T12:52:49.632 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T12:52:49.632 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T12:52:49.632 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T12:52:49.632 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T12:52:49.632 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T12:52:49.632 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T12:52:49.632 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T12:52:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T12:52:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:50.070 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:52:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T12:52:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T12:52:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T12:52:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T12:52:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T12:52:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T12:52:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T12:52:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T12:52:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T12:52:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T12:52:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T12:52:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:50.071 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:52:50.495 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-rook 2024-09-13T12:52:51.077 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:22 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:51.213 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:52:51.215 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-09-13T12:52:51.215 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:51.215 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:52:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 49 k 2024-09-13T12:52:51.215 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-09-13T12:52:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T12:52:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T12:52:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T12:52:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T12:52:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T12:52:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T12:52:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T12:52:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T12:52:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T12:52:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T12:52:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T12:52:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T12:52:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:51.216 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:52:51.216 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:51.216 INFO:teuthology.orchestra.run.smithi167.stdout:Install 13 Packages 2024-09-13T12:52:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:51.217 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 2.2 M 2024-09-13T12:52:51.217 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 26 M 2024-09-13T12:52:51.217 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:52:51.710 INFO:teuthology.orchestra.run.smithi167.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 183 kB/s | 19 kB 00:00 2024-09-13T12:52:51.711 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4917.gb0201 12/12 2024-09-13T12:52:51.711 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4917.gb0201 1/12 2024-09-13T12:52:51.711 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T12:52:51.711 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T12:52:51.711 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T12:52:51.711 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T12:52:51.712 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T12:52:51.712 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T12:52:51.712 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T12:52:51.712 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T12:52:51.712 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T12:52:51.712 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T12:52:51.735 INFO:teuthology.orchestra.run.smithi167.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 197 kB/s | 26 kB 00:00 2024-09-13T12:52:51.760 INFO:teuthology.orchestra.run.smithi167.stdout:(3/13): ceph-mgr-rook-19.3.0-4917.gb0201b8c.el9 317 kB/s | 49 kB 00:00 2024-09-13T12:52:51.852 INFO:teuthology.orchestra.run.smithi167.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-13T12:52:51.877 INFO:teuthology.orchestra.run.smithi167.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-13T12:52:51.902 INFO:teuthology.orchestra.run.smithi167.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-13T12:52:51.945 INFO:teuthology.orchestra.run.smithi167.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-13T12:52:51.987 INFO:teuthology.orchestra.run.smithi167.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 168 kB/s | 14 kB 00:00 2024-09-13T12:52:52.012 INFO:teuthology.orchestra.run.smithi167.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 239 kB/s | 32 kB 00:00 2024-09-13T12:52:52.046 INFO:teuthology.orchestra.run.smithi167.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-13T12:52:52.071 INFO:teuthology.orchestra.run.smithi167.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.8 MB/s | 220 kB 00:00 2024-09-13T12:52:52.096 INFO:teuthology.orchestra.run.smithi167.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-13T12:52:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T12:52:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:52.110 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:52:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T12:52:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T12:52:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T12:52:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T12:52:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T12:52:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T12:52:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T12:52:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T12:52:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T12:52:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T12:52:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T12:52:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:52.111 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:52:52.129 INFO:teuthology.orchestra.run.smithi167.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.3 MB/s | 1.0 MB 00:00 2024-09-13T12:52:52.132 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:52.132 INFO:teuthology.orchestra.run.smithi167.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-09-13T12:52:52.196 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:52:52.212 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:52:52.212 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:52:52.305 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:52:52.305 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:52:52.456 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-rook 2024-09-13T12:52:52.818 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:52:52.977 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T12:52:53.039 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T12:52:53.051 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:24 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:53.094 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T12:52:53.132 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T12:52:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T12:52:53.193 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:52:53.194 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-13T12:52:53.194 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:53.194 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:52:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 49 k 2024-09-13T12:52:53.194 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-13T12:52:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T12:52:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T12:52:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T12:52:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T12:52:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T12:52:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T12:52:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T12:52:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T12:52:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T12:52:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T12:52:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T12:52:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T12:52:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:53.195 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:52:53.196 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:53.196 INFO:teuthology.orchestra.run.smithi052.stdout:Install 13 Packages 2024-09-13T12:52:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:53.196 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.2 M 2024-09-13T12:52:53.196 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 26 M 2024-09-13T12:52:53.196 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:52:53.277 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T12:52:53.384 INFO:teuthology.orchestra.run.smithi052.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 307 kB/s | 26 kB 00:00 2024-09-13T12:52:53.385 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T12:52:53.409 INFO:teuthology.orchestra.run.smithi052.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 177 kB/s | 19 kB 00:00 2024-09-13T12:52:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T12:52:53.501 INFO:teuthology.orchestra.run.smithi052.stdout:(3/13): ceph-mgr-rook-19.3.0-4917.gb0201b8c.el9 246 kB/s | 49 kB 00:00 2024-09-13T12:52:53.526 INFO:teuthology.orchestra.run.smithi052.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-13T12:52:53.551 INFO:teuthology.orchestra.run.smithi052.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-13T12:52:53.589 INFO:teuthology.orchestra.run.smithi052.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 917 kB/s | 54 kB 00:00 2024-09-13T12:52:53.618 INFO:teuthology.orchestra.run.smithi052.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-13T12:52:53.644 INFO:teuthology.orchestra.run.smithi052.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 247 kB/s | 14 kB 00:00 2024-09-13T12:52:53.678 INFO:teuthology.orchestra.run.smithi052.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.8 MB/s | 220 kB 00:00 2024-09-13T12:52:53.703 INFO:teuthology.orchestra.run.smithi052.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-13T12:52:53.728 INFO:teuthology.orchestra.run.smithi052.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-13T12:52:53.753 INFO:teuthology.orchestra.run.smithi052.stdout:(12/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.1 MB/s | 279 kB 00:00 2024-09-13T12:52:53.795 INFO:teuthology.orchestra.run.smithi052.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-13T12:52:53.797 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:53.797 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.7 MB/s | 2.2 MB 00:00 2024-09-13T12:52:53.864 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:52:53.880 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:52:53.880 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:52:53.979 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:52:53.979 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:52:54.154 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T12:52:54.206 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T12:52:54.240 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T12:52:54.288 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T12:52:54.318 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-rook-2:19.3.0-4917.gb0201b8c.el9.noarch 13/13 2024-09-13T12:52:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:52:54.672 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T12:52:54.724 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T12:52:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T12:52:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T12:52:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T12:52:54.965 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T12:52:55.075 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T12:52:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4917.gb0201b8c.el9.noarch 13/13 2024-09-13T12:52:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4917.gb0201b8c.el9.noarch 1/13 2024-09-13T12:52:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T12:52:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T12:52:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T12:52:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T12:52:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T12:52:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T12:52:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T12:52:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T12:52:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T12:52:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T12:52:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T12:52:55.136 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T12:52:55.408 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T12:52:55.409 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:55.409 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:52:55.409 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:55.409 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T12:52:55.409 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T12:52:55.409 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T12:52:55.409 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T12:52:55.409 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T12:52:55.409 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T12:52:55.409 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T12:52:55.409 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T12:52:55.410 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T12:52:55.410 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T12:52:55.410 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T12:52:55.410 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T12:52:55.410 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:55.410 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:52:55.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T12:52:56.007 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T12:52:56.054 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T12:52:56.088 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T12:52:56.137 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T12:52:56.179 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-rook-2:19.3.0-4917.gb0201b8c.el9.noarch 13/13 2024-09-13T12:52:56.221 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:27 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:56.356 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:52:56.357 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:56.357 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-09-13T12:52:56.357 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:56.357 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:52:56.357 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 154 k 2024-09-13T12:52:56.357 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-09-13T12:52:56.357 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T12:52:56.358 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T12:52:56.358 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T12:52:56.358 INFO:teuthology.orchestra.run.smithi167.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T12:52:56.358 INFO:teuthology.orchestra.run.smithi167.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T12:52:56.358 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:56.358 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:52:56.358 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:52:56.359 INFO:teuthology.orchestra.run.smithi167.stdout:Install 6 Packages 2024-09-13T12:52:56.359 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:52:56.360 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 7.0 M 2024-09-13T12:52:56.360 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 33 M 2024-09-13T12:52:56.360 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:52:56.617 INFO:teuthology.orchestra.run.smithi167.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4917.gb0201b8c.e 727 kB/s | 154 kB 00:00 2024-09-13T12:52:56.642 INFO:teuthology.orchestra.run.smithi167.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-09-13T12:52:56.701 INFO:teuthology.orchestra.run.smithi167.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-13T12:52:56.726 INFO:teuthology.orchestra.run.smithi167.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 689 kB/s | 58 kB 00:00 2024-09-13T12:52:56.751 INFO:teuthology.orchestra.run.smithi167.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-13T12:52:56.808 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4917.gb0201b8c.el9.noarch 13/13 2024-09-13T12:52:56.808 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4917.gb0201b8c.el9.noarch 1/13 2024-09-13T12:52:56.808 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T12:52:56.809 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T12:52:56.809 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T12:52:56.809 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T12:52:56.809 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T12:52:56.809 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T12:52:56.809 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T12:52:56.809 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T12:52:56.809 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T12:52:56.809 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T12:52:56.809 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T12:52:57.149 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T12:52:57.149 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:57.149 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:52:57.149 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:52:57.149 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T12:52:57.150 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T12:52:57.150 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T12:52:57.150 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T12:52:57.150 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T12:52:57.150 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T12:52:57.150 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T12:52:57.150 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T12:52:57.150 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T12:52:57.150 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T12:52:57.150 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T12:52:57.150 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T12:52:57.150 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:57.151 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:52:57.387 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T12:52:57.976 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:28 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:52:58.116 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:52:58.116 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:58.116 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-13T12:52:58.117 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:58.117 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:52:58.117 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 154 k 2024-09-13T12:52:58.117 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-13T12:52:58.117 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T12:52:58.117 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T12:52:58.117 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T12:52:58.117 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T12:52:58.117 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T12:52:58.117 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:58.117 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:52:58.117 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:52:58.118 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-09-13T12:52:58.118 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:52:58.119 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 7.0 M 2024-09-13T12:52:58.119 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 33 M 2024-09-13T12:52:58.119 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:52:58.505 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 2.0 MB/s | 248 kB 00:00 2024-09-13T12:52:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-13T12:52:58.568 INFO:teuthology.orchestra.run.smithi167.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.8 MB/s | 6.0 MB 00:02 2024-09-13T12:52:58.570 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:58.570 INFO:teuthology.orchestra.run.smithi167.stdout:Total 3.2 MB/s | 7.0 MB 00:02 2024-09-13T12:52:58.587 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-13T12:52:58.613 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-13T12:52:58.638 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): ceph-mgr-cephadm-19.3.0-4917.gb0201b8c.e 607 kB/s | 154 kB 00:00 2024-09-13T12:52:58.646 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:52:58.657 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:52:58.657 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:52:58.757 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:52:58.757 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:52:59.068 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:52:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T12:52:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T12:52:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T12:52:59.906 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T12:52:59.922 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.9 MB/s | 6.0 MB 00:01 2024-09-13T12:52:59.924 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:52:59.924 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.9 MB/s | 7.0 MB 00:01 2024-09-13T12:52:59.974 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T12:52:59.999 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 6/6 2024-09-13T12:53:00.000 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:53:00.012 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:53:00.012 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:53:00.121 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:53:00.121 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:53:00.458 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:53:00.645 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T12:53:00.701 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T12:53:00.828 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 6/6 2024-09-13T12:53:00.829 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 1/6 2024-09-13T12:53:00.829 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T12:53:00.829 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T12:53:00.829 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T12:53:00.830 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T12:53:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T12:53:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:01.100 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:53:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:53:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T12:53:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T12:53:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T12:53:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T12:53:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T12:53:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:01.100 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:01.262 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T12:53:01.354 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T12:53:01.360 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-fuse 2024-09-13T12:53:01.430 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T12:53:01.460 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 6/6 2024-09-13T12:53:01.933 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:32 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:02.066 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:02.067 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:02.067 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-09-13T12:53:02.067 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:02.067 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:53:02.067 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 900 k 2024-09-13T12:53:02.067 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-09-13T12:53:02.067 INFO:teuthology.orchestra.run.smithi167.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T12:53:02.067 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:02.068 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:53:02.068 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:02.068 INFO:teuthology.orchestra.run.smithi167.stdout:Install 2 Packages 2024-09-13T12:53:02.068 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:02.068 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 979 k 2024-09-13T12:53:02.068 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 2.9 M 2024-09-13T12:53:02.068 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:53:02.214 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 6/6 2024-09-13T12:53:02.215 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 1/6 2024-09-13T12:53:02.215 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T12:53:02.215 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T12:53:02.215 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T12:53:02.215 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T12:53:02.437 INFO:teuthology.orchestra.run.smithi167.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 872 kB/s | 80 kB 00:00 2024-09-13T12:53:02.491 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T12:53:02.491 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:02.491 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:53:02.492 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:53:02.492 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T12:53:02.492 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T12:53:02.492 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T12:53:02.492 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T12:53:02.492 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T12:53:02.492 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:02.492 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:02.671 INFO:teuthology.orchestra.run.smithi167.stdout:(2/2): ceph-fuse-19.3.0-4917.gb0201b8c.el9.x86_ 2.7 MB/s | 900 kB 00:00 2024-09-13T12:53:02.671 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:02.672 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.6 MB/s | 979 kB 00:00 2024-09-13T12:53:02.684 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:53:02.706 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:53:02.706 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:53:02.744 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse 2024-09-13T12:53:02.787 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:53:02.788 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:53:02.982 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:53:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T12:53:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 2/2 2024-09-13T12:53:03.335 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:03.474 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:03.476 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:03.476 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-13T12:53:03.476 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:03.476 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:53:03.477 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 900 k 2024-09-13T12:53:03.477 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-13T12:53:03.477 INFO:teuthology.orchestra.run.smithi052.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T12:53:03.477 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:03.477 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:53:03.477 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:03.478 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2024-09-13T12:53:03.478 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:03.478 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 979 k 2024-09-13T12:53:03.478 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.9 M 2024-09-13T12:53:03.478 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:53:03.723 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 543 kB/s | 80 kB 00:00 2024-09-13T12:53:03.897 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 2/2 2024-09-13T12:53:03.898 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/2 2024-09-13T12:53:03.906 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-fuse-19.3.0-4917.gb0201b8c.el9.x86_ 2.7 MB/s | 900 kB 00:00 2024-09-13T12:53:03.907 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:03.907 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.2 MB/s | 979 kB 00:00 2024-09-13T12:53:03.920 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:53:03.943 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:53:03.943 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:53:04.030 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:53:04.030 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:53:04.159 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T12:53:04.159 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:04.159 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:53:04.159 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T12:53:04.159 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:04.159 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:04.225 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:53:04.372 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T12:53:04.428 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-volume 2024-09-13T12:53:04.442 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 2/2 2024-09-13T12:53:05.003 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 2/2 2024-09-13T12:53:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/2 2024-09-13T12:53:05.141 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:05.142 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:05.142 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-09-13T12:53:05.142 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:05.142 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:53:05.142 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-volume noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 285 k 2024-09-13T12:53:05.142 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:05.142 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:53:05.143 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:05.143 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-09-13T12:53:05.143 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:05.143 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 285 k 2024-09-13T12:53:05.143 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 1.3 M 2024-09-13T12:53:05.143 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:53:05.378 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-volume-19.3.0-4917.gb0201b8c.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-13T12:53:05.379 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T12:53:05.379 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:05.379 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:53:05.379 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T12:53:05.379 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:05.380 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:05.380 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:05.380 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-13T12:53:05.380 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:53:05.388 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:53:05.388 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:53:05.416 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:53:05.416 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:53:05.637 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:53:05.638 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-volume 2024-09-13T12:53:05.697 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-volume-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:53:05.713 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-volume-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:53:05.713 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:53:05.713 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T12:53:05.713 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:06.235 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:06.377 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:06.377 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:06.378 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-13T12:53:06.378 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:06.378 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:53:06.378 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume noarch 2:19.3.0-4917.gb0201b8c.el9 ceph-noarch 285 k 2024-09-13T12:53:06.378 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:06.378 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:53:06.378 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:06.378 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-13T12:53:06.378 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:06.378 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 285 k 2024-09-13T12:53:06.378 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.3 M 2024-09-13T12:53:06.378 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:53:06.610 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-volume-19.3.0-4917.gb0201b8c.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-13T12:53:06.611 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:06.611 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-13T12:53:06.611 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:53:06.620 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:53:06.621 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:53:06.653 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:53:06.653 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:53:06.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-volume-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:53:06.699 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:06.699 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:53:06.699 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-volume-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:53:06.699 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:06.699 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:06.891 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:53:06.971 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-volume-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:53:06.988 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-volume-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:53:06.988 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:53:06.988 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T12:53:06.988 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:06.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados-devel 2024-09-13T12:53:07.578 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:07.718 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:07.719 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:07.719 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-09-13T12:53:07.719 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:07.719 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:53:07.719 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 127 k 2024-09-13T12:53:07.719 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:07.719 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:53:07.719 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:07.719 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-09-13T12:53:07.720 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:07.720 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 127 k 2024-09-13T12:53:07.720 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 456 k 2024-09-13T12:53:07.720 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:53:07.926 INFO:teuthology.orchestra.run.smithi167.stdout:librados-devel-19.3.0-4917.gb0201b8c.el9.x86_64 616 kB/s | 127 kB 00:00 2024-09-13T12:53:07.927 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:07.927 INFO:teuthology.orchestra.run.smithi167.stdout:Total 611 kB/s | 127 kB 00:00 2024-09-13T12:53:07.927 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:53:07.932 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:53:07.932 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:53:07.996 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:53:07.997 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:53:08.013 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-volume-2:19.3.0-4917.gb0201b8c.el9.noarch 1/1 2024-09-13T12:53:08.014 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:08.014 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:53:08.014 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume-2:19.3.0-4917.gb0201b8c.el9.noarch 2024-09-13T12:53:08.014 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:08.014 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:53:08.256 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librados-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:08.309 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel 2024-09-13T12:53:08.685 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:08.907 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:08.923 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:08.923 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:08.923 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:53:08.923 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:53:08.923 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:08.923 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:09.047 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:09.048 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:09.048 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-09-13T12:53:09.048 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:09.049 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:53:09.049 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 127 k 2024-09-13T12:53:09.049 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:09.049 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:53:09.049 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:09.049 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-13T12:53:09.049 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:09.049 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 127 k 2024-09-13T12:53:09.049 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 456 k 2024-09-13T12:53:09.049 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:53:09.182 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs2 2024-09-13T12:53:09.256 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-19.3.0-4917.gb0201b8c.el9.x86_64 612 kB/s | 127 kB 00:00 2024-09-13T12:53:09.256 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:09.257 INFO:teuthology.orchestra.run.smithi052.stdout:Total 607 kB/s | 127 kB 00:00 2024-09-13T12:53:09.257 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:53:09.263 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:53:09.263 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:53:09.333 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:53:09.333 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:53:09.490 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:53:09.591 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:09.764 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:09.858 INFO:teuthology.orchestra.run.smithi167.stdout:Package libcephfs2-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:09.903 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:09.904 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-09-13T12:53:09.904 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:09.979 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs-devel 2024-09-13T12:53:10.038 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:10.322 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:10.322 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:10.322 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:53:10.322 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:53:10.322 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:10.322 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:10.543 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:41 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:10.618 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2 2024-09-13T12:53:10.676 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:10.677 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:10.678 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-09-13T12:53:10.678 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:10.678 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:53:10.678 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 32 k 2024-09-13T12:53:10.678 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:10.678 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:53:10.678 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:10.678 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-09-13T12:53:10.678 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:10.678 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 32 k 2024-09-13T12:53:10.678 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 142 k 2024-09-13T12:53:10.678 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:53:10.830 INFO:teuthology.orchestra.run.smithi167.stdout:libcephfs-devel-19.3.0-4917.gb0201b8c.el9.x86_6 209 kB/s | 32 kB 00:00 2024-09-13T12:53:10.830 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:10.831 INFO:teuthology.orchestra.run.smithi167.stdout:Total 207 kB/s | 32 kB 00:00 2024-09-13T12:53:10.831 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:53:10.834 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:53:10.834 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:53:10.850 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:53:10.870 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:53:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:53:11.052 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:11.216 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:11.313 INFO:teuthology.orchestra.run.smithi052.stdout:Package libcephfs2-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:11.359 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:11.360 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-13T12:53:11.360 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:11.429 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs-devel 2024-09-13T12:53:11.457 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:11.706 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:11.707 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:11.707 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:53:11.707 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:53:11.707 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:11.707 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:11.871 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados2 2024-09-13T12:53:12.022 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:12.161 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:12.162 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:12.162 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-09-13T12:53:12.162 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:12.162 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:53:12.162 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 32 k 2024-09-13T12:53:12.162 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:12.162 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:53:12.162 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:12.163 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-13T12:53:12.163 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:12.163 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 32 k 2024-09-13T12:53:12.163 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 142 k 2024-09-13T12:53:12.163 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:53:12.301 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs-devel-19.3.0-4917.gb0201b8c.el9.x86_6 230 kB/s | 32 kB 00:00 2024-09-13T12:53:12.301 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:12.302 INFO:teuthology.orchestra.run.smithi052.stdout:Total 228 kB/s | 32 kB 00:00 2024-09-13T12:53:12.302 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:53:12.305 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:53:12.305 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:53:12.326 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:53:12.327 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:53:12.438 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:12.458 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:53:12.528 INFO:teuthology.orchestra.run.smithi167.stdout:Package librados2-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:12.571 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:12.572 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-09-13T12:53:12.573 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:12.577 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:12.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librbd1 2024-09-13T12:53:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:13.188 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:13.188 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:13.188 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:53:13.188 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:53:13.188 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:13.188 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:13.221 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:13.314 INFO:teuthology.orchestra.run.smithi167.stdout:Package librbd1-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:13.357 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:13.358 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-09-13T12:53:13.358 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:13.370 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2 2024-09-13T12:53:13.440 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rados 2024-09-13T12:53:13.963 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:14.010 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:14.056 INFO:teuthology.orchestra.run.smithi052.stdout:Package librados2-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:14.101 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rados-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:14.102 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:14.103 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-13T12:53:14.103 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:14.145 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:14.146 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-09-13T12:53:14.146 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:14.179 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1 2024-09-13T12:53:14.228 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rgw 2024-09-13T12:53:14.773 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:14.790 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:14.867 INFO:teuthology.orchestra.run.smithi052.stdout:Package librbd1-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:14.881 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:14.913 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:14.914 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-13T12:53:14.914 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:14.925 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:14.926 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-09-13T12:53:14.926 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:14.984 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados 2024-09-13T12:53:15.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-cephfs 2024-09-13T12:53:15.578 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:15.584 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:15.672 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rados-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:15.675 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-cephfs-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:15.718 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:15.719 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:15.719 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-09-13T12:53:15.719 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:15.720 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-13T12:53:15.720 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:15.794 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw 2024-09-13T12:53:15.798 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rbd 2024-09-13T12:53:16.372 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:16.384 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:16.464 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:16.481 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rgw-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:16.507 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:16.509 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-09-13T12:53:16.509 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:16.524 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:16.525 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-13T12:53:16.525 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:16.588 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-fuse 2024-09-13T12:53:16.602 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs 2024-09-13T12:53:17.155 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:17.192 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:17.285 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-cephfs-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:17.290 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:17.291 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:17.291 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-09-13T12:53:17.291 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:17.291 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:53:17.291 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 85 k 2024-09-13T12:53:17.291 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:17.291 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:53:17.291 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:17.292 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-09-13T12:53:17.292 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:17.292 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 85 k 2024-09-13T12:53:17.292 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 231 k 2024-09-13T12:53:17.292 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:53:17.331 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:17.332 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-13T12:53:17.333 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:17.408 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd 2024-09-13T12:53:17.483 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-fuse-19.3.0-4917.gb0201b8c.el9.x86_64.rpm 446 kB/s | 85 kB 00:00 2024-09-13T12:53:17.484 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:17.484 INFO:teuthology.orchestra.run.smithi167.stdout:Total 443 kB/s | 85 kB 00:00 2024-09-13T12:53:17.484 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:53:17.491 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:53:17.491 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:53:17.556 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:53:17.557 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:53:17.712 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:53:17.797 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:18.001 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:18.094 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 is already installed. 2024-09-13T12:53:18.141 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:18.141 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-13T12:53:18.142 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:18.170 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:18.208 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse 2024-09-13T12:53:18.433 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:18.433 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:18.433 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:53:18.433 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:53:18.433 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:18.433 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:18.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-mirror 2024-09-13T12:53:18.799 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:18.940 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:18.941 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:18.941 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-09-13T12:53:18.941 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:18.941 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:53:18.941 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 85 k 2024-09-13T12:53:18.941 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:18.941 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:53:18.942 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:18.942 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-13T12:53:18.942 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:18.942 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 85 k 2024-09-13T12:53:18.942 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 231 k 2024-09-13T12:53:18.942 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:53:19.145 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-19.3.0-4917.gb0201b8c.el9.x86_64.rpm 418 kB/s | 85 kB 00:00 2024-09-13T12:53:19.146 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:19.146 INFO:teuthology.orchestra.run.smithi052.stdout:Total 415 kB/s | 85 kB 00:00 2024-09-13T12:53:19.146 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:53:19.154 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:53:19.154 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:53:19.222 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:53:19.222 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:53:19.294 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:50 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:19.398 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:53:19.435 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:19.435 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:19.435 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-09-13T12:53:19.436 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:19.436 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:53:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 3.2 M 2024-09-13T12:53:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:19.436 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:53:19.436 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:19.436 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-09-13T12:53:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:19.436 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 3.2 M 2024-09-13T12:53:19.436 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 14 M 2024-09-13T12:53:19.436 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:53:19.516 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:19.838 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-mirror-19.3.0-4917.gb0201b8c.el9.x86_64.rpm 7.9 MB/s | 3.2 MB 00:00 2024-09-13T12:53:19.838 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:19.838 INFO:teuthology.orchestra.run.smithi167.stdout:Total 7.9 MB/s | 3.2 MB 00:00 2024-09-13T12:53:19.839 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:53:19.848 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:53:19.848 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:53:19.894 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:19.931 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:53:19.931 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:53:20.170 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:20.170 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:20.170 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:53:20.170 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:53:20.170 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:20.170 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:20.428 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror 2024-09-13T12:53:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:53:20.505 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-mirror-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:20.532 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:20.532 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:53:20.532 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T12:53:20.532 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T12:53:20.532 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T12:53:20.532 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:21.031 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:21.172 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:21.173 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:21.173 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-09-13T12:53:21.173 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:21.173 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:53:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 3.2 M 2024-09-13T12:53:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:21.174 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:53:21.174 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:21.174 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-13T12:53:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:21.174 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.2 M 2024-09-13T12:53:21.174 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 14 M 2024-09-13T12:53:21.174 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:53:21.591 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-19.3.0-4917.gb0201b8c.el9.x86_64.rpm 7.6 MB/s | 3.2 MB 00:00 2024-09-13T12:53:21.592 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:21.592 INFO:teuthology.orchestra.run.smithi052.stdout:Total 7.6 MB/s | 3.2 MB 00:00 2024-09-13T12:53:21.592 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:53:21.603 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:53:21.603 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:53:21.681 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-mirror-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:21.681 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:21.681 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:53:21.681 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:53:21.681 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:21.681 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:21.693 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:53:21.693 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:53:21.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-nbd 2024-09-13T12:53:22.220 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:53:22.275 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:22.302 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:22.303 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:53:22.303 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T12:53:22.303 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T12:53:22.303 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T12:53:22.303 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:22.476 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:53 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:22.607 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-09-13T12:53:22.608 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:22.608 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-09-13T12:53:22.608 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:22.608 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-09-13T12:53:22.608 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 170 k 2024-09-13T12:53:22.608 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:22.608 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-09-13T12:53:22.608 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-09-13T12:53:22.609 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-09-13T12:53:22.609 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:22.609 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 170 k 2024-09-13T12:53:22.609 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 490 k 2024-09-13T12:53:22.609 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-09-13T12:53:22.833 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-nbd-19.3.0-4917.gb0201b8c.el9.x86_64.rpm 761 kB/s | 170 kB 00:00 2024-09-13T12:53:22.834 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:22.834 INFO:teuthology.orchestra.run.smithi167.stdout:Total 756 kB/s | 170 kB 00:00 2024-09-13T12:53:22.834 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-09-13T12:53:22.842 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-09-13T12:53:22.842 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-09-13T12:53:22.908 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-09-13T12:53:22.909 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-09-13T12:53:23.072 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-09-13T12:53:23.189 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-nbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:23.442 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:23.442 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:23.442 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:53:23.443 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:53:23.443 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:23.443 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:23.642 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:23.669 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd 2024-09-13T12:53:23.921 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-nbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:23.921 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:23.921 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-09-13T12:53:23.921 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:53:23.922 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:53:23.922 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-09-13T12:53:24.147 DEBUG:teuthology.parallel:result is None 2024-09-13T12:53:24.271 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:55 ago on Fri 13 Sep 2024 12:51:29 PM UTC. 2024-09-13T12:53:24.408 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-13T12:53:24.409 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:24.409 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-09-13T12:53:24.409 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:24.409 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-13T12:53:24.409 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd x86_64 2:19.3.0-4917.gb0201b8c.el9 ceph 170 k 2024-09-13T12:53:24.409 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:24.409 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-13T12:53:24.410 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-13T12:53:24.410 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-13T12:53:24.410 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:24.410 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 170 k 2024-09-13T12:53:24.410 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 490 k 2024-09-13T12:53:24.410 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-13T12:53:24.628 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-19.3.0-4917.gb0201b8c.el9.x86_64.rpm 782 kB/s | 170 kB 00:00 2024-09-13T12:53:24.628 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:53:24.628 INFO:teuthology.orchestra.run.smithi052.stdout:Total 777 kB/s | 170 kB 00:00 2024-09-13T12:53:24.628 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-13T12:53:24.637 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-13T12:53:24.637 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-13T12:53:24.705 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-13T12:53:24.706 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-13T12:53:24.873 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-13T12:53:24.971 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:25.363 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:25.627 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 1/1 2024-09-13T12:53:25.627 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:25.628 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-13T12:53:25.628 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-2:19.3.0-4917.gb0201b8c.el9.x86_64 2024-09-13T12:53:25.628 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:53:25.628 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-13T12:53:25.873 DEBUG:teuthology.parallel:result is None 2024-09-13T12:53:25.874 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b0201b8c79733293453e7f10a10c7fa43119222b 2024-09-13T12:53:26.025 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T12:53:26.048 INFO:teuthology.orchestra.run.smithi052.stdout:19.3.0-4917.gb0201b8c.el9 2024-09-13T12:53:26.048 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4917.gb0201b8c.el9 2024-09-13T12:53:26.048 INFO:teuthology.task.install:The correct ceph version 19.3.0-4917.gb0201b8c is installed. 2024-09-13T12:53:26.050 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b0201b8c79733293453e7f10a10c7fa43119222b 2024-09-13T12:53:26.197 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T12:53:26.223 INFO:teuthology.orchestra.run.smithi167.stdout:19.3.0-4917.gb0201b8c.el9 2024-09-13T12:53:26.223 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4917.gb0201b8c.el9 2024-09-13T12:53:26.224 INFO:teuthology.task.install:The correct ceph version 19.3.0-4917.gb0201b8c is installed. 2024-09-13T12:53:26.226 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T12:53:26.226 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:53:26.226 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T12:53:26.256 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-09-13T12:53:26.256 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T12:53:26.298 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T12:53:26.299 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:53:26.299 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T12:53:26.325 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T12:53:26.390 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-09-13T12:53:26.390 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T12:53:26.419 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T12:53:26.488 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T12:53:26.488 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:53:26.488 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T12:53:26.513 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T12:53:26.579 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-09-13T12:53:26.579 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T12:53:26.606 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T12:53:26.673 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T12:53:26.673 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:53:26.673 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T12:53:26.699 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T12:53:26.765 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-09-13T12:53:26.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T12:53:26.791 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T12:53:26.857 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-13T12:53:26.964 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': 'b0201b8c79733293453e7f10a10c7fa43119222b', 'cephadm_mode': 'root'} 2024-09-13T12:53:26.964 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b 2024-09-13T12:53:26.964 INFO:tasks.cephadm:Cluster fsid is 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:53:26.964 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-13T12:53:26.965 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.52:6789]', 'mon.c': '[v1:172.21.15.52:6790]', 'mon.b': '[v1:172.21.15.167:6789]'} 2024-09-13T12:53:26.965 INFO:tasks.cephadm:First mon is mon.a on smithi052 2024-09-13T12:53:26.965 INFO:tasks.cephadm:First mgr is y 2024-09-13T12:53:26.965 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-13T12:53:26.965 DEBUG:teuthology.orchestra.run.smithi052:> sudo hostname $(hostname -s) 2024-09-13T12:53:26.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo hostname $(hostname -s) 2024-09-13T12:53:27.018 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-13T12:53:27.018 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b0201b8c79733293453e7f10a10c7fa43119222b 2024-09-13T12:53:27.171 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'b0201b8c79733293453e7f10a10c7fa43119222b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83042/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4917-gb0201b8c', 'node_name': '172.21.5.40+adami10', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4917.gb0201b8c'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-13 11:13:17.876231', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-main-2024-09-13-0840', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-13T12:53:27.322 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-guits-main-2024-09-13-0840, sha1 b0201b8c79733293453e7f10a10c7fa43119222b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=b0201b8c79733293453e7f10a10c7fa43119222b 2024-09-13T12:53:27.324 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/x86_64/flavors/default/cephadm 2024-09-13T12:53:27.324 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/x86_64/flavors/default/cephadm 2024-09-13T12:53:27.324 DEBUG:teuthology.orchestra.run.smithi052:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T12:53:27.673 INFO:teuthology.orchestra.run.smithi052.stdout:-rw-r--r--. 1 ubuntu ubuntu 804201 Sep 13 12:53 /home/ubuntu/cephtest/cephadm 2024-09-13T12:53:27.673 DEBUG:teuthology.orchestra.run.smithi167:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-13-0840/b0201b8c79733293453e7f10a10c7fa43119222b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T12:53:28.029 INFO:teuthology.orchestra.run.smithi167.stdout:-rw-r--r--. 1 ubuntu ubuntu 804201 Sep 13 12:53 /home/ubuntu/cephtest/cephadm 2024-09-13T12:53:28.029 DEBUG:teuthology.orchestra.run.smithi052:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T12:53:28.050 DEBUG:teuthology.orchestra.run.smithi167:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T12:53:28.081 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b on all hosts... 2024-09-13T12:53:28.081 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b pull 2024-09-13T12:53:28.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b pull 2024-09-13T12:53:29.715 INFO:teuthology.orchestra.run.smithi052.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b... 2024-09-13T12:53:29.824 INFO:teuthology.orchestra.run.smithi167.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b... 2024-09-13T12:54:59.339 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T12:54:59.340 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_version": "ceph version 19.3.0-4917-gb0201b8c (b0201b8c79733293453e7f10a10c7fa43119222b) squid (dev)", 2024-09-13T12:54:59.340 INFO:teuthology.orchestra.run.smithi052.stdout: "image_id": "1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031", 2024-09-13T12:54:59.340 INFO:teuthology.orchestra.run.smithi052.stdout: "repo_digests": [ 2024-09-13T12:54:59.341 INFO:teuthology.orchestra.run.smithi052.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b" 2024-09-13T12:54:59.341 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T12:54:59.341 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T12:55:07.788 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T12:55:07.788 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph_version": "ceph version 19.3.0-4917-gb0201b8c (b0201b8c79733293453e7f10a10c7fa43119222b) squid (dev)", 2024-09-13T12:55:07.788 INFO:teuthology.orchestra.run.smithi167.stdout: "image_id": "1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031", 2024-09-13T12:55:07.789 INFO:teuthology.orchestra.run.smithi167.stdout: "repo_digests": [ 2024-09-13T12:55:07.789 INFO:teuthology.orchestra.run.smithi167.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b" 2024-09-13T12:55:07.789 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T12:55:07.789 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T12:55:07.818 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph 2024-09-13T12:55:07.852 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /etc/ceph 2024-09-13T12:55:07.883 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /etc/ceph 2024-09-13T12:55:07.923 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 777 /etc/ceph 2024-09-13T12:55:07.950 INFO:tasks.cephadm:Writing seed config... 2024-09-13T12:55:07.951 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-13T12:55:07.951 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-13T12:55:07.951 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-09-13T12:55:07.951 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-09-13T12:55:07.951 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-09-13T12:55:07.952 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-09-13T12:55:07.952 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-13T12:55:07.952 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-13T12:55:07.952 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-13T12:55:07.952 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-13T12:55:07.952 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-13T12:55:07.952 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-13T12:55:07.952 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-13T12:55:07.952 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-13T12:55:07.953 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-13T12:55:07.953 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-13T12:55:07.953 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-13T12:55:07.953 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-13T12:55:07.953 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-13T12:55:07.953 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-13T12:55:07.953 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-13T12:55:07.953 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-13T12:55:07.954 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-13T12:55:07.954 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-13T12:55:07.955 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:55:07.955 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-13T12:55:07.984 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2b415b38-71cf-11ef-bceb-c7b262605968 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-13T12:55:07.984 DEBUG:teuthology.orchestra.run.smithi052:mon.a> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.a.service 2024-09-13T12:55:08.027 DEBUG:teuthology.orchestra.run.smithi052:mgr.y> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mgr.y.service 2024-09-13T12:55:08.070 INFO:tasks.cephadm:Bootstrapping... 2024-09-13T12:55:08.070 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b -v bootstrap --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.52:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-13T12:55:08.290 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:55:08.290 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b', '-v', 'bootstrap', '--fsid', '2b415b38-71cf-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.52:6789]', '--skip-admin-label'] 2024-09-13T12:55:08.291 INFO:teuthology.orchestra.run.smithi052.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-13T12:55:08.291 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying podman|docker is present... 2024-09-13T12:55:08.315 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 5.2.2 2024-09-13T12:55:08.316 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying lvm2 is present... 2024-09-13T12:55:08.316 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying time synchronization is in place... 2024-09-13T12:55:08.323 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T12:55:08.324 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T12:55:08.330 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T12:55:08.330 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-09-13T12:55:08.337 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout enabled 2024-09-13T12:55:08.343 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout active 2024-09-13T12:55:08.343 INFO:teuthology.orchestra.run.smithi052.stdout:Unit chronyd.service is enabled and running 2024-09-13T12:55:08.344 INFO:teuthology.orchestra.run.smithi052.stdout:Repeating the final host check... 2024-09-13T12:55:08.367 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 5.2.2 2024-09-13T12:55:08.368 INFO:teuthology.orchestra.run.smithi052.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-13T12:55:08.368 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl is present 2024-09-13T12:55:08.368 INFO:teuthology.orchestra.run.smithi052.stdout:lvcreate is present 2024-09-13T12:55:08.374 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T12:55:08.374 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T12:55:08.381 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T12:55:08.381 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-09-13T12:55:08.387 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout enabled 2024-09-13T12:55:08.394 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout active 2024-09-13T12:55:08.394 INFO:teuthology.orchestra.run.smithi052.stdout:Unit chronyd.service is enabled and running 2024-09-13T12:55:08.394 INFO:teuthology.orchestra.run.smithi052.stdout:Host looks OK 2024-09-13T12:55:08.394 INFO:teuthology.orchestra.run.smithi052.stdout:Cluster fsid: 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:55:08.395 INFO:teuthology.orchestra.run.smithi052.stdout:Acquiring lock 140421577105024 on /run/cephadm/2b415b38-71cf-11ef-bceb-c7b262605968.lock 2024-09-13T12:55:08.395 INFO:teuthology.orchestra.run.smithi052.stdout:Lock 140421577105024 acquired on /run/cephadm/2b415b38-71cf-11ef-bceb-c7b262605968.lock 2024-09-13T12:55:08.395 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying IP 172.21.15.52 port 6789 ... 2024-09-13T12:55:08.395 INFO:teuthology.orchestra.run.smithi052.stdout:Base mon IP(s) is [172.21.15.52:6789], mon addrv is [v1:172.21.15.52:6789] 2024-09-13T12:55:08.399 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.52 metric 100 2024-09-13T12:55:08.399 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.52 metric 100 2024-09-13T12:55:08.401 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-13T12:55:08.401 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-13T12:55:08.401 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1787sec hoplimit 64 pref medium 2024-09-13T12:55:08.404 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-13T12:55:08.405 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-13T12:55:08.405 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T12:55:08.405 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-13T12:55:08.405 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15de/64 scope link 2024-09-13T12:55:08.405 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T12:55:08.405 INFO:teuthology.orchestra.run.smithi052.stdout:Mon IP `172.21.15.52` is in CIDR network `172.21.0.0/20` 2024-09-13T12:55:08.405 INFO:teuthology.orchestra.run.smithi052.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-09-13T12:55:08.405 INFO:teuthology.orchestra.run.smithi052.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-13T12:55:08.406 INFO:teuthology.orchestra.run.smithi052.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b... 2024-09-13T12:55:08.814 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 2024-09-13T12:55:08.814 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b... 2024-09-13T12:55:08.814 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Getting image source signatures 2024-09-13T12:55:08.814 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying blob sha256:ed0c205fe07e55e926825d2d80c92003b1e841ce1f2d91029cfd1b95c3d1cf26 2024-09-13T12:55:08.814 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-13T12:55:08.814 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying config sha256:1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 2024-09-13T12:55:08.814 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-13T12:55:09.744 INFO:teuthology.orchestra.run.smithi052.stdout:ceph: stdout ceph version 19.3.0-4917-gb0201b8c (b0201b8c79733293453e7f10a10c7fa43119222b) squid (dev) 2024-09-13T12:55:09.745 INFO:teuthology.orchestra.run.smithi052.stdout:Ceph version: ceph version 19.3.0-4917-gb0201b8c (b0201b8c79733293453e7f10a10c7fa43119222b) squid (dev) 2024-09-13T12:55:09.745 INFO:teuthology.orchestra.run.smithi052.stdout:Extracting ceph user uid/gid from container image... 2024-09-13T12:55:13.779 INFO:teuthology.orchestra.run.smithi052.stdout:stat: stdout 167 167 2024-09-13T12:55:13.779 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial keys... 2024-09-13T12:55:21.863 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQA0NuRmotYcHxAAOBe+mXXPkBR8OD3sYUYf7A== 2024-09-13T12:55:22.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQA6NuRm32SoDBAAyQtqlovBV1kyFx+x0DOG+Q== 2024-09-13T12:55:23.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQA7NuRmgSEKDRAABptbcyyveJ6J0Qswk3YBFw== 2024-09-13T12:55:23.834 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial monmap... 2024-09-13T12:55:24.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T12:55:24.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-13T12:55:24.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:55:24.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T12:55:24.722 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool for a [v1:172.21.15.52:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T12:55:24.722 INFO:teuthology.orchestra.run.smithi052.stdout:setting min_mon_release = quincy 2024-09-13T12:55:24.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: set fsid to 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:55:24.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T12:55:24.722 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:55:24.722 INFO:teuthology.orchestra.run.smithi052.stdout:Creating mon... 2024-09-13T12:55:25.744 INFO:teuthology.orchestra.run.smithi052.stdout:create mon.a on 2024-09-13T12:55:25.955 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-13T12:55:26.098 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-13T12:55:26.302 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2b415b38-71cf-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-2b415b38-71cf-11ef-bceb-c7b262605968.target. 2024-09-13T12:55:26.302 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-2b415b38-71cf-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-2b415b38-71cf-11ef-bceb-c7b262605968.target. 2024-09-13T12:55:26.532 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.a 2024-09-13T12:55:26.532 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-13T12:55:26.725 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2b415b38-71cf-11ef-bceb-c7b262605968.target.wants/ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-2b415b38-71cf-11ef-bceb-c7b262605968@.service. 2024-09-13T12:55:27.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:26 smithi052 podman[29084]: 2024-09-13 12:55:26.87509512 +0000 UTC m=+0.017354875 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b 2024-09-13T12:55:27.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:27 smithi052 podman[29084]: 2024-09-13 12:55:27.650632646 +0000 UTC m=+0.792892393 container create 6fa23f9f9642d59b05516f04d25208a3a36d92b09ab967ad78ab9a3aae06972a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T12:55:27.968 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T12:55:27.968 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-09-13T12:55:27.975 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T12:55:27.975 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-09-13T12:55:27.975 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-09-13T12:55:27.975 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T12:55:27.976 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mon to start... 2024-09-13T12:55:27.976 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mon... 2024-09-13T12:55:28.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:27 smithi052 podman[29084]: 2024-09-13 12:55:27.852872053 +0000 UTC m=+0.995131798 container init 6fa23f9f9642d59b05516f04d25208a3a36d92b09ab967ad78ab9a3aae06972a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-13T12:55:28.201 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:27 smithi052 podman[29084]: 2024-09-13 12:55:27.856210333 +0000 UTC m=+0.998470077 container start 6fa23f9f9642d59b05516f04d25208a3a36d92b09ab967ad78ab9a3aae06972a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8) 2024-09-13T12:55:28.201 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:27 smithi052 ceph-mon[29106]: mkfs 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:55:28.201 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:27 smithi052 ceph-mon[29106]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T12:55:28.201 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:27 smithi052 bash[29084]: 6fa23f9f9642d59b05516f04d25208a3a36d92b09ab967ad78ab9a3aae06972a 2024-09-13T12:55:28.201 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:27 smithi052 systemd[1]: Started Ceph mon.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T12:55:28.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout cluster: 2024-09-13T12:55:28.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout id: 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:55:28.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-13T12:55:28.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-09-13T12:55:28.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout services: 2024-09-13T12:55:28.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.579802s) 2024-09-13T12:55:28.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-13T12:55:28.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-13T12:55:28.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-09-13T12:55:28.982 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout data: 2024-09-13T12:55:28.982 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-13T12:55:28.982 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-13T12:55:28.982 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-13T12:55:28.982 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout pgs: 2024-09-13T12:55:28.982 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-09-13T12:55:28.982 INFO:teuthology.orchestra.run.smithi052.stdout:mon is available 2024-09-13T12:55:28.982 INFO:teuthology.orchestra.run.smithi052.stdout:Assimilating anything we can from ceph.conf... 2024-09-13T12:55:29.220 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:28 smithi052 ceph-mon[29106]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T12:55:29.220 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:28 smithi052 ceph-mon[29106]: monmap epoch 1 2024-09-13T12:55:29.220 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:28 smithi052 ceph-mon[29106]: fsid 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:55:29.220 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:28 smithi052 ceph-mon[29106]: last_changed 2024-09-13T12:55:24.138668+0000 2024-09-13T12:55:29.220 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:28 smithi052 ceph-mon[29106]: created 2024-09-13T12:55:24.138668+0000 2024-09-13T12:55:29.221 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:28 smithi052 ceph-mon[29106]: min_mon_release 19 (squid) 2024-09-13T12:55:29.221 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:28 smithi052 ceph-mon[29106]: election_strategy: 1 2024-09-13T12:55:29.221 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:28 smithi052 ceph-mon[29106]: 0: v1:172.21.15.52:6789/0 mon.a 2024-09-13T12:55:29.221 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:28 smithi052 ceph-mon[29106]: fsmap 2024-09-13T12:55:29.221 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:28 smithi052 ceph-mon[29106]: osdmap e1: 0 total, 0 up, 0 in 2024-09-13T12:55:29.221 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:28 smithi052 ceph-mon[29106]: mgrmap e1: no daemons active 2024-09-13T12:55:29.221 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:28 smithi052 ceph-mon[29106]: from='client.? v1:172.21.15.52:0/2064248487' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-13T12:55:29.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-09-13T12:55:29.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T12:55:29.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout fsid = 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:55:29.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-13T12:55:29.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.52:6789] 2024-09-13T12:55:29.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T12:55:29.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T12:55:29.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T12:55:29.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T12:55:29.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-09-13T12:55:29.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T12:55:29.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T12:55:29.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-09-13T12:55:29.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T12:55:29.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T12:55:29.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T12:55:29.980 INFO:teuthology.orchestra.run.smithi052.stdout:Generating new minimal ceph.conf... 2024-09-13T12:55:30.184 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:29 smithi052 ceph-mon[29106]: from='client.? v1:172.21.15.52:0/432174258' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-13T12:55:30.184 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:29 smithi052 ceph-mon[29106]: from='client.? v1:172.21.15.52:0/432174258' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-13T12:55:31.001 INFO:teuthology.orchestra.run.smithi052.stdout:Restarting the monitor... 2024-09-13T12:55:31.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:30 smithi052 ceph-mon[29106]: from='client.? v1:172.21.15.52:0/2425493181' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:55:31.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:31 smithi052 systemd[1]: Stopping Ceph mon.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T12:55:31.626 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:31 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a[29102]: 2024-09-13T12:55:31.283+0000 7f9aaf36f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T12:55:31.626 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:31 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a[29102]: 2024-09-13T12:55:31.283+0000 7f9aaf36f640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-13T12:55:31.626 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:31 smithi052 podman[29347]: 2024-09-13 12:55:31.417208417 +0000 UTC m=+0.241336018 container died 6fa23f9f9642d59b05516f04d25208a3a36d92b09ab967ad78ab9a3aae06972a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-13T12:55:31.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:31 smithi052 podman[29347]: 2024-09-13 12:55:31.724977546 +0000 UTC m=+0.549105146 container cleanup 6fa23f9f9642d59b05516f04d25208a3a36d92b09ab967ad78ab9a3aae06972a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-13T12:55:31.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:31 smithi052 bash[29347]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a 2024-09-13T12:55:32.184 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:31 smithi052 podman[29359]: 2024-09-13 12:55:31.926019904 +0000 UTC m=+0.504928716 container remove 6fa23f9f9642d59b05516f04d25208a3a36d92b09ab967ad78ab9a3aae06972a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T12:55:32.184 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-13T12:55:32.184 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 systemd[1]: Stopped Ceph mon.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T12:55:32.184 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 systemd[1]: Starting Ceph mon.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T12:55:32.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 podman[29452]: 2024-09-13 12:55:32.360128438 +0000 UTC m=+0.105910997 container create 17b694ca3a37bff268f92cc01ddb74df39f87fc959b645e75df9aafba04906b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T12:55:32.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 podman[29452]: 2024-09-13 12:55:32.270696765 +0000 UTC m=+0.016479294 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b 2024-09-13T12:55:32.667 INFO:teuthology.orchestra.run.smithi052.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-13T12:55:32.911 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 podman[29452]: 2024-09-13 12:55:32.570633739 +0000 UTC m=+0.316416256 container init 17b694ca3a37bff268f92cc01ddb74df39f87fc959b645e75df9aafba04906b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T12:55:32.911 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 podman[29452]: 2024-09-13 12:55:32.573918711 +0000 UTC m=+0.319701239 container start 17b694ca3a37bff268f92cc01ddb74df39f87fc959b645e75df9aafba04906b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-13T12:55:32.911 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T12:55:32.911 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: ceph version 19.3.0-4917-gb0201b8c (b0201b8c79733293453e7f10a10c7fa43119222b) squid (dev), process ceph-mon, pid 2 2024-09-13T12:55:32.911 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: pidfile_write: ignore empty --pid-file 2024-09-13T12:55:32.911 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: load: jerasure load: lrc 2024-09-13T12:55:32.911 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: RocksDB version: 7.9.2 2024-09-13T12:55:32.911 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Git sha 0 2024-09-13T12:55:32.911 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Compile date 2024-09-13 10:06:52 2024-09-13T12:55:32.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: DB SUMMARY 2024-09-13T12:55:32.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: DB Session ID: KPHEK8BCS26K23DIEA8U 2024-09-13T12:55:32.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: CURRENT file: CURRENT 2024-09-13T12:55:32.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T12:55:32.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-13T12:55:32.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-13T12:55:32.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 92886 ; 2024-09-13T12:55:32.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.error_if_exists: 0 2024-09-13T12:55:32.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.create_if_missing: 0 2024-09-13T12:55:32.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.paranoid_checks: 1 2024-09-13T12:55:32.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T12:55:32.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T12:55:32.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T12:55:32.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.env: 0x5600a914c120 2024-09-13T12:55:32.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T12:55:32.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.info_log: 0x5600ab91f6a0 2024-09-13T12:55:32.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T12:55:32.914 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.statistics: (nil) 2024-09-13T12:55:32.914 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.use_fsync: 0 2024-09-13T12:55:32.914 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_log_file_size: 0 2024-09-13T12:55:32.914 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T12:55:32.915 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T12:55:32.915 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T12:55:32.915 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T12:55:32.915 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.allow_fallocate: 1 2024-09-13T12:55:32.915 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T12:55:32.915 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T12:55:32.915 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.use_direct_reads: 0 2024-09-13T12:55:32.915 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T12:55:32.915 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T12:55:32.916 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.db_log_dir: 2024-09-13T12:55:32.916 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.wal_dir: 2024-09-13T12:55:32.916 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T12:55:32.916 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T12:55:32.916 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T12:55:32.916 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T12:55:32.916 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T12:55:32.916 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T12:55:32.916 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T12:55:32.916 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T12:55:32.916 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.write_buffer_manager: 0x5600ab923b80 2024-09-13T12:55:32.916 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T12:55:32.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T12:55:32.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T12:55:32.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T12:55:32.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T12:55:32.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T12:55:32.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T12:55:32.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T12:55:32.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.unordered_write: 0 2024-09-13T12:55:32.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T12:55:32.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T12:55:32.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T12:55:32.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T12:55:32.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.row_cache: None 2024-09-13T12:55:32.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.wal_filter: None 2024-09-13T12:55:32.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T12:55:32.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T12:55:32.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.two_write_queues: 0 2024-09-13T12:55:32.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T12:55:32.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.wal_compression: 0 2024-09-13T12:55:32.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.atomic_flush: 0 2024-09-13T12:55:32.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T12:55:32.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T12:55:32.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T12:55:32.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.log_readahead_size: 0 2024-09-13T12:55:32.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T12:55:32.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T12:55:32.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T12:55:32.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T12:55:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T12:55:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T12:55:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T12:55:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_background_jobs: 2 2024-09-13T12:55:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_background_compactions: -1 2024-09-13T12:55:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_subcompactions: 1 2024-09-13T12:55:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T12:55:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T12:55:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T12:55:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T12:55:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T12:55:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T12:55:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_open_files: -1 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_background_flushes: -1 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Compression algorithms supported: 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: kZSTD supported: 0 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: kXpressCompression supported: 0 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: kBZip2Compression supported: 0 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: kLZ4Compression supported: 1 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: kZlibCompression supported: 1 2024-09-13T12:55:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T12:55:32.922 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: kSnappyCompression supported: 1 2024-09-13T12:55:32.922 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T12:55:32.922 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T12:55:32.922 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-13T12:55:32.922 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T12:55:32.922 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T12:55:32.922 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.merge_operator: 2024-09-13T12:55:32.922 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compaction_filter: None 2024-09-13T12:55:32.923 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T12:55:32.923 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T12:55:32.923 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T12:55:32.923 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T12:55:32.923 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5600ab91f2c0) 2024-09-13T12:55:32.923 INFO:journalctl@ceph.mon.a.smithi052.stdout: cache_index_and_filter_blocks: 1 2024-09-13T12:55:32.923 INFO:journalctl@ceph.mon.a.smithi052.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T12:55:32.923 INFO:journalctl@ceph.mon.a.smithi052.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T12:55:32.923 INFO:journalctl@ceph.mon.a.smithi052.stdout: pin_top_level_index_and_filter: 1 2024-09-13T12:55:32.923 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_type: 0 2024-09-13T12:55:32.923 INFO:journalctl@ceph.mon.a.smithi052.stdout: data_block_index_type: 0 2024-09-13T12:55:32.923 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_shortening: 1 2024-09-13T12:55:32.923 INFO:journalctl@ceph.mon.a.smithi052.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T12:55:32.924 INFO:journalctl@ceph.mon.a.smithi052.stdout: checksum: 4 2024-09-13T12:55:32.924 INFO:journalctl@ceph.mon.a.smithi052.stdout: no_block_cache: 0 2024-09-13T12:55:32.924 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache: 0x5600ab913610 2024-09-13T12:55:32.924 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_name: BinnedLRUCache 2024-09-13T12:55:32.924 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_options: 2024-09-13T12:55:32.924 INFO:journalctl@ceph.mon.a.smithi052.stdout: capacity : 536870912 2024-09-13T12:55:32.924 INFO:journalctl@ceph.mon.a.smithi052.stdout: num_shard_bits : 4 2024-09-13T12:55:32.924 INFO:journalctl@ceph.mon.a.smithi052.stdout: strict_capacity_limit : 0 2024-09-13T12:55:32.924 INFO:journalctl@ceph.mon.a.smithi052.stdout: high_pri_pool_ratio: 0.000 2024-09-13T12:55:32.924 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_compressed: (nil) 2024-09-13T12:55:32.924 INFO:journalctl@ceph.mon.a.smithi052.stdout: persistent_cache: (nil) 2024-09-13T12:55:32.924 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_size: 4096 2024-09-13T12:55:32.924 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_size_deviation: 10 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_restart_interval: 16 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_block_restart_interval: 1 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: metadata_block_size: 4096 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: partition_filters: 0 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: use_delta_encoding: 1 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: filter_policy: bloomfilter 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: whole_key_filtering: 1 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: verify_compression: 0 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: read_amp_bytes_per_bit: 0 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: format_version: 5 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: enable_index_compression: 1 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_align: 0 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: max_auto_readahead_size: 262144 2024-09-13T12:55:32.925 INFO:journalctl@ceph.mon.a.smithi052.stdout: prepopulate_block_cache: 0 2024-09-13T12:55:32.926 INFO:journalctl@ceph.mon.a.smithi052.stdout: initial_auto_readahead_size: 8192 2024-09-13T12:55:32.926 INFO:journalctl@ceph.mon.a.smithi052.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T12:55:32.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T12:55:32.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T12:55:32.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compression: NoCompression 2024-09-13T12:55:32.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T12:55:32.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T12:55:32.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T12:55:32.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.num_levels: 7 2024-09-13T12:55:32.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T12:55:32.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T12:55:32.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T12:55:32.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T12:55:32.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T12:55:32.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T12:55:32.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T12:55:32.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T12:55:32.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T12:55:32.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T12:55:32.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T12:55:32.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T12:55:32.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T12:55:32.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T12:55:32.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T12:55:32.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T12:55:32.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T12:55:32.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T12:55:32.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T12:55:32.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T12:55:32.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T12:55:32.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T12:55:32.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T12:55:32.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T12:55:32.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T12:55:32.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T12:55:32.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T12:55:32.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T12:55:32.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.inplace_update_support: 0 2024-09-13T12:55:32.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T12:55:32.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T12:55:32.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T12:55:32.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T12:55:32.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.bloom_locality: 0 2024-09-13T12:55:32.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.max_successive_merges: 0 2024-09-13T12:55:32.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T12:55:32.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T12:55:32.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T12:55:32.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T12:55:32.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.ttl: 2592000 2024-09-13T12:55:32.932 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T12:55:32.932 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T12:55:32.932 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T12:55:32.932 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.enable_blob_files: false 2024-09-13T12:55:32.932 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.min_blob_size: 0 2024-09-13T12:55:32.932 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T12:55:32.932 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T12:55:32.932 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T12:55:32.932 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d485cd7b-52a1-4bba-b3ab-b1ea95320e42 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726232132602101, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726232132603043, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 88872, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 287, "table_properties": {"data_size": 86928, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11809, "raw_average_key_size": 48, "raw_value_size": 80400, "raw_average_value_size": 326, "num_data_blocks": 11, "num_entries": 246, "num_filter_entries": 246, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726232132, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d485cd7b-52a1-4bba-b3ab-b1ea95320e42", "db_session_id": "KPHEK8BCS26K23DIEA8U", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726232132603145, "job": 1, "event": "recovery_finished"} 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T12:55:32.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5600ab942e00 2024-09-13T12:55:32.934 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: DB pointer 0x5600aba44000 2024-09-13T12:55:32.934 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T12:55:32.934 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T12:55:32.934 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** DB Stats ** 2024-09-13T12:55:32.934 INFO:journalctl@ceph.mon.a.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T12:55:32.934 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T12:55:32.934 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T12:55:32.934 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T12:55:32.934 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T12:55:32.934 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T12:55:32.934 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T12:55:32.934 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-09-13T12:55:32.934 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** Compaction Stats [default] ** 2024-09-13T12:55:32.936 INFO:journalctl@ceph.mon.a.smithi052.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T12:55:32.936 INFO:journalctl@ceph.mon.a.smithi052.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T12:55:32.936 INFO:journalctl@ceph.mon.a.smithi052.stdout: L0 2/0 88.62 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T12:55:32.936 INFO:journalctl@ceph.mon.a.smithi052.stdout: Sum 2/0 88.62 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T12:55:32.936 INFO:journalctl@ceph.mon.a.smithi052.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T12:55:32.936 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-09-13T12:55:32.937 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** Compaction Stats [default] ** 2024-09-13T12:55:32.937 INFO:journalctl@ceph.mon.a.smithi052.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T12:55:32.937 INFO:journalctl@ceph.mon.a.smithi052.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T12:55:32.937 INFO:journalctl@ceph.mon.a.smithi052.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T12:55:32.937 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-09-13T12:55:32.937 INFO:journalctl@ceph.mon.a.smithi052.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T12:55:32.937 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-09-13T12:55:32.937 INFO:journalctl@ceph.mon.a.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T12:55:32.937 INFO:journalctl@ceph.mon.a.smithi052.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T12:55:32.937 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T12:55:32.937 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T12:55:32.938 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T12:55:32.938 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T12:55:32.938 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative compaction: 0.00 GB write, 26.83 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T12:55:32.938 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval compaction: 0.00 GB write, 26.83 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T12:55:32.938 INFO:journalctl@ceph.mon.a.smithi052.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T12:55:32.938 INFO:journalctl@ceph.mon.a.smithi052.stdout: Block cache BinnedLRUCache@0x5600ab913610#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-13T12:55:32.938 INFO:journalctl@ceph.mon.a.smithi052.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-13T12:55:32.938 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-09-13T12:55:32.938 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T12:55:32.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: starting mon.a rank 0 at public addrs v1:172.21.15.52:6789/0 at bind addrs v1:172.21.15.52:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:55:32.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: mon.a@-1(???) e1 preinit fsid 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:55:32.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: mon.a@-1(???).mds e1 new map 2024-09-13T12:55:32.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: mon.a@-1(???).mds e1 print_map 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout: e1 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout: btime 2024-09-13T12:55:27:893730+0000 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout: legacy client fscid: -1 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout: No filesystems configured 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 ceph-mon[29471]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 bash[29452]: 17b694ca3a37bff268f92cc01ddb74df39f87fc959b645e75df9aafba04906b7 2024-09-13T12:55:32.939 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:32 smithi052 systemd[1]: Started Ceph mon.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T12:55:33.864 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:33 smithi052 ceph-mon[29471]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T12:55:33.864 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:33 smithi052 ceph-mon[29471]: monmap epoch 1 2024-09-13T12:55:33.864 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:33 smithi052 ceph-mon[29471]: fsid 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:55:33.864 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:33 smithi052 ceph-mon[29471]: last_changed 2024-09-13T12:55:24.138668+0000 2024-09-13T12:55:33.864 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:33 smithi052 ceph-mon[29471]: created 2024-09-13T12:55:24.138668+0000 2024-09-13T12:55:33.864 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:33 smithi052 ceph-mon[29471]: min_mon_release 19 (squid) 2024-09-13T12:55:33.864 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:33 smithi052 ceph-mon[29471]: election_strategy: 1 2024-09-13T12:55:33.864 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:33 smithi052 ceph-mon[29471]: 0: v1:172.21.15.52:6789/0 mon.a 2024-09-13T12:55:33.864 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:33 smithi052 ceph-mon[29471]: fsmap 2024-09-13T12:55:33.865 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:33 smithi052 ceph-mon[29471]: osdmap e1: 0 total, 0 up, 0 in 2024-09-13T12:55:33.865 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:33 smithi052 ceph-mon[29471]: mgrmap e1: no daemons active 2024-09-13T12:55:33.865 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:33 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/434608764' entity='client.admin' 2024-09-13T12:55:33.866 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-13T12:55:33.896 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-13T12:55:33.896 INFO:teuthology.orchestra.run.smithi052.stdout:Creating mgr... 2024-09-13T12:55:33.896 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-13T12:55:33.897 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-13T12:55:34.163 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mgr.y 2024-09-13T12:55:34.163 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-13T12:55:34.320 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2b415b38-71cf-11ef-bceb-c7b262605968.target.wants/ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-2b415b38-71cf-11ef-bceb-c7b262605968@.service. 2024-09-13T12:55:34.503 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:34 smithi052 systemd[1]: Starting Ceph mgr.y for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T12:55:34.818 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:34 smithi052 podman[29682]: 2024-09-13 12:55:34.455669488 +0000 UTC m=+0.017165801 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b 2024-09-13T12:55:34.818 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:34 smithi052 podman[29682]: 2024-09-13 12:55:34.573673156 +0000 UTC m=+0.135169515 container create e79c228297b0de198d0fe51dd2076c97da9e42cff564972fee130c88f6d1d9ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-13T12:55:34.818 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:34 smithi052 podman[29682]: 2024-09-13 12:55:34.742412836 +0000 UTC m=+0.303909136 container init e79c228297b0de198d0fe51dd2076c97da9e42cff564972fee130c88f6d1d9ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-13T12:55:34.818 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:34 smithi052 podman[29682]: 2024-09-13 12:55:34.745347362 +0000 UTC m=+0.306843663 container start e79c228297b0de198d0fe51dd2076c97da9e42cff564972fee130c88f6d1d9ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8) 2024-09-13T12:55:34.818 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:34 smithi052 bash[29682]: e79c228297b0de198d0fe51dd2076c97da9e42cff564972fee130c88f6d1d9ea 2024-09-13T12:55:34.832 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T12:55:34.832 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-09-13T12:55:34.840 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T12:55:34.840 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-09-13T12:55:34.840 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-09-13T12:55:34.840 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T12:55:34.849 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T12:55:34.849 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-09-13T12:55:34.857 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T12:55:34.857 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-09-13T12:55:34.857 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-09-13T12:55:34.857 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-13T12:55:34.857 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr to start... 2024-09-13T12:55:34.857 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr... 2024-09-13T12:55:35.069 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:34 smithi052 systemd[1]: Started Ceph mgr.y for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T12:55:35.632 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:35 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1136564345' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-13T12:55:35.913 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-09-13T12:55:35.913 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-09-13T12:55:35.913 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "2b415b38-71cf-11ef-bceb-c7b262605968", 2024-09-13T12:55:35.914 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T12:55:35.914 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T12:55:35.914 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T12:55:35.914 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T12:55:35.914 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:35.914 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T12:55:35.914 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T12:55:35.914 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-09-13T12:55:35.914 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-09-13T12:55:35.914 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T12:55:35.914 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T12:55:35.915 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-09-13T12:55:35.915 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-13T12:55:35.915 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T12:55:35.915 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T12:55:35.915 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-13T12:55:35.915 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T12:55:35.915 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:35.915 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T12:55:35.915 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T12:55:35.916 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T12:55:35.916 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T12:55:35.916 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T12:55:35.916 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T12:55:35.916 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T12:55:35.916 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T12:55:35.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:35.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T12:55:35.919 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "btime": "2024-09-13T12:55:27:893730+0000", 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T12:55:35.920 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T12:55:27.894391+0000", 2024-09-13T12:55:35.921 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T12:55:35.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:35.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T12:55:35.922 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-09-13T12:55:35.922 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (1/15)... 2024-09-13T12:55:38.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4048842756' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-13T12:55:38.989 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-09-13T12:55:38.989 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "2b415b38-71cf-11ef-bceb-c7b262605968", 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-09-13T12:55:38.990 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-13T12:55:38.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T12:55:38.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T12:55:38.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-13T12:55:38.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T12:55:38.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:38.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T12:55:38.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T12:55:38.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T12:55:38.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T12:55:38.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T12:55:38.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T12:55:38.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T12:55:38.991 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T12:55:38.994 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "btime": "2024-09-13T12:55:27:893730+0000", 2024-09-13T12:55:38.995 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T12:55:38.996 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T12:55:38.996 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:38.996 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T12:55:38.996 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-13T12:55:38.996 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T12:55:38.996 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T12:55:38.996 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T12:55:38.996 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T12:55:38.996 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T12:55:38.996 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-09-13T12:55:38.996 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T12:55:38.996 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:38.996 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T12:55:38.997 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T12:55:38.997 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T12:55:27.894391+0000", 2024-09-13T12:55:38.997 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T12:55:38.997 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:38.997 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T12:55:38.997 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-09-13T12:55:38.997 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (2/15)... 2024-09-13T12:55:40.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: Activating manager daemon y 2024-09-13T12:55:40.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: mgrmap e2: y(active, starting, since 0.00333576s) 2024-09-13T12:55:40.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: from='mgr.14100 v1:172.21.15.52:0/935991963' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-13T12:55:40.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: from='mgr.14100 v1:172.21.15.52:0/935991963' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-13T12:55:40.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: from='mgr.14100 v1:172.21.15.52:0/935991963' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-13T12:55:40.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: from='mgr.14100 v1:172.21.15.52:0/935991963' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-13T12:55:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: from='mgr.14100 v1:172.21.15.52:0/935991963' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-13T12:55:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: from='mgr.14100 v1:172.21.15.52:0/935991963' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-13T12:55:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: from='mgr.14100 v1:172.21.15.52:0/935991963' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:55:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: from='mgr.14100 v1:172.21.15.52:0/935991963' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-13T12:55:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: Manager daemon y is now available 2024-09-13T12:55:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: from='mgr.14100 v1:172.21.15.52:0/935991963' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T12:55:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: from='mgr.14100 v1:172.21.15.52:0/935991963' entity='mgr.y' 2024-09-13T12:55:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: from='mgr.14100 v1:172.21.15.52:0/935991963' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T12:55:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: from='mgr.14100 v1:172.21.15.52:0/935991963' entity='mgr.y' 2024-09-13T12:55:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:40 smithi052 ceph-mon[29471]: from='mgr.14100 v1:172.21.15.52:0/935991963' entity='mgr.y' 2024-09-13T12:55:42.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-09-13T12:55:42.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-09-13T12:55:42.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "2b415b38-71cf-11ef-bceb-c7b262605968", 2024-09-13T12:55:42.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T12:55:42.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T12:55:42.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T12:55:42.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T12:55:42.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:42.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T12:55:42.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T12:55:42.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-09-13T12:55:42.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-09-13T12:55:42.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T12:55:42.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T12:55:42.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-09-13T12:55:42.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-13T12:55:42.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T12:55:42.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T12:55:42.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-13T12:55:42.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T12:55:42.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:42.211 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T12:55:42.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T12:55:42.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T12:55:42.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T12:55:42.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T12:55:42.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T12:55:42.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T12:55:42.212 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T12:55:42.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:42.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T12:55:42.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T12:55:42.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T12:55:42.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T12:55:42.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T12:55:42.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T12:55:42.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T12:55:42.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T12:55:42.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T12:55:42.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:42.216 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T12:55:42.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T12:55:42.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "btime": "2024-09-13T12:55:27:893730+0000", 2024-09-13T12:55:42.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T12:55:42.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T12:55:42.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:42.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T12:55:42.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T12:55:42.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T12:55:42.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T12:55:42.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T12:55:42.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T12:55:42.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T12:55:42.218 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-09-13T12:55:42.218 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T12:55:42.218 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:42.218 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T12:55:42.218 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T12:55:42.218 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T12:55:27.894391+0000", 2024-09-13T12:55:42.218 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T12:55:42.218 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-09-13T12:55:42.218 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T12:55:42.218 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-09-13T12:55:42.218 INFO:teuthology.orchestra.run.smithi052.stdout:mgr is available 2024-09-13T12:55:42.238 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:41 smithi052 ceph-mon[29471]: mgrmap e3: y(active, since 1.00744s) 2024-09-13T12:55:42.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:41 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1465711996' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-13T12:55:43.238 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:42 smithi052 ceph-mon[29471]: mgrmap e4: y(active, since 2s) 2024-09-13T12:55:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1582372844' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-13T12:55:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-09-13T12:55:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T12:55:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout fsid = 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:55:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-13T12:55:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.52:6789] 2024-09-13T12:55:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T12:55:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T12:55:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T12:55:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T12:55:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-09-13T12:55:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T12:55:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T12:55:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-09-13T12:55:43.324 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T12:55:43.324 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T12:55:43.324 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T12:55:43.324 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling cephadm module... 2024-09-13T12:55:44.238 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3304456642' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-13T12:55:44.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:44 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ignoring --setuser ceph since I am not root 2024-09-13T12:55:44.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:44 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ignoring --setgroup ceph since I am not root 2024-09-13T12:55:45.238 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3304456642' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-13T12:55:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:44 smithi052 ceph-mon[29471]: mgrmap e5: y(active, since 4s) 2024-09-13T12:55:45.892 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-09-13T12:55:45.892 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-13T12:55:45.892 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T12:55:45.893 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-13T12:55:45.893 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T12:55:45.893 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-09-13T12:55:45.893 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for the mgr to restart... 2024-09-13T12:55:45.893 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr epoch 5... 2024-09-13T12:55:46.147 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:45 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/758436236' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-13T12:55:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:49 smithi052 ceph-mon[29471]: Active manager daemon y restarted 2024-09-13T12:55:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:49 smithi052 ceph-mon[29471]: Activating manager daemon y 2024-09-13T12:55:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:49 smithi052 ceph-mon[29471]: osdmap e2: 0 total, 0 up, 0 in 2024-09-13T12:55:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:49 smithi052 ceph-mon[29471]: mgrmap e6: y(active, starting, since 0.00366073s) 2024-09-13T12:55:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:49 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:55:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:49 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-13T12:55:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:49 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-13T12:55:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:49 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-13T12:55:49.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:49 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-13T12:55:49.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:49 smithi052 ceph-mon[29471]: Manager daemon y is now available 2024-09-13T12:55:50.807 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-09-13T12:55:50.807 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-13T12:55:50.807 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T12:55:50.808 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-09-13T12:55:50.808 INFO:teuthology.orchestra.run.smithi052.stdout:mgr epoch 5 is available 2024-09-13T12:55:50.808 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying orchestrator module is enabled... 2024-09-13T12:55:51.395 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' 2024-09-13T12:55:51.395 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' 2024-09-13T12:55:51.395 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-mon[29471]: Found migration_current of "None". Setting to last migration. 2024-09-13T12:55:51.395 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' 2024-09-13T12:55:51.396 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' 2024-09-13T12:55:51.396 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:55:51.396 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:55:51.396 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T12:55:51.396 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T12:55:51.396 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-mon[29471]: from='client.14122 v1:172.21.15.52:0/3943388542' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-13T12:55:51.396 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-mon[29471]: mgrmap e7: y(active, since 1.0061s) 2024-09-13T12:55:51.396 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-mon[29471]: from='client.14122 v1:172.21.15.52:0/3943388542' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-13T12:55:51.739 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: 2024-09-13T12:55:51.393+0000 7fae2a9d7640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:12:55:51] ENGINE Error in HTTPServer.serve 2024-09-13T12:55:51.739 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: Traceback (most recent call last): 2024-09-13T12:55:51.739 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T12:55:51.739 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: self._connections.run(self.expiration_interval) 2024-09-13T12:55:51.739 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T12:55:51.739 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: self._run(expiration_interval) 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: s = self.context.wrap_socket( 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: return self.sslsocket_class._create( 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: self.do_handshake() 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: self._sslobj.do_handshake() 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T12:55:51.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: 2024-09-13T12:55:52.369 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:52 smithi052 ceph-mon[29471]: [13/Sep/2024:12:55:51] ENGINE Bus STARTING 2024-09-13T12:55:52.369 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:52 smithi052 ceph-mon[29471]: [13/Sep/2024:12:55:51] ENGINE Serving on https://172.21.15.52:7150 2024-09-13T12:55:52.369 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:52 smithi052 ceph-mon[29471]: [13/Sep/2024:12:55:51] ENGINE Error in HTTPServer.serve 2024-09-13T12:55:52.369 INFO:journalctl@ceph.mon.a.smithi052.stdout: Traceback (most recent call last): 2024-09-13T12:55:52.369 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T12:55:52.369 INFO:journalctl@ceph.mon.a.smithi052.stdout: self._connections.run(self.expiration_interval) 2024-09-13T12:55:52.369 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T12:55:52.369 INFO:journalctl@ceph.mon.a.smithi052.stdout: self._run(expiration_interval) 2024-09-13T12:55:52.370 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T12:55:52.370 INFO:journalctl@ceph.mon.a.smithi052.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T12:55:52.370 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T12:55:52.370 INFO:journalctl@ceph.mon.a.smithi052.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T12:55:52.370 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T12:55:52.370 INFO:journalctl@ceph.mon.a.smithi052.stdout: s = self.context.wrap_socket( 2024-09-13T12:55:52.370 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T12:55:52.370 INFO:journalctl@ceph.mon.a.smithi052.stdout: return self.sslsocket_class._create( 2024-09-13T12:55:52.370 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T12:55:52.370 INFO:journalctl@ceph.mon.a.smithi052.stdout: self.do_handshake() 2024-09-13T12:55:52.370 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T12:55:52.370 INFO:journalctl@ceph.mon.a.smithi052.stdout: self._sslobj.do_handshake() 2024-09-13T12:55:52.370 INFO:journalctl@ceph.mon.a.smithi052.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T12:55:52.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:52 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2275125862' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-13T12:55:52.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:52 smithi052 ceph-mon[29471]: [13/Sep/2024:12:55:51] ENGINE Serving on http://172.21.15.52:8765 2024-09-13T12:55:52.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:52 smithi052 ceph-mon[29471]: [13/Sep/2024:12:55:51] ENGINE Bus STARTED 2024-09-13T12:55:52.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:52 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:55:52.631 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-13T12:55:52.631 INFO:teuthology.orchestra.run.smithi052.stdout:Setting orchestrator backend to cephadm... 2024-09-13T12:55:53.238 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2275125862' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-13T12:55:53.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:53 smithi052 ceph-mon[29471]: mgrmap e8: y(active, since 2s) 2024-09-13T12:55:54.434 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:54 smithi052 ceph-mon[29471]: from='client.14132 v1:172.21.15.52:0/4159186946' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:55:54.435 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:54 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' 2024-09-13T12:55:54.435 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:54 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:55:54.950 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-13T12:55:54.950 INFO:teuthology.orchestra.run.smithi052.stdout:Generating ssh key... 2024-09-13T12:55:55.702 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-mon[29471]: from='client.14134 v1:172.21.15.52:0/3670281422' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:55:55.979 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: Generating public/private rsa key pair. 2024-09-13T12:55:55.979 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: Your identification has been saved in /tmp/tmpr23fxpex/key 2024-09-13T12:55:55.979 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: Your public key has been saved in /tmp/tmpr23fxpex/key.pub 2024-09-13T12:55:55.979 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: The key fingerprint is: 2024-09-13T12:55:55.979 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: SHA256:K0C5KRQWLbZ/rSLp06y2Rshqkd1GFpx9S2m0um3RkTA ceph-2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:55:55.979 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: The key's randomart image is: 2024-09-13T12:55:55.979 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: +---[RSA 3072]----+ 2024-09-13T12:55:55.979 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: | +o. o .E. | 2024-09-13T12:55:55.979 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: | .o..= . =+ . | 2024-09-13T12:55:55.979 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: | ..oo . +..o | 2024-09-13T12:55:55.980 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: | ... = ... . | 2024-09-13T12:55:55.980 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: |..+.B oS. . | 2024-09-13T12:55:55.980 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: |.+.o.+. +.. | 2024-09-13T12:55:55.980 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: |...+...o.o | 2024-09-13T12:55:55.980 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: |..* + ... | 2024-09-13T12:55:55.980 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: |.++= . | 2024-09-13T12:55:55.980 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:55:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: +----[SHA256]-----+ 2024-09-13T12:55:56.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:56 smithi052 ceph-mon[29471]: from='client.14136 v1:172.21.15.52:0/715016816' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:55:56.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:56 smithi052 ceph-mon[29471]: Generating ssh key... 2024-09-13T12:55:56.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:56 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' 2024-09-13T12:55:56.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:56 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' 2024-09-13T12:55:57.355 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCf+Mldulp0z2V3/U7+MkVsxW0hM+z8qPOElkNS9dAVwm0pahzf1GpKv899EkyCitvAnLKHxADFhwfP/hTiP+OrfE8aulrVsdv5eBk+gbFiY6p4X7SIxmODFgSlTfShROqDMZ/0DP8EktS2bG2SBi7Z+nZIYfm/y0y0i2+holqowBnmGbUgERKuy3a8PvW9vmyO1TajaNt/KMcTInaIbn/6YeOZB+IeTkz+NolieEu0cIqnInkbROOjwmfCw1hTcW9Ttp4pl3wx5zuDdV9OY8r88/aPueD4ii5YUp225gpurJmSx+uefIvHTSTlQVN0telGBkRKMxVfXJQiPQtDLn4E/Bk8bNOF8iegOGk4y2JIpbDWQHI0M1qGYVY5Z34rU+K6SNstY+TplH9H+kj6Pqon71hu970tKjIqNW6sTDVaUrsSkqA8ormzvwOzUuYfg2lToFm+zfQtddd3fgUPz+LSYqs/ys4Np/G/aE0YfUYjqzywqKSMyJDY+6AYdXwu2TE= ceph-2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:55:57.355 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-13T12:55:57.356 INFO:teuthology.orchestra.run.smithi052.stdout:Adding key to root@localhost authorized_keys... 2024-09-13T12:55:57.366 INFO:teuthology.orchestra.run.smithi052.stdout:Adding host smithi052... 2024-09-13T12:55:57.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:57 smithi052 ceph-mon[29471]: from='client.14138 v1:172.21.15.52:0/2367681065' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:55:58.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:55:58 smithi052 ceph-mon[29471]: from='client.14140 v1:172.21.15.52:0/664332918' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi052", "addr": "172.21.15.52", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:56:00.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:00 smithi052 ceph-mon[29471]: Deploying cephadm binary to smithi052 2024-09-13T12:56:02.330 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Added host 'smithi052' with addr '172.21.15.52' 2024-09-13T12:56:02.330 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying unmanaged mon service... 2024-09-13T12:56:02.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:02 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' 2024-09-13T12:56:02.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:02 smithi052 ceph-mon[29471]: Added host smithi052 2024-09-13T12:56:02.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:02 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:56:04.206 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:03 smithi052 ceph-mon[29471]: from='client.14142 v1:172.21.15.52:0/2980730926' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:56:04.207 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:03 smithi052 ceph-mon[29471]: Saving service mon spec with placement count:5 2024-09-13T12:56:04.207 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:03 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' 2024-09-13T12:56:04.645 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-13T12:56:04.645 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying unmanaged mgr service... 2024-09-13T12:56:05.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:05 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' 2024-09-13T12:56:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:05 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' 2024-09-13T12:56:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:05 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' 2024-09-13T12:56:05.760 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-13T12:56:06.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:06 smithi052 ceph-mon[29471]: from='client.14144 v1:172.21.15.52:0/3397826976' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:56:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:06 smithi052 ceph-mon[29471]: Saving service mgr spec with placement count:2 2024-09-13T12:56:07.557 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:07 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2417375319' entity='client.admin' 2024-09-13T12:56:08.073 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling the dashboard module... 2024-09-13T12:56:08.642 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3013445866' entity='client.admin' 2024-09-13T12:56:09.863 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:09 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4056051240' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-13T12:56:09.864 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:09 smithi052 ceph-mon[29471]: from='mgr.14118 v1:172.21.15.52:0/2251712203' entity='mgr.y' 2024-09-13T12:56:09.864 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:09 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ignoring --setuser ceph since I am not root 2024-09-13T12:56:09.864 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:09 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ignoring --setgroup ceph since I am not root 2024-09-13T12:56:10.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:10 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4056051240' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-13T12:56:10.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:10 smithi052 ceph-mon[29471]: mgrmap e9: y(active, since 20s) 2024-09-13T12:56:11.322 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-09-13T12:56:11.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-13T12:56:11.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T12:56:11.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-13T12:56:11.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T12:56:11.323 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-09-13T12:56:11.323 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for the mgr to restart... 2024-09-13T12:56:11.324 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr epoch 9... 2024-09-13T12:56:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1945528737' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-13T12:56:15.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:14 smithi052 ceph-mon[29471]: Active manager daemon y restarted 2024-09-13T12:56:15.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:14 smithi052 ceph-mon[29471]: Activating manager daemon y 2024-09-13T12:56:15.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:14 smithi052 ceph-mon[29471]: osdmap e3: 0 total, 0 up, 0 in 2024-09-13T12:56:15.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:14 smithi052 ceph-mon[29471]: mgrmap e10: y(active, starting, since 0.00322361s) 2024-09-13T12:56:15.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:14 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:56:15.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:14 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-13T12:56:15.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:14 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-13T12:56:15.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:14 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-13T12:56:15.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:14 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-13T12:56:15.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:14 smithi052 ceph-mon[29471]: Manager daemon y is now available 2024-09-13T12:56:16.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:15 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:56:16.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:15 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T12:56:16.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:15 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T12:56:16.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:15 smithi052 ceph-mon[29471]: mgrmap e11: y(active, since 1.00635s) 2024-09-13T12:56:16.680 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-09-13T12:56:16.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-13T12:56:16.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T12:56:16.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-09-13T12:56:16.681 INFO:teuthology.orchestra.run.smithi052.stdout:mgr epoch 9 is available 2024-09-13T12:56:16.681 INFO:teuthology.orchestra.run.smithi052.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-13T12:56:16.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-mon[29471]: [13/Sep/2024:12:56:16] ENGINE Bus STARTING 2024-09-13T12:56:16.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: 2024-09-13T12:56:16.765+0000 7fde6d483640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:12:56:16] ENGINE Error in HTTPServer.serve 2024-09-13T12:56:16.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: Traceback (most recent call last): 2024-09-13T12:56:16.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T12:56:16.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: self._connections.run(self.expiration_interval) 2024-09-13T12:56:16.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T12:56:16.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: self._run(expiration_interval) 2024-09-13T12:56:16.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T12:56:16.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T12:56:16.990 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T12:56:16.990 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T12:56:16.990 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T12:56:16.990 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: s = self.context.wrap_socket( 2024-09-13T12:56:16.990 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T12:56:16.990 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: return self.sslsocket_class._create( 2024-09-13T12:56:16.990 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T12:56:16.990 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: self.do_handshake() 2024-09-13T12:56:16.990 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T12:56:16.990 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: self._sslobj.do_handshake() 2024-09-13T12:56:16.990 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T12:56:16.990 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 12:56:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: 2024-09-13T12:56:18.125 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:17 smithi052 ceph-mon[29471]: [13/Sep/2024:12:56:16] ENGINE Serving on http://172.21.15.52:8765 2024-09-13T12:56:18.125 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:17 smithi052 ceph-mon[29471]: [13/Sep/2024:12:56:16] ENGINE Serving on https://172.21.15.52:7150 2024-09-13T12:56:18.125 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:17 smithi052 ceph-mon[29471]: [13/Sep/2024:12:56:16] ENGINE Bus STARTED 2024-09-13T12:56:18.125 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:17 smithi052 ceph-mon[29471]: [13/Sep/2024:12:56:16] ENGINE Error in HTTPServer.serve 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: Traceback (most recent call last): 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: self._connections.run(self.expiration_interval) 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: self._run(expiration_interval) 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: s = self.context.wrap_socket( 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: return self.sslsocket_class._create( 2024-09-13T12:56:18.126 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T12:56:18.127 INFO:journalctl@ceph.mon.a.smithi052.stdout: self.do_handshake() 2024-09-13T12:56:18.127 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T12:56:18.127 INFO:journalctl@ceph.mon.a.smithi052.stdout: self._sslobj.do_handshake() 2024-09-13T12:56:18.127 INFO:journalctl@ceph.mon.a.smithi052.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T12:56:18.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:17 smithi052 ceph-mon[29471]: mgrmap e12: y(active, since 2s) 2024-09-13T12:56:18.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:17 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:18.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:17 smithi052 ceph-mon[29471]: from='client.14164 v1:172.21.15.52:0/3147945907' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:56:18.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:17 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:19.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIFEDCCAvigAwIBAgIUZkheJjBi9TtnrBXunXaJaGHG7LowDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkxMzEyNTYxOFoXDTM0\nMDkxNDEyNTYxOFowFzEVMBMGA1UEAwwMMTcyLjIxLjE1LjUyMIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEAroD5s1TnIOPceakx23kqK6wZpkVlPrwwQfGn\nNLAVCUl6wn11Ik3KfOfzSbH/oXk5v4jvRdpLwcSys/wi5ekagt8iR9uD64aYRjP/\noWijqAYMQn6Nc1Rp1nUCexLZN5MwPFYmj05rhLKdCYRyLaXbygZ7q/B6hka0DXDp\nv1TgkguqLtmm17jtDe1J9tUdqHPOfq1UjdyjUUdK9Z0H+SRM9BVhGbGO/L4tXm9M\nRKu6guEC6u520+udMvI8K7DEANjPqMBkwqydOvnx8WWaFycThEVgt8UG+4EBxkp6\nktch55W8fn6qzjPj6iOIQe0hD2YkKa3+Ci9psIC9daXSHncApDM3vXaYexkLAWsP\nIp08huKIoBFQ6jUpzKND8EK1GOLGaL/s7RT6UYcMqr+ijnGRLiL4Ep1eT/HcvOyp\n7CU9v3qgfUdFgXJ27SB7vREF++T6DDXtnH7zkkujWu0X2ybqY+zU2FH3+3sNgqFQ\n9t1MeY5rVm3ksFdyv5JnTIN1yHOdXJd+B44y3srcNDFbdwQ5zHvX1dPygB9pTHnO\nwQLNICoS79WIZrsBk+Q9A9z6gj+5+QgxVysCt8hrytFRSdIUBvWOHIY26rcl4s00\nJpmNHapfVRr0Qs7U52xwr6jt8NA6NBwOHiJUdHhwIsC/GlfxMf4gwlU4H+q4SjgS\n2vcPNVkCAwEAAaNUMFIwQgYDVR0RBDswOYIec21pdGhpMDUyLmZyb250LnNlcGlh\nLmNlcGguY29tghFkYXNoYm9hcmRfc2VydmVyc4cErBUPNDAMBgNVHRMBAf8EAjAA\nMA0GCSqGSIb3DQEBCwUAA4ICAQAg0baq3vt9rhMuFIKocxj9qcrT/ygv/qaLdhWA\nPbJmH1bpKcVPVIaUP0f0hhc2v/o1uDWxhh4b1Wcig92vs37/LmzBuE5LQc8U6EBq\n+Y/ZIhN2gtmzU6YOJQgX/1ZGM6qk/DbF9ABHWcPdXAf3Px3STl+lg/8/p4w9ZqxN\nFJVAXRuHlMt7eQwxmt4RV+GKaYqzEvoPDesh/BE5rFBT3/Y0CjB2nc4idAEJdxYx\nlWlySQqFix8rRdVLajAJp6xq40+ti9VKAxEf0zmG2BwuBEKEpkf107K9SsfpPZXs\nYpyxQA92cw6WGm3mE1peOua3awE/2s9JabHZDyM2AGFMgUeEAYchM4G8FcIP9MKS\nN+k7P9Q5kKsgDpCeNnS9UjQTLOLkIfDfFcWV5ck4E6j/MfYfZiVQbwOjRh4ao5ch\nm6KsWn2b05yK7dVCrkSJi7nnI6Oy7KeSuCYe+HsMyq9DPb5YkcQI6sn1nGhP+NvB\nvRlgd9RT10db8zx6y0Ia5dZic71nXr+R42qlCGdGAvW3OMmwVccqwgeYR1sC3Rb2\nE96nRLX5T2+K5skSLR6Oi8xbTXOFe/VqomAjagD1Qpsc8d/VkPaXze23kLHDK9Hf\n9cZ+B6UU0i7PLyt0/yL/+gxNGlKjZNbNKiKyrZsrbjKqkmyA+ZzRIjC6QujxjmSq\n/K/E6g==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKQIBAAKCAgEAroD5s1TnIOPceakx23kqK6wZpkVlPrwwQfGnNLAVCUl6wn11\nIk3KfOfzSbH/oXk5v4jvRdpLwcSys/wi5ekagt8iR9uD64aYRjP/oWijqAYMQn6N\nc1Rp1nUCexLZN5MwPFYmj05rhLKdCYRyLaXbygZ7q/B6hka0DXDpv1TgkguqLtmm\n17jtDe1J9tUdqHPOfq1UjdyjUUdK9Z0H+SRM9BVhGbGO/L4tXm9MRKu6guEC6u52\n0+udMvI8K7DEANjPqMBkwqydOvnx8WWaFycThEVgt8UG+4EBxkp6ktch55W8fn6q\nzjPj6iOIQe0hD2YkKa3+Ci9psIC9daXSHncApDM3vXaYexkLAWsPIp08huKIoBFQ\n6jUpzKND8EK1GOLGaL/s7RT6UYcMqr+ijnGRLiL4Ep1eT/HcvOyp7CU9v3qgfUdF\ngXJ27SB7vREF++T6DDXtnH7zkkujWu0X2ybqY+zU2FH3+3sNgqFQ9t1MeY5rVm3k\nsFdyv5JnTIN1yHOdXJd+B44y3srcNDFbdwQ5zHvX1dPygB9pTHnOwQLNICoS79WI\nZrsBk+Q9A9z6gj+5+QgxVysCt8hrytFRSdIUBvWOHIY26rcl4s00JpmNHapfVRr0\nQs7U52xwr6jt8NA6NBwOHiJUdHhwIsC/GlfxMf4gwlU4H+q4SjgS2vcPNVkCAwEA\nAQKCAgAvNToX/yyF087l5NVVk/pKFUVdG2Hv7lY5yMecs9QFDqlUqz8AeD4Cu3cw\nJSTMSLDMNZ3RbmRXN8DGezvvqbef1wB9ow+VrHRObavGvTmk73nqlTTsEdaOC1qP\n9AHqjHmcoQvOcf8ZR36/P+LpoXiLt2RGmAh1gfQcvMKnna0hJWG1SfFeY6tRvLNp\nQK9Ezot67qY0uI9AIkLPnls1VdIxqAT8P/I4QrSgWwitfS24UWoKJ1fjLdb5nNmV\nUj08I2tOMjowBiWE+nAvB3b/s3UOr/whyv4AFR7nR2fQiL2l2nbDW74WiagrQ9CN\nUTrD4TSASA81Ly1u7+0RQjpNPpFf5/ZQ2IfzAGxIVBWswGxFkJQjD+dNwn9Ylsuw\n7elg4F445K3NmzPlUEyL/ckIgnRqShk00iv+RuSmlDUlBmnXjHSGQ+gXcZFDH60B\nn3Xsg6/5QOi7PIeIdgkXb1AuCwEcR1pSpfL8jOkJaBxGJODzJ7tUI3qBQYhFTxfx\nxPDgyStMFpimdqAP8YWo4O1Xhh53IPBSu3uCNLhHNhSiMT5KcJRp5PI51ejztnPD\nOzdPwwO9C6KzT298W0faRs6LCiz5zmOd1u2hJKffmk4LVJraFKT/A1NtgcvN3rZ7\ngDV0jPhRFCAAo7o9xSBl6QS5coLJBk8JYA4/aTv5yfOpICElTwKCAQEA3VQInEDE\nBb7+QB8RBbrGUzmS6gD5/wVPOfeVZ0LToH1AZrW01dt1eh/wbyM0Skgn/z0VHjS6\nU8mH4rShUrilNGjne8lN0gBToyugIe9ipP+PuBSh03F28ugU4D7ivau0z0Zho76k\nivAYL3CO/yxEqSJ+lr4erLjl5GLCGRA4EEXaVORoOKs/gN+zLSols9Mkrdh1FbYa\n8zsKleGCb4DBFb/ckl0h0Afs6FWuyHdfmNmtBjHOEtFxf7ZjumgQBQz/PcOzYu0A\noE9mBlen14CFdIVT7o/uqgX2dsTr3P9Lq9ckg3MQkzCWhUfYHXiYURHJcOFEiUzo\nVyotLYr0lAqYdwKCAQEAydch9wskvjeMYq4n9O2e2K3Bkig6Zy0z5FwPwAskO57i\n++Rp98iZd+2ywC811PCD61pDWvr/P8m70QkbUt87lM0t7WmDJCkND9U8/U2RrqIv\nJHY8Cir1h4Gww7nIouX+i8xksycs0fx5sp6RBGj3H0BA8GNplx/WNYatEkWoRRN3\nsTN09yl0trn/ltk5hqno5eUpOiZWKrysHUvETHeOcvfrvokxNNW38blvRe4JUEUP\nYR7C15gqBctVxHGvjoUXibzoH3LGvsBOF54hV2QuHkkiOjJXsU37dOsr6j2hjEAH\n24e+cYFePn+vMZRBn+ONTFXtYabThSdVtXhelPHkrwKCAQEArmAw0HqudiB+TKKt\nVKhM7PElcLwIOwZB31fTHCIoff9wp/ttWDipm+hiFA3WetfSu2OyqbO9QZo8Nz89\nAuF7N4mXJ434hAXB34LVjSb/vyOiZdPLk72Hi+WR3QNuvST1NYEsGEhxdw2m4Gl8\nIEsOZXtJ5RYxMMAUUBYG+pwur5N5avzxqAQz6iNVFbZpMf6enWE3Dj4FibDwrBma\nHKOCfNJGxAIgS7iSzJIFPOfgJ5sImSLQSz88fvSCKjEmvfP5fP4c7Unw/lzP8xtI\nn58ghictc31u+gHsprKLafix89dzayVS38SeMq/h84Y561VqMkcpKkuvmk0efw1F\nmQWoDQKCAQB2Nvo9ptkuT3p5kWb1nPacEah94fHS3Da4WPMgl2XxRW3L+kvWRDce\ndqlqxjvRK80X2xX4A2ncAltuYtcOApDy5fkwQ8v7Mc/kpg/Ci9v3XCYgRN+FVQPZ\nCaxt9jH8tpgPaPbRtvAZTLY6V4S4AII8W3WtVYxSZuMfFnHnccTMMEq6DllDKZSi\nMO3xe+ilZn5xp3YuCKYPAdIMloOyBZP44hvAzLl1KHdDrEfvvSrXSJtpVornmseB\n/UWOeMdJwcm3clzBjoe40Qjr7EG63YHlWxAg5btWrUXJhLODzfUAH8B6Y1a3Q/3e\nvQnvD9YfvzbSrsEJoJviMNeSIZ3DvqkhAoIBAQDC7ZtsHfU37Rj0zcqy/rBTlme8\nmZrZZpXiKBZk+AtbQhkvuoaoPO0VaqvlRuvvrS4mVTK7qymZp/qF4Yrs6NdPrioL\n7PbSRGKGWyiY/XXQRS5qMUdYDwHkwcm5qVXj8QJ7pqDGIljipOwgjZYrzXj3Sjpg\nOUPJSV2EMi1gUuYnIDFw5zag3c/resAsx+EhSZfwSOdh3GmtDiGf1o7tF7jdLXD0\nTDqtCpM/u7ZFV3uREqsRodos0qJ5WBaiyPJGXLvNVBr3pexqOHjhjw7+GnjN2fj6\nA0cx2s8Nw9HeYCuQheuLvACzpZEytZKv7kb9TEGkD8BOdfXypVcUgDrY50jV\n-----END RSA PRIVATE KEY-----\n"} 2024-09-13T12:56:20.818 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-13T12:56:21.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:21 smithi052 ceph-mon[29471]: from='client.14166 v1:172.21.15.52:0/3212096414' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:56:21.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:21 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:21.974 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-13T12:56:21.974 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial admin user... 2024-09-13T12:56:22.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:22 smithi052 ceph-mon[29471]: from='client.14168 v1:172.21.15.52:0/1134639676' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:56:22.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:22 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:23.422 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$7isVdMrg4Wamwiu1tSaIDeYH5hVTpFz/av7EZWD.7ZeZbYo6IBQVS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726232182, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-13T12:56:23.422 INFO:teuthology.orchestra.run.smithi052.stdout:Fetching dashboard port number... 2024-09-13T12:56:23.688 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:23 smithi052 ceph-mon[29471]: from='client.14170 v1:172.21.15.52:0/3943084789' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:56:23.688 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:23 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:24.525 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:24 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4184363479' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-13T12:56:24.529 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 8443 2024-09-13T12:56:24.540 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T12:56:24.540 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-09-13T12:56:24.548 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T12:56:24.548 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-09-13T12:56:24.548 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-09-13T12:56:24.548 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-13T12:56:24.550 INFO:teuthology.orchestra.run.smithi052.stdout:Ceph Dashboard is now available at: 2024-09-13T12:56:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:56:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: URL: https://smithi052.front.sepia.ceph.com:8443/ 2024-09-13T12:56:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: User: admin 2024-09-13T12:56:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: Password: 7wbsxrohto 2024-09-13T12:56:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:56:24.551 INFO:teuthology.orchestra.run.smithi052.stdout:Saving cluster configuration to /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config directory 2024-09-13T12:56:25.742 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-13T12:56:25.742 INFO:teuthology.orchestra.run.smithi052.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-13T12:56:25.742 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout:Or, if you are only running a single cluster on this host: 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout: ceph telemetry on 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout:For more information see: 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-13T12:56:25.743 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:56:25.744 INFO:teuthology.orchestra.run.smithi052.stdout:Bootstrap complete. 2024-09-13T12:56:25.784 INFO:tasks.cephadm:Fetching config... 2024-09-13T12:56:25.785 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:56:25.785 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-13T12:56:25.801 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-13T12:56:25.801 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:56:25.801 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-13T12:56:25.858 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-13T12:56:25.859 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:56:25.859 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-13T12:56:25.925 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-13T12:56:25.926 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:56:25.926 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-13T12:56:25.982 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-13T12:56:25.982 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCf+Mldulp0z2V3/U7+MkVsxW0hM+z8qPOElkNS9dAVwm0pahzf1GpKv899EkyCitvAnLKHxADFhwfP/hTiP+OrfE8aulrVsdv5eBk+gbFiY6p4X7SIxmODFgSlTfShROqDMZ/0DP8EktS2bG2SBi7Z+nZIYfm/y0y0i2+holqowBnmGbUgERKuy3a8PvW9vmyO1TajaNt/KMcTInaIbn/6YeOZB+IeTkz+NolieEu0cIqnInkbROOjwmfCw1hTcW9Ttp4pl3wx5zuDdV9OY8r88/aPueD4ii5YUp225gpurJmSx+uefIvHTSTlQVN0telGBkRKMxVfXJQiPQtDLn4E/Bk8bNOF8iegOGk4y2JIpbDWQHI0M1qGYVY5Z34rU+K6SNstY+TplH9H+kj6Pqon71hu970tKjIqNW6sTDVaUrsSkqA8ormzvwOzUuYfg2lToFm+zfQtddd3fgUPz+LSYqs/ys4Np/G/aE0YfUYjqzywqKSMyJDY+6AYdXwu2TE= ceph-2b415b38-71cf-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T12:56:26.060 INFO:teuthology.orchestra.run.smithi052.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCf+Mldulp0z2V3/U7+MkVsxW0hM+z8qPOElkNS9dAVwm0pahzf1GpKv899EkyCitvAnLKHxADFhwfP/hTiP+OrfE8aulrVsdv5eBk+gbFiY6p4X7SIxmODFgSlTfShROqDMZ/0DP8EktS2bG2SBi7Z+nZIYfm/y0y0i2+holqowBnmGbUgERKuy3a8PvW9vmyO1TajaNt/KMcTInaIbn/6YeOZB+IeTkz+NolieEu0cIqnInkbROOjwmfCw1hTcW9Ttp4pl3wx5zuDdV9OY8r88/aPueD4ii5YUp225gpurJmSx+uefIvHTSTlQVN0telGBkRKMxVfXJQiPQtDLn4E/Bk8bNOF8iegOGk4y2JIpbDWQHI0M1qGYVY5Z34rU+K6SNstY+TplH9H+kj6Pqon71hu970tKjIqNW6sTDVaUrsSkqA8ormzvwOzUuYfg2lToFm+zfQtddd3fgUPz+LSYqs/ys4Np/G/aE0YfUYjqzywqKSMyJDY+6AYdXwu2TE= ceph-2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:56:26.077 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCf+Mldulp0z2V3/U7+MkVsxW0hM+z8qPOElkNS9dAVwm0pahzf1GpKv899EkyCitvAnLKHxADFhwfP/hTiP+OrfE8aulrVsdv5eBk+gbFiY6p4X7SIxmODFgSlTfShROqDMZ/0DP8EktS2bG2SBi7Z+nZIYfm/y0y0i2+holqowBnmGbUgERKuy3a8PvW9vmyO1TajaNt/KMcTInaIbn/6YeOZB+IeTkz+NolieEu0cIqnInkbROOjwmfCw1hTcW9Ttp4pl3wx5zuDdV9OY8r88/aPueD4ii5YUp225gpurJmSx+uefIvHTSTlQVN0telGBkRKMxVfXJQiPQtDLn4E/Bk8bNOF8iegOGk4y2JIpbDWQHI0M1qGYVY5Z34rU+K6SNstY+TplH9H+kj6Pqon71hu970tKjIqNW6sTDVaUrsSkqA8ormzvwOzUuYfg2lToFm+zfQtddd3fgUPz+LSYqs/ys4Np/G/aE0YfUYjqzywqKSMyJDY+6AYdXwu2TE= ceph-2b415b38-71cf-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T12:56:26.175 INFO:teuthology.orchestra.run.smithi167.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCf+Mldulp0z2V3/U7+MkVsxW0hM+z8qPOElkNS9dAVwm0pahzf1GpKv899EkyCitvAnLKHxADFhwfP/hTiP+OrfE8aulrVsdv5eBk+gbFiY6p4X7SIxmODFgSlTfShROqDMZ/0DP8EktS2bG2SBi7Z+nZIYfm/y0y0i2+holqowBnmGbUgERKuy3a8PvW9vmyO1TajaNt/KMcTInaIbn/6YeOZB+IeTkz+NolieEu0cIqnInkbROOjwmfCw1hTcW9Ttp4pl3wx5zuDdV9OY8r88/aPueD4ii5YUp225gpurJmSx+uefIvHTSTlQVN0telGBkRKMxVfXJQiPQtDLn4E/Bk8bNOF8iegOGk4y2JIpbDWQHI0M1qGYVY5Z34rU+K6SNstY+TplH9H+kj6Pqon71hu970tKjIqNW6sTDVaUrsSkqA8ormzvwOzUuYfg2lToFm+zfQtddd3fgUPz+LSYqs/ys4Np/G/aE0YfUYjqzywqKSMyJDY+6AYdXwu2TE= ceph-2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:56:26.189 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-13T12:56:26.448 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:56:26.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:26 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3195192920' entity='client.admin' 2024-09-13T12:56:27.657 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-13T12:56:27.658 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-13T12:56:28.058 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:56:28.273 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:28 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/163843716' entity='client.admin' 2024-09-13T12:56:28.273 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:28 smithi052 ceph-mon[29471]: mgrmap e13: y(active, since 12s) 2024-09-13T12:56:28.273 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:28 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:28.273 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:28 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:28.273 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:28 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"} : dispatch 2024-09-13T12:56:28.273 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:28 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:29.389 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:29 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:56:29.389 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:29 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:29.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:29 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:29.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:29 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:56:29.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:29 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:56:29.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:29 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:56:29.416 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi167 2024-09-13T12:56:29.416 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-09-13T12:56:29.417 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/etc/ceph/ceph.conf 2024-09-13T12:56:29.435 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-09-13T12:56:29.436 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T12:56:29.494 INFO:tasks.cephadm:Adding host smithi167 to orchestrator... 2024-09-13T12:56:29.494 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch host add smithi167 2024-09-13T12:56:29.745 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:56:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:30 smithi052 ceph-mon[29471]: from='client.14178 v1:172.21.15.52:0/4023830078' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:56:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:30 smithi052 ceph-mon[29471]: Updating smithi052:/etc/ceph/ceph.conf 2024-09-13T12:56:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:30 smithi052 ceph-mon[29471]: Updating smithi052:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T12:56:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:30 smithi052 ceph-mon[29471]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-09-13T12:56:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:30 smithi052 ceph-mon[29471]: Updating smithi052:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T12:56:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:30 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:30 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:30 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:31.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:31 smithi052 ceph-mon[29471]: from='client.14180 v1:172.21.15.52:0/2172715187' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi167", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:56:32.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:32 smithi052 ceph-mon[29471]: Deploying cephadm binary to smithi167 2024-09-13T12:56:34.113 INFO:teuthology.orchestra.run.smithi052.stdout:Added host 'smithi167' with addr '172.21.15.167' 2024-09-13T12:56:34.680 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-13T12:56:34.934 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:56:35.340 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:35 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:35.340 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:35 smithi052 ceph-mon[29471]: Added host smithi167 2024-09-13T12:56:35.340 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:35 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:56:35.340 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:35 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:35.341 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:35 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:35.622 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:56:35.622 INFO:teuthology.orchestra.run.smithi052.stdout:[{"addr": "172.21.15.52", "hostname": "smithi052", "labels": [], "status": ""}, {"addr": "172.21.15.167", "hostname": "smithi167", "labels": [], "status": ""}] 2024-09-13T12:56:36.206 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-13T12:56:36.207 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-13T12:56:36.460 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:56:36.685 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:36 smithi052 ceph-mon[29471]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:36.686 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:36 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:36.686 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:36 smithi052 ceph-mon[29471]: from='client.14182 v1:172.21.15.52:0/1219482654' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T12:56:37.434 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2024-09-13T12:56:37.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:37 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3075472285' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-13T12:56:38.043 INFO:tasks.cephadm:Adding mon.a on smithi052 2024-09-13T12:56:38.043 INFO:tasks.cephadm:Adding mon.c on smithi052 2024-09-13T12:56:38.043 INFO:tasks.cephadm:Adding mon.b on smithi167 2024-09-13T12:56:38.044 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi052:[v1:172.21.15.52:6789]=a;smithi052:[v1:172.21.15.52:6790]=c;smithi167:[v1:172.21.15.167:6789]=b' 2024-09-13T12:56:38.375 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T12:56:38.421 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T12:56:38.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:38 smithi052 ceph-mon[29471]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3075472285' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-13T12:56:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:38 smithi052 ceph-mon[29471]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T12:56:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:38 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:38 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:38 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:38 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:38 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi167", "name": "osd_memory_target"} : dispatch 2024-09-13T12:56:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:38 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:56:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:38 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:56:39.137 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled mon update... 2024-09-13T12:56:39.694 DEBUG:teuthology.orchestra.run.smithi052:mon.c> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.c.service 2024-09-13T12:56:39.697 DEBUG:teuthology.orchestra.run.smithi167:mon.b> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.b.service 2024-09-13T12:56:39.699 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T12:56:39.700 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T12:56:39.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:39 smithi052 ceph-mon[29471]: Updating smithi167:/etc/ceph/ceph.conf 2024-09-13T12:56:39.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:39 smithi052 ceph-mon[29471]: Updating smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T12:56:39.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:39 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:39.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:39 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:39.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:39 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:39.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:39 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:39.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:39 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:56:39.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:39 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:56:39.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:39 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:56:39.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:39 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:39.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:39 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:56:39.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:39 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:56:40.347 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T12:56:40.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:40 smithi052 ceph-mon[29471]: Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2024-09-13T12:56:40.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:40 smithi052 ceph-mon[29471]: Updating smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T12:56:40.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:40 smithi052 ceph-mon[29471]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:40.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:40 smithi052 ceph-mon[29471]: from='client.14186 v1:172.21.15.167:0/2454574402' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:[v1:172.21.15.52:6789]=a;smithi052:[v1:172.21.15.52:6790]=c;smithi167:[v1:172.21.15.167:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:56:40.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:40 smithi052 ceph-mon[29471]: Saving service mon spec with placement smithi052:[v1:172.21.15.52:6789]=a;smithi052:[v1:172.21.15.52:6790]=c;smithi167:[v1:172.21.15.167:6789]=b;count:3 2024-09-13T12:56:40.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:40 smithi052 ceph-mon[29471]: Deploying daemon mon.b on smithi167 2024-09-13T12:56:40.992 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:56:40.992 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":1,"fsid":"2b415b38-71cf-11ef-bceb-c7b262605968","modified":"2024-09-13T12:55:24.138668Z","created":"2024-09-13T12:55:24.138668Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T12:56:40.992 INFO:teuthology.orchestra.run.smithi167.stderr:dumped monmap epoch 1 2024-09-13T12:56:41.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:41 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/27378577' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T12:56:42.633 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T12:56:42.633 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T12:56:42.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:42 smithi052 ceph-mon[29471]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:42.933 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T12:56:43.779 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:56:43.779 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":1,"fsid":"2b415b38-71cf-11ef-bceb-c7b262605968","modified":"2024-09-13T12:55:24.138668Z","created":"2024-09-13T12:55:24.138668Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T12:56:43.780 INFO:teuthology.orchestra.run.smithi167.stderr:dumped monmap epoch 1 2024-09-13T12:56:44.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:44 smithi052 ceph-mon[29471]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:44.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/2221745528' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T12:56:45.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:44 smithi167 systemd[1]: Started Ceph mon.b for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T12:56:45.342 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T12:56:45.342 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T12:56:45.599 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T12:56:49.965 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:56:49.965 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":2,"fsid":"2b415b38-71cf-11ef-bceb-c7b262605968","modified":"2024-09-13T12:56:44.683156Z","created":"2024-09-13T12:55:24.138668Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-13T12:56:49.965 INFO:teuthology.orchestra.run.smithi167.stderr:dumped monmap epoch 2 2024-09-13T12:56:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:56:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: mon.a calling monitor election 2024-09-13T12:56:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: mon.b calling monitor election 2024-09-13T12:56:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:50.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:50.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:50.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:50.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:50.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T12:56:50.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: monmap epoch 2 2024-09-13T12:56:50.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: fsid 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:56:50.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: last_changed 2024-09-13T12:56:44.683156+0000 2024-09-13T12:56:50.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: created 2024-09-13T12:55:24.138668+0000 2024-09-13T12:56:50.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: min_mon_release 19 (squid) 2024-09-13T12:56:50.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: election_strategy: 1 2024-09-13T12:56:50.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: 0: v1:172.21.15.52:6789/0 mon.a 2024-09-13T12:56:50.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: 1: v1:172.21.15.167:6789/0 mon.b 2024-09-13T12:56:50.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: fsmap 2024-09-13T12:56:50.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T12:56:50.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: mgrmap e13: y(active, since 34s) 2024-09-13T12:56:50.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: overall HEALTH_OK 2024-09-13T12:56:50.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:50.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:50.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:50.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:56:50.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:56:50.109 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:56:50.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:50.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: mon.a calling monitor election 2024-09-13T12:56:50.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:50.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:50.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:50.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: mon.b calling monitor election 2024-09-13T12:56:50.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:50.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:50.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:50.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:50.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:50.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T12:56:50.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: monmap epoch 2 2024-09-13T12:56:50.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: fsid 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:56:50.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: last_changed 2024-09-13T12:56:44.683156+0000 2024-09-13T12:56:50.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: created 2024-09-13T12:55:24.138668+0000 2024-09-13T12:56:50.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: min_mon_release 19 (squid) 2024-09-13T12:56:50.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: election_strategy: 1 2024-09-13T12:56:50.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: 0: v1:172.21.15.52:6789/0 mon.a 2024-09-13T12:56:50.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: 1: v1:172.21.15.167:6789/0 mon.b 2024-09-13T12:56:50.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: fsmap 2024-09-13T12:56:50.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T12:56:50.112 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: mgrmap e13: y(active, since 34s) 2024-09-13T12:56:50.112 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: overall HEALTH_OK 2024-09-13T12:56:50.112 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:50.112 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:50.112 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:50.112 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:56:50.112 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:56:50.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:50 smithi167 ceph-mon[33178]: Deploying daemon mon.c on smithi052 2024-09-13T12:56:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:50 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/1061599883' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T12:56:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:50 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:51.059 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:50 smithi052 ceph-mon[29471]: Deploying daemon mon.c on smithi052 2024-09-13T12:56:51.059 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:50 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/1061599883' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T12:56:51.059 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:50 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:51.533 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T12:56:51.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T12:56:51.791 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T12:56:51.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:51 smithi167 ceph-mon[33178]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:52.086 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:51 smithi052 ceph-mon[29471]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:52.438 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:56:52.438 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":2,"fsid":"2b415b38-71cf-11ef-bceb-c7b262605968","modified":"2024-09-13T12:56:44.683156Z","created":"2024-09-13T12:55:24.138668Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-13T12:56:52.438 INFO:teuthology.orchestra.run.smithi167.stderr:dumped monmap epoch 2 2024-09-13T12:56:52.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 systemd[1]: Starting Ceph mon.c for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T12:56:52.775 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/630321588' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T12:56:52.775 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 podman[36664]: 2024-09-13 12:56:52.416815854 +0000 UTC m=+0.016589404 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T12:56:52.775 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 podman[36664]: 2024-09-13 12:56:52.539143663 +0000 UTC m=+0.138917255 container create cf921afd375aaa9878bfd4307955fb2573c16fad07a9fd31ef0acebb62e60e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T12:56:52.775 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 podman[36664]: 2024-09-13 12:56:52.774743587 +0000 UTC m=+0.374517137 container init cf921afd375aaa9878bfd4307955fb2573c16fad07a9fd31ef0acebb62e60e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-c, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-13T12:56:53.004 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:52 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/630321588' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T12:56:53.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 podman[36664]: 2024-09-13 12:56:52.778229451 +0000 UTC m=+0.378003003 container start cf921afd375aaa9878bfd4307955fb2573c16fad07a9fd31ef0acebb62e60e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-13T12:56:53.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T12:56:53.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: ceph version 19.3.0-4917-gb0201b8c (b0201b8c79733293453e7f10a10c7fa43119222b) squid (dev), process ceph-mon, pid 2 2024-09-13T12:56:53.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: pidfile_write: ignore empty --pid-file 2024-09-13T12:56:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: load: jerasure load: lrc 2024-09-13T12:56:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: RocksDB version: 7.9.2 2024-09-13T12:56:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Git sha 0 2024-09-13T12:56:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Compile date 2024-09-13 10:06:52 2024-09-13T12:56:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: DB SUMMARY 2024-09-13T12:56:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: DB Session ID: 9HC4UIAQUSV064OEE557 2024-09-13T12:56:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: CURRENT file: CURRENT 2024-09-13T12:56:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T12:56:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-13T12:56:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-13T12:56:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-09-13T12:56:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.error_if_exists: 0 2024-09-13T12:56:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.create_if_missing: 0 2024-09-13T12:56:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.paranoid_checks: 1 2024-09-13T12:56:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T12:56:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T12:56:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T12:56:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.env: 0x56435e1f6120 2024-09-13T12:56:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T12:56:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.info_log: 0x56435fd5f6e0 2024-09-13T12:56:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T12:56:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.statistics: (nil) 2024-09-13T12:56:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.use_fsync: 0 2024-09-13T12:56:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_log_file_size: 0 2024-09-13T12:56:53.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T12:56:53.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T12:56:53.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T12:56:53.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T12:56:53.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.allow_fallocate: 1 2024-09-13T12:56:53.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T12:56:53.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T12:56:53.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.use_direct_reads: 0 2024-09-13T12:56:53.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T12:56:53.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T12:56:53.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.db_log_dir: 2024-09-13T12:56:53.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.wal_dir: 2024-09-13T12:56:53.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T12:56:53.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T12:56:53.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T12:56:53.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T12:56:53.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T12:56:53.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T12:56:53.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T12:56:53.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T12:56:53.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.write_buffer_manager: 0x56435fd63b80 2024-09-13T12:56:53.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T12:56:53.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T12:56:53.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T12:56:53.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T12:56:53.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T12:56:53.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T12:56:53.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T12:56:53.246 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T12:56:53.246 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.unordered_write: 0 2024-09-13T12:56:53.246 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T12:56:53.246 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T12:56:53.246 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T12:56:53.246 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T12:56:53.246 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.row_cache: None 2024-09-13T12:56:53.246 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.wal_filter: None 2024-09-13T12:56:53.246 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T12:56:53.246 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T12:56:53.247 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.two_write_queues: 0 2024-09-13T12:56:53.247 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T12:56:53.247 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.wal_compression: 0 2024-09-13T12:56:53.247 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.atomic_flush: 0 2024-09-13T12:56:53.247 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T12:56:53.247 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T12:56:53.247 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T12:56:53.247 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.log_readahead_size: 0 2024-09-13T12:56:53.247 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T12:56:53.247 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T12:56:53.247 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T12:56:53.247 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T12:56:53.248 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T12:56:53.248 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T12:56:53.248 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T12:56:53.248 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_background_jobs: 2 2024-09-13T12:56:53.248 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_background_compactions: -1 2024-09-13T12:56:53.248 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_subcompactions: 1 2024-09-13T12:56:53.248 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T12:56:53.248 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T12:56:53.248 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T12:56:53.248 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T12:56:53.248 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T12:56:53.248 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_open_files: -1 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_background_flushes: -1 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Compression algorithms supported: 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: kZSTD supported: 0 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: kXpressCompression supported: 0 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: kBZip2Compression supported: 0 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T12:56:53.249 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: kLZ4Compression supported: 1 2024-09-13T12:56:53.250 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: kZlibCompression supported: 1 2024-09-13T12:56:53.250 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T12:56:53.250 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: kSnappyCompression supported: 1 2024-09-13T12:56:53.250 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T12:56:53.250 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T12:56:53.250 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-13T12:56:53.250 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T12:56:53.250 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T12:56:53.250 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.merge_operator: 2024-09-13T12:56:53.250 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compaction_filter: None 2024-09-13T12:56:53.250 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T12:56:53.250 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T12:56:53.250 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T12:56:53.251 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T12:56:53.251 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56435fd5f300) 2024-09-13T12:56:53.251 INFO:journalctl@ceph.mon.c.smithi052.stdout: cache_index_and_filter_blocks: 1 2024-09-13T12:56:53.251 INFO:journalctl@ceph.mon.c.smithi052.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T12:56:53.251 INFO:journalctl@ceph.mon.c.smithi052.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T12:56:53.251 INFO:journalctl@ceph.mon.c.smithi052.stdout: pin_top_level_index_and_filter: 1 2024-09-13T12:56:53.251 INFO:journalctl@ceph.mon.c.smithi052.stdout: index_type: 0 2024-09-13T12:56:53.251 INFO:journalctl@ceph.mon.c.smithi052.stdout: data_block_index_type: 0 2024-09-13T12:56:53.251 INFO:journalctl@ceph.mon.c.smithi052.stdout: index_shortening: 1 2024-09-13T12:56:53.251 INFO:journalctl@ceph.mon.c.smithi052.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T12:56:53.251 INFO:journalctl@ceph.mon.c.smithi052.stdout: checksum: 4 2024-09-13T12:56:53.251 INFO:journalctl@ceph.mon.c.smithi052.stdout: no_block_cache: 0 2024-09-13T12:56:53.251 INFO:journalctl@ceph.mon.c.smithi052.stdout: block_cache: 0x56435fd53610 2024-09-13T12:56:53.252 INFO:journalctl@ceph.mon.c.smithi052.stdout: block_cache_name: BinnedLRUCache 2024-09-13T12:56:53.252 INFO:journalctl@ceph.mon.c.smithi052.stdout: block_cache_options: 2024-09-13T12:56:53.252 INFO:journalctl@ceph.mon.c.smithi052.stdout: capacity : 536870912 2024-09-13T12:56:53.252 INFO:journalctl@ceph.mon.c.smithi052.stdout: num_shard_bits : 4 2024-09-13T12:56:53.252 INFO:journalctl@ceph.mon.c.smithi052.stdout: strict_capacity_limit : 0 2024-09-13T12:56:53.252 INFO:journalctl@ceph.mon.c.smithi052.stdout: high_pri_pool_ratio: 0.000 2024-09-13T12:56:53.252 INFO:journalctl@ceph.mon.c.smithi052.stdout: block_cache_compressed: (nil) 2024-09-13T12:56:53.252 INFO:journalctl@ceph.mon.c.smithi052.stdout: persistent_cache: (nil) 2024-09-13T12:56:53.252 INFO:journalctl@ceph.mon.c.smithi052.stdout: block_size: 4096 2024-09-13T12:56:53.252 INFO:journalctl@ceph.mon.c.smithi052.stdout: block_size_deviation: 10 2024-09-13T12:56:53.252 INFO:journalctl@ceph.mon.c.smithi052.stdout: block_restart_interval: 16 2024-09-13T12:56:53.252 INFO:journalctl@ceph.mon.c.smithi052.stdout: index_block_restart_interval: 1 2024-09-13T12:56:53.253 INFO:journalctl@ceph.mon.c.smithi052.stdout: metadata_block_size: 4096 2024-09-13T12:56:53.253 INFO:journalctl@ceph.mon.c.smithi052.stdout: partition_filters: 0 2024-09-13T12:56:53.253 INFO:journalctl@ceph.mon.c.smithi052.stdout: use_delta_encoding: 1 2024-09-13T12:56:53.253 INFO:journalctl@ceph.mon.c.smithi052.stdout: filter_policy: bloomfilter 2024-09-13T12:56:53.253 INFO:journalctl@ceph.mon.c.smithi052.stdout: whole_key_filtering: 1 2024-09-13T12:56:53.253 INFO:journalctl@ceph.mon.c.smithi052.stdout: verify_compression: 0 2024-09-13T12:56:53.253 INFO:journalctl@ceph.mon.c.smithi052.stdout: read_amp_bytes_per_bit: 0 2024-09-13T12:56:53.253 INFO:journalctl@ceph.mon.c.smithi052.stdout: format_version: 5 2024-09-13T12:56:53.253 INFO:journalctl@ceph.mon.c.smithi052.stdout: enable_index_compression: 1 2024-09-13T12:56:53.253 INFO:journalctl@ceph.mon.c.smithi052.stdout: block_align: 0 2024-09-13T12:56:53.253 INFO:journalctl@ceph.mon.c.smithi052.stdout: max_auto_readahead_size: 262144 2024-09-13T12:56:53.253 INFO:journalctl@ceph.mon.c.smithi052.stdout: prepopulate_block_cache: 0 2024-09-13T12:56:53.253 INFO:journalctl@ceph.mon.c.smithi052.stdout: initial_auto_readahead_size: 8192 2024-09-13T12:56:53.254 INFO:journalctl@ceph.mon.c.smithi052.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T12:56:53.254 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T12:56:53.254 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T12:56:53.254 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compression: NoCompression 2024-09-13T12:56:53.254 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T12:56:53.254 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T12:56:53.254 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T12:56:53.254 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.num_levels: 7 2024-09-13T12:56:53.254 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T12:56:53.254 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T12:56:53.254 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T12:56:53.254 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T12:56:53.254 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T12:56:53.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T12:56:53.256 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T12:56:53.256 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T12:56:53.256 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T12:56:53.256 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T12:56:53.256 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T12:56:53.256 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T12:56:53.256 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T12:56:53.256 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T12:56:53.256 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T12:56:53.256 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T12:56:53.256 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T12:56:53.256 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T12:56:53.256 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T12:56:53.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T12:56:53.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T12:56:53.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T12:56:53.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T12:56:53.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T12:56:53.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T12:56:53.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T12:56:53.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T12:56:53.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T12:56:53.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T12:56:53.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T12:56:53.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T12:56:53.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T12:56:53.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T12:56:53.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T12:56:53.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T12:56:53.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T12:56:53.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T12:56:53.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T12:56:53.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T12:56:53.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T12:56:53.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T12:56:53.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.inplace_update_support: 0 2024-09-13T12:56:53.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.bloom_locality: 0 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.max_successive_merges: 0 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.ttl: 2592000 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T12:56:53.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.enable_blob_files: false 2024-09-13T12:56:53.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.min_blob_size: 0 2024-09-13T12:56:53.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T12:56:53.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T12:56:53.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T12:56:53.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T12:56:53.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T12:56:53.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T12:56:53.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T12:56:53.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T12:56:53.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T12:56:53.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-13T12:56:53.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4980d608-dd6c-4094-92fe-62111a280876 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726232212805354, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726232212805751, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1703, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726232212, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4980d608-dd6c-4094-92fe-62111a280876", "db_session_id": "9HC4UIAQUSV064OEE557", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726232212805850, "job": 1, "event": "recovery_finished"} 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56435fd82e00 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: DB pointer 0x56435fe84000 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout: ** DB Stats ** 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T12:56:53.261 INFO:journalctl@ceph.mon.c.smithi052.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T12:56:53.262 INFO:journalctl@ceph.mon.c.smithi052.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T12:56:53.262 INFO:journalctl@ceph.mon.c.smithi052.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T12:56:53.262 INFO:journalctl@ceph.mon.c.smithi052.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T12:56:53.262 INFO:journalctl@ceph.mon.c.smithi052.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T12:56:53.262 INFO:journalctl@ceph.mon.c.smithi052.stdout: 2024-09-13T12:56:53.262 INFO:journalctl@ceph.mon.c.smithi052.stdout: ** Compaction Stats [default] ** 2024-09-13T12:56:53.262 INFO:journalctl@ceph.mon.c.smithi052.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T12:56:53.262 INFO:journalctl@ceph.mon.c.smithi052.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T12:56:53.262 INFO:journalctl@ceph.mon.c.smithi052.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T12:56:53.262 INFO:journalctl@ceph.mon.c.smithi052.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T12:56:53.262 INFO:journalctl@ceph.mon.c.smithi052.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T12:56:53.262 INFO:journalctl@ceph.mon.c.smithi052.stdout: 2024-09-13T12:56:53.262 INFO:journalctl@ceph.mon.c.smithi052.stdout: ** Compaction Stats [default] ** 2024-09-13T12:56:53.263 INFO:journalctl@ceph.mon.c.smithi052.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T12:56:53.263 INFO:journalctl@ceph.mon.c.smithi052.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T12:56:53.263 INFO:journalctl@ceph.mon.c.smithi052.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T12:56:53.263 INFO:journalctl@ceph.mon.c.smithi052.stdout: 2024-09-13T12:56:53.263 INFO:journalctl@ceph.mon.c.smithi052.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T12:56:53.263 INFO:journalctl@ceph.mon.c.smithi052.stdout: 2024-09-13T12:56:53.263 INFO:journalctl@ceph.mon.c.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T12:56:53.263 INFO:journalctl@ceph.mon.c.smithi052.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T12:56:53.263 INFO:journalctl@ceph.mon.c.smithi052.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T12:56:53.263 INFO:journalctl@ceph.mon.c.smithi052.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T12:56:53.263 INFO:journalctl@ceph.mon.c.smithi052.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T12:56:53.263 INFO:journalctl@ceph.mon.c.smithi052.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T12:56:53.264 INFO:journalctl@ceph.mon.c.smithi052.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T12:56:53.264 INFO:journalctl@ceph.mon.c.smithi052.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T12:56:53.264 INFO:journalctl@ceph.mon.c.smithi052.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T12:56:53.264 INFO:journalctl@ceph.mon.c.smithi052.stdout: Block cache BinnedLRUCache@0x56435fd53610#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-09-13T12:56:53.264 INFO:journalctl@ceph.mon.c.smithi052.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-13T12:56:53.264 INFO:journalctl@ceph.mon.c.smithi052.stdout: 2024-09-13T12:56:53.264 INFO:journalctl@ceph.mon.c.smithi052.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T12:56:53.264 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-13T12:56:53.264 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: using public_addrv v1:172.21.15.52:6790/0 2024-09-13T12:56:53.264 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: starting mon.c rank -1 at public addrs v1:172.21.15.52:6790/0 at bind addrs v1:172.21.15.52:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:56:53.264 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(???) e0 preinit fsid 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:56:53.264 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(synchronizing).mds e1 new map 2024-09-13T12:56:53.264 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout: e1 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout: btime 2024-09-13T12:55:27:893730+0000 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout: legacy client fscid: -1 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout: 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout: No filesystems configured 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-13T12:56:53.265 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: Updating smithi167:/etc/ceph/ceph.conf 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: Updating smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:53.266 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:56:53.267 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:56:53.267 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2024-09-13T12:56:53.267 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: Updating smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T12:56:53.267 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:53.267 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='client.14186 v1:172.21.15.167:0/2454574402' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:[v1:172.21.15.52:6789]=a;smithi052:[v1:172.21.15.52:6790]=c;smithi167:[v1:172.21.15.167:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:56:53.267 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: Saving service mon spec with placement smithi052:[v1:172.21.15.52:6789]=a;smithi052:[v1:172.21.15.52:6790]=c;smithi167:[v1:172.21.15.167:6789]=b;count:3 2024-09-13T12:56:53.267 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: Deploying daemon mon.b on smithi167 2024-09-13T12:56:53.267 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/27378577' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T12:56:53.267 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:53.267 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:53.267 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/2221745528' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T12:56:53.267 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:56:53.267 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:53.268 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.a calling monitor election 2024-09-13T12:56:53.268 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:53.268 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:53.268 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:53.268 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.b calling monitor election 2024-09-13T12:56:53.268 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:53.268 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:53.268 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:53.268 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:53.268 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:53.268 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T12:56:53.268 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: monmap epoch 2 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: fsid 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: last_changed 2024-09-13T12:56:44.683156+0000 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: created 2024-09-13T12:55:24.138668+0000 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: min_mon_release 19 (squid) 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: election_strategy: 1 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: 0: v1:172.21.15.52:6789/0 mon.a 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: 1: v1:172.21.15.167:6789/0 mon.b 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: fsmap 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mgrmap e13: y(active, since 34s) 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: overall HEALTH_OK 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:53.269 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:53.270 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:56:53.270 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:56:53.270 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: Deploying daemon mon.c on smithi052 2024-09-13T12:56:53.272 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/1061599883' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T12:56:53.272 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:53.272 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:53.272 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/630321588' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T12:56:53.272 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 ceph-mon[36682]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-09-13T12:56:53.272 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 bash[36664]: cf921afd375aaa9878bfd4307955fb2573c16fad07a9fd31ef0acebb62e60e5e 2024-09-13T12:56:53.272 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:52 smithi052 systemd[1]: Started Ceph mon.c for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T12:56:54.029 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T12:56:54.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T12:56:54.275 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T12:56:58.119 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:56:58.119 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":3,"fsid":"2b415b38-71cf-11ef-bceb-c7b262605968","modified":"2024-09-13T12:56:52.825958Z","created":"2024-09-13T12:55:24.138668Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6790","nonce":0}]},"addr":"172.21.15.52:6790/0","public_addr":"172.21.15.52:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-13T12:56:58.119 INFO:teuthology.orchestra.run.smithi167.stderr:dumped monmap epoch 3 2024-09-13T12:56:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:56:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: mon.a calling monitor election 2024-09-13T12:56:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: mon.b calling monitor election 2024-09-13T12:56:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:58.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: mon.c calling monitor election 2024-09-13T12:56:58.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:58.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T12:56:58.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: monmap epoch 3 2024-09-13T12:56:58.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: fsid 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:56:58.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: last_changed 2024-09-13T12:56:52.825958+0000 2024-09-13T12:56:58.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: created 2024-09-13T12:55:24.138668+0000 2024-09-13T12:56:58.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: min_mon_release 19 (squid) 2024-09-13T12:56:58.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: election_strategy: 1 2024-09-13T12:56:58.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: 0: v1:172.21.15.52:6789/0 mon.a 2024-09-13T12:56:58.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: 1: v1:172.21.15.167:6789/0 mon.b 2024-09-13T12:56:58.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: 2: v1:172.21.15.52:6790/0 mon.c 2024-09-13T12:56:58.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: fsmap 2024-09-13T12:56:58.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T12:56:58.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: mgrmap e13: y(active, since 43s) 2024-09-13T12:56:58.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: overall HEALTH_OK 2024-09-13T12:56:58.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:58.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:58.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:58.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:58.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:58.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:56:58.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: mon.a calling monitor election 2024-09-13T12:56:58.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: mon.b calling monitor election 2024-09-13T12:56:58.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:58.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:58.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: mon.c calling monitor election 2024-09-13T12:56:58.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: monmap epoch 3 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: fsid 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: last_changed 2024-09-13T12:56:52.825958+0000 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: created 2024-09-13T12:55:24.138668+0000 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: min_mon_release 19 (squid) 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: election_strategy: 1 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: 0: v1:172.21.15.52:6789/0 mon.a 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: 1: v1:172.21.15.167:6789/0 mon.b 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: 2: v1:172.21.15.52:6790/0 mon.c 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: fsmap 2024-09-13T12:56:58.244 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T12:56:58.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: mgrmap e13: y(active, since 43s) 2024-09-13T12:56:58.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: overall HEALTH_OK 2024-09-13T12:56:58.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:58.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:58.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:58.245 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:56:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: mon.a calling monitor election 2024-09-13T12:56:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: mon.b calling monitor election 2024-09-13T12:56:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: mon.c calling monitor election 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: monmap epoch 3 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: fsid 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: last_changed 2024-09-13T12:56:52.825958+0000 2024-09-13T12:56:58.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: created 2024-09-13T12:55:24.138668+0000 2024-09-13T12:56:58.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: min_mon_release 19 (squid) 2024-09-13T12:56:58.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: election_strategy: 1 2024-09-13T12:56:58.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: 0: v1:172.21.15.52:6789/0 mon.a 2024-09-13T12:56:58.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: 1: v1:172.21.15.167:6789/0 mon.b 2024-09-13T12:56:58.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: 2: v1:172.21.15.52:6790/0 mon.c 2024-09-13T12:56:58.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: fsmap 2024-09-13T12:56:58.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T12:56:58.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: mgrmap e13: y(active, since 43s) 2024-09-13T12:56:58.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: overall HEALTH_OK 2024-09-13T12:56:58.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:58.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:58.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:58.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:56:58.698 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-13T12:56:58.698 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-13T12:56:58.966 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:56:58.969 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:58 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:56:58.969 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:58 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/2730145031' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T12:56:58.969 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:58 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:59.225 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:58 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:56:59.225 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:58 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/2730145031' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T12:56:59.225 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:58 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:59.226 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:58 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:56:59.226 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:58 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/2730145031' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T12:56:59.226 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:58 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:56:59.647 INFO:teuthology.orchestra.run.smithi052.stdout:# minimal ceph.conf for 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:56:59.647 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2024-09-13T12:56:59.647 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T12:56:59.647 INFO:teuthology.orchestra.run.smithi052.stdout: mon_host = 172.21.15.52:6789/0 172.21.15.167:6789/0 v1:172.21.15.52:6790/0 2024-09-13T12:57:00.213 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:00.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:00.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:00.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: Updating smithi052:/etc/ceph/ceph.conf 2024-09-13T12:57:00.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: Updating smithi167:/etc/ceph/ceph.conf 2024-09-13T12:57:00.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: Updating smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T12:57:00.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: Updating smithi052:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T12:57:00.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1709533215' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:00.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.215 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.215 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.215 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.215 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.215 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.215 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.215 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.215 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.215 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:57:00.215 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T12:57:00.215 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:00.216 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:00.216 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.216 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.216 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.216 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.216 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:00.216 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:00.216 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: Updating smithi052:/etc/ceph/ceph.conf 2024-09-13T12:57:00.216 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: Updating smithi167:/etc/ceph/ceph.conf 2024-09-13T12:57:00.216 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: Updating smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T12:57:00.216 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: Updating smithi052:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T12:57:00.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1709533215' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:00.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:57:00.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T12:57:00.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:56:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:00.243 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-13T12:57:00.243 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:57:00.243 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T12:57:00.271 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:57:00.271 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T12:57:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:00.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:00.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: Updating smithi052:/etc/ceph/ceph.conf 2024-09-13T12:57:00.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: Updating smithi167:/etc/ceph/ceph.conf 2024-09-13T12:57:00.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: Updating smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T12:57:00.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: Updating smithi052:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T12:57:00.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1709533215' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:00.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.324 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.324 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.324 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.324 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:00.324 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:57:00.324 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T12:57:00.325 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:56:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:00.339 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-09-13T12:57:00.339 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T12:57:00.370 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-09-13T12:57:00.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T12:57:00.437 INFO:tasks.cephadm:Adding mgr.y on smithi052 2024-09-13T12:57:00.437 INFO:tasks.cephadm:Adding mgr.x on smithi167 2024-09-13T12:57:00.437 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi052=y;smithi167=x' 2024-09-13T12:57:00.725 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T12:57:01.332 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled mgr update... 2024-09-13T12:57:01.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[29471]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T12:57:01.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[29471]: Reconfiguring daemon mon.a on smithi052 2024-09-13T12:57:01.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:01.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:01.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:57:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T12:57:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[36682]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T12:57:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[36682]: Reconfiguring daemon mon.a on smithi052 2024-09-13T12:57:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:57:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T12:57:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:01 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:01.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:01 smithi167 ceph-mon[33178]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T12:57:01.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:01 smithi167 ceph-mon[33178]: Reconfiguring daemon mon.a on smithi052 2024-09-13T12:57:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:01 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:01 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:01 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:01 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:57:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:01 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T12:57:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:01 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:01.871 DEBUG:teuthology.orchestra.run.smithi167:mgr.x> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mgr.x.service 2024-09-13T12:57:01.875 DEBUG:tasks.cephadm:set 0 configs 2024-09-13T12:57:01.875 INFO:tasks.cephadm:Deploying OSDs... 2024-09-13T12:57:01.875 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T12:57:01.875 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T12:57:01.893 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T12:57:01.893 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 2024-09-13T12:57:01.951 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T12:57:01.951 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:57:01.951 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 980 Links: 1 2024-09-13T12:57:01.951 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:57:01.951 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:57:01.951 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-09-13 12:56:52.235580078 +0000 2024-09-13T12:57:01.951 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-09-13 12:51:07.208497003 +0000 2024-09-13T12:57:01.951 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-09-13 12:51:07.208497003 +0000 2024-09-13T12:57:01.951 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-09-13 12:51:07.208497003 +0000 2024-09-13T12:57:01.952 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T12:57:02.018 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-09-13T12:57:02.018 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-09-13T12:57:02.018 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000112494 s, 4.6 MB/s 2024-09-13T12:57:02.020 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T12:57:02.078 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 2024-09-13T12:57:02.135 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T12:57:02.136 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:57:02.136 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-09-13T12:57:02.136 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:57:02.136 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:57:02.136 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-09-13 12:56:52.236580067 +0000 2024-09-13T12:57:02.136 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-09-13 12:51:07.207496996 +0000 2024-09-13T12:57:02.136 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-09-13 12:51:07.207496996 +0000 2024-09-13T12:57:02.136 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-09-13 12:51:07.207496996 +0000 2024-09-13T12:57:02.136 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T12:57:02.200 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-09-13T12:57:02.200 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-09-13T12:57:02.200 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000109066 s, 4.7 MB/s 2024-09-13T12:57:02.201 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T12:57:02.259 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 2024-09-13T12:57:02.316 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T12:57:02.316 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:57:02.316 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 986 Links: 1 2024-09-13T12:57:02.316 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:57:02.316 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:57:02.316 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-09-13 12:56:52.236580067 +0000 2024-09-13T12:57:02.317 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-09-13 12:51:07.209497010 +0000 2024-09-13T12:57:02.317 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-09-13 12:51:07.209497010 +0000 2024-09-13T12:57:02.317 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-09-13 12:51:07.209497010 +0000 2024-09-13T12:57:02.317 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T12:57:02.381 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-09-13T12:57:02.381 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-09-13T12:57:02.381 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000105998 s, 4.8 MB/s 2024-09-13T12:57:02.382 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T12:57:02.425 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:02 smithi052 ceph-mon[29471]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:02.426 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:02 smithi052 ceph-mon[29471]: Reconfiguring mon.c (monmap changed)... 2024-09-13T12:57:02.426 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:02 smithi052 ceph-mon[29471]: Reconfiguring daemon mon.c on smithi052 2024-09-13T12:57:02.426 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:02 smithi052 ceph-mon[29471]: from='client.24109 v1:172.21.15.167:0/429910480' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi052=y;smithi167=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:57:02.426 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:02 smithi052 ceph-mon[29471]: Saving service mgr spec with placement smithi052=y;smithi167=x;count:2 2024-09-13T12:57:02.426 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:02 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:02.426 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:02 smithi052 ceph-mon[36682]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:02.426 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:02 smithi052 ceph-mon[36682]: Reconfiguring mon.c (monmap changed)... 2024-09-13T12:57:02.426 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:02 smithi052 ceph-mon[36682]: Reconfiguring daemon mon.c on smithi052 2024-09-13T12:57:02.426 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:02 smithi052 ceph-mon[36682]: from='client.24109 v1:172.21.15.167:0/429910480' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi052=y;smithi167=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:57:02.427 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:02 smithi052 ceph-mon[36682]: Saving service mgr spec with placement smithi052=y;smithi167=x;count:2 2024-09-13T12:57:02.427 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:02 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:02.440 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 2024-09-13T12:57:02.496 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T12:57:02.496 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:57:02.496 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 991 Links: 1 2024-09-13T12:57:02.496 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:57:02.496 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:57:02.496 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-09-13 12:56:52.237580055 +0000 2024-09-13T12:57:02.496 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-09-13 12:51:07.209497010 +0000 2024-09-13T12:57:02.496 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-09-13 12:51:07.209497010 +0000 2024-09-13T12:57:02.497 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-09-13 12:51:07.209497010 +0000 2024-09-13T12:57:02.497 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T12:57:02.560 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-09-13T12:57:02.561 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-09-13T12:57:02.561 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000106447 s, 4.8 MB/s 2024-09-13T12:57:02.562 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T12:57:02.619 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-09-13T12:57:02.619 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T12:57:02.635 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T12:57:02.636 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_1 2024-09-13T12:57:02.691 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T12:57:02.691 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:57:02.692 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 966 Links: 1 2024-09-13T12:57:02.692 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:57:02.692 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:57:02.692 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-09-13 12:56:43.670783206 +0000 2024-09-13T12:57:02.692 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-09-13 12:51:08.365846836 +0000 2024-09-13T12:57:02.692 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-09-13 12:51:08.365846836 +0000 2024-09-13T12:57:02.692 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-09-13 12:51:08.365846836 +0000 2024-09-13T12:57:02.692 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T12:57:02.753 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:02 smithi167 ceph-mon[33178]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:02.754 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:02 smithi167 ceph-mon[33178]: Reconfiguring mon.c (monmap changed)... 2024-09-13T12:57:02.754 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:02 smithi167 ceph-mon[33178]: Reconfiguring daemon mon.c on smithi052 2024-09-13T12:57:02.754 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:02 smithi167 ceph-mon[33178]: from='client.24109 v1:172.21.15.167:0/429910480' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi052=y;smithi167=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:57:02.754 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:02 smithi167 ceph-mon[33178]: Saving service mgr spec with placement smithi052=y;smithi167=x;count:2 2024-09-13T12:57:02.754 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:02 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:02.757 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-09-13T12:57:02.757 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-09-13T12:57:02.757 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000214707 s, 2.4 MB/s 2024-09-13T12:57:02.758 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T12:57:02.816 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_2 2024-09-13T12:57:02.872 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T12:57:02.872 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:57:02.872 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 975 Links: 1 2024-09-13T12:57:02.872 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:57:02.872 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:57:02.872 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-09-13 12:56:43.671783188 +0000 2024-09-13T12:57:02.872 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-09-13 12:51:08.366846818 +0000 2024-09-13T12:57:02.873 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-09-13 12:51:08.366846818 +0000 2024-09-13T12:57:02.873 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-09-13 12:51:08.366846818 +0000 2024-09-13T12:57:02.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T12:57:02.937 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-09-13T12:57:02.937 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-09-13T12:57:02.937 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000198921 s, 2.6 MB/s 2024-09-13T12:57:02.938 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T12:57:02.996 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_3 2024-09-13T12:57:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T12:57:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:57:03.053 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 979 Links: 1 2024-09-13T12:57:03.053 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:57:03.054 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:57:03.054 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-09-13 12:56:43.671783188 +0000 2024-09-13T12:57:03.054 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-09-13 12:51:08.366846818 +0000 2024-09-13T12:57:03.054 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-09-13 12:51:08.366846818 +0000 2024-09-13T12:57:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-09-13 12:51:08.366846818 +0000 2024-09-13T12:57:03.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T12:57:03.117 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-09-13T12:57:03.117 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-09-13T12:57:03.117 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000174385 s, 2.9 MB/s 2024-09-13T12:57:03.119 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T12:57:03.175 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_4 2024-09-13T12:57:03.232 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T12:57:03.232 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:57:03.232 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 980 Links: 1 2024-09-13T12:57:03.232 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:57:03.232 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:57:03.232 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-09-13 12:56:43.672783170 +0000 2024-09-13T12:57:03.232 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-09-13 12:51:08.366846818 +0000 2024-09-13T12:57:03.233 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-09-13 12:51:08.366846818 +0000 2024-09-13T12:57:03.233 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-09-13 12:51:08.366846818 +0000 2024-09-13T12:57:03.233 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T12:57:03.296 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-09-13T12:57:03.297 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-09-13T12:57:03.297 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000201034 s, 2.5 MB/s 2024-09-13T12:57:03.298 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T12:57:03.354 INFO:tasks.cephadm:Deploying osd.0 on smithi052 with /dev/vg_nvme/lv_4... 2024-09-13T12:57:03.354 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-13T12:57:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:03 smithi167 ceph-mon[33178]: Reconfiguring mon.b (monmap changed)... 2024-09-13T12:57:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:57:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T12:57:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:03 smithi167 ceph-mon[33178]: Reconfiguring daemon mon.b on smithi167 2024-09-13T12:57:03.621 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:57:03.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:03.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:03.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[29471]: Reconfiguring mon.b (monmap changed)... 2024-09-13T12:57:03.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:57:03.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T12:57:03.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:03.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[29471]: Reconfiguring daemon mon.b on smithi167 2024-09-13T12:57:03.711 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:03.711 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:03.711 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[36682]: Reconfiguring mon.b (monmap changed)... 2024-09-13T12:57:03.711 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T12:57:03.711 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T12:57:03.711 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:03.711 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:03 smithi052 ceph-mon[36682]: Reconfiguring daemon mon.b on smithi167 2024-09-13T12:57:04.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:04 smithi167 ceph-mon[33178]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:04 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:04 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:04 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:04 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:04 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:04 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:04 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T12:57:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:04 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T12:57:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:04 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T12:57:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:04 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:04.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[29471]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:04.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:04.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:04.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:04.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:04.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:04.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:04.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T12:57:04.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T12:57:04.906 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T12:57:04.906 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:04.906 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[36682]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:04.906 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:04.906 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:04.906 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:04.906 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:04.906 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:04.906 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:04.907 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T12:57:04.907 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T12:57:04.907 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T12:57:04.907 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:04 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:05.570 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 12:57:05 smithi167 systemd[1]: Starting Ceph mgr.x for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T12:57:05.921 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:05 smithi167 ceph-mon[33178]: Deploying daemon mgr.x on smithi167 2024-09-13T12:57:05.921 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 12:57:05 smithi167 podman[34657]: 2024-09-13 12:57:05.600912936 +0000 UTC m=+0.015681353 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T12:57:05.921 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 12:57:05 smithi167 podman[34657]: 2024-09-13 12:57:05.718919361 +0000 UTC m=+0.133687811 container create 91caf7b5c2432b3014080c8f2c589b2df9a925faf933889bd280ae23a4af0a5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD) 2024-09-13T12:57:05.922 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 12:57:05 smithi167 podman[34657]: 2024-09-13 12:57:05.921165115 +0000 UTC m=+0.335933529 container init 91caf7b5c2432b3014080c8f2c589b2df9a925faf933889bd280ae23a4af0a5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8) 2024-09-13T12:57:05.971 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:05 smithi052 ceph-mon[29471]: Deploying daemon mgr.x on smithi167 2024-09-13T12:57:05.972 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:05 smithi052 ceph-mon[36682]: Deploying daemon mgr.x on smithi167 2024-09-13T12:57:06.250 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:57:06.276 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_4 2024-09-13T12:57:06.320 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 12:57:05 smithi167 podman[34657]: 2024-09-13 12:57:05.924031693 +0000 UTC m=+0.338800101 container start 91caf7b5c2432b3014080c8f2c589b2df9a925faf933889bd280ae23a4af0a5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-13T12:57:06.320 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 12:57:06 smithi167 bash[34657]: 91caf7b5c2432b3014080c8f2c589b2df9a925faf933889bd280ae23a4af0a5e 2024-09-13T12:57:06.321 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 12:57:06 smithi167 systemd[1]: Started Ceph mgr.x for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T12:57:06.530 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:57:06.901 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:06 smithi167 ceph-mon[33178]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:06.901 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:06 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:06.901 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:06 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:06.901 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:06 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:06.901 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:06 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:06.901 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:06 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:06.909 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:06 smithi052 ceph-mon[29471]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:06.909 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:06 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:06.909 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:06 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:06.909 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:06 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:06.909 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:06 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:06.909 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:06 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:06.909 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:06 smithi052 ceph-mon[36682]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:06.909 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:06 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:06.909 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:06 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:06.910 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:06 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:06.910 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:06 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:06.910 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:06 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:07.835 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:57:07.835 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:57:07.835 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:07.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:07.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:07.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:07.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:07.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:07.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T12:57:07.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T12:57:07.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:07.836 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:57:07.836 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:57:07.837 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:07.837 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:07.837 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:07.837 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:07.837 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:07.837 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:07.837 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T12:57:07.837 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T12:57:07.837 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:08.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:57:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:57:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T12:57:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T12:57:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:08.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:08 smithi052 ceph-mon[29471]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:08.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:08 smithi052 ceph-mon[29471]: from='client.14226 v1:172.21.15.52:0/47358691' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:57:08.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:08 smithi052 ceph-mon[29471]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T12:57:08.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:08 smithi052 ceph-mon[29471]: Reconfiguring daemon mgr.y on smithi052 2024-09-13T12:57:08.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:08 smithi052 ceph-mon[36682]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:08.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:08 smithi052 ceph-mon[36682]: from='client.14226 v1:172.21.15.52:0/47358691' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:57:08.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:08 smithi052 ceph-mon[36682]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T12:57:08.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:08 smithi052 ceph-mon[36682]: Reconfiguring daemon mgr.y on smithi052 2024-09-13T12:57:09.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:08 smithi167 ceph-mon[33178]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:09.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:08 smithi167 ceph-mon[33178]: from='client.14226 v1:172.21.15.52:0/47358691' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:57:09.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:08 smithi167 ceph-mon[33178]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T12:57:09.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:08 smithi167 ceph-mon[33178]: Reconfiguring daemon mgr.y on smithi052 2024-09-13T12:57:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[29471]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:10.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[36682]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:10.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:10.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:10.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:10.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:10.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:10.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:10 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:10 smithi167 ceph-mon[33178]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:10 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:10 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:10 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:10 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:10 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:10 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/424614505' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d52dca1a-94c1-4e9e-93f5-687c038cd7c7"} : dispatch 2024-09-13T12:57:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[29471]: from='client.34121 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d52dca1a-94c1-4e9e-93f5-687c038cd7c7"} : dispatch 2024-09-13T12:57:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[29471]: from='client.34121 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d52dca1a-94c1-4e9e-93f5-687c038cd7c7"}]': finished 2024-09-13T12:57:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[29471]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T12:57:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[29471]: Standby manager daemon x started 2024-09-13T12:57:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[29471]: from='mgr.? v1:172.21.15.167:0/789552094' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-13T12:57:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[29471]: from='mgr.? v1:172.21.15.167:0/789552094' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-13T12:57:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[29471]: from='mgr.? v1:172.21.15.167:0/789552094' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-13T12:57:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[29471]: from='mgr.? v1:172.21.15.167:0/789552094' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-13T12:57:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3213781389' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:57:11.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:11.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/424614505' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d52dca1a-94c1-4e9e-93f5-687c038cd7c7"} : dispatch 2024-09-13T12:57:11.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[36682]: from='client.34121 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d52dca1a-94c1-4e9e-93f5-687c038cd7c7"} : dispatch 2024-09-13T12:57:11.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[36682]: from='client.34121 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d52dca1a-94c1-4e9e-93f5-687c038cd7c7"}]': finished 2024-09-13T12:57:11.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[36682]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T12:57:11.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:11.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[36682]: Standby manager daemon x started 2024-09-13T12:57:11.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[36682]: from='mgr.? v1:172.21.15.167:0/789552094' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-13T12:57:11.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[36682]: from='mgr.? v1:172.21.15.167:0/789552094' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-13T12:57:11.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[36682]: from='mgr.? v1:172.21.15.167:0/789552094' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-13T12:57:11.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[36682]: from='mgr.? v1:172.21.15.167:0/789552094' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-13T12:57:11.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3213781389' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:57:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:11 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/424614505' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d52dca1a-94c1-4e9e-93f5-687c038cd7c7"} : dispatch 2024-09-13T12:57:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:11 smithi167 ceph-mon[33178]: from='client.34121 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d52dca1a-94c1-4e9e-93f5-687c038cd7c7"} : dispatch 2024-09-13T12:57:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:11 smithi167 ceph-mon[33178]: from='client.34121 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d52dca1a-94c1-4e9e-93f5-687c038cd7c7"}]': finished 2024-09-13T12:57:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:11 smithi167 ceph-mon[33178]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T12:57:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:11 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:11 smithi167 ceph-mon[33178]: Standby manager daemon x started 2024-09-13T12:57:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:11 smithi167 ceph-mon[33178]: from='mgr.? v1:172.21.15.167:0/789552094' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-13T12:57:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:11 smithi167 ceph-mon[33178]: from='mgr.? v1:172.21.15.167:0/789552094' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-13T12:57:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:11 smithi167 ceph-mon[33178]: from='mgr.? v1:172.21.15.167:0/789552094' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-13T12:57:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:11 smithi167 ceph-mon[33178]: from='mgr.? v1:172.21.15.167:0/789552094' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-13T12:57:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3213781389' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:57:12.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:12 smithi052 ceph-mon[29471]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:12.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:12 smithi052 ceph-mon[29471]: mgrmap e14: y(active, since 56s), standbys: x 2024-09-13T12:57:12.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:12 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-13T12:57:12.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:12 smithi052 ceph-mon[36682]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:12.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:12 smithi052 ceph-mon[36682]: mgrmap e14: y(active, since 56s), standbys: x 2024-09-13T12:57:12.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:12 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-13T12:57:12.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:12 smithi167 ceph-mon[33178]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:12.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:12 smithi167 ceph-mon[33178]: mgrmap e14: y(active, since 56s), standbys: x 2024-09-13T12:57:12.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:12 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-13T12:57:14.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:14 smithi052 ceph-mon[36682]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:14.788 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:14 smithi052 ceph-mon[29471]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:15.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:14 smithi167 ceph-mon[33178]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:16.721 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:16 smithi052 ceph-mon[29471]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:16.722 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:16 smithi052 ceph-mon[36682]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:17.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:16 smithi167 ceph-mon[33178]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:17.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:17 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T12:57:17.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:17 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:17.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:17 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T12:57:17.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:17 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:18.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:17 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T12:57:18.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:17 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:18.924 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:18 smithi052 ceph-mon[29471]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:18.924 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:18 smithi052 ceph-mon[29471]: Deploying daemon osd.0 on smithi052 2024-09-13T12:57:18.924 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:18 smithi052 ceph-mon[36682]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:18.924 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:18 smithi052 ceph-mon[36682]: Deploying daemon osd.0 on smithi052 2024-09-13T12:57:19.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:18 smithi167 ceph-mon[33178]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:19.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:18 smithi167 ceph-mon[33178]: Deploying daemon osd.0 on smithi052 2024-09-13T12:57:19.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:19 smithi052 ceph-mon[29471]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:19 smithi052 ceph-mon[36682]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:20.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:19 smithi167 ceph-mon[33178]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:22.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:21 smithi052 ceph-mon[36682]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:22.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:21 smithi052 ceph-mon[29471]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:22.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:21 smithi167 ceph-mon[33178]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:24.195 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:23 smithi052 ceph-mon[29471]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:24.196 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:23 smithi052 ceph-mon[36682]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:24.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:23 smithi167 ceph-mon[33178]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:25.158 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:24 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:25.158 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:24 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:25.158 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:24 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:25.159 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:24 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:25.159 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:24 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:25.159 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:24 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:25.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:24 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:25.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:24 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:25.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:24 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:26.161 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:25 smithi052 ceph-mon[29471]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:26.161 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:25 smithi052 ceph-mon[36682]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:26.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:25 smithi167 ceph-mon[33178]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:28.399 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:28 smithi167 ceph-mon[33178]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:28.399 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:28 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:28.399 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:28 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:28.399 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:28 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:28.399 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:28 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:28.399 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:28 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:28.417 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:28 smithi052 ceph-mon[29471]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:28.417 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:28 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:28.417 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:28 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:28.417 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:28 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:28.417 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:28 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:28.417 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:28 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:28.417 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:28 smithi052 ceph-mon[36682]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:28.417 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:28 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:28.417 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:28 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:28.418 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:28 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:28.418 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:28 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:28.418 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:28 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:29.948 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 0 on host 'smithi052' 2024-09-13T12:57:30.218 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:30 smithi052 ceph-mon[36682]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:30.218 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:30 smithi052 ceph-mon[36682]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T12:57:30.218 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:30 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:30.219 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:30 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:30.219 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:30 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:30.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:30 smithi052 ceph-mon[29471]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:30.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:30 smithi052 ceph-mon[29471]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T12:57:30.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:30 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:30.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:30 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:30.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:30 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:30.507 DEBUG:teuthology.orchestra.run.smithi052:osd.0> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.0.service 2024-09-13T12:57:30.510 INFO:tasks.cephadm:Deploying osd.1 on smithi052 with /dev/vg_nvme/lv_3... 2024-09-13T12:57:30.510 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-13T12:57:30.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:30 smithi167 ceph-mon[33178]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:30.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:30 smithi167 ceph-mon[33178]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T12:57:30.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:30 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:30.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:30 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:30.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:30 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:31.180 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:57:31.364 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:31 smithi052 ceph-mon[29471]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T12:57:31.364 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:31 smithi052 ceph-mon[29471]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T12:57:31.365 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:31 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:31.365 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:31 smithi052 ceph-mon[29471]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:57:31.365 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:31 smithi052 ceph-mon[36682]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T12:57:31.365 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:31 smithi052 ceph-mon[36682]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T12:57:31.365 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:31 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:31.365 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:31 smithi052 ceph-mon[36682]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:57:31.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:31 smithi167 ceph-mon[33178]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T12:57:31.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:31 smithi167 ceph-mon[33178]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T12:57:31.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:31 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:31.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:31 smithi167 ceph-mon[33178]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:57:31.739 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 12:57:31 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0[39971]: 2024-09-13T12:57:31.362+0000 7f1d69cd0640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T12:57:31.739 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 12:57:31 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0[39971]: 2024-09-13T12:57:31.364+0000 7f1d61aef640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:57:32.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:32 smithi052 ceph-mon[29471]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:32.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:32 smithi052 ceph-mon[29471]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-09-13T12:57:32.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:32 smithi052 ceph-mon[29471]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T12:57:32.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:32 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:32.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:32 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:32.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:32 smithi052 ceph-mon[29471]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' 2024-09-13T12:57:32.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:32 smithi052 ceph-mon[36682]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:32.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:32 smithi052 ceph-mon[36682]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-09-13T12:57:32.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:32 smithi052 ceph-mon[36682]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T12:57:32.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:32 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:32.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:32 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:32.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:32 smithi052 ceph-mon[36682]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' 2024-09-13T12:57:32.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:32 smithi167 ceph-mon[33178]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:32.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:32 smithi167 ceph-mon[33178]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-09-13T12:57:32.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:32 smithi167 ceph-mon[33178]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T12:57:32.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:32 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:32.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:32 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:32.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:32 smithi167 ceph-mon[33178]: from='osd.0 v1:172.21.15.52:6801/542298937' entity='osd.0' 2024-09-13T12:57:33.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:33 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T12:57:33.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:33 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T12:57:33.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:33.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:33 smithi052 ceph-mon[29471]: osd.0 v1:172.21.15.52:6801/542298937 boot 2024-09-13T12:57:33.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:33 smithi052 ceph-mon[29471]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T12:57:33.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:33.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:33 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T12:57:33.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:33 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T12:57:33.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:33.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:33 smithi052 ceph-mon[36682]: osd.0 v1:172.21.15.52:6801/542298937 boot 2024-09-13T12:57:33.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:33 smithi052 ceph-mon[36682]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T12:57:33.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:33.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:33 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T12:57:33.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:33 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T12:57:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:33 smithi167 ceph-mon[33178]: osd.0 v1:172.21.15.52:6801/542298937 boot 2024-09-13T12:57:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:33 smithi167 ceph-mon[33178]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T12:57:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T12:57:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[36682]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[36682]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T12:57:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T12:57:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[29471]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[29471]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T12:57:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T12:57:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:34 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:35.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:34 smithi167 ceph-mon[33178]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T12:57:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:34 smithi167 ceph-mon[33178]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T12:57:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:34 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:34 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:34 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T12:57:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:34 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:34 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:34 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:34 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:35.752 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:35 smithi052 ceph-mon[36682]: Detected new or changed devices on smithi052 2024-09-13T12:57:35.752 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:35 smithi052 ceph-mon[36682]: Adjusting osd_memory_target on smithi052 to 16068M 2024-09-13T12:57:35.752 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:35 smithi052 ceph-mon[29471]: Detected new or changed devices on smithi052 2024-09-13T12:57:35.753 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:35 smithi052 ceph-mon[29471]: Adjusting osd_memory_target on smithi052 to 16068M 2024-09-13T12:57:35.755 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:57:35.785 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_3 2024-09-13T12:57:36.040 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:57:36.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:35 smithi167 ceph-mon[33178]: Detected new or changed devices on smithi052 2024-09-13T12:57:36.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:35 smithi167 ceph-mon[33178]: Adjusting osd_memory_target on smithi052 to 16068M 2024-09-13T12:57:36.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:36 smithi052 ceph-mon[29471]: pgmap v39: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:36 smithi052 ceph-mon[36682]: pgmap v39: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:37.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:36 smithi167 ceph-mon[33178]: pgmap v39: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:37.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:37 smithi052 ceph-mon[29471]: from='client.14253 v1:172.21.15.52:0/1324209342' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:57:37.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:37 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:57:37.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:37 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:57:37.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:37 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:37.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:37 smithi052 ceph-mon[29471]: pgmap v40: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:37.982 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:37 smithi052 ceph-mon[36682]: from='client.14253 v1:172.21.15.52:0/1324209342' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:57:37.982 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:37 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:57:37.982 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:37 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:57:37.982 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:37 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:37.982 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:37 smithi052 ceph-mon[36682]: pgmap v40: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:38.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:37 smithi167 ceph-mon[33178]: from='client.14253 v1:172.21.15.52:0/1324209342' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:57:38.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:37 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:57:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:37 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:57:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:37 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:37 smithi167 ceph-mon[33178]: pgmap v40: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:39.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3933019381' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b40a3574-3c6f-4810-a651-8d64aee6801b"} : dispatch 2024-09-13T12:57:39.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3933019381' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b40a3574-3c6f-4810-a651-8d64aee6801b"}]': finished 2024-09-13T12:57:39.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:38 smithi052 ceph-mon[29471]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T12:57:39.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:38 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:57:39.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:38 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3933019381' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b40a3574-3c6f-4810-a651-8d64aee6801b"} : dispatch 2024-09-13T12:57:39.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:38 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3933019381' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b40a3574-3c6f-4810-a651-8d64aee6801b"}]': finished 2024-09-13T12:57:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:38 smithi052 ceph-mon[36682]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T12:57:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:38 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:57:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:38 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3933019381' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b40a3574-3c6f-4810-a651-8d64aee6801b"} : dispatch 2024-09-13T12:57:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:38 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3933019381' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b40a3574-3c6f-4810-a651-8d64aee6801b"}]': finished 2024-09-13T12:57:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:38 smithi167 ceph-mon[33178]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T12:57:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:38 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:57:40.238 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:39 smithi052 ceph-mon[29471]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:40.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:39 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2875475194' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:57:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:39 smithi052 ceph-mon[36682]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:39 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2875475194' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:57:40.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:39 smithi167 ceph-mon[33178]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:40.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:39 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2875475194' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:57:42.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:41 smithi052 ceph-mon[29471]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:42.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:41 smithi052 ceph-mon[36682]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:42.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:41 smithi167 ceph-mon[33178]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:44.139 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:43 smithi052 ceph-mon[29471]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:44.139 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:43 smithi052 ceph-mon[36682]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:44.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:43 smithi167 ceph-mon[33178]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:46.238 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:45 smithi052 ceph-mon[29471]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:46.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:45 smithi052 ceph-mon[36682]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:46.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:45 smithi167 ceph-mon[33178]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:47.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:46 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T12:57:47.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:46 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:47.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:46 smithi052 ceph-mon[29471]: Deploying daemon osd.1 on smithi052 2024-09-13T12:57:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:46 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T12:57:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:46 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:46 smithi052 ceph-mon[36682]: Deploying daemon osd.1 on smithi052 2024-09-13T12:57:47.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:46 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T12:57:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:46 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:46 smithi167 ceph-mon[33178]: Deploying daemon osd.1 on smithi052 2024-09-13T12:57:48.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:47 smithi052 ceph-mon[29471]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:47 smithi052 ceph-mon[36682]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:48.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:47 smithi167 ceph-mon[33178]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:50.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:49 smithi052 ceph-mon[29471]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:50.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:49 smithi052 ceph-mon[36682]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:50.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:49 smithi167 ceph-mon[33178]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:52.176 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:51 smithi052 ceph-mon[36682]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:52.177 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:51 smithi052 ceph-mon[29471]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:52.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:51 smithi167 ceph-mon[33178]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:53.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:53 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:53.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:53 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:53.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:53 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:53.933 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:53 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:53.934 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:53 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:53.934 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:53 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:54.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:53 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:54.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:53 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:54.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:53 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:54.873 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:54 smithi052 ceph-mon[29471]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:54.873 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:54 smithi052 ceph-mon[36682]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:55.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:54 smithi167 ceph-mon[33178]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:56 smithi052 ceph-mon[36682]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:56 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:56 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:56 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:56 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:56 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:56.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:56 smithi052 ceph-mon[29471]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:56.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:56 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:56.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:56 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:56.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:56 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:56.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:56 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:56.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:56 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:57.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:56 smithi167 ceph-mon[33178]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:57.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:56 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:57.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:56 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:57.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:56 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:57:57.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:56 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:57:57.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:56 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:58.593 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 1 on host 'smithi052' 2024-09-13T12:57:58.898 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:58 smithi052 ceph-mon[36682]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:58.898 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:58 smithi052 ceph-mon[36682]: from='osd.1 v1:172.21.15.52:6805/26289431' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T12:57:58.898 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:58 smithi052 ceph-mon[36682]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T12:57:58.898 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:58 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:58.898 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:58 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:58.898 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:58 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:58.898 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:58 smithi052 ceph-mon[29471]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:58.898 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:58 smithi052 ceph-mon[29471]: from='osd.1 v1:172.21.15.52:6805/26289431' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T12:57:58.898 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:58 smithi052 ceph-mon[29471]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T12:57:58.899 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:58 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:58.899 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:58 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:58.899 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:58 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:59.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:58 smithi167 ceph-mon[33178]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:57:59.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:58 smithi167 ceph-mon[33178]: from='osd.1 v1:172.21.15.52:6805/26289431' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T12:57:59.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:58 smithi167 ceph-mon[33178]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T12:57:59.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:58 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:57:59.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:58 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:59.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:58 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:57:59.245 DEBUG:teuthology.orchestra.run.smithi052:osd.1> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.1.service 2024-09-13T12:57:59.250 INFO:tasks.cephadm:Deploying osd.2 on smithi052 with /dev/vg_nvme/lv_2... 2024-09-13T12:57:59.250 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-13T12:57:59.870 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:57:59.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-mon[29471]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T12:57:59.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-mon[29471]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T12:57:59.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-mon[29471]: from='osd.1 v1:172.21.15.52:6805/26289431' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:57:59.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:57:59.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-mon[29471]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:57:59.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-mon[29471]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-09-13T12:57:59.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-mon[36682]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T12:57:59.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-mon[36682]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T12:57:59.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-mon[36682]: from='osd.1 v1:172.21.15.52:6805/26289431' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:57:59.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:57:59.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-mon[36682]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:57:59.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-mon[36682]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-09-13T12:57:59.990 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1[45353]: 2024-09-13T12:57:59.645+0000 7fe37a3c2640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T12:57:59.990 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 12:57:59 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1[45353]: 2024-09-13T12:57:59.648+0000 7fe3761e9640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:58:00.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:59 smithi167 ceph-mon[33178]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T12:58:00.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:59 smithi167 ceph-mon[33178]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T12:58:00.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:59 smithi167 ceph-mon[33178]: from='osd.1 v1:172.21.15.52:6805/26289431' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:58:00.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:58:00.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:59 smithi167 ceph-mon[33178]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:58:00.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:57:59 smithi167 ceph-mon[33178]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-09-13T12:58:00.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:00 smithi052 ceph-mon[29471]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:58:00.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:00 smithi052 ceph-mon[29471]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T12:58:00.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:00 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:58:00.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:00 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:58:00.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:00 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:58:00.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:00 smithi052 ceph-mon[36682]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:58:00.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:00 smithi052 ceph-mon[36682]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T12:58:00.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:00 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:58:00.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:00 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:58:00.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:00 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:58:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:00 smithi167 ceph-mon[33178]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:58:01.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:00 smithi167 ceph-mon[33178]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T12:58:01.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:00 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:58:01.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:00 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:58:01.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:00 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:58:01.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:01 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T12:58:01.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:01 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T12:58:01.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:01 smithi052 ceph-mon[29471]: osd.1 v1:172.21.15.52:6805/26289431 boot 2024-09-13T12:58:01.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:01 smithi052 ceph-mon[29471]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T12:58:01.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:01 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:58:01.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:01 smithi052 ceph-mon[29471]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:58:01.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:01 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T12:58:01.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:01 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T12:58:01.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:01 smithi052 ceph-mon[36682]: osd.1 v1:172.21.15.52:6805/26289431 boot 2024-09-13T12:58:01.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:01 smithi052 ceph-mon[36682]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T12:58:01.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:01 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:58:01.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:01 smithi052 ceph-mon[36682]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:58:02.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:01 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T12:58:02.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:01 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T12:58:02.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:01 smithi167 ceph-mon[33178]: osd.1 v1:172.21.15.52:6805/26289431 boot 2024-09-13T12:58:02.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:01 smithi167 ceph-mon[33178]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T12:58:02.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:01 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T12:58:02.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:01 smithi167 ceph-mon[33178]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-13T12:58:02.811 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:02 smithi052 ceph-mon[36682]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T12:58:02.811 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:02 smithi052 ceph-mon[29471]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T12:58:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:02 smithi167 ceph-mon[33178]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T12:58:04.140 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[36682]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:04.140 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[36682]: Detected new or changed devices on smithi052 2024-09-13T12:58:04.140 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:04.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:04.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:04.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:04.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[36682]: Adjusting osd_memory_target on smithi052 to 8034M 2024-09-13T12:58:04.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:04.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:04.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:58:04.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:04.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[29471]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:04.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[29471]: Detected new or changed devices on smithi052 2024-09-13T12:58:04.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:04.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:04.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:04.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:04.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[29471]: Adjusting osd_memory_target on smithi052 to 8034M 2024-09-13T12:58:04.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:04.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:04.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:58:04.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:03 smithi167 ceph-mon[33178]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:03 smithi167 ceph-mon[33178]: Detected new or changed devices on smithi052 2024-09-13T12:58:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:03 smithi167 ceph-mon[33178]: Adjusting osd_memory_target on smithi052 to 8034M 2024-09-13T12:58:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:58:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:58:04.448 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_2 2024-09-13T12:58:04.705 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:58:05.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:05 smithi052 ceph-mon[29471]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:05.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:05 smithi052 ceph-mon[29471]: from='client.14280 v1:172.21.15.52:0/2691954312' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:58:05.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:05 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:58:05.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:05 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:58:05.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:05 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:05.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:05 smithi052 ceph-mon[36682]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:05.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:05 smithi052 ceph-mon[36682]: from='client.14280 v1:172.21.15.52:0/2691954312' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:58:05.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:05 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:58:05.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:05 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:58:05.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:05 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:06.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:05 smithi167 ceph-mon[33178]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:06.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:05 smithi167 ceph-mon[33178]: from='client.14280 v1:172.21.15.52:0/2691954312' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:58:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:05 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:58:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:05 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:58:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:05 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:08.162 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:07 smithi052 ceph-mon[36682]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:08.163 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:07 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3723407041' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1e8e49a0-ccf8-4b7f-839b-104b64601408"} : dispatch 2024-09-13T12:58:08.163 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:07 smithi052 ceph-mon[36682]: from='client.34175 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1e8e49a0-ccf8-4b7f-839b-104b64601408"} : dispatch 2024-09-13T12:58:08.163 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:07 smithi052 ceph-mon[36682]: from='client.34175 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e8e49a0-ccf8-4b7f-839b-104b64601408"}]': finished 2024-09-13T12:58:08.163 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:07 smithi052 ceph-mon[36682]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T12:58:08.163 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:08.163 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:07 smithi052 ceph-mon[29471]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:08.163 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:07 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3723407041' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1e8e49a0-ccf8-4b7f-839b-104b64601408"} : dispatch 2024-09-13T12:58:08.163 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:07 smithi052 ceph-mon[29471]: from='client.34175 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1e8e49a0-ccf8-4b7f-839b-104b64601408"} : dispatch 2024-09-13T12:58:08.163 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:07 smithi052 ceph-mon[29471]: from='client.34175 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e8e49a0-ccf8-4b7f-839b-104b64601408"}]': finished 2024-09-13T12:58:08.163 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:07 smithi052 ceph-mon[29471]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T12:58:08.163 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:08.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:07 smithi167 ceph-mon[33178]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:07 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3723407041' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1e8e49a0-ccf8-4b7f-839b-104b64601408"} : dispatch 2024-09-13T12:58:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:07 smithi167 ceph-mon[33178]: from='client.34175 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1e8e49a0-ccf8-4b7f-839b-104b64601408"} : dispatch 2024-09-13T12:58:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:07 smithi167 ceph-mon[33178]: from='client.34175 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e8e49a0-ccf8-4b7f-839b-104b64601408"}]': finished 2024-09-13T12:58:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:07 smithi167 ceph-mon[33178]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T12:58:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:09.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/229369390' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:58:09.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/229369390' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:58:09.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/229369390' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:58:10.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:09 smithi167 ceph-mon[33178]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:10.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:09 smithi052 ceph-mon[29471]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:10.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:09 smithi052 ceph-mon[36682]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:12.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:11 smithi052 ceph-mon[29471]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:12.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:11 smithi052 ceph-mon[36682]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:12.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:11 smithi167 ceph-mon[33178]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:14.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:14 smithi052 ceph-mon[36682]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:14.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:14 smithi052 ceph-mon[29471]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:15.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:14 smithi167 ceph-mon[33178]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:15.972 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:15 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T12:58:15.972 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:15 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:15.972 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:15 smithi052 ceph-mon[36682]: Deploying daemon osd.2 on smithi052 2024-09-13T12:58:15.972 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:15 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T12:58:15.972 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:15 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:15.972 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:15 smithi052 ceph-mon[29471]: Deploying daemon osd.2 on smithi052 2024-09-13T12:58:16.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:15 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T12:58:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:15 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:15 smithi167 ceph-mon[33178]: Deploying daemon osd.2 on smithi052 2024-09-13T12:58:16.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:16 smithi052 ceph-mon[36682]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:16.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:16 smithi052 ceph-mon[29471]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:17.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:16 smithi167 ceph-mon[33178]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:18.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:18 smithi052 ceph-mon[29471]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:18.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:18 smithi052 ceph-mon[36682]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:19.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:18 smithi167 ceph-mon[33178]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:20.896 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:20 smithi052 ceph-mon[36682]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:20.897 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:20 smithi052 ceph-mon[29471]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:21.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:20 smithi167 ceph-mon[33178]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:22.876 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:22 smithi052 ceph-mon[29471]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:22.877 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:22 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:58:22.877 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:22 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:22.877 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:22 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:22.877 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:22 smithi052 ceph-mon[36682]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:22.877 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:22 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:58:22.877 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:22 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:22.877 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:22 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:23.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:22 smithi167 ceph-mon[33178]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:23.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:22 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:58:23.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:22 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:23.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:22 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:23.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:23 smithi052 ceph-mon[29471]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:23.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:23 smithi052 ceph-mon[36682]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:24.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:23 smithi167 ceph-mon[33178]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:25 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:25.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:25 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:25.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:25 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:25.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:25 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:58:25.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:25 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:25.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:25 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:25.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:25 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:25.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:25 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:25.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:25 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:58:25.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:25 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:26.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:25 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:26.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:25 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:26.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:25 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:26.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:25 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:58:26.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:25 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:26.982 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:26 smithi052 ceph-mon[36682]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:26.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:26 smithi052 ceph-mon[29471]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:27.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:26 smithi167 ceph-mon[33178]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:27.656 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 2 on host 'smithi052' 2024-09-13T12:58:27.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:27 smithi052 ceph-mon[29471]: from='osd.2 v1:172.21.15.52:6809/3960240393' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T12:58:27.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:27 smithi052 ceph-mon[36682]: from='osd.2 v1:172.21.15.52:6809/3960240393' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T12:58:28.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:27 smithi167 ceph-mon[33178]: from='osd.2 v1:172.21.15.52:6809/3960240393' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T12:58:28.325 DEBUG:teuthology.orchestra.run.smithi052:osd.2> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.2.service 2024-09-13T12:58:28.327 INFO:tasks.cephadm:Deploying osd.3 on smithi052 with /dev/vg_nvme/lv_1... 2024-09-13T12:58:28.327 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-13T12:58:28.972 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:28 smithi167 ceph-mon[33178]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:28.972 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:28 smithi167 ceph-mon[33178]: from='osd.2 v1:172.21.15.52:6809/3960240393' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T12:58:28.972 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:28 smithi167 ceph-mon[33178]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T12:58:28.972 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:28 smithi167 ceph-mon[33178]: from='osd.2 v1:172.21.15.52:6809/3960240393' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:58:28.972 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:28 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:28.972 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:28 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:58:28.972 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:28 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:28.972 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:28 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:28.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[36682]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:28.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[36682]: from='osd.2 v1:172.21.15.52:6809/3960240393' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T12:58:28.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[36682]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T12:58:28.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[36682]: from='osd.2 v1:172.21.15.52:6809/3960240393' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:58:28.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:28.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:58:28.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:28.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:28.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[29471]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:28.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[29471]: from='osd.2 v1:172.21.15.52:6809/3960240393' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T12:58:28.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[29471]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T12:58:28.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[29471]: from='osd.2 v1:172.21.15.52:6809/3960240393' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:58:28.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:28.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:58:28.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:28.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:28.990 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2[50706]: 2024-09-13T12:58:28.634+0000 7f75563cf640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T12:58:28.991 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 12:58:28 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2[50706]: 2024-09-13T12:58:28.636+0000 7f75521f6640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:58:29.057 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:58:29.769 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:29 smithi052 ceph-mon[29471]: from='osd.2 v1:172.21.15.52:6809/3960240393' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-09-13T12:58:29.769 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:29 smithi052 ceph-mon[29471]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T12:58:29.769 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:29 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:29.769 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:29 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:29.770 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:29 smithi052 ceph-mon[36682]: from='osd.2 v1:172.21.15.52:6809/3960240393' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-09-13T12:58:29.770 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:29 smithi052 ceph-mon[36682]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T12:58:29.770 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:29 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:29.770 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:29 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:29 smithi167 ceph-mon[33178]: from='osd.2 v1:172.21.15.52:6809/3960240393' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-09-13T12:58:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:29 smithi167 ceph-mon[33178]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T12:58:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:29 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:29 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:30 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T12:58:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:30 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T12:58:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:30 smithi052 ceph-mon[36682]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:30 smithi052 ceph-mon[36682]: osd.2 v1:172.21.15.52:6809/3960240393 boot 2024-09-13T12:58:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:30 smithi052 ceph-mon[36682]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T12:58:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:30 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:30 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T12:58:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:30 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T12:58:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:30 smithi052 ceph-mon[29471]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:30 smithi052 ceph-mon[29471]: osd.2 v1:172.21.15.52:6809/3960240393 boot 2024-09-13T12:58:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:30 smithi052 ceph-mon[29471]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T12:58:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:30 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:30 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T12:58:31.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:30 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T12:58:31.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:30 smithi167 ceph-mon[33178]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-13T12:58:31.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:30 smithi167 ceph-mon[33178]: osd.2 v1:172.21.15.52:6809/3960240393 boot 2024-09-13T12:58:31.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:30 smithi167 ceph-mon[33178]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T12:58:31.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:30 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T12:58:31.899 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:31 smithi052 ceph-mon[36682]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T12:58:31.899 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:31 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-13T12:58:31.899 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:31 smithi052 ceph-mon[29471]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T12:58:31.899 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:31 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-13T12:58:32.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:31 smithi167 ceph-mon[33178]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T12:58:32.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:31 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-13T12:58:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[29471]: pgmap v77: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T12:58:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[29471]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T12:58:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-13T12:58:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:32.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:32.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:32.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:32.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:32.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:32.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:58:32.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:32.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[36682]: pgmap v77: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:32.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T12:58:32.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[36682]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T12:58:32.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-13T12:58:32.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:32.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:32.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:32.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:32.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:32.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:32.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:32.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:58:32.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:32 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:33.068 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54753]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T12:58:33.069 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54753]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T12:58:33.069 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54753]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T12:58:33.069 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54753]: pam_unix(sudo:session): session closed for user root 2024-09-13T12:58:33.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:32 smithi167 ceph-mon[33178]: pgmap v77: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:32 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T12:58:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:32 smithi167 ceph-mon[33178]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T12:58:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:32 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-13T12:58:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:32 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:32 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:32 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:32 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:32 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T12:58:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:32 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:32 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:32 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:58:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:32 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:33.322 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54778]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T12:58:33.322 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54778]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T12:58:33.323 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54778]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T12:58:33.323 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54778]: pam_unix(sudo:session): session closed for user root 2024-09-13T12:58:33.323 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54757]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T12:58:33.323 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54757]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T12:58:33.323 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54757]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T12:58:33.323 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54757]: pam_unix(sudo:session): session closed for user root 2024-09-13T12:58:33.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54772]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T12:58:33.324 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54772]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T12:58:33.324 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54772]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T12:58:33.324 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54772]: pam_unix(sudo:session): session closed for user root 2024-09-13T12:58:33.324 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54775]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T12:58:33.324 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54775]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T12:58:33.324 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54775]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T12:58:33.324 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54775]: pam_unix(sudo:session): session closed for user root 2024-09-13T12:58:33.324 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54760]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T12:58:33.324 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54760]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T12:58:33.324 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54760]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T12:58:33.325 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54760]: pam_unix(sudo:session): session closed for user root 2024-09-13T12:58:33.325 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54763]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T12:58:33.325 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54763]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T12:58:33.325 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54763]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T12:58:33.325 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54763]: pam_unix(sudo:session): session closed for user root 2024-09-13T12:58:33.325 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54766]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T12:58:33.325 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54766]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T12:58:33.325 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54766]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T12:58:33.325 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54766]: pam_unix(sudo:session): session closed for user root 2024-09-13T12:58:33.325 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54769]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T12:58:33.326 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54769]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T12:58:33.326 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54769]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T12:58:33.326 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54769]: pam_unix(sudo:session): session closed for user root 2024-09-13T12:58:33.492 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T12:58:33.517 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_1 2024-09-13T12:58:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 sudo[35116]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T12:58:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 sudo[35116]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T12:58:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 sudo[35116]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T12:58:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 sudo[35116]: pam_unix(sudo:session): session closed for user root 2024-09-13T12:58:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 sudo[35119]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T12:58:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 sudo[35119]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T12:58:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 sudo[35119]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T12:58:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 sudo[35119]: pam_unix(sudo:session): session closed for user root 2024-09-13T12:58:33.663 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54781]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T12:58:33.664 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54781]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T12:58:33.664 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54781]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T12:58:33.664 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 sudo[54781]: pam_unix(sudo:session): session closed for user root 2024-09-13T12:58:33.664 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: Detected new or changed devices on smithi052 2024-09-13T12:58:33.664 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: Adjusting osd_memory_target on smithi052 to 5356M 2024-09-13T12:58:33.664 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T12:58:33.664 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T12:58:33.664 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:33.664 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:33.664 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:58:33.665 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:58:33.665 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:58:33.665 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T12:58:33.665 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T12:58:33.665 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:58:33.776 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T12:58:33.984 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:58:33.984 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T12:58:33.984 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:58:33.984 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T12:58:33.984 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:33.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:58:33.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T12:58:33.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:58:33.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:58:33.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[29471]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T12:58:33.985 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: Detected new or changed devices on smithi052 2024-09-13T12:58:33.985 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: Adjusting osd_memory_target on smithi052 to 5356M 2024-09-13T12:58:33.985 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T12:58:33.985 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T12:58:33.985 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:33.985 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:58:33.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:58:33.987 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:33 smithi052 ceph-mon[36682]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T12:58:34.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: Detected new or changed devices on smithi052 2024-09-13T12:58:34.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: Adjusting osd_memory_target on smithi052 to 5356M 2024-09-13T12:58:34.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T12:58:34.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T12:58:34.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:34.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:34.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:58:34.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:58:34.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:58:34.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T12:58:34.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T12:58:34.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:58:34.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:58:34.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T12:58:34.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:58:34.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T12:58:34.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:34.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T12:58:34.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T12:58:34.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T12:58:34.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T12:58:34.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:33 smithi167 ceph-mon[33178]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T12:58:34.937 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:34 smithi052 ceph-mon[36682]: osdmap e22: 3 total, 3 up, 3 in 2024-09-13T12:58:34.938 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:34 smithi052 ceph-mon[36682]: from='client.14316 v1:172.21.15.52:0/3614486912' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:58:34.938 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:34 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:58:34.938 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:34 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:58:34.938 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:34 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:34.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:34 smithi052 ceph-mon[29471]: osdmap e22: 3 total, 3 up, 3 in 2024-09-13T12:58:34.939 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:34 smithi052 ceph-mon[29471]: from='client.14316 v1:172.21.15.52:0/3614486912' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:58:34.939 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:34 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:58:34.939 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:34 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:58:34.939 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:34 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:35.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:34 smithi167 ceph-mon[33178]: osdmap e22: 3 total, 3 up, 3 in 2024-09-13T12:58:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:34 smithi167 ceph-mon[33178]: from='client.14316 v1:172.21.15.52:0/3614486912' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:58:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:34 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:58:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:34 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:58:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:34 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:35.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:35 smithi052 ceph-mon[29471]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:35 smithi052 ceph-mon[36682]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:36.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:35 smithi167 ceph-mon[33178]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:36.924 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:36 smithi052 ceph-mon[36682]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T12:58:36.924 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:36 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3640637032' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9f6f1d22-be68-4d86-9d4b-86fd9e87d141"} : dispatch 2024-09-13T12:58:36.924 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:36 smithi052 ceph-mon[29471]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T12:58:36.924 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:36 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3640637032' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9f6f1d22-be68-4d86-9d4b-86fd9e87d141"} : dispatch 2024-09-13T12:58:37.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:36 smithi167 ceph-mon[33178]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T12:58:37.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:36 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3640637032' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9f6f1d22-be68-4d86-9d4b-86fd9e87d141"} : dispatch 2024-09-13T12:58:37.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:37 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3640637032' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f6f1d22-be68-4d86-9d4b-86fd9e87d141"}]': finished 2024-09-13T12:58:37.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:37 smithi052 ceph-mon[29471]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T12:58:37.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:37 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:37.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:37 smithi052 ceph-mon[29471]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:37.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:37 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/714525261' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:58:37.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:37 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3640637032' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f6f1d22-be68-4d86-9d4b-86fd9e87d141"}]': finished 2024-09-13T12:58:37.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:37 smithi052 ceph-mon[36682]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T12:58:37.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:37 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:37.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:37 smithi052 ceph-mon[36682]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:37.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:37 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/714525261' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:58:38.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:37 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3640637032' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f6f1d22-be68-4d86-9d4b-86fd9e87d141"}]': finished 2024-09-13T12:58:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:37 smithi167 ceph-mon[33178]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T12:58:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:37 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:37 smithi167 ceph-mon[33178]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:37 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/714525261' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:58:40.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:39 smithi052 ceph-mon[29471]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:39 smithi052 ceph-mon[36682]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:40.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:39 smithi167 ceph-mon[33178]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:42.153 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:41 smithi052 ceph-mon[36682]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:42.153 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:41 smithi052 ceph-mon[29471]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:42.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:41 smithi167 ceph-mon[33178]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:43.921 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:43 smithi052 ceph-mon[36682]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:43.922 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:43 smithi052 ceph-mon[29471]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:44.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:43 smithi167 ceph-mon[33178]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:44 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T12:58:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:44 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:44 smithi052 ceph-mon[29471]: Deploying daemon osd.3 on smithi052 2024-09-13T12:58:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:44 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T12:58:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:44 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:44 smithi052 ceph-mon[36682]: Deploying daemon osd.3 on smithi052 2024-09-13T12:58:45.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:44 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T12:58:45.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:44 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:45.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:44 smithi167 ceph-mon[33178]: Deploying daemon osd.3 on smithi052 2024-09-13T12:58:46.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:45 smithi052 ceph-mon[29471]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:46.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:45 smithi052 ceph-mon[36682]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:46.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:45 smithi167 ceph-mon[33178]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:47 smithi052 ceph-mon[36682]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:48.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:47 smithi052 ceph-mon[29471]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:48.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:47 smithi167 ceph-mon[33178]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:50.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:49 smithi052 ceph-mon[29471]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:50.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:49 smithi052 ceph-mon[36682]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:50.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:49 smithi167 ceph-mon[33178]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:52.081 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:51 smithi052 ceph-mon[36682]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:52.081 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:51 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:58:52.081 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:51 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:52.081 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:51 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:52.081 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:51 smithi052 ceph-mon[29471]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:52.081 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:51 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:58:52.081 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:51 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:52.081 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:51 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:52.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:51 smithi167 ceph-mon[33178]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:52.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:51 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:58:52.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:51 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:52.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:51 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:54.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[36682]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:54.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T12:58:54.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T12:58:54.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[36682]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T12:58:54.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:54.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:54.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:54.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:54.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:58:54.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:54.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[29471]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:54.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T12:58:54.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T12:58:54.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[29471]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T12:58:54.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:54.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:54.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:54.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:54.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:58:54.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:55.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:54 smithi167 ceph-mon[33178]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:54 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T12:58:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:54 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T12:58:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:54 smithi167 ceph-mon[33178]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T12:58:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:58:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:58:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:56.832 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:56 smithi052 ceph-mon[36682]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:56.832 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:56 smithi052 ceph-mon[36682]: from='osd.3 v1:172.21.15.52:6813/2025522885' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T12:58:56.832 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:56 smithi052 ceph-mon[36682]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T12:58:56.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:56 smithi052 ceph-mon[29471]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:56.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:56 smithi052 ceph-mon[29471]: from='osd.3 v1:172.21.15.52:6813/2025522885' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T12:58:56.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:56 smithi052 ceph-mon[29471]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T12:58:56.880 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 3 on host 'smithi052' 2024-09-13T12:58:57.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:56 smithi167 ceph-mon[33178]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:57.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:56 smithi167 ceph-mon[33178]: from='osd.3 v1:172.21.15.52:6813/2025522885' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T12:58:57.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:56 smithi167 ceph-mon[33178]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T12:58:57.627 DEBUG:teuthology.orchestra.run.smithi052:osd.3> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.3.service 2024-09-13T12:58:57.629 INFO:tasks.cephadm:Deploying osd.4 on smithi167 with /dev/vg_nvme/lv_4... 2024-09-13T12:58:57.629 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-13T12:58:57.873 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T12:58:57.933 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[36682]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T12:58:57.934 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[36682]: osdmap e25: 4 total, 3 up, 4 in 2024-09-13T12:58:57.934 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:57.934 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[36682]: from='osd.3 v1:172.21.15.52:6813/2025522885' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:58:57.934 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[36682]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:58:57.934 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[36682]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:57.934 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:58:57.934 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:57.934 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:57.934 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[29471]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T12:58:57.934 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[29471]: osdmap e25: 4 total, 3 up, 4 in 2024-09-13T12:58:57.935 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:57.935 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[29471]: from='osd.3 v1:172.21.15.52:6813/2025522885' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:58:57.935 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[29471]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:58:57.935 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[29471]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:57.935 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:58:57.935 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:57.935 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:57.994 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:57 smithi167 ceph-mon[33178]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T12:58:57.994 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:57 smithi167 ceph-mon[33178]: osdmap e25: 4 total, 3 up, 4 in 2024-09-13T12:58:57.994 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:57.994 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:57 smithi167 ceph-mon[33178]: from='osd.3 v1:172.21.15.52:6813/2025522885' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:58:57.994 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:57 smithi167 ceph-mon[33178]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]} : dispatch 2024-09-13T12:58:57.994 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:57 smithi167 ceph-mon[33178]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:57.995 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:58:57.995 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:57.995 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:57 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:58:58.239 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3[56695]: 2024-09-13T12:58:57.932+0000 7f0be3df1640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T12:58:58.239 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 12:58:57 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3[56695]: 2024-09-13T12:58:57.935+0000 7f0bdbc10640 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:58:58.985 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:58 smithi052 ceph-mon[36682]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-09-13T12:58:58.985 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:58 smithi052 ceph-mon[36682]: osdmap e26: 4 total, 3 up, 4 in 2024-09-13T12:58:58.985 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:58 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:58.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:58 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:58.986 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:58 smithi052 ceph-mon[36682]: from='osd.3 ' entity='osd.3' 2024-09-13T12:58:58.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:58 smithi052 ceph-mon[29471]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-09-13T12:58:58.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:58 smithi052 ceph-mon[29471]: osdmap e26: 4 total, 3 up, 4 in 2024-09-13T12:58:58.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:58 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:58.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:58 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:58.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:58 smithi052 ceph-mon[29471]: from='osd.3 ' entity='osd.3' 2024-09-13T12:58:59.009 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:58 smithi167 ceph-mon[33178]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-09-13T12:58:59.009 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:58 smithi167 ceph-mon[33178]: osdmap e26: 4 total, 3 up, 4 in 2024-09-13T12:58:59.009 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:58 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:59.009 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:58 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:59.009 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:58 smithi167 ceph-mon[33178]: from='osd.3 ' entity='osd.3' 2024-09-13T12:58:59.871 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:59 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T12:58:59.872 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:59 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T12:58:59.872 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:59.872 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:59 smithi167 ceph-mon[33178]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:59.872 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:59 smithi167 ceph-mon[33178]: osd.3 v1:172.21.15.52:6813/2025522885 boot 2024-09-13T12:58:59.872 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:59 smithi167 ceph-mon[33178]: osdmap e27: 4 total, 4 up, 4 in 2024-09-13T12:58:59.872 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:58:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:59.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T12:58:59.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T12:58:59.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:59.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[36682]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:59.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[36682]: osd.3 v1:172.21.15.52:6813/2025522885 boot 2024-09-13T12:58:59.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[36682]: osdmap e27: 4 total, 4 up, 4 in 2024-09-13T12:58:59.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:59.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T12:58:59.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T12:58:59.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:58:59.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[29471]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-13T12:58:59.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[29471]: osd.3 v1:172.21.15.52:6813/2025522885 boot 2024-09-13T12:58:59.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[29471]: osdmap e27: 4 total, 4 up, 4 in 2024-09-13T12:58:59.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:58:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T12:59:00.492 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:59:00.520 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi167:vg_nvme/lv_4 2024-09-13T12:59:00.771 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T12:59:01.144 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:00 smithi167 ceph-mon[33178]: osdmap e28: 4 total, 4 up, 4 in 2024-09-13T12:59:01.144 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:00 smithi167 ceph-mon[33178]: Detected new or changed devices on smithi052 2024-09-13T12:59:01.144 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:00 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:01.144 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:00 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:01.144 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:00 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:01.145 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:00 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:01.145 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:00 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:01.145 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:00 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:01.145 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:00 smithi167 ceph-mon[33178]: Adjusting osd_memory_target on smithi052 to 4017M 2024-09-13T12:59:01.145 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:00 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:01.145 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:00 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:01.145 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:00 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:01.145 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:00 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[36682]: osdmap e28: 4 total, 4 up, 4 in 2024-09-13T12:59:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[36682]: Detected new or changed devices on smithi052 2024-09-13T12:59:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[36682]: Adjusting osd_memory_target on smithi052 to 4017M 2024-09-13T12:59:01.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:01.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:01.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:01.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[29471]: osdmap e28: 4 total, 4 up, 4 in 2024-09-13T12:59:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[29471]: Detected new or changed devices on smithi052 2024-09-13T12:59:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:01.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[29471]: Adjusting osd_memory_target on smithi052 to 4017M 2024-09-13T12:59:01.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:01.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:01.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:01.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:00 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:02.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:02 smithi052 ceph-mon[29471]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:02.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:02 smithi052 ceph-mon[29471]: from='client.24235 v1:172.21.15.167:0/1352724071' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:59:02.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:02 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:59:02.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:02 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:59:02.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:02 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:02.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:02 smithi052 ceph-mon[36682]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:02.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:02 smithi052 ceph-mon[36682]: from='client.24235 v1:172.21.15.167:0/1352724071' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:59:02.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:02 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:59:02.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:02 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:59:02.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:02 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:02.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:02 smithi167 ceph-mon[33178]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:02.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:02 smithi167 ceph-mon[33178]: from='client.24235 v1:172.21.15.167:0/1352724071' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:59:02.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:02 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:59:02.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:02 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:59:02.742 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:02 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:04.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[36682]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:04.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/2934240213' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1326d850-caf4-484e-90a1-052f54f9f306"} : dispatch 2024-09-13T12:59:04.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[36682]: from='client.24241 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1326d850-caf4-484e-90a1-052f54f9f306"} : dispatch 2024-09-13T12:59:04.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[36682]: from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1326d850-caf4-484e-90a1-052f54f9f306"}]': finished 2024-09-13T12:59:04.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[36682]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T12:59:04.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:04.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/1859819892' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:59:04.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[29471]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:04.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/2934240213' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1326d850-caf4-484e-90a1-052f54f9f306"} : dispatch 2024-09-13T12:59:04.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[29471]: from='client.24241 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1326d850-caf4-484e-90a1-052f54f9f306"} : dispatch 2024-09-13T12:59:04.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[29471]: from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1326d850-caf4-484e-90a1-052f54f9f306"}]': finished 2024-09-13T12:59:04.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[29471]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T12:59:04.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:04.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:04 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/1859819892' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:59:05.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:04 smithi167 ceph-mon[33178]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:05.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:04 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/2934240213' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1326d850-caf4-484e-90a1-052f54f9f306"} : dispatch 2024-09-13T12:59:05.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:04 smithi167 ceph-mon[33178]: from='client.24241 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1326d850-caf4-484e-90a1-052f54f9f306"} : dispatch 2024-09-13T12:59:05.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:04 smithi167 ceph-mon[33178]: from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1326d850-caf4-484e-90a1-052f54f9f306"}]': finished 2024-09-13T12:59:05.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:04 smithi167 ceph-mon[33178]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T12:59:05.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:04 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:05.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:04 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/1859819892' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:59:06.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:06 smithi052 ceph-mon[29471]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:06.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:06 smithi052 ceph-mon[36682]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:07.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:06 smithi167 ceph-mon[33178]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:08.961 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:08 smithi167 ceph-mon[33178]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:08.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:08 smithi052 ceph-mon[29471]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:08 smithi052 ceph-mon[36682]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:10.755 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:10 smithi167 ceph-mon[33178]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:10.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:10 smithi052 ceph-mon[29471]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:10.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:10 smithi052 ceph-mon[36682]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:11.721 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:11 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T12:59:11.721 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:11 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:11 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T12:59:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:11 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:11 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T12:59:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:11 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:12.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:12 smithi052 ceph-mon[29471]: Deploying daemon osd.4 on smithi167 2024-09-13T12:59:12.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:12 smithi052 ceph-mon[29471]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:12.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:12 smithi052 ceph-mon[36682]: Deploying daemon osd.4 on smithi167 2024-09-13T12:59:12.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:12 smithi052 ceph-mon[36682]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:13.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:12 smithi167 ceph-mon[33178]: Deploying daemon osd.4 on smithi167 2024-09-13T12:59:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:12 smithi167 ceph-mon[33178]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:13.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:13 smithi167 ceph-mon[33178]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:13.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:13 smithi052 ceph-mon[29471]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:13.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:13 smithi052 ceph-mon[36682]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:16.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:15 smithi052 ceph-mon[29471]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:16.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:15 smithi052 ceph-mon[36682]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:16.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:15 smithi167 ceph-mon[33178]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:18.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:17 smithi052 ceph-mon[29471]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:18.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:17 smithi052 ceph-mon[36682]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:18.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:17 smithi167 ceph-mon[33178]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:19.220 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:18 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:59:19.220 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:18 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:19.220 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:18 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:18 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:59:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:18 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:18 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:19.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:18 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:59:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:18 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:18 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:20.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:19 smithi167 ceph-mon[33178]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:20.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:19 smithi052 ceph-mon[36682]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:20.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:19 smithi052 ceph-mon[29471]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:21.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:21 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:21.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:21 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:21.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:21 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:21.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:21 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:21.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:21 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:21.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:21 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:21.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:21 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:21.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:21 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:21.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:21 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:21.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:21 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:22.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:21 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:21 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:21 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:21 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:21 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:22.709 INFO:teuthology.orchestra.run.smithi167.stdout:Created osd(s) 4 on host 'smithi167' 2024-09-13T12:59:22.928 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:22 smithi167 ceph-mon[33178]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:22.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:22 smithi052 ceph-mon[36682]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:22.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:22 smithi052 ceph-mon[29471]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:23.254 DEBUG:teuthology.orchestra.run.smithi167:osd.4> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.4.service 2024-09-13T12:59:23.257 INFO:tasks.cephadm:Deploying osd.5 on smithi167 with /dev/vg_nvme/lv_3... 2024-09-13T12:59:23.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-13T12:59:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:23 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T12:59:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:23 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T12:59:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:23 smithi167 ceph-mon[33178]: osdmap e30: 5 total, 4 up, 5 in 2024-09-13T12:59:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:23 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:23 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:59:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:23 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:23 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:23 smithi167 ceph-mon[33178]: from='osd.4 v1:172.21.15.167:6800/2137138858' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-13T12:59:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:23 smithi167 ceph-mon[33178]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-13T12:59:23.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T12:59:23.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T12:59:23.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[29471]: osdmap e30: 5 total, 4 up, 5 in 2024-09-13T12:59:23.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:23.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:59:23.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:23.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:23.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[29471]: from='osd.4 v1:172.21.15.167:6800/2137138858' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-13T12:59:23.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[29471]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-13T12:59:23.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T12:59:23.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T12:59:23.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[36682]: osdmap e30: 5 total, 4 up, 5 in 2024-09-13T12:59:23.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:23.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:59:23.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:23.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:23.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[36682]: from='osd.4 v1:172.21.15.167:6800/2137138858' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-13T12:59:23.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:23 smithi052 ceph-mon[36682]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-13T12:59:24.385 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T12:59:24.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:24 smithi052 ceph-mon[29471]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:24.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:24 smithi052 ceph-mon[29471]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T12:59:24.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:24 smithi052 ceph-mon[29471]: from='osd.4 v1:172.21.15.167:6800/2137138858' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T12:59:24.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:24 smithi052 ceph-mon[29471]: osdmap e31: 5 total, 4 up, 5 in 2024-09-13T12:59:24.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:24 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:24.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:24 smithi052 ceph-mon[29471]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T12:59:24.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:24 smithi052 ceph-mon[36682]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:24.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:24 smithi052 ceph-mon[36682]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T12:59:24.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:24 smithi052 ceph-mon[36682]: from='osd.4 v1:172.21.15.167:6800/2137138858' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T12:59:24.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:24 smithi052 ceph-mon[36682]: osdmap e31: 5 total, 4 up, 5 in 2024-09-13T12:59:24.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:24 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:24.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:24 smithi052 ceph-mon[36682]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T12:59:25.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:24 smithi167 ceph-mon[33178]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:25.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:24 smithi167 ceph-mon[33178]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T12:59:25.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:24 smithi167 ceph-mon[33178]: from='osd.4 v1:172.21.15.167:6800/2137138858' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T12:59:25.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:24 smithi167 ceph-mon[33178]: osdmap e31: 5 total, 4 up, 5 in 2024-09-13T12:59:25.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:24 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:25.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:24 smithi167 ceph-mon[33178]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T12:59:25.071 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 12:59:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[37339]: 2024-09-13T12:59:24.713+0000 7f945322e640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T12:59:25.071 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 12:59:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[37339]: 2024-09-13T12:59:24.717+0000 7f944f055640 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:59:25.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:25 smithi167 ceph-mon[33178]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-09-13T12:59:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:25 smithi167 ceph-mon[33178]: osdmap e32: 5 total, 4 up, 5 in 2024-09-13T12:59:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:25 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:25 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:25 smithi167 ceph-mon[33178]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:25 smithi052 ceph-mon[29471]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-09-13T12:59:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:25 smithi052 ceph-mon[29471]: osdmap e32: 5 total, 4 up, 5 in 2024-09-13T12:59:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:25 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:25 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:25 smithi052 ceph-mon[29471]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:25 smithi052 ceph-mon[36682]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-09-13T12:59:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:25 smithi052 ceph-mon[36682]: osdmap e32: 5 total, 4 up, 5 in 2024-09-13T12:59:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:25 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:25 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:25 smithi052 ceph-mon[36682]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-13T12:59:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T12:59:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T12:59:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: osd.4 v1:172.21.15.167:6800/2137138858 boot 2024-09-13T12:59:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T12:59:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: Detected new or changed devices on smithi167 2024-09-13T12:59:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:26.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:26.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: Adjusting osd_memory_target on smithi167 to 17088M 2024-09-13T12:59:26.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:26.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:26.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:26.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:26 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T12:59:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T12:59:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: osd.4 v1:172.21.15.167:6800/2137138858 boot 2024-09-13T12:59:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T12:59:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: Detected new or changed devices on smithi167 2024-09-13T12:59:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:26.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:26.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: Adjusting osd_memory_target on smithi167 to 17088M 2024-09-13T12:59:26.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:26.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:26.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:26.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:26.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T12:59:26.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T12:59:26.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: osd.4 v1:172.21.15.167:6800/2137138858 boot 2024-09-13T12:59:26.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T12:59:26.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T12:59:26.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: Detected new or changed devices on smithi167 2024-09-13T12:59:26.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:26.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:26.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:26.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: Adjusting osd_memory_target on smithi167 to 17088M 2024-09-13T12:59:26.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:26.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:26.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:26.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:26 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:27.970 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:27 smithi167 ceph-mon[33178]: osdmap e34: 5 total, 5 up, 5 in 2024-09-13T12:59:27.971 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:27 smithi167 ceph-mon[33178]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:27.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:27 smithi052 ceph-mon[29471]: osdmap e34: 5 total, 5 up, 5 in 2024-09-13T12:59:27.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:27 smithi052 ceph-mon[29471]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:27.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:27 smithi052 ceph-mon[36682]: osdmap e34: 5 total, 5 up, 5 in 2024-09-13T12:59:27.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:27 smithi052 ceph-mon[36682]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:28.241 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:59:28.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi167:vg_nvme/lv_3 2024-09-13T12:59:28.507 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T12:59:28.867 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:28 smithi167 ceph-mon[33178]: osdmap e35: 5 total, 5 up, 5 in 2024-09-13T12:59:28.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:28 smithi052 ceph-mon[36682]: osdmap e35: 5 total, 5 up, 5 in 2024-09-13T12:59:28.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:28 smithi052 ceph-mon[29471]: osdmap e35: 5 total, 5 up, 5 in 2024-09-13T12:59:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:29 smithi052 ceph-mon[36682]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:29 smithi052 ceph-mon[36682]: from='client.24262 v1:172.21.15.167:0/3595448735' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:59:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:29 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:59:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:29 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:59:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:29 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:29.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:29 smithi052 ceph-mon[29471]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:29.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:29 smithi052 ceph-mon[29471]: from='client.24262 v1:172.21.15.167:0/3595448735' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:59:29.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:29 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:59:29.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:29 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:59:29.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:29 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:29 smithi167 ceph-mon[33178]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:29 smithi167 ceph-mon[33178]: from='client.24262 v1:172.21.15.167:0/3595448735' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:59:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:29 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:59:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:29 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:59:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:29 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:32.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:31 smithi167 ceph-mon[33178]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:31 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/3620592919' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2"} : dispatch 2024-09-13T12:59:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:31 smithi167 ceph-mon[33178]: from='client.24268 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2"} : dispatch 2024-09-13T12:59:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:31 smithi167 ceph-mon[33178]: from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2"}]': finished 2024-09-13T12:59:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:31 smithi167 ceph-mon[33178]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T12:59:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:31 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:32.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:31 smithi052 ceph-mon[29471]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:32.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:31 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/3620592919' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2"} : dispatch 2024-09-13T12:59:32.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:31 smithi052 ceph-mon[29471]: from='client.24268 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2"} : dispatch 2024-09-13T12:59:32.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:31 smithi052 ceph-mon[29471]: from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2"}]': finished 2024-09-13T12:59:32.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:31 smithi052 ceph-mon[29471]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T12:59:32.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:31 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:32.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:31 smithi052 ceph-mon[36682]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:32.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:31 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/3620592919' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2"} : dispatch 2024-09-13T12:59:32.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:31 smithi052 ceph-mon[36682]: from='client.24268 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2"} : dispatch 2024-09-13T12:59:32.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:31 smithi052 ceph-mon[36682]: from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2"}]': finished 2024-09-13T12:59:32.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:31 smithi052 ceph-mon[36682]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T12:59:32.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:31 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:33.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:32 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/3152633633' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:59:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:32 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/3152633633' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:59:33.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:32 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/3152633633' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T12:59:34.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:33 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T12:59:34.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:33 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T12:59:34.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:33 smithi052 ceph-mon[36682]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-09-13T12:59:34.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:33 smithi052 ceph-mon[36682]: osdmap e37: 6 total, 5 up, 6 in 2024-09-13T12:59:34.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:33 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:34.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:33 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T12:59:34.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:33 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T12:59:34.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:33 smithi052 ceph-mon[29471]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-09-13T12:59:34.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:33 smithi052 ceph-mon[29471]: osdmap e37: 6 total, 5 up, 6 in 2024-09-13T12:59:34.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:33 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:34.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:33 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T12:59:34.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:33 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T12:59:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:33 smithi167 ceph-mon[33178]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-09-13T12:59:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:33 smithi167 ceph-mon[33178]: osdmap e37: 6 total, 5 up, 6 in 2024-09-13T12:59:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:33 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:36.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:35 smithi052 ceph-mon[36682]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-13T12:59:36.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:35 smithi052 ceph-mon[36682]: 1.0 scrub starts 2024-09-13T12:59:36.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:35 smithi052 ceph-mon[36682]: 1.0 scrub ok 2024-09-13T12:59:36.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:35 smithi052 ceph-mon[29471]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-13T12:59:36.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:35 smithi052 ceph-mon[29471]: 1.0 scrub starts 2024-09-13T12:59:36.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:35 smithi052 ceph-mon[29471]: 1.0 scrub ok 2024-09-13T12:59:36.243 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:35 smithi167 ceph-mon[33178]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-13T12:59:36.243 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:35 smithi167 ceph-mon[33178]: 1.0 scrub starts 2024-09-13T12:59:36.243 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:35 smithi167 ceph-mon[33178]: 1.0 scrub ok 2024-09-13T12:59:38.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:37 smithi167 ceph-mon[33178]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T12:59:38.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:37 smithi052 ceph-mon[36682]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T12:59:38.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:37 smithi052 ceph-mon[29471]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T12:59:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:39 smithi052 ceph-mon[29471]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T12:59:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:39 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T12:59:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:39 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:39 smithi052 ceph-mon[29471]: Deploying daemon osd.5 on smithi167 2024-09-13T12:59:39.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:39 smithi052 ceph-mon[36682]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T12:59:39.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:39 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T12:59:39.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:39 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:39.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:39 smithi052 ceph-mon[36682]: Deploying daemon osd.5 on smithi167 2024-09-13T12:59:40.221 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:39 smithi167 ceph-mon[33178]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T12:59:40.221 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:39 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T12:59:40.221 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:39 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:40.222 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:39 smithi167 ceph-mon[33178]: Deploying daemon osd.5 on smithi167 2024-09-13T12:59:42.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:41 smithi167 ceph-mon[33178]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-13T12:59:42.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:41 smithi052 ceph-mon[29471]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-13T12:59:42.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:41 smithi052 ceph-mon[36682]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-13T12:59:44.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:43 smithi052 ceph-mon[29471]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:44.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:43 smithi052 ceph-mon[36682]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:44.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:43 smithi167 ceph-mon[33178]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:46.066 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:45 smithi167 ceph-mon[33178]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:46.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:45 smithi052 ceph-mon[36682]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:46.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:45 smithi052 ceph-mon[29471]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:47.035 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:46 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:59:47.035 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:46 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:47.035 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:46 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:46 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:59:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:46 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:46 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:47.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:46 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:59:47.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:46 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:47.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:46 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:48.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:48 smithi167 ceph-mon[33178]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:48.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:48 smithi052 ceph-mon[29471]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:48 smithi052 ceph-mon[36682]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:50.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:49 smithi052 ceph-mon[29471]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:50.257 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:50.257 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:50.257 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:50.257 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:50.257 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:50.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:49 smithi052 ceph-mon[36682]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:50.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:49 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:50.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:49 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:50.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:49 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:50.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:49 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:50.258 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:49 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:50.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:49 smithi167 ceph-mon[33178]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:50.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:50.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:50.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:50.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:50.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:51.389 INFO:teuthology.orchestra.run.smithi167.stdout:Created osd(s) 5 on host 'smithi167' 2024-09-13T12:59:52.023 DEBUG:teuthology.orchestra.run.smithi167:osd.5> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.5.service 2024-09-13T12:59:52.026 INFO:tasks.cephadm:Deploying osd.6 on smithi167 with /dev/vg_nvme/lv_2... 2024-09-13T12:59:52.026 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-13T12:59:52.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:51 smithi052 ceph-mon[29471]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:52.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:51 smithi052 ceph-mon[29471]: from='osd.5 v1:172.21.15.167:6804/2906884337' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-13T12:59:52.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:51 smithi052 ceph-mon[29471]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-13T12:59:52.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:51 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:59:52.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:51 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:52.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:51 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:52.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:51 smithi052 ceph-mon[36682]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:52.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:51 smithi052 ceph-mon[36682]: from='osd.5 v1:172.21.15.167:6804/2906884337' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-13T12:59:52.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:51 smithi052 ceph-mon[36682]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-13T12:59:52.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:51 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:59:52.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:51 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:52.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:51 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:52.290 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:51 smithi167 ceph-mon[33178]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:52.290 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:51 smithi167 ceph-mon[33178]: from='osd.5 v1:172.21.15.167:6804/2906884337' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-13T12:59:52.290 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:51 smithi167 ceph-mon[33178]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-13T12:59:52.290 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:51 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T12:59:52.290 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:51 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:52.290 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:51 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:53.094 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T12:59:53.249 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:52 smithi167 ceph-mon[33178]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T12:59:53.249 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:52 smithi167 ceph-mon[33178]: osdmap e38: 6 total, 5 up, 6 in 2024-09-13T12:59:53.249 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:52 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:53.249 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:52 smithi167 ceph-mon[33178]: from='osd.5 v1:172.21.15.167:6804/2906884337' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T12:59:53.250 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:52 smithi167 ceph-mon[33178]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T12:59:53.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:52 smithi052 ceph-mon[29471]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T12:59:53.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:52 smithi052 ceph-mon[29471]: osdmap e38: 6 total, 5 up, 6 in 2024-09-13T12:59:53.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:52 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:53.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:52 smithi052 ceph-mon[29471]: from='osd.5 v1:172.21.15.167:6804/2906884337' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T12:59:53.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:52 smithi052 ceph-mon[29471]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T12:59:53.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:52 smithi052 ceph-mon[36682]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T12:59:53.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:52 smithi052 ceph-mon[36682]: osdmap e38: 6 total, 5 up, 6 in 2024-09-13T12:59:53.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:52 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:53.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:52 smithi052 ceph-mon[36682]: from='osd.5 v1:172.21.15.167:6804/2906884337' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T12:59:53.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:52 smithi052 ceph-mon[36682]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T12:59:53.555 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 12:59:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T12:59:53.449+0000 7f68c53da640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T12:59:53.555 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 12:59:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T12:59:53.453+0000 7f68bd1f9640 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:59:54.200 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:53 smithi167 ceph-mon[33178]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:54.200 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:53 smithi167 ceph-mon[33178]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-09-13T12:59:54.200 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:53 smithi167 ceph-mon[33178]: osdmap e39: 6 total, 5 up, 6 in 2024-09-13T12:59:54.200 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:53 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:54.201 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:53 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:54.201 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:53 smithi167 ceph-mon[33178]: from='osd.5 ' entity='osd.5' 2024-09-13T12:59:54.201 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:53 smithi167 ceph-mon[33178]: osd.5 v1:172.21.15.167:6804/2906884337 boot 2024-09-13T12:59:54.201 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:53 smithi167 ceph-mon[33178]: osdmap e40: 6 total, 6 up, 6 in 2024-09-13T12:59:54.201 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:53 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:54.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[29471]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:54.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[29471]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-09-13T12:59:54.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[29471]: osdmap e39: 6 total, 5 up, 6 in 2024-09-13T12:59:54.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:54.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:54.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[29471]: from='osd.5 ' entity='osd.5' 2024-09-13T12:59:54.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[29471]: osd.5 v1:172.21.15.167:6804/2906884337 boot 2024-09-13T12:59:54.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[29471]: osdmap e40: 6 total, 6 up, 6 in 2024-09-13T12:59:54.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:54.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[36682]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-13T12:59:54.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[36682]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-09-13T12:59:54.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[36682]: osdmap e39: 6 total, 5 up, 6 in 2024-09-13T12:59:54.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:54.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:54.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[36682]: from='osd.5 ' entity='osd.5' 2024-09-13T12:59:54.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[36682]: osd.5 v1:172.21.15.167:6804/2906884337 boot 2024-09-13T12:59:54.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[36682]: osdmap e40: 6 total, 6 up, 6 in 2024-09-13T12:59:54.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:53 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T12:59:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:55 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T12:59:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:55 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T12:59:55.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:55 smithi167 ceph-mon[33178]: osdmap e41: 6 total, 6 up, 6 in 2024-09-13T12:59:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:54 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T12:59:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:54 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T12:59:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:54 smithi052 ceph-mon[36682]: osdmap e41: 6 total, 6 up, 6 in 2024-09-13T12:59:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:54 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T12:59:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:54 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T12:59:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:54 smithi052 ceph-mon[29471]: osdmap e41: 6 total, 6 up, 6 in 2024-09-13T12:59:56.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:56 smithi167 ceph-mon[33178]: pgmap v140: 1 pgs: 1 unknown; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-09-13T12:59:56.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:56 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:56.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:56 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:56.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:56 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:56.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:56 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:56.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:56 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:56.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:56 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:56.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:56 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:56.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:56 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[29471]: pgmap v140: 1 pgs: 1 unknown; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-09-13T12:59:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[36682]: pgmap v140: 1 pgs: 1 unknown; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-09-13T12:59:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T12:59:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T12:59:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:56 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T12:59:57.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:57 smithi167 ceph-mon[33178]: Detected new or changed devices on smithi167 2024-09-13T12:59:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:57 smithi167 ceph-mon[33178]: Adjusting osd_memory_target on smithi167 to 8544M 2024-09-13T12:59:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:57 smithi167 ceph-mon[33178]: osdmap e42: 6 total, 6 up, 6 in 2024-09-13T12:59:57.377 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T12:59:57.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi167:vg_nvme/lv_2 2024-09-13T12:59:57.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:57 smithi052 ceph-mon[36682]: Detected new or changed devices on smithi167 2024-09-13T12:59:57.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:57 smithi052 ceph-mon[36682]: Adjusting osd_memory_target on smithi167 to 8544M 2024-09-13T12:59:57.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:57 smithi052 ceph-mon[36682]: osdmap e42: 6 total, 6 up, 6 in 2024-09-13T12:59:57.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:57 smithi052 ceph-mon[29471]: Detected new or changed devices on smithi167 2024-09-13T12:59:57.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:57 smithi052 ceph-mon[29471]: Adjusting osd_memory_target on smithi167 to 8544M 2024-09-13T12:59:57.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:57 smithi052 ceph-mon[29471]: osdmap e42: 6 total, 6 up, 6 in 2024-09-13T12:59:57.646 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T12:59:58.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:58 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T12:59:58.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:58 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T12:59:58.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:58 smithi167 ceph-mon[33178]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-09-13T12:59:58.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:58 smithi167 ceph-mon[33178]: osdmap e43: 6 total, 6 up, 6 in 2024-09-13T12:59:58.389 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:58 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T12:59:58.389 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:58 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T12:59:58.389 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:58 smithi052 ceph-mon[36682]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-09-13T12:59:58.389 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:58 smithi052 ceph-mon[36682]: osdmap e43: 6 total, 6 up, 6 in 2024-09-13T12:59:58.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:58 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T12:59:58.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:58 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T12:59:58.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:58 smithi052 ceph-mon[29471]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-09-13T12:59:58.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:58 smithi052 ceph-mon[29471]: osdmap e43: 6 total, 6 up, 6 in 2024-09-13T12:59:59.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:59 smithi167 ceph-mon[33178]: from='client.24289 v1:172.21.15.167:0/3270956206' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:59:59.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:59:59.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:59:59.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 12:59:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:59.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:59 smithi052 ceph-mon[29471]: from='client.24289 v1:172.21.15.167:0/3270956206' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:59:59.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:59:59.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:59:59.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 12:59:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T12:59:59.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:59 smithi052 ceph-mon[36682]: from='client.24289 v1:172.21.15.167:0/3270956206' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T12:59:59.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T12:59:59.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T12:59:59.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 12:59:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:00.242 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:00 smithi167 ceph-mon[33178]: pgmap v145: 1 pgs: 1 unknown; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:00.242 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:00 smithi167 ceph-mon[33178]: overall HEALTH_OK 2024-09-13T13:00:00.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:00 smithi052 ceph-mon[29471]: pgmap v145: 1 pgs: 1 unknown; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:00.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:00 smithi052 ceph-mon[29471]: overall HEALTH_OK 2024-09-13T13:00:00.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:00 smithi052 ceph-mon[36682]: pgmap v145: 1 pgs: 1 unknown; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:00.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:00 smithi052 ceph-mon[36682]: overall HEALTH_OK 2024-09-13T13:00:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:01 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/1248588689' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2471907d-fcbd-44e0-9d29-54151ada9df0"} : dispatch 2024-09-13T13:00:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:01 smithi167 ceph-mon[33178]: from='client.24295 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2471907d-fcbd-44e0-9d29-54151ada9df0"} : dispatch 2024-09-13T13:00:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:01 smithi167 ceph-mon[33178]: from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2471907d-fcbd-44e0-9d29-54151ada9df0"}]': finished 2024-09-13T13:00:01.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:01 smithi167 ceph-mon[33178]: osdmap e44: 7 total, 6 up, 7 in 2024-09-13T13:00:01.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:01 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:01.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:01 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/1248588689' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2471907d-fcbd-44e0-9d29-54151ada9df0"} : dispatch 2024-09-13T13:00:01.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:01 smithi052 ceph-mon[29471]: from='client.24295 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2471907d-fcbd-44e0-9d29-54151ada9df0"} : dispatch 2024-09-13T13:00:01.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:01 smithi052 ceph-mon[29471]: from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2471907d-fcbd-44e0-9d29-54151ada9df0"}]': finished 2024-09-13T13:00:01.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:01 smithi052 ceph-mon[29471]: osdmap e44: 7 total, 6 up, 7 in 2024-09-13T13:00:01.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:01 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:01.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:01 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/1248588689' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2471907d-fcbd-44e0-9d29-54151ada9df0"} : dispatch 2024-09-13T13:00:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:01 smithi052 ceph-mon[36682]: from='client.24295 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2471907d-fcbd-44e0-9d29-54151ada9df0"} : dispatch 2024-09-13T13:00:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:01 smithi052 ceph-mon[36682]: from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2471907d-fcbd-44e0-9d29-54151ada9df0"}]': finished 2024-09-13T13:00:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:01 smithi052 ceph-mon[36682]: osdmap e44: 7 total, 6 up, 7 in 2024-09-13T13:00:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:01 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:02.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:02 smithi167 ceph-mon[33178]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-13T13:00:02.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:02 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/3227018071' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T13:00:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:02 smithi052 ceph-mon[29471]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-13T13:00:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:02 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/3227018071' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T13:00:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:02 smithi052 ceph-mon[36682]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-13T13:00:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:02 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/3227018071' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T13:00:04.176 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:04 smithi167 ceph-mon[33178]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-09-13T13:00:04.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:04 smithi052 ceph-mon[29471]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-09-13T13:00:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:04 smithi052 ceph-mon[36682]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-09-13T13:00:06.218 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:06 smithi167 ceph-mon[33178]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T13:00:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:06 smithi052 ceph-mon[29471]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T13:00:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:06 smithi052 ceph-mon[36682]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T13:00:07.300 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:07 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:00:07.300 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:07 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:00:07.300 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:07 smithi167 ceph-mon[33178]: osdmap e45: 7 total, 6 up, 7 in 2024-09-13T13:00:07.300 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:07.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:07 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:00:07.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:07 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:00:07.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:07 smithi052 ceph-mon[29471]: osdmap e45: 7 total, 6 up, 7 in 2024-09-13T13:00:07.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:07 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:00:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:07 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:00:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:07 smithi052 ceph-mon[36682]: osdmap e45: 7 total, 6 up, 7 in 2024-09-13T13:00:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:08 smithi167 ceph-mon[33178]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T13:00:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:08 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T13:00:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:08 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:08 smithi167 ceph-mon[33178]: Deploying daemon osd.6 on smithi167 2024-09-13T13:00:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:08 smithi052 ceph-mon[36682]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T13:00:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:08 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T13:00:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:08 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:08 smithi052 ceph-mon[36682]: Deploying daemon osd.6 on smithi167 2024-09-13T13:00:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:08 smithi052 ceph-mon[29471]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T13:00:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:08 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T13:00:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:08 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:08 smithi052 ceph-mon[29471]: Deploying daemon osd.6 on smithi167 2024-09-13T13:00:10.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:10 smithi167 ceph-mon[33178]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:10 smithi052 ceph-mon[36682]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:10 smithi052 ceph-mon[29471]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:12 smithi167 ceph-mon[33178]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:12.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:12 smithi052 ceph-mon[36682]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:12 smithi052 ceph-mon[29471]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:14.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:14 smithi052 ceph-mon[36682]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:14.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:14 smithi052 ceph-mon[29471]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:15.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:14 smithi167 ceph-mon[33178]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:15.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:15 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:15.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:15 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:15.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:15 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:15.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:15 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:15.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:15 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:15.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:15 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:15.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:15 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:15.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:15 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:15.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:15 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:16.813 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:16 smithi167 ceph-mon[33178]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:16.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:16 smithi052 ceph-mon[36682]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:16.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:16 smithi052 ceph-mon[29471]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:18 smithi052 ceph-mon[29471]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:18 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:18 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:18 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:18 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:18.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:18 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:18.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:18 smithi052 ceph-mon[36682]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:18.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:18 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:18.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:18 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:18.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:18 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:18.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:18 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:18.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:18 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:18.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:18 smithi167 ceph-mon[33178]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:18.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:18 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:18.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:18 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:18.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:18 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:18.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:18 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:18.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:18 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:19.572 INFO:teuthology.orchestra.run.smithi167.stdout:Created osd(s) 6 on host 'smithi167' 2024-09-13T13:00:20.276 DEBUG:teuthology.orchestra.run.smithi167:osd.6> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.6.service 2024-09-13T13:00:20.279 INFO:tasks.cephadm:Deploying osd.7 on smithi167 with /dev/vg_nvme/lv_1... 2024-09-13T13:00:20.279 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-13T13:00:20.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:20 smithi167 ceph-mon[33178]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:20 smithi167 ceph-mon[33178]: from='osd.6 v1:172.21.15.167:6808/756383534' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T13:00:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:20 smithi167 ceph-mon[33178]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T13:00:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:20 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:20 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:20 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:20 smithi052 ceph-mon[36682]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:20 smithi052 ceph-mon[36682]: from='osd.6 v1:172.21.15.167:6808/756383534' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T13:00:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:20 smithi052 ceph-mon[36682]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T13:00:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:20 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:20 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:20 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:20.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:20 smithi052 ceph-mon[29471]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:20.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:20 smithi052 ceph-mon[29471]: from='osd.6 v1:172.21.15.167:6808/756383534' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T13:00:20.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:20 smithi052 ceph-mon[29471]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T13:00:20.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:20 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:20.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:20 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:20.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:20 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:20.973 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:00:21.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:21 smithi052 ceph-mon[29471]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T13:00:21.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:21 smithi052 ceph-mon[29471]: from='osd.6 v1:172.21.15.167:6808/756383534' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:00:21.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:21 smithi052 ceph-mon[29471]: osdmap e46: 7 total, 6 up, 7 in 2024-09-13T13:00:21.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:21 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:21.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:21 smithi052 ceph-mon[29471]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:00:21.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:21 smithi052 ceph-mon[36682]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T13:00:21.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:21 smithi052 ceph-mon[36682]: from='osd.6 v1:172.21.15.167:6808/756383534' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:00:21.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:21 smithi052 ceph-mon[36682]: osdmap e46: 7 total, 6 up, 7 in 2024-09-13T13:00:21.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:21 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:21.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:21 smithi052 ceph-mon[36682]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:00:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:21 smithi167 ceph-mon[33178]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T13:00:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:21 smithi167 ceph-mon[33178]: from='osd.6 v1:172.21.15.167:6808/756383534' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:00:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:21 smithi167 ceph-mon[33178]: osdmap e46: 7 total, 6 up, 7 in 2024-09-13T13:00:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:21 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:21 smithi167 ceph-mon[33178]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:00:21.821 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:00:21 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[48313]: 2024-09-13T13:00:21.461+0000 7f05c20f6640 -1 osd.6 0 waiting for initial osdmap 2024-09-13T13:00:21.822 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:00:21 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[48313]: 2024-09-13T13:00:21.465+0000 7f05bdf1d640 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:00:22.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:22 smithi052 ceph-mon[29471]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:22.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:22 smithi052 ceph-mon[29471]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-09-13T13:00:22.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:22 smithi052 ceph-mon[29471]: osdmap e47: 7 total, 6 up, 7 in 2024-09-13T13:00:22.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:22 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:22.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:22 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:22.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:22 smithi052 ceph-mon[36682]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:22.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:22 smithi052 ceph-mon[36682]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-09-13T13:00:22.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:22 smithi052 ceph-mon[36682]: osdmap e47: 7 total, 6 up, 7 in 2024-09-13T13:00:22.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:22 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:22.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:22 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:22.752 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:22 smithi167 ceph-mon[33178]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-13T13:00:22.752 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:22 smithi167 ceph-mon[33178]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-09-13T13:00:22.752 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:22 smithi167 ceph-mon[33178]: osdmap e47: 7 total, 6 up, 7 in 2024-09-13T13:00:22.752 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:22 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:22.752 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:22 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:23 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:00:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:23 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:00:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:23 smithi052 ceph-mon[36682]: osd.6 v1:172.21.15.167:6808/756383534 boot 2024-09-13T13:00:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:23 smithi052 ceph-mon[36682]: osdmap e48: 7 total, 7 up, 7 in 2024-09-13T13:00:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:23 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:23 smithi052 ceph-mon[36682]: osdmap e49: 7 total, 7 up, 7 in 2024-09-13T13:00:23.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:23 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:00:23.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:23 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:00:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:23 smithi052 ceph-mon[29471]: osd.6 v1:172.21.15.167:6808/756383534 boot 2024-09-13T13:00:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:23 smithi052 ceph-mon[29471]: osdmap e48: 7 total, 7 up, 7 in 2024-09-13T13:00:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:23 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:23 smithi052 ceph-mon[29471]: osdmap e49: 7 total, 7 up, 7 in 2024-09-13T13:00:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:23 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:00:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:23 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:00:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:23 smithi167 ceph-mon[33178]: osd.6 v1:172.21.15.167:6808/756383534 boot 2024-09-13T13:00:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:23 smithi167 ceph-mon[33178]: osdmap e48: 7 total, 7 up, 7 in 2024-09-13T13:00:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:23 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:00:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:23 smithi167 ceph-mon[33178]: osdmap e49: 7 total, 7 up, 7 in 2024-09-13T13:00:24.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:24 smithi167 ceph-mon[33178]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:24.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:24 smithi167 ceph-mon[33178]: osdmap e50: 7 total, 7 up, 7 in 2024-09-13T13:00:24.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:24 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:24.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:24 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:24.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:24 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:24.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:24 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:24.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:24 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:24.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:24 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:24.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:24 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:24.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:24 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:24.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:24 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:24.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[29471]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:24.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[29471]: osdmap e50: 7 total, 7 up, 7 in 2024-09-13T13:00:24.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:24.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:24.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:24.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:24.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:24.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:24.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:24.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:24.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:24.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[36682]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:24.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[36682]: osdmap e50: 7 total, 7 up, 7 in 2024-09-13T13:00:24.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:24.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:24.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:24.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:24.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:24.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:24.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:24.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:24.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:24 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:25.901 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:25 smithi167 ceph-mon[33178]: Detected new or changed devices on smithi167 2024-09-13T13:00:25.901 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:25 smithi167 ceph-mon[33178]: Adjusting osd_memory_target on smithi167 to 5696M 2024-09-13T13:00:25.902 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:25 smithi167 ceph-mon[33178]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-09-13T13:00:25.932 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi167:vg_nvme/lv_1 2024-09-13T13:00:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:25 smithi052 ceph-mon[36682]: Detected new or changed devices on smithi167 2024-09-13T13:00:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:25 smithi052 ceph-mon[36682]: Adjusting osd_memory_target on smithi167 to 5696M 2024-09-13T13:00:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:25 smithi052 ceph-mon[36682]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:25 smithi052 ceph-mon[29471]: Detected new or changed devices on smithi167 2024-09-13T13:00:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:25 smithi052 ceph-mon[29471]: Adjusting osd_memory_target on smithi167 to 5696M 2024-09-13T13:00:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:25 smithi052 ceph-mon[29471]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:26.179 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:00:27.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:27 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:00:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:27 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:00:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:27 smithi167 ceph-mon[33178]: osdmap e51: 7 total, 7 up, 7 in 2024-09-13T13:00:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:27 smithi167 ceph-mon[33178]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:27 smithi167 ceph-mon[33178]: from='client.24316 v1:172.21.15.167:0/256756389' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:00:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:27 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T13:00:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:27 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T13:00:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:27 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:27.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:00:27.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:00:27.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[29471]: osdmap e51: 7 total, 7 up, 7 in 2024-09-13T13:00:27.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[29471]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:27.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[29471]: from='client.24316 v1:172.21.15.167:0/256756389' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:00:27.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T13:00:27.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T13:00:27.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:27.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:00:27.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:00:27.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[36682]: osdmap e51: 7 total, 7 up, 7 in 2024-09-13T13:00:27.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[36682]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:27.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[36682]: from='client.24316 v1:172.21.15.167:0/256756389' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:00:27.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-13T13:00:27.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-13T13:00:27.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:27 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:29 smithi167 ceph-mon[33178]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:29 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/474049073' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "97d2c6d7-7989-4c11-91e0-32ee87e76c6d"} : dispatch 2024-09-13T13:00:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:29 smithi167 ceph-mon[33178]: from='client.24322 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "97d2c6d7-7989-4c11-91e0-32ee87e76c6d"} : dispatch 2024-09-13T13:00:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:29 smithi167 ceph-mon[33178]: from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97d2c6d7-7989-4c11-91e0-32ee87e76c6d"}]': finished 2024-09-13T13:00:30.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:29 smithi167 ceph-mon[33178]: osdmap e52: 8 total, 7 up, 8 in 2024-09-13T13:00:30.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:29 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:30.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:29 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/747942386' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T13:00:30.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[36682]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:30.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/474049073' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "97d2c6d7-7989-4c11-91e0-32ee87e76c6d"} : dispatch 2024-09-13T13:00:30.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[36682]: from='client.24322 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "97d2c6d7-7989-4c11-91e0-32ee87e76c6d"} : dispatch 2024-09-13T13:00:30.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[36682]: from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97d2c6d7-7989-4c11-91e0-32ee87e76c6d"}]': finished 2024-09-13T13:00:30.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[36682]: osdmap e52: 8 total, 7 up, 8 in 2024-09-13T13:00:30.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:30.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/747942386' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T13:00:30.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[29471]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:30.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/474049073' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "97d2c6d7-7989-4c11-91e0-32ee87e76c6d"} : dispatch 2024-09-13T13:00:30.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[29471]: from='client.24322 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "97d2c6d7-7989-4c11-91e0-32ee87e76c6d"} : dispatch 2024-09-13T13:00:30.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[29471]: from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97d2c6d7-7989-4c11-91e0-32ee87e76c6d"}]': finished 2024-09-13T13:00:30.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[29471]: osdmap e52: 8 total, 7 up, 8 in 2024-09-13T13:00:30.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:30.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:29 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/747942386' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-13T13:00:32.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:31 smithi052 ceph-mon[36682]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:32.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:31 smithi052 ceph-mon[29471]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:32.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:31 smithi167 ceph-mon[33178]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:34.936 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:34 smithi167 ceph-mon[33178]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:34.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:34 smithi052 ceph-mon[29471]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:34 smithi052 ceph-mon[36682]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:36.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:36 smithi167 ceph-mon[33178]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:36.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:36 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T13:00:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:36 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:36.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:36 smithi052 ceph-mon[36682]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:36.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:36 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T13:00:36.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:36 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:36.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:36 smithi052 ceph-mon[29471]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:36.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:36 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T13:00:36.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:36 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:37.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:37 smithi052 ceph-mon[29471]: Deploying daemon osd.7 on smithi167 2024-09-13T13:00:37.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:37 smithi052 ceph-mon[36682]: Deploying daemon osd.7 on smithi167 2024-09-13T13:00:38.019 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:37 smithi167 ceph-mon[33178]: Deploying daemon osd.7 on smithi167 2024-09-13T13:00:38.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:38 smithi052 ceph-mon[36682]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:38.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:38 smithi052 ceph-mon[29471]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:39.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:38 smithi167 ceph-mon[33178]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:39 smithi052 ceph-mon[29471]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:39.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:39 smithi052 ceph-mon[36682]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:40.039 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:39 smithi167 ceph-mon[33178]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:42.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:41 smithi052 ceph-mon[29471]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:42.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:41 smithi052 ceph-mon[36682]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:42.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:41 smithi167 ceph-mon[33178]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:44.238 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:43 smithi052 ceph-mon[29471]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:44.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:43 smithi052 ceph-mon[36682]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:44.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:43 smithi167 ceph-mon[33178]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:44 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:44 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:44 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:44 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:44 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:44 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:45.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:44 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:44 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:44 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:46.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:46 smithi167 ceph-mon[33178]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:46.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:46 smithi052 ceph-mon[29471]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:46.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:46 smithi052 ceph-mon[36682]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:48.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:47 smithi167 ceph-mon[33178]: 1.0 scrub starts 2024-09-13T13:00:48.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:47 smithi167 ceph-mon[33178]: 1.0 scrub ok 2024-09-13T13:00:48.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:47 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:48.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:47 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:48.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:47 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:48.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:47 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:48.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:47 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:48.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:47 smithi167 ceph-mon[33178]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[36682]: 1.0 scrub starts 2024-09-13T13:00:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[36682]: 1.0 scrub ok 2024-09-13T13:00:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[36682]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[29471]: 1.0 scrub starts 2024-09-13T13:00:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[29471]: 1.0 scrub ok 2024-09-13T13:00:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:47 smithi052 ceph-mon[29471]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:49.766 INFO:teuthology.orchestra.run.smithi167.stdout:Created osd(s) 7 on host 'smithi167' 2024-09-13T13:00:50.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:49 smithi167 ceph-mon[33178]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:50.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:49 smithi167 ceph-mon[33178]: from='osd.7 v1:172.21.15.167:6812/403155155' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-13T13:00:50.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:49 smithi167 ceph-mon[33178]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-13T13:00:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:49 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:50.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:49 smithi052 ceph-mon[29471]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:50.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:49 smithi052 ceph-mon[29471]: from='osd.7 v1:172.21.15.167:6812/403155155' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-13T13:00:50.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:49 smithi052 ceph-mon[29471]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-13T13:00:50.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:50.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:50.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:49 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:50.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:49 smithi052 ceph-mon[36682]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:50.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:49 smithi052 ceph-mon[36682]: from='osd.7 v1:172.21.15.167:6812/403155155' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-13T13:00:50.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:49 smithi052 ceph-mon[36682]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-13T13:00:50.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:49 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:50.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:49 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:50.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:49 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:50.393 DEBUG:teuthology.orchestra.run.smithi167:osd.7> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.7.service 2024-09-13T13:00:50.396 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-13T13:00:50.396 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-13T13:00:50.655 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:00:50.920 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:50 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:00:50.920 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:50 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:00:50.920 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:50 smithi167 ceph-mon[33178]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T13:00:50.920 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:50 smithi167 ceph-mon[33178]: osdmap e53: 8 total, 7 up, 8 in 2024-09-13T13:00:50.920 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:50 smithi167 ceph-mon[33178]: from='osd.7 v1:172.21.15.167:6812/403155155' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:00:50.920 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:50 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:50.920 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:50 smithi167 ceph-mon[33178]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:00:50.920 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:50 smithi167 ceph-mon[33178]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-09-13T13:00:50.921 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:00:50 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:00:50.913+0000 7f823f33b640 -1 osd.7 0 waiting for initial osdmap 2024-09-13T13:00:50.921 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:00:50 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:00:50.919+0000 7f823b162640 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:00:51.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:00:51.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:00:51.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[36682]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T13:00:51.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[36682]: osdmap e53: 8 total, 7 up, 8 in 2024-09-13T13:00:51.031 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[36682]: from='osd.7 v1:172.21.15.167:6812/403155155' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:00:51.031 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:51.031 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[36682]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:00:51.031 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[36682]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-09-13T13:00:51.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:00:51.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:00:51.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[29471]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T13:00:51.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[29471]: osdmap e53: 8 total, 7 up, 8 in 2024-09-13T13:00:51.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[29471]: from='osd.7 v1:172.21.15.167:6812/403155155' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:00:51.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:51.032 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[29471]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:00:51.032 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:50 smithi052 ceph-mon[29471]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-09-13T13:00:51.408 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:00:51.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[29471]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:51.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[29471]: osdmap e54: 8 total, 7 up, 8 in 2024-09-13T13:00:51.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:51.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:51.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2596498332' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-13T13:00:51.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[29471]: osd.7 v1:172.21.15.167:6812/403155155 boot 2024-09-13T13:00:51.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[29471]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T13:00:51.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[36682]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[36682]: osdmap e54: 8 total, 7 up, 8 in 2024-09-13T13:00:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2596498332' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-13T13:00:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[36682]: osd.7 v1:172.21.15.167:6812/403155155 boot 2024-09-13T13:00:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[36682]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T13:00:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:51 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:52.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:51 smithi167 ceph-mon[33178]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T13:00:52.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:51 smithi167 ceph-mon[33178]: osdmap e54: 8 total, 7 up, 8 in 2024-09-13T13:00:52.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:51 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:52.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:51 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:52.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:51 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2596498332' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-13T13:00:52.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:51 smithi167 ceph-mon[33178]: osd.7 v1:172.21.15.167:6812/403155155 boot 2024-09-13T13:00:52.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:51 smithi167 ceph-mon[33178]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T13:00:52.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:51 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:00:52.122 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":54,"num_osds":8,"num_up_osds":7,"osd_up_since":1726232422,"num_in_osds":8,"osd_in_since":1726232429,"num_remapped_pgs":0} 2024-09-13T13:00:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:52 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:00:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:52 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:00:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:52 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:00:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:52 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:00:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:52 smithi052 ceph-mon[36682]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T13:00:52.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:52 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:00:52.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:52 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:00:52.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:52 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:00:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:52 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:00:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:52 smithi052 ceph-mon[29471]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T13:00:53.115 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:52 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:00:53.115 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:52 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:00:53.116 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:52 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:00:53.116 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:52 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:00:53.116 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:52 smithi167 ceph-mon[33178]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T13:00:53.123 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-13T13:00:53.387 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:00:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:00:54.333 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:00:54.333 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: Detected new or changed devices on smithi167 2024-09-13T13:00:54.333 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:54.334 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:54.334 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:54.334 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:54.334 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:54.334 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:54.334 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: Adjusting osd_memory_target on smithi167 to 4272M 2024-09-13T13:00:54.334 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:54.334 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:54.334 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:54.334 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:54.334 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T13:00:54.334 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1915704376' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: Detected new or changed devices on smithi167 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: Adjusting osd_memory_target on smithi167 to 4272M 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:54.335 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T13:00:54.336 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:54 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1915704376' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-13T13:00:54.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:00:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: Detected new or changed devices on smithi167 2024-09-13T13:00:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:00:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: Adjusting osd_memory_target on smithi167 to 4272M 2024-09-13T13:00:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:54.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T13:00:54.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:54 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1915704376' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-13T13:00:54.655 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1726232451,"num_in_osds":8,"osd_in_since":1726232429,"num_remapped_pgs":0} 2024-09-13T13:00:54.656 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T13:00:54.915 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:00:55.554 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:00:55.554 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":57,"fsid":"2b415b38-71cf-11ef-bceb-c7b262605968","created":"2024-09-13T12:55:27.893882+0000","modified":"2024-09-13T13:00:53.916902+0000","last_up_change":"2024-09-13T13:00:51.908805+0000","last_in_change":"2024-09-13T13:00:29.195772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T12:58:31.271893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d52dca1a-94c1-4e9e-93f5-687c038cd7c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6801","nonce":542298937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6802","nonce":542298937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6804","nonce":542298937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6803","nonce":542298937}]},"public_addr":"172.21.15.52:6801/542298937","cluster_addr":"172.21.15.52:6802/542298937","heartbeat_back_addr":"172.21.15.52:6804/542298937","heartbeat_front_addr":"172.21.15.52:6803/542298937","state":["exists","up"]},{"osd":1,"uuid":"b40a3574-3c6f-4810-a651-8d64aee6801b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6805","nonce":26289431}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6806","nonce":26289431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6808","nonce":26289431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6807","nonce":26289431}]},"public_addr":"172.21.15.52:6805/26289431","cluster_addr":"172.21.15.52:6806/26289431","heartbeat_back_addr":"172.21.15.52:6808/26289431","heartbeat_front_addr":"172.21.15.52:6807/26289431","state":["exists","up"]},{"osd":2,"uuid":"1e8e49a0-ccf8-4b7f-839b-104b64601408","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6809","nonce":3960240393}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6810","nonce":3960240393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6812","nonce":3960240393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6811","nonce":3960240393}]},"public_addr":"172.21.15.52:6809/3960240393","cluster_addr":"172.21.15.52:6810/3960240393","heartbeat_back_addr":"172.21.15.52:6812/3960240393","heartbeat_front_addr":"172.21.15.52:6811/3960240393","state":["exists","up"]},{"osd":3,"uuid":"9f6f1d22-be68-4d86-9d4b-86fd9e87d141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6813","nonce":2025522885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6814","nonce":2025522885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6816","nonce":2025522885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6815","nonce":2025522885}]},"public_addr":"172.21.15.52:6813/2025522885","cluster_addr":"172.21.15.52:6814/2025522885","heartbeat_back_addr":"172.21.15.52:6816/2025522885","heartbeat_front_addr":"172.21.15.52:6815/2025522885","state":["exists","up"]},{"osd":4,"uuid":"1326d850-caf4-484e-90a1-052f54f9f306","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6800","nonce":2137138858}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6801","nonce":2137138858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6803","nonce":2137138858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6802","nonce":2137138858}]},"public_addr":"172.21.15.167:6800/2137138858","cluster_addr":"172.21.15.167:6801/2137138858","heartbeat_back_addr":"172.21.15.167:6803/2137138858","heartbeat_front_addr":"172.21.15.167:6802/2137138858","state":["exists","up"]},{"osd":5,"uuid":"ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6804","nonce":2906884337}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6805","nonce":2906884337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6807","nonce":2906884337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6806","nonce":2906884337}]},"public_addr":"172.21.15.167:6804/2906884337","cluster_addr":"172.21.15.167:6805/2906884337","heartbeat_back_addr":"172.21.15.167:6807/2906884337","heartbeat_front_addr":"172.21.15.167:6806/2906884337","state":["exists","up"]},{"osd":6,"uuid":"2471907d-fcbd-44e0-9d29-54151ada9df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6808","nonce":756383534}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6809","nonce":756383534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6811","nonce":756383534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6810","nonce":756383534}]},"public_addr":"172.21.15.167:6808/756383534","cluster_addr":"172.21.15.167:6809/756383534","heartbeat_back_addr":"172.21.15.167:6811/756383534","heartbeat_front_addr":"172.21.15.167:6810/756383534","state":["exists","up"]},{"osd":7,"uuid":"97d2c6d7-7989-4c11-91e0-32ee87e76c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6812","nonce":403155155}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6813","nonce":403155155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6815","nonce":403155155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6814","nonce":403155155}]},"public_addr":"172.21.15.167:6812/403155155","cluster_addr":"172.21.15.167:6813/403155155","heartbeat_back_addr":"172.21.15.167:6815/403155155","heartbeat_front_addr":"172.21.15.167:6814/403155155","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:59:56.739358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:26.353499+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:51.946027+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:05.753950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:49.128440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:59:52.343188+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:20.527020+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:50.239295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/1933060029":"2024-09-14T12:56:14.802531+0000","172.21.15.52:6800/1437783219":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/3080924817":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/4248579079":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/4151877300":"2024-09-14T12:55:49.232470+0000","172.21.15.52:0/1471436716":"2024-09-14T12:55:49.232470+0000","172.21.15.52:6800/910649698":"2024-09-14T12:55:49.232470+0000","172.21.15.52:0/1887735845":"2024-09-14T12:55:49.232470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:00:56.305 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:56 smithi052 ceph-mon[36682]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:00:56.306 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:56 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2882755926' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:00:56.306 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:56 smithi052 ceph-mon[29471]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:00:56.306 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:56 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2882755926' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:00:56.307 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T12:58:31.271893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T13:00:56.308 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-13T13:00:56.568 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:00:56.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:56 smithi167 ceph-mon[33178]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:00:56.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:56 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2882755926' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:00:57.198 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2024-09-13T13:00:57.895 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi052 2024-09-13T13:00:57.895 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi052=foo.a' 2024-09-13T13:00:58.158 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:00:58.279 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:58 smithi167 ceph-mon[33178]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:00:58.279 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:58 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/997085812' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-13T13:00:58.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:58 smithi052 ceph-mon[29471]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:00:58.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:58 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/997085812' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-13T13:00:58.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:58 smithi052 ceph-mon[36682]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:00:58.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:58 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/997085812' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-13T13:00:58.805 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled rgw.foo.a update... 2024-09-13T13:00:59.384 DEBUG:teuthology.orchestra.run.smithi052:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@rgw.foo.a.service 2024-09-13T13:00:59.386 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi167 2024-09-13T13:00:59.386 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-13T13:00:59.408 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:00:59.409 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:00:59.409 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[36682]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T13:00:59.409 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:59.409 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:59.409 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:59.409 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:59.409 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:59.409 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-13T13:00:59.409 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-13T13:00:59.409 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:59.409 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:59.410 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:00:59.410 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:00:59.410 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[29471]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T13:00:59.410 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:59.410 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:59.410 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:59.410 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:59.410 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:59.410 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-13T13:00:59.410 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-13T13:00:59.410 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:59.410 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:00:59 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:59.411 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:59 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:00:59.411 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:59 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:00:59.411 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:59 smithi167 ceph-mon[33178]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T13:00:59.411 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:59.411 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:00:59.411 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:59.411 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:00:59.411 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:59.411 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-13T13:00:59.411 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-13T13:00:59.412 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:00:59.412 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:00:59 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:00:59.637 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:01:00.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:00 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:01:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:00 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:01:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:00 smithi167 ceph-mon[33178]: from='client.24367 v1:172.21.15.167:0/995547217' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi052=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:00 smithi167 ceph-mon[33178]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-09-13T13:01:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:00 smithi167 ceph-mon[33178]: Deploying daemon rgw.foo.a on smithi052 2024-09-13T13:01:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:00 smithi167 ceph-mon[33178]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:00.425 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:00 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:01:00.425 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:00 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:01:00.425 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:00 smithi052 ceph-mon[36682]: from='client.24367 v1:172.21.15.167:0/995547217' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi052=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:00.425 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:00 smithi052 ceph-mon[36682]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-09-13T13:01:00.425 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:00 smithi052 ceph-mon[36682]: Deploying daemon rgw.foo.a on smithi052 2024-09-13T13:01:00.425 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:00 smithi052 ceph-mon[36682]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:00.426 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:00 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:01:00.426 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:00 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:01:00.426 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:00 smithi052 ceph-mon[29471]: from='client.24367 v1:172.21.15.167:0/995547217' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi052=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:00.426 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:00 smithi052 ceph-mon[29471]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-09-13T13:01:00.426 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:00 smithi052 ceph-mon[29471]: Deploying daemon rgw.foo.a on smithi052 2024-09-13T13:01:00.426 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:00 smithi052 ceph-mon[29471]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:00.739 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:01:00 smithi052 systemd[1]: Starting Ceph rgw.foo.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:01:01.128 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:01:00 smithi052 podman[60991]: 2024-09-13 13:01:00.735029873 +0000 UTC m=+0.017224974 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:01:01.128 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:01:00 smithi052 podman[60991]: 2024-09-13 13:01:00.841437457 +0000 UTC m=+0.123632551 container create 213c35ecfe3fb5c3a47bb119660d8884fbc00d73801688aaf63cad32a113b7d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-13T13:01:01.129 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:01:01 smithi052 podman[60991]: 2024-09-13 13:01:01.061131859 +0000 UTC m=+0.343326946 container init 213c35ecfe3fb5c3a47bb119660d8884fbc00d73801688aaf63cad32a113b7d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-rgw-foo-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-13T13:01:01.129 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:01:01 smithi052 podman[60991]: 2024-09-13 13:01:01.064145386 +0000 UTC m=+0.346340472 container start 213c35ecfe3fb5c3a47bb119660d8884fbc00d73801688aaf63cad32a113b7d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-rgw-foo-a, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-13T13:01:01.129 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:01:01 smithi052 bash[60991]: 213c35ecfe3fb5c3a47bb119660d8884fbc00d73801688aaf63cad32a113b7d9 2024-09-13T13:01:01.173 INFO:teuthology.orchestra.run.smithi167.stderr:pool 'datapool' created 2024-09-13T13:01:01.487 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:01 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/3936964369' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-13T13:01:01.487 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:01 smithi167 ceph-mon[33178]: from='client.24373 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-13T13:01:01.489 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:01:01 smithi052 systemd[1]: Started Ceph rgw.foo.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:01:01.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:01 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/3936964369' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-13T13:01:01.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:01 smithi052 ceph-mon[29471]: from='client.24373 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-13T13:01:01.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:01 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/3936964369' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-13T13:01:01.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:01 smithi052 ceph-mon[36682]: from='client.24373 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-13T13:01:01.805 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- rbd pool init datapool 2024-09-13T13:01:02.100 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:01:02.209 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[36682]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:02.209 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[36682]: from='client.24373 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-13T13:01:02.209 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[36682]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T13:01:02.209 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1608406760' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-13T13:01:02.209 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[36682]: from='client.34376 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-13T13:01:02.210 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.210 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.210 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.210 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[36682]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-09-13T13:01:02.210 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.210 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.210 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:01:02.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[29471]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:02.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[29471]: from='client.24373 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-13T13:01:02.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[29471]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T13:01:02.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1608406760' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-13T13:01:02.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[29471]: from='client.34376 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-13T13:01:02.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[29471]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-09-13T13:01:02.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:02 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:01:02.508 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:02 smithi167 ceph-mon[33178]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:02.508 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:02 smithi167 ceph-mon[33178]: from='client.24373 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-13T13:01:02.508 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:02 smithi167 ceph-mon[33178]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T13:01:02.508 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:02 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1608406760' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-13T13:01:02.508 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:02 smithi167 ceph-mon[33178]: from='client.34376 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-13T13:01:02.508 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:02 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.509 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:02 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.509 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:02 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.509 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:02 smithi167 ceph-mon[33178]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-09-13T13:01:02.509 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:02 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.509 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:02 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:02.509 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:02 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:01:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[36682]: from='client.34376 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-13T13:01:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[36682]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T13:01:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/4027666595' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-13T13:01:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[36682]: from='client.24397 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-13T13:01:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:03.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:03.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:01:03.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:01:03.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[36682]: Checking dashboard <-> RGW credentials 2024-09-13T13:01:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[29471]: from='client.34376 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-13T13:01:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[29471]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T13:01:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/4027666595' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-13T13:01:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[29471]: from='client.24397 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-13T13:01:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:01:03.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:01:03.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:03 smithi052 ceph-mon[29471]: Checking dashboard <-> RGW credentials 2024-09-13T13:01:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:03 smithi167 ceph-mon[33178]: from='client.34376 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-13T13:01:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:03 smithi167 ceph-mon[33178]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T13:01:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:03 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/4027666595' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-13T13:01:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:03 smithi167 ceph-mon[33178]: from='client.24397 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-13T13:01:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:01:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:03 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:01:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:03 smithi167 ceph-mon[33178]: Checking dashboard <-> RGW credentials 2024-09-13T13:01:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:04 smithi052 ceph-mon[36682]: pgmap v194: 36 pgs: 14 creating+peering, 3 active+clean, 19 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:04 smithi052 ceph-mon[36682]: from='client.24397 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-13T13:01:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:04 smithi052 ceph-mon[36682]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T13:01:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:04 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-13T13:01:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:04 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-13T13:01:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:04 smithi052 ceph-mon[36682]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:01:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:04 smithi052 ceph-mon[29471]: pgmap v194: 36 pgs: 14 creating+peering, 3 active+clean, 19 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:04 smithi052 ceph-mon[29471]: from='client.24397 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-13T13:01:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:04 smithi052 ceph-mon[29471]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T13:01:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:04 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-13T13:01:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:04 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-13T13:01:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:04 smithi052 ceph-mon[29471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:01:04.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:04 smithi167 ceph-mon[33178]: pgmap v194: 36 pgs: 14 creating+peering, 3 active+clean, 19 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:04.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:04 smithi167 ceph-mon[33178]: from='client.24397 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-13T13:01:04.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:04 smithi167 ceph-mon[33178]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T13:01:04.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:04 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-13T13:01:04.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:04 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-13T13:01:04.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:04 smithi167 ceph-mon[33178]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:01:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:05 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-13T13:01:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:05 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-13T13:01:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:05 smithi052 ceph-mon[29471]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T13:01:05.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:05 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-13T13:01:05.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:05 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-13T13:01:05.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:05 smithi052 ceph-mon[36682]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T13:01:05.493 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:05 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-13T13:01:05.494 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:05 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-13T13:01:05.494 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:05 smithi167 ceph-mon[33178]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T13:01:05.740 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.167 --placement '1;smithi167=iscsi.a' 2024-09-13T13:01:05.991 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:01:06.423 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:06 smithi167 ceph-mon[33178]: pgmap v197: 68 pgs: 16 creating+peering, 28 active+clean, 24 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-13T13:01:06.423 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:06 smithi167 ceph-mon[33178]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:01:06.423 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:06 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:01:06.423 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:06 smithi167 ceph-mon[33178]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T13:01:06.423 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:06 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-13T13:01:06.423 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:06 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-13T13:01:06.423 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:06 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[36682]: pgmap v197: 68 pgs: 16 creating+peering, 28 active+clean, 24 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-13T13:01:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[36682]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:01:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:01:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[36682]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T13:01:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-13T13:01:06.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-13T13:01:06.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[29471]: pgmap v197: 68 pgs: 16 creating+peering, 28 active+clean, 24 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-13T13:01:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[29471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:01:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:01:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[29471]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T13:01:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-13T13:01:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-13T13:01:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:06 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:06.628 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled iscsi.datapool update... 2024-09-13T13:01:07.251 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:07 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-13T13:01:07.251 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:07 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-13T13:01:07.252 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:07 smithi167 ceph-mon[33178]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T13:01:07.252 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:07 smithi167 ceph-mon[33178]: from='client.24430 v1:172.21.15.167:0/2551252552' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.167", "placement": "1;smithi167=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:07.252 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:07 smithi167 ceph-mon[33178]: Saving service iscsi.datapool spec with placement smithi167=iscsi.a;count:1 2024-09-13T13:01:07.252 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:07 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:07.252 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:07 smithi167 ceph-mon[33178]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T13:01:07.252 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:07 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-13T13:01:07.252 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:07 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-13T13:01:07.278 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-13T13:01:07.278 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T13:01:07.278 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-13T13:01:07.313 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-09-13T13:01:07.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-13T13:01:07.340 DEBUG:teuthology.orchestra.run.smithi167:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@iscsi.iscsi.a.service 2024-09-13T13:01:07.383 INFO:tasks.cephadm:Adding prometheus.a on smithi167 2024-09-13T13:01:07.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi167=a' 2024-09-13T13:01:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-13T13:01:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-13T13:01:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[36682]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T13:01:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[36682]: from='client.24430 v1:172.21.15.167:0/2551252552' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.167", "placement": "1;smithi167=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[36682]: Saving service iscsi.datapool spec with placement smithi167=iscsi.a;count:1 2024-09-13T13:01:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[36682]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T13:01:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-13T13:01:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-13T13:01:07.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-13T13:01:07.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-13T13:01:07.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[29471]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T13:01:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[29471]: from='client.24430 v1:172.21.15.167:0/2551252552' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.167", "placement": "1;smithi167=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[29471]: Saving service iscsi.datapool spec with placement smithi167=iscsi.a;count:1 2024-09-13T13:01:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[29471]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T13:01:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-13T13:01:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:07 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-13T13:01:07.667 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:01:08.282 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled prometheus update... 2024-09-13T13:01:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[36682]: pgmap v200: 100 pgs: 14 creating+peering, 59 active+clean, 27 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-13T13:01:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-13T13:01:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-13T13:01:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[36682]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T13:01:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-13T13:01:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-13T13:01:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:08.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[29471]: pgmap v200: 100 pgs: 14 creating+peering, 59 active+clean, 27 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-13T13:01:08.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-13T13:01:08.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-13T13:01:08.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[29471]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T13:01:08.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-13T13:01:08.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-13T13:01:08.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:08 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:09.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:08 smithi167 ceph-mon[33178]: pgmap v200: 100 pgs: 14 creating+peering, 59 active+clean, 27 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-13T13:01:09.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-13T13:01:09.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-13T13:01:09.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:08 smithi167 ceph-mon[33178]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T13:01:09.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-13T13:01:09.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-13T13:01:09.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:08 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:09.120 DEBUG:teuthology.orchestra.run.smithi167:prometheus.a> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-13T13:01:09.122 INFO:tasks.cephadm:Adding node-exporter.a on smithi052 2024-09-13T13:01:09.122 INFO:tasks.cephadm:Adding node-exporter.b on smithi167 2024-09-13T13:01:09.123 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi052=a;smithi167=b' 2024-09-13T13:01:09.452 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:01:09.846 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:09 smithi167 ceph-mon[33178]: from='client.24436 v1:172.21.15.167:0/1309339475' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi167=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:09.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:09 smithi167 ceph-mon[33178]: Saving service prometheus spec with placement smithi167=a;count:1 2024-09-13T13:01:09.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:09 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-13T13:01:09.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:09 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-13T13:01:09.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:09 smithi167 ceph-mon[33178]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T13:01:09.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:09 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:09.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:09 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:09.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:09 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:09.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:09 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-13T13:01:09.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:09 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-13T13:01:09.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:09 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:01:09.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[29471]: from='client.24436 v1:172.21.15.167:0/1309339475' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi167=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:09.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[29471]: Saving service prometheus spec with placement smithi167=a;count:1 2024-09-13T13:01:09.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-13T13:01:09.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-13T13:01:09.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[29471]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T13:01:09.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:09.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:09.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:09.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-13T13:01:09.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-13T13:01:09.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:01:09.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[36682]: from='client.24436 v1:172.21.15.167:0/1309339475' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi167=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:09.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[36682]: Saving service prometheus spec with placement smithi167=a;count:1 2024-09-13T13:01:09.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1708534747' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-13T13:01:09.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2628990817' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-13T13:01:09.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[36682]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T13:01:09.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:09.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:09.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:09.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-13T13:01:09.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-13T13:01:09.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:09 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:01:10.148 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled node-exporter update... 2024-09-13T13:01:10.755 DEBUG:teuthology.orchestra.run.smithi052:node-exporter.a> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-13T13:01:10.759 DEBUG:teuthology.orchestra.run.smithi167:node-exporter.b> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-13T13:01:10.761 INFO:tasks.cephadm:Adding alertmanager.a on smithi052 2024-09-13T13:01:10.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi052=a' 2024-09-13T13:01:10.928 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:10 smithi167 ceph-mon[33178]: pgmap v203: 132 pgs: 12 creating+peering, 84 active+clean, 36 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-13T13:01:10.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:10 smithi167 ceph-mon[33178]: Deploying daemon iscsi.iscsi.a on smithi167 2024-09-13T13:01:10.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:10 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:10.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:10 smithi052 ceph-mon[36682]: pgmap v203: 132 pgs: 12 creating+peering, 84 active+clean, 36 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-13T13:01:10.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:10 smithi052 ceph-mon[36682]: Deploying daemon iscsi.iscsi.a on smithi167 2024-09-13T13:01:10.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:10 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:10.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:10 smithi052 ceph-mon[29471]: pgmap v203: 132 pgs: 12 creating+peering, 84 active+clean, 36 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-13T13:01:10.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:10 smithi052 ceph-mon[29471]: Deploying daemon iscsi.iscsi.a on smithi167 2024-09-13T13:01:10.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:10 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:11.158 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:01:11.970 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled alertmanager update... 2024-09-13T13:01:11.970 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:11 smithi167 ceph-mon[33178]: from='client.24457 v1:172.21.15.167:0/306948394' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi052=a;smithi167=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:11.970 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:11 smithi167 ceph-mon[33178]: Saving service node-exporter spec with placement smithi052=a;smithi167=b;count:2 2024-09-13T13:01:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:11 smithi052 ceph-mon[29471]: from='client.24457 v1:172.21.15.167:0/306948394' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi052=a;smithi167=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:11 smithi052 ceph-mon[29471]: Saving service node-exporter spec with placement smithi052=a;smithi167=b;count:2 2024-09-13T13:01:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:11 smithi052 ceph-mon[36682]: from='client.24457 v1:172.21.15.167:0/306948394' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi052=a;smithi167=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:11 smithi052 ceph-mon[36682]: Saving service node-exporter spec with placement smithi052=a;smithi167=b;count:2 2024-09-13T13:01:12.637 DEBUG:teuthology.orchestra.run.smithi052:alertmanager.a> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-13T13:01:12.640 INFO:tasks.cephadm:Adding grafana.a on smithi167 2024-09-13T13:01:12.640 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi167=a' 2024-09-13T13:01:12.859 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:12 smithi167 ceph-mon[33178]: pgmap v205: 132 pgs: 5 creating+peering, 113 active+clean, 14 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 3 KiB/s wr, 61 op/s 2024-09-13T13:01:12.859 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:12 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:12.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:12 smithi052 ceph-mon[36682]: pgmap v205: 132 pgs: 5 creating+peering, 113 active+clean, 14 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 3 KiB/s wr, 61 op/s 2024-09-13T13:01:12.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:12 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:12.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:12 smithi052 ceph-mon[29471]: pgmap v205: 132 pgs: 5 creating+peering, 113 active+clean, 14 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 3 KiB/s wr, 61 op/s 2024-09-13T13:01:12.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:12 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:13.261 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:01:13.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:13 smithi167 ceph-mon[33178]: from='client.24463 v1:172.21.15.167:0/763621863' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi052=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:13.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:13 smithi167 ceph-mon[33178]: Saving service alertmanager spec with placement smithi052=a;count:1 2024-09-13T13:01:13.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:13 smithi167 ceph-mon[33178]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T13:01:13.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:13 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:13.918 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:13 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:13.918 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:13 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:13.918 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:13 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:13.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[29471]: from='client.24463 v1:172.21.15.167:0/763621863' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi052=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:13.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[29471]: Saving service alertmanager spec with placement smithi052=a;count:1 2024-09-13T13:01:13.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[29471]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T13:01:13.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:13.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:13.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:13.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:13.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[36682]: from='client.24463 v1:172.21.15.167:0/763621863' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi052=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:13.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[36682]: Saving service alertmanager spec with placement smithi052=a;count:1 2024-09-13T13:01:13.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[36682]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T13:01:13.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:13.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:13.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:13.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:13 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:14.436 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled grafana update... 2024-09-13T13:01:14.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:14 smithi052 ceph-mon[36682]: pgmap v207: 132 pgs: 5 creating+peering, 127 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 5.3 KiB/s wr, 155 op/s 2024-09-13T13:01:14.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:14 smithi052 ceph-mon[36682]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-13T13:01:14.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:14 smithi052 ceph-mon[36682]: Deploying daemon prometheus.a on smithi167 2024-09-13T13:01:14.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:14 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/2757887406' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-13T13:01:14.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:14 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:14.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:14 smithi052 ceph-mon[29471]: pgmap v207: 132 pgs: 5 creating+peering, 127 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 5.3 KiB/s wr, 155 op/s 2024-09-13T13:01:14.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:14 smithi052 ceph-mon[29471]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-13T13:01:14.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:14 smithi052 ceph-mon[29471]: Deploying daemon prometheus.a on smithi167 2024-09-13T13:01:14.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:14 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/2757887406' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-13T13:01:14.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:14 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:15.014 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:14 smithi167 ceph-mon[33178]: pgmap v207: 132 pgs: 5 creating+peering, 127 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 5.3 KiB/s wr, 155 op/s 2024-09-13T13:01:15.014 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:14 smithi167 ceph-mon[33178]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-13T13:01:15.015 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:14 smithi167 ceph-mon[33178]: Deploying daemon prometheus.a on smithi167 2024-09-13T13:01:15.015 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:14 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/2757887406' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-13T13:01:15.015 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:14 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:15.044 DEBUG:teuthology.orchestra.run.smithi167:grafana.a> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@grafana.a.service 2024-09-13T13:01:15.046 INFO:tasks.cephadm:Setting up client nodes... 2024-09-13T13:01:15.047 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T13:01:15.319 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:15.703 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:15 smithi052 ceph-mon[29471]: from='client.24499 v1:172.21.15.167:0/1118828223' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi167=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:15.703 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:15 smithi052 ceph-mon[29471]: Saving service grafana spec with placement smithi167=a;count:1 2024-09-13T13:01:15.703 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:15 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:01:15.703 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:15 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:01:15.703 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:15 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:15.703 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:15 smithi052 ceph-mon[36682]: from='client.24499 v1:172.21.15.167:0/1118828223' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi167=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:15.703 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:15 smithi052 ceph-mon[36682]: Saving service grafana spec with placement smithi167=a;count:1 2024-09-13T13:01:15.703 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:15 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:01:15.703 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:15 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:01:15.704 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:15 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:16.066 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2024-09-13T13:01:16.066 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQCcN+RmYWO0AxAA6u0nSjheswq31vANYY3rFA== 2024-09-13T13:01:16.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:15 smithi167 ceph-mon[33178]: from='client.24499 v1:172.21.15.167:0/1118828223' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi167=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T13:01:16.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:15 smithi167 ceph-mon[33178]: Saving service grafana spec with placement smithi167=a;count:1 2024-09-13T13:01:16.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:15 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:01:16.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:15 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:01:16.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:15 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:16 smithi052 ceph-mon[36682]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 5.4 KiB/s wr, 167 op/s 2024-09-13T13:01:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:16 smithi052 ceph-mon[36682]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T13:01:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:16 smithi052 ceph-mon[36682]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-13T13:01:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:16 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1890808244' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-13T13:01:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:16 smithi052 ceph-mon[36682]: from='client.34499 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-13T13:01:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:16 smithi052 ceph-mon[36682]: from='client.34499 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T13:01:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:16 smithi052 ceph-mon[29471]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 5.4 KiB/s wr, 167 op/s 2024-09-13T13:01:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:16 smithi052 ceph-mon[29471]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T13:01:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:16 smithi052 ceph-mon[29471]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-13T13:01:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:16 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1890808244' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-13T13:01:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:16 smithi052 ceph-mon[29471]: from='client.34499 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-13T13:01:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:16 smithi052 ceph-mon[29471]: from='client.34499 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T13:01:16.776 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-13T13:01:16.776 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-13T13:01:16.776 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T13:01:16.818 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T13:01:17.067 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:01:17.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:16 smithi167 ceph-mon[33178]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 5.4 KiB/s wr, 167 op/s 2024-09-13T13:01:17.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:16 smithi167 ceph-mon[33178]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T13:01:17.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:16 smithi167 ceph-mon[33178]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-13T13:01:17.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:16 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1890808244' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-13T13:01:17.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:16 smithi167 ceph-mon[33178]: from='client.34499 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-13T13:01:17.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:16 smithi167 ceph-mon[33178]: from='client.34499 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T13:01:18.248 INFO:teuthology.orchestra.run.smithi167.stdout:[client.1] 2024-09-13T13:01:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQCeN+Rm0WZ+DhAAixZ19GpnYyqFrm/UPrcngg== 2024-09-13T13:01:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:18 smithi052 ceph-mon[36682]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.4 KiB/s wr, 145 op/s 2024-09-13T13:01:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:18 smithi052 ceph-mon[36682]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T13:01:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:18 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.167:0/980886112' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-13T13:01:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:18 smithi052 ceph-mon[36682]: from='client.24511 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-13T13:01:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:18 smithi052 ceph-mon[36682]: from='client.24511 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T13:01:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:18 smithi052 ceph-mon[29471]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.4 KiB/s wr, 145 op/s 2024-09-13T13:01:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:18 smithi052 ceph-mon[29471]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T13:01:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:18 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.167:0/980886112' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-13T13:01:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:18 smithi052 ceph-mon[29471]: from='client.24511 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-13T13:01:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:18 smithi052 ceph-mon[29471]: from='client.24511 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T13:01:19.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:18 smithi167 ceph-mon[33178]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.4 KiB/s wr, 145 op/s 2024-09-13T13:01:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:18 smithi167 ceph-mon[33178]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T13:01:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:18 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.167:0/980886112' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-13T13:01:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:18 smithi167 ceph-mon[33178]: from='client.24511 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-13T13:01:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:18 smithi167 ceph-mon[33178]: from='client.24511 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T13:01:19.133 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-09-13T13:01:19.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-13T13:01:19.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-13T13:01:19.175 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph config log 1 --format=json 2024-09-13T13:01:19.443 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:20.058 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:01:20.857 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:20 smithi052 ceph-mon[36682]: pgmap v212: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 1.3 KiB/s wr, 41 op/s 2024-09-13T13:01:20.857 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:20 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/379636644' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-13T13:01:20.857 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:20 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:20.858 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:20 smithi052 ceph-mon[29471]: pgmap v212: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 1.3 KiB/s wr, 41 op/s 2024-09-13T13:01:20.858 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:20 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/379636644' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-13T13:01:20.858 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:20 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:20.888 INFO:teuthology.orchestra.run.smithi052.stdout:[{"version":23,"timestamp":"2024-09-13T13:01:09.431647+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/RGW_API_SECRET_KEY","new_value":"45nFvYdDz3NjHUyE0Ktf0G68AnT2ECrVfI8T7ODN"}]}] 2024-09-13T13:01:20.889 INFO:tasks.ceph_manager:config epoch is 23 2024-09-13T13:01:20.889 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T13:01:20.889 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T13:01:20.889 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T13:01:21.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:20 smithi167 ceph-mon[33178]: pgmap v212: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 1.3 KiB/s wr, 41 op/s 2024-09-13T13:01:21.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:20 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/379636644' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-13T13:01:21.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:20 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:21.145 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:21.826 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:01:21.834 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:21 smithi052 ceph-mon[36682]: pgmap v213: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 1023 B/s wr, 31 op/s 2024-09-13T13:01:21.834 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:21 smithi052 ceph-mon[29471]: pgmap v213: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 1023 B/s wr, 31 op/s 2024-09-13T13:01:21.941 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:21 smithi167 ceph-mon[33178]: pgmap v213: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 1023 B/s wr, 31 op/s 2024-09-13T13:01:22.659 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6800","nonce":326637643}]},"active_addr":"172.21.15.52:6800/326637643","active_change":"2024-09-13T12:56:14.802637+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24115,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.52:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":991228238}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":3415403690}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1009267179}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":455730280}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2104786675}]}]} 2024-09-13T13:01:22.664 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T13:01:22.664 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T13:01:22.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T13:01:22.923 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:22 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/292352388' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T13:01:22.923 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:22 smithi167 ceph-mon[33178]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T13:01:22.935 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:22.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:22 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/292352388' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T13:01:22.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:22 smithi052 ceph-mon[36682]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T13:01:22.950 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:22 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/292352388' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T13:01:22.950 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:22 smithi052 ceph-mon[29471]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T13:01:23.599 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:01:23.600 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":71,"fsid":"2b415b38-71cf-11ef-bceb-c7b262605968","created":"2024-09-13T12:55:27.893882+0000","modified":"2024-09-13T13:01:22.674044+0000","last_up_change":"2024-09-13T13:00:51.908805+0000","last_in_change":"2024-09-13T13:00:29.195772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T12:58:31.271893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-13T13:01:00.306747+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-13T13:01:01.103523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-13T13:01:02.249986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-13T13:01:04.203764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-13T13:01:06.293676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d52dca1a-94c1-4e9e-93f5-687c038cd7c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6801","nonce":542298937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6802","nonce":542298937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6804","nonce":542298937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6803","nonce":542298937}]},"public_addr":"172.21.15.52:6801/542298937","cluster_addr":"172.21.15.52:6802/542298937","heartbeat_back_addr":"172.21.15.52:6804/542298937","heartbeat_front_addr":"172.21.15.52:6803/542298937","state":["exists","up"]},{"osd":1,"uuid":"b40a3574-3c6f-4810-a651-8d64aee6801b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6805","nonce":26289431}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6806","nonce":26289431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6808","nonce":26289431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6807","nonce":26289431}]},"public_addr":"172.21.15.52:6805/26289431","cluster_addr":"172.21.15.52:6806/26289431","heartbeat_back_addr":"172.21.15.52:6808/26289431","heartbeat_front_addr":"172.21.15.52:6807/26289431","state":["exists","up"]},{"osd":2,"uuid":"1e8e49a0-ccf8-4b7f-839b-104b64601408","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6809","nonce":3960240393}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6810","nonce":3960240393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6812","nonce":3960240393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6811","nonce":3960240393}]},"public_addr":"172.21.15.52:6809/3960240393","cluster_addr":"172.21.15.52:6810/3960240393","heartbeat_back_addr":"172.21.15.52:6812/3960240393","heartbeat_front_addr":"172.21.15.52:6811/3960240393","state":["exists","up"]},{"osd":3,"uuid":"9f6f1d22-be68-4d86-9d4b-86fd9e87d141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6813","nonce":2025522885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6814","nonce":2025522885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6816","nonce":2025522885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6815","nonce":2025522885}]},"public_addr":"172.21.15.52:6813/2025522885","cluster_addr":"172.21.15.52:6814/2025522885","heartbeat_back_addr":"172.21.15.52:6816/2025522885","heartbeat_front_addr":"172.21.15.52:6815/2025522885","state":["exists","up"]},{"osd":4,"uuid":"1326d850-caf4-484e-90a1-052f54f9f306","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6800","nonce":2137138858}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6801","nonce":2137138858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6803","nonce":2137138858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6802","nonce":2137138858}]},"public_addr":"172.21.15.167:6800/2137138858","cluster_addr":"172.21.15.167:6801/2137138858","heartbeat_back_addr":"172.21.15.167:6803/2137138858","heartbeat_front_addr":"172.21.15.167:6802/2137138858","state":["exists","up"]},{"osd":5,"uuid":"ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6804","nonce":2906884337}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6805","nonce":2906884337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6807","nonce":2906884337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6806","nonce":2906884337}]},"public_addr":"172.21.15.167:6804/2906884337","cluster_addr":"172.21.15.167:6805/2906884337","heartbeat_back_addr":"172.21.15.167:6807/2906884337","heartbeat_front_addr":"172.21.15.167:6806/2906884337","state":["exists","up"]},{"osd":6,"uuid":"2471907d-fcbd-44e0-9d29-54151ada9df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6808","nonce":756383534}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6809","nonce":756383534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6811","nonce":756383534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6810","nonce":756383534}]},"public_addr":"172.21.15.167:6808/756383534","cluster_addr":"172.21.15.167:6809/756383534","heartbeat_back_addr":"172.21.15.167:6811/756383534","heartbeat_front_addr":"172.21.15.167:6810/756383534","state":["exists","up"]},{"osd":7,"uuid":"97d2c6d7-7989-4c11-91e0-32ee87e76c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6812","nonce":403155155}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6813","nonce":403155155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6815","nonce":403155155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6814","nonce":403155155}]},"public_addr":"172.21.15.167:6812/403155155","cluster_addr":"172.21.15.167:6813/403155155","heartbeat_back_addr":"172.21.15.167:6815/403155155","heartbeat_front_addr":"172.21.15.167:6814/403155155","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:58.129304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:26.353499+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:51.946027+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:01:15.006021+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:49.128440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:58.475493+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:20.527020+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:50.239295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/1933060029":"2024-09-14T12:56:14.802531+0000","172.21.15.52:6800/1437783219":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/3080924817":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/4248579079":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/4151877300":"2024-09-14T12:55:49.232470+0000","172.21.15.52:0/1471436716":"2024-09-14T12:55:49.232470+0000","172.21.15.52:6800/910649698":"2024-09-14T12:55:49.232470+0000","172.21.15.52:0/1887735845":"2024-09-14T12:55:49.232470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:01:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:23 smithi167 ceph-mon[33178]: pgmap v215: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 142 B/s wr, 2 op/s 2024-09-13T13:01:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:23 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:01:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:23 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3288667004' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:01:23.970 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:23 smithi052 ceph-mon[36682]: pgmap v215: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 142 B/s wr, 2 op/s 2024-09-13T13:01:23.970 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:23 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:01:23.970 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:23 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3288667004' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:01:23.971 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:23 smithi052 ceph-mon[29471]: pgmap v215: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 142 B/s wr, 2 op/s 2024-09-13T13:01:23.971 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:23 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:01:23.971 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:23 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3288667004' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:01:24.268 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T13:01:24.268 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T13:01:24.526 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:24.959 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:24 smithi167 systemd[1]: Starting Ceph prometheus.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:01:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:01:25.202 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":71,"fsid":"2b415b38-71cf-11ef-bceb-c7b262605968","created":"2024-09-13T12:55:27.893882+0000","modified":"2024-09-13T13:01:22.674044+0000","last_up_change":"2024-09-13T13:00:51.908805+0000","last_in_change":"2024-09-13T13:00:29.195772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T12:58:31.271893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-13T13:01:00.306747+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-13T13:01:01.103523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-13T13:01:02.249986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-13T13:01:04.203764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-13T13:01:06.293676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d52dca1a-94c1-4e9e-93f5-687c038cd7c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6801","nonce":542298937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6802","nonce":542298937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6804","nonce":542298937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6803","nonce":542298937}]},"public_addr":"172.21.15.52:6801/542298937","cluster_addr":"172.21.15.52:6802/542298937","heartbeat_back_addr":"172.21.15.52:6804/542298937","heartbeat_front_addr":"172.21.15.52:6803/542298937","state":["exists","up"]},{"osd":1,"uuid":"b40a3574-3c6f-4810-a651-8d64aee6801b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6805","nonce":26289431}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6806","nonce":26289431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6808","nonce":26289431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6807","nonce":26289431}]},"public_addr":"172.21.15.52:6805/26289431","cluster_addr":"172.21.15.52:6806/26289431","heartbeat_back_addr":"172.21.15.52:6808/26289431","heartbeat_front_addr":"172.21.15.52:6807/26289431","state":["exists","up"]},{"osd":2,"uuid":"1e8e49a0-ccf8-4b7f-839b-104b64601408","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6809","nonce":3960240393}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6810","nonce":3960240393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6812","nonce":3960240393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6811","nonce":3960240393}]},"public_addr":"172.21.15.52:6809/3960240393","cluster_addr":"172.21.15.52:6810/3960240393","heartbeat_back_addr":"172.21.15.52:6812/3960240393","heartbeat_front_addr":"172.21.15.52:6811/3960240393","state":["exists","up"]},{"osd":3,"uuid":"9f6f1d22-be68-4d86-9d4b-86fd9e87d141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6813","nonce":2025522885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6814","nonce":2025522885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6816","nonce":2025522885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6815","nonce":2025522885}]},"public_addr":"172.21.15.52:6813/2025522885","cluster_addr":"172.21.15.52:6814/2025522885","heartbeat_back_addr":"172.21.15.52:6816/2025522885","heartbeat_front_addr":"172.21.15.52:6815/2025522885","state":["exists","up"]},{"osd":4,"uuid":"1326d850-caf4-484e-90a1-052f54f9f306","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6800","nonce":2137138858}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6801","nonce":2137138858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6803","nonce":2137138858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6802","nonce":2137138858}]},"public_addr":"172.21.15.167:6800/2137138858","cluster_addr":"172.21.15.167:6801/2137138858","heartbeat_back_addr":"172.21.15.167:6803/2137138858","heartbeat_front_addr":"172.21.15.167:6802/2137138858","state":["exists","up"]},{"osd":5,"uuid":"ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6804","nonce":2906884337}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6805","nonce":2906884337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6807","nonce":2906884337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6806","nonce":2906884337}]},"public_addr":"172.21.15.167:6804/2906884337","cluster_addr":"172.21.15.167:6805/2906884337","heartbeat_back_addr":"172.21.15.167:6807/2906884337","heartbeat_front_addr":"172.21.15.167:6806/2906884337","state":["exists","up"]},{"osd":6,"uuid":"2471907d-fcbd-44e0-9d29-54151ada9df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6808","nonce":756383534}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6809","nonce":756383534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6811","nonce":756383534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6810","nonce":756383534}]},"public_addr":"172.21.15.167:6808/756383534","cluster_addr":"172.21.15.167:6809/756383534","heartbeat_back_addr":"172.21.15.167:6811/756383534","heartbeat_front_addr":"172.21.15.167:6810/756383534","state":["exists","up"]},{"osd":7,"uuid":"97d2c6d7-7989-4c11-91e0-32ee87e76c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6812","nonce":403155155}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6813","nonce":403155155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6815","nonce":403155155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6814","nonce":403155155}]},"public_addr":"172.21.15.167:6812/403155155","cluster_addr":"172.21.15.167:6813/403155155","heartbeat_back_addr":"172.21.15.167:6815/403155155","heartbeat_front_addr":"172.21.15.167:6814/403155155","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:58.129304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:26.353499+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:51.946027+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:01:15.006021+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:49.128440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:58.475493+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:20.527020+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:00:50.239295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/1933060029":"2024-09-14T12:56:14.802531+0000","172.21.15.52:6800/1437783219":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/3080924817":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/4248579079":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/4151877300":"2024-09-14T12:55:49.232470+0000","172.21.15.52:0/1471436716":"2024-09-14T12:55:49.232470+0000","172.21.15.52:6800/910649698":"2024-09-14T12:55:49.232470+0000","172.21.15.52:0/1887735845":"2024-09-14T12:55:49.232470+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:01:25.216 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:24 smithi167 podman[59024]: 2024-09-13 13:01:24.958812019 +0000 UTC m=+0.105156610 container create ba51a23b04422e6c397e781589b781dc7fd9f933ad50acc068a07ee18d7438e5 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T13:01:25.216 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:24 smithi167 podman[59024]: 2024-09-13 13:01:24.870302879 +0000 UTC m=+0.016647483 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-13T13:01:25.216 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 podman[59024]: 2024-09-13 13:01:25.169165297 +0000 UTC m=+0.315509893 container init ba51a23b04422e6c397e781589b781dc7fd9f933ad50acc068a07ee18d7438e5 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T13:01:25.216 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 podman[59024]: 2024-09-13 13:01:25.171838263 +0000 UTC m=+0.318182853 container start ba51a23b04422e6c397e781589b781dc7fd9f933ad50acc068a07ee18d7438e5 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T13:01:25.216 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.191Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-13T13:01:25.216 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.191Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-13T13:01:25.216 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.191Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi167 (none))" 2024-09-13T13:01:25.216 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.191Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T13:01:25.216 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.191Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T13:01:25.216 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.193Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T13:01:25.216 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.193Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-13T13:01:25.216 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.195Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T13:01:25.217 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.196Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T13:01:25.217 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.197Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T13:01:25.217 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.197Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.672µs 2024-09-13T13:01:25.217 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.197Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T13:01:25.217 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.198Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-13T13:01:25.217 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.198Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=21.708µs wal_replay_duration=276.468µs wbl_replay_duration=111ns total_replay_duration=315.909µs 2024-09-13T13:01:25.217 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.199Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T13:01:25.217 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.199Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-13T13:01:25.217 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.199Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T13:01:25.217 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.215Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.617313ms db_storage=814ns remote_storage=1.091µs web_handler=179ns query_engine=555ns scrape=406.394µs scrape_sd=95.946µs notify=685ns notify_sd=560ns rules=14.783713ms tracing=6µs 2024-09-13T13:01:25.217 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.215Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-13T13:01:25.218 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:01:25.215Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T13:01:25.570 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 bash[59024]: ba51a23b04422e6c397e781589b781dc7fd9f933ad50acc068a07ee18d7438e5 2024-09-13T13:01:25.571 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:01:25 smithi167 systemd[1]: Started Ceph prometheus.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:01:25.944 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T13:01:25.945 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T13:01:25.945 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T13:01:25.946 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T13:01:25.946 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T13:01:25.947 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T13:01:25.948 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T13:01:25.948 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T13:01:26.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:25 smithi052 ceph-mon[36682]: pgmap v216: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-09-13T13:01:26.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:25 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2646521389' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:01:26.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:25 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:26.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:25 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:26.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:25 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:26.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:25 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-13T13:01:26.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:25 smithi052 ceph-mon[29471]: pgmap v216: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-09-13T13:01:26.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:25 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2646521389' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:01:26.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:25 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:26.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:25 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:26.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:25 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:26.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:25 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-13T13:01:26.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-mon[33178]: pgmap v216: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-09-13T13:01:26.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2646521389' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:01:26.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:26.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:26.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' 2024-09-13T13:01:26.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:25 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-13T13:01:26.558 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:26.562 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:26.563 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:26.563 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:26.565 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:26.566 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:26.567 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:26.585 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:26.727 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ignoring --setuser ceph since I am not root 2024-09-13T13:01:26.727 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ignoring --setgroup ceph since I am not root 2024-09-13T13:01:26.820 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 13:01:26 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x[34672]: ignoring --setuser ceph since I am not root 2024-09-13T13:01:26.820 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 13:01:26 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x[34672]: ignoring --setgroup ceph since I am not root 2024-09-13T13:01:27.654 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:27 smithi052 ceph-mon[36682]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T13:01:27.655 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:27 smithi052 ceph-mon[36682]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-13T13:01:27.655 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:27 smithi052 ceph-mon[29471]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T13:01:27.655 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:27 smithi052 ceph-mon[29471]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-13T13:01:27.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:27 smithi167 ceph-mon[33178]: from='mgr.14152 v1:172.21.15.52:0/800061561' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T13:01:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:27 smithi167 ceph-mon[33178]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-13T13:01:28.933 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:28 smithi052 ceph-mon[36682]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T13:01:28.937 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:28 smithi052 ceph-mon[29471]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T13:01:29.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:28 smithi167 ceph-mon[33178]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T13:01:29.977 INFO:teuthology.orchestra.run.smithi052.stdout:115964117025 2024-09-13T13:01:29.978 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T13:01:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:31 smithi167 ceph-mon[33178]: Standby manager daemon x restarted 2024-09-13T13:01:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:31 smithi167 ceph-mon[33178]: Standby manager daemon x started 2024-09-13T13:01:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:31 smithi167 ceph-mon[33178]: from='mgr.? v1:172.21.15.167:0/2121169355' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-13T13:01:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:31 smithi167 ceph-mon[33178]: from='mgr.? v1:172.21.15.167:0/2121169355' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-13T13:01:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:31 smithi167 ceph-mon[33178]: from='mgr.? v1:172.21.15.167:0/2121169355' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-13T13:01:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:31 smithi167 ceph-mon[33178]: from='mgr.? v1:172.21.15.167:0/2121169355' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-13T13:01:31.821 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 13:01:31 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x[34672]: [13/Sep/2024:13:01:31] ENGINE Bus STARTING 2024-09-13T13:01:31.821 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 13:01:31 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x[34672]: CherryPy Checker: 2024-09-13T13:01:31.822 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 13:01:31 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x[34672]: The Application mounted at '' has an empty config. 2024-09-13T13:01:31.822 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 13:01:31 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x[34672]: 2024-09-13T13:01:31.822 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 13:01:31 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x[34672]: [13/Sep/2024:13:01:31] ENGINE Serving on http://:::9283 2024-09-13T13:01:31.822 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 13:01:31 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x[34672]: [13/Sep/2024:13:01:31] ENGINE Bus STARTED 2024-09-13T13:01:31.850 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:31 smithi052 ceph-mon[36682]: Standby manager daemon x restarted 2024-09-13T13:01:31.850 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:31 smithi052 ceph-mon[36682]: Standby manager daemon x started 2024-09-13T13:01:31.850 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:31 smithi052 ceph-mon[36682]: from='mgr.? v1:172.21.15.167:0/2121169355' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-13T13:01:31.850 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:31 smithi052 ceph-mon[36682]: from='mgr.? v1:172.21.15.167:0/2121169355' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-13T13:01:31.850 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:31 smithi052 ceph-mon[36682]: from='mgr.? v1:172.21.15.167:0/2121169355' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-13T13:01:31.850 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:31 smithi052 ceph-mon[36682]: from='mgr.? v1:172.21.15.167:0/2121169355' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-13T13:01:31.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:31 smithi052 ceph-mon[29471]: Standby manager daemon x restarted 2024-09-13T13:01:31.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:31 smithi052 ceph-mon[29471]: Standby manager daemon x started 2024-09-13T13:01:31.851 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:31 smithi052 ceph-mon[29471]: from='mgr.? v1:172.21.15.167:0/2121169355' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-13T13:01:31.851 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:31 smithi052 ceph-mon[29471]: from='mgr.? v1:172.21.15.167:0/2121169355' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-13T13:01:31.851 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:31 smithi052 ceph-mon[29471]: from='mgr.? v1:172.21.15.167:0/2121169355' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-13T13:01:31.851 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:31 smithi052 ceph-mon[29471]: from='mgr.? v1:172.21.15.167:0/2121169355' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-13T13:01:31.891 INFO:teuthology.orchestra.run.smithi052.stdout:206158430222 2024-09-13T13:01:31.892 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T13:01:32.785 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-13T13:01:32.786 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: Active manager daemon y restarted 2024-09-13T13:01:32.786 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: Activating manager daemon y 2024-09-13T13:01:32.786 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T13:01:32.786 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: mgrmap e19: y(active, starting, since 0.00921369s), standbys: x 2024-09-13T13:01:32.786 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T13:01:32.786 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T13:01:32.786 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T13:01:32.786 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-13T13:01:32.786 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-13T13:01:32.786 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: Manager daemon y is now available 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:32.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:01:32.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:01:32.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T13:01:32.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T13:01:32.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T13:01:32.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T13:01:32.788 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-13T13:01:32.788 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: Active manager daemon y restarted 2024-09-13T13:01:32.788 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: Activating manager daemon y 2024-09-13T13:01:32.789 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T13:01:32.789 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: mgrmap e19: y(active, starting, since 0.00921369s), standbys: x 2024-09-13T13:01:32.789 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T13:01:32.789 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T13:01:32.789 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T13:01:32.789 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-13T13:01:32.789 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-13T13:01:32.789 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T13:01:32.789 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T13:01:32.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T13:01:32.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T13:01:32.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T13:01:32.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T13:01:32.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:01:32.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:01:32.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-13T13:01:32.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-13T13:01:32.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-13T13:01:32.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: Manager daemon y is now available 2024-09-13T13:01:32.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:01:32.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:32.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:01:32.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:01:32.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T13:01:32.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T13:01:32.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T13:01:32.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T13:01:32.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-13T13:01:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: Active manager daemon y restarted 2024-09-13T13:01:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: Activating manager daemon y 2024-09-13T13:01:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T13:01:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: mgrmap e19: y(active, starting, since 0.00921369s), standbys: x 2024-09-13T13:01:32.824 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T13:01:32.824 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T13:01:32.824 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T13:01:32.824 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: Manager daemon y is now available 2024-09-13T13:01:32.825 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:01:32.826 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:32.826 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:01:32.826 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:01:32.826 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T13:01:32.826 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T13:01:32.826 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T13:01:32.826 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:32 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T13:01:33.117 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:32] ENGINE Bus STARTING 2024-09-13T13:01:33.117 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: CherryPy Checker: 2024-09-13T13:01:33.117 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: The Application mounted at '' has an empty config. 2024-09-13T13:01:33.117 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:32 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: 2024-09-13T13:01:33.117 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:33] ENGINE Serving on http://:::9283 2024-09-13T13:01:33.118 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:33] ENGINE Bus STARTED 2024-09-13T13:01:33.163 INFO:teuthology.orchestra.run.smithi052.stdout:55834574891 2024-09-13T13:01:33.163 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T13:01:33.521 INFO:teuthology.orchestra.run.smithi052.stdout:34359738418 2024-09-13T13:01:33.522 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T13:01:33.784 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-mon[36682]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T13:01:33.784 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-mon[36682]: mgrmap e20: y(active, since 1.01253s), standbys: x 2024-09-13T13:01:33.784 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-mon[36682]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:01:33.784 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-mon[36682]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:33.784 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:01:33.784 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-mon[29471]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T13:01:33.785 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-mon[29471]: mgrmap e20: y(active, since 1.01253s), standbys: x 2024-09-13T13:01:33.785 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-mon[29471]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:01:33.785 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-mon[29471]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:33.785 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:01:33.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:33 smithi167 ceph-mon[33178]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T13:01:33.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:33 smithi167 ceph-mon[33178]: mgrmap e20: y(active, since 1.01253s), standbys: x 2024-09-13T13:01:33.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:33 smithi167 ceph-mon[33178]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:01:33.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:33 smithi167 ceph-mon[33178]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:33 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:01:34.074 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: 2024-09-13T13:01:33.914+0000 7f81c9701640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:13:01:33] ENGINE Error in HTTPServer.serve 2024-09-13T13:01:34.074 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: Traceback (most recent call last): 2024-09-13T13:01:34.074 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T13:01:34.074 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: self._connections.run(self.expiration_interval) 2024-09-13T13:01:34.074 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T13:01:34.074 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: self._run(expiration_interval) 2024-09-13T13:01:34.074 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T13:01:34.074 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T13:01:34.074 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T13:01:34.074 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T13:01:34.074 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T13:01:34.075 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: s = self.context.wrap_socket( 2024-09-13T13:01:34.075 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T13:01:34.075 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: return self.sslsocket_class._create( 2024-09-13T13:01:34.075 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T13:01:34.075 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: self.do_handshake() 2024-09-13T13:01:34.075 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T13:01:34.075 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: self._sslobj.do_handshake() 2024-09-13T13:01:34.075 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T13:01:34.075 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:33 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: 2024-09-13T13:01:34.104 INFO:teuthology.orchestra.run.smithi052.stdout:77309411364 2024-09-13T13:01:34.104 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T13:01:34.585 INFO:teuthology.orchestra.run.smithi052.stdout:141733920794 2024-09-13T13:01:34.586 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T13:01:34.605 INFO:teuthology.orchestra.run.smithi052.stdout:236223201288 2024-09-13T13:01:34.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T13:01:34.863 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[29471]: [13/Sep/2024:13:01:33] ENGINE Bus STARTING 2024-09-13T13:01:35.026 INFO:teuthology.orchestra.run.smithi052.stdout:171798691860 2024-09-13T13:01:35.027 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T13:01:35.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:34 smithi167 ceph-mon[33178]: [13/Sep/2024:13:01:33] ENGINE Bus STARTING 2024-09-13T13:01:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:34 smithi167 ceph-mon[33178]: [13/Sep/2024:13:01:33] ENGINE Serving on http://172.21.15.52:8765 2024-09-13T13:01:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:34 smithi167 ceph-mon[33178]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:34 smithi167 ceph-mon[33178]: [13/Sep/2024:13:01:33] ENGINE Serving on https://172.21.15.52:7150 2024-09-13T13:01:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:34 smithi167 ceph-mon[33178]: [13/Sep/2024:13:01:33] ENGINE Bus STARTED 2024-09-13T13:01:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:34 smithi167 ceph-mon[33178]: [13/Sep/2024:13:01:33] ENGINE Error in HTTPServer.serve 2024-09-13T13:01:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout: Traceback (most recent call last): 2024-09-13T13:01:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T13:01:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout: self._connections.run(self.expiration_interval) 2024-09-13T13:01:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T13:01:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout: self._run(expiration_interval) 2024-09-13T13:01:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T13:01:35.072 INFO:journalctl@ceph.mon.b.smithi167.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T13:01:35.072 INFO:journalctl@ceph.mon.b.smithi167.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T13:01:35.072 INFO:journalctl@ceph.mon.b.smithi167.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T13:01:35.072 INFO:journalctl@ceph.mon.b.smithi167.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T13:01:35.072 INFO:journalctl@ceph.mon.b.smithi167.stdout: s = self.context.wrap_socket( 2024-09-13T13:01:35.072 INFO:journalctl@ceph.mon.b.smithi167.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T13:01:35.072 INFO:journalctl@ceph.mon.b.smithi167.stdout: return self.sslsocket_class._create( 2024-09-13T13:01:35.072 INFO:journalctl@ceph.mon.b.smithi167.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T13:01:35.072 INFO:journalctl@ceph.mon.b.smithi167.stdout: self.do_handshake() 2024-09-13T13:01:35.072 INFO:journalctl@ceph.mon.b.smithi167.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T13:01:35.072 INFO:journalctl@ceph.mon.b.smithi167.stdout: self._sslobj.do_handshake() 2024-09-13T13:01:35.072 INFO:journalctl@ceph.mon.b.smithi167.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T13:01:35.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:34 smithi167 ceph-mon[33178]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-13T13:01:35.139 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[36682]: [13/Sep/2024:13:01:33] ENGINE Bus STARTING 2024-09-13T13:01:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[36682]: [13/Sep/2024:13:01:33] ENGINE Serving on http://172.21.15.52:8765 2024-09-13T13:01:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[36682]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[36682]: [13/Sep/2024:13:01:33] ENGINE Serving on https://172.21.15.52:7150 2024-09-13T13:01:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[36682]: [13/Sep/2024:13:01:33] ENGINE Bus STARTED 2024-09-13T13:01:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[36682]: [13/Sep/2024:13:01:33] ENGINE Error in HTTPServer.serve 2024-09-13T13:01:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout: Traceback (most recent call last): 2024-09-13T13:01:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: self._connections.run(self.expiration_interval) 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: self._run(expiration_interval) 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: s = self.context.wrap_socket( 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: return self.sslsocket_class._create( 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: self.do_handshake() 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T13:01:35.240 INFO:journalctl@ceph.mon.c.smithi052.stdout: self._sslobj.do_handshake() 2024-09-13T13:01:35.241 INFO:journalctl@ceph.mon.c.smithi052.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T13:01:35.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[36682]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-13T13:01:35.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[29471]: [13/Sep/2024:13:01:33] ENGINE Serving on http://172.21.15.52:8765 2024-09-13T13:01:35.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[29471]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:35.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[29471]: [13/Sep/2024:13:01:33] ENGINE Serving on https://172.21.15.52:7150 2024-09-13T13:01:35.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[29471]: [13/Sep/2024:13:01:33] ENGINE Bus STARTED 2024-09-13T13:01:35.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[29471]: [13/Sep/2024:13:01:33] ENGINE Error in HTTPServer.serve 2024-09-13T13:01:35.241 INFO:journalctl@ceph.mon.a.smithi052.stdout: Traceback (most recent call last): 2024-09-13T13:01:35.241 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T13:01:35.241 INFO:journalctl@ceph.mon.a.smithi052.stdout: self._connections.run(self.expiration_interval) 2024-09-13T13:01:35.241 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: self._run(expiration_interval) 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: s = self.context.wrap_socket( 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: return self.sslsocket_class._create( 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: self.do_handshake() 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: self._sslobj.do_handshake() 2024-09-13T13:01:35.242 INFO:journalctl@ceph.mon.a.smithi052.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T13:01:35.243 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:34 smithi052 ceph-mon[29471]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-13T13:01:35.436 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:35.436 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:35.889 INFO:teuthology.orchestra.run.smithi052.stdout:115964117026 2024-09-13T13:01:35.905 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:36.434 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:36.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:01:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:01:36.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2308812955' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T13:01:36.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:36 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2308812955' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T13:01:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:36.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:36.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:36 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2308812955' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T13:01:37.023 INFO:teuthology.orchestra.run.smithi052.stdout:206158430223 2024-09-13T13:01:37.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:37 smithi052 ceph-mon[29471]: Adjusting osd_memory_target on smithi167 to 3760M 2024-09-13T13:01:37.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:37 smithi052 ceph-mon[29471]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:37.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:37 smithi052 ceph-mon[29471]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-13T13:01:37.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:37 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3017556971' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T13:01:37.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:37 smithi052 ceph-mon[36682]: Adjusting osd_memory_target on smithi167 to 3760M 2024-09-13T13:01:37.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:37 smithi052 ceph-mon[36682]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:37.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:37 smithi052 ceph-mon[36682]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-13T13:01:37.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:37 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3017556971' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T13:01:37.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:37 smithi167 ceph-mon[33178]: Adjusting osd_memory_target on smithi167 to 3760M 2024-09-13T13:01:37.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:37 smithi167 ceph-mon[33178]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:01:37.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:37 smithi167 ceph-mon[33178]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-13T13:01:37.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:37 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3017556971' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T13:01:37.630 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117025 got 115964117026 for osd.3 2024-09-13T13:01:37.630 DEBUG:teuthology.parallel:result is None 2024-09-13T13:01:38.079 INFO:teuthology.orchestra.run.smithi052.stdout:34359738419 2024-09-13T13:01:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:38 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:01:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:38 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:01:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:38 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1884238347' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T13:01:38.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:38 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:01:38.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:38 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:01:38.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1884238347' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T13:01:38.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:38 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:01:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:38 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:01:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:38 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1884238347' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T13:01:39.009 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:39.158 INFO:teuthology.orchestra.run.smithi052.stdout:77309411366 2024-09-13T13:01:39.387 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430222 got 206158430223 for osd.6 2024-09-13T13:01:39.388 DEBUG:teuthology.parallel:result is None 2024-09-13T13:01:39.461 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:39 smithi052 ceph-mon[36682]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-13T13:01:39.461 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:39 smithi052 ceph-mon[36682]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T13:01:39.461 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:39 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1217312447' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T13:01:39.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:39 smithi052 ceph-mon[29471]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-13T13:01:39.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:39 smithi052 ceph-mon[29471]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T13:01:39.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:39 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1217312447' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T13:01:39.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:39 smithi167 ceph-mon[33178]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-13T13:01:39.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:39 smithi167 ceph-mon[33178]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T13:01:39.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:39 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1217312447' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T13:01:40.239 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411364 got 77309411366 for osd.2 2024-09-13T13:01:40.239 DEBUG:teuthology.parallel:result is None 2024-09-13T13:01:40.321 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:40.321 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:40.357 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738419 for osd.0 2024-09-13T13:01:40.357 DEBUG:teuthology.parallel:result is None 2024-09-13T13:01:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:41 smithi167 ceph-mon[33178]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T13:01:41.337 INFO:teuthology.orchestra.run.smithi052.stdout:141733920796 2024-09-13T13:01:41.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:41 smithi052 ceph-mon[36682]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T13:01:41.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:41 smithi052 ceph-mon[29471]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T13:01:41.550 INFO:teuthology.orchestra.run.smithi052.stdout:236223201290 2024-09-13T13:01:41.859 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920794 got 141733920796 for osd.4 2024-09-13T13:01:41.859 DEBUG:teuthology.parallel:result is None 2024-09-13T13:01:42.258 INFO:teuthology.orchestra.run.smithi052.stdout:171798691862 2024-09-13T13:01:42.349 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1808294054' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T13:01:42.349 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4010563437' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T13:01:42.349 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:42 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1808294054' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T13:01:42.350 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:42 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4010563437' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T13:01:42.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:42 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1808294054' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T13:01:42.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:42 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4010563437' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T13:01:42.959 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691860 got 171798691862 for osd.5 2024-09-13T13:01:42.959 DEBUG:teuthology.parallel:result is None 2024-09-13T13:01:43.392 INFO:teuthology.orchestra.run.smithi052.stdout:55834574893 2024-09-13T13:01:43.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:01:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T13:01:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3582141225' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T13:01:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T13:01:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:01:43.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:43 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:01:43.583 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T13:01:43.583 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3582141225' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T13:01:43.583 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.583 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.583 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T13:01:43.583 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.583 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.583 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:01:43.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:01:43.584 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T13:01:43.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3582141225' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T13:01:43.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T13:01:43.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.586 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.586 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.586 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T13:01:43.586 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:43.586 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:01:43.586 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:43 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:01:43.719 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201288 got 236223201290 for osd.7 2024-09-13T13:01:43.720 DEBUG:teuthology.parallel:result is None 2024-09-13T13:01:44.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[29471]: Adjusting osd_memory_target on smithi052 to 3761M 2024-09-13T13:01:44.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[29471]: Updating smithi052:/etc/ceph/ceph.conf 2024-09-13T13:01:44.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[29471]: Updating smithi167:/etc/ceph/ceph.conf 2024-09-13T13:01:44.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[29471]: Updating smithi052:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T13:01:44.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[29471]: Updating smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T13:01:44.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3026275321' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T13:01:44.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:01:44.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.344 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.344 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[36682]: Adjusting osd_memory_target on smithi052 to 3761M 2024-09-13T13:01:44.344 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[36682]: Updating smithi052:/etc/ceph/ceph.conf 2024-09-13T13:01:44.344 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[36682]: Updating smithi167:/etc/ceph/ceph.conf 2024-09-13T13:01:44.344 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[36682]: Updating smithi052:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T13:01:44.344 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[36682]: Updating smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T13:01:44.344 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3026275321' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T13:01:44.344 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:01:44.344 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.344 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.344 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.345 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.345 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:44 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.371 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574891 got 55834574893 for osd.1 2024-09-13T13:01:44.371 DEBUG:teuthology.parallel:result is None 2024-09-13T13:01:44.371 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T13:01:44.372 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:01:44.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:44 smithi167 ceph-mon[33178]: Adjusting osd_memory_target on smithi052 to 3761M 2024-09-13T13:01:44.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:44 smithi167 ceph-mon[33178]: Updating smithi052:/etc/ceph/ceph.conf 2024-09-13T13:01:44.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:44 smithi167 ceph-mon[33178]: Updating smithi167:/etc/ceph/ceph.conf 2024-09-13T13:01:44.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:44 smithi167 ceph-mon[33178]: Updating smithi052:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T13:01:44.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:44 smithi167 ceph-mon[33178]: Updating smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T13:01:44.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3026275321' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T13:01:44.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:44 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:01:44.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:44 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:44 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:44 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:44 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:44 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:44.641 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:45.345 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:01:45.345 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:45 smithi052 ceph-mon[36682]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-09-13T13:01:45.345 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:45 smithi052 ceph-mon[36682]: Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2024-09-13T13:01:45.345 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:45 smithi052 ceph-mon[36682]: Updating smithi052:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T13:01:45.346 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:45 smithi052 ceph-mon[36682]: Updating smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T13:01:45.346 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:45 smithi052 ceph-mon[36682]: Deploying daemon alertmanager.a on smithi052 2024-09-13T13:01:45.346 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:45 smithi052 ceph-mon[36682]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T13:01:45.346 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:45 smithi052 ceph-mon[29471]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-09-13T13:01:45.347 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:45 smithi052 ceph-mon[29471]: Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2024-09-13T13:01:45.347 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:45 smithi052 ceph-mon[29471]: Updating smithi052:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T13:01:45.347 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:45 smithi052 ceph-mon[29471]: Updating smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T13:01:45.347 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:45 smithi052 ceph-mon[29471]: Deploying daemon alertmanager.a on smithi052 2024-09-13T13:01:45.347 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:45 smithi052 ceph-mon[29471]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T13:01:45.351 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:01:45.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:45 smithi167 ceph-mon[33178]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-09-13T13:01:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:45 smithi167 ceph-mon[33178]: Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2024-09-13T13:01:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:45 smithi167 ceph-mon[33178]: Updating smithi052:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T13:01:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:45 smithi167 ceph-mon[33178]: Updating smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T13:01:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:45 smithi167 ceph-mon[33178]: Deploying daemon alertmanager.a on smithi052 2024-09-13T13:01:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:45 smithi167 ceph-mon[33178]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T13:01:46.417 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:46 smithi052 ceph-mon[36682]: from='client.14787 v1:172.21.15.52:0/160948439' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:01:46.417 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:01:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:01:46.418 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:46 smithi052 ceph-mon[29471]: from='client.14787 v1:172.21.15.52:0/160948439' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:01:46.444 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-13T13:01:43.871426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":883,"num_read_kb":740,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236380,"kb_used_data":7676,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724836,"statfs":{"total":767960285184,"available":767718232064,"internally_reserved":0,"allocated":7860224,"data_stored":4161609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":234016306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":186,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001708"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798870+0000","last_change":"2024-09-13T13:01:02.184818+0000","last_active":"2024-09-13T13:01:42.798870+0000","last_peered":"2024-09-13T13:01:42.798870+0000","last_clean":"2024-09-13T13:01:42.798870+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:01:42.798870+0000","last_undegraded":"2024-09-13T13:01:42.798870+0000","last_fullsized":"2024-09-13T13:01:42.798870+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.229908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":54,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.239854+0000","last_change":"2024-09-13T13:01:04.194784+0000","last_active":"2024-09-13T13:01:38.239854+0000","last_peered":"2024-09-13T13:01:38.239854+0000","last_clean":"2024-09-13T13:01:38.239854+0000","last_became_active":"2024-09-13T13:01:04.194444+0000","last_became_peered":"2024-09-13T13:01:04.194444+0000","last_unstale":"2024-09-13T13:01:38.239854+0000","last_undegraded":"2024-09-13T13:01:38.239854+0000","last_fullsized":"2024-09-13T13:01:38.239854+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.654746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.239895+0000","last_change":"2024-09-13T13:01:06.207968+0000","last_active":"2024-09-13T13:01:38.239895+0000","last_peered":"2024-09-13T13:01:38.239895+0000","last_clean":"2024-09-13T13:01:38.239895+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:01:38.239895+0000","last_undegraded":"2024-09-13T13:01:38.239895+0000","last_fullsized":"2024-09-13T13:01:38.239895+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.690479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240015+0000","last_change":"2024-09-13T13:01:08.214110+0000","last_active":"2024-09-13T13:01:38.240015+0000","last_peered":"2024-09-13T13:01:38.240015+0000","last_clean":"2024-09-13T13:01:38.240015+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:01:38.240015+0000","last_undegraded":"2024-09-13T13:01:38.240015+0000","last_fullsized":"2024-09-13T13:01:38.240015+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.859045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684177+0000","last_change":"2024-09-13T13:01:08.534893+0000","last_active":"2024-09-13T13:01:42.684177+0000","last_peered":"2024-09-13T13:01:42.684177+0000","last_clean":"2024-09-13T13:01:42.684177+0000","last_became_active":"2024-09-13T13:01:08.534558+0000","last_became_peered":"2024-09-13T13:01:08.534558+0000","last_unstale":"2024-09-13T13:01:42.684177+0000","last_undegraded":"2024-09-13T13:01:42.684177+0000","last_fullsized":"2024-09-13T13:01:42.684177+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.865474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684529+0000","last_change":"2024-09-13T13:01:02.183744+0000","last_active":"2024-09-13T13:01:42.684529+0000","last_peered":"2024-09-13T13:01:42.684529+0000","last_clean":"2024-09-13T13:01:42.684529+0000","last_became_active":"2024-09-13T13:01:02.183415+0000","last_became_peered":"2024-09-13T13:01:02.183415+0000","last_unstale":"2024-09-13T13:01:42.684529+0000","last_undegraded":"2024-09-13T13:01:42.684529+0000","last_fullsized":"2024-09-13T13:01:42.684529+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.940777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":65,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683479+0000","last_change":"2024-09-13T13:01:04.187590+0000","last_active":"2024-09-13T13:01:42.683479+0000","last_peered":"2024-09-13T13:01:42.683479+0000","last_clean":"2024-09-13T13:01:42.683479+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:01:42.683479+0000","last_undegraded":"2024-09-13T13:01:42.683479+0000","last_fullsized":"2024-09-13T13:01:42.683479+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.466652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.239850+0000","last_change":"2024-09-13T13:01:06.207701+0000","last_active":"2024-09-13T13:01:38.239850+0000","last_peered":"2024-09-13T13:01:38.239850+0000","last_clean":"2024-09-13T13:01:38.239850+0000","last_became_active":"2024-09-13T13:01:06.207429+0000","last_became_peered":"2024-09-13T13:01:06.207429+0000","last_unstale":"2024-09-13T13:01:38.239850+0000","last_undegraded":"2024-09-13T13:01:38.239850+0000","last_fullsized":"2024-09-13T13:01:38.239850+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.420247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":48,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275960+0000","last_change":"2024-09-13T13:01:02.184512+0000","last_active":"2024-09-13T13:01:38.275960+0000","last_peered":"2024-09-13T13:01:38.275960+0000","last_clean":"2024-09-13T13:01:38.275960+0000","last_became_active":"2024-09-13T13:01:02.184241+0000","last_became_peered":"2024-09-13T13:01:02.184241+0000","last_unstale":"2024-09-13T13:01:38.275960+0000","last_undegraded":"2024-09-13T13:01:38.275960+0000","last_fullsized":"2024-09-13T13:01:38.275960+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.275782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":54,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240316+0000","last_change":"2024-09-13T13:01:04.189504+0000","last_active":"2024-09-13T13:01:38.240316+0000","last_peered":"2024-09-13T13:01:38.240316+0000","last_clean":"2024-09-13T13:01:38.240316+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:01:38.240316+0000","last_undegraded":"2024-09-13T13:01:38.240316+0000","last_fullsized":"2024-09-13T13:01:38.240316+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:25.303489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275275+0000","last_change":"2024-09-13T13:01:06.205759+0000","last_active":"2024-09-13T13:01:38.275275+0000","last_peered":"2024-09-13T13:01:38.275275+0000","last_clean":"2024-09-13T13:01:38.275275+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:01:38.275275+0000","last_undegraded":"2024-09-13T13:01:38.275275+0000","last_fullsized":"2024-09-13T13:01:38.275275+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.930018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798267+0000","last_change":"2024-09-13T13:01:08.210285+0000","last_active":"2024-09-13T13:01:42.798267+0000","last_peered":"2024-09-13T13:01:42.798267+0000","last_clean":"2024-09-13T13:01:42.798267+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:01:42.798267+0000","last_undegraded":"2024-09-13T13:01:42.798267+0000","last_fullsized":"2024-09-13T13:01:42.798267+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:24.867052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.274842+0000","last_change":"2024-09-13T13:01:02.184552+0000","last_active":"2024-09-13T13:01:38.274842+0000","last_peered":"2024-09-13T13:01:38.274842+0000","last_clean":"2024-09-13T13:01:38.274842+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:01:38.274842+0000","last_undegraded":"2024-09-13T13:01:38.274842+0000","last_fullsized":"2024-09-13T13:01:38.274842+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:05.277064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":48,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240247+0000","last_change":"2024-09-13T13:01:04.196359+0000","last_active":"2024-09-13T13:01:38.240247+0000","last_peered":"2024-09-13T13:01:38.240247+0000","last_clean":"2024-09-13T13:01:38.240247+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:01:38.240247+0000","last_undegraded":"2024-09-13T13:01:38.240247+0000","last_fullsized":"2024-09-13T13:01:38.240247+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.916135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835910+0000","last_change":"2024-09-13T13:01:06.210845+0000","last_active":"2024-09-13T13:01:42.835910+0000","last_peered":"2024-09-13T13:01:42.835910+0000","last_clean":"2024-09-13T13:01:42.835910+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:01:42.835910+0000","last_undegraded":"2024-09-13T13:01:42.835910+0000","last_fullsized":"2024-09-13T13:01:42.835910+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.182827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.274126+0000","last_change":"2024-09-13T13:01:08.213608+0000","last_active":"2024-09-13T13:01:38.274126+0000","last_peered":"2024-09-13T13:01:38.274126+0000","last_clean":"2024-09-13T13:01:38.274126+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:01:38.274126+0000","last_undegraded":"2024-09-13T13:01:38.274126+0000","last_fullsized":"2024-09-13T13:01:38.274126+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.927149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":31,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240747+0000","last_change":"2024-09-13T13:01:08.533336+0000","last_active":"2024-09-13T13:01:38.240747+0000","last_peered":"2024-09-13T13:01:38.240747+0000","last_clean":"2024-09-13T13:01:38.240747+0000","last_became_active":"2024-09-13T13:01:08.533051+0000","last_became_peered":"2024-09-13T13:01:08.533051+0000","last_unstale":"2024-09-13T13:01:38.240747+0000","last_undegraded":"2024-09-13T13:01:38.240747+0000","last_fullsized":"2024-09-13T13:01:38.240747+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:31.568605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798921+0000","last_change":"2024-09-13T13:01:02.183913+0000","last_active":"2024-09-13T13:01:42.798921+0000","last_peered":"2024-09-13T13:01:42.798921+0000","last_clean":"2024-09-13T13:01:42.798921+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:01:42.798921+0000","last_undegraded":"2024-09-13T13:01:42.798921+0000","last_fullsized":"2024-09-13T13:01:42.798921+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.747175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":65,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683345+0000","last_change":"2024-09-13T13:01:04.191966+0000","last_active":"2024-09-13T13:01:42.683345+0000","last_peered":"2024-09-13T13:01:42.683345+0000","last_clean":"2024-09-13T13:01:42.683345+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:01:42.683345+0000","last_undegraded":"2024-09-13T13:01:42.683345+0000","last_fullsized":"2024-09-13T13:01:42.683345+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:05.477313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241442+0000","last_change":"2024-09-13T13:01:06.209495+0000","last_active":"2024-09-13T13:01:38.241442+0000","last_peered":"2024-09-13T13:01:38.241442+0000","last_clean":"2024-09-13T13:01:38.241442+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:01:38.241442+0000","last_undegraded":"2024-09-13T13:01:38.241442+0000","last_fullsized":"2024-09-13T13:01:38.241442+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.608220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684652+0000","last_change":"2024-09-13T13:01:08.533133+0000","last_active":"2024-09-13T13:01:42.684652+0000","last_peered":"2024-09-13T13:01:42.684652+0000","last_clean":"2024-09-13T13:01:42.684652+0000","last_became_active":"2024-09-13T13:01:08.532753+0000","last_became_peered":"2024-09-13T13:01:08.532753+0000","last_unstale":"2024-09-13T13:01:42.684652+0000","last_undegraded":"2024-09-13T13:01:42.684652+0000","last_fullsized":"2024-09-13T13:01:42.684652+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.674547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240594+0000","last_change":"2024-09-13T13:01:02.173465+0000","last_active":"2024-09-13T13:01:38.240594+0000","last_peered":"2024-09-13T13:01:38.240594+0000","last_clean":"2024-09-13T13:01:38.240594+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:01:38.240594+0000","last_undegraded":"2024-09-13T13:01:38.240594+0000","last_fullsized":"2024-09-13T13:01:38.240594+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:28.027613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684184+0000","last_change":"2024-09-13T13:01:04.195126+0000","last_active":"2024-09-13T13:01:42.684184+0000","last_peered":"2024-09-13T13:01:42.684184+0000","last_clean":"2024-09-13T13:01:42.684184+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:01:42.684184+0000","last_undegraded":"2024-09-13T13:01:42.684184+0000","last_fullsized":"2024-09-13T13:01:42.684184+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.577922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240293+0000","last_change":"2024-09-13T13:01:06.207164+0000","last_active":"2024-09-13T13:01:38.240293+0000","last_peered":"2024-09-13T13:01:38.240293+0000","last_clean":"2024-09-13T13:01:38.240293+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:01:38.240293+0000","last_undegraded":"2024-09-13T13:01:38.240293+0000","last_fullsized":"2024-09-13T13:01:38.240293+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.747378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":33,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836058+0000","last_change":"2024-09-13T13:01:08.210300+0000","last_active":"2024-09-13T13:01:42.836058+0000","last_peered":"2024-09-13T13:01:42.836058+0000","last_clean":"2024-09-13T13:01:42.836058+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:01:42.836058+0000","last_undegraded":"2024-09-13T13:01:42.836058+0000","last_fullsized":"2024-09-13T13:01:42.836058+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.173643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":43,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240059+0000","last_change":"2024-09-13T13:01:02.180198+0000","last_active":"2024-09-13T13:01:38.240059+0000","last_peered":"2024-09-13T13:01:38.240059+0000","last_clean":"2024-09-13T13:01:38.240059+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:01:38.240059+0000","last_undegraded":"2024-09-13T13:01:38.240059+0000","last_fullsized":"2024-09-13T13:01:38.240059+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:09.281312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798872+0000","last_change":"2024-09-13T13:01:04.208045+0000","last_active":"2024-09-13T13:01:42.798872+0000","last_peered":"2024-09-13T13:01:42.798872+0000","last_clean":"2024-09-13T13:01:42.798872+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:01:42.798872+0000","last_undegraded":"2024-09-13T13:01:42.798872+0000","last_fullsized":"2024-09-13T13:01:42.798872+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.686327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684767+0000","last_change":"2024-09-13T13:01:06.211329+0000","last_active":"2024-09-13T13:01:42.684767+0000","last_peered":"2024-09-13T13:01:42.684767+0000","last_clean":"2024-09-13T13:01:42.684767+0000","last_became_active":"2024-09-13T13:01:06.211034+0000","last_became_peered":"2024-09-13T13:01:06.211034+0000","last_unstale":"2024-09-13T13:01:42.684767+0000","last_undegraded":"2024-09-13T13:01:42.684767+0000","last_fullsized":"2024-09-13T13:01:42.684767+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:33.073914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":63,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241047+0000","last_change":"2024-09-13T13:01:04.198918+0000","last_active":"2024-09-13T13:01:38.241047+0000","last_peered":"2024-09-13T13:01:38.241047+0000","last_clean":"2024-09-13T13:01:38.241047+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:01:38.241047+0000","last_undegraded":"2024-09-13T13:01:38.241047+0000","last_fullsized":"2024-09-13T13:01:38.241047+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:04.636447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684050+0000","last_change":"2024-09-13T13:01:02.186847+0000","last_active":"2024-09-13T13:01:42.684050+0000","last_peered":"2024-09-13T13:01:42.684050+0000","last_clean":"2024-09-13T13:01:42.684050+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:01:42.684050+0000","last_undegraded":"2024-09-13T13:01:42.684050+0000","last_fullsized":"2024-09-13T13:01:42.684050+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.191632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":51,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240712+0000","last_change":"2024-09-13T13:01:06.206080+0000","last_active":"2024-09-13T13:01:38.240712+0000","last_peered":"2024-09-13T13:01:38.240712+0000","last_clean":"2024-09-13T13:01:38.240712+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:01:38.240712+0000","last_undegraded":"2024-09-13T13:01:38.240712+0000","last_fullsized":"2024-09-13T13:01:38.240712+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.738297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275593+0000","last_change":"2024-09-13T13:01:08.216021+0000","last_active":"2024-09-13T13:01:38.275593+0000","last_peered":"2024-09-13T13:01:38.275593+0000","last_clean":"2024-09-13T13:01:38.275593+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:01:38.275593+0000","last_undegraded":"2024-09-13T13:01:38.275593+0000","last_fullsized":"2024-09-13T13:01:38.275593+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:08.377856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":72,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683806+0000","last_change":"2024-09-13T13:01:04.209310+0000","last_active":"2024-09-13T13:01:42.683806+0000","last_peered":"2024-09-13T13:01:42.683806+0000","last_clean":"2024-09-13T13:01:42.683806+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:01:42.683806+0000","last_undegraded":"2024-09-13T13:01:42.683806+0000","last_fullsized":"2024-09-13T13:01:42.683806+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.025302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":50,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.685036+0000","last_change":"2024-09-13T13:01:02.181001+0000","last_active":"2024-09-13T13:01:42.685036+0000","last_peered":"2024-09-13T13:01:42.685036+0000","last_clean":"2024-09-13T13:01:42.685036+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:01:42.685036+0000","last_undegraded":"2024-09-13T13:01:42.685036+0000","last_fullsized":"2024-09-13T13:01:42.685036+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.444743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.239876+0000","last_change":"2024-09-13T13:01:06.209027+0000","last_active":"2024-09-13T13:01:38.239876+0000","last_peered":"2024-09-13T13:01:38.239876+0000","last_clean":"2024-09-13T13:01:38.239876+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:01:38.239876+0000","last_undegraded":"2024-09-13T13:01:38.239876+0000","last_fullsized":"2024-09-13T13:01:38.239876+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.392689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240168+0000","last_change":"2024-09-13T13:01:08.204782+0000","last_active":"2024-09-13T13:01:38.240168+0000","last_peered":"2024-09-13T13:01:38.240168+0000","last_clean":"2024-09-13T13:01:38.240168+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:01:38.240168+0000","last_undegraded":"2024-09-13T13:01:38.240168+0000","last_fullsized":"2024-09-13T13:01:38.240168+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.812714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":59,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.239915+0000","last_change":"2024-09-13T13:01:04.187117+0000","last_active":"2024-09-13T13:01:38.239915+0000","last_peered":"2024-09-13T13:01:38.239915+0000","last_clean":"2024-09-13T13:01:38.239915+0000","last_became_active":"2024-09-13T13:01:04.186768+0000","last_became_peered":"2024-09-13T13:01:04.186768+0000","last_unstale":"2024-09-13T13:01:38.239915+0000","last_undegraded":"2024-09-13T13:01:38.239915+0000","last_fullsized":"2024-09-13T13:01:38.239915+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.593884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":45,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.799045+0000","last_change":"2024-09-13T13:01:02.187792+0000","last_active":"2024-09-13T13:01:42.799045+0000","last_peered":"2024-09-13T13:01:42.799045+0000","last_clean":"2024-09-13T13:01:42.799045+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:01:42.799045+0000","last_undegraded":"2024-09-13T13:01:42.799045+0000","last_fullsized":"2024-09-13T13:01:42.799045+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:08.968885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683605+0000","last_change":"2024-09-13T13:01:06.211211+0000","last_active":"2024-09-13T13:01:42.683605+0000","last_peered":"2024-09-13T13:01:42.683605+0000","last_clean":"2024-09-13T13:01:42.683605+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:01:42.683605+0000","last_undegraded":"2024-09-13T13:01:42.683605+0000","last_fullsized":"2024-09-13T13:01:42.683605+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.975106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835987+0000","last_change":"2024-09-13T13:01:08.533116+0000","last_active":"2024-09-13T13:01:42.835987+0000","last_peered":"2024-09-13T13:01:42.835987+0000","last_clean":"2024-09-13T13:01:42.835987+0000","last_became_active":"2024-09-13T13:01:08.532859+0000","last_became_peered":"2024-09-13T13:01:08.532859+0000","last_unstale":"2024-09-13T13:01:42.835987+0000","last_undegraded":"2024-09-13T13:01:42.835987+0000","last_fullsized":"2024-09-13T13:01:42.835987+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:24.307795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":53,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240822+0000","last_change":"2024-09-13T13:01:04.199011+0000","last_active":"2024-09-13T13:01:38.240822+0000","last_peered":"2024-09-13T13:01:38.240822+0000","last_clean":"2024-09-13T13:01:38.240822+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:01:38.240822+0000","last_undegraded":"2024-09-13T13:01:38.240822+0000","last_fullsized":"2024-09-13T13:01:38.240822+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.407716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275995+0000","last_change":"2024-09-13T13:01:02.181597+0000","last_active":"2024-09-13T13:01:38.275995+0000","last_peered":"2024-09-13T13:01:38.275995+0000","last_clean":"2024-09-13T13:01:38.275995+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:01:38.275995+0000","last_undegraded":"2024-09-13T13:01:38.275995+0000","last_fullsized":"2024-09-13T13:01:38.275995+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.285259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798594+0000","last_change":"2024-09-13T13:01:06.216352+0000","last_active":"2024-09-13T13:01:42.798594+0000","last_peered":"2024-09-13T13:01:42.798594+0000","last_clean":"2024-09-13T13:01:42.798594+0000","last_became_active":"2024-09-13T13:01:06.216043+0000","last_became_peered":"2024-09-13T13:01:06.216043+0000","last_unstale":"2024-09-13T13:01:42.798594+0000","last_undegraded":"2024-09-13T13:01:42.798594+0000","last_fullsized":"2024-09-13T13:01:42.798594+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.006398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798385+0000","last_change":"2024-09-13T13:01:08.211109+0000","last_active":"2024-09-13T13:01:42.798385+0000","last_peered":"2024-09-13T13:01:42.798385+0000","last_clean":"2024-09-13T13:01:42.798385+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:01:42.798385+0000","last_undegraded":"2024-09-13T13:01:42.798385+0000","last_fullsized":"2024-09-13T13:01:42.798385+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.759653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":76,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798717+0000","last_change":"2024-09-13T13:01:04.207894+0000","last_active":"2024-09-13T13:01:42.798717+0000","last_peered":"2024-09-13T13:01:42.798717+0000","last_clean":"2024-09-13T13:01:42.798717+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:01:42.798717+0000","last_undegraded":"2024-09-13T13:01:42.798717+0000","last_fullsized":"2024-09-13T13:01:42.798717+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.794771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241382+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:01:38.241382+0000","last_peered":"2024-09-13T13:01:38.241382+0000","last_clean":"2024-09-13T13:01:38.241382+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:01:38.241382+0000","last_undegraded":"2024-09-13T13:01:38.241382+0000","last_fullsized":"2024-09-13T13:01:38.241382+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.453481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684301+0000","last_change":"2024-09-13T13:01:06.213926+0000","last_active":"2024-09-13T13:01:42.684301+0000","last_peered":"2024-09-13T13:01:42.684301+0000","last_clean":"2024-09-13T13:01:42.684301+0000","last_became_active":"2024-09-13T13:01:06.213617+0000","last_became_peered":"2024-09-13T13:01:06.213617+0000","last_unstale":"2024-09-13T13:01:42.684301+0000","last_undegraded":"2024-09-13T13:01:42.684301+0000","last_fullsized":"2024-09-13T13:01:42.684301+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.400692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.239855+0000","last_change":"2024-09-13T13:01:08.532524+0000","last_active":"2024-09-13T13:01:38.239855+0000","last_peered":"2024-09-13T13:01:38.239855+0000","last_clean":"2024-09-13T13:01:38.239855+0000","last_became_active":"2024-09-13T13:01:08.532145+0000","last_became_peered":"2024-09-13T13:01:08.532145+0000","last_unstale":"2024-09-13T13:01:38.239855+0000","last_undegraded":"2024-09-13T13:01:38.239855+0000","last_fullsized":"2024-09-13T13:01:38.239855+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:16.922652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":95,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241343+0000","last_change":"2024-09-13T13:01:04.197527+0000","last_active":"2024-09-13T13:01:38.241343+0000","last_peered":"2024-09-13T13:01:38.241343+0000","last_clean":"2024-09-13T13:01:38.241343+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:01:38.241343+0000","last_undegraded":"2024-09-13T13:01:38.241343+0000","last_fullsized":"2024-09-13T13:01:38.241343+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.799330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240454+0000","last_change":"2024-09-13T13:01:02.181566+0000","last_active":"2024-09-13T13:01:38.240454+0000","last_peered":"2024-09-13T13:01:38.240454+0000","last_clean":"2024-09-13T13:01:38.240454+0000","last_became_active":"2024-09-13T13:01:02.180760+0000","last_became_peered":"2024-09-13T13:01:02.180760+0000","last_unstale":"2024-09-13T13:01:38.240454+0000","last_undegraded":"2024-09-13T13:01:38.240454+0000","last_fullsized":"2024-09-13T13:01:38.240454+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.355888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":50,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.274764+0000","last_change":"2024-09-13T13:01:05.199998+0000","last_active":"2024-09-13T13:01:38.274764+0000","last_peered":"2024-09-13T13:01:38.274764+0000","last_clean":"2024-09-13T13:01:38.274764+0000","last_became_active":"2024-09-13T13:01:02.185089+0000","last_became_peered":"2024-09-13T13:01:02.185089+0000","last_unstale":"2024-09-13T13:01:38.274764+0000","last_undegraded":"2024-09-13T13:01:38.274764+0000","last_fullsized":"2024-09-13T13:01:38.274764+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.802534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798623+0000","last_change":"2024-09-13T13:01:06.216206+0000","last_active":"2024-09-13T13:01:42.798623+0000","last_peered":"2024-09-13T13:01:42.798623+0000","last_clean":"2024-09-13T13:01:42.798623+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:01:42.798623+0000","last_undegraded":"2024-09-13T13:01:42.798623+0000","last_fullsized":"2024-09-13T13:01:42.798623+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.187742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684361+0000","last_change":"2024-09-13T13:01:08.203445+0000","last_active":"2024-09-13T13:01:42.684361+0000","last_peered":"2024-09-13T13:01:42.684361+0000","last_clean":"2024-09-13T13:01:42.684361+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:01:42.684361+0000","last_undegraded":"2024-09-13T13:01:42.684361+0000","last_fullsized":"2024-09-13T13:01:42.684361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.430979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":65,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241139+0000","last_change":"2024-09-13T13:01:04.192923+0000","last_active":"2024-09-13T13:01:38.241139+0000","last_peered":"2024-09-13T13:01:38.241139+0000","last_clean":"2024-09-13T13:01:38.241139+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:01:38.241139+0000","last_undegraded":"2024-09-13T13:01:38.241139+0000","last_fullsized":"2024-09-13T13:01:38.241139+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.839147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240430+0000","last_change":"2024-09-13T13:01:02.185582+0000","last_active":"2024-09-13T13:01:38.240430+0000","last_peered":"2024-09-13T13:01:38.240430+0000","last_clean":"2024-09-13T13:01:38.240430+0000","last_became_active":"2024-09-13T13:01:02.185263+0000","last_became_peered":"2024-09-13T13:01:02.185263+0000","last_unstale":"2024-09-13T13:01:38.240430+0000","last_undegraded":"2024-09-13T13:01:38.240430+0000","last_fullsized":"2024-09-13T13:01:38.240430+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:28.109915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":51,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683107+0000","last_change":"2024-09-13T13:01:05.202103+0000","last_active":"2024-09-13T13:01:42.683107+0000","last_peered":"2024-09-13T13:01:42.683107+0000","last_clean":"2024-09-13T13:01:42.683107+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:01:42.683107+0000","last_undegraded":"2024-09-13T13:01:42.683107+0000","last_fullsized":"2024-09-13T13:01:42.683107+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.268994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683782+0000","last_change":"2024-09-13T13:01:06.205805+0000","last_active":"2024-09-13T13:01:42.683782+0000","last_peered":"2024-09-13T13:01:42.683782+0000","last_clean":"2024-09-13T13:01:42.683782+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:01:42.683782+0000","last_undegraded":"2024-09-13T13:01:42.683782+0000","last_fullsized":"2024-09-13T13:01:42.683782+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.555922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836396+0000","last_change":"2024-09-13T13:01:08.532390+0000","last_active":"2024-09-13T13:01:42.836396+0000","last_peered":"2024-09-13T13:01:42.836396+0000","last_clean":"2024-09-13T13:01:42.836396+0000","last_became_active":"2024-09-13T13:01:08.532087+0000","last_became_peered":"2024-09-13T13:01:08.532087+0000","last_unstale":"2024-09-13T13:01:42.836396+0000","last_undegraded":"2024-09-13T13:01:42.836396+0000","last_fullsized":"2024-09-13T13:01:42.836396+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.390192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798984+0000","last_change":"2024-09-13T13:01:04.193042+0000","last_active":"2024-09-13T13:01:42.798984+0000","last_peered":"2024-09-13T13:01:42.798984+0000","last_clean":"2024-09-13T13:01:42.798984+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:01:42.798984+0000","last_undegraded":"2024-09-13T13:01:42.798984+0000","last_fullsized":"2024-09-13T13:01:42.798984+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.415798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798373+0000","last_change":"2024-09-13T13:01:02.186331+0000","last_active":"2024-09-13T13:01:42.798373+0000","last_peered":"2024-09-13T13:01:42.798373+0000","last_clean":"2024-09-13T13:01:42.798373+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:01:42.798373+0000","last_undegraded":"2024-09-13T13:01:42.798373+0000","last_fullsized":"2024-09-13T13:01:42.798373+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.787015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":88,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835877+0000","last_change":"2024-09-13T13:01:05.200818+0000","last_active":"2024-09-13T13:01:42.835877+0000","last_peered":"2024-09-13T13:01:42.835877+0000","last_clean":"2024-09-13T13:01:42.835877+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:01:42.835877+0000","last_undegraded":"2024-09-13T13:01:42.835877+0000","last_fullsized":"2024-09-13T13:01:42.835877+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.587568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275829+0000","last_change":"2024-09-13T13:01:06.212898+0000","last_active":"2024-09-13T13:01:38.275829+0000","last_peered":"2024-09-13T13:01:38.275829+0000","last_clean":"2024-09-13T13:01:38.275829+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:01:38.275829+0000","last_undegraded":"2024-09-13T13:01:38.275829+0000","last_fullsized":"2024-09-13T13:01:38.275829+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.387930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241155+0000","last_change":"2024-09-13T13:01:08.213728+0000","last_active":"2024-09-13T13:01:38.241155+0000","last_peered":"2024-09-13T13:01:38.241155+0000","last_clean":"2024-09-13T13:01:38.241155+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:01:38.241155+0000","last_undegraded":"2024-09-13T13:01:38.241155+0000","last_fullsized":"2024-09-13T13:01:38.241155+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.763739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":71,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684817+0000","last_change":"2024-09-13T13:01:04.208034+0000","last_active":"2024-09-13T13:01:42.684817+0000","last_peered":"2024-09-13T13:01:42.684817+0000","last_clean":"2024-09-13T13:01:42.684817+0000","last_became_active":"2024-09-13T13:01:04.207653+0000","last_became_peered":"2024-09-13T13:01:04.207653+0000","last_unstale":"2024-09-13T13:01:42.684817+0000","last_undegraded":"2024-09-13T13:01:42.684817+0000","last_fullsized":"2024-09-13T13:01:42.684817+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.433729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683990+0000","last_change":"2024-09-13T13:01:02.180002+0000","last_active":"2024-09-13T13:01:42.683990+0000","last_peered":"2024-09-13T13:01:42.683990+0000","last_clean":"2024-09-13T13:01:42.683990+0000","last_became_active":"2024-09-13T13:01:02.179698+0000","last_became_peered":"2024-09-13T13:01:02.179698+0000","last_unstale":"2024-09-13T13:01:42.683990+0000","last_undegraded":"2024-09-13T13:01:42.683990+0000","last_fullsized":"2024-09-13T13:01:42.683990+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:03.938319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":81,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836091+0000","last_change":"2024-09-13T13:00:53.927016+0000","last_active":"2024-09-13T13:01:42.836091+0000","last_peered":"2024-09-13T13:01:42.836091+0000","last_clean":"2024-09-13T13:01:42.836091+0000","last_became_active":"2024-09-13T13:00:53.922110+0000","last_became_peered":"2024-09-13T13:00:53.922110+0000","last_unstale":"2024-09-13T13:01:42.836091+0000","last_undegraded":"2024-09-13T13:01:42.836091+0000","last_fullsized":"2024-09-13T13:01:42.836091+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-13T13:00:46.618619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:00:46.618619+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.913095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836230+0000","last_change":"2024-09-13T13:01:06.207054+0000","last_active":"2024-09-13T13:01:42.836230+0000","last_peered":"2024-09-13T13:01:42.836230+0000","last_clean":"2024-09-13T13:01:42.836230+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:01:42.836230+0000","last_undegraded":"2024-09-13T13:01:42.836230+0000","last_fullsized":"2024-09-13T13:01:42.836230+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.955181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275743+0000","last_change":"2024-09-13T13:01:08.209367+0000","last_active":"2024-09-13T13:01:38.275743+0000","last_peered":"2024-09-13T13:01:38.275743+0000","last_clean":"2024-09-13T13:01:38.275743+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:01:38.275743+0000","last_undegraded":"2024-09-13T13:01:38.275743+0000","last_fullsized":"2024-09-13T13:01:38.275743+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.418279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":57,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240780+0000","last_change":"2024-09-13T13:01:04.196954+0000","last_active":"2024-09-13T13:01:38.240780+0000","last_peered":"2024-09-13T13:01:38.240780+0000","last_clean":"2024-09-13T13:01:38.240780+0000","last_became_active":"2024-09-13T13:01:04.196363+0000","last_became_peered":"2024-09-13T13:01:04.196363+0000","last_unstale":"2024-09-13T13:01:38.240780+0000","last_undegraded":"2024-09-13T13:01:38.240780+0000","last_fullsized":"2024-09-13T13:01:38.240780+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.318889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240228+0000","last_change":"2024-09-13T13:01:02.180296+0000","last_active":"2024-09-13T13:01:38.240228+0000","last_peered":"2024-09-13T13:01:38.240228+0000","last_clean":"2024-09-13T13:01:38.240228+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:01:38.240228+0000","last_undegraded":"2024-09-13T13:01:38.240228+0000","last_fullsized":"2024-09-13T13:01:38.240228+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.041720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275565+0000","last_change":"2024-09-13T13:01:06.209895+0000","last_active":"2024-09-13T13:01:38.275565+0000","last_peered":"2024-09-13T13:01:38.275565+0000","last_clean":"2024-09-13T13:01:38.275565+0000","last_became_active":"2024-09-13T13:01:06.206590+0000","last_became_peered":"2024-09-13T13:01:06.206590+0000","last_unstale":"2024-09-13T13:01:38.275565+0000","last_undegraded":"2024-09-13T13:01:38.275565+0000","last_fullsized":"2024-09-13T13:01:38.275565+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.901892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684715+0000","last_change":"2024-09-13T13:01:08.533475+0000","last_active":"2024-09-13T13:01:42.684715+0000","last_peered":"2024-09-13T13:01:42.684715+0000","last_clean":"2024-09-13T13:01:42.684715+0000","last_became_active":"2024-09-13T13:01:08.533119+0000","last_became_peered":"2024-09-13T13:01:08.533119+0000","last_unstale":"2024-09-13T13:01:42.684715+0000","last_undegraded":"2024-09-13T13:01:42.684715+0000","last_fullsized":"2024-09-13T13:01:42.684715+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.681416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":66,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240773+0000","last_change":"2024-09-13T13:01:04.195092+0000","last_active":"2024-09-13T13:01:38.240773+0000","last_peered":"2024-09-13T13:01:38.240773+0000","last_clean":"2024-09-13T13:01:38.240773+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:01:38.240773+0000","last_undegraded":"2024-09-13T13:01:38.240773+0000","last_fullsized":"2024-09-13T13:01:38.240773+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.170610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684860+0000","last_change":"2024-09-13T13:01:02.181492+0000","last_active":"2024-09-13T13:01:42.684860+0000","last_peered":"2024-09-13T13:01:42.684860+0000","last_clean":"2024-09-13T13:01:42.684860+0000","last_became_active":"2024-09-13T13:01:02.180946+0000","last_became_peered":"2024-09-13T13:01:02.180946+0000","last_unstale":"2024-09-13T13:01:42.684860+0000","last_undegraded":"2024-09-13T13:01:42.684860+0000","last_fullsized":"2024-09-13T13:01:42.684860+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:15.300817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241178+0000","last_change":"2024-09-13T13:01:06.208593+0000","last_active":"2024-09-13T13:01:38.241178+0000","last_peered":"2024-09-13T13:01:38.241178+0000","last_clean":"2024-09-13T13:01:38.241178+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:01:38.241178+0000","last_undegraded":"2024-09-13T13:01:38.241178+0000","last_fullsized":"2024-09-13T13:01:38.241178+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.535525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683087+0000","last_change":"2024-09-13T13:01:08.204029+0000","last_active":"2024-09-13T13:01:42.683087+0000","last_peered":"2024-09-13T13:01:42.683087+0000","last_clean":"2024-09-13T13:01:42.683087+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:01:42.683087+0000","last_undegraded":"2024-09-13T13:01:42.683087+0000","last_fullsized":"2024-09-13T13:01:42.683087+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.251232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":53,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240367+0000","last_change":"2024-09-13T13:01:04.189975+0000","last_active":"2024-09-13T13:01:38.240367+0000","last_peered":"2024-09-13T13:01:38.240367+0000","last_clean":"2024-09-13T13:01:38.240367+0000","last_became_active":"2024-09-13T13:01:04.189380+0000","last_became_peered":"2024-09-13T13:01:04.189380+0000","last_unstale":"2024-09-13T13:01:38.240367+0000","last_undegraded":"2024-09-13T13:01:38.240367+0000","last_fullsized":"2024-09-13T13:01:38.240367+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:03.186596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684594+0000","last_change":"2024-09-13T13:01:02.188134+0000","last_active":"2024-09-13T13:01:42.684594+0000","last_peered":"2024-09-13T13:01:42.684594+0000","last_clean":"2024-09-13T13:01:42.684594+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:01:42.684594+0000","last_undegraded":"2024-09-13T13:01:42.684594+0000","last_fullsized":"2024-09-13T13:01:42.684594+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.160057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.682804+0000","last_change":"2024-09-13T13:01:06.206335+0000","last_active":"2024-09-13T13:01:42.682804+0000","last_peered":"2024-09-13T13:01:42.682804+0000","last_clean":"2024-09-13T13:01:42.682804+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:01:42.682804+0000","last_undegraded":"2024-09-13T13:01:42.682804+0000","last_fullsized":"2024-09-13T13:01:42.682804+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.752225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240587+0000","last_change":"2024-09-13T13:01:08.213034+0000","last_active":"2024-09-13T13:01:38.240587+0000","last_peered":"2024-09-13T13:01:38.240587+0000","last_clean":"2024-09-13T13:01:38.240587+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:01:38.240587+0000","last_undegraded":"2024-09-13T13:01:38.240587+0000","last_fullsized":"2024-09-13T13:01:38.240587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.453460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798661+0000","last_change":"2024-09-13T13:01:04.192921+0000","last_active":"2024-09-13T13:01:42.798661+0000","last_peered":"2024-09-13T13:01:42.798661+0000","last_clean":"2024-09-13T13:01:42.798661+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:01:42.798661+0000","last_undegraded":"2024-09-13T13:01:42.798661+0000","last_fullsized":"2024-09-13T13:01:42.798661+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.518566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.274995+0000","last_change":"2024-09-13T13:01:02.180807+0000","last_active":"2024-09-13T13:01:38.274995+0000","last_peered":"2024-09-13T13:01:38.274995+0000","last_clean":"2024-09-13T13:01:38.274995+0000","last_became_active":"2024-09-13T13:01:02.180541+0000","last_became_peered":"2024-09-13T13:01:02.180541+0000","last_unstale":"2024-09-13T13:01:38.274995+0000","last_undegraded":"2024-09-13T13:01:38.274995+0000","last_fullsized":"2024-09-13T13:01:38.274995+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:28.541518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683348+0000","last_change":"2024-09-13T13:01:06.207416+0000","last_active":"2024-09-13T13:01:42.683348+0000","last_peered":"2024-09-13T13:01:42.683348+0000","last_clean":"2024-09-13T13:01:42.683348+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:01:42.683348+0000","last_undegraded":"2024-09-13T13:01:42.683348+0000","last_fullsized":"2024-09-13T13:01:42.683348+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.336110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798352+0000","last_change":"2024-09-13T13:01:08.211229+0000","last_active":"2024-09-13T13:01:42.798352+0000","last_peered":"2024-09-13T13:01:42.798352+0000","last_clean":"2024-09-13T13:01:42.798352+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:01:42.798352+0000","last_undegraded":"2024-09-13T13:01:42.798352+0000","last_fullsized":"2024-09-13T13:01:42.798352+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:35.720723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":71,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684492+0000","last_change":"2024-09-13T13:01:04.208073+0000","last_active":"2024-09-13T13:01:42.684492+0000","last_peered":"2024-09-13T13:01:42.684492+0000","last_clean":"2024-09-13T13:01:42.684492+0000","last_became_active":"2024-09-13T13:01:04.207702+0000","last_became_peered":"2024-09-13T13:01:04.207702+0000","last_unstale":"2024-09-13T13:01:42.684492+0000","last_undegraded":"2024-09-13T13:01:42.684492+0000","last_fullsized":"2024-09-13T13:01:42.684492+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.215067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836652+0000","last_change":"2024-09-13T13:01:02.178687+0000","last_active":"2024-09-13T13:01:42.836652+0000","last_peered":"2024-09-13T13:01:42.836652+0000","last_clean":"2024-09-13T13:01:42.836652+0000","last_became_active":"2024-09-13T13:01:02.178317+0000","last_became_peered":"2024-09-13T13:01:02.178317+0000","last_unstale":"2024-09-13T13:01:42.836652+0000","last_undegraded":"2024-09-13T13:01:42.836652+0000","last_fullsized":"2024-09-13T13:01:42.836652+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.914639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.682821+0000","last_change":"2024-09-13T13:01:06.203813+0000","last_active":"2024-09-13T13:01:42.682821+0000","last_peered":"2024-09-13T13:01:42.682821+0000","last_clean":"2024-09-13T13:01:42.682821+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:01:42.682821+0000","last_undegraded":"2024-09-13T13:01:42.682821+0000","last_fullsized":"2024-09-13T13:01:42.682821+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:31.552430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835928+0000","last_change":"2024-09-13T13:01:08.212073+0000","last_active":"2024-09-13T13:01:42.835928+0000","last_peered":"2024-09-13T13:01:42.835928+0000","last_clean":"2024-09-13T13:01:42.835928+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:01:42.835928+0000","last_undegraded":"2024-09-13T13:01:42.835928+0000","last_fullsized":"2024-09-13T13:01:42.835928+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:35.915325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":61,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240101+0000","last_change":"2024-09-13T13:01:04.196002+0000","last_active":"2024-09-13T13:01:38.240101+0000","last_peered":"2024-09-13T13:01:38.240101+0000","last_clean":"2024-09-13T13:01:38.240101+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:01:38.240101+0000","last_undegraded":"2024-09-13T13:01:38.240101+0000","last_fullsized":"2024-09-13T13:01:38.240101+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.222887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836727+0000","last_change":"2024-09-13T13:01:02.187229+0000","last_active":"2024-09-13T13:01:42.836727+0000","last_peered":"2024-09-13T13:01:42.836727+0000","last_clean":"2024-09-13T13:01:42.836727+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:01:42.836727+0000","last_undegraded":"2024-09-13T13:01:42.836727+0000","last_fullsized":"2024-09-13T13:01:42.836727+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:02.438861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.682690+0000","last_change":"2024-09-13T13:01:06.211172+0000","last_active":"2024-09-13T13:01:42.682690+0000","last_peered":"2024-09-13T13:01:42.682690+0000","last_clean":"2024-09-13T13:01:42.682690+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:01:42.682690+0000","last_undegraded":"2024-09-13T13:01:42.682690+0000","last_fullsized":"2024-09-13T13:01:42.682690+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.354291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684274+0000","last_change":"2024-09-13T13:01:08.211530+0000","last_active":"2024-09-13T13:01:42.684274+0000","last_peered":"2024-09-13T13:01:42.684274+0000","last_clean":"2024-09-13T13:01:42.684274+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:01:42.684274+0000","last_undegraded":"2024-09-13T13:01:42.684274+0000","last_fullsized":"2024-09-13T13:01:42.684274+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.539593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":63,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.273915+0000","last_change":"2024-09-13T13:01:04.209162+0000","last_active":"2024-09-13T13:01:38.273915+0000","last_peered":"2024-09-13T13:01:38.273915+0000","last_clean":"2024-09-13T13:01:38.273915+0000","last_became_active":"2024-09-13T13:01:04.208731+0000","last_became_peered":"2024-09-13T13:01:04.208731+0000","last_unstale":"2024-09-13T13:01:38.273915+0000","last_undegraded":"2024-09-13T13:01:38.273915+0000","last_fullsized":"2024-09-13T13:01:38.273915+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.356204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836671+0000","last_change":"2024-09-13T13:01:02.183335+0000","last_active":"2024-09-13T13:01:42.836671+0000","last_peered":"2024-09-13T13:01:42.836671+0000","last_clean":"2024-09-13T13:01:42.836671+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:01:42.836671+0000","last_undegraded":"2024-09-13T13:01:42.836671+0000","last_fullsized":"2024-09-13T13:01:42.836671+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:20.269754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835884+0000","last_change":"2024-09-13T13:01:06.210716+0000","last_active":"2024-09-13T13:01:42.835884+0000","last_peered":"2024-09-13T13:01:42.835884+0000","last_clean":"2024-09-13T13:01:42.835884+0000","last_became_active":"2024-09-13T13:01:06.210441+0000","last_became_peered":"2024-09-13T13:01:06.210441+0000","last_unstale":"2024-09-13T13:01:42.835884+0000","last_undegraded":"2024-09-13T13:01:42.835884+0000","last_fullsized":"2024-09-13T13:01:42.835884+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.206296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.274525+0000","last_change":"2024-09-13T13:01:08.532185+0000","last_active":"2024-09-13T13:01:38.274525+0000","last_peered":"2024-09-13T13:01:38.274525+0000","last_clean":"2024-09-13T13:01:38.274525+0000","last_became_active":"2024-09-13T13:01:08.531895+0000","last_became_peered":"2024-09-13T13:01:08.531895+0000","last_unstale":"2024-09-13T13:01:38.274525+0000","last_undegraded":"2024-09-13T13:01:38.274525+0000","last_fullsized":"2024-09-13T13:01:38.274525+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.747075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684258+0000","last_change":"2024-09-13T13:01:02.179217+0000","last_active":"2024-09-13T13:01:42.684258+0000","last_peered":"2024-09-13T13:01:42.684258+0000","last_clean":"2024-09-13T13:01:42.684258+0000","last_became_active":"2024-09-13T13:01:02.178934+0000","last_became_peered":"2024-09-13T13:01:02.178934+0000","last_unstale":"2024-09-13T13:01:42.684258+0000","last_undegraded":"2024-09-13T13:01:42.684258+0000","last_fullsized":"2024-09-13T13:01:42.684258+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.675331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":49,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684880+0000","last_change":"2024-09-13T13:01:04.209208+0000","last_active":"2024-09-13T13:01:42.684880+0000","last_peered":"2024-09-13T13:01:42.684880+0000","last_clean":"2024-09-13T13:01:42.684880+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:01:42.684880+0000","last_undegraded":"2024-09-13T13:01:42.684880+0000","last_fullsized":"2024-09-13T13:01:42.684880+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.286499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275567+0000","last_change":"2024-09-13T13:01:06.208468+0000","last_active":"2024-09-13T13:01:38.275567+0000","last_peered":"2024-09-13T13:01:38.275567+0000","last_clean":"2024-09-13T13:01:38.275567+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:01:38.275567+0000","last_undegraded":"2024-09-13T13:01:38.275567+0000","last_fullsized":"2024-09-13T13:01:38.275567+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.411667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836239+0000","last_change":"2024-09-13T13:01:08.532411+0000","last_active":"2024-09-13T13:01:42.836239+0000","last_peered":"2024-09-13T13:01:42.836239+0000","last_clean":"2024-09-13T13:01:42.836239+0000","last_became_active":"2024-09-13T13:01:08.532178+0000","last_became_peered":"2024-09-13T13:01:08.532178+0000","last_unstale":"2024-09-13T13:01:42.836239+0000","last_undegraded":"2024-09-13T13:01:42.836239+0000","last_fullsized":"2024-09-13T13:01:42.836239+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:15.835392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798990+0000","last_change":"2024-09-13T13:01:04.208866+0000","last_active":"2024-09-13T13:01:42.798990+0000","last_peered":"2024-09-13T13:01:42.798990+0000","last_clean":"2024-09-13T13:01:42.798990+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:01:42.798990+0000","last_undegraded":"2024-09-13T13:01:42.798990+0000","last_fullsized":"2024-09-13T13:01:42.798990+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.015284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836142+0000","last_change":"2024-09-13T13:01:02.181825+0000","last_active":"2024-09-13T13:01:42.836142+0000","last_peered":"2024-09-13T13:01:42.836142+0000","last_clean":"2024-09-13T13:01:42.836142+0000","last_became_active":"2024-09-13T13:01:02.181548+0000","last_became_peered":"2024-09-13T13:01:02.181548+0000","last_unstale":"2024-09-13T13:01:42.836142+0000","last_undegraded":"2024-09-13T13:01:42.836142+0000","last_fullsized":"2024-09-13T13:01:42.836142+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.710586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684732+0000","last_change":"2024-09-13T13:01:06.210328+0000","last_active":"2024-09-13T13:01:42.684732+0000","last_peered":"2024-09-13T13:01:42.684732+0000","last_clean":"2024-09-13T13:01:42.684732+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:01:42.684732+0000","last_undegraded":"2024-09-13T13:01:42.684732+0000","last_fullsized":"2024-09-13T13:01:42.684732+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.781482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.682977+0000","last_change":"2024-09-13T13:01:08.205690+0000","last_active":"2024-09-13T13:01:42.682977+0000","last_peered":"2024-09-13T13:01:42.682977+0000","last_clean":"2024-09-13T13:01:42.682977+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:01:42.682977+0000","last_undegraded":"2024-09-13T13:01:42.682977+0000","last_fullsized":"2024-09-13T13:01:42.682977+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.215451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":54,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275935+0000","last_change":"2024-09-13T13:01:04.208351+0000","last_active":"2024-09-13T13:01:38.275935+0000","last_peered":"2024-09-13T13:01:38.275935+0000","last_clean":"2024-09-13T13:01:38.275935+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:01:38.275935+0000","last_undegraded":"2024-09-13T13:01:38.275935+0000","last_fullsized":"2024-09-13T13:01:38.275935+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.268797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798481+0000","last_change":"2024-09-13T13:01:02.185440+0000","last_active":"2024-09-13T13:01:42.798481+0000","last_peered":"2024-09-13T13:01:42.798481+0000","last_clean":"2024-09-13T13:01:42.798481+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:01:42.798481+0000","last_undegraded":"2024-09-13T13:01:42.798481+0000","last_fullsized":"2024-09-13T13:01:42.798481+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.667174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684853+0000","last_change":"2024-09-13T13:01:06.210251+0000","last_active":"2024-09-13T13:01:42.684853+0000","last_peered":"2024-09-13T13:01:42.684853+0000","last_clean":"2024-09-13T13:01:42.684853+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:01:42.684853+0000","last_undegraded":"2024-09-13T13:01:42.684853+0000","last_fullsized":"2024-09-13T13:01:42.684853+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.308767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240922+0000","last_change":"2024-09-13T13:01:08.206113+0000","last_active":"2024-09-13T13:01:38.240922+0000","last_peered":"2024-09-13T13:01:38.240922+0000","last_clean":"2024-09-13T13:01:38.240922+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:01:38.240922+0000","last_undegraded":"2024-09-13T13:01:38.240922+0000","last_fullsized":"2024-09-13T13:01:38.240922+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.315612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.685126+0000","last_change":"2024-09-13T13:01:04.208466+0000","last_active":"2024-09-13T13:01:42.685126+0000","last_peered":"2024-09-13T13:01:42.685126+0000","last_clean":"2024-09-13T13:01:42.685126+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:01:42.685126+0000","last_undegraded":"2024-09-13T13:01:42.685126+0000","last_fullsized":"2024-09-13T13:01:42.685126+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.373274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836358+0000","last_change":"2024-09-13T13:01:02.180706+0000","last_active":"2024-09-13T13:01:42.836358+0000","last_peered":"2024-09-13T13:01:42.836358+0000","last_clean":"2024-09-13T13:01:42.836358+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:01:42.836358+0000","last_undegraded":"2024-09-13T13:01:42.836358+0000","last_fullsized":"2024-09-13T13:01:42.836358+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.494760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":51,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275805+0000","last_change":"2024-09-13T13:01:06.206440+0000","last_active":"2024-09-13T13:01:38.275805+0000","last_peered":"2024-09-13T13:01:38.275805+0000","last_clean":"2024-09-13T13:01:38.275805+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:01:38.275805+0000","last_undegraded":"2024-09-13T13:01:38.275805+0000","last_fullsized":"2024-09-13T13:01:38.275805+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.104127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798869+0000","last_change":"2024-09-13T13:01:08.215072+0000","last_active":"2024-09-13T13:01:42.798869+0000","last_peered":"2024-09-13T13:01:42.798869+0000","last_clean":"2024-09-13T13:01:42.798869+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:01:42.798869+0000","last_undegraded":"2024-09-13T13:01:42.798869+0000","last_fullsized":"2024-09-13T13:01:42.798869+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.276961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":57,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240219+0000","last_change":"2024-09-13T13:01:04.197213+0000","last_active":"2024-09-13T13:01:38.240219+0000","last_peered":"2024-09-13T13:01:38.240219+0000","last_clean":"2024-09-13T13:01:38.240219+0000","last_became_active":"2024-09-13T13:01:04.196857+0000","last_became_peered":"2024-09-13T13:01:04.196857+0000","last_unstale":"2024-09-13T13:01:38.240219+0000","last_undegraded":"2024-09-13T13:01:38.240219+0000","last_fullsized":"2024-09-13T13:01:38.240219+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.316586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240510+0000","last_change":"2024-09-13T13:01:02.181939+0000","last_active":"2024-09-13T13:01:38.240510+0000","last_peered":"2024-09-13T13:01:38.240510+0000","last_clean":"2024-09-13T13:01:38.240510+0000","last_became_active":"2024-09-13T13:01:02.180817+0000","last_became_peered":"2024-09-13T13:01:02.180817+0000","last_unstale":"2024-09-13T13:01:38.240510+0000","last_undegraded":"2024-09-13T13:01:38.240510+0000","last_fullsized":"2024-09-13T13:01:38.240510+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.328756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240678+0000","last_change":"2024-09-13T13:01:06.209945+0000","last_active":"2024-09-13T13:01:38.240678+0000","last_peered":"2024-09-13T13:01:38.240678+0000","last_clean":"2024-09-13T13:01:38.240678+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:01:38.240678+0000","last_undegraded":"2024-09-13T13:01:38.240678+0000","last_fullsized":"2024-09-13T13:01:38.240678+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.087308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683212+0000","last_change":"2024-09-13T13:01:08.205355+0000","last_active":"2024-09-13T13:01:42.683212+0000","last_peered":"2024-09-13T13:01:42.683212+0000","last_clean":"2024-09-13T13:01:42.683212+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:01:42.683212+0000","last_undegraded":"2024-09-13T13:01:42.683212+0000","last_fullsized":"2024-09-13T13:01:42.683212+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.257400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":54,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240995+0000","last_change":"2024-09-13T13:01:04.196437+0000","last_active":"2024-09-13T13:01:38.240995+0000","last_peered":"2024-09-13T13:01:38.240995+0000","last_clean":"2024-09-13T13:01:38.240995+0000","last_became_active":"2024-09-13T13:01:04.196045+0000","last_became_peered":"2024-09-13T13:01:04.196045+0000","last_unstale":"2024-09-13T13:01:38.240995+0000","last_undegraded":"2024-09-13T13:01:38.240995+0000","last_fullsized":"2024-09-13T13:01:38.240995+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:24.629192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836556+0000","last_change":"2024-09-13T13:01:02.182107+0000","last_active":"2024-09-13T13:01:42.836556+0000","last_peered":"2024-09-13T13:01:42.836556+0000","last_clean":"2024-09-13T13:01:42.836556+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:01:42.836556+0000","last_undegraded":"2024-09-13T13:01:42.836556+0000","last_fullsized":"2024-09-13T13:01:42.836556+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.840261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684359+0000","last_change":"2024-09-13T13:01:06.211993+0000","last_active":"2024-09-13T13:01:42.684359+0000","last_peered":"2024-09-13T13:01:42.684359+0000","last_clean":"2024-09-13T13:01:42.684359+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:01:42.684359+0000","last_undegraded":"2024-09-13T13:01:42.684359+0000","last_fullsized":"2024-09-13T13:01:42.684359+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.375348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798189+0000","last_change":"2024-09-13T13:01:08.210271+0000","last_active":"2024-09-13T13:01:42.798189+0000","last_peered":"2024-09-13T13:01:42.798189+0000","last_clean":"2024-09-13T13:01:42.798189+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:01:42.798189+0000","last_undegraded":"2024-09-13T13:01:42.798189+0000","last_fullsized":"2024-09-13T13:01:42.798189+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.691464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683346+0000","last_change":"2024-09-13T13:01:04.208421+0000","last_active":"2024-09-13T13:01:42.683346+0000","last_peered":"2024-09-13T13:01:42.683346+0000","last_clean":"2024-09-13T13:01:42.683346+0000","last_became_active":"2024-09-13T13:01:04.207841+0000","last_became_peered":"2024-09-13T13:01:04.207841+0000","last_unstale":"2024-09-13T13:01:42.683346+0000","last_undegraded":"2024-09-13T13:01:42.683346+0000","last_fullsized":"2024-09-13T13:01:42.683346+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.504022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275645+0000","last_change":"2024-09-13T13:01:02.185420+0000","last_active":"2024-09-13T13:01:38.275645+0000","last_peered":"2024-09-13T13:01:38.275645+0000","last_clean":"2024-09-13T13:01:38.275645+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:01:38.275645+0000","last_undegraded":"2024-09-13T13:01:38.275645+0000","last_fullsized":"2024-09-13T13:01:38.275645+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.931940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835666+0000","last_change":"2024-09-13T13:01:06.208696+0000","last_active":"2024-09-13T13:01:42.835666+0000","last_peered":"2024-09-13T13:01:42.835666+0000","last_clean":"2024-09-13T13:01:42.835666+0000","last_became_active":"2024-09-13T13:01:06.208426+0000","last_became_peered":"2024-09-13T13:01:06.208426+0000","last_unstale":"2024-09-13T13:01:42.835666+0000","last_undegraded":"2024-09-13T13:01:42.835666+0000","last_fullsized":"2024-09-13T13:01:42.835666+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.767606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684505+0000","last_change":"2024-09-13T13:01:08.535233+0000","last_active":"2024-09-13T13:01:42.684505+0000","last_peered":"2024-09-13T13:01:42.684505+0000","last_clean":"2024-09-13T13:01:42.684505+0000","last_became_active":"2024-09-13T13:01:08.534673+0000","last_became_peered":"2024-09-13T13:01:08.534673+0000","last_unstale":"2024-09-13T13:01:42.684505+0000","last_undegraded":"2024-09-13T13:01:42.684505+0000","last_fullsized":"2024-09-13T13:01:42.684505+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:33.330172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683709+0000","last_change":"2024-09-13T13:01:04.195434+0000","last_active":"2024-09-13T13:01:42.683709+0000","last_peered":"2024-09-13T13:01:42.683709+0000","last_clean":"2024-09-13T13:01:42.683709+0000","last_became_active":"2024-09-13T13:01:04.195125+0000","last_became_peered":"2024-09-13T13:01:04.195125+0000","last_unstale":"2024-09-13T13:01:42.683709+0000","last_undegraded":"2024-09-13T13:01:42.683709+0000","last_fullsized":"2024-09-13T13:01:42.683709+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.430862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798913+0000","last_change":"2024-09-13T13:01:02.186003+0000","last_active":"2024-09-13T13:01:42.798913+0000","last_peered":"2024-09-13T13:01:42.798913+0000","last_clean":"2024-09-13T13:01:42.798913+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:01:42.798913+0000","last_undegraded":"2024-09-13T13:01:42.798913+0000","last_fullsized":"2024-09-13T13:01:42.798913+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.400129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684258+0000","last_change":"2024-09-13T13:01:06.211514+0000","last_active":"2024-09-13T13:01:42.684258+0000","last_peered":"2024-09-13T13:01:42.684258+0000","last_clean":"2024-09-13T13:01:42.684258+0000","last_became_active":"2024-09-13T13:01:06.211243+0000","last_became_peered":"2024-09-13T13:01:06.211243+0000","last_unstale":"2024-09-13T13:01:42.684258+0000","last_undegraded":"2024-09-13T13:01:42.684258+0000","last_fullsized":"2024-09-13T13:01:42.684258+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:16.563704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835865+0000","last_change":"2024-09-13T13:01:08.210205+0000","last_active":"2024-09-13T13:01:42.835865+0000","last_peered":"2024-09-13T13:01:42.835865+0000","last_clean":"2024-09-13T13:01:42.835865+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:01:42.835865+0000","last_undegraded":"2024-09-13T13:01:42.835865+0000","last_fullsized":"2024-09-13T13:01:42.835865+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.043981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241243+0000","last_change":"2024-09-13T13:01:08.214778+0000","last_active":"2024-09-13T13:01:38.241243+0000","last_peered":"2024-09-13T13:01:38.241243+0000","last_clean":"2024-09-13T13:01:38.241243+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:01:38.241243+0000","last_undegraded":"2024-09-13T13:01:38.241243+0000","last_fullsized":"2024-09-13T13:01:38.241243+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:32.480474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683966+0000","last_change":"2024-09-13T13:01:02.187888+0000","last_active":"2024-09-13T13:01:42.683966+0000","last_peered":"2024-09-13T13:01:42.683966+0000","last_clean":"2024-09-13T13:01:42.683966+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:01:42.683966+0000","last_undegraded":"2024-09-13T13:01:42.683966+0000","last_fullsized":"2024-09-13T13:01:42.683966+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.807870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684810+0000","last_change":"2024-09-13T13:01:04.195160+0000","last_active":"2024-09-13T13:01:42.684810+0000","last_peered":"2024-09-13T13:01:42.684810+0000","last_clean":"2024-09-13T13:01:42.684810+0000","last_became_active":"2024-09-13T13:01:04.194855+0000","last_became_peered":"2024-09-13T13:01:04.194855+0000","last_unstale":"2024-09-13T13:01:42.684810+0000","last_undegraded":"2024-09-13T13:01:42.684810+0000","last_fullsized":"2024-09-13T13:01:42.684810+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:20.224127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798784+0000","last_change":"2024-09-13T13:01:06.211596+0000","last_active":"2024-09-13T13:01:42.798784+0000","last_peered":"2024-09-13T13:01:42.798784+0000","last_clean":"2024-09-13T13:01:42.798784+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:01:42.798784+0000","last_undegraded":"2024-09-13T13:01:42.798784+0000","last_fullsized":"2024-09-13T13:01:42.798784+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.427559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201291,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28348,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716804,"statfs":{"total":95995035648,"available":95966007296,"internally_reserved":0,"allocated":1351680,"data_stored":890010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430225,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28332,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716820,"statfs":{"total":95995035648,"available":95966023680,"internally_reserved":0,"allocated":1335296,"data_stored":888381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":745472,"data_stored":298372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920797,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27780,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717372,"statfs":{"total":95995035648,"available":95966588928,"internally_reserved":0,"allocated":770048,"data_stored":298618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117028,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31944,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713208,"statfs":{"total":95995035648,"available":95962324992,"internally_reserved":0,"allocated":774144,"data_stored":299448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411367,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27740,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":729088,"data_stored":297712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":798720,"data_stored":299349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712640,"statfs":{"total":95995035648,"available":95961743360,"internally_reserved":0,"allocated":1355776,"data_stored":889719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:01:46.452 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:01:46.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:46 smithi167 ceph-mon[33178]: from='client.14787 v1:172.21.15.52:0/160948439' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:01:46.729 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:47.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:47 smithi052 ceph-mon[29471]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:01:47.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:47 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:01:47.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:47 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:01:47.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:47 smithi052 ceph-mon[36682]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:01:47.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:47 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:01:47.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:47 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:01:47.526 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:01:47.527 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:01:47.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:47 smithi167 ceph-mon[33178]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:01:47.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:47 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:01:47.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:47 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:01:48.522 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:48 smithi052 ceph-mon[36682]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T13:01:48.522 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:48 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:01:48.522 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:48 smithi052 ceph-mon[36682]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T13:01:48.523 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:48 smithi052 ceph-mon[29471]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T13:01:48.523 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:48 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:01:48.523 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:48 smithi052 ceph-mon[29471]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T13:01:48.547 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-13T13:01:47.261542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":883,"num_read_kb":740,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236380,"kb_used_data":7676,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724836,"statfs":{"total":767960285184,"available":767718232064,"internally_reserved":0,"allocated":7860224,"data_stored":4161609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":234016306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.022935"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798870+0000","last_change":"2024-09-13T13:01:02.184818+0000","last_active":"2024-09-13T13:01:42.798870+0000","last_peered":"2024-09-13T13:01:42.798870+0000","last_clean":"2024-09-13T13:01:42.798870+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:01:42.798870+0000","last_undegraded":"2024-09-13T13:01:42.798870+0000","last_fullsized":"2024-09-13T13:01:42.798870+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.229908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":54,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.239854+0000","last_change":"2024-09-13T13:01:04.194784+0000","last_active":"2024-09-13T13:01:38.239854+0000","last_peered":"2024-09-13T13:01:38.239854+0000","last_clean":"2024-09-13T13:01:38.239854+0000","last_became_active":"2024-09-13T13:01:04.194444+0000","last_became_peered":"2024-09-13T13:01:04.194444+0000","last_unstale":"2024-09-13T13:01:38.239854+0000","last_undegraded":"2024-09-13T13:01:38.239854+0000","last_fullsized":"2024-09-13T13:01:38.239854+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.654746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.239895+0000","last_change":"2024-09-13T13:01:06.207968+0000","last_active":"2024-09-13T13:01:38.239895+0000","last_peered":"2024-09-13T13:01:38.239895+0000","last_clean":"2024-09-13T13:01:38.239895+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:01:38.239895+0000","last_undegraded":"2024-09-13T13:01:38.239895+0000","last_fullsized":"2024-09-13T13:01:38.239895+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.690479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240015+0000","last_change":"2024-09-13T13:01:08.214110+0000","last_active":"2024-09-13T13:01:38.240015+0000","last_peered":"2024-09-13T13:01:38.240015+0000","last_clean":"2024-09-13T13:01:38.240015+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:01:38.240015+0000","last_undegraded":"2024-09-13T13:01:38.240015+0000","last_fullsized":"2024-09-13T13:01:38.240015+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.859045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684177+0000","last_change":"2024-09-13T13:01:08.534893+0000","last_active":"2024-09-13T13:01:42.684177+0000","last_peered":"2024-09-13T13:01:42.684177+0000","last_clean":"2024-09-13T13:01:42.684177+0000","last_became_active":"2024-09-13T13:01:08.534558+0000","last_became_peered":"2024-09-13T13:01:08.534558+0000","last_unstale":"2024-09-13T13:01:42.684177+0000","last_undegraded":"2024-09-13T13:01:42.684177+0000","last_fullsized":"2024-09-13T13:01:42.684177+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.865474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684529+0000","last_change":"2024-09-13T13:01:02.183744+0000","last_active":"2024-09-13T13:01:42.684529+0000","last_peered":"2024-09-13T13:01:42.684529+0000","last_clean":"2024-09-13T13:01:42.684529+0000","last_became_active":"2024-09-13T13:01:02.183415+0000","last_became_peered":"2024-09-13T13:01:02.183415+0000","last_unstale":"2024-09-13T13:01:42.684529+0000","last_undegraded":"2024-09-13T13:01:42.684529+0000","last_fullsized":"2024-09-13T13:01:42.684529+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.940777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":65,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683479+0000","last_change":"2024-09-13T13:01:04.187590+0000","last_active":"2024-09-13T13:01:42.683479+0000","last_peered":"2024-09-13T13:01:42.683479+0000","last_clean":"2024-09-13T13:01:42.683479+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:01:42.683479+0000","last_undegraded":"2024-09-13T13:01:42.683479+0000","last_fullsized":"2024-09-13T13:01:42.683479+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.466652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.239850+0000","last_change":"2024-09-13T13:01:06.207701+0000","last_active":"2024-09-13T13:01:38.239850+0000","last_peered":"2024-09-13T13:01:38.239850+0000","last_clean":"2024-09-13T13:01:38.239850+0000","last_became_active":"2024-09-13T13:01:06.207429+0000","last_became_peered":"2024-09-13T13:01:06.207429+0000","last_unstale":"2024-09-13T13:01:38.239850+0000","last_undegraded":"2024-09-13T13:01:38.239850+0000","last_fullsized":"2024-09-13T13:01:38.239850+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.420247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":48,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275960+0000","last_change":"2024-09-13T13:01:02.184512+0000","last_active":"2024-09-13T13:01:38.275960+0000","last_peered":"2024-09-13T13:01:38.275960+0000","last_clean":"2024-09-13T13:01:38.275960+0000","last_became_active":"2024-09-13T13:01:02.184241+0000","last_became_peered":"2024-09-13T13:01:02.184241+0000","last_unstale":"2024-09-13T13:01:38.275960+0000","last_undegraded":"2024-09-13T13:01:38.275960+0000","last_fullsized":"2024-09-13T13:01:38.275960+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.275782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":54,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240316+0000","last_change":"2024-09-13T13:01:04.189504+0000","last_active":"2024-09-13T13:01:38.240316+0000","last_peered":"2024-09-13T13:01:38.240316+0000","last_clean":"2024-09-13T13:01:38.240316+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:01:38.240316+0000","last_undegraded":"2024-09-13T13:01:38.240316+0000","last_fullsized":"2024-09-13T13:01:38.240316+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:25.303489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275275+0000","last_change":"2024-09-13T13:01:06.205759+0000","last_active":"2024-09-13T13:01:38.275275+0000","last_peered":"2024-09-13T13:01:38.275275+0000","last_clean":"2024-09-13T13:01:38.275275+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:01:38.275275+0000","last_undegraded":"2024-09-13T13:01:38.275275+0000","last_fullsized":"2024-09-13T13:01:38.275275+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.930018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798267+0000","last_change":"2024-09-13T13:01:08.210285+0000","last_active":"2024-09-13T13:01:42.798267+0000","last_peered":"2024-09-13T13:01:42.798267+0000","last_clean":"2024-09-13T13:01:42.798267+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:01:42.798267+0000","last_undegraded":"2024-09-13T13:01:42.798267+0000","last_fullsized":"2024-09-13T13:01:42.798267+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:24.867052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.274842+0000","last_change":"2024-09-13T13:01:02.184552+0000","last_active":"2024-09-13T13:01:38.274842+0000","last_peered":"2024-09-13T13:01:38.274842+0000","last_clean":"2024-09-13T13:01:38.274842+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:01:38.274842+0000","last_undegraded":"2024-09-13T13:01:38.274842+0000","last_fullsized":"2024-09-13T13:01:38.274842+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:05.277064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":48,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240247+0000","last_change":"2024-09-13T13:01:04.196359+0000","last_active":"2024-09-13T13:01:38.240247+0000","last_peered":"2024-09-13T13:01:38.240247+0000","last_clean":"2024-09-13T13:01:38.240247+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:01:38.240247+0000","last_undegraded":"2024-09-13T13:01:38.240247+0000","last_fullsized":"2024-09-13T13:01:38.240247+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.916135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835910+0000","last_change":"2024-09-13T13:01:06.210845+0000","last_active":"2024-09-13T13:01:42.835910+0000","last_peered":"2024-09-13T13:01:42.835910+0000","last_clean":"2024-09-13T13:01:42.835910+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:01:42.835910+0000","last_undegraded":"2024-09-13T13:01:42.835910+0000","last_fullsized":"2024-09-13T13:01:42.835910+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.182827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.274126+0000","last_change":"2024-09-13T13:01:08.213608+0000","last_active":"2024-09-13T13:01:38.274126+0000","last_peered":"2024-09-13T13:01:38.274126+0000","last_clean":"2024-09-13T13:01:38.274126+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:01:38.274126+0000","last_undegraded":"2024-09-13T13:01:38.274126+0000","last_fullsized":"2024-09-13T13:01:38.274126+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.927149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":31,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240747+0000","last_change":"2024-09-13T13:01:08.533336+0000","last_active":"2024-09-13T13:01:38.240747+0000","last_peered":"2024-09-13T13:01:38.240747+0000","last_clean":"2024-09-13T13:01:38.240747+0000","last_became_active":"2024-09-13T13:01:08.533051+0000","last_became_peered":"2024-09-13T13:01:08.533051+0000","last_unstale":"2024-09-13T13:01:38.240747+0000","last_undegraded":"2024-09-13T13:01:38.240747+0000","last_fullsized":"2024-09-13T13:01:38.240747+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:31.568605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798921+0000","last_change":"2024-09-13T13:01:02.183913+0000","last_active":"2024-09-13T13:01:42.798921+0000","last_peered":"2024-09-13T13:01:42.798921+0000","last_clean":"2024-09-13T13:01:42.798921+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:01:42.798921+0000","last_undegraded":"2024-09-13T13:01:42.798921+0000","last_fullsized":"2024-09-13T13:01:42.798921+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.747175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":65,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683345+0000","last_change":"2024-09-13T13:01:04.191966+0000","last_active":"2024-09-13T13:01:42.683345+0000","last_peered":"2024-09-13T13:01:42.683345+0000","last_clean":"2024-09-13T13:01:42.683345+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:01:42.683345+0000","last_undegraded":"2024-09-13T13:01:42.683345+0000","last_fullsized":"2024-09-13T13:01:42.683345+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:05.477313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241442+0000","last_change":"2024-09-13T13:01:06.209495+0000","last_active":"2024-09-13T13:01:38.241442+0000","last_peered":"2024-09-13T13:01:38.241442+0000","last_clean":"2024-09-13T13:01:38.241442+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:01:38.241442+0000","last_undegraded":"2024-09-13T13:01:38.241442+0000","last_fullsized":"2024-09-13T13:01:38.241442+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.608220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684652+0000","last_change":"2024-09-13T13:01:08.533133+0000","last_active":"2024-09-13T13:01:42.684652+0000","last_peered":"2024-09-13T13:01:42.684652+0000","last_clean":"2024-09-13T13:01:42.684652+0000","last_became_active":"2024-09-13T13:01:08.532753+0000","last_became_peered":"2024-09-13T13:01:08.532753+0000","last_unstale":"2024-09-13T13:01:42.684652+0000","last_undegraded":"2024-09-13T13:01:42.684652+0000","last_fullsized":"2024-09-13T13:01:42.684652+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.674547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240594+0000","last_change":"2024-09-13T13:01:02.173465+0000","last_active":"2024-09-13T13:01:38.240594+0000","last_peered":"2024-09-13T13:01:38.240594+0000","last_clean":"2024-09-13T13:01:38.240594+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:01:38.240594+0000","last_undegraded":"2024-09-13T13:01:38.240594+0000","last_fullsized":"2024-09-13T13:01:38.240594+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:28.027613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684184+0000","last_change":"2024-09-13T13:01:04.195126+0000","last_active":"2024-09-13T13:01:42.684184+0000","last_peered":"2024-09-13T13:01:42.684184+0000","last_clean":"2024-09-13T13:01:42.684184+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:01:42.684184+0000","last_undegraded":"2024-09-13T13:01:42.684184+0000","last_fullsized":"2024-09-13T13:01:42.684184+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.577922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240293+0000","last_change":"2024-09-13T13:01:06.207164+0000","last_active":"2024-09-13T13:01:38.240293+0000","last_peered":"2024-09-13T13:01:38.240293+0000","last_clean":"2024-09-13T13:01:38.240293+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:01:38.240293+0000","last_undegraded":"2024-09-13T13:01:38.240293+0000","last_fullsized":"2024-09-13T13:01:38.240293+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.747378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":33,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836058+0000","last_change":"2024-09-13T13:01:08.210300+0000","last_active":"2024-09-13T13:01:42.836058+0000","last_peered":"2024-09-13T13:01:42.836058+0000","last_clean":"2024-09-13T13:01:42.836058+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:01:42.836058+0000","last_undegraded":"2024-09-13T13:01:42.836058+0000","last_fullsized":"2024-09-13T13:01:42.836058+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.173643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":43,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240059+0000","last_change":"2024-09-13T13:01:02.180198+0000","last_active":"2024-09-13T13:01:38.240059+0000","last_peered":"2024-09-13T13:01:38.240059+0000","last_clean":"2024-09-13T13:01:38.240059+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:01:38.240059+0000","last_undegraded":"2024-09-13T13:01:38.240059+0000","last_fullsized":"2024-09-13T13:01:38.240059+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:09.281312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798872+0000","last_change":"2024-09-13T13:01:04.208045+0000","last_active":"2024-09-13T13:01:42.798872+0000","last_peered":"2024-09-13T13:01:42.798872+0000","last_clean":"2024-09-13T13:01:42.798872+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:01:42.798872+0000","last_undegraded":"2024-09-13T13:01:42.798872+0000","last_fullsized":"2024-09-13T13:01:42.798872+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.686327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684767+0000","last_change":"2024-09-13T13:01:06.211329+0000","last_active":"2024-09-13T13:01:42.684767+0000","last_peered":"2024-09-13T13:01:42.684767+0000","last_clean":"2024-09-13T13:01:42.684767+0000","last_became_active":"2024-09-13T13:01:06.211034+0000","last_became_peered":"2024-09-13T13:01:06.211034+0000","last_unstale":"2024-09-13T13:01:42.684767+0000","last_undegraded":"2024-09-13T13:01:42.684767+0000","last_fullsized":"2024-09-13T13:01:42.684767+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:33.073914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":63,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241047+0000","last_change":"2024-09-13T13:01:04.198918+0000","last_active":"2024-09-13T13:01:38.241047+0000","last_peered":"2024-09-13T13:01:38.241047+0000","last_clean":"2024-09-13T13:01:38.241047+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:01:38.241047+0000","last_undegraded":"2024-09-13T13:01:38.241047+0000","last_fullsized":"2024-09-13T13:01:38.241047+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:04.636447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684050+0000","last_change":"2024-09-13T13:01:02.186847+0000","last_active":"2024-09-13T13:01:42.684050+0000","last_peered":"2024-09-13T13:01:42.684050+0000","last_clean":"2024-09-13T13:01:42.684050+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:01:42.684050+0000","last_undegraded":"2024-09-13T13:01:42.684050+0000","last_fullsized":"2024-09-13T13:01:42.684050+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.191632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":51,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240712+0000","last_change":"2024-09-13T13:01:06.206080+0000","last_active":"2024-09-13T13:01:38.240712+0000","last_peered":"2024-09-13T13:01:38.240712+0000","last_clean":"2024-09-13T13:01:38.240712+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:01:38.240712+0000","last_undegraded":"2024-09-13T13:01:38.240712+0000","last_fullsized":"2024-09-13T13:01:38.240712+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.738297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275593+0000","last_change":"2024-09-13T13:01:08.216021+0000","last_active":"2024-09-13T13:01:38.275593+0000","last_peered":"2024-09-13T13:01:38.275593+0000","last_clean":"2024-09-13T13:01:38.275593+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:01:38.275593+0000","last_undegraded":"2024-09-13T13:01:38.275593+0000","last_fullsized":"2024-09-13T13:01:38.275593+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:08.377856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":72,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683806+0000","last_change":"2024-09-13T13:01:04.209310+0000","last_active":"2024-09-13T13:01:42.683806+0000","last_peered":"2024-09-13T13:01:42.683806+0000","last_clean":"2024-09-13T13:01:42.683806+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:01:42.683806+0000","last_undegraded":"2024-09-13T13:01:42.683806+0000","last_fullsized":"2024-09-13T13:01:42.683806+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.025302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":50,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.685036+0000","last_change":"2024-09-13T13:01:02.181001+0000","last_active":"2024-09-13T13:01:42.685036+0000","last_peered":"2024-09-13T13:01:42.685036+0000","last_clean":"2024-09-13T13:01:42.685036+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:01:42.685036+0000","last_undegraded":"2024-09-13T13:01:42.685036+0000","last_fullsized":"2024-09-13T13:01:42.685036+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.444743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.239876+0000","last_change":"2024-09-13T13:01:06.209027+0000","last_active":"2024-09-13T13:01:38.239876+0000","last_peered":"2024-09-13T13:01:38.239876+0000","last_clean":"2024-09-13T13:01:38.239876+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:01:38.239876+0000","last_undegraded":"2024-09-13T13:01:38.239876+0000","last_fullsized":"2024-09-13T13:01:38.239876+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.392689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240168+0000","last_change":"2024-09-13T13:01:08.204782+0000","last_active":"2024-09-13T13:01:38.240168+0000","last_peered":"2024-09-13T13:01:38.240168+0000","last_clean":"2024-09-13T13:01:38.240168+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:01:38.240168+0000","last_undegraded":"2024-09-13T13:01:38.240168+0000","last_fullsized":"2024-09-13T13:01:38.240168+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.812714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":59,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.239915+0000","last_change":"2024-09-13T13:01:04.187117+0000","last_active":"2024-09-13T13:01:38.239915+0000","last_peered":"2024-09-13T13:01:38.239915+0000","last_clean":"2024-09-13T13:01:38.239915+0000","last_became_active":"2024-09-13T13:01:04.186768+0000","last_became_peered":"2024-09-13T13:01:04.186768+0000","last_unstale":"2024-09-13T13:01:38.239915+0000","last_undegraded":"2024-09-13T13:01:38.239915+0000","last_fullsized":"2024-09-13T13:01:38.239915+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.593884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":45,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.799045+0000","last_change":"2024-09-13T13:01:02.187792+0000","last_active":"2024-09-13T13:01:42.799045+0000","last_peered":"2024-09-13T13:01:42.799045+0000","last_clean":"2024-09-13T13:01:42.799045+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:01:42.799045+0000","last_undegraded":"2024-09-13T13:01:42.799045+0000","last_fullsized":"2024-09-13T13:01:42.799045+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:08.968885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683605+0000","last_change":"2024-09-13T13:01:06.211211+0000","last_active":"2024-09-13T13:01:42.683605+0000","last_peered":"2024-09-13T13:01:42.683605+0000","last_clean":"2024-09-13T13:01:42.683605+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:01:42.683605+0000","last_undegraded":"2024-09-13T13:01:42.683605+0000","last_fullsized":"2024-09-13T13:01:42.683605+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.975106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835987+0000","last_change":"2024-09-13T13:01:08.533116+0000","last_active":"2024-09-13T13:01:42.835987+0000","last_peered":"2024-09-13T13:01:42.835987+0000","last_clean":"2024-09-13T13:01:42.835987+0000","last_became_active":"2024-09-13T13:01:08.532859+0000","last_became_peered":"2024-09-13T13:01:08.532859+0000","last_unstale":"2024-09-13T13:01:42.835987+0000","last_undegraded":"2024-09-13T13:01:42.835987+0000","last_fullsized":"2024-09-13T13:01:42.835987+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:24.307795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":53,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240822+0000","last_change":"2024-09-13T13:01:04.199011+0000","last_active":"2024-09-13T13:01:38.240822+0000","last_peered":"2024-09-13T13:01:38.240822+0000","last_clean":"2024-09-13T13:01:38.240822+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:01:38.240822+0000","last_undegraded":"2024-09-13T13:01:38.240822+0000","last_fullsized":"2024-09-13T13:01:38.240822+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.407716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275995+0000","last_change":"2024-09-13T13:01:02.181597+0000","last_active":"2024-09-13T13:01:38.275995+0000","last_peered":"2024-09-13T13:01:38.275995+0000","last_clean":"2024-09-13T13:01:38.275995+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:01:38.275995+0000","last_undegraded":"2024-09-13T13:01:38.275995+0000","last_fullsized":"2024-09-13T13:01:38.275995+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.285259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798594+0000","last_change":"2024-09-13T13:01:06.216352+0000","last_active":"2024-09-13T13:01:42.798594+0000","last_peered":"2024-09-13T13:01:42.798594+0000","last_clean":"2024-09-13T13:01:42.798594+0000","last_became_active":"2024-09-13T13:01:06.216043+0000","last_became_peered":"2024-09-13T13:01:06.216043+0000","last_unstale":"2024-09-13T13:01:42.798594+0000","last_undegraded":"2024-09-13T13:01:42.798594+0000","last_fullsized":"2024-09-13T13:01:42.798594+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.006398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798385+0000","last_change":"2024-09-13T13:01:08.211109+0000","last_active":"2024-09-13T13:01:42.798385+0000","last_peered":"2024-09-13T13:01:42.798385+0000","last_clean":"2024-09-13T13:01:42.798385+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:01:42.798385+0000","last_undegraded":"2024-09-13T13:01:42.798385+0000","last_fullsized":"2024-09-13T13:01:42.798385+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.759653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":76,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798717+0000","last_change":"2024-09-13T13:01:04.207894+0000","last_active":"2024-09-13T13:01:42.798717+0000","last_peered":"2024-09-13T13:01:42.798717+0000","last_clean":"2024-09-13T13:01:42.798717+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:01:42.798717+0000","last_undegraded":"2024-09-13T13:01:42.798717+0000","last_fullsized":"2024-09-13T13:01:42.798717+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.794771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241382+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:01:38.241382+0000","last_peered":"2024-09-13T13:01:38.241382+0000","last_clean":"2024-09-13T13:01:38.241382+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:01:38.241382+0000","last_undegraded":"2024-09-13T13:01:38.241382+0000","last_fullsized":"2024-09-13T13:01:38.241382+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.453481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684301+0000","last_change":"2024-09-13T13:01:06.213926+0000","last_active":"2024-09-13T13:01:42.684301+0000","last_peered":"2024-09-13T13:01:42.684301+0000","last_clean":"2024-09-13T13:01:42.684301+0000","last_became_active":"2024-09-13T13:01:06.213617+0000","last_became_peered":"2024-09-13T13:01:06.213617+0000","last_unstale":"2024-09-13T13:01:42.684301+0000","last_undegraded":"2024-09-13T13:01:42.684301+0000","last_fullsized":"2024-09-13T13:01:42.684301+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.400692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.239855+0000","last_change":"2024-09-13T13:01:08.532524+0000","last_active":"2024-09-13T13:01:38.239855+0000","last_peered":"2024-09-13T13:01:38.239855+0000","last_clean":"2024-09-13T13:01:38.239855+0000","last_became_active":"2024-09-13T13:01:08.532145+0000","last_became_peered":"2024-09-13T13:01:08.532145+0000","last_unstale":"2024-09-13T13:01:38.239855+0000","last_undegraded":"2024-09-13T13:01:38.239855+0000","last_fullsized":"2024-09-13T13:01:38.239855+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:16.922652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":95,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241343+0000","last_change":"2024-09-13T13:01:04.197527+0000","last_active":"2024-09-13T13:01:38.241343+0000","last_peered":"2024-09-13T13:01:38.241343+0000","last_clean":"2024-09-13T13:01:38.241343+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:01:38.241343+0000","last_undegraded":"2024-09-13T13:01:38.241343+0000","last_fullsized":"2024-09-13T13:01:38.241343+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.799330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240454+0000","last_change":"2024-09-13T13:01:02.181566+0000","last_active":"2024-09-13T13:01:38.240454+0000","last_peered":"2024-09-13T13:01:38.240454+0000","last_clean":"2024-09-13T13:01:38.240454+0000","last_became_active":"2024-09-13T13:01:02.180760+0000","last_became_peered":"2024-09-13T13:01:02.180760+0000","last_unstale":"2024-09-13T13:01:38.240454+0000","last_undegraded":"2024-09-13T13:01:38.240454+0000","last_fullsized":"2024-09-13T13:01:38.240454+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.355888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":50,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.274764+0000","last_change":"2024-09-13T13:01:05.199998+0000","last_active":"2024-09-13T13:01:38.274764+0000","last_peered":"2024-09-13T13:01:38.274764+0000","last_clean":"2024-09-13T13:01:38.274764+0000","last_became_active":"2024-09-13T13:01:02.185089+0000","last_became_peered":"2024-09-13T13:01:02.185089+0000","last_unstale":"2024-09-13T13:01:38.274764+0000","last_undegraded":"2024-09-13T13:01:38.274764+0000","last_fullsized":"2024-09-13T13:01:38.274764+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.802534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798623+0000","last_change":"2024-09-13T13:01:06.216206+0000","last_active":"2024-09-13T13:01:42.798623+0000","last_peered":"2024-09-13T13:01:42.798623+0000","last_clean":"2024-09-13T13:01:42.798623+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:01:42.798623+0000","last_undegraded":"2024-09-13T13:01:42.798623+0000","last_fullsized":"2024-09-13T13:01:42.798623+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.187742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684361+0000","last_change":"2024-09-13T13:01:08.203445+0000","last_active":"2024-09-13T13:01:42.684361+0000","last_peered":"2024-09-13T13:01:42.684361+0000","last_clean":"2024-09-13T13:01:42.684361+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:01:42.684361+0000","last_undegraded":"2024-09-13T13:01:42.684361+0000","last_fullsized":"2024-09-13T13:01:42.684361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.430979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":65,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241139+0000","last_change":"2024-09-13T13:01:04.192923+0000","last_active":"2024-09-13T13:01:38.241139+0000","last_peered":"2024-09-13T13:01:38.241139+0000","last_clean":"2024-09-13T13:01:38.241139+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:01:38.241139+0000","last_undegraded":"2024-09-13T13:01:38.241139+0000","last_fullsized":"2024-09-13T13:01:38.241139+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.839147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240430+0000","last_change":"2024-09-13T13:01:02.185582+0000","last_active":"2024-09-13T13:01:38.240430+0000","last_peered":"2024-09-13T13:01:38.240430+0000","last_clean":"2024-09-13T13:01:38.240430+0000","last_became_active":"2024-09-13T13:01:02.185263+0000","last_became_peered":"2024-09-13T13:01:02.185263+0000","last_unstale":"2024-09-13T13:01:38.240430+0000","last_undegraded":"2024-09-13T13:01:38.240430+0000","last_fullsized":"2024-09-13T13:01:38.240430+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:28.109915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":51,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683107+0000","last_change":"2024-09-13T13:01:05.202103+0000","last_active":"2024-09-13T13:01:42.683107+0000","last_peered":"2024-09-13T13:01:42.683107+0000","last_clean":"2024-09-13T13:01:42.683107+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:01:42.683107+0000","last_undegraded":"2024-09-13T13:01:42.683107+0000","last_fullsized":"2024-09-13T13:01:42.683107+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.268994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683782+0000","last_change":"2024-09-13T13:01:06.205805+0000","last_active":"2024-09-13T13:01:42.683782+0000","last_peered":"2024-09-13T13:01:42.683782+0000","last_clean":"2024-09-13T13:01:42.683782+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:01:42.683782+0000","last_undegraded":"2024-09-13T13:01:42.683782+0000","last_fullsized":"2024-09-13T13:01:42.683782+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.555922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836396+0000","last_change":"2024-09-13T13:01:08.532390+0000","last_active":"2024-09-13T13:01:42.836396+0000","last_peered":"2024-09-13T13:01:42.836396+0000","last_clean":"2024-09-13T13:01:42.836396+0000","last_became_active":"2024-09-13T13:01:08.532087+0000","last_became_peered":"2024-09-13T13:01:08.532087+0000","last_unstale":"2024-09-13T13:01:42.836396+0000","last_undegraded":"2024-09-13T13:01:42.836396+0000","last_fullsized":"2024-09-13T13:01:42.836396+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.390192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798984+0000","last_change":"2024-09-13T13:01:04.193042+0000","last_active":"2024-09-13T13:01:42.798984+0000","last_peered":"2024-09-13T13:01:42.798984+0000","last_clean":"2024-09-13T13:01:42.798984+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:01:42.798984+0000","last_undegraded":"2024-09-13T13:01:42.798984+0000","last_fullsized":"2024-09-13T13:01:42.798984+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.415798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798373+0000","last_change":"2024-09-13T13:01:02.186331+0000","last_active":"2024-09-13T13:01:42.798373+0000","last_peered":"2024-09-13T13:01:42.798373+0000","last_clean":"2024-09-13T13:01:42.798373+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:01:42.798373+0000","last_undegraded":"2024-09-13T13:01:42.798373+0000","last_fullsized":"2024-09-13T13:01:42.798373+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.787015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":88,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835877+0000","last_change":"2024-09-13T13:01:05.200818+0000","last_active":"2024-09-13T13:01:42.835877+0000","last_peered":"2024-09-13T13:01:42.835877+0000","last_clean":"2024-09-13T13:01:42.835877+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:01:42.835877+0000","last_undegraded":"2024-09-13T13:01:42.835877+0000","last_fullsized":"2024-09-13T13:01:42.835877+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.587568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275829+0000","last_change":"2024-09-13T13:01:06.212898+0000","last_active":"2024-09-13T13:01:38.275829+0000","last_peered":"2024-09-13T13:01:38.275829+0000","last_clean":"2024-09-13T13:01:38.275829+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:01:38.275829+0000","last_undegraded":"2024-09-13T13:01:38.275829+0000","last_fullsized":"2024-09-13T13:01:38.275829+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.387930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241155+0000","last_change":"2024-09-13T13:01:08.213728+0000","last_active":"2024-09-13T13:01:38.241155+0000","last_peered":"2024-09-13T13:01:38.241155+0000","last_clean":"2024-09-13T13:01:38.241155+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:01:38.241155+0000","last_undegraded":"2024-09-13T13:01:38.241155+0000","last_fullsized":"2024-09-13T13:01:38.241155+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.763739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":71,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684817+0000","last_change":"2024-09-13T13:01:04.208034+0000","last_active":"2024-09-13T13:01:42.684817+0000","last_peered":"2024-09-13T13:01:42.684817+0000","last_clean":"2024-09-13T13:01:42.684817+0000","last_became_active":"2024-09-13T13:01:04.207653+0000","last_became_peered":"2024-09-13T13:01:04.207653+0000","last_unstale":"2024-09-13T13:01:42.684817+0000","last_undegraded":"2024-09-13T13:01:42.684817+0000","last_fullsized":"2024-09-13T13:01:42.684817+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.433729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683990+0000","last_change":"2024-09-13T13:01:02.180002+0000","last_active":"2024-09-13T13:01:42.683990+0000","last_peered":"2024-09-13T13:01:42.683990+0000","last_clean":"2024-09-13T13:01:42.683990+0000","last_became_active":"2024-09-13T13:01:02.179698+0000","last_became_peered":"2024-09-13T13:01:02.179698+0000","last_unstale":"2024-09-13T13:01:42.683990+0000","last_undegraded":"2024-09-13T13:01:42.683990+0000","last_fullsized":"2024-09-13T13:01:42.683990+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:03.938319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":81,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836091+0000","last_change":"2024-09-13T13:00:53.927016+0000","last_active":"2024-09-13T13:01:42.836091+0000","last_peered":"2024-09-13T13:01:42.836091+0000","last_clean":"2024-09-13T13:01:42.836091+0000","last_became_active":"2024-09-13T13:00:53.922110+0000","last_became_peered":"2024-09-13T13:00:53.922110+0000","last_unstale":"2024-09-13T13:01:42.836091+0000","last_undegraded":"2024-09-13T13:01:42.836091+0000","last_fullsized":"2024-09-13T13:01:42.836091+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-13T13:00:46.618619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:00:46.618619+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.913095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836230+0000","last_change":"2024-09-13T13:01:06.207054+0000","last_active":"2024-09-13T13:01:42.836230+0000","last_peered":"2024-09-13T13:01:42.836230+0000","last_clean":"2024-09-13T13:01:42.836230+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:01:42.836230+0000","last_undegraded":"2024-09-13T13:01:42.836230+0000","last_fullsized":"2024-09-13T13:01:42.836230+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.955181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275743+0000","last_change":"2024-09-13T13:01:08.209367+0000","last_active":"2024-09-13T13:01:38.275743+0000","last_peered":"2024-09-13T13:01:38.275743+0000","last_clean":"2024-09-13T13:01:38.275743+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:01:38.275743+0000","last_undegraded":"2024-09-13T13:01:38.275743+0000","last_fullsized":"2024-09-13T13:01:38.275743+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.418279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":57,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240780+0000","last_change":"2024-09-13T13:01:04.196954+0000","last_active":"2024-09-13T13:01:38.240780+0000","last_peered":"2024-09-13T13:01:38.240780+0000","last_clean":"2024-09-13T13:01:38.240780+0000","last_became_active":"2024-09-13T13:01:04.196363+0000","last_became_peered":"2024-09-13T13:01:04.196363+0000","last_unstale":"2024-09-13T13:01:38.240780+0000","last_undegraded":"2024-09-13T13:01:38.240780+0000","last_fullsized":"2024-09-13T13:01:38.240780+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.318889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240228+0000","last_change":"2024-09-13T13:01:02.180296+0000","last_active":"2024-09-13T13:01:38.240228+0000","last_peered":"2024-09-13T13:01:38.240228+0000","last_clean":"2024-09-13T13:01:38.240228+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:01:38.240228+0000","last_undegraded":"2024-09-13T13:01:38.240228+0000","last_fullsized":"2024-09-13T13:01:38.240228+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.041720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275565+0000","last_change":"2024-09-13T13:01:06.209895+0000","last_active":"2024-09-13T13:01:38.275565+0000","last_peered":"2024-09-13T13:01:38.275565+0000","last_clean":"2024-09-13T13:01:38.275565+0000","last_became_active":"2024-09-13T13:01:06.206590+0000","last_became_peered":"2024-09-13T13:01:06.206590+0000","last_unstale":"2024-09-13T13:01:38.275565+0000","last_undegraded":"2024-09-13T13:01:38.275565+0000","last_fullsized":"2024-09-13T13:01:38.275565+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.901892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684715+0000","last_change":"2024-09-13T13:01:08.533475+0000","last_active":"2024-09-13T13:01:42.684715+0000","last_peered":"2024-09-13T13:01:42.684715+0000","last_clean":"2024-09-13T13:01:42.684715+0000","last_became_active":"2024-09-13T13:01:08.533119+0000","last_became_peered":"2024-09-13T13:01:08.533119+0000","last_unstale":"2024-09-13T13:01:42.684715+0000","last_undegraded":"2024-09-13T13:01:42.684715+0000","last_fullsized":"2024-09-13T13:01:42.684715+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.681416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":66,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240773+0000","last_change":"2024-09-13T13:01:04.195092+0000","last_active":"2024-09-13T13:01:38.240773+0000","last_peered":"2024-09-13T13:01:38.240773+0000","last_clean":"2024-09-13T13:01:38.240773+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:01:38.240773+0000","last_undegraded":"2024-09-13T13:01:38.240773+0000","last_fullsized":"2024-09-13T13:01:38.240773+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.170610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684860+0000","last_change":"2024-09-13T13:01:02.181492+0000","last_active":"2024-09-13T13:01:42.684860+0000","last_peered":"2024-09-13T13:01:42.684860+0000","last_clean":"2024-09-13T13:01:42.684860+0000","last_became_active":"2024-09-13T13:01:02.180946+0000","last_became_peered":"2024-09-13T13:01:02.180946+0000","last_unstale":"2024-09-13T13:01:42.684860+0000","last_undegraded":"2024-09-13T13:01:42.684860+0000","last_fullsized":"2024-09-13T13:01:42.684860+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:15.300817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241178+0000","last_change":"2024-09-13T13:01:06.208593+0000","last_active":"2024-09-13T13:01:38.241178+0000","last_peered":"2024-09-13T13:01:38.241178+0000","last_clean":"2024-09-13T13:01:38.241178+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:01:38.241178+0000","last_undegraded":"2024-09-13T13:01:38.241178+0000","last_fullsized":"2024-09-13T13:01:38.241178+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.535525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683087+0000","last_change":"2024-09-13T13:01:08.204029+0000","last_active":"2024-09-13T13:01:42.683087+0000","last_peered":"2024-09-13T13:01:42.683087+0000","last_clean":"2024-09-13T13:01:42.683087+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:01:42.683087+0000","last_undegraded":"2024-09-13T13:01:42.683087+0000","last_fullsized":"2024-09-13T13:01:42.683087+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.251232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":53,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240367+0000","last_change":"2024-09-13T13:01:04.189975+0000","last_active":"2024-09-13T13:01:38.240367+0000","last_peered":"2024-09-13T13:01:38.240367+0000","last_clean":"2024-09-13T13:01:38.240367+0000","last_became_active":"2024-09-13T13:01:04.189380+0000","last_became_peered":"2024-09-13T13:01:04.189380+0000","last_unstale":"2024-09-13T13:01:38.240367+0000","last_undegraded":"2024-09-13T13:01:38.240367+0000","last_fullsized":"2024-09-13T13:01:38.240367+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:03.186596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684594+0000","last_change":"2024-09-13T13:01:02.188134+0000","last_active":"2024-09-13T13:01:42.684594+0000","last_peered":"2024-09-13T13:01:42.684594+0000","last_clean":"2024-09-13T13:01:42.684594+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:01:42.684594+0000","last_undegraded":"2024-09-13T13:01:42.684594+0000","last_fullsized":"2024-09-13T13:01:42.684594+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.160057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.682804+0000","last_change":"2024-09-13T13:01:06.206335+0000","last_active":"2024-09-13T13:01:42.682804+0000","last_peered":"2024-09-13T13:01:42.682804+0000","last_clean":"2024-09-13T13:01:42.682804+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:01:42.682804+0000","last_undegraded":"2024-09-13T13:01:42.682804+0000","last_fullsized":"2024-09-13T13:01:42.682804+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.752225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240587+0000","last_change":"2024-09-13T13:01:08.213034+0000","last_active":"2024-09-13T13:01:38.240587+0000","last_peered":"2024-09-13T13:01:38.240587+0000","last_clean":"2024-09-13T13:01:38.240587+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:01:38.240587+0000","last_undegraded":"2024-09-13T13:01:38.240587+0000","last_fullsized":"2024-09-13T13:01:38.240587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.453460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798661+0000","last_change":"2024-09-13T13:01:04.192921+0000","last_active":"2024-09-13T13:01:42.798661+0000","last_peered":"2024-09-13T13:01:42.798661+0000","last_clean":"2024-09-13T13:01:42.798661+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:01:42.798661+0000","last_undegraded":"2024-09-13T13:01:42.798661+0000","last_fullsized":"2024-09-13T13:01:42.798661+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.518566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.274995+0000","last_change":"2024-09-13T13:01:02.180807+0000","last_active":"2024-09-13T13:01:38.274995+0000","last_peered":"2024-09-13T13:01:38.274995+0000","last_clean":"2024-09-13T13:01:38.274995+0000","last_became_active":"2024-09-13T13:01:02.180541+0000","last_became_peered":"2024-09-13T13:01:02.180541+0000","last_unstale":"2024-09-13T13:01:38.274995+0000","last_undegraded":"2024-09-13T13:01:38.274995+0000","last_fullsized":"2024-09-13T13:01:38.274995+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:28.541518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683348+0000","last_change":"2024-09-13T13:01:06.207416+0000","last_active":"2024-09-13T13:01:42.683348+0000","last_peered":"2024-09-13T13:01:42.683348+0000","last_clean":"2024-09-13T13:01:42.683348+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:01:42.683348+0000","last_undegraded":"2024-09-13T13:01:42.683348+0000","last_fullsized":"2024-09-13T13:01:42.683348+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.336110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798352+0000","last_change":"2024-09-13T13:01:08.211229+0000","last_active":"2024-09-13T13:01:42.798352+0000","last_peered":"2024-09-13T13:01:42.798352+0000","last_clean":"2024-09-13T13:01:42.798352+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:01:42.798352+0000","last_undegraded":"2024-09-13T13:01:42.798352+0000","last_fullsized":"2024-09-13T13:01:42.798352+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:35.720723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":71,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684492+0000","last_change":"2024-09-13T13:01:04.208073+0000","last_active":"2024-09-13T13:01:42.684492+0000","last_peered":"2024-09-13T13:01:42.684492+0000","last_clean":"2024-09-13T13:01:42.684492+0000","last_became_active":"2024-09-13T13:01:04.207702+0000","last_became_peered":"2024-09-13T13:01:04.207702+0000","last_unstale":"2024-09-13T13:01:42.684492+0000","last_undegraded":"2024-09-13T13:01:42.684492+0000","last_fullsized":"2024-09-13T13:01:42.684492+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.215067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836652+0000","last_change":"2024-09-13T13:01:02.178687+0000","last_active":"2024-09-13T13:01:42.836652+0000","last_peered":"2024-09-13T13:01:42.836652+0000","last_clean":"2024-09-13T13:01:42.836652+0000","last_became_active":"2024-09-13T13:01:02.178317+0000","last_became_peered":"2024-09-13T13:01:02.178317+0000","last_unstale":"2024-09-13T13:01:42.836652+0000","last_undegraded":"2024-09-13T13:01:42.836652+0000","last_fullsized":"2024-09-13T13:01:42.836652+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.914639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.682821+0000","last_change":"2024-09-13T13:01:06.203813+0000","last_active":"2024-09-13T13:01:42.682821+0000","last_peered":"2024-09-13T13:01:42.682821+0000","last_clean":"2024-09-13T13:01:42.682821+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:01:42.682821+0000","last_undegraded":"2024-09-13T13:01:42.682821+0000","last_fullsized":"2024-09-13T13:01:42.682821+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:31.552430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835928+0000","last_change":"2024-09-13T13:01:08.212073+0000","last_active":"2024-09-13T13:01:42.835928+0000","last_peered":"2024-09-13T13:01:42.835928+0000","last_clean":"2024-09-13T13:01:42.835928+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:01:42.835928+0000","last_undegraded":"2024-09-13T13:01:42.835928+0000","last_fullsized":"2024-09-13T13:01:42.835928+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:35.915325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":61,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240101+0000","last_change":"2024-09-13T13:01:04.196002+0000","last_active":"2024-09-13T13:01:38.240101+0000","last_peered":"2024-09-13T13:01:38.240101+0000","last_clean":"2024-09-13T13:01:38.240101+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:01:38.240101+0000","last_undegraded":"2024-09-13T13:01:38.240101+0000","last_fullsized":"2024-09-13T13:01:38.240101+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.222887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836727+0000","last_change":"2024-09-13T13:01:02.187229+0000","last_active":"2024-09-13T13:01:42.836727+0000","last_peered":"2024-09-13T13:01:42.836727+0000","last_clean":"2024-09-13T13:01:42.836727+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:01:42.836727+0000","last_undegraded":"2024-09-13T13:01:42.836727+0000","last_fullsized":"2024-09-13T13:01:42.836727+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:02.438861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.682690+0000","last_change":"2024-09-13T13:01:06.211172+0000","last_active":"2024-09-13T13:01:42.682690+0000","last_peered":"2024-09-13T13:01:42.682690+0000","last_clean":"2024-09-13T13:01:42.682690+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:01:42.682690+0000","last_undegraded":"2024-09-13T13:01:42.682690+0000","last_fullsized":"2024-09-13T13:01:42.682690+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.354291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684274+0000","last_change":"2024-09-13T13:01:08.211530+0000","last_active":"2024-09-13T13:01:42.684274+0000","last_peered":"2024-09-13T13:01:42.684274+0000","last_clean":"2024-09-13T13:01:42.684274+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:01:42.684274+0000","last_undegraded":"2024-09-13T13:01:42.684274+0000","last_fullsized":"2024-09-13T13:01:42.684274+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.539593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":63,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.273915+0000","last_change":"2024-09-13T13:01:04.209162+0000","last_active":"2024-09-13T13:01:38.273915+0000","last_peered":"2024-09-13T13:01:38.273915+0000","last_clean":"2024-09-13T13:01:38.273915+0000","last_became_active":"2024-09-13T13:01:04.208731+0000","last_became_peered":"2024-09-13T13:01:04.208731+0000","last_unstale":"2024-09-13T13:01:38.273915+0000","last_undegraded":"2024-09-13T13:01:38.273915+0000","last_fullsized":"2024-09-13T13:01:38.273915+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.356204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836671+0000","last_change":"2024-09-13T13:01:02.183335+0000","last_active":"2024-09-13T13:01:42.836671+0000","last_peered":"2024-09-13T13:01:42.836671+0000","last_clean":"2024-09-13T13:01:42.836671+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:01:42.836671+0000","last_undegraded":"2024-09-13T13:01:42.836671+0000","last_fullsized":"2024-09-13T13:01:42.836671+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:20.269754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835884+0000","last_change":"2024-09-13T13:01:06.210716+0000","last_active":"2024-09-13T13:01:42.835884+0000","last_peered":"2024-09-13T13:01:42.835884+0000","last_clean":"2024-09-13T13:01:42.835884+0000","last_became_active":"2024-09-13T13:01:06.210441+0000","last_became_peered":"2024-09-13T13:01:06.210441+0000","last_unstale":"2024-09-13T13:01:42.835884+0000","last_undegraded":"2024-09-13T13:01:42.835884+0000","last_fullsized":"2024-09-13T13:01:42.835884+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.206296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.274525+0000","last_change":"2024-09-13T13:01:08.532185+0000","last_active":"2024-09-13T13:01:38.274525+0000","last_peered":"2024-09-13T13:01:38.274525+0000","last_clean":"2024-09-13T13:01:38.274525+0000","last_became_active":"2024-09-13T13:01:08.531895+0000","last_became_peered":"2024-09-13T13:01:08.531895+0000","last_unstale":"2024-09-13T13:01:38.274525+0000","last_undegraded":"2024-09-13T13:01:38.274525+0000","last_fullsized":"2024-09-13T13:01:38.274525+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.747075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684258+0000","last_change":"2024-09-13T13:01:02.179217+0000","last_active":"2024-09-13T13:01:42.684258+0000","last_peered":"2024-09-13T13:01:42.684258+0000","last_clean":"2024-09-13T13:01:42.684258+0000","last_became_active":"2024-09-13T13:01:02.178934+0000","last_became_peered":"2024-09-13T13:01:02.178934+0000","last_unstale":"2024-09-13T13:01:42.684258+0000","last_undegraded":"2024-09-13T13:01:42.684258+0000","last_fullsized":"2024-09-13T13:01:42.684258+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.675331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":49,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684880+0000","last_change":"2024-09-13T13:01:04.209208+0000","last_active":"2024-09-13T13:01:42.684880+0000","last_peered":"2024-09-13T13:01:42.684880+0000","last_clean":"2024-09-13T13:01:42.684880+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:01:42.684880+0000","last_undegraded":"2024-09-13T13:01:42.684880+0000","last_fullsized":"2024-09-13T13:01:42.684880+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.286499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275567+0000","last_change":"2024-09-13T13:01:06.208468+0000","last_active":"2024-09-13T13:01:38.275567+0000","last_peered":"2024-09-13T13:01:38.275567+0000","last_clean":"2024-09-13T13:01:38.275567+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:01:38.275567+0000","last_undegraded":"2024-09-13T13:01:38.275567+0000","last_fullsized":"2024-09-13T13:01:38.275567+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.411667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836239+0000","last_change":"2024-09-13T13:01:08.532411+0000","last_active":"2024-09-13T13:01:42.836239+0000","last_peered":"2024-09-13T13:01:42.836239+0000","last_clean":"2024-09-13T13:01:42.836239+0000","last_became_active":"2024-09-13T13:01:08.532178+0000","last_became_peered":"2024-09-13T13:01:08.532178+0000","last_unstale":"2024-09-13T13:01:42.836239+0000","last_undegraded":"2024-09-13T13:01:42.836239+0000","last_fullsized":"2024-09-13T13:01:42.836239+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:15.835392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798990+0000","last_change":"2024-09-13T13:01:04.208866+0000","last_active":"2024-09-13T13:01:42.798990+0000","last_peered":"2024-09-13T13:01:42.798990+0000","last_clean":"2024-09-13T13:01:42.798990+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:01:42.798990+0000","last_undegraded":"2024-09-13T13:01:42.798990+0000","last_fullsized":"2024-09-13T13:01:42.798990+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.015284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836142+0000","last_change":"2024-09-13T13:01:02.181825+0000","last_active":"2024-09-13T13:01:42.836142+0000","last_peered":"2024-09-13T13:01:42.836142+0000","last_clean":"2024-09-13T13:01:42.836142+0000","last_became_active":"2024-09-13T13:01:02.181548+0000","last_became_peered":"2024-09-13T13:01:02.181548+0000","last_unstale":"2024-09-13T13:01:42.836142+0000","last_undegraded":"2024-09-13T13:01:42.836142+0000","last_fullsized":"2024-09-13T13:01:42.836142+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.710586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684732+0000","last_change":"2024-09-13T13:01:06.210328+0000","last_active":"2024-09-13T13:01:42.684732+0000","last_peered":"2024-09-13T13:01:42.684732+0000","last_clean":"2024-09-13T13:01:42.684732+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:01:42.684732+0000","last_undegraded":"2024-09-13T13:01:42.684732+0000","last_fullsized":"2024-09-13T13:01:42.684732+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.781482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.682977+0000","last_change":"2024-09-13T13:01:08.205690+0000","last_active":"2024-09-13T13:01:42.682977+0000","last_peered":"2024-09-13T13:01:42.682977+0000","last_clean":"2024-09-13T13:01:42.682977+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:01:42.682977+0000","last_undegraded":"2024-09-13T13:01:42.682977+0000","last_fullsized":"2024-09-13T13:01:42.682977+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.215451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":54,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275935+0000","last_change":"2024-09-13T13:01:04.208351+0000","last_active":"2024-09-13T13:01:38.275935+0000","last_peered":"2024-09-13T13:01:38.275935+0000","last_clean":"2024-09-13T13:01:38.275935+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:01:38.275935+0000","last_undegraded":"2024-09-13T13:01:38.275935+0000","last_fullsized":"2024-09-13T13:01:38.275935+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.268797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798481+0000","last_change":"2024-09-13T13:01:02.185440+0000","last_active":"2024-09-13T13:01:42.798481+0000","last_peered":"2024-09-13T13:01:42.798481+0000","last_clean":"2024-09-13T13:01:42.798481+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:01:42.798481+0000","last_undegraded":"2024-09-13T13:01:42.798481+0000","last_fullsized":"2024-09-13T13:01:42.798481+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.667174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684853+0000","last_change":"2024-09-13T13:01:06.210251+0000","last_active":"2024-09-13T13:01:42.684853+0000","last_peered":"2024-09-13T13:01:42.684853+0000","last_clean":"2024-09-13T13:01:42.684853+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:01:42.684853+0000","last_undegraded":"2024-09-13T13:01:42.684853+0000","last_fullsized":"2024-09-13T13:01:42.684853+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.308767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240922+0000","last_change":"2024-09-13T13:01:08.206113+0000","last_active":"2024-09-13T13:01:38.240922+0000","last_peered":"2024-09-13T13:01:38.240922+0000","last_clean":"2024-09-13T13:01:38.240922+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:01:38.240922+0000","last_undegraded":"2024-09-13T13:01:38.240922+0000","last_fullsized":"2024-09-13T13:01:38.240922+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.315612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.685126+0000","last_change":"2024-09-13T13:01:04.208466+0000","last_active":"2024-09-13T13:01:42.685126+0000","last_peered":"2024-09-13T13:01:42.685126+0000","last_clean":"2024-09-13T13:01:42.685126+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:01:42.685126+0000","last_undegraded":"2024-09-13T13:01:42.685126+0000","last_fullsized":"2024-09-13T13:01:42.685126+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.373274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836358+0000","last_change":"2024-09-13T13:01:02.180706+0000","last_active":"2024-09-13T13:01:42.836358+0000","last_peered":"2024-09-13T13:01:42.836358+0000","last_clean":"2024-09-13T13:01:42.836358+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:01:42.836358+0000","last_undegraded":"2024-09-13T13:01:42.836358+0000","last_fullsized":"2024-09-13T13:01:42.836358+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.494760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":51,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275805+0000","last_change":"2024-09-13T13:01:06.206440+0000","last_active":"2024-09-13T13:01:38.275805+0000","last_peered":"2024-09-13T13:01:38.275805+0000","last_clean":"2024-09-13T13:01:38.275805+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:01:38.275805+0000","last_undegraded":"2024-09-13T13:01:38.275805+0000","last_fullsized":"2024-09-13T13:01:38.275805+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.104127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798869+0000","last_change":"2024-09-13T13:01:08.215072+0000","last_active":"2024-09-13T13:01:42.798869+0000","last_peered":"2024-09-13T13:01:42.798869+0000","last_clean":"2024-09-13T13:01:42.798869+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:01:42.798869+0000","last_undegraded":"2024-09-13T13:01:42.798869+0000","last_fullsized":"2024-09-13T13:01:42.798869+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.276961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":57,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240219+0000","last_change":"2024-09-13T13:01:04.197213+0000","last_active":"2024-09-13T13:01:38.240219+0000","last_peered":"2024-09-13T13:01:38.240219+0000","last_clean":"2024-09-13T13:01:38.240219+0000","last_became_active":"2024-09-13T13:01:04.196857+0000","last_became_peered":"2024-09-13T13:01:04.196857+0000","last_unstale":"2024-09-13T13:01:38.240219+0000","last_undegraded":"2024-09-13T13:01:38.240219+0000","last_fullsized":"2024-09-13T13:01:38.240219+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.316586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240510+0000","last_change":"2024-09-13T13:01:02.181939+0000","last_active":"2024-09-13T13:01:38.240510+0000","last_peered":"2024-09-13T13:01:38.240510+0000","last_clean":"2024-09-13T13:01:38.240510+0000","last_became_active":"2024-09-13T13:01:02.180817+0000","last_became_peered":"2024-09-13T13:01:02.180817+0000","last_unstale":"2024-09-13T13:01:38.240510+0000","last_undegraded":"2024-09-13T13:01:38.240510+0000","last_fullsized":"2024-09-13T13:01:38.240510+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.328756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":34,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240678+0000","last_change":"2024-09-13T13:01:06.209945+0000","last_active":"2024-09-13T13:01:38.240678+0000","last_peered":"2024-09-13T13:01:38.240678+0000","last_clean":"2024-09-13T13:01:38.240678+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:01:38.240678+0000","last_undegraded":"2024-09-13T13:01:38.240678+0000","last_fullsized":"2024-09-13T13:01:38.240678+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.087308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683212+0000","last_change":"2024-09-13T13:01:08.205355+0000","last_active":"2024-09-13T13:01:42.683212+0000","last_peered":"2024-09-13T13:01:42.683212+0000","last_clean":"2024-09-13T13:01:42.683212+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:01:42.683212+0000","last_undegraded":"2024-09-13T13:01:42.683212+0000","last_fullsized":"2024-09-13T13:01:42.683212+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.257400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":54,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.240995+0000","last_change":"2024-09-13T13:01:04.196437+0000","last_active":"2024-09-13T13:01:38.240995+0000","last_peered":"2024-09-13T13:01:38.240995+0000","last_clean":"2024-09-13T13:01:38.240995+0000","last_became_active":"2024-09-13T13:01:04.196045+0000","last_became_peered":"2024-09-13T13:01:04.196045+0000","last_unstale":"2024-09-13T13:01:38.240995+0000","last_undegraded":"2024-09-13T13:01:38.240995+0000","last_fullsized":"2024-09-13T13:01:38.240995+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:24.629192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.836556+0000","last_change":"2024-09-13T13:01:02.182107+0000","last_active":"2024-09-13T13:01:42.836556+0000","last_peered":"2024-09-13T13:01:42.836556+0000","last_clean":"2024-09-13T13:01:42.836556+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:01:42.836556+0000","last_undegraded":"2024-09-13T13:01:42.836556+0000","last_fullsized":"2024-09-13T13:01:42.836556+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.840261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684359+0000","last_change":"2024-09-13T13:01:06.211993+0000","last_active":"2024-09-13T13:01:42.684359+0000","last_peered":"2024-09-13T13:01:42.684359+0000","last_clean":"2024-09-13T13:01:42.684359+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:01:42.684359+0000","last_undegraded":"2024-09-13T13:01:42.684359+0000","last_fullsized":"2024-09-13T13:01:42.684359+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.375348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798189+0000","last_change":"2024-09-13T13:01:08.210271+0000","last_active":"2024-09-13T13:01:42.798189+0000","last_peered":"2024-09-13T13:01:42.798189+0000","last_clean":"2024-09-13T13:01:42.798189+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:01:42.798189+0000","last_undegraded":"2024-09-13T13:01:42.798189+0000","last_fullsized":"2024-09-13T13:01:42.798189+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.691464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683346+0000","last_change":"2024-09-13T13:01:04.208421+0000","last_active":"2024-09-13T13:01:42.683346+0000","last_peered":"2024-09-13T13:01:42.683346+0000","last_clean":"2024-09-13T13:01:42.683346+0000","last_became_active":"2024-09-13T13:01:04.207841+0000","last_became_peered":"2024-09-13T13:01:04.207841+0000","last_unstale":"2024-09-13T13:01:42.683346+0000","last_undegraded":"2024-09-13T13:01:42.683346+0000","last_fullsized":"2024-09-13T13:01:42.683346+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.504022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":42,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.275645+0000","last_change":"2024-09-13T13:01:02.185420+0000","last_active":"2024-09-13T13:01:38.275645+0000","last_peered":"2024-09-13T13:01:38.275645+0000","last_clean":"2024-09-13T13:01:38.275645+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:01:38.275645+0000","last_undegraded":"2024-09-13T13:01:38.275645+0000","last_fullsized":"2024-09-13T13:01:38.275645+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.931940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835666+0000","last_change":"2024-09-13T13:01:06.208696+0000","last_active":"2024-09-13T13:01:42.835666+0000","last_peered":"2024-09-13T13:01:42.835666+0000","last_clean":"2024-09-13T13:01:42.835666+0000","last_became_active":"2024-09-13T13:01:06.208426+0000","last_became_peered":"2024-09-13T13:01:06.208426+0000","last_unstale":"2024-09-13T13:01:42.835666+0000","last_undegraded":"2024-09-13T13:01:42.835666+0000","last_fullsized":"2024-09-13T13:01:42.835666+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.767606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684505+0000","last_change":"2024-09-13T13:01:08.535233+0000","last_active":"2024-09-13T13:01:42.684505+0000","last_peered":"2024-09-13T13:01:42.684505+0000","last_clean":"2024-09-13T13:01:42.684505+0000","last_became_active":"2024-09-13T13:01:08.534673+0000","last_became_peered":"2024-09-13T13:01:08.534673+0000","last_unstale":"2024-09-13T13:01:42.684505+0000","last_undegraded":"2024-09-13T13:01:42.684505+0000","last_fullsized":"2024-09-13T13:01:42.684505+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:33.330172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683709+0000","last_change":"2024-09-13T13:01:04.195434+0000","last_active":"2024-09-13T13:01:42.683709+0000","last_peered":"2024-09-13T13:01:42.683709+0000","last_clean":"2024-09-13T13:01:42.683709+0000","last_became_active":"2024-09-13T13:01:04.195125+0000","last_became_peered":"2024-09-13T13:01:04.195125+0000","last_unstale":"2024-09-13T13:01:42.683709+0000","last_undegraded":"2024-09-13T13:01:42.683709+0000","last_fullsized":"2024-09-13T13:01:42.683709+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.430862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798913+0000","last_change":"2024-09-13T13:01:02.186003+0000","last_active":"2024-09-13T13:01:42.798913+0000","last_peered":"2024-09-13T13:01:42.798913+0000","last_clean":"2024-09-13T13:01:42.798913+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:01:42.798913+0000","last_undegraded":"2024-09-13T13:01:42.798913+0000","last_fullsized":"2024-09-13T13:01:42.798913+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.400129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684258+0000","last_change":"2024-09-13T13:01:06.211514+0000","last_active":"2024-09-13T13:01:42.684258+0000","last_peered":"2024-09-13T13:01:42.684258+0000","last_clean":"2024-09-13T13:01:42.684258+0000","last_became_active":"2024-09-13T13:01:06.211243+0000","last_became_peered":"2024-09-13T13:01:06.211243+0000","last_unstale":"2024-09-13T13:01:42.684258+0000","last_undegraded":"2024-09-13T13:01:42.684258+0000","last_fullsized":"2024-09-13T13:01:42.684258+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:16.563704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.835865+0000","last_change":"2024-09-13T13:01:08.210205+0000","last_active":"2024-09-13T13:01:42.835865+0000","last_peered":"2024-09-13T13:01:42.835865+0000","last_clean":"2024-09-13T13:01:42.835865+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:01:42.835865+0000","last_undegraded":"2024-09-13T13:01:42.835865+0000","last_fullsized":"2024-09-13T13:01:42.835865+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.043981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":30,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T13:01:38.241243+0000","last_change":"2024-09-13T13:01:08.214778+0000","last_active":"2024-09-13T13:01:38.241243+0000","last_peered":"2024-09-13T13:01:38.241243+0000","last_clean":"2024-09-13T13:01:38.241243+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:01:38.241243+0000","last_undegraded":"2024-09-13T13:01:38.241243+0000","last_fullsized":"2024-09-13T13:01:38.241243+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:32.480474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.683966+0000","last_change":"2024-09-13T13:01:02.187888+0000","last_active":"2024-09-13T13:01:42.683966+0000","last_peered":"2024-09-13T13:01:42.683966+0000","last_clean":"2024-09-13T13:01:42.683966+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:01:42.683966+0000","last_undegraded":"2024-09-13T13:01:42.683966+0000","last_fullsized":"2024-09-13T13:01:42.683966+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.807870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.684810+0000","last_change":"2024-09-13T13:01:04.195160+0000","last_active":"2024-09-13T13:01:42.684810+0000","last_peered":"2024-09-13T13:01:42.684810+0000","last_clean":"2024-09-13T13:01:42.684810+0000","last_became_active":"2024-09-13T13:01:04.194855+0000","last_became_peered":"2024-09-13T13:01:04.194855+0000","last_unstale":"2024-09-13T13:01:42.684810+0000","last_undegraded":"2024-09-13T13:01:42.684810+0000","last_fullsized":"2024-09-13T13:01:42.684810+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:20.224127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T13:01:42.798784+0000","last_change":"2024-09-13T13:01:06.211596+0000","last_active":"2024-09-13T13:01:42.798784+0000","last_peered":"2024-09-13T13:01:42.798784+0000","last_clean":"2024-09-13T13:01:42.798784+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:01:42.798784+0000","last_undegraded":"2024-09-13T13:01:42.798784+0000","last_fullsized":"2024-09-13T13:01:42.798784+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.427559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201291,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28348,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716804,"statfs":{"total":95995035648,"available":95966007296,"internally_reserved":0,"allocated":1351680,"data_stored":890010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430225,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28332,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716820,"statfs":{"total":95995035648,"available":95966023680,"internally_reserved":0,"allocated":1335296,"data_stored":888381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":745472,"data_stored":298372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920797,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27780,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717372,"statfs":{"total":95995035648,"available":95966588928,"internally_reserved":0,"allocated":770048,"data_stored":298618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117028,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31944,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713208,"statfs":{"total":95995035648,"available":95962324992,"internally_reserved":0,"allocated":774144,"data_stored":299448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411367,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27740,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":729088,"data_stored":297712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":798720,"data_stored":299349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712640,"statfs":{"total":95995035648,"available":95961743360,"internally_reserved":0,"allocated":1355776,"data_stored":889719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:01:48.553 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T13:01:48.553 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T13:01:48.553 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T13:01:48.553 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T13:01:48.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:48 smithi167 ceph-mon[33178]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T13:01:48.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:48 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:01:48.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:48 smithi167 ceph-mon[33178]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T13:01:49.091 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:49.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:49 smithi052 ceph-mon[36682]: from='client.14793 v1:172.21.15.52:0/2283503395' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:01:49.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:49 smithi052 ceph-mon[36682]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T13:01:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:49 smithi052 ceph-mon[29471]: from='client.14793 v1:172.21.15.52:0/2283503395' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:01:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:49 smithi052 ceph-mon[29471]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T13:01:49.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:49 smithi167 ceph-mon[33178]: from='client.14793 v1:172.21.15.52:0/2283503395' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:01:49.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:49 smithi167 ceph-mon[33178]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T13:01:50.019 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:01:50.019 INFO:teuthology.orchestra.run.smithi052.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T13:01:50.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:50 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1310408347' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T13:01:50.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:50 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1310408347' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T13:01:50.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:50 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1310408347' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T13:01:51.475 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T13:01:51.476 INFO:tasks.cephadm:Setup complete, yielding 2024-09-13T13:01:51.476 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-13T13:01:51.492 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T13:01:51.492 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-main-2024-09-13-0840'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': 'b0201b8c79733293453e7f10a10c7fa43119222b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b0201b8c79733293453e7f10a10c7fa43119222b'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'wip-guits-main-2024-09-13-0840', 'sha1': 'b0201b8c79733293453e7f10a10c7fa43119222b'}} 2024-09-13T13:01:51.492 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T13:01:51.492 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-13T13:01:51.520 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:51 smithi052 ceph-mon[36682]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T13:01:51.520 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:51 smithi052 ceph-mon[29471]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T13:01:51.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:51 smithi167 ceph-mon[33178]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-13T13:01:53.262 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:53.403 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:53 smithi052 ceph-mon[36682]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:01:53.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:53 smithi052 ceph-mon[29471]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:01:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:53 smithi167 ceph-mon[33178]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:01:53.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:01:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:01:54.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:53 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:53] ENGINE Bus STOPPING 2024-09-13T13:01:54.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:54 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T13:01:54.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:54 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:54] ENGINE Bus STOPPED 2024-09-13T13:01:54.240 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:54 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:54] ENGINE Bus STARTING 2024-09-13T13:01:54.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:54 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:01:54.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:54 smithi167 ceph-mon[33178]: from='client.34682 ' entity='client.admin' 2024-09-13T13:01:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:54 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:01:54.629 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:54 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:01:54.629 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:54 smithi052 ceph-mon[36682]: from='client.34682 ' entity='client.admin' 2024-09-13T13:01:54.629 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:54 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:01:54.629 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:54 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:54] ENGINE Serving on http://:::9283 2024-09-13T13:01:54.630 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:54 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:54] ENGINE Bus STARTED 2024-09-13T13:01:54.630 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:54 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:01:54.630 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:54 smithi052 ceph-mon[29471]: from='client.34682 ' entity='client.admin' 2024-09-13T13:01:54.630 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:54 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:01:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-mon[36682]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:01:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-mon[36682]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:01:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-mon[29471]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:01:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-mon[29471]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:01:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:55.559 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-13T13:01:55.559 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T13:01:55.559 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:01:55.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:55 smithi167 ceph-mon[33178]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:01:55.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:55 smithi167 ceph-mon[33178]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:01:55.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:55 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:55 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:55 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:55 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:55.776 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:55] ENGINE Bus STOPPING 2024-09-13T13:01:55.865 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:56.116 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T13:01:56.117 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:55] ENGINE Bus STOPPED 2024-09-13T13:01:56.117 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:55] ENGINE Bus STARTING 2024-09-13T13:01:56.117 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:55] ENGINE Serving on http://:::9283 2024-09-13T13:01:56.117 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:01:55] ENGINE Bus STARTED 2024-09-13T13:01:56.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:01:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:01:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:01:56.526 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:01:56.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:56 smithi167 ceph-mon[33178]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-13T13:01:56.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:56 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:56.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:56 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:56.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:56 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-13T13:01:56.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:56 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:56.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:56 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:01:56.931 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-mon[36682]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-13T13:01:56.932 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:56.932 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:56.932 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-13T13:01:56.932 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:56.932 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:01:56.932 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-mon[29471]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-13T13:01:56.932 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:56.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:56.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-13T13:01:56.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:56.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:01:56.933 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:01:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[68036]: ts=2024-09-13T13:01:56.747Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00000899s 2024-09-13T13:01:57.117 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-13T13:01:55.874577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":893,"num_read_kb":750,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236676,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724540,"statfs":{"total":767960285184,"available":767717928960,"internally_reserved":0,"allocated":8155136,"data_stored":4314937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":234016306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.613035"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685552+0000","last_change":"2024-09-13T13:01:02.184818+0000","last_active":"2024-09-13T13:01:47.685552+0000","last_peered":"2024-09-13T13:01:47.685552+0000","last_clean":"2024-09-13T13:01:47.685552+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:01:47.685552+0000","last_undegraded":"2024-09-13T13:01:47.685552+0000","last_fullsized":"2024-09-13T13:01:47.685552+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.229908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684822+0000","last_change":"2024-09-13T13:01:04.194784+0000","last_active":"2024-09-13T13:01:47.684822+0000","last_peered":"2024-09-13T13:01:47.684822+0000","last_clean":"2024-09-13T13:01:47.684822+0000","last_became_active":"2024-09-13T13:01:04.194444+0000","last_became_peered":"2024-09-13T13:01:04.194444+0000","last_unstale":"2024-09-13T13:01:47.684822+0000","last_undegraded":"2024-09-13T13:01:47.684822+0000","last_fullsized":"2024-09-13T13:01:47.684822+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.654746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686249+0000","last_change":"2024-09-13T13:01:06.207968+0000","last_active":"2024-09-13T13:01:47.686249+0000","last_peered":"2024-09-13T13:01:47.686249+0000","last_clean":"2024-09-13T13:01:47.686249+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:01:47.686249+0000","last_undegraded":"2024-09-13T13:01:47.686249+0000","last_fullsized":"2024-09-13T13:01:47.686249+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.690479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685023+0000","last_change":"2024-09-13T13:01:08.214110+0000","last_active":"2024-09-13T13:01:47.685023+0000","last_peered":"2024-09-13T13:01:47.685023+0000","last_clean":"2024-09-13T13:01:47.685023+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:01:47.685023+0000","last_undegraded":"2024-09-13T13:01:47.685023+0000","last_fullsized":"2024-09-13T13:01:47.685023+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.859045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995992+0000","last_change":"2024-09-13T13:01:08.534893+0000","last_active":"2024-09-13T13:01:47.995992+0000","last_peered":"2024-09-13T13:01:47.995992+0000","last_clean":"2024-09-13T13:01:47.995992+0000","last_became_active":"2024-09-13T13:01:08.534558+0000","last_became_peered":"2024-09-13T13:01:08.534558+0000","last_unstale":"2024-09-13T13:01:47.995992+0000","last_undegraded":"2024-09-13T13:01:47.995992+0000","last_fullsized":"2024-09-13T13:01:47.995992+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.865474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996526+0000","last_change":"2024-09-13T13:01:02.183744+0000","last_active":"2024-09-13T13:01:47.996526+0000","last_peered":"2024-09-13T13:01:47.996526+0000","last_clean":"2024-09-13T13:01:47.996526+0000","last_became_active":"2024-09-13T13:01:02.183415+0000","last_became_peered":"2024-09-13T13:01:02.183415+0000","last_unstale":"2024-09-13T13:01:47.996526+0000","last_undegraded":"2024-09-13T13:01:47.996526+0000","last_fullsized":"2024-09-13T13:01:47.996526+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.940777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":68,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995788+0000","last_change":"2024-09-13T13:01:04.187590+0000","last_active":"2024-09-13T13:01:47.995788+0000","last_peered":"2024-09-13T13:01:47.995788+0000","last_clean":"2024-09-13T13:01:47.995788+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:01:47.995788+0000","last_undegraded":"2024-09-13T13:01:47.995788+0000","last_fullsized":"2024-09-13T13:01:47.995788+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.466652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684811+0000","last_change":"2024-09-13T13:01:06.207701+0000","last_active":"2024-09-13T13:01:47.684811+0000","last_peered":"2024-09-13T13:01:47.684811+0000","last_clean":"2024-09-13T13:01:47.684811+0000","last_became_active":"2024-09-13T13:01:06.207429+0000","last_became_peered":"2024-09-13T13:01:06.207429+0000","last_unstale":"2024-09-13T13:01:47.684811+0000","last_undegraded":"2024-09-13T13:01:47.684811+0000","last_fullsized":"2024-09-13T13:01:47.684811+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.420247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":53,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151307+0000","last_change":"2024-09-13T13:01:02.184512+0000","last_active":"2024-09-13T13:01:48.151307+0000","last_peered":"2024-09-13T13:01:48.151307+0000","last_clean":"2024-09-13T13:01:48.151307+0000","last_became_active":"2024-09-13T13:01:02.184241+0000","last_became_peered":"2024-09-13T13:01:02.184241+0000","last_unstale":"2024-09-13T13:01:48.151307+0000","last_undegraded":"2024-09-13T13:01:48.151307+0000","last_fullsized":"2024-09-13T13:01:48.151307+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.275782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685708+0000","last_change":"2024-09-13T13:01:04.189504+0000","last_active":"2024-09-13T13:01:47.685708+0000","last_peered":"2024-09-13T13:01:47.685708+0000","last_clean":"2024-09-13T13:01:47.685708+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:01:47.685708+0000","last_undegraded":"2024-09-13T13:01:47.685708+0000","last_fullsized":"2024-09-13T13:01:47.685708+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:25.303489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149681+0000","last_change":"2024-09-13T13:01:06.205759+0000","last_active":"2024-09-13T13:01:48.149681+0000","last_peered":"2024-09-13T13:01:48.149681+0000","last_clean":"2024-09-13T13:01:48.149681+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:01:48.149681+0000","last_undegraded":"2024-09-13T13:01:48.149681+0000","last_fullsized":"2024-09-13T13:01:48.149681+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.930018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684263+0000","last_change":"2024-09-13T13:01:08.210285+0000","last_active":"2024-09-13T13:01:47.684263+0000","last_peered":"2024-09-13T13:01:47.684263+0000","last_clean":"2024-09-13T13:01:47.684263+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:01:47.684263+0000","last_undegraded":"2024-09-13T13:01:47.684263+0000","last_fullsized":"2024-09-13T13:01:47.684263+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:24.867052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150075+0000","last_change":"2024-09-13T13:01:02.184552+0000","last_active":"2024-09-13T13:01:48.150075+0000","last_peered":"2024-09-13T13:01:48.150075+0000","last_clean":"2024-09-13T13:01:48.150075+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:01:48.150075+0000","last_undegraded":"2024-09-13T13:01:48.150075+0000","last_fullsized":"2024-09-13T13:01:48.150075+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:05.277064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":54,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685089+0000","last_change":"2024-09-13T13:01:04.196359+0000","last_active":"2024-09-13T13:01:47.685089+0000","last_peered":"2024-09-13T13:01:47.685089+0000","last_clean":"2024-09-13T13:01:47.685089+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:01:47.685089+0000","last_undegraded":"2024-09-13T13:01:47.685089+0000","last_fullsized":"2024-09-13T13:01:47.685089+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.916135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150044+0000","last_change":"2024-09-13T13:01:06.210845+0000","last_active":"2024-09-13T13:01:48.150044+0000","last_peered":"2024-09-13T13:01:48.150044+0000","last_clean":"2024-09-13T13:01:48.150044+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:01:48.150044+0000","last_undegraded":"2024-09-13T13:01:48.150044+0000","last_fullsized":"2024-09-13T13:01:48.150044+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.182827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149616+0000","last_change":"2024-09-13T13:01:08.213608+0000","last_active":"2024-09-13T13:01:48.149616+0000","last_peered":"2024-09-13T13:01:48.149616+0000","last_clean":"2024-09-13T13:01:48.149616+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:01:48.149616+0000","last_undegraded":"2024-09-13T13:01:48.149616+0000","last_fullsized":"2024-09-13T13:01:48.149616+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.927149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":37,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685600+0000","last_change":"2024-09-13T13:01:08.533336+0000","last_active":"2024-09-13T13:01:47.685600+0000","last_peered":"2024-09-13T13:01:47.685600+0000","last_clean":"2024-09-13T13:01:47.685600+0000","last_became_active":"2024-09-13T13:01:08.533051+0000","last_became_peered":"2024-09-13T13:01:08.533051+0000","last_unstale":"2024-09-13T13:01:47.685600+0000","last_undegraded":"2024-09-13T13:01:47.685600+0000","last_fullsized":"2024-09-13T13:01:47.685600+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:31.568605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685736+0000","last_change":"2024-09-13T13:01:02.183913+0000","last_active":"2024-09-13T13:01:47.685736+0000","last_peered":"2024-09-13T13:01:47.685736+0000","last_clean":"2024-09-13T13:01:47.685736+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:01:47.685736+0000","last_undegraded":"2024-09-13T13:01:47.685736+0000","last_fullsized":"2024-09-13T13:01:47.685736+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.747175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":69,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150076+0000","last_change":"2024-09-13T13:01:04.191966+0000","last_active":"2024-09-13T13:01:48.150076+0000","last_peered":"2024-09-13T13:01:48.150076+0000","last_clean":"2024-09-13T13:01:48.150076+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:01:48.150076+0000","last_undegraded":"2024-09-13T13:01:48.150076+0000","last_fullsized":"2024-09-13T13:01:48.150076+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:05.477313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686922+0000","last_change":"2024-09-13T13:01:06.209495+0000","last_active":"2024-09-13T13:01:47.686922+0000","last_peered":"2024-09-13T13:01:47.686922+0000","last_clean":"2024-09-13T13:01:47.686922+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:01:47.686922+0000","last_undegraded":"2024-09-13T13:01:47.686922+0000","last_fullsized":"2024-09-13T13:01:47.686922+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.608220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996469+0000","last_change":"2024-09-13T13:01:08.533133+0000","last_active":"2024-09-13T13:01:47.996469+0000","last_peered":"2024-09-13T13:01:47.996469+0000","last_clean":"2024-09-13T13:01:47.996469+0000","last_became_active":"2024-09-13T13:01:08.532753+0000","last_became_peered":"2024-09-13T13:01:08.532753+0000","last_unstale":"2024-09-13T13:01:47.996469+0000","last_undegraded":"2024-09-13T13:01:47.996469+0000","last_fullsized":"2024-09-13T13:01:47.996469+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.674547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685839+0000","last_change":"2024-09-13T13:01:02.173465+0000","last_active":"2024-09-13T13:01:47.685839+0000","last_peered":"2024-09-13T13:01:47.685839+0000","last_clean":"2024-09-13T13:01:47.685839+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:01:47.685839+0000","last_undegraded":"2024-09-13T13:01:47.685839+0000","last_fullsized":"2024-09-13T13:01:47.685839+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:28.027613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":66,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996200+0000","last_change":"2024-09-13T13:01:04.195126+0000","last_active":"2024-09-13T13:01:47.996200+0000","last_peered":"2024-09-13T13:01:47.996200+0000","last_clean":"2024-09-13T13:01:47.996200+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:01:47.996200+0000","last_undegraded":"2024-09-13T13:01:47.996200+0000","last_fullsized":"2024-09-13T13:01:47.996200+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.577922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685142+0000","last_change":"2024-09-13T13:01:06.207164+0000","last_active":"2024-09-13T13:01:47.685142+0000","last_peered":"2024-09-13T13:01:47.685142+0000","last_clean":"2024-09-13T13:01:47.685142+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:01:47.685142+0000","last_undegraded":"2024-09-13T13:01:47.685142+0000","last_fullsized":"2024-09-13T13:01:47.685142+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.747378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150287+0000","last_change":"2024-09-13T13:01:08.210300+0000","last_active":"2024-09-13T13:01:48.150287+0000","last_peered":"2024-09-13T13:01:48.150287+0000","last_clean":"2024-09-13T13:01:48.150287+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:01:48.150287+0000","last_undegraded":"2024-09-13T13:01:48.150287+0000","last_fullsized":"2024-09-13T13:01:48.150287+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.173643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":49,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686330+0000","last_change":"2024-09-13T13:01:02.180198+0000","last_active":"2024-09-13T13:01:47.686330+0000","last_peered":"2024-09-13T13:01:47.686330+0000","last_clean":"2024-09-13T13:01:47.686330+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:01:47.686330+0000","last_undegraded":"2024-09-13T13:01:47.686330+0000","last_fullsized":"2024-09-13T13:01:47.686330+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:09.281312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685868+0000","last_change":"2024-09-13T13:01:04.208045+0000","last_active":"2024-09-13T13:01:47.685868+0000","last_peered":"2024-09-13T13:01:47.685868+0000","last_clean":"2024-09-13T13:01:47.685868+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:01:47.685868+0000","last_undegraded":"2024-09-13T13:01:47.685868+0000","last_fullsized":"2024-09-13T13:01:47.685868+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.686327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152494+0000","last_change":"2024-09-13T13:01:06.211329+0000","last_active":"2024-09-13T13:01:48.152494+0000","last_peered":"2024-09-13T13:01:48.152494+0000","last_clean":"2024-09-13T13:01:48.152494+0000","last_became_active":"2024-09-13T13:01:06.211034+0000","last_became_peered":"2024-09-13T13:01:06.211034+0000","last_unstale":"2024-09-13T13:01:48.152494+0000","last_undegraded":"2024-09-13T13:01:48.152494+0000","last_fullsized":"2024-09-13T13:01:48.152494+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:33.073914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":69,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.687164+0000","last_change":"2024-09-13T13:01:04.198918+0000","last_active":"2024-09-13T13:01:47.687164+0000","last_peered":"2024-09-13T13:01:47.687164+0000","last_clean":"2024-09-13T13:01:47.687164+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:01:47.687164+0000","last_undegraded":"2024-09-13T13:01:47.687164+0000","last_fullsized":"2024-09-13T13:01:47.687164+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:04.636447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995777+0000","last_change":"2024-09-13T13:01:02.186847+0000","last_active":"2024-09-13T13:01:47.995777+0000","last_peered":"2024-09-13T13:01:47.995777+0000","last_clean":"2024-09-13T13:01:47.995777+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:01:47.995777+0000","last_undegraded":"2024-09-13T13:01:47.995777+0000","last_fullsized":"2024-09-13T13:01:47.995777+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.191632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685765+0000","last_change":"2024-09-13T13:01:06.206080+0000","last_active":"2024-09-13T13:01:47.685765+0000","last_peered":"2024-09-13T13:01:47.685765+0000","last_clean":"2024-09-13T13:01:47.685765+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:01:47.685765+0000","last_undegraded":"2024-09-13T13:01:47.685765+0000","last_fullsized":"2024-09-13T13:01:47.685765+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.738297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150573+0000","last_change":"2024-09-13T13:01:08.216021+0000","last_active":"2024-09-13T13:01:48.150573+0000","last_peered":"2024-09-13T13:01:48.150573+0000","last_clean":"2024-09-13T13:01:48.150573+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:01:48.150573+0000","last_undegraded":"2024-09-13T13:01:48.150573+0000","last_fullsized":"2024-09-13T13:01:48.150573+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:08.377856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":75,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995614+0000","last_change":"2024-09-13T13:01:04.209310+0000","last_active":"2024-09-13T13:01:47.995614+0000","last_peered":"2024-09-13T13:01:47.995614+0000","last_clean":"2024-09-13T13:01:47.995614+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:01:47.995614+0000","last_undegraded":"2024-09-13T13:01:47.995614+0000","last_fullsized":"2024-09-13T13:01:47.995614+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.025302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":53,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996682+0000","last_change":"2024-09-13T13:01:02.181001+0000","last_active":"2024-09-13T13:01:47.996682+0000","last_peered":"2024-09-13T13:01:47.996682+0000","last_clean":"2024-09-13T13:01:47.996682+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:01:47.996682+0000","last_undegraded":"2024-09-13T13:01:47.996682+0000","last_fullsized":"2024-09-13T13:01:47.996682+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.444743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684864+0000","last_change":"2024-09-13T13:01:06.209027+0000","last_active":"2024-09-13T13:01:47.684864+0000","last_peered":"2024-09-13T13:01:47.684864+0000","last_clean":"2024-09-13T13:01:47.684864+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:01:47.684864+0000","last_undegraded":"2024-09-13T13:01:47.684864+0000","last_fullsized":"2024-09-13T13:01:47.684864+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.392689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685570+0000","last_change":"2024-09-13T13:01:08.204782+0000","last_active":"2024-09-13T13:01:47.685570+0000","last_peered":"2024-09-13T13:01:47.685570+0000","last_clean":"2024-09-13T13:01:47.685570+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:01:47.685570+0000","last_undegraded":"2024-09-13T13:01:47.685570+0000","last_fullsized":"2024-09-13T13:01:47.685570+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.812714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684901+0000","last_change":"2024-09-13T13:01:04.187117+0000","last_active":"2024-09-13T13:01:47.684901+0000","last_peered":"2024-09-13T13:01:47.684901+0000","last_clean":"2024-09-13T13:01:47.684901+0000","last_became_active":"2024-09-13T13:01:04.186768+0000","last_became_peered":"2024-09-13T13:01:04.186768+0000","last_unstale":"2024-09-13T13:01:47.684901+0000","last_undegraded":"2024-09-13T13:01:47.684901+0000","last_fullsized":"2024-09-13T13:01:47.684901+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.593884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686132+0000","last_change":"2024-09-13T13:01:02.187792+0000","last_active":"2024-09-13T13:01:47.686132+0000","last_peered":"2024-09-13T13:01:47.686132+0000","last_clean":"2024-09-13T13:01:47.686132+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:01:47.686132+0000","last_undegraded":"2024-09-13T13:01:47.686132+0000","last_fullsized":"2024-09-13T13:01:47.686132+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:08.968885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995507+0000","last_change":"2024-09-13T13:01:06.211211+0000","last_active":"2024-09-13T13:01:47.995507+0000","last_peered":"2024-09-13T13:01:47.995507+0000","last_clean":"2024-09-13T13:01:47.995507+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:01:47.995507+0000","last_undegraded":"2024-09-13T13:01:47.995507+0000","last_fullsized":"2024-09-13T13:01:47.995507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.975106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150283+0000","last_change":"2024-09-13T13:01:08.533116+0000","last_active":"2024-09-13T13:01:48.150283+0000","last_peered":"2024-09-13T13:01:48.150283+0000","last_clean":"2024-09-13T13:01:48.150283+0000","last_became_active":"2024-09-13T13:01:08.532859+0000","last_became_peered":"2024-09-13T13:01:08.532859+0000","last_unstale":"2024-09-13T13:01:48.150283+0000","last_undegraded":"2024-09-13T13:01:48.150283+0000","last_fullsized":"2024-09-13T13:01:48.150283+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:24.307795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686115+0000","last_change":"2024-09-13T13:01:04.199011+0000","last_active":"2024-09-13T13:01:47.686115+0000","last_peered":"2024-09-13T13:01:47.686115+0000","last_clean":"2024-09-13T13:01:47.686115+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:01:47.686115+0000","last_undegraded":"2024-09-13T13:01:47.686115+0000","last_fullsized":"2024-09-13T13:01:47.686115+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.407716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151447+0000","last_change":"2024-09-13T13:01:02.181597+0000","last_active":"2024-09-13T13:01:48.151447+0000","last_peered":"2024-09-13T13:01:48.151447+0000","last_clean":"2024-09-13T13:01:48.151447+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:01:48.151447+0000","last_undegraded":"2024-09-13T13:01:48.151447+0000","last_fullsized":"2024-09-13T13:01:48.151447+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.285259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685208+0000","last_change":"2024-09-13T13:01:06.216352+0000","last_active":"2024-09-13T13:01:47.685208+0000","last_peered":"2024-09-13T13:01:47.685208+0000","last_clean":"2024-09-13T13:01:47.685208+0000","last_became_active":"2024-09-13T13:01:06.216043+0000","last_became_peered":"2024-09-13T13:01:06.216043+0000","last_unstale":"2024-09-13T13:01:47.685208+0000","last_undegraded":"2024-09-13T13:01:47.685208+0000","last_fullsized":"2024-09-13T13:01:47.685208+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.006398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684721+0000","last_change":"2024-09-13T13:01:08.211109+0000","last_active":"2024-09-13T13:01:47.684721+0000","last_peered":"2024-09-13T13:01:47.684721+0000","last_clean":"2024-09-13T13:01:47.684721+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:01:47.684721+0000","last_undegraded":"2024-09-13T13:01:47.684721+0000","last_fullsized":"2024-09-13T13:01:47.684721+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.759653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":79,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685451+0000","last_change":"2024-09-13T13:01:04.207894+0000","last_active":"2024-09-13T13:01:47.685451+0000","last_peered":"2024-09-13T13:01:47.685451+0000","last_clean":"2024-09-13T13:01:47.685451+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:01:47.685451+0000","last_undegraded":"2024-09-13T13:01:47.685451+0000","last_fullsized":"2024-09-13T13:01:47.685451+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.794771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686808+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:01:47.686808+0000","last_peered":"2024-09-13T13:01:47.686808+0000","last_clean":"2024-09-13T13:01:47.686808+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:01:47.686808+0000","last_undegraded":"2024-09-13T13:01:47.686808+0000","last_fullsized":"2024-09-13T13:01:47.686808+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.453481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151656+0000","last_change":"2024-09-13T13:01:06.213926+0000","last_active":"2024-09-13T13:01:48.151656+0000","last_peered":"2024-09-13T13:01:48.151656+0000","last_clean":"2024-09-13T13:01:48.151656+0000","last_became_active":"2024-09-13T13:01:06.213617+0000","last_became_peered":"2024-09-13T13:01:06.213617+0000","last_unstale":"2024-09-13T13:01:48.151656+0000","last_undegraded":"2024-09-13T13:01:48.151656+0000","last_fullsized":"2024-09-13T13:01:48.151656+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.400692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684269+0000","last_change":"2024-09-13T13:01:08.532524+0000","last_active":"2024-09-13T13:01:47.684269+0000","last_peered":"2024-09-13T13:01:47.684269+0000","last_clean":"2024-09-13T13:01:47.684269+0000","last_became_active":"2024-09-13T13:01:08.532145+0000","last_became_peered":"2024-09-13T13:01:08.532145+0000","last_unstale":"2024-09-13T13:01:47.684269+0000","last_undegraded":"2024-09-13T13:01:47.684269+0000","last_fullsized":"2024-09-13T13:01:47.684269+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:16.922652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":103,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686635+0000","last_change":"2024-09-13T13:01:04.197527+0000","last_active":"2024-09-13T13:01:47.686635+0000","last_peered":"2024-09-13T13:01:47.686635+0000","last_clean":"2024-09-13T13:01:47.686635+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:01:47.686635+0000","last_undegraded":"2024-09-13T13:01:47.686635+0000","last_fullsized":"2024-09-13T13:01:47.686635+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.799330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685637+0000","last_change":"2024-09-13T13:01:02.181566+0000","last_active":"2024-09-13T13:01:47.685637+0000","last_peered":"2024-09-13T13:01:47.685637+0000","last_clean":"2024-09-13T13:01:47.685637+0000","last_became_active":"2024-09-13T13:01:02.180760+0000","last_became_peered":"2024-09-13T13:01:02.180760+0000","last_unstale":"2024-09-13T13:01:47.685637+0000","last_undegraded":"2024-09-13T13:01:47.685637+0000","last_fullsized":"2024-09-13T13:01:47.685637+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.355888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150711+0000","last_change":"2024-09-13T13:01:05.199998+0000","last_active":"2024-09-13T13:01:48.150711+0000","last_peered":"2024-09-13T13:01:48.150711+0000","last_clean":"2024-09-13T13:01:48.150711+0000","last_became_active":"2024-09-13T13:01:02.185089+0000","last_became_peered":"2024-09-13T13:01:02.185089+0000","last_unstale":"2024-09-13T13:01:48.150711+0000","last_undegraded":"2024-09-13T13:01:48.150711+0000","last_fullsized":"2024-09-13T13:01:48.150711+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.802534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685181+0000","last_change":"2024-09-13T13:01:06.216206+0000","last_active":"2024-09-13T13:01:47.685181+0000","last_peered":"2024-09-13T13:01:47.685181+0000","last_clean":"2024-09-13T13:01:47.685181+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:01:47.685181+0000","last_undegraded":"2024-09-13T13:01:47.685181+0000","last_fullsized":"2024-09-13T13:01:47.685181+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.187742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996267+0000","last_change":"2024-09-13T13:01:08.203445+0000","last_active":"2024-09-13T13:01:47.996267+0000","last_peered":"2024-09-13T13:01:47.996267+0000","last_clean":"2024-09-13T13:01:47.996267+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:01:47.996267+0000","last_undegraded":"2024-09-13T13:01:47.996267+0000","last_fullsized":"2024-09-13T13:01:47.996267+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.430979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":71,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.687208+0000","last_change":"2024-09-13T13:01:04.192923+0000","last_active":"2024-09-13T13:01:47.687208+0000","last_peered":"2024-09-13T13:01:47.687208+0000","last_clean":"2024-09-13T13:01:47.687208+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:01:47.687208+0000","last_undegraded":"2024-09-13T13:01:47.687208+0000","last_fullsized":"2024-09-13T13:01:47.687208+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.839147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684894+0000","last_change":"2024-09-13T13:01:02.185582+0000","last_active":"2024-09-13T13:01:47.684894+0000","last_peered":"2024-09-13T13:01:47.684894+0000","last_clean":"2024-09-13T13:01:47.684894+0000","last_became_active":"2024-09-13T13:01:02.185263+0000","last_became_peered":"2024-09-13T13:01:02.185263+0000","last_unstale":"2024-09-13T13:01:47.684894+0000","last_undegraded":"2024-09-13T13:01:47.684894+0000","last_fullsized":"2024-09-13T13:01:47.684894+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:28.109915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149563+0000","last_change":"2024-09-13T13:01:05.202103+0000","last_active":"2024-09-13T13:01:48.149563+0000","last_peered":"2024-09-13T13:01:48.149563+0000","last_clean":"2024-09-13T13:01:48.149563+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:01:48.149563+0000","last_undegraded":"2024-09-13T13:01:48.149563+0000","last_fullsized":"2024-09-13T13:01:48.149563+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.268994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150018+0000","last_change":"2024-09-13T13:01:06.205805+0000","last_active":"2024-09-13T13:01:48.150018+0000","last_peered":"2024-09-13T13:01:48.150018+0000","last_clean":"2024-09-13T13:01:48.150018+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:01:48.150018+0000","last_undegraded":"2024-09-13T13:01:48.150018+0000","last_fullsized":"2024-09-13T13:01:48.150018+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.555922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150818+0000","last_change":"2024-09-13T13:01:08.532390+0000","last_active":"2024-09-13T13:01:48.150818+0000","last_peered":"2024-09-13T13:01:48.150818+0000","last_clean":"2024-09-13T13:01:48.150818+0000","last_became_active":"2024-09-13T13:01:08.532087+0000","last_became_peered":"2024-09-13T13:01:08.532087+0000","last_unstale":"2024-09-13T13:01:48.150818+0000","last_undegraded":"2024-09-13T13:01:48.150818+0000","last_fullsized":"2024-09-13T13:01:48.150818+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.390192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685985+0000","last_change":"2024-09-13T13:01:04.193042+0000","last_active":"2024-09-13T13:01:47.685985+0000","last_peered":"2024-09-13T13:01:47.685985+0000","last_clean":"2024-09-13T13:01:47.685985+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:01:47.685985+0000","last_undegraded":"2024-09-13T13:01:47.685985+0000","last_fullsized":"2024-09-13T13:01:47.685985+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.415798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685509+0000","last_change":"2024-09-13T13:01:02.186331+0000","last_active":"2024-09-13T13:01:47.685509+0000","last_peered":"2024-09-13T13:01:47.685509+0000","last_clean":"2024-09-13T13:01:47.685509+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:01:47.685509+0000","last_undegraded":"2024-09-13T13:01:47.685509+0000","last_fullsized":"2024-09-13T13:01:47.685509+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.787015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":101,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:51.498899+0000","last_change":"2024-09-13T13:01:05.200818+0000","last_active":"2024-09-13T13:01:51.498899+0000","last_peered":"2024-09-13T13:01:51.498899+0000","last_clean":"2024-09-13T13:01:51.498899+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:01:51.498899+0000","last_undegraded":"2024-09-13T13:01:51.498899+0000","last_fullsized":"2024-09-13T13:01:51.498899+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.587568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151611+0000","last_change":"2024-09-13T13:01:06.212898+0000","last_active":"2024-09-13T13:01:48.151611+0000","last_peered":"2024-09-13T13:01:48.151611+0000","last_clean":"2024-09-13T13:01:48.151611+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:01:48.151611+0000","last_undegraded":"2024-09-13T13:01:48.151611+0000","last_fullsized":"2024-09-13T13:01:48.151611+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.387930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686408+0000","last_change":"2024-09-13T13:01:08.213728+0000","last_active":"2024-09-13T13:01:47.686408+0000","last_peered":"2024-09-13T13:01:47.686408+0000","last_clean":"2024-09-13T13:01:47.686408+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:01:47.686408+0000","last_undegraded":"2024-09-13T13:01:47.686408+0000","last_fullsized":"2024-09-13T13:01:47.686408+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.763739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":76,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152088+0000","last_change":"2024-09-13T13:01:04.208034+0000","last_active":"2024-09-13T13:01:48.152088+0000","last_peered":"2024-09-13T13:01:48.152088+0000","last_clean":"2024-09-13T13:01:48.152088+0000","last_became_active":"2024-09-13T13:01:04.207653+0000","last_became_peered":"2024-09-13T13:01:04.207653+0000","last_unstale":"2024-09-13T13:01:48.152088+0000","last_undegraded":"2024-09-13T13:01:48.152088+0000","last_fullsized":"2024-09-13T13:01:48.152088+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.433729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995823+0000","last_change":"2024-09-13T13:01:02.180002+0000","last_active":"2024-09-13T13:01:47.995823+0000","last_peered":"2024-09-13T13:01:47.995823+0000","last_clean":"2024-09-13T13:01:47.995823+0000","last_became_active":"2024-09-13T13:01:02.179698+0000","last_became_peered":"2024-09-13T13:01:02.179698+0000","last_unstale":"2024-09-13T13:01:47.995823+0000","last_undegraded":"2024-09-13T13:01:47.995823+0000","last_fullsized":"2024-09-13T13:01:47.995823+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:03.938319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":84,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151302+0000","last_change":"2024-09-13T13:00:53.927016+0000","last_active":"2024-09-13T13:01:48.151302+0000","last_peered":"2024-09-13T13:01:48.151302+0000","last_clean":"2024-09-13T13:01:48.151302+0000","last_became_active":"2024-09-13T13:00:53.922110+0000","last_became_peered":"2024-09-13T13:00:53.922110+0000","last_unstale":"2024-09-13T13:01:48.151302+0000","last_undegraded":"2024-09-13T13:01:48.151302+0000","last_fullsized":"2024-09-13T13:01:48.151302+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-13T13:00:46.618619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:00:46.618619+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.913095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150670+0000","last_change":"2024-09-13T13:01:06.207054+0000","last_active":"2024-09-13T13:01:48.150670+0000","last_peered":"2024-09-13T13:01:48.150670+0000","last_clean":"2024-09-13T13:01:48.150670+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:01:48.150670+0000","last_undegraded":"2024-09-13T13:01:48.150670+0000","last_fullsized":"2024-09-13T13:01:48.150670+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.955181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151924+0000","last_change":"2024-09-13T13:01:08.209367+0000","last_active":"2024-09-13T13:01:48.151924+0000","last_peered":"2024-09-13T13:01:48.151924+0000","last_clean":"2024-09-13T13:01:48.151924+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:01:48.151924+0000","last_undegraded":"2024-09-13T13:01:48.151924+0000","last_fullsized":"2024-09-13T13:01:48.151924+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.418279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685894+0000","last_change":"2024-09-13T13:01:04.196954+0000","last_active":"2024-09-13T13:01:47.685894+0000","last_peered":"2024-09-13T13:01:47.685894+0000","last_clean":"2024-09-13T13:01:47.685894+0000","last_became_active":"2024-09-13T13:01:04.196363+0000","last_became_peered":"2024-09-13T13:01:04.196363+0000","last_unstale":"2024-09-13T13:01:47.685894+0000","last_undegraded":"2024-09-13T13:01:47.685894+0000","last_fullsized":"2024-09-13T13:01:47.685894+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.318889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685831+0000","last_change":"2024-09-13T13:01:02.180296+0000","last_active":"2024-09-13T13:01:47.685831+0000","last_peered":"2024-09-13T13:01:47.685831+0000","last_clean":"2024-09-13T13:01:47.685831+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:01:47.685831+0000","last_undegraded":"2024-09-13T13:01:47.685831+0000","last_fullsized":"2024-09-13T13:01:47.685831+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.041720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151227+0000","last_change":"2024-09-13T13:01:06.209895+0000","last_active":"2024-09-13T13:01:48.151227+0000","last_peered":"2024-09-13T13:01:48.151227+0000","last_clean":"2024-09-13T13:01:48.151227+0000","last_became_active":"2024-09-13T13:01:06.206590+0000","last_became_peered":"2024-09-13T13:01:06.206590+0000","last_unstale":"2024-09-13T13:01:48.151227+0000","last_undegraded":"2024-09-13T13:01:48.151227+0000","last_fullsized":"2024-09-13T13:01:48.151227+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.901892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995757+0000","last_change":"2024-09-13T13:01:08.533475+0000","last_active":"2024-09-13T13:01:47.995757+0000","last_peered":"2024-09-13T13:01:47.995757+0000","last_clean":"2024-09-13T13:01:47.995757+0000","last_became_active":"2024-09-13T13:01:08.533119+0000","last_became_peered":"2024-09-13T13:01:08.533119+0000","last_unstale":"2024-09-13T13:01:47.995757+0000","last_undegraded":"2024-09-13T13:01:47.995757+0000","last_fullsized":"2024-09-13T13:01:47.995757+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.681416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":72,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685709+0000","last_change":"2024-09-13T13:01:04.195092+0000","last_active":"2024-09-13T13:01:47.685709+0000","last_peered":"2024-09-13T13:01:47.685709+0000","last_clean":"2024-09-13T13:01:47.685709+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:01:47.685709+0000","last_undegraded":"2024-09-13T13:01:47.685709+0000","last_fullsized":"2024-09-13T13:01:47.685709+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.170610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152248+0000","last_change":"2024-09-13T13:01:02.181492+0000","last_active":"2024-09-13T13:01:48.152248+0000","last_peered":"2024-09-13T13:01:48.152248+0000","last_clean":"2024-09-13T13:01:48.152248+0000","last_became_active":"2024-09-13T13:01:02.180946+0000","last_became_peered":"2024-09-13T13:01:02.180946+0000","last_unstale":"2024-09-13T13:01:48.152248+0000","last_undegraded":"2024-09-13T13:01:48.152248+0000","last_fullsized":"2024-09-13T13:01:48.152248+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:15.300817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686474+0000","last_change":"2024-09-13T13:01:06.208593+0000","last_active":"2024-09-13T13:01:47.686474+0000","last_peered":"2024-09-13T13:01:47.686474+0000","last_clean":"2024-09-13T13:01:47.686474+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:01:47.686474+0000","last_undegraded":"2024-09-13T13:01:47.686474+0000","last_fullsized":"2024-09-13T13:01:47.686474+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.535525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149895+0000","last_change":"2024-09-13T13:01:08.204029+0000","last_active":"2024-09-13T13:01:48.149895+0000","last_peered":"2024-09-13T13:01:48.149895+0000","last_clean":"2024-09-13T13:01:48.149895+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:01:48.149895+0000","last_undegraded":"2024-09-13T13:01:48.149895+0000","last_fullsized":"2024-09-13T13:01:48.149895+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.251232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685192+0000","last_change":"2024-09-13T13:01:04.189975+0000","last_active":"2024-09-13T13:01:47.685192+0000","last_peered":"2024-09-13T13:01:47.685192+0000","last_clean":"2024-09-13T13:01:47.685192+0000","last_became_active":"2024-09-13T13:01:04.189380+0000","last_became_peered":"2024-09-13T13:01:04.189380+0000","last_unstale":"2024-09-13T13:01:47.685192+0000","last_undegraded":"2024-09-13T13:01:47.685192+0000","last_fullsized":"2024-09-13T13:01:47.685192+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:03.186596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996388+0000","last_change":"2024-09-13T13:01:02.188134+0000","last_active":"2024-09-13T13:01:47.996388+0000","last_peered":"2024-09-13T13:01:47.996388+0000","last_clean":"2024-09-13T13:01:47.996388+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:01:47.996388+0000","last_undegraded":"2024-09-13T13:01:47.996388+0000","last_fullsized":"2024-09-13T13:01:47.996388+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.160057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149939+0000","last_change":"2024-09-13T13:01:06.206335+0000","last_active":"2024-09-13T13:01:48.149939+0000","last_peered":"2024-09-13T13:01:48.149939+0000","last_clean":"2024-09-13T13:01:48.149939+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:01:48.149939+0000","last_undegraded":"2024-09-13T13:01:48.149939+0000","last_fullsized":"2024-09-13T13:01:48.149939+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.752225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685420+0000","last_change":"2024-09-13T13:01:08.213034+0000","last_active":"2024-09-13T13:01:47.685420+0000","last_peered":"2024-09-13T13:01:47.685420+0000","last_clean":"2024-09-13T13:01:47.685420+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:01:47.685420+0000","last_undegraded":"2024-09-13T13:01:47.685420+0000","last_fullsized":"2024-09-13T13:01:47.685420+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.453460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685311+0000","last_change":"2024-09-13T13:01:04.192921+0000","last_active":"2024-09-13T13:01:47.685311+0000","last_peered":"2024-09-13T13:01:47.685311+0000","last_clean":"2024-09-13T13:01:47.685311+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:01:47.685311+0000","last_undegraded":"2024-09-13T13:01:47.685311+0000","last_fullsized":"2024-09-13T13:01:47.685311+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.518566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150300+0000","last_change":"2024-09-13T13:01:02.180807+0000","last_active":"2024-09-13T13:01:48.150300+0000","last_peered":"2024-09-13T13:01:48.150300+0000","last_clean":"2024-09-13T13:01:48.150300+0000","last_became_active":"2024-09-13T13:01:02.180541+0000","last_became_peered":"2024-09-13T13:01:02.180541+0000","last_unstale":"2024-09-13T13:01:48.150300+0000","last_undegraded":"2024-09-13T13:01:48.150300+0000","last_fullsized":"2024-09-13T13:01:48.150300+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:28.541518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149485+0000","last_change":"2024-09-13T13:01:06.207416+0000","last_active":"2024-09-13T13:01:48.149485+0000","last_peered":"2024-09-13T13:01:48.149485+0000","last_clean":"2024-09-13T13:01:48.149485+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:01:48.149485+0000","last_undegraded":"2024-09-13T13:01:48.149485+0000","last_fullsized":"2024-09-13T13:01:48.149485+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.336110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685339+0000","last_change":"2024-09-13T13:01:08.211229+0000","last_active":"2024-09-13T13:01:47.685339+0000","last_peered":"2024-09-13T13:01:47.685339+0000","last_clean":"2024-09-13T13:01:47.685339+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:01:47.685339+0000","last_undegraded":"2024-09-13T13:01:47.685339+0000","last_fullsized":"2024-09-13T13:01:47.685339+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:35.720723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":74,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151927+0000","last_change":"2024-09-13T13:01:04.208073+0000","last_active":"2024-09-13T13:01:48.151927+0000","last_peered":"2024-09-13T13:01:48.151927+0000","last_clean":"2024-09-13T13:01:48.151927+0000","last_became_active":"2024-09-13T13:01:04.207702+0000","last_became_peered":"2024-09-13T13:01:04.207702+0000","last_unstale":"2024-09-13T13:01:48.151927+0000","last_undegraded":"2024-09-13T13:01:48.151927+0000","last_fullsized":"2024-09-13T13:01:48.151927+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.215067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152342+0000","last_change":"2024-09-13T13:01:02.178687+0000","last_active":"2024-09-13T13:01:48.152342+0000","last_peered":"2024-09-13T13:01:48.152342+0000","last_clean":"2024-09-13T13:01:48.152342+0000","last_became_active":"2024-09-13T13:01:02.178317+0000","last_became_peered":"2024-09-13T13:01:02.178317+0000","last_unstale":"2024-09-13T13:01:48.152342+0000","last_undegraded":"2024-09-13T13:01:48.152342+0000","last_fullsized":"2024-09-13T13:01:48.152342+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.914639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149751+0000","last_change":"2024-09-13T13:01:06.203813+0000","last_active":"2024-09-13T13:01:48.149751+0000","last_peered":"2024-09-13T13:01:48.149751+0000","last_clean":"2024-09-13T13:01:48.149751+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:01:48.149751+0000","last_undegraded":"2024-09-13T13:01:48.149751+0000","last_fullsized":"2024-09-13T13:01:48.149751+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:31.552430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151220+0000","last_change":"2024-09-13T13:01:08.212073+0000","last_active":"2024-09-13T13:01:48.151220+0000","last_peered":"2024-09-13T13:01:48.151220+0000","last_clean":"2024-09-13T13:01:48.151220+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:01:48.151220+0000","last_undegraded":"2024-09-13T13:01:48.151220+0000","last_fullsized":"2024-09-13T13:01:48.151220+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:35.915325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685079+0000","last_change":"2024-09-13T13:01:04.196002+0000","last_active":"2024-09-13T13:01:47.685079+0000","last_peered":"2024-09-13T13:01:47.685079+0000","last_clean":"2024-09-13T13:01:47.685079+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:01:47.685079+0000","last_undegraded":"2024-09-13T13:01:47.685079+0000","last_fullsized":"2024-09-13T13:01:47.685079+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.222887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152092+0000","last_change":"2024-09-13T13:01:02.187229+0000","last_active":"2024-09-13T13:01:48.152092+0000","last_peered":"2024-09-13T13:01:48.152092+0000","last_clean":"2024-09-13T13:01:48.152092+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:01:48.152092+0000","last_undegraded":"2024-09-13T13:01:48.152092+0000","last_fullsized":"2024-09-13T13:01:48.152092+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:02.438861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149432+0000","last_change":"2024-09-13T13:01:06.211172+0000","last_active":"2024-09-13T13:01:48.149432+0000","last_peered":"2024-09-13T13:01:48.149432+0000","last_clean":"2024-09-13T13:01:48.149432+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:01:48.149432+0000","last_undegraded":"2024-09-13T13:01:48.149432+0000","last_fullsized":"2024-09-13T13:01:48.149432+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.354291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996008+0000","last_change":"2024-09-13T13:01:08.211530+0000","last_active":"2024-09-13T13:01:47.996008+0000","last_peered":"2024-09-13T13:01:47.996008+0000","last_clean":"2024-09-13T13:01:47.996008+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:01:47.996008+0000","last_undegraded":"2024-09-13T13:01:47.996008+0000","last_fullsized":"2024-09-13T13:01:47.996008+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.539593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":68,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149423+0000","last_change":"2024-09-13T13:01:04.209162+0000","last_active":"2024-09-13T13:01:48.149423+0000","last_peered":"2024-09-13T13:01:48.149423+0000","last_clean":"2024-09-13T13:01:48.149423+0000","last_became_active":"2024-09-13T13:01:04.208731+0000","last_became_peered":"2024-09-13T13:01:04.208731+0000","last_unstale":"2024-09-13T13:01:48.149423+0000","last_undegraded":"2024-09-13T13:01:48.149423+0000","last_fullsized":"2024-09-13T13:01:48.149423+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.356204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152226+0000","last_change":"2024-09-13T13:01:02.183335+0000","last_active":"2024-09-13T13:01:48.152226+0000","last_peered":"2024-09-13T13:01:48.152226+0000","last_clean":"2024-09-13T13:01:48.152226+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:01:48.152226+0000","last_undegraded":"2024-09-13T13:01:48.152226+0000","last_fullsized":"2024-09-13T13:01:48.152226+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:20.269754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149989+0000","last_change":"2024-09-13T13:01:06.210716+0000","last_active":"2024-09-13T13:01:48.149989+0000","last_peered":"2024-09-13T13:01:48.149989+0000","last_clean":"2024-09-13T13:01:48.149989+0000","last_became_active":"2024-09-13T13:01:06.210441+0000","last_became_peered":"2024-09-13T13:01:06.210441+0000","last_unstale":"2024-09-13T13:01:48.149989+0000","last_undegraded":"2024-09-13T13:01:48.149989+0000","last_fullsized":"2024-09-13T13:01:48.149989+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.206296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149578+0000","last_change":"2024-09-13T13:01:08.532185+0000","last_active":"2024-09-13T13:01:48.149578+0000","last_peered":"2024-09-13T13:01:48.149578+0000","last_clean":"2024-09-13T13:01:48.149578+0000","last_became_active":"2024-09-13T13:01:08.531895+0000","last_became_peered":"2024-09-13T13:01:08.531895+0000","last_unstale":"2024-09-13T13:01:48.149578+0000","last_undegraded":"2024-09-13T13:01:48.149578+0000","last_fullsized":"2024-09-13T13:01:48.149578+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.747075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150376+0000","last_change":"2024-09-13T13:01:02.179217+0000","last_active":"2024-09-13T13:01:48.150376+0000","last_peered":"2024-09-13T13:01:48.150376+0000","last_clean":"2024-09-13T13:01:48.150376+0000","last_became_active":"2024-09-13T13:01:02.178934+0000","last_became_peered":"2024-09-13T13:01:02.178934+0000","last_unstale":"2024-09-13T13:01:48.150376+0000","last_undegraded":"2024-09-13T13:01:48.150376+0000","last_fullsized":"2024-09-13T13:01:48.150376+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.675331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996611+0000","last_change":"2024-09-13T13:01:04.209208+0000","last_active":"2024-09-13T13:01:47.996611+0000","last_peered":"2024-09-13T13:01:47.996611+0000","last_clean":"2024-09-13T13:01:47.996611+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:01:47.996611+0000","last_undegraded":"2024-09-13T13:01:47.996611+0000","last_fullsized":"2024-09-13T13:01:47.996611+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.286499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150114+0000","last_change":"2024-09-13T13:01:06.208468+0000","last_active":"2024-09-13T13:01:48.150114+0000","last_peered":"2024-09-13T13:01:48.150114+0000","last_clean":"2024-09-13T13:01:48.150114+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:01:48.150114+0000","last_undegraded":"2024-09-13T13:01:48.150114+0000","last_fullsized":"2024-09-13T13:01:48.150114+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.411667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150544+0000","last_change":"2024-09-13T13:01:08.532411+0000","last_active":"2024-09-13T13:01:48.150544+0000","last_peered":"2024-09-13T13:01:48.150544+0000","last_clean":"2024-09-13T13:01:48.150544+0000","last_became_active":"2024-09-13T13:01:08.532178+0000","last_became_peered":"2024-09-13T13:01:08.532178+0000","last_unstale":"2024-09-13T13:01:48.150544+0000","last_undegraded":"2024-09-13T13:01:48.150544+0000","last_fullsized":"2024-09-13T13:01:48.150544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:15.835392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686003+0000","last_change":"2024-09-13T13:01:04.208866+0000","last_active":"2024-09-13T13:01:47.686003+0000","last_peered":"2024-09-13T13:01:47.686003+0000","last_clean":"2024-09-13T13:01:47.686003+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:01:47.686003+0000","last_undegraded":"2024-09-13T13:01:47.686003+0000","last_fullsized":"2024-09-13T13:01:47.686003+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.015284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150370+0000","last_change":"2024-09-13T13:01:02.181825+0000","last_active":"2024-09-13T13:01:48.150370+0000","last_peered":"2024-09-13T13:01:48.150370+0000","last_clean":"2024-09-13T13:01:48.150370+0000","last_became_active":"2024-09-13T13:01:02.181548+0000","last_became_peered":"2024-09-13T13:01:02.181548+0000","last_unstale":"2024-09-13T13:01:48.150370+0000","last_undegraded":"2024-09-13T13:01:48.150370+0000","last_fullsized":"2024-09-13T13:01:48.150370+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.710586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996537+0000","last_change":"2024-09-13T13:01:06.210328+0000","last_active":"2024-09-13T13:01:47.996537+0000","last_peered":"2024-09-13T13:01:47.996537+0000","last_clean":"2024-09-13T13:01:47.996537+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:01:47.996537+0000","last_undegraded":"2024-09-13T13:01:47.996537+0000","last_fullsized":"2024-09-13T13:01:47.996537+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.781482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149780+0000","last_change":"2024-09-13T13:01:08.205690+0000","last_active":"2024-09-13T13:01:48.149780+0000","last_peered":"2024-09-13T13:01:48.149780+0000","last_clean":"2024-09-13T13:01:48.149780+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:01:48.149780+0000","last_undegraded":"2024-09-13T13:01:48.149780+0000","last_fullsized":"2024-09-13T13:01:48.149780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.215451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151154+0000","last_change":"2024-09-13T13:01:04.208351+0000","last_active":"2024-09-13T13:01:48.151154+0000","last_peered":"2024-09-13T13:01:48.151154+0000","last_clean":"2024-09-13T13:01:48.151154+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:01:48.151154+0000","last_undegraded":"2024-09-13T13:01:48.151154+0000","last_fullsized":"2024-09-13T13:01:48.151154+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.268797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685047+0000","last_change":"2024-09-13T13:01:02.185440+0000","last_active":"2024-09-13T13:01:47.685047+0000","last_peered":"2024-09-13T13:01:47.685047+0000","last_clean":"2024-09-13T13:01:47.685047+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:01:47.685047+0000","last_undegraded":"2024-09-13T13:01:47.685047+0000","last_fullsized":"2024-09-13T13:01:47.685047+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.667174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996034+0000","last_change":"2024-09-13T13:01:06.210251+0000","last_active":"2024-09-13T13:01:47.996034+0000","last_peered":"2024-09-13T13:01:47.996034+0000","last_clean":"2024-09-13T13:01:47.996034+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:01:47.996034+0000","last_undegraded":"2024-09-13T13:01:47.996034+0000","last_fullsized":"2024-09-13T13:01:47.996034+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.308767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685613+0000","last_change":"2024-09-13T13:01:08.206113+0000","last_active":"2024-09-13T13:01:47.685613+0000","last_peered":"2024-09-13T13:01:47.685613+0000","last_clean":"2024-09-13T13:01:47.685613+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:01:47.685613+0000","last_undegraded":"2024-09-13T13:01:47.685613+0000","last_fullsized":"2024-09-13T13:01:47.685613+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.315612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996401+0000","last_change":"2024-09-13T13:01:04.208466+0000","last_active":"2024-09-13T13:01:47.996401+0000","last_peered":"2024-09-13T13:01:47.996401+0000","last_clean":"2024-09-13T13:01:47.996401+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:01:47.996401+0000","last_undegraded":"2024-09-13T13:01:47.996401+0000","last_fullsized":"2024-09-13T13:01:47.996401+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.373274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151566+0000","last_change":"2024-09-13T13:01:02.180706+0000","last_active":"2024-09-13T13:01:48.151566+0000","last_peered":"2024-09-13T13:01:48.151566+0000","last_clean":"2024-09-13T13:01:48.151566+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:01:48.151566+0000","last_undegraded":"2024-09-13T13:01:48.151566+0000","last_fullsized":"2024-09-13T13:01:48.151566+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.494760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150871+0000","last_change":"2024-09-13T13:01:06.206440+0000","last_active":"2024-09-13T13:01:48.150871+0000","last_peered":"2024-09-13T13:01:48.150871+0000","last_clean":"2024-09-13T13:01:48.150871+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:01:48.150871+0000","last_undegraded":"2024-09-13T13:01:48.150871+0000","last_fullsized":"2024-09-13T13:01:48.150871+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.104127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685534+0000","last_change":"2024-09-13T13:01:08.215072+0000","last_active":"2024-09-13T13:01:47.685534+0000","last_peered":"2024-09-13T13:01:47.685534+0000","last_clean":"2024-09-13T13:01:47.685534+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:01:47.685534+0000","last_undegraded":"2024-09-13T13:01:47.685534+0000","last_fullsized":"2024-09-13T13:01:47.685534+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.276961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685087+0000","last_change":"2024-09-13T13:01:04.197213+0000","last_active":"2024-09-13T13:01:47.685087+0000","last_peered":"2024-09-13T13:01:47.685087+0000","last_clean":"2024-09-13T13:01:47.685087+0000","last_became_active":"2024-09-13T13:01:04.196857+0000","last_became_peered":"2024-09-13T13:01:04.196857+0000","last_unstale":"2024-09-13T13:01:47.685087+0000","last_undegraded":"2024-09-13T13:01:47.685087+0000","last_fullsized":"2024-09-13T13:01:47.685087+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.316586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685397+0000","last_change":"2024-09-13T13:01:02.181939+0000","last_active":"2024-09-13T13:01:47.685397+0000","last_peered":"2024-09-13T13:01:47.685397+0000","last_clean":"2024-09-13T13:01:47.685397+0000","last_became_active":"2024-09-13T13:01:02.180817+0000","last_became_peered":"2024-09-13T13:01:02.180817+0000","last_unstale":"2024-09-13T13:01:47.685397+0000","last_undegraded":"2024-09-13T13:01:47.685397+0000","last_fullsized":"2024-09-13T13:01:47.685397+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.328756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685953+0000","last_change":"2024-09-13T13:01:06.209945+0000","last_active":"2024-09-13T13:01:47.685953+0000","last_peered":"2024-09-13T13:01:47.685953+0000","last_clean":"2024-09-13T13:01:47.685953+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:01:47.685953+0000","last_undegraded":"2024-09-13T13:01:47.685953+0000","last_fullsized":"2024-09-13T13:01:47.685953+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.087308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995515+0000","last_change":"2024-09-13T13:01:08.205355+0000","last_active":"2024-09-13T13:01:47.995515+0000","last_peered":"2024-09-13T13:01:47.995515+0000","last_clean":"2024-09-13T13:01:47.995515+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:01:47.995515+0000","last_undegraded":"2024-09-13T13:01:47.995515+0000","last_fullsized":"2024-09-13T13:01:47.995515+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.257400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686273+0000","last_change":"2024-09-13T13:01:04.196437+0000","last_active":"2024-09-13T13:01:47.686273+0000","last_peered":"2024-09-13T13:01:47.686273+0000","last_clean":"2024-09-13T13:01:47.686273+0000","last_became_active":"2024-09-13T13:01:04.196045+0000","last_became_peered":"2024-09-13T13:01:04.196045+0000","last_unstale":"2024-09-13T13:01:47.686273+0000","last_undegraded":"2024-09-13T13:01:47.686273+0000","last_fullsized":"2024-09-13T13:01:47.686273+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:24.629192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152201+0000","last_change":"2024-09-13T13:01:02.182107+0000","last_active":"2024-09-13T13:01:48.152201+0000","last_peered":"2024-09-13T13:01:48.152201+0000","last_clean":"2024-09-13T13:01:48.152201+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:01:48.152201+0000","last_undegraded":"2024-09-13T13:01:48.152201+0000","last_fullsized":"2024-09-13T13:01:48.152201+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.840261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996416+0000","last_change":"2024-09-13T13:01:06.211993+0000","last_active":"2024-09-13T13:01:47.996416+0000","last_peered":"2024-09-13T13:01:47.996416+0000","last_clean":"2024-09-13T13:01:47.996416+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:01:47.996416+0000","last_undegraded":"2024-09-13T13:01:47.996416+0000","last_fullsized":"2024-09-13T13:01:47.996416+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.375348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684160+0000","last_change":"2024-09-13T13:01:08.210271+0000","last_active":"2024-09-13T13:01:47.684160+0000","last_peered":"2024-09-13T13:01:47.684160+0000","last_clean":"2024-09-13T13:01:47.684160+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:01:47.684160+0000","last_undegraded":"2024-09-13T13:01:47.684160+0000","last_fullsized":"2024-09-13T13:01:47.684160+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.691464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995642+0000","last_change":"2024-09-13T13:01:04.208421+0000","last_active":"2024-09-13T13:01:47.995642+0000","last_peered":"2024-09-13T13:01:47.995642+0000","last_clean":"2024-09-13T13:01:47.995642+0000","last_became_active":"2024-09-13T13:01:04.207841+0000","last_became_peered":"2024-09-13T13:01:04.207841+0000","last_unstale":"2024-09-13T13:01:47.995642+0000","last_undegraded":"2024-09-13T13:01:47.995642+0000","last_fullsized":"2024-09-13T13:01:47.995642+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.504022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150242+0000","last_change":"2024-09-13T13:01:02.185420+0000","last_active":"2024-09-13T13:01:48.150242+0000","last_peered":"2024-09-13T13:01:48.150242+0000","last_clean":"2024-09-13T13:01:48.150242+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:01:48.150242+0000","last_undegraded":"2024-09-13T13:01:48.150242+0000","last_fullsized":"2024-09-13T13:01:48.150242+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.931940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149554+0000","last_change":"2024-09-13T13:01:06.208696+0000","last_active":"2024-09-13T13:01:48.149554+0000","last_peered":"2024-09-13T13:01:48.149554+0000","last_clean":"2024-09-13T13:01:48.149554+0000","last_became_active":"2024-09-13T13:01:06.208426+0000","last_became_peered":"2024-09-13T13:01:06.208426+0000","last_unstale":"2024-09-13T13:01:48.149554+0000","last_undegraded":"2024-09-13T13:01:48.149554+0000","last_fullsized":"2024-09-13T13:01:48.149554+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.767606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996259+0000","last_change":"2024-09-13T13:01:08.535233+0000","last_active":"2024-09-13T13:01:47.996259+0000","last_peered":"2024-09-13T13:01:47.996259+0000","last_clean":"2024-09-13T13:01:47.996259+0000","last_became_active":"2024-09-13T13:01:08.534673+0000","last_became_peered":"2024-09-13T13:01:08.534673+0000","last_unstale":"2024-09-13T13:01:47.996259+0000","last_undegraded":"2024-09-13T13:01:47.996259+0000","last_fullsized":"2024-09-13T13:01:47.996259+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:33.330172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":49,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995528+0000","last_change":"2024-09-13T13:01:04.195434+0000","last_active":"2024-09-13T13:01:47.995528+0000","last_peered":"2024-09-13T13:01:47.995528+0000","last_clean":"2024-09-13T13:01:47.995528+0000","last_became_active":"2024-09-13T13:01:04.195125+0000","last_became_peered":"2024-09-13T13:01:04.195125+0000","last_unstale":"2024-09-13T13:01:47.995528+0000","last_undegraded":"2024-09-13T13:01:47.995528+0000","last_fullsized":"2024-09-13T13:01:47.995528+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.430862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685638+0000","last_change":"2024-09-13T13:01:02.186003+0000","last_active":"2024-09-13T13:01:47.685638+0000","last_peered":"2024-09-13T13:01:47.685638+0000","last_clean":"2024-09-13T13:01:47.685638+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:01:47.685638+0000","last_undegraded":"2024-09-13T13:01:47.685638+0000","last_fullsized":"2024-09-13T13:01:47.685638+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.400129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150966+0000","last_change":"2024-09-13T13:01:06.211514+0000","last_active":"2024-09-13T13:01:48.150966+0000","last_peered":"2024-09-13T13:01:48.150966+0000","last_clean":"2024-09-13T13:01:48.150966+0000","last_became_active":"2024-09-13T13:01:06.211243+0000","last_became_peered":"2024-09-13T13:01:06.211243+0000","last_unstale":"2024-09-13T13:01:48.150966+0000","last_undegraded":"2024-09-13T13:01:48.150966+0000","last_fullsized":"2024-09-13T13:01:48.150966+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:16.563704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149886+0000","last_change":"2024-09-13T13:01:08.210205+0000","last_active":"2024-09-13T13:01:48.149886+0000","last_peered":"2024-09-13T13:01:48.149886+0000","last_clean":"2024-09-13T13:01:48.149886+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:01:48.149886+0000","last_undegraded":"2024-09-13T13:01:48.149886+0000","last_fullsized":"2024-09-13T13:01:48.149886+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.043981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686712+0000","last_change":"2024-09-13T13:01:08.214778+0000","last_active":"2024-09-13T13:01:47.686712+0000","last_peered":"2024-09-13T13:01:47.686712+0000","last_clean":"2024-09-13T13:01:47.686712+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:01:47.686712+0000","last_undegraded":"2024-09-13T13:01:47.686712+0000","last_fullsized":"2024-09-13T13:01:47.686712+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:32.480474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995775+0000","last_change":"2024-09-13T13:01:02.187888+0000","last_active":"2024-09-13T13:01:47.995775+0000","last_peered":"2024-09-13T13:01:47.995775+0000","last_clean":"2024-09-13T13:01:47.995775+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:01:47.995775+0000","last_undegraded":"2024-09-13T13:01:47.995775+0000","last_fullsized":"2024-09-13T13:01:47.995775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.807870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152523+0000","last_change":"2024-09-13T13:01:04.195160+0000","last_active":"2024-09-13T13:01:48.152523+0000","last_peered":"2024-09-13T13:01:48.152523+0000","last_clean":"2024-09-13T13:01:48.152523+0000","last_became_active":"2024-09-13T13:01:04.194855+0000","last_became_peered":"2024-09-13T13:01:04.194855+0000","last_unstale":"2024-09-13T13:01:48.152523+0000","last_undegraded":"2024-09-13T13:01:48.152523+0000","last_fullsized":"2024-09-13T13:01:48.152523+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:20.224127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685740+0000","last_change":"2024-09-13T13:01:06.211596+0000","last_active":"2024-09-13T13:01:47.685740+0000","last_peered":"2024-09-13T13:01:47.685740+0000","last_clean":"2024-09-13T13:01:47.685740+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:01:47.685740+0000","last_undegraded":"2024-09-13T13:01:47.685740+0000","last_fullsized":"2024-09-13T13:01:47.685740+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.427559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201293,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28384,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716768,"statfs":{"total":95995035648,"available":95965970432,"internally_reserved":0,"allocated":1388544,"data_stored":909176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430227,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1372160,"data_stored":907547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691865,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31952,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713200,"statfs":{"total":95995035648,"available":95962316800,"internally_reserved":0,"allocated":782336,"data_stored":317538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920799,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":806912,"data_stored":317784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117030,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713172,"statfs":{"total":95995035648,"available":95962288128,"internally_reserved":0,"allocated":811008,"data_stored":318614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27780,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717372,"statfs":{"total":95995035648,"available":95966588928,"internally_reserved":0,"allocated":765952,"data_stored":316878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":835584,"data_stored":318515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32548,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712604,"statfs":{"total":95995035648,"available":95961706496,"internally_reserved":0,"allocated":1392640,"data_stored":908885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:01:57.123 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:01:57.383 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:57.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:57 smithi052 ceph-mon[36682]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T13:01:57.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:57 smithi052 ceph-mon[36682]: Deploying daemon grafana.a on smithi167 2024-09-13T13:01:57.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:57 smithi052 ceph-mon[36682]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:01:57.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:57 smithi052 ceph-mon[36682]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:01:57.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:57 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:57.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:57 smithi052 ceph-mon[29471]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T13:01:57.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:57 smithi052 ceph-mon[29471]: Deploying daemon grafana.a on smithi167 2024-09-13T13:01:57.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:57 smithi052 ceph-mon[29471]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:01:57.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:57 smithi052 ceph-mon[29471]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:01:57.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:57 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:57.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:57 smithi167 ceph-mon[33178]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T13:01:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:57 smithi167 ceph-mon[33178]: Deploying daemon grafana.a on smithi167 2024-09-13T13:01:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:57 smithi167 ceph-mon[33178]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:01:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:57 smithi167 ceph-mon[33178]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:01:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:57 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:01:58.009 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:01:58.011 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:01:58.606 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:58 smithi052 ceph-mon[36682]: from='client.34688 v1:172.21.15.52:0/1928990499' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:01:58.606 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:58 smithi052 ceph-mon[29471]: from='client.34688 v1:172.21.15.52:0/1928990499' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:01:58.640 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T13:01:57.874984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":898,"num_read_kb":755,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236676,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724540,"statfs":{"total":767960285184,"available":767717928960,"internally_reserved":0,"allocated":8155136,"data_stored":4314937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":234016306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.191637"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685552+0000","last_change":"2024-09-13T13:01:02.184818+0000","last_active":"2024-09-13T13:01:47.685552+0000","last_peered":"2024-09-13T13:01:47.685552+0000","last_clean":"2024-09-13T13:01:47.685552+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:01:47.685552+0000","last_undegraded":"2024-09-13T13:01:47.685552+0000","last_fullsized":"2024-09-13T13:01:47.685552+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.229908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684822+0000","last_change":"2024-09-13T13:01:04.194784+0000","last_active":"2024-09-13T13:01:47.684822+0000","last_peered":"2024-09-13T13:01:47.684822+0000","last_clean":"2024-09-13T13:01:47.684822+0000","last_became_active":"2024-09-13T13:01:04.194444+0000","last_became_peered":"2024-09-13T13:01:04.194444+0000","last_unstale":"2024-09-13T13:01:47.684822+0000","last_undegraded":"2024-09-13T13:01:47.684822+0000","last_fullsized":"2024-09-13T13:01:47.684822+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.654746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686249+0000","last_change":"2024-09-13T13:01:06.207968+0000","last_active":"2024-09-13T13:01:47.686249+0000","last_peered":"2024-09-13T13:01:47.686249+0000","last_clean":"2024-09-13T13:01:47.686249+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:01:47.686249+0000","last_undegraded":"2024-09-13T13:01:47.686249+0000","last_fullsized":"2024-09-13T13:01:47.686249+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.690479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685023+0000","last_change":"2024-09-13T13:01:08.214110+0000","last_active":"2024-09-13T13:01:47.685023+0000","last_peered":"2024-09-13T13:01:47.685023+0000","last_clean":"2024-09-13T13:01:47.685023+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:01:47.685023+0000","last_undegraded":"2024-09-13T13:01:47.685023+0000","last_fullsized":"2024-09-13T13:01:47.685023+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.859045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995992+0000","last_change":"2024-09-13T13:01:08.534893+0000","last_active":"2024-09-13T13:01:47.995992+0000","last_peered":"2024-09-13T13:01:47.995992+0000","last_clean":"2024-09-13T13:01:47.995992+0000","last_became_active":"2024-09-13T13:01:08.534558+0000","last_became_peered":"2024-09-13T13:01:08.534558+0000","last_unstale":"2024-09-13T13:01:47.995992+0000","last_undegraded":"2024-09-13T13:01:47.995992+0000","last_fullsized":"2024-09-13T13:01:47.995992+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.865474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996526+0000","last_change":"2024-09-13T13:01:02.183744+0000","last_active":"2024-09-13T13:01:47.996526+0000","last_peered":"2024-09-13T13:01:47.996526+0000","last_clean":"2024-09-13T13:01:47.996526+0000","last_became_active":"2024-09-13T13:01:02.183415+0000","last_became_peered":"2024-09-13T13:01:02.183415+0000","last_unstale":"2024-09-13T13:01:47.996526+0000","last_undegraded":"2024-09-13T13:01:47.996526+0000","last_fullsized":"2024-09-13T13:01:47.996526+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.940777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":68,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995788+0000","last_change":"2024-09-13T13:01:04.187590+0000","last_active":"2024-09-13T13:01:47.995788+0000","last_peered":"2024-09-13T13:01:47.995788+0000","last_clean":"2024-09-13T13:01:47.995788+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:01:47.995788+0000","last_undegraded":"2024-09-13T13:01:47.995788+0000","last_fullsized":"2024-09-13T13:01:47.995788+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.466652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684811+0000","last_change":"2024-09-13T13:01:06.207701+0000","last_active":"2024-09-13T13:01:47.684811+0000","last_peered":"2024-09-13T13:01:47.684811+0000","last_clean":"2024-09-13T13:01:47.684811+0000","last_became_active":"2024-09-13T13:01:06.207429+0000","last_became_peered":"2024-09-13T13:01:06.207429+0000","last_unstale":"2024-09-13T13:01:47.684811+0000","last_undegraded":"2024-09-13T13:01:47.684811+0000","last_fullsized":"2024-09-13T13:01:47.684811+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.420247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":53,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151307+0000","last_change":"2024-09-13T13:01:02.184512+0000","last_active":"2024-09-13T13:01:48.151307+0000","last_peered":"2024-09-13T13:01:48.151307+0000","last_clean":"2024-09-13T13:01:48.151307+0000","last_became_active":"2024-09-13T13:01:02.184241+0000","last_became_peered":"2024-09-13T13:01:02.184241+0000","last_unstale":"2024-09-13T13:01:48.151307+0000","last_undegraded":"2024-09-13T13:01:48.151307+0000","last_fullsized":"2024-09-13T13:01:48.151307+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.275782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685708+0000","last_change":"2024-09-13T13:01:04.189504+0000","last_active":"2024-09-13T13:01:47.685708+0000","last_peered":"2024-09-13T13:01:47.685708+0000","last_clean":"2024-09-13T13:01:47.685708+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:01:47.685708+0000","last_undegraded":"2024-09-13T13:01:47.685708+0000","last_fullsized":"2024-09-13T13:01:47.685708+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:25.303489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149681+0000","last_change":"2024-09-13T13:01:06.205759+0000","last_active":"2024-09-13T13:01:48.149681+0000","last_peered":"2024-09-13T13:01:48.149681+0000","last_clean":"2024-09-13T13:01:48.149681+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:01:48.149681+0000","last_undegraded":"2024-09-13T13:01:48.149681+0000","last_fullsized":"2024-09-13T13:01:48.149681+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.930018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684263+0000","last_change":"2024-09-13T13:01:08.210285+0000","last_active":"2024-09-13T13:01:47.684263+0000","last_peered":"2024-09-13T13:01:47.684263+0000","last_clean":"2024-09-13T13:01:47.684263+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:01:47.684263+0000","last_undegraded":"2024-09-13T13:01:47.684263+0000","last_fullsized":"2024-09-13T13:01:47.684263+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:24.867052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150075+0000","last_change":"2024-09-13T13:01:02.184552+0000","last_active":"2024-09-13T13:01:48.150075+0000","last_peered":"2024-09-13T13:01:48.150075+0000","last_clean":"2024-09-13T13:01:48.150075+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:01:48.150075+0000","last_undegraded":"2024-09-13T13:01:48.150075+0000","last_fullsized":"2024-09-13T13:01:48.150075+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:05.277064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":54,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685089+0000","last_change":"2024-09-13T13:01:04.196359+0000","last_active":"2024-09-13T13:01:47.685089+0000","last_peered":"2024-09-13T13:01:47.685089+0000","last_clean":"2024-09-13T13:01:47.685089+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:01:47.685089+0000","last_undegraded":"2024-09-13T13:01:47.685089+0000","last_fullsized":"2024-09-13T13:01:47.685089+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.916135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150044+0000","last_change":"2024-09-13T13:01:06.210845+0000","last_active":"2024-09-13T13:01:48.150044+0000","last_peered":"2024-09-13T13:01:48.150044+0000","last_clean":"2024-09-13T13:01:48.150044+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:01:48.150044+0000","last_undegraded":"2024-09-13T13:01:48.150044+0000","last_fullsized":"2024-09-13T13:01:48.150044+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.182827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149616+0000","last_change":"2024-09-13T13:01:08.213608+0000","last_active":"2024-09-13T13:01:48.149616+0000","last_peered":"2024-09-13T13:01:48.149616+0000","last_clean":"2024-09-13T13:01:48.149616+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:01:48.149616+0000","last_undegraded":"2024-09-13T13:01:48.149616+0000","last_fullsized":"2024-09-13T13:01:48.149616+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.927149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":37,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685600+0000","last_change":"2024-09-13T13:01:08.533336+0000","last_active":"2024-09-13T13:01:47.685600+0000","last_peered":"2024-09-13T13:01:47.685600+0000","last_clean":"2024-09-13T13:01:47.685600+0000","last_became_active":"2024-09-13T13:01:08.533051+0000","last_became_peered":"2024-09-13T13:01:08.533051+0000","last_unstale":"2024-09-13T13:01:47.685600+0000","last_undegraded":"2024-09-13T13:01:47.685600+0000","last_fullsized":"2024-09-13T13:01:47.685600+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:31.568605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685736+0000","last_change":"2024-09-13T13:01:02.183913+0000","last_active":"2024-09-13T13:01:47.685736+0000","last_peered":"2024-09-13T13:01:47.685736+0000","last_clean":"2024-09-13T13:01:47.685736+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:01:47.685736+0000","last_undegraded":"2024-09-13T13:01:47.685736+0000","last_fullsized":"2024-09-13T13:01:47.685736+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.747175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":69,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150076+0000","last_change":"2024-09-13T13:01:04.191966+0000","last_active":"2024-09-13T13:01:48.150076+0000","last_peered":"2024-09-13T13:01:48.150076+0000","last_clean":"2024-09-13T13:01:48.150076+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:01:48.150076+0000","last_undegraded":"2024-09-13T13:01:48.150076+0000","last_fullsized":"2024-09-13T13:01:48.150076+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:05.477313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686922+0000","last_change":"2024-09-13T13:01:06.209495+0000","last_active":"2024-09-13T13:01:47.686922+0000","last_peered":"2024-09-13T13:01:47.686922+0000","last_clean":"2024-09-13T13:01:47.686922+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:01:47.686922+0000","last_undegraded":"2024-09-13T13:01:47.686922+0000","last_fullsized":"2024-09-13T13:01:47.686922+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.608220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996469+0000","last_change":"2024-09-13T13:01:08.533133+0000","last_active":"2024-09-13T13:01:47.996469+0000","last_peered":"2024-09-13T13:01:47.996469+0000","last_clean":"2024-09-13T13:01:47.996469+0000","last_became_active":"2024-09-13T13:01:08.532753+0000","last_became_peered":"2024-09-13T13:01:08.532753+0000","last_unstale":"2024-09-13T13:01:47.996469+0000","last_undegraded":"2024-09-13T13:01:47.996469+0000","last_fullsized":"2024-09-13T13:01:47.996469+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.674547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685839+0000","last_change":"2024-09-13T13:01:02.173465+0000","last_active":"2024-09-13T13:01:47.685839+0000","last_peered":"2024-09-13T13:01:47.685839+0000","last_clean":"2024-09-13T13:01:47.685839+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:01:47.685839+0000","last_undegraded":"2024-09-13T13:01:47.685839+0000","last_fullsized":"2024-09-13T13:01:47.685839+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:28.027613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":66,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996200+0000","last_change":"2024-09-13T13:01:04.195126+0000","last_active":"2024-09-13T13:01:47.996200+0000","last_peered":"2024-09-13T13:01:47.996200+0000","last_clean":"2024-09-13T13:01:47.996200+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:01:47.996200+0000","last_undegraded":"2024-09-13T13:01:47.996200+0000","last_fullsized":"2024-09-13T13:01:47.996200+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.577922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685142+0000","last_change":"2024-09-13T13:01:06.207164+0000","last_active":"2024-09-13T13:01:47.685142+0000","last_peered":"2024-09-13T13:01:47.685142+0000","last_clean":"2024-09-13T13:01:47.685142+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:01:47.685142+0000","last_undegraded":"2024-09-13T13:01:47.685142+0000","last_fullsized":"2024-09-13T13:01:47.685142+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.747378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150287+0000","last_change":"2024-09-13T13:01:08.210300+0000","last_active":"2024-09-13T13:01:48.150287+0000","last_peered":"2024-09-13T13:01:48.150287+0000","last_clean":"2024-09-13T13:01:48.150287+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:01:48.150287+0000","last_undegraded":"2024-09-13T13:01:48.150287+0000","last_fullsized":"2024-09-13T13:01:48.150287+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.173643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":49,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686330+0000","last_change":"2024-09-13T13:01:02.180198+0000","last_active":"2024-09-13T13:01:47.686330+0000","last_peered":"2024-09-13T13:01:47.686330+0000","last_clean":"2024-09-13T13:01:47.686330+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:01:47.686330+0000","last_undegraded":"2024-09-13T13:01:47.686330+0000","last_fullsized":"2024-09-13T13:01:47.686330+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:09.281312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685868+0000","last_change":"2024-09-13T13:01:04.208045+0000","last_active":"2024-09-13T13:01:47.685868+0000","last_peered":"2024-09-13T13:01:47.685868+0000","last_clean":"2024-09-13T13:01:47.685868+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:01:47.685868+0000","last_undegraded":"2024-09-13T13:01:47.685868+0000","last_fullsized":"2024-09-13T13:01:47.685868+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.686327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:52.240596+0000","last_change":"2024-09-13T13:01:06.211329+0000","last_active":"2024-09-13T13:01:52.240596+0000","last_peered":"2024-09-13T13:01:52.240596+0000","last_clean":"2024-09-13T13:01:52.240596+0000","last_became_active":"2024-09-13T13:01:06.211034+0000","last_became_peered":"2024-09-13T13:01:06.211034+0000","last_unstale":"2024-09-13T13:01:52.240596+0000","last_undegraded":"2024-09-13T13:01:52.240596+0000","last_fullsized":"2024-09-13T13:01:52.240596+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:33.073914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":69,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.687164+0000","last_change":"2024-09-13T13:01:04.198918+0000","last_active":"2024-09-13T13:01:47.687164+0000","last_peered":"2024-09-13T13:01:47.687164+0000","last_clean":"2024-09-13T13:01:47.687164+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:01:47.687164+0000","last_undegraded":"2024-09-13T13:01:47.687164+0000","last_fullsized":"2024-09-13T13:01:47.687164+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:04.636447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995777+0000","last_change":"2024-09-13T13:01:02.186847+0000","last_active":"2024-09-13T13:01:47.995777+0000","last_peered":"2024-09-13T13:01:47.995777+0000","last_clean":"2024-09-13T13:01:47.995777+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:01:47.995777+0000","last_undegraded":"2024-09-13T13:01:47.995777+0000","last_fullsized":"2024-09-13T13:01:47.995777+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.191632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:52.240546+0000","last_change":"2024-09-13T13:01:06.206080+0000","last_active":"2024-09-13T13:01:52.240546+0000","last_peered":"2024-09-13T13:01:52.240546+0000","last_clean":"2024-09-13T13:01:52.240546+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:01:52.240546+0000","last_undegraded":"2024-09-13T13:01:52.240546+0000","last_fullsized":"2024-09-13T13:01:52.240546+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.738297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150573+0000","last_change":"2024-09-13T13:01:08.216021+0000","last_active":"2024-09-13T13:01:48.150573+0000","last_peered":"2024-09-13T13:01:48.150573+0000","last_clean":"2024-09-13T13:01:48.150573+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:01:48.150573+0000","last_undegraded":"2024-09-13T13:01:48.150573+0000","last_fullsized":"2024-09-13T13:01:48.150573+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:08.377856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":75,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995614+0000","last_change":"2024-09-13T13:01:04.209310+0000","last_active":"2024-09-13T13:01:47.995614+0000","last_peered":"2024-09-13T13:01:47.995614+0000","last_clean":"2024-09-13T13:01:47.995614+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:01:47.995614+0000","last_undegraded":"2024-09-13T13:01:47.995614+0000","last_fullsized":"2024-09-13T13:01:47.995614+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.025302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":53,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996682+0000","last_change":"2024-09-13T13:01:02.181001+0000","last_active":"2024-09-13T13:01:47.996682+0000","last_peered":"2024-09-13T13:01:47.996682+0000","last_clean":"2024-09-13T13:01:47.996682+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:01:47.996682+0000","last_undegraded":"2024-09-13T13:01:47.996682+0000","last_fullsized":"2024-09-13T13:01:47.996682+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.444743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684864+0000","last_change":"2024-09-13T13:01:06.209027+0000","last_active":"2024-09-13T13:01:47.684864+0000","last_peered":"2024-09-13T13:01:47.684864+0000","last_clean":"2024-09-13T13:01:47.684864+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:01:47.684864+0000","last_undegraded":"2024-09-13T13:01:47.684864+0000","last_fullsized":"2024-09-13T13:01:47.684864+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.392689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685570+0000","last_change":"2024-09-13T13:01:08.204782+0000","last_active":"2024-09-13T13:01:47.685570+0000","last_peered":"2024-09-13T13:01:47.685570+0000","last_clean":"2024-09-13T13:01:47.685570+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:01:47.685570+0000","last_undegraded":"2024-09-13T13:01:47.685570+0000","last_fullsized":"2024-09-13T13:01:47.685570+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.812714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684901+0000","last_change":"2024-09-13T13:01:04.187117+0000","last_active":"2024-09-13T13:01:47.684901+0000","last_peered":"2024-09-13T13:01:47.684901+0000","last_clean":"2024-09-13T13:01:47.684901+0000","last_became_active":"2024-09-13T13:01:04.186768+0000","last_became_peered":"2024-09-13T13:01:04.186768+0000","last_unstale":"2024-09-13T13:01:47.684901+0000","last_undegraded":"2024-09-13T13:01:47.684901+0000","last_fullsized":"2024-09-13T13:01:47.684901+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.593884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686132+0000","last_change":"2024-09-13T13:01:02.187792+0000","last_active":"2024-09-13T13:01:47.686132+0000","last_peered":"2024-09-13T13:01:47.686132+0000","last_clean":"2024-09-13T13:01:47.686132+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:01:47.686132+0000","last_undegraded":"2024-09-13T13:01:47.686132+0000","last_fullsized":"2024-09-13T13:01:47.686132+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:08.968885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:52.240449+0000","last_change":"2024-09-13T13:01:06.211211+0000","last_active":"2024-09-13T13:01:52.240449+0000","last_peered":"2024-09-13T13:01:52.240449+0000","last_clean":"2024-09-13T13:01:52.240449+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:01:52.240449+0000","last_undegraded":"2024-09-13T13:01:52.240449+0000","last_fullsized":"2024-09-13T13:01:52.240449+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.975106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150283+0000","last_change":"2024-09-13T13:01:08.533116+0000","last_active":"2024-09-13T13:01:48.150283+0000","last_peered":"2024-09-13T13:01:48.150283+0000","last_clean":"2024-09-13T13:01:48.150283+0000","last_became_active":"2024-09-13T13:01:08.532859+0000","last_became_peered":"2024-09-13T13:01:08.532859+0000","last_unstale":"2024-09-13T13:01:48.150283+0000","last_undegraded":"2024-09-13T13:01:48.150283+0000","last_fullsized":"2024-09-13T13:01:48.150283+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:24.307795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686115+0000","last_change":"2024-09-13T13:01:04.199011+0000","last_active":"2024-09-13T13:01:47.686115+0000","last_peered":"2024-09-13T13:01:47.686115+0000","last_clean":"2024-09-13T13:01:47.686115+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:01:47.686115+0000","last_undegraded":"2024-09-13T13:01:47.686115+0000","last_fullsized":"2024-09-13T13:01:47.686115+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.407716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151447+0000","last_change":"2024-09-13T13:01:02.181597+0000","last_active":"2024-09-13T13:01:48.151447+0000","last_peered":"2024-09-13T13:01:48.151447+0000","last_clean":"2024-09-13T13:01:48.151447+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:01:48.151447+0000","last_undegraded":"2024-09-13T13:01:48.151447+0000","last_fullsized":"2024-09-13T13:01:48.151447+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.285259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:52.240396+0000","last_change":"2024-09-13T13:01:06.216352+0000","last_active":"2024-09-13T13:01:52.240396+0000","last_peered":"2024-09-13T13:01:52.240396+0000","last_clean":"2024-09-13T13:01:52.240396+0000","last_became_active":"2024-09-13T13:01:06.216043+0000","last_became_peered":"2024-09-13T13:01:06.216043+0000","last_unstale":"2024-09-13T13:01:52.240396+0000","last_undegraded":"2024-09-13T13:01:52.240396+0000","last_fullsized":"2024-09-13T13:01:52.240396+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.006398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684721+0000","last_change":"2024-09-13T13:01:08.211109+0000","last_active":"2024-09-13T13:01:47.684721+0000","last_peered":"2024-09-13T13:01:47.684721+0000","last_clean":"2024-09-13T13:01:47.684721+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:01:47.684721+0000","last_undegraded":"2024-09-13T13:01:47.684721+0000","last_fullsized":"2024-09-13T13:01:47.684721+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:34.759653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":79,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685451+0000","last_change":"2024-09-13T13:01:04.207894+0000","last_active":"2024-09-13T13:01:47.685451+0000","last_peered":"2024-09-13T13:01:47.685451+0000","last_clean":"2024-09-13T13:01:47.685451+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:01:47.685451+0000","last_undegraded":"2024-09-13T13:01:47.685451+0000","last_fullsized":"2024-09-13T13:01:47.685451+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.794771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686808+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:01:47.686808+0000","last_peered":"2024-09-13T13:01:47.686808+0000","last_clean":"2024-09-13T13:01:47.686808+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:01:47.686808+0000","last_undegraded":"2024-09-13T13:01:47.686808+0000","last_fullsized":"2024-09-13T13:01:47.686808+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.453481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151656+0000","last_change":"2024-09-13T13:01:06.213926+0000","last_active":"2024-09-13T13:01:48.151656+0000","last_peered":"2024-09-13T13:01:48.151656+0000","last_clean":"2024-09-13T13:01:48.151656+0000","last_became_active":"2024-09-13T13:01:06.213617+0000","last_became_peered":"2024-09-13T13:01:06.213617+0000","last_unstale":"2024-09-13T13:01:48.151656+0000","last_undegraded":"2024-09-13T13:01:48.151656+0000","last_fullsized":"2024-09-13T13:01:48.151656+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.400692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684269+0000","last_change":"2024-09-13T13:01:08.532524+0000","last_active":"2024-09-13T13:01:47.684269+0000","last_peered":"2024-09-13T13:01:47.684269+0000","last_clean":"2024-09-13T13:01:47.684269+0000","last_became_active":"2024-09-13T13:01:08.532145+0000","last_became_peered":"2024-09-13T13:01:08.532145+0000","last_unstale":"2024-09-13T13:01:47.684269+0000","last_undegraded":"2024-09-13T13:01:47.684269+0000","last_fullsized":"2024-09-13T13:01:47.684269+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:16.922652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":104,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:52.240449+0000","last_change":"2024-09-13T13:01:04.197527+0000","last_active":"2024-09-13T13:01:52.240449+0000","last_peered":"2024-09-13T13:01:52.240449+0000","last_clean":"2024-09-13T13:01:52.240449+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:01:52.240449+0000","last_undegraded":"2024-09-13T13:01:52.240449+0000","last_fullsized":"2024-09-13T13:01:52.240449+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.799330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685637+0000","last_change":"2024-09-13T13:01:02.181566+0000","last_active":"2024-09-13T13:01:47.685637+0000","last_peered":"2024-09-13T13:01:47.685637+0000","last_clean":"2024-09-13T13:01:47.685637+0000","last_became_active":"2024-09-13T13:01:02.180760+0000","last_became_peered":"2024-09-13T13:01:02.180760+0000","last_unstale":"2024-09-13T13:01:47.685637+0000","last_undegraded":"2024-09-13T13:01:47.685637+0000","last_fullsized":"2024-09-13T13:01:47.685637+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.355888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150711+0000","last_change":"2024-09-13T13:01:05.199998+0000","last_active":"2024-09-13T13:01:48.150711+0000","last_peered":"2024-09-13T13:01:48.150711+0000","last_clean":"2024-09-13T13:01:48.150711+0000","last_became_active":"2024-09-13T13:01:02.185089+0000","last_became_peered":"2024-09-13T13:01:02.185089+0000","last_unstale":"2024-09-13T13:01:48.150711+0000","last_undegraded":"2024-09-13T13:01:48.150711+0000","last_fullsized":"2024-09-13T13:01:48.150711+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.802534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685181+0000","last_change":"2024-09-13T13:01:06.216206+0000","last_active":"2024-09-13T13:01:47.685181+0000","last_peered":"2024-09-13T13:01:47.685181+0000","last_clean":"2024-09-13T13:01:47.685181+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:01:47.685181+0000","last_undegraded":"2024-09-13T13:01:47.685181+0000","last_fullsized":"2024-09-13T13:01:47.685181+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.187742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996267+0000","last_change":"2024-09-13T13:01:08.203445+0000","last_active":"2024-09-13T13:01:47.996267+0000","last_peered":"2024-09-13T13:01:47.996267+0000","last_clean":"2024-09-13T13:01:47.996267+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:01:47.996267+0000","last_undegraded":"2024-09-13T13:01:47.996267+0000","last_fullsized":"2024-09-13T13:01:47.996267+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.430979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":71,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.687208+0000","last_change":"2024-09-13T13:01:04.192923+0000","last_active":"2024-09-13T13:01:47.687208+0000","last_peered":"2024-09-13T13:01:47.687208+0000","last_clean":"2024-09-13T13:01:47.687208+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:01:47.687208+0000","last_undegraded":"2024-09-13T13:01:47.687208+0000","last_fullsized":"2024-09-13T13:01:47.687208+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.839147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684894+0000","last_change":"2024-09-13T13:01:02.185582+0000","last_active":"2024-09-13T13:01:47.684894+0000","last_peered":"2024-09-13T13:01:47.684894+0000","last_clean":"2024-09-13T13:01:47.684894+0000","last_became_active":"2024-09-13T13:01:02.185263+0000","last_became_peered":"2024-09-13T13:01:02.185263+0000","last_unstale":"2024-09-13T13:01:47.684894+0000","last_undegraded":"2024-09-13T13:01:47.684894+0000","last_fullsized":"2024-09-13T13:01:47.684894+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:28.109915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149563+0000","last_change":"2024-09-13T13:01:05.202103+0000","last_active":"2024-09-13T13:01:48.149563+0000","last_peered":"2024-09-13T13:01:48.149563+0000","last_clean":"2024-09-13T13:01:48.149563+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:01:48.149563+0000","last_undegraded":"2024-09-13T13:01:48.149563+0000","last_fullsized":"2024-09-13T13:01:48.149563+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.268994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150018+0000","last_change":"2024-09-13T13:01:06.205805+0000","last_active":"2024-09-13T13:01:48.150018+0000","last_peered":"2024-09-13T13:01:48.150018+0000","last_clean":"2024-09-13T13:01:48.150018+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:01:48.150018+0000","last_undegraded":"2024-09-13T13:01:48.150018+0000","last_fullsized":"2024-09-13T13:01:48.150018+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.555922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150818+0000","last_change":"2024-09-13T13:01:08.532390+0000","last_active":"2024-09-13T13:01:48.150818+0000","last_peered":"2024-09-13T13:01:48.150818+0000","last_clean":"2024-09-13T13:01:48.150818+0000","last_became_active":"2024-09-13T13:01:08.532087+0000","last_became_peered":"2024-09-13T13:01:08.532087+0000","last_unstale":"2024-09-13T13:01:48.150818+0000","last_undegraded":"2024-09-13T13:01:48.150818+0000","last_fullsized":"2024-09-13T13:01:48.150818+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.390192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685985+0000","last_change":"2024-09-13T13:01:04.193042+0000","last_active":"2024-09-13T13:01:47.685985+0000","last_peered":"2024-09-13T13:01:47.685985+0000","last_clean":"2024-09-13T13:01:47.685985+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:01:47.685985+0000","last_undegraded":"2024-09-13T13:01:47.685985+0000","last_fullsized":"2024-09-13T13:01:47.685985+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.415798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685509+0000","last_change":"2024-09-13T13:01:02.186331+0000","last_active":"2024-09-13T13:01:47.685509+0000","last_peered":"2024-09-13T13:01:47.685509+0000","last_clean":"2024-09-13T13:01:47.685509+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:01:47.685509+0000","last_undegraded":"2024-09-13T13:01:47.685509+0000","last_fullsized":"2024-09-13T13:01:47.685509+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.787015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":106,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:56.506086+0000","last_change":"2024-09-13T13:01:05.200818+0000","last_active":"2024-09-13T13:01:56.506086+0000","last_peered":"2024-09-13T13:01:56.506086+0000","last_clean":"2024-09-13T13:01:56.506086+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:01:56.506086+0000","last_undegraded":"2024-09-13T13:01:56.506086+0000","last_fullsized":"2024-09-13T13:01:56.506086+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.587568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151611+0000","last_change":"2024-09-13T13:01:06.212898+0000","last_active":"2024-09-13T13:01:48.151611+0000","last_peered":"2024-09-13T13:01:48.151611+0000","last_clean":"2024-09-13T13:01:48.151611+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:01:48.151611+0000","last_undegraded":"2024-09-13T13:01:48.151611+0000","last_fullsized":"2024-09-13T13:01:48.151611+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.387930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686408+0000","last_change":"2024-09-13T13:01:08.213728+0000","last_active":"2024-09-13T13:01:47.686408+0000","last_peered":"2024-09-13T13:01:47.686408+0000","last_clean":"2024-09-13T13:01:47.686408+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:01:47.686408+0000","last_undegraded":"2024-09-13T13:01:47.686408+0000","last_fullsized":"2024-09-13T13:01:47.686408+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.763739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":77,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:52.240616+0000","last_change":"2024-09-13T13:01:04.208034+0000","last_active":"2024-09-13T13:01:52.240616+0000","last_peered":"2024-09-13T13:01:52.240616+0000","last_clean":"2024-09-13T13:01:52.240616+0000","last_became_active":"2024-09-13T13:01:04.207653+0000","last_became_peered":"2024-09-13T13:01:04.207653+0000","last_unstale":"2024-09-13T13:01:52.240616+0000","last_undegraded":"2024-09-13T13:01:52.240616+0000","last_fullsized":"2024-09-13T13:01:52.240616+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.433729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995823+0000","last_change":"2024-09-13T13:01:02.180002+0000","last_active":"2024-09-13T13:01:47.995823+0000","last_peered":"2024-09-13T13:01:47.995823+0000","last_clean":"2024-09-13T13:01:47.995823+0000","last_became_active":"2024-09-13T13:01:02.179698+0000","last_became_peered":"2024-09-13T13:01:02.179698+0000","last_unstale":"2024-09-13T13:01:47.995823+0000","last_undegraded":"2024-09-13T13:01:47.995823+0000","last_fullsized":"2024-09-13T13:01:47.995823+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:03.938319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":84,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151302+0000","last_change":"2024-09-13T13:00:53.927016+0000","last_active":"2024-09-13T13:01:48.151302+0000","last_peered":"2024-09-13T13:01:48.151302+0000","last_clean":"2024-09-13T13:01:48.151302+0000","last_became_active":"2024-09-13T13:00:53.922110+0000","last_became_peered":"2024-09-13T13:00:53.922110+0000","last_unstale":"2024-09-13T13:01:48.151302+0000","last_undegraded":"2024-09-13T13:01:48.151302+0000","last_fullsized":"2024-09-13T13:01:48.151302+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-13T13:00:46.618619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:00:46.618619+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.913095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150670+0000","last_change":"2024-09-13T13:01:06.207054+0000","last_active":"2024-09-13T13:01:48.150670+0000","last_peered":"2024-09-13T13:01:48.150670+0000","last_clean":"2024-09-13T13:01:48.150670+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:01:48.150670+0000","last_undegraded":"2024-09-13T13:01:48.150670+0000","last_fullsized":"2024-09-13T13:01:48.150670+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.955181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151924+0000","last_change":"2024-09-13T13:01:08.209367+0000","last_active":"2024-09-13T13:01:48.151924+0000","last_peered":"2024-09-13T13:01:48.151924+0000","last_clean":"2024-09-13T13:01:48.151924+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:01:48.151924+0000","last_undegraded":"2024-09-13T13:01:48.151924+0000","last_fullsized":"2024-09-13T13:01:48.151924+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.418279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685894+0000","last_change":"2024-09-13T13:01:04.196954+0000","last_active":"2024-09-13T13:01:47.685894+0000","last_peered":"2024-09-13T13:01:47.685894+0000","last_clean":"2024-09-13T13:01:47.685894+0000","last_became_active":"2024-09-13T13:01:04.196363+0000","last_became_peered":"2024-09-13T13:01:04.196363+0000","last_unstale":"2024-09-13T13:01:47.685894+0000","last_undegraded":"2024-09-13T13:01:47.685894+0000","last_fullsized":"2024-09-13T13:01:47.685894+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.318889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685831+0000","last_change":"2024-09-13T13:01:02.180296+0000","last_active":"2024-09-13T13:01:47.685831+0000","last_peered":"2024-09-13T13:01:47.685831+0000","last_clean":"2024-09-13T13:01:47.685831+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:01:47.685831+0000","last_undegraded":"2024-09-13T13:01:47.685831+0000","last_fullsized":"2024-09-13T13:01:47.685831+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.041720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151227+0000","last_change":"2024-09-13T13:01:06.209895+0000","last_active":"2024-09-13T13:01:48.151227+0000","last_peered":"2024-09-13T13:01:48.151227+0000","last_clean":"2024-09-13T13:01:48.151227+0000","last_became_active":"2024-09-13T13:01:06.206590+0000","last_became_peered":"2024-09-13T13:01:06.206590+0000","last_unstale":"2024-09-13T13:01:48.151227+0000","last_undegraded":"2024-09-13T13:01:48.151227+0000","last_fullsized":"2024-09-13T13:01:48.151227+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.901892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995757+0000","last_change":"2024-09-13T13:01:08.533475+0000","last_active":"2024-09-13T13:01:47.995757+0000","last_peered":"2024-09-13T13:01:47.995757+0000","last_clean":"2024-09-13T13:01:47.995757+0000","last_became_active":"2024-09-13T13:01:08.533119+0000","last_became_peered":"2024-09-13T13:01:08.533119+0000","last_unstale":"2024-09-13T13:01:47.995757+0000","last_undegraded":"2024-09-13T13:01:47.995757+0000","last_fullsized":"2024-09-13T13:01:47.995757+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.681416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":72,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685709+0000","last_change":"2024-09-13T13:01:04.195092+0000","last_active":"2024-09-13T13:01:47.685709+0000","last_peered":"2024-09-13T13:01:47.685709+0000","last_clean":"2024-09-13T13:01:47.685709+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:01:47.685709+0000","last_undegraded":"2024-09-13T13:01:47.685709+0000","last_fullsized":"2024-09-13T13:01:47.685709+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.170610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152248+0000","last_change":"2024-09-13T13:01:02.181492+0000","last_active":"2024-09-13T13:01:48.152248+0000","last_peered":"2024-09-13T13:01:48.152248+0000","last_clean":"2024-09-13T13:01:48.152248+0000","last_became_active":"2024-09-13T13:01:02.180946+0000","last_became_peered":"2024-09-13T13:01:02.180946+0000","last_unstale":"2024-09-13T13:01:48.152248+0000","last_undegraded":"2024-09-13T13:01:48.152248+0000","last_fullsized":"2024-09-13T13:01:48.152248+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:15.300817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686474+0000","last_change":"2024-09-13T13:01:06.208593+0000","last_active":"2024-09-13T13:01:47.686474+0000","last_peered":"2024-09-13T13:01:47.686474+0000","last_clean":"2024-09-13T13:01:47.686474+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:01:47.686474+0000","last_undegraded":"2024-09-13T13:01:47.686474+0000","last_fullsized":"2024-09-13T13:01:47.686474+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.535525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149895+0000","last_change":"2024-09-13T13:01:08.204029+0000","last_active":"2024-09-13T13:01:48.149895+0000","last_peered":"2024-09-13T13:01:48.149895+0000","last_clean":"2024-09-13T13:01:48.149895+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:01:48.149895+0000","last_undegraded":"2024-09-13T13:01:48.149895+0000","last_fullsized":"2024-09-13T13:01:48.149895+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.251232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685192+0000","last_change":"2024-09-13T13:01:04.189975+0000","last_active":"2024-09-13T13:01:47.685192+0000","last_peered":"2024-09-13T13:01:47.685192+0000","last_clean":"2024-09-13T13:01:47.685192+0000","last_became_active":"2024-09-13T13:01:04.189380+0000","last_became_peered":"2024-09-13T13:01:04.189380+0000","last_unstale":"2024-09-13T13:01:47.685192+0000","last_undegraded":"2024-09-13T13:01:47.685192+0000","last_fullsized":"2024-09-13T13:01:47.685192+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:03.186596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996388+0000","last_change":"2024-09-13T13:01:02.188134+0000","last_active":"2024-09-13T13:01:47.996388+0000","last_peered":"2024-09-13T13:01:47.996388+0000","last_clean":"2024-09-13T13:01:47.996388+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:01:47.996388+0000","last_undegraded":"2024-09-13T13:01:47.996388+0000","last_fullsized":"2024-09-13T13:01:47.996388+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.160057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:52.240396+0000","last_change":"2024-09-13T13:01:06.206335+0000","last_active":"2024-09-13T13:01:52.240396+0000","last_peered":"2024-09-13T13:01:52.240396+0000","last_clean":"2024-09-13T13:01:52.240396+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:01:52.240396+0000","last_undegraded":"2024-09-13T13:01:52.240396+0000","last_fullsized":"2024-09-13T13:01:52.240396+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.752225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685420+0000","last_change":"2024-09-13T13:01:08.213034+0000","last_active":"2024-09-13T13:01:47.685420+0000","last_peered":"2024-09-13T13:01:47.685420+0000","last_clean":"2024-09-13T13:01:47.685420+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:01:47.685420+0000","last_undegraded":"2024-09-13T13:01:47.685420+0000","last_fullsized":"2024-09-13T13:01:47.685420+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.453460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685311+0000","last_change":"2024-09-13T13:01:04.192921+0000","last_active":"2024-09-13T13:01:47.685311+0000","last_peered":"2024-09-13T13:01:47.685311+0000","last_clean":"2024-09-13T13:01:47.685311+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:01:47.685311+0000","last_undegraded":"2024-09-13T13:01:47.685311+0000","last_fullsized":"2024-09-13T13:01:47.685311+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.518566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150300+0000","last_change":"2024-09-13T13:01:02.180807+0000","last_active":"2024-09-13T13:01:48.150300+0000","last_peered":"2024-09-13T13:01:48.150300+0000","last_clean":"2024-09-13T13:01:48.150300+0000","last_became_active":"2024-09-13T13:01:02.180541+0000","last_became_peered":"2024-09-13T13:01:02.180541+0000","last_unstale":"2024-09-13T13:01:48.150300+0000","last_undegraded":"2024-09-13T13:01:48.150300+0000","last_fullsized":"2024-09-13T13:01:48.150300+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:28.541518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149485+0000","last_change":"2024-09-13T13:01:06.207416+0000","last_active":"2024-09-13T13:01:48.149485+0000","last_peered":"2024-09-13T13:01:48.149485+0000","last_clean":"2024-09-13T13:01:48.149485+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:01:48.149485+0000","last_undegraded":"2024-09-13T13:01:48.149485+0000","last_fullsized":"2024-09-13T13:01:48.149485+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.336110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685339+0000","last_change":"2024-09-13T13:01:08.211229+0000","last_active":"2024-09-13T13:01:47.685339+0000","last_peered":"2024-09-13T13:01:47.685339+0000","last_clean":"2024-09-13T13:01:47.685339+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:01:47.685339+0000","last_undegraded":"2024-09-13T13:01:47.685339+0000","last_fullsized":"2024-09-13T13:01:47.685339+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:35.720723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":74,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151927+0000","last_change":"2024-09-13T13:01:04.208073+0000","last_active":"2024-09-13T13:01:48.151927+0000","last_peered":"2024-09-13T13:01:48.151927+0000","last_clean":"2024-09-13T13:01:48.151927+0000","last_became_active":"2024-09-13T13:01:04.207702+0000","last_became_peered":"2024-09-13T13:01:04.207702+0000","last_unstale":"2024-09-13T13:01:48.151927+0000","last_undegraded":"2024-09-13T13:01:48.151927+0000","last_fullsized":"2024-09-13T13:01:48.151927+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.215067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152342+0000","last_change":"2024-09-13T13:01:02.178687+0000","last_active":"2024-09-13T13:01:48.152342+0000","last_peered":"2024-09-13T13:01:48.152342+0000","last_clean":"2024-09-13T13:01:48.152342+0000","last_became_active":"2024-09-13T13:01:02.178317+0000","last_became_peered":"2024-09-13T13:01:02.178317+0000","last_unstale":"2024-09-13T13:01:48.152342+0000","last_undegraded":"2024-09-13T13:01:48.152342+0000","last_fullsized":"2024-09-13T13:01:48.152342+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.914639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149751+0000","last_change":"2024-09-13T13:01:06.203813+0000","last_active":"2024-09-13T13:01:48.149751+0000","last_peered":"2024-09-13T13:01:48.149751+0000","last_clean":"2024-09-13T13:01:48.149751+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:01:48.149751+0000","last_undegraded":"2024-09-13T13:01:48.149751+0000","last_fullsized":"2024-09-13T13:01:48.149751+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:31.552430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151220+0000","last_change":"2024-09-13T13:01:08.212073+0000","last_active":"2024-09-13T13:01:48.151220+0000","last_peered":"2024-09-13T13:01:48.151220+0000","last_clean":"2024-09-13T13:01:48.151220+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:01:48.151220+0000","last_undegraded":"2024-09-13T13:01:48.151220+0000","last_fullsized":"2024-09-13T13:01:48.151220+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:35.915325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685079+0000","last_change":"2024-09-13T13:01:04.196002+0000","last_active":"2024-09-13T13:01:47.685079+0000","last_peered":"2024-09-13T13:01:47.685079+0000","last_clean":"2024-09-13T13:01:47.685079+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:01:47.685079+0000","last_undegraded":"2024-09-13T13:01:47.685079+0000","last_fullsized":"2024-09-13T13:01:47.685079+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.222887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152092+0000","last_change":"2024-09-13T13:01:02.187229+0000","last_active":"2024-09-13T13:01:48.152092+0000","last_peered":"2024-09-13T13:01:48.152092+0000","last_clean":"2024-09-13T13:01:48.152092+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:01:48.152092+0000","last_undegraded":"2024-09-13T13:01:48.152092+0000","last_fullsized":"2024-09-13T13:01:48.152092+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:02.438861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149432+0000","last_change":"2024-09-13T13:01:06.211172+0000","last_active":"2024-09-13T13:01:48.149432+0000","last_peered":"2024-09-13T13:01:48.149432+0000","last_clean":"2024-09-13T13:01:48.149432+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:01:48.149432+0000","last_undegraded":"2024-09-13T13:01:48.149432+0000","last_fullsized":"2024-09-13T13:01:48.149432+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.354291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996008+0000","last_change":"2024-09-13T13:01:08.211530+0000","last_active":"2024-09-13T13:01:47.996008+0000","last_peered":"2024-09-13T13:01:47.996008+0000","last_clean":"2024-09-13T13:01:47.996008+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:01:47.996008+0000","last_undegraded":"2024-09-13T13:01:47.996008+0000","last_fullsized":"2024-09-13T13:01:47.996008+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:11.539593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":68,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149423+0000","last_change":"2024-09-13T13:01:04.209162+0000","last_active":"2024-09-13T13:01:48.149423+0000","last_peered":"2024-09-13T13:01:48.149423+0000","last_clean":"2024-09-13T13:01:48.149423+0000","last_became_active":"2024-09-13T13:01:04.208731+0000","last_became_peered":"2024-09-13T13:01:04.208731+0000","last_unstale":"2024-09-13T13:01:48.149423+0000","last_undegraded":"2024-09-13T13:01:48.149423+0000","last_fullsized":"2024-09-13T13:01:48.149423+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.356204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152226+0000","last_change":"2024-09-13T13:01:02.183335+0000","last_active":"2024-09-13T13:01:48.152226+0000","last_peered":"2024-09-13T13:01:48.152226+0000","last_clean":"2024-09-13T13:01:48.152226+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:01:48.152226+0000","last_undegraded":"2024-09-13T13:01:48.152226+0000","last_fullsized":"2024-09-13T13:01:48.152226+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:20.269754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:52.240572+0000","last_change":"2024-09-13T13:01:06.210716+0000","last_active":"2024-09-13T13:01:52.240572+0000","last_peered":"2024-09-13T13:01:52.240572+0000","last_clean":"2024-09-13T13:01:52.240572+0000","last_became_active":"2024-09-13T13:01:06.210441+0000","last_became_peered":"2024-09-13T13:01:06.210441+0000","last_unstale":"2024-09-13T13:01:52.240572+0000","last_undegraded":"2024-09-13T13:01:52.240572+0000","last_fullsized":"2024-09-13T13:01:52.240572+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:22.206296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149578+0000","last_change":"2024-09-13T13:01:08.532185+0000","last_active":"2024-09-13T13:01:48.149578+0000","last_peered":"2024-09-13T13:01:48.149578+0000","last_clean":"2024-09-13T13:01:48.149578+0000","last_became_active":"2024-09-13T13:01:08.531895+0000","last_became_peered":"2024-09-13T13:01:08.531895+0000","last_unstale":"2024-09-13T13:01:48.149578+0000","last_undegraded":"2024-09-13T13:01:48.149578+0000","last_fullsized":"2024-09-13T13:01:48.149578+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.747075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150376+0000","last_change":"2024-09-13T13:01:02.179217+0000","last_active":"2024-09-13T13:01:48.150376+0000","last_peered":"2024-09-13T13:01:48.150376+0000","last_clean":"2024-09-13T13:01:48.150376+0000","last_became_active":"2024-09-13T13:01:02.178934+0000","last_became_peered":"2024-09-13T13:01:02.178934+0000","last_unstale":"2024-09-13T13:01:48.150376+0000","last_undegraded":"2024-09-13T13:01:48.150376+0000","last_fullsized":"2024-09-13T13:01:48.150376+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.675331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996611+0000","last_change":"2024-09-13T13:01:04.209208+0000","last_active":"2024-09-13T13:01:47.996611+0000","last_peered":"2024-09-13T13:01:47.996611+0000","last_clean":"2024-09-13T13:01:47.996611+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:01:47.996611+0000","last_undegraded":"2024-09-13T13:01:47.996611+0000","last_fullsized":"2024-09-13T13:01:47.996611+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.286499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150114+0000","last_change":"2024-09-13T13:01:06.208468+0000","last_active":"2024-09-13T13:01:48.150114+0000","last_peered":"2024-09-13T13:01:48.150114+0000","last_clean":"2024-09-13T13:01:48.150114+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:01:48.150114+0000","last_undegraded":"2024-09-13T13:01:48.150114+0000","last_fullsized":"2024-09-13T13:01:48.150114+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.411667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150544+0000","last_change":"2024-09-13T13:01:08.532411+0000","last_active":"2024-09-13T13:01:48.150544+0000","last_peered":"2024-09-13T13:01:48.150544+0000","last_clean":"2024-09-13T13:01:48.150544+0000","last_became_active":"2024-09-13T13:01:08.532178+0000","last_became_peered":"2024-09-13T13:01:08.532178+0000","last_unstale":"2024-09-13T13:01:48.150544+0000","last_undegraded":"2024-09-13T13:01:48.150544+0000","last_fullsized":"2024-09-13T13:01:48.150544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:15.835392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686003+0000","last_change":"2024-09-13T13:01:04.208866+0000","last_active":"2024-09-13T13:01:47.686003+0000","last_peered":"2024-09-13T13:01:47.686003+0000","last_clean":"2024-09-13T13:01:47.686003+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:01:47.686003+0000","last_undegraded":"2024-09-13T13:01:47.686003+0000","last_fullsized":"2024-09-13T13:01:47.686003+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.015284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150370+0000","last_change":"2024-09-13T13:01:02.181825+0000","last_active":"2024-09-13T13:01:48.150370+0000","last_peered":"2024-09-13T13:01:48.150370+0000","last_clean":"2024-09-13T13:01:48.150370+0000","last_became_active":"2024-09-13T13:01:02.181548+0000","last_became_peered":"2024-09-13T13:01:02.181548+0000","last_unstale":"2024-09-13T13:01:48.150370+0000","last_undegraded":"2024-09-13T13:01:48.150370+0000","last_fullsized":"2024-09-13T13:01:48.150370+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:13.710586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996537+0000","last_change":"2024-09-13T13:01:06.210328+0000","last_active":"2024-09-13T13:01:47.996537+0000","last_peered":"2024-09-13T13:01:47.996537+0000","last_clean":"2024-09-13T13:01:47.996537+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:01:47.996537+0000","last_undegraded":"2024-09-13T13:01:47.996537+0000","last_fullsized":"2024-09-13T13:01:47.996537+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.781482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149780+0000","last_change":"2024-09-13T13:01:08.205690+0000","last_active":"2024-09-13T13:01:48.149780+0000","last_peered":"2024-09-13T13:01:48.149780+0000","last_clean":"2024-09-13T13:01:48.149780+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:01:48.149780+0000","last_undegraded":"2024-09-13T13:01:48.149780+0000","last_fullsized":"2024-09-13T13:01:48.149780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:14.215451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151154+0000","last_change":"2024-09-13T13:01:04.208351+0000","last_active":"2024-09-13T13:01:48.151154+0000","last_peered":"2024-09-13T13:01:48.151154+0000","last_clean":"2024-09-13T13:01:48.151154+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:01:48.151154+0000","last_undegraded":"2024-09-13T13:01:48.151154+0000","last_fullsized":"2024-09-13T13:01:48.151154+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.268797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685047+0000","last_change":"2024-09-13T13:01:02.185440+0000","last_active":"2024-09-13T13:01:47.685047+0000","last_peered":"2024-09-13T13:01:47.685047+0000","last_clean":"2024-09-13T13:01:47.685047+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:01:47.685047+0000","last_undegraded":"2024-09-13T13:01:47.685047+0000","last_fullsized":"2024-09-13T13:01:47.685047+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.667174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:52.240593+0000","last_change":"2024-09-13T13:01:06.210251+0000","last_active":"2024-09-13T13:01:52.240593+0000","last_peered":"2024-09-13T13:01:52.240593+0000","last_clean":"2024-09-13T13:01:52.240593+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:01:52.240593+0000","last_undegraded":"2024-09-13T13:01:52.240593+0000","last_fullsized":"2024-09-13T13:01:52.240593+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.308767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685613+0000","last_change":"2024-09-13T13:01:08.206113+0000","last_active":"2024-09-13T13:01:47.685613+0000","last_peered":"2024-09-13T13:01:47.685613+0000","last_clean":"2024-09-13T13:01:47.685613+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:01:47.685613+0000","last_undegraded":"2024-09-13T13:01:47.685613+0000","last_fullsized":"2024-09-13T13:01:47.685613+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:10.315612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996401+0000","last_change":"2024-09-13T13:01:04.208466+0000","last_active":"2024-09-13T13:01:47.996401+0000","last_peered":"2024-09-13T13:01:47.996401+0000","last_clean":"2024-09-13T13:01:47.996401+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:01:47.996401+0000","last_undegraded":"2024-09-13T13:01:47.996401+0000","last_fullsized":"2024-09-13T13:01:47.996401+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.373274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.151566+0000","last_change":"2024-09-13T13:01:02.180706+0000","last_active":"2024-09-13T13:01:48.151566+0000","last_peered":"2024-09-13T13:01:48.151566+0000","last_clean":"2024-09-13T13:01:48.151566+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:01:48.151566+0000","last_undegraded":"2024-09-13T13:01:48.151566+0000","last_fullsized":"2024-09-13T13:01:48.151566+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.494760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:52.240605+0000","last_change":"2024-09-13T13:01:06.206440+0000","last_active":"2024-09-13T13:01:52.240605+0000","last_peered":"2024-09-13T13:01:52.240605+0000","last_clean":"2024-09-13T13:01:52.240605+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:01:52.240605+0000","last_undegraded":"2024-09-13T13:01:52.240605+0000","last_fullsized":"2024-09-13T13:01:52.240605+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.104127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685534+0000","last_change":"2024-09-13T13:01:08.215072+0000","last_active":"2024-09-13T13:01:47.685534+0000","last_peered":"2024-09-13T13:01:47.685534+0000","last_clean":"2024-09-13T13:01:47.685534+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:01:47.685534+0000","last_undegraded":"2024-09-13T13:01:47.685534+0000","last_fullsized":"2024-09-13T13:01:47.685534+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.276961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685087+0000","last_change":"2024-09-13T13:01:04.197213+0000","last_active":"2024-09-13T13:01:47.685087+0000","last_peered":"2024-09-13T13:01:47.685087+0000","last_clean":"2024-09-13T13:01:47.685087+0000","last_became_active":"2024-09-13T13:01:04.196857+0000","last_became_peered":"2024-09-13T13:01:04.196857+0000","last_unstale":"2024-09-13T13:01:47.685087+0000","last_undegraded":"2024-09-13T13:01:47.685087+0000","last_fullsized":"2024-09-13T13:01:47.685087+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.316586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685397+0000","last_change":"2024-09-13T13:01:02.181939+0000","last_active":"2024-09-13T13:01:47.685397+0000","last_peered":"2024-09-13T13:01:47.685397+0000","last_clean":"2024-09-13T13:01:47.685397+0000","last_became_active":"2024-09-13T13:01:02.180817+0000","last_became_peered":"2024-09-13T13:01:02.180817+0000","last_unstale":"2024-09-13T13:01:47.685397+0000","last_undegraded":"2024-09-13T13:01:47.685397+0000","last_fullsized":"2024-09-13T13:01:47.685397+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.328756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685953+0000","last_change":"2024-09-13T13:01:06.209945+0000","last_active":"2024-09-13T13:01:47.685953+0000","last_peered":"2024-09-13T13:01:47.685953+0000","last_clean":"2024-09-13T13:01:47.685953+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:01:47.685953+0000","last_undegraded":"2024-09-13T13:01:47.685953+0000","last_fullsized":"2024-09-13T13:01:47.685953+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.087308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995515+0000","last_change":"2024-09-13T13:01:08.205355+0000","last_active":"2024-09-13T13:01:47.995515+0000","last_peered":"2024-09-13T13:01:47.995515+0000","last_clean":"2024-09-13T13:01:47.995515+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:01:47.995515+0000","last_undegraded":"2024-09-13T13:01:47.995515+0000","last_fullsized":"2024-09-13T13:01:47.995515+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:21.257400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686273+0000","last_change":"2024-09-13T13:01:04.196437+0000","last_active":"2024-09-13T13:01:47.686273+0000","last_peered":"2024-09-13T13:01:47.686273+0000","last_clean":"2024-09-13T13:01:47.686273+0000","last_became_active":"2024-09-13T13:01:04.196045+0000","last_became_peered":"2024-09-13T13:01:04.196045+0000","last_unstale":"2024-09-13T13:01:47.686273+0000","last_undegraded":"2024-09-13T13:01:47.686273+0000","last_fullsized":"2024-09-13T13:01:47.686273+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:24.629192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152201+0000","last_change":"2024-09-13T13:01:02.182107+0000","last_active":"2024-09-13T13:01:48.152201+0000","last_peered":"2024-09-13T13:01:48.152201+0000","last_clean":"2024-09-13T13:01:48.152201+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:01:48.152201+0000","last_undegraded":"2024-09-13T13:01:48.152201+0000","last_fullsized":"2024-09-13T13:01:48.152201+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:18.840261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996416+0000","last_change":"2024-09-13T13:01:06.211993+0000","last_active":"2024-09-13T13:01:47.996416+0000","last_peered":"2024-09-13T13:01:47.996416+0000","last_clean":"2024-09-13T13:01:47.996416+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:01:47.996416+0000","last_undegraded":"2024-09-13T13:01:47.996416+0000","last_fullsized":"2024-09-13T13:01:47.996416+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:29.375348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.684160+0000","last_change":"2024-09-13T13:01:08.210271+0000","last_active":"2024-09-13T13:01:47.684160+0000","last_peered":"2024-09-13T13:01:47.684160+0000","last_clean":"2024-09-13T13:01:47.684160+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:01:47.684160+0000","last_undegraded":"2024-09-13T13:01:47.684160+0000","last_fullsized":"2024-09-13T13:01:47.684160+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:19.691464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995642+0000","last_change":"2024-09-13T13:01:04.208421+0000","last_active":"2024-09-13T13:01:47.995642+0000","last_peered":"2024-09-13T13:01:47.995642+0000","last_clean":"2024-09-13T13:01:47.995642+0000","last_became_active":"2024-09-13T13:01:04.207841+0000","last_became_peered":"2024-09-13T13:01:04.207841+0000","last_unstale":"2024-09-13T13:01:47.995642+0000","last_undegraded":"2024-09-13T13:01:47.995642+0000","last_fullsized":"2024-09-13T13:01:47.995642+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:07.504022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150242+0000","last_change":"2024-09-13T13:01:02.185420+0000","last_active":"2024-09-13T13:01:48.150242+0000","last_peered":"2024-09-13T13:01:48.150242+0000","last_clean":"2024-09-13T13:01:48.150242+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:01:48.150242+0000","last_undegraded":"2024-09-13T13:01:48.150242+0000","last_fullsized":"2024-09-13T13:01:48.150242+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:30.931940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149554+0000","last_change":"2024-09-13T13:01:06.208696+0000","last_active":"2024-09-13T13:01:48.149554+0000","last_peered":"2024-09-13T13:01:48.149554+0000","last_clean":"2024-09-13T13:01:48.149554+0000","last_became_active":"2024-09-13T13:01:06.208426+0000","last_became_peered":"2024-09-13T13:01:06.208426+0000","last_unstale":"2024-09-13T13:01:48.149554+0000","last_undegraded":"2024-09-13T13:01:48.149554+0000","last_fullsized":"2024-09-13T13:01:48.149554+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:27.767606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.996259+0000","last_change":"2024-09-13T13:01:08.535233+0000","last_active":"2024-09-13T13:01:47.996259+0000","last_peered":"2024-09-13T13:01:47.996259+0000","last_clean":"2024-09-13T13:01:47.996259+0000","last_became_active":"2024-09-13T13:01:08.534673+0000","last_became_peered":"2024-09-13T13:01:08.534673+0000","last_unstale":"2024-09-13T13:01:47.996259+0000","last_undegraded":"2024-09-13T13:01:47.996259+0000","last_fullsized":"2024-09-13T13:01:47.996259+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:33.330172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":49,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995528+0000","last_change":"2024-09-13T13:01:04.195434+0000","last_active":"2024-09-13T13:01:47.995528+0000","last_peered":"2024-09-13T13:01:47.995528+0000","last_clean":"2024-09-13T13:01:47.995528+0000","last_became_active":"2024-09-13T13:01:04.195125+0000","last_became_peered":"2024-09-13T13:01:04.195125+0000","last_unstale":"2024-09-13T13:01:47.995528+0000","last_undegraded":"2024-09-13T13:01:47.995528+0000","last_fullsized":"2024-09-13T13:01:47.995528+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:17.430862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685638+0000","last_change":"2024-09-13T13:01:02.186003+0000","last_active":"2024-09-13T13:01:47.685638+0000","last_peered":"2024-09-13T13:01:47.685638+0000","last_clean":"2024-09-13T13:01:47.685638+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:01:47.685638+0000","last_undegraded":"2024-09-13T13:01:47.685638+0000","last_fullsized":"2024-09-13T13:01:47.685638+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:06.400129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.150966+0000","last_change":"2024-09-13T13:01:06.211514+0000","last_active":"2024-09-13T13:01:48.150966+0000","last_peered":"2024-09-13T13:01:48.150966+0000","last_clean":"2024-09-13T13:01:48.150966+0000","last_became_active":"2024-09-13T13:01:06.211243+0000","last_became_peered":"2024-09-13T13:01:06.211243+0000","last_unstale":"2024-09-13T13:01:48.150966+0000","last_undegraded":"2024-09-13T13:01:48.150966+0000","last_fullsized":"2024-09-13T13:01:48.150966+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:16.563704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.149886+0000","last_change":"2024-09-13T13:01:08.210205+0000","last_active":"2024-09-13T13:01:48.149886+0000","last_peered":"2024-09-13T13:01:48.149886+0000","last_clean":"2024-09-13T13:01:48.149886+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:01:48.149886+0000","last_undegraded":"2024-09-13T13:01:48.149886+0000","last_fullsized":"2024-09-13T13:01:48.149886+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:26.043981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.686712+0000","last_change":"2024-09-13T13:01:08.214778+0000","last_active":"2024-09-13T13:01:47.686712+0000","last_peered":"2024-09-13T13:01:47.686712+0000","last_clean":"2024-09-13T13:01:47.686712+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:01:47.686712+0000","last_undegraded":"2024-09-13T13:01:47.686712+0000","last_fullsized":"2024-09-13T13:01:47.686712+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:32.480474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.995775+0000","last_change":"2024-09-13T13:01:02.187888+0000","last_active":"2024-09-13T13:01:47.995775+0000","last_peered":"2024-09-13T13:01:47.995775+0000","last_clean":"2024-09-13T13:01:47.995775+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:01:47.995775+0000","last_undegraded":"2024-09-13T13:01:47.995775+0000","last_fullsized":"2024-09-13T13:01:47.995775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:23.807870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:48.152523+0000","last_change":"2024-09-13T13:01:04.195160+0000","last_active":"2024-09-13T13:01:48.152523+0000","last_peered":"2024-09-13T13:01:48.152523+0000","last_clean":"2024-09-13T13:01:48.152523+0000","last_became_active":"2024-09-13T13:01:04.194855+0000","last_became_peered":"2024-09-13T13:01:04.194855+0000","last_unstale":"2024-09-13T13:01:48.152523+0000","last_undegraded":"2024-09-13T13:01:48.152523+0000","last_fullsized":"2024-09-13T13:01:48.152523+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:20.224127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-13T13:01:47.685740+0000","last_change":"2024-09-13T13:01:06.211596+0000","last_active":"2024-09-13T13:01:47.685740+0000","last_peered":"2024-09-13T13:01:47.685740+0000","last_clean":"2024-09-13T13:01:47.685740+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:01:47.685740+0000","last_undegraded":"2024-09-13T13:01:47.685740+0000","last_fullsized":"2024-09-13T13:01:47.685740+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:02:12.427559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201294,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28384,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716768,"statfs":{"total":95995035648,"available":95965970432,"internally_reserved":0,"allocated":1388544,"data_stored":909176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430228,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1372160,"data_stored":907547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691866,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31952,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713200,"statfs":{"total":95995035648,"available":95962316800,"internally_reserved":0,"allocated":782336,"data_stored":317538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920800,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":806912,"data_stored":317784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117031,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713172,"statfs":{"total":95995035648,"available":95962288128,"internally_reserved":0,"allocated":811008,"data_stored":318614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27780,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717372,"statfs":{"total":95995035648,"available":95966588928,"internally_reserved":0,"allocated":765952,"data_stored":316878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":835584,"data_stored":318515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32548,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712604,"statfs":{"total":95995035648,"available":95961706496,"internally_reserved":0,"allocated":1392640,"data_stored":908885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:01:58.645 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T13:01:58.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd dump 2024-09-13T13:01:58.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:58 smithi167 ceph-mon[33178]: from='client.34688 v1:172.21.15.52:0/1928990499' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:01:58.912 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:01:59.541 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 79 2024-09-13T13:01:59.541 INFO:teuthology.orchestra.run.smithi052.stdout:fsid 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T13:01:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:created 2024-09-13T12:55:27.893882+0000 2024-09-13T13:01:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:modified 2024-09-13T13:01:58.689534+0000 2024-09-13T13:01:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-13T13:01:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:crush_version 18 2024-09-13T13:01:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:full_ratio 0.95 2024-09-13T13:01:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:backfillfull_ratio 0.9 2024-09-13T13:01:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:nearfull_ratio 0.85 2024-09-13T13:01:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:require_min_compat_client luminous 2024-09-13T13:01:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:min_compat_client jewel 2024-09-13T13:01:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:require_osd_release squid 2024-09-13T13:01:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:stretch_mode_enabled false 2024-09-13T13:01:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 63 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 61 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 63 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 65 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 67 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:max_osd 8 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0 up in weight 1 up_from 8 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.52:6801/542298937 v1:172.21.15.52:6802/542298937 exists,up d52dca1a-94c1-4e9e-93f5-687c038cd7c7 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1 up in weight 1 up_from 13 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.52:6805/26289431 v1:172.21.15.52:6806/26289431 exists,up b40a3574-3c6f-4810-a651-8d64aee6801b 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2 up in weight 1 up_from 18 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.52:6809/3960240393 v1:172.21.15.52:6810/3960240393 exists,up 1e8e49a0-ccf8-4b7f-839b-104b64601408 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3 up in weight 1 up_from 27 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.52:6813/2025522885 v1:172.21.15.52:6814/2025522885 exists,up 9f6f1d22-be68-4d86-9d4b-86fd9e87d141 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4 up in weight 1 up_from 33 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6800/2137138858 v1:172.21.15.167:6801/2137138858 exists,up 1326d850-caf4-484e-90a1-052f54f9f306 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5 up in weight 1 up_from 40 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6804/2906884337 v1:172.21.15.167:6805/2906884337 exists,up ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6 up in weight 1 up_from 48 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6808/756383534 v1:172.21.15.167:6809/756383534 exists,up 2471907d-fcbd-44e0-9d29-54151ada9df0 2024-09-13T13:01:59.543 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7 up in weight 1 up_from 55 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6812/403155155 v1:172.21.15.167:6813/403155155 exists,up 97d2c6d7-7989-4c11-91e0-32ee87e76c6d 2024-09-13T13:01:59.544 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/2104786675 expires 2024-09-14T13:01:31.849627+0000 2024-09-13T13:01:59.544 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:6800/910649698 expires 2024-09-14T12:55:49.232470+0000 2024-09-13T13:01:59.544 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/1471436716 expires 2024-09-14T12:55:49.232470+0000 2024-09-13T13:01:59.544 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/4151877300 expires 2024-09-14T12:55:49.232470+0000 2024-09-13T13:01:59.544 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/3080924817 expires 2024-09-14T12:56:14.802531+0000 2024-09-13T13:01:59.544 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/4248579079 expires 2024-09-14T12:56:14.802531+0000 2024-09-13T13:01:59.544 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/455730280 expires 2024-09-14T13:01:31.849627+0000 2024-09-13T13:01:59.544 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/1887735845 expires 2024-09-14T12:55:49.232470+0000 2024-09-13T13:01:59.544 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:6800/1437783219 expires 2024-09-14T12:56:14.802531+0000 2024-09-13T13:01:59.544 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:6800/326637643 expires 2024-09-14T13:01:31.849627+0000 2024-09-13T13:01:59.544 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/991228238 expires 2024-09-14T13:01:31.849627+0000 2024-09-13T13:01:59.544 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/3415403690 expires 2024-09-14T13:01:31.849627+0000 2024-09-13T13:01:59.544 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/1933060029 expires 2024-09-14T12:56:14.802531+0000 2024-09-13T13:01:59.545 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/1009267179 expires 2024-09-14T13:01:31.849627+0000 2024-09-13T13:01:59.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:59 smithi052 ceph-mon[29471]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T13:01:59.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:59 smithi052 ceph-mon[29471]: from='client.14817 v1:172.21.15.52:0/3303026828' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:01:59.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:59 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:01:59.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:59 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:01:59.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:01:59 smithi052 ceph-mon[29471]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T13:01:59.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:59 smithi052 ceph-mon[36682]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T13:01:59.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:59 smithi052 ceph-mon[36682]: from='client.14817 v1:172.21.15.52:0/3303026828' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:01:59.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:59 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:01:59.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:59 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:01:59.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:01:59 smithi052 ceph-mon[36682]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T13:01:59.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:59 smithi167 ceph-mon[33178]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T13:01:59.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:59 smithi167 ceph-mon[33178]: from='client.14817 v1:172.21.15.52:0/3303026828' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:01:59.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:59 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:01:59.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:59 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:01:59.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:01:59 smithi167 ceph-mon[33178]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T13:02:00.228 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.52:6801/542298937 v1:172.21.15.52:6802/542298937 exists,up d52dca1a-94c1-4e9e-93f5-687c038cd7c7', 'osd.1 up in weight 1 up_from 13 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.52:6805/26289431 v1:172.21.15.52:6806/26289431 exists,up b40a3574-3c6f-4810-a651-8d64aee6801b', 'osd.2 up in weight 1 up_from 18 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.52:6809/3960240393 v1:172.21.15.52:6810/3960240393 exists,up 1e8e49a0-ccf8-4b7f-839b-104b64601408', 'osd.3 up in weight 1 up_from 27 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.52:6813/2025522885 v1:172.21.15.52:6814/2025522885 exists,up 9f6f1d22-be68-4d86-9d4b-86fd9e87d141', 'osd.4 up in weight 1 up_from 33 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6800/2137138858 v1:172.21.15.167:6801/2137138858 exists,up 1326d850-caf4-484e-90a1-052f54f9f306', 'osd.5 up in weight 1 up_from 40 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6804/2906884337 v1:172.21.15.167:6805/2906884337 exists,up ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2', 'osd.6 up in weight 1 up_from 48 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6808/756383534 v1:172.21.15.167:6809/756383534 exists,up 2471907d-fcbd-44e0-9d29-54151ada9df0', 'osd.7 up in weight 1 up_from 55 up_thru 65 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6812/403155155 v1:172.21.15.167:6813/403155155 exists,up 97d2c6d7-7989-4c11-91e0-32ee87e76c6d'] 2024-09-13T13:02:00.228 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon mon.a config show 2024-09-13T13:02:00.488 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:00.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:00 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/432346215' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T13:02:00.849 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:00 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/432346215' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T13:02:00.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:00 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/432346215' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T13:02:00.947 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:02:00.947 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mon.a", 2024-09-13T13:02:00.947 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster": "ceph", 2024-09-13T13:02:00.947 INFO:teuthology.orchestra.run.smithi052.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-13T13:02:00.948 INFO:teuthology.orchestra.run.smithi052.stdout: "admin_socket_mode": "", 2024-09-13T13:02:00.948 INFO:teuthology.orchestra.run.smithi052.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-13T13:02:00.948 INFO:teuthology.orchestra.run.smithi052.stdout: "auth_client_required": "cephx, none", 2024-09-13T13:02:00.948 INFO:teuthology.orchestra.run.smithi052.stdout: "auth_cluster_required": "cephx", 2024-09-13T13:02:00.948 INFO:teuthology.orchestra.run.smithi052.stdout: "auth_debug": "true", 2024-09-13T13:02:00.948 INFO:teuthology.orchestra.run.smithi052.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-13T13:02:00.948 INFO:teuthology.orchestra.run.smithi052.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-13T13:02:00.948 INFO:teuthology.orchestra.run.smithi052.stdout: "auth_service_required": "cephx", 2024-09-13T13:02:00.948 INFO:teuthology.orchestra.run.smithi052.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-13T13:02:00.948 INFO:teuthology.orchestra.run.smithi052.stdout: "auth_supported": "", 2024-09-13T13:02:00.948 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_aio": "true", 2024-09-13T13:02:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-13T13:02:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_aio_poll_ms": "250", 2024-09-13T13:02:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_aio_reap_max": "16", 2024-09-13T13:02:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_async_discard_threads": "0", 2024-09-13T13:02:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_block_size": "4096", 2024-09-13T13:02:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_debug_aio": "false", 2024-09-13T13:02:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-13T13:02:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-13T13:02:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_debug_inflight_ios": "false", 2024-09-13T13:02:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_enable_discard": "false", 2024-09-13T13:02:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_flock_retry": "3", 2024-09-13T13:02:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-13T13:02:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_inject_crash": "0", 2024-09-13T13:02:00.950 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-13T13:02:00.950 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_ioring": "false", 2024-09-13T13:02:00.950 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_ioring_hipri": "false", 2024-09-13T13:02:00.950 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-13T13:02:00.950 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-13T13:02:00.950 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-13T13:02:00.950 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-13T13:02:00.950 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-13T13:02:00.950 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-13T13:02:00.950 INFO:teuthology.orchestra.run.smithi052.stdout: "bdev_type": "", 2024-09-13T13:02:00.950 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_alloc_size": "1048576", 2024-09-13T13:02:00.950 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_allocator": "hybrid", 2024-09-13T13:02:00.951 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_buffered_io": "true", 2024-09-13T13:02:00.951 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_check_for_zeros": "false", 2024-09-13T13:02:00.951 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-13T13:02:00.951 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-13T13:02:00.951 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_compact_log_sync": "false", 2024-09-13T13:02:00.951 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-13T13:02:00.951 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-13T13:02:00.951 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-13T13:02:00.951 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-13T13:02:00.951 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_max_log_runway": "4194304", 2024-09-13T13:02:00.951 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_max_prefetch": "1048576", 2024-09-13T13:02:00.951 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_min_flush_size": "524288", 2024-09-13T13:02:00.951 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_min_log_runway": "1048576", 2024-09-13T13:02:00.952 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_replay_recovery": "false", 2024-09-13T13:02:00.952 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-13T13:02:00.952 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-13T13:02:00.952 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_sync_write": "false", 2024-09-13T13:02:00.952 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-13T13:02:00.952 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-13T13:02:00.952 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-13T13:02:00.952 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_allocation_from_file": "true", 2024-09-13T13:02:00.952 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_allocator": "hybrid", 2024-09-13T13:02:00.952 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_async_db_compaction": "true", 2024-09-13T13:02:00.952 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-13T13:02:00.952 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-13T13:02:00.953 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-13T13:02:00.953 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-13T13:02:00.953 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_bdev_label_multi": "true", 2024-09-13T13:02:00.953 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-13T13:02:00.953 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-13T13:02:00.953 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-13T13:02:00.953 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-13T13:02:00.953 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-13T13:02:00.953 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_block_create": "true", 2024-09-13T13:02:00.953 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_block_db_create": "false", 2024-09-13T13:02:00.953 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_block_db_path": "", 2024-09-13T13:02:00.953 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_block_db_size": "0", 2024-09-13T13:02:00.953 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_block_path": "", 2024-09-13T13:02:00.954 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_block_preallocate_file": "false", 2024-09-13T13:02:00.954 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_block_size": "107374182400", 2024-09-13T13:02:00.954 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_block_wal_create": "false", 2024-09-13T13:02:00.954 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_block_wal_path": "", 2024-09-13T13:02:00.954 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_block_wal_size": "100663296", 2024-09-13T13:02:00.954 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_bluefs": "true", 2024-09-13T13:02:00.954 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-13T13:02:00.954 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-13T13:02:00.954 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-13T13:02:00.954 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-13T13:02:00.954 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-13T13:02:00.954 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T13:02:00.954 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T13:02:00.955 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-13T13:02:00.955 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T13:02:00.955 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_autotune": "true", 2024-09-13T13:02:00.955 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-13T13:02:00.955 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-13T13:02:00.955 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-13T13:02:00.955 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-13T13:02:00.955 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_size": "0", 2024-09-13T13:02:00.955 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-13T13:02:00.955 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-13T13:02:00.955 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-13T13:02:00.955 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-13T13:02:00.955 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_type": "2q", 2024-09-13T13:02:00.956 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-13T13:02:00.956 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_clone_cow": "true", 2024-09-13T13:02:00.956 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-13T13:02:00.956 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-13T13:02:00.956 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-13T13:02:00.956 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-13T13:02:00.956 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-13T13:02:00.956 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-13T13:02:00.956 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-13T13:02:00.956 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_compression_mode": "none", 2024-09-13T13:02:00.956 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-13T13:02:00.956 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_csum_type": "crc32c", 2024-09-13T13:02:00.957 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-13T13:02:00.957 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-13T13:02:00.957 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_freelist": "false", 2024-09-13T13:02:00.957 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-13T13:02:00.957 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-13T13:02:00.957 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-13T13:02:00.957 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-13T13:02:00.957 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-13T13:02:00.957 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-13T13:02:00.957 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-13T13:02:00.957 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-13T13:02:00.957 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-13T13:02:00.957 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-13T13:02:00.958 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-13T13:02:00.958 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-13T13:02:00.958 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_small_allocations": "0", 2024-09-13T13:02:00.958 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-13T13:02:00.958 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_default_buffered_read": "true", 2024-09-13T13:02:00.958 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_default_buffered_write": "false", 2024-09-13T13:02:00.958 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-13T13:02:00.958 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-13T13:02:00.958 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-13T13:02:00.958 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-13T13:02:00.958 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-13T13:02:00.958 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-13T13:02:00.958 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-13T13:02:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-13T13:02:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-13T13:02:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fail_eio": "false", 2024-09-13T13:02:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-13T13:02:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-13T13:02:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-13T13:02:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-13T13:02:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-13T13:02:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-13T13:02:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_on_mount": "false", 2024-09-13T13:02:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-13T13:02:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_on_umount": "false", 2024-09-13T13:02:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-13T13:02:00.960 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-13T13:02:00.960 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-13T13:02:00.960 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-13T13:02:00.960 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-13T13:02:00.960 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-13T13:02:00.960 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-13T13:02:00.960 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-13T13:02:00.960 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_ignore_data_csum": "false", 2024-09-13T13:02:00.960 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-13T13:02:00.960 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-13T13:02:00.960 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-13T13:02:00.960 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-13T13:02:00.960 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_log_op_age": "5.000000", 2024-09-13T13:02:00.961 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_max_alloc_size": "0", 2024-09-13T13:02:00.961 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_max_blob_size": "0", 2024-09-13T13:02:00.961 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-13T13:02:00.961 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-13T13:02:00.961 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-13T13:02:00.961 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_max_deferred_txc": "32", 2024-09-13T13:02:00.961 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_min_alloc_size": "0", 2024-09-13T13:02:00.961 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-13T13:02:00.961 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-13T13:02:00.961 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_nid_prealloc": "1024", 2024-09-13T13:02:00.962 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-13T13:02:00.962 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-13T13:02:00.962 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-13T13:02:00.962 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-13T13:02:00.963 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_retry_disk_reads": "3", 2024-09-13T13:02:00.963 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_rocksdb_cf": "true", 2024-09-13T13:02:00.963 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-13T13:02:00.963 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-13T13:02:00.963 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-13T13:02:00.963 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-13T13:02:00.964 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-13T13:02:00.964 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-13T13:02:00.964 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-13T13:02:00.964 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-13T13:02:00.964 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_spdk_mem": "512", 2024-09-13T13:02:00.964 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-13T13:02:00.964 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-13T13:02:00.964 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-13T13:02:00.964 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-13T13:02:00.964 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-13T13:02:00.964 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-13T13:02:00.964 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-13T13:02:00.965 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_tracing": "false", 2024-09-13T13:02:00.965 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-13T13:02:00.965 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-13T13:02:00.965 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-13T13:02:00.965 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-13T13:02:00.965 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-13T13:02:00.965 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-13T13:02:00.965 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-13T13:02:00.965 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-13T13:02:00.965 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-13T13:02:00.965 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_write_v2": "false", 2024-09-13T13:02:00.965 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_write_v2_random": "false", 2024-09-13T13:02:00.965 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_zero_block_detection": "false", 2024-09-13T13:02:00.966 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-13T13:02:00.966 INFO:teuthology.orchestra.run.smithi052.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-13T13:02:00.966 INFO:teuthology.orchestra.run.smithi052.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-13T13:02:00.966 INFO:teuthology.orchestra.run.smithi052.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-13T13:02:00.966 INFO:teuthology.orchestra.run.smithi052.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-13T13:02:00.966 INFO:teuthology.orchestra.run.smithi052.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-13T13:02:00.966 INFO:teuthology.orchestra.run.smithi052.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-13T13:02:00.966 INFO:teuthology.orchestra.run.smithi052.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-13T13:02:00.966 INFO:teuthology.orchestra.run.smithi052.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-13T13:02:00.966 INFO:teuthology.orchestra.run.smithi052.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-13T13:02:00.966 INFO:teuthology.orchestra.run.smithi052.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-13T13:02:00.966 INFO:teuthology.orchestra.run.smithi052.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-13T13:02:00.966 INFO:teuthology.orchestra.run.smithi052.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-13T13:02:00.967 INFO:teuthology.orchestra.run.smithi052.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-13T13:02:00.967 INFO:teuthology.orchestra.run.smithi052.stdout: "cephx_cluster_require_signatures": "false", 2024-09-13T13:02:00.967 INFO:teuthology.orchestra.run.smithi052.stdout: "cephx_cluster_require_version": "2", 2024-09-13T13:02:00.967 INFO:teuthology.orchestra.run.smithi052.stdout: "cephx_require_signatures": "false", 2024-09-13T13:02:00.967 INFO:teuthology.orchestra.run.smithi052.stdout: "cephx_require_version": "2", 2024-09-13T13:02:00.967 INFO:teuthology.orchestra.run.smithi052.stdout: "cephx_service_require_signatures": "false", 2024-09-13T13:02:00.967 INFO:teuthology.orchestra.run.smithi052.stdout: "cephx_service_require_version": "2", 2024-09-13T13:02:00.967 INFO:teuthology.orchestra.run.smithi052.stdout: "cephx_sign_messages": "true", 2024-09-13T13:02:00.967 INFO:teuthology.orchestra.run.smithi052.stdout: "chdir": "", 2024-09-13T13:02:00.967 INFO:teuthology.orchestra.run.smithi052.stdout: "client_acl_type": "", 2024-09-13T13:02:00.967 INFO:teuthology.orchestra.run.smithi052.stdout: "client_asio_thread_count": "2", 2024-09-13T13:02:00.967 INFO:teuthology.orchestra.run.smithi052.stdout: "client_cache_mid": "0.750000", 2024-09-13T13:02:00.967 INFO:teuthology.orchestra.run.smithi052.stdout: "client_cache_size": "16384", 2024-09-13T13:02:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_caps_release_delay": "5", 2024-09-13T13:02:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_check_pool_perm": "true", 2024-09-13T13:02:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-13T13:02:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_debug_force_sync_read": "false", 2024-09-13T13:02:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_debug_getattr_caps": "false", 2024-09-13T13:02:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_debug_inject_features": "", 2024-09-13T13:02:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_debug_inject_tick_delay": "0", 2024-09-13T13:02:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-13T13:02:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_die_on_failed_remount": "false", 2024-09-13T13:02:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_dirsize_rbytes": "true", 2024-09-13T13:02:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_force_lazyio": "false", 2024-09-13T13:02:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_fs": "", 2024-09-13T13:02:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-13T13:02:00.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_inject_release_failure": "false", 2024-09-13T13:02:00.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_max_inline_size": "4096", 2024-09-13T13:02:00.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-13T13:02:00.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_mds_namespace": "", 2024-09-13T13:02:00.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_metadata": "", 2024-09-13T13:02:00.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_mount_gid": "-1", 2024-09-13T13:02:00.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_mount_timeout": "300", 2024-09-13T13:02:00.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_mount_uid": "-1", 2024-09-13T13:02:00.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_mountpoint": "/", 2024-09-13T13:02:00.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_notify_timeout": "10", 2024-09-13T13:02:00.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_oc": "true", 2024-09-13T13:02:00.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_oc_max_dirty": "104857600", 2024-09-13T13:02:00.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-13T13:02:00.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_oc_max_objects": "1000", 2024-09-13T13:02:00.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_oc_size": "209715200", 2024-09-13T13:02:00.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_oc_target_dirty": "8388608", 2024-09-13T13:02:00.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_permissions": "true", 2024-09-13T13:02:00.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_quota": "true", 2024-09-13T13:02:00.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_quota_df": "true", 2024-09-13T13:02:00.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_readahead_max_bytes": "0", 2024-09-13T13:02:00.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_readahead_max_periods": "4", 2024-09-13T13:02:00.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_readahead_min": "131072", 2024-09-13T13:02:00.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_reconnect_stale": "false", 2024-09-13T13:02:00.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_shutdown_timeout": "30", 2024-09-13T13:02:00.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_snapdir": ".snap", 2024-09-13T13:02:00.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_tick_interval": "1", 2024-09-13T13:02:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: "client_trace": "", 2024-09-13T13:02:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: "client_try_dentry_invalidate": "false", 2024-09-13T13:02:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: "client_use_faked_inos": "false", 2024-09-13T13:02:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: "client_use_random_mds": "false", 2024-09-13T13:02:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: "clog_to_graylog": "false", 2024-09-13T13:02:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-13T13:02:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: "clog_to_graylog_port": "12201", 2024-09-13T13:02:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: "clog_to_monitors": "default=true", 2024-09-13T13:02:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: "clog_to_syslog": "false", 2024-09-13T13:02:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-13T13:02:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: "clog_to_syslog_level": "info", 2024-09-13T13:02:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "-", 2024-09-13T13:02:00.971 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_network": "", 2024-09-13T13:02:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_network_interface": "", 2024-09-13T13:02:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "compressor_zlib_isal": "false", 2024-09-13T13:02:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "compressor_zlib_level": "5", 2024-09-13T13:02:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "compressor_zlib_winsize": "-15", 2024-09-13T13:02:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "compressor_zstd_level": "1", 2024-09-13T13:02:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b", 2024-09-13T13:02:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-13T13:02:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "crimson_alien_op_num_threads": "6", 2024-09-13T13:02:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-13T13:02:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-13T13:02:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-13T13:02:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "crimson_osd_stat_interval": "0", 2024-09-13T13:02:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "crimson_seastar_cpu_cores": "", 2024-09-13T13:02:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "crimson_seastar_num_threads": "0", 2024-09-13T13:02:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_location": "", 2024-09-13T13:02:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_location_hook": "", 2024-09-13T13:02:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_location_hook_timeout": "10", 2024-09-13T13:02:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "daemonize": "false", 2024-09-13T13:02:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "daos_pool": "tank", 2024-09-13T13:02:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-13T13:02:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-13T13:02:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-13T13:02:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_alienstore": "0/5", 2024-09-13T13:02:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_allow_any_pool_priority": "false", 2024-09-13T13:02:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_asok": "1/5", 2024-09-13T13:02:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_asok_assert_abort": "false", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_asserts_on_shutdown": "true", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_auth": "1/5", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_bdev": "1/3", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_bluefs": "1/5", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_bluestore": "1/5", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_buffer": "0/1", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_ceph_dedup": "0/5", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_ceph_exporter": "1/5", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_cephfs_mirror": "0/5", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_cephsqlite": "0/5", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_client": "0/5", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_compressor": "1/5", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_context": "0/1", 2024-09-13T13:02:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_crimson_interrupt": "0/5", 2024-09-13T13:02:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_crush": "1/1", 2024-09-13T13:02:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_crypto": "1/5", 2024-09-13T13:02:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_cyanstore": "0/5", 2024-09-13T13:02:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_deliberately_leak_memory": "false", 2024-09-13T13:02:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_disable_randomized_ping": "false", 2024-09-13T13:02:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_dpdk": "1/5", 2024-09-13T13:02:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_eventtrace": "1/5", 2024-09-13T13:02:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_filer": "0/1", 2024-09-13T13:02:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_filestore": "1/3", 2024-09-13T13:02:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_finisher": "1/1", 2024-09-13T13:02:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_fuse": "1/5", 2024-09-13T13:02:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_heartbeat_testing_span": "0", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_heartbeatmap": "1/5", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_javaclient": "1/5", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_journal": "1/3", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_journaler": "0/5", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_kstore": "1/5", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_lockdep": "0/1", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_mclock": "1/5", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_mds": "1/5", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_mds_balancer": "1/5", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_mds_locker": "1/5", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_mds_log": "1/5", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_mds_log_expire": "1/5", 2024-09-13T13:02:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_mds_migrator": "1/5", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_mds_quiesce": "3/5", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_memstore": "1/5", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_mgr": "2/5", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_mgrc": "1/5", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_mon": "20/20", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_monc": "0/10", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_ms": "1/1", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_none": "0/5", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_objclass": "0/5", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_objectcacher": "0/5", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_objecter": "0/1", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_optracker": "0/5", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_osd": "1/5", 2024-09-13T13:02:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_paxos": "20/20", 2024-09-13T13:02:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_perfcounter": "1/5", 2024-09-13T13:02:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_prioritycache": "1/5", 2024-09-13T13:02:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rados": "0/5", 2024-09-13T13:02:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rbd": "0/5", 2024-09-13T13:02:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rbd_mirror": "0/5", 2024-09-13T13:02:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rbd_pwl": "0/5", 2024-09-13T13:02:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rbd_replay": "0/5", 2024-09-13T13:02:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_refs": "0/0", 2024-09-13T13:02:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_reserver": "1/1", 2024-09-13T13:02:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rgw": "1/5", 2024-09-13T13:02:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rgw_access": "1/5", 2024-09-13T13:02:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rgw_datacache": "1/5", 2024-09-13T13:02:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rgw_dbstore": "1/5", 2024-09-13T13:02:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rgw_flight": "1/5", 2024-09-13T13:02:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-13T13:02:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rgw_notification": "1/5", 2024-09-13T13:02:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rgw_sync": "1/5", 2024-09-13T13:02:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_rocksdb": "4/5", 2024-09-13T13:02:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore": "0/5", 2024-09-13T13:02:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore_backref": "0/5", 2024-09-13T13:02:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore_cache": "0/5", 2024-09-13T13:02:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore_cleaner": "0/5", 2024-09-13T13:02:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore_device": "0/5", 2024-09-13T13:02:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore_epm": "0/5", 2024-09-13T13:02:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-13T13:02:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore_journal": "0/5", 2024-09-13T13:02:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore_lba": "0/5", 2024-09-13T13:02:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore_odata": "0/5", 2024-09-13T13:02:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore_omap": "0/5", 2024-09-13T13:02:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore_onode": "0/5", 2024-09-13T13:02:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore_t": "0/5", 2024-09-13T13:02:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_seastore_tm": "0/5", 2024-09-13T13:02:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_striper": "0/1", 2024-09-13T13:02:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_test": "0/5", 2024-09-13T13:02:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_throttle": "1/1", 2024-09-13T13:02:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_timer": "0/1", 2024-09-13T13:02:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_tp": "0/5", 2024-09-13T13:02:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "debug_trace": "1/5", 2024-09-13T13:02:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-13T13:02:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "device_failure_prediction_mode": "none", 2024-09-13T13:02:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-13T13:02:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-13T13:02:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "err_to_graylog": "false", 2024-09-13T13:02:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "err_to_journald": "false", 2024-09-13T13:02:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "err_to_stderr": "true", 2024-09-13T13:02:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "err_to_syslog": "false", 2024-09-13T13:02:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "event_tracing": "false", 2024-09-13T13:02:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "exporter_addr": "0.0.0.0", 2024-09-13T13:02:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "exporter_cert_file": "", 2024-09-13T13:02:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "exporter_http_port": "9926", 2024-09-13T13:02:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "exporter_key_file": "", 2024-09-13T13:02:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "exporter_prio_limit": "5", 2024-09-13T13:02:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-13T13:02:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "exporter_sort_metrics": "true", 2024-09-13T13:02:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "exporter_stats_period": "5", 2024-09-13T13:02:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "fake_statfs_for_testing": "0", 2024-09-13T13:02:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "fatal_signal_handlers": "true", 2024-09-13T13:02:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "filer_max_purge_ops": "10", 2024-09-13T13:02:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "filer_max_truncate_ops": "128", 2024-09-13T13:02:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_apply_finisher_threads": "1", 2024-09-13T13:02:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_blackhole": "false", 2024-09-13T13:02:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_btrfs_clone_range": "true", 2024-09-13T13:02:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_btrfs_snap": "true", 2024-09-13T13:02:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_caller_concurrency": "10", 2024-09-13T13:02:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_collect_device_partition_information": "true", 2024-09-13T13:02:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_commit_timeout": "600.000000", 2024-09-13T13:02:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_debug_inject_read_err": "false", 2024-09-13T13:02:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_debug_omap_check": "false", 2024-09-13T13:02:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_debug_verify_split": "false", 2024-09-13T13:02:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_dump_file": "", 2024-09-13T13:02:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-13T13:02:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-13T13:02:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_fadvise": "true", 2024-09-13T13:02:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_fail_eio": "true", 2024-09-13T13:02:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_fd_cache_shards": "16", 2024-09-13T13:02:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_fd_cache_size": "128", 2024-09-13T13:02:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_fiemap": "false", 2024-09-13T13:02:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_fiemap_threshold": "4096", 2024-09-13T13:02:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-13T13:02:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-13T13:02:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_inject_stall": "0", 2024-09-13T13:02:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_journal_parallel": "false", 2024-09-13T13:02:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_journal_trailing": "false", 2024-09-13T13:02:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_journal_writeahead": "false", 2024-09-13T13:02:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_kill_at": "0", 2024-09-13T13:02:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-13T13:02:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-13T13:02:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-13T13:02:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-13T13:02:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-13T13:02:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_inline_xattrs": "0", 2024-09-13T13:02:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-13T13:02:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-13T13:02:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-13T13:02:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-13T13:02:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_xattr_value_size": "0", 2024-09-13T13:02:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-13T13:02:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-13T13:02:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-13T13:02:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_merge_threshold": "-10", 2024-09-13T13:02:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-13T13:02:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_odsync_write": "false", 2024-09-13T13:02:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_omap_backend": "rocksdb", 2024-09-13T13:02:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_omap_backend_path": "", 2024-09-13T13:02:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-13T13:02:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-13T13:02:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-13T13:02:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_op_thread_timeout": "60", 2024-09-13T13:02:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_op_threads": "2", 2024-09-13T13:02:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_punch_hole": "false", 2024-09-13T13:02:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-13T13:02:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-13T13:02:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-13T13:02:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-13T13:02:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-13T13:02:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-13T13:02:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-13T13:02:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-13T13:02:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-13T13:02:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_queue_max_ops": "50", 2024-09-13T13:02:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-13T13:02:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_seek_data_hole": "false", 2024-09-13T13:02:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_sloppy_crc": "false", 2024-09-13T13:02:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-13T13:02:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_splice": "false", 2024-09-13T13:02:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_split_multiple": "2", 2024-09-13T13:02:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_split_rand_factor": "20", 2024-09-13T13:02:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_update_to": "1000", 2024-09-13T13:02:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-13T13:02:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-13T13:02:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-13T13:02:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-13T13:02:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-13T13:02:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-13T13:02:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_wbthrottle_enable": "true", 2024-09-13T13:02:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-13T13:02:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-13T13:02:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-13T13:02:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-13T13:02:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-13T13:02:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-13T13:02:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_xfs_extsize": "false", 2024-09-13T13:02:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: "filestore_zfs_snap": "false", 2024-09-13T13:02:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: "fio_dir": "/tmp/fio", 2024-09-13T13:02:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: "fsid": "2b415b38-71cf-11ef-bceb-c7b262605968", 2024-09-13T13:02:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_allow_other": "true", 2024-09-13T13:02:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_atomic_o_trunc": "true", 2024-09-13T13:02:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_big_writes": "true", 2024-09-13T13:02:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_debug": "false", 2024-09-13T13:02:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_default_permissions": "false", 2024-09-13T13:02:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_disable_pagecache": "false", 2024-09-13T13:02:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_max_write": "0", 2024-09-13T13:02:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_multithreaded": "true", 2024-09-13T13:02:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_require_active_mds": "true", 2024-09-13T13:02:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_set_user_groups": "true", 2024-09-13T13:02:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_splice_move": "true", 2024-09-13T13:02:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_splice_read": "true", 2024-09-13T13:02:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_splice_write": "true", 2024-09-13T13:02:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-13T13:02:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: "fuse_use_invalidate_cb": "true", 2024-09-13T13:02:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-13T13:02:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: "gss_target_name": "ceph", 2024-09-13T13:02:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_file": "", 2024-09-13T13:02:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_inject_failure": "0", 2024-09-13T13:02:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_interval": "5", 2024-09-13T13:02:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: "host": "smithi052", 2024-09-13T13:02:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-13T13:02:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-13T13:02:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-13T13:02:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-13T13:02:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-13T13:02:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-13T13:02:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-13T13:02:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-13T13:02:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-13T13:02:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-13T13:02:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-13T13:02:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-13T13:02:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "inject_early_sigterm": "false", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "jaeger_agent_port": "6799", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "jaeger_tracing_enable": "false", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_aio": "true", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_align_min_size": "65536", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_block_align": "true", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_block_size": "4096", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_dio": "true", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_discard": "false", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_force_aio": "false", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_ignore_corruption": "false", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_max_write_bytes": "10485760", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_max_write_entries": "100", 2024-09-13T13:02:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_replay_from": "0", 2024-09-13T13:02:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-13T13:02:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-13T13:02:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-13T13:02:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-13T13:02:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_write_header_frequency": "0", 2024-09-13T13:02:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: "journal_zero_on_create": "false", 2024-09-13T13:02:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: "journaler_prefetch_periods": "10", 2024-09-13T13:02:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: "journaler_prezero_periods": "5", 2024-09-13T13:02:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: "journaler_write_head_interval": "15", 2024-09-13T13:02:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "", 2024-09-13T13:02:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: "keyfile": "", 2024-09-13T13:02:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-13T13:02:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: "kstore_backend": "rocksdb", 2024-09-13T13:02:00.994 INFO:teuthology.orchestra.run.smithi052.stdout: "kstore_default_stripe_size": "65536", 2024-09-13T13:02:00.994 INFO:teuthology.orchestra.run.smithi052.stdout: "kstore_fsck_on_mount": "false", 2024-09-13T13:02:00.994 INFO:teuthology.orchestra.run.smithi052.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-13T13:02:00.994 INFO:teuthology.orchestra.run.smithi052.stdout: "kstore_max_bytes": "67108864", 2024-09-13T13:02:00.994 INFO:teuthology.orchestra.run.smithi052.stdout: "kstore_max_ops": "512", 2024-09-13T13:02:00.994 INFO:teuthology.orchestra.run.smithi052.stdout: "kstore_nid_prealloc": "1024", 2024-09-13T13:02:00.994 INFO:teuthology.orchestra.run.smithi052.stdout: "kstore_onode_map_size": "1024", 2024-09-13T13:02:00.994 INFO:teuthology.orchestra.run.smithi052.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-13T13:02:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: "kstore_sync_submit_transaction": "false", 2024-09-13T13:02:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: "kstore_sync_transaction": "false", 2024-09-13T13:02:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: "librados_thread_count": "2", 2024-09-13T13:02:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: "lockdep": "false", 2024-09-13T13:02:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: "lockdep_force_backtrace": "false", 2024-09-13T13:02:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: "log_coarse_timestamps": "true", 2024-09-13T13:02:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-13T13:02:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: "log_flush_on_exit": "false", 2024-09-13T13:02:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: "log_graylog_host": "127.0.0.1", 2024-09-13T13:02:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: "log_graylog_port": "12201", 2024-09-13T13:02:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: "log_max_new": "1000", 2024-09-13T13:02:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: "log_max_recent": "10000", 2024-09-13T13:02:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: "log_stderr_prefix": "", 2024-09-13T13:02:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: "log_stop_at_utilization": "0.970000", 2024-09-13T13:02:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": "true", 2024-09-13T13:02:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_graylog": "false", 2024-09-13T13:02:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_journald": "false", 2024-09-13T13:02:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_stderr": "false", 2024-09-13T13:02:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_syslog": "false", 2024-09-13T13:02:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: "max_rotating_auth_attempts": "10", 2024-09-13T13:02:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-13T13:02:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_action_on_write_error": "1", 2024-09-13T13:02:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_alternate_name_max": "8192", 2024-09-13T13:02:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_asio_thread_count": "2", 2024-09-13T13:02:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_export_pin": "true", 2024-09-13T13:02:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_fragment_dirs": "true", 2024-09-13T13:02:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-13T13:02:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_fragment_interval": "5", 2024-09-13T13:02:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-13T13:02:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-13T13:02:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_interval": "10", 2024-09-13T13:02:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_max": "-1", 2024-09-13T13:02:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_max_until": "-1", 2024-09-13T13:02:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_merge_size": "50", 2024-09-13T13:02:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_midchunk": "0.300000", 2024-09-13T13:02:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-13T13:02:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_min_start": "0.200000", 2024-09-13T13:02:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_minchunk": "0.001000", 2024-09-13T13:02:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_mode": "0", 2024-09-13T13:02:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_need_max": "1.200000", 2024-09-13T13:02:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_need_min": "0.800000", 2024-09-13T13:02:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_overload_epochs": "2", 2024-09-13T13:02:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-13T13:02:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-13T13:02:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_split_bits": "3", 2024-09-13T13:02:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-13T13:02:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_split_size": "10000", 2024-09-13T13:02:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-13T13:02:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_target_decay": "10.000000", 2024-09-13T13:02:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-13T13:02:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_beacon_grace": "15.000000", 2024-09-13T13:02:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_beacon_interval": "4.000000", 2024-09-13T13:02:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-13T13:02:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-13T13:02:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cache_mid": "0.700000", 2024-09-13T13:02:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-13T13:02:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cache_quiesce_delay": "0", 2024-09-13T13:02:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-13T13:02:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-13T13:02:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-13T13:02:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cache_release_free_interval": "10", 2024-09-13T13:02:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cache_reservation": "0.050000", 2024-09-13T13:02:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-13T13:02:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cache_trim_interval": "1", 2024-09-13T13:02:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cache_trim_threshold": "262144", 2024-09-13T13:02:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-13T13:02:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-13T13:02:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-13T13:02:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_client_prealloc_inos": "1000", 2024-09-13T13:02:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-13T13:02:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_connect_bootstrapping": "false", 2024-09-13T13:02:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_damage_table_max_entries": "10000", 2024-09-13T13:02:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-13T13:02:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_debug_auth_pins": "false", 2024-09-13T13:02:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_debug_frag": "false", 2024-09-13T13:02:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_debug_scatterstat": "false", 2024-09-13T13:02:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_debug_subtrees": "false", 2024-09-13T13:02:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_decay_halflife": "5.000000", 2024-09-13T13:02:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_default_dir_hash": "2", 2024-09-13T13:02:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_defer_session_stale": "true", 2024-09-13T13:02:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_deny_all_reconnect": "false", 2024-09-13T13:02:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_dir_keys_per_op": "16384", 2024-09-13T13:02:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_dir_max_commit_size": "10", 2024-09-13T13:02:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_dir_max_entries": "0", 2024-09-13T13:02:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_dir_prefetch": "true", 2024-09-13T13:02:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-13T13:02:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-13T13:02:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_dump_cache_on_map": "false", 2024-09-13T13:02:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-13T13:02:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-13T13:02:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_early_reply": "true", 2024-09-13T13:02:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_enable_op_tracker": "true", 2024-09-13T13:02:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_enforce_unique_name": "true", 2024-09-13T13:02:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-13T13:02:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-13T13:02:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_export_ephemeral_random": "true", 2024-09-13T13:02:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-13T13:02:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-13T13:02:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_health_summarize_threshold": "10", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_inject_health_dummy": "false", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_inject_migrator_session_race": "false", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-13T13:02:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-13T13:02:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_join_fs": "", 2024-09-13T13:02:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_journal_format": "1", 2024-09-13T13:02:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-13T13:02:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_kill_create_at": "0", 2024-09-13T13:02:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_kill_dirfrag_at": "0", 2024-09-13T13:02:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_kill_export_at": "0", 2024-09-13T13:02:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_kill_import_at": "0", 2024-09-13T13:02:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_kill_journal_at": "0", 2024-09-13T13:02:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_kill_journal_expire_at": "0", 2024-09-13T13:02:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_kill_journal_replay_at": "0", 2024-09-13T13:02:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_kill_link_at": "0", 2024-09-13T13:02:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_kill_mdstable_at": "0", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_kill_openc_at": "0", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_kill_rename_at": "0", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_kill_shutdown_at": "0", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log_event_large_threshold": "524288", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log_events_per_segment": "1024", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log_max_events": "-1", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log_max_segments": "128", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log_pause": "false", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log_segment_size": "0", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-13T13:02:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log_trim_threshold": "128", 2024-09-13T13:02:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-13T13:02:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log_warn_factor": "2.000000", 2024-09-13T13:02:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_max_caps_per_client": "1048576", 2024-09-13T13:02:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_max_completed_flushes": "100000", 2024-09-13T13:02:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_max_completed_requests": "100000", 2024-09-13T13:02:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_max_export_size": "20971520", 2024-09-13T13:02:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_max_file_recover": "32", 2024-09-13T13:02:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_max_purge_files": "64", 2024-09-13T13:02:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_max_purge_ops": "8192", 2024-09-13T13:02:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-13T13:02:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-13T13:02:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_max_snaps_per_dir": "100", 2024-09-13T13:02:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_metrics_update_interval": "2", 2024-09-13T13:02:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_min_caps_per_client": "100", 2024-09-13T13:02:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_min_caps_working_set": "10000", 2024-09-13T13:02:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-13T13:02:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_numa_node": "-1", 2024-09-13T13:02:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-13T13:02:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_op_complaint_time": "30.000000", 2024-09-13T13:02:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_op_history_duration": "600", 2024-09-13T13:02:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_op_history_size": "20", 2024-09-13T13:02:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_op_history_slow_op_size": "20", 2024-09-13T13:02:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-13T13:02:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_op_log_threshold": "5", 2024-09-13T13:02:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_ping_grace": "15", 2024-09-13T13:02:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_ping_interval": "5", 2024-09-13T13:02:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-13T13:02:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-13T13:02:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_recall_max_caps": "30000", 2024-09-13T13:02:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-13T13:02:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-13T13:02:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-13T13:02:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_recall_warning_threshold": "262144", 2024-09-13T13:02:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-13T13:02:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_replay_interval": "1.000000", 2024-09-13T13:02:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-13T13:02:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-13T13:02:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_root_ino_gid": "0", 2024-09-13T13:02:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_root_ino_uid": "0", 2024-09-13T13:02:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-13T13:02:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_scrub_stats_review_period": "1", 2024-09-13T13:02:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-13T13:02:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-13T13:02:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-13T13:02:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-13T13:02:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-13T13:02:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-13T13:02:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-13T13:02:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-13T13:02:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-13T13:02:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-13T13:02:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_shutdown_check": "0", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_skip_ino": "0", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_snap_max_uid": "4294967294", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_snap_min_uid": "0", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_snap_rstat": "false", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_standby_replay_damaged": "false", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_symlink_recovery": "true", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_thrash_exports": "0", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_thrash_fragments": "0", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_tick_interval": "5.000000", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_valgrind_exit": "false", 2024-09-13T13:02:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_verify_backtrace": "1", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_verify_scatter": "false", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_wipe_sessions": "false", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "mempool_debug": "false", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "memstore_device_bytes": "1073741824", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "memstore_page_set": "false", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "memstore_page_size": "65536", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_client_bytes": "134217728", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_client_messages": "512", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-13T13:02:01.011 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_disabled_modules": "", 2024-09-13T13:02:01.012 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_enable_op_tracker": "true", 2024-09-13T13:02:01.012 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-13T13:02:01.012 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_max_pg_creating": "1024", 2024-09-13T13:02:01.012 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_max_pg_num_change": "128", 2024-09-13T13:02:01.012 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_mds_bytes": "134217728", 2024-09-13T13:02:01.012 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_mds_messages": "128", 2024-09-13T13:02:01.012 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-13T13:02:01.012 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_mon_bytes": "134217728", 2024-09-13T13:02:01.012 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_mon_messages": "128", 2024-09-13T13:02:01.012 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-13T13:02:01.012 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-13T13:02:01.012 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_op_history_duration": "600", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_op_history_size": "20", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_op_log_threshold": "5", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_osd_bytes": "536870912", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_osd_messages": "8192", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_pool": "true", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_standby_modules": "true", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_stats_period": "5", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_stats_threshold": "5", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_tick_period": "2", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-13T13:02:01.013 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_allow_pool_delete": "true", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_allow_pool_size_one": "false", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_auth_validate_all_caps": "true", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_client_bytes": "104857600", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_client_directed_command_retry": "2", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_client_hunt_parallel": "3", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_client_log_interval": "1.000000", 2024-09-13T13:02:01.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_client_ping_interval": "10.000000", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_client_target_rank": "-1", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_cluster_log_level": "debug", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_cluster_log_to_file": "true", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_cluster_log_to_journald": "true", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-13T13:02:01.015 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-13T13:02:01.016 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_compact_on_bootstrap": "false", 2024-09-13T13:02:01.016 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_compact_on_start": "false", 2024-09-13T13:02:01.016 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_compact_on_trim": "true", 2024-09-13T13:02:01.016 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-13T13:02:01.016 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-13T13:02:01.016 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-13T13:02:01.016 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_cpu_threads": "4", 2024-09-13T13:02:01.016 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_crush_min_required_version": "hammer", 2024-09-13T13:02:01.016 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_daemon_bytes": "419430400", 2024-09-13T13:02:01.016 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-13T13:02:01.016 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_data_avail_crit": "5", 2024-09-13T13:02:01.016 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_data_avail_warn": "5", 2024-09-13T13:02:01.016 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_data_size_warn": "16106127360", 2024-09-13T13:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-13T13:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-13T13:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_debug_dump_json": "false", 2024-09-13T13:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-13T13:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_debug_dump_transactions": "false", 2024-09-13T13:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_debug_extra_checks": "false", 2024-09-13T13:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-13T13:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-13T13:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_debug_no_require_reef": "false", 2024-09-13T13:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_debug_no_require_squid": "false", 2024-09-13T13:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-13T13:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-13T13:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-13T13:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_down_mkfs_grace": "60", 2024-09-13T13:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_down_uptime_grace": "60", 2024-09-13T13:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_election_default_strategy": "1", 2024-09-13T13:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_election_timeout": "5.000000", 2024-09-13T13:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-13T13:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_elector_ping_divisor": "2", 2024-09-13T13:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-13T13:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_enable_op_tracker": "true", 2024-09-13T13:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_fake_pool_delete": "false", 2024-09-13T13:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_force_quorum_join": "false", 2024-09-13T13:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-13T13:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_globalid_prealloc": "10000", 2024-09-13T13:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_health_detail_to_clog": "true", 2024-09-13T13:02:01.019 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_health_log_update_period": "5", 2024-09-13T13:02:01.019 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_health_max_detail": "50", 2024-09-13T13:02:01.019 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_health_to_clog": "true", 2024-09-13T13:02:01.019 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_health_to_clog_interval": "600", 2024-09-13T13:02:01.019 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-13T13:02:01.019 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_host": "172.21.15.52:6789/0", 2024-09-13T13:02:01.019 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_host_override": "", 2024-09-13T13:02:01.019 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_initial_members": "", 2024-09-13T13:02:01.019 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-13T13:02:01.019 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-13T13:02:01.019 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-13T13:02:01.019 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-13T13:02:01.020 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-13T13:02:01.020 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_lease": "5.000000", 2024-09-13T13:02:01.020 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-13T13:02:01.020 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-13T13:02:01.020 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_log_full_interval": "50", 2024-09-13T13:02:01.020 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_log_max": "10000", 2024-09-13T13:02:01.020 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_log_max_summary": "50", 2024-09-13T13:02:01.020 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-13T13:02:01.020 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_max_log_epochs": "500", 2024-09-13T13:02:01.020 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-13T13:02:01.020 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-13T13:02:01.020 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-13T13:02:01.020 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_max_osd": "10000", 2024-09-13T13:02:01.021 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_max_pg_per_osd": "10000", 2024-09-13T13:02:01.021 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_max_pool_pg_num": "65536", 2024-09-13T13:02:01.021 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-13T13:02:01.021 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-13T13:02:01.021 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_mds_force_trim_to": "0", 2024-09-13T13:02:01.021 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_mds_skip_sanity": "false", 2024-09-13T13:02:01.021 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_memory_autotune": "true", 2024-09-13T13:02:01.021 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_memory_target": "2147483648", 2024-09-13T13:02:01.021 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_mgr_beacon_grace": "30", 2024-09-13T13:02:01.021 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-13T13:02:01.021 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_mgr_digest_period": "5", 2024-09-13T13:02:01.021 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_mgr_inactive_grace": "60", 2024-09-13T13:02:01.021 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-13T13:02:01.022 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-13T13:02:01.022 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_min_osdmap_epochs": "50", 2024-09-13T13:02:01.022 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-13T13:02:01.022 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-13T13:02:01.022 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_op_complaint_time": "30", 2024-09-13T13:02:01.022 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_op_history_duration": "600", 2024-09-13T13:02:01.022 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_op_history_size": "20", 2024-09-13T13:02:01.022 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_op_history_slow_op_size": "20", 2024-09-13T13:02:01.022 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-13T13:02:01.022 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_op_log_threshold": "5", 2024-09-13T13:02:01.022 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-13T13:02:01.022 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-13T13:02:01.022 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-13T13:02:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_auto_mark_in": "false", 2024-09-13T13:02:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-13T13:02:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-13T13:02:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-13T13:02:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_cache_size": "500", 2024-09-13T13:02:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-13T13:02:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-13T13:02:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-13T13:02:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_down_out_interval": "600", 2024-09-13T13:02:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-13T13:02:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-13T13:02:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_force_trim_to": "0", 2024-09-13T13:02:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_min_down_reporters": "2", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-13T13:02:01.024 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_report_timeout": "900", 2024-09-13T13:02:01.025 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-13T13:02:01.025 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-13T13:02:01.025 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-13T13:02:01.026 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-13T13:02:01.026 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-13T13:02:01.026 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-13T13:02:01.026 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-13T13:02:01.026 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-13T13:02:01.027 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-13T13:02:01.027 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_pg_stuck_threshold": "60", 2024-09-13T13:02:01.027 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-13T13:02:01.027 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-13T13:02:01.027 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-13T13:02:01.027 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-13T13:02:01.027 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-13T13:02:01.027 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-13T13:02:01.027 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_probe_timeout": "2.000000", 2024-09-13T13:02:01.027 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_reweight_max_change": "0.050000", 2024-09-13T13:02:01.027 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_reweight_max_osds": "4", 2024-09-13T13:02:01.027 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-13T13:02:01.027 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-13T13:02:01.028 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-13T13:02:01.028 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-13T13:02:01.028 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-13T13:02:01.028 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_scrub_interval": "86400", 2024-09-13T13:02:01.028 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_scrub_max_keys": "100", 2024-09-13T13:02:01.028 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_scrub_timeout": "300", 2024-09-13T13:02:01.028 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_session_timeout": "300", 2024-09-13T13:02:01.028 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_smart_report_timeout": "5", 2024-09-13T13:02:01.028 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_stat_smooth_intervals": "6", 2024-09-13T13:02:01.028 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-13T13:02:01.028 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_stretch_pool_min_size": "2", 2024-09-13T13:02:01.028 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_stretch_pool_size": "4", 2024-09-13T13:02:01.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-13T13:02:01.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-13T13:02:01.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_sync_debug": "false", 2024-09-13T13:02:01.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-13T13:02:01.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-13T13:02:01.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_sync_provider_kill_at": "0", 2024-09-13T13:02:01.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_sync_requester_kill_at": "0", 2024-09-13T13:02:01.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_sync_timeout": "60.000000", 2024-09-13T13:02:01.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_target_pg_per_osd": "100", 2024-09-13T13:02:01.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_tick_interval": "5", 2024-09-13T13:02:01.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_timecheck_interval": "300.000000", 2024-09-13T13:02:01.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-13T13:02:01.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_use_min_delay_socket": "false", 2024-09-13T13:02:01.030 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_older_version_delay": "604800", 2024-09-13T13:02:01.030 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-13T13:02:01.030 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-13T13:02:01.030 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-13T13:02:01.030 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-13T13:02:01.030 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-13T13:02:01.030 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-13T13:02:01.030 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-13T13:02:01.030 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_misplaced": "false", 2024-09-13T13:02:01.030 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-13T13:02:01.030 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_older_version": "true", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "monmap": "", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-13T13:02:01.031 INFO:teuthology.orchestra.run.smithi052.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T13:02:01.032 INFO:teuthology.orchestra.run.smithi052.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-13T13:02:01.032 INFO:teuthology.orchestra.run.smithi052.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-13T13:02:01.032 INFO:teuthology.orchestra.run.smithi052.stdout: "motr_tracing_enabled": "false", 2024-09-13T13:02:01.032 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_op_threads": "3", 2024-09-13T13:02:01.032 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-13T13:02:01.032 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_cm": "false", 2024-09-13T13:02:01.032 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_device_name": "", 2024-09-13T13:02:01.032 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_dscp": "96", 2024-09-13T13:02:01.032 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-13T13:02:01.032 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-13T13:02:01.032 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_local_gid": "", 2024-09-13T13:02:01.032 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-13T13:02:01.032 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_port_num": "1", 2024-09-13T13:02:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-13T13:02:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-13T13:02:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-13T13:02:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-13T13:02:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_sl": "3", 2024-09-13T13:02:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_support_srq": "true", 2024-09-13T13:02:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_rdma_type": "ib", 2024-09-13T13:02:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_async_reap_threshold": "5", 2024-09-13T13:02:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_bind_before_connect": "false", 2024-09-13T13:02:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_bind_ipv4": "true", 2024-09-13T13:02:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_bind_ipv6": "false", 2024-09-13T13:02:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_bind_msgr1": "true", 2024-09-13T13:02:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_bind_msgr2": "false", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_bind_port_max": "7568", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_bind_port_min": "6800", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_bind_retry_count": "3", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_bind_retry_delay": "5", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_blackhole_client": "false", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_blackhole_mds": "false", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_blackhole_mgr": "false", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_blackhole_mon": "false", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_blackhole_osd": "false", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_client_mode": "crc secure", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_cluster_mode": "crc secure", 2024-09-13T13:02:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_cluster_type": "", 2024-09-13T13:02:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_compress_secure": "false", 2024-09-13T13:02:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_connection_idle_timeout": "900", 2024-09-13T13:02:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_connection_ready_timeout": "10", 2024-09-13T13:02:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_crc_data": "true", 2024-09-13T13:02:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_crc_header": "true", 2024-09-13T13:02:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_die_on_bad_msg": "false", 2024-09-13T13:02:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_die_on_bug": "true", 2024-09-13T13:02:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_die_on_old_message": "true", 2024-09-13T13:02:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_die_on_skipped_message": "false", 2024-09-13T13:02:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-13T13:02:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-13T13:02:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_coremask": "0xF", 2024-09-13T13:02:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_enable_tso": "true", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_hugepages": "", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_lro": "true", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_memory_channel": "4", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_pmd": "", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_port_id": "0", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-13T13:02:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-13T13:02:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_dump_on_send": "false", 2024-09-13T13:02:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_initial_backoff": "0.200000", 2024-09-13T13:02:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_inject_delay_max": "1.000000", 2024-09-13T13:02:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-13T13:02:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_inject_delay_type": "", 2024-09-13T13:02:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-13T13:02:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_inject_network_congestion": "0", 2024-09-13T13:02:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_inject_socket_failures": "0", 2024-09-13T13:02:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_learn_addr_from_peer": "true", 2024-09-13T13:02:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_max_accept_failures": "4", 2024-09-13T13:02:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_max_backoff": "15.000000", 2024-09-13T13:02:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_mon_client_mode": "secure crc", 2024-09-13T13:02:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_mon_service_mode": "secure crc", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_osd_compress_min_size": "1024", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_osd_compress_mode": "none", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_pq_min_cost": "65536", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_public_type": "", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_service_mode": "crc secure", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_tcp_listen_backlog": "512", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_tcp_nodelay": "true", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_tcp_rcvbuf": "0", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-13T13:02:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ms_type": "async+posix", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "no_config_file": "false", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "objecter_completion_locks_per_session": "32", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "objecter_inflight_ops": "1024", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "objecter_tick_interval": "5.000000", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "objecter_timeout": "10.000000", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "objectstore_blackhole": "false", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-13T13:02:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "openssl_engine_opts": "", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_agent_delay_time": "5.000000", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_agent_hist_halflife": "1000", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_agent_max_low_ops": "2", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_agent_max_ops": "4", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_agent_slop": "0.020000", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_asio_thread_count": "2", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_async_recovery_min_cost": "100", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-13T13:02:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_backfill_scan_max": "512", 2024-09-13T13:02:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_backfill_scan_min": "64", 2024-09-13T13:02:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_backoff_on_degraded": "false", 2024-09-13T13:02:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_backoff_on_peering": "false", 2024-09-13T13:02:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_backoff_on_unfound": "true", 2024-09-13T13:02:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_beacon_report_interval": "300", 2024-09-13T13:02:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_bench_duration": "30", 2024-09-13T13:02:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-13T13:02:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_bench_max_block_size": "67108864", 2024-09-13T13:02:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-13T13:02:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_blkin_trace_all": "false", 2024-09-13T13:02:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-13T13:02:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-13T13:02:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-13T13:02:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-13T13:02:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_check_for_log_corruption": "false", 2024-09-13T13:02:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-13T13:02:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T13:02:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-13T13:02:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T13:02:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_class_update_on_start": "true", 2024-09-13T13:02:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_client_message_cap": "256", 2024-09-13T13:02:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_client_message_size_cap": "524288000", 2024-09-13T13:02:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_client_op_priority": "63", 2024-09-13T13:02:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_client_watch_timeout": "30", 2024-09-13T13:02:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_command_max_records": "256", 2024-09-13T13:02:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_compact_on_start": "false", 2024-09-13T13:02:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-13T13:02:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-13T13:02:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-13T13:02:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_crush_update_on_start": "true", 2024-09-13T13:02:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_crush_update_weight_set": "true", 2024-09-13T13:02:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-13T13:02:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-13T13:02:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-13T13:02:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-13T13:02:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-13T13:02:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_feed_pullee": "-1", 2024-09-13T13:02:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-13T13:02:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-13T13:02:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-13T13:02:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_misdirected_ops": "false", 2024-09-13T13:02:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_no_acting_change": "false", 2024-09-13T13:02:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_no_purge_strays": "false", 2024-09-13T13:02:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_op_order": "false", 2024-09-13T13:02:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-13T13:02:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-13T13:02:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-13T13:02:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-13T13:02:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_shutdown": "false", 2024-09-13T13:02:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-13T13:02:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-13T13:02:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_trim_objects": "false", 2024-09-13T13:02:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-13T13:02:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-13T13:02:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_verify_snaps": "false", 2024-09-13T13:02:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-13T13:02:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-13T13:02:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_deep_scrub_keys": "1024", 2024-09-13T13:02:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-13T13:02:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-13T13:02:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-13T13:02:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_deep_scrub_stride": "524288", 2024-09-13T13:02:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-13T13:02:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-13T13:02:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_default_notify_timeout": "30", 2024-09-13T13:02:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_delete_sleep": "0.000000", 2024-09-13T13:02:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-13T13:02:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-13T13:02:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-13T13:02:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_discard_disconnected_ops": "true", 2024-09-13T13:02:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_ec_partial_reads": "true", 2024-09-13T13:02:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_enable_op_tracker": "true", 2024-09-13T13:02:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-13T13:02:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-13T13:02:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-13T13:02:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-13T13:02:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_fast_info": "true", 2024-09-13T13:02:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_fast_shutdown": "true", 2024-09-13T13:02:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-13T13:02:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-13T13:02:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-13T13:02:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-13T13:02:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-13T13:02:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_function_tracing": "false", 2024-09-13T13:02:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_heartbeat_grace": "20", 2024-09-13T13:02:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_heartbeat_interval": "6", 2024-09-13T13:02:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-13T13:02:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_heartbeat_min_peers": "10", 2024-09-13T13:02:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_heartbeat_min_size": "2000", 2024-09-13T13:02:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_heartbeat_stale": "600", 2024-09-13T13:02:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-13T13:02:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_hit_set_max_size": "100000", 2024-09-13T13:02:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_hit_set_min_size": "1000", 2024-09-13T13:02:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-13T13:02:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-13T13:02:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-13T13:02:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-13T13:02:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-13T13:02:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-13T13:02:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_journal_size": "5120", 2024-09-13T13:02:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_kill_backfill_at": "0", 2024-09-13T13:02:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-13T13:02:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_map_cache_size": "50", 2024-09-13T13:02:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_map_dedup": "true", 2024-09-13T13:02:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_map_message_max": "40", 2024-09-13T13:02:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-13T13:02:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_map_share_max_epochs": "40", 2024-09-13T13:02:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_attr_name_len": "100", 2024-09-13T13:02:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_attr_size": "0", 2024-09-13T13:02:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_backfills": "1", 2024-09-13T13:02:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_markdown_count": "5", 2024-09-13T13:02:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_markdown_period": "600", 2024-09-13T13:02:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_object_name_len": "2048", 2024-09-13T13:02:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_object_namespace_len": "256", 2024-09-13T13:02:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_object_size": "134217728", 2024-09-13T13:02:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-13T13:02:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-13T13:02:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_pg_blocked_by": "16", 2024-09-13T13:02:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_pg_log_entries": "10000", 2024-09-13T13:02:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-13T13:02:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_pgls": "1024", 2024-09-13T13:02:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_push_cost": "8388608", 2024-09-13T13:02:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_push_objects": "10", 2024-09-13T13:02:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_scrubs": "3", 2024-09-13T13:02:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-13T13:02:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_trimming_pgs": "2", 2024-09-13T13:02:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_write_op_reply_len": "64", 2024-09-13T13:02:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_max_write_size": "90", 2024-09-13T13:02:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-13T13:02:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-13T13:02:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-13T13:02:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-13T13:02:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-13T13:02:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-13T13:02:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-13T13:02:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-13T13:02:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_profile": "balanced", 2024-09-13T13:02:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-13T13:02:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-13T13:02:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_memory_base": "805306368", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_memory_cache_min": "134217728", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_memory_target": "4294967296", 2024-09-13T13:02:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_memory_target_autotune": "false", 2024-09-13T13:02:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-13T13:02:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_min_pg_log_entries": "250", 2024-09-13T13:02:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_min_recovery_priority": "0", 2024-09-13T13:02:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-13T13:02:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-13T13:02:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mon_report_interval": "5", 2024-09-13T13:02:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-13T13:02:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-13T13:02:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_num_cache_shards": "32", 2024-09-13T13:02:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_num_op_tracker_shard": "32", 2024-09-13T13:02:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_numa_auto_affinity": "true", 2024-09-13T13:02:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_numa_node": "-1", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_numa_prefer_iface": "true", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_objecter_finishers": "1", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_objectstore": "bluestore", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_objectstore_fuse": "false", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_objectstore_tracing": "false", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_complaint_time": "30.000000", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_history_duration": "600", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_history_size": "20", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_history_slow_op_size": "20", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_log_threshold": "5", 2024-09-13T13:02:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_num_shards": "0", 2024-09-13T13:02:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_num_shards_hdd": "5", 2024-09-13T13:02:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_num_shards_ssd": "8", 2024-09-13T13:02:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-13T13:02:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-13T13:02:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-13T13:02:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-13T13:02:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_pq_min_cost": "65536", 2024-09-13T13:02:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-13T13:02:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_queue_cut_off": "high", 2024-09-13T13:02:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-13T13:02:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_op_thread_timeout": "15", 2024-09-13T13:02:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_open_classes_on_start": "true", 2024-09-13T13:02:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_os_flags": "0", 2024-09-13T13:02:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_peering_op_priority": "255", 2024-09-13T13:02:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pg_delete_cost": "1048576", 2024-09-13T13:02:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pg_delete_priority": "5", 2024-09-13T13:02:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-13T13:02:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-13T13:02:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-13T13:02:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pg_log_trim_max": "10000", 2024-09-13T13:02:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pg_log_trim_min": "100", 2024-09-13T13:02:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-13T13:02:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-13T13:02:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-13T13:02:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-13T13:02:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-13T13:02:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-13T13:02:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-13T13:02:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-13T13:02:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-13T13:02:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-13T13:02:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_crimson": "false", 2024-09-13T13:02:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-13T13:02:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-13T13:02:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-13T13:02:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-13T13:02:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-13T13:02:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-13T13:02:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-13T13:02:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-13T13:02:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_flags": "0", 2024-09-13T13:02:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-13T13:02:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_min_size": "0", 2024-09-13T13:02:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-13T13:02:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_pg_num": "32", 2024-09-13T13:02:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_pgp_num": "0", 2024-09-13T13:02:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-13T13:02:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_read_ratio": "70", 2024-09-13T13:02:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_size": "3", 2024-09-13T13:02:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_default_type": "replicated", 2024-09-13T13:02:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-13T13:02:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-13T13:02:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_push_per_object_cost": "1000", 2024-09-13T13:02:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-13T13:02:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recover_clone_overlap": "true", 2024-09-13T13:02:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-13T13:02:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_cost": "20971520", 2024-09-13T13:02:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-13T13:02:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_max_active": "0", 2024-09-13T13:02:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-13T13:02:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-13T13:02:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-13T13:02:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-13T13:02:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_max_single_start": "1", 2024-09-13T13:02:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_op_priority": "3", 2024-09-13T13:02:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-13T13:02:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_priority": "5", 2024-09-13T13:02:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-13T13:02:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_sleep": "0.000000", 2024-09-13T13:02:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-13T13:02:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-13T13:02:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-13T13:02:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_repair_during_recovery": "false", 2024-09-13T13:02:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_requested_scrub_priority": "120", 2024-09-13T13:02:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-13T13:02:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-13T13:02:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_auto_repair": "false", 2024-09-13T13:02:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-13T13:02:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-13T13:02:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_begin_hour": "0", 2024-09-13T13:02:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_begin_week_day": "0", 2024-09-13T13:02:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_chunk_max": "25", 2024-09-13T13:02:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_chunk_min": "5", 2024-09-13T13:02:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_cost": "52428800", 2024-09-13T13:02:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-13T13:02:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_during_recovery": "false", 2024-09-13T13:02:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_end_hour": "0", 2024-09-13T13:02:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_end_week_day": "0", 2024-09-13T13:02:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_event_cost": "4096", 2024-09-13T13:02:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-13T13:02:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-13T13:02:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_invalid_stats": "true", 2024-09-13T13:02:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-13T13:02:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-13T13:02:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_max_preemptions": "5", 2024-09-13T13:02:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-13T13:02:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_priority": "5", 2024-09-13T13:02:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-13T13:02:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_retry_delay": "30", 2024-09-13T13:02:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-13T13:02:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-13T13:02:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_retry_trimming": "10", 2024-09-13T13:02:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_scrub_sleep": "0.000000", 2024-09-13T13:02:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_skip_data_digest": "false", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_smart_report_timeout": "5", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_snap_trim_cost": "1048576", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_snap_trim_priority": "5", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-13T13:02:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-13T13:02:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_target_transaction_size": "30", 2024-09-13T13:02:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-13T13:02:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-13T13:02:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-13T13:02:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-13T13:02:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-13T13:02:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-13T13:02:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-13T13:02:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-13T13:02:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-13T13:02:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-13T13:02:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_tracing": "false", 2024-09-13T13:02:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_use_stale_snap": "false", 2024-09-13T13:02:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-13T13:02:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "osdc_blkin_trace_all": "false", 2024-09-13T13:02:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "paxos_kill_at": "0", 2024-09-13T13:02:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "paxos_max_join_drift": "10", 2024-09-13T13:02:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "paxos_min": "500", 2024-09-13T13:02:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "paxos_min_wait": "0.050000", 2024-09-13T13:02:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "paxos_propose_interval": "1.000000", 2024-09-13T13:02:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "paxos_service_trim_max": "500", 2024-09-13T13:02:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-13T13:02:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "paxos_service_trim_min": "10", 2024-09-13T13:02:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "paxos_stash_full_interval": "25", 2024-09-13T13:02:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "paxos_trim_max": "500", 2024-09-13T13:02:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "paxos_trim_min": "250", 2024-09-13T13:02:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "perf": "true", 2024-09-13T13:02:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "pid_file": "", 2024-09-13T13:02:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-13T13:02:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-13T13:02:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "-", 2024-09-13T13:02:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrv": "", 2024-09-13T13:02:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "public_bind_addr": "-", 2024-09-13T13:02:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "public_network": "", 2024-09-13T13:02:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "public_network_interface": "", 2024-09-13T13:02:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "qat_compressor_busy_polling": "false", 2024-09-13T13:02:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "qat_compressor_enabled": "false", 2024-09-13T13:02:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "qat_compressor_session_max_number": "256", 2024-09-13T13:02:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "rados_mon_op_timeout": "0", 2024-09-13T13:02:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "rados_osd_op_timeout": "0", 2024-09-13T13:02:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "rados_tracing": "false", 2024-09-13T13:02:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_atime_update_interval": "60", 2024-09-13T13:02:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-13T13:02:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_balance_parent_reads": "false", 2024-09-13T13:02:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_balance_snap_reads": "false", 2024-09-13T13:02:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_blkin_trace_all": "false", 2024-09-13T13:02:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-13T13:02:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-13T13:02:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_cache": "true", 2024-09-13T13:02:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-13T13:02:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-13T13:02:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-13T13:02:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-13T13:02:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_cache_policy": "writearound", 2024-09-13T13:02:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_cache_size": "33554432", 2024-09-13T13:02:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-13T13:02:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-13T13:02:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_clone_copy_on_read": "false", 2024-09-13T13:02:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_compression_hint": "none", 2024-09-13T13:02:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_concurrent_management_ops": "10", 2024-09-13T13:02:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-13T13:02:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_default_clone_format": "auto", 2024-09-13T13:02:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_default_data_pool": "", 2024-09-13T13:02:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_default_features": "61", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_default_format": "2", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_default_map_options": "", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_default_order": "22", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_default_pool": "rbd", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_default_stripe_count": "0", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_default_stripe_unit": "0", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_enable_alloc_hint": "true", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_io_scheduler": "simple", 2024-09-13T13:02:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-13T13:02:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-13T13:02:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-13T13:02:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-13T13:02:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-13T13:02:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-13T13:02:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-13T13:02:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-13T13:02:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-13T13:02:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_journal_order": "24", 2024-09-13T13:02:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_journal_pool": "", 2024-09-13T13:02:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_journal_splay_width": "4", 2024-09-13T13:02:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_localize_parent_reads": "false", 2024-09-13T13:02:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_localize_snap_reads": "false", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-13T13:02:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-13T13:02:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_mtime_update_interval": "60", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_non_blocking_aio": "true", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_op_thread_timeout": "60", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_op_threads": "1", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_parent_cache_enabled": "false", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_plugins": "", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_bps_burst": "0", 2024-09-13T13:02:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_bps_limit": "0", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_exclude_ops": "0", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_iops_burst": "0", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_iops_limit": "0", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-13T13:02:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_read_from_replica_policy": "default", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_skip_partial_discard": "true", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_tracing": "false", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_validate_names": "true", 2024-09-13T13:02:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_validate_pool": "true", 2024-09-13T13:02:01.076 INFO:teuthology.orchestra.run.smithi052.stdout: "restapi_base_url": "", 2024-09-13T13:02:01.076 INFO:teuthology.orchestra.run.smithi052.stdout: "restapi_log_level": "", 2024-09-13T13:02:01.076 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-13T13:02:01.076 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-13T13:02:01.076 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_acl_grants_max_num": "100", 2024-09-13T13:02:01.076 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_admin_entry": "admin", 2024-09-13T13:02:01.076 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-13T13:02:01.076 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_asio_assert_yielding": "false", 2024-09-13T13:02:01.076 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_backend_store": "rados", 2024-09-13T13:02:01.076 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_barbican_url": "", 2024-09-13T13:02:01.076 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_beast_enable_async": "true", 2024-09-13T13:02:01.076 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_bucket_counters_cache": "false", 2024-09-13T13:02:01.077 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-13T13:02:01.077 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-13T13:02:01.077 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-13T13:02:01.077 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-13T13:02:01.077 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-13T13:02:01.077 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-13T13:02:01.077 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-13T13:02:01.077 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-13T13:02:01.077 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_cache_enabled": "true", 2024-09-13T13:02:01.077 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_cache_expiry_interval": "900", 2024-09-13T13:02:01.077 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_cache_lru_size": "25000", 2024-09-13T13:02:01.077 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_config_store": "rados", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_content_length_compat": "false", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_copy_obj_progress": "true", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_cors_rules_max_num": "100", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_cross_domain_policy": "", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_kmip_addr": "", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_kmip_password": "", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-13T13:02:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_kmip_username": "", 2024-09-13T13:02:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_require_ssl": "true", 2024-09-13T13:02:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-13T13:02:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-13T13:02:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-13T13:02:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-13T13:02:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-13T13:02:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-13T13:02:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-13T13:02:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-13T13:02:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-13T13:02:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-13T13:02:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-13T13:02:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-13T13:02:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-13T13:02:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-13T13:02:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-13T13:02:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_vault_addr": "", 2024-09-13T13:02:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_vault_auth": "token", 2024-09-13T13:02:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_vault_namespace": "", 2024-09-13T13:02:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_vault_prefix": "", 2024-09-13T13:02:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-13T13:02:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-13T13:02:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-13T13:02:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-13T13:02:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_vault_token_file": "", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_curl_buffersize": "524288", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_curl_low_speed_time": "300", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-13T13:02:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-13T13:02:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-13T13:02:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-13T13:02:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-13T13:02:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-13T13:02:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-13T13:02:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-13T13:02:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-13T13:02:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-13T13:02:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-13T13:02:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_data_log_changes_size": "1000", 2024-09-13T13:02:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_data_log_num_shards": "128", 2024-09-13T13:02:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-13T13:02:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_data_log_window": "30", 2024-09-13T13:02:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-13T13:02:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-13T13:02:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-13T13:02:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-13T13:02:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-13T13:02:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-13T13:02:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-13T13:02:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-13T13:02:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-13T13:02:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-13T13:02:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-13T13:02:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-13T13:02:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_disable_s3select": "false", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-13T13:02:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dns_name": "", 2024-09-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dns_s3website_name": "", 2024-09-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dynamic_resharding": "true", 2024-09-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_enable_gc_threads": "true", 2024-09-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_enable_lc_threads": "true", 2024-09-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_enable_mdsearch": "true", 2024-09-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_enable_ops_log": "false", 2024-09-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_enable_quota_threads": "true", 2024-09-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_enable_static_website": "false", 2024-09-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_enable_usage_log": "false", 2024-09-13T13:02:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_enforce_swift_acls": "true", 2024-09-13T13:02:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_exit_timeout_secs": "120", 2024-09-13T13:02:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_expose_bucket": "false", 2024-09-13T13:02:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_extended_http_attrs": "", 2024-09-13T13:02:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_filter": "none", 2024-09-13T13:02:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-13T13:02:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_frontends": "beast port=7480", 2024-09-13T13:02:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-13T13:02:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_gc_max_deferred": "50", 2024-09-13T13:02:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-13T13:02:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_gc_max_objs": "32", 2024-09-13T13:02:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-13T13:02:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-13T13:02:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_gc_processor_period": "3600", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_init_timeout": "300", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_kafka_connection_idle": "300", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-13T13:02:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-13T13:02:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-13T13:02:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_admin_domain": "", 2024-09-13T13:02:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_admin_password": "", 2024-09-13T13:02:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_admin_password_path": "", 2024-09-13T13:02:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_admin_project": "", 2024-09-13T13:02:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_admin_tenant": "", 2024-09-13T13:02:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_admin_token": "", 2024-09-13T13:02:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_admin_token_path": "", 2024-09-13T13:02:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_admin_user": "", 2024-09-13T13:02:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_barbican_domain": "", 2024-09-13T13:02:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_barbican_password": "", 2024-09-13T13:02:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_barbican_project": "", 2024-09-13T13:02:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-13T13:02:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_barbican_user": "", 2024-09-13T13:02:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-13T13:02:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-13T13:02:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-13T13:02:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-13T13:02:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-13T13:02:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_url": "", 2024-09-13T13:02:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-13T13:02:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_lc_debug_interval": "-1", 2024-09-13T13:02:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_lc_lock_max_time": "90", 2024-09-13T13:02:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_lc_max_objs": "32", 2024-09-13T13:02:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_lc_max_rules": "1000", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_lc_max_worker": "3", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_lc_thread_delay": "0", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_ldap_dnattr": "uid", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_ldap_searchfilter": "", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_ldap_uri": "", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-13T13:02:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_log_http_headers": "", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_log_object_name_utc": "false", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_max_attr_name_len": "0", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_max_attr_size": "0", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_max_chunk_size": "4194304", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_max_listing_results": "5000", 2024-09-13T13:02:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_max_notify_retries": "10", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_max_put_param_size": "1048576", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_max_put_size": "5368709120", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_max_slo_entries": "1000", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_md_log_max_shards": "64", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_mp_lock_max_time": "600", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-13T13:02:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-13T13:02:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-13T13:02:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-13T13:02:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-13T13:02:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-13T13:02:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_nfs_max_gc": "300", 2024-09-13T13:02:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-13T13:02:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-13T13:02:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-13T13:02:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-13T13:02:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-13T13:02:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-13T13:02:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-13T13:02:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_num_async_rados_threads": "32", 2024-09-13T13:02:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_num_control_oids": "8", 2024-09-13T13:02:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_numa_node": "-1", 2024-09-13T13:02:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-13T13:02:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-13T13:02:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_objexp_chunk_size": "100", 2024-09-13T13:02:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_objexp_gc_interval": "600", 2024-09-13T13:02:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-13T13:02:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-13T13:02:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-13T13:02:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_op_thread_timeout": "600", 2024-09-13T13:02:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_op_tracing": "false", 2024-09-13T13:02:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_opa_token": "", 2024-09-13T13:02:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_opa_url": "", 2024-09-13T13:02:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_opa_verify_ssl": "true", 2024-09-13T13:02:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-13T13:02:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-13T13:02:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_ops_log_rados": "false", 2024-09-13T13:02:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_ops_log_socket_path": "", 2024-09-13T13:02:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-13T13:02:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-13T13:02:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-13T13:02:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_period_push_interval": "2.000000", 2024-09-13T13:02:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-13T13:02:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-13T13:02:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-13T13:02:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-13T13:02:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_posix_cache_lanes": "3", 2024-09-13T13:02:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-13T13:02:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-13T13:02:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_posix_cache_partitions": "3", 2024-09-13T13:02:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-13T13:02:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_print_continue": "true", 2024-09-13T13:02:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-13T13:02:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-13T13:02:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-13T13:02:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-13T13:02:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-13T13:02:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_rados_tracing": "false", 2024-09-13T13:02:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_realm": "", 2024-09-13T13:02:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_realm_id": "", 2024-09-13T13:02:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-13T13:02:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_region": "", 2024-09-13T13:02:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-13T13:02:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-13T13:02:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-13T13:02:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-13T13:02:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_request_uri": "", 2024-09-13T13:02:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_reshard_batch_size": "64", 2024-09-13T13:02:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-13T13:02:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-13T13:02:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_reshard_max_aio": "128", 2024-09-13T13:02:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_reshard_num_logs": "16", 2024-09-13T13:02:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-13T13:02:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-13T13:02:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_reshard_thread_interval": "600", 2024-09-13T13:02:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-13T13:02:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_resolve_cname": "false", 2024-09-13T13:02:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-13T13:02:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_run_sync_thread": "true", 2024-09-13T13:02:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-13T13:02:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-13T13:02:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-13T13:02:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-13T13:02:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_scheduler_type": "throttler", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_script_uri": "", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_service_provider_name": "", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sts_client_id": "", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sts_client_secret": "", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sts_entry": "sts", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sts_key": "sts", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-13T13:02:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sts_min_session_duration": "900", 2024-09-13T13:02:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sts_token_introspection_url": "", 2024-09-13T13:02:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_swift_account_in_url": "false", 2024-09-13T13:02:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_swift_auth_entry": "auth", 2024-09-13T13:02:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_swift_auth_url": "", 2024-09-13T13:02:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_swift_custom_header": "", 2024-09-13T13:02:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-13T13:02:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_swift_need_stats": "true", 2024-09-13T13:02:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_swift_tenant_name": "", 2024-09-13T13:02:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_swift_token_expiration": "86400", 2024-09-13T13:02:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_swift_url": "", 2024-09-13T13:02:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_swift_url_prefix": "swift", 2024-09-13T13:02:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-13T13:02:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-13T13:02:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-13T13:02:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sync_lease_period": "120", 2024-09-13T13:02:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-13T13:02:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-13T13:02:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-13T13:02:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-13T13:02:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-13T13:02:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-13T13:02:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-13T13:02:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-13T13:02:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-13T13:02:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_thread_pool_size": "512", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_torrent_comment": "", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_torrent_createby": "", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_torrent_encoding": "", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_torrent_flag": "false", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_torrent_origin": "", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_torrent_tracker": "", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_trust_forwarded_https": "false", 2024-09-13T13:02:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-13T13:02:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-13T13:02:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_usage_max_shards": "32", 2024-09-13T13:02:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_usage_max_user_shards": "1", 2024-09-13T13:02:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_use_opa_authz": "false", 2024-09-13T13:02:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_user_counters_cache": "false", 2024-09-13T13:02:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-13T13:02:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-13T13:02:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-13T13:02:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_user_max_buckets": "1000", 2024-09-13T13:02:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_user_policies_max_num": "100", 2024-09-13T13:02:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-13T13:02:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-13T13:02:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-13T13:02:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-13T13:02:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_user_unique_email": "true", 2024-09-13T13:02:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_verify_ssl": "true", 2024-09-13T13:02:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-13T13:02:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_zone": "", 2024-09-13T13:02:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_zone_id": "", 2024-09-13T13:02:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-13T13:02:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_zonegroup": "", 2024-09-13T13:02:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_zonegroup_id": "", 2024-09-13T13:02:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-13T13:02:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "rgwlc_auto_session_clear": "true", 2024-09-13T13:02:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-13T13:02:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_block_size": "4096", 2024-09-13T13:02:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-13T13:02:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-13T13:02:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-13T13:02:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-13T13:02:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-13T13:02:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_cache_size": "536870912", 2024-09-13T13:02:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-13T13:02:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-13T13:02:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-13T13:02:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-13T13:02:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-13T13:02:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-13T13:02:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_index_type": "binary_search", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_partition_filters": "false", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_perf": "false", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "run_dir": "/var/run/ceph", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_block_create": "true", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_cache_lru_size": "67108864", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-13T13:02:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_default_max_object_size": "16777216", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_device_size": "53687091200", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_full_integrity_check": "false", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_journal_batch_capacity": "16", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_main_device_type": "SSD", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-13T13:02:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-13T13:02:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-13T13:02:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-13T13:02:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "seastore_segment_size": "67108864", 2024-09-13T13:02:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "set_keepcaps": "false", 2024-09-13T13:02:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "setgroup": "ceph", 2024-09-13T13:02:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "setuser": "ceph", 2024-09-13T13:02:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "setuser_match_path": "", 2024-09-13T13:02:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-13T13:02:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "thp": "false", 2024-09-13T13:02:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "threadpool_default_timeout": "60", 2024-09-13T13:02:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-13T13:02:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "throttler_perf_counter": "true", 2024-09-13T13:02:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "tmp_dir": "/tmp", 2024-09-13T13:02:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-13T13:02:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "uadk_compressor_enabled": "false", 2024-09-13T13:02:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-13T13:02:01.109 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:02:01.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:01 smithi167 ceph-mon[33178]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T13:02:01.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:01 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:02:01.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:01 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:02:01.884 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:01 smithi052 ceph-mon[36682]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T13:02:01.884 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:01 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:02:01.884 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:01 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:02:01.884 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:01 smithi052 ceph-mon[29471]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T13:02:01.885 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:01 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:02:01.885 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:01 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:02:01.918 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-13T13:02:02.179 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:02.554 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:02 smithi052 ceph-mon[36682]: osdmap e80: 8 total, 8 up, 8 in 2024-09-13T13:02:02.554 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:02 smithi052 ceph-mon[36682]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:02:02.554 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:02 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:02:02.554 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:02 smithi052 ceph-mon[29471]: osdmap e80: 8 total, 8 up, 8 in 2024-09-13T13:02:02.554 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:02 smithi052 ceph-mon[29471]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:02:02.554 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:02 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:02:02.647 INFO:teuthology.orchestra.run.smithi052.stdout:mon.a: {} 2024-09-13T13:02:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T13:02:02.656 INFO:teuthology.orchestra.run.smithi052.stdout:mon.b: {} 2024-09-13T13:02:02.656 INFO:teuthology.orchestra.run.smithi052.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T13:02:02.664 INFO:teuthology.orchestra.run.smithi052.stdout:mon.c: {} 2024-09-13T13:02:02.664 INFO:teuthology.orchestra.run.smithi052.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T13:02:02.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:02 smithi167 ceph-mon[33178]: osdmap e80: 8 total, 8 up, 8 in 2024-09-13T13:02:02.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:02 smithi167 ceph-mon[33178]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:02:02.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:02 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:02:03.262 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-13T13:02:03.287 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T13:02:03.291 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T13:02:03.292 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-13T13:02:03.292 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-13T13:02:03.292 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-13T13:02:03.292 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T13:02:03.293 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:02:03.293 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-13T13:02:03.293 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-13T13:02:03.293 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-13T13:02:03.294 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:03.294 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-13T13:02:03.294 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:02:03.295 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-13T13:02:03.295 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T13:02:03.295 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-13T13:02:03.295 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:02:03.327 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:03.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:03.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:03.458 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:03.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:03.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:02:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:02:03.561 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:03.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:03.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:03.608 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:03.610 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:03.616 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:03.687 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:03.709 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:03.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:03.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:03.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:03 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3548788392' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T13:02:03.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:03 smithi167 ceph-mon[33178]: 3.e scrub starts 2024-09-13T13:02:03.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:03 smithi167 ceph-mon[33178]: 3.e scrub ok 2024-09-13T13:02:03.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:03 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:03.833 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:03.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:03.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:03.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:03 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3548788392' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T13:02:03.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:03 smithi052 ceph-mon[36682]: 3.e scrub starts 2024-09-13T13:02:03.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:03 smithi052 ceph-mon[36682]: 3.e scrub ok 2024-09-13T13:02:03.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:03 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:03.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:03 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3548788392' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T13:02:03.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:03 smithi052 ceph-mon[29471]: 3.e scrub starts 2024-09-13T13:02:03.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:03 smithi052 ceph-mon[29471]: 3.e scrub ok 2024-09-13T13:02:03.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:03 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:03.935 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:03.957 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:03.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:04.059 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:04.082 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:04.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:04.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:04.184 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:04.193 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:02:04.193 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:02:04.206 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:04.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:04.211 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:02:04.211 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:02:04.229 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:02:04.229 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:02:04.246 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:02:04.246 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:02:04.262 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:02:04.263 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:02:04.279 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:02:04.279 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:02:04.298 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:02:04.299 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:02:04.308 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:04.320 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:02:04.320 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:02:04.334 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:04.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:04.436 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:04.459 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:04.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:04.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:04.561 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:04.583 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:04.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:04.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:04.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:04.707 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:04.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:04.792 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:04 smithi167 ceph-mon[33178]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:02:04.793 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:04 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2815736167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:04.793 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:04 smithi167 ceph-mon[33178]: 3.2 scrub starts 2024-09-13T13:02:04.793 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:04 smithi167 ceph-mon[33178]: 3.2 scrub ok 2024-09-13T13:02:04.808 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:04.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:04.831 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:04.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:04.833 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:04 smithi052 ceph-mon[36682]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:02:04.833 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:04 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2815736167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:04.833 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:04 smithi052 ceph-mon[36682]: 3.2 scrub starts 2024-09-13T13:02:04.833 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:04 smithi052 ceph-mon[36682]: 3.2 scrub ok 2024-09-13T13:02:04.834 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:04 smithi052 ceph-mon[29471]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:02:04.834 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:04 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2815736167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:04.834 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:04 smithi052 ceph-mon[29471]: 3.2 scrub starts 2024-09-13T13:02:04.834 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:04 smithi052 ceph-mon[29471]: 3.2 scrub ok 2024-09-13T13:02:04.834 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:02:04 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[68036]: ts=2024-09-13T13:02:04.748Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.000888721s 2024-09-13T13:02:04.842 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:02:04.842 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:02:04.932 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:04.953 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:04.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:04.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:05.055 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:05.077 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:05.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:05.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:05.179 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:05.202 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:05.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:05.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:05.324 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:05.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:05.426 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:05.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:05.448 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:05.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:05.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:05.573 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:05.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:05.602 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T13:02:05.675 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:05.697 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:05.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:05.799 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:05.823 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:05.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:05.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:05.954 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:05.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:06.056 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:06.079 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:06.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:06.170 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:02:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:02:06.181 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:06.205 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:06.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:06.301 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 systemd[1]: Starting Ceph grafana.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:02:06.307 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:06.330 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:06.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:06.345 INFO:teuthology.orchestra.run.smithi052.stderr:no valid command found; 10 closest matches: 2024-09-13T13:02:06.345 INFO:teuthology.orchestra.run.smithi052.stderr:0 2024-09-13T13:02:06.345 INFO:teuthology.orchestra.run.smithi052.stderr:1 2024-09-13T13:02:06.345 INFO:teuthology.orchestra.run.smithi052.stderr:2 2024-09-13T13:02:06.345 INFO:teuthology.orchestra.run.smithi052.stderr:abort 2024-09-13T13:02:06.345 INFO:teuthology.orchestra.run.smithi052.stderr:assert 2024-09-13T13:02:06.345 INFO:teuthology.orchestra.run.smithi052.stderr:bench [] [] [] [] 2024-09-13T13:02:06.345 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs debug_inject_read_zeros 2024-09-13T13:02:06.345 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs files list 2024-09-13T13:02:06.346 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs stats 2024-09-13T13:02:06.346 INFO:teuthology.orchestra.run.smithi052.stderr:bluestore allocator dump block 2024-09-13T13:02:06.346 INFO:teuthology.orchestra.run.smithi052.stderr:Error EINVAL: invalid command 2024-09-13T13:02:06.432 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:06.454 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:06.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:06.505 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:06.556 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:06.577 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:06.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:06.581 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 podman[60780]: 2024-09-13 13:02:06.203369685 +0000 UTC m=+0.016569901 image pull c8b91775d855b99270fc5d22f3c6737e8cca01ef4c25c8b0362295e0746fa39b quay.io/ceph/grafana:10.4.0 2024-09-13T13:02:06.582 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 podman[60780]: 2024-09-13 13:02:06.382702855 +0000 UTC m=+0.195903070 container create e7993e166a4d71ddd2dc76860cc84404ee0818305461bd91dc884432945a27ee (image=quay.io/ceph/grafana:10.4.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T13:02:06.679 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:06.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:06.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:06.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:06.805 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:06.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:06.827 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:06.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:06.834 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 podman[60780]: 2024-09-13 13:02:06.610343228 +0000 UTC m=+0.423543442 container init e7993e166a4d71ddd2dc76860cc84404ee0818305461bd91dc884432945a27ee (image=quay.io/ceph/grafana:10.4.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T13:02:06.834 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 podman[60780]: 2024-09-13 13:02:06.613292092 +0000 UTC m=+0.426492306 container start e7993e166a4d71ddd2dc76860cc84404ee0818305461bd91dc884432945a27ee (image=quay.io/ceph/grafana:10.4.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T13:02:06.834 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 bash[60780]: e7993e166a4d71ddd2dc76860cc84404ee0818305461bd91dc884432945a27ee 2024-09-13T13:02:06.835 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 systemd[1]: Started Ceph grafana.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:02:06.835 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741463752Z level=info msg="Starting Grafana" version=10.4.0 commit=03f502a94d17f7dc4e6c34acdf8428aedd986e4c branch=HEAD compiled=2024-09-13T13:02:06Z 2024-09-13T13:02:06.835 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741704605Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-13T13:02:06.835 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741713494Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-13T13:02:06.835 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741718516Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-13T13:02:06.835 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741723085Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-13T13:02:06.835 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741727323Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-13T13:02:06.835 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741731708Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-13T13:02:06.835 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741736459Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-13T13:02:06.835 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741741129Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-13T13:02:06.835 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741746298Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-13T13:02:06.836 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741750776Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-13T13:02:06.836 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741755287Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-13T13:02:06.836 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741760389Z level=info msg=Target target=[all] 2024-09-13T13:02:06.836 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741769413Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-13T13:02:06.837 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741774057Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-13T13:02:06.837 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741778488Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-13T13:02:06.837 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741782858Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-13T13:02:06.837 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741797876Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-13T13:02:06.837 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=settings t=2024-09-13T13:02:06.741802686Z level=info msg="App mode production" 2024-09-13T13:02:06.838 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=sqlstore t=2024-09-13T13:02:06.742044862Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-13T13:02:06.838 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=sqlstore t=2024-09-13T13:02:06.74206134Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-13T13:02:06.838 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.742566425Z level=info msg="Starting DB migrations" 2024-09-13T13:02:06.838 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.743424991Z level=info msg="Executing migration" id="create migration_log table" 2024-09-13T13:02:06.838 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.744292267Z level=info msg="Migration successfully executed" id="create migration_log table" duration=865.347µs 2024-09-13T13:02:06.838 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.74474962Z level=info msg="Executing migration" id="create user table" 2024-09-13T13:02:06.838 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.745284654Z level=info msg="Migration successfully executed" id="create user table" duration=534.964µs 2024-09-13T13:02:06.838 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.745710471Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-13T13:02:06.838 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.746381451Z level=info msg="Migration successfully executed" id="add unique index user.login" duration=672.54µs 2024-09-13T13:02:06.838 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.746827444Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-13T13:02:06.839 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.747282064Z level=info msg="Migration successfully executed" id="add unique index user.email" duration=454.689µs 2024-09-13T13:02:06.839 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.747677137Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-13T13:02:06.839 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.748142124Z level=info msg="Migration successfully executed" id="drop index UQE_user_login - v1" duration=465.143µs 2024-09-13T13:02:06.839 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.748580531Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-13T13:02:06.839 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.749018218Z level=info msg="Migration successfully executed" id="drop index UQE_user_email - v1" duration=437.618µs 2024-09-13T13:02:06.839 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.749450722Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-13T13:02:06.839 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.751544888Z level=info msg="Migration successfully executed" id="Rename table user to user_v1 - v1" duration=2.093148ms 2024-09-13T13:02:06.839 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.752039234Z level=info msg="Executing migration" id="create user table v2" 2024-09-13T13:02:06.840 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.752759808Z level=info msg="Migration successfully executed" id="create user table v2" duration=719.954µs 2024-09-13T13:02:06.840 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.753145159Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-13T13:02:06.840 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.753676573Z level=info msg="Migration successfully executed" id="create index UQE_user_login - v2" duration=530.378µs 2024-09-13T13:02:06.840 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.754122951Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-13T13:02:06.840 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.754716795Z level=info msg="Migration successfully executed" id="create index UQE_user_email - v2" duration=593.149µs 2024-09-13T13:02:06.840 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.755210153Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-13T13:02:06.840 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.755504752Z level=info msg="Migration successfully executed" id="copy data_source v1 to v2" duration=294.811µs 2024-09-13T13:02:06.840 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.755853521Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-13T13:02:06.840 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.756232539Z level=info msg="Migration successfully executed" id="Drop old table user_v1" duration=378.767µs 2024-09-13T13:02:06.840 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.756738199Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-13T13:02:06.841 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.757472924Z level=info msg="Migration successfully executed" id="Add column help_flags1 to user table" duration=732.13µs 2024-09-13T13:02:06.841 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.757833483Z level=info msg="Executing migration" id="Update user table charset" 2024-09-13T13:02:06.841 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.757853467Z level=info msg="Migration successfully executed" id="Update user table charset" duration=20.257µs 2024-09-13T13:02:06.841 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.758663739Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-13T13:02:06.841 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.759369728Z level=info msg="Migration successfully executed" id="Add last_seen_at column to user" duration=705.601µs 2024-09-13T13:02:06.841 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.75971823Z level=info msg="Executing migration" id="Add missing user data" 2024-09-13T13:02:06.841 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.759870978Z level=info msg="Migration successfully executed" id="Add missing user data" duration=152.73µs 2024-09-13T13:02:06.842 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.760739116Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-13T13:02:06.842 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.761497666Z level=info msg="Migration successfully executed" id="Add is_disabled column to user" duration=760.598µs 2024-09-13T13:02:06.842 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.761884445Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-13T13:02:06.842 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.762355515Z level=info msg="Migration successfully executed" id="Add index user.login/user.email" duration=471.045µs 2024-09-13T13:02:06.842 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.762712073Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-13T13:02:06.842 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.763433257Z level=info msg="Migration successfully executed" id="Add is_service_account column to user" duration=720.848µs 2024-09-13T13:02:06.842 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.763781464Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-13T13:02:06.843 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.768943046Z level=info msg="Migration successfully executed" id="Update is_service_account column to nullable" duration=5.161315ms 2024-09-13T13:02:06.843 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.769290362Z level=info msg="Executing migration" id="Add uid column to user" 2024-09-13T13:02:06.843 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.77002107Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=728.266µs 2024-09-13T13:02:06.843 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.770411546Z level=info msg="Executing migration" id="Update uid column values for users" 2024-09-13T13:02:06.843 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.770540122Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=129.894µs 2024-09-13T13:02:06.843 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.771307763Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-09-13T13:02:06.843 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.771757969Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=449.927µs 2024-09-13T13:02:06.843 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.77437468Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-13T13:02:06.843 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.774907509Z level=info msg="Migration successfully executed" id="create temp user table v1-7" duration=532.995µs 2024-09-13T13:02:06.843 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.777473993Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-13T13:02:06.843 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.778002003Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v1-7" duration=530.539µs 2024-09-13T13:02:06.844 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.78023713Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-13T13:02:06.844 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.7806979Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v1-7" duration=463.017µs 2024-09-13T13:02:06.844 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.782957394Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-13T13:02:06.844 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.783437559Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v1-7" duration=480.351µs 2024-09-13T13:02:06.844 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.785692166Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-13T13:02:06.844 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.786165781Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v1-7" duration=471.567µs 2024-09-13T13:02:06.844 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.788554071Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-13T13:02:06.844 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.788586311Z level=info msg="Migration successfully executed" id="Update temp_user table charset" duration=33.046µs 2024-09-13T13:02:06.844 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.791367573Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-13T13:02:06.844 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.791956483Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_email - v1" duration=591.156µs 2024-09-13T13:02:06.844 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.794162143Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-13T13:02:06.845 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.794608891Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_org_id - v1" duration=447.532µs 2024-09-13T13:02:06.845 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.796922305Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-13T13:02:06.845 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.797362664Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_code - v1" duration=440.16µs 2024-09-13T13:02:06.845 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.799918003Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-13T13:02:06.845 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.800338258Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_status - v1" duration=420.221µs 2024-09-13T13:02:06.845 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.803088205Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-13T13:02:06.845 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.805217444Z level=info msg="Migration successfully executed" id="Rename table temp_user to temp_user_tmp_qwerty - v1" duration=2.127493ms 2024-09-13T13:02:06.845 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.805663206Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-13T13:02:06.845 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.806206483Z level=info msg="Migration successfully executed" id="create temp_user v2" duration=543.032µs 2024-09-13T13:02:06.845 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.808278389Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-13T13:02:06.845 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.808775854Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v2" duration=497.554µs 2024-09-13T13:02:06.846 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.811012877Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-13T13:02:06.846 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.81148431Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v2" duration=471.394µs 2024-09-13T13:02:06.846 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.813850292Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-13T13:02:06.846 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.814314453Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v2" duration=464.057µs 2024-09-13T13:02:06.846 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.816809501Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-13T13:02:06.846 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.817284934Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v2" duration=475.584µs 2024-09-13T13:02:06.846 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.819696345Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-13T13:02:06.846 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.819974183Z level=info msg="Migration successfully executed" id="copy temp_user v1 to v2" duration=277.838µs 2024-09-13T13:02:06.846 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.822397679Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-13T13:02:06.846 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.822813506Z level=info msg="Migration successfully executed" id="drop temp_user_tmp_qwerty" duration=415.642µs 2024-09-13T13:02:06.848 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.825032019Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-13T13:02:06.848 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.825291445Z level=info msg="Migration successfully executed" id="Set created for temp users that will otherwise prematurely expire" duration=259.491µs 2024-09-13T13:02:06.848 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.827867965Z level=info msg="Executing migration" id="create star table" 2024-09-13T13:02:06.848 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.828423526Z level=info msg="Migration successfully executed" id="create star table" duration=558.094µs 2024-09-13T13:02:06.848 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.828762109Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-13T13:02:06.850 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.829290424Z level=info msg="Migration successfully executed" id="add unique index star.user_id_dashboard_id" duration=507.758µs 2024-09-13T13:02:06.850 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.831573227Z level=info msg="Executing migration" id="create org table v1" 2024-09-13T13:02:06.850 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.832073662Z level=info msg="Migration successfully executed" id="create org table v1" duration=500.373µs 2024-09-13T13:02:06.850 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.834042281Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-13T13:02:06.851 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.834517015Z level=info msg="Migration successfully executed" id="create index UQE_org_name - v1" duration=474.474µs 2024-09-13T13:02:06.851 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.836758298Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-13T13:02:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T13:02:06.885 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T13:02:06.885 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-13T13:02:06.929 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:06.975 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:06.975 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:06.995 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:06.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:06.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:07.084 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-mon[33178]: 4.f scrub starts 2024-09-13T13:02:07.085 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-mon[33178]: 4.f scrub ok 2024-09-13T13:02:07.085 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-mon[33178]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:02:07.085 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:07.085 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:07.085 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:07.085 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:07.086 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.837416514Z level=info msg="Migration successfully executed" id="create org_user table v1" duration=658.413µs 2024-09-13T13:02:07.086 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.83943803Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-13T13:02:07.086 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.840113813Z level=info msg="Migration successfully executed" id="create index IDX_org_user_org_id - v1" duration=675.691µs 2024-09-13T13:02:07.086 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.842126864Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-13T13:02:07.086 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.842826631Z level=info msg="Migration successfully executed" id="create index UQE_org_user_org_id_user_id - v1" duration=699.441µs 2024-09-13T13:02:07.086 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.844709036Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-13T13:02:07.086 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.845267476Z level=info msg="Migration successfully executed" id="create index IDX_org_user_user_id - v1" duration=558.918µs 2024-09-13T13:02:07.086 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.847342491Z level=info msg="Executing migration" id="Update org table charset" 2024-09-13T13:02:07.086 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.847376866Z level=info msg="Migration successfully executed" id="Update org table charset" duration=35.075µs 2024-09-13T13:02:07.086 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.850149514Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-13T13:02:07.087 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.850202519Z level=info msg="Migration successfully executed" id="Update org_user table charset" duration=54.769µs 2024-09-13T13:02:07.087 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.853014876Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-13T13:02:07.087 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.853207793Z level=info msg="Migration successfully executed" id="Migrate all Read Only Viewers to Viewers" duration=195.232µs 2024-09-13T13:02:07.087 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.855396925Z level=info msg="Executing migration" id="create dashboard table" 2024-09-13T13:02:07.087 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.855956234Z level=info msg="Migration successfully executed" id="create dashboard table" duration=557.868µs 2024-09-13T13:02:07.087 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.857994875Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-13T13:02:07.087 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.858512421Z level=info msg="Migration successfully executed" id="add index dashboard.account_id" duration=517.391µs 2024-09-13T13:02:07.087 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.860883673Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-13T13:02:07.087 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.861537666Z level=info msg="Migration successfully executed" id="add unique index dashboard_account_id_slug" duration=655.745µs 2024-09-13T13:02:07.087 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.863587336Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-13T13:02:07.087 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.864027396Z level=info msg="Migration successfully executed" id="create dashboard_tag table" duration=439.786µs 2024-09-13T13:02:07.087 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.866468437Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-13T13:02:07.087 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.867080839Z level=info msg="Migration successfully executed" id="add unique index dashboard_tag.dasboard_id_term" duration=614.904µs 2024-09-13T13:02:07.088 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.867429728Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-13T13:02:07.088 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.867887103Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" duration=457.371µs 2024-09-13T13:02:07.088 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.868266112Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-13T13:02:07.088 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.871709302Z level=info msg="Migration successfully executed" id="Rename table dashboard to dashboard_v1 - v1" duration=3.4435ms 2024-09-13T13:02:07.088 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.872085938Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-13T13:02:07.088 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.872566675Z level=info msg="Migration successfully executed" id="create dashboard v2" duration=480.512µs 2024-09-13T13:02:07.088 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.873071566Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-13T13:02:07.088 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.873539758Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_org_id - v2" duration=468.073µs 2024-09-13T13:02:07.088 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.873912343Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-13T13:02:07.088 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.874386143Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_org_id_slug - v2" duration=474.728µs 2024-09-13T13:02:07.088 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.874770868Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-13T13:02:07.088 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.87500978Z level=info msg="Migration successfully executed" id="copy dashboard v1 to v2" duration=238.843µs 2024-09-13T13:02:07.089 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.875603573Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-13T13:02:07.089 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.876180981Z level=info msg="Migration successfully executed" id="drop table dashboard_v1" duration=577.469µs 2024-09-13T13:02:07.089 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.876533424Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-13T13:02:07.089 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.876571653Z level=info msg="Migration successfully executed" id="alter dashboard.data to mediumtext v1" duration=38.711µs 2024-09-13T13:02:07.089 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.877361012Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-13T13:02:07.089 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.87849381Z level=info msg="Migration successfully executed" id="Add column updated_by in dashboard - v2" duration=1.132675ms 2024-09-13T13:02:07.089 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.878878366Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-13T13:02:07.089 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.879916068Z level=info msg="Migration successfully executed" id="Add column created_by in dashboard - v2" duration=1.038705ms 2024-09-13T13:02:07.089 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.880268566Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-13T13:02:07.089 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.881353886Z level=info msg="Migration successfully executed" id="Add column gnetId in dashboard" duration=1.085192ms 2024-09-13T13:02:07.089 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.881711481Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-13T13:02:07.090 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.882208721Z level=info msg="Migration successfully executed" id="Add index for gnetId in dashboard" duration=498.254µs 2024-09-13T13:02:07.090 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.8825729Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-13T13:02:07.090 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.883639601Z level=info msg="Migration successfully executed" id="Add column plugin_id in dashboard" duration=1.066479ms 2024-09-13T13:02:07.090 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.8856081Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-13T13:02:07.090 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.886210989Z level=info msg="Migration successfully executed" id="Add index for plugin_id in dashboard" duration=605.045µs 2024-09-13T13:02:07.090 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.888416141Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-13T13:02:07.090 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.888910901Z level=info msg="Migration successfully executed" id="Add index for dashboard_id in dashboard_tag" duration=496.948µs 2024-09-13T13:02:07.092 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.891096606Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-13T13:02:07.092 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.891128617Z level=info msg="Migration successfully executed" id="Update dashboard table charset" duration=32.561µs 2024-09-13T13:02:07.092 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.891529978Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-13T13:02:07.092 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.891544976Z level=info msg="Migration successfully executed" id="Update dashboard_tag table charset" duration=14.669µs 2024-09-13T13:02:07.093 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.893871969Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-13T13:02:07.093 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.895052062Z level=info msg="Migration successfully executed" id="Add column folder_id in dashboard" duration=1.17969ms 2024-09-13T13:02:07.093 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.895427502Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-13T13:02:07.093 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.896594259Z level=info msg="Migration successfully executed" id="Add column isFolder in dashboard" duration=1.166641ms 2024-09-13T13:02:07.093 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.896966426Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-13T13:02:07.093 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.898098449Z level=info msg="Migration successfully executed" id="Add column has_acl in dashboard" duration=1.131901ms 2024-09-13T13:02:07.093 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.899905304Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-13T13:02:07.093 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.901036732Z level=info msg="Migration successfully executed" id="Add column uid in dashboard" duration=1.131188ms 2024-09-13T13:02:07.093 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.902591665Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-13T13:02:07.093 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.902747744Z level=info msg="Migration successfully executed" id="Update uid column values in dashboard" duration=156.074µs 2024-09-13T13:02:07.093 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.905428351Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-13T13:02:07.093 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.9059136Z level=info msg="Migration successfully executed" id="Add unique index dashboard_org_id_uid" duration=484.907µs 2024-09-13T13:02:07.094 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.908123642Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-13T13:02:07.094 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.908712241Z level=info msg="Migration successfully executed" id="Remove unique index org_id_slug" duration=590.464µs 2024-09-13T13:02:07.094 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.910655246Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-13T13:02:07.094 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.910697239Z level=info msg="Migration successfully executed" id="Update dashboard title length" duration=45.159µs 2024-09-13T13:02:07.094 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.91315088Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-13T13:02:07.094 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.91365504Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_title_folder_id" duration=504.543µs 2024-09-13T13:02:07.094 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.915757314Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-13T13:02:07.094 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.916250618Z level=info msg="Migration successfully executed" id="create dashboard_provisioning" duration=493.026µs 2024-09-13T13:02:07.094 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.918480311Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-13T13:02:07.094 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.92154897Z level=info msg="Migration successfully executed" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" duration=3.06887ms 2024-09-13T13:02:07.094 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.921920643Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-13T13:02:07.094 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.922375024Z level=info msg="Migration successfully executed" id="create dashboard_provisioning v2" duration=454.271µs 2024-09-13T13:02:07.094 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.924759546Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-13T13:02:07.095 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.925309664Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id - v2" duration=550.857µs 2024-09-13T13:02:07.095 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.927285316Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-13T13:02:07.095 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.927812085Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" duration=526.868µs 2024-09-13T13:02:07.095 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.930020231Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-13T13:02:07.095 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.930255947Z level=info msg="Migration successfully executed" id="copy dashboard_provisioning v1 to v2" duration=236.032µs 2024-09-13T13:02:07.095 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.932543429Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-13T13:02:07.095 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.932929134Z level=info msg="Migration successfully executed" id="drop dashboard_provisioning_tmp_qwerty" duration=385.617µs 2024-09-13T13:02:07.095 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.933276721Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-13T13:02:07.095 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.934488228Z level=info msg="Migration successfully executed" id="Add check_sum column" duration=1.211042ms 2024-09-13T13:02:07.095 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.935880005Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-13T13:02:07.095 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.936347408Z level=info msg="Migration successfully executed" id="Add index for dashboard_title" duration=466.942µs 2024-09-13T13:02:07.095 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.938475962Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-13T13:02:07.096 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.938584957Z level=info msg="Migration successfully executed" id="delete tags for deleted dashboards" duration=109.175µs 2024-09-13T13:02:07.096 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.941152661Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-13T13:02:07.096 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.941257275Z level=info msg="Migration successfully executed" id="delete stars for deleted dashboards" duration=106.084µs 2024-09-13T13:02:07.096 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.944025174Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-13T13:02:07.096 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.94458664Z level=info msg="Migration successfully executed" id="Add index for dashboard_is_folder" duration=563.012µs 2024-09-13T13:02:07.096 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.944966103Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-13T13:02:07.096 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.946348833Z level=info msg="Migration successfully executed" id="Add isPublic for dashboard" duration=1.382558ms 2024-09-13T13:02:07.096 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.946746794Z level=info msg="Executing migration" id="create data_source table" 2024-09-13T13:02:07.096 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.947379917Z level=info msg="Migration successfully executed" id="create data_source table" duration=632.897µs 2024-09-13T13:02:07.096 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.947766372Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-13T13:02:07.096 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.948286228Z level=info msg="Migration successfully executed" id="add index data_source.account_id" duration=520.008µs 2024-09-13T13:02:07.096 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.948651495Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-13T13:02:07.096 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.949146993Z level=info msg="Migration successfully executed" id="add unique index data_source.account_id_name" duration=495.198µs 2024-09-13T13:02:07.097 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.949500361Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-13T13:02:07.097 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.949986441Z level=info msg="Migration successfully executed" id="drop index IDX_data_source_account_id - v1" duration=486.026µs 2024-09-13T13:02:07.097 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.950363823Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-13T13:02:07.097 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.950890546Z level=info msg="Migration successfully executed" id="drop index UQE_data_source_account_id_name - v1" duration=525.392µs 2024-09-13T13:02:07.097 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.951334121Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-13T13:02:07.097 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.954765214Z level=info msg="Migration successfully executed" id="Rename table data_source to data_source_v1 - v1" duration=3.429862ms 2024-09-13T13:02:07.097 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.955169425Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-13T13:02:07.097 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.955756035Z level=info msg="Migration successfully executed" id="create data_source table v2" duration=586.44µs 2024-09-13T13:02:07.097 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.956137877Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-13T13:02:07.097 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.956653174Z level=info msg="Migration successfully executed" id="create index IDX_data_source_org_id - v2" duration=513.425µs 2024-09-13T13:02:07.097 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.957016684Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-13T13:02:07.097 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.957540677Z level=info msg="Migration successfully executed" id="create index UQE_data_source_org_id_name - v2" duration=523.862µs 2024-09-13T13:02:07.097 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.957918917Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-13T13:02:07.098 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.958307938Z level=info msg="Migration successfully executed" id="Drop old table data_source_v1 #2" duration=387.801µs 2024-09-13T13:02:07.098 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.958753665Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-13T13:02:07.098 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.960203103Z level=info msg="Migration successfully executed" id="Add column with_credentials" duration=1.449207ms 2024-09-13T13:02:07.098 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.960574228Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-13T13:02:07.098 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.961983314Z level=info msg="Migration successfully executed" id="Add secure json data column" duration=1.408873ms 2024-09-13T13:02:07.098 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.962373355Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-13T13:02:07.098 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.96239283Z level=info msg="Migration successfully executed" id="Update data_source table charset" duration=19.638µs 2024-09-13T13:02:07.098 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.963241649Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-13T13:02:07.098 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.963397582Z level=info msg="Migration successfully executed" id="Update initial version to 1" duration=155.608µs 2024-09-13T13:02:07.098 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.964222684Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-13T13:02:07.098 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.965708969Z level=info msg="Migration successfully executed" id="Add read_only data column" duration=1.488326ms 2024-09-13T13:02:07.098 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.966089016Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-13T13:02:07.098 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.96622426Z level=info msg="Migration successfully executed" id="Migrate logging ds to loki ds" duration=135.414µs 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.966948263Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.967073611Z level=info msg="Migration successfully executed" id="Update json_data with nulls" duration=125.391µs 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.967818924Z level=info msg="Executing migration" id="Add uid column" 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.969230671Z level=info msg="Migration successfully executed" id="Add uid column" duration=1.412818ms 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.96959609Z level=info msg="Executing migration" id="Update uid value" 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.969731202Z level=info msg="Migration successfully executed" id="Update uid value" duration=134.949µs 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.970449852Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.970999444Z level=info msg="Migration successfully executed" id="Add unique index datasource_org_id_uid" duration=549.346µs 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.971384548Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.971943387Z level=info msg="Migration successfully executed" id="add unique index datasource_org_id_is_default" duration=558.631µs 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.972346142Z level=info msg="Executing migration" id="create api_key table" 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.972917603Z level=info msg="Migration successfully executed" id="create api_key table" duration=571.236µs 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.973285223Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-13T13:02:07.099 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.973824965Z level=info msg="Migration successfully executed" id="add index api_key.account_id" duration=540.371µs 2024-09-13T13:02:07.100 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.974184832Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-13T13:02:07.100 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.974673925Z level=info msg="Migration successfully executed" id="add index api_key.key" duration=489.846µs 2024-09-13T13:02:07.100 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.975075119Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-13T13:02:07.100 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.975599538Z level=info msg="Migration successfully executed" id="add index api_key.account_id_name" duration=524.292µs 2024-09-13T13:02:07.100 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.975976211Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-13T13:02:07.100 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.976456191Z level=info msg="Migration successfully executed" id="drop index IDX_api_key_account_id - v1" duration=478.935µs 2024-09-13T13:02:07.100 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.97681112Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-13T13:02:07.100 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.977311929Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_key - v1" duration=484.414µs 2024-09-13T13:02:07.100 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.977660183Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-13T13:02:07.100 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.978143694Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_account_id_name - v1" duration=483.49µs 2024-09-13T13:02:07.100 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.978517094Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-13T13:02:07.100 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.982276976Z level=info msg="Migration successfully executed" id="Rename table api_key to api_key_v1 - v1" duration=3.759077ms 2024-09-13T13:02:07.100 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.98265504Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-13T13:02:07.101 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.98316914Z level=info msg="Migration successfully executed" id="create api_key table v2" duration=513.927µs 2024-09-13T13:02:07.101 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.983531545Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-13T13:02:07.101 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.984032456Z level=info msg="Migration successfully executed" id="create index IDX_api_key_org_id - v2" duration=500.676µs 2024-09-13T13:02:07.101 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.9844066Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-13T13:02:07.101 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.984899112Z level=info msg="Migration successfully executed" id="create index UQE_api_key_key - v2" duration=492.244µs 2024-09-13T13:02:07.101 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.985237486Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-13T13:02:07.101 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.985733093Z level=info msg="Migration successfully executed" id="create index UQE_api_key_org_id_name - v2" duration=495.367µs 2024-09-13T13:02:07.101 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.986094818Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-13T13:02:07.101 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.986328393Z level=info msg="Migration successfully executed" id="copy api_key v1 to v2" duration=233.512µs 2024-09-13T13:02:07.101 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.986687548Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-13T13:02:07.101 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.987088551Z level=info msg="Migration successfully executed" id="Drop old table api_key_v1" duration=401.282µs 2024-09-13T13:02:07.101 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.987439546Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-13T13:02:07.102 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.987457563Z level=info msg="Migration successfully executed" id="Update api_key table charset" duration=18.193µs 2024-09-13T13:02:07.102 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.987892063Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-13T13:02:07.102 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.98937697Z level=info msg="Migration successfully executed" id="Add expires to api_key table" duration=1.484716ms 2024-09-13T13:02:07.102 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.989713707Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-13T13:02:07.102 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.991206334Z level=info msg="Migration successfully executed" id="Add service account foreign key" duration=1.492455ms 2024-09-13T13:02:07.102 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.991545705Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-13T13:02:07.102 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.991655753Z level=info msg="Migration successfully executed" id="set service account foreign key to nil if 0" duration=110.199µs 2024-09-13T13:02:07.102 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.992088007Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-13T13:02:07.102 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.993641367Z level=info msg="Migration successfully executed" id="Add last_used_at to api_key table" duration=1.554123ms 2024-09-13T13:02:07.102 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.994013642Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-13T13:02:07.102 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.995556738Z level=info msg="Migration successfully executed" id="Add is_revoked column to api_key table" duration=1.542973ms 2024-09-13T13:02:07.102 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.995906784Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-13T13:02:07.102 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.996410818Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v4" duration=504.349µs 2024-09-13T13:02:07.103 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.996752645Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-13T13:02:07.103 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.997150789Z level=info msg="Migration successfully executed" id="drop table dashboard_snapshot_v4 #1" duration=398.147µs 2024-09-13T13:02:07.103 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.997513628Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-13T13:02:07.103 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.998106328Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v5 #2" duration=592.697µs 2024-09-13T13:02:07.103 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.99846844Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-13T13:02:07.103 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.999003163Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_key - v5" duration=535.451µs 2024-09-13T13:02:07.103 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.999353757Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-13T13:02:07.103 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:06.999942288Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_delete_key - v5" duration=588.342µs 2024-09-13T13:02:07.103 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.000327896Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-13T13:02:07.104 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.000883542Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_snapshot_user_id - v5" duration=554.53µs 2024-09-13T13:02:07.104 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.001239562Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-13T13:02:07.104 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.001303507Z level=info msg="Migration successfully executed" id="alter dashboard_snapshot to mediumtext v2" duration=64.201µs 2024-09-13T13:02:07.104 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.001728278Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-13T13:02:07.104 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.001746688Z level=info msg="Migration successfully executed" id="Update dashboard_snapshot table charset" duration=18.434µs 2024-09-13T13:02:07.104 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.002167436Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-13T13:02:07.104 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.003934809Z level=info msg="Migration successfully executed" id="Add column external_delete_url to dashboard_snapshots table" duration=1.767099ms 2024-09-13T13:02:07.104 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.004333391Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-13T13:02:07.104 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.005992664Z level=info msg="Migration successfully executed" id="Add encrypted dashboard json column" duration=1.659038ms 2024-09-13T13:02:07.104 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.006389783Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-13T13:02:07.104 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.006438879Z level=info msg="Migration successfully executed" id="Change dashboard_encrypted column to MEDIUMBLOB" duration=49.916µs 2024-09-13T13:02:07.104 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.006841995Z level=info msg="Executing migration" id="create quota table v1" 2024-09-13T13:02:07.105 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.007316738Z level=info msg="Migration successfully executed" id="create quota table v1" duration=474.731µs 2024-09-13T13:02:07.105 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.007680845Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-13T13:02:07.105 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.008214852Z level=info msg="Migration successfully executed" id="create index UQE_quota_org_id_user_id_target - v1" duration=533.887µs 2024-09-13T13:02:07.105 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.008545127Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-13T13:02:07.105 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.008563483Z level=info msg="Migration successfully executed" id="Update quota table charset" duration=18.558µs 2024-09-13T13:02:07.105 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.00896484Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-13T13:02:07.105 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.009472878Z level=info msg="Migration successfully executed" id="create plugin_setting table" duration=507.931µs 2024-09-13T13:02:07.105 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.00984743Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-13T13:02:07.105 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.010380631Z level=info msg="Migration successfully executed" id="create index UQE_plugin_setting_org_id_plugin_id - v1" duration=533.028µs 2024-09-13T13:02:07.105 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.010729177Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-13T13:02:07.105 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.012396822Z level=info msg="Migration successfully executed" id="Add column plugin_version to plugin_settings" duration=1.667592ms 2024-09-13T13:02:07.105 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.012752724Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-13T13:02:07.106 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.012773757Z level=info msg="Migration successfully executed" id="Update plugin_setting table charset" duration=20.649µs 2024-09-13T13:02:07.106 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.013198127Z level=info msg="Executing migration" id="create session table" 2024-09-13T13:02:07.106 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.013763385Z level=info msg="Migration successfully executed" id="create session table" duration=565.007µs 2024-09-13T13:02:07.106 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.014138328Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-13T13:02:07.106 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.014209372Z level=info msg="Migration successfully executed" id="Drop old table playlist table" duration=71.198µs 2024-09-13T13:02:07.106 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.014918958Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-13T13:02:07.106 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.014987467Z level=info msg="Migration successfully executed" id="Drop old table playlist_item table" duration=68.561µs 2024-09-13T13:02:07.106 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.01538583Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-13T13:02:07.106 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.015865791Z level=info msg="Migration successfully executed" id="create playlist table v2" duration=479.637µs 2024-09-13T13:02:07.106 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.016227738Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-13T13:02:07.106 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.016711107Z level=info msg="Migration successfully executed" id="create playlist item table v2" duration=483.12µs 2024-09-13T13:02:07.106 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.017067116Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-13T13:02:07.106 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.0170861Z level=info msg="Migration successfully executed" id="Update playlist table charset" duration=19.044µs 2024-09-13T13:02:07.107 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.017495762Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-13T13:02:07.107 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.017514109Z level=info msg="Migration successfully executed" id="Update playlist_item table charset" duration=18.574µs 2024-09-13T13:02:07.107 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.017898161Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-09-13T13:02:07.107 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.019713053Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=1.814708ms 2024-09-13T13:02:07.107 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.02009242Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-09-13T13:02:07.107 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.021894896Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=1.802238ms 2024-09-13T13:02:07.107 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.022238724Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-13T13:02:07.107 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.022309645Z level=info msg="Migration successfully executed" id="drop preferences table v2" duration=69.117µs 2024-09-13T13:02:07.107 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.022681413Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-13T13:02:07.107 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.022744619Z level=info msg="Migration successfully executed" id="drop preferences table v3" duration=63.244µs 2024-09-13T13:02:07.107 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.023170212Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-13T13:02:07.107 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.023675782Z level=info msg="Migration successfully executed" id="create preferences table v3" duration=505.382µs 2024-09-13T13:02:07.107 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.02406433Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-13T13:02:07.108 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.024083883Z level=info msg="Migration successfully executed" id="Update preferences table charset" duration=20.856µs 2024-09-13T13:02:07.108 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.024470957Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-13T13:02:07.108 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.026294466Z level=info msg="Migration successfully executed" id="Add column team_id in preferences" duration=1.823282ms 2024-09-13T13:02:07.108 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.026659831Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-13T13:02:07.108 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.026763511Z level=info msg="Migration successfully executed" id="Update team_id column values in preferences" duration=103.953µs 2024-09-13T13:02:07.108 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.027178323Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-13T13:02:07.108 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.029024672Z level=info msg="Migration successfully executed" id="Add column week_start in preferences" duration=1.846074ms 2024-09-13T13:02:07.108 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.029413087Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-13T13:02:07.108 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.031365246Z level=info msg="Migration successfully executed" id="Add column preferences.json_data" duration=1.952082ms 2024-09-13T13:02:07.108 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.031720467Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-13T13:02:07.109 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.031769987Z level=info msg="Migration successfully executed" id="alter preferences.json_data to mediumtext v1" duration=50.217µs 2024-09-13T13:02:07.109 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.032228841Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-13T13:02:07.109 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.03284356Z level=info msg="Migration successfully executed" id="Add preferences index org_id" duration=614.68µs 2024-09-13T13:02:07.109 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.033200763Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-13T13:02:07.109 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.033734975Z level=info msg="Migration successfully executed" id="Add preferences index user_id" duration=533.985µs 2024-09-13T13:02:07.109 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.034089914Z level=info msg="Executing migration" id="create alert table v1" 2024-09-13T13:02:07.109 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.034832476Z level=info msg="Migration successfully executed" id="create alert table v1" duration=742.203µs 2024-09-13T13:02:07.109 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.035187476Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-13T13:02:07.109 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.035769813Z level=info msg="Migration successfully executed" id="add index alert org_id & id " duration=581.982µs 2024-09-13T13:02:07.109 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.036124957Z level=info msg="Executing migration" id="add index alert state" 2024-09-13T13:02:07.109 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.036628311Z level=info msg="Migration successfully executed" id="add index alert state" duration=503.273µs 2024-09-13T13:02:07.109 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.036965462Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-13T13:02:07.110 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.037483951Z level=info msg="Migration successfully executed" id="add index alert dashboard_id" duration=518.292µs 2024-09-13T13:02:07.110 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.037821191Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-13T13:02:07.110 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.038252785Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v1" duration=430.213µs 2024-09-13T13:02:07.110 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.038587249Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-13T13:02:07.110 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.039152869Z level=info msg="Migration successfully executed" id="Add unique index alert_rule_tag.alert_id_tag_id" duration=565.3µs 2024-09-13T13:02:07.110 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.039496312Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-13T13:02:07.110 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.04002339Z level=info msg="Migration successfully executed" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" duration=525.996µs 2024-09-13T13:02:07.110 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.040343299Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-13T13:02:07.110 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.046024914Z level=info msg="Migration successfully executed" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" duration=5.681437ms 2024-09-13T13:02:07.110 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.046358571Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-13T13:02:07.110 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.046837129Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v2" duration=478.494µs 2024-09-13T13:02:07.110 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.047163713Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-13T13:02:07.110 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.047674467Z level=info msg="Migration successfully executed" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" duration=510.588µs 2024-09-13T13:02:07.111 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.048012673Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-13T13:02:07.111 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.048199499Z level=info msg="Migration successfully executed" id="copy alert_rule_tag v1 to v2" duration=186.826µs 2024-09-13T13:02:07.111 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.048501624Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-13T13:02:07.111 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.048848774Z level=info msg="Migration successfully executed" id="drop table alert_rule_tag_v1" duration=346.947µs 2024-09-13T13:02:07.111 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.049204708Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-13T13:02:07.111 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.0496799Z level=info msg="Migration successfully executed" id="create alert_notification table v1" duration=474.987µs 2024-09-13T13:02:07.111 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.050047814Z level=info msg="Executing migration" id="Add column is_default" 2024-09-13T13:02:07.111 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.052075866Z level=info msg="Migration successfully executed" id="Add column is_default" duration=2.026576ms 2024-09-13T13:02:07.111 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.052433656Z level=info msg="Executing migration" id="Add column frequency" 2024-09-13T13:02:07.111 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.05462973Z level=info msg="Migration successfully executed" id="Add column frequency" duration=2.195802ms 2024-09-13T13:02:07.111 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.054985649Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-13T13:02:07.111 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.05737955Z level=info msg="Migration successfully executed" id="Add column send_reminder" duration=2.393811ms 2024-09-13T13:02:07.111 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.057728098Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-13T13:02:07.112 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.059775259Z level=info msg="Migration successfully executed" id="Add column disable_resolve_message" duration=2.047064ms 2024-09-13T13:02:07.112 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.060128243Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-13T13:02:07.112 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.060630191Z level=info msg="Migration successfully executed" id="add index alert_notification org_id & name" duration=501.849µs 2024-09-13T13:02:07.112 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.062822517Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-13T13:02:07.112 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.062838582Z level=info msg="Migration successfully executed" id="Update alert table charset" duration=16.744µs 2024-09-13T13:02:07.112 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.065602735Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-13T13:02:07.112 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.065634792Z level=info msg="Migration successfully executed" id="Update alert_notification table charset" duration=32.639µs 2024-09-13T13:02:07.112 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.068272285Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-13T13:02:07.112 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.068751179Z level=info msg="Migration successfully executed" id="create notification_journal table v1" duration=477.524µs 2024-09-13T13:02:07.112 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.069106435Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-13T13:02:07.112 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.06962251Z level=info msg="Migration successfully executed" id="add index notification_journal org_id & alert_id & notifier_id" duration=515.935µs 2024-09-13T13:02:07.112 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.069979885Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-13T13:02:07.112 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.070486938Z level=info msg="Migration successfully executed" id="drop alert_notification_journal" duration=506.925µs 2024-09-13T13:02:07.113 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.070881974Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-13T13:02:07.113 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.071386545Z level=info msg="Migration successfully executed" id="create alert_notification_state table v1" duration=504.598µs 2024-09-13T13:02:07.113 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.071739162Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-13T13:02:07.113 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.072275867Z level=info msg="Migration successfully executed" id="add index alert_notification_state org_id & alert_id & notifier_id" duration=536.576µs 2024-09-13T13:02:07.113 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.072621246Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-13T13:02:07.113 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.074724297Z level=info msg="Migration successfully executed" id="Add for to alert table" duration=2.102847ms 2024-09-13T13:02:07.113 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.075113465Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-13T13:02:07.113 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.077398216Z level=info msg="Migration successfully executed" id="Add column uid in alert_notification" duration=2.284565ms 2024-09-13T13:02:07.113 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.077752835Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-13T13:02:07.113 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.077872886Z level=info msg="Migration successfully executed" id="Update uid column values in alert_notification" duration=120.168µs 2024-09-13T13:02:07.113 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.078613146Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-13T13:02:07.113 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.079135632Z level=info msg="Migration successfully executed" id="Add unique index alert_notification_org_id_uid" duration=522.342µs 2024-09-13T13:02:07.113 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.07949909Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-13T13:02:07.114 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.080035401Z level=info msg="Migration successfully executed" id="Remove unique index org_id_name" duration=536.865µs 2024-09-13T13:02:07.114 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.080405512Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-13T13:02:07.114 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.082604246Z level=info msg="Migration successfully executed" id="Add column secure_settings in alert_notification" duration=2.198669ms 2024-09-13T13:02:07.114 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.082965233Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-13T13:02:07.114 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:07.115 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.083003941Z level=info msg="Migration successfully executed" id="alter alert.settings to mediumtext" duration=39.108µs 2024-09-13T13:02:07.115 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.083861886Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-13T13:02:07.115 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.084391979Z level=info msg="Migration successfully executed" id="Add non-unique index alert_notification_state_alert_id" duration=530.06µs 2024-09-13T13:02:07.115 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.084784113Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-13T13:02:07.115 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.085412123Z level=info msg="Migration successfully executed" id="Add non-unique index alert_rule_tag_alert_id" duration=628.326µs 2024-09-13T13:02:07.115 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.085772926Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-13T13:02:07.116 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.085850038Z level=info msg="Migration successfully executed" id="Drop old annotation table v4" duration=77.194µs 2024-09-13T13:02:07.116 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.086609986Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-13T13:02:07.116 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.087230921Z level=info msg="Migration successfully executed" id="create annotation table v5" duration=621.221µs 2024-09-13T13:02:07.116 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.087582677Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-13T13:02:07.116 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.088168349Z level=info msg="Migration successfully executed" id="add index annotation 0 v3" duration=585.942µs 2024-09-13T13:02:07.116 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.088514042Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-13T13:02:07.116 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.08904666Z level=info msg="Migration successfully executed" id="add index annotation 1 v3" duration=532.465µs 2024-09-13T13:02:07.116 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.089409743Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-13T13:02:07.116 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.089951258Z level=info msg="Migration successfully executed" id="add index annotation 2 v3" duration=540.172µs 2024-09-13T13:02:07.116 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.090314698Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-13T13:02:07.116 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.090933017Z level=info msg="Migration successfully executed" id="add index annotation 3 v3" duration=618.019µs 2024-09-13T13:02:07.116 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.091300483Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-13T13:02:07.116 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.091955182Z level=info msg="Migration successfully executed" id="add index annotation 4 v3" duration=653.777µs 2024-09-13T13:02:07.117 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.092329607Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-13T13:02:07.117 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.092352636Z level=info msg="Migration successfully executed" id="Update annotation table charset" duration=22.031µs 2024-09-13T13:02:07.137 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:07.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:07.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:07.224 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[36682]: 4.f scrub starts 2024-09-13T13:02:07.224 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[36682]: 4.f scrub ok 2024-09-13T13:02:07.224 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[36682]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:02:07.224 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:07.224 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:07.224 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:07.225 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:07.225 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[29471]: 4.f scrub starts 2024-09-13T13:02:07.225 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[29471]: 4.f scrub ok 2024-09-13T13:02:07.225 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[29471]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:02:07.225 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:07.225 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:07.225 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:07.225 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:06 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:07.239 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:07.261 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:07.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:07.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:07.353 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.093176604Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-13T13:02:07.353 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.095643694Z level=info msg="Migration successfully executed" id="Add column region_id to annotation table" duration=2.467439ms 2024-09-13T13:02:07.353 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.09601393Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-13T13:02:07.353 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.09654401Z level=info msg="Migration successfully executed" id="Drop category_id index" duration=530.042µs 2024-09-13T13:02:07.353 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.096886179Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-13T13:02:07.353 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.099157903Z level=info msg="Migration successfully executed" id="Add column tags to annotation table" duration=2.272001ms 2024-09-13T13:02:07.353 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.099566951Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-13T13:02:07.353 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.100009217Z level=info msg="Migration successfully executed" id="Create annotation_tag table v2" duration=442.191µs 2024-09-13T13:02:07.353 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.100401107Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-13T13:02:07.353 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.101103082Z level=info msg="Migration successfully executed" id="Add unique index annotation_tag.annotation_id_tag_id" duration=702.033µs 2024-09-13T13:02:07.354 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.101461857Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-13T13:02:07.354 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.102028306Z level=info msg="Migration successfully executed" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" duration=567.318µs 2024-09-13T13:02:07.354 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.102373025Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-13T13:02:07.354 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.108894201Z level=info msg="Migration successfully executed" id="Rename table annotation_tag to annotation_tag_v2 - v2" duration=6.520673ms 2024-09-13T13:02:07.354 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.109255584Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-13T13:02:07.354 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.109716853Z level=info msg="Migration successfully executed" id="Create annotation_tag table v3" duration=461.313µs 2024-09-13T13:02:07.354 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.110136276Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-13T13:02:07.355 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.110687551Z level=info msg="Migration successfully executed" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" duration=551.185µs 2024-09-13T13:02:07.355 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.111054308Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-13T13:02:07.355 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.111253882Z level=info msg="Migration successfully executed" id="copy annotation_tag v2 to v3" duration=199.075µs 2024-09-13T13:02:07.355 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.111833183Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-13T13:02:07.355 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.112206801Z level=info msg="Migration successfully executed" id="drop table annotation_tag_v2" duration=373.623µs 2024-09-13T13:02:07.355 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.112549776Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-13T13:02:07.355 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.112655411Z level=info msg="Migration successfully executed" id="Update alert annotations and set TEXT to empty" duration=105.855µs 2024-09-13T13:02:07.355 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.113391133Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-13T13:02:07.355 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.11573632Z level=info msg="Migration successfully executed" id="Add created time to annotation table" duration=2.344843ms 2024-09-13T13:02:07.355 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.116089207Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-13T13:02:07.355 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.118632631Z level=info msg="Migration successfully executed" id="Add updated time to annotation table" duration=2.542285ms 2024-09-13T13:02:07.355 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.11900834Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-13T13:02:07.356 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.119563817Z level=info msg="Migration successfully executed" id="Add index for created in annotation table" duration=555.015µs 2024-09-13T13:02:07.356 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.119911503Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-13T13:02:07.356 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.120430785Z level=info msg="Migration successfully executed" id="Add index for updated in annotation table" duration=520.419µs 2024-09-13T13:02:07.356 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.122819874Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-13T13:02:07.356 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.122962903Z level=info msg="Migration successfully executed" id="Convert existing annotations from seconds to milliseconds" duration=143.054µs 2024-09-13T13:02:07.356 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.125655686Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-13T13:02:07.356 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.128102395Z level=info msg="Migration successfully executed" id="Add epoch_end column" duration=2.447127ms 2024-09-13T13:02:07.356 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.128493733Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-13T13:02:07.356 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.129034487Z level=info msg="Migration successfully executed" id="Add index for epoch_end" duration=540.434µs 2024-09-13T13:02:07.356 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.129380038Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-13T13:02:07.356 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.12948376Z level=info msg="Migration successfully executed" id="Make epoch_end the same as epoch" duration=103.835µs 2024-09-13T13:02:07.356 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.132326303Z level=info msg="Executing migration" id="Move region to single row" 2024-09-13T13:02:07.357 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.132537626Z level=info msg="Migration successfully executed" id="Move region to single row" duration=211.191µs 2024-09-13T13:02:07.357 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.135229249Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-13T13:02:07.357 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.135797941Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch from annotation table" duration=568.641µs 2024-09-13T13:02:07.357 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.138074634Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-13T13:02:07.357 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.13891956Z level=info msg="Migration successfully executed" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" duration=845.784µs 2024-09-13T13:02:07.357 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.139276639Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-13T13:02:07.357 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.139862894Z level=info msg="Migration successfully executed" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" duration=586.102µs 2024-09-13T13:02:07.357 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.142095522Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-13T13:02:07.357 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.142624942Z level=info msg="Migration successfully executed" id="Add index for org_id_epoch_end_epoch on annotation table" duration=529.104µs 2024-09-13T13:02:07.357 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.144934437Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-13T13:02:07.357 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.145460034Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch_epoch_end from annotation table" duration=525.604µs 2024-09-13T13:02:07.357 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.14763916Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-13T13:02:07.357 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.148187567Z level=info msg="Migration successfully executed" id="Add index for alert_id on annotation table" duration=548.263µs 2024-09-13T13:02:07.358 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.148536601Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-13T13:02:07.358 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.148576156Z level=info msg="Migration successfully executed" id="Increase tags column to length 4096" duration=39.754µs 2024-09-13T13:02:07.358 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.151318237Z level=info msg="Executing migration" id="create test_data table" 2024-09-13T13:02:07.358 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.151865351Z level=info msg="Migration successfully executed" id="create test_data table" duration=546.733µs 2024-09-13T13:02:07.358 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.1543073Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-13T13:02:07.358 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.155016012Z level=info msg="Migration successfully executed" id="create dashboard_version table v1" duration=710.969µs 2024-09-13T13:02:07.358 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.15709604Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-13T13:02:07.358 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.157624579Z level=info msg="Migration successfully executed" id="add index dashboard_version.dashboard_id" duration=528.554µs 2024-09-13T13:02:07.358 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.15799431Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-13T13:02:07.358 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.158545682Z level=info msg="Migration successfully executed" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" duration=551.131µs 2024-09-13T13:02:07.358 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.160652568Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-13T13:02:07.358 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.160764863Z level=info msg="Migration successfully executed" id="Set dashboard version to 1 where 0" duration=112.394µs 2024-09-13T13:02:07.358 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.163230383Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-13T13:02:07.359 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.163459603Z level=info msg="Migration successfully executed" id="save existing dashboard data in dashboard_version table v1" duration=229.159µs 2024-09-13T13:02:07.359 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.165842478Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-13T13:02:07.360 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.165882399Z level=info msg="Migration successfully executed" id="alter dashboard_version.data to mediumtext v1" duration=40.317µs 2024-09-13T13:02:07.360 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.166332764Z level=info msg="Executing migration" id="create team table" 2024-09-13T13:02:07.360 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.166809917Z level=info msg="Migration successfully executed" id="create team table" duration=476.848µs 2024-09-13T13:02:07.360 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.168950811Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-13T13:02:07.360 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.169573481Z level=info msg="Migration successfully executed" id="add index team.org_id" duration=623.118µs 2024-09-13T13:02:07.360 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.17151877Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-13T13:02:07.360 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.172103917Z level=info msg="Migration successfully executed" id="add unique index team_org_id_name" duration=584.719µs 2024-09-13T13:02:07.360 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.174323255Z level=info msg="Executing migration" id="Add column uid in team" 2024-09-13T13:02:07.360 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.176922398Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=2.601195ms 2024-09-13T13:02:07.360 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.177288965Z level=info msg="Executing migration" id="Update uid column values in team" 2024-09-13T13:02:07.361 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.177419992Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=131.014µs 2024-09-13T13:02:07.361 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.180036279Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-09-13T13:02:07.361 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.180615855Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=579.49µs 2024-09-13T13:02:07.361 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.18237306Z level=info msg="Executing migration" id="create team member table" 2024-09-13T13:02:07.361 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.18288808Z level=info msg="Migration successfully executed" id="create team member table" duration=514.826µs 2024-09-13T13:02:07.361 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.184929757Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-13T13:02:07.361 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.185489765Z level=info msg="Migration successfully executed" id="add index team_member.org_id" duration=558.933µs 2024-09-13T13:02:07.361 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.185861917Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-13T13:02:07.361 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.186424551Z level=info msg="Migration successfully executed" id="add unique index team_member_org_id_team_id_user_id" duration=562.492µs 2024-09-13T13:02:07.361 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.188635501Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-13T13:02:07.361 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.189215749Z level=info msg="Migration successfully executed" id="add index team_member.team_id" duration=580.001µs 2024-09-13T13:02:07.361 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.191133201Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-13T13:02:07.361 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.193944867Z level=info msg="Migration successfully executed" id="Add column email to team table" duration=2.811283ms 2024-09-13T13:02:07.362 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.19428983Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-13T13:02:07.362 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.197251736Z level=info msg="Migration successfully executed" id="Add column external to team_member table" duration=2.961781ms 2024-09-13T13:02:07.362 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.197615773Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-13T13:02:07.362 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.200198855Z level=info msg="Migration successfully executed" id="Add column permission to team_member table" duration=2.583817ms 2024-09-13T13:02:07.362 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.200564688Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-13T13:02:07.362 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.20118926Z level=info msg="Migration successfully executed" id="create dashboard acl table" duration=624.412µs 2024-09-13T13:02:07.362 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.203321776Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-13T13:02:07.362 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.203941563Z level=info msg="Migration successfully executed" id="add index dashboard_acl_dashboard_id" duration=619.537µs 2024-09-13T13:02:07.362 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.204304149Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-13T13:02:07.362 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.204966176Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_user_id" duration=661.647µs 2024-09-13T13:02:07.362 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.205358517Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-13T13:02:07.362 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.205991694Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_team_id" duration=633.201µs 2024-09-13T13:02:07.362 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.206350575Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-13T13:02:07.363 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.206914365Z level=info msg="Migration successfully executed" id="add index dashboard_acl_user_id" duration=563.543µs 2024-09-13T13:02:07.363 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.207279049Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-13T13:02:07.363 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.207852566Z level=info msg="Migration successfully executed" id="add index dashboard_acl_team_id" duration=573.122µs 2024-09-13T13:02:07.363 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.208199953Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-13T13:02:07.363 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.20877714Z level=info msg="Migration successfully executed" id="add index dashboard_acl_org_id_role" duration=576.876µs 2024-09-13T13:02:07.363 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.209134154Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-13T13:02:07.363 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.209724869Z level=info msg="Migration successfully executed" id="add index dashboard_permission" duration=590.198µs 2024-09-13T13:02:07.363 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.210086497Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-13T13:02:07.363 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.210428817Z level=info msg="Migration successfully executed" id="save default acl rules in dashboard_acl table" duration=342.192µs 2024-09-13T13:02:07.363 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.210947418Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-13T13:02:07.363 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.211095356Z level=info msg="Migration successfully executed" id="delete acl rules for deleted dashboards and folders" duration=149.488µs 2024-09-13T13:02:07.363 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.211711491Z level=info msg="Executing migration" id="create tag table" 2024-09-13T13:02:07.363 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.212231747Z level=info msg="Migration successfully executed" id="create tag table" duration=520.015µs 2024-09-13T13:02:07.364 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.212607106Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-13T13:02:07.364 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.21316375Z level=info msg="Migration successfully executed" id="add index tag.key_value" duration=556.454µs 2024-09-13T13:02:07.364 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.213545511Z level=info msg="Executing migration" id="create login attempt table" 2024-09-13T13:02:07.364 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.214032184Z level=info msg="Migration successfully executed" id="create login attempt table" duration=486.351µs 2024-09-13T13:02:07.364 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.214421468Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-13T13:02:07.364 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.21498247Z level=info msg="Migration successfully executed" id="add index login_attempt.username" duration=560.611µs 2024-09-13T13:02:07.364 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.215326796Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-13T13:02:07.364 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.215889394Z level=info msg="Migration successfully executed" id="drop index IDX_login_attempt_username - v1" duration=562.623µs 2024-09-13T13:02:07.364 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.216232369Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-13T13:02:07.364 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.224336597Z level=info msg="Migration successfully executed" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" duration=8.104124ms 2024-09-13T13:02:07.364 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.224702966Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-13T13:02:07.364 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.225194131Z level=info msg="Migration successfully executed" id="create login_attempt v2" duration=490.114µs 2024-09-13T13:02:07.364 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.225553091Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-13T13:02:07.365 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.226111177Z level=info msg="Migration successfully executed" id="create index IDX_login_attempt_username - v2" duration=556.581µs 2024-09-13T13:02:07.365 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.226459445Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-13T13:02:07.365 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.226667552Z level=info msg="Migration successfully executed" id="copy login_attempt v1 to v2" duration=208.007µs 2024-09-13T13:02:07.365 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.227353538Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-13T13:02:07.365 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.22778097Z level=info msg="Migration successfully executed" id="drop login_attempt_tmp_qwerty" duration=428.283µs 2024-09-13T13:02:07.365 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.228229637Z level=info msg="Executing migration" id="create user auth table" 2024-09-13T13:02:07.365 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.228698044Z level=info msg="Migration successfully executed" id="create user auth table" duration=468.379µs 2024-09-13T13:02:07.365 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.229117307Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-13T13:02:07.365 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.229725266Z level=info msg="Migration successfully executed" id="create index IDX_user_auth_auth_module_auth_id - v1" duration=607.687µs 2024-09-13T13:02:07.365 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.230086328Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-13T13:02:07.365 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.230135101Z level=info msg="Migration successfully executed" id="alter user_auth.auth_id to length 190" duration=48.712µs 2024-09-13T13:02:07.365 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.230901484Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-13T13:02:07.366 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.233927617Z level=info msg="Migration successfully executed" id="Add OAuth access token to user_auth" duration=3.026742ms 2024-09-13T13:02:07.366 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.234276037Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-13T13:02:07.366 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.237216323Z level=info msg="Migration successfully executed" id="Add OAuth refresh token to user_auth" duration=2.940259ms 2024-09-13T13:02:07.366 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.237571692Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-13T13:02:07.366 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.240469131Z level=info msg="Migration successfully executed" id="Add OAuth token type to user_auth" duration=2.896575ms 2024-09-13T13:02:07.366 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.240866022Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-13T13:02:07.366 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.244545259Z level=info msg="Migration successfully executed" id="Add OAuth expiry to user_auth" duration=3.677446ms 2024-09-13T13:02:07.366 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.244941311Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-13T13:02:07.366 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.245605273Z level=info msg="Migration successfully executed" id="Add index to user_id column in user_auth" duration=663.27µs 2024-09-13T13:02:07.366 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.246029887Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-13T13:02:07.366 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.249239525Z level=info msg="Migration successfully executed" id="Add OAuth ID token to user_auth" duration=3.208056ms 2024-09-13T13:02:07.366 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.249641103Z level=info msg="Executing migration" id="create server_lock table" 2024-09-13T13:02:07.366 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.250220338Z level=info msg="Migration successfully executed" id="create server_lock table" duration=578.392µs 2024-09-13T13:02:07.367 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.250581131Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-13T13:02:07.367 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.251164421Z level=info msg="Migration successfully executed" id="add index server_lock.operation_uid" duration=583.037µs 2024-09-13T13:02:07.367 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.251507522Z level=info msg="Executing migration" id="create user auth token table" 2024-09-13T13:02:07.367 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.252066037Z level=info msg="Migration successfully executed" id="create user auth token table" duration=558.171µs 2024-09-13T13:02:07.367 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.25240653Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-13T13:02:07.367 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.252997392Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.auth_token" duration=590.685µs 2024-09-13T13:02:07.367 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.253337242Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-13T13:02:07.367 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.253920257Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.prev_auth_token" duration=582.693µs 2024-09-13T13:02:07.367 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.254257562Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-13T13:02:07.367 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.254927855Z level=info msg="Migration successfully executed" id="add index user_auth_token.user_id" duration=669.625µs 2024-09-13T13:02:07.367 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.255303691Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-13T13:02:07.367 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.258510612Z level=info msg="Migration successfully executed" id="Add revoked_at to the user auth token" duration=3.204383ms 2024-09-13T13:02:07.368 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.258870406Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-09-13T13:02:07.368 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.25945839Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=587.88µs 2024-09-13T13:02:07.368 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.261807178Z level=info msg="Executing migration" id="create cache_data table" 2024-09-13T13:02:07.368 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.262418779Z level=info msg="Migration successfully executed" id="create cache_data table" duration=611.459µs 2024-09-13T13:02:07.368 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.264783566Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-13T13:02:07.368 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.265378457Z level=info msg="Migration successfully executed" id="add unique index cache_data.cache_key" duration=593.049µs 2024-09-13T13:02:07.368 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.267866897Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-13T13:02:07.368 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.268437137Z level=info msg="Migration successfully executed" id="create short_url table v1" duration=570.767µs 2024-09-13T13:02:07.368 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.27054169Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-13T13:02:07.368 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.27128723Z level=info msg="Migration successfully executed" id="add index short_url.org_id-uid" duration=745.939µs 2024-09-13T13:02:07.368 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.273269313Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-13T13:02:07.368 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.273314193Z level=info msg="Migration successfully executed" id="alter table short_url alter column created_by type to bigint" duration=45.156µs 2024-09-13T13:02:07.368 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.276109712Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-13T13:02:07.369 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.27619426Z level=info msg="Migration successfully executed" id="delete alert_definition table" duration=85.147µs 2024-09-13T13:02:07.369 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.278992533Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-13T13:02:07.369 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.279549423Z level=info msg="Migration successfully executed" id="recreate alert_definition table" duration=557.191µs 2024-09-13T13:02:07.369 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.281950424Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-13T13:02:07.369 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.282609397Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and title columns" duration=659.164µs 2024-09-13T13:02:07.369 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.284884031Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-13T13:02:07.369 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.285547555Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and uid columns" duration=663.376µs 2024-09-13T13:02:07.369 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.287618715Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-13T13:02:07.369 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.287668057Z level=info msg="Migration successfully executed" id="alter alert_definition table data column to mediumtext in mysql" duration=49.48µs 2024-09-13T13:02:07.369 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.290584772Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-13T13:02:07.369 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.291349662Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and title columns" duration=767.204µs 2024-09-13T13:02:07.369 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.293457828Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-13T13:02:07.370 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:07.371 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.294103296Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and uid columns" duration=651.519µs 2024-09-13T13:02:07.371 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.296059569Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-13T13:02:07.371 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.296729876Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and title columns" duration=670.111µs 2024-09-13T13:02:07.371 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.299062051Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-13T13:02:07.371 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.299742098Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and uid columns" duration=682.166µs 2024-09-13T13:02:07.371 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.301783293Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-13T13:02:07.371 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.305345766Z level=info msg="Migration successfully executed" id="Add column paused in alert_definition" duration=3.564466ms 2024-09-13T13:02:07.371 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.305707271Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-13T13:02:07.371 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.306388606Z level=info msg="Migration successfully executed" id="drop alert_definition table" duration=681.266µs 2024-09-13T13:02:07.371 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.308594501Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-13T13:02:07.371 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.308681476Z level=info msg="Migration successfully executed" id="delete alert_definition_version table" duration=87.159µs 2024-09-13T13:02:07.372 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.311306867Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-13T13:02:07.372 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.311927263Z level=info msg="Migration successfully executed" id="recreate alert_definition_version table" duration=620.162µs 2024-09-13T13:02:07.372 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.312281673Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-13T13:02:07.372 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.312901652Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_id and version columns" duration=619.865µs 2024-09-13T13:02:07.372 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.315059026Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-13T13:02:07.372 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.315701335Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_uid and version columns" duration=642.029µs 2024-09-13T13:02:07.372 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.317750025Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-13T13:02:07.372 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.317824138Z level=info msg="Migration successfully executed" id="alter alert_definition_version table data column to mediumtext in mysql" duration=72.593µs 2024-09-13T13:02:07.372 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.318226863Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-13T13:02:07.372 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.318888002Z level=info msg="Migration successfully executed" id="drop alert_definition_version table" duration=661.143µs 2024-09-13T13:02:07.372 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.320417708Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-13T13:02:07.373 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.321046623Z level=info msg="Migration successfully executed" id="create alert_instance table" duration=628.572µs 2024-09-13T13:02:07.373 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.32310176Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-13T13:02:07.373 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.323742641Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" duration=640.612µs 2024-09-13T13:02:07.373 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.325839197Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-13T13:02:07.373 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.326453276Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, current_state columns" duration=614.022µs 2024-09-13T13:02:07.373 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.326813275Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-13T13:02:07.373 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.330133532Z level=info msg="Migration successfully executed" id="add column current_state_end to alert_instance" duration=3.319105ms 2024-09-13T13:02:07.373 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.330482746Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-13T13:02:07.373 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.331072866Z level=info msg="Migration successfully executed" id="remove index def_org_id, def_uid, current_state on alert_instance" duration=590.058µs 2024-09-13T13:02:07.373 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.33141828Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-13T13:02:07.373 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.33199711Z level=info msg="Migration successfully executed" id="remove index def_org_id, current_state on alert_instance" duration=578.752µs 2024-09-13T13:02:07.373 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.334036819Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-13T13:02:07.374 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.352128038Z level=info msg="Migration successfully executed" id="rename def_org_id to rule_org_id in alert_instance" duration=18.087458ms 2024-09-13T13:02:07.374 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.352602341Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-13T13:02:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:07.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:07.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:07.499 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:07.522 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:07.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:07.607 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.369472155Z level=info msg="Migration successfully executed" id="rename def_uid to rule_uid in alert_instance" duration=16.870074ms 2024-09-13T13:02:07.607 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.369866628Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-13T13:02:07.607 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.370490607Z level=info msg="Migration successfully executed" id="add index rule_org_id, rule_uid, current_state on alert_instance" duration=623.806µs 2024-09-13T13:02:07.607 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.372541333Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-13T13:02:07.607 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.373148441Z level=info msg="Migration successfully executed" id="add index rule_org_id, current_state on alert_instance" duration=606.66µs 2024-09-13T13:02:07.607 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.375286874Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-13T13:02:07.607 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.378652342Z level=info msg="Migration successfully executed" id="add current_reason column related to current_state" duration=3.368058ms 2024-09-13T13:02:07.607 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.379034119Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-09-13T13:02:07.608 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.382363255Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=3.329019ms 2024-09-13T13:02:07.608 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.382753932Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-13T13:02:07.608 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.383399406Z level=info msg="Migration successfully executed" id="create alert_rule table" duration=643.605µs 2024-09-13T13:02:07.608 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.38378428Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-13T13:02:07.608 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.384423418Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and title columns" duration=638.816µs 2024-09-13T13:02:07.608 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.386387197Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-13T13:02:07.608 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.387041196Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and uid columns" duration=653.758µs 2024-09-13T13:02:07.608 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.389036352Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-13T13:02:07.608 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.389749074Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" duration=712.606µs 2024-09-13T13:02:07.608 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.39015439Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-13T13:02:07.608 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.39020113Z level=info msg="Migration successfully executed" id="alter alert_rule table data column to mediumtext in mysql" duration=44.701µs 2024-09-13T13:02:07.608 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.390650101Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-13T13:02:07.609 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.394082088Z level=info msg="Migration successfully executed" id="add column for to alert_rule" duration=3.431704ms 2024-09-13T13:02:07.609 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.394460378Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-13T13:02:07.609 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.397842085Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule" duration=3.380925ms 2024-09-13T13:02:07.609 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.398213836Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-13T13:02:07.609 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.401579845Z level=info msg="Migration successfully executed" id="add column labels to alert_rule" duration=3.365891ms 2024-09-13T13:02:07.609 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.401990632Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-13T13:02:07.609 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.402602044Z level=info msg="Migration successfully executed" id="remove unique index from alert_rule on org_id, title columns" duration=611.437µs 2024-09-13T13:02:07.609 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.402984156Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-13T13:02:07.609 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.403620189Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespase_uid and title columns" duration=635.728µs 2024-09-13T13:02:07.609 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.403989886Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-13T13:02:07.609 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.407270766Z level=info msg="Migration successfully executed" id="add dashboard_uid column to alert_rule" duration=3.280755ms 2024-09-13T13:02:07.609 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.407653392Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-13T13:02:07.609 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.410971923Z level=info msg="Migration successfully executed" id="add panel_id column to alert_rule" duration=3.318346ms 2024-09-13T13:02:07.611 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.411357047Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-13T13:02:07.611 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.411995171Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" duration=637.938µs 2024-09-13T13:02:07.611 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.412418496Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-13T13:02:07.611 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.415767118Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule" duration=3.348313ms 2024-09-13T13:02:07.611 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.416156229Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-13T13:02:07.611 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.419569147Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule table" duration=3.412754ms 2024-09-13T13:02:07.611 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.419972895Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-13T13:02:07.611 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.420019355Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule table" duration=45.99µs 2024-09-13T13:02:07.611 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.420857922Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-13T13:02:07.611 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.421595682Z level=info msg="Migration successfully executed" id="create alert_rule_version table" duration=737.659µs 2024-09-13T13:02:07.611 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.422001144Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-13T13:02:07.611 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.422635592Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" duration=634.24µs 2024-09-13T13:02:07.612 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.423012205Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-13T13:02:07.612 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.423699611Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" duration=687.332µs 2024-09-13T13:02:07.612 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.424101966Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-13T13:02:07.612 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.42414803Z level=info msg="Migration successfully executed" id="alter alert_rule_version table data column to mediumtext in mysql" duration=45.828µs 2024-09-13T13:02:07.612 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.424942272Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-13T13:02:07.612 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.428509452Z level=info msg="Migration successfully executed" id="add column for to alert_rule_version" duration=3.566994ms 2024-09-13T13:02:07.612 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.429019342Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-13T13:02:07.612 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.432568953Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule_version" duration=3.549457ms 2024-09-13T13:02:07.612 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.432952672Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-13T13:02:07.612 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.436505667Z level=info msg="Migration successfully executed" id="add column labels to alert_rule_version" duration=3.550291ms 2024-09-13T13:02:07.612 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.43690557Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-13T13:02:07.612 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.440446333Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule_version" duration=3.538888ms 2024-09-13T13:02:07.612 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.440835255Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-13T13:02:07.613 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.444370523Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule_versions table" duration=3.535039ms 2024-09-13T13:02:07.613 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.444756389Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-13T13:02:07.613 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.444813717Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule_version table" duration=56.796µs 2024-09-13T13:02:07.613 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.445661431Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-13T13:02:07.613 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.44619114Z level=info msg="Migration successfully executed" id=create_alert_configuration_table duration=529.572µs 2024-09-13T13:02:07.613 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.446577178Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-13T13:02:07.613 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.450284973Z level=info msg="Migration successfully executed" id="Add column default in alert_configuration" duration=3.707508ms 2024-09-13T13:02:07.613 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.450660854Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-13T13:02:07.613 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.450708188Z level=info msg="Migration successfully executed" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" duration=45.024µs 2024-09-13T13:02:07.613 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.451175325Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-13T13:02:07.613 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.454886993Z level=info msg="Migration successfully executed" id="add column org_id in alert_configuration" duration=3.711429ms 2024-09-13T13:02:07.613 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.455251298Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-13T13:02:07.614 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.455876537Z level=info msg="Migration successfully executed" id="add index in alert_configuration table on org_id column" duration=626.074µs 2024-09-13T13:02:07.614 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.45624644Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-13T13:02:07.614 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.459936344Z level=info msg="Migration successfully executed" id="add configuration_hash column to alert_configuration" duration=3.689511ms 2024-09-13T13:02:07.614 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.460358213Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-13T13:02:07.614 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.460910712Z level=info msg="Migration successfully executed" id=create_ngalert_configuration_table duration=552.278µs 2024-09-13T13:02:07.614 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.461284096Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-13T13:02:07.614 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.461908284Z level=info msg="Migration successfully executed" id="add index in ngalert_configuration on org_id column" duration=623.945µs 2024-09-13T13:02:07.614 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.462334348Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-13T13:02:07.614 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.465946113Z level=info msg="Migration successfully executed" id="add column send_alerts_to in ngalert_configuration" duration=3.611656ms 2024-09-13T13:02:07.614 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.466368488Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-13T13:02:07.614 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.46689991Z level=info msg="Migration successfully executed" id="create provenance_type table" duration=531.199µs 2024-09-13T13:02:07.614 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.467341393Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-13T13:02:07.614 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.467984245Z level=info msg="Migration successfully executed" id="add index to uniquify (record_key, record_type, org_id) columns" duration=642.578µs 2024-09-13T13:02:07.615 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.468410173Z level=info msg="Executing migration" id="create alert_image table" 2024-09-13T13:02:07.615 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.468962411Z level=info msg="Migration successfully executed" id="create alert_image table" duration=552.142µs 2024-09-13T13:02:07.615 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.469360862Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-13T13:02:07.615 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.469986211Z level=info msg="Migration successfully executed" id="add unique index on token to alert_image table" duration=625.122µs 2024-09-13T13:02:07.615 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.470359355Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-13T13:02:07.615 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.470406915Z level=info msg="Migration successfully executed" id="support longer URLs in alert_image table" duration=47.098µs 2024-09-13T13:02:07.615 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.473214735Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-13T13:02:07.615 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.473871509Z level=info msg="Migration successfully executed" id=create_alert_configuration_history_table duration=656.729µs 2024-09-13T13:02:07.615 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.474286141Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-13T13:02:07.615 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.474932527Z level=info msg="Migration successfully executed" id="drop non-unique orgID index on alert_configuration" duration=646.29µs 2024-09-13T13:02:07.615 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.477044177Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-13T13:02:07.615 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.477294788Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-13T13:02:07.615 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.479871826Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-13T13:02:07.616 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.480167462Z level=info msg="Migration successfully executed" id="extract alertmanager configuration history to separate table" duration=295.564µs 2024-09-13T13:02:07.616 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.482687835Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-13T13:02:07.616 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.483327748Z level=info msg="Migration successfully executed" id="add unique index on orgID to alert_configuration" duration=639.649µs 2024-09-13T13:02:07.616 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.483689854Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-09-13T13:02:07.616 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.487468763Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=3.77869ms 2024-09-13T13:02:07.616 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.487867305Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-13T13:02:07.616 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.488592518Z level=info msg="Migration successfully executed" id="create library_element table v1" duration=725.184µs 2024-09-13T13:02:07.616 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.490722791Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-13T13:02:07.616 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.491425729Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_id-name-kind" duration=702.651µs 2024-09-13T13:02:07.616 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.493688709Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-13T13:02:07.616 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.494265787Z level=info msg="Migration successfully executed" id="create library_element_connection table v1" duration=576.864µs 2024-09-13T13:02:07.617 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.494656136Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-13T13:02:07.617 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.495341112Z level=info msg="Migration successfully executed" id="add index library_element_connection element_id-kind-connection_id" duration=684.787µs 2024-09-13T13:02:07.617 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.497773398Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-13T13:02:07.617 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.498502318Z level=info msg="Migration successfully executed" id="add unique index library_element org_id_uid" duration=731.133µs 2024-09-13T13:02:07.617 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.500611478Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-13T13:02:07.617 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.500645466Z level=info msg="Migration successfully executed" id="increase max description length to 2048" duration=34.624µs 2024-09-13T13:02:07.617 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.503353641Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-09-13T13:02:07.617 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.503414285Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=60.331µs 2024-09-13T13:02:07.617 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.503856497Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-13T13:02:07.617 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.504046001Z level=info msg="Migration successfully executed" id="clone move dashboard alerts to unified alerting" duration=189.57µs 2024-09-13T13:02:07.617 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.506708992Z level=info msg="Executing migration" id="create data_keys table" 2024-09-13T13:02:07.617 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.507370816Z level=info msg="Migration successfully executed" id="create data_keys table" duration=660.571µs 2024-09-13T13:02:07.617 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.509579409Z level=info msg="Executing migration" id="create secrets table" 2024-09-13T13:02:07.618 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.510152345Z level=info msg="Migration successfully executed" id="create secrets table" duration=573.127µs 2024-09-13T13:02:07.618 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.510602236Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-13T13:02:07.618 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.533073265Z level=info msg="Migration successfully executed" id="rename data_keys name column to id" duration=22.469868ms 2024-09-13T13:02:07.618 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.533472571Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-13T13:02:07.618 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.537667985Z level=info msg="Migration successfully executed" id="add name column into data_keys" duration=4.19618ms 2024-09-13T13:02:07.618 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.538044182Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-13T13:02:07.618 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.538158251Z level=info msg="Migration successfully executed" id="copy data_keys id column values into name" duration=114.306µs 2024-09-13T13:02:07.618 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.540952867Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-13T13:02:07.618 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.562329656Z level=info msg="Migration successfully executed" id="rename data_keys name column to label" duration=21.37874ms 2024-09-13T13:02:07.618 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.562722959Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-13T13:02:07.618 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.584102778Z level=info msg="Migration successfully executed" id="rename data_keys id column back to name" duration=21.378729ms 2024-09-13T13:02:07.618 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.584509245Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-13T13:02:07.619 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.585119266Z level=info msg="Migration successfully executed" id="create kv_store table v1" duration=610.004µs 2024-09-13T13:02:07.619 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.587174762Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-13T13:02:07.619 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.587893149Z level=info msg="Migration successfully executed" id="add index kv_store.org_id-namespace-key" duration=718.115µs 2024-09-13T13:02:07.619 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.589965545Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-13T13:02:07.619 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.59011834Z level=info msg="Migration successfully executed" id="update dashboard_uid and panel_id from existing annotations" duration=152.86µs 2024-09-13T13:02:07.619 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.592755831Z level=info msg="Executing migration" id="create permission table" 2024-09-13T13:02:07.619 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.593450219Z level=info msg="Migration successfully executed" id="create permission table" duration=694.329µs 2024-09-13T13:02:07.619 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.593868333Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-13T13:02:07.619 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.594498747Z level=info msg="Migration successfully executed" id="add unique index permission.role_id" duration=630.24µs 2024-09-13T13:02:07.619 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.596690475Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-13T13:02:07.619 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.597454055Z level=info msg="Migration successfully executed" id="add unique index role_id_action_scope" duration=765.391µs 2024-09-13T13:02:07.619 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.599369111Z level=info msg="Executing migration" id="create role table" 2024-09-13T13:02:07.620 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.600070711Z level=info msg="Migration successfully executed" id="create role table" duration=707.886µs 2024-09-13T13:02:07.620 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.601918897Z level=info msg="Executing migration" id="add column display_name" 2024-09-13T13:02:07.620 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.606398734Z level=info msg="Migration successfully executed" id="add column display_name" duration=4.479709ms 2024-09-13T13:02:07.620 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.606836067Z level=info msg="Executing migration" id="add column group_name" 2024-09-13T13:02:07.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:07.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:07.649 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:07.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:07.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:07.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:07.772 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:07.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:07.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:07.894 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:07.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:07.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:07.903 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.611118506Z level=info msg="Migration successfully executed" id="add column group_name" duration=4.28262ms 2024-09-13T13:02:07.904 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.611479759Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-13T13:02:07.904 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.612112367Z level=info msg="Migration successfully executed" id="add index role.org_id" duration=632.601µs 2024-09-13T13:02:07.904 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.614095329Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-13T13:02:07.904 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.614852682Z level=info msg="Migration successfully executed" id="add unique index role_org_id_name" duration=756.478µs 2024-09-13T13:02:07.904 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.616838393Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-13T13:02:07.904 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.617489588Z level=info msg="Migration successfully executed" id="add index role_org_id_uid" duration=651.061µs 2024-09-13T13:02:07.904 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.617856432Z level=info msg="Executing migration" id="create team role table" 2024-09-13T13:02:07.904 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.618400113Z level=info msg="Migration successfully executed" id="create team role table" duration=543.58µs 2024-09-13T13:02:07.904 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.620481855Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-13T13:02:07.904 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.621156881Z level=info msg="Migration successfully executed" id="add index team_role.org_id" duration=674.868µs 2024-09-13T13:02:07.904 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.623031046Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-13T13:02:07.905 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.623723104Z level=info msg="Migration successfully executed" id="add unique index team_role_org_id_team_id_role_id" duration=690.269µs 2024-09-13T13:02:07.905 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.62411547Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-13T13:02:07.905 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.624781459Z level=info msg="Migration successfully executed" id="add index team_role.team_id" duration=664.85µs 2024-09-13T13:02:07.905 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.625159002Z level=info msg="Executing migration" id="create user role table" 2024-09-13T13:02:07.905 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.625738432Z level=info msg="Migration successfully executed" id="create user role table" duration=577.233µs 2024-09-13T13:02:07.905 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.626133085Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-13T13:02:07.906 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.626796875Z level=info msg="Migration successfully executed" id="add index user_role.org_id" duration=650.505µs 2024-09-13T13:02:07.906 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.627161712Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-13T13:02:07.906 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.627828329Z level=info msg="Migration successfully executed" id="add unique index user_role_org_id_user_id_role_id" duration=666.404µs 2024-09-13T13:02:07.906 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.628181296Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-13T13:02:07.906 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.628818744Z level=info msg="Migration successfully executed" id="add index user_role.user_id" duration=638.332µs 2024-09-13T13:02:07.907 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.629175798Z level=info msg="Executing migration" id="create builtin role table" 2024-09-13T13:02:07.907 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.629721369Z level=info msg="Migration successfully executed" id="create builtin role table" duration=545.378µs 2024-09-13T13:02:07.907 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.630125752Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-13T13:02:07.907 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.63077914Z level=info msg="Migration successfully executed" id="add index builtin_role.role_id" duration=651.725µs 2024-09-13T13:02:07.907 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.631144889Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-13T13:02:07.907 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.631807115Z level=info msg="Migration successfully executed" id="add index builtin_role.name" duration=662.188µs 2024-09-13T13:02:07.907 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.632159817Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-13T13:02:07.907 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.637023006Z level=info msg="Migration successfully executed" id="Add column org_id to builtin_role table" duration=4.862458ms 2024-09-13T13:02:07.907 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.6373866Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-13T13:02:07.907 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.638054948Z level=info msg="Migration successfully executed" id="add index builtin_role.org_id" duration=668.503µs 2024-09-13T13:02:07.908 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.638408996Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-13T13:02:07.908 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.639078326Z level=info msg="Migration successfully executed" id="add unique index builtin_role_org_id_role_id_role" duration=669.014µs 2024-09-13T13:02:07.908 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.639429006Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-13T13:02:07.908 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.640217565Z level=info msg="Migration successfully executed" id="Remove unique index role_org_id_uid" duration=788.376µs 2024-09-13T13:02:07.908 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.640560911Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-13T13:02:07.908 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.641212649Z level=info msg="Migration successfully executed" id="add unique index role.uid" duration=651.556µs 2024-09-13T13:02:07.908 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.6415489Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-13T13:02:07.908 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.642057695Z level=info msg="Migration successfully executed" id="create seed assignment table" duration=508.412µs 2024-09-13T13:02:07.908 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.642405357Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-13T13:02:07.908 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.643082882Z level=info msg="Migration successfully executed" id="add unique index builtin_role_role_name" duration=677.269µs 2024-09-13T13:02:07.908 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.643451252Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-13T13:02:07.908 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.648364322Z level=info msg="Migration successfully executed" id="add column hidden to role table" duration=4.914357ms 2024-09-13T13:02:07.908 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.648737534Z level=info msg="Executing migration" id="permission kind migration" 2024-09-13T13:02:07.909 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.653442935Z level=info msg="Migration successfully executed" id="permission kind migration" duration=4.705171ms 2024-09-13T13:02:07.909 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.653818248Z level=info msg="Executing migration" id="permission attribute migration" 2024-09-13T13:02:07.909 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.658339035Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=4.520631ms 2024-09-13T13:02:07.909 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.658693828Z level=info msg="Executing migration" id="permission identifier migration" 2024-09-13T13:02:07.909 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.663218163Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=4.524175ms 2024-09-13T13:02:07.909 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.66358084Z level=info msg="Executing migration" id="add permission identifier index" 2024-09-13T13:02:07.909 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.664276759Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=696.047µs 2024-09-13T13:02:07.909 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.664643239Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-09-13T13:02:07.909 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.665367642Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=724.057µs 2024-09-13T13:02:07.909 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.665719225Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-09-13T13:02:07.909 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.666366239Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=647.111µs 2024-09-13T13:02:07.910 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.66673892Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-13T13:02:07.911 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.66732736Z level=info msg="Migration successfully executed" id="create query_history table v1" duration=588.297µs 2024-09-13T13:02:07.911 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.667682839Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-13T13:02:07.911 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.668347767Z level=info msg="Migration successfully executed" id="add index query_history.org_id-created_by-datasource_uid" duration=664.822µs 2024-09-13T13:02:07.911 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.668704722Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-13T13:02:07.911 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.668752871Z level=info msg="Migration successfully executed" id="alter table query_history alter column created_by type to bigint" duration=48.314µs 2024-09-13T13:02:07.911 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.669164019Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-09-13T13:02:07.911 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.669187076Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=23.425µs 2024-09-13T13:02:07.911 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.669549823Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-13T13:02:07.911 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.669818309Z level=info msg="Migration successfully executed" id="teams permissions migration" duration=268.41µs 2024-09-13T13:02:07.911 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.670203839Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-13T13:02:07.911 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.670530993Z level=info msg="Migration successfully executed" id="dashboard permissions" duration=327.75µs 2024-09-13T13:02:07.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.670914151Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-13T13:02:07.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.671334288Z level=info msg="Migration successfully executed" id="dashboard permissions uid scopes" duration=420.162µs 2024-09-13T13:02:07.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.671822193Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-13T13:02:07.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.671976382Z level=info msg="Migration successfully executed" id="drop managed folder create actions" duration=152.559µs 2024-09-13T13:02:07.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.672366235Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-13T13:02:07.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.67267425Z level=info msg="Migration successfully executed" id="alerting notification permissions" duration=290.603µs 2024-09-13T13:02:07.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.67306794Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-13T13:02:07.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.673571261Z level=info msg="Migration successfully executed" id="create query_history_star table v1" duration=503.18µs 2024-09-13T13:02:07.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.673907997Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-13T13:02:07.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.67455387Z level=info msg="Migration successfully executed" id="add index query_history.user_id-query_uid" duration=645.459µs 2024-09-13T13:02:07.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.674883761Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-13T13:02:07.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.679508455Z level=info msg="Migration successfully executed" id="add column org_id in query_history_star" duration=4.624468ms 2024-09-13T13:02:07.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.679864358Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-13T13:02:07.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.679903108Z level=info msg="Migration successfully executed" id="alter table query_history_star_mig column user_id type to bigint" duration=39.179µs 2024-09-13T13:02:07.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.682908932Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-13T13:02:07.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.68369813Z level=info msg="Migration successfully executed" id="create correlation table v1" duration=791.128µs 2024-09-13T13:02:07.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.685586208Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-13T13:02:07.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.686256348Z level=info msg="Migration successfully executed" id="add index correlations.uid" duration=671.316µs 2024-09-13T13:02:07.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.688577855Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-13T13:02:07.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.68925514Z level=info msg="Migration successfully executed" id="add index correlations.source_uid" duration=678.632µs 2024-09-13T13:02:07.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.689589255Z level=info msg="Executing migration" id="add correlation config column" 2024-09-13T13:02:07.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.694717697Z level=info msg="Migration successfully executed" id="add correlation config column" duration=5.125745ms 2024-09-13T13:02:07.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.695183524Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-09-13T13:02:07.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.695939881Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=757.365µs 2024-09-13T13:02:07.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.696283049Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-09-13T13:02:07.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.696985157Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=702.442µs 2024-09-13T13:02:07.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.702935068Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-09-13T13:02:07.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.716328906Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=13.393633ms 2024-09-13T13:02:07.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.716691813Z level=info msg="Executing migration" id="create correlation v2" 2024-09-13T13:02:07.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.717433093Z level=info msg="Migration successfully executed" id="create correlation v2" duration=740.968µs 2024-09-13T13:02:07.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.717800419Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-09-13T13:02:07.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.718459699Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=659.335µs 2024-09-13T13:02:07.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.718820946Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-09-13T13:02:07.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.719532839Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=711.771µs 2024-09-13T13:02:07.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.719892146Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-09-13T13:02:07.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.720562188Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=669.95µs 2024-09-13T13:02:07.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.720982882Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-09-13T13:02:07.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.721149604Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=166.824µs 2024-09-13T13:02:07.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.721828344Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-09-13T13:02:07.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.722363274Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=534.838µs 2024-09-13T13:02:07.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.72271922Z level=info msg="Executing migration" id="add provisioning column" 2024-09-13T13:02:07.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.727532265Z level=info msg="Migration successfully executed" id="add provisioning column" duration=4.812906ms 2024-09-13T13:02:07.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.727906131Z level=info msg="Executing migration" id="create entity_events table" 2024-09-13T13:02:07.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.72843223Z level=info msg="Migration successfully executed" id="create entity_events table" duration=524.088µs 2024-09-13T13:02:07.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.72880573Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-13T13:02:07.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.72948295Z level=info msg="Migration successfully executed" id="create dashboard public config v1" duration=677.047µs 2024-09-13T13:02:07.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.729844141Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-13T13:02:07.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.730110151Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-13T13:02:07.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.730673438Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T13:02:07.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.730975686Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T13:02:07.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.733659858Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-13T13:02:07.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.73422323Z level=info msg="Migration successfully executed" id="Drop old dashboard public config table" duration=563.205µs 2024-09-13T13:02:07.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.734594762Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-13T13:02:07.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.735252216Z level=info msg="Migration successfully executed" id="recreate dashboard public config v1" duration=657.37µs 2024-09-13T13:02:07.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.737505682Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-13T13:02:07.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.738198277Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v1" duration=692.269µs 2024-09-13T13:02:07.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.740114938Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T13:02:07.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.740834467Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" duration=719.07µs 2024-09-13T13:02:07.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.742842931Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-13T13:02:07.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.743497077Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_public_config_uid - v2" duration=653.979µs 2024-09-13T13:02:07.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.743861583Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-13T13:02:07.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.744533937Z level=info msg="Migration successfully executed" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=672.413µs 2024-09-13T13:02:07.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.746651967Z level=info msg="Executing migration" id="Drop public config table" 2024-09-13T13:02:07.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.747228741Z level=info msg="Migration successfully executed" id="Drop public config table" duration=576.615µs 2024-09-13T13:02:07.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.749701291Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-13T13:02:07.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.750472866Z level=info msg="Migration successfully executed" id="Recreate dashboard public config v2" duration=773.948µs 2024-09-13T13:02:07.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.75227538Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-13T13:02:07.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.752961738Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v2" duration=686.288µs 2024-09-13T13:02:07.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.753300507Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-13T13:02:07.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.75399426Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=693.299µs 2024-09-13T13:02:07.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.75611099Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-13T13:02:07.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.756780128Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_access_token - v2" duration=670.89µs 2024-09-13T13:02:07.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.7588941Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-13T13:02:07.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.776034565Z level=info msg="Migration successfully executed" id="Rename table dashboard_public_config to dashboard_public - v2" duration=17.161169ms 2024-09-13T13:02:07.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.776520542Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-13T13:02:07.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.781636192Z level=info msg="Migration successfully executed" id="add annotations_enabled column" duration=5.116007ms 2024-09-13T13:02:07.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.782059736Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-13T13:02:07.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.786853939Z level=info msg="Migration successfully executed" id="add time_selection_enabled column" duration=4.794035ms 2024-09-13T13:02:07.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.787221835Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-13T13:02:07.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.787363718Z level=info msg="Migration successfully executed" id="delete orphaned public dashboards" duration=142.088µs 2024-09-13T13:02:07.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.790030538Z level=info msg="Executing migration" id="add share column" 2024-09-13T13:02:07.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.794816847Z level=info msg="Migration successfully executed" id="add share column" duration=4.7861ms 2024-09-13T13:02:07.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.795164119Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-09-13T13:02:07.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.79529411Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=130.177µs 2024-09-13T13:02:07.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.795674166Z level=info msg="Executing migration" id="create file table" 2024-09-13T13:02:07.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.796278567Z level=info msg="Migration successfully executed" id="create file table" duration=604.251µs 2024-09-13T13:02:07.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.798425291Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-13T13:02:07.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.799115589Z level=info msg="Migration successfully executed" id="file table idx: path natural pk" duration=691.596µs 2024-09-13T13:02:07.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.801307934Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-13T13:02:07.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.802122558Z level=info msg="Migration successfully executed" id="file table idx: parent_folder_path_hash fast folder retrieval" duration=816.779µs 2024-09-13T13:02:07.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.803750626Z level=info msg="Executing migration" id="create file_meta table" 2024-09-13T13:02:07.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.804268781Z level=info msg="Migration successfully executed" id="create file_meta table" duration=517.962µs 2024-09-13T13:02:07.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.804622645Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-13T13:02:07.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.805300494Z level=info msg="Migration successfully executed" id="file table idx: path key" duration=677.598µs 2024-09-13T13:02:07.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.807483826Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-13T13:02:07.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.807539386Z level=info msg="Migration successfully executed" id="set path collation in file table" duration=55.895µs 2024-09-13T13:02:07.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.809987184Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-09-13T13:02:07.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.81002507Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=38.226µs 2024-09-13T13:02:07.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.812743301Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-13T13:02:07.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.813069067Z level=info msg="Migration successfully executed" id="managed permissions migration" duration=326.936µs 2024-09-13T13:02:07.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.813446648Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-13T13:02:07.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.813566236Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions migration" duration=119.755µs 2024-09-13T13:02:07.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.81592937Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-13T13:02:07.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.816732215Z level=info msg="Migration successfully executed" id="RBAC action name migrator" duration=802.705µs 2024-09-13T13:02:07.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.817210328Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-13T13:02:07.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.822118584Z level=info msg="Migration successfully executed" id="Add UID column to playlist" duration=4.90797ms 2024-09-13T13:02:07.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.822477008Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-13T13:02:07.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.822578528Z level=info msg="Migration successfully executed" id="Update uid column values in playlist" duration=101.533µs 2024-09-13T13:02:07.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.825245385Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-13T13:02:07.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.826005098Z level=info msg="Migration successfully executed" id="Add index for uid in playlist" duration=759.23µs 2024-09-13T13:02:07.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.826358543Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-13T13:02:07.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.82658604Z level=info msg="Migration successfully executed" id="update group index for alert rules" duration=228.052µs 2024-09-13T13:02:07.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.82903553Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-13T13:02:07.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.82916142Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated migration" duration=124.738µs 2024-09-13T13:02:07.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.831889942Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-13T13:02:07.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.832170098Z level=info msg="Migration successfully executed" id="admin only folder/dashboard permission" duration=280.017µs 2024-09-13T13:02:07.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.834576957Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-13T13:02:07.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.839479428Z level=info msg="Migration successfully executed" id="add action column to seed_assignment" duration=4.902165ms 2024-09-13T13:02:07.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.839848459Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-13T13:02:07.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.844731214Z level=info msg="Migration successfully executed" id="add scope column to seed_assignment" duration=4.88262ms 2024-09-13T13:02:07.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.845091294Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-13T13:02:07.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.845755085Z level=info msg="Migration successfully executed" id="remove unique index builtin_role_role_name before nullable update" duration=663.931µs 2024-09-13T13:02:07.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.846112819Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-13T13:02:07.924 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.903344903Z level=info msg="Migration successfully executed" id="update seed_assignment role_name column to nullable" duration=57.228602ms 2024-09-13T13:02:07.924 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.903931805Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-13T13:02:07.924 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.904776481Z level=info msg="Migration successfully executed" id="add unique index builtin_role_name back" duration=844.854µs 2024-09-13T13:02:07.924 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.905137152Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-13T13:02:07.924 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.905834631Z level=info msg="Migration successfully executed" id="add unique index builtin_role_action_scope" duration=697.226µs 2024-09-13T13:02:07.924 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.906262637Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-13T13:02:07.997 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:08.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:08.117 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[29471]: 4.1c scrub starts 2024-09-13T13:02:08.117 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[29471]: 4.1c scrub ok 2024-09-13T13:02:08.117 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[29471]: Deploying daemon node-exporter.a on smithi052 2024-09-13T13:02:08.117 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[29471]: 3.b scrub starts 2024-09-13T13:02:08.117 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[29471]: 3.b scrub ok 2024-09-13T13:02:08.117 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:08.118 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3012957211' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:02:08.118 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[36682]: 4.1c scrub starts 2024-09-13T13:02:08.118 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[36682]: 4.1c scrub ok 2024-09-13T13:02:08.118 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[36682]: Deploying daemon node-exporter.a on smithi052 2024-09-13T13:02:08.118 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[36682]: 3.b scrub starts 2024-09-13T13:02:08.118 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[36682]: 3.b scrub ok 2024-09-13T13:02:08.118 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:08.118 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:07 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3012957211' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:02:08.121 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:08.149 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:08.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:08.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-mon[33178]: 4.1c scrub starts 2024-09-13T13:02:08.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-mon[33178]: 4.1c scrub ok 2024-09-13T13:02:08.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-mon[33178]: Deploying daemon node-exporter.a on smithi052 2024-09-13T13:02:08.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-mon[33178]: 3.b scrub starts 2024-09-13T13:02:08.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-mon[33178]: 3.b scrub ok 2024-09-13T13:02:08.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:08.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3012957211' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:02:08.197 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.923393211Z level=info msg="Migration successfully executed" id="add primary key to seed_assigment" duration=17.131028ms 2024-09-13T13:02:08.197 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.923810582Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-09-13T13:02:08.197 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.928941829Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=5.131618ms 2024-09-13T13:02:08.197 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.929294935Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-09-13T13:02:08.197 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.929491247Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=197.153µs 2024-09-13T13:02:08.197 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.929891882Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-09-13T13:02:08.197 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.930006312Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=115.039µs 2024-09-13T13:02:08.198 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.930695438Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-13T13:02:08.198 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.930848308Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated fixed migration" duration=153.287µs 2024-09-13T13:02:08.198 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.931606738Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-09-13T13:02:08.198 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.931739024Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=131.947µs 2024-09-13T13:02:08.198 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.932511354Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-13T13:02:08.198 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.932654816Z level=info msg="Migration successfully executed" id="migrate external alertmanagers to datsourcse" duration=143.56µs 2024-09-13T13:02:08.198 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.933294655Z level=info msg="Executing migration" id="create folder table" 2024-09-13T13:02:08.198 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.933908794Z level=info msg="Migration successfully executed" id="create folder table" duration=613.944µs 2024-09-13T13:02:08.198 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.934271994Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-13T13:02:08.198 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.935100296Z level=info msg="Migration successfully executed" id="Add index for parent_uid" duration=828.386µs 2024-09-13T13:02:08.198 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.935474092Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-13T13:02:08.198 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.93621907Z level=info msg="Migration successfully executed" id="Add unique index for folder.uid and folder.org_id" duration=746.917µs 2024-09-13T13:02:08.199 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.936589914Z level=info msg="Executing migration" id="Update folder title length" 2024-09-13T13:02:08.199 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.936605749Z level=info msg="Migration successfully executed" id="Update folder title length" duration=16.835µs 2024-09-13T13:02:08.199 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.937446531Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-13T13:02:08.199 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.938202064Z level=info msg="Migration successfully executed" id="Add unique index for folder.title and folder.parent_uid" duration=755.7µs 2024-09-13T13:02:08.199 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.938555363Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-09-13T13:02:08.199 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.939247098Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=689.833µs 2024-09-13T13:02:08.199 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.93960595Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-09-13T13:02:08.199 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.940354049Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=747.988µs 2024-09-13T13:02:08.199 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.940734415Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-09-13T13:02:08.199 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.94103779Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=303.404µs 2024-09-13T13:02:08.199 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.941603065Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-09-13T13:02:08.199 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.941803239Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=200.134µs 2024-09-13T13:02:08.199 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.942482563Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-09-13T13:02:08.200 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.943193191Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=710.576µs 2024-09-13T13:02:08.200 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.943676613Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-09-13T13:02:08.200 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.944385963Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=709.239µs 2024-09-13T13:02:08.201 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.944742474Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-09-13T13:02:08.201 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.945452396Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=708.378µs 2024-09-13T13:02:08.201 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.945803064Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-09-13T13:02:08.201 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.946492838Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=690.24µs 2024-09-13T13:02:08.201 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.946854815Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-09-13T13:02:08.201 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.947571748Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=715.399µs 2024-09-13T13:02:08.201 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.947935912Z level=info msg="Executing migration" id="create anon_device table" 2024-09-13T13:02:08.201 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.94848659Z level=info msg="Migration successfully executed" id="create anon_device table" duration=551.434µs 2024-09-13T13:02:08.201 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.948850298Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-09-13T13:02:08.201 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.949601352Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=749.524µs 2024-09-13T13:02:08.201 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.949987094Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-09-13T13:02:08.201 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.950668411Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=681.591µs 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.951056605Z level=info msg="Executing migration" id="create signing_key table" 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.951689899Z level=info msg="Migration successfully executed" id="create signing_key table" duration=633.18µs 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.952082636Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.952793926Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=702.542µs 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.953140261Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.953848614Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=708.594µs 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.954186631Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.954361789Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=175.344µs 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.955029429Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.960173196Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=5.143609ms 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.96053654Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.961001141Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=465.387µs 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.961401551Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-09-13T13:02:08.202 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.962130182Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=728.32µs 2024-09-13T13:02:08.203 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.964495355Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-09-13T13:02:08.203 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.965268393Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=777.19µs 2024-09-13T13:02:08.203 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.967380134Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-09-13T13:02:08.203 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.968227253Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=849.591µs 2024-09-13T13:02:08.203 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.970091359Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-09-13T13:02:08.203 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.970882468Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=791.102µs 2024-09-13T13:02:08.203 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.97298188Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-09-13T13:02:08.203 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.973678471Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=696.382µs 2024-09-13T13:02:08.203 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.975719065Z level=info msg="Executing migration" id="create sso_setting table" 2024-09-13T13:02:08.203 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.976569402Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=850.234µs 2024-09-13T13:02:08.203 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.978667535Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-09-13T13:02:08.203 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.979215021Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=548.403µs 2024-09-13T13:02:08.204 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.981487781Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-09-13T13:02:08.204 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.98167334Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=185.642µs 2024-09-13T13:02:08.204 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.98453773Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-09-13T13:02:08.204 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.984621689Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=87.288µs 2024-09-13T13:02:08.204 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.986913309Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-09-13T13:02:08.204 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.992176942Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.263428ms 2024-09-13T13:02:08.204 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.992591645Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-09-13T13:02:08.204 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.998014179Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=5.422243ms 2024-09-13T13:02:08.204 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.99838759Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-09-13T13:02:08.204 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:07.998607059Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=219.492µs 2024-09-13T13:02:08.204 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=migrator t=2024-09-13T13:02:08.001367598Z level=info msg="migrations completed" performed=547 skipped=0 duration=1.257972018s 2024-09-13T13:02:08.204 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=sqlstore t=2024-09-13T13:02:08.002355205Z level=info msg="Created default organization" 2024-09-13T13:02:08.205 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=secrets t=2024-09-13T13:02:08.004423002Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-13T13:02:08.205 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=plugin.store t=2024-09-13T13:02:08.022909413Z level=info msg="Loading plugins..." 2024-09-13T13:02:08.205 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=local.finder t=2024-09-13T13:02:08.062361601Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-13T13:02:08.205 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=plugin.store t=2024-09-13T13:02:08.062379566Z level=info msg="Plugins loaded" count=55 duration=39.471686ms 2024-09-13T13:02:08.205 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=query_data t=2024-09-13T13:02:08.064442908Z level=info msg="Query Service initialization" 2024-09-13T13:02:08.205 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=live.push_http t=2024-09-13T13:02:08.066873274Z level=info msg="Live Push Gateway initialization" 2024-09-13T13:02:08.205 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=ngalert.migration t=2024-09-13T13:02:08.067843832Z level=info msg=Starting 2024-09-13T13:02:08.205 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=ngalert.migration t=2024-09-13T13:02:08.068123278Z level=info msg="Applying transition" currentType=Legacy desiredType=UnifiedAlerting cleanOnDowngrade=false cleanOnUpgrade=false 2024-09-13T13:02:08.205 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=ngalert.migration orgID=1 t=2024-09-13T13:02:08.068384002Z level=info msg="Migrating alerts for organisation" 2024-09-13T13:02:08.205 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=ngalert.migration orgID=1 t=2024-09-13T13:02:08.068841018Z level=info msg="Alerts found to migrate" alerts=0 2024-09-13T13:02:08.205 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=ngalert.migration t=2024-09-13T13:02:08.06994157Z level=info msg="Completed alerting migration" 2024-09-13T13:02:08.205 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=ngalert.state.manager t=2024-09-13T13:02:08.080549593Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-09-13T13:02:08.205 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=infra.usagestats.collector t=2024-09-13T13:02:08.08197837Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=provisioning.datasources t=2024-09-13T13:02:08.083679932Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=provisioning.datasources t=2024-09-13T13:02:08.095910124Z level=info msg="inserting datasource from configuration" name=Loki uid=P8E80F9AEF21F6940 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=provisioning.alerting t=2024-09-13T13:02:08.108755416Z level=info msg="starting to provision alerting" 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=provisioning.alerting t=2024-09-13T13:02:08.108765466Z level=info msg="finished to provision alerting" 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=ngalert.state.manager t=2024-09-13T13:02:08.108840006Z level=info msg="Warming state cache for startup" 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=grafanaStorageLogger t=2024-09-13T13:02:08.108984939Z level=info msg="Storage starting" 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=ngalert.state.manager t=2024-09-13T13:02:08.109117708Z level=info msg="State cache has been initialized" states=0 duration=276.911µs 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=ngalert.multiorg.alertmanager t=2024-09-13T13:02:08.108850561Z level=info msg="Starting MultiOrg Alertmanager" 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=ngalert.scheduler t=2024-09-13T13:02:08.109144299Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=ticker t=2024-09-13T13:02:08.10918404Z level=info msg=starting first_tick=2024-09-13T13:02:10Z 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=http.server t=2024-09-13T13:02:08.111187848Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=http.server t=2024-09-13T13:02:08.111416704Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=provisioning.dashboard t=2024-09-13T13:02:08.12896482Z level=info msg="starting to provision dashboards" 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=grafana.update.checker t=2024-09-13T13:02:08.195282664Z level=info msg="Update check succeeded" duration=86.220475ms 2024-09-13T13:02:08.206 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=plugins.update.checker t=2024-09-13T13:02:08.199545364Z level=info msg="Update check succeeded" duration=90.333956ms 2024-09-13T13:02:08.207 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:02:08.210 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:08.252 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:08.266 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:02:08.267 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:02:08.277 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:08.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:08.352 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:02:08.380 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:08.402 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:08.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:08.457 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=sqlstore.transactions t=2024-09-13T13:02:08.21980052Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T13:02:08.457 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=sqlstore.transactions t=2024-09-13T13:02:08.230211536Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-13T13:02:08.457 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=sqlstore.transactions t=2024-09-13T13:02:08.240676601Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-13T13:02:08.458 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=sqlstore.transactions t=2024-09-13T13:02:08.251260744Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-09-13T13:02:08.458 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=sqlstore.transactions t=2024-09-13T13:02:08.262178686Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T13:02:08.458 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=provisioning.dashboard t=2024-09-13T13:02:08.327592857Z level=info msg="finished to provision dashboards" 2024-09-13T13:02:08.458 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=grafana-apiserver t=2024-09-13T13:02:08.457216891Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-09-13T13:02:08.503 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:08.525 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:08.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:08.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:08.627 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:08.649 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:08.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:08.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:08.772 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:08.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:08.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:08.820 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=grafana-apiserver t=2024-09-13T13:02:08.457668714Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-09-13T13:02:08.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:08.896 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:08.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:08.997 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:09.019 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:09.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:09.121 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:09.142 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:09.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:09.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:09.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[29471]: 3.1c scrub starts 2024-09-13T13:02:09.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[29471]: 3.1c scrub ok 2024-09-13T13:02:09.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[29471]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:02:09.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[29471]: 3.8 scrub starts 2024-09-13T13:02:09.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[29471]: 3.8 scrub ok 2024-09-13T13:02:09.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:02:09.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3012957211' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:02:09.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[29471]: osdmap e81: 8 total, 8 up, 8 in 2024-09-13T13:02:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[36682]: 3.1c scrub starts 2024-09-13T13:02:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[36682]: 3.1c scrub ok 2024-09-13T13:02:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[36682]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:02:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[36682]: 3.8 scrub starts 2024-09-13T13:02:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[36682]: 3.8 scrub ok 2024-09-13T13:02:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:02:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3012957211' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:02:09.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:08 smithi052 ceph-mon[36682]: osdmap e81: 8 total, 8 up, 8 in 2024-09-13T13:02:09.244 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:09.270 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:09.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:09.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-mon[33178]: 3.1c scrub starts 2024-09-13T13:02:09.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-mon[33178]: 3.1c scrub ok 2024-09-13T13:02:09.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-mon[33178]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:02:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-mon[33178]: 3.8 scrub starts 2024-09-13T13:02:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-mon[33178]: 3.8 scrub ok 2024-09-13T13:02:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:02:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3012957211' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:02:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:08 smithi167 ceph-mon[33178]: osdmap e81: 8 total, 8 up, 8 in 2024-09-13T13:02:09.333 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T13:02:09.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:09.393 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:09.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:09.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:09.462 INFO:teuthology.orchestra.run.smithi052.stderr:no valid command found; 10 closest matches: 2024-09-13T13:02:09.462 INFO:teuthology.orchestra.run.smithi052.stderr:0 2024-09-13T13:02:09.462 INFO:teuthology.orchestra.run.smithi052.stderr:1 2024-09-13T13:02:09.462 INFO:teuthology.orchestra.run.smithi052.stderr:2 2024-09-13T13:02:09.462 INFO:teuthology.orchestra.run.smithi052.stderr:abort 2024-09-13T13:02:09.462 INFO:teuthology.orchestra.run.smithi052.stderr:assert 2024-09-13T13:02:09.462 INFO:teuthology.orchestra.run.smithi052.stderr:bench [] [] [] [] 2024-09-13T13:02:09.462 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs debug_inject_read_zeros 2024-09-13T13:02:09.462 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs files list 2024-09-13T13:02:09.462 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs stats 2024-09-13T13:02:09.462 INFO:teuthology.orchestra.run.smithi052.stderr:bluestore allocator dump block 2024-09-13T13:02:09.463 INFO:teuthology.orchestra.run.smithi052.stderr:Error EINVAL: invalid command 2024-09-13T13:02:09.495 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:09.517 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:09.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:09.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:09.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:09.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:09.642 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:09.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:09.743 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:09.765 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:09.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:09.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:09.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:09.889 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:09.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:09.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:10.012 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:10.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:10.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:10.018 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:09 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:02:10.019 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:09 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:02:10.019 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:09 smithi052 ceph-mon[36682]: osdmap e82: 8 total, 8 up, 8 in 2024-09-13T13:02:10.019 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:09 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:02:10.019 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:09 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:02:10.019 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:09 smithi052 ceph-mon[29471]: osdmap e82: 8 total, 8 up, 8 in 2024-09-13T13:02:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T13:02:10.020 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T13:02:10.020 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-13T13:02:10.114 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:10.138 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:10.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:10.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:10.240 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:10.262 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:10.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:10.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:09 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:02:10.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:09 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:02:10.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:09 smithi167 ceph-mon[33178]: osdmap e82: 8 total, 8 up, 8 in 2024-09-13T13:02:10.363 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:10.385 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:10.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:10.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:10.487 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:10.509 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:10.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:10.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:10.632 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:10.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:10.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:10.736 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:10.760 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:10.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:10.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:10.862 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:10.871 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:10.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:10.884 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:10.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:10.894 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:10.985 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:11.008 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:11.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:11.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:11.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:10 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:02:11.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:10 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:02:11.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:10 smithi052 ceph-mon[36682]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:02:11.031 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:10 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1054143134' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-13T13:02:11.031 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:10 smithi052 ceph-mon[36682]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T13:02:11.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:10 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:02:11.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:10 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:02:11.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:10 smithi052 ceph-mon[29471]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:02:11.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:10 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1054143134' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-13T13:02:11.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:10 smithi052 ceph-mon[29471]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T13:02:11.110 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:11.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:11.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:11.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:11.233 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:11.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:11.260 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:11.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:11.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:10 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:02:11.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:10 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:02:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:10 smithi167 ceph-mon[33178]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:02:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:10 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1054143134' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-13T13:02:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:10 smithi167 ceph-mon[33178]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T13:02:11.358 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'unique_pool_0' created 2024-09-13T13:02:11.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:11.382 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:11.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:11.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:11.459 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:02:11.459 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:02:11.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:11.505 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:11.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:11.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:11.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:11.630 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:11.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:11.672 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:11 smithi052 bash[70077]: Getting image source signatures 2024-09-13T13:02:11.673 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:11 smithi052 bash[70077]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-13T13:02:11.673 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:11 smithi052 bash[70077]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-13T13:02:11.673 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:11 smithi052 bash[70077]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-13T13:02:11.732 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:11.755 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:11.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:11.857 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:11.880 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:11.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:11.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:11.982 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:12.005 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:12.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:12.085 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:12.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:12.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:12.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:12.233 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:12.258 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:12.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:12.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:12.359 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:12.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:12.390 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:12.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:12.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:12.515 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:12.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:12.617 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:12.640 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:12.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:12.658 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:12 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1054143134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T13:02:12.658 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:12 smithi052 ceph-mon[36682]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T13:02:12.658 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:12 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1054143134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T13:02:12.658 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:12 smithi052 ceph-mon[29471]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T13:02:12.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:12.765 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:12.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:12.792 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:02:12.805 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:02:12.805 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:02:12.805 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:02:12.805 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:02:12.805 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:02:12.806 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:02:12.811 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:02:12.811 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:02:12.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:12 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1054143134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T13:02:12.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:12 smithi167 ceph-mon[33178]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T13:02:12.829 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:02:12.829 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:02:12.847 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:02:12.847 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:02:12.865 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:02:12.865 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:02:12.867 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:12.883 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:02:12.883 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:02:12.891 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:12.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:12.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:13.019 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:13.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:13.120 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:13.147 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:13.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:13.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:13.249 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:13.271 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:13.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:13.373 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:13.395 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:13.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:13.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:13.497 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:13.523 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:13.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:13.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:13.613 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:13 smithi052 ceph-mon[36682]: pgmap v34: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:02:13.660 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:13 smithi052 ceph-mon[36682]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:02:13.661 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:13 smithi052 ceph-mon[36682]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T13:02:13.661 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:13 smithi052 ceph-mon[36682]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T13:02:13.661 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2672851272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:13.661 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:13 smithi052 ceph-mon[29471]: pgmap v34: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:02:13.661 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:13 smithi052 ceph-mon[29471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:02:13.661 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:13 smithi052 ceph-mon[29471]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T13:02:13.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:13 smithi052 ceph-mon[29471]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T13:02:13.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2672851272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:13.663 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T13:02:13.663 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:13.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:13.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:13.786 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:13.808 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:13.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:13.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:13.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:02:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:02:13.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:13 smithi167 ceph-mon[33178]: pgmap v34: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T13:02:13.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:13 smithi167 ceph-mon[33178]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:02:13.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:13 smithi167 ceph-mon[33178]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T13:02:13.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:13 smithi167 ceph-mon[33178]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T13:02:13.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2672851272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:13.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:13.936 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:13.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:13.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:13.989 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:13 smithi052 bash[70077]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-13T13:02:14.038 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:14.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:14.062 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:14.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:14.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:14.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:14.190 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:14.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:14.291 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:14.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:14.315 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:14.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:14.367 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:14 smithi052 bash[70077]: Writing manifest to image destination 2024-09-13T13:02:14.416 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:14.420 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-13T13:02:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:14.440 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:14.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:14.541 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:14.566 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:14.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:14.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:14.668 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:14.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:14.691 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:14.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:14 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:14.739 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:14 smithi052 podman[70077]: 2024-09-13 13:02:14.439012208 +0000 UTC m=+3.521720014 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-13T13:02:14.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:14 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:14.792 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:14.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:14.815 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:14.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:14.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:14 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:14.917 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:14.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:14.940 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:14.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:15.041 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:15.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:15.108 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:14 smithi052 podman[70077]: 2024-09-13 13:02:14.990461136 +0000 UTC m=+4.073168973 container create 537bfc380255800bd7c5f4abb731e376ab876e2f0966684535ce6f09de19e8ea (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T13:02:15.165 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:15.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:15.188 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:15.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:15.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:15.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:15.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:15.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:15.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:15.440 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:15.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-mon[36682]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:02:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:15.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-mon[29471]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:02:15.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:15.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:15.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:15.490 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 podman[70077]: 2024-09-13 13:02:15.209437749 +0000 UTC m=+4.292145543 container init 537bfc380255800bd7c5f4abb731e376ab876e2f0966684535ce6f09de19e8ea (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T13:02:15.490 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 podman[70077]: 2024-09-13 13:02:15.212461341 +0000 UTC m=+4.295169141 container start 537bfc380255800bd7c5f4abb731e376ab876e2f0966684535ce6f09de19e8ea (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T13:02:15.490 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.216Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-13T13:02:15.490 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.216Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-13T13:02:15.490 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T13:02:15.490 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T13:02:15.490 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T13:02:15.491 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T13:02:15.491 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T13:02:15.491 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T13:02:15.491 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T13:02:15.491 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T13:02:15.491 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T13:02:15.491 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T13:02:15.491 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T13:02:15.491 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T13:02:15.491 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T13:02:15.491 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T13:02:15.491 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T13:02:15.492 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T13:02:15.492 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T13:02:15.492 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T13:02:15.492 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T13:02:15.492 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T13:02:15.492 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T13:02:15.492 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T13:02:15.492 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T13:02:15.492 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T13:02:15.492 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T13:02:15.492 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T13:02:15.492 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T13:02:15.492 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T13:02:15.493 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T13:02:15.493 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T13:02:15.493 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T13:02:15.493 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T13:02:15.493 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T13:02:15.493 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T13:02:15.493 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T13:02:15.493 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T13:02:15.493 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T13:02:15.494 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T13:02:15.494 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T13:02:15.494 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T13:02:15.494 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T13:02:15.494 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T13:02:15.494 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.217Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T13:02:15.494 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.218Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T13:02:15.494 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.218Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T13:02:15.494 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.218Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T13:02:15.494 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.218Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T13:02:15.495 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.218Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T13:02:15.495 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.218Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T13:02:15.495 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.218Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-13T13:02:15.495 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a[70788]: ts=2024-09-13T13:02:15.218Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T13:02:15.495 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 bash[70077]: 537bfc380255800bd7c5f4abb731e376ab876e2f0966684535ce6f09de19e8ea 2024-09-13T13:02:15.495 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:02:15 smithi052 systemd[1]: Started Ceph node-exporter.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:02:15.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:15.564 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:15.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:15.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:15.611 INFO:teuthology.orchestra.run.smithi052.stderr:no valid command found; 10 closest matches: 2024-09-13T13:02:15.611 INFO:teuthology.orchestra.run.smithi052.stderr:0 2024-09-13T13:02:15.611 INFO:teuthology.orchestra.run.smithi052.stderr:1 2024-09-13T13:02:15.611 INFO:teuthology.orchestra.run.smithi052.stderr:2 2024-09-13T13:02:15.611 INFO:teuthology.orchestra.run.smithi052.stderr:abort 2024-09-13T13:02:15.611 INFO:teuthology.orchestra.run.smithi052.stderr:assert 2024-09-13T13:02:15.611 INFO:teuthology.orchestra.run.smithi052.stderr:bench [] [] [] [] 2024-09-13T13:02:15.612 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs debug_inject_read_zeros 2024-09-13T13:02:15.612 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs files list 2024-09-13T13:02:15.612 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs stats 2024-09-13T13:02:15.612 INFO:teuthology.orchestra.run.smithi052.stderr:bluestore allocator dump block 2024-09-13T13:02:15.612 INFO:teuthology.orchestra.run.smithi052.stderr:Error EINVAL: invalid command 2024-09-13T13:02:15.665 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:15.687 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:15.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:15.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:15.711 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:15 smithi167 ceph-mon[33178]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:02:15.712 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:15 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:15.712 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:15 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:15.712 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:15 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:15.789 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:15.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:15.813 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:15.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:15.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:15.937 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:15.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:16.039 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:16.062 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:16.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:16.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:16.187 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:16.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:16.289 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:16.312 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:16.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:16.373 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:16 smithi167 bash[61727]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-13T13:02:16.414 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:16.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:16.437 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:16.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:16.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:16 smithi052 ceph-mon[29471]: Deploying daemon node-exporter.b on smithi167 2024-09-13T13:02:16.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:16 smithi052 ceph-mon[36682]: Deploying daemon node-exporter.b on smithi167 2024-09-13T13:02:16.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:02:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:02:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:02:16.538 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:16.561 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:16.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:16.632 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T13:02:16.633 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T13:02:16.633 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-13T13:02:16.640 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:16.645 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:16.649 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:16.664 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:16.669 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:16 smithi167 ceph-mon[33178]: Deploying daemon node-exporter.b on smithi167 2024-09-13T13:02:16.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:16.687 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:16.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:16.788 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:16.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:16.811 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:16.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:16.892 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:02:16.912 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:16.934 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:16.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:17.036 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:17.059 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:17.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:17.070 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:16 smithi167 bash[61727]: Getting image source signatures 2024-09-13T13:02:17.070 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:16 smithi167 bash[61727]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-13T13:02:17.070 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:16 smithi167 bash[61727]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-13T13:02:17.070 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:16 smithi167 bash[61727]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-13T13:02:17.160 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:17.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:17.182 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:17.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:17.283 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:17.305 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:17.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:17.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:17.406 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:17.428 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:17.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:17 smithi052 ceph-mon[36682]: pgmap v38: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 926 B/s rd, 0 op/s 2024-09-13T13:02:17.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:17 smithi052 ceph-mon[29471]: pgmap v38: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 926 B/s rd, 0 op/s 2024-09-13T13:02:17.531 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:17.554 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:17.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:17.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:17.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:17.678 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:17.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:17.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:17.780 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:17.804 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:17.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:17.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:17.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:17 smithi167 ceph-mon[33178]: pgmap v38: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 926 B/s rd, 0 op/s 2024-09-13T13:02:17.828 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:17.906 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:17.928 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:17.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:17.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:18.030 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:18.054 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:18.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:18.155 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:18.180 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:18.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:18.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:18.282 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:18.306 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:18.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:18.390 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:02:18.390 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:02:18.390 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 0 2024-09-13T13:02:18.390 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:02:18.406 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-13T13:02:18.406 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:02:18.407 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:18.433 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:18.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:18.535 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:18.560 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:18.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:18.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:18 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:18 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:02:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:18 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2925861733' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:02:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:18 smithi167 ceph-mon[33178]: from='client.34760 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:02:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:18 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1479759963' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:02:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:18 smithi167 ceph-mon[33178]: from='client.34766 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:02:18.626 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:18 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:18.626 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:18 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:02:18.626 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:18 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2925861733' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:02:18.627 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:18 smithi052 ceph-mon[36682]: from='client.34760 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:02:18.627 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:18 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1479759963' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:02:18.627 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:18 smithi052 ceph-mon[36682]: from='client.34766 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:02:18.627 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:18 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:18.627 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:18 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:02:18.627 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:18 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2925861733' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:02:18.627 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:18 smithi052 ceph-mon[29471]: from='client.34760 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:02:18.627 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:18 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1479759963' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:02:18.627 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:18 smithi052 ceph-mon[29471]: from='client.34766 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:02:18.662 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:18.684 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:18.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:18.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:18.788 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:18.811 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:18.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:18.913 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:18.935 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:18.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:19.037 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:19.059 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:19.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:19.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:19.161 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:19.183 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:19.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:19.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:19.285 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:19.307 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:19.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:19.408 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:19.434 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:19.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:19.522 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T13:02:19.536 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:19.561 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:19.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:19.620 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:02:19.620 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:02:19.620 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:02:19.620 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:02:19.620 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:02:19.620 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:02:19.620 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:02:19.621 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:02:19.621 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:02:19.621 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:02:19.621 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:02:19.621 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:02:19.621 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:02:19.621 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:02:19.622 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:02:19.622 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:02:19.663 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:19.688 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:19.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:19.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:19 smithi052 ceph-mon[29471]: pgmap v39: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T13:02:19.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:19 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:02:19.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:19 smithi052 ceph-mon[29471]: from='client.34760 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:02:19.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:19 smithi052 ceph-mon[29471]: from='client.34766 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:02:19.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:19 smithi052 ceph-mon[29471]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T13:02:19.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:19 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3257900406' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:19.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:19 smithi052 ceph-mon[36682]: pgmap v39: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T13:02:19.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:19 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:02:19.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:19 smithi052 ceph-mon[36682]: from='client.34760 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:02:19.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:19 smithi052 ceph-mon[36682]: from='client.34766 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:02:19.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:19 smithi052 ceph-mon[36682]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T13:02:19.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:19 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3257900406' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:19.790 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:19.814 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:19.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:19.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:19 smithi167 ceph-mon[33178]: pgmap v39: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T13:02:19.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:19 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:02:19.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:19 smithi167 ceph-mon[33178]: from='client.34760 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:02:19.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:19 smithi167 ceph-mon[33178]: from='client.34766 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:02:19.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:19 smithi167 ceph-mon[33178]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T13:02:19.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:19 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3257900406' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:19.915 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:19.938 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:19.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:19.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:20.040 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:20.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:20.064 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:20.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:20.165 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:20.187 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:20.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:20.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:20.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:20.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:20.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:20.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:20.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:20.440 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:20.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:20.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:20.541 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:20.563 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:20.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:20.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:20.665 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:20.692 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:20.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:20.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:20 smithi052 ceph-mon[36682]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:02:20.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:20 smithi052 ceph-mon[29471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:02:20.794 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:20.819 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:20.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:20.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:20 smithi167 ceph-mon[33178]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:02:20.921 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:20.945 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:20.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:20.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:21.046 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:21.072 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:21.073 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:21.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:21.174 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:21.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:21.201 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:21.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:21.302 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:21.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:21.326 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:21.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:21.427 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:21.453 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:21.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:21.537 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:21.555 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:21.578 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:21.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:21.679 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:21.702 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:21.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:21.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:21 smithi052 ceph-mon[36682]: pgmap v41: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T13:02:21.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:21 smithi052 ceph-mon[29471]: pgmap v41: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T13:02:21.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:21.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:21 smithi167 ceph-mon[33178]: pgmap v41: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-13T13:02:21.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:21.839 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:21.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:21.940 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:21.963 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:21.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:21.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:22.042 DEBUG:teuthology.orchestra.run.smithi052:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-13T13:02:22.045 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T13:02:22.045 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T13:02:22.065 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:22.091 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:22.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:22.135 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:hints = 1 2024-09-13T13:02:22.135 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T13:02:22.135 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Object prefix: benchmark_data_smithi052_71865 2024-09-13T13:02:22.135 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:02:22.135 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 0 0 0 0 0 0 - 0 2024-09-13T13:02:22.192 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:22.223 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:22.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:22.325 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:22.347 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:22 smithi167 bash[61727]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-13T13:02:22.354 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:22.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:22.406 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:02:22.406 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:02:22.455 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:22.473 INFO:teuthology.orchestra.run.smithi052.stderr:no valid command found; 10 closest matches: 2024-09-13T13:02:22.473 INFO:teuthology.orchestra.run.smithi052.stderr:0 2024-09-13T13:02:22.474 INFO:teuthology.orchestra.run.smithi052.stderr:1 2024-09-13T13:02:22.474 INFO:teuthology.orchestra.run.smithi052.stderr:2 2024-09-13T13:02:22.474 INFO:teuthology.orchestra.run.smithi052.stderr:abort 2024-09-13T13:02:22.474 INFO:teuthology.orchestra.run.smithi052.stderr:assert 2024-09-13T13:02:22.474 INFO:teuthology.orchestra.run.smithi052.stderr:bench [] [] [] [] 2024-09-13T13:02:22.474 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs debug_inject_read_zeros 2024-09-13T13:02:22.474 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs files list 2024-09-13T13:02:22.474 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs stats 2024-09-13T13:02:22.474 INFO:teuthology.orchestra.run.smithi052.stderr:bluestore allocator dump block 2024-09-13T13:02:22.474 INFO:teuthology.orchestra.run.smithi052.stderr:Error EINVAL: invalid command 2024-09-13T13:02:22.485 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:22.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:22.502 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:02:22.587 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:22.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:22.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:22.639 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:22 smithi167 bash[61727]: Writing manifest to image destination 2024-09-13T13:02:22.718 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:22.771 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:22.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:22.872 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:22.907 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:22.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:22.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:22.989 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:22 smithi167 podman[61727]: 2024-09-13 13:02:22.613108582 +0000 UTC m=+6.406991449 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-13T13:02:22.990 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:22 smithi167 podman[61727]: 2024-09-13 13:02:22.744867951 +0000 UTC m=+6.538750808 container create 5c4f411e5c51c291675468676bc3d8525e5d9e1e95e6e26c40c440af1e14c610 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T13:02:22.990 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:22 smithi167 podman[61727]: 2024-09-13 13:02:22.989539962 +0000 UTC m=+6.783422815 container init 5c4f411e5c51c291675468676bc3d8525e5d9e1e95e6e26c40c440af1e14c610 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T13:02:23.009 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:23.049 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:23.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:23.135 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 1 16 3127 3111 194.427 194.438 0.00432296 0.00513007 2024-09-13T13:02:23.150 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:23.194 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:23.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:23.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:23.229 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T13:02:23.230 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T13:02:23.230 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-13T13:02:23.296 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:23.321 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:22 smithi167 podman[61727]: 2024-09-13 13:02:22.994913536 +0000 UTC m=+6.788796388 container start 5c4f411e5c51c291675468676bc3d8525e5d9e1e95e6e26c40c440af1e14c610 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T13:02:23.321 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.000Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-13T13:02:23.321 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.000Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-13T13:02:23.321 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.002Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T13:02:23.321 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.002Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T13:02:23.321 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T13:02:23.322 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T13:02:23.322 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T13:02:23.322 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T13:02:23.322 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T13:02:23.322 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T13:02:23.322 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T13:02:23.322 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T13:02:23.322 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T13:02:23.322 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T13:02:23.322 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T13:02:23.322 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T13:02:23.323 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T13:02:23.323 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T13:02:23.323 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T13:02:23.323 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T13:02:23.323 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T13:02:23.324 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T13:02:23.324 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T13:02:23.324 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T13:02:23.324 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T13:02:23.324 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T13:02:23.324 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T13:02:23.324 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T13:02:23.324 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T13:02:23.324 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T13:02:23.324 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T13:02:23.325 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T13:02:23.325 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T13:02:23.325 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T13:02:23.325 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T13:02:23.325 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T13:02:23.325 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T13:02:23.325 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T13:02:23.325 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T13:02:23.325 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T13:02:23.325 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T13:02:23.325 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T13:02:23.325 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T13:02:23.326 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T13:02:23.326 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T13:02:23.326 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T13:02:23.326 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T13:02:23.326 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T13:02:23.326 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T13:02:23.326 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T13:02:23.326 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.003Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T13:02:23.326 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.004Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-13T13:02:23.326 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b[62180]: ts=2024-09-13T13:02:23.004Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T13:02:23.327 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 bash[61727]: 5c4f411e5c51c291675468676bc3d8525e5d9e1e95e6e26c40c440af1e14c610 2024-09-13T13:02:23.327 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:02:23 smithi167 systemd[1]: Started Ceph node-exporter.b for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:02:23.367 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:23.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:23.464 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:02:23.469 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:23.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[36682]: pgmap v42: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:02:23.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[36682]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T13:02:23.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:02:23.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[29471]: pgmap v42: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:02:23.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[29471]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T13:02:23.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:23 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:02:23.562 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:23.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:23.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:23.664 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:23.687 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:23.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:23.789 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:23.818 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:23.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:23.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-mon[33178]: pgmap v42: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:02:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-mon[33178]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T13:02:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:02:23.822 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:02:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:02:23.920 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:23.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:23.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:24.054 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:24.093 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:24.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:24.135 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 2 16 5654 5638 176.175 157.938 0.00302189 0.00566306 2024-09-13T13:02:24.195 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:24.232 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:24.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:24.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:24.334 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:24.339 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T13:02:24.363 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:24.401 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:24.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:24.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:24.503 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:24.550 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:24.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:24.630 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:24 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:24.631 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:24 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:24.652 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:24.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:24.697 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:24.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:24.798 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:24.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:24 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:24.822 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:24.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:24.924 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:24.968 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:24.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:25.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:25.097 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:25.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:25.133 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:25.143 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 3 16 8101 8085 168.425 152.938 0.0124814 0.00592156 2024-09-13T13:02:25.143 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:25.198 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:25.249 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:25.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:25.350 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:25.397 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:25.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:25.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:25.499 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:25.536 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:25.537 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:25 smithi052 ceph-mon[29471]: pgmap v44: 148 pgs: 148 active+clean; 140 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 17 MiB/s wr, 280 op/s 2024-09-13T13:02:25.537 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:25 smithi052 ceph-mon[36682]: pgmap v44: 148 pgs: 148 active+clean; 140 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 17 MiB/s wr, 280 op/s 2024-09-13T13:02:25.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:25.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:25.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:25 smithi167 ceph-mon[33178]: pgmap v44: 148 pgs: 148 active+clean; 140 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 17 MiB/s wr, 280 op/s 2024-09-13T13:02:25.639 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:25.683 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:25.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:25.785 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:25.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:25.828 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:25.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:25.864 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:25.929 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:25.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:25.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:25.971 INFO:teuthology.orchestra.run.smithi052.stderr:no valid command found; 10 closest matches: 2024-09-13T13:02:25.971 INFO:teuthology.orchestra.run.smithi052.stderr:0 2024-09-13T13:02:25.971 INFO:teuthology.orchestra.run.smithi052.stderr:1 2024-09-13T13:02:25.971 INFO:teuthology.orchestra.run.smithi052.stderr:2 2024-09-13T13:02:25.971 INFO:teuthology.orchestra.run.smithi052.stderr:abort 2024-09-13T13:02:25.971 INFO:teuthology.orchestra.run.smithi052.stderr:assert 2024-09-13T13:02:25.971 INFO:teuthology.orchestra.run.smithi052.stderr:bench [] [] [] [] 2024-09-13T13:02:25.971 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs debug_inject_read_zeros 2024-09-13T13:02:25.971 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs files list 2024-09-13T13:02:25.971 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs stats 2024-09-13T13:02:25.971 INFO:teuthology.orchestra.run.smithi052.stderr:bluestore allocator dump block 2024-09-13T13:02:25.971 INFO:teuthology.orchestra.run.smithi052.stderr:Error EINVAL: invalid command 2024-09-13T13:02:26.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:26.085 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:26.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:26.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:26.150 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 4 16 10208 10192 158.662 131.688 0.00317492 0.0062655 2024-09-13T13:02:26.187 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:26.227 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:26.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:26.329 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:26.375 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:26.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:26.392 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:02:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:02:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:02:26.398 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:02:26.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:26.505 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:26.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:26.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:26.639 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:26.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:26.667 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:26 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3458823043' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:02:26.667 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:26 smithi052 ceph-mon[36682]: from='client.24787 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:02:26.668 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:26 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3458823043' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:02:26.668 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:26 smithi052 ceph-mon[29471]: from='client.24787 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:02:26.740 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:26.749 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:26 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3458823043' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:02:26.749 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:26 smithi167 ceph-mon[33178]: from='client.24787 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:02:26.772 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:26.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:26.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:26.874 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:26.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:26.922 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:26.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:27.023 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:27.063 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:27.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:27.150 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 5 16 12422 12406 154.614 138.375 0.00630584 0.00645787 2024-09-13T13:02:27.164 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:27.204 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:27.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:27.306 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:27.335 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:27.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:27.437 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:27.466 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:27.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:27.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:27.558 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:02:27.558 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:02:27.558 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:02:27.558 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:02:27.558 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:02:27.559 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:02:27.559 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:02:27.559 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:02:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:02:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:02:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:02:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:02:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:02:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:02:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:02:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:02:27.568 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:27.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:27.602 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:27.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:27.703 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:27.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:27.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:27.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:27.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[36682]: pgmap v45: 148 pgs: 148 active+clean; 140 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 17 MiB/s wr, 279 op/s 2024-09-13T13:02:27.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:02:27.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[36682]: from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:02:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[36682]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T13:02:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/157008679' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[36682]: 5.16 scrub starts 2024-09-13T13:02:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[36682]: 5.16 scrub ok 2024-09-13T13:02:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[29471]: pgmap v45: 148 pgs: 148 active+clean; 140 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 17 MiB/s wr, 279 op/s 2024-09-13T13:02:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:02:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[29471]: from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:02:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[29471]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T13:02:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/157008679' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[29471]: 5.16 scrub starts 2024-09-13T13:02:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[29471]: 5.16 scrub ok 2024-09-13T13:02:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:27 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:27.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:27 smithi167 ceph-mon[33178]: pgmap v45: 148 pgs: 148 active+clean; 140 MiB data, 388 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 17 MiB/s wr, 279 op/s 2024-09-13T13:02:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:27 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:02:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:27 smithi167 ceph-mon[33178]: from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:02:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:27 smithi167 ceph-mon[33178]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T13:02:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:27 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/157008679' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:27 smithi167 ceph-mon[33178]: 5.16 scrub starts 2024-09-13T13:02:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:27 smithi167 ceph-mon[33178]: 5.16 scrub ok 2024-09-13T13:02:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:27 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:27.838 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:27.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:27.869 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:27.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:27.970 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:28.009 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:28.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:28.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:28.060 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:02:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:02:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:02:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:02:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-13T13:02:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T12:58:32.655077+0000", 2024-09-13T13:02:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 235.401711971, 2024-09-13T13:02:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.35374267199999998, 2024-09-13T13:02:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.14158", 2024-09-13T13:02:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/1009267179", 2024-09-13T13:02:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 1 2024-09-13T13:02:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:32.655077+0000", 2024-09-13T13:02:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:32.655077+0000", 2024-09-13T13:02:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:32.655079+0000", 2024-09-13T13:02:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8500000000000001e-06 2024-09-13T13:02:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:32.655093+0000", 2024-09-13T13:02:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4413000000000001e-05 2024-09-13T13:02:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:32.655112+0000", 2024-09-13T13:02:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8680000000000001e-05 2024-09-13T13:02:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:32.655120+0000", 2024-09-13T13:02:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2190000000000004e-06 2024-09-13T13:02:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:32.655190+0000", 2024-09-13T13:02:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0104000000000005e-05 2024-09-13T13:02:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:02:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:32.655198+0000", 2024-09-13T13:02:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.191e-06 2024-09-13T13:02:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:33.007779+0000", 2024-09-13T13:02:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.35258104099999998 2024-09-13T13:02:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:33.007910+0000", 2024-09-13T13:02:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000131113 2024-09-13T13:02:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:02:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:33.008033+0000", 2024-09-13T13:02:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012258599999999999 2024-09-13T13:02:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:33.008374+0000", 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00034065700000000001 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:33.008586+0000", 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021231399999999999 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:33.008764+0000", 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017794800000000001 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:33.008796+0000", 2024-09-13T13:02:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2230999999999997e-05 2024-09-13T13:02:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T12:58:33.008820+0000", 2024-09-13T13:02:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3325000000000001e-05 2024-09-13T13:02:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.14493.0:43 4.2 4:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T13:02:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:01:04.184427+0000", 2024-09-13T13:02:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 83.872362000999999, 2024-09-13T13:02:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014360346, 2024-09-13T13:02:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:02:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.14493", 2024-09-13T13:02:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:02:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 43 2024-09-13T13:02:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.184427+0000", 2024-09-13T13:02:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.184427+0000", 2024-09-13T13:02:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.184431+0000", 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7900000000000001e-06 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.184442+0000", 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1354e-05 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.184455+0000", 2024-09-13T13:02:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2897999999999999e-05 2024-09-13T13:02:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.184485+0000", 2024-09-13T13:02:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.968e-05 2024-09-13T13:02:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.189585+0000", 2024-09-13T13:02:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051008329999999999 2024-09-13T13:02:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:02:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.189593+0000", 2024-09-13T13:02:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0560000000000005e-06 2024-09-13T13:02:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.198660+0000", 2024-09-13T13:02:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0090664600000000001 2024-09-13T13:02:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.198787+0000", 2024-09-13T13:02:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012727499999999999 2024-09-13T13:02:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.14535.0:43 4.2 4:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:01:04.185953+0000", 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 83.870835428000007, 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.013601993999999999, 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.14535", 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/1708534747", 2024-09-13T13:02:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 43 2024-09-13T13:02:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.185953+0000", 2024-09-13T13:02:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.185953+0000", 2024-09-13T13:02:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.185956+0000", 2024-09-13T13:02:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2029999999999999e-06 2024-09-13T13:02:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.185964+0000", 2024-09-13T13:02:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5129999999999996e-06 2024-09-13T13:02:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.185983+0000", 2024-09-13T13:02:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8603000000000001e-05 2024-09-13T13:02:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.185992+0000", 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1540000000000008e-06 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.189616+0000", 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036242750000000002 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.189621+0000", 2024-09-13T13:02:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1950000000000002e-06 2024-09-13T13:02:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.199458+0000", 2024-09-13T13:02:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0098362109999999992 2024-09-13T13:02:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:01:04.199555+0000", 2024-09-13T13:02:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7839999999999998e-05 2024-09-13T13:02:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:11488 7.b 7:d89c6e84:::benchmark_data_smithi052_71865_object11487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:26.703332+0000", 2024-09-13T13:02:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 1.3534564929999999, 2024-09-13T13:02:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0088198489999999994, 2024-09-13T13:02:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 11488 2024-09-13T13:02:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.703332+0000", 2024-09-13T13:02:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.703332+0000", 2024-09-13T13:02:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.703335+0000", 2024-09-13T13:02:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.181e-06 2024-09-13T13:02:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.703397+0000", 2024-09-13T13:02:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2262999999999997e-05 2024-09-13T13:02:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.703415+0000", 2024-09-13T13:02:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8284999999999999e-05 2024-09-13T13:02:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.703457+0000", 2024-09-13T13:02:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1593e-05 2024-09-13T13:02:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.704153+0000", 2024-09-13T13:02:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00069678700000000004 2024-09-13T13:02:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.707971+0000", 2024-09-13T13:02:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003817619 2024-09-13T13:02:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:02:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.708109+0000", 2024-09-13T13:02:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000137916 2024-09-13T13:02:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.712004+0000", 2024-09-13T13:02:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003894645 2024-09-13T13:02:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.712017+0000", 2024-09-13T13:02:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2862e-05 2024-09-13T13:02:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.712051+0000", 2024-09-13T13:02:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4941000000000003e-05 2024-09-13T13:02:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.712109+0000", 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7867e-05 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.712152+0000", 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2889999999999998e-05 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:11917 7.b 7:d74f7c7b:::benchmark_data_smithi052_71865_object11916:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:26.886441+0000", 2024-09-13T13:02:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 1.1703477760000001, 2024-09-13T13:02:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.059697234000000002, 2024-09-13T13:02:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 11917 2024-09-13T13:02:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886441+0000", 2024-09-13T13:02:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886441+0000", 2024-09-13T13:02:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886444+0000", 2024-09-13T13:02:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1130000000000001e-06 2024-09-13T13:02:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886476+0000", 2024-09-13T13:02:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2135999999999999e-05 2024-09-13T13:02:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886497+0000", 2024-09-13T13:02:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1053999999999999e-05 2024-09-13T13:02:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886507+0000", 2024-09-13T13:02:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0098000000000001e-05 2024-09-13T13:02:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886584+0000", 2024-09-13T13:02:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6298999999999996e-05 2024-09-13T13:02:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886717+0000", 2024-09-13T13:02:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000133597 2024-09-13T13:02:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:02:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886898+0000", 2024-09-13T13:02:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018069900000000001 2024-09-13T13:02:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.888957+0000", 2024-09-13T13:02:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020587259999999999 2024-09-13T13:02:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.889441+0000", 2024-09-13T13:02:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00048412599999999998 2024-09-13T13:02:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946024+0000", 2024-09-13T13:02:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.056582722000000002 2024-09-13T13:02:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946092+0000", 2024-09-13T13:02:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8387999999999996e-05 2024-09-13T13:02:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946138+0000", 2024-09-13T13:02:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6276000000000002e-05 2024-09-13T13:02:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:11919 7.b 7:d93aaddb:::benchmark_data_smithi052_71865_object11918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:26.886533+0000", 2024-09-13T13:02:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 1.170255847, 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.059706377999999997, 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 11919 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886533+0000", 2024-09-13T13:02:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886533+0000", 2024-09-13T13:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886534+0000", 2024-09-13T13:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0400000000000005e-07 2024-09-13T13:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886559+0000", 2024-09-13T13:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4607e-05 2024-09-13T13:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886567+0000", 2024-09-13T13:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1159999999999999e-06 2024-09-13T13:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886573+0000", 2024-09-13T13:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4559999999999997e-06 2024-09-13T13:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.887607+0000", 2024-09-13T13:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010338190000000001 2024-09-13T13:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.887721+0000", 2024-09-13T13:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113917 2024-09-13T13:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:02:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.887861+0000", 2024-09-13T13:02:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013968699999999999 2024-09-13T13:02:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.888962+0000", 2024-09-13T13:02:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001101517 2024-09-13T13:02:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.889481+0000", 2024-09-13T13:02:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00051924600000000001 2024-09-13T13:02:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946172+0000", 2024-09-13T13:02:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.056690469 2024-09-13T13:02:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946210+0000", 2024-09-13T13:02:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8656999999999999e-05 2024-09-13T13:02:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946239+0000", 2024-09-13T13:02:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8983e-05 2024-09-13T13:02:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:11946 7.b 7:de51ce50:::benchmark_data_smithi052_71865_object11945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:26.908326+0000", 2024-09-13T13:02:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 1.1484631620000001, 2024-09-13T13:02:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.038006949999999998, 2024-09-13T13:02:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 11946 2024-09-13T13:02:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.908326+0000", 2024-09-13T13:02:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.908326+0000", 2024-09-13T13:02:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.908328+0000", 2024-09-13T13:02:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7080000000000002e-06 2024-09-13T13:02:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.908356+0000", 2024-09-13T13:02:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7257e-05 2024-09-13T13:02:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.908371+0000", 2024-09-13T13:02:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5735000000000001e-05 2024-09-13T13:02:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.908379+0000", 2024-09-13T13:02:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0199999999999994e-06 2024-09-13T13:02:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.908442+0000", 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2995000000000002e-05 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.908571+0000", 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012829700000000001 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.908703+0000", 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013201900000000001 2024-09-13T13:02:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.909768+0000", 2024-09-13T13:02:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010652839999999999 2024-09-13T13:02:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.910330+0000", 2024-09-13T13:02:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00056162300000000003 2024-09-13T13:02:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946268+0000", 2024-09-13T13:02:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.035937913000000002 2024-09-13T13:02:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946305+0000", 2024-09-13T13:02:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7611000000000003e-05 2024-09-13T13:02:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946333+0000", 2024-09-13T13:02:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7487999999999998e-05 2024-09-13T13:02:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:11954 7.b 7:d52ff045:::benchmark_data_smithi052_71865_object11953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:26.915104+0000", 2024-09-13T13:02:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 1.1416847619999999, 2024-09-13T13:02:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.031329382000000003, 2024-09-13T13:02:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 11954 2024-09-13T13:02:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.915104+0000", 2024-09-13T13:02:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.915104+0000", 2024-09-13T13:02:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.915107+0000", 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.508e-06 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.915133+0000", 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6412000000000001e-05 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.915151+0000", 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8456000000000001e-05 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.915160+0000", 2024-09-13T13:02:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3049999999999993e-06 2024-09-13T13:02:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.915229+0000", 2024-09-13T13:02:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9164000000000004e-05 2024-09-13T13:02:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.915374+0000", 2024-09-13T13:02:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014483200000000001 2024-09-13T13:02:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:02:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.915525+0000", 2024-09-13T13:02:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000150836 2024-09-13T13:02:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.916633+0000", 2024-09-13T13:02:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001108064 2024-09-13T13:02:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.916834+0000", 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020165 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946362+0000", 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.029527332 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946405+0000", 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3293e-05 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946433+0000", 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.853e-05 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:12036 7.b 7:dba7ae60:::benchmark_data_smithi052_71865_object12035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:26.995709+0000", 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 1.0610795180000001, 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011484203, 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 12036 2024-09-13T13:02:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.995709+0000", 2024-09-13T13:02:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.995709+0000", 2024-09-13T13:02:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.995711+0000", 2024-09-13T13:02:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9800000000000001e-06 2024-09-13T13:02:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.995735+0000", 2024-09-13T13:02:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3896e-05 2024-09-13T13:02:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.995751+0000", 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5917999999999999e-05 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.995760+0000", 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7299999999999994e-06 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.995813+0000", 2024-09-13T13:02:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3359999999999997e-05 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.996048+0000", 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000234992 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.996168+0000", 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120093 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.997112+0000", 2024-09-13T13:02:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00094416300000000003 2024-09-13T13:02:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.997404+0000", 2024-09-13T13:02:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00029148200000000003 2024-09-13T13:02:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.007050+0000", 2024-09-13T13:02:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0096462049999999997 2024-09-13T13:02:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.007149+0000", 2024-09-13T13:02:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9024999999999995e-05 2024-09-13T13:02:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.007194+0000", 2024-09-13T13:02:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4359e-05 2024-09-13T13:02:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:12126 7.b 7:d5f64625:::benchmark_data_smithi052_71865_object12125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:27.034061+0000", 2024-09-13T13:02:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 1.0227277269999999, 2024-09-13T13:02:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011316746000000001, 2024-09-13T13:02:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 12126 2024-09-13T13:02:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034061+0000", 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034061+0000", 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034062+0000", 2024-09-13T13:02:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8599999999999997e-07 2024-09-13T13:02:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034088+0000", 2024-09-13T13:02:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5749999999999999e-05 2024-09-13T13:02:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034095+0000", 2024-09-13T13:02:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8299999999999996e-06 2024-09-13T13:02:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034101+0000", 2024-09-13T13:02:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4519999999999996e-06 2024-09-13T13:02:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.035519+0000", 2024-09-13T13:02:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00141829 2024-09-13T13:02:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.035632+0000", 2024-09-13T13:02:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113004 2024-09-13T13:02:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:02:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.035771+0000", 2024-09-13T13:02:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000138813 2024-09-13T13:02:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.039386+0000", 2024-09-13T13:02:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036152100000000002 2024-09-13T13:02:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.039434+0000", 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7639999999999998e-05 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.041878+0000", 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00244458 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.043792+0000", 2024-09-13T13:02:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001913447 2024-09-13T13:02:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.045378+0000", 2024-09-13T13:02:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0015859439999999999 2024-09-13T13:02:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:12128 7.b 7:da3bd7d1:::benchmark_data_smithi052_71865_object12127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:27.034110+0000", 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 1.022679093, 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011385149000000001, 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 12128 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034110+0000", 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034110+0000", 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034111+0000", 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9599999999999998e-07 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034134+0000", 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3370000000000002e-05 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034141+0000", 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9789999999999996e-06 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034147+0000", 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6590000000000004e-06 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.037647+0000", 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0035007609999999998 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.037772+0000", 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012432599999999999 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.037913+0000", 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000141586 2024-09-13T13:02:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.039473+0000", 2024-09-13T13:02:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0015596200000000001 2024-09-13T13:02:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.039514+0000", 2024-09-13T13:02:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1125999999999999e-05 2024-09-13T13:02:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.045395+0000", 2024-09-13T13:02:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058810360000000001 2024-09-13T13:02:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.045448+0000", 2024-09-13T13:02:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3149000000000002e-05 2024-09-13T13:02:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.045495+0000", 2024-09-13T13:02:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6740999999999999e-05 2024-09-13T13:02:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:12129 7.b 7:db2d4fd6:::benchmark_data_smithi052_71865_object12128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:27.034154+0000", 2024-09-13T13:02:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 1.022634743, 2024-09-13T13:02:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.01147011, 2024-09-13T13:02:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 12129 2024-09-13T13:02:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034154+0000", 2024-09-13T13:02:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034154+0000", 2024-09-13T13:02:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034155+0000", 2024-09-13T13:02:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5199999999999996e-07 2024-09-13T13:02:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034206+0000", 2024-09-13T13:02:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1570999999999998e-05 2024-09-13T13:02:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034215+0000", 2024-09-13T13:02:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0259999999999996e-06 2024-09-13T13:02:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.034222+0000", 2024-09-13T13:02:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4640000000000003e-06 2024-09-13T13:02:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.039547+0000", 2024-09-13T13:02:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053254890000000001 2024-09-13T13:02:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.039668+0000", 2024-09-13T13:02:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120396 2024-09-13T13:02:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:02:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.039788+0000", 2024-09-13T13:02:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000119981 2024-09-13T13:02:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.041845+0000", 2024-09-13T13:02:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020572030000000001 2024-09-13T13:02:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.045503+0000", 2024-09-13T13:02:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036581859999999999 2024-09-13T13:02:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.045546+0000", 2024-09-13T13:02:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2463999999999999e-05 2024-09-13T13:02:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.045591+0000", 2024-09-13T13:02:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5538999999999999e-05 2024-09-13T13:02:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.045624+0000", 2024-09-13T13:02:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3139000000000002e-05 2024-09-13T13:02:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:13306 7.b 7:dd5290b8:::benchmark_data_smithi052_71865_object13305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:27.463126+0000", 2024-09-13T13:02:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.59366248399999999, 2024-09-13T13:02:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.012852916000000001, 2024-09-13T13:02:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 13306 2024-09-13T13:02:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.463126+0000", 2024-09-13T13:02:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.463126+0000", 2024-09-13T13:02:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.463129+0000", 2024-09-13T13:02:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.554e-06 2024-09-13T13:02:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.463165+0000", 2024-09-13T13:02:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5880000000000002e-05 2024-09-13T13:02:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.463177+0000", 2024-09-13T13:02:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.202e-05 2024-09-13T13:02:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.463186+0000", 2024-09-13T13:02:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9830000000000002e-06 2024-09-13T13:02:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.464422+0000", 2024-09-13T13:02:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012361869999999999 2024-09-13T13:02:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.464580+0000", 2024-09-13T13:02:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015828899999999999 2024-09-13T13:02:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:02:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.464721+0000", 2024-09-13T13:02:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014039200000000001 2024-09-13T13:02:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.468912+0000", 2024-09-13T13:02:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041912 2024-09-13T13:02:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.468958+0000", 2024-09-13T13:02:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6375000000000002e-05 2024-09-13T13:02:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.475562+0000", 2024-09-13T13:02:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0066032929999999997 2024-09-13T13:02:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.475924+0000", 2024-09-13T13:02:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000362006 2024-09-13T13:02:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.475979+0000", 2024-09-13T13:02:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5736999999999998e-05 2024-09-13T13:02:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:13313 7.0 e89/84 7:0e8a44fc:::benchmark_data_smithi052_71865_object13312:head v 89'799, mlcod=89'795)", 2024-09-13T13:02:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:27.463788+0000", 2024-09-13T13:02:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.59300134100000002, 2024-09-13T13:02:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.013388639000000001, 2024-09-13T13:02:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.463788+0000", 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.463788+0000", 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.463789+0000", 2024-09-13T13:02:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1689999999999999e-06 2024-09-13T13:02:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.152 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.463890+0000", 2024-09-13T13:02:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101544 2024-09-13T13:02:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.463899+0000", 2024-09-13T13:02:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7430000000000008e-06 2024-09-13T13:02:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.463905+0000", 2024-09-13T13:02:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1979999999999998e-06 2024-09-13T13:02:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.475765+0000", 2024-09-13T13:02:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011859590999999999 2024-09-13T13:02:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.475782+0000", 2024-09-13T13:02:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7716000000000001e-05 2024-09-13T13:02:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.477135+0000", 2024-09-13T13:02:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001352776 2024-09-13T13:02:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.477176+0000", 2024-09-13T13:02:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0902000000000002e-05 2024-09-13T13:02:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:13305 7.2 e89/84 7:466dd32a:::benchmark_data_smithi052_71865_object13304:head v 89'825, mlcod=89'823)", 2024-09-13T13:02:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:27.465542+0000", 2024-09-13T13:02:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.59124642500000002, 2024-09-13T13:02:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.01011248, 2024-09-13T13:02:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465542+0000", 2024-09-13T13:02:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465542+0000", 2024-09-13T13:02:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465544+0000", 2024-09-13T13:02:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.421e-06 2024-09-13T13:02:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465575+0000", 2024-09-13T13:02:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1278000000000001e-05 2024-09-13T13:02:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465586+0000", 2024-09-13T13:02:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0562e-05 2024-09-13T13:02:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465593+0000", 2024-09-13T13:02:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2840000000000001e-06 2024-09-13T13:02:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465664+0000", 2024-09-13T13:02:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0610000000000003e-05 2024-09-13T13:02:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465690+0000", 2024-09-13T13:02:28.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6699e-05 2024-09-13T13:02:28.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.475400+0000", 2024-09-13T13:02:28.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0097098370000000007 2024-09-13T13:02:28.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.475655+0000", 2024-09-13T13:02:28.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000254789 2024-09-13T13:02:28.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.161 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.161 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:13309 7.2 e89/84 7:4ba08063:::benchmark_data_smithi052_71865_object13308:head v 89'826, mlcod=89'823)", 2024-09-13T13:02:28.161 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:27.465614+0000", 2024-09-13T13:02:28.161 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.59117498899999998, 2024-09-13T13:02:28.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010074783, 2024-09-13T13:02:28.161 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.161 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.161 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.162 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465614+0000", 2024-09-13T13:02:28.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465614+0000", 2024-09-13T13:02:28.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465615+0000", 2024-09-13T13:02:28.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3399999999999999e-07 2024-09-13T13:02:28.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465639+0000", 2024-09-13T13:02:28.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4411999999999999e-05 2024-09-13T13:02:28.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465646+0000", 2024-09-13T13:02:28.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4749999999999996e-06 2024-09-13T13:02:28.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465652+0000", 2024-09-13T13:02:28.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8540000000000002e-06 2024-09-13T13:02:28.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.466233+0000", 2024-09-13T13:02:28.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00058069000000000003 2024-09-13T13:02:28.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.466250+0000", 2024-09-13T13:02:28.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7158000000000001e-05 2024-09-13T13:02:28.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.475660+0000", 2024-09-13T13:02:28.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0094102539999999998 2024-09-13T13:02:28.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.475689+0000", 2024-09-13T13:02:28.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8306e-05 2024-09-13T13:02:28.166 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.166 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.166 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.166 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:13312 7.d e89/84 7:b634f4be:::benchmark_data_smithi052_71865_object13311:head v 89'818, mlcod=89'816)", 2024-09-13T13:02:28.166 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:27.465672+0000", 2024-09-13T13:02:28.166 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.59111644100000005, 2024-09-13T13:02:28.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010325330000000001, 2024-09-13T13:02:28.166 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.166 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.166 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.167 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465672+0000", 2024-09-13T13:02:28.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465672+0000", 2024-09-13T13:02:28.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465673+0000", 2024-09-13T13:02:28.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8700000000000005e-07 2024-09-13T13:02:28.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465705+0000", 2024-09-13T13:02:28.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2233000000000001e-05 2024-09-13T13:02:28.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465715+0000", 2024-09-13T13:02:28.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3260000000000002e-06 2024-09-13T13:02:28.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.465720+0000", 2024-09-13T13:02:28.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8770000000000001e-06 2024-09-13T13:02:28.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.474619+0000", 2024-09-13T13:02:28.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0088988660000000001 2024-09-13T13:02:28.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.474652+0000", 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3590000000000002e-05 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.475955+0000", 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001302767 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.475998+0000", 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2883999999999998e-05 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.170 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.171 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:13718 7.b 7:d5816016:::benchmark_data_smithi052_71865_object13717:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:28.171 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:27.610423+0000", 2024-09-13T13:02:28.171 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.44636553499999998, 2024-09-13T13:02:28.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0089915389999999998, 2024-09-13T13:02:28.171 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.171 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.171 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.171 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.171 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:28.172 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:28.172 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 13718 2024-09-13T13:02:28.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.172 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.610423+0000", 2024-09-13T13:02:28.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.610423+0000", 2024-09-13T13:02:28.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.610425+0000", 2024-09-13T13:02:28.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.407e-06 2024-09-13T13:02:28.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.610448+0000", 2024-09-13T13:02:28.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2966999999999999e-05 2024-09-13T13:02:28.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.610460+0000", 2024-09-13T13:02:28.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2774e-05 2024-09-13T13:02:28.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.610467+0000", 2024-09-13T13:02:28.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0269999999999998e-06 2024-09-13T13:02:28.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.612505+0000", 2024-09-13T13:02:28.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020372960000000001 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.612623+0000", 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011867900000000001 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.612701+0000", 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7218000000000005e-05 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:28.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.615159+0000", 2024-09-13T13:02:28.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024580639999999998 2024-09-13T13:02:28.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.176 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.176 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.615205+0000", 2024-09-13T13:02:28.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6081000000000001e-05 2024-09-13T13:02:28.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.176 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.176 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.619248+0000", 2024-09-13T13:02:28.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0040433229999999997 2024-09-13T13:02:28.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.619366+0000", 2024-09-13T13:02:28.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117613 2024-09-13T13:02:28.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.619415+0000", 2024-09-13T13:02:28.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9089999999999999e-05 2024-09-13T13:02:28.177 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.177 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.177 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:13721 7.b 7:d0429836:::benchmark_data_smithi052_71865_object13720:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:27.610944+0000", 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.44584512599999998, 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0088794359999999992, 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 13721 2024-09-13T13:02:28.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.179 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.610944+0000", 2024-09-13T13:02:28.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.610944+0000", 2024-09-13T13:02:28.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.610945+0000", 2024-09-13T13:02:28.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5600000000000004e-07 2024-09-13T13:02:28.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.610964+0000", 2024-09-13T13:02:28.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8943000000000001e-05 2024-09-13T13:02:28.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.610973+0000", 2024-09-13T13:02:28.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0909999999999999e-06 2024-09-13T13:02:28.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.610980+0000", 2024-09-13T13:02:28.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5510000000000002e-06 2024-09-13T13:02:28.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.613694+0000", 2024-09-13T13:02:28.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00271325 2024-09-13T13:02:28.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.613759+0000", 2024-09-13T13:02:28.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5145000000000001e-05 2024-09-13T13:02:28.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:02:28.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.613835+0000", 2024-09-13T13:02:28.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6478999999999997e-05 2024-09-13T13:02:28.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:28.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.615219+0000", 2024-09-13T13:02:28.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001383954 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.615253+0000", 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3756e-05 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.619506+0000", 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0042531640000000002 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.619692+0000", 2024-09-13T13:02:28.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018568999999999999 2024-09-13T13:02:28.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.184 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.619823+0000", 2024-09-13T13:02:28.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000131457 2024-09-13T13:02:28.184 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.184 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.184 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.184 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:14422 7.9 e90/84 7:934c1e79:::benchmark_data_smithi052_71865_object14421:head v 90'940, mlcod=90'937)", 2024-09-13T13:02:28.184 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:27.869775+0000", 2024-09-13T13:02:28.185 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.18701357900000001, 2024-09-13T13:02:28.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0092942560000000007, 2024-09-13T13:02:28.185 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:28.185 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:28.185 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:28.185 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:28.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:28.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.869775+0000", 2024-09-13T13:02:28.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:28.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.869775+0000", 2024-09-13T13:02:28.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:28.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:28.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.869776+0000", 2024-09-13T13:02:28.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.37e-07 2024-09-13T13:02:28.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:28.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.869801+0000", 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5497999999999999e-05 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.869809+0000", 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4499999999999998e-06 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.869814+0000", 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3519999999999997e-06 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.870512+0000", 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00069782100000000005 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.870532+0000", 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9791000000000001e-05 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.878980+0000", 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0084479710000000003 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:28.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:28.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:28.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:27.879070+0000", 2024-09-13T13:02:28.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9735999999999996e-05 2024-09-13T13:02:28.189 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.189 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.189 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.189 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:28.189 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:28.189 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:02:28.189 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 6 16 15096 15080 156.692 167.125 0.00969474 0.00637043 2024-09-13T13:02:28.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:28.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:28.322 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:28.350 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:28.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:28.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:28.452 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:28.496 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:28.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:28.598 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:28.646 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:28.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:28.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:28.748 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:28.786 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:28.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:28.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:28.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:28.920 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:28.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:28.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:28 smithi052 ceph-mon[36682]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T13:02:28.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:28 smithi052 ceph-mon[36682]: pgmap v48: 148 pgs: 148 active+clean; 902 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 150 MiB/s wr, 2.40k op/s 2024-09-13T13:02:28.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:28 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:28.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:28 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:28.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:28 smithi052 ceph-mon[29471]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T13:02:28.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:28 smithi052 ceph-mon[29471]: pgmap v48: 148 pgs: 148 active+clean; 902 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 150 MiB/s wr, 2.40k op/s 2024-09-13T13:02:28.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:28 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:28.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:28 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:29.021 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:29.025 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:28 smithi167 ceph-mon[33178]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T13:02:29.026 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:28 smithi167 ceph-mon[33178]: pgmap v48: 148 pgs: 148 active+clean; 902 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 150 MiB/s wr, 2.40k op/s 2024-09-13T13:02:29.026 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:28 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:29.026 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:28 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:29.063 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:29.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:29.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:29.150 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 7 16 17613 17597 156.778 157.312 0.00449299 0.00637063 2024-09-13T13:02:29.167 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:29.205 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:29.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:29.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:29.307 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:29.346 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:29.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:29.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:29.448 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:29.480 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:29.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:29.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T13:02:29.488 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T13:02:29.488 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-13T13:02:29.501 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:02:29.582 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:29.631 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:29.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:29.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:29.732 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:29.763 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:29.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:29.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:29.865 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:29.895 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:29.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[36682]: 6.1 scrub starts 2024-09-13T13:02:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[36682]: 6.1 scrub ok 2024-09-13T13:02:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[36682]: 6.a scrub starts 2024-09-13T13:02:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[36682]: 6.a scrub ok 2024-09-13T13:02:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:02:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:02:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[36682]: 3.d scrub starts 2024-09-13T13:02:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[36682]: 3.d scrub ok 2024-09-13T13:02:29.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[36682]: osdmap e91: 8 total, 8 up, 8 in 2024-09-13T13:02:29.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[29471]: 6.1 scrub starts 2024-09-13T13:02:29.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[29471]: 6.1 scrub ok 2024-09-13T13:02:29.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[29471]: 6.a scrub starts 2024-09-13T13:02:29.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[29471]: 6.a scrub ok 2024-09-13T13:02:29.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:02:29.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:02:29.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[29471]: 3.d scrub starts 2024-09-13T13:02:29.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[29471]: 3.d scrub ok 2024-09-13T13:02:29.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:29 smithi052 ceph-mon[29471]: osdmap e91: 8 total, 8 up, 8 in 2024-09-13T13:02:29.997 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:30.044 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:30.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:29 smithi167 ceph-mon[33178]: 6.1 scrub starts 2024-09-13T13:02:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:29 smithi167 ceph-mon[33178]: 6.1 scrub ok 2024-09-13T13:02:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:29 smithi167 ceph-mon[33178]: 6.a scrub starts 2024-09-13T13:02:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:29 smithi167 ceph-mon[33178]: 6.a scrub ok 2024-09-13T13:02:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:29 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:02:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:29 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:02:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:29 smithi167 ceph-mon[33178]: 3.d scrub starts 2024-09-13T13:02:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:29 smithi167 ceph-mon[33178]: 3.d scrub ok 2024-09-13T13:02:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:29 smithi167 ceph-mon[33178]: osdmap e91: 8 total, 8 up, 8 in 2024-09-13T13:02:30.145 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:30.150 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 8 16 19795 19779 154.231 136.375 0.00674259 0.00647701 2024-09-13T13:02:30.175 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:30.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:30.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:30.306 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:30.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:30.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:30.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:30.437 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:30.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:30.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:30.569 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:30.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:30.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:30.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:30.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:30.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:30.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:30.802 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:30.832 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:30.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:30.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:30.934 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:30.957 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:30.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:30.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:30 smithi052 ceph-mon[36682]: 4.9 scrub starts 2024-09-13T13:02:30.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:30 smithi052 ceph-mon[36682]: 4.9 scrub ok 2024-09-13T13:02:30.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:30 smithi052 ceph-mon[36682]: 4.3 scrub starts 2024-09-13T13:02:30.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:30 smithi052 ceph-mon[36682]: 4.3 scrub ok 2024-09-13T13:02:30.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:30 smithi052 ceph-mon[36682]: pgmap v50: 148 pgs: 148 active+clean; 902 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-09-13T13:02:30.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:30 smithi052 ceph-mon[29471]: 4.9 scrub starts 2024-09-13T13:02:30.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:30 smithi052 ceph-mon[29471]: 4.9 scrub ok 2024-09-13T13:02:30.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:30 smithi052 ceph-mon[29471]: 4.3 scrub starts 2024-09-13T13:02:30.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:30 smithi052 ceph-mon[29471]: 4.3 scrub ok 2024-09-13T13:02:30.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:30 smithi052 ceph-mon[29471]: pgmap v50: 148 pgs: 148 active+clean; 902 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-09-13T13:02:31.058 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:30 smithi167 ceph-mon[33178]: 4.9 scrub starts 2024-09-13T13:02:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:30 smithi167 ceph-mon[33178]: 4.9 scrub ok 2024-09-13T13:02:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:30 smithi167 ceph-mon[33178]: 4.3 scrub starts 2024-09-13T13:02:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:30 smithi167 ceph-mon[33178]: 4.3 scrub ok 2024-09-13T13:02:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:30 smithi167 ceph-mon[33178]: pgmap v50: 148 pgs: 148 active+clean; 902 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-09-13T13:02:31.098 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:31.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:31.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:31.150 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 9 15 22136 22121 153.358 146.375 0.00928571 0.00651503 2024-09-13T13:02:31.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:31.229 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:31.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:31.331 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:31.365 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:31.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:31.467 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:31.508 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:31.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:31.610 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:31.640 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:31.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:31.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:31 smithi052 ceph-mon[36682]: 5.19 scrub starts 2024-09-13T13:02:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:31 smithi052 ceph-mon[29471]: 5.19 scrub starts 2024-09-13T13:02:31.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:31.765 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:31.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:31.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:31.895 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:31.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:31.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:31.997 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:32.026 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:32.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:32.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:31 smithi167 ceph-mon[33178]: 5.19 scrub starts 2024-09-13T13:02:32.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:31 smithi167 ceph-mon[33178]: 5.19 scrub ok 2024-09-13T13:02:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:31 smithi167 ceph-mon[33178]: 5.15 scrub starts 2024-09-13T13:02:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:31 smithi167 ceph-mon[33178]: 5.15 scrub ok 2024-09-13T13:02:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:31 smithi167 ceph-mon[33178]: 5.1e scrub starts 2024-09-13T13:02:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:31 smithi167 ceph-mon[33178]: 5.1e scrub ok 2024-09-13T13:02:32.127 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:32.151 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 10 16 24330 24314 151.729 137.062 0.0101707 0.0065825 2024-09-13T13:02:32.164 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:32.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:32.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:31 smithi052 ceph-mon[36682]: 5.19 scrub ok 2024-09-13T13:02:32.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:31 smithi052 ceph-mon[36682]: 5.15 scrub starts 2024-09-13T13:02:32.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:31 smithi052 ceph-mon[36682]: 5.15 scrub ok 2024-09-13T13:02:32.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:31 smithi052 ceph-mon[36682]: 5.1e scrub starts 2024-09-13T13:02:32.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:31 smithi052 ceph-mon[36682]: 5.1e scrub ok 2024-09-13T13:02:32.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:31 smithi052 ceph-mon[29471]: 5.19 scrub ok 2024-09-13T13:02:32.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:31 smithi052 ceph-mon[29471]: 5.15 scrub starts 2024-09-13T13:02:32.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:31 smithi052 ceph-mon[29471]: 5.15 scrub ok 2024-09-13T13:02:32.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:31 smithi052 ceph-mon[29471]: 5.1e scrub starts 2024-09-13T13:02:32.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:31 smithi052 ceph-mon[29471]: 5.1e scrub ok 2024-09-13T13:02:32.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:32.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:32.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:32.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:32.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:32.426 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:32.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:32.475 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:02:32.528 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:32.577 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:32.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:32.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:32.678 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:32.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:32.707 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:32.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:32.808 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:32.839 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:32.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:32.941 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:32.981 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:32.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[36682]: pgmap v51: 148 pgs: 148 active+clean; 902 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-09-13T13:02:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-13T13:02:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]} : dispatch 2024-09-13T13:02:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-13T13:02:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-13T13:02:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]} : dispatch 2024-09-13T13:02:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-13T13:02:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:02:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-13T13:02:32.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-09-13T13:02:32.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-13T13:02:32.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[36682]: osdmap e92: 8 total, 8 up, 8 in 2024-09-13T13:02:32.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[29471]: pgmap v51: 148 pgs: 148 active+clean; 902 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-09-13T13:02:32.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-13T13:02:32.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]} : dispatch 2024-09-13T13:02:32.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-13T13:02:32.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-13T13:02:32.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]} : dispatch 2024-09-13T13:02:32.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-13T13:02:32.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:32.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:02:32.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-13T13:02:32.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-09-13T13:02:32.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-13T13:02:32.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:32 smithi052 ceph-mon[29471]: osdmap e92: 8 total, 8 up, 8 in 2024-09-13T13:02:33.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:32 smithi167 ceph-mon[33178]: pgmap v51: 148 pgs: 148 active+clean; 902 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-09-13T13:02:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-13T13:02:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]} : dispatch 2024-09-13T13:02:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-13T13:02:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:32 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-13T13:02:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:32 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]} : dispatch 2024-09-13T13:02:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:32 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-13T13:02:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:32 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:02:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:32 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-13T13:02:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:32 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-09-13T13:02:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:32 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-13T13:02:33.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:32 smithi167 ceph-mon[33178]: osdmap e92: 8 total, 8 up, 8 in 2024-09-13T13:02:33.083 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:33.124 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:33.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:33.151 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 11 16 26350 26334 149.414 126.25 0.00486552 0.00668649 2024-09-13T13:02:33.226 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:33.256 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:33.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:33.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:33.358 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:33.398 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:33.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:33.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:33.499 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:33.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:33.544 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:33.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:33.645 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:33.674 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:33.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:33.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:33.776 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:33.805 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:33.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:33.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:33.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:33 smithi167 ceph-mon[33178]: 5.e scrub starts 2024-09-13T13:02:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:33 smithi167 ceph-mon[33178]: 5.e scrub ok 2024-09-13T13:02:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:33 smithi167 ceph-mon[33178]: 3.1b scrub starts 2024-09-13T13:02:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:33 smithi167 ceph-mon[33178]: 3.1b scrub ok 2024-09-13T13:02:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:33 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:02:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:33 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:02:33.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:02:33 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:02:33.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:33.933 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:33.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:33.978 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T13:02:33.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:33 smithi052 ceph-mon[36682]: 5.e scrub starts 2024-09-13T13:02:33.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:33 smithi052 ceph-mon[36682]: 5.e scrub ok 2024-09-13T13:02:33.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:33 smithi052 ceph-mon[36682]: 3.1b scrub starts 2024-09-13T13:02:33.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:33 smithi052 ceph-mon[36682]: 3.1b scrub ok 2024-09-13T13:02:33.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:33 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:02:33.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:33 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:02:33.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:33 smithi052 ceph-mon[29471]: 5.e scrub starts 2024-09-13T13:02:33.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:33 smithi052 ceph-mon[29471]: 5.e scrub ok 2024-09-13T13:02:33.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:33 smithi052 ceph-mon[29471]: 3.1b scrub starts 2024-09-13T13:02:33.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:33 smithi052 ceph-mon[29471]: 3.1b scrub ok 2024-09-13T13:02:33.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:33 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:02:33.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:33 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:02:33.994 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:34.005 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:34.035 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:34.061 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:34.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:34.153 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 12 16 28372 28356 147.482 126.375 0.00413959 0.00677196 2024-09-13T13:02:34.163 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:34.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:34.219 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:34.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:34.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:34.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:34.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:34.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:34.446 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:34.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:34.468 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:34.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:34.569 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:34.619 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:34.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:34.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:34.721 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:34.751 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:34.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:34.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:34.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[36682]: 5.18 scrub starts 2024-09-13T13:02:34.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[36682]: 5.18 scrub ok 2024-09-13T13:02:34.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:34.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[36682]: osdmap e93: 8 total, 8 up, 8 in 2024-09-13T13:02:34.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[36682]: pgmap v54: 148 pgs: 3 peering, 145 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 118 MiB/s wr, 1.88k op/s 2024-09-13T13:02:34.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[36682]: 6.5 scrub starts 2024-09-13T13:02:34.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[36682]: 6.5 scrub ok 2024-09-13T13:02:34.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[29471]: 5.18 scrub starts 2024-09-13T13:02:34.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[29471]: 5.18 scrub ok 2024-09-13T13:02:34.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:34.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[29471]: osdmap e93: 8 total, 8 up, 8 in 2024-09-13T13:02:34.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[29471]: pgmap v54: 148 pgs: 3 peering, 145 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 118 MiB/s wr, 1.88k op/s 2024-09-13T13:02:34.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[29471]: 6.5 scrub starts 2024-09-13T13:02:34.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:34 smithi052 ceph-mon[29471]: 6.5 scrub ok 2024-09-13T13:02:34.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:34.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:34.984 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:35.011 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:35.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:35.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:35.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:34 smithi167 ceph-mon[33178]: 5.18 scrub starts 2024-09-13T13:02:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:34 smithi167 ceph-mon[33178]: 5.18 scrub ok 2024-09-13T13:02:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:34 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:34 smithi167 ceph-mon[33178]: osdmap e93: 8 total, 8 up, 8 in 2024-09-13T13:02:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:34 smithi167 ceph-mon[33178]: pgmap v54: 148 pgs: 3 peering, 145 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 118 MiB/s wr, 1.88k op/s 2024-09-13T13:02:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:34 smithi167 ceph-mon[33178]: 6.5 scrub starts 2024-09-13T13:02:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:34 smithi167 ceph-mon[33178]: 6.5 scrub ok 2024-09-13T13:02:35.113 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:35.152 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 13 16 30153 30137 144.703 111.312 0.00853757 0.00690452 2024-09-13T13:02:35.158 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:35.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:35.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:35.260 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:35.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:35.288 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:35.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:35.347 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:35.389 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:35.415 INFO:teuthology.orchestra.run.smithi052.stderr:no valid command found; 10 closest matches: 2024-09-13T13:02:35.415 INFO:teuthology.orchestra.run.smithi052.stderr:0 2024-09-13T13:02:35.415 INFO:teuthology.orchestra.run.smithi052.stderr:1 2024-09-13T13:02:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:2 2024-09-13T13:02:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:abort 2024-09-13T13:02:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:assert 2024-09-13T13:02:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:bench [] [] [] [] 2024-09-13T13:02:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs debug_inject_read_zeros 2024-09-13T13:02:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs files list 2024-09-13T13:02:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs stats 2024-09-13T13:02:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:bluestore allocator dump block 2024-09-13T13:02:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:Error EINVAL: invalid command 2024-09-13T13:02:35.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:35.465 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:35.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:35.566 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:35.594 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:35.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:35.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:35.696 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:35.726 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:35.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:02:35.828 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:35.858 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:35.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:35.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:35.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:35.984 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:35.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:36.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:35 smithi167 ceph-mon[33178]: 6.e scrub starts 2024-09-13T13:02:36.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:35 smithi167 ceph-mon[33178]: 6.e scrub ok 2024-09-13T13:02:36.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:35 smithi167 ceph-mon[33178]: 4.1e scrub starts 2024-09-13T13:02:36.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:35 smithi167 ceph-mon[33178]: 4.1e scrub ok 2024-09-13T13:02:36.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:35 smithi167 ceph-mon[33178]: 5.a scrub starts 2024-09-13T13:02:36.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:35 smithi167 ceph-mon[33178]: 5.a scrub ok 2024-09-13T13:02:36.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:35 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1039917715' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:02:36.085 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:36.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:36.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:36.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:36.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[36682]: 6.e scrub starts 2024-09-13T13:02:36.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[36682]: 6.e scrub ok 2024-09-13T13:02:36.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[36682]: 4.1e scrub starts 2024-09-13T13:02:36.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[36682]: 4.1e scrub ok 2024-09-13T13:02:36.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[36682]: 5.a scrub starts 2024-09-13T13:02:36.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[36682]: 5.a scrub ok 2024-09-13T13:02:36.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1039917715' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:02:36.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[29471]: 6.e scrub starts 2024-09-13T13:02:36.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[29471]: 6.e scrub ok 2024-09-13T13:02:36.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[29471]: 4.1e scrub starts 2024-09-13T13:02:36.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[29471]: 4.1e scrub ok 2024-09-13T13:02:36.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[29471]: 5.a scrub starts 2024-09-13T13:02:36.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[29471]: 5.a scrub ok 2024-09-13T13:02:36.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:35 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1039917715' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:02:36.152 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 14 16 31960 31944 142.435 112.938 0.00472718 0.00701498 2024-09-13T13:02:36.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:36.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:36.245 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:36.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:36.346 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:36.386 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:36.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:36.487 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:36.488 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:02:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:02:36.527 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:36.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:36.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:36.629 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:36.695 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:36.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:36.797 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:36.827 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:36.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:36.929 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:36.970 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:36.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:36.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[36682]: 3.14 scrub starts 2024-09-13T13:02:36.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[36682]: 3.14 scrub ok 2024-09-13T13:02:36.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[36682]: 5.12 scrub starts 2024-09-13T13:02:36.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[36682]: 5.12 scrub ok 2024-09-13T13:02:36.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[36682]: 4.12 scrub starts 2024-09-13T13:02:36.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[36682]: 4.12 scrub ok 2024-09-13T13:02:36.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[36682]: 5.5 scrub starts 2024-09-13T13:02:36.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[36682]: 5.5 scrub ok 2024-09-13T13:02:36.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:02:36.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:02:36.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1039917715' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:02:36.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[36682]: osdmap e94: 8 total, 8 up, 8 in 2024-09-13T13:02:36.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[36682]: pgmap v56: 148 pgs: 3 peering, 145 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 118 MiB/s wr, 1.88k op/s 2024-09-13T13:02:36.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[29471]: 3.14 scrub starts 2024-09-13T13:02:36.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[29471]: 3.14 scrub ok 2024-09-13T13:02:36.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[29471]: 5.12 scrub starts 2024-09-13T13:02:36.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[29471]: 5.12 scrub ok 2024-09-13T13:02:36.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[29471]: 4.12 scrub starts 2024-09-13T13:02:36.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[29471]: 4.12 scrub ok 2024-09-13T13:02:36.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[29471]: 5.5 scrub starts 2024-09-13T13:02:36.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[29471]: 5.5 scrub ok 2024-09-13T13:02:36.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:02:36.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:02:36.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1039917715' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:02:36.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[29471]: osdmap e94: 8 total, 8 up, 8 in 2024-09-13T13:02:36.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:36 smithi052 ceph-mon[29471]: pgmap v56: 148 pgs: 3 peering, 145 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 118 MiB/s wr, 1.88k op/s 2024-09-13T13:02:37.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:36 smithi167 ceph-mon[33178]: 3.14 scrub starts 2024-09-13T13:02:37.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:36 smithi167 ceph-mon[33178]: 3.14 scrub ok 2024-09-13T13:02:37.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:36 smithi167 ceph-mon[33178]: 5.12 scrub starts 2024-09-13T13:02:37.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:36 smithi167 ceph-mon[33178]: 5.12 scrub ok 2024-09-13T13:02:37.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:36 smithi167 ceph-mon[33178]: 4.12 scrub starts 2024-09-13T13:02:37.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:36 smithi167 ceph-mon[33178]: 4.12 scrub ok 2024-09-13T13:02:37.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:36 smithi167 ceph-mon[33178]: 5.5 scrub starts 2024-09-13T13:02:37.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:36 smithi167 ceph-mon[33178]: 5.5 scrub ok 2024-09-13T13:02:37.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:36 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:02:37.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:36 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:02:37.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:36 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1039917715' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:02:37.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:36 smithi167 ceph-mon[33178]: osdmap e94: 8 total, 8 up, 8 in 2024-09-13T13:02:37.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:36 smithi167 ceph-mon[33178]: pgmap v56: 148 pgs: 3 peering, 145 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 118 MiB/s wr, 1.88k op/s 2024-09-13T13:02:37.073 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:37.099 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:37.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:37.152 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 15 15 33995 33980 141.423 127.25 0.00434501 0.00706531 2024-09-13T13:02:37.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:37.228 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:37.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:37.319 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:02:37.319 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:02:37.330 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:37.336 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:02:37.337 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:02:37.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:37.378 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:37.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:37.379 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:02:37.379 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:02:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T13:02:37.403 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T13:02:37.404 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-13T13:02:37.419 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:02:37.419 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:02:37.434 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:37.456 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:02:37.456 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:02:37.479 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:37.490 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:02:37.490 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:02:37.505 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:37.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:37.530 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:02:37.531 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:02:37.569 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:02:37.569 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:02:37.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:37.632 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:37.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:37.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:37.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:37.764 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:37.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:37.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:37.866 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:37.894 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:37.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:37.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:37.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:38.022 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:38.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:38.029 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[29471]: 5.1 scrub starts 2024-09-13T13:02:38.029 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[29471]: 5.1 scrub ok 2024-09-13T13:02:38.029 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[29471]: 6.4 scrub starts 2024-09-13T13:02:38.029 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[29471]: 6.4 scrub ok 2024-09-13T13:02:38.029 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[29471]: 6.10 scrub starts 2024-09-13T13:02:38.029 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[29471]: 6.10 scrub ok 2024-09-13T13:02:38.029 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[29471]: 5.b scrub starts 2024-09-13T13:02:38.029 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[29471]: 5.b scrub ok 2024-09-13T13:02:38.030 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[29471]: 4.14 scrub starts 2024-09-13T13:02:38.030 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[29471]: 4.14 scrub ok 2024-09-13T13:02:38.030 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1424295630' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:38.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[36682]: 5.1 scrub starts 2024-09-13T13:02:38.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[36682]: 5.1 scrub ok 2024-09-13T13:02:38.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[36682]: 6.4 scrub starts 2024-09-13T13:02:38.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[36682]: 6.4 scrub ok 2024-09-13T13:02:38.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[36682]: 6.10 scrub starts 2024-09-13T13:02:38.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[36682]: 6.10 scrub ok 2024-09-13T13:02:38.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[36682]: 5.b scrub starts 2024-09-13T13:02:38.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[36682]: 5.b scrub ok 2024-09-13T13:02:38.030 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[36682]: 4.14 scrub starts 2024-09-13T13:02:38.031 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[36682]: 4.14 scrub ok 2024-09-13T13:02:38.031 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:37 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1424295630' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:38.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:37 smithi167 ceph-mon[33178]: 5.1 scrub starts 2024-09-13T13:02:38.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:37 smithi167 ceph-mon[33178]: 5.1 scrub ok 2024-09-13T13:02:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:37 smithi167 ceph-mon[33178]: 6.4 scrub starts 2024-09-13T13:02:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:37 smithi167 ceph-mon[33178]: 6.4 scrub ok 2024-09-13T13:02:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:37 smithi167 ceph-mon[33178]: 6.10 scrub starts 2024-09-13T13:02:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:37 smithi167 ceph-mon[33178]: 6.10 scrub ok 2024-09-13T13:02:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:37 smithi167 ceph-mon[33178]: 5.b scrub starts 2024-09-13T13:02:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:37 smithi167 ceph-mon[33178]: 5.b scrub ok 2024-09-13T13:02:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:37 smithi167 ceph-mon[33178]: 4.14 scrub starts 2024-09-13T13:02:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:37 smithi167 ceph-mon[33178]: 4.14 scrub ok 2024-09-13T13:02:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:37 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1424295630' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:38.124 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:38.152 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 16 16 35569 35553 138.729 98.3125 0.00997847 0.00720191 2024-09-13T13:02:38.168 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:38.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:38.271 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:38.307 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:38.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:38.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:38.438 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:38.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:38.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:38.540 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:38.588 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:38.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:38.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:38.678 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:02:38.690 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:38.717 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:38.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:38.815 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:02:38.815 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:02:38.818 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:38.830 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:38.842 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:38.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:38.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:38.944 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:38.972 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:38.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:38.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:38.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:38 smithi052 ceph-mon[36682]: 3.1 scrub starts 2024-09-13T13:02:38.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:38 smithi052 ceph-mon[36682]: 3.1 scrub ok 2024-09-13T13:02:38.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:38 smithi052 ceph-mon[36682]: pgmap v57: 148 pgs: 148 active+clean; 2.0 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 191 MiB/s wr, 3.05k op/s; 36 B/s, 1 objects/s recovering 2024-09-13T13:02:38.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:38 smithi052 ceph-mon[29471]: 3.1 scrub starts 2024-09-13T13:02:38.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:38 smithi052 ceph-mon[29471]: 3.1 scrub ok 2024-09-13T13:02:38.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:38 smithi052 ceph-mon[29471]: pgmap v57: 148 pgs: 148 active+clean; 2.0 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 191 MiB/s wr, 3.05k op/s; 36 B/s, 1 objects/s recovering 2024-09-13T13:02:39.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:38 smithi167 ceph-mon[33178]: 3.1 scrub starts 2024-09-13T13:02:39.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:38 smithi167 ceph-mon[33178]: 3.1 scrub ok 2024-09-13T13:02:39.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:38 smithi167 ceph-mon[33178]: pgmap v57: 148 pgs: 148 active+clean; 2.0 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 191 MiB/s wr, 3.05k op/s; 36 B/s, 1 objects/s recovering 2024-09-13T13:02:39.074 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:39.119 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:39.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:39.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:39.153 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 17 15 37325 37310 137.023 109.812 0.0130654 0.00729264 2024-09-13T13:02:39.223 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:39.264 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:39.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:39.366 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:39.407 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:39.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:39.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:39.509 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:39.534 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T13:02:39.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:39.538 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:39.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:39.639 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:39.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:39.680 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:39.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:39.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:39.807 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:39.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:39.909 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:39.937 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:39.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:39.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:40.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:40.067 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:40.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:40.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:40.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:39 smithi167 ceph-mon[33178]: 3.1f scrub starts 2024-09-13T13:02:40.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:39 smithi167 ceph-mon[33178]: 3.1f scrub ok 2024-09-13T13:02:40.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:39 smithi167 ceph-mon[33178]: 6.c scrub starts 2024-09-13T13:02:40.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:39 smithi167 ceph-mon[33178]: 6.c scrub ok 2024-09-13T13:02:40.113 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:39 smithi052 ceph-mon[29471]: 3.1f scrub starts 2024-09-13T13:02:40.113 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:39 smithi052 ceph-mon[29471]: 3.1f scrub ok 2024-09-13T13:02:40.113 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:39 smithi052 ceph-mon[29471]: 6.c scrub starts 2024-09-13T13:02:40.113 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:39 smithi052 ceph-mon[29471]: 6.c scrub ok 2024-09-13T13:02:40.113 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:39 smithi052 ceph-mon[36682]: 3.1f scrub starts 2024-09-13T13:02:40.113 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:39 smithi052 ceph-mon[36682]: 3.1f scrub ok 2024-09-13T13:02:40.113 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:39 smithi052 ceph-mon[36682]: 6.c scrub starts 2024-09-13T13:02:40.113 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:39 smithi052 ceph-mon[36682]: 6.c scrub ok 2024-09-13T13:02:40.153 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 18 16 39037 39021 135.353 106.938 0.00262842 0.00737003 2024-09-13T13:02:40.169 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:40.191 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:40.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:40.293 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:40.332 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:40.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:40.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:40.434 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:40.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:40.456 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:40.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:40.500 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:02:40.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:40.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:40.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:40.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:40.714 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:40.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:40.816 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:40.863 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:40.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:40.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:40.993 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:40.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:41.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:40 smithi167 ceph-mon[33178]: 4.4 scrub starts 2024-09-13T13:02:41.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:40 smithi167 ceph-mon[33178]: 4.4 scrub ok 2024-09-13T13:02:41.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:40 smithi167 ceph-mon[33178]: 6.7 scrub starts 2024-09-13T13:02:41.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:40 smithi167 ceph-mon[33178]: 6.7 scrub ok 2024-09-13T13:02:41.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:40 smithi167 ceph-mon[33178]: 4.1f scrub starts 2024-09-13T13:02:41.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:40 smithi167 ceph-mon[33178]: 4.1f scrub ok 2024-09-13T13:02:41.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:40 smithi167 ceph-mon[33178]: 6.18 scrub starts 2024-09-13T13:02:41.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:40 smithi167 ceph-mon[33178]: 6.18 scrub ok 2024-09-13T13:02:41.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:40 smithi167 ceph-mon[33178]: pgmap v58: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 180 MiB/s wr, 2.88k op/s; 30 B/s, 1 objects/s recovering 2024-09-13T13:02:41.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:40 smithi167 ceph-mon[33178]: 3.15 scrub starts 2024-09-13T13:02:41.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:40 smithi167 ceph-mon[33178]: 3.15 scrub ok 2024-09-13T13:02:41.095 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:41.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:41.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:41.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:41.153 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 19 16 40657 40641 133.558 101.25 0.00217511 0.00747231 2024-09-13T13:02:41.228 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:41.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[36682]: 4.4 scrub starts 2024-09-13T13:02:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[36682]: 4.4 scrub ok 2024-09-13T13:02:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[36682]: 6.7 scrub starts 2024-09-13T13:02:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[36682]: 6.7 scrub ok 2024-09-13T13:02:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[36682]: 4.1f scrub starts 2024-09-13T13:02:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[36682]: 4.1f scrub ok 2024-09-13T13:02:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[36682]: 6.18 scrub starts 2024-09-13T13:02:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[36682]: 6.18 scrub ok 2024-09-13T13:02:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[36682]: pgmap v58: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 180 MiB/s wr, 2.88k op/s; 30 B/s, 1 objects/s recovering 2024-09-13T13:02:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[36682]: 3.15 scrub starts 2024-09-13T13:02:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[36682]: 3.15 scrub ok 2024-09-13T13:02:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[29471]: 4.4 scrub starts 2024-09-13T13:02:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[29471]: 4.4 scrub ok 2024-09-13T13:02:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[29471]: 6.7 scrub starts 2024-09-13T13:02:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[29471]: 6.7 scrub ok 2024-09-13T13:02:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[29471]: 4.1f scrub starts 2024-09-13T13:02:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[29471]: 4.1f scrub ok 2024-09-13T13:02:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[29471]: 6.18 scrub starts 2024-09-13T13:02:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[29471]: 6.18 scrub ok 2024-09-13T13:02:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[29471]: pgmap v58: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 180 MiB/s wr, 2.88k op/s; 30 B/s, 1 objects/s recovering 2024-09-13T13:02:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[29471]: 3.15 scrub starts 2024-09-13T13:02:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:40 smithi052 ceph-mon[29471]: 3.15 scrub ok 2024-09-13T13:02:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:41.269 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:41.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:41.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:41.401 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:41.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:41.504 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:41.532 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:41.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:41.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:41.637 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:41.677 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:41.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:41.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:41.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:41.807 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:41.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:41.909 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:41.939 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:41.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:42.041 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:42.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:41 smithi167 ceph-mon[33178]: 4.1b scrub starts 2024-09-13T13:02:42.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:41 smithi167 ceph-mon[33178]: 4.1b scrub ok 2024-09-13T13:02:42.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:41 smithi167 ceph-mon[33178]: 3.0 scrub starts 2024-09-13T13:02:42.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:41 smithi167 ceph-mon[33178]: 3.0 scrub ok 2024-09-13T13:02:42.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:41 smithi167 ceph-mon[33178]: 6.16 scrub starts 2024-09-13T13:02:42.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:41 smithi167 ceph-mon[33178]: 6.16 scrub ok 2024-09-13T13:02:42.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:41 smithi167 ceph-mon[33178]: 3.1e scrub starts 2024-09-13T13:02:42.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:41 smithi167 ceph-mon[33178]: 3.1e scrub ok 2024-09-13T13:02:42.081 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:42.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:42.154 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:02:42.154097+0000 min lat: 0.00146492 max lat: 0.102926 avg lat: 0.00757201 2024-09-13T13:02:42.154 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:02:42.154 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 20 16 42221 42205 131.769 97.75 0.00393958 0.00757201 2024-09-13T13:02:42.183 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:42.224 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:42.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:42.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:42.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[36682]: 4.1b scrub starts 2024-09-13T13:02:42.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[36682]: 4.1b scrub ok 2024-09-13T13:02:42.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[36682]: 3.0 scrub starts 2024-09-13T13:02:42.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[36682]: 3.0 scrub ok 2024-09-13T13:02:42.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[36682]: 6.16 scrub starts 2024-09-13T13:02:42.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[36682]: 6.16 scrub ok 2024-09-13T13:02:42.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[36682]: 3.1e scrub starts 2024-09-13T13:02:42.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[36682]: 3.1e scrub ok 2024-09-13T13:02:42.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[29471]: 4.1b scrub starts 2024-09-13T13:02:42.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[29471]: 4.1b scrub ok 2024-09-13T13:02:42.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[29471]: 3.0 scrub starts 2024-09-13T13:02:42.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[29471]: 3.0 scrub ok 2024-09-13T13:02:42.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[29471]: 6.16 scrub starts 2024-09-13T13:02:42.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[29471]: 6.16 scrub ok 2024-09-13T13:02:42.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[29471]: 3.1e scrub starts 2024-09-13T13:02:42.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:41 smithi052 ceph-mon[29471]: 3.1e scrub ok 2024-09-13T13:02:42.327 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:42.369 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:42.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:42.471 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:42.513 INFO:teuthology.orchestra.run.smithi052.stderr:no valid command found; 10 closest matches: 2024-09-13T13:02:42.513 INFO:teuthology.orchestra.run.smithi052.stderr:0 2024-09-13T13:02:42.513 INFO:teuthology.orchestra.run.smithi052.stderr:1 2024-09-13T13:02:42.515 INFO:teuthology.orchestra.run.smithi052.stderr:2 2024-09-13T13:02:42.515 INFO:teuthology.orchestra.run.smithi052.stderr:abort 2024-09-13T13:02:42.515 INFO:teuthology.orchestra.run.smithi052.stderr:assert 2024-09-13T13:02:42.515 INFO:teuthology.orchestra.run.smithi052.stderr:bench [] [] [] [] 2024-09-13T13:02:42.515 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs debug_inject_read_zeros 2024-09-13T13:02:42.515 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs files list 2024-09-13T13:02:42.516 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs stats 2024-09-13T13:02:42.516 INFO:teuthology.orchestra.run.smithi052.stderr:bluestore allocator dump block 2024-09-13T13:02:42.516 INFO:teuthology.orchestra.run.smithi052.stderr:Error EINVAL: invalid command 2024-09-13T13:02:42.518 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:42.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:42.621 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:42.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:42.665 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:42.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:42.766 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:42.794 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:42.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:42.896 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:42.923 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:42.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:42.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:43.025 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:43.054 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:43.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:43.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:43.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:42 smithi167 ceph-mon[33178]: 3.12 scrub starts 2024-09-13T13:02:43.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:42 smithi167 ceph-mon[33178]: 3.12 scrub ok 2024-09-13T13:02:43.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:42 smithi167 ceph-mon[33178]: pgmap v59: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 627 B/s rd, 72 MiB/s wr, 1.16k op/s; 26 B/s, 1 objects/s recovering 2024-09-13T13:02:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:42 smithi167 ceph-mon[33178]: 6.11 scrub starts 2024-09-13T13:02:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:42 smithi167 ceph-mon[33178]: 6.11 scrub ok 2024-09-13T13:02:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:42 smithi167 ceph-mon[33178]: 5.1b scrub starts 2024-09-13T13:02:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:42 smithi167 ceph-mon[33178]: 5.1b scrub ok 2024-09-13T13:02:43.154 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 21 16 43891 43875 130.465 104.375 0.00917807 0.00765727 2024-09-13T13:02:43.155 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:43.172 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:43.176 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:43.196 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:43.237 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:43.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:43.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[29471]: 3.12 scrub starts 2024-09-13T13:02:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[29471]: 3.12 scrub ok 2024-09-13T13:02:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[29471]: pgmap v59: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 627 B/s rd, 72 MiB/s wr, 1.16k op/s; 26 B/s, 1 objects/s recovering 2024-09-13T13:02:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[29471]: 6.11 scrub starts 2024-09-13T13:02:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[29471]: 6.11 scrub ok 2024-09-13T13:02:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[29471]: 5.1b scrub starts 2024-09-13T13:02:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[29471]: 5.1b scrub ok 2024-09-13T13:02:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[36682]: 3.12 scrub starts 2024-09-13T13:02:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[36682]: 3.12 scrub ok 2024-09-13T13:02:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[36682]: pgmap v59: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 627 B/s rd, 72 MiB/s wr, 1.16k op/s; 26 B/s, 1 objects/s recovering 2024-09-13T13:02:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[36682]: 6.11 scrub starts 2024-09-13T13:02:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[36682]: 6.11 scrub ok 2024-09-13T13:02:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[36682]: 5.1b scrub starts 2024-09-13T13:02:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:42 smithi052 ceph-mon[36682]: 5.1b scrub ok 2024-09-13T13:02:43.339 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:43.368 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:43.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:43.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:43.470 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:43.510 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:43.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:43.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:43.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:43.639 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:43.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:43.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:43.741 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:43.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:43.768 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:43.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:43.803 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T13:02:43.803 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T13:02:43.804 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-13T13:02:43.807 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:02:43 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:02:43.870 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:43.898 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:43.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:44.001 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:44.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:44.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:44.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:43 smithi167 ceph-mon[33178]: 4.16 scrub starts 2024-09-13T13:02:44.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:43 smithi167 ceph-mon[33178]: 4.16 scrub ok 2024-09-13T13:02:44.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:43 smithi167 ceph-mon[33178]: 4.a scrub starts 2024-09-13T13:02:44.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:43 smithi167 ceph-mon[33178]: 4.a scrub ok 2024-09-13T13:02:44.131 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:44.154 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 22 16 45548 45532 129.242 103.562 0.0102409 0.00773183 2024-09-13T13:02:44.182 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:44.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:44.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:44.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:43 smithi052 ceph-mon[36682]: 4.16 scrub starts 2024-09-13T13:02:44.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:43 smithi052 ceph-mon[36682]: 4.16 scrub ok 2024-09-13T13:02:44.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:43 smithi052 ceph-mon[36682]: 4.a scrub starts 2024-09-13T13:02:44.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:43 smithi052 ceph-mon[36682]: 4.a scrub ok 2024-09-13T13:02:44.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:43 smithi052 ceph-mon[29471]: 4.16 scrub starts 2024-09-13T13:02:44.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:43 smithi052 ceph-mon[29471]: 4.16 scrub ok 2024-09-13T13:02:44.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:43 smithi052 ceph-mon[29471]: 4.a scrub starts 2024-09-13T13:02:44.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:43 smithi052 ceph-mon[29471]: 4.a scrub ok 2024-09-13T13:02:44.284 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:44.327 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:44.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:44.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:44.466 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:44.517 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:44.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:44.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:44.653 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:44.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:44.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:44.755 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:44.808 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:44.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:44.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:44.938 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:44.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:45.040 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:45.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: 4.13 scrub starts 2024-09-13T13:02:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: 4.13 scrub ok 2024-09-13T13:02:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: 6.1d scrub starts 2024-09-13T13:02:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: 6.1d scrub ok 2024-09-13T13:02:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: 6.0 scrub starts 2024-09-13T13:02:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: 6.0 scrub ok 2024-09-13T13:02:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: pgmap v60: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.77k op/s; 21 B/s, 1 objects/s recovering 2024-09-13T13:02:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: 4.1 scrub starts 2024-09-13T13:02:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: 4.1 scrub ok 2024-09-13T13:02:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: 3.f scrub starts 2024-09-13T13:02:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: 3.f scrub ok 2024-09-13T13:02:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: 4.19 scrub starts 2024-09-13T13:02:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:44 smithi167 ceph-mon[33178]: 4.19 scrub ok 2024-09-13T13:02:45.080 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:45.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:45.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:45.154 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 23 16 47031 47015 127.654 92.6875 0.0022861 0.00780508 2024-09-13T13:02:45.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:45.207 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:45.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:45.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: 4.13 scrub starts 2024-09-13T13:02:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: 4.13 scrub ok 2024-09-13T13:02:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: 6.1d scrub starts 2024-09-13T13:02:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: 6.1d scrub ok 2024-09-13T13:02:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: 6.0 scrub starts 2024-09-13T13:02:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: 6.0 scrub ok 2024-09-13T13:02:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: pgmap v60: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.77k op/s; 21 B/s, 1 objects/s recovering 2024-09-13T13:02:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: 4.1 scrub starts 2024-09-13T13:02:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: 4.1 scrub ok 2024-09-13T13:02:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: 3.f scrub starts 2024-09-13T13:02:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: 3.f scrub ok 2024-09-13T13:02:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: 4.19 scrub starts 2024-09-13T13:02:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[36682]: 4.19 scrub ok 2024-09-13T13:02:45.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: 4.13 scrub starts 2024-09-13T13:02:45.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: 4.13 scrub ok 2024-09-13T13:02:45.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: 6.1d scrub starts 2024-09-13T13:02:45.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: 6.1d scrub ok 2024-09-13T13:02:45.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:45.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: 6.0 scrub starts 2024-09-13T13:02:45.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: 6.0 scrub ok 2024-09-13T13:02:45.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: pgmap v60: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.77k op/s; 21 B/s, 1 objects/s recovering 2024-09-13T13:02:45.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: 4.1 scrub starts 2024-09-13T13:02:45.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: 4.1 scrub ok 2024-09-13T13:02:45.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: 3.f scrub starts 2024-09-13T13:02:45.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: 3.f scrub ok 2024-09-13T13:02:45.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: 4.19 scrub starts 2024-09-13T13:02:45.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:44 smithi052 ceph-mon[29471]: 4.19 scrub ok 2024-09-13T13:02:45.309 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:45.336 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:45.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:45.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:45.438 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:45.499 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:45.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:45.600 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:45.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:45.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:45.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:45.731 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:45.781 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:45.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:45.883 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:45.913 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:45.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:46.015 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:46.044 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:46.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:46.146 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:46.155 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 24 16 48522 48506 126.211 93.1875 0.0100507 0.00791781 2024-09-13T13:02:46.183 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:46.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:46.286 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:46.326 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:46.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:46.427 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:46.457 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:46.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:46.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:46.474 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:02:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:02:46.559 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:46.598 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:46.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:46.601 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:02:46.601 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:02:46.701 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:46.740 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:46.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:46.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:46.871 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:46.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:46.973 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:47.015 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:47.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:47.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:47.117 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:47.145 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:47.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:47.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:47.155 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 25 16 50098 50082 125.103 98.5 0.00463448 0.00798742 2024-09-13T13:02:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[36682]: 3.9 scrub starts 2024-09-13T13:02:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[36682]: 3.9 scrub ok 2024-09-13T13:02:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[36682]: 5.1d scrub starts 2024-09-13T13:02:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[36682]: 5.1d scrub ok 2024-09-13T13:02:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[36682]: 5.2 scrub starts 2024-09-13T13:02:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[36682]: 5.2 scrub ok 2024-09-13T13:02:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[36682]: pgmap v61: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1010 B/s rd, 109 MiB/s wr, 1.75k op/s; 21 B/s, 1 objects/s recovering 2024-09-13T13:02:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[36682]: 5.9 scrub starts 2024-09-13T13:02:47.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[36682]: 5.9 scrub ok 2024-09-13T13:02:47.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[36682]: 3.18 scrub starts 2024-09-13T13:02:47.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[36682]: 3.18 scrub ok 2024-09-13T13:02:47.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[36682]: 3.c scrub starts 2024-09-13T13:02:47.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[36682]: 3.c scrub ok 2024-09-13T13:02:47.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[29471]: 3.9 scrub starts 2024-09-13T13:02:47.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[29471]: 3.9 scrub ok 2024-09-13T13:02:47.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[29471]: 5.1d scrub starts 2024-09-13T13:02:47.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[29471]: 5.1d scrub ok 2024-09-13T13:02:47.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[29471]: 5.2 scrub starts 2024-09-13T13:02:47.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[29471]: 5.2 scrub ok 2024-09-13T13:02:47.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[29471]: pgmap v61: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1010 B/s rd, 109 MiB/s wr, 1.75k op/s; 21 B/s, 1 objects/s recovering 2024-09-13T13:02:47.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[29471]: 5.9 scrub starts 2024-09-13T13:02:47.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[29471]: 5.9 scrub ok 2024-09-13T13:02:47.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[29471]: 3.18 scrub starts 2024-09-13T13:02:47.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[29471]: 3.18 scrub ok 2024-09-13T13:02:47.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[29471]: 3.c scrub starts 2024-09-13T13:02:47.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:46 smithi052 ceph-mon[29471]: 3.c scrub ok 2024-09-13T13:02:47.247 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:47.284 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:47.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:46 smithi167 ceph-mon[33178]: 3.9 scrub starts 2024-09-13T13:02:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:46 smithi167 ceph-mon[33178]: 3.9 scrub ok 2024-09-13T13:02:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:46 smithi167 ceph-mon[33178]: 5.1d scrub starts 2024-09-13T13:02:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:46 smithi167 ceph-mon[33178]: 5.1d scrub ok 2024-09-13T13:02:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:46 smithi167 ceph-mon[33178]: 5.2 scrub starts 2024-09-13T13:02:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:46 smithi167 ceph-mon[33178]: 5.2 scrub ok 2024-09-13T13:02:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:46 smithi167 ceph-mon[33178]: pgmap v61: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1010 B/s rd, 109 MiB/s wr, 1.75k op/s; 21 B/s, 1 objects/s recovering 2024-09-13T13:02:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:46 smithi167 ceph-mon[33178]: 5.9 scrub starts 2024-09-13T13:02:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:46 smithi167 ceph-mon[33178]: 5.9 scrub ok 2024-09-13T13:02:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:46 smithi167 ceph-mon[33178]: 3.18 scrub starts 2024-09-13T13:02:47.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:46 smithi167 ceph-mon[33178]: 3.18 scrub ok 2024-09-13T13:02:47.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:46 smithi167 ceph-mon[33178]: 3.c scrub starts 2024-09-13T13:02:47.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:46 smithi167 ceph-mon[33178]: 3.c scrub ok 2024-09-13T13:02:47.351 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T13:02:47.386 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:47.462 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:47.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:47.563 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:47.592 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:47.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:47.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:47.694 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:47.744 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:47.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:47.846 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:47.887 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:47.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:47.988 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:48.035 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:48.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:48.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:48.079 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:02:48.079 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:02:48.118 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:02:48.119 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:02:48.139 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:48.156 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 26 16 51444 51428 123.527 84.125 0.00737355 0.00808949 2024-09-13T13:02:48.161 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:02:48.161 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:02:48.164 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:48.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:48.201 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:02:48.201 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:02:48.239 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:02:48.239 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:02:48.265 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:48.273 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:02:48.273 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:02:48.310 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:48.311 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:02:48.311 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:02:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:48.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:48.354 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:02:48.355 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:02:48.412 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:48.439 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:48.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:48.541 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:48.569 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:48.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:48.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:48.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:48.694 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:48.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[36682]: 5.1f scrub starts 2024-09-13T13:02:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[36682]: 5.1f scrub ok 2024-09-13T13:02:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:02:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2028583267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3490162794' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:02:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[36682]: from='client.24820 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:02:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[29471]: 5.1f scrub starts 2024-09-13T13:02:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[29471]: 5.1f scrub ok 2024-09-13T13:02:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:02:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2028583267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3490162794' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:02:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:48 smithi052 ceph-mon[29471]: from='client.24820 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:02:48.796 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:48.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:48 smithi167 ceph-mon[33178]: 5.1f scrub starts 2024-09-13T13:02:48.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:48 smithi167 ceph-mon[33178]: 5.1f scrub ok 2024-09-13T13:02:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:48 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:48 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:02:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:48 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2028583267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:02:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:48 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3490162794' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:02:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:48 smithi167 ceph-mon[33178]: from='client.24820 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:02:48.835 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:48.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:48.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:48.936 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:48.965 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:48.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:49.067 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:49.110 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:49.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:49.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:49.156 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 27 16 52815 52799 122.127 85.6875 0.0117518 0.00818256 2024-09-13T13:02:49.211 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:49.238 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:49.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:49.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:49.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:49.366 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:49.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:49.468 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:49.535 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:49.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:49.637 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:49.663 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:49.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:49 smithi052 ceph-mon[36682]: pgmap v62: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s; 18 B/s, 0 objects/s recovering 2024-09-13T13:02:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:49 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:02:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:49 smithi052 ceph-mon[36682]: from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:02:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:49 smithi052 ceph-mon[36682]: osdmap e95: 8 total, 8 up, 8 in 2024-09-13T13:02:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:49 smithi052 ceph-mon[29471]: pgmap v62: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s; 18 B/s, 0 objects/s recovering 2024-09-13T13:02:49.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:49 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:02:49.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:49 smithi052 ceph-mon[29471]: from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:02:49.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:49 smithi052 ceph-mon[29471]: osdmap e95: 8 total, 8 up, 8 in 2024-09-13T13:02:49.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:49.791 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:49.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:49.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:49.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:49 smithi167 ceph-mon[33178]: pgmap v62: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s; 18 B/s, 0 objects/s recovering 2024-09-13T13:02:49.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:49 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:02:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:49 smithi167 ceph-mon[33178]: from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:02:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:49 smithi167 ceph-mon[33178]: osdmap e95: 8 total, 8 up, 8 in 2024-09-13T13:02:49.833 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:02:49.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:49.945 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:49.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:49.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:50.047 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:50.108 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:50.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:50.156 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 28 16 54300 54284 121.078 92.8125 0.0050243 0.00825381 2024-09-13T13:02:50.210 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:50.237 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:50.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:50.339 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:50.370 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:50.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:50.471 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:50.496 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:50.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:50.522 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:50.598 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:50.649 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:50.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:50.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:50.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:50.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:50.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[36682]: pgmap v64: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-13T13:02:50.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:02:50.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:02:50.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:50.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[36682]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T13:02:50.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[36682]: Reconfiguring daemon alertmanager.a on smithi052 2024-09-13T13:02:50.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:50.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:50.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[29471]: pgmap v64: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-13T13:02:50.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:02:50.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:02:50.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:50.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[29471]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T13:02:50.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:50 smithi052 ceph-mon[29471]: Reconfiguring daemon alertmanager.a on smithi052 2024-09-13T13:02:50.751 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:50.804 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:50.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:50.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:50 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:50.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:50 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:50.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:50 smithi167 ceph-mon[33178]: pgmap v64: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-13T13:02:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:50 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:02:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:50 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:02:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:50 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:02:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:50 smithi167 ceph-mon[33178]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T13:02:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:50 smithi167 ceph-mon[33178]: Reconfiguring daemon alertmanager.a on smithi052 2024-09-13T13:02:50.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:50.933 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:50.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:50.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:51.035 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:51.061 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:51.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:51.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:51.156 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 29 16 55701 55685 119.923 87.5625 0.00852373 0.00833317 2024-09-13T13:02:51.163 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:51.204 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:51.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:51.305 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:51.331 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:51.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:51.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:51.457 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:51.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:51.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:51.559 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:51.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:51.590 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:51.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:51.691 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:51.722 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:51.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:51.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:51.824 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:51.867 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:51.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:51.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:51.969 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:52.015 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:52.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:52.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:52.117 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:52.147 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:52.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:52.156 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 30 16 57242 57226 119.136 96.3125 0.00958687 0.00838827 2024-09-13T13:02:52.249 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:52.291 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:52.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:52.393 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:52.443 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:52.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:52.515 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:02:52.545 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:52.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:52.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:52.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:52.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:52.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:52.802 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:52.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:52.864 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:52.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:52.965 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:53.009 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:53.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:53.111 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:53.157 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 31 16 58693 58677 118.219 90.6875 0.017979 0.00845363 2024-09-13T13:02:53.162 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:53.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:53.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:53.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:52 smithi052 ceph-mon[36682]: pgmap v65: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-13T13:02:53.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:52 smithi052 ceph-mon[36682]: osdmap e96: 8 total, 8 up, 8 in 2024-09-13T13:02:53.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:52 smithi052 ceph-mon[29471]: pgmap v65: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-13T13:02:53.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:52 smithi052 ceph-mon[29471]: osdmap e96: 8 total, 8 up, 8 in 2024-09-13T13:02:53.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:53.291 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:53.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:53.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:52 smithi167 ceph-mon[33178]: pgmap v65: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-13T13:02:53.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:52 smithi167 ceph-mon[33178]: osdmap e96: 8 total, 8 up, 8 in 2024-09-13T13:02:53.393 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:53.431 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:53.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:53.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:53.490 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:02:53.533 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:53.577 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:53.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:53.609 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:53.679 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:53.739 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:53.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:53.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:53.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:02:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:02:53.841 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:02:53.889 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:53.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:53.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:02:53.991 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:54.036 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:54.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:54.138 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:54.158 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 32 16 60054 60038 117.18 85.0625 0.00535543 0.00852829 2024-09-13T13:02:54.167 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:54.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:54.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:54.269 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:54.315 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:54.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:54.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:54.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:54.445 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:54.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:54.547 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:54.592 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:54.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:54.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:54.721 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:54.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:54.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:54.804 INFO:teuthology.orchestra.run.smithi052.stderr:no valid command found; 10 closest matches: 2024-09-13T13:02:54.804 INFO:teuthology.orchestra.run.smithi052.stderr:0 2024-09-13T13:02:54.804 INFO:teuthology.orchestra.run.smithi052.stderr:1 2024-09-13T13:02:54.804 INFO:teuthology.orchestra.run.smithi052.stderr:2 2024-09-13T13:02:54.804 INFO:teuthology.orchestra.run.smithi052.stderr:abort 2024-09-13T13:02:54.804 INFO:teuthology.orchestra.run.smithi052.stderr:assert 2024-09-13T13:02:54.804 INFO:teuthology.orchestra.run.smithi052.stderr:bench [] [] [] [] 2024-09-13T13:02:54.805 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs debug_inject_read_zeros 2024-09-13T13:02:54.805 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs files list 2024-09-13T13:02:54.805 INFO:teuthology.orchestra.run.smithi052.stderr:bluefs stats 2024-09-13T13:02:54.805 INFO:teuthology.orchestra.run.smithi052.stderr:bluestore allocator dump block 2024-09-13T13:02:54.805 INFO:teuthology.orchestra.run.smithi052.stderr:Error EINVAL: invalid command 2024-09-13T13:02:54.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:54.854 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:54.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:54.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:54.956 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:54.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:54 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:54.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:54 smithi052 ceph-mon[36682]: pgmap v67: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-13T13:02:54.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:54 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:54.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:54 smithi052 ceph-mon[29471]: pgmap v67: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-13T13:02:54.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:54.999 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:54.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:55.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:55.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:55.129 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:55.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:55.158 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 33 16 61404 61388 116.186 84.375 0.00997258 0.0086013 2024-09-13T13:02:55.230 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:55.257 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:55.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:55.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:54 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:02:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:54 smithi167 ceph-mon[33178]: pgmap v67: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-13T13:02:55.359 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:55.405 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:55.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:55.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:55.507 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:55.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:55.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:55.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:55.635 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:55.660 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:55.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:55.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:55.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:55.788 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:55.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:55.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:55.890 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:55.892 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:02:55 smithi052 systemd[1]: Stopping Ceph alertmanager.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:02:55.938 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:55.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:56.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:56.066 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:56.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:56.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:56.158 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 34 16 62625 62609 115.014 76.3125 0.00636886 0.00868859 2024-09-13T13:02:56.168 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:56.197 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:56.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:56.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:56.284 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T13:02:56.284 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T13:02:56.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:56.321 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:56.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:56.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:56.423 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:56.448 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:56.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:56.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:56.460 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:02:56.460 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:02:56.460 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:02:56.460 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:02:56.460 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:11909 7.e 7:7b5c689f:::benchmark_data_smithi052_71865_object11908:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:26.881006+0000", 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.088799226999999, 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.065572517999999996, 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 11909 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.462 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.462 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.881006+0000", 2024-09-13T13:02:56.462 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.462 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.462 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.462 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.881006+0000", 2024-09-13T13:02:56.462 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.462 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.462 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.462 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.881009+0000", 2024-09-13T13:02:56.462 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3379999999999999e-06 2024-09-13T13:02:56.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.463 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.463 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.881034+0000", 2024-09-13T13:02:56.463 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5202e-05 2024-09-13T13:02:56.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.463 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.463 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.881051+0000", 2024-09-13T13:02:56.463 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7102999999999999e-05 2024-09-13T13:02:56.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.463 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.464 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.881064+0000", 2024-09-13T13:02:56.464 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2575000000000001e-05 2024-09-13T13:02:56.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.464 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.464 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.881130+0000", 2024-09-13T13:02:56.464 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6326000000000002e-05 2024-09-13T13:02:56.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.464 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.464 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.881282+0000", 2024-09-13T13:02:56.464 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000152378 2024-09-13T13:02:56.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.465 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.465 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.881430+0000", 2024-09-13T13:02:56.465 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014787099999999999 2024-09-13T13:02:56.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.465 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.465 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.883657+0000", 2024-09-13T13:02:56.465 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002226502 2024-09-13T13:02:56.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.465 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.466 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.889259+0000", 2024-09-13T13:02:56.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0056027919999999997 2024-09-13T13:02:56.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.466 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.466 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.466 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946450+0000", 2024-09-13T13:02:56.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.057190223999999998 2024-09-13T13:02:56.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.466 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.466 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.466 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946537+0000", 2024-09-13T13:02:56.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7621999999999996e-05 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946579+0000", 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1585000000000002e-05 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:11920 7.e 7:73991523:::benchmark_data_smithi052_71865_object11919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:26.886550+0000", 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.083255114, 2024-09-13T13:02:56.467 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.060164226000000001, 2024-09-13T13:02:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 11920 2024-09-13T13:02:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886550+0000", 2024-09-13T13:02:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.469 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.469 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886550+0000", 2024-09-13T13:02:56.469 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.469 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.469 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886553+0000", 2024-09-13T13:02:56.469 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1009999999999999e-06 2024-09-13T13:02:56.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886577+0000", 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4009e-05 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886594+0000", 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7028e-05 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886602+0000", 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.799e-06 2024-09-13T13:02:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886659+0000", 2024-09-13T13:02:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7070000000000001e-05 2024-09-13T13:02:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886774+0000", 2024-09-13T13:02:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114945 2024-09-13T13:02:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886911+0000", 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000136198 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.888027+0000", 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011161439999999999 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.889297+0000", 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001269795 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946612+0000", 2024-09-13T13:02:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.057315193 2024-09-13T13:02:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946683+0000", 2024-09-13T13:02:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1333000000000003e-05 2024-09-13T13:02:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946715+0000", 2024-09-13T13:02:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1610999999999999e-05 2024-09-13T13:02:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:11938 7.e 7:74a0a033:::benchmark_data_smithi052_71865_object11937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:26.900171+0000", 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.069634334, 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.047028012000000001, 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 11938 2024-09-13T13:02:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.900171+0000", 2024-09-13T13:02:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.900171+0000", 2024-09-13T13:02:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.900174+0000", 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3779999999999999e-06 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.900197+0000", 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3745999999999999e-05 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.900214+0000", 2024-09-13T13:02:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6371e-05 2024-09-13T13:02:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.900222+0000", 2024-09-13T13:02:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.96e-06 2024-09-13T13:02:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.900279+0000", 2024-09-13T13:02:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7113e-05 2024-09-13T13:02:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.900374+0000", 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4747999999999995e-05 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.900453+0000", 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9305e-05 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.901891+0000", 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001438099 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.902089+0000", 2024-09-13T13:02:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019790199999999999 2024-09-13T13:02:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946867+0000", 2024-09-13T13:02:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.044778114000000001 2024-09-13T13:02:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.947136+0000", 2024-09-13T13:02:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00026912300000000002 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.947199+0000", 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3152999999999997e-05 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:11945 7.e 7:798f7930:::benchmark_data_smithi052_71865_object11944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:26.907337+0000", 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.062468532, 2024-09-13T13:02:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.040422434, 2024-09-13T13:02:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 11945 2024-09-13T13:02:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.907337+0000", 2024-09-13T13:02:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.907337+0000", 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.907340+0000", 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7e-06 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.907367+0000", 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7067999999999999e-05 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.907383+0000", 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6478999999999999e-05 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.907392+0000", 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4509999999999992e-06 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.907461+0000", 2024-09-13T13:02:56.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9051e-05 2024-09-13T13:02:56.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.907587+0000", 2024-09-13T13:02:56.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012610200000000001 2024-09-13T13:02:56.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.907739+0000", 2024-09-13T13:02:56.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015188199999999999 2024-09-13T13:02:56.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.910710+0000", 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029713389999999999 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.910981+0000", 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027065599999999998 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.947367+0000", 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.036386151999999998 2024-09-13T13:02:56.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.947692+0000", 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00032558100000000003 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.947759+0000", 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6972999999999997e-05 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:21681 7.e 7:75122e2d:::benchmark_data_smithi052_71865_object21680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:30.925923+0000", 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 25.043882710999998, 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.044824910000000003, 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 21681 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.925923+0000", 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.925923+0000", 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.925926+0000", 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7980000000000001e-06 2024-09-13T13:02:56.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.489 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.489 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.925960+0000", 2024-09-13T13:02:56.489 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4476e-05 2024-09-13T13:02:56.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.489 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.489 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.925973+0000", 2024-09-13T13:02:56.489 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3091000000000001e-05 2024-09-13T13:02:56.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.489 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.489 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.925982+0000", 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8540000000000003e-06 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.926043+0000", 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1253999999999993e-05 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.926163+0000", 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000119734 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.926293+0000", 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000130078 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.927386+0000", 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010925170000000001 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.927972+0000", 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00058637900000000005 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.492 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.492 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.970640+0000", 2024-09-13T13:02:56.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.042668425000000003 2024-09-13T13:02:56.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.492 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.492 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.970704+0000", 2024-09-13T13:02:56.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3479999999999995e-05 2024-09-13T13:02:56.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.492 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:30.970748+0000", 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3823999999999999e-05 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:22477 7.e 7:78069d87:::benchmark_data_smithi052_71865_object22476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:31.285317+0000", 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 24.684488877, 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.058367704999999999, 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.493 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 22477 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.285317+0000", 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.285317+0000", 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.285319+0000", 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8700000000000001e-06 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.285344+0000", 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5405000000000002e-05 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.495 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.496 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.285355+0000", 2024-09-13T13:02:56.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0861e-05 2024-09-13T13:02:56.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.496 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.496 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.285362+0000", 2024-09-13T13:02:56.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3699999999999997e-06 2024-09-13T13:02:56.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.496 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.496 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.285410+0000", 2024-09-13T13:02:56.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7450999999999997e-05 2024-09-13T13:02:56.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.285522+0000", 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112428 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.285652+0000", 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012961200000000001 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.286518+0000", 2024-09-13T13:02:56.497 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00086620700000000002 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.287059+0000", 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00054080399999999998 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.343549+0000", 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.056490375000000002 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.343644+0000", 2024-09-13T13:02:56.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4435e-05 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.343684+0000", 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0886999999999997e-05 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:23372 7.e 7:7774cb23:::benchmark_data_smithi052_71865_object23371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:31.708407+0000", 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 24.261398897999999, 2024-09-13T13:02:56.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.040841874, 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 23372 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.708407+0000", 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.708407+0000", 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.708408+0000", 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.66e-06 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.708432+0000", 2024-09-13T13:02:56.501 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3666e-05 2024-09-13T13:02:56.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.502 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.502 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.708441+0000", 2024-09-13T13:02:56.502 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3430000000000002e-06 2024-09-13T13:02:56.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.502 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.502 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.708448+0000", 2024-09-13T13:02:56.502 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5889999999999999e-06 2024-09-13T13:02:56.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.502 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.708499+0000", 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1065e-05 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.708601+0000", 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010180599999999999 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.708726+0000", 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012530700000000001 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.503 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.709648+0000", 2024-09-13T13:02:56.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00092172200000000004 2024-09-13T13:02:56.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.504 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.709834+0000", 2024-09-13T13:02:56.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018661900000000001 2024-09-13T13:02:56.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.504 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.749087+0000", 2024-09-13T13:02:56.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.039252096 2024-09-13T13:02:56.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.749199+0000", 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112012 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.749249+0000", 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9988999999999999e-05 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:24428 7.e 7:764166ae:::benchmark_data_smithi052_71865_object24427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:32.193493+0000", 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 23.776312809, 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.032830595999999997, 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 24428 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.193493+0000", 2024-09-13T13:02:56.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.193493+0000", 2024-09-13T13:02:56.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.193494+0000", 2024-09-13T13:02:56.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5680000000000001e-06 2024-09-13T13:02:56.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.508 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.508 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.193524+0000", 2024-09-13T13:02:56.508 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9342000000000001e-05 2024-09-13T13:02:56.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.508 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.508 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.193535+0000", 2024-09-13T13:02:56.508 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1328000000000001e-05 2024-09-13T13:02:56.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.508 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.193543+0000", 2024-09-13T13:02:56.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5619999999999998e-06 2024-09-13T13:02:56.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.194150+0000", 2024-09-13T13:02:56.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00060735599999999998 2024-09-13T13:02:56.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.194251+0000", 2024-09-13T13:02:56.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101383 2024-09-13T13:02:56.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.194370+0000", 2024-09-13T13:02:56.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011876500000000001 2024-09-13T13:02:56.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.195869+0000", 2024-09-13T13:02:56.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001499039 2024-09-13T13:02:56.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.195895+0000", 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5871999999999999e-05 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.226229+0000", 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.030333995999999998 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.226283+0000", 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4011000000000002e-05 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.226323+0000", 2024-09-13T13:02:56.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0373999999999997e-05 2024-09-13T13:02:56.512 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.512 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.512 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.512 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:25068 7.e 7:7f7f90d5:::benchmark_data_smithi052_71865_object25067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:02:56.512 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:32.487947+0000", 2024-09-13T13:02:56.512 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 23.481858773999999, 2024-09-13T13:02:56.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.059837394000000002, 2024-09-13T13:02:56.512 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.512 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.512 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.513 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.513 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.513 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.513 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 25068 2024-09-13T13:02:56.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.513 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.487947+0000", 2024-09-13T13:02:56.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.487947+0000", 2024-09-13T13:02:56.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.487949+0000", 2024-09-13T13:02:56.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3810000000000002e-06 2024-09-13T13:02:56.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.487975+0000", 2024-09-13T13:02:56.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6225999999999999e-05 2024-09-13T13:02:56.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.515 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.487988+0000", 2024-09-13T13:02:56.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3029e-05 2024-09-13T13:02:56.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.515 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.487997+0000", 2024-09-13T13:02:56.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5709999999999998e-06 2024-09-13T13:02:56.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.515 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.488059+0000", 2024-09-13T13:02:56.516 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1556999999999994e-05 2024-09-13T13:02:56.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.488187+0000", 2024-09-13T13:02:56.516 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012892000000000001 2024-09-13T13:02:56.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.488316+0000", 2024-09-13T13:02:56.516 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012864899999999999 2024-09-13T13:02:56.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.489475+0000", 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001158699 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.489651+0000", 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017654199999999999 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.547635+0000", 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.057983641000000002 2024-09-13T13:02:56.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.547739+0000", 2024-09-13T13:02:56.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103545 2024-09-13T13:02:56.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:32.547784+0000", 2024-09-13T13:02:56.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5633999999999997e-05 2024-09-13T13:02:56.518 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.518 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.518 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.519 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:29282 7.e 7:7215e9ef:::benchmark_data_smithi052_71865_object29281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-13T13:02:56.519 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:34.657740+0000", 2024-09-13T13:02:56.519 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 21.312065567000001, 2024-09-13T13:02:56.519 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.034127188000000003, 2024-09-13T13:02:56.519 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.519 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.519 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.519 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.520 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.520 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.520 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 29282 2024-09-13T13:02:56.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.520 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.520 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.657740+0000", 2024-09-13T13:02:56.520 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.520 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.657740+0000", 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.657741+0000", 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.088e-06 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.657762+0000", 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1061e-05 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.657775+0000", 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2489e-05 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.657781+0000", 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5189999999999998e-06 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.660295+0000", 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002513398 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.660390+0000", 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5397999999999997e-05 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.660505+0000", 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011493 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.662748+0000", 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002243227 2024-09-13T13:02:56.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.664945+0000", 2024-09-13T13:02:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0021964739999999999 2024-09-13T13:02:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.691751+0000", 2024-09-13T13:02:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.026806409 2024-09-13T13:02:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.691837+0000", 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6394e-05 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.691867+0000", 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9801000000000001e-05 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:29299 7.8 7:170eb23c:::benchmark_data_smithi052_71865_object29298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:34.671246+0000", 2024-09-13T13:02:56.525 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 21.298560041999998, 2024-09-13T13:02:56.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.035367021999999998, 2024-09-13T13:02:56.526 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.526 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.526 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.526 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.526 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.526 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.526 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 29299 2024-09-13T13:02:56.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.526 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.671246+0000", 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.671246+0000", 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.671249+0000", 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.527 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.671277+0000", 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8914000000000001e-05 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.671295+0000", 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7445999999999999e-05 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.671305+0000", 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8789999999999998e-06 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.671371+0000", 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6194999999999999e-05 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.671491+0000", 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011995299999999999 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.671625+0000", 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013438999999999999 2024-09-13T13:02:56.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.672802+0000", 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011766400000000001 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.687535+0000", 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014732567 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.706465+0000", 2024-09-13T13:02:56.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.018930802 2024-09-13T13:02:56.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.706567+0000", 2024-09-13T13:02:56.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101393 2024-09-13T13:02:56.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.706613+0000", 2024-09-13T13:02:56.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5852000000000001e-05 2024-09-13T13:02:56.531 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.531 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.532 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.532 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:29742 7.8 7:11d6db7a:::benchmark_data_smithi052_71865_object29741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-13T13:02:56.532 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:34.912099+0000", 2024-09-13T13:02:56.532 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 21.057706392, 2024-09-13T13:02:56.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.035113121999999997, 2024-09-13T13:02:56.532 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.532 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.532 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.532 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.532 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.532 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 29742 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.912099+0000", 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.912099+0000", 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.912101+0000", 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.849e-06 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.912125+0000", 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3927e-05 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.912136+0000", 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1104000000000001e-05 2024-09-13T13:02:56.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.912143+0000", 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2649999999999999e-06 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.913858+0000", 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0017149750000000001 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.913969+0000", 2024-09-13T13:02:56.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110753 2024-09-13T13:02:56.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:02:56.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.914086+0000", 2024-09-13T13:02:56.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117092 2024-09-13T13:02:56.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.916147+0000", 2024-09-13T13:02:56.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020606560000000001 2024-09-13T13:02:56.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.929525+0000", 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.01337852 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.947081+0000", 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.017555404 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.947168+0000", 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7687000000000003e-05 2024-09-13T13:02:56.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.947212+0000", 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3890000000000002e-05 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:31545 7.e 7:73d1b538:::benchmark_data_smithi052_71865_object31544:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:35.935482+0000", 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 20.034323399000002, 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.041840580000000002, 2024-09-13T13:02:56.538 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 31545 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.935482+0000", 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.935482+0000", 2024-09-13T13:02:56.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.935492+0000", 2024-09-13T13:02:56.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6660000000000003e-06 2024-09-13T13:02:56.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.935516+0000", 2024-09-13T13:02:56.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4456000000000001e-05 2024-09-13T13:02:56.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.935527+0000", 2024-09-13T13:02:56.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0672e-05 2024-09-13T13:02:56.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.935534+0000", 2024-09-13T13:02:56.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3209999999999999e-06 2024-09-13T13:02:56.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.936729+0000", 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011951399999999999 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.936844+0000", 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011417399999999999 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.936987+0000", 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014379799999999999 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.938238+0000", 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012501669999999999 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.943796+0000", 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.005558637 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.977216+0000", 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.033419702000000003 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.977268+0000", 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2543000000000002e-05 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:35.977323+0000", 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4304e-05 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.544 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:51449 7.e 7:7632f73f:::benchmark_data_smithi052_71865_object51448:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T13:02:56.545 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.159462+0000", 2024-09-13T13:02:56.545 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 7.8103433200000003, 2024-09-13T13:02:56.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.033410855000000003, 2024-09-13T13:02:56.545 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.545 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.545 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.545 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.545 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.545 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.545 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 51449 2024-09-13T13:02:56.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.545 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.159462+0000", 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.159462+0000", 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.159465+0000", 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8449999999999999e-06 2024-09-13T13:02:56.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.159492+0000", 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6455e-05 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.159505+0000", 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3183999999999999e-05 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.159513+0000", 2024-09-13T13:02:56.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2789999999999998e-06 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.160452+0000", 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00093867899999999999 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.160571+0000", 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011977 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.160708+0000", 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000136105 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.165097+0000", 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0043898909999999999 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.165139+0000", 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1411e-05 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.192778+0000", 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.02763941 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.192829+0000", 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1181999999999999e-05 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.192873+0000", 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3643999999999998e-05 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.550 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.551 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.551 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:51453 7.e 7:7e5fddbf:::benchmark_data_smithi052_71865_object51452:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T13:02:56.551 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.160354+0000", 2024-09-13T13:02:56.551 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 7.8094510939999999, 2024-09-13T13:02:56.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.036660702000000003, 2024-09-13T13:02:56.551 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.551 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.551 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.551 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.551 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.551 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 51453 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.160354+0000", 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.160354+0000", 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.160356+0000", 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5680000000000001e-06 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.160380+0000", 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4136000000000001e-05 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.160391+0000", 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0923e-05 2024-09-13T13:02:56.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.160399+0000", 2024-09-13T13:02:56.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0209999999999999e-06 2024-09-13T13:02:56.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.163267+0000", 2024-09-13T13:02:56.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0028676130000000002 2024-09-13T13:02:56.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.163380+0000", 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113471 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.163507+0000", 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000127234 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.165274+0000", 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0017666800000000001 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.165641+0000", 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00036664899999999998 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.196855+0000", 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.031214627000000002 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.196971+0000", 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011517 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.197015+0000", 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4610000000000001e-05 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:52587 7.e 7:798f87ce:::benchmark_data_smithi052_71865_object52586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.957417+0000", 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 7.0123890329999998, 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.064968235999999999, 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.557 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.558 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.558 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.558 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.558 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.558 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 52587 2024-09-13T13:02:56.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.558 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957417+0000", 2024-09-13T13:02:56.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957417+0000", 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957418+0000", 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.917e-06 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957442+0000", 2024-09-13T13:02:56.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3181999999999999e-05 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957452+0000", 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0387999999999999e-05 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957459+0000", 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9260000000000003e-06 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957936+0000", 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00047749700000000001 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958056+0000", 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011951 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958255+0000", 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019871600000000001 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.960384+0000", 2024-09-13T13:02:56.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0021292720000000002 2024-09-13T13:02:56.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.960529+0000", 2024-09-13T13:02:56.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000144962 2024-09-13T13:02:56.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.022213+0000", 2024-09-13T13:02:56.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.061683644000000003 2024-09-13T13:02:56.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.022336+0000", 2024-09-13T13:02:56.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012357700000000001 2024-09-13T13:02:56.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.022385+0000", 2024-09-13T13:02:56.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8644999999999999e-05 2024-09-13T13:02:56.563 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.563 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.563 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:53356 7.e 7:7ccb6de1:::benchmark_data_smithi052_71865_object53355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:49.491341+0000", 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 6.4784646410000004, 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.033391704000000001, 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 53356 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.564 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.491341+0000", 2024-09-13T13:02:56.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.491341+0000", 2024-09-13T13:02:56.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.491343+0000", 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9980000000000002e-06 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.491369+0000", 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5896000000000001e-05 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.491379+0000", 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9830000000000008e-06 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.491386+0000", 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2960000000000001e-06 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.492797+0000", 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0014105739999999999 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.492894+0000", 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7332999999999998e-05 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.493027+0000", 2024-09-13T13:02:56.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013283300000000001 2024-09-13T13:02:56.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.495185+0000", 2024-09-13T13:02:56.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0021584400000000002 2024-09-13T13:02:56.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.495197+0000", 2024-09-13T13:02:56.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1575e-05 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.524505+0000", 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.029308280999999999 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.524676+0000", 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000171015 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.524733+0000", 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6480000000000001e-05 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:53371 7.e 7:7dc80d25:::benchmark_data_smithi052_71865_object53370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:49.500422+0000", 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 6.469383444, 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.030340255999999999, 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 53371 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.500422+0000", 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.500422+0000", 2024-09-13T13:02:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.500424+0000", 2024-09-13T13:02:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2680000000000001e-06 2024-09-13T13:02:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.500449+0000", 2024-09-13T13:02:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5020000000000001e-05 2024-09-13T13:02:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.500466+0000", 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6897000000000001e-05 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.500475+0000", 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5010000000000004e-06 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.501655+0000", 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001180462 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.501762+0000", 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107062 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.501889+0000", 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012656899999999999 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.503759+0000", 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001869814 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.504101+0000", 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00034211799999999998 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.530663+0000", 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.026562558 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.530718+0000", 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5102999999999997e-05 2024-09-13T13:02:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.530762+0000", 2024-09-13T13:02:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3884000000000002e-05 2024-09-13T13:02:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:55550 7.e 7:7f39fec4:::benchmark_data_smithi052_71865_object55549:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:02:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:51.041699+0000", 2024-09-13T13:02:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 4.9281068269999997, 2024-09-13T13:02:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.034864391000000002, 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 55550 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.041699+0000", 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.041699+0000", 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.041700+0000", 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.389e-06 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.041724+0000", 2024-09-13T13:02:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3385999999999999e-05 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.041733+0000", 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4229999999999994e-06 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.041740+0000", 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3900000000000004e-06 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.042176+0000", 2024-09-13T13:02:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00043544599999999998 2024-09-13T13:02:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.042293+0000", 2024-09-13T13:02:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117007 2024-09-13T13:02:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:02:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.042412+0000", 2024-09-13T13:02:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000119329 2024-09-13T13:02:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.044379+0000", 2024-09-13T13:02:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019671210000000001 2024-09-13T13:02:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.048359+0000", 2024-09-13T13:02:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039797490000000003 2024-09-13T13:02:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.076471+0000", 2024-09-13T13:02:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.028111918999999999 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.076524+0000", 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3232000000000001e-05 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:51.076563+0000", 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8999999999999999e-05 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:60366 7.8 7:123f92df:::benchmark_data_smithi052_71865_object60365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T13:02:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:54.371005+0000", 2024-09-13T13:02:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 1.5988005279999999, 2024-09-13T13:02:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.029902756999999999, 2024-09-13T13:02:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:02:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:02:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:02:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:02:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:02:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:02:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 60366 2024-09-13T13:02:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.371005+0000", 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.371005+0000", 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:02:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.371007+0000", 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9439999999999999e-06 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.371031+0000", 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4212000000000001e-05 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.371045+0000", 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4100000000000001e-05 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:02:56.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.371053+0000", 2024-09-13T13:02:56.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6290000000000001e-06 2024-09-13T13:02:56.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:02:56.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.371242+0000", 2024-09-13T13:02:56.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018921500000000001 2024-09-13T13:02:56.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.371354+0000", 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112266 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.371495+0000", 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014037199999999999 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.373668+0000", 2024-09-13T13:02:56.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0021736339999999998 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.373699+0000", 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0691999999999997e-05 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.400812+0000", 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.027112496 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:02:56.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.400865+0000", 2024-09-13T13:02:56.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3767000000000002e-05 2024-09-13T13:02:56.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:02:56.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:02:56.591 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:02:56.591 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:54.400908+0000", 2024-09-13T13:02:56.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2429999999999999e-05 2024-09-13T13:02:56.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.591 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:02:56.591 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:02:56.591 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:02:56.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:56.593 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:02:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:02:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:02:56.618 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:56.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:56.721 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:56.748 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:56.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:56.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:56.787 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:56.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:56.874 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:56.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:56.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:56.976 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:02:57.028 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:57.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:57.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:02:57.130 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:57.157 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:57.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:57.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:57.158 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 35 16 63804 63788 113.833 73.6875 0.00756791 0.00877955 2024-09-13T13:02:57.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:56 smithi052 ceph-mon[36682]: pgmap v68: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T13:02:57.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:56 smithi052 ceph-mon[29471]: pgmap v68: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T13:02:57.258 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:57.301 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:57.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:57.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:56 smithi167 ceph-mon[33178]: pgmap v68: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T13:02:57.405 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:57.448 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:57.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:57.460 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T13:02:57.495 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:02:57.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:57.576 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:57.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:57.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:57.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:57.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:57.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:57.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:57.856 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:57.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:57.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:57.958 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:58.007 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:58.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:58.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:58.111 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:58.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:58.140 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:58.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:58.158 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 36 16 64994 64978 112.738 74.375 0.00361369 0.00886302 2024-09-13T13:02:58.241 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:58.286 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:58.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:58.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:58.412 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:58.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:58.514 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:02:58.543 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:58.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:58.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:02:58.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:58.679 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:58.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:58.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:58.781 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:58.826 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:58.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:58.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:58.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:58.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:58.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:58 smithi052 ceph-mon[36682]: pgmap v69: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-13T13:02:58.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:58 smithi052 ceph-mon[29471]: pgmap v69: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-13T13:02:59.054 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:59.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:58 smithi167 ceph-mon[33178]: pgmap v69: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-13T13:02:59.080 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:59.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:02:59.158 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 37 16 66194 66178 111.718 75 0.00962978 0.00894589 2024-09-13T13:02:59.182 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:59.210 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:59.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:59.312 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:02:59.347 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:59.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:59.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:02:59.449 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:02:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:59.494 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:59.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:02:59.595 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:02:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:59.643 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:02:59.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:02:59.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:59 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1146520060' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:02:59.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:02:59 smithi052 ceph-mon[36682]: from='client.24838 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:02:59.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:59 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1146520060' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:02:59.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:02:59 smithi052 ceph-mon[29471]: from='client.24838 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:02:59.739 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:02:59 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[68036]: ts=2024-09-13T13:02:59.486Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T13:02:59.740 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:02:59 smithi052 podman[76045]: 2024-09-13 13:02:59.52998142 +0000 UTC m=+1.697997446 container died d5ffde3514bfbfc85a53fa4eb0ffc90c778eaa60a24c10e7b4179991131a5f22 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T13:02:59.744 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:02:59.775 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:59.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:59.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:02:59.877 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:02:59.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:02:59.910 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:02:59.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:00.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:00.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:00.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:00.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:00.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:59 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1146520060' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:00.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:02:59 smithi167 ceph-mon[33178]: from='client.24838 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:00.151 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:00.158 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 38 16 67517 67501 110.955 82.6875 0.0120897 0.00900731 2024-09-13T13:03:00.195 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:00.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:00.239 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:02:59 smithi052 podman[76045]: 2024-09-13 13:02:59.982579982 +0000 UTC m=+2.150595997 container cleanup d5ffde3514bfbfc85a53fa4eb0ffc90c778eaa60a24c10e7b4179991131a5f22 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T13:03:00.239 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:02:59 smithi052 bash[76045]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T13:03:00.297 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:00.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:00.339 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:00.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:00.440 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:00.472 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:00.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:00.577 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:00.618 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:00.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:00.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:00.639 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:03:00.639 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:00 smithi052 podman[76204]: 2024-09-13 13:03:00.370810807 +0000 UTC m=+0.836342844 container remove d5ffde3514bfbfc85a53fa4eb0ffc90c778eaa60a24c10e7b4179991131a5f22 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T13:03:00.639 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:00 smithi052 podman[76204]: 2024-09-13 13:03:00.478492473 +0000 UTC m=+0.944024510 volume remove ea397d3f52bf5b9b865f0a80ed68cee6676cd751b98aca42e02059116c19d12e 2024-09-13T13:03:00.720 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:00.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:00.754 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:00.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:00.855 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:00.884 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:00.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:00.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:00.965 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:00 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:00.965 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:00 smithi052 ceph-mon[36682]: from='client.24838 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:00.965 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:00 smithi052 ceph-mon[36682]: osdmap e97: 8 total, 8 up, 8 in 2024-09-13T13:03:00.965 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:00 smithi052 ceph-mon[36682]: pgmap v71: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-13T13:03:00.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:00 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:00.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:00 smithi052 ceph-mon[29471]: from='client.24838 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:00.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:00 smithi052 ceph-mon[29471]: osdmap e97: 8 total, 8 up, 8 in 2024-09-13T13:03:00.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:00 smithi052 ceph-mon[29471]: pgmap v71: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-13T13:03:00.986 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:01.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:01.021 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:01.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:00 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:00 smithi167 ceph-mon[33178]: from='client.24838 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:00 smithi167 ceph-mon[33178]: osdmap e97: 8 total, 8 up, 8 in 2024-09-13T13:03:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:00 smithi167 ceph-mon[33178]: pgmap v71: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-13T13:03:01.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:01.147 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:01.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:01.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:01.159 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 39 16 68855 68839 110.252 83.625 0.0287105 0.00906425 2024-09-13T13:03:01.249 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:01.286 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T13:03:01.286 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:03:01.286 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:03:01.292 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:01.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:01.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:01.393 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:01.398 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:01.461 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:01.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:01.484 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:03:01.484 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:03:01.519 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:03:01.519 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:03:01.556 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:03:01.556 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:03:01.563 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:01.595 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:03:01.596 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:03:01.629 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:03:01.629 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:03:01.634 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:01.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:01.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:01.665 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:03:01.666 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:03:01.701 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:03:01.702 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:03:01.736 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:01.740 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:03:01.740 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:03:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:01.765 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:01.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:01.815 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:01 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/823094860' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:01.815 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:01 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/823094860' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:01.866 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:01.952 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:01.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:02.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:02.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:01 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/823094860' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:02.081 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:02.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:02.159 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:03:02.159219+0000 min lat: 0.00146492 max lat: 0.231867 avg lat: 0.00914721 2024-09-13T13:03:02.159 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:03:02.159 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 40 16 69985 69969 109.262 70.625 0.0070133 0.00914721 2024-09-13T13:03:02.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:02.211 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:02.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:02.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:02.313 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:02.354 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:02.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:02.456 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:02.477 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:02.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:02.579 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:02.618 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:02.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:02.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:02.720 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:02.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:02.783 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:02.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:02.884 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:02.912 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:02.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:02.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:03.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:03.016 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:02 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:03:03.016 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:02 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:03:03.016 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:02 smithi052 ceph-mon[36682]: osdmap e98: 8 total, 8 up, 8 in 2024-09-13T13:03:03.016 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:02 smithi052 ceph-mon[36682]: pgmap v73: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 790 op/s 2024-09-13T13:03:03.016 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:02 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:03.016 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:02 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:03:03.017 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:02 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:03:03.017 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:02 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:03:03.017 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:02 smithi052 ceph-mon[29471]: osdmap e98: 8 total, 8 up, 8 in 2024-09-13T13:03:03.017 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:02 smithi052 ceph-mon[29471]: pgmap v73: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 790 op/s 2024-09-13T13:03:03.017 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:02 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:03.017 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:02 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:03:03.039 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:03.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:03.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:02 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:03:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:02 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:03:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:02 smithi167 ceph-mon[33178]: osdmap e98: 8 total, 8 up, 8 in 2024-09-13T13:03:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:02 smithi167 ceph-mon[33178]: pgmap v73: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 790 op/s 2024-09-13T13:03:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:02 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:02 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:03:03.141 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:03.159 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 41 16 71236 71220 108.504 78.1875 0.00957878 0.00921135 2024-09-13T13:03:03.169 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:03.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:03.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:03.271 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:03.316 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:03.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:03.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:03.418 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:03.447 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:03.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:03.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:03.548 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:03.576 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:03.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:03.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:03.705 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:03.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:03.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:03.807 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:03.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:03:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:03:03.867 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:03:03.871 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:03:03.871 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:03:03.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:03.872 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:03.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:03.873 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:03.973 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:04.028 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:04.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:04.130 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:04.159 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 42 16 72409 72393 107.666 73.3125 0.0216003 0.0092808 2024-09-13T13:03:04.174 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:04.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:04.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:04.222 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:03 smithi052 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T13:03:04.222 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:03 smithi052 systemd[1]: Stopped Ceph alertmanager.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:03:04.222 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:03 smithi052 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.265s CPU time. 2024-09-13T13:03:04.222 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:03 smithi052 systemd[1]: Starting Ceph alertmanager.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:03:04.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:04.304 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:04.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:04.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:04.406 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:04.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:04.447 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:04.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:04.548 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:04.574 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:04.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:04.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:04.676 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:04.714 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:04.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:04.816 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:04.859 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:04.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:04.961 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:04.983 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:04.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:05.084 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:05.124 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:05.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:05.159 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 43 16 73627 73611 106.933 76.125 0.0130015 0.0093457 2024-09-13T13:03:05.174 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T13:03:05.226 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:04 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:04 smithi052 ceph-mon[29471]: pgmap v74: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-13T13:03:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:04 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:03:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:04 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:03:05.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:04 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:05.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:04 smithi052 ceph-mon[36682]: pgmap v74: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-13T13:03:05.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:04 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:03:05.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:04 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:03:05.252 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:05.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:05.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:05.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:04 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:05.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:04 smithi167 ceph-mon[33178]: pgmap v74: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-13T13:03:05.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:04 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:03:05.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:04 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:03:05.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:05.380 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:05.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:05.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:05.482 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:05.547 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:03:05.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:05.556 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:05.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:05.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:05.685 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:05.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:05.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:05.786 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:05.839 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:05.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:05.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:05.896 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:05 smithi052 podman[76829]: 2024-09-13 13:03:05.439077728 +0000 UTC m=+0.040769989 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-13T13:03:05.896 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:05 smithi052 podman[76829]: 2024-09-13 13:03:05.554505011 +0000 UTC m=+0.156197263 volume create 3b90f463cad43caaec15b3f968c518f912358b1b734fbd8b3ece0993a5080383 2024-09-13T13:03:05.896 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:05 smithi052 podman[76829]: 2024-09-13 13:03:05.638492782 +0000 UTC m=+0.240185036 container create 5dfe3cbc32b00d0b7eea5249089d69a696a2b933add18233e68f9c5fa73d84a6 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T13:03:05.941 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:05.968 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:05.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:06.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:06.096 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:06.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:06.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:06.160 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 44 16 74726 74710 106.061 68.6875 0.0112614 0.00942283 2024-09-13T13:03:06.198 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:06.228 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:06.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:06.254 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:03:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:03:06.254 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:06 smithi052 podman[76829]: 2024-09-13 13:03:06.227570573 +0000 UTC m=+0.829262826 container init 5dfe3cbc32b00d0b7eea5249089d69a696a2b933add18233e68f9c5fa73d84a6 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T13:03:06.255 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:06 smithi052 podman[76829]: 2024-09-13 13:03:06.234508877 +0000 UTC m=+0.836201131 container start 5dfe3cbc32b00d0b7eea5249089d69a696a2b933add18233e68f9c5fa73d84a6 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T13:03:06.255 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:06 smithi052 ceph-mon[29471]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T13:03:06.255 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:06 smithi052 ceph-mon[36682]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T13:03:06.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:06 smithi167 ceph-mon[33178]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T13:03:06.331 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:06.356 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:06.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:06.392 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:06.458 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:06.472 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:06.537 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:06.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:06.552 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:03:06.553 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[76921]: ts=2024-09-13T13:03:06.253Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-13T13:03:06.553 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[76921]: ts=2024-09-13T13:03:06.253Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-13T13:03:06.553 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[76921]: ts=2024-09-13T13:03:06.255Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.52 port=9094 2024-09-13T13:03:06.553 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[76921]: ts=2024-09-13T13:03:06.258Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T13:03:06.553 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[76921]: ts=2024-09-13T13:03:06.304Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T13:03:06.553 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[76921]: ts=2024-09-13T13:03:06.304Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T13:03:06.553 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[76921]: ts=2024-09-13T13:03:06.307Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-13T13:03:06.553 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[76921]: ts=2024-09-13T13:03:06.308Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T13:03:06.553 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:06 smithi052 bash[76829]: 5dfe3cbc32b00d0b7eea5249089d69a696a2b933add18233e68f9c5fa73d84a6 2024-09-13T13:03:06.554 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:06 smithi052 systemd[1]: Started Ceph alertmanager.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:03:06.561 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:03:06.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:06.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:06.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:06.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:06.767 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:06.809 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:06.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:06.911 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:06.935 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:06.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:06.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:07.036 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:07.085 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:07.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:07.161 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 45 16 75748 75732 105.124 63.875 0.0118222 0.00950748 2024-09-13T13:03:07.187 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:07.231 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:07.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:07.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:07.333 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:07.362 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:07.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:07.464 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:07.503 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:07.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:07.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:07.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:07.622 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:07 smithi052 ceph-mon[36682]: pgmap v76: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 830 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-13T13:03:07.622 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:07 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:07.622 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:07 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:07.622 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:07 smithi052 ceph-mon[36682]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T13:03:07.624 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:07 smithi052 ceph-mon[29471]: pgmap v76: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 830 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-13T13:03:07.624 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:07 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:07.624 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:07 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:07.624 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:07 smithi052 ceph-mon[29471]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T13:03:07.647 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:07.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:07.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:07.680 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-09-13T13:03:05.898461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4646574619,"num_objects":71091,"num_object_clones":0,"num_object_copies":213273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71091,"num_whiteouts":0,"num_read":963,"num_read_kb":820,"num_write":71441,"num_write_kb":4538761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":590588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71444,"ondisk_log_size":71444,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13937472,"kb_used_data":13503324,"kb_used_omap":12,"kb_used_meta":433971,"kb_avail":736023744,"statfs":{"total":767960285184,"available":753688313856,"internally_reserved":0,"allocated":13827403776,"data_stored":13822677889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":444386866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,5,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":411631616,"num_objects":6281,"num_object_clones":0,"num_object_copies":18843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6281,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6281,"num_write_kb":401984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.162438"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748369+0000","last_change":"2024-09-13T13:02:38.651382+0000","last_active":"2024-09-13T13:03:01.748369+0000","last_peered":"2024-09-13T13:03:01.748369+0000","last_clean":"2024-09-13T13:03:01.748369+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:03:01.748369+0000","last_undegraded":"2024-09-13T13:03:01.748369+0000","last_fullsized":"2024-09-13T13:03:01.748369+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:38.651314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:38.651314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:56.124890+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":100,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.740312+0000","last_change":"2024-09-13T13:01:04.194784+0000","last_active":"2024-09-13T13:03:01.740312+0000","last_peered":"2024-09-13T13:03:01.740312+0000","last_clean":"2024-09-13T13:03:01.740312+0000","last_became_active":"2024-09-13T13:01:04.194444+0000","last_became_peered":"2024-09-13T13:01:04.194444+0000","last_unstale":"2024-09-13T13:03:01.740312+0000","last_undegraded":"2024-09-13T13:03:01.740312+0000","last_fullsized":"2024-09-13T13:03:01.740312+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751291+0000","last_change":"2024-09-13T13:02:30.248974+0000","last_active":"2024-09-13T13:03:01.751291+0000","last_peered":"2024-09-13T13:03:01.751291+0000","last_clean":"2024-09-13T13:03:01.751291+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:03:01.751291+0000","last_undegraded":"2024-09-13T13:03:01.751291+0000","last_fullsized":"2024-09-13T13:03:01.751291+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:30.248905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:30.248905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:58.193471+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":76,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.740489+0000","last_change":"2024-09-13T13:01:08.214110+0000","last_active":"2024-09-13T13:03:01.740489+0000","last_peered":"2024-09-13T13:03:01.740489+0000","last_clean":"2024-09-13T13:03:01.740489+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:03:01.740489+0000","last_undegraded":"2024-09-13T13:03:01.740489+0000","last_fullsized":"2024-09-13T13:03:01.740489+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:54.034112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.749796+0000","last_change":"2024-09-13T13:01:08.534893+0000","last_active":"2024-09-13T13:03:01.749796+0000","last_peered":"2024-09-13T13:03:01.749796+0000","last_clean":"2024-09-13T13:03:01.749796+0000","last_became_active":"2024-09-13T13:01:08.534558+0000","last_became_peered":"2024-09-13T13:01:08.534558+0000","last_unstale":"2024-09-13T13:03:01.749796+0000","last_undegraded":"2024-09-13T13:03:01.749796+0000","last_fullsized":"2024-09-13T13:03:01.749796+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:18.285353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.749468+0000","last_change":"2024-09-13T13:02:41.233440+0000","last_active":"2024-09-13T13:03:01.749468+0000","last_peered":"2024-09-13T13:03:01.749468+0000","last_clean":"2024-09-13T13:03:01.749468+0000","last_became_active":"2024-09-13T13:01:02.183415+0000","last_became_peered":"2024-09-13T13:01:02.183415+0000","last_unstale":"2024-09-13T13:03:01.749468+0000","last_undegraded":"2024-09-13T13:03:01.749468+0000","last_fullsized":"2024-09-13T13:03:01.749468+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.233259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.233259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:46.761922+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":116,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747031+0000","last_change":"2024-09-13T13:02:44.263453+0000","last_active":"2024-09-13T13:03:01.747031+0000","last_peered":"2024-09-13T13:03:01.747031+0000","last_clean":"2024-09-13T13:03:01.747031+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:03:01.747031+0000","last_undegraded":"2024-09-13T13:03:01.747031+0000","last_fullsized":"2024-09-13T13:03:01.747031+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:02:44.263371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:44.263371+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:47.888219+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.739653+0000","last_change":"2024-09-13T13:02:33.021732+0000","last_active":"2024-09-13T13:03:01.739653+0000","last_peered":"2024-09-13T13:03:01.739653+0000","last_clean":"2024-09-13T13:03:01.739653+0000","last_became_active":"2024-09-13T13:01:06.207429+0000","last_became_peered":"2024-09-13T13:01:06.207429+0000","last_unstale":"2024-09-13T13:03:01.739653+0000","last_undegraded":"2024-09-13T13:03:01.739653+0000","last_fullsized":"2024-09-13T13:03:01.739653+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.021662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.021662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:50.826118+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":93,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748201+0000","last_change":"2024-09-13T13:01:02.184512+0000","last_active":"2024-09-13T13:03:01.748201+0000","last_peered":"2024-09-13T13:03:01.748201+0000","last_clean":"2024-09-13T13:03:01.748201+0000","last_became_active":"2024-09-13T13:01:02.184241+0000","last_became_peered":"2024-09-13T13:01:02.184241+0000","last_unstale":"2024-09-13T13:03:01.748201+0000","last_undegraded":"2024-09-13T13:03:01.748201+0000","last_fullsized":"2024-09-13T13:03:01.748201+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:21.585957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":100,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.740758+0000","last_change":"2024-09-13T13:01:04.189504+0000","last_active":"2024-09-13T13:03:01.740758+0000","last_peered":"2024-09-13T13:03:01.740758+0000","last_clean":"2024-09-13T13:03:01.740758+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:03:01.740758+0000","last_undegraded":"2024-09-13T13:03:01.740758+0000","last_fullsized":"2024-09-13T13:03:01.740758+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:50.993391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747729+0000","last_change":"2024-09-13T13:02:42.364052+0000","last_active":"2024-09-13T13:03:01.747729+0000","last_peered":"2024-09-13T13:03:01.747729+0000","last_clean":"2024-09-13T13:03:01.747729+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:03:01.747729+0000","last_undegraded":"2024-09-13T13:03:01.747729+0000","last_fullsized":"2024-09-13T13:03:01.747729+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:42.363956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.363956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:52.370671+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":83,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.742382+0000","last_change":"2024-09-13T13:02:39.607712+0000","last_active":"2024-09-13T13:03:01.742382+0000","last_peered":"2024-09-13T13:03:01.742382+0000","last_clean":"2024-09-13T13:03:01.742382+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:03:01.742382+0000","last_undegraded":"2024-09-13T13:03:01.742382+0000","last_fullsized":"2024-09-13T13:03:01.742382+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.607646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.607646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:41.310813+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748271+0000","last_change":"2024-09-13T13:02:07.530533+0000","last_active":"2024-09-13T13:03:01.748271+0000","last_peered":"2024-09-13T13:03:01.748271+0000","last_clean":"2024-09-13T13:03:01.748271+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:03:01.748271+0000","last_undegraded":"2024-09-13T13:03:01.748271+0000","last_fullsized":"2024-09-13T13:03:01.748271+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:07.530485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:07.530485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:29.298503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":102,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.741383+0000","last_change":"2024-09-13T13:02:39.998234+0000","last_active":"2024-09-13T13:03:01.741383+0000","last_peered":"2024-09-13T13:03:01.741383+0000","last_clean":"2024-09-13T13:03:01.741383+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:03:01.741383+0000","last_undegraded":"2024-09-13T13:03:01.741383+0000","last_fullsized":"2024-09-13T13:03:01.741383+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:02:39.998182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.998182+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:45.305542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.745627+0000","last_change":"2024-09-13T13:01:06.210845+0000","last_active":"2024-09-13T13:03:01.745627+0000","last_peered":"2024-09-13T13:03:01.745627+0000","last_clean":"2024-09-13T13:03:01.745627+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:03:01.745627+0000","last_undegraded":"2024-09-13T13:03:01.745627+0000","last_fullsized":"2024-09-13T13:03:01.745627+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747384+0000","last_change":"2024-09-13T13:01:08.213608+0000","last_active":"2024-09-13T13:03:01.747384+0000","last_peered":"2024-09-13T13:03:01.747384+0000","last_clean":"2024-09-13T13:03:01.747384+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:03:01.747384+0000","last_undegraded":"2024-09-13T13:03:01.747384+0000","last_fullsized":"2024-09-13T13:03:01.747384+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:54.473917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":77,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.741273+0000","last_change":"2024-09-13T13:01:08.533336+0000","last_active":"2024-09-13T13:03:01.741273+0000","last_peered":"2024-09-13T13:03:01.741273+0000","last_clean":"2024-09-13T13:03:01.741273+0000","last_became_active":"2024-09-13T13:01:08.533051+0000","last_became_peered":"2024-09-13T13:01:08.533051+0000","last_unstale":"2024-09-13T13:03:01.741273+0000","last_undegraded":"2024-09-13T13:03:01.741273+0000","last_fullsized":"2024-09-13T13:03:01.741273+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.743988+0000","last_change":"2024-09-13T13:02:32.590355+0000","last_active":"2024-09-13T13:03:01.743988+0000","last_peered":"2024-09-13T13:03:01.743988+0000","last_clean":"2024-09-13T13:03:01.743988+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:03:01.743988+0000","last_undegraded":"2024-09-13T13:03:01.743988+0000","last_fullsized":"2024-09-13T13:03:01.743988+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:32.590282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.590282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:46.543654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":117,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.744961+0000","last_change":"2024-09-13T13:02:06.468336+0000","last_active":"2024-09-13T13:03:01.744961+0000","last_peered":"2024-09-13T13:03:01.744961+0000","last_clean":"2024-09-13T13:03:01.744961+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:03:01.744961+0000","last_undegraded":"2024-09-13T13:03:01.744961+0000","last_fullsized":"2024-09-13T13:03:01.744961+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:02:06.468276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:06.468276+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:17.206748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751800+0000","last_change":"2024-09-13T13:02:45.202865+0000","last_active":"2024-09-13T13:03:01.751800+0000","last_peered":"2024-09-13T13:03:01.751800+0000","last_clean":"2024-09-13T13:03:01.751800+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:03:01.751800+0000","last_undegraded":"2024-09-13T13:03:01.751800+0000","last_fullsized":"2024-09-13T13:03:01.751800+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.202802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.202802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:04:14.916146+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747483+0000","last_change":"2024-09-13T13:01:08.533133+0000","last_active":"2024-09-13T13:03:01.747483+0000","last_peered":"2024-09-13T13:03:01.747483+0000","last_clean":"2024-09-13T13:03:01.747483+0000","last_became_active":"2024-09-13T13:01:08.532753+0000","last_became_peered":"2024-09-13T13:01:08.532753+0000","last_unstale":"2024-09-13T13:03:01.747483+0000","last_undegraded":"2024-09-13T13:03:01.747483+0000","last_fullsized":"2024-09-13T13:03:01.747483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.740894+0000","last_change":"2024-09-13T13:01:02.173465+0000","last_active":"2024-09-13T13:03:01.740894+0000","last_peered":"2024-09-13T13:03:01.740894+0000","last_clean":"2024-09-13T13:03:01.740894+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:03:01.740894+0000","last_undegraded":"2024-09-13T13:03:01.740894+0000","last_fullsized":"2024-09-13T13:03:01.740894+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:54.997878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":106,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747031+0000","last_change":"2024-09-13T13:01:04.195126+0000","last_active":"2024-09-13T13:03:01.747031+0000","last_peered":"2024-09-13T13:03:01.747031+0000","last_clean":"2024-09-13T13:03:01.747031+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:03:01.747031+0000","last_undegraded":"2024-09-13T13:03:01.747031+0000","last_fullsized":"2024-09-13T13:03:01.747031+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":80,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.741165+0000","last_change":"2024-09-13T13:01:06.207164+0000","last_active":"2024-09-13T13:03:01.741165+0000","last_peered":"2024-09-13T13:03:01.741165+0000","last_clean":"2024-09-13T13:03:01.741165+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:03:01.741165+0000","last_undegraded":"2024-09-13T13:03:01.741165+0000","last_fullsized":"2024-09-13T13:03:01.741165+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":76,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.745682+0000","last_change":"2024-09-13T13:01:08.210300+0000","last_active":"2024-09-13T13:03:01.745682+0000","last_peered":"2024-09-13T13:03:01.745682+0000","last_clean":"2024-09-13T13:03:01.745682+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:03:01.745682+0000","last_undegraded":"2024-09-13T13:03:01.745682+0000","last_fullsized":"2024-09-13T13:03:01.745682+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:55.939397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":89,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751424+0000","last_change":"2024-09-13T13:01:02.180198+0000","last_active":"2024-09-13T13:03:01.751424+0000","last_peered":"2024-09-13T13:03:01.751424+0000","last_clean":"2024-09-13T13:03:01.751424+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:03:01.751424+0000","last_undegraded":"2024-09-13T13:03:01.751424+0000","last_fullsized":"2024-09-13T13:03:01.751424+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:11.185744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":106,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751431+0000","last_change":"2024-09-13T13:02:34.593511+0000","last_active":"2024-09-13T13:03:01.751431+0000","last_peered":"2024-09-13T13:03:01.751431+0000","last_clean":"2024-09-13T13:03:01.751431+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:03:01.751431+0000","last_undegraded":"2024-09-13T13:03:01.751431+0000","last_fullsized":"2024-09-13T13:03:01.751431+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:34.593431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:34.593431+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:49.042524+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":123,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.752419+0000","last_change":"2024-09-13T13:02:46.604729+0000","last_active":"2024-09-13T13:03:01.752419+0000","last_peered":"2024-09-13T13:03:01.752419+0000","last_clean":"2024-09-13T13:03:01.752419+0000","last_became_active":"2024-09-13T13:01:06.211034+0000","last_became_peered":"2024-09-13T13:01:06.211034+0000","last_unstale":"2024-09-13T13:03:01.752419+0000","last_undegraded":"2024-09-13T13:03:01.752419+0000","last_fullsized":"2024-09-13T13:03:01.752419+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:46.604669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.604669+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:49.762907+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"98'4443","reported_seq":4482,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.870603+0000","last_change":"2024-09-13T13:02:12.377991+0000","last_active":"2024-09-13T13:03:02.870603+0000","last_peered":"2024-09-13T13:03:02.870603+0000","last_clean":"2024-09-13T13:03:02.870603+0000","last_became_active":"2024-09-13T13:02:12.377552+0000","last_became_peered":"2024-09-13T13:02:12.377552+0000","last_unstale":"2024-09-13T13:03:02.870603+0000","last_undegraded":"2024-09-13T13:03:02.870603+0000","last_fullsized":"2024-09-13T13:03:02.870603+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4443,"log_dups_size":0,"ondisk_log_size":4443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:26.751377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291176448,"num_objects":4443,"num_object_clones":0,"num_object_copies":13329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":284352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":117,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751558+0000","last_change":"2024-09-13T13:02:05.179720+0000","last_active":"2024-09-13T13:03:01.751558+0000","last_peered":"2024-09-13T13:03:01.751558+0000","last_clean":"2024-09-13T13:03:01.751558+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:03:01.751558+0000","last_undegraded":"2024-09-13T13:03:01.751558+0000","last_fullsized":"2024-09-13T13:03:01.751558+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:02:05.179667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:05.179667+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:18.126160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748827+0000","last_change":"2024-09-13T13:02:08.195640+0000","last_active":"2024-09-13T13:03:01.748827+0000","last_peered":"2024-09-13T13:03:01.748827+0000","last_clean":"2024-09-13T13:03:01.748827+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:03:01.748827+0000","last_undegraded":"2024-09-13T13:03:01.748827+0000","last_fullsized":"2024-09-13T13:03:01.748827+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:08.195574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:08.195574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:22.347842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":121,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.741520+0000","last_change":"2024-09-13T13:02:32.013587+0000","last_active":"2024-09-13T13:03:01.741520+0000","last_peered":"2024-09-13T13:03:01.741520+0000","last_clean":"2024-09-13T13:03:01.741520+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:03:01.741520+0000","last_undegraded":"2024-09-13T13:03:01.741520+0000","last_fullsized":"2024-09-13T13:03:01.741520+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:32.013505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.013505+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:47.148595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747754+0000","last_change":"2024-09-13T13:01:08.216021+0000","last_active":"2024-09-13T13:03:01.747754+0000","last_peered":"2024-09-13T13:03:01.747754+0000","last_clean":"2024-09-13T13:03:01.747754+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:03:01.747754+0000","last_undegraded":"2024-09-13T13:03:01.747754+0000","last_fullsized":"2024-09-13T13:03:01.747754+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:13.477055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"98'4374","reported_seq":4412,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.887096+0000","last_change":"2024-09-13T13:02:12.370004+0000","last_active":"2024-09-13T13:03:02.887096+0000","last_peered":"2024-09-13T13:03:02.887096+0000","last_clean":"2024-09-13T13:03:02.887096+0000","last_became_active":"2024-09-13T13:02:12.369723+0000","last_became_peered":"2024-09-13T13:02:12.369723+0000","last_unstale":"2024-09-13T13:03:02.887096+0000","last_undegraded":"2024-09-13T13:03:02.887096+0000","last_fullsized":"2024-09-13T13:03:02.887096+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4374,"log_dups_size":0,"ondisk_log_size":4374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:21.693053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286654464,"num_objects":4374,"num_object_clones":0,"num_object_copies":13122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4374,"num_write_kb":279936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":115,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748592+0000","last_change":"2024-09-13T13:01:04.209310+0000","last_active":"2024-09-13T13:03:01.748592+0000","last_peered":"2024-09-13T13:03:01.748592+0000","last_clean":"2024-09-13T13:03:01.748592+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:03:01.748592+0000","last_undegraded":"2024-09-13T13:03:01.748592+0000","last_fullsized":"2024-09-13T13:03:01.748592+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:19.301960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":93,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747807+0000","last_change":"2024-09-13T13:01:02.181001+0000","last_active":"2024-09-13T13:03:01.747807+0000","last_peered":"2024-09-13T13:03:01.747807+0000","last_clean":"2024-09-13T13:03:01.747807+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:03:01.747807+0000","last_undegraded":"2024-09-13T13:03:01.747807+0000","last_fullsized":"2024-09-13T13:03:01.747807+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:14.246599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.740467+0000","last_change":"2024-09-13T13:02:36.005760+0000","last_active":"2024-09-13T13:03:01.740467+0000","last_peered":"2024-09-13T13:03:01.740467+0000","last_clean":"2024-09-13T13:03:01.740467+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:03:01.740467+0000","last_undegraded":"2024-09-13T13:03:01.740467+0000","last_fullsized":"2024-09-13T13:03:01.740467+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.005697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.005697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:04:04.684149+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":76,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.740678+0000","last_change":"2024-09-13T13:01:08.204782+0000","last_active":"2024-09-13T13:03:01.740678+0000","last_peered":"2024-09-13T13:03:01.740678+0000","last_clean":"2024-09-13T13:03:01.740678+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:03:01.740678+0000","last_undegraded":"2024-09-13T13:03:01.740678+0000","last_fullsized":"2024-09-13T13:03:01.740678+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:24.968620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"98'4416","reported_seq":4455,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.901855+0000","last_change":"2024-09-13T13:02:12.373817+0000","last_active":"2024-09-13T13:03:02.901855+0000","last_peered":"2024-09-13T13:03:02.901855+0000","last_clean":"2024-09-13T13:03:02.901855+0000","last_became_active":"2024-09-13T13:02:12.373492+0000","last_became_peered":"2024-09-13T13:02:12.373492+0000","last_unstale":"2024-09-13T13:03:02.901855+0000","last_undegraded":"2024-09-13T13:03:02.901855+0000","last_fullsized":"2024-09-13T13:03:02.901855+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4416,"log_dups_size":0,"ondisk_log_size":4416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:34.091045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":4416,"num_object_clones":0,"num_object_copies":13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4416,"num_write_kb":282624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":113,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.740495+0000","last_change":"2024-09-13T13:02:43.947701+0000","last_active":"2024-09-13T13:03:01.740495+0000","last_peered":"2024-09-13T13:03:01.740495+0000","last_clean":"2024-09-13T13:03:01.740495+0000","last_became_active":"2024-09-13T13:01:04.186768+0000","last_became_peered":"2024-09-13T13:01:04.186768+0000","last_unstale":"2024-09-13T13:03:01.740495+0000","last_undegraded":"2024-09-13T13:03:01.740495+0000","last_fullsized":"2024-09-13T13:03:01.740495+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:02:43.947615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.947615+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:54.423849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751700+0000","last_change":"2024-09-13T13:01:02.187792+0000","last_active":"2024-09-13T13:03:01.751700+0000","last_peered":"2024-09-13T13:03:01.751700+0000","last_clean":"2024-09-13T13:03:01.751700+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:03:01.751700+0000","last_undegraded":"2024-09-13T13:03:01.751700+0000","last_fullsized":"2024-09-13T13:03:01.751700+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:17.474572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":112,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748368+0000","last_change":"2024-09-13T13:01:06.211211+0000","last_active":"2024-09-13T13:03:01.748368+0000","last_peered":"2024-09-13T13:03:01.748368+0000","last_clean":"2024-09-13T13:03:01.748368+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:03:01.748368+0000","last_undegraded":"2024-09-13T13:03:01.748368+0000","last_fullsized":"2024-09-13T13:03:01.748368+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:50.215869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.745653+0000","last_change":"2024-09-13T13:01:08.533116+0000","last_active":"2024-09-13T13:03:01.745653+0000","last_peered":"2024-09-13T13:03:01.745653+0000","last_clean":"2024-09-13T13:03:01.745653+0000","last_became_active":"2024-09-13T13:01:08.532859+0000","last_became_peered":"2024-09-13T13:01:08.532859+0000","last_unstale":"2024-09-13T13:03:01.745653+0000","last_undegraded":"2024-09-13T13:03:01.745653+0000","last_fullsized":"2024-09-13T13:03:01.745653+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:51.959052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"98'4442","reported_seq":4481,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.839242+0000","last_change":"2024-09-13T13:02:12.376266+0000","last_active":"2024-09-13T13:03:02.839242+0000","last_peered":"2024-09-13T13:03:02.839242+0000","last_clean":"2024-09-13T13:03:02.839242+0000","last_became_active":"2024-09-13T13:02:12.375925+0000","last_became_peered":"2024-09-13T13:02:12.375925+0000","last_unstale":"2024-09-13T13:03:02.839242+0000","last_undegraded":"2024-09-13T13:03:02.839242+0000","last_fullsized":"2024-09-13T13:03:02.839242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:38.173999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291110912,"num_objects":4442,"num_object_clones":0,"num_object_copies":13326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4442,"num_write_kb":284288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":99,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.759217+0000","last_change":"2024-09-13T13:01:04.199011+0000","last_active":"2024-09-13T13:03:01.759217+0000","last_peered":"2024-09-13T13:03:01.759217+0000","last_clean":"2024-09-13T13:03:01.759217+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:03:01.759217+0000","last_undegraded":"2024-09-13T13:03:01.759217+0000","last_fullsized":"2024-09-13T13:03:01.759217+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:20.233718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748287+0000","last_change":"2024-09-13T13:01:02.181597+0000","last_active":"2024-09-13T13:03:01.748287+0000","last_peered":"2024-09-13T13:03:01.748287+0000","last_clean":"2024-09-13T13:03:01.748287+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:03:01.748287+0000","last_undegraded":"2024-09-13T13:03:01.748287+0000","last_fullsized":"2024-09-13T13:03:01.748287+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:50.378099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":112,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.743504+0000","last_change":"2024-09-13T13:01:06.216352+0000","last_active":"2024-09-13T13:03:01.743504+0000","last_peered":"2024-09-13T13:03:01.743504+0000","last_clean":"2024-09-13T13:03:01.743504+0000","last_became_active":"2024-09-13T13:01:06.216043+0000","last_became_peered":"2024-09-13T13:01:06.216043+0000","last_unstale":"2024-09-13T13:03:01.743504+0000","last_undegraded":"2024-09-13T13:03:01.743504+0000","last_fullsized":"2024-09-13T13:03:01.743504+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:19.411651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":83,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.742587+0000","last_change":"2024-09-13T13:02:43.558487+0000","last_active":"2024-09-13T13:03:01.742587+0000","last_peered":"2024-09-13T13:03:01.742587+0000","last_clean":"2024-09-13T13:03:01.742587+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:03:01.742587+0000","last_undegraded":"2024-09-13T13:03:01.742587+0000","last_fullsized":"2024-09-13T13:03:01.742587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:43.558418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.558418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:56.409125+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"98'4334","reported_seq":4373,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.830924+0000","last_change":"2024-09-13T13:02:12.379536+0000","last_active":"2024-09-13T13:03:02.830924+0000","last_peered":"2024-09-13T13:03:02.830924+0000","last_clean":"2024-09-13T13:03:02.830924+0000","last_became_active":"2024-09-13T13:02:12.379226+0000","last_became_peered":"2024-09-13T13:02:12.379226+0000","last_unstale":"2024-09-13T13:03:02.830924+0000","last_undegraded":"2024-09-13T13:03:02.830924+0000","last_fullsized":"2024-09-13T13:03:02.830924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4334,"log_dups_size":0,"ondisk_log_size":4334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:36.051392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284033024,"num_objects":4334,"num_object_clones":0,"num_object_copies":13002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4334,"num_write_kb":277376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":127,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.743657+0000","last_change":"2024-09-13T13:02:29.532879+0000","last_active":"2024-09-13T13:03:01.743657+0000","last_peered":"2024-09-13T13:03:01.743657+0000","last_clean":"2024-09-13T13:03:01.743657+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:03:01.743657+0000","last_undegraded":"2024-09-13T13:03:01.743657+0000","last_fullsized":"2024-09-13T13:03:01.743657+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:29.532779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.532779+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:43.559043+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.752412+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:03:01.752412+0000","last_peered":"2024-09-13T13:03:01.752412+0000","last_clean":"2024-09-13T13:03:01.752412+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:03:01.752412+0000","last_undegraded":"2024-09-13T13:03:01.752412+0000","last_fullsized":"2024-09-13T13:03:01.752412+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:19.265543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751769+0000","last_change":"2024-09-13T13:02:45.563761+0000","last_active":"2024-09-13T13:03:01.751769+0000","last_peered":"2024-09-13T13:03:01.751769+0000","last_clean":"2024-09-13T13:03:01.751769+0000","last_became_active":"2024-09-13T13:01:06.213617+0000","last_became_peered":"2024-09-13T13:01:06.213617+0000","last_unstale":"2024-09-13T13:03:01.751769+0000","last_undegraded":"2024-09-13T13:03:01.751769+0000","last_fullsized":"2024-09-13T13:03:01.751769+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.563708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.563708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:56.915105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":84,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751203+0000","last_change":"2024-09-13T13:02:28.275837+0000","last_active":"2024-09-13T13:03:01.751203+0000","last_peered":"2024-09-13T13:03:01.751203+0000","last_clean":"2024-09-13T13:03:01.751203+0000","last_became_active":"2024-09-13T13:01:08.532145+0000","last_became_peered":"2024-09-13T13:01:08.532145+0000","last_unstale":"2024-09-13T13:03:01.751203+0000","last_undegraded":"2024-09-13T13:03:01.751203+0000","last_fullsized":"2024-09-13T13:03:01.751203+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:28.275768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.275768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:56.618217+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"98'4482","reported_seq":4521,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.875072+0000","last_change":"2024-09-13T13:02:12.379467+0000","last_active":"2024-09-13T13:03:02.875072+0000","last_peered":"2024-09-13T13:03:02.875072+0000","last_clean":"2024-09-13T13:03:02.875072+0000","last_became_active":"2024-09-13T13:02:12.379195+0000","last_became_peered":"2024-09-13T13:02:12.379195+0000","last_unstale":"2024-09-13T13:03:02.875072+0000","last_undegraded":"2024-09-13T13:03:02.875072+0000","last_fullsized":"2024-09-13T13:03:02.875072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:30.262637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293732352,"num_objects":4482,"num_object_clones":0,"num_object_copies":13446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4482,"num_write_kb":286848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":165,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.752329+0000","last_change":"2024-09-13T13:02:39.183839+0000","last_active":"2024-09-13T13:03:01.752329+0000","last_peered":"2024-09-13T13:03:01.752329+0000","last_clean":"2024-09-13T13:03:01.752329+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:03:01.752329+0000","last_undegraded":"2024-09-13T13:03:01.752329+0000","last_fullsized":"2024-09-13T13:03:01.752329+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:02:39.183740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.183740+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:59.142302+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.741940+0000","last_change":"2024-09-13T13:01:02.181566+0000","last_active":"2024-09-13T13:03:01.741940+0000","last_peered":"2024-09-13T13:03:01.741940+0000","last_clean":"2024-09-13T13:03:01.741940+0000","last_became_active":"2024-09-13T13:01:02.180760+0000","last_became_peered":"2024-09-13T13:01:02.180760+0000","last_unstale":"2024-09-13T13:03:01.741940+0000","last_undegraded":"2024-09-13T13:03:01.741940+0000","last_fullsized":"2024-09-13T13:03:01.741940+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:16.907740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747693+0000","last_change":"2024-09-13T13:01:05.199998+0000","last_active":"2024-09-13T13:03:01.747693+0000","last_peered":"2024-09-13T13:03:01.747693+0000","last_clean":"2024-09-13T13:03:01.747693+0000","last_became_active":"2024-09-13T13:01:02.185089+0000","last_became_peered":"2024-09-13T13:01:02.185089+0000","last_unstale":"2024-09-13T13:03:01.747693+0000","last_undegraded":"2024-09-13T13:03:01.747693+0000","last_fullsized":"2024-09-13T13:03:01.747693+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:17.571894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751429+0000","last_change":"2024-09-13T13:02:35.566666+0000","last_active":"2024-09-13T13:03:01.751429+0000","last_peered":"2024-09-13T13:03:01.751429+0000","last_clean":"2024-09-13T13:03:01.751429+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:03:01.751429+0000","last_undegraded":"2024-09-13T13:03:01.751429+0000","last_fullsized":"2024-09-13T13:03:01.751429+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:35.566602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:35.566602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:36.317282+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.749195+0000","last_change":"2024-09-13T13:01:08.203445+0000","last_active":"2024-09-13T13:03:01.749195+0000","last_peered":"2024-09-13T13:03:01.749195+0000","last_clean":"2024-09-13T13:03:01.749195+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:03:01.749195+0000","last_undegraded":"2024-09-13T13:03:01.749195+0000","last_fullsized":"2024-09-13T13:03:01.749195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:59.433605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"98'4461","reported_seq":4500,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.891197+0000","last_change":"2024-09-13T13:02:12.380398+0000","last_active":"2024-09-13T13:03:02.891197+0000","last_peered":"2024-09-13T13:03:02.891197+0000","last_clean":"2024-09-13T13:03:02.891197+0000","last_became_active":"2024-09-13T13:02:12.379331+0000","last_became_peered":"2024-09-13T13:02:12.379331+0000","last_unstale":"2024-09-13T13:03:02.891197+0000","last_undegraded":"2024-09-13T13:03:02.891197+0000","last_fullsized":"2024-09-13T13:03:02.891197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4461,"log_dups_size":0,"ondisk_log_size":4461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:18.368446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292356096,"num_objects":4461,"num_object_clones":0,"num_object_copies":13383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4461,"num_write_kb":285504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":111,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751655+0000","last_change":"2024-09-13T13:01:04.192923+0000","last_active":"2024-09-13T13:03:01.751655+0000","last_peered":"2024-09-13T13:03:01.751655+0000","last_clean":"2024-09-13T13:03:01.751655+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:03:01.751655+0000","last_undegraded":"2024-09-13T13:03:01.751655+0000","last_fullsized":"2024-09-13T13:03:01.751655+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:14.131977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":97,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.754442+0000","last_change":"2024-09-13T13:02:40.214244+0000","last_active":"2024-09-13T13:03:01.754442+0000","last_peered":"2024-09-13T13:03:01.754442+0000","last_clean":"2024-09-13T13:03:01.754442+0000","last_became_active":"2024-09-13T13:01:02.185263+0000","last_became_peered":"2024-09-13T13:01:02.185263+0000","last_unstale":"2024-09-13T13:03:01.754442+0000","last_undegraded":"2024-09-13T13:03:01.754442+0000","last_fullsized":"2024-09-13T13:03:01.754442+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:40.214163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:40.214163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:56.625186+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.742200+0000","last_change":"2024-09-13T13:01:05.202103+0000","last_active":"2024-09-13T13:03:01.742200+0000","last_peered":"2024-09-13T13:03:01.742200+0000","last_clean":"2024-09-13T13:03:01.742200+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:03:01.742200+0000","last_undegraded":"2024-09-13T13:03:01.742200+0000","last_fullsized":"2024-09-13T13:03:01.742200+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:14.436354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":80,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.756017+0000","last_change":"2024-09-13T13:01:06.205805+0000","last_active":"2024-09-13T13:03:01.756017+0000","last_peered":"2024-09-13T13:03:01.756017+0000","last_clean":"2024-09-13T13:03:01.756017+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:03:01.756017+0000","last_undegraded":"2024-09-13T13:03:01.756017+0000","last_fullsized":"2024-09-13T13:03:01.756017+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:10.423722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":83,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.746708+0000","last_change":"2024-09-13T13:02:33.994802+0000","last_active":"2024-09-13T13:03:01.746708+0000","last_peered":"2024-09-13T13:03:01.746708+0000","last_clean":"2024-09-13T13:03:01.746708+0000","last_became_active":"2024-09-13T13:01:08.532087+0000","last_became_peered":"2024-09-13T13:01:08.532087+0000","last_unstale":"2024-09-13T13:03:01.746708+0000","last_undegraded":"2024-09-13T13:03:01.746708+0000","last_fullsized":"2024-09-13T13:03:01.746708+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.994729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.994729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:36.510995+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"98'4493","reported_seq":4532,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.841093+0000","last_change":"2024-09-13T13:02:12.380059+0000","last_active":"2024-09-13T13:03:02.841093+0000","last_peered":"2024-09-13T13:03:02.841093+0000","last_clean":"2024-09-13T13:03:02.841093+0000","last_became_active":"2024-09-13T13:02:12.379779+0000","last_became_peered":"2024-09-13T13:02:12.379779+0000","last_unstale":"2024-09-13T13:03:02.841093+0000","last_undegraded":"2024-09-13T13:03:02.841093+0000","last_fullsized":"2024-09-13T13:03:02.841093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:26.549018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294453248,"num_objects":4493,"num_object_clones":0,"num_object_copies":13479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4493,"num_write_kb":287552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":101,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751517+0000","last_change":"2024-09-13T13:01:04.193042+0000","last_active":"2024-09-13T13:03:01.751517+0000","last_peered":"2024-09-13T13:03:01.751517+0000","last_clean":"2024-09-13T13:03:01.751517+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:03:01.751517+0000","last_undegraded":"2024-09-13T13:03:01.751517+0000","last_fullsized":"2024-09-13T13:03:01.751517+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:13.393738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.743409+0000","last_change":"2024-09-13T13:02:37.616313+0000","last_active":"2024-09-13T13:03:01.743409+0000","last_peered":"2024-09-13T13:03:01.743409+0000","last_clean":"2024-09-13T13:03:01.743409+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:03:01.743409+0000","last_undegraded":"2024-09-13T13:03:01.743409+0000","last_fullsized":"2024-09-13T13:03:01.743409+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:37.616224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.616224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:55.222698+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":212,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.588139+0000","last_change":"2024-09-13T13:01:05.200818+0000","last_active":"2024-09-13T13:03:02.588139+0000","last_peered":"2024-09-13T13:03:02.588139+0000","last_clean":"2024-09-13T13:03:02.588139+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:03:02.588139+0000","last_undegraded":"2024-09-13T13:03:02.588139+0000","last_fullsized":"2024-09-13T13:03:02.588139+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:24.013349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":115,"num_read_kb":110,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748678+0000","last_change":"2024-09-13T13:01:06.212898+0000","last_active":"2024-09-13T13:03:01.748678+0000","last_peered":"2024-09-13T13:03:01.748678+0000","last_clean":"2024-09-13T13:03:01.748678+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:03:01.748678+0000","last_undegraded":"2024-09-13T13:03:01.748678+0000","last_fullsized":"2024-09-13T13:03:01.748678+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:15.497741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":84,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751999+0000","last_change":"2024-09-13T13:02:36.222607+0000","last_active":"2024-09-13T13:03:01.751999+0000","last_peered":"2024-09-13T13:03:01.751999+0000","last_clean":"2024-09-13T13:03:01.751999+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:03:01.751999+0000","last_undegraded":"2024-09-13T13:03:01.751999+0000","last_fullsized":"2024-09-13T13:03:01.751999+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.222329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:59.124572+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"98'4471","reported_seq":4510,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.863623+0000","last_change":"2024-09-13T13:02:12.379556+0000","last_active":"2024-09-13T13:03:02.863623+0000","last_peered":"2024-09-13T13:03:02.863623+0000","last_clean":"2024-09-13T13:03:02.863623+0000","last_became_active":"2024-09-13T13:02:12.379255+0000","last_became_peered":"2024-09-13T13:02:12.379255+0000","last_unstale":"2024-09-13T13:03:02.863623+0000","last_undegraded":"2024-09-13T13:03:02.863623+0000","last_fullsized":"2024-09-13T13:03:02.863623+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4471,"log_dups_size":0,"ondisk_log_size":4471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:36.193910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293011456,"num_objects":4471,"num_object_clones":0,"num_object_copies":13413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4471,"num_write_kb":286144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":130,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.752281+0000","last_change":"2024-09-13T13:01:04.208034+0000","last_active":"2024-09-13T13:03:01.752281+0000","last_peered":"2024-09-13T13:03:01.752281+0000","last_clean":"2024-09-13T13:03:01.752281+0000","last_became_active":"2024-09-13T13:01:04.207653+0000","last_became_peered":"2024-09-13T13:01:04.207653+0000","last_unstale":"2024-09-13T13:03:01.752281+0000","last_undegraded":"2024-09-13T13:03:01.752281+0000","last_fullsized":"2024-09-13T13:03:01.752281+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:24.711240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.759725+0000","last_change":"2024-09-13T13:02:04.228072+0000","last_active":"2024-09-13T13:03:01.759725+0000","last_peered":"2024-09-13T13:03:01.759725+0000","last_clean":"2024-09-13T13:03:01.759725+0000","last_became_active":"2024-09-13T13:01:02.179698+0000","last_became_peered":"2024-09-13T13:01:02.179698+0000","last_unstale":"2024-09-13T13:03:01.759725+0000","last_undegraded":"2024-09-13T13:03:01.759725+0000","last_fullsized":"2024-09-13T13:03:01.759725+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:04.228002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:04.228002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:05.452908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":125,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.745511+0000","last_change":"2024-09-13T13:00:53.927016+0000","last_active":"2024-09-13T13:03:01.745511+0000","last_peered":"2024-09-13T13:03:01.745511+0000","last_clean":"2024-09-13T13:03:01.745511+0000","last_became_active":"2024-09-13T13:00:53.922110+0000","last_became_peered":"2024-09-13T13:00:53.922110+0000","last_unstale":"2024-09-13T13:03:01.745511+0000","last_undegraded":"2024-09-13T13:03:01.745511+0000","last_fullsized":"2024-09-13T13:03:01.745511+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-13T13:00:46.618619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:00:46.618619+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:09.010941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.746110+0000","last_change":"2024-09-13T13:01:06.207054+0000","last_active":"2024-09-13T13:03:01.746110+0000","last_peered":"2024-09-13T13:03:01.746110+0000","last_clean":"2024-09-13T13:03:01.746110+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:03:01.746110+0000","last_undegraded":"2024-09-13T13:03:01.746110+0000","last_fullsized":"2024-09-13T13:03:01.746110+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:18.081142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":83,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748532+0000","last_change":"2024-09-13T13:02:39.434636+0000","last_active":"2024-09-13T13:03:01.748532+0000","last_peered":"2024-09-13T13:03:01.748532+0000","last_clean":"2024-09-13T13:03:01.748532+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:03:01.748532+0000","last_undegraded":"2024-09-13T13:03:01.748532+0000","last_fullsized":"2024-09-13T13:03:01.748532+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.434575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.434575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:57.673272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"98'4445","reported_seq":4484,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.801906+0000","last_change":"2024-09-13T13:02:12.377904+0000","last_active":"2024-09-13T13:03:02.801906+0000","last_peered":"2024-09-13T13:03:02.801906+0000","last_clean":"2024-09-13T13:03:02.801906+0000","last_became_active":"2024-09-13T13:02:12.377581+0000","last_became_peered":"2024-09-13T13:02:12.377581+0000","last_unstale":"2024-09-13T13:03:02.801906+0000","last_undegraded":"2024-09-13T13:03:02.801906+0000","last_fullsized":"2024-09-13T13:03:02.801906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4445,"log_dups_size":0,"ondisk_log_size":4445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:26.803962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291307520,"num_objects":4445,"num_object_clones":0,"num_object_copies":13335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4445,"num_write_kb":284480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":103,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.741577+0000","last_change":"2024-09-13T13:01:04.196954+0000","last_active":"2024-09-13T13:03:01.741577+0000","last_peered":"2024-09-13T13:03:01.741577+0000","last_clean":"2024-09-13T13:03:01.741577+0000","last_became_active":"2024-09-13T13:01:04.196363+0000","last_became_peered":"2024-09-13T13:01:04.196363+0000","last_unstale":"2024-09-13T13:03:01.741577+0000","last_undegraded":"2024-09-13T13:03:01.741577+0000","last_fullsized":"2024-09-13T13:03:01.741577+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:59.061632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":96,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.740779+0000","last_change":"2024-09-13T13:02:45.015634+0000","last_active":"2024-09-13T13:03:01.740779+0000","last_peered":"2024-09-13T13:03:01.740779+0000","last_clean":"2024-09-13T13:03:01.740779+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:03:01.740779+0000","last_undegraded":"2024-09-13T13:03:01.740779+0000","last_fullsized":"2024-09-13T13:03:01.740779+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.015577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.015577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:49.187000+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748439+0000","last_change":"2024-09-13T13:01:06.209895+0000","last_active":"2024-09-13T13:03:01.748439+0000","last_peered":"2024-09-13T13:03:01.748439+0000","last_clean":"2024-09-13T13:03:01.748439+0000","last_became_active":"2024-09-13T13:01:06.206590+0000","last_became_peered":"2024-09-13T13:01:06.206590+0000","last_unstale":"2024-09-13T13:03:01.748439+0000","last_undegraded":"2024-09-13T13:03:01.748439+0000","last_fullsized":"2024-09-13T13:03:01.748439+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:23.590626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":83,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.761078+0000","last_change":"2024-09-13T13:02:39.213534+0000","last_active":"2024-09-13T13:03:01.761078+0000","last_peered":"2024-09-13T13:03:01.761078+0000","last_clean":"2024-09-13T13:03:01.761078+0000","last_became_active":"2024-09-13T13:01:08.533119+0000","last_became_peered":"2024-09-13T13:01:08.533119+0000","last_unstale":"2024-09-13T13:03:01.761078+0000","last_undegraded":"2024-09-13T13:03:01.761078+0000","last_fullsized":"2024-09-13T13:03:01.761078+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.213470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.213470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:04:08.233584+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"98'4263","reported_seq":4302,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.882537+0000","last_change":"2024-09-13T13:02:12.375804+0000","last_active":"2024-09-13T13:03:02.882537+0000","last_peered":"2024-09-13T13:03:02.882537+0000","last_clean":"2024-09-13T13:03:02.882537+0000","last_became_active":"2024-09-13T13:02:12.375509+0000","last_became_peered":"2024-09-13T13:02:12.375509+0000","last_unstale":"2024-09-13T13:03:02.882537+0000","last_undegraded":"2024-09-13T13:03:02.882537+0000","last_fullsized":"2024-09-13T13:03:02.882537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4263,"log_dups_size":0,"ondisk_log_size":4263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:20.534774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279379968,"num_objects":4263,"num_object_clones":0,"num_object_copies":12789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4263,"num_write_kb":272832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":112,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.741571+0000","last_change":"2024-09-13T13:01:04.195092+0000","last_active":"2024-09-13T13:03:01.741571+0000","last_peered":"2024-09-13T13:03:01.741571+0000","last_clean":"2024-09-13T13:03:01.741571+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:03:01.741571+0000","last_undegraded":"2024-09-13T13:03:01.741571+0000","last_fullsized":"2024-09-13T13:03:01.741571+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751999+0000","last_change":"2024-09-13T13:01:02.181492+0000","last_active":"2024-09-13T13:03:01.751999+0000","last_peered":"2024-09-13T13:03:01.751999+0000","last_clean":"2024-09-13T13:03:01.751999+0000","last_became_active":"2024-09-13T13:01:02.180946+0000","last_became_peered":"2024-09-13T13:01:02.180946+0000","last_unstale":"2024-09-13T13:03:01.751999+0000","last_undegraded":"2024-09-13T13:03:01.751999+0000","last_fullsized":"2024-09-13T13:03:01.751999+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:19.598689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":80,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.752075+0000","last_change":"2024-09-13T13:01:06.208593+0000","last_active":"2024-09-13T13:03:01.752075+0000","last_peered":"2024-09-13T13:03:01.752075+0000","last_clean":"2024-09-13T13:03:01.752075+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:03:01.752075+0000","last_undegraded":"2024-09-13T13:03:01.752075+0000","last_fullsized":"2024-09-13T13:03:01.752075+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:22.221007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":76,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.745667+0000","last_change":"2024-09-13T13:01:08.204029+0000","last_active":"2024-09-13T13:03:01.745667+0000","last_peered":"2024-09-13T13:03:01.745667+0000","last_clean":"2024-09-13T13:03:01.745667+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:03:01.745667+0000","last_undegraded":"2024-09-13T13:03:01.745667+0000","last_fullsized":"2024-09-13T13:03:01.745667+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:21.588201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"98'4472","reported_seq":4511,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.906415+0000","last_change":"2024-09-13T13:02:12.380514+0000","last_active":"2024-09-13T13:03:02.906415+0000","last_peered":"2024-09-13T13:03:02.906415+0000","last_clean":"2024-09-13T13:03:02.906415+0000","last_became_active":"2024-09-13T13:02:12.380229+0000","last_became_peered":"2024-09-13T13:02:12.380229+0000","last_unstale":"2024-09-13T13:03:02.906415+0000","last_undegraded":"2024-09-13T13:03:02.906415+0000","last_fullsized":"2024-09-13T13:03:02.906415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4472,"log_dups_size":0,"ondisk_log_size":4472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:22.406559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293076992,"num_objects":4472,"num_object_clones":0,"num_object_copies":13416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4472,"num_write_kb":286208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":99,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.741491+0000","last_change":"2024-09-13T13:01:04.189975+0000","last_active":"2024-09-13T13:03:01.741491+0000","last_peered":"2024-09-13T13:03:01.741491+0000","last_clean":"2024-09-13T13:03:01.741491+0000","last_became_active":"2024-09-13T13:01:04.189380+0000","last_became_peered":"2024-09-13T13:01:04.189380+0000","last_unstale":"2024-09-13T13:03:01.741491+0000","last_undegraded":"2024-09-13T13:03:01.741491+0000","last_fullsized":"2024-09-13T13:03:01.741491+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:14.865741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.750200+0000","last_change":"2024-09-13T13:02:07.165721+0000","last_active":"2024-09-13T13:03:01.750200+0000","last_peered":"2024-09-13T13:03:01.750200+0000","last_clean":"2024-09-13T13:03:01.750200+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:03:01.750200+0000","last_undegraded":"2024-09-13T13:03:01.750200+0000","last_fullsized":"2024-09-13T13:03:01.750200+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:07.165676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:07.165676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:20.879678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":113,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.754438+0000","last_change":"2024-09-13T13:01:06.206335+0000","last_active":"2024-09-13T13:03:01.754438+0000","last_peered":"2024-09-13T13:03:01.754438+0000","last_clean":"2024-09-13T13:03:01.754438+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:03:01.754438+0000","last_undegraded":"2024-09-13T13:03:01.754438+0000","last_fullsized":"2024-09-13T13:03:01.754438+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:13.466648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":84,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.741181+0000","last_change":"2024-09-13T13:02:33.997780+0000","last_active":"2024-09-13T13:03:01.741181+0000","last_peered":"2024-09-13T13:03:01.741181+0000","last_clean":"2024-09-13T13:03:01.741181+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:03:01.741181+0000","last_undegraded":"2024-09-13T13:03:01.741181+0000","last_fullsized":"2024-09-13T13:03:01.741181+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.997740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.997740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:56.739408+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"98'4443","reported_seq":4482,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.835707+0000","last_change":"2024-09-13T13:02:12.376094+0000","last_active":"2024-09-13T13:03:02.835707+0000","last_peered":"2024-09-13T13:03:02.835707+0000","last_clean":"2024-09-13T13:03:02.835707+0000","last_became_active":"2024-09-13T13:02:12.375310+0000","last_became_peered":"2024-09-13T13:02:12.375310+0000","last_unstale":"2024-09-13T13:03:02.835707+0000","last_undegraded":"2024-09-13T13:03:02.835707+0000","last_fullsized":"2024-09-13T13:03:02.835707+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4443,"log_dups_size":0,"ondisk_log_size":4443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:26.180190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291176448,"num_objects":4443,"num_object_clones":0,"num_object_copies":13329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":284352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":99,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.743556+0000","last_change":"2024-09-13T13:01:04.192921+0000","last_active":"2024-09-13T13:03:01.743556+0000","last_peered":"2024-09-13T13:03:01.743556+0000","last_clean":"2024-09-13T13:03:01.743556+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:03:01.743556+0000","last_undegraded":"2024-09-13T13:03:01.743556+0000","last_fullsized":"2024-09-13T13:03:01.743556+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:12.409533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748385+0000","last_change":"2024-09-13T13:02:46.305914+0000","last_active":"2024-09-13T13:03:01.748385+0000","last_peered":"2024-09-13T13:03:01.748385+0000","last_clean":"2024-09-13T13:03:01.748385+0000","last_became_active":"2024-09-13T13:01:02.180541+0000","last_became_peered":"2024-09-13T13:01:02.180541+0000","last_unstale":"2024-09-13T13:03:01.748385+0000","last_undegraded":"2024-09-13T13:03:01.748385+0000","last_fullsized":"2024-09-13T13:03:01.748385+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.305843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.305843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:04:02.213134+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.744572+0000","last_change":"2024-09-13T13:02:34.637576+0000","last_active":"2024-09-13T13:03:01.744572+0000","last_peered":"2024-09-13T13:03:01.744572+0000","last_clean":"2024-09-13T13:03:01.744572+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:03:01.744572+0000","last_undegraded":"2024-09-13T13:03:01.744572+0000","last_fullsized":"2024-09-13T13:03:01.744572+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:34.637509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:34.637509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:54.048663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.743229+0000","last_change":"2024-09-13T13:01:08.211229+0000","last_active":"2024-09-13T13:03:01.743229+0000","last_peered":"2024-09-13T13:03:01.743229+0000","last_clean":"2024-09-13T13:03:01.743229+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:03:01.743229+0000","last_undegraded":"2024-09-13T13:03:01.743229+0000","last_fullsized":"2024-09-13T13:03:01.743229+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:52.528142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"98'4486","reported_seq":4525,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.876567+0000","last_change":"2024-09-13T13:02:12.379755+0000","last_active":"2024-09-13T13:03:02.876567+0000","last_peered":"2024-09-13T13:03:02.876567+0000","last_clean":"2024-09-13T13:03:02.876567+0000","last_became_active":"2024-09-13T13:02:12.379442+0000","last_became_peered":"2024-09-13T13:02:12.379442+0000","last_unstale":"2024-09-13T13:03:02.876567+0000","last_undegraded":"2024-09-13T13:03:02.876567+0000","last_fullsized":"2024-09-13T13:03:02.876567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4486,"log_dups_size":0,"ondisk_log_size":4486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:16.717891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293994496,"num_objects":4486,"num_object_clones":0,"num_object_copies":13458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4486,"num_write_kb":287104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":122,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751879+0000","last_change":"2024-09-13T13:02:42.593376+0000","last_active":"2024-09-13T13:03:01.751879+0000","last_peered":"2024-09-13T13:03:01.751879+0000","last_clean":"2024-09-13T13:03:01.751879+0000","last_became_active":"2024-09-13T13:01:04.207702+0000","last_became_peered":"2024-09-13T13:01:04.207702+0000","last_unstale":"2024-09-13T13:03:01.751879+0000","last_undegraded":"2024-09-13T13:03:01.751879+0000","last_fullsized":"2024-09-13T13:03:01.751879+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:42.593284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.593284+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:04:11.597045+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747015+0000","last_change":"2024-09-13T13:02:29.012390+0000","last_active":"2024-09-13T13:03:01.747015+0000","last_peered":"2024-09-13T13:03:01.747015+0000","last_clean":"2024-09-13T13:03:01.747015+0000","last_became_active":"2024-09-13T13:01:02.178317+0000","last_became_peered":"2024-09-13T13:01:02.178317+0000","last_unstale":"2024-09-13T13:03:01.747015+0000","last_undegraded":"2024-09-13T13:03:01.747015+0000","last_fullsized":"2024-09-13T13:03:01.747015+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:29.012312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.012312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:56.417738+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.744310+0000","last_change":"2024-09-13T13:02:36.652073+0000","last_active":"2024-09-13T13:03:01.744310+0000","last_peered":"2024-09-13T13:03:01.744310+0000","last_clean":"2024-09-13T13:03:01.744310+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:03:01.744310+0000","last_undegraded":"2024-09-13T13:03:01.744310+0000","last_fullsized":"2024-09-13T13:03:01.744310+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.651818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.651818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:43.126075+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.745441+0000","last_change":"2024-09-13T13:01:08.212073+0000","last_active":"2024-09-13T13:03:01.745441+0000","last_peered":"2024-09-13T13:03:01.745441+0000","last_clean":"2024-09-13T13:03:01.745441+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:03:01.745441+0000","last_undegraded":"2024-09-13T13:03:01.745441+0000","last_fullsized":"2024-09-13T13:03:01.745441+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"98'4379","reported_seq":4418,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.902055+0000","last_change":"2024-09-13T13:02:12.376680+0000","last_active":"2024-09-13T13:03:02.902055+0000","last_peered":"2024-09-13T13:03:02.902055+0000","last_clean":"2024-09-13T13:03:02.902055+0000","last_became_active":"2024-09-13T13:02:12.376334+0000","last_became_peered":"2024-09-13T13:02:12.376334+0000","last_unstale":"2024-09-13T13:03:02.902055+0000","last_undegraded":"2024-09-13T13:03:02.902055+0000","last_fullsized":"2024-09-13T13:03:02.902055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:26.707085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286982144,"num_objects":4379,"num_object_clones":0,"num_object_copies":13137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4379,"num_write_kb":280256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":115,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.740681+0000","last_change":"2024-09-13T13:02:28.990089+0000","last_active":"2024-09-13T13:03:01.740681+0000","last_peered":"2024-09-13T13:03:01.740681+0000","last_clean":"2024-09-13T13:03:01.740681+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:03:01.740681+0000","last_undegraded":"2024-09-13T13:03:01.740681+0000","last_fullsized":"2024-09-13T13:03:01.740681+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:02:28.990005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.990005+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:56.465528+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747043+0000","last_change":"2024-09-13T13:02:03.099308+0000","last_active":"2024-09-13T13:03:01.747043+0000","last_peered":"2024-09-13T13:03:01.747043+0000","last_clean":"2024-09-13T13:03:01.747043+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:03:01.747043+0000","last_undegraded":"2024-09-13T13:03:01.747043+0000","last_fullsized":"2024-09-13T13:03:01.747043+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:03.099251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:03.099251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:26.603092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":80,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748322+0000","last_change":"2024-09-13T13:01:06.211172+0000","last_active":"2024-09-13T13:03:01.748322+0000","last_peered":"2024-09-13T13:03:01.748322+0000","last_clean":"2024-09-13T13:03:01.748322+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:03:01.748322+0000","last_undegraded":"2024-09-13T13:03:01.748322+0000","last_fullsized":"2024-09-13T13:03:01.748322+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:23.599900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.749888+0000","last_change":"2024-09-13T13:01:08.211530+0000","last_active":"2024-09-13T13:03:01.749888+0000","last_peered":"2024-09-13T13:03:01.749888+0000","last_clean":"2024-09-13T13:03:01.749888+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:03:01.749888+0000","last_undegraded":"2024-09-13T13:03:01.749888+0000","last_fullsized":"2024-09-13T13:03:01.749888+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:17.274269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"98'4488","reported_seq":4527,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:02.854789+0000","last_change":"2024-09-13T13:02:12.376919+0000","last_active":"2024-09-13T13:03:02.854789+0000","last_peered":"2024-09-13T13:03:02.854789+0000","last_clean":"2024-09-13T13:03:02.854789+0000","last_became_active":"2024-09-13T13:02:12.376643+0000","last_became_peered":"2024-09-13T13:02:12.376643+0000","last_unstale":"2024-09-13T13:03:02.854789+0000","last_undegraded":"2024-09-13T13:03:02.854789+0000","last_fullsized":"2024-09-13T13:03:02.854789+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:14.341212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294125568,"num_objects":4488,"num_object_clones":0,"num_object_copies":13464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4488,"num_write_kb":287232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":108,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747364+0000","last_change":"2024-09-13T13:01:04.209162+0000","last_active":"2024-09-13T13:03:01.747364+0000","last_peered":"2024-09-13T13:03:01.747364+0000","last_clean":"2024-09-13T13:03:01.747364+0000","last_became_active":"2024-09-13T13:01:04.208731+0000","last_became_peered":"2024-09-13T13:01:04.208731+0000","last_unstale":"2024-09-13T13:03:01.747364+0000","last_undegraded":"2024-09-13T13:03:01.747364+0000","last_fullsized":"2024-09-13T13:03:01.747364+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:22.557714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":107,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.746393+0000","last_change":"2024-09-13T13:02:43.962441+0000","last_active":"2024-09-13T13:03:01.746393+0000","last_peered":"2024-09-13T13:03:01.746393+0000","last_clean":"2024-09-13T13:03:01.746393+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:03:01.746393+0000","last_undegraded":"2024-09-13T13:03:01.746393+0000","last_fullsized":"2024-09-13T13:03:01.746393+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:02:43.962338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.962338+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:04:08.822545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":123,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.745407+0000","last_change":"2024-09-13T13:02:45.959334+0000","last_active":"2024-09-13T13:03:01.745407+0000","last_peered":"2024-09-13T13:03:01.745407+0000","last_clean":"2024-09-13T13:03:01.745407+0000","last_became_active":"2024-09-13T13:01:06.210441+0000","last_became_peered":"2024-09-13T13:01:06.210441+0000","last_unstale":"2024-09-13T13:03:01.745407+0000","last_undegraded":"2024-09-13T13:03:01.745407+0000","last_fullsized":"2024-09-13T13:03:01.745407+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:45.959285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.959285+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:04:02.336780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":83,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747530+0000","last_change":"2024-09-13T13:02:28.530023+0000","last_active":"2024-09-13T13:03:01.747530+0000","last_peered":"2024-09-13T13:03:01.747530+0000","last_clean":"2024-09-13T13:03:01.747530+0000","last_became_active":"2024-09-13T13:01:08.531895+0000","last_became_peered":"2024-09-13T13:01:08.531895+0000","last_unstale":"2024-09-13T13:03:01.747530+0000","last_undegraded":"2024-09-13T13:03:01.747530+0000","last_fullsized":"2024-09-13T13:03:01.747530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:28.529954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.529954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:44.575376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751703+0000","last_change":"2024-09-13T13:01:02.179217+0000","last_active":"2024-09-13T13:03:01.751703+0000","last_peered":"2024-09-13T13:03:01.751703+0000","last_clean":"2024-09-13T13:03:01.751703+0000","last_became_active":"2024-09-13T13:01:02.178934+0000","last_became_peered":"2024-09-13T13:01:02.178934+0000","last_unstale":"2024-09-13T13:03:01.751703+0000","last_undegraded":"2024-09-13T13:03:01.751703+0000","last_fullsized":"2024-09-13T13:03:01.751703+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:21.696616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":92,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747685+0000","last_change":"2024-09-13T13:01:04.209208+0000","last_active":"2024-09-13T13:03:01.747685+0000","last_peered":"2024-09-13T13:03:01.747685+0000","last_clean":"2024-09-13T13:03:01.747685+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:03:01.747685+0000","last_undegraded":"2024-09-13T13:03:01.747685+0000","last_fullsized":"2024-09-13T13:03:01.747685+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:11.228914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748673+0000","last_change":"2024-09-13T13:02:26.581497+0000","last_active":"2024-09-13T13:03:01.748673+0000","last_peered":"2024-09-13T13:03:01.748673+0000","last_clean":"2024-09-13T13:03:01.748673+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:03:01.748673+0000","last_undegraded":"2024-09-13T13:03:01.748673+0000","last_fullsized":"2024-09-13T13:03:01.748673+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:26.581434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:26.581434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:52.538043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.746624+0000","last_change":"2024-09-13T13:01:08.532411+0000","last_active":"2024-09-13T13:03:01.746624+0000","last_peered":"2024-09-13T13:03:01.746624+0000","last_clean":"2024-09-13T13:03:01.746624+0000","last_became_active":"2024-09-13T13:01:08.532178+0000","last_became_peered":"2024-09-13T13:01:08.532178+0000","last_unstale":"2024-09-13T13:03:01.746624+0000","last_undegraded":"2024-09-13T13:03:01.746624+0000","last_fullsized":"2024-09-13T13:03:01.746624+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:25.003970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":107,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751612+0000","last_change":"2024-09-13T13:02:42.589951+0000","last_active":"2024-09-13T13:03:01.751612+0000","last_peered":"2024-09-13T13:03:01.751612+0000","last_clean":"2024-09-13T13:03:01.751612+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:03:01.751612+0000","last_undegraded":"2024-09-13T13:03:01.751612+0000","last_fullsized":"2024-09-13T13:03:01.751612+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:42.589866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.589866+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:43.304111+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.745508+0000","last_change":"2024-09-13T13:01:02.181825+0000","last_active":"2024-09-13T13:03:01.745508+0000","last_peered":"2024-09-13T13:03:01.745508+0000","last_clean":"2024-09-13T13:03:01.745508+0000","last_became_active":"2024-09-13T13:01:02.181548+0000","last_became_peered":"2024-09-13T13:01:02.181548+0000","last_unstale":"2024-09-13T13:03:01.745508+0000","last_undegraded":"2024-09-13T13:03:01.745508+0000","last_fullsized":"2024-09-13T13:03:01.745508+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:21.115596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747562+0000","last_change":"2024-09-13T13:01:06.210328+0000","last_active":"2024-09-13T13:03:01.747562+0000","last_peered":"2024-09-13T13:03:01.747562+0000","last_clean":"2024-09-13T13:03:01.747562+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:03:01.747562+0000","last_undegraded":"2024-09-13T13:03:01.747562+0000","last_fullsized":"2024-09-13T13:03:01.747562+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:24.341363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":76,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.744548+0000","last_change":"2024-09-13T13:01:08.205690+0000","last_active":"2024-09-13T13:03:01.744548+0000","last_peered":"2024-09-13T13:03:01.744548+0000","last_clean":"2024-09-13T13:03:01.744548+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:03:01.744548+0000","last_undegraded":"2024-09-13T13:03:01.744548+0000","last_fullsized":"2024-09-13T13:03:01.744548+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:19.543874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":99,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748111+0000","last_change":"2024-09-13T13:01:04.208351+0000","last_active":"2024-09-13T13:03:01.748111+0000","last_peered":"2024-09-13T13:03:01.748111+0000","last_clean":"2024-09-13T13:03:01.748111+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:03:01.748111+0000","last_undegraded":"2024-09-13T13:03:01.748111+0000","last_fullsized":"2024-09-13T13:03:01.748111+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:14.514329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.743196+0000","last_change":"2024-09-13T13:02:41.622369+0000","last_active":"2024-09-13T13:03:01.743196+0000","last_peered":"2024-09-13T13:03:01.743196+0000","last_clean":"2024-09-13T13:03:01.743196+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:03:01.743196+0000","last_undegraded":"2024-09-13T13:03:01.743196+0000","last_fullsized":"2024-09-13T13:03:01.743196+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.622292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.622292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:53.892482+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":112,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.761258+0000","last_change":"2024-09-13T13:01:06.210251+0000","last_active":"2024-09-13T13:03:01.761258+0000","last_peered":"2024-09-13T13:03:01.761258+0000","last_clean":"2024-09-13T13:03:01.761258+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:03:01.761258+0000","last_undegraded":"2024-09-13T13:03:01.761258+0000","last_fullsized":"2024-09-13T13:03:01.761258+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":76,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.741895+0000","last_change":"2024-09-13T13:01:08.206113+0000","last_active":"2024-09-13T13:03:01.741895+0000","last_peered":"2024-09-13T13:03:01.741895+0000","last_clean":"2024-09-13T13:03:01.741895+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:03:01.741895+0000","last_undegraded":"2024-09-13T13:03:01.741895+0000","last_fullsized":"2024-09-13T13:03:01.741895+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:20.932765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":106,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.761507+0000","last_change":"2024-09-13T13:02:37.222400+0000","last_active":"2024-09-13T13:03:01.761507+0000","last_peered":"2024-09-13T13:03:01.761507+0000","last_clean":"2024-09-13T13:03:01.761507+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:03:01.761507+0000","last_undegraded":"2024-09-13T13:03:01.761507+0000","last_fullsized":"2024-09-13T13:03:01.761507+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:37.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.222329+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:52.865293+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.745986+0000","last_change":"2024-09-13T13:01:02.180706+0000","last_active":"2024-09-13T13:03:01.745986+0000","last_peered":"2024-09-13T13:03:01.745986+0000","last_clean":"2024-09-13T13:03:01.745986+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:03:01.745986+0000","last_undegraded":"2024-09-13T13:03:01.745986+0000","last_fullsized":"2024-09-13T13:03:01.745986+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:12.978829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":120,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747963+0000","last_change":"2024-09-13T13:02:30.469448+0000","last_active":"2024-09-13T13:03:01.747963+0000","last_peered":"2024-09-13T13:03:01.747963+0000","last_clean":"2024-09-13T13:03:01.747963+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:03:01.747963+0000","last_undegraded":"2024-09-13T13:03:01.747963+0000","last_fullsized":"2024-09-13T13:03:01.747963+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:30.469386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:30.469386+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:34.257322+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":83,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751087+0000","last_change":"2024-09-13T13:02:40.615749+0000","last_active":"2024-09-13T13:03:01.751087+0000","last_peered":"2024-09-13T13:03:01.751087+0000","last_clean":"2024-09-13T13:03:01.751087+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:03:01.751087+0000","last_undegraded":"2024-09-13T13:03:01.751087+0000","last_fullsized":"2024-09-13T13:03:01.751087+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:40.615674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:40.615674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:58.047473+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":111,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.741507+0000","last_change":"2024-09-13T13:02:42.961405+0000","last_active":"2024-09-13T13:03:01.741507+0000","last_peered":"2024-09-13T13:03:01.741507+0000","last_clean":"2024-09-13T13:03:01.741507+0000","last_became_active":"2024-09-13T13:01:04.196857+0000","last_became_peered":"2024-09-13T13:01:04.196857+0000","last_unstale":"2024-09-13T13:03:01.741507+0000","last_undegraded":"2024-09-13T13:03:01.741507+0000","last_fullsized":"2024-09-13T13:03:01.741507+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:02:42.961328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.961328+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:44.564319+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":96,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.741863+0000","last_change":"2024-09-13T13:02:35.015368+0000","last_active":"2024-09-13T13:03:01.741863+0000","last_peered":"2024-09-13T13:03:01.741863+0000","last_clean":"2024-09-13T13:03:01.741863+0000","last_became_active":"2024-09-13T13:01:02.180817+0000","last_became_peered":"2024-09-13T13:01:02.180817+0000","last_unstale":"2024-09-13T13:03:01.741863+0000","last_undegraded":"2024-09-13T13:03:01.741863+0000","last_fullsized":"2024-09-13T13:03:01.741863+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:35.015309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:35.015309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:50.142703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":88,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.759078+0000","last_change":"2024-09-13T13:02:35.221809+0000","last_active":"2024-09-13T13:03:01.759078+0000","last_peered":"2024-09-13T13:03:01.759078+0000","last_clean":"2024-09-13T13:03:01.759078+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:03:01.759078+0000","last_undegraded":"2024-09-13T13:03:01.759078+0000","last_fullsized":"2024-09-13T13:03:01.759078+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:35.221628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:35.221628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:40.700679+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":83,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.746708+0000","last_change":"2024-09-13T13:02:42.208021+0000","last_active":"2024-09-13T13:03:01.746708+0000","last_peered":"2024-09-13T13:03:01.746708+0000","last_clean":"2024-09-13T13:03:01.746708+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:03:01.746708+0000","last_undegraded":"2024-09-13T13:03:01.746708+0000","last_fullsized":"2024-09-13T13:03:01.746708+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:42.207956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.207956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:52.791966+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":40,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.747386+0000","last_change":"2024-09-13T13:02:35.448454+0000","last_active":"2024-09-13T13:03:01.747386+0000","last_peered":"2024-09-13T13:03:01.747386+0000","last_clean":"2024-09-13T13:03:01.747386+0000","last_became_active":"2024-09-13T13:02:33.739584+0000","last_became_peered":"2024-09-13T13:02:33.739584+0000","last_unstale":"2024-09-13T13:03:01.747386+0000","last_undegraded":"2024-09-13T13:03:01.747386+0000","last_fullsized":"2024-09-13T13:03:01.747386+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:35.448404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:35.448404+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:57.864321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.746929+0000","last_change":"2024-09-13T13:02:39.921391+0000","last_active":"2024-09-13T13:03:01.746929+0000","last_peered":"2024-09-13T13:03:01.746929+0000","last_clean":"2024-09-13T13:03:01.746929+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:03:01.746929+0000","last_undegraded":"2024-09-13T13:03:01.746929+0000","last_fullsized":"2024-09-13T13:03:01.746929+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.921325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.921325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:04:00.788546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.749988+0000","last_change":"2024-09-13T13:01:06.211993+0000","last_active":"2024-09-13T13:03:01.749988+0000","last_peered":"2024-09-13T13:03:01.749988+0000","last_clean":"2024-09-13T13:03:01.749988+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:03:01.749988+0000","last_undegraded":"2024-09-13T13:03:01.749988+0000","last_fullsized":"2024-09-13T13:03:01.749988+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:53.305128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":83,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.742213+0000","last_change":"2024-09-13T13:02:36.586702+0000","last_active":"2024-09-13T13:03:01.742213+0000","last_peered":"2024-09-13T13:03:01.742213+0000","last_clean":"2024-09-13T13:03:01.742213+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:03:01.742213+0000","last_undegraded":"2024-09-13T13:03:01.742213+0000","last_fullsized":"2024-09-13T13:03:01.742213+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.586654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.586654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:42.698549+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":102,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.746859+0000","last_change":"2024-09-13T13:01:04.208421+0000","last_active":"2024-09-13T13:03:01.746859+0000","last_peered":"2024-09-13T13:03:01.746859+0000","last_clean":"2024-09-13T13:03:01.746859+0000","last_became_active":"2024-09-13T13:01:04.207841+0000","last_became_peered":"2024-09-13T13:01:04.207841+0000","last_unstale":"2024-09-13T13:03:01.746859+0000","last_undegraded":"2024-09-13T13:03:01.746859+0000","last_fullsized":"2024-09-13T13:03:01.746859+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:09.197339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748756+0000","last_change":"2024-09-13T13:01:02.185420+0000","last_active":"2024-09-13T13:03:01.748756+0000","last_peered":"2024-09-13T13:03:01.748756+0000","last_clean":"2024-09-13T13:03:01.748756+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:03:01.748756+0000","last_undegraded":"2024-09-13T13:03:01.748756+0000","last_fullsized":"2024-09-13T13:03:01.748756+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:51.427620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.745211+0000","last_change":"2024-09-13T13:01:06.208696+0000","last_active":"2024-09-13T13:03:01.745211+0000","last_peered":"2024-09-13T13:03:01.745211+0000","last_clean":"2024-09-13T13:03:01.745211+0000","last_became_active":"2024-09-13T13:01:06.208426+0000","last_became_peered":"2024-09-13T13:01:06.208426+0000","last_unstale":"2024-09-13T13:03:01.745211+0000","last_undegraded":"2024-09-13T13:03:01.745211+0000","last_fullsized":"2024-09-13T13:03:01.745211+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:04:01.016499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.750126+0000","last_change":"2024-09-13T13:01:08.535233+0000","last_active":"2024-09-13T13:03:01.750126+0000","last_peered":"2024-09-13T13:03:01.750126+0000","last_clean":"2024-09-13T13:03:01.750126+0000","last_became_active":"2024-09-13T13:01:08.534673+0000","last_became_peered":"2024-09-13T13:01:08.534673+0000","last_unstale":"2024-09-13T13:03:01.750126+0000","last_undegraded":"2024-09-13T13:03:01.750126+0000","last_fullsized":"2024-09-13T13:03:01.750126+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":102,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748485+0000","last_change":"2024-09-13T13:02:33.767463+0000","last_active":"2024-09-13T13:03:01.748485+0000","last_peered":"2024-09-13T13:03:01.748485+0000","last_clean":"2024-09-13T13:03:01.748485+0000","last_became_active":"2024-09-13T13:02:33.743715+0000","last_became_peered":"2024-09-13T13:02:33.743715+0000","last_unstale":"2024-09-13T13:03:01.748485+0000","last_undegraded":"2024-09-13T13:03:01.748485+0000","last_fullsized":"2024-09-13T13:03:01.748485+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:51.241486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748453+0000","last_change":"2024-09-13T13:01:02.186003+0000","last_active":"2024-09-13T13:03:01.748453+0000","last_peered":"2024-09-13T13:03:01.748453+0000","last_clean":"2024-09-13T13:03:01.748453+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:03:01.748453+0000","last_undegraded":"2024-09-13T13:03:01.748453+0000","last_fullsized":"2024-09-13T13:03:01.748453+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:08.412185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":79,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751857+0000","last_change":"2024-09-13T13:01:06.211514+0000","last_active":"2024-09-13T13:03:01.751857+0000","last_peered":"2024-09-13T13:03:01.751857+0000","last_clean":"2024-09-13T13:03:01.751857+0000","last_became_active":"2024-09-13T13:01:06.211243+0000","last_became_peered":"2024-09-13T13:01:06.211243+0000","last_unstale":"2024-09-13T13:03:01.751857+0000","last_undegraded":"2024-09-13T13:03:01.751857+0000","last_fullsized":"2024-09-13T13:03:01.751857+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:20.648463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.745410+0000","last_change":"2024-09-13T13:01:08.210205+0000","last_active":"2024-09-13T13:03:01.745410+0000","last_peered":"2024-09-13T13:03:01.745410+0000","last_clean":"2024-09-13T13:03:01.745410+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:03:01.745410+0000","last_undegraded":"2024-09-13T13:03:01.745410+0000","last_fullsized":"2024-09-13T13:03:01.745410+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:53.907342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":84,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751574+0000","last_change":"2024-09-13T13:02:43.234909+0000","last_active":"2024-09-13T13:03:01.751574+0000","last_peered":"2024-09-13T13:03:01.751574+0000","last_clean":"2024-09-13T13:03:01.751574+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:03:01.751574+0000","last_undegraded":"2024-09-13T13:03:01.751574+0000","last_fullsized":"2024-09-13T13:03:01.751574+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:43.234847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.234847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:04:01.526096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.748740+0000","last_change":"2024-09-13T13:02:46.246602+0000","last_active":"2024-09-13T13:03:01.748740+0000","last_peered":"2024-09-13T13:03:01.748740+0000","last_clean":"2024-09-13T13:03:01.748740+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:03:01.748740+0000","last_undegraded":"2024-09-13T13:03:01.748740+0000","last_fullsized":"2024-09-13T13:03:01.748740+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.246540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.246540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:04:10.457167+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":129,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.752402+0000","last_change":"2024-09-13T13:02:39.557347+0000","last_active":"2024-09-13T13:03:01.752402+0000","last_peered":"2024-09-13T13:03:01.752402+0000","last_clean":"2024-09-13T13:03:01.752402+0000","last_became_active":"2024-09-13T13:02:33.735381+0000","last_became_peered":"2024-09-13T13:02:33.735381+0000","last_unstale":"2024-09-13T13:03:01.752402+0000","last_undegraded":"2024-09-13T13:03:01.752402+0000","last_fullsized":"2024-09-13T13:03:01.752402+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:02:39.557251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.557251+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:46.327035+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":87,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T13:03:01.751346+0000","last_change":"2024-09-13T13:02:30.512536+0000","last_active":"2024-09-13T13:03:01.751346+0000","last_peered":"2024-09-13T13:03:01.751346+0000","last_clean":"2024-09-13T13:03:01.751346+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:03:01.751346+0000","last_undegraded":"2024-09-13T13:03:01.751346+0000","last_fullsized":"2024-09-13T13:03:01.751346+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:30.512433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:30.512433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:03:57.883777+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4645978112,"num_objects":70892,"num_object_clones":0,"num_object_copies":212676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70892,"num_write_kb":4537088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13939769344,"data_stored":13939769344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70892,"ondisk_log_size":70892,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":110,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201307,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1458896,"kb_used_data":1409660,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92286256,"statfs":{"total":95995035648,"available":94501126144,"internally_reserved":0,"allocated":1443491840,"data_stored":1442901761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430241,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1473212,"kb_used_data":1423980,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92271940,"statfs":{"total":95995035648,"available":94486466560,"internally_reserved":0,"allocated":1458155520,"data_stored":1457580196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691879,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1158788,"kb_used_data":1109552,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92586364,"statfs":{"total":95995035648,"available":94808436736,"internally_reserved":0,"allocated":1136181248,"data_stored":1135601753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920813,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1449436,"kb_used_data":1400196,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92295716,"statfs":{"total":95995035648,"available":94510813184,"internally_reserved":0,"allocated":1433800704,"data_stored":1433200865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117044,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2595040,"kb_used_data":2527752,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91150112,"statfs":{"total":95995035648,"available":93337714688,"internally_reserved":0,"allocated":2588418048,"data_stored":2587814943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411383,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1730864,"kb_used_data":1677472,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92014288,"statfs":{"total":95995035648,"available":94222630912,"internally_reserved":0,"allocated":1717731328,"data_stored":1717167557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1739504,"kb_used_data":1686104,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92005648,"statfs":{"total":95995035648,"available":94213783552,"internally_reserved":0,"allocated":1726570496,"data_stored":1725950688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2331732,"kb_used_data":2268608,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91413420,"statfs":{"total":95995035648,"available":93607342080,"internally_reserved":0,"allocated":2323054592,"data_stored":2322460126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2332688384,"data_stored":2332688384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1745354752,"data_stored":1745354752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1727856640,"data_stored":1727856640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2610364416,"data_stored":2610364416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1454964736,"data_stored":1454964736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1144324096,"data_stored":1144324096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1461583872,"data_stored":1461583872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1462632448,"data_stored":1462632448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:03:07.753 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:07.781 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:07.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:07.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:07 smithi167 ceph-mon[33178]: pgmap v76: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 830 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-13T13:03:07.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:07 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:07.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:07 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:07.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:07 smithi167 ceph-mon[33178]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T13:03:07.882 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:07.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:07.928 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:07.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:08.029 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:08.072 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:08.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:08.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:08.161 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 46 15 76917 76902 104.428 73.125 0.00400502 0.0095697 2024-09-13T13:03:08.174 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:08.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:08.207 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:08.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:08.307 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:08.336 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:08.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:08.437 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:08.469 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:08.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:08.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:08.571 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:08.612 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:08 smithi052 ceph-mon[36682]: from='client.34847 v1:172.21.15.52:0/3992091752' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:03:08.612 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:08 smithi052 ceph-mon[36682]: Reconfiguring daemon prometheus.a on smithi167 2024-09-13T13:03:08.612 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:08 smithi052 ceph-mon[36682]: osdmap e100: 8 total, 8 up, 8 in 2024-09-13T13:03:08.612 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2661220636' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:03:08.613 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:08 smithi052 ceph-mon[36682]: from='client.24853 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:03:08.613 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:08 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[76921]: ts=2024-09-13T13:03:08.258Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000399655s 2024-09-13T13:03:08.613 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:08 smithi052 ceph-mon[29471]: from='client.34847 v1:172.21.15.52:0/3992091752' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:03:08.613 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:08 smithi052 ceph-mon[29471]: Reconfiguring daemon prometheus.a on smithi167 2024-09-13T13:03:08.613 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:08 smithi052 ceph-mon[29471]: osdmap e100: 8 total, 8 up, 8 in 2024-09-13T13:03:08.613 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2661220636' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:03:08.613 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:08 smithi052 ceph-mon[29471]: from='client.24853 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:03:08.615 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:08.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:08.678 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:03:08.678 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:03:08.705 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:08.717 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:08.734 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:03:08.743 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:08.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:08.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:08.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:08 smithi167 ceph-mon[33178]: from='client.34847 v1:172.21.15.52:0/3992091752' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:03:08.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:08 smithi167 ceph-mon[33178]: Reconfiguring daemon prometheus.a on smithi167 2024-09-13T13:03:08.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:08 smithi167 ceph-mon[33178]: osdmap e100: 8 total, 8 up, 8 in 2024-09-13T13:03:08.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2661220636' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:03:08.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:08 smithi167 ceph-mon[33178]: from='client.24853 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:03:08.844 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:08.874 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:08.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:08.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:08.975 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:09.004 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:09.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:09.106 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:09.131 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:09.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:09.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:09.161 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 47 16 78077 78061 103.748 72.4375 0.00508588 0.00963302 2024-09-13T13:03:09.233 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:09.280 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:09.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:09.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:09.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:09.409 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:09.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:09.511 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:09.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:09.550 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:09.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:09.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[36682]: pgmap v78: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.6 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-13T13:03:09.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:03:09.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:03:09.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:09.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[36682]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:03:09.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[36682]: osdmap e101: 8 total, 8 up, 8 in 2024-09-13T13:03:09.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[36682]: 1.0 scrub starts 2024-09-13T13:03:09.604 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[36682]: 1.0 scrub ok 2024-09-13T13:03:09.605 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[29471]: pgmap v78: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.6 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-13T13:03:09.605 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:03:09.605 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:03:09.605 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:09.605 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[29471]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:03:09.605 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[29471]: osdmap e101: 8 total, 8 up, 8 in 2024-09-13T13:03:09.605 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[29471]: 1.0 scrub starts 2024-09-13T13:03:09.605 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:09 smithi052 ceph-mon[29471]: 1.0 scrub ok 2024-09-13T13:03:09.652 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:09.697 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:09.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:09.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:09.761 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:09 smithi167 ceph-mon[33178]: pgmap v78: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.6 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-13T13:03:09.761 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:09 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:03:09.761 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:09 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:03:09.761 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:09 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:09.761 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:09 smithi167 ceph-mon[33178]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:03:09.761 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:09 smithi167 ceph-mon[33178]: osdmap e101: 8 total, 8 up, 8 in 2024-09-13T13:03:09.761 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:09 smithi167 ceph-mon[33178]: 1.0 scrub starts 2024-09-13T13:03:09.762 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:09 smithi167 ceph-mon[33178]: 1.0 scrub ok 2024-09-13T13:03:09.799 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:09.826 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:09.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:09.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:09.927 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:09.978 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:09.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:10.034 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T13:03:10.080 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:10.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:10.106 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:10.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:10.161 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 48 16 79228 79212 103.085 71.9375 0.0137787 0.00969476 2024-09-13T13:03:10.206 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:10.232 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:10.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:10.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:10.334 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:10.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:10.361 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:10.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:10.462 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:10.489 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:10.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:10.591 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:10.629 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:10.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:10.731 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:10.753 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:10.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:10.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:10.854 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:10.892 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:10.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:10.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:10.995 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:03:11.022 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:11.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:11.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:03:11.125 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:11.155 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:11.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:11.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:11.161 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 49 16 80381 80365 102.452 72.0625 0.0175579 0.00975492 2024-09-13T13:03:11.258 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:11.293 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:11.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:11.394 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:11.439 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:11.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:11.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:11.541 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:11.570 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:11.570 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 systemd[1]: Stopping Ceph prometheus.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:03:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:11 smithi167 ceph-mon[33178]: pgmap v80: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 955 op/s 2024-09-13T13:03:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:11.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:11.573 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:03:11.573 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:03:11.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:03:11.614 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:03:11.614 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:03:11.622 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:11.652 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:11 smithi052 ceph-mon[29471]: pgmap v80: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 955 op/s 2024-09-13T13:03:11.653 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:11 smithi052 ceph-mon[36682]: pgmap v80: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 955 op/s 2024-09-13T13:03:11.656 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:03:11.657 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:03:11.672 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:11.696 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:03:11.696 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:03:11.723 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:11.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:11.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:11.727 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:03:11.728 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:03:11.765 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:03:11.766 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:03:11.802 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:03:11.803 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:03:11.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:11.836 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:03:11.836 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:03:11.851 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:11.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:11.953 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:11.980 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:11.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:12.005 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:03:11.719Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T13:03:12.005 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:03:11.719Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-13T13:03:12.005 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:03:11.719Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-13T13:03:12.005 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:03:11.719Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T13:03:12.005 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:03:11.719Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-13T13:03:12.006 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:03:11.719Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-13T13:03:12.006 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:03:11.719Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T13:03:12.006 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:03:11.719Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-13T13:03:12.006 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:03:11.719Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-13T13:03:12.006 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:03:11.720Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T13:03:12.006 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:03:11.720Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-13T13:03:12.006 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[59039]: ts=2024-09-13T13:03:11.720Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-13T13:03:12.006 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:11 smithi167 podman[66150]: 2024-09-13 13:03:11.729853852 +0000 UTC m=+0.158936206 container died ba51a23b04422e6c397e781589b781dc7fd9f933ad50acc068a07ee18d7438e5 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T13:03:12.082 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:12.110 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:12.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:12.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:12.161 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 50 16 81372 81356 101.642 61.9375 0.0147935 0.00983366 2024-09-13T13:03:12.213 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:12.264 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:12.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:12.321 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:12 smithi167 podman[66150]: 2024-09-13 13:03:12.112020584 +0000 UTC m=+0.541102934 container cleanup ba51a23b04422e6c397e781589b781dc7fd9f933ad50acc068a07ee18d7438e5 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T13:03:12.321 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:12 smithi167 bash[66150]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T13:03:12.366 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:12.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:12.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:12.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:12.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:12.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:12.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:12.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:12.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:12.663 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:12.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:12.687 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T13:03:12.687 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:03:12.687 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T13:03:12.688 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-13T13:03:12.688 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.6 2024-09-13T13:03:12.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:12 smithi167 ceph-mon[33178]: 5.1c scrub starts 2024-09-13T13:03:12.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:12 smithi167 ceph-mon[33178]: 5.1c scrub ok 2024-09-13T13:03:12.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:12 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1342592725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:12.724 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:12 smithi167 podman[66163]: 2024-09-13 13:03:12.430882801 +0000 UTC m=+0.699499328 container remove ba51a23b04422e6c397e781589b781dc7fd9f933ad50acc068a07ee18d7438e5 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T13:03:12.724 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:12 smithi167 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T13:03:12.724 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:12 smithi167 systemd[1]: Stopped Ceph prometheus.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:03:12.725 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:12 smithi167 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.183s CPU time. 2024-09-13T13:03:12.725 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:12 smithi167 systemd[1]: Starting Ceph prometheus.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:03:12.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:12 smithi052 ceph-mon[36682]: 5.1c scrub starts 2024-09-13T13:03:12.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:12 smithi052 ceph-mon[36682]: 5.1c scrub ok 2024-09-13T13:03:12.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:12 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1342592725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:12.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:12 smithi052 ceph-mon[29471]: 5.1c scrub starts 2024-09-13T13:03:12.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:12 smithi052 ceph-mon[29471]: 5.1c scrub ok 2024-09-13T13:03:12.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:12 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1342592725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:12.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:12.807 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:12.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:12.909 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:12.940 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:12.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:13.042 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:13.070 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:12 smithi167 systemd[1]: Stopping Ceph osd.6 for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:03:13.071 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:12 smithi167 podman[66361]: 2024-09-13 13:03:12.803276256 +0000 UTC m=+0.032442770 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-13T13:03:13.071 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:12 smithi167 podman[66361]: 2024-09-13 13:03:12.940655695 +0000 UTC m=+0.169822188 container create dcd09dadbde1fe64b21d69d218c5d29a69c99171a5eeb4c1764fac25b2800599 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T13:03:13.085 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:13.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:13.161 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 51 16 82540 82524 101.081 73 0.00438541 0.00988788 2024-09-13T13:03:13.188 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:13.215 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:13.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:13.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:13.337 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 podman[66361]: 2024-09-13 13:03:13.227602887 +0000 UTC m=+0.456769380 container init dcd09dadbde1fe64b21d69d218c5d29a69c99171a5eeb4c1764fac25b2800599 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T13:03:13.337 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 podman[66361]: 2024-09-13 13:03:13.232278594 +0000 UTC m=+0.461445091 container start dcd09dadbde1fe64b21d69d218c5d29a69c99171a5eeb4c1764fac25b2800599 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T13:03:13.337 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.258Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-13T13:03:13.337 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.258Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-13T13:03:13.338 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.258Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi167 (none))" 2024-09-13T13:03:13.338 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.258Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T13:03:13.338 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.258Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T13:03:13.338 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.260Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T13:03:13.338 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.261Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-13T13:03:13.338 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.264Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T13:03:13.338 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.264Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T13:03:13.338 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.265Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T13:03:13.338 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.265Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.745µs 2024-09-13T13:03:13.338 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.265Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T13:03:13.338 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.266Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-13T13:03:13.338 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.266Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-13T13:03:13.338 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.266Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.491µs wal_replay_duration=473.334µs wbl_replay_duration=114ns total_replay_duration=515.259µs 2024-09-13T13:03:13.339 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.268Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T13:03:13.339 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.268Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-13T13:03:13.339 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.268Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T13:03:13.339 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.288Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.431373ms db_storage=1.068µs remote_storage=1.58µs web_handler=381ns query_engine=673ns scrape=486.979µs scrape_sd=118.109µs notify=16.46µs notify_sd=13.146µs rules=19.312789ms tracing=8.329µs 2024-09-13T13:03:13.339 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.288Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-13T13:03:13.339 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:03:13.288Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T13:03:13.339 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 bash[66361]: dcd09dadbde1fe64b21d69d218c5d29a69c99171a5eeb4c1764fac25b2800599 2024-09-13T13:03:13.348 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:13.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:13.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:13.450 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:13.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:13.492 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:13.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:13.593 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:13.632 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:13.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:13.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:13.705 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:03:13 smithi167 systemd[1]: Started Ceph prometheus.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:03:13.706 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:03:13.706 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-mon[33178]: pgmap v81: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 741 B/s rd, 52 MiB/s wr, 831 op/s 2024-09-13T13:03:13.706 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-mon[33178]: 4.d scrub starts 2024-09-13T13:03:13.706 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-mon[33178]: 4.d scrub ok 2024-09-13T13:03:13.706 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-mon[33178]: 3.19 scrub starts 2024-09-13T13:03:13.706 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-mon[33178]: 3.19 scrub ok 2024-09-13T13:03:13.706 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-mon[33178]: 4.11 scrub starts 2024-09-13T13:03:13.706 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-mon[33178]: 4.11 scrub ok 2024-09-13T13:03:13.707 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-mon[33178]: osdmap e102: 8 total, 8 up, 8 in 2024-09-13T13:03:13.707 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:13.707 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:13.707 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[48313]: 2024-09-13T13:03:13.548+0000 7f05c310b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:03:13.707 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[48313]: 2024-09-13T13:03:13.548+0000 7f05c310b640 -1 osd.6 102 *** Got signal Terminated *** 2024-09-13T13:03:13.707 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[48313]: 2024-09-13T13:03:13.548+0000 7f05c310b640 -1 osd.6 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T13:03:13.728 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:13] ENGINE Bus STOPPING 2024-09-13T13:03:13.728 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[36682]: pgmap v81: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 741 B/s rd, 52 MiB/s wr, 831 op/s 2024-09-13T13:03:13.729 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[36682]: 4.d scrub starts 2024-09-13T13:03:13.729 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[36682]: 4.d scrub ok 2024-09-13T13:03:13.729 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[36682]: 3.19 scrub starts 2024-09-13T13:03:13.729 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[36682]: 3.19 scrub ok 2024-09-13T13:03:13.729 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[36682]: 4.11 scrub starts 2024-09-13T13:03:13.729 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[36682]: 4.11 scrub ok 2024-09-13T13:03:13.729 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[36682]: osdmap e102: 8 total, 8 up, 8 in 2024-09-13T13:03:13.729 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:13.729 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:13.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[29471]: pgmap v81: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 741 B/s rd, 52 MiB/s wr, 831 op/s 2024-09-13T13:03:13.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[29471]: 4.d scrub starts 2024-09-13T13:03:13.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[29471]: 4.d scrub ok 2024-09-13T13:03:13.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[29471]: 3.19 scrub starts 2024-09-13T13:03:13.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[29471]: 3.19 scrub ok 2024-09-13T13:03:13.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[29471]: 4.11 scrub starts 2024-09-13T13:03:13.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[29471]: 4.11 scrub ok 2024-09-13T13:03:13.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[29471]: osdmap e102: 8 total, 8 up, 8 in 2024-09-13T13:03:13.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:13.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:13.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:13.761 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:13.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:13.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:13.805 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:03:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:03:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:03:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:03:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:11913 7.a 7:5f95dae2:::benchmark_data_smithi052_71865_object11912:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:03:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:26.882452+0000", 2024-09-13T13:03:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 46.505497644000002, 2024-09-13T13:03:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.064360618999999994, 2024-09-13T13:03:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 11913 2024-09-13T13:03:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882452+0000", 2024-09-13T13:03:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882452+0000", 2024-09-13T13:03:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882454+0000", 2024-09-13T13:03:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.663e-06 2024-09-13T13:03:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882478+0000", 2024-09-13T13:03:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4311000000000002e-05 2024-09-13T13:03:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882489+0000", 2024-09-13T13:03:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0739e-05 2024-09-13T13:03:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882496+0000", 2024-09-13T13:03:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7800000000000003e-06 2024-09-13T13:03:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882592+0000", 2024-09-13T13:03:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5927999999999994e-05 2024-09-13T13:03:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882706+0000", 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114673 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,5", 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882831+0000", 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012411900000000001 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886617+0000", 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003786583 2024-09-13T13:03:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886768+0000", 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015053899999999999 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946723+0000", 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.059954933000000002 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946774+0000", 2024-09-13T13:03:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1728999999999999e-05 2024-09-13T13:03:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946813+0000", 2024-09-13T13:03:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8621999999999997e-05 2024-09-13T13:03:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:11915 7.a 7:5618ba99:::benchmark_data_smithi052_71865_object11914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:03:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:26.882516+0000", 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 46.505433871999998, 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.064400218999999995, 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 11915 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882516+0000", 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882516+0000", 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882517+0000", 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5300000000000003e-07 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882537+0000", 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0001e-05 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882543+0000", 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5980000000000002e-06 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.882548+0000", 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0100000000000003e-06 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.883653+0000", 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011044760000000001 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.883766+0000", 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011265299999999999 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,5", 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.883902+0000", 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013598500000000001 2024-09-13T13:03:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886623+0000", 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027212669999999999 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.886821+0000", 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000198106 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946845+0000", 2024-09-13T13:03:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.060024474000000001 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946886+0000", 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0438999999999997e-05 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:26.946916+0000", 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0457e-05 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:22476 7.2 7:42f39f32:::benchmark_data_smithi052_71865_object22475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:31.285153+0000", 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 42.102796927999997, 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.062853415999999995, 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 22476 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.285153+0000", 2024-09-13T13:03:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.285153+0000", 2024-09-13T13:03:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.285155+0000", 2024-09-13T13:03:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.429e-06 2024-09-13T13:03:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.285177+0000", 2024-09-13T13:03:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.247e-05 2024-09-13T13:03:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.342183+0000", 2024-09-13T13:03:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.057005866000000002 2024-09-13T13:03:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.342193+0000", 2024-09-13T13:03:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7750000000000004e-06 2024-09-13T13:03:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.342254+0000", 2024-09-13T13:03:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0964000000000001e-05 2024-09-13T13:03:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.342392+0000", 2024-09-13T13:03:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013799299999999999 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.342532+0000", 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000140135 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.346021+0000", 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0034892019999999998 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.346057+0000", 2024-09-13T13:03:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6513000000000001e-05 2024-09-13T13:03:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.347918+0000", 2024-09-13T13:03:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018609950000000001 2024-09-13T13:03:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.347971+0000", 2024-09-13T13:03:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2312999999999999e-05 2024-09-13T13:03:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:31.348006+0000", 2024-09-13T13:03:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5760999999999999e-05 2024-09-13T13:03:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:28935 7.2 7:4a358106:::benchmark_data_smithi052_71865_object28934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-13T13:03:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:34.434304+0000", 2024-09-13T13:03:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 38.953646323000001, 2024-09-13T13:03:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.063293178000000005, 2024-09-13T13:03:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 28935 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434304+0000", 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434304+0000", 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434305+0000", 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5710000000000001e-06 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434330+0000", 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4680000000000001e-05 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434341+0000", 2024-09-13T13:03:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1024e-05 2024-09-13T13:03:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434348+0000", 2024-09-13T13:03:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1799999999999999e-06 2024-09-13T13:03:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434430+0000", 2024-09-13T13:03:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2185e-05 2024-09-13T13:03:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434526+0000", 2024-09-13T13:03:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5249999999999998e-05 2024-09-13T13:03:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:03:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434645+0000", 2024-09-13T13:03:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000119041 2024-09-13T13:03:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.437060+0000", 2024-09-13T13:03:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002415183 2024-09-13T13:03:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.493547+0000", 2024-09-13T13:03:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.056486847999999999 2024-09-13T13:03:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.497317+0000", 2024-09-13T13:03:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037707359999999998 2024-09-13T13:03:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.497553+0000", 2024-09-13T13:03:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000235718 2024-09-13T13:03:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.497597+0000", 2024-09-13T13:03:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3761999999999999e-05 2024-09-13T13:03:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:28938 7.2 7:498382bc:::benchmark_data_smithi052_71865_object28937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:34.434449+0000", 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 38.953501478, 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.066155847000000004, 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 28938 2024-09-13T13:03:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434449+0000", 2024-09-13T13:03:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434449+0000", 2024-09-13T13:03:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434450+0000", 2024-09-13T13:03:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.359e-06 2024-09-13T13:03:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434473+0000", 2024-09-13T13:03:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3023999999999998e-05 2024-09-13T13:03:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434483+0000", 2024-09-13T13:03:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0077e-05 2024-09-13T13:03:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.434489+0000", 2024-09-13T13:03:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4960000000000001e-06 2024-09-13T13:03:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.437182+0000", 2024-09-13T13:03:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026928070000000002 2024-09-13T13:03:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.492201+0000", 2024-09-13T13:03:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.055018682999999999 2024-09-13T13:03:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:03:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.492367+0000", 2024-09-13T13:03:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000166 2024-09-13T13:03:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.497636+0000", 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0052690020000000001 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.497668+0000", 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2443000000000001e-05 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.500527+0000", 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002858644 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.500568+0000", 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0813999999999999e-05 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:34.500604+0000", 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6498000000000002e-05 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:52577 7.3 7:c25c147d:::benchmark_data_smithi052_71865_object52576:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:03:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.952773+0000", 2024-09-13T13:03:13.839 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 24.435176835, 2024-09-13T13:03:13.839 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.073450979, 2024-09-13T13:03:13.839 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.839 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.839 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.839 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:13.839 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:13.839 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:13.839 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 52577 2024-09-13T13:03:13.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.839 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.839 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.839 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.840 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.952773+0000", 2024-09-13T13:03:13.840 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.840 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.840 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.840 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.952773+0000", 2024-09-13T13:03:13.840 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.840 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.840 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.840 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.952775+0000", 2024-09-13T13:03:13.840 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.703e-06 2024-09-13T13:03:13.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.952798+0000", 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2833999999999999e-05 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.952808+0000", 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0017000000000001e-05 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.952814+0000", 2024-09-13T13:03:13.841 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6730000000000003e-06 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.952989+0000", 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017484400000000001 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.955036+0000", 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020471159999999999 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:03:13.842 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.955192+0000", 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015525200000000001 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026091+0000", 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.070899441999999993 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026102+0000", 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0818999999999999e-05 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.843 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026136+0000", 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3816999999999998e-05 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026183+0000", 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7172999999999997e-05 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026224+0000", 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1288999999999997e-05 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.845 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.845 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.845 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.845 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:52579 7.3 7:cf1d4f0b:::benchmark_data_smithi052_71865_object52578:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:03:13.845 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.952838+0000", 2024-09-13T13:03:13.845 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 24.435112406999998, 2024-09-13T13:03:13.845 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.073526280999999999, 2024-09-13T13:03:13.845 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.845 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.845 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.845 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:13.845 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:13.845 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 52579 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.952838+0000", 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.952838+0000", 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.952838+0000", 2024-09-13T13:03:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1800000000000005e-07 2024-09-13T13:03:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.952859+0000", 2024-09-13T13:03:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0230000000000001e-05 2024-09-13T13:03:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.952865+0000", 2024-09-13T13:03:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3459999999999999e-06 2024-09-13T13:03:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.952870+0000", 2024-09-13T13:03:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4620000000000004e-06 2024-09-13T13:03:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957740+0000", 2024-09-13T13:03:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048697310000000004 2024-09-13T13:03:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958013+0000", 2024-09-13T13:03:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027256600000000002 2024-09-13T13:03:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:03:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958132+0000", 2024-09-13T13:03:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000119101 2024-09-13T13:03:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026106+0000", 2024-09-13T13:03:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.067974437999999998 2024-09-13T13:03:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026260+0000", 2024-09-13T13:03:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015401799999999999 2024-09-13T13:03:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026293+0000", 2024-09-13T13:03:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2319e-05 2024-09-13T13:03:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026333+0000", 2024-09-13T13:03:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0759000000000001e-05 2024-09-13T13:03:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026364+0000", 2024-09-13T13:03:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0493000000000001e-05 2024-09-13T13:03:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:52581 7.3 7:c085ce27:::benchmark_data_smithi052_71865_object52580:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.954546+0000", 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 24.433404398, 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.077166910000000005, 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 52581 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.954546+0000", 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.954546+0000", 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.954547+0000", 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5910000000000001e-06 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.954570+0000", 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2697e-05 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.954580+0000", 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0122999999999999e-05 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.954587+0000", 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2609999999999996e-06 2024-09-13T13:03:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.023180+0000", 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.068592852999999995 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.023295+0000", 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011482899999999999 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.023418+0000", 2024-09-13T13:03:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012348 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026400+0000", 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029816320000000001 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.030921+0000", 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045212519999999999 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.030975+0000", 2024-09-13T13:03:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3171000000000003e-05 2024-09-13T13:03:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.031664+0000", 2024-09-13T13:03:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00068910900000000003 2024-09-13T13:03:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.031713+0000", 2024-09-13T13:03:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8912000000000002e-05 2024-09-13T13:03:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:52586 7.a 7:5edaf02d:::benchmark_data_smithi052_71865_object52585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:03:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.956792+0000", 2024-09-13T13:03:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 24.431158475, 2024-09-13T13:03:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.070572977999999995, 2024-09-13T13:03:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 52586 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.956792+0000", 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.956792+0000", 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.956793+0000", 2024-09-13T13:03:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1570000000000001e-06 2024-09-13T13:03:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.956817+0000", 2024-09-13T13:03:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3912000000000001e-05 2024-09-13T13:03:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.956825+0000", 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4819999999999994e-06 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.956831+0000", 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3910000000000003e-06 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957506+0000", 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00067493499999999999 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957709+0000", 2024-09-13T13:03:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000202313 2024-09-13T13:03:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,5", 2024-09-13T13:03:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958387+0000", 2024-09-13T13:03:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00067789900000000004 2024-09-13T13:03:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.024711+0000", 2024-09-13T13:03:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.066323940999999997 2024-09-13T13:03:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.024744+0000", 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3206e-05 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.027284+0000", 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025406410000000002 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.027327+0000", 2024-09-13T13:03:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2256999999999999e-05 2024-09-13T13:03:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.027365+0000", 2024-09-13T13:03:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7843999999999999e-05 2024-09-13T13:03:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:52588 7.3 7:c5e70aaa:::benchmark_data_smithi052_71865_object52587:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:03:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.957194+0000", 2024-09-13T13:03:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 24.430756133999999, 2024-09-13T13:03:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.074806290999999997, 2024-09-13T13:03:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 52588 2024-09-13T13:03:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957194+0000", 2024-09-13T13:03:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957194+0000", 2024-09-13T13:03:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957195+0000", 2024-09-13T13:03:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1540000000000001e-06 2024-09-13T13:03:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957217+0000", 2024-09-13T13:03:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2376999999999999e-05 2024-09-13T13:03:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957225+0000", 2024-09-13T13:03:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5639999999999999e-06 2024-09-13T13:03:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.957231+0000", 2024-09-13T13:03:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2179999999999996e-06 2024-09-13T13:03:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026427+0000", 2024-09-13T13:03:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.069196250000000001 2024-09-13T13:03:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026531+0000", 2024-09-13T13:03:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103522 2024-09-13T13:03:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:03:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026651+0000", 2024-09-13T13:03:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012036699999999999 2024-09-13T13:03:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.031727+0000", 2024-09-13T13:03:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.005075558 2024-09-13T13:03:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.031762+0000", 2024-09-13T13:03:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5364999999999998e-05 2024-09-13T13:03:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.031800+0000", 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7954999999999998e-05 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.031882+0000", 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2000000000000001e-05 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.032000+0000", 2024-09-13T13:03:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117961 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:52591 7.3 7:c0542627:::benchmark_data_smithi052_71865_object52590:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.958124+0000", 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 24.429825997999998, 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.077973423, 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 52591 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958124+0000", 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958124+0000", 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958125+0000", 2024-09-13T13:03:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4640000000000001e-06 2024-09-13T13:03:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958152+0000", 2024-09-13T13:03:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6265999999999999e-05 2024-09-13T13:03:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958166+0000", 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4688999999999999e-05 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958174+0000", 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9049999999999997e-06 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.028638+0000", 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.070464099000000002 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.028739+0000", 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000100789 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.028857+0000", 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117465 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.031772+0000", 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002915071 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.032045+0000", 2024-09-13T13:03:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027313400000000002 2024-09-13T13:03:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.035995+0000", 2024-09-13T13:03:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003950037 2024-09-13T13:03:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.036051+0000", 2024-09-13T13:03:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6215000000000003e-05 2024-09-13T13:03:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.036097+0000", 2024-09-13T13:03:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6288999999999997e-05 2024-09-13T13:03:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:52577 7.3 e95/84 ondisk, result = 0)", 2024-09-13T13:03:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.958124+0000", 2024-09-13T13:03:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 24.429826129999999, 2024-09-13T13:03:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.067982998000000003, 2024-09-13T13:03:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:03:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958124+0000", 2024-09-13T13:03:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958124+0000", 2024-09-13T13:03:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958125+0000", 2024-09-13T13:03:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9900000000000009e-07 2024-09-13T13:03:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958129+0000", 2024-09-13T13:03:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4730000000000002e-06 2024-09-13T13:03:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958140+0000", 2024-09-13T13:03:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0457e-05 2024-09-13T13:03:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958147+0000", 2024-09-13T13:03:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8719999999999996e-06 2024-09-13T13:03:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026078+0000", 2024-09-13T13:03:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.067931536000000001 2024-09-13T13:03:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026085+0000", 2024-09-13T13:03:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.939e-06 2024-09-13T13:03:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026107+0000", 2024-09-13T13:03:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1722e-05 2024-09-13T13:03:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:52592 7.a 7:5f6efd2d:::benchmark_data_smithi052_71865_object52591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:03:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.958189+0000", 2024-09-13T13:03:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 24.429760939000001, 2024-09-13T13:03:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.077913403000000006, 2024-09-13T13:03:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 52592 2024-09-13T13:03:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958189+0000", 2024-09-13T13:03:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958189+0000", 2024-09-13T13:03:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958190+0000", 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4499999999999996e-07 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958215+0000", 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5199e-05 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958223+0000", 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8339999999999999e-06 2024-09-13T13:03:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958230+0000", 2024-09-13T13:03:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5919999999999997e-06 2024-09-13T13:03:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.022472+0000", 2024-09-13T13:03:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.06424291 2024-09-13T13:03:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.022623+0000", 2024-09-13T13:03:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000150555 2024-09-13T13:03:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,5", 2024-09-13T13:03:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.022852+0000", 2024-09-13T13:03:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00022940999999999999 2024-09-13T13:03:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.025030+0000", 2024-09-13T13:03:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0021775420000000002 2024-09-13T13:03:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.029741+0000", 2024-09-13T13:03:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047113370000000003 2024-09-13T13:03:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.036020+0000", 2024-09-13T13:03:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0062783520000000001 2024-09-13T13:03:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.036065+0000", 2024-09-13T13:03:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5540000000000001e-05 2024-09-13T13:03:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.036102+0000", 2024-09-13T13:03:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7286999999999997e-05 2024-09-13T13:03:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:52579 7.3 e95/84 ondisk, result = 0)", 2024-09-13T13:03:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.960597+0000", 2024-09-13T13:03:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 24.427353423, 2024-09-13T13:03:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.065770782999999999, 2024-09-13T13:03:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:03:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.960597+0000", 2024-09-13T13:03:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.960597+0000", 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.960598+0000", 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5060000000000001e-06 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.960603+0000", 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4259999999999996e-06 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.022790+0000", 2024-09-13T13:03:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.062187226999999998 2024-09-13T13:03:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.022798+0000", 2024-09-13T13:03:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3659999999999992e-06 2024-09-13T13:03:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026284+0000", 2024-09-13T13:03:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0034854880000000001 2024-09-13T13:03:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026289+0000", 2024-09-13T13:03:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1739999999999999e-06 2024-09-13T13:03:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026367+0000", 2024-09-13T13:03:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8596000000000004e-05 2024-09-13T13:03:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:52593 7.6 e95/84 7:6eabb69d:::benchmark_data_smithi052_71865_object52592:head v 95'3352, mlcod=95'3350)", 2024-09-13T13:03:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.961043+0000", 2024-09-13T13:03:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 24.426906973000001, 2024-09-13T13:03:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.062561438999999996, 2024-09-13T13:03:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.961043+0000", 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.961043+0000", 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.961044+0000", 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1999999999999999e-06 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.961069+0000", 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4757999999999999e-05 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.020859+0000", 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.059789515000000001 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.020870+0000", 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1443999999999999e-05 2024-09-13T13:03:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.020929+0000", 2024-09-13T13:03:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9373000000000002e-05 2024-09-13T13:03:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.020961+0000", 2024-09-13T13:03:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1472e-05 2024-09-13T13:03:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.023548+0000", 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025870149999999998 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.023604+0000", 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6662000000000001e-05 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:52577 7.3 e95/84 ondisk, result = 0)", 2024-09-13T13:03:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.961200+0000", 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 24.426749940000001, 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.065029383999999996, 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.961200+0000", 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.961200+0000", 2024-09-13T13:03:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.961201+0000", 2024-09-13T13:03:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0440000000000001e-06 2024-09-13T13:03:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.961205+0000", 2024-09-13T13:03:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.335e-06 2024-09-13T13:03:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.020854+0000", 2024-09-13T13:03:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.059648158999999999 2024-09-13T13:03:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.020868+0000", 2024-09-13T13:03:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4049000000000001e-05 2024-09-13T13:03:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026126+0000", 2024-09-13T13:03:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0052584800000000003 2024-09-13T13:03:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026131+0000", 2024-09-13T13:03:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.339e-06 2024-09-13T13:03:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.026229+0000", 2024-09-13T13:03:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7978000000000003e-05 2024-09-13T13:03:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:81742 7.7 e101/84 7:ee739748:::benchmark_data_smithi052_71865_object81741:head v 101'5163, mlcod=101'5160)", 2024-09-13T13:03:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:12.469304+0000", 2024-09-13T13:03:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.91864603199999995, 2024-09-13T13:03:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.075122159999999993, 2024-09-13T13:03:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.469304+0000", 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.469304+0000", 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.469305+0000", 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0240000000000001e-06 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.469357+0000", 2024-09-13T13:03:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2033000000000001e-05 2024-09-13T13:03:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.469367+0000", 2024-09-13T13:03:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0263e-05 2024-09-13T13:03:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.469374+0000", 2024-09-13T13:03:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0500000000000003e-06 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.471331+0000", 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019565559999999999 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.471351+0000", 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9768000000000001e-05 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.544245+0000", 2024-09-13T13:03:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.072894308000000005 2024-09-13T13:03:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.544426+0000", 2024-09-13T13:03:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000181158 2024-09-13T13:03:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:81746 7.6 e101/84 7:6d98524e:::benchmark_data_smithi052_71865_object81745:head v 101'5092, mlcod=101'5090)", 2024-09-13T13:03:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:12.472243+0000", 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.91570723300000001, 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.063566167000000007, 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.472243+0000", 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.472243+0000", 2024-09-13T13:03:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.472245+0000", 2024-09-13T13:03:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7579999999999999e-06 2024-09-13T13:03:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.472266+0000", 2024-09-13T13:03:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1107999999999998e-05 2024-09-13T13:03:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.472274+0000", 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6279999999999994e-06 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.472281+0000", 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3289999999999999e-06 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.472362+0000", 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1692000000000003e-05 2024-09-13T13:03:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.472380+0000", 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7597000000000001e-05 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.475119+0000", 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027387480000000001 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.535809+0000", 2024-09-13T13:03:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.060690306999999999 2024-09-13T13:03:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:81751 7.2 7:488e80a4:::benchmark_data_smithi052_71865_object81750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-13T13:03:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:12.475609+0000", 2024-09-13T13:03:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.91234078500000004, 2024-09-13T13:03:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.074688003000000003, 2024-09-13T13:03:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 81751 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.475609+0000", 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.475609+0000", 2024-09-13T13:03:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.475611+0000", 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7320000000000001e-06 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.475630+0000", 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8607999999999999e-05 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.535757+0000", 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.060127710000000001 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.535767+0000", 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6730000000000004e-06 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.535849+0000", 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2175000000000005e-05 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.536003+0000", 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000153974 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.542579+0000", 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0065756959999999998 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.550090+0000", 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0075116979999999998 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.550105+0000", 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4448e-05 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.550183+0000", 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7918999999999997e-05 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.550254+0000", 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0949000000000001e-05 2024-09-13T13:03:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.550297+0000", 2024-09-13T13:03:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3421000000000003e-05 2024-09-13T13:03:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:81736 7.f e101/84 7:f971c8ab:::benchmark_data_smithi052_71865_object81735:head v 101'5127, mlcod=101'5125)", 2024-09-13T13:03:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:12.477387+0000", 2024-09-13T13:03:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.91056275900000005, 2024-09-13T13:03:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.067072544999999997, 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.477387+0000", 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.477387+0000", 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.477389+0000", 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.477413+0000", 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4260000000000002e-05 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.535721+0000", 2024-09-13T13:03:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.058307491000000003 2024-09-13T13:03:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.535737+0000", 2024-09-13T13:03:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6368999999999999e-05 2024-09-13T13:03:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.536262+0000", 2024-09-13T13:03:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00052508199999999996 2024-09-13T13:03:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.536283+0000", 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0795e-05 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.544431+0000", 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0081482640000000005 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:12.544460+0000", 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8744000000000001e-05 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.922 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:13.922 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:13.922 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:03:13.922 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:13.958 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:13.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:13.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:13.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T13:03:13.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:13] ENGINE Bus STOPPED 2024-09-13T13:03:13.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:13] ENGINE Bus STARTING 2024-09-13T13:03:13.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:13] ENGINE Serving on http://:::9283 2024-09-13T13:03:13.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:13] ENGINE Bus STARTED 2024-09-13T13:03:13.989 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:13 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:13] ENGINE Bus STOPPING 2024-09-13T13:03:14.060 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:14.103 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:14.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:14.161 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 52 16 83578 83562 100.385 64.875 0.0103431 0.00995525 2024-09-13T13:03:14.205 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:14.235 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:14.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:14.337 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:14.377 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:14.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:14.479 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:14.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:14.515 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:14.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:14.616 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:14.639 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:14.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-13T13:03:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T13:03:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi052.front.sepia.ceph.com:9093"} : dispatch 2024-09-13T13:03:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi052.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T13:03:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-13T13:03:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T13:03:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi167.front.sepia.ceph.com:9095"} : dispatch 2024-09-13T13:03:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi167.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T13:03:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:14.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-13T13:03:14.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi167.front.sepia.ceph.com:3000"} : dispatch 2024-09-13T13:03:14.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: osd.6 marked itself down and dead 2024-09-13T13:03:14.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:14.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:03:14.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:03:14.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:03:14.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T13:03:14.740 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:14] ENGINE Bus STOPPED 2024-09-13T13:03:14.741 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:14] ENGINE Bus STARTING 2024-09-13T13:03:14.741 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:14] ENGINE Serving on http://:::9283 2024-09-13T13:03:14.741 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:14] ENGINE Bus STARTED 2024-09-13T13:03:14.741 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:14] ENGINE Bus STOPPING 2024-09-13T13:03:14.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-13T13:03:14.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T13:03:14.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi052.front.sepia.ceph.com:9093"} : dispatch 2024-09-13T13:03:14.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi052.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T13:03:14.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:14.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-13T13:03:14.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T13:03:14.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi167.front.sepia.ceph.com:9095"} : dispatch 2024-09-13T13:03:14.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi167.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T13:03:14.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:14.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-13T13:03:14.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi167.front.sepia.ceph.com:3000"} : dispatch 2024-09-13T13:03:14.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: osd.6 marked itself down and dead 2024-09-13T13:03:14.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:14.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:03:14.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:03:14.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:03:14.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:14.765 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:14.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:14.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:14.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:03:14.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-13T13:03:14.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T13:03:14.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi052.front.sepia.ceph.com:9093"} : dispatch 2024-09-13T13:03:14.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi052.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T13:03:14.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:14.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-13T13:03:14.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T13:03:14.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi167.front.sepia.ceph.com:9095"} : dispatch 2024-09-13T13:03:14.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi167.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T13:03:14.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:14.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-13T13:03:14.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi167.front.sepia.ceph.com:3000"} : dispatch 2024-09-13T13:03:14.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: osd.6 marked itself down and dead 2024-09-13T13:03:14.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:14.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:03:14.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:03:14.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:14 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:03:14.867 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:14.891 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:14.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:14.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:15.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:15.015 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:15.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:15.073 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T13:03:15.073 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:14] ENGINE Bus STOPPED 2024-09-13T13:03:15.073 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:14 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:14] ENGINE Bus STARTING 2024-09-13T13:03:15.073 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:15] ENGINE Serving on http://:::9283 2024-09-13T13:03:15.073 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:03:15] ENGINE Bus STARTED 2024-09-13T13:03:15.116 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:15.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:15.139 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:15.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:15.161 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 53 16 83989 83973 98.9758 25.6875 0.0032924 0.00998296 2024-09-13T13:03:15.240 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:15.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:15.263 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:15.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:15.364 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:15.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:15.386 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:15.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:15.421 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:15.487 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:15.510 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:15.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:15.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:15.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T13:03:15.770 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi167.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T13:03:15.771 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:15.771 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:03:15.771 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: pgmap v83: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.6 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-09-13T13:03:15.771 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: 4.7 scrub starts 2024-09-13T13:03:15.771 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: 4.7 scrub ok 2024-09-13T13:03:15.771 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: 5.6 scrub starts 2024-09-13T13:03:15.771 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: 5.6 scrub ok 2024-09-13T13:03:15.771 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:03:15.771 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: 4.17 scrub starts 2024-09-13T13:03:15.771 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: 4.17 scrub ok 2024-09-13T13:03:15.771 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T13:03:15.772 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: osdmap e103: 8 total, 7 up, 8 in 2024-09-13T13:03:15.772 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:03:15.772 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: 5.d scrub starts 2024-09-13T13:03:15.772 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[36682]: 5.d scrub ok 2024-09-13T13:03:15.772 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:15.773 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T13:03:15.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi167.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T13:03:15.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:15.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:03:15.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: pgmap v83: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.6 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-09-13T13:03:15.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: 4.7 scrub starts 2024-09-13T13:03:15.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: 4.7 scrub ok 2024-09-13T13:03:15.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: 5.6 scrub starts 2024-09-13T13:03:15.775 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: 5.6 scrub ok 2024-09-13T13:03:15.775 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:03:15.775 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: 4.17 scrub starts 2024-09-13T13:03:15.775 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: 4.17 scrub ok 2024-09-13T13:03:15.775 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T13:03:15.775 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: osdmap e103: 8 total, 7 up, 8 in 2024-09-13T13:03:15.775 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:03:15.775 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: 5.d scrub starts 2024-09-13T13:03:15.776 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:15 smithi052 ceph-mon[29471]: 5.d scrub ok 2024-09-13T13:03:15.797 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T13:03:15.797 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi167.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T13:03:15.797 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:15.797 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:03:15.797 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: pgmap v83: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.6 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-09-13T13:03:15.798 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: 4.7 scrub starts 2024-09-13T13:03:15.798 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: 4.7 scrub ok 2024-09-13T13:03:15.798 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: 5.6 scrub starts 2024-09-13T13:03:15.798 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: 5.6 scrub ok 2024-09-13T13:03:15.798 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:03:15.798 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: 4.17 scrub starts 2024-09-13T13:03:15.798 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: 4.17 scrub ok 2024-09-13T13:03:15.799 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T13:03:15.799 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: osdmap e103: 8 total, 7 up, 8 in 2024-09-13T13:03:15.799 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:03:15.799 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: 5.d scrub starts 2024-09-13T13:03:15.799 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:15 smithi167 ceph-mon[33178]: 5.d scrub ok 2024-09-13T13:03:15.801 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:15.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:15.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:15.903 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:15.958 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:15.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:15.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:16.060 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:16.070 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:15 smithi167 podman[66476]: 2024-09-13 13:03:15.795992469 +0000 UTC m=+2.389669225 container died 69e88afd58cf271b574572058b79362c6e8a8eedde0cab9d00a75c9ae70856f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-13T13:03:16.096 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:16.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:16.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:16.163 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 54 16 84661 84645 97.921 42 0.0192087 0.010207 2024-09-13T13:03:16.198 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:16.256 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:16.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:16.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:16.357 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:16.383 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:16.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:16.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:16.482 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:16 smithi167 podman[66476]: 2024-09-13 13:03:16.204645128 +0000 UTC m=+2.798321891 container cleanup 69e88afd58cf271b574572058b79362c6e8a8eedde0cab9d00a75c9ae70856f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-13T13:03:16.482 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:16 smithi167 bash[66476]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6 2024-09-13T13:03:16.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:16.508 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:16.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:16.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:16.541 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:03:16.544 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:03:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[76921]: ts=2024-09-13T13:03:16.262Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003871602s 2024-09-13T13:03:16.579 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:03:16.610 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:16.649 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:16.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:16.751 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:16.796 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:16.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:16.812 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:16 smithi052 ceph-mon[29471]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:03:16.813 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:16 smithi052 ceph-mon[29471]: 3.17 scrub starts 2024-09-13T13:03:16.813 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:16 smithi052 ceph-mon[29471]: 3.17 scrub ok 2024-09-13T13:03:16.813 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:16 smithi052 ceph-mon[29471]: osdmap e104: 8 total, 7 up, 8 in 2024-09-13T13:03:16.813 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:16 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3878258693' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:16.815 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:16 smithi052 ceph-mon[36682]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:03:16.815 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:16 smithi052 ceph-mon[36682]: 3.17 scrub starts 2024-09-13T13:03:16.815 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:16 smithi052 ceph-mon[36682]: 3.17 scrub ok 2024-09-13T13:03:16.815 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:16 smithi052 ceph-mon[36682]: osdmap e104: 8 total, 7 up, 8 in 2024-09-13T13:03:16.815 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:16 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3878258693' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:16.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:16 smithi167 ceph-mon[33178]: from='mon.? v1:172.21.15.52:0/825563574' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T13:03:16.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:16 smithi167 ceph-mon[33178]: 3.17 scrub starts 2024-09-13T13:03:16.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:16 smithi167 ceph-mon[33178]: 3.17 scrub ok 2024-09-13T13:03:16.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:16 smithi167 ceph-mon[33178]: osdmap e104: 8 total, 7 up, 8 in 2024-09-13T13:03:16.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:16 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3878258693' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:16.821 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:16 smithi167 podman[66733]: 2024-09-13 13:03:16.481834164 +0000 UTC m=+1.413895072 container remove 69e88afd58cf271b574572058b79362c6e8a8eedde0cab9d00a75c9ae70856f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-13T13:03:16.898 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:16.943 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:16.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:16.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:17.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:17.071 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:17.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:17.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:17.138 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:03:17.138 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:03:17.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.34781.0:85735 7.8 e105/84 7:1063e54a:::benchmark_data_smithi052_71865_object85734:head v 105'5424, mlcod=105'5421)", 2024-09-13T13:03:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:17.133177+0000", 2024-09-13T13:03:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0033411809999999999, 2024-09-13T13:03:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033679270000000002, 2024-09-13T13:03:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:03:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.133177+0000", 2024-09-13T13:03:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.133177+0000", 2024-09-13T13:03:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.133178+0000", 2024-09-13T13:03:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7230000000000001e-06 2024-09-13T13:03:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.133291+0000", 2024-09-13T13:03:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112105 2024-09-13T13:03:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.133305+0000", 2024-09-13T13:03:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4901e-05 2024-09-13T13:03:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.133313+0000", 2024-09-13T13:03:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9740000000000002e-06 2024-09-13T13:03:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.133381+0000", 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7978000000000005e-05 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.133402+0000", 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0177e-05 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.34781.0:85736 7.8 e105/84 7:111b9a41:::benchmark_data_smithi052_71865_object85735:head v 105'5425, mlcod=105'5421)", 2024-09-13T13:03:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:17.135329+0000", 2024-09-13T13:03:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00118928, 2024-09-13T13:03:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001249113, 2024-09-13T13:03:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:03:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.135329+0000", 2024-09-13T13:03:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.135329+0000", 2024-09-13T13:03:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.135331+0000", 2024-09-13T13:03:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9549999999999999e-06 2024-09-13T13:03:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.135418+0000", 2024-09-13T13:03:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7849999999999994e-05 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.135437+0000", 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9059000000000002e-05 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.135446+0000", 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5169999999999999e-06 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.135919+0000", 2024-09-13T13:03:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00047284300000000003 2024-09-13T13:03:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.136008+0000", 2024-09-13T13:03:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8769000000000003e-05 2024-09-13T13:03:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.34781.0:85727 7.4 e105/84 7:21306d6a:::benchmark_data_smithi052_71865_object85726:head v 105'5364, mlcod=105'5362)", 2024-09-13T13:03:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:17.136151+0000", 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00036709699999999999, 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045354899999999998, 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.136151+0000", 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.136151+0000", 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.136152+0000", 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2699999999999999e-06 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.136198+0000", 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5905000000000002e-05 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.136208+0000", 2024-09-13T13:03:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8120000000000002e-06 2024-09-13T13:03:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.136216+0000", 2024-09-13T13:03:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.755e-06 2024-09-13T13:03:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.136269+0000", 2024-09-13T13:03:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2374999999999999e-05 2024-09-13T13:03:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:17.136288+0000", 2024-09-13T13:03:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9491999999999999e-05 2024-09-13T13:03:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:03:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 3 2024-09-13T13:03:17.150 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:03:17.162 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 55 16 85763 85747 97.3931 68.875 0.0173324 0.0102617 2024-09-13T13:03:17.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:17.208 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:17.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:17.309 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:17.335 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:17.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:17.361 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:03:17.437 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:17.460 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:17.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:17.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:17.562 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:17.628 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:17.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:17.729 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:17.808 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:17.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:17.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:17 smithi167 ceph-mon[33178]: pgmap v86: 148 pgs: 11 stale+active+clean, 137 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 931 op/s 2024-09-13T13:03:17.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:17 smithi167 ceph-mon[33178]: 4.b scrub starts 2024-09-13T13:03:17.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:17 smithi167 ceph-mon[33178]: 4.b scrub ok 2024-09-13T13:03:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:17 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:03:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:17 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3878258693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:17 smithi167 ceph-mon[33178]: osdmap e105: 8 total, 7 up, 8 in 2024-09-13T13:03:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:17 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:17 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:03:17.868 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:17.897 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T13:03:17.914 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:17.976 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:17.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[36682]: pgmap v86: 148 pgs: 11 stale+active+clean, 137 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 931 op/s 2024-09-13T13:03:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[36682]: 4.b scrub starts 2024-09-13T13:03:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[36682]: 4.b scrub ok 2024-09-13T13:03:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:03:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3878258693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[36682]: osdmap e105: 8 total, 7 up, 8 in 2024-09-13T13:03:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:03:17.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[29471]: pgmap v86: 148 pgs: 11 stale+active+clean, 137 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 931 op/s 2024-09-13T13:03:17.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[29471]: 4.b scrub starts 2024-09-13T13:03:17.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[29471]: 4.b scrub ok 2024-09-13T13:03:17.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:03:17.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3878258693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:17.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[29471]: osdmap e105: 8 total, 7 up, 8 in 2024-09-13T13:03:17.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:17.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:17 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:03:18.077 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:18.124 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:18.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:18.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:18.163 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 56 16 86812 86796 96.8248 65.5625 0.00814627 0.0103231 2024-09-13T13:03:18.226 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:18.250 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:18.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:18.352 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:18.397 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:18.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:18.499 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:18.529 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:03:18.529 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:03:18.569 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:03:18.569 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:03:18.570 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:18 smithi167 podman[67162]: 2024-09-13 13:03:18.041525615 +0000 UTC m=+0.027945136 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:03:18.570 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:18 smithi167 podman[67162]: 2024-09-13 13:03:18.178368999 +0000 UTC m=+0.164788505 container create 2ad278ae5db7505baef6e29b24bb4f02c86907196ca11f47f4afcd718718445f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-13T13:03:18.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:18.572 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:18.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:18.611 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:03:18.611 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:03:18.646 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:03:18.646 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:03:18.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:18.678 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:03:18.678 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:03:18.697 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:18.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:18.712 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:03:18.712 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:03:18.713 INFO:teuthology.orchestra.run.smithi052.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T13:03:18.745 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:03:18.745 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:03:18.800 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:18.840 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:18.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:18.942 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:18.944 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:03:18.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:18.986 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:18.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[36682]: osdmap e106: 8 total, 7 up, 8 in 2024-09-13T13:03:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[36682]: pgmap v89: 148 pgs: 14 active+undersized, 6 stale+active+clean, 16 active+undersized+degraded, 112 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 767 B/s rd, 13 MiB/s wr, 209 op/s; 5486/249729 objects degraded (2.197%) 2024-09-13T13:03:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[36682]: 2.1 scrub starts 2024-09-13T13:03:18.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[36682]: 2.1 scrub ok 2024-09-13T13:03:18.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2131748737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:18.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[36682]: 3.7 scrub starts 2024-09-13T13:03:18.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[36682]: 3.7 scrub ok 2024-09-13T13:03:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[29471]: osdmap e106: 8 total, 7 up, 8 in 2024-09-13T13:03:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[29471]: pgmap v89: 148 pgs: 14 active+undersized, 6 stale+active+clean, 16 active+undersized+degraded, 112 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 767 B/s rd, 13 MiB/s wr, 209 op/s; 5486/249729 objects degraded (2.197%) 2024-09-13T13:03:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[29471]: 2.1 scrub starts 2024-09-13T13:03:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[29471]: 2.1 scrub ok 2024-09-13T13:03:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2131748737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[29471]: 3.7 scrub starts 2024-09-13T13:03:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:18 smithi052 ceph-mon[29471]: 3.7 scrub ok 2024-09-13T13:03:19.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:18 smithi167 ceph-mon[33178]: osdmap e106: 8 total, 7 up, 8 in 2024-09-13T13:03:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:18 smithi167 ceph-mon[33178]: pgmap v89: 148 pgs: 14 active+undersized, 6 stale+active+clean, 16 active+undersized+degraded, 112 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 767 B/s rd, 13 MiB/s wr, 209 op/s; 5486/249729 objects degraded (2.197%) 2024-09-13T13:03:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:18 smithi167 ceph-mon[33178]: 2.1 scrub starts 2024-09-13T13:03:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:18 smithi167 ceph-mon[33178]: 2.1 scrub ok 2024-09-13T13:03:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:18 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2131748737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:18 smithi167 ceph-mon[33178]: 3.7 scrub starts 2024-09-13T13:03:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:18 smithi167 ceph-mon[33178]: 3.7 scrub ok 2024-09-13T13:03:19.071 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:18 smithi167 podman[67162]: 2024-09-13 13:03:18.707179475 +0000 UTC m=+0.693598989 container init 2ad278ae5db7505baef6e29b24bb4f02c86907196ca11f47f4afcd718718445f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8) 2024-09-13T13:03:19.071 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:18 smithi167 podman[67162]: 2024-09-13 13:03:18.711706675 +0000 UTC m=+0.698126182 container start 2ad278ae5db7505baef6e29b24bb4f02c86907196ca11f47f4afcd718718445f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-deactivate, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.build-date=20240731) 2024-09-13T13:03:19.071 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:18 smithi167 podman[67162]: 2024-09-13 13:03:18.890808205 +0000 UTC m=+0.877227718 container attach 2ad278ae5db7505baef6e29b24bb4f02c86907196ca11f47f4afcd718718445f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.build-date=20240731) 2024-09-13T13:03:19.088 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:19.130 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:19.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:19.162 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 57 16 87848 87832 96.2609 64.75 0.0133933 0.0103827 2024-09-13T13:03:19.231 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:19.279 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:19.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:19.380 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:19.407 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:19.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:19.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:03:19.477 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:03:19.508 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:19.550 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:19.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:19.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:19.652 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:19.677 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:19.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:19.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:19.722 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:03:19.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:19 smithi052 ceph-mon[29471]: 4.6 scrub starts 2024-09-13T13:03:19.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:19 smithi052 ceph-mon[29471]: 4.6 scrub ok 2024-09-13T13:03:19.780 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:19.807 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:19.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:19.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:19.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:19 smithi167 ceph-mon[33178]: 4.6 scrub starts 2024-09-13T13:03:19.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:19 smithi167 ceph-mon[33178]: 4.6 scrub ok 2024-09-13T13:03:19.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:19 smithi167 ceph-mon[33178]: Health check failed: Degraded data redundancy: 5486/249729 objects degraded (2.197%), 16 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:19.874 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:03:19.874 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:03:19.909 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:19.932 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:19.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:19.976 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:19 smithi052 ceph-mon[36682]: 4.6 scrub starts 2024-09-13T13:03:19.977 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:19 smithi052 ceph-mon[36682]: 4.6 scrub ok 2024-09-13T13:03:19.977 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:19 smithi052 ceph-mon[36682]: Health check failed: Degraded data redundancy: 5486/249729 objects degraded (2.197%), 16 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:19.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:19 smithi052 ceph-mon[29471]: Health check failed: Degraded data redundancy: 5486/249729 objects degraded (2.197%), 16 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:20.034 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:20.082 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:20.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:20.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:20.162 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 58 15 88955 88940 95.7952 69.25 0.0155184 0.0104335 2024-09-13T13:03:20.185 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:20.214 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:20.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:20.316 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:20.362 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:20.385 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:20.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:20.474 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:20 smithi167 podman[67297]: 2024-09-13 13:03:20.1344517 +0000 UTC m=+1.167765813 container died 2ad278ae5db7505baef6e29b24bb4f02c86907196ca11f47f4afcd718718445f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T13:03:20.487 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:20.512 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:20.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:20.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:20.615 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:20.664 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:20.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:20.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:20.779 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:20 smithi052 ceph-mon[36682]: 4.1c scrub starts 2024-09-13T13:03:20.779 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:20 smithi052 ceph-mon[36682]: 4.1c scrub ok 2024-09-13T13:03:20.779 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:20 smithi052 ceph-mon[36682]: pgmap v90: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 380 B/s rd, 50 MiB/s wr, 805 op/s; 27326/260208 objects degraded (10.502%); 12 KiB/s, 0 objects/s recovering 2024-09-13T13:03:20.779 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:20 smithi052 ceph-mon[36682]: 6.b scrub starts 2024-09-13T13:03:20.779 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:20 smithi052 ceph-mon[36682]: 6.b scrub ok 2024-09-13T13:03:20.780 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:20 smithi052 ceph-mon[29471]: 4.1c scrub starts 2024-09-13T13:03:20.780 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:20 smithi052 ceph-mon[29471]: 4.1c scrub ok 2024-09-13T13:03:20.780 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:20 smithi052 ceph-mon[29471]: pgmap v90: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 380 B/s rd, 50 MiB/s wr, 805 op/s; 27326/260208 objects degraded (10.502%); 12 KiB/s, 0 objects/s recovering 2024-09-13T13:03:20.780 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:20 smithi052 ceph-mon[29471]: 6.b scrub starts 2024-09-13T13:03:20.780 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:20 smithi052 ceph-mon[29471]: 6.b scrub ok 2024-09-13T13:03:20.789 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:20.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:20.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:20.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.6.service' 2024-09-13T13:03:20.891 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:20.932 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:20.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:20.936 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:03:20.937 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-13T13:03:20.937 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2024-09-13T13:03:20.937 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi167.front.sepia.ceph.com 2024-09-13T13:03:20.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -n osd.6 --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2024-09-13T13:03:21.002 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:20 smithi167 ceph-mon[33178]: 4.1c scrub starts 2024-09-13T13:03:21.002 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:20 smithi167 ceph-mon[33178]: 4.1c scrub ok 2024-09-13T13:03:21.002 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:20 smithi167 ceph-mon[33178]: pgmap v90: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 380 B/s rd, 50 MiB/s wr, 805 op/s; 27326/260208 objects degraded (10.502%); 12 KiB/s, 0 objects/s recovering 2024-09-13T13:03:21.002 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:20 smithi167 ceph-mon[33178]: 6.b scrub starts 2024-09-13T13:03:21.002 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:20 smithi167 ceph-mon[33178]: 6.b scrub ok 2024-09-13T13:03:21.034 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:21.059 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:21.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:21.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:21.162 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:21.163 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 59 16 90033 90017 95.3117 67.3125 0.0080021 0.0104871 2024-09-13T13:03:21.208 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:21.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:21.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:21.310 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:21.335 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:21.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:21.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:21.437 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:21.479 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:03:21.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:21.481 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:21.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:21.582 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:21.606 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:21.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:21.707 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:21.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T13:03:21.733 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:21.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:21.836 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:21.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:21.890 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:21.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:21.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:21 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:03:21.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:21 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:03:21.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:21 smithi167 ceph-mon[33178]: osdmap e107: 8 total, 7 up, 8 in 2024-09-13T13:03:21.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:21 smithi167 ceph-mon[33178]: 6.14 scrub starts 2024-09-13T13:03:21.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:21 smithi167 ceph-mon[33178]: 6.14 scrub ok 2024-09-13T13:03:21.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:21 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:21.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:21 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:21.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:03:21.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:03:21.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[36682]: osdmap e107: 8 total, 7 up, 8 in 2024-09-13T13:03:21.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[36682]: 6.14 scrub starts 2024-09-13T13:03:21.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[36682]: 6.14 scrub ok 2024-09-13T13:03:21.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:21.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:21.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:03:21.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:03:21.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[29471]: osdmap e107: 8 total, 7 up, 8 in 2024-09-13T13:03:21.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[29471]: 6.14 scrub starts 2024-09-13T13:03:21.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[29471]: 6.14 scrub ok 2024-09-13T13:03:21.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:21.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:21 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:21.991 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:22.030 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:22.046 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:22.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:22.151 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:22.163 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:03:22.163728+0000 min lat: 0.00146492 max lat: 1.03648 avg lat: 0.0105462 2024-09-13T13:03:22.163 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:03:22.163 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 60 16 91035 91019 94.767 62.625 0.0044735 0.0105462 2024-09-13T13:03:22.199 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:22.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:22.301 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:22.325 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:22.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:22.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:22.428 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:22.454 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:22.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:22.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:22.464 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/osd.6/config 2024-09-13T13:03:22.520 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:03:22.557 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:22.607 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:22.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:22.708 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:22.726 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:03:22.727 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:03:22.728 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:03:22.753 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:22.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:22.776 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:03:22.777 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:03:22.777 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/697127019' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:03:22.777 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[36682]: from='client.24877 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:03:22.777 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[36682]: pgmap v92: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 340 B/s rd, 45 MiB/s wr, 721 op/s; 27326/260208 objects degraded (10.502%); 11 KiB/s, 0 objects/s recovering 2024-09-13T13:03:22.777 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[36682]: 6.f scrub starts 2024-09-13T13:03:22.777 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[36682]: 6.f scrub ok 2024-09-13T13:03:22.777 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[36682]: 4.0 scrub starts 2024-09-13T13:03:22.778 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[36682]: 4.0 scrub ok 2024-09-13T13:03:22.778 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1468752974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:22.778 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:22.778 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[36682]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:03:22.778 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[36682]: osdmap e108: 8 total, 7 up, 8 in 2024-09-13T13:03:22.778 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/697127019' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:03:22.778 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[29471]: from='client.24877 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:03:22.778 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[29471]: pgmap v92: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 340 B/s rd, 45 MiB/s wr, 721 op/s; 27326/260208 objects degraded (10.502%); 11 KiB/s, 0 objects/s recovering 2024-09-13T13:03:22.778 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[29471]: 6.f scrub starts 2024-09-13T13:03:22.778 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[29471]: 6.f scrub ok 2024-09-13T13:03:22.778 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[29471]: 4.0 scrub starts 2024-09-13T13:03:22.779 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[29471]: 4.0 scrub ok 2024-09-13T13:03:22.779 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1468752974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:22.779 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:22.779 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[29471]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:03:22.779 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:22 smithi052 ceph-mon[29471]: osdmap e108: 8 total, 7 up, 8 in 2024-09-13T13:03:22.819 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:03:22.820 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:03:22.855 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:22.866 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:03:22.866 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:03:22.874 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:22 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/697127019' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:03:22.874 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:22 smithi167 ceph-mon[33178]: from='client.24877 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:03:22.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:22 smithi167 ceph-mon[33178]: pgmap v92: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 340 B/s rd, 45 MiB/s wr, 721 op/s; 27326/260208 objects degraded (10.502%); 11 KiB/s, 0 objects/s recovering 2024-09-13T13:03:22.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:22 smithi167 ceph-mon[33178]: 6.f scrub starts 2024-09-13T13:03:22.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:22 smithi167 ceph-mon[33178]: 6.f scrub ok 2024-09-13T13:03:22.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:22 smithi167 ceph-mon[33178]: 4.0 scrub starts 2024-09-13T13:03:22.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:22 smithi167 ceph-mon[33178]: 4.0 scrub ok 2024-09-13T13:03:22.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:22 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1468752974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:22.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:22 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:22.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:22 smithi167 ceph-mon[33178]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:03:22.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:22 smithi167 ceph-mon[33178]: osdmap e108: 8 total, 7 up, 8 in 2024-09-13T13:03:22.898 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:22.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:22.900 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:03:22.900 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:03:22.931 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:03:22.932 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:03:22.936 INFO:teuthology.orchestra.run.smithi052.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T13:03:22.969 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:03:22.969 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:03:23.001 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:23.038 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:23.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:23.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:23.160 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:03:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=infra.usagestats t=2024-09-13T13:03:23.119555032Z level=info msg="Usage stats are ready to report" 2024-09-13T13:03:23.163 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 61 16 92096 92080 94.3006 66.3125 0.00816706 0.0105994 2024-09-13T13:03:23.172 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:23.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:23.274 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:23.323 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:23.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:23.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:23.425 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:23.473 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:23.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:23.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:23.575 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:23.616 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:23.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:23.718 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:23.743 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:23.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:23.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:23.845 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:23.883 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:23.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:23.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:23.943 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:23 smithi167 ceph-mon[33178]: 3.6 scrub starts 2024-09-13T13:03:23.943 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:23 smithi167 ceph-mon[33178]: 3.6 scrub ok 2024-09-13T13:03:23.943 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:03:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:03:23.985 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:24.011 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:24.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:24.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:24.073 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:03:24.074 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:03:24.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:03:24.074 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:03:24.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.074 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:21143 7.1 7:84711343:::benchmark_data_smithi052_71865_object21142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:03:24.074 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:30.693622+0000", 2024-09-13T13:03:24.074 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 53.376908493999998, 2024-09-13T13:03:24.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031268601, 2024-09-13T13:03:24.074 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.074 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.074 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.075 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.075 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.075 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.075 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21143 2024-09-13T13:03:24.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.075 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.693622+0000", 2024-09-13T13:03:24.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.693622+0000", 2024-09-13T13:03:24.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.693624+0000", 2024-09-13T13:03:24.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.669e-06 2024-09-13T13:03:24.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.693656+0000", 2024-09-13T13:03:24.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2415000000000001e-05 2024-09-13T13:03:24.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.693671+0000", 2024-09-13T13:03:24.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4962e-05 2024-09-13T13:03:24.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.693679+0000", 2024-09-13T13:03:24.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4399999999999999e-06 2024-09-13T13:03:24.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.693727+0000", 2024-09-13T13:03:24.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7882e-05 2024-09-13T13:03:24.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.693864+0000", 2024-09-13T13:03:24.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013752400000000001 2024-09-13T13:03:24.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.693984+0000", 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120143 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.695749+0000", 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00176516 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.695934+0000", 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018454699999999999 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.724767+0000", 2024-09-13T13:03:24.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028833030999999999 2024-09-13T13:03:24.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.724842+0000", 2024-09-13T13:03:24.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4598000000000006e-05 2024-09-13T13:03:24.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.724891+0000", 2024-09-13T13:03:24.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9230000000000001e-05 2024-09-13T13:03:24.081 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.081 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.081 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.081 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:21147 7.1 7:89df79e8:::benchmark_data_smithi052_71865_object21146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:03:24.081 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:30.694000+0000", 2024-09-13T13:03:24.081 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 53.376531135999997, 2024-09-13T13:03:24.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032474535999999998, 2024-09-13T13:03:24.081 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.081 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.081 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21147 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.694000+0000", 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.694000+0000", 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.694001+0000", 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4610000000000001e-06 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.694058+0000", 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7318000000000002e-05 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.694069+0000", 2024-09-13T13:03:24.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1019000000000001e-05 2024-09-13T13:03:24.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.694077+0000", 2024-09-13T13:03:24.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2729999999999997e-06 2024-09-13T13:03:24.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.694775+0000", 2024-09-13T13:03:24.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00069855000000000004 2024-09-13T13:03:24.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.694958+0000", 2024-09-13T13:03:24.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018299399999999999 2024-09-13T13:03:24.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.695102+0000", 2024-09-13T13:03:24.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014410099999999999 2024-09-13T13:03:24.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.696033+0000", 2024-09-13T13:03:24.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093096499999999996 2024-09-13T13:03:24.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.697080+0000", 2024-09-13T13:03:24.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010469590000000001 2024-09-13T13:03:24.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.726411+0000", 2024-09-13T13:03:24.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029330568000000001 2024-09-13T13:03:24.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.726448+0000", 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7518000000000002e-05 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.726474+0000", 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5809999999999999e-05 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.087 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:22467 7.1 7:83ca2fe7:::benchmark_data_smithi052_71865_object22466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:03:24.088 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:31.281822+0000", 2024-09-13T13:03:24.088 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.788708274999998, 2024-09-13T13:03:24.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059630491000000001, 2024-09-13T13:03:24.088 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.088 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.088 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.091 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22467 2024-09-13T13:03:24.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.091 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281822+0000", 2024-09-13T13:03:24.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281822+0000", 2024-09-13T13:03:24.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281824+0000", 2024-09-13T13:03:24.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-06 2024-09-13T13:03:24.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281853+0000", 2024-09-13T13:03:24.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8864999999999999e-05 2024-09-13T13:03:24.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281882+0000", 2024-09-13T13:03:24.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9249000000000001e-05 2024-09-13T13:03:24.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281890+0000", 2024-09-13T13:03:24.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4320000000000002e-06 2024-09-13T13:03:24.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281939+0000", 2024-09-13T13:03:24.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9039000000000003e-05 2024-09-13T13:03:24.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.282052+0000", 2024-09-13T13:03:24.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113272 2024-09-13T13:03:24.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.282179+0000", 2024-09-13T13:03:24.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012745500000000001 2024-09-13T13:03:24.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.283337+0000", 2024-09-13T13:03:24.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011573530000000001 2024-09-13T13:03:24.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.285171+0000", 2024-09-13T13:03:24.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001834727 2024-09-13T13:03:24.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.341389+0000", 2024-09-13T13:03:24.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056217894999999997 2024-09-13T13:03:24.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.341424+0000", 2024-09-13T13:03:24.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5172999999999997e-05 2024-09-13T13:03:24.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.341453+0000", 2024-09-13T13:03:24.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8441000000000001e-05 2024-09-13T13:03:24.099 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.099 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.099 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.099 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:23363 7.1 7:87698e45:::benchmark_data_smithi052_71865_object23362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:03:24.100 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:31.706271+0000", 2024-09-13T13:03:24.100 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.364259668999999, 2024-09-13T13:03:24.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054295215000000001, 2024-09-13T13:03:24.100 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.100 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.100 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.100 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.100 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.100 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.100 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 23363 2024-09-13T13:03:24.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.101 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706271+0000", 2024-09-13T13:03:24.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706271+0000", 2024-09-13T13:03:24.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706279+0000", 2024-09-13T13:03:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4540000000000007e-06 2024-09-13T13:03:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706326+0000", 2024-09-13T13:03:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6665000000000001e-05 2024-09-13T13:03:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706340+0000", 2024-09-13T13:03:24.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4341999999999999e-05 2024-09-13T13:03:24.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706348+0000", 2024-09-13T13:03:24.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7109999999999995e-06 2024-09-13T13:03:24.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706409+0000", 2024-09-13T13:03:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1068999999999994e-05 2024-09-13T13:03:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706533+0000", 2024-09-13T13:03:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347700000000001 2024-09-13T13:03:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706669+0000", 2024-09-13T13:03:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000136348 2024-09-13T13:03:24.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.708364+0000", 2024-09-13T13:03:24.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001695041 2024-09-13T13:03:24.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.709134+0000", 2024-09-13T13:03:24.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077030700000000002 2024-09-13T13:03:24.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.760488+0000", 2024-09-13T13:03:24.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051353584000000001 2024-09-13T13:03:24.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.760533+0000", 2024-09-13T13:03:24.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5139000000000003e-05 2024-09-13T13:03:24.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.760566+0000", 2024-09-13T13:03:24.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3077999999999997e-05 2024-09-13T13:03:24.107 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.107 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.108 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.108 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:23367 7.1 7:89345dda:::benchmark_data_smithi052_71865_object23366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:03:24.108 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:31.706735+0000", 2024-09-13T13:03:24.108 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.363795881000001, 2024-09-13T13:03:24.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056305055999999999, 2024-09-13T13:03:24.108 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.109 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.109 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.109 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.109 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.109 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.109 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 23367 2024-09-13T13:03:24.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.109 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706735+0000", 2024-09-13T13:03:24.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706735+0000", 2024-09-13T13:03:24.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706736+0000", 2024-09-13T13:03:24.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.305e-06 2024-09-13T13:03:24.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706768+0000", 2024-09-13T13:03:24.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2001000000000001e-05 2024-09-13T13:03:24.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706784+0000", 2024-09-13T13:03:24.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5546e-05 2024-09-13T13:03:24.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.706799+0000", 2024-09-13T13:03:24.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5502000000000002e-05 2024-09-13T13:03:24.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.707465+0000", 2024-09-13T13:03:24.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066586100000000001 2024-09-13T13:03:24.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.707580+0000", 2024-09-13T13:03:24.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011497099999999999 2024-09-13T13:03:24.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.707727+0000", 2024-09-13T13:03:24.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147126 2024-09-13T13:03:24.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.708505+0000", 2024-09-13T13:03:24.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077767799999999996 2024-09-13T13:03:24.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.709318+0000", 2024-09-13T13:03:24.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00081338600000000004 2024-09-13T13:03:24.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.762973+0000", 2024-09-13T13:03:24.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053654372999999998 2024-09-13T13:03:24.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.763009+0000", 2024-09-13T13:03:24.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6158999999999997e-05 2024-09-13T13:03:24.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.763040+0000", 2024-09-13T13:03:24.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1148000000000001e-05 2024-09-13T13:03:24.116 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.116 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.117 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.117 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:23373 7.1 7:8d96342f:::benchmark_data_smithi052_71865_object23372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:03:24.117 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:31.709272+0000", 2024-09-13T13:03:24.117 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.361258763999999, 2024-09-13T13:03:24.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054051222000000003, 2024-09-13T13:03:24.117 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.117 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.117 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.117 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.117 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.118 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.118 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 23373 2024-09-13T13:03:24.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.118 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.709272+0000", 2024-09-13T13:03:24.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.709272+0000", 2024-09-13T13:03:24.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.709274+0000", 2024-09-13T13:03:24.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7859999999999999e-06 2024-09-13T13:03:24.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.709332+0000", 2024-09-13T13:03:24.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8257000000000001e-05 2024-09-13T13:03:24.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.709344+0000", 2024-09-13T13:03:24.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.208e-05 2024-09-13T13:03:24.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.709351+0000", 2024-09-13T13:03:24.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5440000000000004e-06 2024-09-13T13:03:24.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.709385+0000", 2024-09-13T13:03:24.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4198e-05 2024-09-13T13:03:24.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.709459+0000", 2024-09-13T13:03:24.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4466000000000002e-05 2024-09-13T13:03:24.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.709541+0000", 2024-09-13T13:03:24.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2131000000000004e-05 2024-09-13T13:03:24.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.710193+0000", 2024-09-13T13:03:24.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00065136299999999996 2024-09-13T13:03:24.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.710344+0000", 2024-09-13T13:03:24.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015117599999999999 2024-09-13T13:03:24.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.763263+0000", 2024-09-13T13:03:24.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052918685 2024-09-13T13:03:24.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.763300+0000", 2024-09-13T13:03:24.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7178000000000002e-05 2024-09-13T13:03:24.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.763323+0000", 2024-09-13T13:03:24.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3357999999999999e-05 2024-09-13T13:03:24.124 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.124 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.124 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.125 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:25071 7.1 7:81fbd844:::benchmark_data_smithi052_71865_object25070:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:03:24.125 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:32.488468+0000", 2024-09-13T13:03:24.125 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.582062817000001, 2024-09-13T13:03:24.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041581064000000001, 2024-09-13T13:03:24.125 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.125 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.125 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.125 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 25071 2024-09-13T13:03:24.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.126 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.488468+0000", 2024-09-13T13:03:24.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.488468+0000", 2024-09-13T13:03:24.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.488470+0000", 2024-09-13T13:03:24.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2029999999999999e-06 2024-09-13T13:03:24.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.488567+0000", 2024-09-13T13:03:24.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6581000000000002e-05 2024-09-13T13:03:24.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.488579+0000", 2024-09-13T13:03:24.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2741999999999999e-05 2024-09-13T13:03:24.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.488587+0000", 2024-09-13T13:03:24.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5209999999999997e-06 2024-09-13T13:03:24.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.488640+0000", 2024-09-13T13:03:24.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3508000000000003e-05 2024-09-13T13:03:24.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.488753+0000", 2024-09-13T13:03:24.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112272 2024-09-13T13:03:24.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.488883+0000", 2024-09-13T13:03:24.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013009100000000001 2024-09-13T13:03:24.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.489774+0000", 2024-09-13T13:03:24.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089106299999999999 2024-09-13T13:03:24.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.489993+0000", 2024-09-13T13:03:24.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000219106 2024-09-13T13:03:24.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.529954+0000", 2024-09-13T13:03:24.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039961429999999999 2024-09-13T13:03:24.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.530008+0000", 2024-09-13T13:03:24.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3604999999999999e-05 2024-09-13T13:03:24.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:32.530049+0000", 2024-09-13T13:03:24.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0942000000000002e-05 2024-09-13T13:03:24.132 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.132 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.132 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.132 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:28936 7.1 7:8cc6c498:::benchmark_data_smithi052_71865_object28935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-13T13:03:24.132 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:34.434379+0000", 2024-09-13T13:03:24.132 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 49.636151644999998, 2024-09-13T13:03:24.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061581934999999997, 2024-09-13T13:03:24.132 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.133 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.133 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.133 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.133 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.133 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.133 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 28936 2024-09-13T13:03:24.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.133 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434379+0000", 2024-09-13T13:03:24.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434379+0000", 2024-09-13T13:03:24.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434381+0000", 2024-09-13T13:03:24.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.933e-06 2024-09-13T13:03:24.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434412+0000", 2024-09-13T13:03:24.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0580999999999997e-05 2024-09-13T13:03:24.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434427+0000", 2024-09-13T13:03:24.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5919000000000001e-05 2024-09-13T13:03:24.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434435+0000", 2024-09-13T13:03:24.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5599999999999996e-06 2024-09-13T13:03:24.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434513+0000", 2024-09-13T13:03:24.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7610999999999999e-05 2024-09-13T13:03:24.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434638+0000", 2024-09-13T13:03:24.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000125683 2024-09-13T13:03:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434781+0000", 2024-09-13T13:03:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014221000000000001 2024-09-13T13:03:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.437744+0000", 2024-09-13T13:03:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029634409999999998 2024-09-13T13:03:24.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.437765+0000", 2024-09-13T13:03:24.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0554999999999999e-05 2024-09-13T13:03:24.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.495871+0000", 2024-09-13T13:03:24.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058106564999999999 2024-09-13T13:03:24.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.495912+0000", 2024-09-13T13:03:24.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1106999999999998e-05 2024-09-13T13:03:24.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.495961+0000", 2024-09-13T13:03:24.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8770000000000002e-05 2024-09-13T13:03:24.139 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.139 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.139 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.140 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:28940 7.1 7:8444ec31:::benchmark_data_smithi052_71865_object28939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-13T13:03:24.140 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:34.436157+0000", 2024-09-13T13:03:24.140 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 49.634373453000002, 2024-09-13T13:03:24.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062002834, 2024-09-13T13:03:24.140 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.140 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.140 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.140 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.140 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.140 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.140 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 28940 2024-09-13T13:03:24.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.141 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436157+0000", 2024-09-13T13:03:24.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436157+0000", 2024-09-13T13:03:24.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436159+0000", 2024-09-13T13:03:24.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7880000000000001e-06 2024-09-13T13:03:24.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436240+0000", 2024-09-13T13:03:24.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0631999999999996e-05 2024-09-13T13:03:24.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436250+0000", 2024-09-13T13:03:24.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0227000000000001e-05 2024-09-13T13:03:24.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436257+0000", 2024-09-13T13:03:24.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.17e-06 2024-09-13T13:03:24.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436743+0000", 2024-09-13T13:03:24.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00048641599999999999 2024-09-13T13:03:24.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436875+0000", 2024-09-13T13:03:24.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013162800000000001 2024-09-13T13:03:24.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.437006+0000", 2024-09-13T13:03:24.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013089500000000001 2024-09-13T13:03:24.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.437829+0000", 2024-09-13T13:03:24.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00082255999999999996 2024-09-13T13:03:24.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.438239+0000", 2024-09-13T13:03:24.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000410331 2024-09-13T13:03:24.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.498083+0000", 2024-09-13T13:03:24.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059844616000000003 2024-09-13T13:03:24.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.498118+0000", 2024-09-13T13:03:24.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4981000000000003e-05 2024-09-13T13:03:24.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.498160+0000", 2024-09-13T13:03:24.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.159e-05 2024-09-13T13:03:24.146 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.147 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.147 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.147 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:29300 7.1 7:89cd464f:::benchmark_data_smithi052_71865_object29299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-13T13:03:24.147 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:34.672770+0000", 2024-09-13T13:03:24.147 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 49.397760918000003, 2024-09-13T13:03:24.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046752710000000003, 2024-09-13T13:03:24.147 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.147 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.147 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 29300 2024-09-13T13:03:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.672770+0000", 2024-09-13T13:03:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.672770+0000", 2024-09-13T13:03:24.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.672771+0000", 2024-09-13T13:03:24.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.302e-06 2024-09-13T13:03:24.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.716125+0000", 2024-09-13T13:03:24.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043354203000000001 2024-09-13T13:03:24.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.716139+0000", 2024-09-13T13:03:24.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3818000000000001e-05 2024-09-13T13:03:24.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.716147+0000", 2024-09-13T13:03:24.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0639999999999994e-06 2024-09-13T13:03:24.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.716202+0000", 2024-09-13T13:03:24.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4833999999999997e-05 2024-09-13T13:03:24.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.716299+0000", 2024-09-13T13:03:24.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6900999999999999e-05 2024-09-13T13:03:24.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.716380+0000", 2024-09-13T13:03:24.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1599000000000003e-05 2024-09-13T13:03:24.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.718622+0000", 2024-09-13T13:03:24.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022417639999999998 2024-09-13T13:03:24.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.718654+0000", 2024-09-13T13:03:24.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1346000000000001e-05 2024-09-13T13:03:24.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.719459+0000", 2024-09-13T13:03:24.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080518899999999999 2024-09-13T13:03:24.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.719494+0000", 2024-09-13T13:03:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5021000000000002e-05 2024-09-13T13:03:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.719522+0000", 2024-09-13T13:03:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8668999999999999e-05 2024-09-13T13:03:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:68853 7.1 7:8c95639d:::benchmark_data_smithi052_71865_object68852:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-13T13:03:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:01.156879+0000", 2024-09-13T13:03:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.913651549000001, 2024-09-13T13:03:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037148421000000001, 2024-09-13T13:03:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 68853 2024-09-13T13:03:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.156 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.156879+0000", 2024-09-13T13:03:24.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.156879+0000", 2024-09-13T13:03:24.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.156881+0000", 2024-09-13T13:03:24.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7320000000000001e-06 2024-09-13T13:03:24.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.156932+0000", 2024-09-13T13:03:24.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1554000000000001e-05 2024-09-13T13:03:24.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.156942+0000", 2024-09-13T13:03:24.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0009e-05 2024-09-13T13:03:24.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.156947+0000", 2024-09-13T13:03:24.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0370000000000002e-06 2024-09-13T13:03:24.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.156986+0000", 2024-09-13T13:03:24.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8996999999999999e-05 2024-09-13T13:03:24.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.157075+0000", 2024-09-13T13:03:24.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8355000000000004e-05 2024-09-13T13:03:24.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.157176+0000", 2024-09-13T13:03:24.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010123 2024-09-13T13:03:24.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.159503+0000", 2024-09-13T13:03:24.160 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:24.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023273629999999998 2024-09-13T13:03:24.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.159680+0000", 2024-09-13T13:03:24.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000176167 2024-09-13T13:03:24.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.193893+0000", 2024-09-13T13:03:24.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034213022000000003 2024-09-13T13:03:24.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.193963+0000", 2024-09-13T13:03:24.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0711999999999997e-05 2024-09-13T13:03:24.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:01.194028+0000", 2024-09-13T13:03:24.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4243000000000001e-05 2024-09-13T13:03:24.163 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.163 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.163 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.164 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:83026 7.1 7:82e2b293:::benchmark_data_smithi052_71865_object83025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:03:24.164 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:13.636040+0000", 2024-09-13T13:03:24.164 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.434490198000001, 2024-09-13T13:03:24.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037215992000000003, 2024-09-13T13:03:24.164 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.164 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.164 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.165 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.165 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 83026 2024-09-13T13:03:24.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.165 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.636040+0000", 2024-09-13T13:03:24.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.636040+0000", 2024-09-13T13:03:24.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.636048+0000", 2024-09-13T13:03:24.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5820000000000004e-06 2024-09-13T13:03:24.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.636135+0000", 2024-09-13T13:03:24.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7326999999999999e-05 2024-09-13T13:03:24.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.636151+0000", 2024-09-13T13:03:24.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5999999999999999e-05 2024-09-13T13:03:24.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.636159+0000", 2024-09-13T13:03:24.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3409999999999997e-06 2024-09-13T13:03:24.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.636210+0000", 2024-09-13T13:03:24.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1187000000000003e-05 2024-09-13T13:03:24.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.636325+0000", 2024-09-13T13:03:24.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011482 2024-09-13T13:03:24.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.636478+0000", 2024-09-13T13:03:24.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015291 2024-09-13T13:03:24.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.639059+0000", 2024-09-13T13:03:24.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025808670000000001 2024-09-13T13:03:24.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.640489+0000", 2024-09-13T13:03:24.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014305030000000001 2024-09-13T13:03:24.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.673126+0000", 2024-09-13T13:03:24.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032637358999999998 2024-09-13T13:03:24.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.673209+0000", 2024-09-13T13:03:24.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2981999999999999e-05 2024-09-13T13:03:24.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.673256+0000", 2024-09-13T13:03:24.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7114000000000003e-05 2024-09-13T13:03:24.171 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.171 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.171 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.172 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:83030 7.1 7:8a91167c:::benchmark_data_smithi052_71865_object83029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:03:24.172 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:13.640927+0000", 2024-09-13T13:03:24.172 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.429603479000001, 2024-09-13T13:03:24.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032507576000000003, 2024-09-13T13:03:24.172 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.172 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.172 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.172 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.172 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.172 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.172 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 83030 2024-09-13T13:03:24.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.173 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.640927+0000", 2024-09-13T13:03:24.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.640927+0000", 2024-09-13T13:03:24.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.640929+0000", 2024-09-13T13:03:24.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.029e-06 2024-09-13T13:03:24.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.640965+0000", 2024-09-13T13:03:24.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6074000000000001e-05 2024-09-13T13:03:24.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.640979+0000", 2024-09-13T13:03:24.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3266e-05 2024-09-13T13:03:24.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.640986+0000", 2024-09-13T13:03:24.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6120000000000001e-06 2024-09-13T13:03:24.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.641039+0000", 2024-09-13T13:03:24.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3134999999999999e-05 2024-09-13T13:03:24.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.641160+0000", 2024-09-13T13:03:24.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121175 2024-09-13T13:03:24.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.641313+0000", 2024-09-13T13:03:24.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015268200000000001 2024-09-13T13:03:24.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.643347+0000", 2024-09-13T13:03:24.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020338890000000001 2024-09-13T13:03:24.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.643365+0000", 2024-09-13T13:03:24.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7864999999999999e-05 2024-09-13T13:03:24.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.673303+0000", 2024-09-13T13:03:24.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029937900999999999 2024-09-13T13:03:24.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.673393+0000", 2024-09-13T13:03:24.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0217999999999996e-05 2024-09-13T13:03:24.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:13.673435+0000", 2024-09-13T13:03:24.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1730000000000002e-05 2024-09-13T13:03:24.180 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.180 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.180 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.180 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:83867 7.1 7:806392f8:::benchmark_data_smithi052_71865_object83866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:03:24.180 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:14.439379+0000", 2024-09-13T13:03:24.180 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.6311521570000007, 2024-09-13T13:03:24.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034986982, 2024-09-13T13:03:24.180 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.181 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.181 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.181 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 83867 2024-09-13T13:03:24.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.181 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.439379+0000", 2024-09-13T13:03:24.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.439379+0000", 2024-09-13T13:03:24.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.439381+0000", 2024-09-13T13:03:24.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.097e-06 2024-09-13T13:03:24.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.439449+0000", 2024-09-13T13:03:24.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8573999999999998e-05 2024-09-13T13:03:24.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.439462+0000", 2024-09-13T13:03:24.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3305000000000001e-05 2024-09-13T13:03:24.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.439469+0000", 2024-09-13T13:03:24.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9330000000000004e-06 2024-09-13T13:03:24.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.441845+0000", 2024-09-13T13:03:24.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023758910000000002 2024-09-13T13:03:24.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.441985+0000", 2024-09-13T13:03:24.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013982600000000001 2024-09-13T13:03:24.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.442129+0000", 2024-09-13T13:03:24.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000143366 2024-09-13T13:03:24.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.444938+0000", 2024-09-13T13:03:24.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028093089999999999 2024-09-13T13:03:24.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.446368+0000", 2024-09-13T13:03:24.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001430136 2024-09-13T13:03:24.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.474229+0000", 2024-09-13T13:03:24.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.027860558000000001 2024-09-13T13:03:24.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.474321+0000", 2024-09-13T13:03:24.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2319999999999997e-05 2024-09-13T13:03:24.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.474365+0000", 2024-09-13T13:03:24.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4666999999999998e-05 2024-09-13T13:03:24.188 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.188 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.188 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.188 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:83878 7.1 7:88a04359:::benchmark_data_smithi052_71865_object83877:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:03:24.188 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:14.448298+0000", 2024-09-13T13:03:24.188 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.6222322939999998, 2024-09-13T13:03:24.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034336430000000001, 2024-09-13T13:03:24.188 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.188 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.188 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.189 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.189 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.189 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.189 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 83878 2024-09-13T13:03:24.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.189 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.448298+0000", 2024-09-13T13:03:24.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.448298+0000", 2024-09-13T13:03:24.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.448301+0000", 2024-09-13T13:03:24.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1890000000000001e-06 2024-09-13T13:03:24.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.448366+0000", 2024-09-13T13:03:24.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5139000000000001e-05 2024-09-13T13:03:24.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.448376+0000", 2024-09-13T13:03:24.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0492999999999999e-05 2024-09-13T13:03:24.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.448381+0000", 2024-09-13T13:03:24.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1039999999999998e-06 2024-09-13T13:03:24.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.448426+0000", 2024-09-13T13:03:24.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4611999999999999e-05 2024-09-13T13:03:24.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.448554+0000", 2024-09-13T13:03:24.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000128276 2024-09-13T13:03:24.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:03:24.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.448704+0000", 2024-09-13T13:03:24.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014963300000000001 2024-09-13T13:03:24.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.450974+0000", 2024-09-13T13:03:24.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022702629999999998 2024-09-13T13:03:24.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.451636+0000", 2024-09-13T13:03:24.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066216799999999998 2024-09-13T13:03:24.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.482529+0000", 2024-09-13T13:03:24.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030892859000000002 2024-09-13T13:03:24.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.482589+0000", 2024-09-13T13:03:24.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0356999999999998e-05 2024-09-13T13:03:24.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.482635+0000", 2024-09-13T13:03:24.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5337000000000003e-05 2024-09-13T13:03:24.195 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.195 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.195 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.195 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:83969 7.1 7:819e28b6:::benchmark_data_smithi052_71865_object83968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:03:24.196 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:14.547136+0000", 2024-09-13T13:03:24.196 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.5233945230000003, 2024-09-13T13:03:24.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0019013699999999, 2024-09-13T13:03:24.196 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.196 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.196 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.196 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.196 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.196 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.196 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 83969 2024-09-13T13:03:24.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.197 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547136+0000", 2024-09-13T13:03:24.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547136+0000", 2024-09-13T13:03:24.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547138+0000", 2024-09-13T13:03:24.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2859999999999998e-06 2024-09-13T13:03:24.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547171+0000", 2024-09-13T13:03:24.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2586000000000003e-05 2024-09-13T13:03:24.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547182+0000", 2024-09-13T13:03:24.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1336e-05 2024-09-13T13:03:24.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547191+0000", 2024-09-13T13:03:24.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.67e-06 2024-09-13T13:03:24.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547351+0000", 2024-09-13T13:03:24.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000160032 2024-09-13T13:03:24.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:03:24.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547364+0000", 2024-09-13T13:03:24.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3118e-05 2024-09-13T13:03:24.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.545177+0000", 2024-09-13T13:03:24.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99781304699999995 2024-09-13T13:03:24.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.545295+0000", 2024-09-13T13:03:24.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118134 2024-09-13T13:03:24.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:03:24.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.545394+0000", 2024-09-13T13:03:24.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8840999999999998e-05 2024-09-13T13:03:24.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.547566+0000", 2024-09-13T13:03:24.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021715749999999998 2024-09-13T13:03:24.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.548948+0000", 2024-09-13T13:03:24.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013822719999999999 2024-09-13T13:03:24.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.548984+0000", 2024-09-13T13:03:24.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5725000000000001e-05 2024-09-13T13:03:24.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.549038+0000", 2024-09-13T13:03:24.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3748000000000001e-05 2024-09-13T13:03:24.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.203 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.204 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14493.0:1218 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94064716857216 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:03:24.204 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:15.061508+0000", 2024-09-13T13:03:24.204 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.0090225089999993, 2024-09-13T13:03:24.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.48102383999999998, 2024-09-13T13:03:24.204 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.204 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.204 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:03:24.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14493", 2024-09-13T13:03:24.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:03:24.205 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1218 2024-09-13T13:03:24.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.205 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061508+0000", 2024-09-13T13:03:24.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061508+0000", 2024-09-13T13:03:24.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061511+0000", 2024-09-13T13:03:24.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3750000000000001e-06 2024-09-13T13:03:24.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061517+0000", 2024-09-13T13:03:24.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1999999999999999e-06 2024-09-13T13:03:24.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061528+0000", 2024-09-13T13:03:24.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1642e-05 2024-09-13T13:03:24.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061537+0000", 2024-09-13T13:03:24.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7110000000000001e-06 2024-09-13T13:03:24.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061610+0000", 2024-09-13T13:03:24.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3126000000000004e-05 2024-09-13T13:03:24.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:03:24.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061621+0000", 2024-09-13T13:03:24.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1112e-05 2024-09-13T13:03:24.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.537588+0000", 2024-09-13T13:03:24.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.47596680699999999 2024-09-13T13:03:24.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.537736+0000", 2024-09-13T13:03:24.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147887 2024-09-13T13:03:24.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.542532+0000", 2024-09-13T13:03:24.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00479598 2024-09-13T13:03:24.210 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.210 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.210 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.210 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14493.0:1219 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94064721783040 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:03:24.210 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:15.061557+0000", 2024-09-13T13:03:24.210 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.0089731709999992, 2024-09-13T13:03:24.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.481215114, 2024-09-13T13:03:24.210 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.210 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.211 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:03:24.211 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.211 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14493", 2024-09-13T13:03:24.211 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:03:24.211 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1219 2024-09-13T13:03:24.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.211 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061557+0000", 2024-09-13T13:03:24.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061557+0000", 2024-09-13T13:03:24.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061558+0000", 2024-09-13T13:03:24.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3499999999999996e-07 2024-09-13T13:03:24.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061561+0000", 2024-09-13T13:03:24.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7240000000000001e-06 2024-09-13T13:03:24.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061567+0000", 2024-09-13T13:03:24.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4459999999999998e-06 2024-09-13T13:03:24.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061572+0000", 2024-09-13T13:03:24.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7029999999999997e-06 2024-09-13T13:03:24.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061623+0000", 2024-09-13T13:03:24.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1078999999999997e-05 2024-09-13T13:03:24.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:03:24.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061629+0000", 2024-09-13T13:03:24.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4600000000000002e-06 2024-09-13T13:03:24.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.539806+0000", 2024-09-13T13:03:24.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.47817701200000001 2024-09-13T13:03:24.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.539938+0000", 2024-09-13T13:03:24.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000132757 2024-09-13T13:03:24.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.542773+0000", 2024-09-13T13:03:24.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028342979999999999 2024-09-13T13:03:24.216 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.216 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.216 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.217 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.34781.0:84251 7.3 e104/84 7:c353d3e7:::benchmark_data_smithi052_71865_object84250:head v 104'5204, mlcod=104'5200)", 2024-09-13T13:03:24.217 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:15.775475+0000", 2024-09-13T13:03:24.217 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2950556110000004, 2024-09-13T13:03:24.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040243292999999999, 2024-09-13T13:03:24.217 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.217 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.217 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.217 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.775475+0000", 2024-09-13T13:03:24.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.775475+0000", 2024-09-13T13:03:24.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.775476+0000", 2024-09-13T13:03:24.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.325e-06 2024-09-13T13:03:24.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.812040+0000", 2024-09-13T13:03:24.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036563423999999997 2024-09-13T13:03:24.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.812053+0000", 2024-09-13T13:03:24.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3213e-05 2024-09-13T13:03:24.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.812059+0000", 2024-09-13T13:03:24.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4729999999999997e-06 2024-09-13T13:03:24.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.812112+0000", 2024-09-13T13:03:24.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2540000000000002e-05 2024-09-13T13:03:24.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.812129+0000", 2024-09-13T13:03:24.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7401000000000002e-05 2024-09-13T13:03:24.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.815676+0000", 2024-09-13T13:03:24.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035466400000000002 2024-09-13T13:03:24.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.815718+0000", 2024-09-13T13:03:24.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2277000000000002e-05 2024-09-13T13:03:24.222 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.222 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.222 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.222 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:86983 7.1 7:80eeb3f2:::benchmark_data_smithi052_71865_object86982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T13:03:24.222 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:18.331469+0000", 2024-09-13T13:03:24.222 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.7390611849999997, 2024-09-13T13:03:24.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041467298999999999, 2024-09-13T13:03:24.223 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:24.223 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:24.223 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:24.223 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:24.223 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:24.223 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:24.223 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 86983 2024-09-13T13:03:24.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.223 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:24.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:24.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:18.331469+0000", 2024-09-13T13:03:24.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:24.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:18.331469+0000", 2024-09-13T13:03:24.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:24.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:24.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:18.331471+0000", 2024-09-13T13:03:24.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9929999999999998e-06 2024-09-13T13:03:24.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:24.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:18.331525+0000", 2024-09-13T13:03:24.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3720000000000001e-05 2024-09-13T13:03:24.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:24.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:18.331538+0000", 2024-09-13T13:03:24.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3190000000000001e-05 2024-09-13T13:03:24.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:24.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:18.331544+0000", 2024-09-13T13:03:24.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4559999999999999e-06 2024-09-13T13:03:24.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:24.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:18.331594+0000", 2024-09-13T13:03:24.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0343999999999998e-05 2024-09-13T13:03:24.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:24.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:18.331720+0000", 2024-09-13T13:03:24.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000126272 2024-09-13T13:03:24.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:03:24.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:18.331883+0000", 2024-09-13T13:03:24.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016212600000000001 2024-09-13T13:03:24.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:24.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:18.333972+0000", 2024-09-13T13:03:24.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002089207 2024-09-13T13:03:24.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:24.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:18.372790+0000", 2024-09-13T13:03:24.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038817745000000001 2024-09-13T13:03:24.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:24.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:18.372841+0000", 2024-09-13T13:03:24.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1150999999999999e-05 2024-09-13T13:03:24.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:24.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:24.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:24.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:18.372937+0000", 2024-09-13T13:03:24.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6094999999999994e-05 2024-09-13T13:03:24.229 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.229 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.229 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.229 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:24.229 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:24.229 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:03:24.230 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 62 16 93250 93234 93.942 72.125 0.00455805 0.0106389 2024-09-13T13:03:24.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:23 smithi052 ceph-mon[36682]: 3.6 scrub starts 2024-09-13T13:03:24.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:23 smithi052 ceph-mon[36682]: 3.6 scrub ok 2024-09-13T13:03:24.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:23 smithi052 ceph-mon[29471]: 3.6 scrub starts 2024-09-13T13:03:24.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:23 smithi052 ceph-mon[29471]: 3.6 scrub ok 2024-09-13T13:03:24.255 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:24.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:24.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:03:24.323 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:03:24.357 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:24.383 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:24.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:24.485 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:24.513 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:24.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:24.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:24.640 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:24.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:24.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:24.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:24.768 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:24.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:24.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:24.870 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:24.916 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:24.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:25.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:25.026 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:24 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:25.027 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:24 smithi167 ceph-mon[33178]: pgmap v94: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 28918/276105 objects degraded (10.474%); 10 KiB/s, 0 objects/s recovering 2024-09-13T13:03:25.054 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:25.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:25.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:25.156 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:25.165 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 63 16 94432 94416 93.6229 73.875 0.00856412 0.0106754 2024-09-13T13:03:25.194 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:25.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:25.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:25.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:24 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:25.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:24 smithi052 ceph-mon[29471]: pgmap v94: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 28918/276105 objects degraded (10.474%); 10 KiB/s, 0 objects/s recovering 2024-09-13T13:03:25.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:24 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:25.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:24 smithi052 ceph-mon[36682]: pgmap v94: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 28918/276105 objects degraded (10.474%); 10 KiB/s, 0 objects/s recovering 2024-09-13T13:03:25.295 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:25.343 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:25.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:25.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:25.445 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:25.486 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:25.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:25.531 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:03:25.587 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:25.631 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:25.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:25.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:25.733 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:25.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:25.782 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:25.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:25.817 INFO:teuthology.orchestra.run.smithi167.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-13T13:03:25.884 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:25.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:25.953 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:25.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:25 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 28918/276105 objects degraded (10.474%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:25 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 28918/276105 objects degraded (10.474%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:26.006 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:26.054 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:26.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:25 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 28918/276105 objects degraded (10.474%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:26.103 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:26.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:26.168 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 64 16 95496 95480 93.1959 66.5 0.011514 0.0107252 2024-09-13T13:03:26.204 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:26.227 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:26.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:26.324 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:03:26.329 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:26.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T13:03:26.419 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:26.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:26.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:26.488 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:03:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:03:26.521 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:26.547 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:26.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:26.649 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:26.690 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:26.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:26.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:26.792 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:26.833 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:26.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:26.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:26.901 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:26 smithi167 ceph-mon[33178]: pgmap v95: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 511 B/s rd, 69 MiB/s wr, 1.10k op/s; 28918/276105 objects degraded (10.474%); 8.0 KiB/s, 0 objects/s recovering 2024-09-13T13:03:26.936 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:26.945 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:03:26.980 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:26.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:26.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:27.082 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:27.111 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:27.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:27.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:27.167 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 65 16 96386 96370 92.6181 55.625 0.00667498 0.0107923 2024-09-13T13:03:27.206 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:26 smithi052 ceph-mon[36682]: pgmap v95: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 511 B/s rd, 69 MiB/s wr, 1.10k op/s; 28918/276105 objects degraded (10.474%); 8.0 KiB/s, 0 objects/s recovering 2024-09-13T13:03:27.206 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:26 smithi052 ceph-mon[29471]: pgmap v95: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 511 B/s rd, 69 MiB/s wr, 1.10k op/s; 28918/276105 objects degraded (10.474%); 8.0 KiB/s, 0 objects/s recovering 2024-09-13T13:03:27.213 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:27.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:27.258 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:27.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:27.297 INFO:teuthology.orchestra.run.smithi167.stdout:7.b 2024-09-13T13:03:27.297 INFO:teuthology.orchestra.run.smithi167.stdout:6.5 2024-09-13T13:03:27.297 INFO:teuthology.orchestra.run.smithi167.stdout:6.3 2024-09-13T13:03:27.297 INFO:teuthology.orchestra.run.smithi167.stdout:6.1f 2024-09-13T13:03:27.297 INFO:teuthology.orchestra.run.smithi167.stdout:6.c 2024-09-13T13:03:27.297 INFO:teuthology.orchestra.run.smithi167.stdout:6.1b 2024-09-13T13:03:27.297 INFO:teuthology.orchestra.run.smithi167.stdout:6.1 2024-09-13T13:03:27.297 INFO:teuthology.orchestra.run.smithi167.stdout:6.15 2024-09-13T13:03:27.297 INFO:teuthology.orchestra.run.smithi167.stdout:5.9 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:5.3 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:3.a 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:3.14 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:3.3 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:3.1e 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:3.1d 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:6.13 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:5.1f 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:3.2 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:6.1e 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:5.f 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:4.8 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:3.10 2024-09-13T13:03:27.298 INFO:teuthology.orchestra.run.smithi167.stdout:7.1 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:4.1f 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:3.d 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:7.d 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:3.11 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:3.0 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:5.2 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:5.10 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:1.0 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:4.12 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:7.7 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:5.11 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:4.13 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:7.c 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:4.a 2024-09-13T13:03:27.299 INFO:teuthology.orchestra.run.smithi167.stdout:4.10 2024-09-13T13:03:27.300 INFO:teuthology.orchestra.run.smithi167.stdout:4.18 2024-09-13T13:03:27.300 INFO:teuthology.orchestra.run.smithi167.stdout:3.c 2024-09-13T13:03:27.300 INFO:teuthology.orchestra.run.smithi167.stdout:4.c 2024-09-13T13:03:27.300 INFO:teuthology.orchestra.run.smithi167.stdout:4.1 2024-09-13T13:03:27.300 INFO:teuthology.orchestra.run.smithi167.stdout:4.11 2024-09-13T13:03:27.300 INFO:teuthology.orchestra.run.smithi167.stdout:4.5 2024-09-13T13:03:27.300 INFO:teuthology.orchestra.run.smithi167.stdout:6.a 2024-09-13T13:03:27.300 INFO:teuthology.orchestra.run.smithi167.stdout:2.2 2024-09-13T13:03:27.300 INFO:teuthology.orchestra.run.smithi167.stdout:4.e 2024-09-13T13:03:27.300 INFO:teuthology.orchestra.run.smithi167.stdout:5.18 2024-09-13T13:03:27.360 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:27.402 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:27.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:27.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:27.504 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:27.526 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:27.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:27.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:27.629 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:27.655 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:27.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:27.756 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:27.818 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:27.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:27.919 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:27.945 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:27.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:28.047 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:28.096 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:28.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:28.100 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:28.169 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 66 16 97330 97314 92.1082 59 0.0133919 0.0108508 2024-09-13T13:03:28.198 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:28.222 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:28.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:28.324 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:28.364 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:28.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:28.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:28.379 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -n osd.6 --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op export --pgid 3.1e --file /var/log/ceph/exp.3.1e.6 2024-09-13T13:03:28.466 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:28.528 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:28.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:28.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:28.630 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:28.671 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:28.671 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/osd.6/config 2024-09-13T13:03:28.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:28.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:28.770 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:03:28.773 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:28.799 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:28.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:28.800 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:03:28.800 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:03:28.841 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:03:28.842 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:03:28.842 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:28.842 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:28.842 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[36682]: pgmap v96: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 783 op/s; 29228/278997 objects degraded (10.476%) 2024-09-13T13:03:28.842 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:03:28.842 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:03:28.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2472837513' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:28.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[36682]: from='client.24889 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:28.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:03:28.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:03:28.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:28.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:28.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:28.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[29471]: pgmap v96: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 783 op/s; 29228/278997 objects degraded (10.476%) 2024-09-13T13:03:28.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:03:28.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:03:28.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2472837513' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:28.844 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[29471]: from='client.24889 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:28.844 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:03:28.844 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:03:28.844 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:28 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:28.896 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:03:28.896 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:03:28.899 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:28.938 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:03:28.938 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:03:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:28.954 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:28.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:28.970 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:28 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:28.970 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:28 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:28.970 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:28 smithi167 ceph-mon[33178]: pgmap v96: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 783 op/s; 29228/278997 objects degraded (10.476%) 2024-09-13T13:03:28.970 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:28 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:03:28.970 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:28 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:03:28.970 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:28 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2472837513' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:28.971 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:28 smithi167 ceph-mon[33178]: from='client.24889 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:28.971 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:28 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:03:28.971 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:28 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:03:28.971 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:28 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:28.979 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:03:28.979 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:03:29.010 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:03:29.010 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:03:29.011 INFO:teuthology.orchestra.run.smithi052.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T13:03:29.055 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:29.059 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:03:29.060 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:03:29.098 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:29.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:29.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:29.168 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 67 16 98084 98068 91.437 47.125 0.012088 0.0109302 2024-09-13T13:03:29.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:29.226 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:29.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:29.328 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:29.350 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:29.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:29.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:29.452 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:29.489 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:29.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:29.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:29.591 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:29.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:29.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:29.662 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:03:29.662 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:03:29.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:29.757 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:29.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:29.859 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:29.904 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:29.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:29.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:29.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:29 smithi052 ceph-mon[29471]: pgmap v97: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 30540/292056 objects degraded (10.457%) 2024-09-13T13:03:29.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:29 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4106174893' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:29.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:29 smithi052 ceph-mon[29471]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-13T13:03:29.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:29 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:29.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:29 smithi052 ceph-mon[29471]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:29.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:29 smithi052 ceph-mon[29471]: osdmap e109: 8 total, 7 up, 8 in 2024-09-13T13:03:29.987 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:29 smithi052 ceph-mon[36682]: pgmap v97: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 30540/292056 objects degraded (10.457%) 2024-09-13T13:03:29.987 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:29 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4106174893' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:29.987 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:29 smithi052 ceph-mon[36682]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-13T13:03:29.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:29 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:29.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:29 smithi052 ceph-mon[36682]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:29.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:29 smithi052 ceph-mon[36682]: osdmap e109: 8 total, 7 up, 8 in 2024-09-13T13:03:30.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:30.030 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:30.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:29 smithi167 ceph-mon[33178]: pgmap v97: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 30540/292056 objects degraded (10.457%) 2024-09-13T13:03:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:29 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4106174893' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:29 smithi167 ceph-mon[33178]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-13T13:03:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:29 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:29 smithi167 ceph-mon[33178]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:29 smithi167 ceph-mon[33178]: osdmap e109: 8 total, 7 up, 8 in 2024-09-13T13:03:30.132 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:30.156 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:30.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:30.168 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 68 16 98964 98948 90.9013 55 0.0127987 0.0109958 2024-09-13T13:03:30.258 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:30.307 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:30.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:30.386 INFO:teuthology.orchestra.run.smithi167.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-13T13:03:30.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:30.447 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:30.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:30.550 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:30.573 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:30.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:30.675 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:30.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:30.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:30.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:30.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:30.849 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:30.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:30.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:30.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:30.978 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:30.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:30.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:31.080 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:31.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:31.122 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:31.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:31.168 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 69 16 100104 100088 90.6166 71.25 0.00643958 0.0110302 2024-09-13T13:03:31.224 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:31.264 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:31.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:31.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:31.315 INFO:teuthology.orchestra.run.smithi167.stderr:Exporting 3.1e info 3.1e( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) 2024-09-13T13:03:31.315 INFO:teuthology.orchestra.run.smithi167.stderr:Export successful 2024-09-13T13:03:31.366 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:31.402 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:31.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:31.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:03:31.492 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:03:31.505 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:31.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:31.531 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:31.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:31.632 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:31.660 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:31.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:31.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:31.789 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:31.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:31.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:31 smithi167 ceph-mon[33178]: pgmap v99: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 30540/292056 objects degraded (10.457%) 2024-09-13T13:03:31.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:31 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:03:31.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:31 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:03:31.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:31 smithi167 ceph-mon[33178]: osdmap e110: 8 total, 7 up, 8 in 2024-09-13T13:03:31.890 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:31.934 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:31.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:31 smithi052 ceph-mon[36682]: pgmap v99: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 30540/292056 objects degraded (10.457%) 2024-09-13T13:03:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:31 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:03:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:31 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:03:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:31 smithi052 ceph-mon[36682]: osdmap e110: 8 total, 7 up, 8 in 2024-09-13T13:03:31.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:31 smithi052 ceph-mon[29471]: pgmap v99: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 30540/292056 objects degraded (10.457%) 2024-09-13T13:03:31.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:31 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:03:31.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:31 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:03:31.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:31 smithi052 ceph-mon[29471]: osdmap e110: 8 total, 7 up, 8 in 2024-09-13T13:03:32.036 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:32.066 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:32.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:32.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:32.168 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:32.169 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 70 16 101267 101251 90.3604 72.6875 0.0079775 0.0110618 2024-09-13T13:03:32.215 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:32.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:32.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:32.346 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:32.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:32.448 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:32.494 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:32.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:32.595 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:32.608 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:03:32.640 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:32.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:32.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:32.724 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -n osd.6 --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --force --op remove --pgid 3.1e 2024-09-13T13:03:32.746 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:32.804 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:32.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:32.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:32.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:32.939 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T13:03:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:32.941 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:32.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:32.983 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/osd.6/config 2024-09-13T13:03:33.042 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:33.101 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:33.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:33.123 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:33.168 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 71 16 102323 102307 90.0174 66 0.023606 0.011104 2024-09-13T13:03:33.204 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:33.230 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:33.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:33.333 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:33.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:33.372 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:33.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:33.473 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:33.493 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:03:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:33.516 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:33.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:33.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:33.641 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:33.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:33.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:33.743 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:33.766 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:33.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:33.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:33.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:33 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:33 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:03:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:33 smithi167 ceph-mon[33178]: pgmap v101: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 757 B/s rd, 49 MiB/s wr, 787 op/s; 30540/292056 objects degraded (10.457%) 2024-09-13T13:03:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:33 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 30540/292056 objects degraded (10.457%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:33.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:03:33 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:03:33.869 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:33.932 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:33.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:33.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:33 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:33.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:33 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:03:33.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:33 smithi052 ceph-mon[36682]: pgmap v101: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 757 B/s rd, 49 MiB/s wr, 787 op/s; 30540/292056 objects degraded (10.457%) 2024-09-13T13:03:33.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:33 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 30540/292056 objects degraded (10.457%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:33.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:33 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:03:33.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:33 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:03:33.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:33 smithi052 ceph-mon[29471]: pgmap v101: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 757 B/s rd, 49 MiB/s wr, 787 op/s; 30540/292056 objects degraded (10.457%) 2024-09-13T13:03:33.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:33 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 30540/292056 objects degraded (10.457%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:34.035 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:34.046 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:34.060 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:34.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:34.161 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:34.168 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 72 16 103304 103288 89.6182 61.3125 0.0179856 0.0111534 2024-09-13T13:03:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:34.216 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:34.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:34.302 INFO:teuthology.orchestra.run.smithi167.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-13T13:03:34.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:34.335 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:03:34.351 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:34.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:34.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:34.453 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:34.483 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:34.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:34.574 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:03:34.585 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:34.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:34.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:34.711 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:34.717 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:03:34.718 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:03:34.751 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:34.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:34.754 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:03:34.754 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:03:34.788 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:03:34.789 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:03:34.789 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:34 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:34.789 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:34 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1484363848' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:03:34.789 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:34 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:34.789 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:34 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1484363848' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:03:34.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:34 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:34.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:34 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1484363848' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:03:34.825 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:03:34.825 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:03:34.854 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:34.858 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:03:34.858 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:03:34.879 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:34.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:34.890 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:03:34.890 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:03:34.893 INFO:teuthology.orchestra.run.smithi052.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T13:03:34.927 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:03:34.927 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:03:34.981 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:35.005 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:35.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:35.106 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:35.140 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:35.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:35.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:35.169 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 73 16 104299 104283 89.2425 62.1875 0.00314484 0.0111957 2024-09-13T13:03:35.242 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:35.279 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:35.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:35.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:35.382 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:35.419 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:35.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:35.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:35.521 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:35.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:35.566 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:35.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:35.597 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:03:35.598 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:03:35.598 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:03:35.598 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:03:35.598 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:03:35.667 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:35.707 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:35.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:35.808 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:35.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:35 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:35.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:35 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1484363848' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:03:35.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:35 smithi167 ceph-mon[33178]: osdmap e111: 8 total, 7 up, 8 in 2024-09-13T13:03:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:35 smithi167 ceph-mon[33178]: pgmap v103: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 823 op/s; 32072/306861 objects degraded (10.452%) 2024-09-13T13:03:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:35 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3707350436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:35.845 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:35.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:35.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:35.977 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:35.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:35.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:35 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:35.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:35 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1484363848' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:03:35.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:35 smithi052 ceph-mon[36682]: osdmap e111: 8 total, 7 up, 8 in 2024-09-13T13:03:35.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:35 smithi052 ceph-mon[36682]: pgmap v103: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 823 op/s; 32072/306861 objects degraded (10.452%) 2024-09-13T13:03:35.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:35 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3707350436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:35.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:35 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:35.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:35 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1484363848' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:03:35.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:35 smithi052 ceph-mon[29471]: osdmap e111: 8 total, 7 up, 8 in 2024-09-13T13:03:35.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:35 smithi052 ceph-mon[29471]: pgmap v103: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 823 op/s; 32072/306861 objects degraded (10.452%) 2024-09-13T13:03:35.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:35 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3707350436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:36.079 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:36.105 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:36.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:36.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:36.114 INFO:teuthology.orchestra.run.smithi167.stdout: marking collection for removal 2024-09-13T13:03:36.115 INFO:teuthology.orchestra.run.smithi167.stdout:setting '_remove' omap key 2024-09-13T13:03:36.115 INFO:teuthology.orchestra.run.smithi167.stdout:finish_remove_pgs 3.1e_head removing 3.1e 2024-09-13T13:03:36.115 INFO:teuthology.orchestra.run.smithi167.stdout:Remove successful 2024-09-13T13:03:36.169 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 74 16 105326 105310 88.904 64.1875 0.0143364 0.0112426 2024-09-13T13:03:36.207 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:36.252 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:36.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:36.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:36.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:36.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:36.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:36.483 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:36.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:03:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:03:36.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T13:03:36.523 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:36.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:36.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:36.625 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:36.669 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:36.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:36.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:36.770 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:36.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:36 smithi167 ceph-mon[33178]: 7.5 scrub starts 2024-09-13T13:03:36.822 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:36.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:36.924 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:36.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:36.966 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:36.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:36.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:36 smithi052 ceph-mon[36682]: 7.5 scrub starts 2024-09-13T13:03:36.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:36 smithi052 ceph-mon[29471]: 7.5 scrub starts 2024-09-13T13:03:37.067 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:37.109 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:37.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:37.169 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 75 16 106365 106349 88.5845 64.9375 0.00803828 0.0112839 2024-09-13T13:03:37.211 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:37.253 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:37.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:37.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:37.355 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:37.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -n osd.6 --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.1e.6 2024-09-13T13:03:37.398 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:03:37.399 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:37.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:37.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:37.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:37.526 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:37.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:37.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:03:37.556 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:03:37.622 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/osd.6/config 2024-09-13T13:03:37.628 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:37.667 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:37.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:37.769 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:37.811 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:37.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:37.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:37.905 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:37 smithi167 ceph-mon[33178]: 7.e scrub starts 2024-09-13T13:03:37.905 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:37 smithi167 ceph-mon[33178]: pgmap v104: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 648 B/s rd, 39 MiB/s wr, 625 op/s; 32072/306861 objects degraded (10.452%) 2024-09-13T13:03:37.912 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:37.946 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:37.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:37.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:37 smithi052 ceph-mon[36682]: 7.e scrub starts 2024-09-13T13:03:37.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:37 smithi052 ceph-mon[36682]: pgmap v104: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 648 B/s rd, 39 MiB/s wr, 625 op/s; 32072/306861 objects degraded (10.452%) 2024-09-13T13:03:37.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:37 smithi052 ceph-mon[29471]: 7.e scrub starts 2024-09-13T13:03:37.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:37 smithi052 ceph-mon[29471]: pgmap v104: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 648 B/s rd, 39 MiB/s wr, 625 op/s; 32072/306861 objects degraded (10.452%) 2024-09-13T13:03:38.049 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:38.076 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:38.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:38.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:38.169 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 76 16 107395 107379 88.266 64.375 0.0229067 0.0113241 2024-09-13T13:03:38.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:38.202 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:38.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:38.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:38.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:38.330 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:38.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:38.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:38.432 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:38.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:38.475 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:38.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:38.576 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:38.618 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:38.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:38.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:38.720 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:38.745 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:38.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:38.796 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:03:38.847 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:38.872 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:38.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:38.911 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:38 smithi167 ceph-mon[33178]: 7.5 scrub ok 2024-09-13T13:03:38.911 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:38 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 32072/306861 objects degraded (10.452%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:38.911 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:38 smithi167 ceph-mon[33178]: osdmap e112: 8 total, 7 up, 8 in 2024-09-13T13:03:38.912 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:38 smithi167 ceph-mon[33178]: 3.e scrub starts 2024-09-13T13:03:38.912 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:38 smithi167 ceph-mon[33178]: 3.e scrub ok 2024-09-13T13:03:38.935 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:03:38.936 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:03:38.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:03:38.936 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:03:38.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.936 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:11903 7.0 7:0e9989e2:::benchmark_data_smithi052_71865_object11902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:03:38.936 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:26.879291+0000", 2024-09-13T13:03:38.936 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.054121909000003, 2024-09-13T13:03:38.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066779187000000004, 2024-09-13T13:03:38.936 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:38.936 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:38.937 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:38.937 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:38.937 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:38.937 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:38.937 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 11903 2024-09-13T13:03:38.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.937 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:38.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:38.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.879291+0000", 2024-09-13T13:03:38.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.879291+0000", 2024-09-13T13:03:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.879299+0000", 2024-09-13T13:03:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0700000000000007e-06 2024-09-13T13:03:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.879332+0000", 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2434999999999997e-05 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.879346+0000", 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3988999999999999e-05 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.879351+0000", 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7989999999999999e-06 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:38.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.879407+0000", 2024-09-13T13:03:38.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5955999999999998e-05 2024-09-13T13:03:38.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:38.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.879501+0000", 2024-09-13T13:03:38.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3927999999999999e-05 2024-09-13T13:03:38.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.879612+0000", 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110623 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944816+0000", 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065204032999999995 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.945345+0000", 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00052924799999999996 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:38.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.945979+0000", 2024-09-13T13:03:38.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00063349599999999999 2024-09-13T13:03:38.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:38.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.946026+0000", 2024-09-13T13:03:38.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6986e-05 2024-09-13T13:03:38.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:38.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.946070+0000", 2024-09-13T13:03:38.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4623999999999998e-05 2024-09-13T13:03:38.943 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.943 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:38.943 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.943 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.34781.0:11909 7.e e89/84 7:7b5c689f:::benchmark_data_smithi052_71865_object11908:head v 89'745, mlcod=89'743)", 2024-09-13T13:03:38.943 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:26.881560+0000", 2024-09-13T13:03:38.943 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.051853483000002, 2024-09-13T13:03:38.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064379381999999999, 2024-09-13T13:03:38.943 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:38.943 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:38.944 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:38.944 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:38.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:38.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.881560+0000", 2024-09-13T13:03:38.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:38.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.881560+0000", 2024-09-13T13:03:38.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:38.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.881561+0000", 2024-09-13T13:03:38.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.531e-06 2024-09-13T13:03:38.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:38.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.881645+0000", 2024-09-13T13:03:38.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4148999999999997e-05 2024-09-13T13:03:38.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:38.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944275+0000", 2024-09-13T13:03:38.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062629801999999998 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944282+0000", 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1369999999999996e-06 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944334+0000", 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2110000000000001e-05 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:38.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944354+0000", 2024-09-13T13:03:38.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9352e-05 2024-09-13T13:03:38.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:38.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.945902+0000", 2024-09-13T13:03:38.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015484629999999999 2024-09-13T13:03:38.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:38.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.945939+0000", 2024-09-13T13:03:38.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6838000000000002e-05 2024-09-13T13:03:38.947 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.947 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:38.948 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.948 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.34781.0:11903 7.0 e89/84 ondisk, result = 0)", 2024-09-13T13:03:38.948 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:26.883363+0000", 2024-09-13T13:03:38.948 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.050049681000004, 2024-09-13T13:03:38.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061459624999999997, 2024-09-13T13:03:38.948 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:38.948 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:38.948 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:03:38.948 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:38.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.883363+0000", 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.883363+0000", 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.883365+0000", 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3769999999999999e-06 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.883370+0000", 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1510000000000003e-06 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944248+0000", 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060878584999999999 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944263+0000", 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4820999999999999e-05 2024-09-13T13:03:38.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:38.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944802+0000", 2024-09-13T13:03:38.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00053890000000000003 2024-09-13T13:03:38.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:38.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944809+0000", 2024-09-13T13:03:38.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0439999999999998e-06 2024-09-13T13:03:38.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:38.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944823+0000", 2024-09-13T13:03:38.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3747000000000001e-05 2024-09-13T13:03:38.953 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.953 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:38.953 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.953 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:11926 7.0 7:0a0dc003:::benchmark_data_smithi052_71865_object11925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:03:38.953 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:26.892293+0000", 2024-09-13T13:03:38.953 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.041120262999996, 2024-09-13T13:03:38.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057766458, 2024-09-13T13:03:38.953 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:38.953 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:38.953 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:38.954 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:38.954 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:38.954 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:38.954 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 11926 2024-09-13T13:03:38.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.954 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:38.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:38.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.892293+0000", 2024-09-13T13:03:38.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:38.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.892293+0000", 2024-09-13T13:03:38.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:38.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.892302+0000", 2024-09-13T13:03:38.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5910000000000002e-06 2024-09-13T13:03:38.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:38.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.892417+0000", 2024-09-13T13:03:38.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114997 2024-09-13T13:03:38.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:38.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944269+0000", 2024-09-13T13:03:38.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051851302000000002 2024-09-13T13:03:38.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:38.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944281+0000", 2024-09-13T13:03:38.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.276e-05 2024-09-13T13:03:38.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:38.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944848+0000", 2024-09-13T13:03:38.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00056652899999999997 2024-09-13T13:03:38.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:38.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.944932+0000", 2024-09-13T13:03:38.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4134000000000005e-05 2024-09-13T13:03:38.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:38.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.945014+0000", 2024-09-13T13:03:38.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1434000000000007e-05 2024-09-13T13:03:38.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:38.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.946078+0000", 2024-09-13T13:03:38.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001064665 2024-09-13T13:03:38.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.946723+0000", 2024-09-13T13:03:38.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064453600000000002 2024-09-13T13:03:38.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.949994+0000", 2024-09-13T13:03:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032714789999999999 2024-09-13T13:03:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.950028+0000", 2024-09-13T13:03:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4248999999999997e-05 2024-09-13T13:03:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:26.950059+0000", 2024-09-13T13:03:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0781999999999998e-05 2024-09-13T13:03:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:19673 7.0 7:05af4aff:::benchmark_data_smithi052_71865_object19672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:03:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:30.047437+0000", 2024-09-13T13:03:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 68.885975517999995, 2024-09-13T13:03:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058306711999999997, 2024-09-13T13:03:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 19673 2024-09-13T13:03:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.047437+0000", 2024-09-13T13:03:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.047437+0000", 2024-09-13T13:03:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.047439+0000", 2024-09-13T13:03:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5990000000000001e-06 2024-09-13T13:03:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.047496+0000", 2024-09-13T13:03:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7024999999999997e-05 2024-09-13T13:03:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.047506+0000", 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0172e-05 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.047513+0000", 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9619999999999996e-06 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.047566+0000", 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2899000000000003e-05 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:38.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.047690+0000", 2024-09-13T13:03:38.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012362399999999999 2024-09-13T13:03:38.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:38.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.047829+0000", 2024-09-13T13:03:38.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000138837 2024-09-13T13:03:38.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:38.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.048677+0000", 2024-09-13T13:03:38.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00084874099999999997 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.049380+0000", 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070307699999999998 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.105647+0000", 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056266271999999999 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:38.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.105691+0000", 2024-09-13T13:03:38.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4564999999999998e-05 2024-09-13T13:03:38.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:38.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:30.105744+0000", 2024-09-13T13:03:38.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2939000000000003e-05 2024-09-13T13:03:38.966 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.966 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:38.966 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.966 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:22471 7.0 7:088e768e:::benchmark_data_smithi052_71865_object22470:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:31.283094+0000", 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.650318830000003, 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058645054000000002, 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22471 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:38.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.283094+0000", 2024-09-13T13:03:38.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:38.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.283094+0000", 2024-09-13T13:03:38.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:38.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.283096+0000", 2024-09-13T13:03:38.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.869e-06 2024-09-13T13:03:38.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.283129+0000", 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2679999999999999e-05 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.283139+0000", 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0183999999999999e-05 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.283149+0000", 2024-09-13T13:03:38.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6290000000000005e-06 2024-09-13T13:03:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.283299+0000", 2024-09-13T13:03:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015084699999999999 2024-09-13T13:03:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.283403+0000", 2024-09-13T13:03:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010325 2024-09-13T13:03:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:38.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.283521+0000", 2024-09-13T13:03:38.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011874099999999999 2024-09-13T13:03:38.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:38.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.284447+0000", 2024-09-13T13:03:38.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092591899999999998 2024-09-13T13:03:38.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.285253+0000", 2024-09-13T13:03:38.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080586799999999995 2024-09-13T13:03:38.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.341666+0000", 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056412564999999998 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.341710+0000", 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4277000000000003e-05 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.341739+0000", 2024-09-13T13:03:38.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9224999999999999e-05 2024-09-13T13:03:38.973 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.973 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:38.973 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.973 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:28953 7.0 7:0c1a46f8:::benchmark_data_smithi052_71865_object28952:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-13T13:03:38.973 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:34.442560+0000", 2024-09-13T13:03:38.973 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.490852523000001, 2024-09-13T13:03:38.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055681873999999999, 2024-09-13T13:03:38.973 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:38.973 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:38.973 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:38.973 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 28953 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.442560+0000", 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.442560+0000", 2024-09-13T13:03:38.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:38.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.442563+0000", 2024-09-13T13:03:38.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4959999999999999e-06 2024-09-13T13:03:38.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:38.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.442618+0000", 2024-09-13T13:03:38.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5074000000000002e-05 2024-09-13T13:03:38.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:38.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.442630+0000", 2024-09-13T13:03:38.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1793999999999999e-05 2024-09-13T13:03:38.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:38.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.442637+0000", 2024-09-13T13:03:38.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:03:38.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:38.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.442689+0000", 2024-09-13T13:03:38.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1536999999999998e-05 2024-09-13T13:03:38.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:38.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.442767+0000", 2024-09-13T13:03:38.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7910000000000005e-05 2024-09-13T13:03:38.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:38.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.442863+0000", 2024-09-13T13:03:38.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6508999999999994e-05 2024-09-13T13:03:38.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:38.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.443689+0000", 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00082522699999999999 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.444301+0000", 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00061247600000000004 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.498179+0000", 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053878269999999999 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.498214+0000", 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4876999999999998e-05 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.498242+0000", 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8223999999999999e-05 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:52162 7.0 7:00edfe73:::benchmark_data_smithi052_71865_object52161:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:03:38.979 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:48.657756+0000", 2024-09-13T13:03:38.980 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.275657254000002, 2024-09-13T13:03:38.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053909290999999998, 2024-09-13T13:03:38.980 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:38.980 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:38.980 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:38.980 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:38.980 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:38.980 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:38.980 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 52162 2024-09-13T13:03:38.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.980 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:38.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:38.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.657756+0000", 2024-09-13T13:03:38.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:38.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.657756+0000", 2024-09-13T13:03:38.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:38.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.657763+0000", 2024-09-13T13:03:38.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9399999999999996e-06 2024-09-13T13:03:38.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.657832+0000", 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9795999999999994e-05 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.657849+0000", 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6076e-05 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.657857+0000", 2024-09-13T13:03:38.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9540000000000004e-06 2024-09-13T13:03:38.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:38.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.657921+0000", 2024-09-13T13:03:38.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4202000000000006e-05 2024-09-13T13:03:38.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:38.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.658032+0000", 2024-09-13T13:03:38.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111099 2024-09-13T13:03:38.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:38.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.658159+0000", 2024-09-13T13:03:38.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012701800000000001 2024-09-13T13:03:38.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:38.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.659949+0000", 2024-09-13T13:03:38.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017898110000000001 2024-09-13T13:03:38.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.660687+0000", 2024-09-13T13:03:38.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00073822199999999997 2024-09-13T13:03:38.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.711558+0000", 2024-09-13T13:03:38.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050871001999999998 2024-09-13T13:03:38.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:38.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.711620+0000", 2024-09-13T13:03:38.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2136000000000003e-05 2024-09-13T13:03:38.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:38.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:48.711665+0000", 2024-09-13T13:03:38.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5034999999999998e-05 2024-09-13T13:03:38.986 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.986 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:38.986 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.986 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:60359 7.0 7:01b9a01d:::benchmark_data_smithi052_71865_object60358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T13:03:38.986 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:54.366538+0000", 2024-09-13T13:03:38.986 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 44.566874761000001, 2024-09-13T13:03:38.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051815990999999999, 2024-09-13T13:03:38.986 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:38.986 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:38.986 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 60359 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.366538+0000", 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:38.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.366538+0000", 2024-09-13T13:03:38.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:38.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.366540+0000", 2024-09-13T13:03:38.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.702e-06 2024-09-13T13:03:38.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:38.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.366601+0000", 2024-09-13T13:03:38.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1248999999999996e-05 2024-09-13T13:03:38.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:38.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.366613+0000", 2024-09-13T13:03:38.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1588e-05 2024-09-13T13:03:38.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:38.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.366619+0000", 2024-09-13T13:03:38.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3799999999999999e-06 2024-09-13T13:03:38.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:38.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.366689+0000", 2024-09-13T13:03:38.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9509000000000001e-05 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.366774+0000", 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5134000000000002e-05 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.366870+0000", 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6163999999999996e-05 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:38.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.368544+0000", 2024-09-13T13:03:38.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016744080000000001 2024-09-13T13:03:38.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.372650+0000", 2024-09-13T13:03:38.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041053310000000003 2024-09-13T13:03:38.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:38.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.418212+0000", 2024-09-13T13:03:38.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045562000999999998 2024-09-13T13:03:38.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:38.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.418313+0000", 2024-09-13T13:03:38.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101236 2024-09-13T13:03:38.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:38.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:54.418354+0000", 2024-09-13T13:03:38.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1288999999999997e-05 2024-09-13T13:03:38.992 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.992 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:38.992 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:38.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.993 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:81755 7.0 7:0dcad17d:::benchmark_data_smithi052_71865_object81754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-13T13:03:38.993 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:12.477184+0000", 2024-09-13T13:03:38.993 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 26.456229027999999, 2024-09-13T13:03:38.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.068299632999999998, 2024-09-13T13:03:38.993 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:38.993 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:38.993 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:38.993 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:38.993 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:38.993 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:38.993 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81755 2024-09-13T13:03:38.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.994 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:38.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:38.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.477184+0000", 2024-09-13T13:03:38.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:38.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.477184+0000", 2024-09-13T13:03:38.994 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:38.996 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:38 smithi052 ceph-mon[36682]: 7.5 scrub ok 2024-09-13T13:03:38.996 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:38 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 32072/306861 objects degraded (10.452%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:38.996 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:38 smithi052 ceph-mon[36682]: osdmap e112: 8 total, 7 up, 8 in 2024-09-13T13:03:38.996 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:38 smithi052 ceph-mon[36682]: 3.e scrub starts 2024-09-13T13:03:38.996 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:38 smithi052 ceph-mon[36682]: 3.e scrub ok 2024-09-13T13:03:38.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:38 smithi052 ceph-mon[29471]: 7.5 scrub ok 2024-09-13T13:03:38.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:38 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 32072/306861 objects degraded (10.452%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:38.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:38 smithi052 ceph-mon[29471]: osdmap e112: 8 total, 7 up, 8 in 2024-09-13T13:03:38.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:38 smithi052 ceph-mon[29471]: 3.e scrub starts 2024-09-13T13:03:38.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:38 smithi052 ceph-mon[29471]: 3.e scrub ok 2024-09-13T13:03:38.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:38.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:38.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.477187+0000", 2024-09-13T13:03:38.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.571e-06 2024-09-13T13:03:38.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:38.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.477230+0000", 2024-09-13T13:03:38.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2951999999999998e-05 2024-09-13T13:03:38.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:38.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.477245+0000", 2024-09-13T13:03:38.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5177999999999999e-05 2024-09-13T13:03:38.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:38.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.477252+0000", 2024-09-13T13:03:38.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1609999999999997e-06 2024-09-13T13:03:38.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:38.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:38.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.477328+0000", 2024-09-13T13:03:38.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6465999999999996e-05 2024-09-13T13:03:38.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:38.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.477466+0000", 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013739899999999999 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.477598+0000", 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000132378 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.480325+0000", 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027268180000000002 2024-09-13T13:03:39.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.480397+0000", 2024-09-13T13:03:39.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2507000000000002e-05 2024-09-13T13:03:39.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.545416+0000", 2024-09-13T13:03:39.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065018171999999999 2024-09-13T13:03:39.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:39.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.545450+0000", 2024-09-13T13:03:39.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3970000000000002e-05 2024-09-13T13:03:39.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:39.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:12.545484+0000", 2024-09-13T13:03:39.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4060999999999998e-05 2024-09-13T13:03:39.002 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.002 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:39.002 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.002 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:95511 7.0 7:0c9aa372:::benchmark_data_smithi052_71865_object95510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:03:39.002 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:26.176485+0000", 2024-09-13T13:03:39.003 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 12.75692826, 2024-09-13T13:03:39.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060254872000000001, 2024-09-13T13:03:39.003 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:39.003 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:39.003 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:39.003 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:39.003 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:39.003 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:39.003 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95511 2024-09-13T13:03:39.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.003 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:39.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:39.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176485+0000", 2024-09-13T13:03:39.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:39.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176485+0000", 2024-09-13T13:03:39.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:39.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176486+0000", 2024-09-13T13:03:39.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.575e-06 2024-09-13T13:03:39.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:39.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176579+0000", 2024-09-13T13:03:39.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2351999999999999e-05 2024-09-13T13:03:39.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:39.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176590+0000", 2024-09-13T13:03:39.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1581e-05 2024-09-13T13:03:39.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:39.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176595+0000", 2024-09-13T13:03:39.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0900000000000004e-06 2024-09-13T13:03:39.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:39.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176646+0000", 2024-09-13T13:03:39.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0584000000000002e-05 2024-09-13T13:03:39.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:39.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176754+0000", 2024-09-13T13:03:39.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107821 2024-09-13T13:03:39.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176904+0000", 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000150116 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.181461+0000", 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045575859999999998 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.181882+0000", 2024-09-13T13:03:39.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00042067199999999997 2024-09-13T13:03:39.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.236625+0000", 2024-09-13T13:03:39.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054742544999999997 2024-09-13T13:03:39.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:39.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.236687+0000", 2024-09-13T13:03:39.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2335999999999994e-05 2024-09-13T13:03:39.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:39.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.236740+0000", 2024-09-13T13:03:39.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2614000000000001e-05 2024-09-13T13:03:39.009 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.009 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:39.009 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.009 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:95512 7.0 7:0d65727d:::benchmark_data_smithi052_71865_object95511:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:03:39.009 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:26.176817+0000", 2024-09-13T13:03:39.009 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 12.756595611, 2024-09-13T13:03:39.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062221510000000001, 2024-09-13T13:03:39.009 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:39.009 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95512 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176817+0000", 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176817+0000", 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176819+0000", 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3510000000000001e-06 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176893+0000", 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3960000000000003e-05 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:39.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176905+0000", 2024-09-13T13:03:39.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.185e-05 2024-09-13T13:03:39.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:39.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176913+0000", 2024-09-13T13:03:39.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.994e-06 2024-09-13T13:03:39.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:39.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.179504+0000", 2024-09-13T13:03:39.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025910820000000002 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.179586+0000", 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2350999999999998e-05 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.179680+0000", 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4148999999999996e-05 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:39.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.181577+0000", 2024-09-13T13:03:39.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001896378 2024-09-13T13:03:39.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.182000+0000", 2024-09-13T13:03:39.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00042351299999999998 2024-09-13T13:03:39.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.238939+0000", 2024-09-13T13:03:39.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056939084000000001 2024-09-13T13:03:39.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:39.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.238997+0000", 2024-09-13T13:03:39.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8118000000000002e-05 2024-09-13T13:03:39.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:39.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.239039+0000", 2024-09-13T13:03:39.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1680000000000001e-05 2024-09-13T13:03:39.015 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.015 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:39.015 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.016 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:96027 7.0 7:05adc275:::benchmark_data_smithi052_71865_object96026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:03:39.016 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:26.822162+0000", 2024-09-13T13:03:39.016 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 12.111250567999999, 2024-09-13T13:03:39.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063214371000000005, 2024-09-13T13:03:39.016 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:39.016 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:39.016 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:39.016 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:39.016 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:39.016 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:39.016 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 96027 2024-09-13T13:03:39.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.016 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822162+0000", 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822162+0000", 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822164+0000", 2024-09-13T13:03:39.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.933e-06 2024-09-13T13:03:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822224+0000", 2024-09-13T13:03:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9208e-05 2024-09-13T13:03:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822237+0000", 2024-09-13T13:03:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2981e-05 2024-09-13T13:03:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822243+0000", 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3539999999999996e-06 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822293+0000", 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9925000000000001e-05 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822384+0000", 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0672000000000002e-05 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:39.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822479+0000", 2024-09-13T13:03:39.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5152e-05 2024-09-13T13:03:39.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:39.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.827434+0000", 2024-09-13T13:03:39.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049551270000000001 2024-09-13T13:03:39.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.832897+0000", 2024-09-13T13:03:39.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005462822 2024-09-13T13:03:39.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.885299+0000", 2024-09-13T13:03:39.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052402210999999997 2024-09-13T13:03:39.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:39.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.885341+0000", 2024-09-13T13:03:39.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2660000000000002e-05 2024-09-13T13:03:39.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.885377+0000", 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5326e-05 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:96028 7.0 7:00ebffb0:::benchmark_data_smithi052_71865_object96027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:26.822265+0000", 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 12.111147696, 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066489336999999996, 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:39.022 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:39.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:39.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:39.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:39.023 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 96028 2024-09-13T13:03:39.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.023 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:39.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:39.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822265+0000", 2024-09-13T13:03:39.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:39.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822265+0000", 2024-09-13T13:03:39.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:39.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822266+0000", 2024-09-13T13:03:39.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2600000000000001e-07 2024-09-13T13:03:39.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:39.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822305+0000", 2024-09-13T13:03:39.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9289999999999998e-05 2024-09-13T13:03:39.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822311+0000", 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7980000000000002e-06 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.822316+0000", 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8189999999999999e-06 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.824943+0000", 2024-09-13T13:03:39.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026265199999999998 2024-09-13T13:03:39.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:39.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.825067+0000", 2024-09-13T13:03:39.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124747 2024-09-13T13:03:39.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:39.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.825160+0000", 2024-09-13T13:03:39.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2946000000000001e-05 2024-09-13T13:03:39.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.827543+0000", 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002382983 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.832927+0000", 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0053837099999999999 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.888692+0000", 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055764783999999998 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:39.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.888727+0000", 2024-09-13T13:03:39.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.489e-05 2024-09-13T13:03:39.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:39.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.888755+0000", 2024-09-13T13:03:39.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8024000000000001e-05 2024-09-13T13:03:39.028 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.028 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:39.028 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.029 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:96043 7.0 7:08cf77e5:::benchmark_data_smithi052_71865_object96042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:03:39.029 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:26.834010+0000", 2024-09-13T13:03:39.029 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 12.099402662999999, 2024-09-13T13:03:39.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057868199000000002, 2024-09-13T13:03:39.029 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:39.029 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:39.029 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:39.029 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:39.029 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:39.029 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:39.029 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 96043 2024-09-13T13:03:39.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.029 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.834010+0000", 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.834010+0000", 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.834012+0000", 2024-09-13T13:03:39.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.968e-06 2024-09-13T13:03:39.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:39.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.834076+0000", 2024-09-13T13:03:39.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4138000000000002e-05 2024-09-13T13:03:39.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:39.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.834090+0000", 2024-09-13T13:03:39.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3558e-05 2024-09-13T13:03:39.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.834097+0000", 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2599999999999999e-06 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.834146+0000", 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9061999999999999e-05 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.834242+0000", 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5774999999999997e-05 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:39.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.834341+0000", 2024-09-13T13:03:39.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8753999999999997e-05 2024-09-13T13:03:39.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:39.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.836403+0000", 2024-09-13T13:03:39.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020620859999999999 2024-09-13T13:03:39.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.836828+0000", 2024-09-13T13:03:39.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000425518 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.891749+0000", 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054920805000000003 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.891834+0000", 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5118999999999996e-05 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:39.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.891879+0000", 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4156000000000002e-05 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:96045 7.0 7:05d3eeee:::benchmark_data_smithi052_71865_object96044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:26.837034+0000", 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 12.096379027999999, 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054980108, 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:39.035 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 96045 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.837034+0000", 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.837034+0000", 2024-09-13T13:03:39.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.837035+0000", 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3060000000000001e-06 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.837088+0000", 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3127000000000001e-05 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:39.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.837098+0000", 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5689999999999994e-06 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.837103+0000", 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0599999999999998e-06 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.837140+0000", 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7017999999999997e-05 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:39.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.837221+0000", 2024-09-13T13:03:39.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1184000000000001e-05 2024-09-13T13:03:39.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:39.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.837311+0000", 2024-09-13T13:03:39.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9407000000000006e-05 2024-09-13T13:03:39.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:39.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.839051+0000", 2024-09-13T13:03:39.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017408300000000001 2024-09-13T13:03:39.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.839578+0000", 2024-09-13T13:03:39.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00052640800000000002 2024-09-13T13:03:39.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.891925+0000", 2024-09-13T13:03:39.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052346671999999997 2024-09-13T13:03:39.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:39.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.891978+0000", 2024-09-13T13:03:39.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3369999999999999e-05 2024-09-13T13:03:39.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:39.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.892014+0000", 2024-09-13T13:03:39.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6157e-05 2024-09-13T13:03:39.041 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.041 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:39.041 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.041 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:97571 7.0 7:049a00a1:::benchmark_data_smithi052_71865_object97570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:03:39.041 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:28.456045+0000", 2024-09-13T13:03:39.042 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.477367889, 2024-09-13T13:03:39.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051193048999999997, 2024-09-13T13:03:39.042 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:39.042 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:39.042 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:39.042 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:39.042 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:39.042 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:39.042 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 97571 2024-09-13T13:03:39.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.042 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:39.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.456045+0000", 2024-09-13T13:03:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.456045+0000", 2024-09-13T13:03:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.456047+0000", 2024-09-13T13:03:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0779999999999998e-06 2024-09-13T13:03:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.456107+0000", 2024-09-13T13:03:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9960999999999997e-05 2024-09-13T13:03:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.456119+0000", 2024-09-13T13:03:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1778e-05 2024-09-13T13:03:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.456127+0000", 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5089999999999993e-06 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.458688+0000", 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002560321 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.458793+0000", 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010491 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:39.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.458891+0000", 2024-09-13T13:03:39.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7827999999999999e-05 2024-09-13T13:03:39.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:39.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.461189+0000", 2024-09-13T13:03:39.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022985140000000002 2024-09-13T13:03:39.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.496514+0000", 2024-09-13T13:03:39.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035325018999999999 2024-09-13T13:03:39.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.507119+0000", 2024-09-13T13:03:39.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010604674999999999 2024-09-13T13:03:39.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:39.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.507186+0000", 2024-09-13T13:03:39.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7329000000000005e-05 2024-09-13T13:03:39.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:39.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:28.507238+0000", 2024-09-13T13:03:39.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2126999999999997e-05 2024-09-13T13:03:39.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.048 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:39.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.048 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:104135 7.0 7:0acc5318:::benchmark_data_smithi052_71865_object104134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-13T13:03:39.048 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:34.963762+0000", 2024-09-13T13:03:39.048 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.9696508719999999, 2024-09-13T13:03:39.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050218687999999997, 2024-09-13T13:03:39.048 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:39.049 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:39.049 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:39.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:39.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:39.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:39.049 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 104135 2024-09-13T13:03:39.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.049 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:39.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:39.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:34.963762+0000", 2024-09-13T13:03:39.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:39.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:34.963762+0000", 2024-09-13T13:03:39.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:39.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:34.963772+0000", 2024-09-13T13:03:39.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9410000000000002e-06 2024-09-13T13:03:39.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:39.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:34.963822+0000", 2024-09-13T13:03:39.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9950000000000001e-05 2024-09-13T13:03:39.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:39.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:34.963841+0000", 2024-09-13T13:03:39.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8924999999999999e-05 2024-09-13T13:03:39.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:39.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:34.963850+0000", 2024-09-13T13:03:39.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0969999999999995e-06 2024-09-13T13:03:39.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:34.963922+0000", 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2065999999999997e-05 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:34.964045+0000", 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012271999999999999 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:34.964185+0000", 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032399999999999 2024-09-13T13:03:39.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:34.969032+0000", 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048469330000000003 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:34.971903+0000", 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028708140000000002 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:35.013910+0000", 2024-09-13T13:03:39.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042007204999999999 2024-09-13T13:03:39.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:39.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:35.013947+0000", 2024-09-13T13:03:39.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6752999999999999e-05 2024-09-13T13:03:39.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:39.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:35.013981+0000", 2024-09-13T13:03:39.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.396e-05 2024-09-13T13:03:39.054 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.054 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:39.054 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.055 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:105216 7.0 7:0c7da74e:::benchmark_data_smithi052_71865_object105215:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-13T13:03:39.055 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:36.018754+0000", 2024-09-13T13:03:39.055 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.9146592289999997, 2024-09-13T13:03:39.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073909959999999997, 2024-09-13T13:03:39.055 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:39.055 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:39.055 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:39.055 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:39.055 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:39.055 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:39.055 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105216 2024-09-13T13:03:39.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.056 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:39.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:39.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018754+0000", 2024-09-13T13:03:39.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:39.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018754+0000", 2024-09-13T13:03:39.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018756+0000", 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.711e-06 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018808+0000", 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1094000000000002e-05 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018826+0000", 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8502e-05 2024-09-13T13:03:39.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:39.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018834+0000", 2024-09-13T13:03:39.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8099999999999998e-06 2024-09-13T13:03:39.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:39.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018901+0000", 2024-09-13T13:03:39.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6760000000000005e-05 2024-09-13T13:03:39.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:39.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.019041+0000", 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140474 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.019196+0000", 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015521 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.022153+0000", 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029568210000000001 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023370+0000", 2024-09-13T13:03:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001216658 2024-09-13T13:03:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.092552+0000", 2024-09-13T13:03:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069182612000000004 2024-09-13T13:03:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.092613+0000", 2024-09-13T13:03:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0387999999999998e-05 2024-09-13T13:03:39.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.092664+0000", 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.092e-05 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:105223 7.0 7:0994a0f1:::benchmark_data_smithi052_71865_object105222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:36.024390+0000", 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.9090230950000002, 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.068446813999999995, 2024-09-13T13:03:39.061 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105223 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.024390+0000", 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.024390+0000", 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.024392+0000", 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3700000000000002e-06 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.024448+0000", 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6017999999999998e-05 2024-09-13T13:03:39.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:03:39.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.024464+0000", 2024-09-13T13:03:39.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5888000000000001e-05 2024-09-13T13:03:39.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:03:39.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.024472+0000", 2024-09-13T13:03:39.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2849999999999995e-06 2024-09-13T13:03:39.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:03:39.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.024537+0000", 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4461000000000005e-05 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.024685+0000", 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014793299999999999 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.024853+0000", 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016855599999999999 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.027809+0000", 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029555810000000001 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.028179+0000", 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037040899999999998 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.092727+0000", 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064547160000000006 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.092791+0000", 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4529000000000005e-05 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.092837+0000", 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5624000000000002e-05 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:03:39.067 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:03:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:39.069 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:39.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:39.169 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 77 16 108358 108342 87.9011 60.1875 0.0200232 0.011371 2024-09-13T13:03:39.171 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:39.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:39.333 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:39.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:39.393 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:39.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:39.438 INFO:teuthology.orchestra.run.smithi167.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-13T13:03:39.495 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:39.519 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:39.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:39.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:39.557 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:03:39.621 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:39.674 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[29471]: 7.8 scrub starts 2024-09-13T13:03:39.674 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[29471]: 3.b scrub starts 2024-09-13T13:03:39.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[29471]: 3.b scrub ok 2024-09-13T13:03:39.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[29471]: pgmap v106: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 6.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 33659/322065 objects degraded (10.451%) 2024-09-13T13:03:39.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:03:39.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:03:39.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[29471]: 7.e scrub ok 2024-09-13T13:03:39.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[36682]: 7.8 scrub starts 2024-09-13T13:03:39.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[36682]: 3.b scrub starts 2024-09-13T13:03:39.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[36682]: 3.b scrub ok 2024-09-13T13:03:39.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[36682]: pgmap v106: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 6.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 33659/322065 objects degraded (10.451%) 2024-09-13T13:03:39.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:03:39.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:03:39.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:39 smithi052 ceph-mon[36682]: 7.e scrub ok 2024-09-13T13:03:39.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:39.685 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:39.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:39.786 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:39.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:39 smithi167 ceph-mon[33178]: 7.8 scrub starts 2024-09-13T13:03:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:39 smithi167 ceph-mon[33178]: 3.b scrub starts 2024-09-13T13:03:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:39 smithi167 ceph-mon[33178]: 3.b scrub ok 2024-09-13T13:03:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:39 smithi167 ceph-mon[33178]: pgmap v106: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 6.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 33659/322065 objects degraded (10.451%) 2024-09-13T13:03:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:39 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:03:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:39 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:03:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:39 smithi167 ceph-mon[33178]: 7.e scrub ok 2024-09-13T13:03:39.849 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:39.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:39.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:39.976 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:39.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:39.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:40.077 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:40.078 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:40.160 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:40.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:40.171 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 78 16 109158 109142 87.413 50 0.0194726 0.011433 2024-09-13T13:03:40.263 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:40.324 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:40.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:40.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:40.426 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:40.447 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:40.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:40.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:40.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:40.574 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:40.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:40.645 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:03:40.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:40.700 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:40.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:40.802 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:40.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:40.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:40.873 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:40 smithi052 ceph-mon[36682]: 7.3 scrub starts 2024-09-13T13:03:40.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:40 smithi052 ceph-mon[36682]: osdmap e113: 8 total, 7 up, 8 in 2024-09-13T13:03:40.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:40 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2418365176' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:40.874 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:40 smithi052 ceph-mon[29471]: 7.3 scrub starts 2024-09-13T13:03:40.874 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:40 smithi052 ceph-mon[29471]: osdmap e113: 8 total, 7 up, 8 in 2024-09-13T13:03:40.874 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:40 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2418365176' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:40.905 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:40 smithi167 ceph-mon[33178]: 7.3 scrub starts 2024-09-13T13:03:40.905 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:40 smithi167 ceph-mon[33178]: osdmap e113: 8 total, 7 up, 8 in 2024-09-13T13:03:40.905 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:40 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2418365176' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:40.930 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:40.974 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:40.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:40.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:41.076 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:41.122 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:41.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:41.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:41.171 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 79 16 110042 110026 87.006 55.25 0.0149786 0.0114886 2024-09-13T13:03:41.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:41.248 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:41.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:41.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:41.350 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:41.390 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:41.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:41.492 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:41.533 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:41.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:41.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:41.635 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:41.687 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:41.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:41.789 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:41.832 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:41.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:41.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:41.901 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:03:41.902 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:03:41.902 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:03:41.902 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:03:41.902 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[36682]: 7.8 scrub ok 2024-09-13T13:03:41.902 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:03:41.902 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2418365176' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:41.903 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[36682]: osdmap e114: 8 total, 7 up, 8 in 2024-09-13T13:03:41.903 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[36682]: 3.1c scrub starts 2024-09-13T13:03:41.903 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[36682]: 3.1c scrub ok 2024-09-13T13:03:41.903 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[36682]: pgmap v109: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 6.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 845 op/s; 33659/322065 objects degraded (10.451%) 2024-09-13T13:03:41.903 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:03:41.903 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:03:41.903 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[29471]: 7.8 scrub ok 2024-09-13T13:03:41.903 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:03:41.903 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2418365176' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:41.903 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[29471]: osdmap e114: 8 total, 7 up, 8 in 2024-09-13T13:03:41.903 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[29471]: 3.1c scrub starts 2024-09-13T13:03:41.904 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[29471]: 3.1c scrub ok 2024-09-13T13:03:41.904 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:41 smithi052 ceph-mon[29471]: pgmap v109: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 6.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 845 op/s; 33659/322065 objects degraded (10.451%) 2024-09-13T13:03:41.935 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:41.941 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:41 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:03:41.941 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:41 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:03:41.941 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:41 smithi167 ceph-mon[33178]: 7.8 scrub ok 2024-09-13T13:03:41.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:41 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:03:41.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:41 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2418365176' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:41.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:41 smithi167 ceph-mon[33178]: osdmap e114: 8 total, 7 up, 8 in 2024-09-13T13:03:41.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:41 smithi167 ceph-mon[33178]: 3.1c scrub starts 2024-09-13T13:03:41.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:41 smithi167 ceph-mon[33178]: 3.1c scrub ok 2024-09-13T13:03:41.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:41 smithi167 ceph-mon[33178]: pgmap v109: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 6.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 845 op/s; 33659/322065 objects degraded (10.451%) 2024-09-13T13:03:41.943 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:03:41.943 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:03:41.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:41.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:41.984 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:03:41.984 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:03:42.021 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:03:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:03:42.053 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:03:42.053 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:03:42.082 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:42.086 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:03:42.086 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:03:42.088 INFO:teuthology.orchestra.run.smithi052.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T13:03:42.123 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:03:42.124 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:03:42.131 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:42.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:42.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:42.171 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:03:42.171896+0000 min lat: 0.00146492 max lat: 1.03648 avg lat: 0.01153 2024-09-13T13:03:42.172 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:03:42.172 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 80 16 111033 111017 86.6927 61.9375 0.0234344 0.01153 2024-09-13T13:03:42.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:42.263 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:42.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:42.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:42.365 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:42.405 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:42.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:42.507 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:42.550 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:42.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:42.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:42.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T13:03:42.652 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:42.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:42.702 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:42.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:42.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:42 smithi052 ceph-mon[36682]: 5.15 scrub starts 2024-09-13T13:03:42.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:42 smithi052 ceph-mon[36682]: 5.15 scrub ok 2024-09-13T13:03:42.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:42 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2290689538' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:42.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:42 smithi052 ceph-mon[36682]: 7.3 scrub ok 2024-09-13T13:03:42.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:42 smithi052 ceph-mon[36682]: 3.8 scrub starts 2024-09-13T13:03:42.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:42 smithi052 ceph-mon[36682]: 3.8 scrub ok 2024-09-13T13:03:42.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:42 smithi052 ceph-mon[29471]: 5.15 scrub starts 2024-09-13T13:03:42.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:42 smithi052 ceph-mon[29471]: 5.15 scrub ok 2024-09-13T13:03:42.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2290689538' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:42.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:42 smithi052 ceph-mon[29471]: 7.3 scrub ok 2024-09-13T13:03:42.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:42 smithi052 ceph-mon[29471]: 3.8 scrub starts 2024-09-13T13:03:42.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:42 smithi052 ceph-mon[29471]: 3.8 scrub ok 2024-09-13T13:03:42.792 INFO:teuthology.orchestra.run.smithi167.stdout:get_pg_num_history pg_num_history pg_num_history(e102 pg_nums {1={20=1},2={59=3},3={59=32},4={61=32},5={63=32},6={65=32},7={84=16}} deleted_pools ) 2024-09-13T13:03:42.792 INFO:teuthology.orchestra.run.smithi167.stdout:Importing pgid 3.1e 2024-09-13T13:03:42.792 INFO:teuthology.orchestra.run.smithi167.stdout:write_pg epoch 94 info 3.1e( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) 2024-09-13T13:03:42.792 INFO:teuthology.orchestra.run.smithi167.stdout:Import successful 2024-09-13T13:03:42.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:42.838 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:42.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:42.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:42.940 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:42.957 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:03:42.974 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:42.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:42.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:43.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:42 smithi167 ceph-mon[33178]: 5.15 scrub starts 2024-09-13T13:03:43.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:42 smithi167 ceph-mon[33178]: 5.15 scrub ok 2024-09-13T13:03:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:42 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2290689538' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:42 smithi167 ceph-mon[33178]: 7.3 scrub ok 2024-09-13T13:03:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:42 smithi167 ceph-mon[33178]: 3.8 scrub starts 2024-09-13T13:03:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:42 smithi167 ceph-mon[33178]: 3.8 scrub ok 2024-09-13T13:03:43.077 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:43.117 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:43.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:43.172 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 81 16 112176 112160 86.5043 71.4375 0.00739654 0.0115553 2024-09-13T13:03:43.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:43.244 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:43.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:43.346 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:43.386 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:43.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:03:43.449 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:03:43.488 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:43.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:43.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:43.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:43.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:43.646 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:03:43.660 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:43.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:43.662 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:03:43 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:03:43.663 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:43 smithi052 ceph-mon[29471]: pgmap v110: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 6.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 845 op/s; 33659/322065 objects degraded (10.451%) 2024-09-13T13:03:43.762 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:43.764 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:03:43.765 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:03:43.810 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:43.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:43.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:43.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:43.942 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:43.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:43.965 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:43 smithi167 ceph-mon[33178]: pgmap v110: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 6.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 845 op/s; 33659/322065 objects degraded (10.451%) 2024-09-13T13:03:43.965 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:43 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 33659/322065 objects degraded (10.451%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:43.965 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:43 smithi167 ceph-mon[33178]: 7.a scrub starts 2024-09-13T13:03:43.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:43 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 33659/322065 objects degraded (10.451%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:43.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:43 smithi052 ceph-mon[29471]: 7.a scrub starts 2024-09-13T13:03:43.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:43 smithi052 ceph-mon[36682]: pgmap v110: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 6.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 845 op/s; 33659/322065 objects degraded (10.451%) 2024-09-13T13:03:43.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:43 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 33659/322065 objects degraded (10.451%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:43.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:43 smithi052 ceph-mon[36682]: 7.a scrub starts 2024-09-13T13:03:44.044 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:44.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:44.086 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:44.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:44.134 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:44.173 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 82 16 113196 113180 86.2249 63.75 0.0270527 0.0115923 2024-09-13T13:03:44.188 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:44.235 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:44.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:44.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:44.336 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:44.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:44.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:44.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:44.432 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T13:03:44.469 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:44.505 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:44.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:44.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:44.648 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:44.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:44.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:44.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:44.774 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:44.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:44.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:44.876 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:44.915 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:44.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:44.924 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:44 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:44.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/exp.3.1e.6 2024-09-13T13:03:44.981 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:03:44.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:44 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:44 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:45.018 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:45.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:45.062 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:45.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:45.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:45.174 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 83 16 114143 114127 85.8992 59.1875 0.0292197 0.0116366 2024-09-13T13:03:45.189 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:45.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:45.291 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:45.320 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:45.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:45.421 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:45.451 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:03:45.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:45.464 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:45.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:45.556 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:03:45.556 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:03:45.566 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:45.610 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:45.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:45.673 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:03:45.711 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:45.759 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:45.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:45.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:45.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:45 smithi167 ceph-mon[33178]: 5.12 scrub starts 2024-09-13T13:03:45.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:45 smithi167 ceph-mon[33178]: 5.12 scrub ok 2024-09-13T13:03:45.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:45 smithi167 ceph-mon[33178]: pgmap v111: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 89 MiB/s wr, 1.42k op/s; 35158/336423 objects degraded (10.451%) 2024-09-13T13:03:45.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:45 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/524843342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:03:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:45 smithi167 ceph-mon[33178]: 7.2 scrub starts 2024-09-13T13:03:45.861 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:45.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:45.903 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:45.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:45 smithi052 ceph-mon[36682]: 5.12 scrub starts 2024-09-13T13:03:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:45 smithi052 ceph-mon[36682]: 5.12 scrub ok 2024-09-13T13:03:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:45 smithi052 ceph-mon[36682]: pgmap v111: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 89 MiB/s wr, 1.42k op/s; 35158/336423 objects degraded (10.451%) 2024-09-13T13:03:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:45 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/524843342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:03:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:45 smithi052 ceph-mon[36682]: 7.2 scrub starts 2024-09-13T13:03:45.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:45 smithi052 ceph-mon[29471]: 5.12 scrub starts 2024-09-13T13:03:45.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:45 smithi052 ceph-mon[29471]: 5.12 scrub ok 2024-09-13T13:03:45.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:45 smithi052 ceph-mon[29471]: pgmap v111: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 89 MiB/s wr, 1.42k op/s; 35158/336423 objects degraded (10.451%) 2024-09-13T13:03:45.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:45 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/524843342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:03:45.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:45 smithi052 ceph-mon[29471]: 7.2 scrub starts 2024-09-13T13:03:46.004 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:46.030 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:46.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:46.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:46.132 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:46.174 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 84 16 115161 115145 85.6341 63.625 0.0101915 0.0116726 2024-09-13T13:03:46.182 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:46.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:46.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:46.284 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:46.325 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:46.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:46.427 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:46.448 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T13:03:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:46.453 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:46.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:46.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:03:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:03:46.554 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:46.591 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:46.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:46.675 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:46 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:46.675 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:46 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/524843342' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:03:46.691 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:46.720 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:46.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:46.763 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:03:46.821 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:46.869 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:46.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:46.933 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:46 smithi167 ceph-mon[33178]: osdmap e115: 8 total, 7 up, 8 in 2024-09-13T13:03:46.970 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:46 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:46 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/524843342' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:03:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:46 smithi052 ceph-mon[36682]: osdmap e115: 8 total, 7 up, 8 in 2024-09-13T13:03:46.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:46 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:46.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:46 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/524843342' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:03:46.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:46 smithi052 ceph-mon[29471]: osdmap e115: 8 total, 7 up, 8 in 2024-09-13T13:03:46.992 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:46.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:47.094 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:47.130 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:47.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:47.174 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 85 16 116250 116234 85.4273 68.0625 0.0094851 0.0117007 2024-09-13T13:03:47.232 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:47.262 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:47.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:47.348 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:03:47.348 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:03:47.363 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:47.399 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:47.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:47.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:47.502 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:47.526 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:47.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:47.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:47.629 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:47.653 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:47.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:47.755 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:47.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:47.806 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:47.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:47.891 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:47.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:47.930 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:47.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:47.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:47.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:47 smithi052 ceph-mon[29471]: pgmap v113: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 725 B/s rd, 42 MiB/s wr, 679 op/s; 35158/336423 objects degraded (10.451%) 2024-09-13T13:03:47.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:47 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:03:47.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:47 smithi052 ceph-mon[36682]: pgmap v113: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 725 B/s rd, 42 MiB/s wr, 679 op/s; 35158/336423 objects degraded (10.451%) 2024-09-13T13:03:47.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:47 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:03:48.032 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:48.051 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:47 smithi167 ceph-mon[33178]: pgmap v113: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 725 B/s rd, 42 MiB/s wr, 679 op/s; 35158/336423 objects degraded (10.451%) 2024-09-13T13:03:48.051 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:47 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:03:48.058 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:48.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:48.090 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T13:03:48.160 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:48.174 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 86 16 117349 117333 85.2326 68.6875 0.0179903 0.0117271 2024-09-13T13:03:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:48.210 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:48.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:48.311 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:48.360 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:48.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:48.461 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:48.482 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:48.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:48.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:48.584 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:48.626 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:48.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:48.727 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:48.770 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:48.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:48.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:48.872 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:48.896 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:48.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:48.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:48.944 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:03:48.945 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:03:48.945 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:03:48.945 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:03:48.945 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:03:48.945 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:03:48.946 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:03:48.946 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:03:48.946 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:03:48.947 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:03:48.947 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:03:48.947 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:03:48.947 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:03:48.947 INFO:teuthology.orchestra.run.smithi052.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T13:03:48.947 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:03:48.948 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:48 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 35158/336423 objects degraded (10.451%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:48.948 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:48 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1428360932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:48.949 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:48 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 35158/336423 objects degraded (10.451%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:48.949 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:48 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1428360932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:48.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:49.022 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:49.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:49.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:49.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:48 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 35158/336423 objects degraded (10.451%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:49.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:48 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1428360932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:49.125 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:49.149 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:49.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:49.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:49.174 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 87 16 118289 118273 84.9283 58.75 0.00938718 0.0117695 2024-09-13T13:03:49.252 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:49.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:49.274 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:49.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:49.375 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:49.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:49.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:49.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:49.501 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:49.538 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:49.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:49.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:49.640 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:49.661 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:49.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:49.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:49.763 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:49.797 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:03:49.806 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:49.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:49.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:49.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:49.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:49.931 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:49.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:49.961 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-09-13T13:03:49.961 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-09-13T13:03:49.961 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-09-13T13:03:49.961 DEBUG:teuthology.orchestra.run.smithi167:osd.6> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.6.service 2024-09-13T13:03:49.974 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl start ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.6 2024-09-13T13:03:49.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:49 smithi052 ceph-mon[36682]: pgmap v114: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s; 36833/351999 objects degraded (10.464%) 2024-09-13T13:03:49.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:49 smithi052 ceph-mon[29471]: pgmap v114: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s; 36833/351999 objects degraded (10.464%) 2024-09-13T13:03:49.994 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:49 smithi167 ceph-mon[33178]: pgmap v114: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s; 36833/351999 objects degraded (10.464%) 2024-09-13T13:03:50.032 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:50.063 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:50.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:50.165 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:50.175 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 88 16 119218 119202 84.6217 58.0625 0.00990656 0.0118123 2024-09-13T13:03:50.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:50.210 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:50.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:50.289 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:50 smithi167 systemd[1]: Starting Ceph osd.6 for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:03:50.311 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:50.353 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:50.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:50.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:50.455 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:50.488 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:50.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:50.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:50 smithi167 podman[71612]: 2024-09-13 13:03:50.192563579 +0000 UTC m=+0.019915365 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:03:50.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:50 smithi167 podman[71612]: 2024-09-13 13:03:50.317984085 +0000 UTC m=+0.145335861 container create c1fa0461298bf4d7268651d750c20ddc94db4fb2285029fc86d55715b3bf5957 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-13T13:03:50.589 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:50.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:50.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:50.718 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:50.744 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:50.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:50.846 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:50.891 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:50.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:50.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:51.019 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:51.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:51.071 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:50 smithi167 podman[71612]: 2024-09-13 13:03:50.580115383 +0000 UTC m=+0.407467157 container init c1fa0461298bf4d7268651d750c20ddc94db4fb2285029fc86d55715b3bf5957 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-13T13:03:51.071 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:50 smithi167 podman[71612]: 2024-09-13 13:03:50.584701387 +0000 UTC m=+0.412053161 container start c1fa0461298bf4d7268651d750c20ddc94db4fb2285029fc86d55715b3bf5957 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True) 2024-09-13T13:03:51.071 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:50 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate[71627]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T13:03:51.071 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:50 smithi167 bash[71612]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T13:03:51.071 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:50 smithi167 podman[71612]: 2024-09-13 13:03:50.763627233 +0000 UTC m=+0.590979013 container attach c1fa0461298bf4d7268651d750c20ddc94db4fb2285029fc86d55715b3bf5957 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-13T13:03:51.071 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:50 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate[71627]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T13:03:51.071 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:50 smithi167 bash[71612]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T13:03:51.120 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:51.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:51.153 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:51.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:51.176 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 89 16 120098 120082 84.2889 55 0.00986998 0.0118591 2024-09-13T13:03:51.255 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:51.279 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:51.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:51.381 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:51.408 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:51.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:51.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:51.510 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:51.536 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:51.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:51.638 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:51.662 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:51.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:51.763 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:51.803 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:51.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:51.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:51.905 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:51.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:51.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:51 smithi052 ceph-mon[36682]: pgmap v115: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 998 op/s; 36833/351999 objects degraded (10.464%) 2024-09-13T13:03:51.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:51 smithi052 ceph-mon[29471]: pgmap v115: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 998 op/s; 36833/351999 objects degraded (10.464%) 2024-09-13T13:03:52.031 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:52.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:52.068 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:52.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:52.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:51 smithi167 ceph-mon[33178]: pgmap v115: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 998 op/s; 36833/351999 objects degraded (10.464%) 2024-09-13T13:03:52.169 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:52.176 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Total time run: 90.0056 2024-09-13T13:03:52.176 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Total writes made: 121098 2024-09-13T13:03:52.176 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Write size: 65536 2024-09-13T13:03:52.176 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Object size: 65536 2024-09-13T13:03:52.176 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Bandwidth (MB/sec): 84.0906 2024-09-13T13:03:52.176 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev Bandwidth: 30.9328 2024-09-13T13:03:52.176 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max bandwidth (MB/sec): 194.438 2024-09-13T13:03:52.176 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min bandwidth (MB/sec): 25.6875 2024-09-13T13:03:52.176 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Average IOPS: 1345 2024-09-13T13:03:52.176 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev IOPS: 494.925 2024-09-13T13:03:52.176 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max IOPS: 3111 2024-09-13T13:03:52.177 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min IOPS: 411 2024-09-13T13:03:52.177 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Average Latency(s): 0.0118878 2024-09-13T13:03:52.177 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev Latency(s): 0.0142518 2024-09-13T13:03:52.177 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max latency(s): 1.03648 2024-09-13T13:03:52.177 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min latency(s): 0.00146492 2024-09-13T13:03:52.179 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T13:03:52.192 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:52.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:52.294 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:52.335 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:52.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:52.437 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:52.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:52.483 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:52.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:52.584 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:52.627 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:52.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:52.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:52.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:52.755 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:52.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:52.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:52.857 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:52.867 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:03:52.867 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:03:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:52.908 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:52.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:53.009 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:53.025 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:53.030 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:53.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:53.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:53.132 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:53.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:53.191 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:53.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:53.292 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:53.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:53.342 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:53.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:53.425 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate[71627]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T13:03:53.425 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 bash[71612]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T13:03:53.425 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate[71627]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-13T13:03:53.425 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 bash[71612]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-13T13:03:53.426 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate[71627]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-13T13:03:53.426 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 bash[71612]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-13T13:03:53.443 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:53.484 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:53.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:53.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:53.587 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:53.612 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:53.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:53.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:53.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:53.717 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:03:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:03:53.717 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate[71627]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T13:03:53.717 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 bash[71612]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T13:03:53.717 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate[71627]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-13T13:03:53.717 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 bash[71612]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-13T13:03:53.717 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate[71627]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T13:03:53.717 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 bash[71612]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T13:03:53.718 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate[71627]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-13T13:03:53.718 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 bash[71612]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-13T13:03:53.718 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:53 smithi167 podman[71612]: 2024-09-13 13:03:53.461203084 +0000 UTC m=+3.288554866 container died c1fa0461298bf4d7268651d750c20ddc94db4fb2285029fc86d55715b3bf5957 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-13T13:03:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:53.742 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:53.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:53.843 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:53.867 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:53.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:53.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:53.968 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:53 smithi052 ceph-mon[36682]: pgmap v116: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 998 op/s; 36833/351999 objects degraded (10.464%) 2024-09-13T13:03:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:53 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 36833/351999 objects degraded (10.464%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:53.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:53 smithi052 ceph-mon[29471]: pgmap v116: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 998 op/s; 36833/351999 objects degraded (10.464%) 2024-09-13T13:03:53.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:53 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 36833/351999 objects degraded (10.464%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:54.021 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:54.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:54.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:53 smithi167 ceph-mon[33178]: pgmap v116: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 998 op/s; 36833/351999 objects degraded (10.464%) 2024-09-13T13:03:54.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:53 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 36833/351999 objects degraded (10.464%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:54.104 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:03:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:03:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:123216 7.7 7:e1fc6a4a:::benchmark_data_smithi052_71865_object2115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:03:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:54.092197+0000", 2024-09-13T13:03:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.010647733, 2024-09-13T13:03:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010683751999999999, 2024-09-13T13:03:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:03:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 123216 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.092197+0000", 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.092197+0000", 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.092199+0000", 2024-09-13T13:03:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4830000000000002e-06 2024-09-13T13:03:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.092204+0000", 2024-09-13T13:03:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:03:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.092217+0000", 2024-09-13T13:03:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2150999999999999e-05 2024-09-13T13:03:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.092225+0000", 2024-09-13T13:03:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6580000000000008e-06 2024-09-13T13:03:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.093410+0000", 2024-09-13T13:03:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011853110000000001 2024-09-13T13:03:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.093530+0000", 2024-09-13T13:03:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011988499999999999 2024-09-13T13:03:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:03:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.093653+0000", 2024-09-13T13:03:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012217200000000001 2024-09-13T13:03:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.101860+0000", 2024-09-13T13:03:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0082074880000000006 2024-09-13T13:03:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:123217 7.4 7:2db4b01b:::benchmark_data_smithi052_71865_object2116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:03:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:54.093922+0000", 2024-09-13T13:03:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0089221830000000002, 2024-09-13T13:03:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0090378579999999993, 2024-09-13T13:03:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:03:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 123217 2024-09-13T13:03:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.093922+0000", 2024-09-13T13:03:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.093922+0000", 2024-09-13T13:03:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.093924+0000", 2024-09-13T13:03:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6899999999999999e-06 2024-09-13T13:03:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.093929+0000", 2024-09-13T13:03:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3909999999999997e-06 2024-09-13T13:03:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.093946+0000", 2024-09-13T13:03:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6702999999999999e-05 2024-09-13T13:03:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.093954+0000", 2024-09-13T13:03:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1699999999999997e-06 2024-09-13T13:03:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.094010+0000", 2024-09-13T13:03:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5476999999999998e-05 2024-09-13T13:03:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.094117+0000", 2024-09-13T13:03:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107581 2024-09-13T13:03:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:03:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.094229+0000", 2024-09-13T13:03:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011215 2024-09-13T13:03:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.097467+0000", 2024-09-13T13:03:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0032378760000000002 2024-09-13T13:03:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:123215 7.5 e115/84 7:a789a0f0:::benchmark_data_smithi052_71865_object2114:head v 115'7825, mlcod=115'7823)", 2024-09-13T13:03:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:54.094976+0000", 2024-09-13T13:03:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0078684940000000002, 2024-09-13T13:03:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0080673940000000003, 2024-09-13T13:03:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:03:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.094976+0000", 2024-09-13T13:03:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.094976+0000", 2024-09-13T13:03:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.094977+0000", 2024-09-13T13:03:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:03:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.094983+0000", 2024-09-13T13:03:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9460000000000003e-06 2024-09-13T13:03:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.094991+0000", 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2670000000000006e-06 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.094997+0000", 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.037e-06 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095402+0000", 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00040488200000000002 2024-09-13T13:03:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095421+0000", 2024-09-13T13:03:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9270999999999999e-05 2024-09-13T13:03:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:123202 7.6 e115/84)", 2024-09-13T13:03:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:54.095014+0000", 2024-09-13T13:03:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0078307510000000004, 2024-09-13T13:03:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0080741679999999996, 2024-09-13T13:03:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:03:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095014+0000", 2024-09-13T13:03:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095014+0000", 2024-09-13T13:03:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095014+0000", 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.68e-07 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095017+0000", 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0419999999999998e-06 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095023+0000", 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0990000000000004e-06 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095028+0000", 2024-09-13T13:03:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0599999999999998e-06 2024-09-13T13:03:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:123218 7.4 7:2a0f4bf1:::benchmark_data_smithi052_71865_object2117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:03:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:54.095517+0000", 2024-09-13T13:03:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0073267820000000004, 2024-09-13T13:03:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0076107750000000002, 2024-09-13T13:03:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:03:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 123218 2024-09-13T13:03:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095517+0000", 2024-09-13T13:03:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095517+0000", 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095519+0000", 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3629999999999999e-06 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095524+0000", 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8389999999999997e-06 2024-09-13T13:03:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095535+0000", 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1263e-05 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.095543+0000", 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8939999999999993e-06 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.097891+0000", 2024-09-13T13:03:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023483639999999999 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.098015+0000", 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012396900000000001 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.098130+0000", 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114728 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.103086+0000", 2024-09-13T13:03:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049558960000000004 2024-09-13T13:03:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.103098+0000", 2024-09-13T13:03:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2023e-05 2024-09-13T13:03:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:123221 7.7 7:e8650950:::benchmark_data_smithi052_71865_object2120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:03:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:54.102143+0000", 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.00070171899999999995, 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001054756, 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 123221 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.102143+0000", 2024-09-13T13:03:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.102143+0000", 2024-09-13T13:03:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.102145+0000", 2024-09-13T13:03:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0040000000000002e-06 2024-09-13T13:03:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.102150+0000", 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0730000000000004e-06 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.102162+0000", 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1973e-05 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.102169+0000", 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5480000000000004e-06 2024-09-13T13:03:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.102643+0000", 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00047362700000000001 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.102753+0000", 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110536 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.102862+0000", 2024-09-13T13:03:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108924 2024-09-13T13:03:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:123201 7.6 7:61b44ac4:::benchmark_data_smithi052_71865_object2100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:03:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:54.078911+0000", 2024-09-13T13:03:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.023933741000000001, 2024-09-13T13:03:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.024344339, 2024-09-13T13:03:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:03:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 123201 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.078911+0000", 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.078911+0000", 2024-09-13T13:03:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.078911+0000", 2024-09-13T13:03:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.44e-07 2024-09-13T13:03:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.078914+0000", 2024-09-13T13:03:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.65e-06 2024-09-13T13:03:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.078919+0000", 2024-09-13T13:03:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2519999999999999e-06 2024-09-13T13:03:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.078924+0000", 2024-09-13T13:03:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6829999999999999e-06 2024-09-13T13:03:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.079460+0000", 2024-09-13T13:03:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00053646499999999997 2024-09-13T13:03:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.079573+0000", 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113019 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.079686+0000", 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011278200000000001 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.090864+0000", 2024-09-13T13:03:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011178242 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.090875+0000", 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0997e-05 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:123202 7.6 7:6a76b35d:::benchmark_data_smithi052_71865_object2101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:54.078950+0000", 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.023893935000000002, 2024-09-13T13:03:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.024373054000000002, 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 123202 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.078950+0000", 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.078950+0000", 2024-09-13T13:03:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.078951+0000", 2024-09-13T13:03:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:03:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.078954+0000", 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6900000000000001e-06 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.078959+0000", 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8200000000000002e-06 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.078964+0000", 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6920000000000001e-06 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.090899+0000", 2024-09-13T13:03:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011934699 2024-09-13T13:03:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.091004+0000", 2024-09-13T13:03:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010499499999999999 2024-09-13T13:03:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.091109+0000", 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010553899999999999 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:123206 7.e e115/84 7:7aeee117:::benchmark_data_smithi052_71865_object2105:head v 115'7479, mlcod=115'7477)", 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:54.080402+0000", 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.022442519000000001, 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.022978011999999999, 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:03:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.080402+0000", 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.080402+0000", 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:03:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.080403+0000", 2024-09-13T13:03:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.339e-06 2024-09-13T13:03:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:03:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.080409+0000", 2024-09-13T13:03:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3609999999999997e-06 2024-09-13T13:03:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:03:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.080419+0000", 2024-09-13T13:03:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8700000000000004e-06 2024-09-13T13:03:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:03:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.080426+0000", 2024-09-13T13:03:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0759999999999997e-06 2024-09-13T13:03:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:03:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.083037+0000", 2024-09-13T13:03:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026106699999999998 2024-09-13T13:03:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:03:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:03:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:03:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:54.083057+0000", 2024-09-13T13:03:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0186999999999998e-05 2024-09-13T13:03:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:03:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:03:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:03:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 9 2024-09-13T13:03:54.147 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:03:54.147 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:54.174 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:54.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:54.276 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:54.302 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:54.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:54.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:54.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:54.405 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:54 smithi167 podman[71963]: 2024-09-13 13:03:54.120523837 +0000 UTC m=+0.651442951 container remove c1fa0461298bf4d7268651d750c20ddc94db4fb2285029fc86d55715b3bf5957 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T13:03:54.406 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:54 smithi167 podman[72091]: 2024-09-13 13:03:54.266241076 +0000 UTC m=+0.024744921 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:03:54.406 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:54 smithi167 podman[72091]: 2024-09-13 13:03:54.404955602 +0000 UTC m=+0.163459432 container create c52ff0389c24748910bec7e997ee90cbe349a976842f5a59d8714e44a478dd6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-13T13:03:54.428 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:54.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:54.530 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:54.572 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:54.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:54.674 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:54.710 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:54.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:54.720 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:03:54.799 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T13:03:54.812 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:54.856 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:54.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:54.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:54.868 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:03:54.958 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:54.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:54 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:54.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:54 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3678783073' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:54.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:54 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:54.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:54 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3678783073' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:55.014 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:55.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:55.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:55.019 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:54 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:03:55.019 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:54 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3678783073' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:03:55.019 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:54 smithi167 podman[72091]: 2024-09-13 13:03:54.684156476 +0000 UTC m=+0.442660313 container init c52ff0389c24748910bec7e997ee90cbe349a976842f5a59d8714e44a478dd6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-13T13:03:55.019 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:54 smithi167 podman[72091]: 2024-09-13 13:03:54.688797104 +0000 UTC m=+0.447300935 container start c52ff0389c24748910bec7e997ee90cbe349a976842f5a59d8714e44a478dd6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T13:03:55.019 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:54 smithi167 bash[72091]: c52ff0389c24748910bec7e997ee90cbe349a976842f5a59d8714e44a478dd6f 2024-09-13T13:03:55.019 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:54 smithi167 systemd[1]: Started Ceph osd.6 for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:03:55.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:55.129 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:03:55.168 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T13:03:55.173 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:55.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:55.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:55.275 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:55.300 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:55.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:55.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:55.430 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:55.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:55.532 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:55.618 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:55.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:55.645 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:55.670 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:55.674 INFO:teuthology.orchestra.run.smithi167.stderr:no valid command found; 10 closest matches: 2024-09-13T13:03:55.674 INFO:teuthology.orchestra.run.smithi167.stderr:0 2024-09-13T13:03:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:1 2024-09-13T13:03:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:2 2024-09-13T13:03:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:abort 2024-09-13T13:03:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:assert 2024-09-13T13:03:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:config diff 2024-09-13T13:03:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:config diff get 2024-09-13T13:03:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:config get 2024-09-13T13:03:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:config help [] 2024-09-13T13:03:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:config set ... 2024-09-13T13:03:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:admin_socket: invalid command 2024-09-13T13:03:55.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:55.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:55 smithi052 ceph-mon[29471]: pgmap v117: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 1.01k op/s; 37738/361059 objects degraded (10.452%) 2024-09-13T13:03:55.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:55 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:55.765 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:55.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:55.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:55.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:55 smithi167 ceph-mon[33178]: pgmap v117: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 1.01k op/s; 37738/361059 objects degraded (10.452%) 2024-09-13T13:03:55.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:55 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:55 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3678783073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:55 smithi167 ceph-mon[33178]: osdmap e116: 8 total, 7 up, 8 in 2024-09-13T13:03:55.867 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:55.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:55.909 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:55.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:55.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:55 smithi052 ceph-mon[36682]: pgmap v117: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 1.01k op/s; 37738/361059 objects degraded (10.452%) 2024-09-13T13:03:55.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:55 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:03:55.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:55 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3678783073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:55.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:55 smithi052 ceph-mon[36682]: osdmap e116: 8 total, 7 up, 8 in 2024-09-13T13:03:55.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:55 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3678783073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:03:55.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:55 smithi052 ceph-mon[29471]: osdmap e116: 8 total, 7 up, 8 in 2024-09-13T13:03:56.010 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:56.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:56.066 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:56.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:56.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:56.191 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:56.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:56.273 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:03:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:03:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:03:56.293 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:56.320 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:03:56 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:03:56.060+0000 7f9ce44b7640 -1 Falling back to public interface 2024-09-13T13:03:56.326 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:56.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:56.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:56.353 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:03:56.354 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:03:56.390 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:03:56.391 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:03:56.426 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:03:56.426 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:03:56.428 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:56.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:56.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:56.462 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:03:56.462 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:03:56.494 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:03:56.494 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:03:56.526 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:03:56.526 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:03:56.527 INFO:teuthology.orchestra.run.smithi052.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T13:03:56.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:56.558 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:03:56.558 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:03:56.577 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:56.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T13:03:56.611 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-09-13T13:03:56.679 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:56.721 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:56.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:56.823 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:56.864 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:56.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:56.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:56.887 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:56 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1881113698' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:56.888 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:56 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1881113698' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:56.965 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:03:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:57.003 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:57.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:03:57.055 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:03:57.055 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:03:57.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:56 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1881113698' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:03:57.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:57.129 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:57.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:57.231 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:57.257 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:57.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:57.358 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:57.382 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:57.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:57.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:57.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:57.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:57.505 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:57.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:57.553 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:03:57.606 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:57.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:57.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:57.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:03:57.731 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:57.754 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:57.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:57.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:57.881 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:57.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:57.983 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:57.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:57 smithi052 ceph-mon[36682]: pgmap v119: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 1.01k op/s; 37738/361059 objects degraded (10.452%) 2024-09-13T13:03:57.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:57 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 37738/361059 objects degraded (10.452%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:57.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:57 smithi052 ceph-mon[29471]: pgmap v119: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 1.01k op/s; 37738/361059 objects degraded (10.452%) 2024-09-13T13:03:57.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:57 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 37738/361059 objects degraded (10.452%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:58.007 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:58.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:58.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:58.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:57 smithi167 ceph-mon[33178]: pgmap v119: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 1.01k op/s; 37738/361059 objects degraded (10.452%) 2024-09-13T13:03:58.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:57 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 37738/361059 objects degraded (10.452%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:03:58.110 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:58.145 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:58.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:58.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:58.247 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:58.293 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:58.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:58.395 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:58.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:58.437 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:58.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:58.538 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:03:58.552 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:03:58.577 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:58.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:58.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:58.679 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:03:58.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:58.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:03:58.805 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:58.829 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:58.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:58.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:58.931 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:58.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:58.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:59.057 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:03:59.097 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:59.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:03:59.182 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T13:03:59.199 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:59.286 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:59.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:59.316 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:03:59.388 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:59.442 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:59.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:59.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:59.544 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:03:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:59.602 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:59.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:03:59.704 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:03:59.731 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:03:59.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:59.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:03:59.833 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:03:59.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:03:59.875 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:03:59.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:03:59.976 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:03:59.993 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:00.000 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:00.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:00.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:03:59 smithi167 ceph-mon[33178]: pgmap v120: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 1.01k op/s; 36081/345459 objects degraded (10.444%) 2024-09-13T13:04:00.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:00.130 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:00.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:00.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:00.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:03:59 smithi052 ceph-mon[36682]: pgmap v120: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 1.01k op/s; 36081/345459 objects degraded (10.444%) 2024-09-13T13:04:00.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:03:59 smithi052 ceph-mon[29471]: pgmap v120: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 1.01k op/s; 36081/345459 objects degraded (10.444%) 2024-09-13T13:04:00.232 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:00.256 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:00.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:00.358 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:00.404 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:00.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:00.506 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:00.545 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:04:00 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:04:00.183+0000 7f9ce44b7640 -1 osd.6 102 log_to_monitors true 2024-09-13T13:04:00.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:00.549 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:00.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:00.553 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:04:00.651 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:00.663 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:04:00.663 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:04:00.710 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:00.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:00.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:00.781 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:04:00.813 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:00.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:00 smithi167 ceph-mon[33178]: from='osd.6 v1:172.21.15.167:6808/143093129' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T13:04:00.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:00 smithi167 ceph-mon[33178]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T13:04:00.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:00 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2726925434' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:04:00.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:00 smithi167 ceph-mon[33178]: from='client.34934 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:04:00.840 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:00.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:00.942 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:00.973 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:00.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:00.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:01.075 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:01.097 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:01.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:01.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:01.193 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:00 smithi052 ceph-mon[36682]: from='osd.6 v1:172.21.15.167:6808/143093129' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T13:04:01.193 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:00 smithi052 ceph-mon[36682]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T13:04:01.193 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:00 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2726925434' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:04:01.193 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:00 smithi052 ceph-mon[36682]: from='client.34934 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:04:01.194 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:00 smithi052 ceph-mon[29471]: from='osd.6 v1:172.21.15.167:6808/143093129' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T13:04:01.194 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:00 smithi052 ceph-mon[29471]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T13:04:01.194 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:00 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2726925434' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:04:01.194 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:00 smithi052 ceph-mon[29471]: from='client.34934 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:04:01.199 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:01.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:01.231 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:01.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:01.332 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:01.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:01.359 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:01.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:01.460 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:01.484 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:01.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:01.570 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:04:01 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:04:01.278+0000 7f9cd8259640 -1 osd.6 102 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:04:01.586 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:01.612 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T13:04:01.637 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:01.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:01.646 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T13:04:01.739 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:01.766 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:01.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:01.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:01.895 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:01.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:01.904 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:04:01.924 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[36682]: pgmap v121: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 1.01k op/s; 36081/345459 objects degraded (10.444%) 2024-09-13T13:04:01.924 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:04:01.924 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[36682]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T13:04:01.924 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[36682]: from='client.34934 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:04:01.924 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[36682]: from='osd.6 v1:172.21.15.167:6808/143093129' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:04:01.924 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[36682]: osdmap e117: 8 total, 7 up, 8 in 2024-09-13T13:04:01.924 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[36682]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:04:01.925 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[36682]: from='osd.6 ' entity='osd.6' 2024-09-13T13:04:01.925 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[36682]: 6.6 scrub starts 2024-09-13T13:04:01.925 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[36682]: 6.6 scrub ok 2024-09-13T13:04:01.925 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[29471]: pgmap v121: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 1.01k op/s; 36081/345459 objects degraded (10.444%) 2024-09-13T13:04:01.925 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:04:01.925 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[29471]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T13:04:01.925 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[29471]: from='client.34934 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:04:01.925 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[29471]: from='osd.6 v1:172.21.15.167:6808/143093129' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:04:01.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[29471]: osdmap e117: 8 total, 7 up, 8 in 2024-09-13T13:04:01.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[29471]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:04:01.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[29471]: from='osd.6 ' entity='osd.6' 2024-09-13T13:04:01.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[29471]: 6.6 scrub starts 2024-09-13T13:04:01.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:01 smithi052 ceph-mon[29471]: 6.6 scrub ok 2024-09-13T13:04:01.995 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:02.020 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:02.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:02.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:02.026 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:01 smithi167 ceph-mon[33178]: pgmap v121: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 1.01k op/s; 36081/345459 objects degraded (10.444%) 2024-09-13T13:04:02.026 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:01 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:04:02.026 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:01 smithi167 ceph-mon[33178]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T13:04:02.026 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:01 smithi167 ceph-mon[33178]: from='client.34934 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:04:02.026 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:01 smithi167 ceph-mon[33178]: from='osd.6 v1:172.21.15.167:6808/143093129' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:04:02.026 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:01 smithi167 ceph-mon[33178]: osdmap e117: 8 total, 7 up, 8 in 2024-09-13T13:04:02.026 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:01 smithi167 ceph-mon[33178]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:04:02.026 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:01 smithi167 ceph-mon[33178]: from='osd.6 ' entity='osd.6' 2024-09-13T13:04:02.026 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:01 smithi167 ceph-mon[33178]: 6.6 scrub starts 2024-09-13T13:04:02.027 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:01 smithi167 ceph-mon[33178]: 6.6 scrub ok 2024-09-13T13:04:02.122 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:02.166 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:02.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:02.203 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:02.269 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:02.330 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:02.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:02.432 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:02.455 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:02.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:02.521 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:02.558 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:02.582 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:02.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:02.629 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:04:02.629 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:04:02.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.629 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14493.0:1323 5.1f 5.4322fa9f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.629 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.294107+0000", 2024-09-13T13:04:02.629 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.185773933, 2024-09-13T13:04:02.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.185793878, 2024-09-13T13:04:02.629 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.629 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.629 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.630 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.630 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14493", 2024-09-13T13:04:02.630 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:04:02.630 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1323 2024-09-13T13:04:02.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.630 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294107+0000", 2024-09-13T13:04:02.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294107+0000", 2024-09-13T13:04:02.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294109+0000", 2024-09-13T13:04:02.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-09-13T13:04:02.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294115+0000", 2024-09-13T13:04:02.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4030000000000004e-06 2024-09-13T13:04:02.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294130+0000", 2024-09-13T13:04:02.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4558e-05 2024-09-13T13:04:02.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294140+0000", 2024-09-13T13:04:02.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5599999999999999e-06 2024-09-13T13:04:02.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294254+0000", 2024-09-13T13:04:02.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114688 2024-09-13T13:04:02.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-09-13T13:04:02.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294261+0000", 2024-09-13T13:04:02.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7070000000000003e-06 2024-09-13T13:04:02.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.298468+0000", 2024-09-13T13:04:02.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042066710000000004 2024-09-13T13:04:02.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.298478+0000", 2024-09-13T13:04:02.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0334e-05 2024-09-13T13:04:02.635 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.635 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.635 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.636 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14493.0:1324 4.5 4.9649f065 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.636 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.294163+0000", 2024-09-13T13:04:02.636 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.185717769, 2024-09-13T13:04:02.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.18578246900000001, 2024-09-13T13:04:02.636 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.636 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.636 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.636 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.636 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14493", 2024-09-13T13:04:02.636 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:04:02.637 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1324 2024-09-13T13:04:02.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.637 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294163+0000", 2024-09-13T13:04:02.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294163+0000", 2024-09-13T13:04:02.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294164+0000", 2024-09-13T13:04:02.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4900000000000005e-07 2024-09-13T13:04:02.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294167+0000", 2024-09-13T13:04:02.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.23e-06 2024-09-13T13:04:02.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294174+0000", 2024-09-13T13:04:02.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9979999999999997e-06 2024-09-13T13:04:02.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294181+0000", 2024-09-13T13:04:02.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8689999999999998e-06 2024-09-13T13:04:02.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294251+0000", 2024-09-13T13:04:02.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0588999999999998e-05 2024-09-13T13:04:02.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-09-13T13:04:02.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.294258+0000", 2024-09-13T13:04:02.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0530000000000001e-06 2024-09-13T13:04:02.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.314225+0000", 2024-09-13T13:04:02.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01996699 2024-09-13T13:04:02.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.314235+0000", 2024-09-13T13:04:02.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5540000000000003e-06 2024-09-13T13:04:02.642 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.642 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.642 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.642 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:131234 7.7 7.73fda227 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.642 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.308273+0000", 2024-09-13T13:04:02.642 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.171607434, 2024-09-13T13:04:02.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.171709524, 2024-09-13T13:04:02.643 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.643 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.643 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:04:02.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:04:02.643 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 131234 2024-09-13T13:04:02.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.643 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308273+0000", 2024-09-13T13:04:02.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308273+0000", 2024-09-13T13:04:02.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308275+0000", 2024-09-13T13:04:02.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0080000000000001e-06 2024-09-13T13:04:02.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308280+0000", 2024-09-13T13:04:02.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2109999999999997e-06 2024-09-13T13:04:02.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308290+0000", 2024-09-13T13:04:02.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.006e-05 2024-09-13T13:04:02.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308299+0000", 2024-09-13T13:04:02.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6980000000000004e-06 2024-09-13T13:04:02.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308614+0000", 2024-09-13T13:04:02.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031478499999999999 2024-09-13T13:04:02.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308618+0000", 2024-09-13T13:04:02.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5390000000000001e-06 2024-09-13T13:04:02.647 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.647 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.647 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.648 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:131250 7.d 7.6891eaad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.648 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.308405+0000", 2024-09-13T13:04:02.648 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.17147517900000001, 2024-09-13T13:04:02.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17160824399999999, 2024-09-13T13:04:02.648 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.648 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.648 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.648 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.648 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:04:02.648 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:04:02.648 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 131250 2024-09-13T13:04:02.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.649 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308405+0000", 2024-09-13T13:04:02.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308405+0000", 2024-09-13T13:04:02.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308406+0000", 2024-09-13T13:04:02.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.057e-06 2024-09-13T13:04:02.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308411+0000", 2024-09-13T13:04:02.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.69e-06 2024-09-13T13:04:02.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308420+0000", 2024-09-13T13:04:02.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7129999999999994e-06 2024-09-13T13:04:02.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308428+0000", 2024-09-13T13:04:02.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8080000000000005e-06 2024-09-13T13:04:02.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.314521+0000", 2024-09-13T13:04:02.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060935720000000002 2024-09-13T13:04:02.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.314530+0000", 2024-09-13T13:04:02.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0599999999999997e-06 2024-09-13T13:04:02.652 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.652 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.652 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.652 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:131251 7.7 7.97c927 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.652 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.308438+0000", 2024-09-13T13:04:02.652 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.171442227, 2024-09-13T13:04:02.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.171604442, 2024-09-13T13:04:02.652 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.653 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.653 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.653 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.653 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:04:02.653 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:04:02.653 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 131251 2024-09-13T13:04:02.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.653 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308438+0000", 2024-09-13T13:04:02.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308438+0000", 2024-09-13T13:04:02.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308439+0000", 2024-09-13T13:04:02.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8700000000000005e-07 2024-09-13T13:04:02.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308442+0000", 2024-09-13T13:04:02.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9620000000000002e-06 2024-09-13T13:04:02.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308448+0000", 2024-09-13T13:04:02.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5990000000000001e-06 2024-09-13T13:04:02.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308454+0000", 2024-09-13T13:04:02.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2779999999999999e-06 2024-09-13T13:04:02.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308637+0000", 2024-09-13T13:04:02.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018318699999999999 2024-09-13T13:04:02.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.308643+0000", 2024-09-13T13:04:02.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:04:02.657 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.657 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.657 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.657 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:131258 7.c 7.ff3a9a1c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.657 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.314319+0000", 2024-09-13T13:04:02.657 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.16556147800000001, 2024-09-13T13:04:02.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.165752503, 2024-09-13T13:04:02.657 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.657 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.658 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.658 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.658 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:04:02.658 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:04:02.658 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 131258 2024-09-13T13:04:02.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.658 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.314319+0000", 2024-09-13T13:04:02.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.314319+0000", 2024-09-13T13:04:02.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.314321+0000", 2024-09-13T13:04:02.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.894e-06 2024-09-13T13:04:02.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.314326+0000", 2024-09-13T13:04:02.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2440000000000001e-06 2024-09-13T13:04:02.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.314335+0000", 2024-09-13T13:04:02.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.371e-06 2024-09-13T13:04:02.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.314343+0000", 2024-09-13T13:04:02.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2949999999999994e-06 2024-09-13T13:04:02.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.314400+0000", 2024-09-13T13:04:02.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6663999999999998e-05 2024-09-13T13:04:02.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.314409+0000", 2024-09-13T13:04:02.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3589999999999997e-06 2024-09-13T13:04:02.662 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.662 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.662 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.662 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:131276 7.7 7.4bc235c7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.662 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.335329+0000", 2024-09-13T13:04:02.663 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.14455193199999999, 2024-09-13T13:04:02.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14478682200000001, 2024-09-13T13:04:02.663 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.663 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.663 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.663 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.663 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:04:02.663 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:04:02.663 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 131276 2024-09-13T13:04:02.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.663 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335329+0000", 2024-09-13T13:04:02.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335329+0000", 2024-09-13T13:04:02.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335331+0000", 2024-09-13T13:04:02.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.621e-06 2024-09-13T13:04:02.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335345+0000", 2024-09-13T13:04:02.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3312999999999999e-05 2024-09-13T13:04:02.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335363+0000", 2024-09-13T13:04:02.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8445e-05 2024-09-13T13:04:02.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335373+0000", 2024-09-13T13:04:02.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6579999999999997e-06 2024-09-13T13:04:02.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335444+0000", 2024-09-13T13:04:02.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1316000000000006e-05 2024-09-13T13:04:02.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335454+0000", 2024-09-13T13:04:02.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0210999999999999e-05 2024-09-13T13:04:02.667 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.667 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.667 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.667 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:131277 7.d 7.8dd826fd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.667 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.335940+0000", 2024-09-13T13:04:02.667 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.143940702, 2024-09-13T13:04:02.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14420365499999999, 2024-09-13T13:04:02.668 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.668 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.669 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:04:02.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:04:02.669 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 131277 2024-09-13T13:04:02.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.669 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335940+0000", 2024-09-13T13:04:02.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335940+0000", 2024-09-13T13:04:02.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335941+0000", 2024-09-13T13:04:02.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3769999999999999e-06 2024-09-13T13:04:02.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335946+0000", 2024-09-13T13:04:02.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8729999999999998e-06 2024-09-13T13:04:02.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335956+0000", 2024-09-13T13:04:02.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4970000000000007e-06 2024-09-13T13:04:02.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335962+0000", 2024-09-13T13:04:02.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4130000000000003e-06 2024-09-13T13:04:02.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.336020+0000", 2024-09-13T13:04:02.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8075999999999998e-05 2024-09-13T13:04:02.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.336029+0000", 2024-09-13T13:04:02.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7129999999999994e-06 2024-09-13T13:04:02.673 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.673 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.673 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.673 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:131278 7.7 7.3ec43e87 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.674 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.335991+0000", 2024-09-13T13:04:02.674 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.14389004499999999, 2024-09-13T13:04:02.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.144179957, 2024-09-13T13:04:02.674 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.674 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.674 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.674 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.674 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:04:02.674 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:04:02.674 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 131278 2024-09-13T13:04:02.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.675 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335991+0000", 2024-09-13T13:04:02.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335991+0000", 2024-09-13T13:04:02.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335991+0000", 2024-09-13T13:04:02.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2699999999999998e-07 2024-09-13T13:04:02.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.335994+0000", 2024-09-13T13:04:02.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8949999999999998e-06 2024-09-13T13:04:02.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.336001+0000", 2024-09-13T13:04:02.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7460000000000003e-06 2024-09-13T13:04:02.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.336007+0000", 2024-09-13T13:04:02.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0909999999999998e-06 2024-09-13T13:04:02.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.336042+0000", 2024-09-13T13:04:02.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.502e-05 2024-09-13T13:04:02.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.336047+0000", 2024-09-13T13:04:02.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0780000000000003e-06 2024-09-13T13:04:02.679 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.679 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.679 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.679 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:131283 7.c 7.910a6c8c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.679 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.339623+0000", 2024-09-13T13:04:02.680 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.140257513, 2024-09-13T13:04:02.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14057460199999999, 2024-09-13T13:04:02.680 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.680 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.680 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.680 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.680 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:04:02.680 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:04:02.681 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 131283 2024-09-13T13:04:02.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.681 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.339623+0000", 2024-09-13T13:04:02.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.339623+0000", 2024-09-13T13:04:02.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.339625+0000", 2024-09-13T13:04:02.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.674e-06 2024-09-13T13:04:02.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.339631+0000", 2024-09-13T13:04:02.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0530000000000003e-06 2024-09-13T13:04:02.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.339641+0000", 2024-09-13T13:04:02.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0597e-05 2024-09-13T13:04:02.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.339647+0000", 2024-09-13T13:04:02.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.614e-06 2024-09-13T13:04:02.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.339706+0000", 2024-09-13T13:04:02.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9147000000000001e-05 2024-09-13T13:04:02.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.339717+0000", 2024-09-13T13:04:02.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0651e-05 2024-09-13T13:04:02.684 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.684 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.684 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.684 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:131285 7.c 7.a0c6663c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.684 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.340604+0000", 2024-09-13T13:04:02.685 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.13927666999999999, 2024-09-13T13:04:02.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.139620574, 2024-09-13T13:04:02.685 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.685 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.685 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.685 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.685 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:04:02.685 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:04:02.685 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 131285 2024-09-13T13:04:02.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.685 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.340604+0000", 2024-09-13T13:04:02.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.340604+0000", 2024-09-13T13:04:02.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.340605+0000", 2024-09-13T13:04:02.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1179999999999999e-06 2024-09-13T13:04:02.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.340612+0000", 2024-09-13T13:04:02.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8469999999999998e-06 2024-09-13T13:04:02.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.340621+0000", 2024-09-13T13:04:02.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3440000000000007e-06 2024-09-13T13:04:02.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.340626+0000", 2024-09-13T13:04:02.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2399999999999998e-06 2024-09-13T13:04:02.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.340686+0000", 2024-09-13T13:04:02.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9475000000000002e-05 2024-09-13T13:04:02.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.340695+0000", 2024-09-13T13:04:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2259999999999994e-06 2024-09-13T13:04:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:131289 7.d 7.a9850e8d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.349103+0000", 2024-09-13T13:04:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.13077739999999999, 2024-09-13T13:04:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13114815299999999, 2024-09-13T13:04:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 131289 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349103+0000", 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349103+0000", 2024-09-13T13:04:02.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349106+0000", 2024-09-13T13:04:02.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7350000000000001e-06 2024-09-13T13:04:02.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349114+0000", 2024-09-13T13:04:02.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:04:02.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349131+0000", 2024-09-13T13:04:02.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6864e-05 2024-09-13T13:04:02.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349139+0000", 2024-09-13T13:04:02.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3680000000000002e-06 2024-09-13T13:04:02.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349196+0000", 2024-09-13T13:04:02.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7116e-05 2024-09-13T13:04:02.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349205+0000", 2024-09-13T13:04:02.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5730000000000008e-06 2024-09-13T13:04:02.694 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.694 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.694 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.694 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:131291 7.d 7.b1e9f66d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.694 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.349268+0000", 2024-09-13T13:04:02.694 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.130612487, 2024-09-13T13:04:02.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13101460700000001, 2024-09-13T13:04:02.694 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.694 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.694 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.695 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.695 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:04:02.695 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:04:02.695 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 131291 2024-09-13T13:04:02.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.695 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349268+0000", 2024-09-13T13:04:02.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349268+0000", 2024-09-13T13:04:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349269+0000", 2024-09-13T13:04:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.54e-07 2024-09-13T13:04:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349284+0000", 2024-09-13T13:04:02.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4718999999999999e-05 2024-09-13T13:04:02.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349290+0000", 2024-09-13T13:04:02.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3790000000000003e-06 2024-09-13T13:04:02.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349295+0000", 2024-09-13T13:04:02.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4909999999999999e-06 2024-09-13T13:04:02.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349325+0000", 2024-09-13T13:04:02.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0629000000000001e-05 2024-09-13T13:04:02.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.349329+0000", 2024-09-13T13:04:02.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1649999999999999e-06 2024-09-13T13:04:02.699 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.699 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.699 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.699 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:131293 7.7 7.902c5717 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.699 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.352505+0000", 2024-09-13T13:04:02.699 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.12737532900000001, 2024-09-13T13:04:02.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.127804731, 2024-09-13T13:04:02.699 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.699 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.700 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.700 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.700 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:04:02.700 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:04:02.700 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 131293 2024-09-13T13:04:02.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.700 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.352505+0000", 2024-09-13T13:04:02.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.352505+0000", 2024-09-13T13:04:02.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.352506+0000", 2024-09-13T13:04:02.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1400000000000001e-06 2024-09-13T13:04:02.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.352511+0000", 2024-09-13T13:04:02.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.092e-06 2024-09-13T13:04:02.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.352520+0000", 2024-09-13T13:04:02.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9650000000000003e-06 2024-09-13T13:04:02.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.352532+0000", 2024-09-13T13:04:02.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1742e-05 2024-09-13T13:04:02.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.352581+0000", 2024-09-13T13:04:02.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8977999999999998e-05 2024-09-13T13:04:02.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.352589+0000", 2024-09-13T13:04:02.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4109999999999999e-06 2024-09-13T13:04:02.704 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.704 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.704 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.704 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:131295 7.c 7.a11c334c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-13T13:04:02.704 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:02.358525+0000", 2024-09-13T13:04:02.704 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.121355582, 2024-09-13T13:04:02.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.121826563, 2024-09-13T13:04:02.705 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:04:02.705 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:04:02.705 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-09-13T13:04:02.705 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:04:02.705 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:04:02.705 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:04:02.705 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 131295 2024-09-13T13:04:02.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.706 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:04:02.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:04:02.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.358525+0000", 2024-09-13T13:04:02.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:04:02.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.358525+0000", 2024-09-13T13:04:02.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:04:02.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:04:02.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.358527+0000", 2024-09-13T13:04:02.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.471e-06 2024-09-13T13:04:02.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:04:02.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.358535+0000", 2024-09-13T13:04:02.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3590000000000008e-06 2024-09-13T13:04:02.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:04:02.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.358547+0000", 2024-09-13T13:04:02.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2351000000000001e-05 2024-09-13T13:04:02.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:04:02.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.358553+0000", 2024-09-13T13:04:02.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9109999999999998e-06 2024-09-13T13:04:02.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:04:02.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.358602+0000", 2024-09-13T13:04:02.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8727000000000003e-05 2024-09-13T13:04:02.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:04:02.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:04:02.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:04:02.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:02.358608+0000", 2024-09-13T13:04:02.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9689999999999999e-06 2024-09-13T13:04:02.709 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.709 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:04:02.710 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.710 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:04:02.710 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:04:02.710 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 15 2024-09-13T13:04:02.710 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:04:02.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:02.732 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:02.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:02.801 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[36682]: 7.f scrub starts 2024-09-13T13:04:02.802 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[36682]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T13:04:02.802 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[36682]: osd.6 v1:172.21.15.167:6808/143093129 boot 2024-09-13T13:04:02.802 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[36682]: osdmap e118: 8 total, 8 up, 8 in 2024-09-13T13:04:02.802 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:04:02.802 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[36682]: 4.19 scrub starts 2024-09-13T13:04:02.802 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[36682]: 4.19 scrub ok 2024-09-13T13:04:02.802 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:02.802 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:04:02.802 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 36081/345459 objects degraded (10.444%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:02.802 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[36682]: osdmap e119: 8 total, 8 up, 8 in 2024-09-13T13:04:02.802 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[29471]: 7.f scrub starts 2024-09-13T13:04:02.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[29471]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T13:04:02.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[29471]: osd.6 v1:172.21.15.167:6808/143093129 boot 2024-09-13T13:04:02.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[29471]: osdmap e118: 8 total, 8 up, 8 in 2024-09-13T13:04:02.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:04:02.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[29471]: 4.19 scrub starts 2024-09-13T13:04:02.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[29471]: 4.19 scrub ok 2024-09-13T13:04:02.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:02.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:04:02.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 36081/345459 objects degraded (10.444%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:02.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:02 smithi052 ceph-mon[29471]: osdmap e119: 8 total, 8 up, 8 in 2024-09-13T13:04:02.834 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:02.854 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:02.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:02.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:02.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:02.977 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:02.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:03.042 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:04:03.043 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:04:03.059 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:04:03.059 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:04:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:02 smithi167 ceph-mon[33178]: 7.f scrub starts 2024-09-13T13:04:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:02 smithi167 ceph-mon[33178]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T13:04:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:02 smithi167 ceph-mon[33178]: osd.6 v1:172.21.15.167:6808/143093129 boot 2024-09-13T13:04:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:02 smithi167 ceph-mon[33178]: osdmap e118: 8 total, 8 up, 8 in 2024-09-13T13:04:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:02 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T13:04:03.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:02 smithi167 ceph-mon[33178]: 4.19 scrub starts 2024-09-13T13:04:03.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:02 smithi167 ceph-mon[33178]: 4.19 scrub ok 2024-09-13T13:04:03.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:02 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:03.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:02 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:04:03.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:02 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 36081/345459 objects degraded (10.444%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:03.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:02 smithi167 ceph-mon[33178]: osdmap e119: 8 total, 8 up, 8 in 2024-09-13T13:04:03.075 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:04:03.076 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:04:03.079 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:04:03.092 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:04:03.100 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:03.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:03.109 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:04:03.109 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:04:03.126 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:04:03.126 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:04:03.143 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:04:03.143 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:04:03.160 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:04:03.160 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:04:03.203 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:03.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:03.226 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:03.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:03.230 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:04:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:04:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 0 2024-09-13T13:04:03.230 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:04:03.327 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:03.349 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:03.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:03.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:03.451 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:03.472 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:03.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:03.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:03.574 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:03.596 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:03.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:03.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:03.698 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:03.725 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:03.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:03.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:03.827 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:03.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:03.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:03.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:03.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:03.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:03 smithi167 ceph-mon[33178]: pgmap v124: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 643 B/s rd, 0 B/s wr, 653 op/s; 36081/345459 objects degraded (10.444%) 2024-09-13T13:04:03.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:03 smithi167 ceph-mon[33178]: 4.e scrub starts 2024-09-13T13:04:03.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:03 smithi167 ceph-mon[33178]: 4.e scrub ok 2024-09-13T13:04:03.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:03 smithi167 ceph-mon[33178]: 7.f scrub ok 2024-09-13T13:04:03.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:03 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/707673638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:03.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:03 smithi167 ceph-mon[33178]: 6.1f scrub starts 2024-09-13T13:04:03.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:03 smithi167 ceph-mon[33178]: 6.1f scrub ok 2024-09-13T13:04:03.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:03 smithi167 ceph-mon[33178]: osdmap e120: 8 total, 8 up, 8 in 2024-09-13T13:04:03.963 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:04:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:04:04.001 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:04.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:04.098 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[36682]: pgmap v124: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 643 B/s rd, 0 B/s wr, 653 op/s; 36081/345459 objects degraded (10.444%) 2024-09-13T13:04:04.099 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[36682]: 4.e scrub starts 2024-09-13T13:04:04.099 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[36682]: 4.e scrub ok 2024-09-13T13:04:04.099 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[36682]: 7.f scrub ok 2024-09-13T13:04:04.099 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/707673638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:04.099 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[36682]: 6.1f scrub starts 2024-09-13T13:04:04.099 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[36682]: 6.1f scrub ok 2024-09-13T13:04:04.100 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[36682]: osdmap e120: 8 total, 8 up, 8 in 2024-09-13T13:04:04.100 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[29471]: pgmap v124: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 643 B/s rd, 0 B/s wr, 653 op/s; 36081/345459 objects degraded (10.444%) 2024-09-13T13:04:04.100 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[29471]: 4.e scrub starts 2024-09-13T13:04:04.100 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[29471]: 4.e scrub ok 2024-09-13T13:04:04.100 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[29471]: 7.f scrub ok 2024-09-13T13:04:04.100 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/707673638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:04.100 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[29471]: 6.1f scrub starts 2024-09-13T13:04:04.100 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[29471]: 6.1f scrub ok 2024-09-13T13:04:04.101 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:03 smithi052 ceph-mon[29471]: osdmap e120: 8 total, 8 up, 8 in 2024-09-13T13:04:04.102 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:04.140 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:04.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:04.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:04.224 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:04:04.242 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:04.316 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:04.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:04.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:04.418 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:04.459 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T13:04:04.474 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:04.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:04.576 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:04.578 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T13:04:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:04.649 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:04.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:04.750 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:04.795 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:04.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:04.896 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:04.941 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:04.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:04.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:04 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:04.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:04 smithi052 ceph-mon[36682]: 6.15 scrub starts 2024-09-13T13:04:04.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:04 smithi052 ceph-mon[36682]: 6.15 scrub ok 2024-09-13T13:04:04.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:04 smithi052 ceph-mon[36682]: 3.3 scrub starts 2024-09-13T13:04:04.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:04 smithi052 ceph-mon[36682]: 3.3 scrub ok 2024-09-13T13:04:04.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:04 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:04.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:04 smithi052 ceph-mon[29471]: 6.15 scrub starts 2024-09-13T13:04:04.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:04 smithi052 ceph-mon[29471]: 6.15 scrub ok 2024-09-13T13:04:04.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:04 smithi052 ceph-mon[29471]: 3.3 scrub starts 2024-09-13T13:04:04.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:04 smithi052 ceph-mon[29471]: 3.3 scrub ok 2024-09-13T13:04:05.007 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:05.042 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:05.057 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:05.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:04 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:05.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:04 smithi167 ceph-mon[33178]: 6.15 scrub starts 2024-09-13T13:04:05.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:04 smithi167 ceph-mon[33178]: 6.15 scrub ok 2024-09-13T13:04:05.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:04 smithi167 ceph-mon[33178]: 3.3 scrub starts 2024-09-13T13:04:05.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:04 smithi167 ceph-mon[33178]: 3.3 scrub ok 2024-09-13T13:04:05.072 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:05.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:05.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:05.174 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:05.222 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:05.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:05.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:05.323 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:05.384 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:05.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:05.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:05.506 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:05.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:05.541 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:05.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:05.642 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:05.691 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:05.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:05.793 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:05.841 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:05.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:05.944 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:05.974 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:05.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:05.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:05.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:05 smithi052 ceph-mon[29471]: pgmap v127: 148 pgs: 5 remapped+peering, 4 active+undersized, 4 active+undersized+degraded, 135 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 23/333348 objects degraded (0.007%) 2024-09-13T13:04:05.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:05 smithi052 ceph-mon[36682]: pgmap v127: 148 pgs: 5 remapped+peering, 4 active+undersized, 4 active+undersized+degraded, 135 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 23/333348 objects degraded (0.007%) 2024-09-13T13:04:06.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:05 smithi167 ceph-mon[33178]: pgmap v127: 148 pgs: 5 remapped+peering, 4 active+undersized, 4 active+undersized+degraded, 135 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 23/333348 objects degraded (0.007%) 2024-09-13T13:04:06.076 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:06.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:06.137 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:06.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:06.161 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:04:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:04:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:04:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:04:06.162 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:04:06.239 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:06.269 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:06.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:06.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:06.351 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:04:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:04:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:04:06.370 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:06.422 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:06.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:06.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:06.524 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:06.557 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:06.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:06.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:06.659 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:06.688 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:06.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:06.790 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:06.807 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:04:06.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:06.841 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:06.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:06.881 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:04:06.926 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:06 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/188161155' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:06.926 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:06 smithi052 ceph-mon[36682]: from='client.24949 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:06.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:06 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/188161155' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:06.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:06 smithi052 ceph-mon[29471]: from='client.24949 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:06.942 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:06.992 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:06.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:07.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:06 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/188161155' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:07.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:06 smithi167 ceph-mon[33178]: from='client.24949 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:07.095 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:07.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:07.159 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:07.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:07.171 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T13:04:07.260 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:07.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:07.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:07.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:07.386 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:07.414 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:07.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:07.516 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:07.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:07.588 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:07.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:07.692 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:07.735 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:07.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:07.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:07.803 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T13:04:07.837 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:07.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:07.868 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:07.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:07.970 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:07.998 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:07.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:08.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:07 smithi167 ceph-mon[33178]: pgmap v128: 148 pgs: 5 remapped+peering, 143 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 684 op/s 2024-09-13T13:04:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:07 smithi167 ceph-mon[33178]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/333348 objects degraded (0.007%), 4 pgs degraded) 2024-09-13T13:04:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:07 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:04:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:07 smithi167 ceph-mon[33178]: from='client.24949 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:04:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:07 smithi167 ceph-mon[33178]: osdmap e121: 8 total, 8 up, 8 in 2024-09-13T13:04:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:07 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3852485992' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:04:08.101 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:08.133 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:08.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:08.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:08.235 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:08.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:07 smithi052 ceph-mon[36682]: pgmap v128: 148 pgs: 5 remapped+peering, 143 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 684 op/s 2024-09-13T13:04:08.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:07 smithi052 ceph-mon[36682]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/333348 objects degraded (0.007%), 4 pgs degraded) 2024-09-13T13:04:08.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:07 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:04:08.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:07 smithi052 ceph-mon[36682]: from='client.24949 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:04:08.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:07 smithi052 ceph-mon[36682]: osdmap e121: 8 total, 8 up, 8 in 2024-09-13T13:04:08.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:07 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3852485992' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:04:08.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:07 smithi052 ceph-mon[29471]: pgmap v128: 148 pgs: 5 remapped+peering, 143 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 684 op/s 2024-09-13T13:04:08.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:07 smithi052 ceph-mon[29471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/333348 objects degraded (0.007%), 4 pgs degraded) 2024-09-13T13:04:08.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:07 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:04:08.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:07 smithi052 ceph-mon[29471]: from='client.24949 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:04:08.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:07 smithi052 ceph-mon[29471]: osdmap e121: 8 total, 8 up, 8 in 2024-09-13T13:04:08.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:07 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3852485992' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:04:08.264 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:08.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:08.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:08.367 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:08.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:08.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:08.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:08.498 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:08.527 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:08.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:08.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:08.629 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:08.642 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:08.654 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:08.655 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T13:04:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:08.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:08.757 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:08.785 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:08.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:08.887 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:08.945 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:08.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3852485992' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:04:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:08 smithi052 ceph-mon[36682]: osdmap e122: 8 total, 8 up, 8 in 2024-09-13T13:04:08.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3852485992' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:04:08.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:08 smithi052 ceph-mon[29471]: osdmap e122: 8 total, 8 up, 8 in 2024-09-13T13:04:09.047 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:09.104 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:09.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:09.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:09.247 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:09.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:09.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3852485992' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:04:09.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:08 smithi167 ceph-mon[33178]: osdmap e122: 8 total, 8 up, 8 in 2024-09-13T13:04:09.349 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:09.395 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:09.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:09.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:09.497 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:09.533 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:09.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:09.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:09.635 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:09.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:09.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:09.767 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:09.797 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:09.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:09.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:09.899 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:09.944 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:09.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:09.995 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:04:10.046 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:10.072 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:10.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:10.174 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:10.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:10.229 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:10.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:10.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:09 smithi052 ceph-mon[36682]: pgmap v131: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.25k op/s; 12160/323109 objects degraded (3.763%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T13:04:10.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:09 smithi052 ceph-mon[36682]: Health check failed: Degraded data redundancy: 12160/323109 objects degraded (3.763%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:10.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:09 smithi052 ceph-mon[29471]: pgmap v131: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.25k op/s; 12160/323109 objects degraded (3.763%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T13:04:10.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:09 smithi052 ceph-mon[29471]: Health check failed: Degraded data redundancy: 12160/323109 objects degraded (3.763%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:10.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:09 smithi167 ceph-mon[33178]: pgmap v131: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.25k op/s; 12160/323109 objects degraded (3.763%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T13:04:10.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:09 smithi167 ceph-mon[33178]: Health check failed: Degraded data redundancy: 12160/323109 objects degraded (3.763%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:10.330 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:10.355 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:10.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:10.456 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:10.480 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:10.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:10.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:10.583 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:10.619 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:10.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:10.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:10.721 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:10.749 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:10.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:10.851 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:10.877 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:10.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:10.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:10.967 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:04:10.968 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:04:10.968 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:04:10.968 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:04:10.968 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:04:10.968 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:04:10.968 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:04:10.968 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:04:10.969 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:04:10.969 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:04:10.969 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:04:10.969 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:04:10.969 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:04:10.969 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:04:10.969 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:04:10.969 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:04:10.979 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:11.026 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:11.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:11.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:11.128 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:11.158 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:11.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:11.197 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:11.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:10 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2590857517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:11.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:10 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2590857517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:11.260 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:11.319 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:11.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:11.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:10 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2590857517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:11.421 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:11.452 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:11.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:11.554 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:11.584 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:11.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:11.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:11.715 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:11.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:11.817 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:11.835 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:04:11.835 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:04:11.853 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:11.857 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:11.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:11.904 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:11.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:11 smithi052 ceph-mon[29471]: pgmap v132: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 884 B/s rd, 0 B/s wr, 492 op/s; 12160/323109 objects degraded (3.763%); 6.2 MiB/s, 100 objects/s recovering 2024-09-13T13:04:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:11 smithi052 ceph-mon[36682]: pgmap v132: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 884 B/s rd, 0 B/s wr, 492 op/s; 12160/323109 objects degraded (3.763%); 6.2 MiB/s, 100 objects/s recovering 2024-09-13T13:04:12.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:12.038 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:12.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:12.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:12.168 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:12.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:12.270 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:12.317 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:12.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:12.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:12.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:11 smithi167 ceph-mon[33178]: pgmap v132: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 884 B/s rd, 0 B/s wr, 492 op/s; 12160/323109 objects degraded (3.763%); 6.2 MiB/s, 100 objects/s recovering 2024-09-13T13:04:12.418 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:12.453 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:12.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:12.555 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:12.600 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:12.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:12.702 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:12.731 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:12.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:12.832 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:12.863 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:12.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:12.965 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:13.010 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:13.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:13.112 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:13.138 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:13.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:13.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:13.240 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:13.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:13.271 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:13.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:13.345 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T13:04:13.372 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:13.442 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:13.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:13.543 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:13.583 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:13.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:13.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:13.711 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:13.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:13.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:13.813 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:13.853 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:13.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:13.955 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:13.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:13.995 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:13.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:14.041 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:13 smithi052 ceph-mon[36682]: pgmap v133: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 427 op/s; 12160/323109 objects degraded (3.763%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T13:04:14.041 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:13 smithi052 ceph-mon[36682]: osdmap e123: 8 total, 8 up, 8 in 2024-09-13T13:04:14.041 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:13 smithi052 ceph-mon[29471]: pgmap v133: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 427 op/s; 12160/323109 objects degraded (3.763%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T13:04:14.041 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:13 smithi052 ceph-mon[29471]: osdmap e123: 8 total, 8 up, 8 in 2024-09-13T13:04:14.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:04:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:04:14.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:13 smithi167 ceph-mon[33178]: pgmap v133: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 427 op/s; 12160/323109 objects degraded (3.763%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T13:04:14.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:13 smithi167 ceph-mon[33178]: osdmap e123: 8 total, 8 up, 8 in 2024-09-13T13:04:14.096 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:14.159 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:14.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:14.262 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:14.315 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:14.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:14.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:14.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:14.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:14.446 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:14.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:14.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:14.576 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:14.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:14.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:14.678 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:14.697 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:14.770 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:14.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:14.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:14.775 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T13:04:14.783 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:04:14.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:14.903 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:14.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:14.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:14.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:14 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:14.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:14 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3404847680' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:04:14.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:14 smithi052 ceph-mon[29471]: from='client.34961 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:04:14.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:14 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2018818039' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:04:14.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:14 smithi052 ceph-mon[29471]: from='client.34967 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:04:14.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:14 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:14.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:14 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3404847680' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:04:14.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:14 smithi052 ceph-mon[36682]: from='client.34961 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:04:14.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:14 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2018818039' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:04:14.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:14 smithi052 ceph-mon[36682]: from='client.34967 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:04:15.005 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:15.033 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:15.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:15.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:15.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:14 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:15.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:14 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3404847680' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:04:15.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:14 smithi167 ceph-mon[33178]: from='client.34961 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:04:15.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:14 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2018818039' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:04:15.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:14 smithi167 ceph-mon[33178]: from='client.34967 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:04:15.135 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:15.175 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:15.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:15.276 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:15.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:15.305 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:15.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:15.406 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:15.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:15.453 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:15.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:15.554 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:15.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:15.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:15.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:15.672 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:04:15.672 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:04:15.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:15.711 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:15.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:15.813 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:15.844 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:15.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:15.946 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:15.987 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:15.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:16.038 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:15 smithi052 ceph-mon[36682]: pgmap v135: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 986 op/s; 11183/310017 objects degraded (3.607%); 13 MiB/s, 211 objects/s recovering 2024-09-13T13:04:16.039 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:15 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:04:16.039 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:15 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 11183/310017 objects degraded (3.607%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:16.039 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:15 smithi052 ceph-mon[36682]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:04:16.039 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:15 smithi052 ceph-mon[36682]: from='client.34967 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:04:16.039 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:15 smithi052 ceph-mon[36682]: osdmap e124: 8 total, 8 up, 8 in 2024-09-13T13:04:16.040 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:15 smithi052 ceph-mon[29471]: pgmap v135: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 986 op/s; 11183/310017 objects degraded (3.607%); 13 MiB/s, 211 objects/s recovering 2024-09-13T13:04:16.040 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:15 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:04:16.040 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:15 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 11183/310017 objects degraded (3.607%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:16.040 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:15 smithi052 ceph-mon[29471]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:04:16.040 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:15 smithi052 ceph-mon[29471]: from='client.34967 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:04:16.040 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:15 smithi052 ceph-mon[29471]: osdmap e124: 8 total, 8 up, 8 in 2024-09-13T13:04:16.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:15 smithi167 ceph-mon[33178]: pgmap v135: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 986 op/s; 11183/310017 objects degraded (3.607%); 13 MiB/s, 211 objects/s recovering 2024-09-13T13:04:16.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:15 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:04:16.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:15 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 11183/310017 objects degraded (3.607%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:15 smithi167 ceph-mon[33178]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:04:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:15 smithi167 ceph-mon[33178]: from='client.34967 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:04:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:15 smithi167 ceph-mon[33178]: osdmap e124: 8 total, 8 up, 8 in 2024-09-13T13:04:16.090 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:16.133 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:16.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:16.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:16.235 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:16.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:16.279 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:16.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:16.380 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:16.383 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:04:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:04:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:04:16.440 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:16.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:16.445 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T13:04:16.543 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:16.591 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:16.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:16.693 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:16.719 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:16.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:16.741 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:04:16.821 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:16.848 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:16.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:16.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:16.950 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:17.006 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:17.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:17.107 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:17.150 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:17.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:17.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:17.252 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:17.293 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:17.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:17.351 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T13:04:17.395 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:17.454 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:17.481 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:17.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:17.519 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:17.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:17.620 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:17.667 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:17.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:17.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:17.769 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:17.821 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:17.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:17.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:17.923 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:17.953 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:17.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:17.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:17.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:17 smithi052 ceph-mon[29471]: pgmap v137: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 545 op/s; 11183/310017 objects degraded (3.607%); 7.6 MiB/s, 122 objects/s recovering 2024-09-13T13:04:17.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:17 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:04:17.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:17 smithi052 ceph-mon[29471]: osdmap e125: 8 total, 8 up, 8 in 2024-09-13T13:04:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:17 smithi052 ceph-mon[36682]: pgmap v137: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 545 op/s; 11183/310017 objects degraded (3.607%); 7.6 MiB/s, 122 objects/s recovering 2024-09-13T13:04:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:17 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:04:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:17 smithi052 ceph-mon[36682]: osdmap e125: 8 total, 8 up, 8 in 2024-09-13T13:04:18.055 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:18.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:17 smithi167 ceph-mon[33178]: pgmap v137: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 545 op/s; 11183/310017 objects degraded (3.607%); 7.6 MiB/s, 122 objects/s recovering 2024-09-13T13:04:18.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:17 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:04:18.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:17 smithi167 ceph-mon[33178]: osdmap e125: 8 total, 8 up, 8 in 2024-09-13T13:04:18.092 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:18.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:18.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:18.194 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:18.225 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:18.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:18.327 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:18.368 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:18.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:18.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:18.469 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:18.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:18.520 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:18.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:18.621 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:18.648 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:18.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:18.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:18.750 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:18.788 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:18.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:18.890 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:18.919 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:04:18.929 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:18.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:18.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:19.032 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:19.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:18 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:04:19.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:18 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:04:19.088 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:19.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:19.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:19.190 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:19.222 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:19.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:19.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:19.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:18 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:04:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:18 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:04:19.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:18 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:04:19.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:18 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:04:19.323 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:19.379 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:19.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:19.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:19.481 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:19.532 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:19.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:19.633 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:19.662 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:19.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:19.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:19.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:19.792 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:19.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:19.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:19.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:19.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:19.932 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:19.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:20.033 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:20.055 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:20.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:20.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:19 smithi167 ceph-mon[33178]: pgmap v139: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.45k op/s; 10088/297057 objects degraded (3.396%); 22 MiB/s, 344 objects/s recovering 2024-09-13T13:04:20.157 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:20.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:20.209 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:20.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:20.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:19 smithi052 ceph-mon[36682]: pgmap v139: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.45k op/s; 10088/297057 objects degraded (3.396%); 22 MiB/s, 344 objects/s recovering 2024-09-13T13:04:20.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:19 smithi052 ceph-mon[29471]: pgmap v139: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.45k op/s; 10088/297057 objects degraded (3.396%); 22 MiB/s, 344 objects/s recovering 2024-09-13T13:04:20.311 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:20.343 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:20.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:20.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:20.445 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:20.486 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:20.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:20.588 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:20.610 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:20.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:20.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:20.712 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:20.742 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:20.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:20.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:20.844 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:20.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:20.899 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:20.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:21.000 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:21.051 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:21.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:21.152 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:21.195 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:21.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:21.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:21.297 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:21.342 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:21.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:21.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:21.444 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:21.466 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:21.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:21.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:21.606 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:21.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:21.707 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:21.748 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:21.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:21.850 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:21.884 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:21.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:21.986 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:22.019 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:22.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:22.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:21 smithi167 ceph-mon[33178]: pgmap v140: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 10088/297057 objects degraded (3.396%); 16 MiB/s, 260 objects/s recovering 2024-09-13T13:04:22.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:21 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2126575660' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:04:22.072 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T13:04:22.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:21 smithi052 ceph-mon[29471]: pgmap v140: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 10088/297057 objects degraded (3.396%); 16 MiB/s, 260 objects/s recovering 2024-09-13T13:04:22.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:21 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2126575660' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:04:22.074 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:21 smithi052 ceph-mon[36682]: pgmap v140: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 10088/297057 objects degraded (3.396%); 16 MiB/s, 260 objects/s recovering 2024-09-13T13:04:22.074 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:21 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2126575660' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:04:22.120 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:22.169 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:22.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:22.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:22.271 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:22.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:22.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:22.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:22.425 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:22.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:22.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:22.556 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:22.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:22.658 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:22.684 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:22.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:22.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:22.786 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:22.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:22.816 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:22.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:22.917 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:22.947 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:22.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:22.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:23.049 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:23.075 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:23.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:23.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:23.176 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:23.225 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:23.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:23.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:22 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2126575660' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:04:23.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:22 smithi052 ceph-mon[36682]: osdmap e126: 8 total, 8 up, 8 in 2024-09-13T13:04:23.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:22 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 10088/297057 objects degraded (3.396%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:23.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:22 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2126575660' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:04:23.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:22 smithi052 ceph-mon[29471]: osdmap e126: 8 total, 8 up, 8 in 2024-09-13T13:04:23.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:22 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 10088/297057 objects degraded (3.396%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:23.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:22 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2126575660' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:04:23.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:22 smithi167 ceph-mon[33178]: osdmap e126: 8 total, 8 up, 8 in 2024-09-13T13:04:23.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:22 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 10088/297057 objects degraded (3.396%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:23.327 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:23.357 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:23.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:23.459 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:23.497 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:23.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:23.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:23.599 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:23.635 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:23.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:23.737 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:23.764 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:23.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:23.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:23.866 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:23.906 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:23.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:24.008 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:24.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:24.033 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:24.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:24.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:23 smithi167 ceph-mon[33178]: pgmap v142: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 545 op/s; 10088/297057 objects degraded (3.396%); 8.6 MiB/s, 138 objects/s recovering 2024-09-13T13:04:24.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:04:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:04:24.134 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:24.161 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:24.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:24.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:24.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:23 smithi052 ceph-mon[36682]: pgmap v142: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 545 op/s; 10088/297057 objects degraded (3.396%); 8.6 MiB/s, 138 objects/s recovering 2024-09-13T13:04:24.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:23 smithi052 ceph-mon[29471]: pgmap v142: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 545 op/s; 10088/297057 objects degraded (3.396%); 8.6 MiB/s, 138 objects/s recovering 2024-09-13T13:04:24.263 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:24.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:24.286 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:24.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:24.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:24.416 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:24.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:24.518 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:24.544 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:24.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:24.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:24.646 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:24.673 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:24.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:24.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:24.775 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:24.797 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:24.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:24.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:24.930 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:24.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:24.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:25.033 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:25.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:25.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:25.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:25.158 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:25.181 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:25.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:25.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:24 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:25.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:24 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:04:25.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:24 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:04:25.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:24 smithi052 ceph-mon[36682]: osdmap e127: 8 total, 8 up, 8 in 2024-09-13T13:04:25.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:24 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:25.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:24 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:04:25.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:24 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:04:25.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:24 smithi052 ceph-mon[29471]: osdmap e127: 8 total, 8 up, 8 in 2024-09-13T13:04:25.282 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:25.309 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:25.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:25.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:24 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:25.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:24 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:04:25.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:24 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:04:25.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:24 smithi167 ceph-mon[33178]: osdmap e127: 8 total, 8 up, 8 in 2024-09-13T13:04:25.410 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:25.433 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:25.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:25.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:25.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:25.557 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:25.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:25.658 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:25.685 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:25.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:25.786 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:25.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:25.809 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:25.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:25.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:25.934 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:25.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:26.036 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:26.063 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:26.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:26.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:26.165 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:26.212 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:26.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:26.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:26.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:04:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:04:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:04:26.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:25 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:04:26.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:25 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:04:26.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:25 smithi052 ceph-mon[36682]: pgmap v144: 148 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.20k op/s; 8837/285096 objects degraded (3.100%); 21 MiB/s, 369 objects/s recovering 2024-09-13T13:04:26.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:25 smithi052 ceph-mon[36682]: osdmap e128: 8 total, 8 up, 8 in 2024-09-13T13:04:26.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:25 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:04:26.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:25 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:04:26.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:25 smithi052 ceph-mon[29471]: pgmap v144: 148 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.20k op/s; 8837/285096 objects degraded (3.100%); 21 MiB/s, 369 objects/s recovering 2024-09-13T13:04:26.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:25 smithi052 ceph-mon[29471]: osdmap e128: 8 total, 8 up, 8 in 2024-09-13T13:04:26.313 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:26.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:25 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:04:26.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:25 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:04:26.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:25 smithi167 ceph-mon[33178]: pgmap v144: 148 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.20k op/s; 8837/285096 objects degraded (3.100%); 21 MiB/s, 369 objects/s recovering 2024-09-13T13:04:26.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:25 smithi167 ceph-mon[33178]: osdmap e128: 8 total, 8 up, 8 in 2024-09-13T13:04:26.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:26.357 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:26.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:26.458 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:26.502 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:26.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:26.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:26.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:26.630 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:26.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:26.731 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:26.770 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:26.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:26.872 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:26.910 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:26.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:27.011 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:27.053 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:27.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:27.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:27.155 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:27.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:27.184 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:27.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:27.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:26 smithi052 ceph-mon[36682]: osdmap e129: 8 total, 8 up, 8 in 2024-09-13T13:04:27.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:26 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/870625806' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:27.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:26 smithi052 ceph-mon[29471]: osdmap e129: 8 total, 8 up, 8 in 2024-09-13T13:04:27.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:26 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/870625806' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:27.286 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:27.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:26 smithi167 ceph-mon[33178]: osdmap e129: 8 total, 8 up, 8 in 2024-09-13T13:04:27.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:26 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/870625806' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:27.336 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:27.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:27.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:27.438 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:27.474 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:27.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:27.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:27.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:27.602 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:27.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:27.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:27.704 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:27.707 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:04:27.707 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:04:27.707 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:04:27.707 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:04:27.708 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:04:27.708 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:04:27.708 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:04:27.708 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:04:27.708 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:04:27.708 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:04:27.708 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:04:27.708 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:04:27.708 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:04:27.708 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:04:27.709 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:04:27.709 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:04:27.743 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:27.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:27.845 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:27.869 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:27.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:27.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:27.971 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:27.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:27 smithi052 ceph-mon[36682]: pgmap v147: 148 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 827 op/s; 8837/285096 objects degraded (3.100%); 16 MiB/s, 305 objects/s recovering 2024-09-13T13:04:27.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:27 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 8837/285096 objects degraded (3.100%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:27.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:27 smithi052 ceph-mon[36682]: osdmap e130: 8 total, 8 up, 8 in 2024-09-13T13:04:27.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:27 smithi052 ceph-mon[29471]: pgmap v147: 148 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 827 op/s; 8837/285096 objects degraded (3.100%); 16 MiB/s, 305 objects/s recovering 2024-09-13T13:04:27.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:27 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 8837/285096 objects degraded (3.100%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:27.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:27 smithi052 ceph-mon[29471]: osdmap e130: 8 total, 8 up, 8 in 2024-09-13T13:04:27.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:27.998 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:27.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:28.099 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:28.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:28.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:28.226 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:28.265 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:28.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:28.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:28.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:27 smithi167 ceph-mon[33178]: pgmap v147: 148 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 827 op/s; 8837/285096 objects degraded (3.100%); 16 MiB/s, 305 objects/s recovering 2024-09-13T13:04:28.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:27 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 8837/285096 objects degraded (3.100%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:28.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:27 smithi167 ceph-mon[33178]: osdmap e130: 8 total, 8 up, 8 in 2024-09-13T13:04:28.367 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:28.389 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:28.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:28.491 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:28.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:28.533 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:28.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:28.634 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:28.676 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:28.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:28.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:28.777 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:28.804 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:28.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:28.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:28.906 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:28.948 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:28.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:28.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:29.050 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:29.077 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:29.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:29.179 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:29.204 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:29.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:29.306 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:29.350 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:29.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:29.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:29.452 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:29.480 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:29.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:29.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:29.581 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:29.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:29.620 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:29.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:29.722 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:29.749 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:29.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:29.850 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:29.877 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:29.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:29.979 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:30.012 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:30.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:30.114 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:30.140 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:30.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:30.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:30.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:29 smithi052 ceph-mon[36682]: pgmap v149: 148 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 669 op/s; 7323/275466 objects degraded (2.658%); 11 MiB/s, 265 objects/s recovering 2024-09-13T13:04:30.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:29 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:04:30.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:29 smithi052 ceph-mon[29471]: pgmap v149: 148 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 669 op/s; 7323/275466 objects degraded (2.658%); 11 MiB/s, 265 objects/s recovering 2024-09-13T13:04:30.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:29 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:04:30.243 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:30.266 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:29 smithi167 ceph-mon[33178]: pgmap v149: 148 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 669 op/s; 7323/275466 objects degraded (2.658%); 11 MiB/s, 265 objects/s recovering 2024-09-13T13:04:30.266 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:29 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:04:30.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:30.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:30.351 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T13:04:30.368 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:30.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:30.373 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:30.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:30.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:30.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:30.501 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:30.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:30.567 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:30.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:30.669 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:30.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:30.711 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:30.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:30.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:30.837 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:30.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:30.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:30.939 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:30.991 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:30.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:30.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:31.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:31.117 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:31.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:31.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:31.219 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:31.274 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:31.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:31.376 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:31.426 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:31.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:31.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:31.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:31.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:31.560 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:31.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:31.661 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:31.684 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:31.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:31.786 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:31.835 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:31.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:31.937 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:32.008 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:32.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:32.011 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:04:32.011 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:04:32.011 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:04:32.012 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:04:32.012 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:04:32.110 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:32.137 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:32.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:32.238 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:32.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:31 smithi052 ceph-mon[29471]: pgmap v150: 148 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 535 op/s; 7323/275466 objects degraded (2.658%); 9.1 MiB/s, 212 objects/s recovering 2024-09-13T13:04:32.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:31 smithi052 ceph-mon[36682]: pgmap v150: 148 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 535 op/s; 7323/275466 objects degraded (2.658%); 9.1 MiB/s, 212 objects/s recovering 2024-09-13T13:04:32.263 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:32.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:32.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:32.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:31 smithi167 ceph-mon[33178]: pgmap v150: 148 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 535 op/s; 7323/275466 objects degraded (2.658%); 9.1 MiB/s, 212 objects/s recovering 2024-09-13T13:04:32.365 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:32.405 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:32.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:32.506 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:32.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:32.551 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:32.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:32.652 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:32.692 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:32.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:32.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:32.794 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:32.829 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:32.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:32.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:32.848 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:04:32.931 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:32.949 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:04:32.959 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:32.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:32.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:33.061 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:33.109 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:33.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:33.210 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:32 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1932215087' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:32 smithi052 ceph-mon[36682]: from='client.24982 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:04:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:32 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 7323/275466 objects degraded (2.658%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:32 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1932215087' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:32 smithi052 ceph-mon[29471]: from='client.24982 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:04:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:32 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 7323/275466 objects degraded (2.658%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:33.259 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:33.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:33.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:32 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1932215087' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:33.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:32 smithi167 ceph-mon[33178]: from='client.24982 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:33.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:04:33.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:32 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 7323/275466 objects degraded (2.658%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:33.361 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:33.405 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:33.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:33.507 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:33.562 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:33.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:33.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:33.712 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:33.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:33.813 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:33.861 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:33.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:33.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:33.956 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:04:33 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:04:33.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:33.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:33.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:34.093 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:34.123 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:34.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:34.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:34.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:34.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:33 smithi052 ceph-mon[36682]: pgmap v151: 148 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 412 op/s; 7323/275466 objects degraded (2.658%); 7.0 MiB/s, 163 objects/s recovering 2024-09-13T13:04:34.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:33 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:04:34.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:33 smithi052 ceph-mon[36682]: from='client.24982 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:04:34.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:33 smithi052 ceph-mon[36682]: osdmap e131: 8 total, 8 up, 8 in 2024-09-13T13:04:34.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:33 smithi052 ceph-mon[29471]: pgmap v151: 148 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 412 op/s; 7323/275466 objects degraded (2.658%); 7.0 MiB/s, 163 objects/s recovering 2024-09-13T13:04:34.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:33 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:04:34.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:33 smithi052 ceph-mon[29471]: from='client.24982 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:04:34.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:33 smithi052 ceph-mon[29471]: osdmap e131: 8 total, 8 up, 8 in 2024-09-13T13:04:34.268 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:34.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:34.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:33 smithi167 ceph-mon[33178]: pgmap v151: 148 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 412 op/s; 7323/275466 objects degraded (2.658%); 7.0 MiB/s, 163 objects/s recovering 2024-09-13T13:04:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:33 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:04:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:33 smithi167 ceph-mon[33178]: from='client.24982 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:04:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:33 smithi167 ceph-mon[33178]: osdmap e131: 8 total, 8 up, 8 in 2024-09-13T13:04:34.370 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:34.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:34.414 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:34.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:34.515 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:34.525 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:34.556 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:34.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:34.658 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:34.710 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:34.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:34.812 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:34.851 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:34.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:34.952 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:34.995 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:34.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:34.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:35.014 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:35.097 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:35.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:35.152 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:35.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:35.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:34 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:34 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:34 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:35.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:34 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:35.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:34 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:35.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:34 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:35.253 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:35.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:35.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:34 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:35.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:34 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:34 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:35.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:35.430 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:35.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:35.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:35.533 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:35.572 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:35.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:35.674 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:35.724 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:35.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:35.734 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T13:04:35.825 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:35.883 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:35.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:35.986 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:36.008 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:36.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:36.110 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:36.165 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:36.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:36.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:36.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:35 smithi052 ceph-mon[29471]: pgmap v153: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 892 op/s; 6517/263700 objects degraded (2.471%); 13 MiB/s, 259 objects/s recovering 2024-09-13T13:04:36.235 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:04:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:04:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:04:36.235 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:35 smithi052 ceph-mon[36682]: pgmap v153: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 892 op/s; 6517/263700 objects degraded (2.471%); 13 MiB/s, 259 objects/s recovering 2024-09-13T13:04:36.268 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:36.319 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:36.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:36.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:36.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:35 smithi167 ceph-mon[33178]: pgmap v153: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 892 op/s; 6517/263700 objects degraded (2.471%); 13 MiB/s, 259 objects/s recovering 2024-09-13T13:04:36.421 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:36.480 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:36.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:36.582 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:36.606 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:36.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:36.707 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:04:36.707 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:04:36.708 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:36.732 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:36.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:36.763 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:04:36.763 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:04:36.831 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:36.834 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:36.841 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:04:36.842 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:04:36.860 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:36.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:36.912 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:04:36.912 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:04:36.945 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:04:36.945 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:04:36.962 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:36.987 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:36.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:36.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:36.990 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:04:36.990 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:04:37.029 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:04:37.030 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:04:37.050 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:04:37.070 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:04:37.070 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:04:37.089 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:37.101 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:36 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4022508090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:37.102 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:36 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4022508090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:37.139 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:37.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:37.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:37.241 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:37.317 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:37.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:37.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:36 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4022508090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:37.418 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:37.461 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:37.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:37.562 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:37.601 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:37.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:37.703 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:37.726 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:37.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:37.764 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T13:04:37.828 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:37.884 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:37.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:37.986 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:38.043 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:38.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:38.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:38.145 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:38.184 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:38.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:38.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:38.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:37 smithi052 ceph-mon[36682]: pgmap v154: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 797 op/s; 6517/263700 objects degraded (2.471%); 12 MiB/s, 232 objects/s recovering 2024-09-13T13:04:38.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:37 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3055285792' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:04:38.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:37 smithi052 ceph-mon[36682]: from='client.34988 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:04:38.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:37 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 6517/263700 objects degraded (2.471%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:38.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:37 smithi052 ceph-mon[36682]: from='client.34988 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:04:38.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:37 smithi052 ceph-mon[36682]: osdmap e132: 8 total, 8 up, 8 in 2024-09-13T13:04:38.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:37 smithi052 ceph-mon[29471]: pgmap v154: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 797 op/s; 6517/263700 objects degraded (2.471%); 12 MiB/s, 232 objects/s recovering 2024-09-13T13:04:38.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:37 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3055285792' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:04:38.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:37 smithi052 ceph-mon[29471]: from='client.34988 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:04:38.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:37 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 6517/263700 objects degraded (2.471%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:38.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:37 smithi052 ceph-mon[29471]: from='client.34988 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:04:38.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:37 smithi052 ceph-mon[29471]: osdmap e132: 8 total, 8 up, 8 in 2024-09-13T13:04:38.286 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:38.299 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:04:38.299 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:04:38.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:37 smithi167 ceph-mon[33178]: pgmap v154: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 797 op/s; 6517/263700 objects degraded (2.471%); 12 MiB/s, 232 objects/s recovering 2024-09-13T13:04:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:37 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3055285792' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:04:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:37 smithi167 ceph-mon[33178]: from='client.34988 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:04:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:37 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 6517/263700 objects degraded (2.471%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:37 smithi167 ceph-mon[33178]: from='client.34988 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:04:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:37 smithi167 ceph-mon[33178]: osdmap e132: 8 total, 8 up, 8 in 2024-09-13T13:04:38.330 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:38.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:38.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:38.432 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:38.457 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:38.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:38.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:38.559 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:38.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:38.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:38.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:38.690 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:38.729 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:38.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:38.832 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:38.881 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:38.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:38.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:38.983 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:39.025 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:39.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:39.127 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:39.151 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:39.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:39.253 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:39.292 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:39.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:39.394 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:39.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:39.424 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:39.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:39.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:39.552 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:39.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:39.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:39.679 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:39.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:39.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:39.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:39.807 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:39.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:39.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:39.937 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:39.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:39.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:40.039 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:40.064 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:40.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:40.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:40.166 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:40.215 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:40.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:40.317 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:40.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:39 smithi167 ceph-mon[33178]: pgmap v156: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 5623/251310 objects degraded (2.237%); 13 MiB/s, 212 objects/s recovering 2024-09-13T13:04:40.356 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:40.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:40.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:40.458 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:40.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:39 smithi052 ceph-mon[36682]: pgmap v156: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 5623/251310 objects degraded (2.237%); 13 MiB/s, 212 objects/s recovering 2024-09-13T13:04:40.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:39 smithi052 ceph-mon[29471]: pgmap v156: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 5623/251310 objects degraded (2.237%); 13 MiB/s, 212 objects/s recovering 2024-09-13T13:04:40.501 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:40.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:40.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:40.603 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:40.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:40.649 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:40.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:40.749 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:40.804 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:40.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:40.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:40.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:40.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:41.030 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:41.076 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:41.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:41.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:41.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:41.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:41.210 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:41.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:41.311 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:41.327 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T13:04:41.352 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:41.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:41.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:41.454 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:41.511 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:41.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:41.612 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:41.636 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:41.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:41.738 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:41.764 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:41.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:41.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:41.866 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:41.882 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T13:04:41.935 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:41.937 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:41.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:41.940 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:42.039 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:42.064 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:42.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:42.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:42.093 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:41 smithi052 ceph-mon[29471]: pgmap v157: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 5623/251310 objects degraded (2.237%); 13 MiB/s, 212 objects/s recovering 2024-09-13T13:04:42.097 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:42 smithi052 ceph-mon[36682]: pgmap v157: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 5623/251310 objects degraded (2.237%); 13 MiB/s, 212 objects/s recovering 2024-09-13T13:04:42.166 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:42.219 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:42.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:42.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:42.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:41 smithi167 ceph-mon[33178]: pgmap v157: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 5623/251310 objects degraded (2.237%); 13 MiB/s, 212 objects/s recovering 2024-09-13T13:04:42.321 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:42.363 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:42.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:42.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:42.489 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:42.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:42.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:42.617 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:42.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:42.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:42.651 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:04:42.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:42.777 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:42.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:42.880 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:42.976 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:42.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:43.078 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:43.148 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:43.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:43.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:43.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:43 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 5623/251310 objects degraded (2.237%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:43 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 5623/251310 objects degraded (2.237%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:43.250 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:43.263 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:43.277 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:43.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:43 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 5623/251310 objects degraded (2.237%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:43.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:43.357 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:43.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:43.458 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:43.516 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:43.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:43.617 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:43.673 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:43.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:43.774 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:43.801 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:43.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:43.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:43.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:43.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:44.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:44.042 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:04:44.055 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:04:43 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:04:44.056 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T13:04:44.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:44.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:44.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:44.158 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:44.181 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:04:44.182 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:04:44.208 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:44.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:44.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:44.310 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:44.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:44 smithi167 ceph-mon[33178]: pgmap v158: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 524 B/s rd, 0 B/s wr, 751 op/s; 5623/251310 objects degraded (2.237%); 11 MiB/s, 174 objects/s recovering 2024-09-13T13:04:44.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/723488447' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:04:44.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:44 smithi167 ceph-mon[33178]: from='client.34997 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:04:44.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1365455637' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:04:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:44.340 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:44.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:44.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:44 smithi052 ceph-mon[29471]: pgmap v158: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 524 B/s rd, 0 B/s wr, 751 op/s; 5623/251310 objects degraded (2.237%); 11 MiB/s, 174 objects/s recovering 2024-09-13T13:04:44.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/723488447' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:04:44.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:44 smithi052 ceph-mon[29471]: from='client.34997 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:04:44.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1365455637' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:04:44.440 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:44 smithi052 ceph-mon[36682]: pgmap v158: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 524 B/s rd, 0 B/s wr, 751 op/s; 5623/251310 objects degraded (2.237%); 11 MiB/s, 174 objects/s recovering 2024-09-13T13:04:44.440 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/723488447' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:04:44.440 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:44 smithi052 ceph-mon[36682]: from='client.34997 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:04:44.440 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1365455637' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:04:44.441 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:44.490 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:44.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:44.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:44.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:44.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:44.652 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:04:44.652 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:04:44.688 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:04:44.689 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:04:44.718 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:44.740 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:04:44.740 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:04:44.744 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:44.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:44.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:44.788 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:04:44.788 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:04:44.824 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:04:44.824 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:04:44.847 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:44.862 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:04:44.862 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:04:44.874 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:44.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:44.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:44.901 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:04:44.901 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:04:44.937 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:04:44.937 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:04:44.976 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:45.001 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:45.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:45.047 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T13:04:45.103 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:45.160 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:45.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:45.233 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:45.233 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:04:45.233 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[29471]: from='client.34997 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:04:45.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1365455637' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:04:45.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:04:45.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:04:45.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[29471]: osdmap e133: 8 total, 8 up, 8 in 2024-09-13T13:04:45.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1151559219' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:45.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[29471]: osdmap e134: 8 total, 8 up, 8 in 2024-09-13T13:04:45.234 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:45.234 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:04:45.234 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[36682]: from='client.34997 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:04:45.234 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1365455637' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:04:45.234 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:04:45.235 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:04:45.235 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[36682]: osdmap e133: 8 total, 8 up, 8 in 2024-09-13T13:04:45.235 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1151559219' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:45.235 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:45 smithi052 ceph-mon[36682]: osdmap e134: 8 total, 8 up, 8 in 2024-09-13T13:04:45.262 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:45.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:45.327 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:45.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:45.428 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:45.482 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:45.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:45.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:45 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:45.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:45 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:04:45.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:45 smithi167 ceph-mon[33178]: from='client.34997 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:04:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:45 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1365455637' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:04:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:45 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:04:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:45 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:04:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:45 smithi167 ceph-mon[33178]: osdmap e133: 8 total, 8 up, 8 in 2024-09-13T13:04:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:45 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1151559219' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:45 smithi167 ceph-mon[33178]: osdmap e134: 8 total, 8 up, 8 in 2024-09-13T13:04:45.583 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:45.615 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:45.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:45.716 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:45.760 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:45.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:45.862 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:45.896 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:45.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:45.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:45.998 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:46.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:46.040 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:46.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:46.141 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:46.167 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:46.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:46.269 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:46.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:46.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:46.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:46.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:46.423 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:46.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:46.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:46.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:46 smithi052 ceph-mon[29471]: pgmap v160: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 4559/238479 objects degraded (1.912%); 15 MiB/s, 244 objects/s recovering 2024-09-13T13:04:46.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:46 smithi052 ceph-mon[36682]: pgmap v160: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 4559/238479 objects degraded (1.912%); 15 MiB/s, 244 objects/s recovering 2024-09-13T13:04:46.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:04:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:04:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:04:46.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:46.552 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:46.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:46.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:46 smithi167 ceph-mon[33178]: pgmap v160: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 4559/238479 objects degraded (1.912%); 15 MiB/s, 244 objects/s recovering 2024-09-13T13:04:46.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:46.681 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:46.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:46.783 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:46.785 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T13:04:46.827 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:46.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:46.930 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:46.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:46.970 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:46.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:47.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:47.094 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:47.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:47.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:47.196 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:47.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:47.242 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:47.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:47.343 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:47.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:47.388 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:47.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:47.489 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:47.532 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:47.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:47.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:47.658 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:47.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:47.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:47.789 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:47.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:47.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:47.891 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:47.917 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:47.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:47.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:48.019 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:48.048 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:48.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:48.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:48.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:48.174 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:48.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:48.276 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:48.301 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:48.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:48.403 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:48.424 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:48.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:48.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:48 smithi052 ceph-mon[36682]: pgmap v162: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 535 op/s; 4559/238479 objects degraded (1.912%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T13:04:48.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:48 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:04:48.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:48 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:04:48.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:48 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:04:48.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:48 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 4559/238479 objects degraded (1.912%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:48.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:48 smithi052 ceph-mon[36682]: osdmap e135: 8 total, 8 up, 8 in 2024-09-13T13:04:48.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:48 smithi052 ceph-mon[29471]: pgmap v162: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 535 op/s; 4559/238479 objects degraded (1.912%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T13:04:48.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:48 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:04:48.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:48 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:04:48.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:48 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:04:48.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:48 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 4559/238479 objects degraded (1.912%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:48.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:48 smithi052 ceph-mon[29471]: osdmap e135: 8 total, 8 up, 8 in 2024-09-13T13:04:48.527 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:48.550 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:48.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:48.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:48 smithi167 ceph-mon[33178]: pgmap v162: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 535 op/s; 4559/238479 objects degraded (1.912%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T13:04:48.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:48 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:04:48.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:48 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:04:48.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:48 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:04:48.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:48 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 4559/238479 objects degraded (1.912%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:48.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:48 smithi167 ceph-mon[33178]: osdmap e135: 8 total, 8 up, 8 in 2024-09-13T13:04:48.653 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:48.679 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:48.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:48.781 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:48.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:48.813 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:48.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:48.914 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:48.954 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:48.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:49.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:49.079 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:49.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:49.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:49.181 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:49.202 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:49.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:49.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:49.325 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:49.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:49.427 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:49.452 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:49.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:49.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:49.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[36682]: 5.f scrub starts 2024-09-13T13:04:49.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[36682]: 5.f scrub ok 2024-09-13T13:04:49.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[36682]: 5.b scrub starts 2024-09-13T13:04:49.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[36682]: 5.b scrub ok 2024-09-13T13:04:49.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[36682]: osdmap e136: 8 total, 8 up, 8 in 2024-09-13T13:04:49.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[36682]: 4.18 scrub starts 2024-09-13T13:04:49.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[36682]: 4.18 scrub ok 2024-09-13T13:04:49.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[29471]: 5.f scrub starts 2024-09-13T13:04:49.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[29471]: 5.f scrub ok 2024-09-13T13:04:49.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[29471]: 5.b scrub starts 2024-09-13T13:04:49.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[29471]: 5.b scrub ok 2024-09-13T13:04:49.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[29471]: osdmap e136: 8 total, 8 up, 8 in 2024-09-13T13:04:49.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[29471]: 4.18 scrub starts 2024-09-13T13:04:49.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:49 smithi052 ceph-mon[29471]: 4.18 scrub ok 2024-09-13T13:04:49.554 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:49.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:49 smithi167 ceph-mon[33178]: 5.f scrub starts 2024-09-13T13:04:49.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:49 smithi167 ceph-mon[33178]: 5.f scrub ok 2024-09-13T13:04:49.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:49 smithi167 ceph-mon[33178]: 5.b scrub starts 2024-09-13T13:04:49.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:49 smithi167 ceph-mon[33178]: 5.b scrub ok 2024-09-13T13:04:49.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:49 smithi167 ceph-mon[33178]: osdmap e136: 8 total, 8 up, 8 in 2024-09-13T13:04:49.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:49 smithi167 ceph-mon[33178]: 4.18 scrub starts 2024-09-13T13:04:49.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:49 smithi167 ceph-mon[33178]: 4.18 scrub ok 2024-09-13T13:04:49.578 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:49.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:49.681 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:49.705 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:49.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:49.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:49.807 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:49.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:49.849 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:49.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:49.950 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:50.002 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:50.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:50.103 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:50.147 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:50.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:50.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:50.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:50 smithi052 ceph-mon[36682]: 6.d scrub starts 2024-09-13T13:04:50.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:50 smithi052 ceph-mon[36682]: 6.d scrub ok 2024-09-13T13:04:50.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:50 smithi052 ceph-mon[36682]: pgmap v164: 148 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.43k op/s; 3494/225624 objects degraded (1.549%); 15 MiB/s, 443 objects/s recovering 2024-09-13T13:04:50.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:50 smithi052 ceph-mon[36682]: osdmap e137: 8 total, 8 up, 8 in 2024-09-13T13:04:50.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:50 smithi052 ceph-mon[36682]: 6.17 scrub starts 2024-09-13T13:04:50.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:50 smithi052 ceph-mon[36682]: 6.17 scrub ok 2024-09-13T13:04:50.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:50 smithi052 ceph-mon[29471]: 6.d scrub starts 2024-09-13T13:04:50.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:50 smithi052 ceph-mon[29471]: 6.d scrub ok 2024-09-13T13:04:50.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:50 smithi052 ceph-mon[29471]: pgmap v164: 148 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.43k op/s; 3494/225624 objects degraded (1.549%); 15 MiB/s, 443 objects/s recovering 2024-09-13T13:04:50.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:50 smithi052 ceph-mon[29471]: osdmap e137: 8 total, 8 up, 8 in 2024-09-13T13:04:50.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:50 smithi052 ceph-mon[29471]: 6.17 scrub starts 2024-09-13T13:04:50.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:50 smithi052 ceph-mon[29471]: 6.17 scrub ok 2024-09-13T13:04:50.249 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:50.275 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:50.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:50.356 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:50.377 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:50.402 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:50.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:50.504 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:50.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:50.552 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:50.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:50.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:50 smithi167 ceph-mon[33178]: 6.d scrub starts 2024-09-13T13:04:50.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:50 smithi167 ceph-mon[33178]: 6.d scrub ok 2024-09-13T13:04:50.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:50 smithi167 ceph-mon[33178]: pgmap v164: 148 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.43k op/s; 3494/225624 objects degraded (1.549%); 15 MiB/s, 443 objects/s recovering 2024-09-13T13:04:50.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:50 smithi167 ceph-mon[33178]: osdmap e137: 8 total, 8 up, 8 in 2024-09-13T13:04:50.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:50 smithi167 ceph-mon[33178]: 6.17 scrub starts 2024-09-13T13:04:50.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:50 smithi167 ceph-mon[33178]: 6.17 scrub ok 2024-09-13T13:04:50.611 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:50.653 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:50.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:50.718 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:50.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:50.819 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:50.867 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:50.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:50.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:50.969 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:50.997 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:50.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:50.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:51.099 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:51.113 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:04:51.140 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:51.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:51.242 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:51.269 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:51.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:51.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:51.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:51.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:51.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:51.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:51.541 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:51.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:51.616 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:04:51.642 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:51.719 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:51.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:51.820 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:51.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:51.900 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:51.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:52.001 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:52.026 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:52.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:52.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:52.128 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:52.151 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:52.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:52.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:52.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:52.278 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:52.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:52.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:52.310 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:52 smithi052 ceph-mon[36682]: 3.13 scrub starts 2024-09-13T13:04:52.310 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:52 smithi052 ceph-mon[36682]: 3.13 scrub ok 2024-09-13T13:04:52.311 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:52 smithi052 ceph-mon[36682]: pgmap v167: 148 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 904 B/s rd, 0 B/s wr, 758 op/s; 3494/225624 objects degraded (1.549%); 4.3 MiB/s, 282 objects/s recovering 2024-09-13T13:04:52.313 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:52 smithi052 ceph-mon[29471]: 3.13 scrub starts 2024-09-13T13:04:52.313 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:52 smithi052 ceph-mon[29471]: 3.13 scrub ok 2024-09-13T13:04:52.313 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:52 smithi052 ceph-mon[29471]: pgmap v167: 148 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 904 B/s rd, 0 B/s wr, 758 op/s; 3494/225624 objects degraded (1.549%); 4.3 MiB/s, 282 objects/s recovering 2024-09-13T13:04:52.380 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:52.384 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:04:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:04:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:04:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:04:52.384 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:04:52.418 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:52.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:52.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:52.520 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:52.549 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:52.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:52.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:52 smithi167 ceph-mon[33178]: 3.13 scrub starts 2024-09-13T13:04:52.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:52 smithi167 ceph-mon[33178]: 3.13 scrub ok 2024-09-13T13:04:52.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:52 smithi167 ceph-mon[33178]: pgmap v167: 148 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 904 B/s rd, 0 B/s wr, 758 op/s; 3494/225624 objects degraded (1.549%); 4.3 MiB/s, 282 objects/s recovering 2024-09-13T13:04:52.651 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:52.701 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:52.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:52.802 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:52.828 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:52.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:52.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:52.931 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:52.954 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:52.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:53.055 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:53.101 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:53.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:53.202 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:53.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:53.228 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:53.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:53.329 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:53.367 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[36682]: 5.4 scrub starts 2024-09-13T13:04:53.368 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[36682]: 5.4 scrub ok 2024-09-13T13:04:53.368 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:04:53.368 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:04:53.368 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:04:53.368 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:04:53.368 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[36682]: osdmap e138: 8 total, 8 up, 8 in 2024-09-13T13:04:53.368 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 3494/225624 objects degraded (1.549%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:53.368 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:53.368 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:53.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[29471]: 5.4 scrub starts 2024-09-13T13:04:53.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[29471]: 5.4 scrub ok 2024-09-13T13:04:53.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:04:53.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:04:53.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:04:53.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:04:53.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[29471]: osdmap e138: 8 total, 8 up, 8 in 2024-09-13T13:04:53.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 3494/225624 objects degraded (1.549%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:53.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:53.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:53 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:53.376 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:53.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:53.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:53.418 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T13:04:53.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:53.503 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:53.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:53.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:53 smithi167 ceph-mon[33178]: 5.4 scrub starts 2024-09-13T13:04:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:53 smithi167 ceph-mon[33178]: 5.4 scrub ok 2024-09-13T13:04:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:53 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:04:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:53 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:04:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:53 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:04:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:53 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:04:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:53 smithi167 ceph-mon[33178]: osdmap e138: 8 total, 8 up, 8 in 2024-09-13T13:04:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:53 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 3494/225624 objects degraded (1.549%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:53 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:53 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:53.606 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:53.634 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:53.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:53.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:53.736 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:53.800 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:53.811 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:53.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:53.824 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:53.912 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:53.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:53.970 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:53.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:54.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:04:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:04:54.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:54.094 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:54.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:54.196 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:54.219 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:54.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:54.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[36682]: 4.8 scrub starts 2024-09-13T13:04:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[36682]: 4.8 scrub ok 2024-09-13T13:04:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[36682]: pgmap v169: 148 pgs: 1 peering, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 886 op/s; 3494/225339 objects degraded (1.551%); 4.9 MiB/s, 336 objects/s recovering 2024-09-13T13:04:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[36682]: 6.2 scrub starts 2024-09-13T13:04:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[36682]: 6.2 scrub ok 2024-09-13T13:04:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:04:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:04:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:54.223 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[29471]: 4.8 scrub starts 2024-09-13T13:04:54.223 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[29471]: 4.8 scrub ok 2024-09-13T13:04:54.223 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[29471]: pgmap v169: 148 pgs: 1 peering, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 886 op/s; 3494/225339 objects degraded (1.551%); 4.9 MiB/s, 336 objects/s recovering 2024-09-13T13:04:54.223 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[29471]: 6.2 scrub starts 2024-09-13T13:04:54.223 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[29471]: 6.2 scrub ok 2024-09-13T13:04:54.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:04:54.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:04:54.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:54 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:54.321 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:54.366 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:54.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:54.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:54.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:54.491 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:54.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:54.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:54 smithi167 ceph-mon[33178]: 4.8 scrub starts 2024-09-13T13:04:54.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:54 smithi167 ceph-mon[33178]: 4.8 scrub ok 2024-09-13T13:04:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:54 smithi167 ceph-mon[33178]: pgmap v169: 148 pgs: 1 peering, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 886 op/s; 3494/225339 objects degraded (1.551%); 4.9 MiB/s, 336 objects/s recovering 2024-09-13T13:04:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:54 smithi167 ceph-mon[33178]: 6.2 scrub starts 2024-09-13T13:04:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:54 smithi167 ceph-mon[33178]: 6.2 scrub ok 2024-09-13T13:04:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:54 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:04:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:54 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:04:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:54 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:04:54.593 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:54.617 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:54.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:54.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:54.719 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:54.745 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:54.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:54.813 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:04:54.814 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:04:54.846 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:54.875 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:04:54.875 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:04:54.904 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:54.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:54.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:54.914 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:04:54.928 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:04:54.929 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:04:54.962 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:04:54.963 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:04:54.999 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:04:55.000 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:04:55.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:55.030 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:55.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:55.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:55.036 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:04:55.036 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:04:55.075 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:04:55.076 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:04:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:04:55.117 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:04:55.132 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:55.154 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T13:04:55.155 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:04:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:55.172 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:55.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:55.273 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:55.298 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:55.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:55.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:55.400 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:55.424 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:55.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:55.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[36682]: pgmap v170: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 668 op/s; 2681/215721 objects degraded (1.243%); 10 MiB/s, 177 objects/s recovering 2024-09-13T13:04:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[36682]: 4.15 scrub starts 2024-09-13T13:04:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[36682]: 4.15 scrub ok 2024-09-13T13:04:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[36682]: 6.1e scrub starts 2024-09-13T13:04:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[36682]: 6.1e scrub ok 2024-09-13T13:04:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[36682]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-13T13:04:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1243325506' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:04:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4039260403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:55.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/224535607' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[29471]: pgmap v170: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 668 op/s; 2681/215721 objects degraded (1.243%); 10 MiB/s, 177 objects/s recovering 2024-09-13T13:04:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[29471]: 4.15 scrub starts 2024-09-13T13:04:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[29471]: 4.15 scrub ok 2024-09-13T13:04:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[29471]: 6.1e scrub starts 2024-09-13T13:04:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[29471]: 6.1e scrub ok 2024-09-13T13:04:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[29471]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-13T13:04:55.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1243325506' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:04:55.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4039260403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:55.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:55 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/224535607' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:55.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:55.551 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:55.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:55.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:55.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:55 smithi167 ceph-mon[33178]: pgmap v170: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 668 op/s; 2681/215721 objects degraded (1.243%); 10 MiB/s, 177 objects/s recovering 2024-09-13T13:04:55.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:04:55.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:55 smithi167 ceph-mon[33178]: 4.15 scrub starts 2024-09-13T13:04:55.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:55 smithi167 ceph-mon[33178]: 4.15 scrub ok 2024-09-13T13:04:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:55 smithi167 ceph-mon[33178]: 6.1e scrub starts 2024-09-13T13:04:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:55 smithi167 ceph-mon[33178]: 6.1e scrub ok 2024-09-13T13:04:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:55 smithi167 ceph-mon[33178]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-13T13:04:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:55 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1243325506' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:04:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:55 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4039260403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:04:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:55 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/224535607' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:04:55.653 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:55.680 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:55.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:55.783 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:55.807 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:55.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:55.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:55.934 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:55.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:55.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:56.036 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:56.077 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:56.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:56.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:56.203 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:56.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:56.305 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:56.307 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:04:56.307 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:04:56.333 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:56.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:56.435 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:56.464 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:56.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[29471]: 5.7 scrub starts 2024-09-13T13:04:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[29471]: 5.7 scrub ok 2024-09-13T13:04:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:04:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1243325506' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:04:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/224535607' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:04:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[29471]: osdmap e139: 8 total, 8 up, 8 in 2024-09-13T13:04:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:04:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:04:56.490 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:04:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:04:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[36682]: 5.7 scrub starts 2024-09-13T13:04:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[36682]: 5.7 scrub ok 2024-09-13T13:04:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:04:56.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1243325506' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:04:56.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/224535607' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:04:56.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[36682]: osdmap e139: 8 total, 8 up, 8 in 2024-09-13T13:04:56.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:04:56.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:56 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:04:56.565 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:56.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:56 smithi167 ceph-mon[33178]: 5.7 scrub starts 2024-09-13T13:04:56.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:56 smithi167 ceph-mon[33178]: 5.7 scrub ok 2024-09-13T13:04:56.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:56 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:04:56.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:56 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1243325506' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:04:56.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:56 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/224535607' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:04:56.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:56 smithi167 ceph-mon[33178]: osdmap e139: 8 total, 8 up, 8 in 2024-09-13T13:04:56.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:56 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:04:56.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:56 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:04:56.589 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:56.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:56.692 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:56.717 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:56.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:56.818 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:56.842 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:56.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:56.943 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:56.987 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:56.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:56.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:57.090 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:57.111 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:57.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:57.213 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:57.240 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:57.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:57.341 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:57.386 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:57.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:57.488 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:57.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:57 smithi052 ceph-mon[29471]: pgmap v172: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 868 B/s rd, 0 B/s wr, 560 op/s; 2681/215721 objects degraded (1.243%); 8.6 MiB/s, 149 objects/s recovering 2024-09-13T13:04:57.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:57 smithi052 ceph-mon[29471]: 4.1a scrub starts 2024-09-13T13:04:57.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:57 smithi052 ceph-mon[29471]: 4.1a scrub ok 2024-09-13T13:04:57.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:57 smithi052 ceph-mon[29471]: 5.a scrub starts 2024-09-13T13:04:57.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:57 smithi052 ceph-mon[29471]: 5.a scrub ok 2024-09-13T13:04:57.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:57 smithi052 ceph-mon[29471]: osdmap e140: 8 total, 8 up, 8 in 2024-09-13T13:04:57.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:57 smithi052 ceph-mon[36682]: pgmap v172: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 868 B/s rd, 0 B/s wr, 560 op/s; 2681/215721 objects degraded (1.243%); 8.6 MiB/s, 149 objects/s recovering 2024-09-13T13:04:57.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:57 smithi052 ceph-mon[36682]: 4.1a scrub starts 2024-09-13T13:04:57.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:57 smithi052 ceph-mon[36682]: 4.1a scrub ok 2024-09-13T13:04:57.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:57 smithi052 ceph-mon[36682]: 5.a scrub starts 2024-09-13T13:04:57.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:57 smithi052 ceph-mon[36682]: 5.a scrub ok 2024-09-13T13:04:57.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:57 smithi052 ceph-mon[36682]: osdmap e140: 8 total, 8 up, 8 in 2024-09-13T13:04:57.519 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:57.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:57.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:57.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:57 smithi167 ceph-mon[33178]: pgmap v172: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 868 B/s rd, 0 B/s wr, 560 op/s; 2681/215721 objects degraded (1.243%); 8.6 MiB/s, 149 objects/s recovering 2024-09-13T13:04:57.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:57 smithi167 ceph-mon[33178]: 4.1a scrub starts 2024-09-13T13:04:57.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:57 smithi167 ceph-mon[33178]: 4.1a scrub ok 2024-09-13T13:04:57.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:57 smithi167 ceph-mon[33178]: 5.a scrub starts 2024-09-13T13:04:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:57 smithi167 ceph-mon[33178]: 5.a scrub ok 2024-09-13T13:04:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:57 smithi167 ceph-mon[33178]: osdmap e140: 8 total, 8 up, 8 in 2024-09-13T13:04:57.621 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:57.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:57.652 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:57.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:57.753 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:57.779 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:57.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:57.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:57.881 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:57.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:57.921 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:57.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:58.022 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:58.058 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:58.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:58.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:58.161 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:58.193 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:58.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:58.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:58.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:58.318 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:58.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:58.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:58.420 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:04:58.447 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:58.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:58.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:04:58.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:58 smithi052 ceph-mon[36682]: 2.0 scrub starts 2024-09-13T13:04:58.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:58 smithi052 ceph-mon[36682]: 2.0 scrub ok 2024-09-13T13:04:58.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:58 smithi052 ceph-mon[36682]: 3.1a scrub starts 2024-09-13T13:04:58.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:58 smithi052 ceph-mon[36682]: 3.1a scrub ok 2024-09-13T13:04:58.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:58 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 2681/215721 objects degraded (1.243%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:58.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:58 smithi052 ceph-mon[29471]: 2.0 scrub starts 2024-09-13T13:04:58.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:58 smithi052 ceph-mon[29471]: 2.0 scrub ok 2024-09-13T13:04:58.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:58 smithi052 ceph-mon[29471]: 3.1a scrub starts 2024-09-13T13:04:58.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:58 smithi052 ceph-mon[29471]: 3.1a scrub ok 2024-09-13T13:04:58.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:58 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 2681/215721 objects degraded (1.243%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:58.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:04:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:58 smithi167 ceph-mon[33178]: 2.0 scrub starts 2024-09-13T13:04:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:58 smithi167 ceph-mon[33178]: 2.0 scrub ok 2024-09-13T13:04:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:58 smithi167 ceph-mon[33178]: 3.1a scrub starts 2024-09-13T13:04:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:58 smithi167 ceph-mon[33178]: 3.1a scrub ok 2024-09-13T13:04:58.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:58 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 2681/215721 objects degraded (1.243%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T13:04:58.573 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:58.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:58.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:04:58.675 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:04:58.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:58.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:58.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:04:58.803 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:58.847 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:58.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:58.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:58.949 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:04:58.978 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:58.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:58.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:04:59.080 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:59.112 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:59.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:59.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:59.213 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:59.250 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:59.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:59.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:04:59.352 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:04:59.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:59.386 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:04:59.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:04:59.488 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:59.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:59 smithi052 ceph-mon[36682]: 6.3 scrub starts 2024-09-13T13:04:59.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:59 smithi052 ceph-mon[36682]: 6.3 scrub ok 2024-09-13T13:04:59.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:04:59 smithi052 ceph-mon[36682]: pgmap v174: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 920 B/s rd, 0 B/s wr, 625 op/s; 2681/214914 objects degraded (1.247%); 9.1 MiB/s, 146 objects/s recovering 2024-09-13T13:04:59.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:59 smithi052 ceph-mon[29471]: 6.3 scrub starts 2024-09-13T13:04:59.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:59 smithi052 ceph-mon[29471]: 6.3 scrub ok 2024-09-13T13:04:59.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:04:59 smithi052 ceph-mon[29471]: pgmap v174: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 920 B/s rd, 0 B/s wr, 625 op/s; 2681/214914 objects degraded (1.247%); 9.1 MiB/s, 146 objects/s recovering 2024-09-13T13:04:59.530 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:59.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:59 smithi167 ceph-mon[33178]: 6.3 scrub starts 2024-09-13T13:04:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:59 smithi167 ceph-mon[33178]: 6.3 scrub ok 2024-09-13T13:04:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:04:59 smithi167 ceph-mon[33178]: pgmap v174: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 920 B/s rd, 0 B/s wr, 625 op/s; 2681/214914 objects degraded (1.247%); 9.1 MiB/s, 146 objects/s recovering 2024-09-13T13:04:59.632 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:04:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:59.671 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:59.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:04:59.772 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:04:59.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:59.815 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:59.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:04:59.916 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:04:59.945 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:04:59.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:04:59.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:00.047 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:00.077 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:00.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:00.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:00.203 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:00.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:00.305 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:00.328 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:00.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:00.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:00.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:00.455 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:00.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:00.459 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:00 smithi052 ceph-mon[36682]: 6.12 scrub starts 2024-09-13T13:05:00.459 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:00 smithi052 ceph-mon[36682]: 6.12 scrub ok 2024-09-13T13:05:00.460 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:00 smithi052 ceph-mon[36682]: 4.c scrub starts 2024-09-13T13:05:00.460 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:00 smithi052 ceph-mon[36682]: 4.c scrub ok 2024-09-13T13:05:00.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:00 smithi052 ceph-mon[29471]: 6.12 scrub starts 2024-09-13T13:05:00.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:00 smithi052 ceph-mon[29471]: 6.12 scrub ok 2024-09-13T13:05:00.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:00 smithi052 ceph-mon[29471]: 4.c scrub starts 2024-09-13T13:05:00.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:00 smithi052 ceph-mon[29471]: 4.c scrub ok 2024-09-13T13:05:00.471 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:00 smithi167 ceph-mon[33178]: 6.12 scrub starts 2024-09-13T13:05:00.471 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:00 smithi167 ceph-mon[33178]: 6.12 scrub ok 2024-09-13T13:05:00.471 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:00 smithi167 ceph-mon[33178]: 4.c scrub starts 2024-09-13T13:05:00.471 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:00 smithi167 ceph-mon[33178]: 4.c scrub ok 2024-09-13T13:05:00.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:00.587 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:00.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:00.688 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:00.734 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:00.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:00.836 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:00.878 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:00.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:00.980 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:01.033 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:01.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:01.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:01.134 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:01.174 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:01.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:01.275 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:01.317 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:01.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:01.412 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:01 smithi052 ceph-mon[36682]: pgmap v175: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.07k op/s; 2259/203142 objects degraded (1.112%); 11 MiB/s, 177 objects/s recovering 2024-09-13T13:05:01.412 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:01 smithi052 ceph-mon[36682]: 2.2 scrub starts 2024-09-13T13:05:01.412 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:01 smithi052 ceph-mon[36682]: 2.2 scrub ok 2024-09-13T13:05:01.413 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:01 smithi052 ceph-mon[29471]: pgmap v175: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.07k op/s; 2259/203142 objects degraded (1.112%); 11 MiB/s, 177 objects/s recovering 2024-09-13T13:05:01.413 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:01 smithi052 ceph-mon[29471]: 2.2 scrub starts 2024-09-13T13:05:01.413 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:01 smithi052 ceph-mon[29471]: 2.2 scrub ok 2024-09-13T13:05:01.420 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:01.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:01.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:01.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:01.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:01.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:01 smithi167 ceph-mon[33178]: pgmap v175: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.07k op/s; 2259/203142 objects degraded (1.112%); 11 MiB/s, 177 objects/s recovering 2024-09-13T13:05:01.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:01 smithi167 ceph-mon[33178]: 2.2 scrub starts 2024-09-13T13:05:01.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:01 smithi167 ceph-mon[33178]: 2.2 scrub ok 2024-09-13T13:05:01.574 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:01.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:01.677 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:01.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:01.713 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:01.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:01.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:01.847 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:01.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:01.948 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:01.996 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:01.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:02.098 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:02.148 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:02.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:02.249 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:02.272 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:02.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:02.374 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:02.414 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:02.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[36682]: 6.1c scrub starts 2024-09-13T13:05:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[36682]: 6.1c scrub ok 2024-09-13T13:05:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[36682]: 3.1d scrub starts 2024-09-13T13:05:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[36682]: 3.1d scrub ok 2024-09-13T13:05:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[36682]: 6.1a scrub starts 2024-09-13T13:05:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[36682]: 6.1a scrub ok 2024-09-13T13:05:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[36682]: 7.e scrub starts 2024-09-13T13:05:02.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[29471]: 6.1c scrub starts 2024-09-13T13:05:02.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[29471]: 6.1c scrub ok 2024-09-13T13:05:02.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[29471]: 3.1d scrub starts 2024-09-13T13:05:02.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[29471]: 3.1d scrub ok 2024-09-13T13:05:02.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[29471]: 6.1a scrub starts 2024-09-13T13:05:02.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[29471]: 6.1a scrub ok 2024-09-13T13:05:02.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:02 smithi052 ceph-mon[29471]: 7.e scrub starts 2024-09-13T13:05:02.515 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:02.537 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:02.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:02.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:02 smithi167 ceph-mon[33178]: 6.1c scrub starts 2024-09-13T13:05:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:02 smithi167 ceph-mon[33178]: 6.1c scrub ok 2024-09-13T13:05:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:02 smithi167 ceph-mon[33178]: 3.1d scrub starts 2024-09-13T13:05:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:02 smithi167 ceph-mon[33178]: 3.1d scrub ok 2024-09-13T13:05:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:02 smithi167 ceph-mon[33178]: 6.1a scrub starts 2024-09-13T13:05:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:02 smithi167 ceph-mon[33178]: 6.1a scrub ok 2024-09-13T13:05:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:02 smithi167 ceph-mon[33178]: 7.e scrub starts 2024-09-13T13:05:02.638 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:02.668 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:02.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:02.770 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:02.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:02.797 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:02.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:02.898 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:02.924 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:02.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:02.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:03.026 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:03.049 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:03.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:03.151 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:03.188 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:03.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:03.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:03.290 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:03.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:03.336 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:03.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:03.437 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:03.486 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:03.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:03 smithi052 ceph-mon[36682]: 3.11 scrub starts 2024-09-13T13:05:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:03 smithi052 ceph-mon[36682]: 3.11 scrub ok 2024-09-13T13:05:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:03 smithi052 ceph-mon[36682]: pgmap v176: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 554 op/s; 2259/202428 objects degraded (1.116%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T13:05:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:03 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:05:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:03 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:05:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:03 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 2259/202428 objects degraded (1.116%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T13:05:03.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:03 smithi052 ceph-mon[29471]: 3.11 scrub starts 2024-09-13T13:05:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:03 smithi052 ceph-mon[29471]: 3.11 scrub ok 2024-09-13T13:05:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:03 smithi052 ceph-mon[29471]: pgmap v176: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 554 op/s; 2259/202428 objects degraded (1.116%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T13:05:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:03 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:05:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:03 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:05:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:03 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 2259/202428 objects degraded (1.116%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T13:05:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:03 smithi167 ceph-mon[33178]: 3.11 scrub starts 2024-09-13T13:05:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:03 smithi167 ceph-mon[33178]: 3.11 scrub ok 2024-09-13T13:05:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:03 smithi167 ceph-mon[33178]: pgmap v176: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 554 op/s; 2259/202428 objects degraded (1.116%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T13:05:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:03 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:05:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:03 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:05:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:03 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 2259/202428 objects degraded (1.116%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T13:05:03.588 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:03.610 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:03.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:03.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:03.712 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:03.738 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:03.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:03.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:03.861 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:03.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:03.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:03.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:03.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:03.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:04.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:05:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:05:04.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:04.126 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:04.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:04.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:04.254 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:04.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:04.356 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:04.397 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:04.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:04.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:04 smithi052 ceph-mon[36682]: 7.e scrub ok 2024-09-13T13:05:04.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:04 smithi052 ceph-mon[29471]: 7.e scrub ok 2024-09-13T13:05:04.500 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:04.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:04.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:04.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:04 smithi167 ceph-mon[33178]: 7.e scrub ok 2024-09-13T13:05:04.633 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:04.672 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:04.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:04.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:04.774 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:04.798 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:04.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:04.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:04.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:04.927 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:04.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:05.028 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:05.072 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:05.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:05.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:05.201 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:05.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:05.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:05.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:05.331 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:05.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:05.432 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:05.456 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:05.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:05.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:05.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:05 smithi052 ceph-mon[36682]: pgmap v177: 148 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 985 op/s; 1815/190620 objects degraded (0.952%); 6.4 MiB/s, 101 objects/s recovering 2024-09-13T13:05:05.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:05.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:05 smithi052 ceph-mon[36682]: 7.1 scrub starts 2024-09-13T13:05:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:05 smithi052 ceph-mon[29471]: pgmap v177: 148 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 985 op/s; 1815/190620 objects degraded (0.952%); 6.4 MiB/s, 101 objects/s recovering 2024-09-13T13:05:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:05 smithi052 ceph-mon[29471]: 7.1 scrub starts 2024-09-13T13:05:05.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:05 smithi167 ceph-mon[33178]: pgmap v177: 148 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 985 op/s; 1815/190620 objects degraded (0.952%); 6.4 MiB/s, 101 objects/s recovering 2024-09-13T13:05:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:05 smithi167 ceph-mon[33178]: 7.1 scrub starts 2024-09-13T13:05:05.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:05.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:05.684 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:05.724 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:05.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:05.825 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:05.862 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:05.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:05.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:05.964 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:06.002 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:06.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:06.104 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:06.146 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:06.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:06.247 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:06.274 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:06.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:06.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:06.376 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:06.409 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:06.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:06.463 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:06 smithi052 ceph-mon[36682]: 3.5 scrub starts 2024-09-13T13:05:06.464 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:06 smithi052 ceph-mon[36682]: 3.5 scrub ok 2024-09-13T13:05:06.464 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:06 smithi052 ceph-mon[36682]: 7.1 scrub ok 2024-09-13T13:05:06.464 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:05:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:05:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:05:06.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:06 smithi052 ceph-mon[29471]: 3.5 scrub starts 2024-09-13T13:05:06.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:06 smithi052 ceph-mon[29471]: 3.5 scrub ok 2024-09-13T13:05:06.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:06 smithi052 ceph-mon[29471]: 7.1 scrub ok 2024-09-13T13:05:06.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:06.542 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:06.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:06.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:06 smithi167 ceph-mon[33178]: 3.5 scrub starts 2024-09-13T13:05:06.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:06 smithi167 ceph-mon[33178]: 3.5 scrub ok 2024-09-13T13:05:06.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:06 smithi167 ceph-mon[33178]: 7.1 scrub ok 2024-09-13T13:05:06.644 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:06.688 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:06.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:06.789 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:06.834 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:06.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:06.936 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:06.988 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:06.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:06.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:07.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:05:07.090 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:07.133 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:07.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:07.235 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:07.278 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:07.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:07.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:07 smithi167 ceph-mon[33178]: pgmap v178: 148 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 837 op/s; 1815/190620 objects degraded (0.952%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T13:05:07.357 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:05:07.380 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:07.418 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:07.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:07.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:07 smithi052 ceph-mon[29471]: pgmap v178: 148 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 837 op/s; 1815/190620 objects degraded (0.952%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T13:05:07.479 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:07 smithi052 ceph-mon[36682]: pgmap v178: 148 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 837 op/s; 1815/190620 objects degraded (0.952%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T13:05:07.520 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:07.521 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T13:05:07.548 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:07.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:07.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:07.649 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:07.676 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:07.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:07.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:07.781 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:07.833 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:07.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:07.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:07.935 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:07.988 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:07.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:07.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:08.010 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:08.090 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:08.121 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:08.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:08.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:08.224 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:08.269 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:08.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:08.313 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:08 smithi167 ceph-mon[33178]: 7.8 scrub starts 2024-09-13T13:05:08.313 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:08 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 1815/190620 objects degraded (0.952%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T13:05:08.344 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:05:08.344 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:05:08.371 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:08.427 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:08.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:08 smithi052 ceph-mon[36682]: 7.8 scrub starts 2024-09-13T13:05:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:08 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 1815/190620 objects degraded (0.952%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T13:05:08.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:08 smithi052 ceph-mon[29471]: 7.8 scrub starts 2024-09-13T13:05:08.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:08 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 1815/190620 objects degraded (0.952%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T13:05:08.517 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:05:08.528 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:08.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:08.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:08.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:08.680 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:08.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:08.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:08.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:08.806 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:08.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:08.880 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:05:08.908 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:09.002 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:09.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:09.008 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:09.104 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:09.167 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:09.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:09.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:09.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T13:05:09.268 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T13:05:09.269 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:09.316 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:09.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:09.418 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:09.449 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:09.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:09.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:09 smithi052 ceph-mon[36682]: pgmap v179: 148 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 891 B/s rd, 0 B/s wr, 727 op/s; 1815/189879 objects degraded (0.956%); 4.7 MiB/s, 75 objects/s recovering 2024-09-13T13:05:09.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:09 smithi052 ceph-mon[36682]: 5.10 scrub starts 2024-09-13T13:05:09.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:09 smithi052 ceph-mon[36682]: 5.10 scrub ok 2024-09-13T13:05:09.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:09 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/875176398' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:05:09.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:09 smithi052 ceph-mon[36682]: from='client.35012 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:05:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:09 smithi052 ceph-mon[29471]: pgmap v179: 148 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 891 B/s rd, 0 B/s wr, 727 op/s; 1815/189879 objects degraded (0.956%); 4.7 MiB/s, 75 objects/s recovering 2024-09-13T13:05:09.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:09 smithi052 ceph-mon[29471]: 5.10 scrub starts 2024-09-13T13:05:09.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:09 smithi052 ceph-mon[29471]: 5.10 scrub ok 2024-09-13T13:05:09.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:09 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/875176398' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:05:09.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:09 smithi052 ceph-mon[29471]: from='client.35012 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:05:09.550 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:09.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:09 smithi167 ceph-mon[33178]: pgmap v179: 148 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 891 B/s rd, 0 B/s wr, 727 op/s; 1815/189879 objects degraded (0.956%); 4.7 MiB/s, 75 objects/s recovering 2024-09-13T13:05:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:09 smithi167 ceph-mon[33178]: 5.10 scrub starts 2024-09-13T13:05:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:09 smithi167 ceph-mon[33178]: 5.10 scrub ok 2024-09-13T13:05:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:09 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/875176398' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:05:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:09 smithi167 ceph-mon[33178]: from='client.35012 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:05:09.585 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:05:09.591 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:09.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:09.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:09.695 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:09.733 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:09.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:09.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:09.838 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:09.867 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:09.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:09.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:09.969 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:09.996 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:09.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:10.098 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:10.136 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:10.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:10.193 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:05:10.194 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:05:10.203 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:05:10.204 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:05:10.237 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:10.253 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:05:10.253 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:05:10.288 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:05:10.289 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:05:10.295 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:10.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:10.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:10 smithi167 ceph-mon[33178]: 7.8 scrub ok 2024-09-13T13:05:10.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:10 smithi167 ceph-mon[33178]: from='client.35012 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:05:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:10 smithi167 ceph-mon[33178]: osdmap e141: 8 total, 8 up, 8 in 2024-09-13T13:05:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:10 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4199202652' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:10.324 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:05:10.324 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:05:10.362 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:05:10.363 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:05:10.398 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:10.399 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:05:10.399 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:05:10.428 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:10.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:10.430 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:05:10.430 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:05:10.468 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:05:10.468 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:05:10.530 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:10.572 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:10.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:10.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:10.674 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:10.714 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:10.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:10.730 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:10 smithi052 ceph-mon[36682]: 7.8 scrub ok 2024-09-13T13:05:10.730 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:10 smithi052 ceph-mon[36682]: from='client.35012 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:05:10.730 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:10 smithi052 ceph-mon[36682]: osdmap e141: 8 total, 8 up, 8 in 2024-09-13T13:05:10.730 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:10 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4199202652' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:10.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:10 smithi052 ceph-mon[29471]: 7.8 scrub ok 2024-09-13T13:05:10.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:10 smithi052 ceph-mon[29471]: from='client.35012 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:05:10.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:10 smithi052 ceph-mon[29471]: osdmap e141: 8 total, 8 up, 8 in 2024-09-13T13:05:10.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:10 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4199202652' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:10.815 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:10.856 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:10.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:10.909 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T13:05:10.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:10.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:10.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:10.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:10.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T13:05:11.093 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:11.137 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:11.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:11.238 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:11.291 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:11.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:11.300 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:11.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:11.353 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:05:11.401 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:11.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:11.448 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:11.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:11.549 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:11.567 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:11 smithi167 ceph-mon[33178]: pgmap v181: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 880 op/s; 1404/176757 objects degraded (0.794%); 5.3 MiB/s, 98 objects/s recovering 2024-09-13T13:05:11.567 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:11 smithi167 ceph-mon[33178]: 3.14 scrub starts 2024-09-13T13:05:11.567 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:11 smithi167 ceph-mon[33178]: 3.14 scrub ok 2024-09-13T13:05:11.567 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:11 smithi167 ceph-mon[33178]: 5.1a scrub starts 2024-09-13T13:05:11.567 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:11 smithi167 ceph-mon[33178]: 5.1a scrub ok 2024-09-13T13:05:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:11.593 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:11.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:11.616 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:11 smithi052 ceph-mon[36682]: pgmap v181: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 880 op/s; 1404/176757 objects degraded (0.794%); 5.3 MiB/s, 98 objects/s recovering 2024-09-13T13:05:11.616 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:11 smithi052 ceph-mon[36682]: 3.14 scrub starts 2024-09-13T13:05:11.616 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:11 smithi052 ceph-mon[36682]: 3.14 scrub ok 2024-09-13T13:05:11.616 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:11 smithi052 ceph-mon[36682]: 5.1a scrub starts 2024-09-13T13:05:11.616 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:11 smithi052 ceph-mon[36682]: 5.1a scrub ok 2024-09-13T13:05:11.616 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:11 smithi052 ceph-mon[29471]: pgmap v181: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 880 op/s; 1404/176757 objects degraded (0.794%); 5.3 MiB/s, 98 objects/s recovering 2024-09-13T13:05:11.616 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:11 smithi052 ceph-mon[29471]: 3.14 scrub starts 2024-09-13T13:05:11.617 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:11 smithi052 ceph-mon[29471]: 3.14 scrub ok 2024-09-13T13:05:11.617 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:11 smithi052 ceph-mon[29471]: 5.1a scrub starts 2024-09-13T13:05:11.617 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:11 smithi052 ceph-mon[29471]: 5.1a scrub ok 2024-09-13T13:05:11.693 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:11.726 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:11.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:11.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:11.827 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:11.854 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:11.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:11.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:11.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:11.986 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:11.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:11.990 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:12.008 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:05:12.008 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:05:12.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:05:12.009 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:05:12.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.009 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:22467 7.1 7:83ca2fe7:::benchmark_data_smithi052_71865_object22466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:05:12.009 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:31.281822+0000", 2024-09-13T13:05:12.009 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 160.72379765400001, 2024-09-13T13:05:12.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059630491000000001, 2024-09-13T13:05:12.009 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.009 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.009 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.009 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.009 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.010 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.010 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22467 2024-09-13T13:05:12.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.010 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281822+0000", 2024-09-13T13:05:12.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281822+0000", 2024-09-13T13:05:12.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281824+0000", 2024-09-13T13:05:12.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-06 2024-09-13T13:05:12.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281853+0000", 2024-09-13T13:05:12.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8864999999999999e-05 2024-09-13T13:05:12.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281882+0000", 2024-09-13T13:05:12.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9249000000000001e-05 2024-09-13T13:05:12.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281890+0000", 2024-09-13T13:05:12.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4320000000000002e-06 2024-09-13T13:05:12.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281939+0000", 2024-09-13T13:05:12.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9039000000000003e-05 2024-09-13T13:05:12.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.282052+0000", 2024-09-13T13:05:12.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113272 2024-09-13T13:05:12.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:05:12.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.282179+0000", 2024-09-13T13:05:12.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012745500000000001 2024-09-13T13:05:12.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.283337+0000", 2024-09-13T13:05:12.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011573530000000001 2024-09-13T13:05:12.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.285171+0000", 2024-09-13T13:05:12.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001834727 2024-09-13T13:05:12.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.341389+0000", 2024-09-13T13:05:12.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056217894999999997 2024-09-13T13:05:12.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.341424+0000", 2024-09-13T13:05:12.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5172999999999997e-05 2024-09-13T13:05:12.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.341453+0000", 2024-09-13T13:05:12.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8441000000000001e-05 2024-09-13T13:05:12.019 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.019 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.019 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.020 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:28936 7.1 7:8cc6c498:::benchmark_data_smithi052_71865_object28935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-13T13:05:12.020 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:34.434379+0000", 2024-09-13T13:05:12.020 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 157.57124102399999, 2024-09-13T13:05:12.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061581934999999997, 2024-09-13T13:05:12.021 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.021 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.021 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.021 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.021 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.021 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.021 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 28936 2024-09-13T13:05:12.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.021 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434379+0000", 2024-09-13T13:05:12.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434379+0000", 2024-09-13T13:05:12.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434381+0000", 2024-09-13T13:05:12.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.933e-06 2024-09-13T13:05:12.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434412+0000", 2024-09-13T13:05:12.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0580999999999997e-05 2024-09-13T13:05:12.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434427+0000", 2024-09-13T13:05:12.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5919000000000001e-05 2024-09-13T13:05:12.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434435+0000", 2024-09-13T13:05:12.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5599999999999996e-06 2024-09-13T13:05:12.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434513+0000", 2024-09-13T13:05:12.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7610999999999999e-05 2024-09-13T13:05:12.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434638+0000", 2024-09-13T13:05:12.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000125683 2024-09-13T13:05:12.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434781+0000", 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014221000000000001 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.437744+0000", 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029634409999999998 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.437765+0000", 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0554999999999999e-05 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.495871+0000", 2024-09-13T13:05:12.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058106564999999999 2024-09-13T13:05:12.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.495912+0000", 2024-09-13T13:05:12.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1106999999999998e-05 2024-09-13T13:05:12.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.495961+0000", 2024-09-13T13:05:12.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8770000000000002e-05 2024-09-13T13:05:12.028 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.028 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.028 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.028 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:28940 7.1 7:8444ec31:::benchmark_data_smithi052_71865_object28939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-13T13:05:12.028 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:34.436157+0000", 2024-09-13T13:05:12.028 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 157.569462832, 2024-09-13T13:05:12.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062002834, 2024-09-13T13:05:12.028 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.028 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.029 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.029 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.029 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.029 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.029 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 28940 2024-09-13T13:05:12.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.029 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436157+0000", 2024-09-13T13:05:12.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436157+0000", 2024-09-13T13:05:12.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436159+0000", 2024-09-13T13:05:12.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7880000000000001e-06 2024-09-13T13:05:12.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436240+0000", 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0631999999999996e-05 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436250+0000", 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0227000000000001e-05 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436257+0000", 2024-09-13T13:05:12.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.17e-06 2024-09-13T13:05:12.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436743+0000", 2024-09-13T13:05:12.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00048641599999999999 2024-09-13T13:05:12.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436875+0000", 2024-09-13T13:05:12.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013162800000000001 2024-09-13T13:05:12.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:05:12.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.437006+0000", 2024-09-13T13:05:12.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013089500000000001 2024-09-13T13:05:12.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.437829+0000", 2024-09-13T13:05:12.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00082255999999999996 2024-09-13T13:05:12.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.438239+0000", 2024-09-13T13:05:12.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000410331 2024-09-13T13:05:12.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.498083+0000", 2024-09-13T13:05:12.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059844616000000003 2024-09-13T13:05:12.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.498118+0000", 2024-09-13T13:05:12.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4981000000000003e-05 2024-09-13T13:05:12.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.498160+0000", 2024-09-13T13:05:12.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.159e-05 2024-09-13T13:05:12.036 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.036 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.037 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.037 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:83969 7.1 7:819e28b6:::benchmark_data_smithi052_71865_object83968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:05:12.037 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:14.547136+0000", 2024-09-13T13:05:12.037 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 117.458483902, 2024-09-13T13:05:12.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0019013699999999, 2024-09-13T13:05:12.037 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.037 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.037 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.037 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.037 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.037 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.038 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 83969 2024-09-13T13:05:12.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.038 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547136+0000", 2024-09-13T13:05:12.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547136+0000", 2024-09-13T13:05:12.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547138+0000", 2024-09-13T13:05:12.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2859999999999998e-06 2024-09-13T13:05:12.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547171+0000", 2024-09-13T13:05:12.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2586000000000003e-05 2024-09-13T13:05:12.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547182+0000", 2024-09-13T13:05:12.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1336e-05 2024-09-13T13:05:12.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547191+0000", 2024-09-13T13:05:12.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.67e-06 2024-09-13T13:05:12.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547351+0000", 2024-09-13T13:05:12.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000160032 2024-09-13T13:05:12.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:05:12.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547364+0000", 2024-09-13T13:05:12.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3118e-05 2024-09-13T13:05:12.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.545177+0000", 2024-09-13T13:05:12.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99781304699999995 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.545295+0000", 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118134 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.545394+0000", 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8840999999999998e-05 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.547566+0000", 2024-09-13T13:05:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021715749999999998 2024-09-13T13:05:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.548948+0000", 2024-09-13T13:05:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013822719999999999 2024-09-13T13:05:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.548984+0000", 2024-09-13T13:05:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5725000000000001e-05 2024-09-13T13:05:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.549038+0000", 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3748000000000001e-05 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14493.0:1218 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94064716857216 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:15.061508+0000", 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 116.94411188799999, 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.48102383999999998, 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14493", 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1218 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061508+0000", 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061508+0000", 2024-09-13T13:05:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061511+0000", 2024-09-13T13:05:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3750000000000001e-06 2024-09-13T13:05:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061517+0000", 2024-09-13T13:05:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1999999999999999e-06 2024-09-13T13:05:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061528+0000", 2024-09-13T13:05:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1642e-05 2024-09-13T13:05:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061537+0000", 2024-09-13T13:05:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7110000000000001e-06 2024-09-13T13:05:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061610+0000", 2024-09-13T13:05:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3126000000000004e-05 2024-09-13T13:05:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:05:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061621+0000", 2024-09-13T13:05:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1112e-05 2024-09-13T13:05:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.537588+0000", 2024-09-13T13:05:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.47596680699999999 2024-09-13T13:05:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.537736+0000", 2024-09-13T13:05:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147887 2024-09-13T13:05:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.542532+0000", 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00479598 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14493.0:1219 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94064721783040 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:15.061557+0000", 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 116.94406255, 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.481215114, 2024-09-13T13:05:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:05:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14493", 2024-09-13T13:05:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:05:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1219 2024-09-13T13:05:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061557+0000", 2024-09-13T13:05:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061557+0000", 2024-09-13T13:05:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061558+0000", 2024-09-13T13:05:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3499999999999996e-07 2024-09-13T13:05:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061561+0000", 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7240000000000001e-06 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061567+0000", 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4459999999999998e-06 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061572+0000", 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7029999999999997e-06 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061623+0000", 2024-09-13T13:05:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1078999999999997e-05 2024-09-13T13:05:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:05:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061629+0000", 2024-09-13T13:05:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4600000000000002e-06 2024-09-13T13:05:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.539806+0000", 2024-09-13T13:05:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.47817701200000001 2024-09-13T13:05:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.539938+0000", 2024-09-13T13:05:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000132757 2024-09-13T13:05:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.542773+0000", 2024-09-13T13:05:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028342979999999999 2024-09-13T13:05:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:95510 7.1 7:8bdd1589:::benchmark_data_smithi052_71865_object95509:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:05:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:26.176570+0000", 2024-09-13T13:05:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 105.82905049199999, 2024-09-13T13:05:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061999859999999997, 2024-09-13T13:05:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95510 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176570+0000", 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176570+0000", 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176574+0000", 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3680000000000004e-06 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176643+0000", 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8658999999999994e-05 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176657+0000", 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4099e-05 2024-09-13T13:05:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176664+0000", 2024-09-13T13:05:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6720000000000004e-06 2024-09-13T13:05:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176716+0000", 2024-09-13T13:05:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1749999999999997e-05 2024-09-13T13:05:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176857+0000", 2024-09-13T13:05:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000141001 2024-09-13T13:05:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:05:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.177015+0000", 2024-09-13T13:05:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015754399999999999 2024-09-13T13:05:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.179003+0000", 2024-09-13T13:05:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019881690000000001 2024-09-13T13:05:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.238473+0000", 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059470058999999999 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.238511+0000", 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8179999999999997e-05 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.238569+0000", 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8359000000000001e-05 2024-09-13T13:05:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:95515 7.1 7:8f0a2bc8:::benchmark_data_smithi052_71865_object95514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:05:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:26.180498+0000", 2024-09-13T13:05:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 105.82512178499999, 2024-09-13T13:05:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060428145000000003, 2024-09-13T13:05:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95515 2024-09-13T13:05:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180498+0000", 2024-09-13T13:05:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180498+0000", 2024-09-13T13:05:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180500+0000", 2024-09-13T13:05:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0679999999999999e-06 2024-09-13T13:05:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180554+0000", 2024-09-13T13:05:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3749000000000003e-05 2024-09-13T13:05:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180565+0000", 2024-09-13T13:05:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0614e-05 2024-09-13T13:05:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180571+0000", 2024-09-13T13:05:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9490000000000001e-06 2024-09-13T13:05:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180618+0000", 2024-09-13T13:05:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7533e-05 2024-09-13T13:05:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180744+0000", 2024-09-13T13:05:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012591499999999999 2024-09-13T13:05:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:05:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180906+0000", 2024-09-13T13:05:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000162292 2024-09-13T13:05:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.182700+0000", 2024-09-13T13:05:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001793843 2024-09-13T13:05:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.240842+0000", 2024-09-13T13:05:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058141512999999999 2024-09-13T13:05:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.240877+0000", 2024-09-13T13:05:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5500000000000002e-05 2024-09-13T13:05:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.240926+0000", 2024-09-13T13:05:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9169000000000003e-05 2024-09-13T13:05:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:96745 7.1 7:8b8e8d5f:::benchmark_data_smithi052_71865_object96744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:05:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:27.480362+0000", 2024-09-13T13:05:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.52525794, 2024-09-13T13:05:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.068948425999999993, 2024-09-13T13:05:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 96745 2024-09-13T13:05:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480362+0000", 2024-09-13T13:05:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480362+0000", 2024-09-13T13:05:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480364+0000", 2024-09-13T13:05:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1500000000000002e-06 2024-09-13T13:05:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480419+0000", 2024-09-13T13:05:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4797999999999999e-05 2024-09-13T13:05:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480433+0000", 2024-09-13T13:05:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4304999999999999e-05 2024-09-13T13:05:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480441+0000", 2024-09-13T13:05:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1799999999999999e-06 2024-09-13T13:05:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480495+0000", 2024-09-13T13:05:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4583000000000002e-05 2024-09-13T13:05:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480636+0000", 2024-09-13T13:05:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014137899999999999 2024-09-13T13:05:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:05:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480799+0000", 2024-09-13T13:05:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016208999999999999 2024-09-13T13:05:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.483165+0000", 2024-09-13T13:05:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023665209999999999 2024-09-13T13:05:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.549136+0000", 2024-09-13T13:05:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065970735000000003 2024-09-13T13:05:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.549219+0000", 2024-09-13T13:05:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2917000000000006e-05 2024-09-13T13:05:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.549311+0000", 2024-09-13T13:05:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1768000000000006e-05 2024-09-13T13:05:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:105215 7.1 7:8cc92998:::benchmark_data_smithi052_71865_object105214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-13T13:05:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:36.018426+0000", 2024-09-13T13:05:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 95.987194431000006, 2024-09-13T13:05:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069694252999999998, 2024-09-13T13:05:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105215 2024-09-13T13:05:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018426+0000", 2024-09-13T13:05:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018426+0000", 2024-09-13T13:05:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018428+0000", 2024-09-13T13:05:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8119999999999999e-06 2024-09-13T13:05:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018479+0000", 2024-09-13T13:05:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0139999999999998e-05 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018493+0000", 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4552999999999999e-05 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018500+0000", 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3520000000000001e-06 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018559+0000", 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8491e-05 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018707+0000", 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147646 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018875+0000", 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168195 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.021096+0000", 2024-09-13T13:05:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002221425 2024-09-13T13:05:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.087977+0000", 2024-09-13T13:05:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066880870999999995 2024-09-13T13:05:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.088030+0000", 2024-09-13T13:05:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.287e-05 2024-09-13T13:05:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.088120+0000", 2024-09-13T13:05:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9897999999999999e-05 2024-09-13T13:05:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:105222 7.1 7:8c89bd7c:::benchmark_data_smithi052_71865_object105221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-13T13:05:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:36.023584+0000", 2024-09-13T13:05:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 95.982035719999999, 2024-09-13T13:05:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064762324999999996, 2024-09-13T13:05:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105222 2024-09-13T13:05:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023584+0000", 2024-09-13T13:05:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023584+0000", 2024-09-13T13:05:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023586+0000", 2024-09-13T13:05:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.159e-06 2024-09-13T13:05:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023649+0000", 2024-09-13T13:05:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2304000000000005e-05 2024-09-13T13:05:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023662+0000", 2024-09-13T13:05:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3407000000000001e-05 2024-09-13T13:05:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023670+0000", 2024-09-13T13:05:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:05:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023737+0000", 2024-09-13T13:05:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6997999999999998e-05 2024-09-13T13:05:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023891+0000", 2024-09-13T13:05:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015383399999999999 2024-09-13T13:05:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:05:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.024046+0000", 2024-09-13T13:05:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015552099999999999 2024-09-13T13:05:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.026536+0000", 2024-09-13T13:05:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024894330000000001 2024-09-13T13:05:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.088176+0000", 2024-09-13T13:05:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061639988 2024-09-13T13:05:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.088237+0000", 2024-09-13T13:05:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1500000000000004e-05 2024-09-13T13:05:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.088347+0000", 2024-09-13T13:05:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109422 2024-09-13T13:05:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:118706 7.1 7:8cfc8949:::benchmark_data_smithi052_71865_object118705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:05:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:49.554250+0000", 2024-09-13T13:05:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 82.451369827999997, 2024-09-13T13:05:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067355788999999999, 2024-09-13T13:05:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 118706 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554250+0000", 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554250+0000", 2024-09-13T13:05:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554252+0000", 2024-09-13T13:05:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4300000000000001e-06 2024-09-13T13:05:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554309+0000", 2024-09-13T13:05:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6857999999999997e-05 2024-09-13T13:05:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554318+0000", 2024-09-13T13:05:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9380000000000004e-06 2024-09-13T13:05:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554323+0000", 2024-09-13T13:05:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:05:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554369+0000", 2024-09-13T13:05:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5250999999999998e-05 2024-09-13T13:05:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554503+0000", 2024-09-13T13:05:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134096 2024-09-13T13:05:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:05:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554662+0000", 2024-09-13T13:05:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000159674 2024-09-13T13:05:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.556745+0000", 2024-09-13T13:05:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020823880000000001 2024-09-13T13:05:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.621459+0000", 2024-09-13T13:05:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064714516999999999 2024-09-13T13:05:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.621509+0000", 2024-09-13T13:05:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9579000000000001e-05 2024-09-13T13:05:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.621606+0000", 2024-09-13T13:05:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7139999999999995e-05 2024-09-13T13:05:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:118712 7.1 7:89036983:::benchmark_data_smithi052_71865_object118711:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:05:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:49.557026+0000", 2024-09-13T13:05:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 82.448593603999996, 2024-09-13T13:05:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064706738, 2024-09-13T13:05:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 118712 2024-09-13T13:05:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557026+0000", 2024-09-13T13:05:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557026+0000", 2024-09-13T13:05:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557034+0000", 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4270000000000002e-06 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557081+0000", 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7058000000000002e-05 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557090+0000", 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.585e-06 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557095+0000", 2024-09-13T13:05:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4870000000000002e-06 2024-09-13T13:05:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557130+0000", 2024-09-13T13:05:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5389999999999998e-05 2024-09-13T13:05:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557220+0000", 2024-09-13T13:05:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.975e-05 2024-09-13T13:05:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557310+0000", 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0183e-05 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.559010+0000", 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016999669999999999 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.621649+0000", 2024-09-13T13:05:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062639009999999995 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.621684+0000", 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4981999999999998e-05 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.621733+0000", 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8899000000000001e-05 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:119492 7.1 7:88823659:::benchmark_data_smithi052_71865_object119491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:50.481839+0000", 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.523781475999996, 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077838544999999995, 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 119492 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.481839+0000", 2024-09-13T13:05:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.481839+0000", 2024-09-13T13:05:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.481841+0000", 2024-09-13T13:05:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2929999999999999e-06 2024-09-13T13:05:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.481936+0000", 2024-09-13T13:05:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5172000000000003e-05 2024-09-13T13:05:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.481947+0000", 2024-09-13T13:05:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0739e-05 2024-09-13T13:05:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.481953+0000", 2024-09-13T13:05:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5289999999999997e-06 2024-09-13T13:05:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.482005+0000", 2024-09-13T13:05:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1928000000000002e-05 2024-09-13T13:05:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.482123+0000", 2024-09-13T13:05:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011812899999999999 2024-09-13T13:05:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:05:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.482263+0000", 2024-09-13T13:05:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013927899999999999 2024-09-13T13:05:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.484328+0000", 2024-09-13T13:05:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002065356 2024-09-13T13:05:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.559557+0000", 2024-09-13T13:05:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075229375000000001 2024-09-13T13:05:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.559616+0000", 2024-09-13T13:05:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8764000000000002e-05 2024-09-13T13:05:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.559677+0000", 2024-09-13T13:05:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0980999999999998e-05 2024-09-13T13:05:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:120921 7.1 7:8dbade5d:::benchmark_data_smithi052_71865_object120920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:05:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:51.922426+0000", 2024-09-13T13:05:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.083194337999998, 2024-09-13T13:05:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077277414000000003, 2024-09-13T13:05:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 120921 2024-09-13T13:05:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922426+0000", 2024-09-13T13:05:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922426+0000", 2024-09-13T13:05:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922428+0000", 2024-09-13T13:05:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4159999999999998e-06 2024-09-13T13:05:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922470+0000", 2024-09-13T13:05:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2299000000000003e-05 2024-09-13T13:05:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922487+0000", 2024-09-13T13:05:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6701000000000002e-05 2024-09-13T13:05:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922495+0000", 2024-09-13T13:05:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9659999999999996e-06 2024-09-13T13:05:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922552+0000", 2024-09-13T13:05:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6685999999999999e-05 2024-09-13T13:05:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922693+0000", 2024-09-13T13:05:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014088800000000001 2024-09-13T13:05:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:05:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922871+0000", 2024-09-13T13:05:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017871099999999999 2024-09-13T13:05:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.924891+0000", 2024-09-13T13:05:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020191860000000001 2024-09-13T13:05:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.999585+0000", 2024-09-13T13:05:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.074694889 2024-09-13T13:05:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.999645+0000", 2024-09-13T13:05:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9982999999999998e-05 2024-09-13T13:05:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.999703+0000", 2024-09-13T13:05:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7689000000000003e-05 2024-09-13T13:05:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:136555 7.1 7:8fedf1e6:::benchmark_data_smithi052_71865_object15454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:05:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:10.092295+0000", 2024-09-13T13:05:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.913324975000002, 2024-09-13T13:05:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060400885000000001, 2024-09-13T13:05:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 136555 2024-09-13T13:05:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092295+0000", 2024-09-13T13:05:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092295+0000", 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092298+0000", 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6180000000000002e-06 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092304+0000", 2024-09-13T13:05:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1290000000000001e-06 2024-09-13T13:05:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092319+0000", 2024-09-13T13:05:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4902999999999999e-05 2024-09-13T13:05:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092327+0000", 2024-09-13T13:05:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9049999999999997e-06 2024-09-13T13:05:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.096194+0000", 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038671959999999998 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.096319+0000", 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124831 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.096437+0000", 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011856100000000001 2024-09-13T13:05:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.100488+0000", 2024-09-13T13:05:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040505029999999996 2024-09-13T13:05:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.105447+0000", 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049591130000000002 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.151544+0000", 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046097325000000001 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.151607+0000", 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2539999999999994e-05 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.152696+0000", 2024-09-13T13:05:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001089261 2024-09-13T13:05:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:136559 7.1 7:87a02730:::benchmark_data_smithi052_71865_object15458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:05:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:10.092736+0000", 2024-09-13T13:05:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.912883616999999, 2024-09-13T13:05:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061695030999999997, 2024-09-13T13:05:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 136559 2024-09-13T13:05:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092736+0000", 2024-09-13T13:05:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092736+0000", 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092738+0000", 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.195e-06 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092745+0000", 2024-09-13T13:05:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.712e-06 2024-09-13T13:05:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092755+0000", 2024-09-13T13:05:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4360000000000008e-06 2024-09-13T13:05:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092762+0000", 2024-09-13T13:05:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7479999999999996e-06 2024-09-13T13:05:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.098351+0000", 2024-09-13T13:05:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055893660000000001 2024-09-13T13:05:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.098435+0000", 2024-09-13T13:05:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3751000000000005e-05 2024-09-13T13:05:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:05:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.098516+0000", 2024-09-13T13:05:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1769999999999998e-05 2024-09-13T13:05:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.101682+0000", 2024-09-13T13:05:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031659079999999998 2024-09-13T13:05:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.105502+0000", 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038194679999999999 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.153497+0000", 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047995011999999997 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.153552+0000", 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5538999999999998e-05 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.154431+0000", 2024-09-13T13:05:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00087912600000000004 2024-09-13T13:05:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:136928 7.1 7:86ee2aaa:::benchmark_data_smithi052_71865_object15827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:05:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:10.611023+0000", 2024-09-13T13:05:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.394596855000003, 2024-09-13T13:05:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059963874, 2024-09-13T13:05:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 136928 2024-09-13T13:05:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.611023+0000", 2024-09-13T13:05:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.611023+0000", 2024-09-13T13:05:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.611026+0000", 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.418e-06 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.611032+0000", 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8839999999999997e-06 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.611048+0000", 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5136999999999999e-05 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.611057+0000", 2024-09-13T13:05:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2289999999999992e-06 2024-09-13T13:05:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.613408+0000", 2024-09-13T13:05:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023514320000000001 2024-09-13T13:05:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.613505+0000", 2024-09-13T13:05:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6464000000000003e-05 2024-09-13T13:05:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:05:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.613594+0000", 2024-09-13T13:05:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9518000000000006e-05 2024-09-13T13:05:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.621119+0000", 2024-09-13T13:05:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075246599999999999 2024-09-13T13:05:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.626948+0000", 2024-09-13T13:05:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058287349999999998 2024-09-13T13:05:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.670506+0000", 2024-09-13T13:05:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043558056999999997 2024-09-13T13:05:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.670547+0000", 2024-09-13T13:05:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1267999999999999e-05 2024-09-13T13:05:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.670987+0000", 2024-09-13T13:05:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00044007200000000001 2024-09-13T13:05:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:136930 7.1 7:89d55d1f:::benchmark_data_smithi052_71865_object15829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:05:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:10.612698+0000", 2024-09-13T13:05:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.392922298999999, 2024-09-13T13:05:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062650329000000005, 2024-09-13T13:05:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.157 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 136930 2024-09-13T13:05:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.612698+0000", 2024-09-13T13:05:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.612698+0000", 2024-09-13T13:05:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.612700+0000", 2024-09-13T13:05:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0509999999999999e-06 2024-09-13T13:05:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.612707+0000", 2024-09-13T13:05:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5170000000000002e-06 2024-09-13T13:05:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.612722+0000", 2024-09-13T13:05:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4250000000000001e-05 2024-09-13T13:05:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.612730+0000", 2024-09-13T13:05:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2179999999999999e-06 2024-09-13T13:05:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.616173+0000", 2024-09-13T13:05:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034428250000000001 2024-09-13T13:05:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.616309+0000", 2024-09-13T13:05:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000136227 2024-09-13T13:05:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:05:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.616446+0000", 2024-09-13T13:05:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013688400000000001 2024-09-13T13:05:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.626874+0000", 2024-09-13T13:05:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010428203 2024-09-13T13:05:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.626994+0000", 2024-09-13T13:05:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120423 2024-09-13T13:05:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.674597+0000", 2024-09-13T13:05:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047603148999999997 2024-09-13T13:05:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.674657+0000", 2024-09-13T13:05:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9169000000000002e-05 2024-09-13T13:05:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.675348+0000", 2024-09-13T13:05:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00069141299999999999 2024-09-13T13:05:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:155460 7.1 7:83ed900b:::benchmark_data_smithi052_71865_object34359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T13:05:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:34.105592+0000", 2024-09-13T13:05:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.900027704999999, 2024-09-13T13:05:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075149564000000002, 2024-09-13T13:05:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:05:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 155460 2024-09-13T13:05:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105592+0000", 2024-09-13T13:05:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105592+0000", 2024-09-13T13:05:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105595+0000", 2024-09-13T13:05:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0759999999999999e-06 2024-09-13T13:05:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105608+0000", 2024-09-13T13:05:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2848e-05 2024-09-13T13:05:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105630+0000", 2024-09-13T13:05:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2146000000000001e-05 2024-09-13T13:05:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105640+0000", 2024-09-13T13:05:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6860000000000001e-06 2024-09-13T13:05:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.106344+0000", 2024-09-13T13:05:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070417500000000003 2024-09-13T13:05:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.106500+0000", 2024-09-13T13:05:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000155468 2024-09-13T13:05:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:05:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.106631+0000", 2024-09-13T13:05:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131137 2024-09-13T13:05:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.109754+0000", 2024-09-13T13:05:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031233480000000002 2024-09-13T13:05:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.110324+0000", 2024-09-13T13:05:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000569843 2024-09-13T13:05:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:05:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.180592+0000", 2024-09-13T13:05:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070267674000000002 2024-09-13T13:05:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:05:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.180647+0000", 2024-09-13T13:05:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5067e-05 2024-09-13T13:05:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:05:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.180742+0000", 2024-09-13T13:05:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5095999999999999e-05 2024-09-13T13:05:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:12.181 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:05:12.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:12.194 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:12.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:12.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:12.301 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:12 smithi052 ceph-mon[29471]: 6.8 scrub starts 2024-09-13T13:05:12.301 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:12 smithi052 ceph-mon[29471]: 6.8 scrub ok 2024-09-13T13:05:12.301 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:12 smithi052 ceph-mon[36682]: 6.8 scrub starts 2024-09-13T13:05:12.301 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:12 smithi052 ceph-mon[36682]: 6.8 scrub ok 2024-09-13T13:05:12.320 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:12.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:12.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:12.422 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:12.451 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:12.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:12.553 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:12.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:12 smithi167 ceph-mon[33178]: 6.8 scrub starts 2024-09-13T13:05:12.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:12 smithi167 ceph-mon[33178]: 6.8 scrub ok 2024-09-13T13:05:12.596 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:12.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:12.703 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:12.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:12.741 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:12.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:12.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:12.869 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:12.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:12.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T13:05:12.971 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:13.046 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:13.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:13.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:13.148 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:13.198 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:13.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:13.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:13.300 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:13.309 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T13:05:13.312 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:05:13.316 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:05:13.334 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:13.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:13.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:13.342 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:05:13.435 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:13.487 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:13.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:13 smithi167 ceph-mon[33178]: pgmap v182: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 883 op/s; 1404/175956 objects degraded (0.798%); 5.3 MiB/s, 98 objects/s recovering 2024-09-13T13:05:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:13 smithi167 ceph-mon[33178]: 4.1b scrub starts 2024-09-13T13:05:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:13 smithi167 ceph-mon[33178]: 4.1b scrub ok 2024-09-13T13:05:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/33163547' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:05:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:13 smithi167 ceph-mon[33178]: from='client.25036 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:05:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:13 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 1404/175956 objects degraded (0.798%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T13:05:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3950180404' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:05:13.583 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[36682]: pgmap v182: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 883 op/s; 1404/175956 objects degraded (0.798%); 5.3 MiB/s, 98 objects/s recovering 2024-09-13T13:05:13.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[36682]: 4.1b scrub starts 2024-09-13T13:05:13.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[36682]: 4.1b scrub ok 2024-09-13T13:05:13.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/33163547' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:05:13.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[36682]: from='client.25036 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:05:13.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 1404/175956 objects degraded (0.798%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T13:05:13.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3950180404' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:05:13.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[29471]: pgmap v182: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 145 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 883 op/s; 1404/175956 objects degraded (0.798%); 5.3 MiB/s, 98 objects/s recovering 2024-09-13T13:05:13.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[29471]: 4.1b scrub starts 2024-09-13T13:05:13.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[29471]: 4.1b scrub ok 2024-09-13T13:05:13.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/33163547' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:05:13.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[29471]: from='client.25036 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:05:13.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 1404/175956 objects degraded (0.798%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T13:05:13.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3950180404' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:05:13.589 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:13.623 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:13.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:13.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:13.726 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:13.770 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:13.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:13.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:13.872 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:13.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:05:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:05:13.907 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:05:13.907 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:05:13.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:13.907 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:188979 7.0 7.e8da5690 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-13T13:05:13.907 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:13.905379+0000", 2024-09-13T13:05:13.908 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.4208000000000003e-05, 2024-09-13T13:05:13.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012046599999999999, 2024-09-13T13:05:13.908 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:13.908 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:13.908 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:05:13.908 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:13.908 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:13.908 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:13.908 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 188979 2024-09-13T13:05:13.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:13.908 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:13.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:13.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:13.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:13.905379+0000", 2024-09-13T13:05:13.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:13.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:13.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:13.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:13.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:13.905379+0000", 2024-09-13T13:05:13.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:13.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:13.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:13.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:13.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:13.905382+0000", 2024-09-13T13:05:13.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4709999999999999e-06 2024-09-13T13:05:13.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:13.905391+0000", 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5690000000000005e-06 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:13.905407+0000", 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6308999999999999e-05 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:13.905417+0000", 2024-09-13T13:05:13.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8980000000000008e-06 2024-09-13T13:05:13.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:13.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:13.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:13.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:13.905490+0000", 2024-09-13T13:05:13.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2900999999999999e-05 2024-09-13T13:05:13.911 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:13.911 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:13.911 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:13.911 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:13.911 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:05:13.911 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-09-13T13:05:13.911 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:05:13.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:13.913 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:13.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:14.014 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:14.051 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:14.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:14.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:14.153 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:14.197 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:14.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:14.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:14.330 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:14.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:14.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:14.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:14.461 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:14.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:14.562 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:14.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:14 smithi167 ceph-mon[33178]: 3.16 scrub starts 2024-09-13T13:05:14.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:14 smithi167 ceph-mon[33178]: 3.16 scrub ok 2024-09-13T13:05:14.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:14 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:14.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:14 smithi167 ceph-mon[33178]: from='client.25036 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:05:14.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:14 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3950180404' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:05:14.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:14 smithi167 ceph-mon[33178]: osdmap e142: 8 total, 8 up, 8 in 2024-09-13T13:05:14.585 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:14.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:14.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:14.688 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:14.696 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T13:05:14.710 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:14.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:14.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:14.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:14 smithi052 ceph-mon[36682]: 3.16 scrub starts 2024-09-13T13:05:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:14 smithi052 ceph-mon[36682]: 3.16 scrub ok 2024-09-13T13:05:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:14 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:14 smithi052 ceph-mon[36682]: from='client.25036 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:05:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:14 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3950180404' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:05:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:14 smithi052 ceph-mon[36682]: osdmap e142: 8 total, 8 up, 8 in 2024-09-13T13:05:14.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:14 smithi052 ceph-mon[29471]: 3.16 scrub starts 2024-09-13T13:05:14.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:14 smithi052 ceph-mon[29471]: 3.16 scrub ok 2024-09-13T13:05:14.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:14 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:14.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:14 smithi052 ceph-mon[29471]: from='client.25036 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:05:14.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:14 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3950180404' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:05:14.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:14 smithi052 ceph-mon[29471]: osdmap e142: 8 total, 8 up, 8 in 2024-09-13T13:05:14.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:14.834 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:14.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:14.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:14.936 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:14.956 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:05:14.959 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:14.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:15.061 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:15.086 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:15.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:15.189 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:15.213 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:15.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:15.315 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:15.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:15.339 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:15.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:15.354 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:15 smithi167 ceph-mon[33178]: pgmap v184: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 1201/162765 objects degraded (0.738%); 4.8 MiB/s, 152 objects/s recovering 2024-09-13T13:05:15.355 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:15.355 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:15 smithi167 ceph-mon[33178]: osdmap e143: 8 total, 8 up, 8 in 2024-09-13T13:05:15.410 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T13:05:15.410 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:05:15.441 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:15.506 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:15.527 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:15.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:15.628 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:15.638 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:05:15.638 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:05:15.638 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:05:15.638 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:05:15.638 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:05:15.647 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:15 smithi052 ceph-mon[36682]: pgmap v184: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 1201/162765 objects degraded (0.738%); 4.8 MiB/s, 152 objects/s recovering 2024-09-13T13:05:15.647 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:15.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:15 smithi052 ceph-mon[36682]: osdmap e143: 8 total, 8 up, 8 in 2024-09-13T13:05:15.648 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:15 smithi052 ceph-mon[29471]: pgmap v184: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 1201/162765 objects degraded (0.738%); 4.8 MiB/s, 152 objects/s recovering 2024-09-13T13:05:15.648 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:15.648 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:15 smithi052 ceph-mon[29471]: osdmap e143: 8 total, 8 up, 8 in 2024-09-13T13:05:15.657 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:15.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:15.759 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:15.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:15.802 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:15.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:15.904 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:15.958 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:15.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:16.059 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:16.084 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:16.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:16.186 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:16.234 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:05:16.235 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:05:16.235 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:05:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:05:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:05:16.253 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:16.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:16.259 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:16.271 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:05:16.271 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:05:16.310 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:05:16.311 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:05:16.352 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:05:16.352 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:05:16.355 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:16.385 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:05:16.385 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:05:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:16.411 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:16.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:16.422 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:05:16.422 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:05:16.456 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:05:16.456 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:05:16.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T13:05:16.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:16 smithi052 ceph-mon[36682]: osdmap e144: 8 total, 8 up, 8 in 2024-09-13T13:05:16.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:16 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1176719400' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:16.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:16 smithi052 ceph-mon[29471]: osdmap e144: 8 total, 8 up, 8 in 2024-09-13T13:05:16.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:16 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1176719400' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:16.493 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:05:16.493 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:05:16.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:16.551 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:16.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:16.653 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:16.690 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:16.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:16.791 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:16.817 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:05:16.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:16 smithi167 ceph-mon[33178]: osdmap e144: 8 total, 8 up, 8 in 2024-09-13T13:05:16.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:16 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1176719400' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:16.826 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:16.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:16.927 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:16.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:16.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:16.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:17.058 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:17.099 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:05:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:17.101 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:17.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:17.202 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:17.271 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:17.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:17.372 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:17.401 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:17 smithi052 ceph-mon[36682]: pgmap v187: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 777 op/s; 1201/162765 objects degraded (0.738%); 2.1 MiB/s, 113 objects/s recovering 2024-09-13T13:05:17.401 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:17 smithi052 ceph-mon[29471]: pgmap v187: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 777 op/s; 1201/162765 objects degraded (0.738%); 2.1 MiB/s, 113 objects/s recovering 2024-09-13T13:05:17.429 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:05:17.430 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:05:17.432 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:17.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:17.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:17.534 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:17 smithi167 ceph-mon[33178]: pgmap v187: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 777 op/s; 1201/162765 objects degraded (0.738%); 2.1 MiB/s, 113 objects/s recovering 2024-09-13T13:05:17.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:17.589 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:17.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:17.691 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:17.697 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:17.750 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:17.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:17.851 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:17.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:17.879 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:17.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:17.943 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:05:17.946 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:05:17.981 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:18.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:18.028 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:18.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:18.121 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T13:05:18.129 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:18.193 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:18.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:18.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:18.295 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:18.337 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:18.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:18.438 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:18.476 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:05:18.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:18.477 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:18.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:18.578 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:18.622 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:18.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:18.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:18.724 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:18.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:18.766 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:18.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:18.867 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:18.897 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:18 smithi052 ceph-mon[36682]: pgmap v188: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 791 op/s; 1201/161718 objects degraded (0.743%); 2.1 MiB/s, 119 objects/s recovering 2024-09-13T13:05:18.898 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:18 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:05:18.898 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:18 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:05:18.898 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:18 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 1201/162765 objects degraded (0.738%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T13:05:18.898 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:18 smithi052 ceph-mon[36682]: from='client.15165 v1:172.21.15.52:0/3723603960' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:18.898 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:18 smithi052 ceph-mon[29471]: pgmap v188: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 791 op/s; 1201/161718 objects degraded (0.743%); 2.1 MiB/s, 119 objects/s recovering 2024-09-13T13:05:18.898 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:18 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:05:18.898 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:18 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:05:18.898 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:18 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 1201/162765 objects degraded (0.738%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T13:05:18.898 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:18 smithi052 ceph-mon[29471]: from='client.15165 v1:172.21.15.52:0/3723603960' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:18.909 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:18.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:18.951 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-13T13:05:17.663487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3520272947,"num_objects":53906,"num_object_clones":0,"num_object_copies":161718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53906,"num_whiteouts":0,"num_read":1092,"num_read_kb":949,"num_write":189040,"num_write_kb":7751946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12884,"num_bytes_recovered":677970598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89218,"ondisk_log_size":89218,"up":444,"acting":443,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11306724,"kb_used_data":10415012,"kb_used_omap":15,"kb_used_meta":891440,"kb_avail":738654492,"statfs":{"total":767960285184,"available":756382199808,"internally_reserved":0,"allocated":10664972288,"data_stored":10661392941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":912835353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,5,4,8,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-311033856,"num_objects":-4746,"num_object_clones":0,"num_object_copies":-14238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4746,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4746,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":13303808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002206"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":185,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.317974+0000","last_change":"2024-09-13T13:02:38.651382+0000","last_active":"2024-09-13T13:05:13.317974+0000","last_peered":"2024-09-13T13:05:13.317974+0000","last_clean":"2024-09-13T13:05:13.317974+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:05:13.317974+0000","last_undegraded":"2024-09-13T13:05:13.317974+0000","last_fullsized":"2024-09-13T13:05:13.317974+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:38.651314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:38.651314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":206,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262422+0000","last_change":"2024-09-13T13:04:48.899982+0000","last_active":"2024-09-13T13:05:09.262422+0000","last_peered":"2024-09-13T13:05:09.262422+0000","last_clean":"2024-09-13T13:05:09.262422+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:05:09.262422+0000","last_undegraded":"2024-09-13T13:05:09.262422+0000","last_fullsized":"2024-09-13T13:05:09.262422+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:48.899915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.899915+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:00.227319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.272430+0000","last_change":"2024-09-13T13:02:30.248974+0000","last_active":"2024-09-13T13:05:09.272430+0000","last_peered":"2024-09-13T13:05:09.272430+0000","last_clean":"2024-09-13T13:05:09.272430+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:05:09.272430+0000","last_undegraded":"2024-09-13T13:05:09.272430+0000","last_fullsized":"2024-09-13T13:05:09.272430+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:30.248905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:30.248905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":171,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262317+0000","last_change":"2024-09-13T13:05:01.076346+0000","last_active":"2024-09-13T13:05:09.262317+0000","last_peered":"2024-09-13T13:05:09.262317+0000","last_clean":"2024-09-13T13:05:09.262317+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:05:09.262317+0000","last_undegraded":"2024-09-13T13:05:09.262317+0000","last_fullsized":"2024-09-13T13:05:09.262317+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.076275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.076275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:14.180362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":173,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289183+0000","last_change":"2024-09-13T13:04:02.760609+0000","last_active":"2024-09-13T13:05:09.289183+0000","last_peered":"2024-09-13T13:05:09.289183+0000","last_clean":"2024-09-13T13:05:09.289183+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:05:09.289183+0000","last_undegraded":"2024-09-13T13:05:09.289183+0000","last_fullsized":"2024-09-13T13:05:09.289183+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":194,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289750+0000","last_change":"2024-09-13T13:04:02.757267+0000","last_active":"2024-09-13T13:05:09.289750+0000","last_peered":"2024-09-13T13:05:09.289750+0000","last_clean":"2024-09-13T13:05:09.289750+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:05:09.289750+0000","last_undegraded":"2024-09-13T13:05:09.289750+0000","last_fullsized":"2024-09-13T13:05:09.289750+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.233259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.233259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":211,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289977+0000","last_change":"2024-09-13T13:04:02.396436+0000","last_active":"2024-09-13T13:05:09.289977+0000","last_peered":"2024-09-13T13:05:09.289977+0000","last_clean":"2024-09-13T13:05:09.289977+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:05:09.289977+0000","last_undegraded":"2024-09-13T13:05:09.289977+0000","last_fullsized":"2024-09-13T13:05:09.289977+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:02.396342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.396342+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:24.818400+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":186,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262180+0000","last_change":"2024-09-13T13:04:02.765671+0000","last_active":"2024-09-13T13:05:09.262180+0000","last_peered":"2024-09-13T13:05:09.262180+0000","last_clean":"2024-09-13T13:05:09.262180+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:05:09.262180+0000","last_undegraded":"2024-09-13T13:05:09.262180+0000","last_fullsized":"2024-09-13T13:05:09.262180+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.021662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.021662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":199,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.281589+0000","last_change":"2024-09-13T13:05:00.690387+0000","last_active":"2024-09-13T13:05:09.281589+0000","last_peered":"2024-09-13T13:05:09.281589+0000","last_clean":"2024-09-13T13:05:09.281589+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:05:09.281589+0000","last_undegraded":"2024-09-13T13:05:09.281589+0000","last_fullsized":"2024-09-13T13:05:09.281589+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:05:00.690308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:00.690308+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:03.644084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":195,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262489+0000","last_change":"2024-09-13T13:04:56.066382+0000","last_active":"2024-09-13T13:05:09.262489+0000","last_peered":"2024-09-13T13:05:09.262489+0000","last_clean":"2024-09-13T13:05:09.262489+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:05:09.262489+0000","last_undegraded":"2024-09-13T13:05:09.262489+0000","last_fullsized":"2024-09-13T13:05:09.262489+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:56.066326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.066326+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.781998+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.282732+0000","last_change":"2024-09-13T13:02:42.364052+0000","last_active":"2024-09-13T13:05:09.282732+0000","last_peered":"2024-09-13T13:05:09.282732+0000","last_clean":"2024-09-13T13:05:09.282732+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:05:09.282732+0000","last_undegraded":"2024-09-13T13:05:09.282732+0000","last_fullsized":"2024-09-13T13:05:09.282732+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:42.363956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.363956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":172,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.315660+0000","last_change":"2024-09-13T13:02:39.607712+0000","last_active":"2024-09-13T13:05:13.315660+0000","last_peered":"2024-09-13T13:05:13.315660+0000","last_clean":"2024-09-13T13:05:13.315660+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:05:13.315660+0000","last_undegraded":"2024-09-13T13:05:13.315660+0000","last_fullsized":"2024-09-13T13:05:13.315660+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.607646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.607646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":190,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.282356+0000","last_change":"2024-09-13T13:03:40.665853+0000","last_active":"2024-09-13T13:05:09.282356+0000","last_peered":"2024-09-13T13:05:09.282356+0000","last_clean":"2024-09-13T13:05:09.282356+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:05:09.282356+0000","last_undegraded":"2024-09-13T13:05:09.282356+0000","last_fullsized":"2024-09-13T13:05:09.282356+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:40.665805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:40.665805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":197,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:12.093696+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:05:12.093696+0000","last_peered":"2024-09-13T13:05:12.093696+0000","last_clean":"2024-09-13T13:05:12.093696+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:05:12.093696+0000","last_undegraded":"2024-09-13T13:05:12.093696+0000","last_fullsized":"2024-09-13T13:05:12.093696+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:15.096451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:10.383826+0000","last_change":"2024-09-13T13:05:10.383826+0000","last_active":"2024-09-13T13:05:10.383826+0000","last_peered":"2024-09-13T13:05:10.383826+0000","last_clean":"2024-09-13T13:05:10.383826+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:05:10.383826+0000","last_undegraded":"2024-09-13T13:05:10.383826+0000","last_fullsized":"2024-09-13T13:05:10.383826+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.383756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.383756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:33.805546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":162,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.280941+0000","last_change":"2024-09-13T13:01:08.213608+0000","last_active":"2024-09-13T13:05:09.280941+0000","last_peered":"2024-09-13T13:05:09.280941+0000","last_clean":"2024-09-13T13:05:09.280941+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:05:09.280941+0000","last_undegraded":"2024-09-13T13:05:09.280941+0000","last_fullsized":"2024-09-13T13:05:09.280941+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":184,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.263064+0000","last_change":"2024-09-13T13:04:54.016136+0000","last_active":"2024-09-13T13:05:09.263064+0000","last_peered":"2024-09-13T13:05:09.263064+0000","last_clean":"2024-09-13T13:05:09.263064+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:05:09.263064+0000","last_undegraded":"2024-09-13T13:05:09.263064+0000","last_fullsized":"2024-09-13T13:05:09.263064+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:04:54.016064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.016064+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.987760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":184,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.321125+0000","last_change":"2024-09-13T13:02:32.590355+0000","last_active":"2024-09-13T13:05:13.321125+0000","last_peered":"2024-09-13T13:05:13.321125+0000","last_clean":"2024-09-13T13:05:13.321125+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:05:13.321125+0000","last_undegraded":"2024-09-13T13:05:13.321125+0000","last_fullsized":"2024-09-13T13:05:13.321125+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:32.590282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.590282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":212,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.283264+0000","last_change":"2024-09-13T13:03:19.290733+0000","last_active":"2024-09-13T13:05:09.283264+0000","last_peered":"2024-09-13T13:05:09.283264+0000","last_clean":"2024-09-13T13:05:09.283264+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:05:09.283264+0000","last_undegraded":"2024-09-13T13:05:09.283264+0000","last_fullsized":"2024-09-13T13:05:09.283264+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:03:19.290652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:19.290652+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:24.865069+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.274079+0000","last_change":"2024-09-13T13:02:45.202865+0000","last_active":"2024-09-13T13:05:09.274079+0000","last_peered":"2024-09-13T13:05:09.274079+0000","last_clean":"2024-09-13T13:05:09.274079+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:05:09.274079+0000","last_undegraded":"2024-09-13T13:05:09.274079+0000","last_fullsized":"2024-09-13T13:05:09.274079+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.202802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.202802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":181,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.290420+0000","last_change":"2024-09-13T13:04:03.384955+0000","last_active":"2024-09-13T13:05:09.290420+0000","last_peered":"2024-09-13T13:05:09.290420+0000","last_clean":"2024-09-13T13:05:09.290420+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:05:09.290420+0000","last_undegraded":"2024-09-13T13:05:09.290420+0000","last_fullsized":"2024-09-13T13:05:09.290420+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.384901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.384901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:24.976897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":183,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262598+0000","last_change":"2024-09-13T13:04:57.103213+0000","last_active":"2024-09-13T13:05:09.262598+0000","last_peered":"2024-09-13T13:05:09.262598+0000","last_clean":"2024-09-13T13:05:09.262598+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:05:09.262598+0000","last_undegraded":"2024-09-13T13:05:09.262598+0000","last_fullsized":"2024-09-13T13:05:09.262598+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.103167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.103167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.050420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":193,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289152+0000","last_change":"2024-09-13T13:01:04.195126+0000","last_active":"2024-09-13T13:05:09.289152+0000","last_peered":"2024-09-13T13:05:09.289152+0000","last_clean":"2024-09-13T13:05:09.289152+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:05:09.289152+0000","last_undegraded":"2024-09-13T13:05:09.289152+0000","last_fullsized":"2024-09-13T13:05:09.289152+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262991+0000","last_change":"2024-09-13T13:03:11.014220+0000","last_active":"2024-09-13T13:05:09.262991+0000","last_peered":"2024-09-13T13:05:09.262991+0000","last_clean":"2024-09-13T13:05:09.262991+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:05:09.262991+0000","last_undegraded":"2024-09-13T13:05:09.262991+0000","last_fullsized":"2024-09-13T13:05:09.262991+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:11.014157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:11.014157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":171,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288778+0000","last_change":"2024-09-13T13:05:00.457611+0000","last_active":"2024-09-13T13:05:09.288778+0000","last_peered":"2024-09-13T13:05:09.288778+0000","last_clean":"2024-09-13T13:05:09.288778+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:05:09.288778+0000","last_undegraded":"2024-09-13T13:05:09.288778+0000","last_fullsized":"2024-09-13T13:05:09.288778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:05:00.457562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:00.457562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:25.611525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":184,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.272581+0000","last_change":"2024-09-13T13:03:12.276224+0000","last_active":"2024-09-13T13:05:09.272581+0000","last_peered":"2024-09-13T13:05:09.272581+0000","last_clean":"2024-09-13T13:05:09.272581+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:05:09.272581+0000","last_undegraded":"2024-09-13T13:05:09.272581+0000","last_fullsized":"2024-09-13T13:05:09.272581+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:12.276136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.276136+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":196,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.319197+0000","last_change":"2024-09-13T13:02:34.593511+0000","last_active":"2024-09-13T13:05:13.319197+0000","last_peered":"2024-09-13T13:05:13.319197+0000","last_clean":"2024-09-13T13:05:13.319197+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:05:13.319197+0000","last_undegraded":"2024-09-13T13:05:13.319197+0000","last_fullsized":"2024-09-13T13:05:13.319197+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:34.593431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:34.593431+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":189,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321412+0000","last_change":"2024-09-13T13:04:02.765069+0000","last_active":"2024-09-13T13:05:15.321412+0000","last_peered":"2024-09-13T13:05:15.321412+0000","last_clean":"2024-09-13T13:05:15.321412+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:05:15.321412+0000","last_undegraded":"2024-09-13T13:05:15.321412+0000","last_fullsized":"2024-09-13T13:05:15.321412+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:46.604669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.604669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"144'11832","reported_seq":17737,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.860249+0000","last_change":"2024-09-13T13:04:49.780269+0000","last_active":"2024-09-13T13:05:15.860249+0000","last_peered":"2024-09-13T13:05:15.860249+0000","last_clean":"2024-09-13T13:05:15.860249+0000","last_became_active":"2024-09-13T13:04:49.765876+0000","last_became_peered":"2024-09-13T13:04:49.765876+0000","last_unstale":"2024-09-13T13:05:15.860249+0000","last_undegraded":"2024-09-13T13:05:15.860249+0000","last_fullsized":"2024-09-13T13:05:15.860249+0000","mapping_epoch":136,"log_start":"108'5748","ondisk_log_start":"108'5748","created":84,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":6084,"log_dups_size":0,"ondisk_log_size":6084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208797696,"num_objects":3186,"num_object_clones":0,"num_object_copies":9558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11832,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2909,"num_bytes_recovered":147193856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":204,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.272382+0000","last_change":"2024-09-13T13:02:05.179720+0000","last_active":"2024-09-13T13:05:09.272382+0000","last_peered":"2024-09-13T13:05:09.272382+0000","last_clean":"2024-09-13T13:05:09.272382+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:05:09.272382+0000","last_undegraded":"2024-09-13T13:05:09.272382+0000","last_fullsized":"2024-09-13T13:05:09.272382+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:02:05.179667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:05.179667+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":190,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288748+0000","last_change":"2024-09-13T13:03:42.443399+0000","last_active":"2024-09-13T13:05:09.288748+0000","last_peered":"2024-09-13T13:05:09.288748+0000","last_clean":"2024-09-13T13:05:09.288748+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:05:09.288748+0000","last_undegraded":"2024-09-13T13:05:09.288748+0000","last_fullsized":"2024-09-13T13:05:09.288748+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:42.443339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:42.443339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":234,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.263149+0000","last_change":"2024-09-13T13:02:32.013587+0000","last_active":"2024-09-13T13:05:09.263149+0000","last_peered":"2024-09-13T13:05:09.263149+0000","last_clean":"2024-09-13T13:05:09.263149+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:05:09.263149+0000","last_undegraded":"2024-09-13T13:05:09.263149+0000","last_fullsized":"2024-09-13T13:05:09.263149+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:32.013505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.013505+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":170,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.281189+0000","last_change":"2024-09-13T13:04:48.684906+0000","last_active":"2024-09-13T13:05:09.281189+0000","last_peered":"2024-09-13T13:05:09.281189+0000","last_clean":"2024-09-13T13:05:09.281189+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:05:09.281189+0000","last_undegraded":"2024-09-13T13:05:09.281189+0000","last_fullsized":"2024-09-13T13:05:09.281189+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:48.684864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.684864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.491555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"141'11682","reported_seq":11885,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.179840+0000","last_change":"2024-09-13T13:03:41.936543+0000","last_active":"2024-09-13T13:05:13.179840+0000","last_peered":"2024-09-13T13:05:13.179840+0000","last_clean":"2024-09-13T13:05:13.179840+0000","last_became_active":"2024-09-13T13:02:12.369723+0000","last_became_peered":"2024-09-13T13:02:12.369723+0000","last_unstale":"2024-09-13T13:05:13.179840+0000","last_undegraded":"2024-09-13T13:05:13.179840+0000","last_fullsized":"2024-09-13T13:05:13.179840+0000","mapping_epoch":84,"log_start":"114'7000","ondisk_log_start":"114'7000","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6909","last_scrub_stamp":"2024-09-13T13:03:41.936218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:03:41.936218+0000","objects_scrubbed":6836,"log_size":4682,"log_dups_size":0,"ondisk_log_size":4682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225837056,"num_objects":3446,"num_object_clones":0,"num_object_copies":10338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11682,"num_write_kb":484096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":211,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288512+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:05:09.288512+0000","last_peered":"2024-09-13T13:05:09.288512+0000","last_clean":"2024-09-13T13:05:09.288512+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:05:09.288512+0000","last_undegraded":"2024-09-13T13:05:09.288512+0000","last_fullsized":"2024-09-13T13:05:09.288512+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":188,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.290745+0000","last_change":"2024-09-13T13:03:18.516327+0000","last_active":"2024-09-13T13:05:09.290745+0000","last_peered":"2024-09-13T13:05:09.290745+0000","last_clean":"2024-09-13T13:05:09.290745+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:05:09.290745+0000","last_undegraded":"2024-09-13T13:05:09.290745+0000","last_fullsized":"2024-09-13T13:05:09.290745+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:18.516251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.516251+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":176,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262413+0000","last_change":"2024-09-13T13:02:36.005760+0000","last_active":"2024-09-13T13:05:09.262413+0000","last_peered":"2024-09-13T13:05:09.262413+0000","last_clean":"2024-09-13T13:05:09.262413+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:05:09.262413+0000","last_undegraded":"2024-09-13T13:05:09.262413+0000","last_fullsized":"2024-09-13T13:05:09.262413+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.005697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.005697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":171,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262413+0000","last_change":"2024-09-13T13:04:53.006515+0000","last_active":"2024-09-13T13:05:09.262413+0000","last_peered":"2024-09-13T13:05:09.262413+0000","last_clean":"2024-09-13T13:05:09.262413+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:05:09.262413+0000","last_undegraded":"2024-09-13T13:05:09.262413+0000","last_fullsized":"2024-09-13T13:05:09.262413+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:53.006454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.006454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.531241+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"141'11707","reported_seq":11839,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.166990+0000","last_change":"2024-09-13T13:03:45.761068+0000","last_active":"2024-09-13T13:05:13.166990+0000","last_peered":"2024-09-13T13:05:13.166990+0000","last_clean":"2024-09-13T13:05:13.166990+0000","last_became_active":"2024-09-13T13:02:12.373492+0000","last_became_peered":"2024-09-13T13:02:12.373492+0000","last_unstale":"2024-09-13T13:05:13.166990+0000","last_undegraded":"2024-09-13T13:05:13.166990+0000","last_fullsized":"2024-09-13T13:05:13.166990+0000","mapping_epoch":84,"log_start":"114'7000","ondisk_log_start":"114'7000","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":299,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219480064,"num_objects":3349,"num_object_clones":0,"num_object_copies":10047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11707,"num_write_kb":481792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":212,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262508+0000","last_change":"2024-09-13T13:04:02.772914+0000","last_active":"2024-09-13T13:05:09.262508+0000","last_peered":"2024-09-13T13:05:09.262508+0000","last_clean":"2024-09-13T13:05:09.262508+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:05:09.262508+0000","last_undegraded":"2024-09-13T13:05:09.262508+0000","last_fullsized":"2024-09-13T13:05:09.262508+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:02:43.947615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.947615+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":185,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.319478+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:05:13.319478+0000","last_peered":"2024-09-13T13:05:13.319478+0000","last_clean":"2024-09-13T13:05:13.319478+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:05:13.319478+0000","last_undegraded":"2024-09-13T13:05:13.319478+0000","last_fullsized":"2024-09-13T13:05:13.319478+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":226,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:12.298188+0000","last_change":"2024-09-13T13:01:06.211211+0000","last_active":"2024-09-13T13:05:12.298188+0000","last_peered":"2024-09-13T13:05:12.298188+0000","last_clean":"2024-09-13T13:05:12.298188+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:05:12.298188+0000","last_undegraded":"2024-09-13T13:05:12.298188+0000","last_fullsized":"2024-09-13T13:05:12.298188+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":181,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288560+0000","last_change":"2024-09-13T13:04:57.439116+0000","last_active":"2024-09-13T13:05:09.288560+0000","last_peered":"2024-09-13T13:05:09.288560+0000","last_clean":"2024-09-13T13:05:09.288560+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:05:09.288560+0000","last_undegraded":"2024-09-13T13:05:09.288560+0000","last_fullsized":"2024-09-13T13:05:09.288560+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.439053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.439053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:04.992285+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"141'11731","reported_seq":17703,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:12.900841+0000","last_change":"2024-09-13T13:05:05.014474+0000","last_active":"2024-09-13T13:05:12.900841+0000","last_peered":"2024-09-13T13:05:12.900841+0000","last_clean":"2024-09-13T13:05:12.900841+0000","last_became_active":"2024-09-13T13:04:25.922485+0000","last_became_peered":"2024-09-13T13:04:25.922485+0000","last_unstale":"2024-09-13T13:05:12.900841+0000","last_undegraded":"2024-09-13T13:05:12.900841+0000","last_fullsized":"2024-09-13T13:05:12.900841+0000","mapping_epoch":128,"log_start":"114'6900","ondisk_log_start":"114'6900","created":84,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'11284","last_scrub_stamp":"2024-09-13T13:05:05.014171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:05.014171+0000","objects_scrubbed":3783,"log_size":4831,"log_dups_size":0,"ondisk_log_size":4831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:18.664629+0000","scrub_duration":934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216858624,"num_objects":3309,"num_object_clones":0,"num_object_copies":9927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11731,"num_write_kb":481280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2891,"num_bytes_recovered":148570112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":186,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.270609+0000","last_change":"2024-09-13T13:01:04.199011+0000","last_active":"2024-09-13T13:05:09.270609+0000","last_peered":"2024-09-13T13:05:09.270609+0000","last_clean":"2024-09-13T13:05:09.270609+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:05:09.270609+0000","last_undegraded":"2024-09-13T13:05:09.270609+0000","last_fullsized":"2024-09-13T13:05:09.270609+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":182,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.281669+0000","last_change":"2024-09-13T13:05:04.743102+0000","last_active":"2024-09-13T13:05:09.281669+0000","last_peered":"2024-09-13T13:05:09.281669+0000","last_clean":"2024-09-13T13:05:09.281669+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:05:09.281669+0000","last_undegraded":"2024-09-13T13:05:09.281669+0000","last_fullsized":"2024-09-13T13:05:09.281669+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:04.743025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:04.743025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:32.082984+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":241,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.316489+0000","last_change":"2024-09-13T13:04:02.764350+0000","last_active":"2024-09-13T13:05:13.316489+0000","last_peered":"2024-09-13T13:05:13.316489+0000","last_clean":"2024-09-13T13:05:13.316489+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:05:13.316489+0000","last_undegraded":"2024-09-13T13:05:13.316489+0000","last_fullsized":"2024-09-13T13:05:13.316489+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":173,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.321405+0000","last_change":"2024-09-13T13:02:43.558487+0000","last_active":"2024-09-13T13:05:13.321405+0000","last_peered":"2024-09-13T13:05:13.321405+0000","last_clean":"2024-09-13T13:05:13.321405+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:05:13.321405+0000","last_undegraded":"2024-09-13T13:05:13.321405+0000","last_fullsized":"2024-09-13T13:05:13.321405+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:43.558418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.558418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"141'11568","reported_seq":11694,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:12.879551+0000","last_change":"2024-09-13T13:02:12.379536+0000","last_active":"2024-09-13T13:05:12.879551+0000","last_peered":"2024-09-13T13:05:12.879551+0000","last_clean":"2024-09-13T13:05:12.879551+0000","last_became_active":"2024-09-13T13:02:12.379226+0000","last_became_peered":"2024-09-13T13:02:12.379226+0000","last_unstale":"2024-09-13T13:05:12.879551+0000","last_undegraded":"2024-09-13T13:05:12.879551+0000","last_fullsized":"2024-09-13T13:05:12.879551+0000","mapping_epoch":84,"log_start":"108'5700","ondisk_log_start":"108'5700","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5868,"log_dups_size":0,"ondisk_log_size":5868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220987392,"num_objects":3372,"num_object_clones":0,"num_object_copies":10116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11568,"num_write_kb":478080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":216,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.316708+0000","last_change":"2024-09-13T13:02:29.532879+0000","last_active":"2024-09-13T13:05:13.316708+0000","last_peered":"2024-09-13T13:05:13.316708+0000","last_clean":"2024-09-13T13:05:13.316708+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:05:13.316708+0000","last_undegraded":"2024-09-13T13:05:13.316708+0000","last_fullsized":"2024-09-13T13:05:13.316708+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:29.532779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.532779+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":176,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.278495+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:05:09.278495+0000","last_peered":"2024-09-13T13:05:09.278495+0000","last_clean":"2024-09-13T13:05:09.278495+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:05:09.278495+0000","last_undegraded":"2024-09-13T13:05:09.278495+0000","last_fullsized":"2024-09-13T13:05:09.278495+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":142,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.320840+0000","last_change":"2024-09-13T13:04:02.765838+0000","last_active":"2024-09-13T13:05:15.320840+0000","last_peered":"2024-09-13T13:05:15.320840+0000","last_clean":"2024-09-13T13:05:15.320840+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:05:15.320840+0000","last_undegraded":"2024-09-13T13:05:15.320840+0000","last_fullsized":"2024-09-13T13:05:15.320840+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.563708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.563708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":182,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.272336+0000","last_change":"2024-09-13T13:04:02.749391+0000","last_active":"2024-09-13T13:05:09.272336+0000","last_peered":"2024-09-13T13:05:09.272336+0000","last_clean":"2024-09-13T13:05:09.272336+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:05:09.272336+0000","last_undegraded":"2024-09-13T13:05:09.272336+0000","last_fullsized":"2024-09-13T13:05:09.272336+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:28.275768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.275768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"142'11831","reported_seq":14100,"reported_epoch":142,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T13:05:13.412610+0000","last_change":"2024-09-13T13:04:47.345676+0000","last_active":"2024-09-13T13:05:13.412610+0000","last_peered":"2024-09-13T13:05:13.412610+0000","last_clean":"2024-09-13T13:03:14.460355+0000","last_became_active":"2024-09-13T13:04:03.878596+0000","last_became_peered":"2024-09-13T13:04:03.878596+0000","last_unstale":"2024-09-13T13:05:13.412610+0000","last_undegraded":"2024-09-13T13:04:03.760406+0000","last_fullsized":"2024-09-13T13:04:03.744969+0000","mapping_epoch":119,"log_start":"109'6200","ondisk_log_start":"109'6200","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5631,"log_dups_size":0,"ondisk_log_size":5631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218955776,"num_objects":3341,"num_object_clones":0,"num_object_copies":10023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11831,"num_write_kb":485504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1068,"num_bytes_recovered":69926912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":2140}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":278,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.278376+0000","last_change":"2024-09-13T13:02:39.183839+0000","last_active":"2024-09-13T13:05:09.278376+0000","last_peered":"2024-09-13T13:05:09.278376+0000","last_clean":"2024-09-13T13:05:09.278376+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:05:09.278376+0000","last_undegraded":"2024-09-13T13:05:09.278376+0000","last_fullsized":"2024-09-13T13:05:09.278376+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:02:39.183740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.183740+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":195,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262944+0000","last_change":"2024-09-13T13:04:03.850983+0000","last_active":"2024-09-13T13:05:09.262944+0000","last_peered":"2024-09-13T13:05:09.262944+0000","last_clean":"2024-09-13T13:05:09.262944+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:05:09.262944+0000","last_undegraded":"2024-09-13T13:05:09.262944+0000","last_fullsized":"2024-09-13T13:05:09.262944+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.850898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.850898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":202,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.281303+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:05:09.281303+0000","last_peered":"2024-09-13T13:05:09.281303+0000","last_clean":"2024-09-13T13:05:09.281303+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:05:09.281303+0000","last_undegraded":"2024-09-13T13:05:09.281303+0000","last_fullsized":"2024-09-13T13:05:09.281303+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:26.941315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":176,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.326874+0000","last_change":"2024-09-13T13:02:35.566666+0000","last_active":"2024-09-13T13:05:13.326874+0000","last_peered":"2024-09-13T13:05:13.326874+0000","last_clean":"2024-09-13T13:05:13.326874+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:05:13.326874+0000","last_undegraded":"2024-09-13T13:05:13.326874+0000","last_fullsized":"2024-09-13T13:05:13.326874+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:35.566602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:35.566602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":170,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289520+0000","last_change":"2024-09-13T13:04:01.406226+0000","last_active":"2024-09-13T13:05:09.289520+0000","last_peered":"2024-09-13T13:05:09.289520+0000","last_clean":"2024-09-13T13:05:09.289520+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:05:09.289520+0000","last_undegraded":"2024-09-13T13:05:09.289520+0000","last_fullsized":"2024-09-13T13:05:09.289520+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:01.406167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:01.406167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:24.974510+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"142'11777","reported_seq":11905,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.406093+0000","last_change":"2024-09-13T13:02:12.380398+0000","last_active":"2024-09-13T13:05:13.406093+0000","last_peered":"2024-09-13T13:05:13.406093+0000","last_clean":"2024-09-13T13:05:13.406093+0000","last_became_active":"2024-09-13T13:02:12.379331+0000","last_became_peered":"2024-09-13T13:02:12.379331+0000","last_unstale":"2024-09-13T13:05:13.406093+0000","last_undegraded":"2024-09-13T13:05:13.406093+0000","last_fullsized":"2024-09-13T13:05:13.406093+0000","mapping_epoch":84,"log_start":"109'6200","ondisk_log_start":"109'6200","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5577,"log_dups_size":0,"ondisk_log_size":5577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216858624,"num_objects":3309,"num_object_clones":0,"num_object_copies":9927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":482752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":206,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.272530+0000","last_change":"2024-09-13T13:03:14.271569+0000","last_active":"2024-09-13T13:05:09.272530+0000","last_peered":"2024-09-13T13:05:09.272530+0000","last_clean":"2024-09-13T13:05:09.272530+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:05:09.272530+0000","last_undegraded":"2024-09-13T13:05:09.272530+0000","last_fullsized":"2024-09-13T13:05:09.272530+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:03:14.271480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.271480+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":195,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.271109+0000","last_change":"2024-09-13T13:04:02.759283+0000","last_active":"2024-09-13T13:05:09.271109+0000","last_peered":"2024-09-13T13:05:09.271109+0000","last_clean":"2024-09-13T13:05:09.271109+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:05:09.271109+0000","last_undegraded":"2024-09-13T13:05:09.271109+0000","last_fullsized":"2024-09-13T13:05:09.271109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:40.214163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:40.214163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":191,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288035+0000","last_change":"2024-09-13T13:03:18.232951+0000","last_active":"2024-09-13T13:05:09.288035+0000","last_peered":"2024-09-13T13:05:09.288035+0000","last_clean":"2024-09-13T13:05:09.288035+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:05:09.288035+0000","last_undegraded":"2024-09-13T13:05:09.288035+0000","last_fullsized":"2024-09-13T13:05:09.288035+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:03:18.232867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.232867+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:27.894650+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.283171+0000","last_change":"2024-09-13T13:03:14.282866+0000","last_active":"2024-09-13T13:05:09.283171+0000","last_peered":"2024-09-13T13:05:09.283171+0000","last_clean":"2024-09-13T13:05:09.283171+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:05:09.283171+0000","last_undegraded":"2024-09-13T13:05:09.283171+0000","last_fullsized":"2024-09-13T13:05:09.283171+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:20.810360+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":181,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288826+0000","last_change":"2024-09-13T13:04:02.751582+0000","last_active":"2024-09-13T13:05:09.288826+0000","last_peered":"2024-09-13T13:05:09.288826+0000","last_clean":"2024-09-13T13:05:09.288826+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:05:09.288826+0000","last_undegraded":"2024-09-13T13:05:09.288826+0000","last_fullsized":"2024-09-13T13:05:09.288826+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.994729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.994729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"141'11937","reported_seq":12138,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:12.864833+0000","last_change":"2024-09-13T13:03:37.311041+0000","last_active":"2024-09-13T13:05:12.864833+0000","last_peered":"2024-09-13T13:05:12.864833+0000","last_clean":"2024-09-13T13:05:12.864833+0000","last_became_active":"2024-09-13T13:02:12.379779+0000","last_became_peered":"2024-09-13T13:02:12.379779+0000","last_unstale":"2024-09-13T13:05:12.864833+0000","last_undegraded":"2024-09-13T13:05:12.864833+0000","last_fullsized":"2024-09-13T13:05:12.864833+0000","mapping_epoch":84,"log_start":"111'6700","ondisk_log_start":"111'6700","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'6772","last_scrub_stamp":"2024-09-13T13:03:37.310903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:03:37.310903+0000","objects_scrubbed":6730,"log_size":5237,"log_dups_size":0,"ondisk_log_size":5237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226033664,"num_objects":3449,"num_object_clones":0,"num_object_copies":10347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11937,"num_write_kb":492352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":198,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.319261+0000","last_change":"2024-09-13T13:03:18.514436+0000","last_active":"2024-09-13T13:05:13.319261+0000","last_peered":"2024-09-13T13:05:13.319261+0000","last_clean":"2024-09-13T13:05:13.319261+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:05:13.319261+0000","last_undegraded":"2024-09-13T13:05:13.319261+0000","last_fullsized":"2024-09-13T13:05:13.319261+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:03:18.514360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.514360+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":185,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.327072+0000","last_change":"2024-09-13T13:02:37.616313+0000","last_active":"2024-09-13T13:05:13.327072+0000","last_peered":"2024-09-13T13:05:13.327072+0000","last_clean":"2024-09-13T13:05:13.327072+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:05:13.327072+0000","last_undegraded":"2024-09-13T13:05:13.327072+0000","last_fullsized":"2024-09-13T13:05:13.327072+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:37.616224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.616224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":437,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:11.724992+0000","last_change":"2024-09-13T13:04:56.430717+0000","last_active":"2024-09-13T13:05:11.724992+0000","last_peered":"2024-09-13T13:05:11.724992+0000","last_clean":"2024-09-13T13:05:11.724992+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:05:11.724992+0000","last_undegraded":"2024-09-13T13:05:11.724992+0000","last_fullsized":"2024-09-13T13:05:11.724992+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:04:56.430670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.430670+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:59.639039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":245,"num_read_kb":240,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.282084+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:05:09.282084+0000","last_peered":"2024-09-13T13:05:09.282084+0000","last_clean":"2024-09-13T13:05:09.282084+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:05:09.282084+0000","last_undegraded":"2024-09-13T13:05:09.282084+0000","last_fullsized":"2024-09-13T13:05:09.282084+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:19.946825+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":171,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.274674+0000","last_change":"2024-09-13T13:02:36.222607+0000","last_active":"2024-09-13T13:05:09.274674+0000","last_peered":"2024-09-13T13:05:09.274674+0000","last_clean":"2024-09-13T13:05:09.274674+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:05:09.274674+0000","last_undegraded":"2024-09-13T13:05:09.274674+0000","last_fullsized":"2024-09-13T13:05:09.274674+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.222329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"142'11839","reported_seq":11966,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.420817+0000","last_change":"2024-09-13T13:02:12.379556+0000","last_active":"2024-09-13T13:05:13.420817+0000","last_peered":"2024-09-13T13:05:13.420817+0000","last_clean":"2024-09-13T13:05:13.420817+0000","last_became_active":"2024-09-13T13:02:12.379255+0000","last_became_peered":"2024-09-13T13:02:12.379255+0000","last_unstale":"2024-09-13T13:05:13.420817+0000","last_undegraded":"2024-09-13T13:05:13.420817+0000","last_fullsized":"2024-09-13T13:05:13.420817+0000","mapping_epoch":84,"log_start":"109'6200","ondisk_log_start":"109'6200","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215416832,"num_objects":3287,"num_object_clones":0,"num_object_copies":9861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11839,"num_write_kb":484032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":125,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321211+0000","last_change":"2024-09-13T13:04:02.786042+0000","last_active":"2024-09-13T13:05:15.321211+0000","last_peered":"2024-09-13T13:05:15.321211+0000","last_clean":"2024-09-13T13:05:15.321211+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:05:15.321211+0000","last_undegraded":"2024-09-13T13:05:15.321211+0000","last_fullsized":"2024-09-13T13:05:15.321211+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":194,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.291736+0000","last_change":"2024-09-13T13:04:02.765370+0000","last_active":"2024-09-13T13:05:09.291736+0000","last_peered":"2024-09-13T13:05:09.291736+0000","last_clean":"2024-09-13T13:05:09.291736+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:05:09.291736+0000","last_undegraded":"2024-09-13T13:05:09.291736+0000","last_fullsized":"2024-09-13T13:05:09.291736+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:04.228002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:04.228002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":231,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288466+0000","last_change":"2024-09-13T13:04:02.753245+0000","last_active":"2024-09-13T13:05:09.288466+0000","last_peered":"2024-09-13T13:05:09.288466+0000","last_clean":"2024-09-13T13:05:09.288466+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:05:09.288466+0000","last_undegraded":"2024-09-13T13:05:09.288466+0000","last_fullsized":"2024-09-13T13:05:09.288466+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:03:09.054664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:03:09.054664+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288980+0000","last_change":"2024-09-13T13:04:51.395273+0000","last_active":"2024-09-13T13:05:09.288980+0000","last_peered":"2024-09-13T13:05:09.288980+0000","last_clean":"2024-09-13T13:05:09.288980+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:05:09.288980+0000","last_undegraded":"2024-09-13T13:05:09.288980+0000","last_fullsized":"2024-09-13T13:05:09.288980+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:51.395204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:51.395204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:56.927380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":170,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.281963+0000","last_change":"2024-09-13T13:02:39.434636+0000","last_active":"2024-09-13T13:05:09.281963+0000","last_peered":"2024-09-13T13:05:09.281963+0000","last_clean":"2024-09-13T13:05:09.281963+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:05:09.281963+0000","last_undegraded":"2024-09-13T13:05:09.281963+0000","last_fullsized":"2024-09-13T13:05:09.281963+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.434575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.434575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"144'11980","reported_seq":18146,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.849751+0000","last_change":"2024-09-13T13:05:15.349567+0000","last_active":"2024-09-13T13:05:15.849751+0000","last_peered":"2024-09-13T13:05:15.849751+0000","last_clean":"2024-09-13T13:05:15.849751+0000","last_became_active":"2024-09-13T13:05:15.326923+0000","last_became_peered":"2024-09-13T13:05:15.326923+0000","last_unstale":"2024-09-13T13:05:15.849751+0000","last_undegraded":"2024-09-13T13:05:15.849751+0000","last_fullsized":"2024-09-13T13:05:15.849751+0000","mapping_epoch":143,"log_start":"110'6285","ondisk_log_start":"110'6285","created":84,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5695,"log_dups_size":0,"ondisk_log_size":5695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217645080,"num_objects":3322,"num_object_clones":0,"num_object_copies":9966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11980,"num_write_kb":489601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3008,"num_bytes_recovered":155058200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":209,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.263200+0000","last_change":"2024-09-13T13:04:02.832992+0000","last_active":"2024-09-13T13:05:09.263200+0000","last_peered":"2024-09-13T13:05:09.263200+0000","last_clean":"2024-09-13T13:05:09.263200+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:05:09.263200+0000","last_undegraded":"2024-09-13T13:05:09.263200+0000","last_fullsized":"2024-09-13T13:05:09.263200+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:04:02.832935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.832935+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:12.051279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":183,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262800+0000","last_change":"2024-09-13T13:02:45.015634+0000","last_active":"2024-09-13T13:05:09.262800+0000","last_peered":"2024-09-13T13:05:09.262800+0000","last_clean":"2024-09-13T13:05:09.262800+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:05:09.262800+0000","last_undegraded":"2024-09-13T13:05:09.262800+0000","last_fullsized":"2024-09-13T13:05:09.262800+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.015577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.015577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.281844+0000","last_change":"2024-09-13T13:04:47.680651+0000","last_active":"2024-09-13T13:05:09.281844+0000","last_peered":"2024-09-13T13:05:09.281844+0000","last_clean":"2024-09-13T13:05:09.281844+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:05:09.281844+0000","last_undegraded":"2024-09-13T13:05:09.281844+0000","last_fullsized":"2024-09-13T13:05:09.281844+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.680579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.680579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:07.566204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":181,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288856+0000","last_change":"2024-09-13T13:04:02.757378+0000","last_active":"2024-09-13T13:05:09.288856+0000","last_peered":"2024-09-13T13:05:09.288856+0000","last_clean":"2024-09-13T13:05:09.288856+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:05:09.288856+0000","last_undegraded":"2024-09-13T13:05:09.288856+0000","last_fullsized":"2024-09-13T13:05:09.288856+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.213470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.213470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"141'11394","reported_seq":11643,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.055547+0000","last_change":"2024-09-13T13:05:03.355522+0000","last_active":"2024-09-13T13:05:13.055547+0000","last_peered":"2024-09-13T13:05:13.055547+0000","last_clean":"2024-09-13T13:05:13.055547+0000","last_became_active":"2024-09-13T13:02:12.375509+0000","last_became_peered":"2024-09-13T13:02:12.375509+0000","last_unstale":"2024-09-13T13:05:13.055547+0000","last_undegraded":"2024-09-13T13:05:13.055547+0000","last_fullsized":"2024-09-13T13:05:13.055547+0000","mapping_epoch":84,"log_start":"99'4500","ondisk_log_start":"99'4500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'10840","last_scrub_stamp":"2024-09-13T13:05:03.354789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:03.354789+0000","objects_scrubbed":3912,"log_size":6894,"log_dups_size":0,"ondisk_log_size":6894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.025217+0000","scrub_duration":2101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216530944,"num_objects":3304,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11394,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":207,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.263820+0000","last_change":"2024-09-13T13:03:12.028812+0000","last_active":"2024-09-13T13:05:09.263820+0000","last_peered":"2024-09-13T13:05:09.263820+0000","last_clean":"2024-09-13T13:05:09.263820+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:05:09.263820+0000","last_undegraded":"2024-09-13T13:05:09.263820+0000","last_fullsized":"2024-09-13T13:05:09.263820+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:03:12.028719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.028719+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":158,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321009+0000","last_change":"2024-09-13T13:04:02.764657+0000","last_active":"2024-09-13T13:05:15.321009+0000","last_peered":"2024-09-13T13:05:15.321009+0000","last_clean":"2024-09-13T13:05:15.321009+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:05:15.321009+0000","last_undegraded":"2024-09-13T13:05:15.321009+0000","last_fullsized":"2024-09-13T13:05:15.321009+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":167,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.278177+0000","last_change":"2024-09-13T13:01:06.208593+0000","last_active":"2024-09-13T13:05:09.278177+0000","last_peered":"2024-09-13T13:05:09.278177+0000","last_clean":"2024-09-13T13:05:09.278177+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:05:09.278177+0000","last_undegraded":"2024-09-13T13:05:09.278177+0000","last_fullsized":"2024-09-13T13:05:09.278177+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":171,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.283055+0000","last_change":"2024-09-13T13:03:22.282376+0000","last_active":"2024-09-13T13:05:09.283055+0000","last_peered":"2024-09-13T13:05:09.283055+0000","last_clean":"2024-09-13T13:05:09.283055+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:05:09.283055+0000","last_undegraded":"2024-09-13T13:05:09.283055+0000","last_fullsized":"2024-09-13T13:05:09.283055+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:22.282308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.282308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:32.881087+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"142'11825","reported_seq":12030,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.377974+0000","last_change":"2024-09-13T13:04:02.904392+0000","last_active":"2024-09-13T13:05:13.377974+0000","last_peered":"2024-09-13T13:05:13.377974+0000","last_clean":"2024-09-13T13:05:13.377974+0000","last_became_active":"2024-09-13T13:02:12.380229+0000","last_became_peered":"2024-09-13T13:02:12.380229+0000","last_unstale":"2024-09-13T13:05:13.377974+0000","last_undegraded":"2024-09-13T13:05:13.377974+0000","last_fullsized":"2024-09-13T13:05:13.377974+0000","mapping_epoch":84,"log_start":"109'6200","ondisk_log_start":"109'6200","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8189","last_scrub_stamp":"2024-09-13T13:04:02.904297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.904297+0000","objects_scrubbed":6969,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218038272,"num_objects":3327,"num_object_clones":0,"num_object_copies":9981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11825,"num_write_kb":484864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":205,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.263068+0000","last_change":"2024-09-13T13:04:59.091852+0000","last_active":"2024-09-13T13:05:09.263068+0000","last_peered":"2024-09-13T13:05:09.263068+0000","last_clean":"2024-09-13T13:05:09.263068+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:05:09.263068+0000","last_undegraded":"2024-09-13T13:05:09.263068+0000","last_fullsized":"2024-09-13T13:05:09.263068+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:04:59.091751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.091751+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.396141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":190,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289571+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:05:09.289571+0000","last_peered":"2024-09-13T13:05:09.289571+0000","last_clean":"2024-09-13T13:05:09.289571+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:05:09.289571+0000","last_undegraded":"2024-09-13T13:05:09.289571+0000","last_fullsized":"2024-09-13T13:05:09.289571+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":234,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.283357+0000","last_change":"2024-09-13T13:03:15.254020+0000","last_active":"2024-09-13T13:05:09.283357+0000","last_peered":"2024-09-13T13:05:09.283357+0000","last_clean":"2024-09-13T13:05:09.283357+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:05:09.283357+0000","last_undegraded":"2024-09-13T13:05:09.283357+0000","last_fullsized":"2024-09-13T13:05:09.283357+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:03:15.253968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:15.253968+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:21.804124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":171,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262936+0000","last_change":"2024-09-13T13:02:33.997780+0000","last_active":"2024-09-13T13:05:09.262936+0000","last_peered":"2024-09-13T13:05:09.262936+0000","last_clean":"2024-09-13T13:05:09.262936+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:05:09.262936+0000","last_undegraded":"2024-09-13T13:05:09.262936+0000","last_fullsized":"2024-09-13T13:05:09.262936+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.997740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.997740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"141'11783","reported_seq":12032,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.075609+0000","last_change":"2024-09-13T13:05:08.464991+0000","last_active":"2024-09-13T13:05:13.075609+0000","last_peered":"2024-09-13T13:05:13.075609+0000","last_clean":"2024-09-13T13:05:13.075609+0000","last_became_active":"2024-09-13T13:02:12.375310+0000","last_became_peered":"2024-09-13T13:02:12.375310+0000","last_unstale":"2024-09-13T13:05:13.075609+0000","last_undegraded":"2024-09-13T13:05:13.075609+0000","last_fullsized":"2024-09-13T13:05:13.075609+0000","mapping_epoch":84,"log_start":"101'4900","ondisk_log_start":"101'4900","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'11530","last_scrub_stamp":"2024-09-13T13:05:08.464709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:08.464709+0000","objects_scrubbed":3682,"log_size":6883,"log_dups_size":0,"ondisk_log_size":6883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:11.110516+0000","scrub_duration":1226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222363648,"num_objects":3393,"num_object_clones":0,"num_object_copies":10179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11783,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":196,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.316595+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:05:13.316595+0000","last_peered":"2024-09-13T13:05:13.316595+0000","last_clean":"2024-09-13T13:05:13.316595+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:05:13.316595+0000","last_undegraded":"2024-09-13T13:05:13.316595+0000","last_fullsized":"2024-09-13T13:05:13.316595+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":193,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.282485+0000","last_change":"2024-09-13T13:04:02.748907+0000","last_active":"2024-09-13T13:05:09.282485+0000","last_peered":"2024-09-13T13:05:09.282485+0000","last_clean":"2024-09-13T13:05:09.282485+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:05:09.282485+0000","last_undegraded":"2024-09-13T13:05:09.282485+0000","last_fullsized":"2024-09-13T13:05:09.282485+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.305843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.305843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":183,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.283575+0000","last_change":"2024-09-13T13:04:56.183494+0000","last_active":"2024-09-13T13:05:09.283575+0000","last_peered":"2024-09-13T13:05:09.283575+0000","last_clean":"2024-09-13T13:05:09.283575+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:05:09.283575+0000","last_undegraded":"2024-09-13T13:05:09.283575+0000","last_fullsized":"2024-09-13T13:05:09.283575+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:56.183290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.183290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:07.795607+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":164,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.326914+0000","last_change":"2024-09-13T13:01:08.211229+0000","last_active":"2024-09-13T13:05:13.326914+0000","last_peered":"2024-09-13T13:05:13.326914+0000","last_clean":"2024-09-13T13:05:13.326914+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:05:13.326914+0000","last_undegraded":"2024-09-13T13:05:13.326914+0000","last_fullsized":"2024-09-13T13:05:13.326914+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"142'11925","reported_seq":12053,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.351308+0000","last_change":"2024-09-13T13:02:12.379755+0000","last_active":"2024-09-13T13:05:13.351308+0000","last_peered":"2024-09-13T13:05:13.351308+0000","last_clean":"2024-09-13T13:05:13.351308+0000","last_became_active":"2024-09-13T13:02:12.379442+0000","last_became_peered":"2024-09-13T13:02:12.379442+0000","last_unstale":"2024-09-13T13:05:13.351308+0000","last_undegraded":"2024-09-13T13:05:13.351308+0000","last_fullsized":"2024-09-13T13:05:13.351308+0000","mapping_epoch":84,"log_start":"109'6300","ondisk_log_start":"109'6300","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221577216,"num_objects":3381,"num_object_clones":0,"num_object_copies":10143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11925,"num_write_kb":489792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.320882+0000","last_change":"2024-09-13T13:04:02.760376+0000","last_active":"2024-09-13T13:05:15.320882+0000","last_peered":"2024-09-13T13:05:15.320882+0000","last_clean":"2024-09-13T13:05:15.320882+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:05:15.320882+0000","last_undegraded":"2024-09-13T13:05:15.320882+0000","last_fullsized":"2024-09-13T13:05:15.320882+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:42.593284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.593284+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":193,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289010+0000","last_change":"2024-09-13T13:04:02.759538+0000","last_active":"2024-09-13T13:05:09.289010+0000","last_peered":"2024-09-13T13:05:09.289010+0000","last_clean":"2024-09-13T13:05:09.289010+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:05:09.289010+0000","last_undegraded":"2024-09-13T13:05:09.289010+0000","last_fullsized":"2024-09-13T13:05:09.289010+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:29.012312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.012312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":183,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.283569+0000","last_change":"2024-09-13T13:04:47.927556+0000","last_active":"2024-09-13T13:05:09.283569+0000","last_peered":"2024-09-13T13:05:09.283569+0000","last_clean":"2024-09-13T13:05:09.283569+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:05:09.283569+0000","last_undegraded":"2024-09-13T13:05:09.283569+0000","last_fullsized":"2024-09-13T13:05:09.283569+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.927432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.927432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.726893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":170,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:11.420568+0000","last_change":"2024-09-13T13:05:11.420568+0000","last_active":"2024-09-13T13:05:11.420568+0000","last_peered":"2024-09-13T13:05:11.420568+0000","last_clean":"2024-09-13T13:05:11.420568+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:05:11.420568+0000","last_undegraded":"2024-09-13T13:05:11.420568+0000","last_fullsized":"2024-09-13T13:05:11.420568+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:11.420503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:11.420503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:12.545408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"141'11745","reported_seq":11923,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.193081+0000","last_change":"2024-09-13T13:03:45.723433+0000","last_active":"2024-09-13T13:05:13.193081+0000","last_peered":"2024-09-13T13:05:13.193081+0000","last_clean":"2024-09-13T13:05:13.193081+0000","last_became_active":"2024-09-13T13:02:12.376334+0000","last_became_peered":"2024-09-13T13:02:12.376334+0000","last_unstale":"2024-09-13T13:05:13.193081+0000","last_undegraded":"2024-09-13T13:05:13.193081+0000","last_fullsized":"2024-09-13T13:05:13.193081+0000","mapping_epoch":84,"log_start":"114'7100","ondisk_log_start":"114'7100","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":4774,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227737600,"num_objects":3475,"num_object_clones":0,"num_object_copies":10425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11745,"num_write_kb":487040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":202,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262650+0000","last_change":"2024-09-13T13:02:28.990089+0000","last_active":"2024-09-13T13:05:09.262650+0000","last_peered":"2024-09-13T13:05:09.262650+0000","last_clean":"2024-09-13T13:05:09.262650+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:05:09.262650+0000","last_undegraded":"2024-09-13T13:05:09.262650+0000","last_fullsized":"2024-09-13T13:05:09.262650+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:02:28.990005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.990005+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":190,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289198+0000","last_change":"2024-09-13T13:03:37.736138+0000","last_active":"2024-09-13T13:05:09.289198+0000","last_peered":"2024-09-13T13:05:09.289198+0000","last_clean":"2024-09-13T13:05:09.289198+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:05:09.289198+0000","last_undegraded":"2024-09-13T13:05:09.289198+0000","last_fullsized":"2024-09-13T13:05:09.289198+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:37.736073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:37.736073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":167,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.283084+0000","last_change":"2024-09-13T13:01:06.211172+0000","last_active":"2024-09-13T13:05:09.283084+0000","last_peered":"2024-09-13T13:05:09.283084+0000","last_clean":"2024-09-13T13:05:09.283084+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:05:09.283084+0000","last_undegraded":"2024-09-13T13:05:09.283084+0000","last_fullsized":"2024-09-13T13:05:09.283084+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:25.864524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":170,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289273+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:05:09.289273+0000","last_peered":"2024-09-13T13:05:09.289273+0000","last_clean":"2024-09-13T13:05:09.289273+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:05:09.289273+0000","last_undegraded":"2024-09-13T13:05:09.289273+0000","last_fullsized":"2024-09-13T13:05:09.289273+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"141'11935","reported_seq":18076,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.019826+0000","last_change":"2024-09-13T13:04:48.786053+0000","last_active":"2024-09-13T13:05:13.019826+0000","last_peered":"2024-09-13T13:05:13.019826+0000","last_clean":"2024-09-13T13:05:13.019826+0000","last_became_active":"2024-09-13T13:04:48.765805+0000","last_became_peered":"2024-09-13T13:04:48.765805+0000","last_unstale":"2024-09-13T13:05:13.019826+0000","last_undegraded":"2024-09-13T13:05:13.019826+0000","last_fullsized":"2024-09-13T13:05:13.019826+0000","mapping_epoch":135,"log_start":"113'6900","ondisk_log_start":"113'6900","created":84,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226557952,"num_objects":3457,"num_object_clones":0,"num_object_copies":10371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11935,"num_write_kb":492544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3000,"num_bytes_recovered":156631040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":214,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.280837+0000","last_change":"2024-09-13T13:04:52.666294+0000","last_active":"2024-09-13T13:05:09.280837+0000","last_peered":"2024-09-13T13:05:09.280837+0000","last_clean":"2024-09-13T13:05:09.280837+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:05:09.280837+0000","last_undegraded":"2024-09-13T13:05:09.280837+0000","last_fullsized":"2024-09-13T13:05:09.280837+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:52.666222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:52.666222+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:57.559648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":194,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289096+0000","last_change":"2024-09-13T13:02:43.962441+0000","last_active":"2024-09-13T13:05:09.289096+0000","last_peered":"2024-09-13T13:05:09.289096+0000","last_clean":"2024-09-13T13:05:09.289096+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:05:09.289096+0000","last_undegraded":"2024-09-13T13:05:09.289096+0000","last_fullsized":"2024-09-13T13:05:09.289096+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:02:43.962338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.962338+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":250,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288298+0000","last_change":"2024-09-13T13:04:02.759553+0000","last_active":"2024-09-13T13:05:09.288298+0000","last_peered":"2024-09-13T13:05:09.288298+0000","last_clean":"2024-09-13T13:05:09.288298+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:05:09.288298+0000","last_undegraded":"2024-09-13T13:05:09.288298+0000","last_fullsized":"2024-09-13T13:05:09.288298+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:45.959285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.959285+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":181,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.281012+0000","last_change":"2024-09-13T13:04:02.757776+0000","last_active":"2024-09-13T13:05:09.281012+0000","last_peered":"2024-09-13T13:05:09.281012+0000","last_clean":"2024-09-13T13:05:09.281012+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:05:09.281012+0000","last_undegraded":"2024-09-13T13:05:09.281012+0000","last_fullsized":"2024-09-13T13:05:09.281012+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:28.529954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.529954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":158,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.320730+0000","last_change":"2024-09-13T13:04:02.756666+0000","last_active":"2024-09-13T13:05:15.320730+0000","last_peered":"2024-09-13T13:05:15.320730+0000","last_clean":"2024-09-13T13:05:15.320730+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:05:15.320730+0000","last_undegraded":"2024-09-13T13:05:15.320730+0000","last_fullsized":"2024-09-13T13:05:15.320730+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":187,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.290620+0000","last_change":"2024-09-13T13:03:14.504339+0000","last_active":"2024-09-13T13:05:09.290620+0000","last_peered":"2024-09-13T13:05:09.290620+0000","last_clean":"2024-09-13T13:05:09.290620+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:05:09.290620+0000","last_undegraded":"2024-09-13T13:05:09.290620+0000","last_fullsized":"2024-09-13T13:05:09.290620+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:03:14.504263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.504263+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.281583+0000","last_change":"2024-09-13T13:02:26.581497+0000","last_active":"2024-09-13T13:05:09.281583+0000","last_peered":"2024-09-13T13:05:09.281583+0000","last_clean":"2024-09-13T13:05:09.281583+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:05:09.281583+0000","last_undegraded":"2024-09-13T13:05:09.281583+0000","last_fullsized":"2024-09-13T13:05:09.281583+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:26.581434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:26.581434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":181,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288734+0000","last_change":"2024-09-13T13:04:03.746106+0000","last_active":"2024-09-13T13:05:09.288734+0000","last_peered":"2024-09-13T13:05:09.288734+0000","last_clean":"2024-09-13T13:05:09.288734+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:05:09.288734+0000","last_undegraded":"2024-09-13T13:05:09.288734+0000","last_fullsized":"2024-09-13T13:05:09.288734+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.745793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.745793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:15.945770+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":196,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.319374+0000","last_change":"2024-09-13T13:02:42.589951+0000","last_active":"2024-09-13T13:05:13.319374+0000","last_peered":"2024-09-13T13:05:13.319374+0000","last_clean":"2024-09-13T13:05:13.319374+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:05:13.319374+0000","last_undegraded":"2024-09-13T13:05:13.319374+0000","last_fullsized":"2024-09-13T13:05:13.319374+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:42.589866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.589866+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":193,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288485+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:05:09.288485+0000","last_peered":"2024-09-13T13:05:09.288485+0000","last_clean":"2024-09-13T13:05:09.288485+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:05:09.288485+0000","last_undegraded":"2024-09-13T13:05:09.288485+0000","last_fullsized":"2024-09-13T13:05:09.288485+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.542132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.290497+0000","last_change":"2024-09-13T13:01:06.210328+0000","last_active":"2024-09-13T13:05:09.290497+0000","last_peered":"2024-09-13T13:05:09.290497+0000","last_clean":"2024-09-13T13:05:09.290497+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:05:09.290497+0000","last_undegraded":"2024-09-13T13:05:09.290497+0000","last_fullsized":"2024-09-13T13:05:09.290497+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":171,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.282807+0000","last_change":"2024-09-13T13:03:21.281950+0000","last_active":"2024-09-13T13:05:09.282807+0000","last_peered":"2024-09-13T13:05:09.282807+0000","last_clean":"2024-09-13T13:05:09.282807+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:05:09.282807+0000","last_undegraded":"2024-09-13T13:05:09.282807+0000","last_fullsized":"2024-09-13T13:05:09.282807+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:21.281870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:21.281870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:42.889031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":194,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.281505+0000","last_change":"2024-09-13T13:04:53.687731+0000","last_active":"2024-09-13T13:05:09.281505+0000","last_peered":"2024-09-13T13:05:09.281505+0000","last_clean":"2024-09-13T13:05:09.281505+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:05:09.281505+0000","last_undegraded":"2024-09-13T13:05:09.281505+0000","last_fullsized":"2024-09-13T13:05:09.281505+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.632415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":184,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.316246+0000","last_change":"2024-09-13T13:02:41.622369+0000","last_active":"2024-09-13T13:05:13.316246+0000","last_peered":"2024-09-13T13:05:13.316246+0000","last_clean":"2024-09-13T13:05:13.316246+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:05:13.316246+0000","last_undegraded":"2024-09-13T13:05:13.316246+0000","last_fullsized":"2024-09-13T13:05:13.316246+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.622292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.622292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":225,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289068+0000","last_change":"2024-09-13T13:01:06.210251+0000","last_active":"2024-09-13T13:05:09.289068+0000","last_peered":"2024-09-13T13:05:09.289068+0000","last_clean":"2024-09-13T13:05:09.289068+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:05:09.289068+0000","last_undegraded":"2024-09-13T13:05:09.289068+0000","last_fullsized":"2024-09-13T13:05:09.289068+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":171,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.263494+0000","last_change":"2024-09-13T13:04:49.927118+0000","last_active":"2024-09-13T13:05:09.263494+0000","last_peered":"2024-09-13T13:05:09.263494+0000","last_clean":"2024-09-13T13:05:09.263494+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:05:09.263494+0000","last_undegraded":"2024-09-13T13:05:09.263494+0000","last_fullsized":"2024-09-13T13:05:09.263494+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:49.927073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:49.927073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:08.714937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":193,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289348+0000","last_change":"2024-09-13T13:02:37.222400+0000","last_active":"2024-09-13T13:05:09.289348+0000","last_peered":"2024-09-13T13:05:09.289348+0000","last_clean":"2024-09-13T13:05:09.289348+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:05:09.289348+0000","last_undegraded":"2024-09-13T13:05:09.289348+0000","last_fullsized":"2024-09-13T13:05:09.289348+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:37.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.222329+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":182,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288728+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:05:09.288728+0000","last_peered":"2024-09-13T13:05:09.288728+0000","last_clean":"2024-09-13T13:05:09.288728+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:05:09.288728+0000","last_undegraded":"2024-09-13T13:05:09.288728+0000","last_fullsized":"2024-09-13T13:05:09.288728+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:55.131839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":241,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.281369+0000","last_change":"2024-09-13T13:03:41.676060+0000","last_active":"2024-09-13T13:05:09.281369+0000","last_peered":"2024-09-13T13:05:09.281369+0000","last_clean":"2024-09-13T13:05:09.281369+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:05:09.281369+0000","last_undegraded":"2024-09-13T13:05:09.281369+0000","last_fullsized":"2024-09-13T13:05:09.281369+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:03:41.675978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:41.675978+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":172,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.316338+0000","last_change":"2024-09-13T13:02:40.615749+0000","last_active":"2024-09-13T13:05:13.316338+0000","last_peered":"2024-09-13T13:05:13.316338+0000","last_clean":"2024-09-13T13:05:13.316338+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:05:13.316338+0000","last_undegraded":"2024-09-13T13:05:13.316338+0000","last_fullsized":"2024-09-13T13:05:13.316338+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:40.615674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:40.615674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":209,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.262697+0000","last_change":"2024-09-13T13:04:02.764890+0000","last_active":"2024-09-13T13:05:09.262697+0000","last_peered":"2024-09-13T13:05:09.262697+0000","last_clean":"2024-09-13T13:05:09.262697+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:05:09.262697+0000","last_undegraded":"2024-09-13T13:05:09.262697+0000","last_fullsized":"2024-09-13T13:05:09.262697+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:02:42.961328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.961328+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":202,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:10.066345+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:05:10.066345+0000","last_peered":"2024-09-13T13:05:10.066345+0000","last_clean":"2024-09-13T13:05:10.066345+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:05:10.066345+0000","last_undegraded":"2024-09-13T13:05:10.066345+0000","last_fullsized":"2024-09-13T13:05:10.066345+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:39.039395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":183,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.270460+0000","last_change":"2024-09-13T13:03:44.467611+0000","last_active":"2024-09-13T13:05:09.270460+0000","last_peered":"2024-09-13T13:05:09.270460+0000","last_clean":"2024-09-13T13:05:09.270460+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:05:09.270460+0000","last_undegraded":"2024-09-13T13:05:09.270460+0000","last_fullsized":"2024-09-13T13:05:09.270460+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:44.467297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:44.467297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:10.953531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":170,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288575+0000","last_change":"2024-09-13T13:02:42.208021+0000","last_active":"2024-09-13T13:05:09.288575+0000","last_peered":"2024-09-13T13:05:09.288575+0000","last_clean":"2024-09-13T13:05:09.288575+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:05:09.288575+0000","last_undegraded":"2024-09-13T13:05:09.288575+0000","last_fullsized":"2024-09-13T13:05:09.288575+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:42.207956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.207956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":138,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.280940+0000","last_change":"2024-09-13T13:04:02.757522+0000","last_active":"2024-09-13T13:05:09.280940+0000","last_peered":"2024-09-13T13:05:09.280940+0000","last_clean":"2024-09-13T13:05:09.280940+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:05:09.280940+0000","last_undegraded":"2024-09-13T13:05:09.280940+0000","last_fullsized":"2024-09-13T13:05:09.280940+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:35.448404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:35.448404+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":182,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288932+0000","last_change":"2024-09-13T13:02:39.921391+0000","last_active":"2024-09-13T13:05:09.288932+0000","last_peered":"2024-09-13T13:05:09.288932+0000","last_clean":"2024-09-13T13:05:09.288932+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:05:09.288932+0000","last_undegraded":"2024-09-13T13:05:09.288932+0000","last_fullsized":"2024-09-13T13:05:09.288932+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.921325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.921325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289353+0000","last_change":"2024-09-13T13:01:06.211993+0000","last_active":"2024-09-13T13:05:09.289353+0000","last_peered":"2024-09-13T13:05:09.289353+0000","last_clean":"2024-09-13T13:05:09.289353+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:05:09.289353+0000","last_undegraded":"2024-09-13T13:05:09.289353+0000","last_fullsized":"2024-09-13T13:05:09.289353+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":172,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.315604+0000","last_change":"2024-09-13T13:02:36.586702+0000","last_active":"2024-09-13T13:05:13.315604+0000","last_peered":"2024-09-13T13:05:13.315604+0000","last_clean":"2024-09-13T13:05:13.315604+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:05:13.315604+0000","last_undegraded":"2024-09-13T13:05:13.315604+0000","last_fullsized":"2024-09-13T13:05:13.315604+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.586654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.586654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":208,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289844+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:05:09.289844+0000","last_peered":"2024-09-13T13:05:09.289844+0000","last_clean":"2024-09-13T13:05:09.289844+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:05:09.289844+0000","last_undegraded":"2024-09-13T13:05:09.289844+0000","last_fullsized":"2024-09-13T13:05:09.289844+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":182,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:12.597696+0000","last_change":"2024-09-13T13:05:12.597696+0000","last_active":"2024-09-13T13:05:12.597696+0000","last_peered":"2024-09-13T13:05:12.597696+0000","last_clean":"2024-09-13T13:05:12.597696+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:05:12.597696+0000","last_undegraded":"2024-09-13T13:05:12.597696+0000","last_fullsized":"2024-09-13T13:05:12.597696+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:12.597637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.597637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:30.070820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288119+0000","last_change":"2024-09-13T13:05:08.432874+0000","last_active":"2024-09-13T13:05:09.288119+0000","last_peered":"2024-09-13T13:05:09.288119+0000","last_clean":"2024-09-13T13:05:09.288119+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:05:09.288119+0000","last_undegraded":"2024-09-13T13:05:09.288119+0000","last_fullsized":"2024-09-13T13:05:09.288119+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:08.432816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:08.432816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:09.081986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":173,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.289491+0000","last_change":"2024-09-13T13:04:02.760695+0000","last_active":"2024-09-13T13:05:09.289491+0000","last_peered":"2024-09-13T13:05:09.289491+0000","last_clean":"2024-09-13T13:05:09.289491+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:05:09.289491+0000","last_undegraded":"2024-09-13T13:05:09.289491+0000","last_fullsized":"2024-09-13T13:05:09.289491+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":200,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288403+0000","last_change":"2024-09-13T13:04:02.755722+0000","last_active":"2024-09-13T13:05:09.288403+0000","last_peered":"2024-09-13T13:05:09.288403+0000","last_clean":"2024-09-13T13:05:09.288403+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:05:09.288403+0000","last_undegraded":"2024-09-13T13:05:09.288403+0000","last_fullsized":"2024-09-13T13:05:09.288403+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":184,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.318020+0000","last_change":"2024-09-13T13:03:15.466909+0000","last_active":"2024-09-13T13:05:13.318020+0000","last_peered":"2024-09-13T13:05:13.318020+0000","last_clean":"2024-09-13T13:05:13.318020+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:05:13.318020+0000","last_undegraded":"2024-09-13T13:05:13.318020+0000","last_fullsized":"2024-09-13T13:05:13.318020+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:15.466859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:15.466859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":74,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.320885+0000","last_change":"2024-09-13T13:04:02.765046+0000","last_active":"2024-09-13T13:05:15.320885+0000","last_peered":"2024-09-13T13:05:15.320885+0000","last_clean":"2024-09-13T13:05:15.320885+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:05:15.320885+0000","last_undegraded":"2024-09-13T13:05:15.320885+0000","last_fullsized":"2024-09-13T13:05:15.320885+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":170,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288299+0000","last_change":"2024-09-13T13:04:58.468060+0000","last_active":"2024-09-13T13:05:09.288299+0000","last_peered":"2024-09-13T13:05:09.288299+0000","last_clean":"2024-09-13T13:05:09.288299+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:05:09.288299+0000","last_undegraded":"2024-09-13T13:05:09.288299+0000","last_fullsized":"2024-09-13T13:05:09.288299+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:58.468011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:58.468011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.129704+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":171,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.271205+0000","last_change":"2024-09-13T13:02:43.234909+0000","last_active":"2024-09-13T13:05:09.271205+0000","last_peered":"2024-09-13T13:05:09.271205+0000","last_clean":"2024-09-13T13:05:09.271205+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:05:09.271205+0000","last_undegraded":"2024-09-13T13:05:09.271205+0000","last_fullsized":"2024-09-13T13:05:09.271205+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:43.234847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.234847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":182,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-13T13:05:09.288663+0000","last_change":"2024-09-13T13:02:46.246602+0000","last_active":"2024-09-13T13:05:09.288663+0000","last_peered":"2024-09-13T13:05:09.288663+0000","last_clean":"2024-09-13T13:05:09.288663+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:05:09.288663+0000","last_undegraded":"2024-09-13T13:05:09.288663+0000","last_fullsized":"2024-09-13T13:05:09.288663+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.246540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.246540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":184,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321306+0000","last_change":"2024-09-13T13:04:02.756287+0000","last_active":"2024-09-13T13:05:15.321306+0000","last_peered":"2024-09-13T13:05:15.321306+0000","last_clean":"2024-09-13T13:05:15.321306+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:05:15.321306+0000","last_undegraded":"2024-09-13T13:05:15.321306+0000","last_fullsized":"2024-09-13T13:05:15.321306+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:02:39.557251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.557251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":177,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-13T13:05:13.316609+0000","last_change":"2024-09-13T13:02:30.512536+0000","last_active":"2024-09-13T13:05:13.316609+0000","last_peered":"2024-09-13T13:05:13.316609+0000","last_clean":"2024-09-13T13:05:13.316609+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:05:13.316609+0000","last_undegraded":"2024-09-13T13:05:13.316609+0000","last_fullsized":"2024-09-13T13:05:13.316609+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:30.512433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:30.512433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3519676440,"num_objects":53707,"num_object_clones":0,"num_object_copies":161121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188491,"num_write_kb":7750273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12876,"num_bytes_recovered":677380120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10559762432,"data_stored":10559750216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88658,"ondisk_log_size":88658,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":245,"num_read_kb":240,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201333,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1192164,"kb_used_data":1088788,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92552988,"statfs":{"total":95995035648,"available":94774259712,"internally_reserved":0,"allocated":1114918912,"data_stored":1114465540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806140944,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1135508,"kb_used_data":1025860,"kb_used_omap":4,"kb_used_meta":109627,"kb_avail":92609644,"statfs":{"total":95995035648,"available":94832275456,"internally_reserved":0,"allocated":1050480640,"data_stored":1050103051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":112258733},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691905,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":950488,"kb_used_data":879048,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92794664,"statfs":{"total":95995035648,"available":95021735936,"internally_reserved":0,"allocated":900145152,"data_stored":899702364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1202284,"kb_used_data":1098908,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92542868,"statfs":{"total":95995035648,"available":94763896832,"internally_reserved":0,"allocated":1125281792,"data_stored":1124818660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117070,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2108208,"kb_used_data":1950688,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":91636944,"statfs":{"total":95995035648,"available":93836230656,"internally_reserved":0,"allocated":1997504512,"data_stored":1997038114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161282503},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1426888,"kb_used_data":1327672,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92318264,"statfs":{"total":95995035648,"available":94533902336,"internally_reserved":0,"allocated":1359536128,"data_stored":1359109064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1414916,"kb_used_data":1311540,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92330236,"statfs":{"total":95995035648,"available":94546161664,"internally_reserved":0,"allocated":1343016960,"data_stored":1342529787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1876268,"kb_used_data":1732508,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91868884,"statfs":{"total":95995035648,"available":94073737216,"internally_reserved":0,"allocated":1774088192,"data_stored":1773626361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,4,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1754009600,"data_stored":1754005528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1333661696,"data_stored":1333657624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1338507264,"data_stored":1338507264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1979842560,"data_stored":1979842560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1108803584,"data_stored":1108803584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":894107648,"data_stored":894107648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1046024192,"data_stored":1046020120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1104805888,"data_stored":1104805888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:05:18.958 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:05:19.013 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:19.020 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:18 smithi167 ceph-mon[33178]: pgmap v188: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 791 op/s; 1201/161718 objects degraded (0.743%); 2.1 MiB/s, 119 objects/s recovering 2024-09-13T13:05:19.020 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:18 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:05:19.020 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:18 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:05:19.020 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:18 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 1201/162765 objects degraded (0.738%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T13:05:19.020 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:18 smithi167 ceph-mon[33178]: from='client.15165 v1:172.21.15.52:0/3723603960' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:19.046 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:19.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:19.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:19.053 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:05:19.053 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:05:19.149 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:19.194 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:19.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:19.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:19.291 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:05:19.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:19.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:19.324 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:19.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:19.425 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:19.481 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:19.526 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:19.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:19.628 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:19.677 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:19.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:19.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:19.725 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:05:19.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:19.805 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:19.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:19.871 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:19 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1019795561' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:05:19.871 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:19 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1019795561' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:05:19.908 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:19.936 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:19.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:19.964 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:19 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1019795561' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:05:19.966 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T13:05:20.038 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:20.076 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:20.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:20.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:20.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:20.212 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:20.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:20.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:20.347 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:20.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:20.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:20.359 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:05:20.450 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:20.479 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:20.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:20.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:20.581 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:20.629 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:20.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:20.730 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:20.765 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:20 smithi167 ceph-mon[33178]: pgmap v189: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 806 B/s rd, 0 B/s wr, 899 op/s; 823/151020 objects degraded (0.545%); 5.7 MiB/s, 172 objects/s recovering 2024-09-13T13:05:20.765 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:20 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:20.765 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:20 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1019795561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:05:20.765 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:20 smithi167 ceph-mon[33178]: osdmap e145: 8 total, 8 up, 8 in 2024-09-13T13:05:20.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:20.777 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:20.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:20.856 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:20.878 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:20.895 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:05:20.895 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:05:20.936 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:20.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:20.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:20.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:20 smithi052 ceph-mon[29471]: pgmap v189: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 806 B/s rd, 0 B/s wr, 899 op/s; 823/151020 objects degraded (0.545%); 5.7 MiB/s, 172 objects/s recovering 2024-09-13T13:05:20.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:20 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:20.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:20 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1019795561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:05:20.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:20 smithi052 ceph-mon[29471]: osdmap e145: 8 total, 8 up, 8 in 2024-09-13T13:05:20.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:20 smithi052 ceph-mon[36682]: pgmap v189: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 806 B/s rd, 0 B/s wr, 899 op/s; 823/151020 objects degraded (0.545%); 5.7 MiB/s, 172 objects/s recovering 2024-09-13T13:05:20.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:20 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:20.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:20 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1019795561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:05:20.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:20 smithi052 ceph-mon[36682]: osdmap e145: 8 total, 8 up, 8 in 2024-09-13T13:05:21.040 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:21.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:21.092 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:21.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:05:21.176 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:05:21.194 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:21.219 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:21.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:21.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:21.348 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:21.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:21.449 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:21.477 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:21.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:21.579 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:21.617 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:21.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:21.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:21.718 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:21.749 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T13:05:21.751 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:21.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:21.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:21 smithi167 ceph-mon[33178]: from='client.15177 v1:172.21.15.52:0/3134503696' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:21.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:21.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:21.889 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:21.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:21.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:21 smithi052 ceph-mon[36682]: from='client.15177 v1:172.21.15.52:0/3134503696' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:21.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:21 smithi052 ceph-mon[29471]: from='client.15177 v1:172.21.15.52:0/3134503696' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:21.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:22.023 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:22.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:22.062 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:05:22.125 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:22.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:22.152 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:22.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:22.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:22.279 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:22.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:22.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:22.381 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:22.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:22.420 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:22.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:22.507 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:05:22.521 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:22.546 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:22.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:22.579 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-13T13:05:19.725075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3286571571,"num_objects":50340,"num_object_clones":0,"num_object_copies":151020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50340,"num_whiteouts":0,"num_read":1097,"num_read_kb":954,"num_write":192606,"num_write_kb":7751946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13262,"num_bytes_recovered":702743206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88784,"ondisk_log_size":88784,"up":444,"acting":443,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10620388,"kb_used_data":9711900,"kb_used_omap":15,"kb_used_meta":908080,"kb_avail":739340828,"statfs":{"total":767960285184,"available":757085007872,"internally_reserved":0,"allocated":9944985600,"data_stored":9941823964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":929874713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-256573440,"num_objects":-3915,"num_object_clones":0,"num_object_copies":-11745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3915,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3915,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":413,"num_bytes_recovered":24772608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.063316"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":189,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.327722+0000","last_change":"2024-09-13T13:02:38.651382+0000","last_active":"2024-09-13T13:05:15.327722+0000","last_peered":"2024-09-13T13:05:15.327722+0000","last_clean":"2024-09-13T13:05:15.327722+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:05:15.327722+0000","last_undegraded":"2024-09-13T13:05:15.327722+0000","last_fullsized":"2024-09-13T13:05:15.327722+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:38.651314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:38.651314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":212,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.356405+0000","last_change":"2024-09-13T13:04:48.899982+0000","last_active":"2024-09-13T13:05:15.356405+0000","last_peered":"2024-09-13T13:05:15.356405+0000","last_clean":"2024-09-13T13:05:15.356405+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:05:15.356405+0000","last_undegraded":"2024-09-13T13:05:15.356405+0000","last_fullsized":"2024-09-13T13:05:15.356405+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:48.899915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.899915+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:00.227319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":181,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.324019+0000","last_change":"2024-09-13T13:02:30.248974+0000","last_active":"2024-09-13T13:05:15.324019+0000","last_peered":"2024-09-13T13:05:15.324019+0000","last_clean":"2024-09-13T13:05:15.324019+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:05:15.324019+0000","last_undegraded":"2024-09-13T13:05:15.324019+0000","last_fullsized":"2024-09-13T13:05:15.324019+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:30.248905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:30.248905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.351412+0000","last_change":"2024-09-13T13:05:01.076346+0000","last_active":"2024-09-13T13:05:15.351412+0000","last_peered":"2024-09-13T13:05:15.351412+0000","last_clean":"2024-09-13T13:05:15.351412+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:05:15.351412+0000","last_undegraded":"2024-09-13T13:05:15.351412+0000","last_fullsized":"2024-09-13T13:05:15.351412+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.076275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.076275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:14.180362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":179,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321062+0000","last_change":"2024-09-13T13:04:02.760609+0000","last_active":"2024-09-13T13:05:15.321062+0000","last_peered":"2024-09-13T13:05:15.321062+0000","last_clean":"2024-09-13T13:05:15.321062+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:05:15.321062+0000","last_undegraded":"2024-09-13T13:05:15.321062+0000","last_fullsized":"2024-09-13T13:05:15.321062+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":200,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.323320+0000","last_change":"2024-09-13T13:04:02.757267+0000","last_active":"2024-09-13T13:05:15.323320+0000","last_peered":"2024-09-13T13:05:15.323320+0000","last_clean":"2024-09-13T13:05:15.323320+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:05:15.323320+0000","last_undegraded":"2024-09-13T13:05:15.323320+0000","last_fullsized":"2024-09-13T13:05:15.323320+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.233259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.233259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":217,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321422+0000","last_change":"2024-09-13T13:04:02.396436+0000","last_active":"2024-09-13T13:05:15.321422+0000","last_peered":"2024-09-13T13:05:15.321422+0000","last_clean":"2024-09-13T13:05:15.321422+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:05:15.321422+0000","last_undegraded":"2024-09-13T13:05:15.321422+0000","last_fullsized":"2024-09-13T13:05:15.321422+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:02.396342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.396342+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:24.818400+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":192,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352060+0000","last_change":"2024-09-13T13:04:02.765671+0000","last_active":"2024-09-13T13:05:15.352060+0000","last_peered":"2024-09-13T13:05:15.352060+0000","last_clean":"2024-09-13T13:05:15.352060+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:05:15.352060+0000","last_undegraded":"2024-09-13T13:05:15.352060+0000","last_fullsized":"2024-09-13T13:05:15.352060+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.021662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.021662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":205,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352260+0000","last_change":"2024-09-13T13:05:00.690387+0000","last_active":"2024-09-13T13:05:15.352260+0000","last_peered":"2024-09-13T13:05:15.352260+0000","last_clean":"2024-09-13T13:05:15.352260+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:05:15.352260+0000","last_undegraded":"2024-09-13T13:05:15.352260+0000","last_fullsized":"2024-09-13T13:05:15.352260+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:05:00.690308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:00.690308+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:03.644084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":201,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.351705+0000","last_change":"2024-09-13T13:04:56.066382+0000","last_active":"2024-09-13T13:05:15.351705+0000","last_peered":"2024-09-13T13:05:15.351705+0000","last_clean":"2024-09-13T13:05:15.351705+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:05:15.351705+0000","last_undegraded":"2024-09-13T13:05:15.351705+0000","last_fullsized":"2024-09-13T13:05:15.351705+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:56.066326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.066326+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.781998+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":180,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.351709+0000","last_change":"2024-09-13T13:02:42.364052+0000","last_active":"2024-09-13T13:05:15.351709+0000","last_peered":"2024-09-13T13:05:15.351709+0000","last_clean":"2024-09-13T13:05:15.351709+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:05:15.351709+0000","last_undegraded":"2024-09-13T13:05:15.351709+0000","last_fullsized":"2024-09-13T13:05:15.351709+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:42.363956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.363956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":176,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.325638+0000","last_change":"2024-09-13T13:02:39.607712+0000","last_active":"2024-09-13T13:05:15.325638+0000","last_peered":"2024-09-13T13:05:15.325638+0000","last_clean":"2024-09-13T13:05:15.325638+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:05:15.325638+0000","last_undegraded":"2024-09-13T13:05:15.325638+0000","last_fullsized":"2024-09-13T13:05:15.325638+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.607646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.607646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":196,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352624+0000","last_change":"2024-09-13T13:03:40.665853+0000","last_active":"2024-09-13T13:05:15.352624+0000","last_peered":"2024-09-13T13:05:15.352624+0000","last_clean":"2024-09-13T13:05:15.352624+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:05:15.352624+0000","last_undegraded":"2024-09-13T13:05:15.352624+0000","last_fullsized":"2024-09-13T13:05:15.352624+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:40.665805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:40.665805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":203,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352225+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:05:15.352225+0000","last_peered":"2024-09-13T13:05:15.352225+0000","last_clean":"2024-09-13T13:05:15.352225+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:05:15.352225+0000","last_undegraded":"2024-09-13T13:05:15.352225+0000","last_fullsized":"2024-09-13T13:05:15.352225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:15.096451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":180,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353628+0000","last_change":"2024-09-13T13:05:10.383826+0000","last_active":"2024-09-13T13:05:15.353628+0000","last_peered":"2024-09-13T13:05:15.353628+0000","last_clean":"2024-09-13T13:05:15.353628+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:05:15.353628+0000","last_undegraded":"2024-09-13T13:05:15.353628+0000","last_fullsized":"2024-09-13T13:05:15.353628+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.383756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.383756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:33.805546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":168,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.351321+0000","last_change":"2024-09-13T13:01:08.213608+0000","last_active":"2024-09-13T13:05:15.351321+0000","last_peered":"2024-09-13T13:05:15.351321+0000","last_clean":"2024-09-13T13:05:15.351321+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:05:15.351321+0000","last_undegraded":"2024-09-13T13:05:15.351321+0000","last_fullsized":"2024-09-13T13:05:15.351321+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":190,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353049+0000","last_change":"2024-09-13T13:04:54.016136+0000","last_active":"2024-09-13T13:05:15.353049+0000","last_peered":"2024-09-13T13:05:15.353049+0000","last_clean":"2024-09-13T13:05:15.353049+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:05:15.353049+0000","last_undegraded":"2024-09-13T13:05:15.353049+0000","last_fullsized":"2024-09-13T13:05:15.353049+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:04:54.016064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.016064+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.987760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":188,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.326425+0000","last_change":"2024-09-13T13:02:32.590355+0000","last_active":"2024-09-13T13:05:15.326425+0000","last_peered":"2024-09-13T13:05:15.326425+0000","last_clean":"2024-09-13T13:05:15.326425+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:05:15.326425+0000","last_undegraded":"2024-09-13T13:05:15.326425+0000","last_fullsized":"2024-09-13T13:05:15.326425+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:32.590282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.590282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":218,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321222+0000","last_change":"2024-09-13T13:03:19.290733+0000","last_active":"2024-09-13T13:05:15.321222+0000","last_peered":"2024-09-13T13:05:15.321222+0000","last_clean":"2024-09-13T13:05:15.321222+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:05:15.321222+0000","last_undegraded":"2024-09-13T13:05:15.321222+0000","last_fullsized":"2024-09-13T13:05:15.321222+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:03:19.290652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:19.290652+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:24.865069+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":181,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.324166+0000","last_change":"2024-09-13T13:02:45.202865+0000","last_active":"2024-09-13T13:05:15.324166+0000","last_peered":"2024-09-13T13:05:15.324166+0000","last_clean":"2024-09-13T13:05:15.324166+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:05:15.324166+0000","last_undegraded":"2024-09-13T13:05:15.324166+0000","last_fullsized":"2024-09-13T13:05:15.324166+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.202802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.202802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":188,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.323821+0000","last_change":"2024-09-13T13:04:03.384955+0000","last_active":"2024-09-13T13:05:15.323821+0000","last_peered":"2024-09-13T13:05:15.323821+0000","last_clean":"2024-09-13T13:05:15.323821+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:05:15.323821+0000","last_undegraded":"2024-09-13T13:05:15.323821+0000","last_fullsized":"2024-09-13T13:05:15.323821+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.384901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.384901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:24.976897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":189,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.351852+0000","last_change":"2024-09-13T13:04:57.103213+0000","last_active":"2024-09-13T13:05:15.351852+0000","last_peered":"2024-09-13T13:05:15.351852+0000","last_clean":"2024-09-13T13:05:15.351852+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:05:15.351852+0000","last_undegraded":"2024-09-13T13:05:15.351852+0000","last_fullsized":"2024-09-13T13:05:15.351852+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.103167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.103167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.050420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":199,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321923+0000","last_change":"2024-09-13T13:01:04.195126+0000","last_active":"2024-09-13T13:05:15.321923+0000","last_peered":"2024-09-13T13:05:15.321923+0000","last_clean":"2024-09-13T13:05:15.321923+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:05:15.321923+0000","last_undegraded":"2024-09-13T13:05:15.321923+0000","last_fullsized":"2024-09-13T13:05:15.321923+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":181,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352595+0000","last_change":"2024-09-13T13:03:11.014220+0000","last_active":"2024-09-13T13:05:15.352595+0000","last_peered":"2024-09-13T13:05:15.352595+0000","last_clean":"2024-09-13T13:05:15.352595+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:05:15.352595+0000","last_undegraded":"2024-09-13T13:05:15.352595+0000","last_fullsized":"2024-09-13T13:05:15.352595+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:11.014157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:11.014157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.355256+0000","last_change":"2024-09-13T13:05:00.457611+0000","last_active":"2024-09-13T13:05:15.355256+0000","last_peered":"2024-09-13T13:05:15.355256+0000","last_clean":"2024-09-13T13:05:15.355256+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:05:15.355256+0000","last_undegraded":"2024-09-13T13:05:15.355256+0000","last_fullsized":"2024-09-13T13:05:15.355256+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:05:00.457562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:00.457562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:25.611525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":190,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.324179+0000","last_change":"2024-09-13T13:03:12.276224+0000","last_active":"2024-09-13T13:05:15.324179+0000","last_peered":"2024-09-13T13:05:15.324179+0000","last_clean":"2024-09-13T13:05:15.324179+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:05:15.324179+0000","last_undegraded":"2024-09-13T13:05:15.324179+0000","last_fullsized":"2024-09-13T13:05:15.324179+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:12.276136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.276136+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":200,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.326761+0000","last_change":"2024-09-13T13:02:34.593511+0000","last_active":"2024-09-13T13:05:15.326761+0000","last_peered":"2024-09-13T13:05:15.326761+0000","last_clean":"2024-09-13T13:05:15.326761+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:05:15.326761+0000","last_undegraded":"2024-09-13T13:05:15.326761+0000","last_fullsized":"2024-09-13T13:05:15.326761+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:34.593431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:34.593431+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":189,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321412+0000","last_change":"2024-09-13T13:04:02.765069+0000","last_active":"2024-09-13T13:05:15.321412+0000","last_peered":"2024-09-13T13:05:15.321412+0000","last_clean":"2024-09-13T13:05:15.321412+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:05:15.321412+0000","last_undegraded":"2024-09-13T13:05:15.321412+0000","last_fullsized":"2024-09-13T13:05:15.321412+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:46.604669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.604669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"144'11832","reported_seq":17737,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.860249+0000","last_change":"2024-09-13T13:04:49.780269+0000","last_active":"2024-09-13T13:05:15.860249+0000","last_peered":"2024-09-13T13:05:15.860249+0000","last_clean":"2024-09-13T13:05:15.860249+0000","last_became_active":"2024-09-13T13:04:49.765876+0000","last_became_peered":"2024-09-13T13:04:49.765876+0000","last_unstale":"2024-09-13T13:05:15.860249+0000","last_undegraded":"2024-09-13T13:05:15.860249+0000","last_fullsized":"2024-09-13T13:05:15.860249+0000","mapping_epoch":136,"log_start":"108'5748","ondisk_log_start":"108'5748","created":84,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":6084,"log_dups_size":0,"ondisk_log_size":6084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208797696,"num_objects":3186,"num_object_clones":0,"num_object_copies":9558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11832,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2909,"num_bytes_recovered":147193856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":210,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.325256+0000","last_change":"2024-09-13T13:02:05.179720+0000","last_active":"2024-09-13T13:05:15.325256+0000","last_peered":"2024-09-13T13:05:15.325256+0000","last_clean":"2024-09-13T13:05:15.325256+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:05:15.325256+0000","last_undegraded":"2024-09-13T13:05:15.325256+0000","last_fullsized":"2024-09-13T13:05:15.325256+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:02:05.179667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:05.179667+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":196,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.323512+0000","last_change":"2024-09-13T13:03:42.443399+0000","last_active":"2024-09-13T13:05:15.323512+0000","last_peered":"2024-09-13T13:05:15.323512+0000","last_clean":"2024-09-13T13:05:15.323512+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:05:15.323512+0000","last_undegraded":"2024-09-13T13:05:15.323512+0000","last_fullsized":"2024-09-13T13:05:15.323512+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:42.443339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:42.443339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":241,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353164+0000","last_change":"2024-09-13T13:02:32.013587+0000","last_active":"2024-09-13T13:05:15.353164+0000","last_peered":"2024-09-13T13:05:15.353164+0000","last_clean":"2024-09-13T13:05:15.353164+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:05:15.353164+0000","last_undegraded":"2024-09-13T13:05:15.353164+0000","last_fullsized":"2024-09-13T13:05:15.353164+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:32.013505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.013505+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:14.086911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":176,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.351834+0000","last_change":"2024-09-13T13:04:48.684906+0000","last_active":"2024-09-13T13:05:15.351834+0000","last_peered":"2024-09-13T13:05:15.351834+0000","last_clean":"2024-09-13T13:05:15.351834+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:05:15.351834+0000","last_undegraded":"2024-09-13T13:05:15.351834+0000","last_fullsized":"2024-09-13T13:05:15.351834+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:48.684864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.684864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.491555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"144'11958","reported_seq":12167,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:18.176706+0000","last_change":"2024-09-13T13:03:41.936543+0000","last_active":"2024-09-13T13:05:18.176706+0000","last_peered":"2024-09-13T13:05:18.176706+0000","last_clean":"2024-09-13T13:05:18.176706+0000","last_became_active":"2024-09-13T13:02:12.369723+0000","last_became_peered":"2024-09-13T13:02:12.369723+0000","last_unstale":"2024-09-13T13:05:18.176706+0000","last_undegraded":"2024-09-13T13:05:18.176706+0000","last_fullsized":"2024-09-13T13:05:18.176706+0000","mapping_epoch":84,"log_start":"115'7300","ondisk_log_start":"115'7300","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6909","last_scrub_stamp":"2024-09-13T13:03:41.936218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:03:41.936218+0000","objects_scrubbed":6836,"log_size":4658,"log_dups_size":0,"ondisk_log_size":4658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207749120,"num_objects":3170,"num_object_clones":0,"num_object_copies":9510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11958,"num_write_kb":484096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":217,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.323386+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:05:15.323386+0000","last_peered":"2024-09-13T13:05:15.323386+0000","last_clean":"2024-09-13T13:05:15.323386+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:05:15.323386+0000","last_undegraded":"2024-09-13T13:05:15.323386+0000","last_fullsized":"2024-09-13T13:05:15.323386+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":194,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.324486+0000","last_change":"2024-09-13T13:03:18.516327+0000","last_active":"2024-09-13T13:05:15.324486+0000","last_peered":"2024-09-13T13:05:15.324486+0000","last_clean":"2024-09-13T13:05:15.324486+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:05:15.324486+0000","last_undegraded":"2024-09-13T13:05:15.324486+0000","last_fullsized":"2024-09-13T13:05:15.324486+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:18.516251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.516251+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":182,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.351652+0000","last_change":"2024-09-13T13:02:36.005760+0000","last_active":"2024-09-13T13:05:15.351652+0000","last_peered":"2024-09-13T13:05:15.351652+0000","last_clean":"2024-09-13T13:05:15.351652+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:05:15.351652+0000","last_undegraded":"2024-09-13T13:05:15.351652+0000","last_fullsized":"2024-09-13T13:05:15.351652+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.005697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.005697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.351617+0000","last_change":"2024-09-13T13:04:53.006515+0000","last_active":"2024-09-13T13:05:15.351617+0000","last_peered":"2024-09-13T13:05:15.351617+0000","last_clean":"2024-09-13T13:05:15.351617+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:05:15.351617+0000","last_undegraded":"2024-09-13T13:05:15.351617+0000","last_fullsized":"2024-09-13T13:05:15.351617+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:53.006454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.006454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.531241+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"144'11959","reported_seq":12097,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:18.194763+0000","last_change":"2024-09-13T13:03:45.761068+0000","last_active":"2024-09-13T13:05:18.194763+0000","last_peered":"2024-09-13T13:05:18.194763+0000","last_clean":"2024-09-13T13:05:18.194763+0000","last_became_active":"2024-09-13T13:02:12.373492+0000","last_became_peered":"2024-09-13T13:02:12.373492+0000","last_unstale":"2024-09-13T13:05:18.194763+0000","last_undegraded":"2024-09-13T13:05:18.194763+0000","last_fullsized":"2024-09-13T13:05:18.194763+0000","mapping_epoch":84,"log_start":"115'7300","ondisk_log_start":"115'7300","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":299,"log_size":4659,"log_dups_size":0,"ondisk_log_size":4659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202964992,"num_objects":3097,"num_object_clones":0,"num_object_copies":9291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11959,"num_write_kb":481792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":218,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.351735+0000","last_change":"2024-09-13T13:04:02.772914+0000","last_active":"2024-09-13T13:05:15.351735+0000","last_peered":"2024-09-13T13:05:15.351735+0000","last_clean":"2024-09-13T13:05:15.351735+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:05:15.351735+0000","last_undegraded":"2024-09-13T13:05:15.351735+0000","last_fullsized":"2024-09-13T13:05:15.351735+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:02:43.947615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.947615+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":189,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.326880+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:05:15.326880+0000","last_peered":"2024-09-13T13:05:15.326880+0000","last_clean":"2024-09-13T13:05:15.326880+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:05:15.326880+0000","last_undegraded":"2024-09-13T13:05:15.326880+0000","last_fullsized":"2024-09-13T13:05:15.326880+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":233,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.323168+0000","last_change":"2024-09-13T13:01:06.211211+0000","last_active":"2024-09-13T13:05:15.323168+0000","last_peered":"2024-09-13T13:05:15.323168+0000","last_clean":"2024-09-13T13:05:15.323168+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:05:15.323168+0000","last_undegraded":"2024-09-13T13:05:15.323168+0000","last_fullsized":"2024-09-13T13:05:15.323168+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":187,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.354068+0000","last_change":"2024-09-13T13:04:57.439116+0000","last_active":"2024-09-13T13:05:15.354068+0000","last_peered":"2024-09-13T13:05:15.354068+0000","last_clean":"2024-09-13T13:05:15.354068+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:05:15.354068+0000","last_undegraded":"2024-09-13T13:05:15.354068+0000","last_fullsized":"2024-09-13T13:05:15.354068+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.439053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.439053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:04.992285+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"144'11968","reported_seq":17946,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:17.911588+0000","last_change":"2024-09-13T13:05:05.014474+0000","last_active":"2024-09-13T13:05:17.911588+0000","last_peered":"2024-09-13T13:05:17.911588+0000","last_clean":"2024-09-13T13:05:17.911588+0000","last_became_active":"2024-09-13T13:04:25.922485+0000","last_became_peered":"2024-09-13T13:04:25.922485+0000","last_unstale":"2024-09-13T13:05:17.911588+0000","last_undegraded":"2024-09-13T13:05:17.911588+0000","last_fullsized":"2024-09-13T13:05:17.911588+0000","mapping_epoch":128,"log_start":"115'7200","ondisk_log_start":"115'7200","created":84,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'11284","last_scrub_stamp":"2024-09-13T13:05:05.014171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:05.014171+0000","objects_scrubbed":3783,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:18.664629+0000","scrub_duration":934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":3072,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11968,"num_write_kb":481280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2891,"num_bytes_recovered":148570112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":193,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.325104+0000","last_change":"2024-09-13T13:01:04.199011+0000","last_active":"2024-09-13T13:05:15.325104+0000","last_peered":"2024-09-13T13:05:15.325104+0000","last_clean":"2024-09-13T13:05:15.325104+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:05:15.325104+0000","last_undegraded":"2024-09-13T13:05:15.325104+0000","last_fullsized":"2024-09-13T13:05:15.325104+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":189,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352364+0000","last_change":"2024-09-13T13:05:04.743102+0000","last_active":"2024-09-13T13:05:15.352364+0000","last_peered":"2024-09-13T13:05:15.352364+0000","last_clean":"2024-09-13T13:05:15.352364+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:05:15.352364+0000","last_undegraded":"2024-09-13T13:05:15.352364+0000","last_fullsized":"2024-09-13T13:05:15.352364+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:04.743025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:04.743025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:32.082984+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":247,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:17.288513+0000","last_change":"2024-09-13T13:04:02.764350+0000","last_active":"2024-09-13T13:05:17.288513+0000","last_peered":"2024-09-13T13:05:17.288513+0000","last_clean":"2024-09-13T13:05:17.288513+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:05:17.288513+0000","last_undegraded":"2024-09-13T13:05:17.288513+0000","last_fullsized":"2024-09-13T13:05:17.288513+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.325834+0000","last_change":"2024-09-13T13:02:43.558487+0000","last_active":"2024-09-13T13:05:15.325834+0000","last_peered":"2024-09-13T13:05:15.325834+0000","last_clean":"2024-09-13T13:05:15.325834+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:05:15.325834+0000","last_undegraded":"2024-09-13T13:05:15.325834+0000","last_fullsized":"2024-09-13T13:05:15.325834+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:43.558418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.558418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"144'11807","reported_seq":11939,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:17.872292+0000","last_change":"2024-09-13T13:02:12.379536+0000","last_active":"2024-09-13T13:05:17.872292+0000","last_peered":"2024-09-13T13:05:17.872292+0000","last_clean":"2024-09-13T13:05:17.872292+0000","last_became_active":"2024-09-13T13:02:12.379226+0000","last_became_peered":"2024-09-13T13:02:12.379226+0000","last_unstale":"2024-09-13T13:05:17.872292+0000","last_undegraded":"2024-09-13T13:05:17.872292+0000","last_fullsized":"2024-09-13T13:05:17.872292+0000","mapping_epoch":84,"log_start":"108'6000","ondisk_log_start":"108'6000","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5807,"log_dups_size":0,"ondisk_log_size":5807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.567939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205324288,"num_objects":3133,"num_object_clones":0,"num_object_copies":9399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11807,"num_write_kb":478080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":221,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.326265+0000","last_change":"2024-09-13T13:02:29.532879+0000","last_active":"2024-09-13T13:05:15.326265+0000","last_peered":"2024-09-13T13:05:15.326265+0000","last_clean":"2024-09-13T13:05:15.326265+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:05:15.326265+0000","last_undegraded":"2024-09-13T13:05:15.326265+0000","last_fullsized":"2024-09-13T13:05:15.326265+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:29.532779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.532779+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":182,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.324920+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:05:15.324920+0000","last_peered":"2024-09-13T13:05:15.324920+0000","last_clean":"2024-09-13T13:05:15.324920+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:05:15.324920+0000","last_undegraded":"2024-09-13T13:05:15.324920+0000","last_fullsized":"2024-09-13T13:05:15.324920+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":142,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.320840+0000","last_change":"2024-09-13T13:04:02.765838+0000","last_active":"2024-09-13T13:05:15.320840+0000","last_peered":"2024-09-13T13:05:15.320840+0000","last_clean":"2024-09-13T13:05:15.320840+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:05:15.320840+0000","last_undegraded":"2024-09-13T13:05:15.320840+0000","last_fullsized":"2024-09-13T13:05:15.320840+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.563708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.563708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":189,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.323967+0000","last_change":"2024-09-13T13:04:02.749391+0000","last_active":"2024-09-13T13:05:15.323967+0000","last_peered":"2024-09-13T13:05:15.323967+0000","last_clean":"2024-09-13T13:05:15.323967+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:05:15.323967+0000","last_undegraded":"2024-09-13T13:05:15.323967+0000","last_fullsized":"2024-09-13T13:05:15.323967+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:28.275768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.275768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"144'12102","reported_seq":15129,"reported_epoch":144,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T13:05:18.405789+0000","last_change":"2024-09-13T13:04:47.345676+0000","last_active":"2024-09-13T13:05:18.405789+0000","last_peered":"2024-09-13T13:05:18.405789+0000","last_clean":"2024-09-13T13:03:14.460355+0000","last_became_active":"2024-09-13T13:04:03.878596+0000","last_became_peered":"2024-09-13T13:04:03.878596+0000","last_unstale":"2024-09-13T13:05:18.405789+0000","last_undegraded":"2024-09-13T13:04:03.760406+0000","last_fullsized":"2024-09-13T13:04:03.744969+0000","mapping_epoch":119,"log_start":"111'6500","ondisk_log_start":"111'6500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5602,"log_dups_size":0,"ondisk_log_size":5602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201195520,"num_objects":3070,"num_object_clones":0,"num_object_copies":9210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12102,"num_write_kb":485504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":94699520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":2247}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":286,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:17.286689+0000","last_change":"2024-09-13T13:02:39.183839+0000","last_active":"2024-09-13T13:05:17.286689+0000","last_peered":"2024-09-13T13:05:17.286689+0000","last_clean":"2024-09-13T13:05:17.286689+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:05:17.286689+0000","last_undegraded":"2024-09-13T13:05:17.286689+0000","last_fullsized":"2024-09-13T13:05:17.286689+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:02:39.183740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.183740+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":201,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352434+0000","last_change":"2024-09-13T13:04:03.850983+0000","last_active":"2024-09-13T13:05:15.352434+0000","last_peered":"2024-09-13T13:05:15.352434+0000","last_clean":"2024-09-13T13:05:15.352434+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:05:15.352434+0000","last_undegraded":"2024-09-13T13:05:15.352434+0000","last_fullsized":"2024-09-13T13:05:15.352434+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.850898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.850898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":208,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353532+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:05:15.353532+0000","last_peered":"2024-09-13T13:05:15.353532+0000","last_clean":"2024-09-13T13:05:15.353532+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:05:15.353532+0000","last_undegraded":"2024-09-13T13:05:15.353532+0000","last_fullsized":"2024-09-13T13:05:15.353532+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:26.941315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":180,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.326339+0000","last_change":"2024-09-13T13:02:35.566666+0000","last_active":"2024-09-13T13:05:15.326339+0000","last_peered":"2024-09-13T13:05:15.326339+0000","last_clean":"2024-09-13T13:05:15.326339+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:05:15.326339+0000","last_undegraded":"2024-09-13T13:05:15.326339+0000","last_fullsized":"2024-09-13T13:05:15.326339+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:35.566602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:35.566602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":176,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.323052+0000","last_change":"2024-09-13T13:04:01.406226+0000","last_active":"2024-09-13T13:05:15.323052+0000","last_peered":"2024-09-13T13:05:15.323052+0000","last_clean":"2024-09-13T13:05:15.323052+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:05:15.323052+0000","last_undegraded":"2024-09-13T13:05:15.323052+0000","last_fullsized":"2024-09-13T13:05:15.323052+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:01.406167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:01.406167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:24.974510+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"144'12042","reported_seq":12174,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:18.421083+0000","last_change":"2024-09-13T13:02:12.380398+0000","last_active":"2024-09-13T13:05:18.421083+0000","last_peered":"2024-09-13T13:05:18.421083+0000","last_clean":"2024-09-13T13:05:18.421083+0000","last_became_active":"2024-09-13T13:02:12.379331+0000","last_became_peered":"2024-09-13T13:02:12.379331+0000","last_unstale":"2024-09-13T13:05:18.421083+0000","last_undegraded":"2024-09-13T13:05:18.421083+0000","last_fullsized":"2024-09-13T13:05:18.421083+0000","mapping_epoch":84,"log_start":"110'6400","ondisk_log_start":"110'6400","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5642,"log_dups_size":0,"ondisk_log_size":5642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199491584,"num_objects":3044,"num_object_clones":0,"num_object_copies":9132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12042,"num_write_kb":482752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":212,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.325313+0000","last_change":"2024-09-13T13:03:14.271569+0000","last_active":"2024-09-13T13:05:15.325313+0000","last_peered":"2024-09-13T13:05:15.325313+0000","last_clean":"2024-09-13T13:05:15.325313+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:05:15.325313+0000","last_undegraded":"2024-09-13T13:05:15.325313+0000","last_fullsized":"2024-09-13T13:05:15.325313+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:03:14.271480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.271480+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":201,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.324593+0000","last_change":"2024-09-13T13:04:02.759283+0000","last_active":"2024-09-13T13:05:15.324593+0000","last_peered":"2024-09-13T13:05:15.324593+0000","last_clean":"2024-09-13T13:05:15.324593+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:05:15.324593+0000","last_undegraded":"2024-09-13T13:05:15.324593+0000","last_fullsized":"2024-09-13T13:05:15.324593+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:40.214163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:40.214163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":197,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321554+0000","last_change":"2024-09-13T13:03:18.232951+0000","last_active":"2024-09-13T13:05:15.321554+0000","last_peered":"2024-09-13T13:05:15.321554+0000","last_clean":"2024-09-13T13:05:15.321554+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:05:15.321554+0000","last_undegraded":"2024-09-13T13:05:15.321554+0000","last_fullsized":"2024-09-13T13:05:15.321554+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:03:18.232867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.232867+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:27.894650+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":181,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.322471+0000","last_change":"2024-09-13T13:03:14.282866+0000","last_active":"2024-09-13T13:05:15.322471+0000","last_peered":"2024-09-13T13:05:15.322471+0000","last_clean":"2024-09-13T13:05:15.322471+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:05:15.322471+0000","last_undegraded":"2024-09-13T13:05:15.322471+0000","last_fullsized":"2024-09-13T13:05:15.322471+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:20.810360+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":187,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.355539+0000","last_change":"2024-09-13T13:04:02.751582+0000","last_active":"2024-09-13T13:05:15.355539+0000","last_peered":"2024-09-13T13:05:15.355539+0000","last_clean":"2024-09-13T13:05:15.355539+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:05:15.355539+0000","last_undegraded":"2024-09-13T13:05:15.355539+0000","last_fullsized":"2024-09-13T13:05:15.355539+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.994729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.994729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"144'12192","reported_seq":12398,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:17.868834+0000","last_change":"2024-09-13T13:03:37.311041+0000","last_active":"2024-09-13T13:05:17.868834+0000","last_peered":"2024-09-13T13:05:17.868834+0000","last_clean":"2024-09-13T13:05:17.868834+0000","last_became_active":"2024-09-13T13:02:12.379779+0000","last_became_peered":"2024-09-13T13:02:12.379779+0000","last_unstale":"2024-09-13T13:05:17.868834+0000","last_undegraded":"2024-09-13T13:05:17.868834+0000","last_fullsized":"2024-09-13T13:05:17.868834+0000","mapping_epoch":84,"log_start":"114'7000","ondisk_log_start":"114'7000","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'6772","last_scrub_stamp":"2024-09-13T13:03:37.310903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:03:37.310903+0000","objects_scrubbed":6730,"log_size":5192,"log_dups_size":0,"ondisk_log_size":5192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209321984,"num_objects":3194,"num_object_clones":0,"num_object_copies":9582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12192,"num_write_kb":492352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":202,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.326785+0000","last_change":"2024-09-13T13:03:18.514436+0000","last_active":"2024-09-13T13:05:15.326785+0000","last_peered":"2024-09-13T13:05:15.326785+0000","last_clean":"2024-09-13T13:05:15.326785+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:05:15.326785+0000","last_undegraded":"2024-09-13T13:05:15.326785+0000","last_fullsized":"2024-09-13T13:05:15.326785+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:03:18.514360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.514360+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":189,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.326172+0000","last_change":"2024-09-13T13:02:37.616313+0000","last_active":"2024-09-13T13:05:15.326172+0000","last_peered":"2024-09-13T13:05:15.326172+0000","last_clean":"2024-09-13T13:05:15.326172+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:05:15.326172+0000","last_undegraded":"2024-09-13T13:05:15.326172+0000","last_fullsized":"2024-09-13T13:05:15.326172+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:37.616224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.616224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":448,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:16.732312+0000","last_change":"2024-09-13T13:04:56.430717+0000","last_active":"2024-09-13T13:05:16.732312+0000","last_peered":"2024-09-13T13:05:16.732312+0000","last_clean":"2024-09-13T13:05:16.732312+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:05:16.732312+0000","last_undegraded":"2024-09-13T13:05:16.732312+0000","last_fullsized":"2024-09-13T13:05:16.732312+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:04:56.430670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.430670+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:59.639039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":250,"num_read_kb":245,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":180,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352782+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:05:15.352782+0000","last_peered":"2024-09-13T13:05:15.352782+0000","last_clean":"2024-09-13T13:05:15.352782+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:05:15.352782+0000","last_undegraded":"2024-09-13T13:05:15.352782+0000","last_fullsized":"2024-09-13T13:05:15.352782+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:19.946825+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.324353+0000","last_change":"2024-09-13T13:02:36.222607+0000","last_active":"2024-09-13T13:05:15.324353+0000","last_peered":"2024-09-13T13:05:15.324353+0000","last_clean":"2024-09-13T13:05:15.324353+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:05:15.324353+0000","last_undegraded":"2024-09-13T13:05:15.324353+0000","last_fullsized":"2024-09-13T13:05:15.324353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.222329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"144'12075","reported_seq":12206,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:18.406786+0000","last_change":"2024-09-13T13:02:12.379556+0000","last_active":"2024-09-13T13:05:18.406786+0000","last_peered":"2024-09-13T13:05:18.406786+0000","last_clean":"2024-09-13T13:05:18.406786+0000","last_became_active":"2024-09-13T13:02:12.379255+0000","last_became_peered":"2024-09-13T13:02:12.379255+0000","last_unstale":"2024-09-13T13:05:18.406786+0000","last_undegraded":"2024-09-13T13:05:18.406786+0000","last_fullsized":"2024-09-13T13:05:18.406786+0000","mapping_epoch":84,"log_start":"110'6500","ondisk_log_start":"110'6500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5575,"log_dups_size":0,"ondisk_log_size":5575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199950336,"num_objects":3051,"num_object_clones":0,"num_object_copies":9153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12075,"num_write_kb":484032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":125,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321211+0000","last_change":"2024-09-13T13:04:02.786042+0000","last_active":"2024-09-13T13:05:15.321211+0000","last_peered":"2024-09-13T13:05:15.321211+0000","last_clean":"2024-09-13T13:05:15.321211+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:05:15.321211+0000","last_undegraded":"2024-09-13T13:05:15.321211+0000","last_fullsized":"2024-09-13T13:05:15.321211+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":200,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.322810+0000","last_change":"2024-09-13T13:04:02.765370+0000","last_active":"2024-09-13T13:05:15.322810+0000","last_peered":"2024-09-13T13:05:15.322810+0000","last_clean":"2024-09-13T13:05:15.322810+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:05:15.322810+0000","last_undegraded":"2024-09-13T13:05:15.322810+0000","last_fullsized":"2024-09-13T13:05:15.322810+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:04.228002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:04.228002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":237,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353939+0000","last_change":"2024-09-13T13:04:02.753245+0000","last_active":"2024-09-13T13:05:15.353939+0000","last_peered":"2024-09-13T13:05:15.353939+0000","last_clean":"2024-09-13T13:05:15.353939+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:05:15.353939+0000","last_undegraded":"2024-09-13T13:05:15.353939+0000","last_fullsized":"2024-09-13T13:05:15.353939+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:03:09.054664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:03:09.054664+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":181,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.355441+0000","last_change":"2024-09-13T13:04:51.395273+0000","last_active":"2024-09-13T13:05:15.355441+0000","last_peered":"2024-09-13T13:05:15.355441+0000","last_clean":"2024-09-13T13:05:15.355441+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:05:15.355441+0000","last_undegraded":"2024-09-13T13:05:15.355441+0000","last_fullsized":"2024-09-13T13:05:15.355441+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:51.395204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:51.395204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:56.927380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":176,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352650+0000","last_change":"2024-09-13T13:02:39.434636+0000","last_active":"2024-09-13T13:05:15.352650+0000","last_peered":"2024-09-13T13:05:15.352650+0000","last_clean":"2024-09-13T13:05:15.352650+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:05:15.352650+0000","last_undegraded":"2024-09-13T13:05:15.352650+0000","last_fullsized":"2024-09-13T13:05:15.352650+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.434575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.434575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"144'11980","reported_seq":18146,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.849751+0000","last_change":"2024-09-13T13:05:15.349567+0000","last_active":"2024-09-13T13:05:15.849751+0000","last_peered":"2024-09-13T13:05:15.849751+0000","last_clean":"2024-09-13T13:05:15.849751+0000","last_became_active":"2024-09-13T13:05:15.326923+0000","last_became_peered":"2024-09-13T13:05:15.326923+0000","last_unstale":"2024-09-13T13:05:15.849751+0000","last_undegraded":"2024-09-13T13:05:15.849751+0000","last_fullsized":"2024-09-13T13:05:15.849751+0000","mapping_epoch":143,"log_start":"110'6285","ondisk_log_start":"110'6285","created":84,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5695,"log_dups_size":0,"ondisk_log_size":5695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217645080,"num_objects":3322,"num_object_clones":0,"num_object_copies":9966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11980,"num_write_kb":489601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3008,"num_bytes_recovered":155058200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":215,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353211+0000","last_change":"2024-09-13T13:04:02.832992+0000","last_active":"2024-09-13T13:05:15.353211+0000","last_peered":"2024-09-13T13:05:15.353211+0000","last_clean":"2024-09-13T13:05:15.353211+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:05:15.353211+0000","last_undegraded":"2024-09-13T13:05:15.353211+0000","last_fullsized":"2024-09-13T13:05:15.353211+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:04:02.832935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.832935+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":189,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352077+0000","last_change":"2024-09-13T13:02:45.015634+0000","last_active":"2024-09-13T13:05:15.352077+0000","last_peered":"2024-09-13T13:05:15.352077+0000","last_clean":"2024-09-13T13:05:15.352077+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:05:15.352077+0000","last_undegraded":"2024-09-13T13:05:15.352077+0000","last_fullsized":"2024-09-13T13:05:15.352077+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.015577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.015577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":191,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352528+0000","last_change":"2024-09-13T13:04:47.680651+0000","last_active":"2024-09-13T13:05:15.352528+0000","last_peered":"2024-09-13T13:05:15.352528+0000","last_clean":"2024-09-13T13:05:15.352528+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:05:15.352528+0000","last_undegraded":"2024-09-13T13:05:15.352528+0000","last_fullsized":"2024-09-13T13:05:15.352528+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.680579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.680579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:07.566204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":187,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.322139+0000","last_change":"2024-09-13T13:04:02.757378+0000","last_active":"2024-09-13T13:05:15.322139+0000","last_peered":"2024-09-13T13:05:15.322139+0000","last_clean":"2024-09-13T13:05:15.322139+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:05:15.322139+0000","last_undegraded":"2024-09-13T13:05:15.322139+0000","last_fullsized":"2024-09-13T13:05:15.322139+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.213470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.213470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"144'11627","reported_seq":11884,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:18.075911+0000","last_change":"2024-09-13T13:05:03.355522+0000","last_active":"2024-09-13T13:05:18.075911+0000","last_peered":"2024-09-13T13:05:18.075911+0000","last_clean":"2024-09-13T13:05:18.075911+0000","last_became_active":"2024-09-13T13:02:12.375509+0000","last_became_peered":"2024-09-13T13:02:12.375509+0000","last_unstale":"2024-09-13T13:05:18.075911+0000","last_undegraded":"2024-09-13T13:05:18.075911+0000","last_fullsized":"2024-09-13T13:05:18.075911+0000","mapping_epoch":84,"log_start":"101'4800","ondisk_log_start":"101'4800","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'10840","last_scrub_stamp":"2024-09-13T13:05:03.354789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:03.354789+0000","objects_scrubbed":3912,"log_size":6827,"log_dups_size":0,"ondisk_log_size":6827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.025217+0000","scrub_duration":2101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201261056,"num_objects":3071,"num_object_clones":0,"num_object_copies":9213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11627,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":213,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352409+0000","last_change":"2024-09-13T13:03:12.028812+0000","last_active":"2024-09-13T13:05:15.352409+0000","last_peered":"2024-09-13T13:05:15.352409+0000","last_clean":"2024-09-13T13:05:15.352409+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:05:15.352409+0000","last_undegraded":"2024-09-13T13:05:15.352409+0000","last_fullsized":"2024-09-13T13:05:15.352409+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:03:12.028719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.028719+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":158,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321009+0000","last_change":"2024-09-13T13:04:02.764657+0000","last_active":"2024-09-13T13:05:15.321009+0000","last_peered":"2024-09-13T13:05:15.321009+0000","last_clean":"2024-09-13T13:05:15.321009+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:05:15.321009+0000","last_undegraded":"2024-09-13T13:05:15.321009+0000","last_fullsized":"2024-09-13T13:05:15.321009+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":173,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.324496+0000","last_change":"2024-09-13T13:01:06.208593+0000","last_active":"2024-09-13T13:05:15.324496+0000","last_peered":"2024-09-13T13:05:15.324496+0000","last_clean":"2024-09-13T13:05:15.324496+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:05:15.324496+0000","last_undegraded":"2024-09-13T13:05:15.324496+0000","last_fullsized":"2024-09-13T13:05:15.324496+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.322170+0000","last_change":"2024-09-13T13:03:22.282376+0000","last_active":"2024-09-13T13:05:15.322170+0000","last_peered":"2024-09-13T13:05:15.322170+0000","last_clean":"2024-09-13T13:05:15.322170+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:05:15.322170+0000","last_undegraded":"2024-09-13T13:05:15.322170+0000","last_fullsized":"2024-09-13T13:05:15.322170+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:22.282308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.282308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:32.881087+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"144'12088","reported_seq":12297,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:18.413119+0000","last_change":"2024-09-13T13:04:02.904392+0000","last_active":"2024-09-13T13:05:18.413119+0000","last_peered":"2024-09-13T13:05:18.413119+0000","last_clean":"2024-09-13T13:05:18.413119+0000","last_became_active":"2024-09-13T13:02:12.380229+0000","last_became_peered":"2024-09-13T13:02:12.380229+0000","last_unstale":"2024-09-13T13:05:18.413119+0000","last_undegraded":"2024-09-13T13:05:18.413119+0000","last_fullsized":"2024-09-13T13:05:18.413119+0000","mapping_epoch":84,"log_start":"111'6500","ondisk_log_start":"111'6500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8189","last_scrub_stamp":"2024-09-13T13:04:02.904297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.904297+0000","objects_scrubbed":6969,"log_size":5588,"log_dups_size":0,"ondisk_log_size":5588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200802304,"num_objects":3064,"num_object_clones":0,"num_object_copies":9192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12088,"num_write_kb":484864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":211,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352683+0000","last_change":"2024-09-13T13:04:59.091852+0000","last_active":"2024-09-13T13:05:15.352683+0000","last_peered":"2024-09-13T13:05:15.352683+0000","last_clean":"2024-09-13T13:05:15.352683+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:05:15.352683+0000","last_undegraded":"2024-09-13T13:05:15.352683+0000","last_fullsized":"2024-09-13T13:05:15.352683+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:04:59.091751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.091751+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.396141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":196,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321663+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:05:15.321663+0000","last_peered":"2024-09-13T13:05:15.321663+0000","last_clean":"2024-09-13T13:05:15.321663+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:05:15.321663+0000","last_undegraded":"2024-09-13T13:05:15.321663+0000","last_fullsized":"2024-09-13T13:05:15.321663+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":241,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.320952+0000","last_change":"2024-09-13T13:03:15.254020+0000","last_active":"2024-09-13T13:05:15.320952+0000","last_peered":"2024-09-13T13:05:15.320952+0000","last_clean":"2024-09-13T13:05:15.320952+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:05:15.320952+0000","last_undegraded":"2024-09-13T13:05:15.320952+0000","last_fullsized":"2024-09-13T13:05:15.320952+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:03:15.253968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:15.253968+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:21.804124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352973+0000","last_change":"2024-09-13T13:02:33.997780+0000","last_active":"2024-09-13T13:05:15.352973+0000","last_peered":"2024-09-13T13:05:15.352973+0000","last_clean":"2024-09-13T13:05:15.352973+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:05:15.352973+0000","last_undegraded":"2024-09-13T13:05:15.352973+0000","last_fullsized":"2024-09-13T13:05:15.352973+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.997740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.997740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"144'12050","reported_seq":12305,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:18.057644+0000","last_change":"2024-09-13T13:05:08.464991+0000","last_active":"2024-09-13T13:05:18.057644+0000","last_peered":"2024-09-13T13:05:18.057644+0000","last_clean":"2024-09-13T13:05:18.057644+0000","last_became_active":"2024-09-13T13:02:12.375310+0000","last_became_peered":"2024-09-13T13:02:12.375310+0000","last_unstale":"2024-09-13T13:05:18.057644+0000","last_undegraded":"2024-09-13T13:05:18.057644+0000","last_fullsized":"2024-09-13T13:05:18.057644+0000","mapping_epoch":84,"log_start":"102'5200","ondisk_log_start":"102'5200","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'11530","last_scrub_stamp":"2024-09-13T13:05:08.464709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:08.464709+0000","objects_scrubbed":3682,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:11.110516+0000","scrub_duration":1226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204865536,"num_objects":3126,"num_object_clones":0,"num_object_copies":9378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12050,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":200,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.326217+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:05:15.326217+0000","last_peered":"2024-09-13T13:05:15.326217+0000","last_clean":"2024-09-13T13:05:15.326217+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:05:15.326217+0000","last_undegraded":"2024-09-13T13:05:15.326217+0000","last_fullsized":"2024-09-13T13:05:15.326217+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":199,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352769+0000","last_change":"2024-09-13T13:04:02.748907+0000","last_active":"2024-09-13T13:05:15.352769+0000","last_peered":"2024-09-13T13:05:15.352769+0000","last_clean":"2024-09-13T13:05:15.352769+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:05:15.352769+0000","last_undegraded":"2024-09-13T13:05:15.352769+0000","last_fullsized":"2024-09-13T13:05:15.352769+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.305843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.305843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":189,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321367+0000","last_change":"2024-09-13T13:04:56.183494+0000","last_active":"2024-09-13T13:05:15.321367+0000","last_peered":"2024-09-13T13:05:15.321367+0000","last_clean":"2024-09-13T13:05:15.321367+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:05:15.321367+0000","last_undegraded":"2024-09-13T13:05:15.321367+0000","last_fullsized":"2024-09-13T13:05:15.321367+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:56.183290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.183290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:07.795607+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":168,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.326077+0000","last_change":"2024-09-13T13:01:08.211229+0000","last_active":"2024-09-13T13:05:15.326077+0000","last_peered":"2024-09-13T13:05:15.326077+0000","last_clean":"2024-09-13T13:05:15.326077+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:05:15.326077+0000","last_undegraded":"2024-09-13T13:05:15.326077+0000","last_fullsized":"2024-09-13T13:05:15.326077+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"144'12165","reported_seq":12297,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:18.385122+0000","last_change":"2024-09-13T13:02:12.379755+0000","last_active":"2024-09-13T13:05:18.385122+0000","last_peered":"2024-09-13T13:05:18.385122+0000","last_clean":"2024-09-13T13:05:18.385122+0000","last_became_active":"2024-09-13T13:02:12.379442+0000","last_became_peered":"2024-09-13T13:02:12.379442+0000","last_unstale":"2024-09-13T13:05:18.385122+0000","last_undegraded":"2024-09-13T13:05:18.385122+0000","last_fullsized":"2024-09-13T13:05:18.385122+0000","mapping_epoch":84,"log_start":"111'6600","ondisk_log_start":"111'6600","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205848576,"num_objects":3141,"num_object_clones":0,"num_object_copies":9423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":489792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.320882+0000","last_change":"2024-09-13T13:04:02.760376+0000","last_active":"2024-09-13T13:05:15.320882+0000","last_peered":"2024-09-13T13:05:15.320882+0000","last_clean":"2024-09-13T13:05:15.320882+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:05:15.320882+0000","last_undegraded":"2024-09-13T13:05:15.320882+0000","last_fullsized":"2024-09-13T13:05:15.320882+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:42.593284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.593284+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":199,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.355845+0000","last_change":"2024-09-13T13:04:02.759538+0000","last_active":"2024-09-13T13:05:15.355845+0000","last_peered":"2024-09-13T13:05:15.355845+0000","last_clean":"2024-09-13T13:05:15.355845+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:05:15.355845+0000","last_undegraded":"2024-09-13T13:05:15.355845+0000","last_fullsized":"2024-09-13T13:05:15.355845+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:29.012312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.012312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:13.429060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":189,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321850+0000","last_change":"2024-09-13T13:04:47.927556+0000","last_active":"2024-09-13T13:05:15.321850+0000","last_peered":"2024-09-13T13:05:15.321850+0000","last_clean":"2024-09-13T13:05:15.321850+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:05:15.321850+0000","last_undegraded":"2024-09-13T13:05:15.321850+0000","last_fullsized":"2024-09-13T13:05:15.321850+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.927432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.927432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.726893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":176,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353855+0000","last_change":"2024-09-13T13:05:11.420568+0000","last_active":"2024-09-13T13:05:15.353855+0000","last_peered":"2024-09-13T13:05:15.353855+0000","last_clean":"2024-09-13T13:05:15.353855+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:05:15.353855+0000","last_undegraded":"2024-09-13T13:05:15.353855+0000","last_fullsized":"2024-09-13T13:05:15.353855+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:11.420503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:11.420503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:12.545408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"144'12016","reported_seq":12200,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:18.199507+0000","last_change":"2024-09-13T13:03:45.723433+0000","last_active":"2024-09-13T13:05:18.199507+0000","last_peered":"2024-09-13T13:05:18.199507+0000","last_clean":"2024-09-13T13:05:18.199507+0000","last_became_active":"2024-09-13T13:02:12.376334+0000","last_became_peered":"2024-09-13T13:02:12.376334+0000","last_unstale":"2024-09-13T13:05:18.199507+0000","last_undegraded":"2024-09-13T13:05:18.199507+0000","last_fullsized":"2024-09-13T13:05:18.199507+0000","mapping_epoch":84,"log_start":"115'7400","ondisk_log_start":"115'7400","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":4774,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209977344,"num_objects":3204,"num_object_clones":0,"num_object_copies":9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12016,"num_write_kb":487040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":208,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.351935+0000","last_change":"2024-09-13T13:02:28.990089+0000","last_active":"2024-09-13T13:05:15.351935+0000","last_peered":"2024-09-13T13:05:15.351935+0000","last_clean":"2024-09-13T13:05:15.351935+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:05:15.351935+0000","last_undegraded":"2024-09-13T13:05:15.351935+0000","last_fullsized":"2024-09-13T13:05:15.351935+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:02:28.990005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.990005+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":196,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.356062+0000","last_change":"2024-09-13T13:03:37.736138+0000","last_active":"2024-09-13T13:05:15.356062+0000","last_peered":"2024-09-13T13:05:15.356062+0000","last_clean":"2024-09-13T13:05:15.356062+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:05:15.356062+0000","last_undegraded":"2024-09-13T13:05:15.356062+0000","last_fullsized":"2024-09-13T13:05:15.356062+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:37.736073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:37.736073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":173,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.320696+0000","last_change":"2024-09-13T13:01:06.211172+0000","last_active":"2024-09-13T13:05:15.320696+0000","last_peered":"2024-09-13T13:05:15.320696+0000","last_clean":"2024-09-13T13:05:15.320696+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:05:15.320696+0000","last_undegraded":"2024-09-13T13:05:15.320696+0000","last_fullsized":"2024-09-13T13:05:15.320696+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:25.864524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":176,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321135+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:05:15.321135+0000","last_peered":"2024-09-13T13:05:15.321135+0000","last_clean":"2024-09-13T13:05:15.321135+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:05:15.321135+0000","last_undegraded":"2024-09-13T13:05:15.321135+0000","last_fullsized":"2024-09-13T13:05:15.321135+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"144'12196","reported_seq":18343,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:18.019665+0000","last_change":"2024-09-13T13:04:48.786053+0000","last_active":"2024-09-13T13:05:18.019665+0000","last_peered":"2024-09-13T13:05:18.019665+0000","last_clean":"2024-09-13T13:05:18.019665+0000","last_became_active":"2024-09-13T13:04:48.765805+0000","last_became_peered":"2024-09-13T13:04:48.765805+0000","last_unstale":"2024-09-13T13:05:18.019665+0000","last_undegraded":"2024-09-13T13:05:18.019665+0000","last_fullsized":"2024-09-13T13:05:18.019665+0000","mapping_epoch":135,"log_start":"114'7100","ondisk_log_start":"114'7100","created":84,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209453056,"num_objects":3196,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12196,"num_write_kb":492544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3000,"num_bytes_recovered":156631040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":220,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.351336+0000","last_change":"2024-09-13T13:04:52.666294+0000","last_active":"2024-09-13T13:05:15.351336+0000","last_peered":"2024-09-13T13:05:15.351336+0000","last_clean":"2024-09-13T13:05:15.351336+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:05:15.351336+0000","last_undegraded":"2024-09-13T13:05:15.351336+0000","last_fullsized":"2024-09-13T13:05:15.351336+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:52.666222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:52.666222+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:57.559648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":200,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.356319+0000","last_change":"2024-09-13T13:02:43.962441+0000","last_active":"2024-09-13T13:05:15.356319+0000","last_peered":"2024-09-13T13:05:15.356319+0000","last_clean":"2024-09-13T13:05:15.356319+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:05:15.356319+0000","last_undegraded":"2024-09-13T13:05:15.356319+0000","last_fullsized":"2024-09-13T13:05:15.356319+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:02:43.962338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.962338+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":257,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353771+0000","last_change":"2024-09-13T13:04:02.759553+0000","last_active":"2024-09-13T13:05:15.353771+0000","last_peered":"2024-09-13T13:05:15.353771+0000","last_clean":"2024-09-13T13:05:15.353771+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:05:15.353771+0000","last_undegraded":"2024-09-13T13:05:15.353771+0000","last_fullsized":"2024-09-13T13:05:15.353771+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:45.959285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.959285+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":187,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353648+0000","last_change":"2024-09-13T13:04:02.757776+0000","last_active":"2024-09-13T13:05:15.353648+0000","last_peered":"2024-09-13T13:05:15.353648+0000","last_clean":"2024-09-13T13:05:15.353648+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:05:15.353648+0000","last_undegraded":"2024-09-13T13:05:15.353648+0000","last_fullsized":"2024-09-13T13:05:15.353648+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:28.529954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.529954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":158,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.320730+0000","last_change":"2024-09-13T13:04:02.756666+0000","last_active":"2024-09-13T13:05:15.320730+0000","last_peered":"2024-09-13T13:05:15.320730+0000","last_clean":"2024-09-13T13:05:15.320730+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:05:15.320730+0000","last_undegraded":"2024-09-13T13:05:15.320730+0000","last_fullsized":"2024-09-13T13:05:15.320730+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":193,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.324019+0000","last_change":"2024-09-13T13:03:14.504339+0000","last_active":"2024-09-13T13:05:15.324019+0000","last_peered":"2024-09-13T13:05:15.324019+0000","last_clean":"2024-09-13T13:05:15.324019+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:05:15.324019+0000","last_undegraded":"2024-09-13T13:05:15.324019+0000","last_fullsized":"2024-09-13T13:05:15.324019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:03:14.504263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.504263+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":180,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353174+0000","last_change":"2024-09-13T13:02:26.581497+0000","last_active":"2024-09-13T13:05:15.353174+0000","last_peered":"2024-09-13T13:05:15.353174+0000","last_clean":"2024-09-13T13:05:15.353174+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:05:15.353174+0000","last_undegraded":"2024-09-13T13:05:15.353174+0000","last_fullsized":"2024-09-13T13:05:15.353174+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:26.581434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:26.581434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":187,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.355336+0000","last_change":"2024-09-13T13:04:03.746106+0000","last_active":"2024-09-13T13:05:15.355336+0000","last_peered":"2024-09-13T13:05:15.355336+0000","last_clean":"2024-09-13T13:05:15.355336+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:05:15.355336+0000","last_undegraded":"2024-09-13T13:05:15.355336+0000","last_fullsized":"2024-09-13T13:05:15.355336+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.745793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.745793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:15.945770+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":200,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.326846+0000","last_change":"2024-09-13T13:02:42.589951+0000","last_active":"2024-09-13T13:05:15.326846+0000","last_peered":"2024-09-13T13:05:15.326846+0000","last_clean":"2024-09-13T13:05:15.326846+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:05:15.326846+0000","last_undegraded":"2024-09-13T13:05:15.326846+0000","last_fullsized":"2024-09-13T13:05:15.326846+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:42.589866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.589866+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":199,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353845+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:05:15.353845+0000","last_peered":"2024-09-13T13:05:15.353845+0000","last_clean":"2024-09-13T13:05:15.353845+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:05:15.353845+0000","last_undegraded":"2024-09-13T13:05:15.353845+0000","last_fullsized":"2024-09-13T13:05:15.353845+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.542132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":172,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.323866+0000","last_change":"2024-09-13T13:01:06.210328+0000","last_active":"2024-09-13T13:05:15.323866+0000","last_peered":"2024-09-13T13:05:15.323866+0000","last_clean":"2024-09-13T13:05:15.323866+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:05:15.323866+0000","last_undegraded":"2024-09-13T13:05:15.323866+0000","last_fullsized":"2024-09-13T13:05:15.323866+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.320840+0000","last_change":"2024-09-13T13:03:21.281950+0000","last_active":"2024-09-13T13:05:15.320840+0000","last_peered":"2024-09-13T13:05:15.320840+0000","last_clean":"2024-09-13T13:05:15.320840+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:05:15.320840+0000","last_undegraded":"2024-09-13T13:05:15.320840+0000","last_fullsized":"2024-09-13T13:05:15.320840+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:21.281870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:21.281870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:42.889031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":200,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352182+0000","last_change":"2024-09-13T13:04:53.687731+0000","last_active":"2024-09-13T13:05:15.352182+0000","last_peered":"2024-09-13T13:05:15.352182+0000","last_clean":"2024-09-13T13:05:15.352182+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:05:15.352182+0000","last_undegraded":"2024-09-13T13:05:15.352182+0000","last_fullsized":"2024-09-13T13:05:15.352182+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.632415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":188,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.326132+0000","last_change":"2024-09-13T13:02:41.622369+0000","last_active":"2024-09-13T13:05:15.326132+0000","last_peered":"2024-09-13T13:05:15.326132+0000","last_clean":"2024-09-13T13:05:15.326132+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:05:15.326132+0000","last_undegraded":"2024-09-13T13:05:15.326132+0000","last_fullsized":"2024-09-13T13:05:15.326132+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.622292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.622292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":232,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.322316+0000","last_change":"2024-09-13T13:01:06.210251+0000","last_active":"2024-09-13T13:05:15.322316+0000","last_peered":"2024-09-13T13:05:15.322316+0000","last_clean":"2024-09-13T13:05:15.322316+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:05:15.322316+0000","last_undegraded":"2024-09-13T13:05:15.322316+0000","last_fullsized":"2024-09-13T13:05:15.322316+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352677+0000","last_change":"2024-09-13T13:04:49.927118+0000","last_active":"2024-09-13T13:05:15.352677+0000","last_peered":"2024-09-13T13:05:15.352677+0000","last_clean":"2024-09-13T13:05:15.352677+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:05:15.352677+0000","last_undegraded":"2024-09-13T13:05:15.352677+0000","last_fullsized":"2024-09-13T13:05:15.352677+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:49.927073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:49.927073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:08.714937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":199,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.322610+0000","last_change":"2024-09-13T13:02:37.222400+0000","last_active":"2024-09-13T13:05:15.322610+0000","last_peered":"2024-09-13T13:05:15.322610+0000","last_clean":"2024-09-13T13:05:15.322610+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:05:15.322610+0000","last_undegraded":"2024-09-13T13:05:15.322610+0000","last_fullsized":"2024-09-13T13:05:15.322610+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:37.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.222329+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":188,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.354538+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:05:15.354538+0000","last_peered":"2024-09-13T13:05:15.354538+0000","last_clean":"2024-09-13T13:05:15.354538+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:05:15.354538+0000","last_undegraded":"2024-09-13T13:05:15.354538+0000","last_fullsized":"2024-09-13T13:05:15.354538+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:55.131839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":248,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352053+0000","last_change":"2024-09-13T13:03:41.676060+0000","last_active":"2024-09-13T13:05:15.352053+0000","last_peered":"2024-09-13T13:05:15.352053+0000","last_clean":"2024-09-13T13:05:15.352053+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:05:15.352053+0000","last_undegraded":"2024-09-13T13:05:15.352053+0000","last_fullsized":"2024-09-13T13:05:15.352053+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:03:41.675978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:41.675978+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":176,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.326607+0000","last_change":"2024-09-13T13:02:40.615749+0000","last_active":"2024-09-13T13:05:15.326607+0000","last_peered":"2024-09-13T13:05:15.326607+0000","last_clean":"2024-09-13T13:05:15.326607+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:05:15.326607+0000","last_undegraded":"2024-09-13T13:05:15.326607+0000","last_fullsized":"2024-09-13T13:05:15.326607+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:40.615674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:40.615674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":215,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352256+0000","last_change":"2024-09-13T13:04:02.764890+0000","last_active":"2024-09-13T13:05:15.352256+0000","last_peered":"2024-09-13T13:05:15.352256+0000","last_clean":"2024-09-13T13:05:15.352256+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:05:15.352256+0000","last_undegraded":"2024-09-13T13:05:15.352256+0000","last_fullsized":"2024-09-13T13:05:15.352256+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:02:42.961328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.961328+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":208,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.352888+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:05:15.352888+0000","last_peered":"2024-09-13T13:05:15.352888+0000","last_clean":"2024-09-13T13:05:15.352888+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:05:15.352888+0000","last_undegraded":"2024-09-13T13:05:15.352888+0000","last_fullsized":"2024-09-13T13:05:15.352888+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:39.039395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":189,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.324895+0000","last_change":"2024-09-13T13:03:44.467611+0000","last_active":"2024-09-13T13:05:15.324895+0000","last_peered":"2024-09-13T13:05:15.324895+0000","last_clean":"2024-09-13T13:05:15.324895+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:05:15.324895+0000","last_undegraded":"2024-09-13T13:05:15.324895+0000","last_fullsized":"2024-09-13T13:05:15.324895+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:44.467297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:44.467297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":176,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321158+0000","last_change":"2024-09-13T13:02:42.208021+0000","last_active":"2024-09-13T13:05:15.321158+0000","last_peered":"2024-09-13T13:05:15.321158+0000","last_clean":"2024-09-13T13:05:15.321158+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:05:15.321158+0000","last_undegraded":"2024-09-13T13:05:15.321158+0000","last_fullsized":"2024-09-13T13:05:15.321158+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:42.207956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.207956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":144,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.351213+0000","last_change":"2024-09-13T13:04:02.757522+0000","last_active":"2024-09-13T13:05:15.351213+0000","last_peered":"2024-09-13T13:05:15.351213+0000","last_clean":"2024-09-13T13:05:15.351213+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:05:15.351213+0000","last_undegraded":"2024-09-13T13:05:15.351213+0000","last_fullsized":"2024-09-13T13:05:15.351213+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:35.448404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:35.448404+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":188,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.355693+0000","last_change":"2024-09-13T13:02:39.921391+0000","last_active":"2024-09-13T13:05:15.355693+0000","last_peered":"2024-09-13T13:05:15.355693+0000","last_clean":"2024-09-13T13:05:15.355693+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:05:15.355693+0000","last_undegraded":"2024-09-13T13:05:15.355693+0000","last_fullsized":"2024-09-13T13:05:15.355693+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.921325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.921325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":172,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321155+0000","last_change":"2024-09-13T13:01:06.211993+0000","last_active":"2024-09-13T13:05:15.321155+0000","last_peered":"2024-09-13T13:05:15.321155+0000","last_clean":"2024-09-13T13:05:15.321155+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:05:15.321155+0000","last_undegraded":"2024-09-13T13:05:15.321155+0000","last_fullsized":"2024-09-13T13:05:15.321155+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":176,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.325606+0000","last_change":"2024-09-13T13:02:36.586702+0000","last_active":"2024-09-13T13:05:15.325606+0000","last_peered":"2024-09-13T13:05:15.325606+0000","last_clean":"2024-09-13T13:05:15.325606+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:05:15.325606+0000","last_undegraded":"2024-09-13T13:05:15.325606+0000","last_fullsized":"2024-09-13T13:05:15.325606+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.586654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.586654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":214,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321299+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:05:15.321299+0000","last_peered":"2024-09-13T13:05:15.321299+0000","last_clean":"2024-09-13T13:05:15.321299+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:05:15.321299+0000","last_undegraded":"2024-09-13T13:05:15.321299+0000","last_fullsized":"2024-09-13T13:05:15.321299+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":188,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353197+0000","last_change":"2024-09-13T13:05:12.597696+0000","last_active":"2024-09-13T13:05:15.353197+0000","last_peered":"2024-09-13T13:05:15.353197+0000","last_clean":"2024-09-13T13:05:15.353197+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:05:15.353197+0000","last_undegraded":"2024-09-13T13:05:15.353197+0000","last_fullsized":"2024-09-13T13:05:15.353197+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:12.597637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.597637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:30.070820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":191,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353402+0000","last_change":"2024-09-13T13:05:08.432874+0000","last_active":"2024-09-13T13:05:15.353402+0000","last_peered":"2024-09-13T13:05:15.353402+0000","last_clean":"2024-09-13T13:05:15.353402+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:05:15.353402+0000","last_undegraded":"2024-09-13T13:05:15.353402+0000","last_fullsized":"2024-09-13T13:05:15.353402+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:08.432816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:08.432816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:09.081986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":179,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321248+0000","last_change":"2024-09-13T13:04:02.760695+0000","last_active":"2024-09-13T13:05:15.321248+0000","last_peered":"2024-09-13T13:05:15.321248+0000","last_clean":"2024-09-13T13:05:15.321248+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:05:15.321248+0000","last_undegraded":"2024-09-13T13:05:15.321248+0000","last_fullsized":"2024-09-13T13:05:15.321248+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":206,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.323289+0000","last_change":"2024-09-13T13:04:02.755722+0000","last_active":"2024-09-13T13:05:15.323289+0000","last_peered":"2024-09-13T13:05:15.323289+0000","last_clean":"2024-09-13T13:05:15.323289+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:05:15.323289+0000","last_undegraded":"2024-09-13T13:05:15.323289+0000","last_fullsized":"2024-09-13T13:05:15.323289+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":188,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.327766+0000","last_change":"2024-09-13T13:03:15.466909+0000","last_active":"2024-09-13T13:05:15.327766+0000","last_peered":"2024-09-13T13:05:15.327766+0000","last_clean":"2024-09-13T13:05:15.327766+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:05:15.327766+0000","last_undegraded":"2024-09-13T13:05:15.327766+0000","last_fullsized":"2024-09-13T13:05:15.327766+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:15.466859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:15.466859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":74,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.320885+0000","last_change":"2024-09-13T13:04:02.765046+0000","last_active":"2024-09-13T13:05:15.320885+0000","last_peered":"2024-09-13T13:05:15.320885+0000","last_clean":"2024-09-13T13:05:15.320885+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:05:15.320885+0000","last_undegraded":"2024-09-13T13:05:15.320885+0000","last_fullsized":"2024-09-13T13:05:15.320885+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":176,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.353517+0000","last_change":"2024-09-13T13:04:58.468060+0000","last_active":"2024-09-13T13:05:15.353517+0000","last_peered":"2024-09-13T13:05:15.353517+0000","last_clean":"2024-09-13T13:05:15.353517+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:05:15.353517+0000","last_undegraded":"2024-09-13T13:05:15.353517+0000","last_fullsized":"2024-09-13T13:05:15.353517+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:58.468011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:58.468011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.129704+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.323989+0000","last_change":"2024-09-13T13:02:43.234909+0000","last_active":"2024-09-13T13:05:15.323989+0000","last_peered":"2024-09-13T13:05:15.323989+0000","last_clean":"2024-09-13T13:05:15.323989+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:05:15.323989+0000","last_undegraded":"2024-09-13T13:05:15.323989+0000","last_fullsized":"2024-09-13T13:05:15.323989+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:43.234847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.234847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":188,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.323478+0000","last_change":"2024-09-13T13:02:46.246602+0000","last_active":"2024-09-13T13:05:15.323478+0000","last_peered":"2024-09-13T13:05:15.323478+0000","last_clean":"2024-09-13T13:05:15.323478+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:05:15.323478+0000","last_undegraded":"2024-09-13T13:05:15.323478+0000","last_fullsized":"2024-09-13T13:05:15.323478+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.246540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.246540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":184,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.321306+0000","last_change":"2024-09-13T13:04:02.756287+0000","last_active":"2024-09-13T13:05:15.321306+0000","last_peered":"2024-09-13T13:05:15.321306+0000","last_clean":"2024-09-13T13:05:15.321306+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:05:15.321306+0000","last_undegraded":"2024-09-13T13:05:15.321306+0000","last_fullsized":"2024-09-13T13:05:15.321306+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:02:39.557251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.557251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":181,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-13T13:05:15.326703+0000","last_change":"2024-09-13T13:02:30.512536+0000","last_active":"2024-09-13T13:05:15.326703+0000","last_peered":"2024-09-13T13:05:15.326703+0000","last_clean":"2024-09-13T13:05:15.326703+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:05:15.326703+0000","last_undegraded":"2024-09-13T13:05:15.326703+0000","last_fullsized":"2024-09-13T13:05:15.326703+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:30.512433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:30.512433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3285975064,"num_objects":50141,"num_object_clones":0,"num_object_copies":150423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192057,"num_write_kb":7750273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13254,"num_bytes_recovered":702152728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9845551104,"data_stored":9845538888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88224,"ondisk_log_size":88224,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":250,"num_read_kb":245,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201334,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1116140,"kb_used_data":1008604,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92629012,"statfs":{"total":95995035648,"available":94852108288,"internally_reserved":0,"allocated":1032810496,"data_stored":1032416797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806140944,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1135508,"kb_used_data":1025860,"kb_used_omap":4,"kb_used_meta":109627,"kb_avail":92609644,"statfs":{"total":95995035648,"available":94832275456,"internally_reserved":0,"allocated":1050480640,"data_stored":1050103051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":112258733},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691906,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890020,"kb_used_data":814416,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92855132,"statfs":{"total":95995035648,"available":95083655168,"internally_reserved":0,"allocated":833961984,"data_stored":833578869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1124024,"kb_used_data":1016484,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92621128,"statfs":{"total":95995035648,"available":94844035072,"internally_reserved":0,"allocated":1040879616,"data_stored":1040476157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117071,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1964856,"kb_used_data":1803112,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":91780296,"statfs":{"total":95995035648,"available":93983023104,"internally_reserved":0,"allocated":1846386688,"data_stored":1845979963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":165542343},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1330384,"kb_used_data":1226944,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92414768,"statfs":{"total":95995035648,"available":94632722432,"internally_reserved":0,"allocated":1256390656,"data_stored":1256023265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1321484,"kb_used_data":1213820,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92423668,"statfs":{"total":95995035648,"available":94641836032,"internally_reserved":0,"allocated":1242951680,"data_stored":1242524180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1737972,"kb_used_data":1602660,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92007180,"statfs":{"total":95995035648,"available":94215352320,"internally_reserved":0,"allocated":1641123840,"data_stored":1640721682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1622282240,"data_stored":1622278168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1235619840,"data_stored":1235615768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1237319680,"data_stored":1237319680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1829699584,"data_stored":1829699584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1024851968,"data_stored":1024851968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":828899328,"data_stored":828899328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1046024192,"data_stored":1046020120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1020854272,"data_stored":1020854272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:05:22.586 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T13:05:22.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:22.672 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:22.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:22.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:22.710 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:05:22.710 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:05:22.774 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:22.799 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:22.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:22.828 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:22 smithi052 ceph-mon[36682]: pgmap v191: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 697 B/s rd, 0 B/s wr, 616 op/s; 823/149211 objects degraded (0.552%); 3.2 MiB/s, 56 objects/s recovering 2024-09-13T13:05:22.828 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:22 smithi052 ceph-mon[29471]: pgmap v191: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 697 B/s rd, 0 B/s wr, 616 op/s; 823/149211 objects degraded (0.552%); 3.2 MiB/s, 56 objects/s recovering 2024-09-13T13:05:22.901 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:22.924 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:22.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:22.952 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:05:22.952 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:05:22.985 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:05:22.986 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:05:23.021 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:05:23.021 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:05:23.025 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:23.036 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:23.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:22 smithi167 ceph-mon[33178]: pgmap v191: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 697 B/s rd, 0 B/s wr, 616 op/s; 823/149211 objects degraded (0.552%); 3.2 MiB/s, 56 objects/s recovering 2024-09-13T13:05:23.071 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:05:23.071 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:05:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:23.094 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:23.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:23.104 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:05:23.104 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:05:23.139 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:05:23.139 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:05:23.173 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:05:23.173 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:05:23.195 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:23.207 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:05:23.207 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:05:23.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:23.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:23.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:23.322 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:23.365 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:23.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:23.467 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:23.509 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:23.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:23.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:23.568 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T13:05:23.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:23.642 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:23.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:23.743 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:23.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:23.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:23.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:23 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 823/149211 objects degraded (0.552%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:23 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2383926908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:23.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:05:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:05:23.874 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:23.910 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:23.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:23.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:23.918 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:05:23.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:23 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 823/149211 objects degraded (0.552%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:23.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:23 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2383926908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:23.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:23 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 823/149211 objects degraded (0.552%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:23.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:23 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2383926908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:24.012 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:24.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:24.039 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:24.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:24.140 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:24.176 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:24.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:24.279 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:24.323 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:24.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:24.377 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:24.425 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:24.451 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:24.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:24.499 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:05:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:05:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:198421 7.d 7:bff7eedc:::benchmark_data_smithi052_71865_object77320:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-13T13:05:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:24.489593+0000", 2024-09-13T13:05:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0082823029999999995, 2024-09-13T13:05:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0083033440000000007, 2024-09-13T13:05:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:05:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 198421 2024-09-13T13:05:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.489593+0000", 2024-09-13T13:05:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.489593+0000", 2024-09-13T13:05:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.490364+0000", 2024-09-13T13:05:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077041999999999996 2024-09-13T13:05:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.490369+0000", 2024-09-13T13:05:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5450000000000003e-06 2024-09-13T13:05:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.490383+0000", 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3551e-05 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.490390+0000", 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.029e-06 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.490499+0000", 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010890100000000001 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.490596+0000", 2024-09-13T13:05:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7254000000000001e-05 2024-09-13T13:05:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:05:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.490687+0000", 2024-09-13T13:05:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0750999999999999e-05 2024-09-13T13:05:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:05:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.494193+0000", 2024-09-13T13:05:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035062589999999999 2024-09-13T13:05:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:198427 7.d 7:bee14fb1:::benchmark_data_smithi052_71865_object77326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-13T13:05:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:24.495190+0000", 2024-09-13T13:05:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0026856440000000001, 2024-09-13T13:05:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027512140000000001, 2024-09-13T13:05:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:05:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:05:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:05:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:05:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:05:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:05:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 198427 2024-09-13T13:05:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:05:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:05:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.495190+0000", 2024-09-13T13:05:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:05:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.495190+0000", 2024-09-13T13:05:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:05:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:05:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.495192+0000", 2024-09-13T13:05:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.058e-06 2024-09-13T13:05:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:05:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.495199+0000", 2024-09-13T13:05:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2929999999999997e-06 2024-09-13T13:05:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:05:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.495209+0000", 2024-09-13T13:05:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0335e-05 2024-09-13T13:05:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:05:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.495216+0000", 2024-09-13T13:05:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1910000000000003e-06 2024-09-13T13:05:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:05:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.495268+0000", 2024-09-13T13:05:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2018999999999998e-05 2024-09-13T13:05:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:05:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.495373+0000", 2024-09-13T13:05:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010477800000000001 2024-09-13T13:05:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:05:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:05:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:05:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:24.495486+0000", 2024-09-13T13:05:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011361099999999999 2024-09-13T13:05:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:05:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:05:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:05:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 2 2024-09-13T13:05:24.510 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:05:24.553 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:24.599 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:24.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:24.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:24.700 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:24.756 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:24.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:24.766 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:05:24.842 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:24 smithi052 ceph-mon[36682]: pgmap v192: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 583/136734 objects degraded (0.426%); 4.6 MiB/s, 78 objects/s recovering 2024-09-13T13:05:24.842 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:24 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:24.842 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:24 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1788925657' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:05:24.842 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:24 smithi052 ceph-mon[29471]: pgmap v192: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 583/136734 objects degraded (0.426%); 4.6 MiB/s, 78 objects/s recovering 2024-09-13T13:05:24.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:24 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:24.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:24 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1788925657' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:05:24.857 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:24.899 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:24.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:24.999 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:25.036 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:25.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:25.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:24 smithi167 ceph-mon[33178]: pgmap v192: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 583/136734 objects degraded (0.426%); 4.6 MiB/s, 78 objects/s recovering 2024-09-13T13:05:25.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:24 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:25.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:24 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1788925657' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:05:25.138 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:25.178 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:25.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:25.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:25.280 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:25.302 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:25.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:25.405 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:25.429 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:25.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:25.431 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:05:25.431 INFO:teuthology.orchestra.run.smithi052.stdout:{"fsid":"2b415b38-71cf-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 583/136734 objects degraded (0.426%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":507,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":146,"num_osds":8,"num_up_osds":8,"osd_up_since":1726232642,"num_in_osds":8,"osd_in_since":1726232429,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":147},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":148,"num_pools":7,"num_objects":45578,"data_bytes":2974489139,"bytes_used":9969680384,"bytes_avail":757990604800,"bytes_total":767960285184,"degraded_objects":583,"degraded_total":136734,"degraded_ratio":0.0042637529802390045,"recovering_objects_per_sec":78,"recovering_bytes_per_sec":4853061,"recovering_keys_per_sec":0,"num_objects_recovered":653,"num_bytes_recovered":40501248,"num_keys_recovered":0,"read_bytes_sec":1227,"write_bytes_sec":0,"read_op_per_sec":1,"write_op_per_sec":1039},"fsmap":{"epoch":1,"btime":"2024-09-13T12:55:27:893730+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.52:8443/","prometheus":"http://172.21.15.52:9283/"}},"servicemap":{"epoch":15,"modified":"2024-09-13T13:05:01.658350+0000","services":{"rgw":{"daemons":{"summary":"","14493":{"start_epoch":12,"start_stamp":"2024-09-13T13:01:09.302524+0000","gid":14493,"addr":"172.21.15.52:0/2628990817","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-4917-gb0201b8c (b0201b8c79733293453e7f10a10c7fa43119222b) squid (dev)","ceph_version_short":"19.3.0-4917-gb0201b8c","container_hostname":"smithi052","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi052","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024","kernel_version":"5.14.0-505.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32493496","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"434eb8a4-923f-4e38-a7a5-e2b5e50a0eee","zone_name":"default","zonegroup_id":"5289daf0-e99a-4da1-a9e6-98887649b2bb","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T13:05:25.532 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:25.578 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:25.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:25.680 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:25.725 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:25.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:25.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:25.851 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:25.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:25.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:25.953 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:25.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:25 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:25 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1788925657' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:05:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:25 smithi052 ceph-mon[36682]: osdmap e146: 8 total, 8 up, 8 in 2024-09-13T13:05:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:25 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/688898680' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T13:05:25.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:25.990 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:25.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:25.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:25 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:25.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:25 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1788925657' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:05:25.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:25 smithi052 ceph-mon[29471]: osdmap e146: 8 total, 8 up, 8 in 2024-09-13T13:05:25.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:25 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/688898680' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T13:05:26.060 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:05:26.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:25 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:26.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:25 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1788925657' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:05:26.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:25 smithi167 ceph-mon[33178]: osdmap e146: 8 total, 8 up, 8 in 2024-09-13T13:05:26.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:25 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/688898680' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T13:05:26.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:26.118 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:26.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:26.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:26.220 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:26.265 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:26.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:26.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:26.389 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T13:05:26.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:26.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:26.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:26.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:05:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:05:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:05:26.501 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:26.548 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:26.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:26.650 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:26.676 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T13:05:26.676 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:05:26.680 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:26.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:26.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:26.739 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:05:26.769 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:26.782 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:26.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:26.844 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:26.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:26.924 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:26 smithi052 ceph-mon[29471]: pgmap v194: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 583/136734 objects degraded (0.426%); 4.8 MiB/s, 77 objects/s recovering 2024-09-13T13:05:26.924 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:26 smithi052 ceph-mon[29471]: 4.f scrub starts 2024-09-13T13:05:26.925 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:26 smithi052 ceph-mon[29471]: 4.f scrub ok 2024-09-13T13:05:26.925 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:26 smithi052 ceph-mon[36682]: pgmap v194: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 583/136734 objects degraded (0.426%); 4.8 MiB/s, 77 objects/s recovering 2024-09-13T13:05:26.925 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:26 smithi052 ceph-mon[36682]: 4.f scrub starts 2024-09-13T13:05:26.925 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:26 smithi052 ceph-mon[36682]: 4.f scrub ok 2024-09-13T13:05:26.946 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:27.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:27.003 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:27.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:27.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:26 smithi167 ceph-mon[33178]: pgmap v194: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 583/136734 objects degraded (0.426%); 4.8 MiB/s, 77 objects/s recovering 2024-09-13T13:05:27.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:26 smithi167 ceph-mon[33178]: 4.f scrub starts 2024-09-13T13:05:27.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:26 smithi167 ceph-mon[33178]: 4.f scrub ok 2024-09-13T13:05:27.103 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:27.127 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:27.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:27.229 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:27.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:27.258 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:27.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:27.359 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:27.385 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:27.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:27.487 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:27.519 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:27.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:27.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:27.620 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:27.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:27.652 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:27.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:27.677 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:05:27.677 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:05:27.677 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:05:27.677 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:05:27.677 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:05:27.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:27.797 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:27.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:27.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:27.879 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:05:27.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:27.920 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:27.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:27.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:27.999 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:27 smithi167 ceph-mon[33178]: 4.13 scrub starts 2024-09-13T13:05:27.999 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:27 smithi167 ceph-mon[33178]: 4.13 scrub ok 2024-09-13T13:05:27.999 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:27 smithi167 ceph-mon[33178]: 5.1b scrub starts 2024-09-13T13:05:27.999 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:27 smithi167 ceph-mon[33178]: 5.1b scrub ok 2024-09-13T13:05:27.999 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:27 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/121823351' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:27.999 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:27 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 583/136734 objects degraded (0.426%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:28.023 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:28.072 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:28.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:28.174 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:28.211 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:28.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:28.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:28.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:27 smithi052 ceph-mon[36682]: 4.13 scrub starts 2024-09-13T13:05:28.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:27 smithi052 ceph-mon[36682]: 4.13 scrub ok 2024-09-13T13:05:28.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:27 smithi052 ceph-mon[36682]: 5.1b scrub starts 2024-09-13T13:05:28.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:27 smithi052 ceph-mon[36682]: 5.1b scrub ok 2024-09-13T13:05:28.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:27 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/121823351' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:28.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:27 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 583/136734 objects degraded (0.426%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:28.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:27 smithi052 ceph-mon[29471]: 4.13 scrub starts 2024-09-13T13:05:28.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:27 smithi052 ceph-mon[29471]: 4.13 scrub ok 2024-09-13T13:05:28.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:27 smithi052 ceph-mon[29471]: 5.1b scrub starts 2024-09-13T13:05:28.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:27 smithi052 ceph-mon[29471]: 5.1b scrub ok 2024-09-13T13:05:28.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:27 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/121823351' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:28.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:27 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 583/136734 objects degraded (0.426%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:28.314 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:28.340 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:28.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:28.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:28.442 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:28.447 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:05:28.447 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:05:28.447 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:05:28.447 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:05:28.447 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:05:28.448 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:05:28.448 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:05:28.448 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:05:28.448 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:05:28.448 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:05:28.448 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:05:28.448 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:05:28.448 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:05:28.448 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:05:28.449 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:05:28.449 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:05:28.473 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T13:05:28.482 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:28.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:28.585 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:28.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:28.621 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:28.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:28.722 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:28.749 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:28.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:28.820 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:05:28.851 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:28.895 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:28.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:28.997 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:29.026 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:29.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:29.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:28 smithi167 ceph-mon[33178]: 3.f scrub starts 2024-09-13T13:05:29.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:28 smithi167 ceph-mon[33178]: 3.f scrub ok 2024-09-13T13:05:29.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:28 smithi167 ceph-mon[33178]: 5.16 scrub starts 2024-09-13T13:05:29.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:28 smithi167 ceph-mon[33178]: 5.16 scrub ok 2024-09-13T13:05:29.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:28 smithi167 ceph-mon[33178]: pgmap v195: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 672 op/s; 583/134895 objects degraded (0.432%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T13:05:29.128 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:29.162 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:29.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:29.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:28 smithi052 ceph-mon[29471]: 3.f scrub starts 2024-09-13T13:05:29.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:28 smithi052 ceph-mon[29471]: 3.f scrub ok 2024-09-13T13:05:29.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:28 smithi052 ceph-mon[29471]: 5.16 scrub starts 2024-09-13T13:05:29.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:28 smithi052 ceph-mon[29471]: 5.16 scrub ok 2024-09-13T13:05:29.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:28 smithi052 ceph-mon[29471]: pgmap v195: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 672 op/s; 583/134895 objects degraded (0.432%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T13:05:29.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:28 smithi052 ceph-mon[36682]: 3.f scrub starts 2024-09-13T13:05:29.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:28 smithi052 ceph-mon[36682]: 3.f scrub ok 2024-09-13T13:05:29.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:28 smithi052 ceph-mon[36682]: 5.16 scrub starts 2024-09-13T13:05:29.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:28 smithi052 ceph-mon[36682]: 5.16 scrub ok 2024-09-13T13:05:29.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:28 smithi052 ceph-mon[36682]: pgmap v195: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 672 op/s; 583/134895 objects degraded (0.432%); 1.9 MiB/s, 29 objects/s recovering 2024-09-13T13:05:29.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:29.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:29.299 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:29.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:29.400 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:29.401 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:05:29.401 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:05:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:29.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:29.529 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:29.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:29.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:29.656 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:29.697 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:29.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:29.799 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:29.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:29.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:29.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:29.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:29.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:30.057 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:29 smithi167 ceph-mon[33178]: 1.0 scrub starts 2024-09-13T13:05:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:29 smithi167 ceph-mon[33178]: 1.0 scrub ok 2024-09-13T13:05:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:29 smithi167 ceph-mon[33178]: 7.b scrub starts 2024-09-13T13:05:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:29 smithi167 ceph-mon[33178]: 5.8 scrub starts 2024-09-13T13:05:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:29 smithi167 ceph-mon[33178]: 5.8 scrub ok 2024-09-13T13:05:30.092 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:30.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:30.154 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T13:05:30.157 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:29 smithi052 ceph-mon[29471]: 1.0 scrub starts 2024-09-13T13:05:30.157 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:29 smithi052 ceph-mon[29471]: 1.0 scrub ok 2024-09-13T13:05:30.157 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:29 smithi052 ceph-mon[29471]: 7.b scrub starts 2024-09-13T13:05:30.157 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:29 smithi052 ceph-mon[29471]: 5.8 scrub starts 2024-09-13T13:05:30.157 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:29 smithi052 ceph-mon[29471]: 5.8 scrub ok 2024-09-13T13:05:30.157 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:29 smithi052 ceph-mon[36682]: 1.0 scrub starts 2024-09-13T13:05:30.157 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:29 smithi052 ceph-mon[36682]: 1.0 scrub ok 2024-09-13T13:05:30.158 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:29 smithi052 ceph-mon[36682]: 7.b scrub starts 2024-09-13T13:05:30.158 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:29 smithi052 ceph-mon[36682]: 5.8 scrub starts 2024-09-13T13:05:30.158 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:29 smithi052 ceph-mon[36682]: 5.8 scrub ok 2024-09-13T13:05:30.194 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:30.259 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:30.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:30.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:30.387 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:30.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:30.489 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:30.513 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:30.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:30.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:30.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:30.641 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:30.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:30.743 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:30.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:30.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:30.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:30.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:30.898 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:30.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:30.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:31.000 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:31.025 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:31.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:31.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:30 smithi167 ceph-mon[33178]: 5.1 scrub starts 2024-09-13T13:05:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:30 smithi167 ceph-mon[33178]: 5.1 scrub ok 2024-09-13T13:05:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:30 smithi167 ceph-mon[33178]: 3.e scrub starts 2024-09-13T13:05:31.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:30 smithi167 ceph-mon[33178]: 3.e scrub ok 2024-09-13T13:05:31.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:30 smithi167 ceph-mon[33178]: 7.b scrub ok 2024-09-13T13:05:31.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:30 smithi167 ceph-mon[33178]: pgmap v196: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 956 op/s; 326/122523 objects degraded (0.266%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T13:05:31.126 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:31.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:31.155 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:31.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:31.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:30 smithi052 ceph-mon[29471]: 5.1 scrub starts 2024-09-13T13:05:31.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:30 smithi052 ceph-mon[29471]: 5.1 scrub ok 2024-09-13T13:05:31.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:30 smithi052 ceph-mon[29471]: 3.e scrub starts 2024-09-13T13:05:31.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:30 smithi052 ceph-mon[29471]: 3.e scrub ok 2024-09-13T13:05:31.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:30 smithi052 ceph-mon[29471]: 7.b scrub ok 2024-09-13T13:05:31.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:30 smithi052 ceph-mon[29471]: pgmap v196: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 956 op/s; 326/122523 objects degraded (0.266%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T13:05:31.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:30 smithi052 ceph-mon[36682]: 5.1 scrub starts 2024-09-13T13:05:31.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:30 smithi052 ceph-mon[36682]: 5.1 scrub ok 2024-09-13T13:05:31.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:30 smithi052 ceph-mon[36682]: 3.e scrub starts 2024-09-13T13:05:31.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:30 smithi052 ceph-mon[36682]: 3.e scrub ok 2024-09-13T13:05:31.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:30 smithi052 ceph-mon[36682]: 7.b scrub ok 2024-09-13T13:05:31.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:30 smithi052 ceph-mon[36682]: pgmap v196: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 956 op/s; 326/122523 objects degraded (0.266%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T13:05:31.257 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:31.297 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:31.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:31.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:31.399 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:31.442 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:31.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:31.544 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:31.567 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:31.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:31.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:31.674 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:31.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:31.718 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:31.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:31.819 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:31.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:31.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:31.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:31.947 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:31.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:31.988 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:31.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:32.089 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:32.129 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:32.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:32.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:31 smithi052 ceph-mon[29471]: 6.1 scrub starts 2024-09-13T13:05:32.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:31 smithi052 ceph-mon[29471]: 6.1 scrub ok 2024-09-13T13:05:32.131 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:31 smithi052 ceph-mon[36682]: 6.1 scrub starts 2024-09-13T13:05:32.132 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:31 smithi052 ceph-mon[36682]: 6.1 scrub ok 2024-09-13T13:05:32.230 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:32.252 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:32.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:32.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:31 smithi167 ceph-mon[33178]: 6.1 scrub starts 2024-09-13T13:05:32.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:31 smithi167 ceph-mon[33178]: 6.1 scrub ok 2024-09-13T13:05:32.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:32.383 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:32.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:32.485 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:32.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:32.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:32.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:32.663 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:32.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:32.764 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:32.800 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:32.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:32.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:32.902 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:32.942 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:32.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:33.044 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:33.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:32 smithi167 ceph-mon[33178]: 6.7 scrub starts 2024-09-13T13:05:33.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:32 smithi167 ceph-mon[33178]: 6.7 scrub ok 2024-09-13T13:05:33.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:32 smithi167 ceph-mon[33178]: pgmap v197: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.4 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 947 op/s; 326/120795 objects degraded (0.270%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T13:05:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:32 smithi167 ceph-mon[33178]: 5.d scrub starts 2024-09-13T13:05:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:32 smithi167 ceph-mon[33178]: 5.d scrub ok 2024-09-13T13:05:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:05:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:32 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 326/120795 objects degraded (0.270%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:33.085 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:33.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:33.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:33.186 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:33.214 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:33.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[36682]: 6.7 scrub starts 2024-09-13T13:05:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[36682]: 6.7 scrub ok 2024-09-13T13:05:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[36682]: pgmap v197: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.4 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 947 op/s; 326/120795 objects degraded (0.270%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T13:05:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[36682]: 5.d scrub starts 2024-09-13T13:05:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[36682]: 5.d scrub ok 2024-09-13T13:05:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:05:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 326/120795 objects degraded (0.270%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[29471]: 6.7 scrub starts 2024-09-13T13:05:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[29471]: 6.7 scrub ok 2024-09-13T13:05:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[29471]: pgmap v197: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.4 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 947 op/s; 326/120795 objects degraded (0.270%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T13:05:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[29471]: 5.d scrub starts 2024-09-13T13:05:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[29471]: 5.d scrub ok 2024-09-13T13:05:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:05:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:32 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 326/120795 objects degraded (0.270%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:33.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:33.342 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:33.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:33.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:33.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:33.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:33.488 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:33.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:33.589 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:33.611 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:33.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:33.712 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:33.749 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:33.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:33.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:33.851 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:33.896 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:33.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:33.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:34.025 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:34.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:34.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:34.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:33 smithi167 ceph-mon[33178]: 4.1c scrub starts 2024-09-13T13:05:34.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:33 smithi167 ceph-mon[33178]: 4.1c scrub ok 2024-09-13T13:05:34.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:05:33 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:05:34.127 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:34.153 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:34.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:34.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:34.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:33 smithi052 ceph-mon[36682]: 4.1c scrub starts 2024-09-13T13:05:34.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:33 smithi052 ceph-mon[36682]: 4.1c scrub ok 2024-09-13T13:05:34.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:33 smithi052 ceph-mon[29471]: 4.1c scrub starts 2024-09-13T13:05:34.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:33 smithi052 ceph-mon[29471]: 4.1c scrub ok 2024-09-13T13:05:34.254 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:34.296 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:34.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:34.398 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:34.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:34.452 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:34.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:34.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:34.580 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:34.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:34.681 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:34.707 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:34.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:34.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:34.717 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:34.732 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:34.809 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:34.838 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:34.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:34.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[29471]: 4.1 scrub starts 2024-09-13T13:05:34.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[29471]: 4.1 scrub ok 2024-09-13T13:05:34.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[29471]: pgmap v198: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 926 op/s; 67/108951 objects degraded (0.061%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T13:05:34.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:34.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[29471]: 5.19 scrub starts 2024-09-13T13:05:34.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[29471]: 5.19 scrub ok 2024-09-13T13:05:34.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[29471]: 2.1 scrub starts 2024-09-13T13:05:34.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[29471]: 2.1 scrub ok 2024-09-13T13:05:34.875 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[36682]: 4.1 scrub starts 2024-09-13T13:05:34.875 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[36682]: 4.1 scrub ok 2024-09-13T13:05:34.876 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[36682]: pgmap v198: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 926 op/s; 67/108951 objects degraded (0.061%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T13:05:34.876 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:34.876 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[36682]: 5.19 scrub starts 2024-09-13T13:05:34.876 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[36682]: 5.19 scrub ok 2024-09-13T13:05:34.876 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[36682]: 2.1 scrub starts 2024-09-13T13:05:34.876 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:34 smithi052 ceph-mon[36682]: 2.1 scrub ok 2024-09-13T13:05:34.938 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:34.966 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:34.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:34.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:35.053 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:35.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:35.091 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:35.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:35.193 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:35.235 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:35.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:35.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:35.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:34 smithi167 ceph-mon[33178]: 4.1 scrub starts 2024-09-13T13:05:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:34 smithi167 ceph-mon[33178]: 4.1 scrub ok 2024-09-13T13:05:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:34 smithi167 ceph-mon[33178]: pgmap v198: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 926 op/s; 67/108951 objects degraded (0.061%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T13:05:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:34 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:34 smithi167 ceph-mon[33178]: 5.19 scrub starts 2024-09-13T13:05:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:34 smithi167 ceph-mon[33178]: 5.19 scrub ok 2024-09-13T13:05:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:34 smithi167 ceph-mon[33178]: 2.1 scrub starts 2024-09-13T13:05:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:34 smithi167 ceph-mon[33178]: 2.1 scrub ok 2024-09-13T13:05:35.337 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:35.372 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:35.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:35.473 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:35.497 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:35.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:35.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:35.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:35.625 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:35.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:35.727 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:35.750 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:35.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:35.852 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:35.895 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:35.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:35.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:35.996 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:36.019 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[36682]: 4.d scrub starts 2024-09-13T13:05:36.019 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[36682]: 4.d scrub ok 2024-09-13T13:05:36.019 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:05:36.019 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:05:36.019 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[36682]: osdmap e147: 8 total, 8 up, 8 in 2024-09-13T13:05:36.019 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[36682]: 7.5 scrub starts 2024-09-13T13:05:36.020 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[36682]: 6.f scrub starts 2024-09-13T13:05:36.020 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[36682]: 6.f scrub ok 2024-09-13T13:05:36.021 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:05:36.021 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[29471]: 4.d scrub starts 2024-09-13T13:05:36.021 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[29471]: 4.d scrub ok 2024-09-13T13:05:36.021 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:05:36.021 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:05:36.021 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[29471]: osdmap e147: 8 total, 8 up, 8 in 2024-09-13T13:05:36.021 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[29471]: 7.5 scrub starts 2024-09-13T13:05:36.022 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[29471]: 6.f scrub starts 2024-09-13T13:05:36.022 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:35 smithi052 ceph-mon[29471]: 6.f scrub ok 2024-09-13T13:05:36.027 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:05:36.050 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:36.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:36.153 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:36.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:36.189 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:36.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:36.290 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:36.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:35 smithi167 ceph-mon[33178]: 4.d scrub starts 2024-09-13T13:05:36.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:35 smithi167 ceph-mon[33178]: 4.d scrub ok 2024-09-13T13:05:36.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:35 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:05:36.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:35 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:05:36.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:35 smithi167 ceph-mon[33178]: osdmap e147: 8 total, 8 up, 8 in 2024-09-13T13:05:36.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:35 smithi167 ceph-mon[33178]: 7.5 scrub starts 2024-09-13T13:05:36.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:35 smithi167 ceph-mon[33178]: 6.f scrub starts 2024-09-13T13:05:36.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:35 smithi167 ceph-mon[33178]: 6.f scrub ok 2024-09-13T13:05:36.331 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:36.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:36.378 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:05:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:05:36.433 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:36.473 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:36.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:36.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:36.553 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:05:36.553 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:05:36.575 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:36.622 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:36.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:36.671 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:05:36.724 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:36.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:36.778 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:36.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:36.875 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:05:36.879 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:36.937 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:36.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:36.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:36.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[29471]: 4.12 scrub starts 2024-09-13T13:05:36.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[29471]: 4.12 scrub ok 2024-09-13T13:05:36.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[29471]: pgmap v200: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 926 op/s; 67/108951 objects degraded (0.061%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T13:05:36.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[29471]: 7.5 scrub ok 2024-09-13T13:05:36.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/966657627' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:05:36.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[29471]: from='client.25087 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:05:36.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[29471]: from='client.25093 v1:172.21.15.52:0/4130683345' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:36.967 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[36682]: 4.12 scrub starts 2024-09-13T13:05:36.968 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[36682]: 4.12 scrub ok 2024-09-13T13:05:36.968 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[36682]: pgmap v200: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 926 op/s; 67/108951 objects degraded (0.061%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T13:05:36.968 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[36682]: 7.5 scrub ok 2024-09-13T13:05:36.968 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/966657627' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:05:36.968 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[36682]: from='client.25087 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:05:36.968 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:36 smithi052 ceph-mon[36682]: from='client.25093 v1:172.21.15.52:0/4130683345' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:37.020 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-13T13:05:35.670083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2367560243,"num_objects":36317,"num_object_clones":0,"num_object_copies":108951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36317,"num_whiteouts":0,"num_read":1112,"num_read_kb":969,"num_write":206629,"num_write_kb":7751946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14018,"num_bytes_recovered":752288422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89607,"ondisk_log_size":89607,"up":444,"acting":443,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8116456,"kb_used_data":7074792,"kb_used_omap":15,"kb_used_meta":1041328,"kb_avail":741844760,"statfs":{"total":767960285184,"available":759649034240,"internally_reserved":0,"allocated":7244587008,"data_stored":7241332985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":1066320665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-606928896,"num_objects":-9261,"num_object_clones":0,"num_object_copies":-27783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-516,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9261,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9261,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":33816576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003795"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":193,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.776877+0000","last_change":"2024-09-13T13:02:38.651382+0000","last_active":"2024-09-13T13:05:24.776877+0000","last_peered":"2024-09-13T13:05:24.776877+0000","last_clean":"2024-09-13T13:05:24.776877+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:05:24.776877+0000","last_undegraded":"2024-09-13T13:05:24.776877+0000","last_fullsized":"2024-09-13T13:05:24.776877+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:38.651314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:38.651314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":216,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.767526+0000","last_change":"2024-09-13T13:04:48.899982+0000","last_active":"2024-09-13T13:05:24.767526+0000","last_peered":"2024-09-13T13:05:24.767526+0000","last_clean":"2024-09-13T13:05:24.767526+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:05:24.767526+0000","last_undegraded":"2024-09-13T13:05:24.767526+0000","last_fullsized":"2024-09-13T13:05:24.767526+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:48.899915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.899915+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:00.227319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":185,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.790412+0000","last_change":"2024-09-13T13:02:30.248974+0000","last_active":"2024-09-13T13:05:24.790412+0000","last_peered":"2024-09-13T13:05:24.790412+0000","last_clean":"2024-09-13T13:05:24.790412+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:05:24.790412+0000","last_undegraded":"2024-09-13T13:05:24.790412+0000","last_fullsized":"2024-09-13T13:05:24.790412+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:30.248905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:30.248905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.767765+0000","last_change":"2024-09-13T13:05:01.076346+0000","last_active":"2024-09-13T13:05:24.767765+0000","last_peered":"2024-09-13T13:05:24.767765+0000","last_clean":"2024-09-13T13:05:24.767765+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:05:24.767765+0000","last_undegraded":"2024-09-13T13:05:24.767765+0000","last_fullsized":"2024-09-13T13:05:24.767765+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.076275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.076275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:14.180362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":183,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.776488+0000","last_change":"2024-09-13T13:04:02.760609+0000","last_active":"2024-09-13T13:05:24.776488+0000","last_peered":"2024-09-13T13:05:24.776488+0000","last_clean":"2024-09-13T13:05:24.776488+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:05:24.776488+0000","last_undegraded":"2024-09-13T13:05:24.776488+0000","last_fullsized":"2024-09-13T13:05:24.776488+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":204,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.774088+0000","last_change":"2024-09-13T13:04:02.757267+0000","last_active":"2024-09-13T13:05:24.774088+0000","last_peered":"2024-09-13T13:05:24.774088+0000","last_clean":"2024-09-13T13:05:24.774088+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:05:24.774088+0000","last_undegraded":"2024-09-13T13:05:24.774088+0000","last_fullsized":"2024-09-13T13:05:24.774088+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.233259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.233259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":221,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.781408+0000","last_change":"2024-09-13T13:04:02.396436+0000","last_active":"2024-09-13T13:05:24.781408+0000","last_peered":"2024-09-13T13:05:24.781408+0000","last_clean":"2024-09-13T13:05:24.781408+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:05:24.781408+0000","last_undegraded":"2024-09-13T13:05:24.781408+0000","last_fullsized":"2024-09-13T13:05:24.781408+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:02.396342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.396342+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:24.818400+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":196,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.767424+0000","last_change":"2024-09-13T13:04:02.765671+0000","last_active":"2024-09-13T13:05:24.767424+0000","last_peered":"2024-09-13T13:05:24.767424+0000","last_clean":"2024-09-13T13:05:24.767424+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:05:24.767424+0000","last_undegraded":"2024-09-13T13:05:24.767424+0000","last_fullsized":"2024-09-13T13:05:24.767424+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.021662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.021662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:18.072524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":209,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.788027+0000","last_change":"2024-09-13T13:05:00.690387+0000","last_active":"2024-09-13T13:05:24.788027+0000","last_peered":"2024-09-13T13:05:24.788027+0000","last_clean":"2024-09-13T13:05:24.788027+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:05:24.788027+0000","last_undegraded":"2024-09-13T13:05:24.788027+0000","last_fullsized":"2024-09-13T13:05:24.788027+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:05:00.690308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:00.690308+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:03.644084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":205,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.768393+0000","last_change":"2024-09-13T13:04:56.066382+0000","last_active":"2024-09-13T13:05:24.768393+0000","last_peered":"2024-09-13T13:05:24.768393+0000","last_clean":"2024-09-13T13:05:24.768393+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:05:24.768393+0000","last_undegraded":"2024-09-13T13:05:24.768393+0000","last_fullsized":"2024-09-13T13:05:24.768393+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:56.066326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.066326+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.781998+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:26.621813+0000","last_change":"2024-09-13T13:05:26.621813+0000","last_active":"2024-09-13T13:05:26.621813+0000","last_peered":"2024-09-13T13:05:26.621813+0000","last_clean":"2024-09-13T13:05:26.621813+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:05:26.621813+0000","last_undegraded":"2024-09-13T13:05:26.621813+0000","last_fullsized":"2024-09-13T13:05:26.621813+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:26.621732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.621732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:48.964955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":180,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.772142+0000","last_change":"2024-09-13T13:02:39.607712+0000","last_active":"2024-09-13T13:05:24.772142+0000","last_peered":"2024-09-13T13:05:24.772142+0000","last_clean":"2024-09-13T13:05:24.772142+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:05:24.772142+0000","last_undegraded":"2024-09-13T13:05:24.772142+0000","last_fullsized":"2024-09-13T13:05:24.772142+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.607646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.607646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":200,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.787387+0000","last_change":"2024-09-13T13:03:40.665853+0000","last_active":"2024-09-13T13:05:24.787387+0000","last_peered":"2024-09-13T13:05:24.787387+0000","last_clean":"2024-09-13T13:05:24.787387+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:05:24.787387+0000","last_undegraded":"2024-09-13T13:05:24.787387+0000","last_fullsized":"2024-09-13T13:05:24.787387+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:40.665805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:40.665805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":207,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.768015+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:05:24.768015+0000","last_peered":"2024-09-13T13:05:24.768015+0000","last_clean":"2024-09-13T13:05:24.768015+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:05:24.768015+0000","last_undegraded":"2024-09-13T13:05:24.768015+0000","last_fullsized":"2024-09-13T13:05:24.768015+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:15.096451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":184,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.772112+0000","last_change":"2024-09-13T13:05:10.383826+0000","last_active":"2024-09-13T13:05:24.772112+0000","last_peered":"2024-09-13T13:05:24.772112+0000","last_clean":"2024-09-13T13:05:24.772112+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:05:24.772112+0000","last_undegraded":"2024-09-13T13:05:24.772112+0000","last_fullsized":"2024-09-13T13:05:24.772112+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.383756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.383756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:33.805546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":172,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.787097+0000","last_change":"2024-09-13T13:01:08.213608+0000","last_active":"2024-09-13T13:05:24.787097+0000","last_peered":"2024-09-13T13:05:24.787097+0000","last_clean":"2024-09-13T13:05:24.787097+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:05:24.787097+0000","last_undegraded":"2024-09-13T13:05:24.787097+0000","last_fullsized":"2024-09-13T13:05:24.787097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.558201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.769963+0000","last_change":"2024-09-13T13:04:54.016136+0000","last_active":"2024-09-13T13:05:24.769963+0000","last_peered":"2024-09-13T13:05:24.769963+0000","last_clean":"2024-09-13T13:05:24.769963+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:05:24.769963+0000","last_undegraded":"2024-09-13T13:05:24.769963+0000","last_fullsized":"2024-09-13T13:05:24.769963+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:04:54.016064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.016064+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.987760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.775443+0000","last_change":"2024-09-13T13:02:32.590355+0000","last_active":"2024-09-13T13:05:24.775443+0000","last_peered":"2024-09-13T13:05:24.775443+0000","last_clean":"2024-09-13T13:05:24.775443+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:05:24.775443+0000","last_undegraded":"2024-09-13T13:05:24.775443+0000","last_fullsized":"2024-09-13T13:05:24.775443+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:32.590282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.590282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":222,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.765871+0000","last_change":"2024-09-13T13:03:19.290733+0000","last_active":"2024-09-13T13:05:24.765871+0000","last_peered":"2024-09-13T13:05:24.765871+0000","last_clean":"2024-09-13T13:05:24.765871+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:05:24.765871+0000","last_undegraded":"2024-09-13T13:05:24.765871+0000","last_fullsized":"2024-09-13T13:05:24.765871+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:03:19.290652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:19.290652+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:24.865069+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":185,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.784359+0000","last_change":"2024-09-13T13:02:45.202865+0000","last_active":"2024-09-13T13:05:24.784359+0000","last_peered":"2024-09-13T13:05:24.784359+0000","last_clean":"2024-09-13T13:05:24.784359+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:05:24.784359+0000","last_undegraded":"2024-09-13T13:05:24.784359+0000","last_fullsized":"2024-09-13T13:05:24.784359+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.202802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.202802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.774593+0000","last_change":"2024-09-13T13:04:03.384955+0000","last_active":"2024-09-13T13:05:24.774593+0000","last_peered":"2024-09-13T13:05:24.774593+0000","last_clean":"2024-09-13T13:05:24.774593+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:05:24.774593+0000","last_undegraded":"2024-09-13T13:05:24.774593+0000","last_fullsized":"2024-09-13T13:05:24.774593+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.384901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.384901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:24.976897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":193,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.768479+0000","last_change":"2024-09-13T13:04:57.103213+0000","last_active":"2024-09-13T13:05:24.768479+0000","last_peered":"2024-09-13T13:05:24.768479+0000","last_clean":"2024-09-13T13:05:24.768479+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:05:24.768479+0000","last_undegraded":"2024-09-13T13:05:24.768479+0000","last_fullsized":"2024-09-13T13:05:24.768479+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.103167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.103167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.050420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":203,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.773626+0000","last_change":"2024-09-13T13:01:04.195126+0000","last_active":"2024-09-13T13:05:24.773626+0000","last_peered":"2024-09-13T13:05:24.773626+0000","last_clean":"2024-09-13T13:05:24.773626+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:05:24.773626+0000","last_undegraded":"2024-09-13T13:05:24.773626+0000","last_fullsized":"2024-09-13T13:05:24.773626+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":185,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.769506+0000","last_change":"2024-09-13T13:03:11.014220+0000","last_active":"2024-09-13T13:05:24.769506+0000","last_peered":"2024-09-13T13:05:24.769506+0000","last_clean":"2024-09-13T13:05:24.769506+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:05:24.769506+0000","last_undegraded":"2024-09-13T13:05:24.769506+0000","last_fullsized":"2024-09-13T13:05:24.769506+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:11.014157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:11.014157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.773432+0000","last_change":"2024-09-13T13:05:00.457611+0000","last_active":"2024-09-13T13:05:24.773432+0000","last_peered":"2024-09-13T13:05:24.773432+0000","last_clean":"2024-09-13T13:05:24.773432+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:05:24.773432+0000","last_undegraded":"2024-09-13T13:05:24.773432+0000","last_fullsized":"2024-09-13T13:05:24.773432+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:05:00.457562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:00.457562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:25.611525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.790548+0000","last_change":"2024-09-13T13:03:12.276224+0000","last_active":"2024-09-13T13:05:24.790548+0000","last_peered":"2024-09-13T13:05:24.790548+0000","last_clean":"2024-09-13T13:05:24.790548+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:05:24.790548+0000","last_undegraded":"2024-09-13T13:05:24.790548+0000","last_fullsized":"2024-09-13T13:05:24.790548+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:12.276136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.276136+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":204,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.781711+0000","last_change":"2024-09-13T13:02:34.593511+0000","last_active":"2024-09-13T13:05:24.781711+0000","last_peered":"2024-09-13T13:05:24.781711+0000","last_clean":"2024-09-13T13:05:24.781711+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:05:24.781711+0000","last_undegraded":"2024-09-13T13:05:24.781711+0000","last_fullsized":"2024-09-13T13:05:24.781711+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:34.593431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:34.593431+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":196,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.771459+0000","last_change":"2024-09-13T13:04:02.765069+0000","last_active":"2024-09-13T13:05:24.771459+0000","last_peered":"2024-09-13T13:05:24.771459+0000","last_clean":"2024-09-13T13:05:24.771459+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:05:24.771459+0000","last_undegraded":"2024-09-13T13:05:24.771459+0000","last_fullsized":"2024-09-13T13:05:24.771459+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:46.604669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.604669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"146'12723","reported_seq":18632,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:30.866998+0000","last_change":"2024-09-13T13:04:49.780269+0000","last_active":"2024-09-13T13:05:30.866998+0000","last_peered":"2024-09-13T13:05:30.866998+0000","last_clean":"2024-09-13T13:05:30.866998+0000","last_became_active":"2024-09-13T13:04:49.765876+0000","last_became_peered":"2024-09-13T13:04:49.765876+0000","last_unstale":"2024-09-13T13:05:30.866998+0000","last_undegraded":"2024-09-13T13:05:30.866998+0000","last_fullsized":"2024-09-13T13:05:30.866998+0000","mapping_epoch":136,"log_start":"110'6448","ondisk_log_start":"110'6448","created":84,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150405120,"num_objects":2295,"num_object_clones":0,"num_object_copies":6885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12723,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2909,"num_bytes_recovered":147193856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":222,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:25.691233+0000","last_change":"2024-09-13T13:05:25.691233+0000","last_active":"2024-09-13T13:05:25.691233+0000","last_peered":"2024-09-13T13:05:25.691233+0000","last_clean":"2024-09-13T13:05:25.691233+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:05:25.691233+0000","last_undegraded":"2024-09-13T13:05:25.691233+0000","last_fullsized":"2024-09-13T13:05:25.691233+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:05:25.689487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:25.689487+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:53.347573+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":201,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.774273+0000","last_change":"2024-09-13T13:03:42.443399+0000","last_active":"2024-09-13T13:05:24.774273+0000","last_peered":"2024-09-13T13:05:24.774273+0000","last_clean":"2024-09-13T13:05:24.774273+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:05:24.774273+0000","last_undegraded":"2024-09-13T13:05:24.774273+0000","last_fullsized":"2024-09-13T13:05:24.774273+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:42.443339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:42.443339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":248,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:27.286531+0000","last_change":"2024-09-13T13:02:32.013587+0000","last_active":"2024-09-13T13:05:27.286531+0000","last_peered":"2024-09-13T13:05:27.286531+0000","last_clean":"2024-09-13T13:05:27.286531+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:05:27.286531+0000","last_undegraded":"2024-09-13T13:05:27.286531+0000","last_fullsized":"2024-09-13T13:05:27.286531+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:32.013505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.013505+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:14.086911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":180,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.787642+0000","last_change":"2024-09-13T13:04:48.684906+0000","last_active":"2024-09-13T13:05:24.787642+0000","last_peered":"2024-09-13T13:05:24.787642+0000","last_clean":"2024-09-13T13:05:24.787642+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:05:24.787642+0000","last_undegraded":"2024-09-13T13:05:24.787642+0000","last_fullsized":"2024-09-13T13:05:24.787642+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:48.684864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.684864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.491555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"146'12818","reported_seq":13031,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:33.210183+0000","last_change":"2024-09-13T13:03:41.936543+0000","last_active":"2024-09-13T13:05:33.210183+0000","last_peered":"2024-09-13T13:05:33.210183+0000","last_clean":"2024-09-13T13:05:33.210183+0000","last_became_active":"2024-09-13T13:02:12.369723+0000","last_became_peered":"2024-09-13T13:02:12.369723+0000","last_unstale":"2024-09-13T13:05:33.210183+0000","last_undegraded":"2024-09-13T13:05:33.210183+0000","last_fullsized":"2024-09-13T13:05:33.210183+0000","mapping_epoch":84,"log_start":"120'8200","ondisk_log_start":"120'8200","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6909","last_scrub_stamp":"2024-09-13T13:03:41.936218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:03:41.936218+0000","objects_scrubbed":6836,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151388160,"num_objects":2310,"num_object_clones":0,"num_object_copies":6930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12818,"num_write_kb":484096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":221,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.774142+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:05:24.774142+0000","last_peered":"2024-09-13T13:05:24.774142+0000","last_clean":"2024-09-13T13:05:24.774142+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:05:24.774142+0000","last_undegraded":"2024-09-13T13:05:24.774142+0000","last_fullsized":"2024-09-13T13:05:24.774142+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":199,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.775468+0000","last_change":"2024-09-13T13:03:18.516327+0000","last_active":"2024-09-13T13:05:24.775468+0000","last_peered":"2024-09-13T13:05:24.775468+0000","last_clean":"2024-09-13T13:05:24.775468+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:05:24.775468+0000","last_undegraded":"2024-09-13T13:05:24.775468+0000","last_fullsized":"2024-09-13T13:05:24.775468+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:18.516251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.516251+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:29.213998+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:05:29.213998+0000","last_peered":"2024-09-13T13:05:29.213998+0000","last_clean":"2024-09-13T13:05:29.213998+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:05:29.213998+0000","last_undegraded":"2024-09-13T13:05:29.213998+0000","last_fullsized":"2024-09-13T13:05:29.213998+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:40.320454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.768333+0000","last_change":"2024-09-13T13:04:53.006515+0000","last_active":"2024-09-13T13:05:24.768333+0000","last_peered":"2024-09-13T13:05:24.768333+0000","last_clean":"2024-09-13T13:05:24.768333+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:05:24.768333+0000","last_undegraded":"2024-09-13T13:05:24.768333+0000","last_fullsized":"2024-09-13T13:05:24.768333+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:53.006454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.006454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.531241+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"146'12836","reported_seq":12978,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:33.222727+0000","last_change":"2024-09-13T13:03:45.761068+0000","last_active":"2024-09-13T13:05:33.222727+0000","last_peered":"2024-09-13T13:05:33.222727+0000","last_clean":"2024-09-13T13:05:33.222727+0000","last_became_active":"2024-09-13T13:02:12.373492+0000","last_became_peered":"2024-09-13T13:02:12.373492+0000","last_unstale":"2024-09-13T13:05:33.222727+0000","last_undegraded":"2024-09-13T13:05:33.222727+0000","last_fullsized":"2024-09-13T13:05:33.222727+0000","mapping_epoch":84,"log_start":"120'8200","ondisk_log_start":"120'8200","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":299,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145489920,"num_objects":2220,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12836,"num_write_kb":481792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":222,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.767622+0000","last_change":"2024-09-13T13:04:02.772914+0000","last_active":"2024-09-13T13:05:24.767622+0000","last_peered":"2024-09-13T13:05:24.767622+0000","last_clean":"2024-09-13T13:05:24.767622+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:05:24.767622+0000","last_undegraded":"2024-09-13T13:05:24.767622+0000","last_fullsized":"2024-09-13T13:05:24.767622+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:02:43.947615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.947615+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.781974+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:05:24.781974+0000","last_peered":"2024-09-13T13:05:24.781974+0000","last_clean":"2024-09-13T13:05:24.781974+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:05:24.781974+0000","last_undegraded":"2024-09-13T13:05:24.781974+0000","last_fullsized":"2024-09-13T13:05:24.781974+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":240,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:27.287548+0000","last_change":"2024-09-13T13:01:06.211211+0000","last_active":"2024-09-13T13:05:27.287548+0000","last_peered":"2024-09-13T13:05:27.287548+0000","last_clean":"2024-09-13T13:05:27.287548+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:05:27.287548+0000","last_undegraded":"2024-09-13T13:05:27.287548+0000","last_fullsized":"2024-09-13T13:05:27.287548+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.772074+0000","last_change":"2024-09-13T13:04:57.439116+0000","last_active":"2024-09-13T13:05:24.772074+0000","last_peered":"2024-09-13T13:05:24.772074+0000","last_clean":"2024-09-13T13:05:24.772074+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:05:24.772074+0000","last_undegraded":"2024-09-13T13:05:24.772074+0000","last_fullsized":"2024-09-13T13:05:24.772074+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.439053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.439053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:04.992285+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"146'12848","reported_seq":18830,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:32.928414+0000","last_change":"2024-09-13T13:05:05.014474+0000","last_active":"2024-09-13T13:05:32.928414+0000","last_peered":"2024-09-13T13:05:32.928414+0000","last_clean":"2024-09-13T13:05:32.928414+0000","last_became_active":"2024-09-13T13:04:25.922485+0000","last_became_peered":"2024-09-13T13:04:25.922485+0000","last_unstale":"2024-09-13T13:05:32.928414+0000","last_undegraded":"2024-09-13T13:05:32.928414+0000","last_fullsized":"2024-09-13T13:05:32.928414+0000","mapping_epoch":128,"log_start":"116'8000","ondisk_log_start":"116'8000","created":84,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'11284","last_scrub_stamp":"2024-09-13T13:05:05.014171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:05.014171+0000","objects_scrubbed":3783,"log_size":4848,"log_dups_size":0,"ondisk_log_size":4848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:18.664629+0000","scrub_duration":934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143654912,"num_objects":2192,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12848,"num_write_kb":481280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2891,"num_bytes_recovered":148570112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":197,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.783158+0000","last_change":"2024-09-13T13:01:04.199011+0000","last_active":"2024-09-13T13:05:24.783158+0000","last_peered":"2024-09-13T13:05:24.783158+0000","last_clean":"2024-09-13T13:05:24.783158+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:05:24.783158+0000","last_undegraded":"2024-09-13T13:05:24.783158+0000","last_fullsized":"2024-09-13T13:05:24.783158+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.733108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":193,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.788109+0000","last_change":"2024-09-13T13:05:04.743102+0000","last_active":"2024-09-13T13:05:24.788109+0000","last_peered":"2024-09-13T13:05:24.788109+0000","last_clean":"2024-09-13T13:05:24.788109+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:05:24.788109+0000","last_undegraded":"2024-09-13T13:05:24.788109+0000","last_fullsized":"2024-09-13T13:05:24.788109+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:04.743025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:04.743025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:32.082984+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":254,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:27.286906+0000","last_change":"2024-09-13T13:04:02.764350+0000","last_active":"2024-09-13T13:05:27.286906+0000","last_peered":"2024-09-13T13:05:27.286906+0000","last_clean":"2024-09-13T13:05:27.286906+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:05:27.286906+0000","last_undegraded":"2024-09-13T13:05:27.286906+0000","last_fullsized":"2024-09-13T13:05:27.286906+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.772383+0000","last_change":"2024-09-13T13:02:43.558487+0000","last_active":"2024-09-13T13:05:24.772383+0000","last_peered":"2024-09-13T13:05:24.772383+0000","last_clean":"2024-09-13T13:05:24.772383+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:05:24.772383+0000","last_undegraded":"2024-09-13T13:05:24.772383+0000","last_fullsized":"2024-09-13T13:05:24.772383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:43.558418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.558418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"146'12680","reported_seq":12814,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:32.913596+0000","last_change":"2024-09-13T13:02:12.379536+0000","last_active":"2024-09-13T13:05:32.913596+0000","last_peered":"2024-09-13T13:05:32.913596+0000","last_clean":"2024-09-13T13:05:32.913596+0000","last_became_active":"2024-09-13T13:02:12.379226+0000","last_became_peered":"2024-09-13T13:02:12.379226+0000","last_unstale":"2024-09-13T13:05:32.913596+0000","last_undegraded":"2024-09-13T13:05:32.913596+0000","last_fullsized":"2024-09-13T13:05:32.913596+0000","mapping_epoch":84,"log_start":"114'6900","ondisk_log_start":"114'6900","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.567939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148111360,"num_objects":2260,"num_object_clones":0,"num_object_copies":6780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12680,"num_write_kb":478080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":225,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.772954+0000","last_change":"2024-09-13T13:02:29.532879+0000","last_active":"2024-09-13T13:05:24.772954+0000","last_peered":"2024-09-13T13:05:24.772954+0000","last_clean":"2024-09-13T13:05:24.772954+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:05:24.772954+0000","last_undegraded":"2024-09-13T13:05:24.772954+0000","last_fullsized":"2024-09-13T13:05:24.772954+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:29.532779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.532779+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.789967+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:05:24.789967+0000","last_peered":"2024-09-13T13:05:24.789967+0000","last_clean":"2024-09-13T13:05:24.789967+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:05:24.789967+0000","last_undegraded":"2024-09-13T13:05:24.789967+0000","last_fullsized":"2024-09-13T13:05:24.789967+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:20.738953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":146,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.770796+0000","last_change":"2024-09-13T13:04:02.765838+0000","last_active":"2024-09-13T13:05:24.770796+0000","last_peered":"2024-09-13T13:05:24.770796+0000","last_clean":"2024-09-13T13:05:24.770796+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:05:24.770796+0000","last_undegraded":"2024-09-13T13:05:24.770796+0000","last_fullsized":"2024-09-13T13:05:24.770796+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.563708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.563708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":201,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:30.829125+0000","last_change":"2024-09-13T13:05:30.829125+0000","last_active":"2024-09-13T13:05:30.829125+0000","last_peered":"2024-09-13T13:05:30.829125+0000","last_clean":"2024-09-13T13:05:30.829125+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:05:30.829125+0000","last_undegraded":"2024-09-13T13:05:30.829125+0000","last_fullsized":"2024-09-13T13:05:30.829125+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:30.828810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:30.828810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:43.724154+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"146'12995","reported_seq":17540,"reported_epoch":146,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T13:05:33.459927+0000","last_change":"2024-09-13T13:04:47.345676+0000","last_active":"2024-09-13T13:05:33.459927+0000","last_peered":"2024-09-13T13:05:33.459927+0000","last_clean":"2024-09-13T13:03:14.460355+0000","last_became_active":"2024-09-13T13:04:03.878596+0000","last_became_peered":"2024-09-13T13:04:03.878596+0000","last_unstale":"2024-09-13T13:05:33.459927+0000","last_undegraded":"2024-09-13T13:04:03.760406+0000","last_fullsized":"2024-09-13T13:04:03.744969+0000","mapping_epoch":119,"log_start":"115'7400","ondisk_log_start":"115'7400","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5595,"log_dups_size":0,"ondisk_log_size":5595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142671872,"num_objects":2177,"num_object_clones":0,"num_object_copies":6531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12995,"num_write_kb":485504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2202,"num_bytes_recovered":144244736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":2110}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":293,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:27.286540+0000","last_change":"2024-09-13T13:02:39.183839+0000","last_active":"2024-09-13T13:05:27.286540+0000","last_peered":"2024-09-13T13:05:27.286540+0000","last_clean":"2024-09-13T13:05:27.286540+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:05:27.286540+0000","last_undegraded":"2024-09-13T13:05:27.286540+0000","last_fullsized":"2024-09-13T13:05:27.286540+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:02:39.183740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.183740+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":205,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.769078+0000","last_change":"2024-09-13T13:04:03.850983+0000","last_active":"2024-09-13T13:05:24.769078+0000","last_peered":"2024-09-13T13:05:24.769078+0000","last_clean":"2024-09-13T13:05:24.769078+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:05:24.769078+0000","last_undegraded":"2024-09-13T13:05:24.769078+0000","last_fullsized":"2024-09-13T13:05:24.769078+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.850898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.850898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":212,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.787852+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:05:24.787852+0000","last_peered":"2024-09-13T13:05:24.787852+0000","last_clean":"2024-09-13T13:05:24.787852+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:05:24.787852+0000","last_undegraded":"2024-09-13T13:05:24.787852+0000","last_fullsized":"2024-09-13T13:05:24.787852+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:26.941315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":185,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.776320+0000","last_change":"2024-09-13T13:02:35.566666+0000","last_active":"2024-09-13T13:05:24.776320+0000","last_peered":"2024-09-13T13:05:24.776320+0000","last_clean":"2024-09-13T13:05:24.776320+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:05:24.776320+0000","last_undegraded":"2024-09-13T13:05:24.776320+0000","last_fullsized":"2024-09-13T13:05:24.776320+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:35.566602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:35.566602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.773661+0000","last_change":"2024-09-13T13:04:01.406226+0000","last_active":"2024-09-13T13:05:24.773661+0000","last_peered":"2024-09-13T13:05:24.773661+0000","last_clean":"2024-09-13T13:05:24.773661+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:05:24.773661+0000","last_undegraded":"2024-09-13T13:05:24.773661+0000","last_fullsized":"2024-09-13T13:05:24.773661+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:01.406167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:01.406167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"146'12898","reported_seq":13034,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:33.449904+0000","last_change":"2024-09-13T13:02:12.380398+0000","last_active":"2024-09-13T13:05:33.449904+0000","last_peered":"2024-09-13T13:05:33.449904+0000","last_clean":"2024-09-13T13:05:33.449904+0000","last_became_active":"2024-09-13T13:02:12.379331+0000","last_became_peered":"2024-09-13T13:02:12.379331+0000","last_unstale":"2024-09-13T13:05:33.449904+0000","last_undegraded":"2024-09-13T13:05:33.449904+0000","last_fullsized":"2024-09-13T13:05:33.449904+0000","mapping_epoch":84,"log_start":"115'7300","ondisk_log_start":"115'7300","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143392768,"num_objects":2188,"num_object_clones":0,"num_object_copies":6564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12898,"num_write_kb":482752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":217,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.783640+0000","last_change":"2024-09-13T13:03:14.271569+0000","last_active":"2024-09-13T13:05:24.783640+0000","last_peered":"2024-09-13T13:05:24.783640+0000","last_clean":"2024-09-13T13:05:24.783640+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:05:24.783640+0000","last_undegraded":"2024-09-13T13:05:24.783640+0000","last_fullsized":"2024-09-13T13:05:24.783640+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:03:14.271480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.271480+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":205,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.784195+0000","last_change":"2024-09-13T13:04:02.759283+0000","last_active":"2024-09-13T13:05:24.784195+0000","last_peered":"2024-09-13T13:05:24.784195+0000","last_clean":"2024-09-13T13:05:24.784195+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:05:24.784195+0000","last_undegraded":"2024-09-13T13:05:24.784195+0000","last_fullsized":"2024-09-13T13:05:24.784195+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:40.214163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:40.214163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":201,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.765222+0000","last_change":"2024-09-13T13:03:18.232951+0000","last_active":"2024-09-13T13:05:24.765222+0000","last_peered":"2024-09-13T13:05:24.765222+0000","last_clean":"2024-09-13T13:05:24.765222+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:05:24.765222+0000","last_undegraded":"2024-09-13T13:05:24.765222+0000","last_fullsized":"2024-09-13T13:05:24.765222+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:03:18.232867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.232867+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:27.894650+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.769207+0000","last_change":"2024-09-13T13:03:14.282866+0000","last_active":"2024-09-13T13:05:24.769207+0000","last_peered":"2024-09-13T13:05:24.769207+0000","last_clean":"2024-09-13T13:05:24.769207+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:05:24.769207+0000","last_undegraded":"2024-09-13T13:05:24.769207+0000","last_fullsized":"2024-09-13T13:05:24.769207+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.349860+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.773604+0000","last_change":"2024-09-13T13:04:02.751582+0000","last_active":"2024-09-13T13:05:24.773604+0000","last_peered":"2024-09-13T13:05:24.773604+0000","last_clean":"2024-09-13T13:05:24.773604+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:05:24.773604+0000","last_undegraded":"2024-09-13T13:05:24.773604+0000","last_fullsized":"2024-09-13T13:05:24.773604+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.994729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.994729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.398308+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"146'13086","reported_seq":13297,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:32.870268+0000","last_change":"2024-09-13T13:03:37.311041+0000","last_active":"2024-09-13T13:05:32.870268+0000","last_peered":"2024-09-13T13:05:32.870268+0000","last_clean":"2024-09-13T13:05:32.870268+0000","last_became_active":"2024-09-13T13:02:12.379779+0000","last_became_peered":"2024-09-13T13:02:12.379779+0000","last_unstale":"2024-09-13T13:05:32.870268+0000","last_undegraded":"2024-09-13T13:05:32.870268+0000","last_fullsized":"2024-09-13T13:05:32.870268+0000","mapping_epoch":84,"log_start":"116'7900","ondisk_log_start":"116'7900","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'6772","last_scrub_stamp":"2024-09-13T13:03:37.310903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:03:37.310903+0000","objects_scrubbed":6730,"log_size":5186,"log_dups_size":0,"ondisk_log_size":5186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150732800,"num_objects":2300,"num_object_clones":0,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":492352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":207,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.781878+0000","last_change":"2024-09-13T13:03:18.514436+0000","last_active":"2024-09-13T13:05:24.781878+0000","last_peered":"2024-09-13T13:05:24.781878+0000","last_clean":"2024-09-13T13:05:24.781878+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:05:24.781878+0000","last_undegraded":"2024-09-13T13:05:24.781878+0000","last_fullsized":"2024-09-13T13:05:24.781878+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:03:18.514360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.514360+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":193,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.778272+0000","last_change":"2024-09-13T13:02:37.616313+0000","last_active":"2024-09-13T13:05:24.778272+0000","last_peered":"2024-09-13T13:05:24.778272+0000","last_clean":"2024-09-13T13:05:24.778272+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:05:24.778272+0000","last_undegraded":"2024-09-13T13:05:24.778272+0000","last_fullsized":"2024-09-13T13:05:24.778272+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:37.616224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.616224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":467,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:31.747390+0000","last_change":"2024-09-13T13:04:56.430717+0000","last_active":"2024-09-13T13:05:31.747390+0000","last_peered":"2024-09-13T13:05:31.747390+0000","last_clean":"2024-09-13T13:05:31.747390+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:05:31.747390+0000","last_undegraded":"2024-09-13T13:05:31.747390+0000","last_fullsized":"2024-09-13T13:05:31.747390+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:04:56.430670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.430670+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:59.639039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":265,"num_read_kb":260,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":185,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.788457+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:05:24.788457+0000","last_peered":"2024-09-13T13:05:24.788457+0000","last_clean":"2024-09-13T13:05:24.788457+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:05:24.788457+0000","last_undegraded":"2024-09-13T13:05:24.788457+0000","last_fullsized":"2024-09-13T13:05:24.788457+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:19.946825+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.788142+0000","last_change":"2024-09-13T13:02:36.222607+0000","last_active":"2024-09-13T13:05:24.788142+0000","last_peered":"2024-09-13T13:05:24.788142+0000","last_clean":"2024-09-13T13:05:24.788142+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:05:24.788142+0000","last_undegraded":"2024-09-13T13:05:24.788142+0000","last_fullsized":"2024-09-13T13:05:24.788142+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.222329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"146'12871","reported_seq":13007,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:33.425731+0000","last_change":"2024-09-13T13:02:12.379556+0000","last_active":"2024-09-13T13:05:33.425731+0000","last_peered":"2024-09-13T13:05:33.425731+0000","last_clean":"2024-09-13T13:05:33.425731+0000","last_became_active":"2024-09-13T13:02:12.379255+0000","last_became_peered":"2024-09-13T13:02:12.379255+0000","last_unstale":"2024-09-13T13:05:33.425731+0000","last_undegraded":"2024-09-13T13:05:33.425731+0000","last_fullsized":"2024-09-13T13:05:33.425731+0000","mapping_epoch":84,"log_start":"115'7300","ondisk_log_start":"115'7300","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5571,"log_dups_size":0,"ondisk_log_size":5571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147783680,"num_objects":2255,"num_object_clones":0,"num_object_copies":6765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12871,"num_write_kb":484032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":132,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.772976+0000","last_change":"2024-09-13T13:04:02.786042+0000","last_active":"2024-09-13T13:05:24.772976+0000","last_peered":"2024-09-13T13:05:24.772976+0000","last_clean":"2024-09-13T13:05:24.772976+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:05:24.772976+0000","last_undegraded":"2024-09-13T13:05:24.772976+0000","last_fullsized":"2024-09-13T13:05:24.772976+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":204,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.776272+0000","last_change":"2024-09-13T13:04:02.765370+0000","last_active":"2024-09-13T13:05:24.776272+0000","last_peered":"2024-09-13T13:05:24.776272+0000","last_clean":"2024-09-13T13:05:24.776272+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:05:24.776272+0000","last_undegraded":"2024-09-13T13:05:24.776272+0000","last_fullsized":"2024-09-13T13:05:24.776272+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:04.228002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:04.228002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":249,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:28.373960+0000","last_change":"2024-09-13T13:05:28.373960+0000","last_active":"2024-09-13T13:05:28.373960+0000","last_peered":"2024-09-13T13:05:28.373960+0000","last_clean":"2024-09-13T13:05:28.373960+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:05:28.373960+0000","last_undegraded":"2024-09-13T13:05:28.373960+0000","last_fullsized":"2024-09-13T13:05:28.373960+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:05:28.373902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:05:28.373902+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:33.407719+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":185,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.773489+0000","last_change":"2024-09-13T13:04:51.395273+0000","last_active":"2024-09-13T13:05:24.773489+0000","last_peered":"2024-09-13T13:05:24.773489+0000","last_clean":"2024-09-13T13:05:24.773489+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:05:24.773489+0000","last_undegraded":"2024-09-13T13:05:24.773489+0000","last_fullsized":"2024-09-13T13:05:24.773489+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:51.395204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:51.395204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:56.927380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":189,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:31.564860+0000","last_change":"2024-09-13T13:05:31.564860+0000","last_active":"2024-09-13T13:05:31.564860+0000","last_peered":"2024-09-13T13:05:31.564860+0000","last_clean":"2024-09-13T13:05:31.564860+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:05:31.564860+0000","last_undegraded":"2024-09-13T13:05:31.564860+0000","last_fullsized":"2024-09-13T13:05:31.564860+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:31.564783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:31.564783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:41.885010+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"146'12881","reported_seq":19051,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:30.844696+0000","last_change":"2024-09-13T13:05:15.349567+0000","last_active":"2024-09-13T13:05:30.844696+0000","last_peered":"2024-09-13T13:05:30.844696+0000","last_clean":"2024-09-13T13:05:30.844696+0000","last_became_active":"2024-09-13T13:05:15.326923+0000","last_became_peered":"2024-09-13T13:05:15.326923+0000","last_unstale":"2024-09-13T13:05:30.844696+0000","last_undegraded":"2024-09-13T13:05:30.844696+0000","last_fullsized":"2024-09-13T13:05:30.844696+0000","mapping_epoch":143,"log_start":"111'6585","ondisk_log_start":"111'6585","created":84,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":6296,"log_dups_size":0,"ondisk_log_size":6296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158597144,"num_objects":2421,"num_object_clones":0,"num_object_copies":7263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2421,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12881,"num_write_kb":489601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3008,"num_bytes_recovered":155058200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":219,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.770221+0000","last_change":"2024-09-13T13:04:02.832992+0000","last_active":"2024-09-13T13:05:24.770221+0000","last_peered":"2024-09-13T13:05:24.770221+0000","last_clean":"2024-09-13T13:05:24.770221+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:05:24.770221+0000","last_undegraded":"2024-09-13T13:05:24.770221+0000","last_fullsized":"2024-09-13T13:05:24.770221+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:04:02.832935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.832935+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.767889+0000","last_change":"2024-09-13T13:02:45.015634+0000","last_active":"2024-09-13T13:05:24.767889+0000","last_peered":"2024-09-13T13:05:24.767889+0000","last_clean":"2024-09-13T13:05:24.767889+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:05:24.767889+0000","last_undegraded":"2024-09-13T13:05:24.767889+0000","last_fullsized":"2024-09-13T13:05:24.767889+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.015577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.015577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.095405+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.788262+0000","last_change":"2024-09-13T13:04:47.680651+0000","last_active":"2024-09-13T13:05:24.788262+0000","last_peered":"2024-09-13T13:05:24.788262+0000","last_clean":"2024-09-13T13:05:24.788262+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:05:24.788262+0000","last_undegraded":"2024-09-13T13:05:24.788262+0000","last_fullsized":"2024-09-13T13:05:24.788262+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.680579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.680579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:07.566204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":191,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.774324+0000","last_change":"2024-09-13T13:04:02.757378+0000","last_active":"2024-09-13T13:05:24.774324+0000","last_peered":"2024-09-13T13:05:24.774324+0000","last_clean":"2024-09-13T13:05:24.774324+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:05:24.774324+0000","last_undegraded":"2024-09-13T13:05:24.774324+0000","last_fullsized":"2024-09-13T13:05:24.774324+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.213470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.213470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"146'12436","reported_seq":12696,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:33.069198+0000","last_change":"2024-09-13T13:05:03.355522+0000","last_active":"2024-09-13T13:05:33.069198+0000","last_peered":"2024-09-13T13:05:33.069198+0000","last_clean":"2024-09-13T13:05:33.069198+0000","last_became_active":"2024-09-13T13:02:12.375509+0000","last_became_peered":"2024-09-13T13:02:12.375509+0000","last_unstale":"2024-09-13T13:05:33.069198+0000","last_undegraded":"2024-09-13T13:05:33.069198+0000","last_fullsized":"2024-09-13T13:05:33.069198+0000","mapping_epoch":84,"log_start":"108'5600","ondisk_log_start":"108'5600","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'10840","last_scrub_stamp":"2024-09-13T13:05:03.354789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:03.354789+0000","objects_scrubbed":3912,"log_size":6836,"log_dups_size":0,"ondisk_log_size":6836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.025217+0000","scrub_duration":2101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148242432,"num_objects":2262,"num_object_clones":0,"num_object_copies":6786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12436,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":217,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.770612+0000","last_change":"2024-09-13T13:03:12.028812+0000","last_active":"2024-09-13T13:05:24.770612+0000","last_peered":"2024-09-13T13:05:24.770612+0000","last_clean":"2024-09-13T13:05:24.770612+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:05:24.770612+0000","last_undegraded":"2024-09-13T13:05:24.770612+0000","last_fullsized":"2024-09-13T13:05:24.770612+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:03:12.028719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.028719+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":162,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.771062+0000","last_change":"2024-09-13T13:04:02.764657+0000","last_active":"2024-09-13T13:05:24.771062+0000","last_peered":"2024-09-13T13:05:24.771062+0000","last_clean":"2024-09-13T13:05:24.771062+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:05:24.771062+0000","last_undegraded":"2024-09-13T13:05:24.771062+0000","last_fullsized":"2024-09-13T13:05:24.771062+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":177,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.788587+0000","last_change":"2024-09-13T13:01:06.208593+0000","last_active":"2024-09-13T13:05:24.788587+0000","last_peered":"2024-09-13T13:05:24.788587+0000","last_clean":"2024-09-13T13:05:24.788587+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:05:24.788587+0000","last_undegraded":"2024-09-13T13:05:24.788587+0000","last_fullsized":"2024-09-13T13:05:24.788587+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:22.718840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.765486+0000","last_change":"2024-09-13T13:03:22.282376+0000","last_active":"2024-09-13T13:05:24.765486+0000","last_peered":"2024-09-13T13:05:24.765486+0000","last_clean":"2024-09-13T13:05:24.765486+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:05:24.765486+0000","last_undegraded":"2024-09-13T13:05:24.765486+0000","last_fullsized":"2024-09-13T13:05:24.765486+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:22.282308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.282308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:32.881087+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"146'12943","reported_seq":13156,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:33.449498+0000","last_change":"2024-09-13T13:04:02.904392+0000","last_active":"2024-09-13T13:05:33.449498+0000","last_peered":"2024-09-13T13:05:33.449498+0000","last_clean":"2024-09-13T13:05:33.449498+0000","last_became_active":"2024-09-13T13:02:12.380229+0000","last_became_peered":"2024-09-13T13:02:12.380229+0000","last_unstale":"2024-09-13T13:05:33.449498+0000","last_undegraded":"2024-09-13T13:05:33.449498+0000","last_fullsized":"2024-09-13T13:05:33.449498+0000","mapping_epoch":84,"log_start":"115'7300","ondisk_log_start":"115'7300","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8189","last_scrub_stamp":"2024-09-13T13:04:02.904297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.904297+0000","objects_scrubbed":6969,"log_size":5643,"log_dups_size":0,"ondisk_log_size":5643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144769024,"num_objects":2209,"num_object_clones":0,"num_object_copies":6627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12943,"num_write_kb":484864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":215,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.769593+0000","last_change":"2024-09-13T13:04:59.091852+0000","last_active":"2024-09-13T13:05:24.769593+0000","last_peered":"2024-09-13T13:05:24.769593+0000","last_clean":"2024-09-13T13:05:24.769593+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:05:24.769593+0000","last_undegraded":"2024-09-13T13:05:24.769593+0000","last_fullsized":"2024-09-13T13:05:24.769593+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:04:59.091751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.091751+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.396141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":201,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.776996+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:05:24.776996+0000","last_peered":"2024-09-13T13:05:24.776996+0000","last_clean":"2024-09-13T13:05:24.776996+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:05:24.776996+0000","last_undegraded":"2024-09-13T13:05:24.776996+0000","last_fullsized":"2024-09-13T13:05:24.776996+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":256,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:32.256621+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:05:32.256621+0000","last_peered":"2024-09-13T13:05:32.256621+0000","last_clean":"2024-09-13T13:05:32.256621+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:05:32.256621+0000","last_undegraded":"2024-09-13T13:05:32.256621+0000","last_fullsized":"2024-09-13T13:05:32.256621+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:57.041771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.769875+0000","last_change":"2024-09-13T13:02:33.997780+0000","last_active":"2024-09-13T13:05:24.769875+0000","last_peered":"2024-09-13T13:05:24.769875+0000","last_clean":"2024-09-13T13:05:24.769875+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:05:24.769875+0000","last_undegraded":"2024-09-13T13:05:24.769875+0000","last_fullsized":"2024-09-13T13:05:24.769875+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.997740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.997740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.184374+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"146'12971","reported_seq":13230,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:33.072489+0000","last_change":"2024-09-13T13:05:08.464991+0000","last_active":"2024-09-13T13:05:33.072489+0000","last_peered":"2024-09-13T13:05:33.072489+0000","last_clean":"2024-09-13T13:05:33.072489+0000","last_became_active":"2024-09-13T13:02:12.375310+0000","last_became_peered":"2024-09-13T13:02:12.375310+0000","last_unstale":"2024-09-13T13:05:33.072489+0000","last_undegraded":"2024-09-13T13:05:33.072489+0000","last_fullsized":"2024-09-13T13:05:33.072489+0000","mapping_epoch":84,"log_start":"108'6100","ondisk_log_start":"108'6100","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'11530","last_scrub_stamp":"2024-09-13T13:05:08.464709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:08.464709+0000","objects_scrubbed":3682,"log_size":6871,"log_dups_size":0,"ondisk_log_size":6871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:11.110516+0000","scrub_duration":1226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144506880,"num_objects":2205,"num_object_clones":0,"num_object_copies":6615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12971,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":204,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.772852+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:05:24.772852+0000","last_peered":"2024-09-13T13:05:24.772852+0000","last_clean":"2024-09-13T13:05:24.772852+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:05:24.772852+0000","last_undegraded":"2024-09-13T13:05:24.772852+0000","last_fullsized":"2024-09-13T13:05:24.772852+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":204,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.787542+0000","last_change":"2024-09-13T13:04:02.748907+0000","last_active":"2024-09-13T13:05:24.787542+0000","last_peered":"2024-09-13T13:05:24.787542+0000","last_clean":"2024-09-13T13:05:24.787542+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:05:24.787542+0000","last_undegraded":"2024-09-13T13:05:24.787542+0000","last_fullsized":"2024-09-13T13:05:24.787542+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.305843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.305843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.771811+0000","last_change":"2024-09-13T13:04:56.183494+0000","last_active":"2024-09-13T13:05:24.771811+0000","last_peered":"2024-09-13T13:05:24.771811+0000","last_clean":"2024-09-13T13:05:24.771811+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:05:24.771811+0000","last_undegraded":"2024-09-13T13:05:24.771811+0000","last_fullsized":"2024-09-13T13:05:24.771811+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:56.183290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.183290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:07.795607+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":173,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.778230+0000","last_change":"2024-09-13T13:01:08.211229+0000","last_active":"2024-09-13T13:05:24.778230+0000","last_peered":"2024-09-13T13:05:24.778230+0000","last_clean":"2024-09-13T13:05:24.778230+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:05:24.778230+0000","last_undegraded":"2024-09-13T13:05:24.778230+0000","last_fullsized":"2024-09-13T13:05:24.778230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"146'13104","reported_seq":13240,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:33.425140+0000","last_change":"2024-09-13T13:02:12.379755+0000","last_active":"2024-09-13T13:05:33.425140+0000","last_peered":"2024-09-13T13:05:33.425140+0000","last_clean":"2024-09-13T13:05:33.425140+0000","last_became_active":"2024-09-13T13:02:12.379442+0000","last_became_peered":"2024-09-13T13:02:12.379442+0000","last_unstale":"2024-09-13T13:05:33.425140+0000","last_undegraded":"2024-09-13T13:05:33.425140+0000","last_fullsized":"2024-09-13T13:05:33.425140+0000","mapping_epoch":84,"log_start":"115'7500","ondisk_log_start":"115'7500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5604,"log_dups_size":0,"ondisk_log_size":5604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144310272,"num_objects":2202,"num_object_clones":0,"num_object_copies":6606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13104,"num_write_kb":489792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.770886+0000","last_change":"2024-09-13T13:04:02.760376+0000","last_active":"2024-09-13T13:05:24.770886+0000","last_peered":"2024-09-13T13:05:24.770886+0000","last_clean":"2024-09-13T13:05:24.770886+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:05:24.770886+0000","last_undegraded":"2024-09-13T13:05:24.770886+0000","last_fullsized":"2024-09-13T13:05:24.770886+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:42.593284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.593284+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":204,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.774058+0000","last_change":"2024-09-13T13:04:02.759538+0000","last_active":"2024-09-13T13:05:24.774058+0000","last_peered":"2024-09-13T13:05:24.774058+0000","last_clean":"2024-09-13T13:05:24.774058+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:05:24.774058+0000","last_undegraded":"2024-09-13T13:05:24.774058+0000","last_fullsized":"2024-09-13T13:05:24.774058+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:29.012312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.012312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:13.429060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.765450+0000","last_change":"2024-09-13T13:04:47.927556+0000","last_active":"2024-09-13T13:05:24.765450+0000","last_peered":"2024-09-13T13:05:24.765450+0000","last_clean":"2024-09-13T13:05:24.765450+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:05:24.765450+0000","last_undegraded":"2024-09-13T13:05:24.765450+0000","last_fullsized":"2024-09-13T13:05:24.765450+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.927432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.927432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.726893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.773205+0000","last_change":"2024-09-13T13:05:11.420568+0000","last_active":"2024-09-13T13:05:24.773205+0000","last_peered":"2024-09-13T13:05:24.773205+0000","last_clean":"2024-09-13T13:05:24.773205+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:05:24.773205+0000","last_undegraded":"2024-09-13T13:05:24.773205+0000","last_fullsized":"2024-09-13T13:05:24.773205+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:11.420503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:11.420503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:12.545408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"146'12922","reported_seq":13110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:33.222430+0000","last_change":"2024-09-13T13:03:45.723433+0000","last_active":"2024-09-13T13:05:33.222430+0000","last_peered":"2024-09-13T13:05:33.222430+0000","last_clean":"2024-09-13T13:05:33.222430+0000","last_became_active":"2024-09-13T13:02:12.376334+0000","last_became_peered":"2024-09-13T13:02:12.376334+0000","last_unstale":"2024-09-13T13:05:33.222430+0000","last_undegraded":"2024-09-13T13:05:33.222430+0000","last_fullsized":"2024-09-13T13:05:33.222430+0000","mapping_epoch":84,"log_start":"120'8300","ondisk_log_start":"120'8300","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":4774,"log_size":4622,"log_dups_size":0,"ondisk_log_size":4622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150601728,"num_objects":2298,"num_object_clones":0,"num_object_copies":6894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12922,"num_write_kb":487040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":213,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.767778+0000","last_change":"2024-09-13T13:02:28.990089+0000","last_active":"2024-09-13T13:05:24.767778+0000","last_peered":"2024-09-13T13:05:24.767778+0000","last_clean":"2024-09-13T13:05:24.767778+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:05:24.767778+0000","last_undegraded":"2024-09-13T13:05:24.767778+0000","last_fullsized":"2024-09-13T13:05:24.767778+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:02:28.990005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.990005+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.110320+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":209,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:29.345943+0000","last_change":"2024-09-13T13:05:29.345863+0000","last_active":"2024-09-13T13:05:29.345943+0000","last_peered":"2024-09-13T13:05:29.345943+0000","last_clean":"2024-09-13T13:05:29.345943+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:05:29.345943+0000","last_undegraded":"2024-09-13T13:05:29.345943+0000","last_fullsized":"2024-09-13T13:05:29.345943+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.345796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.345796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.394429+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:29.218178+0000","last_change":"2024-09-13T13:05:29.218178+0000","last_active":"2024-09-13T13:05:29.218178+0000","last_peered":"2024-09-13T13:05:29.218178+0000","last_clean":"2024-09-13T13:05:29.218178+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:05:29.218178+0000","last_undegraded":"2024-09-13T13:05:29.218178+0000","last_fullsized":"2024-09-13T13:05:29.218178+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.218119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.218119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:30.944093+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":180,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.776580+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:05:24.776580+0000","last_peered":"2024-09-13T13:05:24.776580+0000","last_clean":"2024-09-13T13:05:24.776580+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:05:24.776580+0000","last_undegraded":"2024-09-13T13:05:24.776580+0000","last_fullsized":"2024-09-13T13:05:24.776580+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"146'13068","reported_seq":19252,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:33.052142+0000","last_change":"2024-09-13T13:05:29.580927+0000","last_active":"2024-09-13T13:05:33.052142+0000","last_peered":"2024-09-13T13:05:33.052142+0000","last_clean":"2024-09-13T13:05:33.052142+0000","last_became_active":"2024-09-13T13:04:48.765805+0000","last_became_peered":"2024-09-13T13:04:48.765805+0000","last_unstale":"2024-09-13T13:05:33.052142+0000","last_undegraded":"2024-09-13T13:05:33.052142+0000","last_fullsized":"2024-09-13T13:05:33.052142+0000","mapping_epoch":135,"log_start":"116'8000","ondisk_log_start":"116'8000","created":84,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'12879","last_scrub_stamp":"2024-09-13T13:05:29.580736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.580736+0000","objects_scrubbed":2542,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:43.250181+0000","scrub_duration":768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152305664,"num_objects":2324,"num_object_clones":0,"num_object_copies":6972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13068,"num_write_kb":492544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3000,"num_bytes_recovered":156631040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":225,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.787199+0000","last_change":"2024-09-13T13:04:52.666294+0000","last_active":"2024-09-13T13:05:24.787199+0000","last_peered":"2024-09-13T13:05:24.787199+0000","last_clean":"2024-09-13T13:05:24.787199+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:05:24.787199+0000","last_undegraded":"2024-09-13T13:05:24.787199+0000","last_fullsized":"2024-09-13T13:05:24.787199+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:52.666222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:52.666222+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:57.559648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":213,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:27.422614+0000","last_change":"2024-09-13T13:05:27.422529+0000","last_active":"2024-09-13T13:05:27.422614+0000","last_peered":"2024-09-13T13:05:27.422614+0000","last_clean":"2024-09-13T13:05:27.422614+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:05:27.422614+0000","last_undegraded":"2024-09-13T13:05:27.422614+0000","last_fullsized":"2024-09-13T13:05:27.422614+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:05:27.422451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:27.422451+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.192286+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":265,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:27.928447+0000","last_change":"2024-09-13T13:04:02.759553+0000","last_active":"2024-09-13T13:05:27.928447+0000","last_peered":"2024-09-13T13:05:27.928447+0000","last_clean":"2024-09-13T13:05:27.928447+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:05:27.928447+0000","last_undegraded":"2024-09-13T13:05:27.928447+0000","last_fullsized":"2024-09-13T13:05:27.928447+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:45.959285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.959285+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.359173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.787557+0000","last_change":"2024-09-13T13:04:02.757776+0000","last_active":"2024-09-13T13:05:24.787557+0000","last_peered":"2024-09-13T13:05:24.787557+0000","last_clean":"2024-09-13T13:05:24.787557+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:05:24.787557+0000","last_undegraded":"2024-09-13T13:05:24.787557+0000","last_fullsized":"2024-09-13T13:05:24.787557+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:28.529954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.529954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.625121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":162,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.771178+0000","last_change":"2024-09-13T13:04:02.756666+0000","last_active":"2024-09-13T13:05:24.771178+0000","last_peered":"2024-09-13T13:05:24.771178+0000","last_clean":"2024-09-13T13:05:24.771178+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:05:24.771178+0000","last_undegraded":"2024-09-13T13:05:24.771178+0000","last_fullsized":"2024-09-13T13:05:24.771178+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":197,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.775340+0000","last_change":"2024-09-13T13:03:14.504339+0000","last_active":"2024-09-13T13:05:24.775340+0000","last_peered":"2024-09-13T13:05:24.775340+0000","last_clean":"2024-09-13T13:05:24.775340+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:05:24.775340+0000","last_undegraded":"2024-09-13T13:05:24.775340+0000","last_fullsized":"2024-09-13T13:05:24.775340+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:03:14.504263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.504263+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:27.619960+0000","last_change":"2024-09-13T13:05:27.619960+0000","last_active":"2024-09-13T13:05:27.619960+0000","last_peered":"2024-09-13T13:05:27.619960+0000","last_clean":"2024-09-13T13:05:27.619960+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:05:27.619960+0000","last_undegraded":"2024-09-13T13:05:27.619960+0000","last_fullsized":"2024-09-13T13:05:27.619960+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:27.619884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:27.619884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.490575+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":191,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.773536+0000","last_change":"2024-09-13T13:04:03.746106+0000","last_active":"2024-09-13T13:05:24.773536+0000","last_peered":"2024-09-13T13:05:24.773536+0000","last_clean":"2024-09-13T13:05:24.773536+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:05:24.773536+0000","last_undegraded":"2024-09-13T13:05:24.773536+0000","last_fullsized":"2024-09-13T13:05:24.773536+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.745793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.745793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":204,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.781942+0000","last_change":"2024-09-13T13:02:42.589951+0000","last_active":"2024-09-13T13:05:24.781942+0000","last_peered":"2024-09-13T13:05:24.781942+0000","last_clean":"2024-09-13T13:05:24.781942+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:05:24.781942+0000","last_undegraded":"2024-09-13T13:05:24.781942+0000","last_fullsized":"2024-09-13T13:05:24.781942+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:42.589866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.589866+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":203,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.772563+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:05:24.772563+0000","last_peered":"2024-09-13T13:05:24.772563+0000","last_clean":"2024-09-13T13:05:24.772563+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:05:24.772563+0000","last_undegraded":"2024-09-13T13:05:24.772563+0000","last_fullsized":"2024-09-13T13:05:24.772563+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.542132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":176,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.774675+0000","last_change":"2024-09-13T13:01:06.210328+0000","last_active":"2024-09-13T13:05:24.774675+0000","last_peered":"2024-09-13T13:05:24.774675+0000","last_clean":"2024-09-13T13:05:24.774675+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:05:24.774675+0000","last_undegraded":"2024-09-13T13:05:24.774675+0000","last_fullsized":"2024-09-13T13:05:24.774675+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.765137+0000","last_change":"2024-09-13T13:03:21.281950+0000","last_active":"2024-09-13T13:05:24.765137+0000","last_peered":"2024-09-13T13:05:24.765137+0000","last_clean":"2024-09-13T13:05:24.765137+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:05:24.765137+0000","last_undegraded":"2024-09-13T13:05:24.765137+0000","last_fullsized":"2024-09-13T13:05:24.765137+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:21.281870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:21.281870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:42.889031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":204,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.787948+0000","last_change":"2024-09-13T13:04:53.687731+0000","last_active":"2024-09-13T13:05:24.787948+0000","last_peered":"2024-09-13T13:05:24.787948+0000","last_clean":"2024-09-13T13:05:24.787948+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:05:24.787948+0000","last_undegraded":"2024-09-13T13:05:24.787948+0000","last_fullsized":"2024-09-13T13:05:24.787948+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.632415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.776024+0000","last_change":"2024-09-13T13:02:41.622369+0000","last_active":"2024-09-13T13:05:24.776024+0000","last_peered":"2024-09-13T13:05:24.776024+0000","last_clean":"2024-09-13T13:05:24.776024+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:05:24.776024+0000","last_undegraded":"2024-09-13T13:05:24.776024+0000","last_fullsized":"2024-09-13T13:05:24.776024+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.622292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.622292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":239,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:27.287929+0000","last_change":"2024-09-13T13:01:06.210251+0000","last_active":"2024-09-13T13:05:27.287929+0000","last_peered":"2024-09-13T13:05:27.287929+0000","last_clean":"2024-09-13T13:05:27.287929+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:05:27.287929+0000","last_undegraded":"2024-09-13T13:05:27.287929+0000","last_fullsized":"2024-09-13T13:05:27.287929+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.770887+0000","last_change":"2024-09-13T13:04:49.927118+0000","last_active":"2024-09-13T13:05:24.770887+0000","last_peered":"2024-09-13T13:05:24.770887+0000","last_clean":"2024-09-13T13:05:24.770887+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:05:24.770887+0000","last_undegraded":"2024-09-13T13:05:24.770887+0000","last_fullsized":"2024-09-13T13:05:24.770887+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:49.927073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:49.927073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:08.714937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":203,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.774832+0000","last_change":"2024-09-13T13:02:37.222400+0000","last_active":"2024-09-13T13:05:24.774832+0000","last_peered":"2024-09-13T13:05:24.774832+0000","last_clean":"2024-09-13T13:05:24.774832+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:05:24.774832+0000","last_undegraded":"2024-09-13T13:05:24.774832+0000","last_fullsized":"2024-09-13T13:05:24.774832+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:37.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.222329+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.772960+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:05:24.772960+0000","last_peered":"2024-09-13T13:05:24.772960+0000","last_clean":"2024-09-13T13:05:24.772960+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:05:24.772960+0000","last_undegraded":"2024-09-13T13:05:24.772960+0000","last_fullsized":"2024-09-13T13:05:24.772960+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:55.131839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":255,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:27.286639+0000","last_change":"2024-09-13T13:03:41.676060+0000","last_active":"2024-09-13T13:05:27.286639+0000","last_peered":"2024-09-13T13:05:27.286639+0000","last_clean":"2024-09-13T13:05:27.286639+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:05:27.286639+0000","last_undegraded":"2024-09-13T13:05:27.286639+0000","last_fullsized":"2024-09-13T13:05:27.286639+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:03:41.675978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:41.675978+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":180,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.779669+0000","last_change":"2024-09-13T13:02:40.615749+0000","last_active":"2024-09-13T13:05:24.779669+0000","last_peered":"2024-09-13T13:05:24.779669+0000","last_clean":"2024-09-13T13:05:24.779669+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:05:24.779669+0000","last_undegraded":"2024-09-13T13:05:24.779669+0000","last_fullsized":"2024-09-13T13:05:24.779669+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:40.615674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:40.615674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":227,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:26.186899+0000","last_change":"2024-09-13T13:05:26.186899+0000","last_active":"2024-09-13T13:05:26.186899+0000","last_peered":"2024-09-13T13:05:26.186899+0000","last_clean":"2024-09-13T13:05:26.186899+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:05:26.186899+0000","last_undegraded":"2024-09-13T13:05:26.186899+0000","last_fullsized":"2024-09-13T13:05:26.186899+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:05:26.186830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.186830+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:29.937721+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":212,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.769706+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:05:24.769706+0000","last_peered":"2024-09-13T13:05:24.769706+0000","last_clean":"2024-09-13T13:05:24.769706+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:05:24.769706+0000","last_undegraded":"2024-09-13T13:05:24.769706+0000","last_fullsized":"2024-09-13T13:05:24.769706+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:39.039395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":193,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.782998+0000","last_change":"2024-09-13T13:03:44.467611+0000","last_active":"2024-09-13T13:05:24.782998+0000","last_peered":"2024-09-13T13:05:24.782998+0000","last_clean":"2024-09-13T13:05:24.782998+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:05:24.782998+0000","last_undegraded":"2024-09-13T13:05:24.782998+0000","last_fullsized":"2024-09-13T13:05:24.782998+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:44.467297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:44.467297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":180,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.781100+0000","last_change":"2024-09-13T13:02:42.208021+0000","last_active":"2024-09-13T13:05:24.781100+0000","last_peered":"2024-09-13T13:05:24.781100+0000","last_clean":"2024-09-13T13:05:24.781100+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:05:24.781100+0000","last_undegraded":"2024-09-13T13:05:24.781100+0000","last_fullsized":"2024-09-13T13:05:24.781100+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:42.207956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.207956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":148,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.787371+0000","last_change":"2024-09-13T13:04:02.757522+0000","last_active":"2024-09-13T13:05:24.787371+0000","last_peered":"2024-09-13T13:05:24.787371+0000","last_clean":"2024-09-13T13:05:24.787371+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:05:24.787371+0000","last_undegraded":"2024-09-13T13:05:24.787371+0000","last_fullsized":"2024-09-13T13:05:24.787371+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:35.448404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:35.448404+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.773981+0000","last_change":"2024-09-13T13:02:39.921391+0000","last_active":"2024-09-13T13:05:24.773981+0000","last_peered":"2024-09-13T13:05:24.773981+0000","last_clean":"2024-09-13T13:05:24.773981+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:05:24.773981+0000","last_undegraded":"2024-09-13T13:05:24.773981+0000","last_fullsized":"2024-09-13T13:05:24.773981+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.921325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.921325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:20.381317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":176,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.776665+0000","last_change":"2024-09-13T13:01:06.211993+0000","last_active":"2024-09-13T13:05:24.776665+0000","last_peered":"2024-09-13T13:05:24.776665+0000","last_clean":"2024-09-13T13:05:24.776665+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:05:24.776665+0000","last_undegraded":"2024-09-13T13:05:24.776665+0000","last_fullsized":"2024-09-13T13:05:24.776665+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":180,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.771919+0000","last_change":"2024-09-13T13:02:36.586702+0000","last_active":"2024-09-13T13:05:24.771919+0000","last_peered":"2024-09-13T13:05:24.771919+0000","last_clean":"2024-09-13T13:05:24.771919+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:05:24.771919+0000","last_undegraded":"2024-09-13T13:05:24.771919+0000","last_fullsized":"2024-09-13T13:05:24.771919+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.586654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.586654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":218,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.781246+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:05:24.781246+0000","last_peered":"2024-09-13T13:05:24.781246+0000","last_clean":"2024-09-13T13:05:24.781246+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:05:24.781246+0000","last_undegraded":"2024-09-13T13:05:24.781246+0000","last_fullsized":"2024-09-13T13:05:24.781246+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.789127+0000","last_change":"2024-09-13T13:05:12.597696+0000","last_active":"2024-09-13T13:05:24.789127+0000","last_peered":"2024-09-13T13:05:24.789127+0000","last_clean":"2024-09-13T13:05:24.789127+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:05:24.789127+0000","last_undegraded":"2024-09-13T13:05:24.789127+0000","last_fullsized":"2024-09-13T13:05:24.789127+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:12.597637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.597637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:30.070820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.773289+0000","last_change":"2024-09-13T13:05:08.432874+0000","last_active":"2024-09-13T13:05:24.773289+0000","last_peered":"2024-09-13T13:05:24.773289+0000","last_clean":"2024-09-13T13:05:24.773289+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:05:24.773289+0000","last_undegraded":"2024-09-13T13:05:24.773289+0000","last_fullsized":"2024-09-13T13:05:24.773289+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:08.432816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:08.432816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:09.081986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":183,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.776856+0000","last_change":"2024-09-13T13:04:02.760695+0000","last_active":"2024-09-13T13:05:24.776856+0000","last_peered":"2024-09-13T13:05:24.776856+0000","last_clean":"2024-09-13T13:05:24.776856+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:05:24.776856+0000","last_undegraded":"2024-09-13T13:05:24.776856+0000","last_fullsized":"2024-09-13T13:05:24.776856+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":210,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.774007+0000","last_change":"2024-09-13T13:04:02.755722+0000","last_active":"2024-09-13T13:05:24.774007+0000","last_peered":"2024-09-13T13:05:24.774007+0000","last_clean":"2024-09-13T13:05:24.774007+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:05:24.774007+0000","last_undegraded":"2024-09-13T13:05:24.774007+0000","last_fullsized":"2024-09-13T13:05:24.774007+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.776973+0000","last_change":"2024-09-13T13:03:15.466909+0000","last_active":"2024-09-13T13:05:24.776973+0000","last_peered":"2024-09-13T13:05:24.776973+0000","last_clean":"2024-09-13T13:05:24.776973+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:05:24.776973+0000","last_undegraded":"2024-09-13T13:05:24.776973+0000","last_fullsized":"2024-09-13T13:05:24.776973+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:15.466859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:15.466859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":78,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.771730+0000","last_change":"2024-09-13T13:04:02.765046+0000","last_active":"2024-09-13T13:05:24.771730+0000","last_peered":"2024-09-13T13:05:24.771730+0000","last_clean":"2024-09-13T13:05:24.771730+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:05:24.771730+0000","last_undegraded":"2024-09-13T13:05:24.771730+0000","last_fullsized":"2024-09-13T13:05:24.771730+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":180,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.772018+0000","last_change":"2024-09-13T13:04:58.468060+0000","last_active":"2024-09-13T13:05:24.772018+0000","last_peered":"2024-09-13T13:05:24.772018+0000","last_clean":"2024-09-13T13:05:24.772018+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:05:24.772018+0000","last_undegraded":"2024-09-13T13:05:24.772018+0000","last_fullsized":"2024-09-13T13:05:24.772018+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:58.468011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:58.468011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.129704+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.783866+0000","last_change":"2024-09-13T13:02:43.234909+0000","last_active":"2024-09-13T13:05:24.783866+0000","last_peered":"2024-09-13T13:05:24.783866+0000","last_clean":"2024-09-13T13:05:24.783866+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:05:24.783866+0000","last_undegraded":"2024-09-13T13:05:24.783866+0000","last_fullsized":"2024-09-13T13:05:24.783866+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:43.234847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.234847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.774235+0000","last_change":"2024-09-13T13:02:46.246602+0000","last_active":"2024-09-13T13:05:24.774235+0000","last_peered":"2024-09-13T13:05:24.774235+0000","last_clean":"2024-09-13T13:05:24.774235+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:05:24.774235+0000","last_undegraded":"2024-09-13T13:05:24.774235+0000","last_fullsized":"2024-09-13T13:05:24.774235+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.246540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.246540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":188,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.770950+0000","last_change":"2024-09-13T13:04:02.756287+0000","last_active":"2024-09-13T13:05:24.770950+0000","last_peered":"2024-09-13T13:05:24.770950+0000","last_clean":"2024-09-13T13:05:24.770950+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:05:24.770950+0000","last_undegraded":"2024-09-13T13:05:24.770950+0000","last_fullsized":"2024-09-13T13:05:24.770950+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:02:39.557251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.557251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":185,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T13:05:24.781542+0000","last_change":"2024-09-13T13:02:30.512536+0000","last_active":"2024-09-13T13:05:24.781542+0000","last_peered":"2024-09-13T13:05:24.781542+0000","last_clean":"2024-09-13T13:05:24.781542+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:05:24.781542+0000","last_undegraded":"2024-09-13T13:05:24.781542+0000","last_fullsized":"2024-09-13T13:05:24.781542+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:30.512433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:30.512433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2366963736,"num_objects":36118,"num_object_clones":0,"num_object_copies":108354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36118,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206080,"num_write_kb":7750273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14010,"num_bytes_recovered":751697944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7134392320,"data_stored":7134380104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89047,"ondisk_log_size":89047,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":265,"num_read_kb":260,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201337,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":844932,"kb_used_data":733236,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92900220,"statfs":{"total":95995035648,"available":95129825280,"internally_reserved":0,"allocated":750833664,"data_stored":750429095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806140947,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":950968,"kb_used_data":801064,"kb_used_omap":4,"kb_used_meta":149883,"kb_avail":92794184,"statfs":{"total":95995035648,"available":95021244416,"internally_reserved":0,"allocated":820289536,"data_stored":819895650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":153480877},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691909,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":705080,"kb_used_data":593384,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93040072,"statfs":{"total":95995035648,"available":95273033728,"internally_reserved":0,"allocated":607625216,"data_stored":607231231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920843,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":843468,"kb_used_data":731708,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92901684,"statfs":{"total":95995035648,"available":95131324416,"internally_reserved":0,"allocated":749268992,"data_stored":748854663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117074,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1457360,"kb_used_data":1299840,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92287792,"statfs":{"total":95995035648,"available":94502699008,"internally_reserved":0,"allocated":1331036160,"data_stored":1330618565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161282503},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":998632,"kb_used_data":891096,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92746520,"statfs":{"total":95995035648,"available":94972436480,"internally_reserved":0,"allocated":912482304,"data_stored":912104043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1003236,"kb_used_data":873364,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92741916,"statfs":{"total":95995035648,"available":94967721984,"internally_reserved":0,"allocated":894324736,"data_stored":893886366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1312780,"kb_used_data":1151100,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92432372,"statfs":{"total":95995035648,"available":94650748928,"internally_reserved":0,"allocated":1178726400,"data_stored":1178313372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1166348288,"data_stored":1166344216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":886968320,"data_stored":886964248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":891944960,"data_stored":891944960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1315635200,"data_stored":1315635200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":732495872,"data_stored":732495872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":601817088,"data_stored":601817088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":803672064,"data_stored":803667992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":735510528,"data_stored":735510528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:05:37.039 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:37.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:37.106 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:37.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:37.209 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:37.234 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:37.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:37.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:37.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:36 smithi167 ceph-mon[33178]: 4.12 scrub starts 2024-09-13T13:05:37.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:36 smithi167 ceph-mon[33178]: 4.12 scrub ok 2024-09-13T13:05:37.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:36 smithi167 ceph-mon[33178]: pgmap v200: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 926 op/s; 67/108951 objects degraded (0.061%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T13:05:37.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:36 smithi167 ceph-mon[33178]: 7.5 scrub ok 2024-09-13T13:05:37.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:36 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/966657627' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:05:37.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:36 smithi167 ceph-mon[33178]: from='client.25087 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:05:37.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:36 smithi167 ceph-mon[33178]: from='client.25093 v1:172.21.15.52:0/4130683345' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:37.336 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:37.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:37.365 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:37.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:37.466 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:37.505 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:37.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:37.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:37.634 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:37.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:37.736 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:37.762 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:37.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:37.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:37.864 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:37.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:37.918 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:37.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:38.019 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:38.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:38.065 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:38.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:38.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:38.193 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:38.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:38.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[29471]: 3.0 scrub starts 2024-09-13T13:05:38.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[29471]: 3.0 scrub ok 2024-09-13T13:05:38.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:05:38.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[29471]: from='client.25087 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:05:38.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[29471]: osdmap e148: 8 total, 8 up, 8 in 2024-09-13T13:05:38.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[29471]: 6.15 scrub starts 2024-09-13T13:05:38.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[29471]: 6.15 scrub ok 2024-09-13T13:05:38.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 67/108951 objects degraded (0.061%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:38.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[36682]: 3.0 scrub starts 2024-09-13T13:05:38.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[36682]: 3.0 scrub ok 2024-09-13T13:05:38.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:05:38.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[36682]: from='client.25087 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:05:38.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[36682]: osdmap e148: 8 total, 8 up, 8 in 2024-09-13T13:05:38.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[36682]: 6.15 scrub starts 2024-09-13T13:05:38.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[36682]: 6.15 scrub ok 2024-09-13T13:05:38.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:37 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 67/108951 objects degraded (0.061%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:38.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:38.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:37 smithi167 ceph-mon[33178]: 3.0 scrub starts 2024-09-13T13:05:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:37 smithi167 ceph-mon[33178]: 3.0 scrub ok 2024-09-13T13:05:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:37 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:05:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:37 smithi167 ceph-mon[33178]: from='client.25087 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:05:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:37 smithi167 ceph-mon[33178]: osdmap e148: 8 total, 8 up, 8 in 2024-09-13T13:05:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:37 smithi167 ceph-mon[33178]: 6.15 scrub starts 2024-09-13T13:05:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:37 smithi167 ceph-mon[33178]: 6.15 scrub ok 2024-09-13T13:05:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:37 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 67/108951 objects degraded (0.061%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:38.322 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:38.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:38.424 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:38.463 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:38.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:38.565 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:38.595 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:38.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:38.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:38.724 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:38.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:38.826 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:38.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:38.865 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:38.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:38.966 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:38.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:38 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:05:38.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:38 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:05:38.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:38 smithi052 ceph-mon[29471]: 3.c scrub starts 2024-09-13T13:05:38.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:38 smithi052 ceph-mon[29471]: 3.c scrub ok 2024-09-13T13:05:38.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:38 smithi052 ceph-mon[29471]: pgmap v202: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 636 op/s; 67/107250 objects degraded (0.062%); 2.0 MiB/s, 32 objects/s recovering 2024-09-13T13:05:38.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:38 smithi052 ceph-mon[29471]: osdmap e149: 8 total, 8 up, 8 in 2024-09-13T13:05:38.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:38 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:05:38.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:38 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:05:38.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:38 smithi052 ceph-mon[36682]: 3.c scrub starts 2024-09-13T13:05:38.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:38 smithi052 ceph-mon[36682]: 3.c scrub ok 2024-09-13T13:05:38.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:38 smithi052 ceph-mon[36682]: pgmap v202: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 636 op/s; 67/107250 objects degraded (0.062%); 2.0 MiB/s, 32 objects/s recovering 2024-09-13T13:05:38.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:38 smithi052 ceph-mon[36682]: osdmap e149: 8 total, 8 up, 8 in 2024-09-13T13:05:38.991 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:38.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:39.094 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:39.099 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T13:05:39.137 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:39.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:39.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:39.236 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:39.241 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:39.269 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:39.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:38 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:05:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:38 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:05:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:38 smithi167 ceph-mon[33178]: 3.c scrub starts 2024-09-13T13:05:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:38 smithi167 ceph-mon[33178]: 3.c scrub ok 2024-09-13T13:05:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:38 smithi167 ceph-mon[33178]: pgmap v202: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 636 op/s; 67/107250 objects degraded (0.062%); 2.0 MiB/s, 32 objects/s recovering 2024-09-13T13:05:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:38 smithi167 ceph-mon[33178]: osdmap e149: 8 total, 8 up, 8 in 2024-09-13T13:05:39.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:39.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:39.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:39.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:39.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:39.544 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:39.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:39.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:39.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:39.698 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:39.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:39.799 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:39.855 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:39.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:39.956 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:39 smithi052 ceph-mon[36682]: 3.1c scrub starts 2024-09-13T13:05:39.957 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:39.957 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:39 smithi052 ceph-mon[29471]: 3.1c scrub starts 2024-09-13T13:05:39.957 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:39 smithi052 ceph-mon[29471]: 3.1c scrub ok 2024-09-13T13:05:39.957 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:39 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:05:39.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:39 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:05:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:39.995 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:39.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:40.027 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:05:40.096 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:40.129 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:40.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:40.231 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:40.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:39 smithi052 ceph-mon[36682]: 3.1c scrub ok 2024-09-13T13:05:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:39 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:05:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:39 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:05:40.259 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:40.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:40.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:39 smithi167 ceph-mon[33178]: 3.1c scrub starts 2024-09-13T13:05:40.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:39 smithi167 ceph-mon[33178]: 3.1c scrub ok 2024-09-13T13:05:40.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:39 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:05:40.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:39 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:05:40.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:40.381 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:05:40.390 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:40.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:40.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:40.492 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:40.582 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:40.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:40.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:40.671 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:05:40.672 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:05:40.684 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:40.707 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:40.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:40.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:40.710 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:40.717 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:05:40.718 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:05:40.727 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:40.759 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:05:40.759 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:05:40.810 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:40.811 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:05:40.811 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:05:40.840 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:05:40.840 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:05:40.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:40.847 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:40.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:40.873 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:05:40.873 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:05:40.910 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:05:40.910 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:05:40.949 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:40.953 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:40 smithi052 ceph-mon[36682]: 5.1c scrub starts 2024-09-13T13:05:40.953 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:40 smithi052 ceph-mon[36682]: 5.1c scrub ok 2024-09-13T13:05:40.953 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:40 smithi052 ceph-mon[36682]: pgmap v204: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 829 op/s; 714 KiB/s, 71 objects/s recovering 2024-09-13T13:05:40.953 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:40 smithi052 ceph-mon[36682]: osdmap e150: 8 total, 8 up, 8 in 2024-09-13T13:05:40.954 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:40 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1596546245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:40.954 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:05:40.954 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:05:40.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:40 smithi052 ceph-mon[29471]: 5.1c scrub starts 2024-09-13T13:05:40.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:40 smithi052 ceph-mon[29471]: 5.1c scrub ok 2024-09-13T13:05:40.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:40 smithi052 ceph-mon[29471]: pgmap v204: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 829 op/s; 714 KiB/s, 71 objects/s recovering 2024-09-13T13:05:40.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:40 smithi052 ceph-mon[29471]: osdmap e150: 8 total, 8 up, 8 in 2024-09-13T13:05:40.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:40 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1596546245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:40.973 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:40.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:40.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:41.074 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:41.117 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:41.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:41.218 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:41.261 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:41.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:41.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:40 smithi167 ceph-mon[33178]: 5.1c scrub starts 2024-09-13T13:05:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:40 smithi167 ceph-mon[33178]: 5.1c scrub ok 2024-09-13T13:05:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:40 smithi167 ceph-mon[33178]: pgmap v204: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 829 op/s; 714 KiB/s, 71 objects/s recovering 2024-09-13T13:05:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:40 smithi167 ceph-mon[33178]: osdmap e150: 8 total, 8 up, 8 in 2024-09-13T13:05:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:40 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1596546245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:41.362 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:41.391 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:41.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:41.493 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:41.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:41.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:41.632 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:41.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:41.674 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:41.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:41.776 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:41.818 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:41.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:41.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:41.920 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:41.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:41.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:42.027 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:05:42.027 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:05:42.028 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:05:42.028 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:05:42.028 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:05:42.054 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:42.089 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:42.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:42.191 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:42.215 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:42.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:42.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:42.339 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:42.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:42.440 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:42.462 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:42.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:42.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:42.564 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:42.586 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:42.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:42.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:42.709 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:42.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:42.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:42.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:42.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:42.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:42.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:42.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:42.951 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:05:42.960 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:05:42.963 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:42.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:42.968 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:43.065 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:43.117 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:43.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:43.218 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:42 smithi052 ceph-mon[29471]: pgmap v206: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 932 op/s; 714 KiB/s, 71 objects/s recovering 2024-09-13T13:05:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:42 smithi052 ceph-mon[29471]: 6.4 scrub starts 2024-09-13T13:05:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:42 smithi052 ceph-mon[29471]: 6.4 scrub ok 2024-09-13T13:05:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:42 smithi052 ceph-mon[29471]: osdmap e151: 8 total, 8 up, 8 in 2024-09-13T13:05:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:42 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:42 smithi052 ceph-mon[29471]: osdmap e152: 8 total, 8 up, 8 in 2024-09-13T13:05:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:42 smithi052 ceph-mon[36682]: pgmap v206: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 932 op/s; 714 KiB/s, 71 objects/s recovering 2024-09-13T13:05:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:42 smithi052 ceph-mon[36682]: 6.4 scrub starts 2024-09-13T13:05:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:42 smithi052 ceph-mon[36682]: 6.4 scrub ok 2024-09-13T13:05:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:42 smithi052 ceph-mon[36682]: osdmap e151: 8 total, 8 up, 8 in 2024-09-13T13:05:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:42 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:42 smithi052 ceph-mon[36682]: osdmap e152: 8 total, 8 up, 8 in 2024-09-13T13:05:43.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:43.260 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:43.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:43.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:42 smithi167 ceph-mon[33178]: pgmap v206: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 932 op/s; 714 KiB/s, 71 objects/s recovering 2024-09-13T13:05:43.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:42 smithi167 ceph-mon[33178]: 6.4 scrub starts 2024-09-13T13:05:43.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:42 smithi167 ceph-mon[33178]: 6.4 scrub ok 2024-09-13T13:05:43.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:42 smithi167 ceph-mon[33178]: osdmap e151: 8 total, 8 up, 8 in 2024-09-13T13:05:43.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:42 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T13:05:43.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:42 smithi167 ceph-mon[33178]: osdmap e152: 8 total, 8 up, 8 in 2024-09-13T13:05:43.360 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:43.401 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:43.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:43.503 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:43.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:43.543 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:43.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:43.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:43.670 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:43.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:43.764 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:05:43.771 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:43.859 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:43.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:43.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:43.961 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:44.026 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:44.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:44.033 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-13T13:05:42.775627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2001082931,"num_objects":30725,"num_object_clones":0,"num_object_copies":92175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30725,"num_whiteouts":0,"num_read":1117,"num_read_kb":974,"num_write":212221,"num_write_kb":7751946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14448,"num_bytes_recovered":756679334,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89799,"ondisk_log_size":89799,"up":444,"acting":443,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7023976,"kb_used_data":6019788,"kb_used_omap":15,"kb_used_meta":1003888,"kb_avail":742937240,"statfs":{"total":767960285184,"available":760767733760,"internally_reserved":0,"allocated":6164262912,"data_stored":6160906174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":1027982105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-329318400,"num_objects":-5025,"num_object_clones":0,"num_object_copies":-15075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5025,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5025,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":4390912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.105108"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":199,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.921482+0000","last_change":"2024-09-13T13:02:38.651382+0000","last_active":"2024-09-13T13:05:37.921482+0000","last_peered":"2024-09-13T13:05:37.921482+0000","last_clean":"2024-09-13T13:05:37.921482+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:05:37.921482+0000","last_undegraded":"2024-09-13T13:05:37.921482+0000","last_fullsized":"2024-09-13T13:05:37.921482+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:38.651314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:38.651314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":222,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.905996+0000","last_change":"2024-09-13T13:04:48.899982+0000","last_active":"2024-09-13T13:05:37.905996+0000","last_peered":"2024-09-13T13:05:37.905996+0000","last_clean":"2024-09-13T13:05:37.905996+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:05:37.905996+0000","last_undegraded":"2024-09-13T13:05:37.905996+0000","last_fullsized":"2024-09-13T13:05:37.905996+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:48.899915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.899915+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:00.227319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":199,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.920920+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:05:37.920920+0000","last_peered":"2024-09-13T13:05:37.920920+0000","last_clean":"2024-09-13T13:05:37.920920+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:05:37.920920+0000","last_undegraded":"2024-09-13T13:05:37.920920+0000","last_fullsized":"2024-09-13T13:05:37.920920+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:47.640365+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":187,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906041+0000","last_change":"2024-09-13T13:05:01.076346+0000","last_active":"2024-09-13T13:05:37.906041+0000","last_peered":"2024-09-13T13:05:37.906041+0000","last_clean":"2024-09-13T13:05:37.906041+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:05:37.906041+0000","last_undegraded":"2024-09-13T13:05:37.906041+0000","last_fullsized":"2024-09-13T13:05:37.906041+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.076275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.076275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:14.180362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":189,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.916123+0000","last_change":"2024-09-13T13:04:02.760609+0000","last_active":"2024-09-13T13:05:37.916123+0000","last_peered":"2024-09-13T13:05:37.916123+0000","last_clean":"2024-09-13T13:05:37.916123+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:05:37.916123+0000","last_undegraded":"2024-09-13T13:05:37.916123+0000","last_fullsized":"2024-09-13T13:05:37.916123+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":210,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.929072+0000","last_change":"2024-09-13T13:04:02.757267+0000","last_active":"2024-09-13T13:05:37.929072+0000","last_peered":"2024-09-13T13:05:37.929072+0000","last_clean":"2024-09-13T13:05:37.929072+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:05:37.929072+0000","last_undegraded":"2024-09-13T13:05:37.929072+0000","last_fullsized":"2024-09-13T13:05:37.929072+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.233259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.233259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":227,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.915449+0000","last_change":"2024-09-13T13:04:02.396436+0000","last_active":"2024-09-13T13:05:37.915449+0000","last_peered":"2024-09-13T13:05:37.915449+0000","last_clean":"2024-09-13T13:05:37.915449+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:05:37.915449+0000","last_undegraded":"2024-09-13T13:05:37.915449+0000","last_fullsized":"2024-09-13T13:05:37.915449+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:02.396342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.396342+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":202,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.905908+0000","last_change":"2024-09-13T13:04:02.765671+0000","last_active":"2024-09-13T13:05:37.905908+0000","last_peered":"2024-09-13T13:05:37.905908+0000","last_clean":"2024-09-13T13:05:37.905908+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:05:37.905908+0000","last_undegraded":"2024-09-13T13:05:37.905908+0000","last_fullsized":"2024-09-13T13:05:37.905908+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.021662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.021662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:18.072524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":213,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.877097+0000","last_change":"2024-09-13T13:05:00.690387+0000","last_active":"2024-09-13T13:05:36.877097+0000","last_peered":"2024-09-13T13:05:36.877097+0000","last_clean":"2024-09-13T13:05:36.877097+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:05:36.877097+0000","last_undegraded":"2024-09-13T13:05:36.877097+0000","last_fullsized":"2024-09-13T13:05:36.877097+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:05:00.690308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:00.690308+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:03.644084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":211,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906244+0000","last_change":"2024-09-13T13:04:56.066382+0000","last_active":"2024-09-13T13:05:37.906244+0000","last_peered":"2024-09-13T13:05:37.906244+0000","last_clean":"2024-09-13T13:05:37.906244+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:05:37.906244+0000","last_undegraded":"2024-09-13T13:05:37.906244+0000","last_fullsized":"2024-09-13T13:05:37.906244+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:56.066326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.066326+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.781998+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":196,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.876967+0000","last_change":"2024-09-13T13:05:26.621813+0000","last_active":"2024-09-13T13:05:36.876967+0000","last_peered":"2024-09-13T13:05:36.876967+0000","last_clean":"2024-09-13T13:05:36.876967+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:05:36.876967+0000","last_undegraded":"2024-09-13T13:05:36.876967+0000","last_fullsized":"2024-09-13T13:05:36.876967+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:26.621732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.621732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:48.964955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":186,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.919561+0000","last_change":"2024-09-13T13:02:39.607712+0000","last_active":"2024-09-13T13:05:37.919561+0000","last_peered":"2024-09-13T13:05:37.919561+0000","last_clean":"2024-09-13T13:05:37.919561+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:05:37.919561+0000","last_undegraded":"2024-09-13T13:05:37.919561+0000","last_fullsized":"2024-09-13T13:05:37.919561+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.607646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.607646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":204,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.876381+0000","last_change":"2024-09-13T13:03:40.665853+0000","last_active":"2024-09-13T13:05:36.876381+0000","last_peered":"2024-09-13T13:05:36.876381+0000","last_clean":"2024-09-13T13:05:36.876381+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:05:36.876381+0000","last_undegraded":"2024-09-13T13:05:36.876381+0000","last_fullsized":"2024-09-13T13:05:36.876381+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:40.665805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:40.665805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":213,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906431+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:05:37.906431+0000","last_peered":"2024-09-13T13:05:37.906431+0000","last_clean":"2024-09-13T13:05:37.906431+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:05:37.906431+0000","last_undegraded":"2024-09-13T13:05:37.906431+0000","last_fullsized":"2024-09-13T13:05:37.906431+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:15.096451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":190,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.923920+0000","last_change":"2024-09-13T13:05:10.383826+0000","last_active":"2024-09-13T13:05:37.923920+0000","last_peered":"2024-09-13T13:05:37.923920+0000","last_clean":"2024-09-13T13:05:37.923920+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:05:37.923920+0000","last_undegraded":"2024-09-13T13:05:37.923920+0000","last_fullsized":"2024-09-13T13:05:37.923920+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.383756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.383756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:33.805546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":176,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.875648+0000","last_change":"2024-09-13T13:01:08.213608+0000","last_active":"2024-09-13T13:05:36.875648+0000","last_peered":"2024-09-13T13:05:36.875648+0000","last_clean":"2024-09-13T13:05:36.875648+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:05:36.875648+0000","last_undegraded":"2024-09-13T13:05:36.875648+0000","last_fullsized":"2024-09-13T13:05:36.875648+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.558201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":200,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906465+0000","last_change":"2024-09-13T13:04:54.016136+0000","last_active":"2024-09-13T13:05:37.906465+0000","last_peered":"2024-09-13T13:05:37.906465+0000","last_clean":"2024-09-13T13:05:37.906465+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:05:37.906465+0000","last_undegraded":"2024-09-13T13:05:37.906465+0000","last_fullsized":"2024-09-13T13:05:37.906465+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:04:54.016064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.016064+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.987760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":198,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.920478+0000","last_change":"2024-09-13T13:02:32.590355+0000","last_active":"2024-09-13T13:05:37.920478+0000","last_peered":"2024-09-13T13:05:37.920478+0000","last_clean":"2024-09-13T13:05:37.920478+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:05:37.920478+0000","last_undegraded":"2024-09-13T13:05:37.920478+0000","last_fullsized":"2024-09-13T13:05:37.920478+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:32.590282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.590282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":236,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.918124+0000","last_change":"2024-09-13T13:05:33.323411+0000","last_active":"2024-09-13T13:05:37.918124+0000","last_peered":"2024-09-13T13:05:37.918124+0000","last_clean":"2024-09-13T13:05:37.918124+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:05:37.918124+0000","last_undegraded":"2024-09-13T13:05:37.918124+0000","last_fullsized":"2024-09-13T13:05:37.918124+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:05:33.323316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.323316+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:36.730491+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":191,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.923969+0000","last_change":"2024-09-13T13:02:45.202865+0000","last_active":"2024-09-13T13:05:37.923969+0000","last_peered":"2024-09-13T13:05:37.923969+0000","last_clean":"2024-09-13T13:05:37.923969+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:05:37.923969+0000","last_undegraded":"2024-09-13T13:05:37.923969+0000","last_fullsized":"2024-09-13T13:05:37.923969+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.202802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.202802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":198,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.919806+0000","last_change":"2024-09-13T13:04:03.384955+0000","last_active":"2024-09-13T13:05:37.919806+0000","last_peered":"2024-09-13T13:05:37.919806+0000","last_clean":"2024-09-13T13:05:37.919806+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:05:37.919806+0000","last_undegraded":"2024-09-13T13:05:37.919806+0000","last_fullsized":"2024-09-13T13:05:37.919806+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.384901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.384901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":199,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906410+0000","last_change":"2024-09-13T13:04:57.103213+0000","last_active":"2024-09-13T13:05:37.906410+0000","last_peered":"2024-09-13T13:05:37.906410+0000","last_clean":"2024-09-13T13:05:37.906410+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:05:37.906410+0000","last_undegraded":"2024-09-13T13:05:37.906410+0000","last_fullsized":"2024-09-13T13:05:37.906410+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.103167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.103167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.050420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":209,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.913830+0000","last_change":"2024-09-13T13:01:04.195126+0000","last_active":"2024-09-13T13:05:37.913830+0000","last_peered":"2024-09-13T13:05:37.913830+0000","last_clean":"2024-09-13T13:05:37.913830+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:05:37.913830+0000","last_undegraded":"2024-09-13T13:05:37.913830+0000","last_fullsized":"2024-09-13T13:05:37.913830+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":191,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906273+0000","last_change":"2024-09-13T13:03:11.014220+0000","last_active":"2024-09-13T13:05:37.906273+0000","last_peered":"2024-09-13T13:05:37.906273+0000","last_clean":"2024-09-13T13:05:37.906273+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:05:37.906273+0000","last_undegraded":"2024-09-13T13:05:37.906273+0000","last_fullsized":"2024-09-13T13:05:37.906273+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:11.014157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:11.014157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":187,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.925663+0000","last_change":"2024-09-13T13:05:00.457611+0000","last_active":"2024-09-13T13:05:37.925663+0000","last_peered":"2024-09-13T13:05:37.925663+0000","last_clean":"2024-09-13T13:05:37.925663+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:05:37.925663+0000","last_undegraded":"2024-09-13T13:05:37.925663+0000","last_fullsized":"2024-09-13T13:05:37.925663+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:05:00.457562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:00.457562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:25.611525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":200,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.922058+0000","last_change":"2024-09-13T13:03:12.276224+0000","last_active":"2024-09-13T13:05:37.922058+0000","last_peered":"2024-09-13T13:05:37.922058+0000","last_clean":"2024-09-13T13:05:37.922058+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:05:37.922058+0000","last_undegraded":"2024-09-13T13:05:37.922058+0000","last_fullsized":"2024-09-13T13:05:37.922058+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:12.276136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.276136+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":210,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.922376+0000","last_change":"2024-09-13T13:02:34.593511+0000","last_active":"2024-09-13T13:05:37.922376+0000","last_peered":"2024-09-13T13:05:37.922376+0000","last_clean":"2024-09-13T13:05:37.922376+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:05:37.922376+0000","last_undegraded":"2024-09-13T13:05:37.922376+0000","last_fullsized":"2024-09-13T13:05:37.922376+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:34.593431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:34.593431+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":206,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T13:05:39.922580+0000","last_change":"2024-09-13T13:04:02.765069+0000","last_active":"2024-09-13T13:05:39.922580+0000","last_peered":"2024-09-13T13:05:39.922580+0000","last_clean":"2024-09-13T13:05:39.922580+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:05:39.922580+0000","last_undegraded":"2024-09-13T13:05:39.922580+0000","last_fullsized":"2024-09-13T13:05:39.922580+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:46.604669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.604669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"150'13293","reported_seq":19210,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T13:05:40.838493+0000","last_change":"2024-09-13T13:04:49.780269+0000","last_active":"2024-09-13T13:05:40.838493+0000","last_peered":"2024-09-13T13:05:40.838493+0000","last_clean":"2024-09-13T13:05:40.838493+0000","last_became_active":"2024-09-13T13:04:49.765876+0000","last_became_peered":"2024-09-13T13:04:49.765876+0000","last_unstale":"2024-09-13T13:05:40.838493+0000","last_undegraded":"2024-09-13T13:05:40.838493+0000","last_fullsized":"2024-09-13T13:05:40.838493+0000","mapping_epoch":136,"log_start":"114'6948","ondisk_log_start":"114'6948","created":84,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":6345,"log_dups_size":0,"ondisk_log_size":6345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113049600,"num_objects":1725,"num_object_clones":0,"num_object_copies":5175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13293,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2909,"num_bytes_recovered":147193856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":229,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.919858+0000","last_change":"2024-09-13T13:05:25.691233+0000","last_active":"2024-09-13T13:05:37.919858+0000","last_peered":"2024-09-13T13:05:37.919858+0000","last_clean":"2024-09-13T13:05:37.919858+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:05:37.919858+0000","last_undegraded":"2024-09-13T13:05:37.919858+0000","last_fullsized":"2024-09-13T13:05:37.919858+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:05:25.689487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:25.689487+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:53.347573+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":207,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.916351+0000","last_change":"2024-09-13T13:03:42.443399+0000","last_active":"2024-09-13T13:05:37.916351+0000","last_peered":"2024-09-13T13:05:37.916351+0000","last_clean":"2024-09-13T13:05:37.916351+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:05:37.916351+0000","last_undegraded":"2024-09-13T13:05:37.916351+0000","last_fullsized":"2024-09-13T13:05:37.916351+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:42.443339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:42.443339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":256,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906751+0000","last_change":"2024-09-13T13:02:32.013587+0000","last_active":"2024-09-13T13:05:37.906751+0000","last_peered":"2024-09-13T13:05:37.906751+0000","last_clean":"2024-09-13T13:05:37.906751+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:05:37.906751+0000","last_undegraded":"2024-09-13T13:05:37.906751+0000","last_fullsized":"2024-09-13T13:05:37.906751+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:32.013505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.013505+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:14.086911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":185,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.876714+0000","last_change":"2024-09-13T13:04:48.684906+0000","last_active":"2024-09-13T13:05:36.876714+0000","last_peered":"2024-09-13T13:05:36.876714+0000","last_clean":"2024-09-13T13:05:36.876714+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:05:36.876714+0000","last_undegraded":"2024-09-13T13:05:36.876714+0000","last_fullsized":"2024-09-13T13:05:36.876714+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:48.684864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.684864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.491555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"149'13135","reported_seq":13354,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:38.211512+0000","last_change":"2024-09-13T13:03:41.936543+0000","last_active":"2024-09-13T13:05:38.211512+0000","last_peered":"2024-09-13T13:05:38.211512+0000","last_clean":"2024-09-13T13:05:38.211512+0000","last_became_active":"2024-09-13T13:02:12.369723+0000","last_became_peered":"2024-09-13T13:02:12.369723+0000","last_unstale":"2024-09-13T13:05:38.211512+0000","last_undegraded":"2024-09-13T13:05:38.211512+0000","last_fullsized":"2024-09-13T13:05:38.211512+0000","mapping_epoch":84,"log_start":"122'8500","ondisk_log_start":"122'8500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6909","last_scrub_stamp":"2024-09-13T13:03:41.936218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:03:41.936218+0000","objects_scrubbed":6836,"log_size":4635,"log_dups_size":0,"ondisk_log_size":4635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130613248,"num_objects":1993,"num_object_clones":0,"num_object_copies":5979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13135,"num_write_kb":484096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":227,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.916122+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:05:37.916122+0000","last_peered":"2024-09-13T13:05:37.916122+0000","last_clean":"2024-09-13T13:05:37.916122+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:05:37.916122+0000","last_undegraded":"2024-09-13T13:05:37.916122+0000","last_fullsized":"2024-09-13T13:05:37.916122+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":205,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.920148+0000","last_change":"2024-09-13T13:03:18.516327+0000","last_active":"2024-09-13T13:05:37.920148+0000","last_peered":"2024-09-13T13:05:37.920148+0000","last_clean":"2024-09-13T13:05:37.920148+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:05:37.920148+0000","last_undegraded":"2024-09-13T13:05:37.920148+0000","last_fullsized":"2024-09-13T13:05:37.920148+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:18.516251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.516251+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":200,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906030+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:05:37.906030+0000","last_peered":"2024-09-13T13:05:37.906030+0000","last_clean":"2024-09-13T13:05:37.906030+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:05:37.906030+0000","last_undegraded":"2024-09-13T13:05:37.906030+0000","last_fullsized":"2024-09-13T13:05:37.906030+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:40.320454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":188,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906138+0000","last_change":"2024-09-13T13:04:53.006515+0000","last_active":"2024-09-13T13:05:37.906138+0000","last_peered":"2024-09-13T13:05:37.906138+0000","last_clean":"2024-09-13T13:05:37.906138+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:05:37.906138+0000","last_undegraded":"2024-09-13T13:05:37.906138+0000","last_fullsized":"2024-09-13T13:05:37.906138+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:53.006454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.006454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.531241+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"149'13141","reported_seq":13289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:38.207356+0000","last_change":"2024-09-13T13:03:45.761068+0000","last_active":"2024-09-13T13:05:38.207356+0000","last_peered":"2024-09-13T13:05:38.207356+0000","last_clean":"2024-09-13T13:05:38.207356+0000","last_became_active":"2024-09-13T13:02:12.373492+0000","last_became_peered":"2024-09-13T13:02:12.373492+0000","last_unstale":"2024-09-13T13:05:38.207356+0000","last_undegraded":"2024-09-13T13:05:38.207356+0000","last_fullsized":"2024-09-13T13:05:38.207356+0000","mapping_epoch":84,"log_start":"122'8500","ondisk_log_start":"122'8500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":299,"log_size":4641,"log_dups_size":0,"ondisk_log_size":4641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125501440,"num_objects":1915,"num_object_clones":0,"num_object_copies":5745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13141,"num_write_kb":481792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":236,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906036+0000","last_change":"2024-09-13T13:05:33.178442+0000","last_active":"2024-09-13T13:05:37.906036+0000","last_peered":"2024-09-13T13:05:37.906036+0000","last_clean":"2024-09-13T13:05:37.906036+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:05:37.906036+0000","last_undegraded":"2024-09-13T13:05:37.906036+0000","last_fullsized":"2024-09-13T13:05:37.906036+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:05:33.178370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.178370+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:57.356161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":200,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.922813+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:05:37.922813+0000","last_peered":"2024-09-13T13:05:37.922813+0000","last_clean":"2024-09-13T13:05:37.922813+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:05:37.922813+0000","last_undegraded":"2024-09-13T13:05:37.922813+0000","last_fullsized":"2024-09-13T13:05:37.922813+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":247,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.913605+0000","last_change":"2024-09-13T13:01:06.211211+0000","last_active":"2024-09-13T13:05:37.913605+0000","last_peered":"2024-09-13T13:05:37.913605+0000","last_clean":"2024-09-13T13:05:37.913605+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:05:37.913605+0000","last_undegraded":"2024-09-13T13:05:37.913605+0000","last_fullsized":"2024-09-13T13:05:37.913605+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":198,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.923888+0000","last_change":"2024-09-13T13:04:57.439116+0000","last_active":"2024-09-13T13:05:37.923888+0000","last_peered":"2024-09-13T13:05:37.923888+0000","last_clean":"2024-09-13T13:05:37.923888+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:05:37.923888+0000","last_undegraded":"2024-09-13T13:05:37.923888+0000","last_fullsized":"2024-09-13T13:05:37.923888+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.439053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.439053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:04.992285+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"149'13172","reported_seq":19160,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.905969+0000","last_change":"2024-09-13T13:05:05.014474+0000","last_active":"2024-09-13T13:05:37.905969+0000","last_peered":"2024-09-13T13:05:37.905969+0000","last_clean":"2024-09-13T13:05:37.905969+0000","last_became_active":"2024-09-13T13:04:25.922485+0000","last_became_peered":"2024-09-13T13:04:25.922485+0000","last_unstale":"2024-09-13T13:05:37.905969+0000","last_undegraded":"2024-09-13T13:05:37.905969+0000","last_fullsized":"2024-09-13T13:05:37.905969+0000","mapping_epoch":128,"log_start":"122'8400","ondisk_log_start":"122'8400","created":84,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'11284","last_scrub_stamp":"2024-09-13T13:05:05.014171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:05.014171+0000","objects_scrubbed":3783,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:18.664629+0000","scrub_duration":934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122421248,"num_objects":1868,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13172,"num_write_kb":481280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2891,"num_bytes_recovered":148570112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":203,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.923829+0000","last_change":"2024-09-13T13:01:04.199011+0000","last_active":"2024-09-13T13:05:37.923829+0000","last_peered":"2024-09-13T13:05:37.923829+0000","last_clean":"2024-09-13T13:05:37.923829+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:05:37.923829+0000","last_undegraded":"2024-09-13T13:05:37.923829+0000","last_fullsized":"2024-09-13T13:05:37.923829+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.733108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":197,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.877199+0000","last_change":"2024-09-13T13:05:04.743102+0000","last_active":"2024-09-13T13:05:36.877199+0000","last_peered":"2024-09-13T13:05:36.877199+0000","last_clean":"2024-09-13T13:05:36.877199+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:05:36.877199+0000","last_undegraded":"2024-09-13T13:05:36.877199+0000","last_fullsized":"2024-09-13T13:05:36.877199+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:04.743025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:04.743025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:32.082984+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":261,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.920062+0000","last_change":"2024-09-13T13:04:02.764350+0000","last_active":"2024-09-13T13:05:37.920062+0000","last_peered":"2024-09-13T13:05:37.920062+0000","last_clean":"2024-09-13T13:05:37.920062+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:05:37.920062+0000","last_undegraded":"2024-09-13T13:05:37.920062+0000","last_fullsized":"2024-09-13T13:05:37.920062+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":187,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.919769+0000","last_change":"2024-09-13T13:02:43.558487+0000","last_active":"2024-09-13T13:05:37.919769+0000","last_peered":"2024-09-13T13:05:37.919769+0000","last_clean":"2024-09-13T13:05:37.919769+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:05:37.919769+0000","last_undegraded":"2024-09-13T13:05:37.919769+0000","last_fullsized":"2024-09-13T13:05:37.919769+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:43.558418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.558418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"148'12978","reported_seq":13118,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.850498+0000","last_change":"2024-09-13T13:02:12.379536+0000","last_active":"2024-09-13T13:05:37.850498+0000","last_peered":"2024-09-13T13:05:37.850498+0000","last_clean":"2024-09-13T13:05:37.850498+0000","last_became_active":"2024-09-13T13:02:12.379226+0000","last_became_peered":"2024-09-13T13:02:12.379226+0000","last_unstale":"2024-09-13T13:05:37.850498+0000","last_undegraded":"2024-09-13T13:05:37.850498+0000","last_fullsized":"2024-09-13T13:05:37.850498+0000","mapping_epoch":84,"log_start":"115'7200","ondisk_log_start":"115'7200","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5778,"log_dups_size":0,"ondisk_log_size":5778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.567939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128581632,"num_objects":1962,"num_object_clones":0,"num_object_copies":5886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12978,"num_write_kb":478080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":231,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.920267+0000","last_change":"2024-09-13T13:02:29.532879+0000","last_active":"2024-09-13T13:05:37.920267+0000","last_peered":"2024-09-13T13:05:37.920267+0000","last_clean":"2024-09-13T13:05:37.920267+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:05:37.920267+0000","last_undegraded":"2024-09-13T13:05:37.920267+0000","last_fullsized":"2024-09-13T13:05:37.920267+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:29.532779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.532779+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":192,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.924141+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:05:37.924141+0000","last_peered":"2024-09-13T13:05:37.924141+0000","last_clean":"2024-09-13T13:05:37.924141+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:05:37.924141+0000","last_undegraded":"2024-09-13T13:05:37.924141+0000","last_fullsized":"2024-09-13T13:05:37.924141+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:20.738953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":154,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T13:05:39.922138+0000","last_change":"2024-09-13T13:04:02.765838+0000","last_active":"2024-09-13T13:05:39.922138+0000","last_peered":"2024-09-13T13:05:39.922138+0000","last_clean":"2024-09-13T13:05:39.922138+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:05:39.922138+0000","last_undegraded":"2024-09-13T13:05:39.922138+0000","last_fullsized":"2024-09-13T13:05:39.922138+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.563708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.563708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":207,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.920824+0000","last_change":"2024-09-13T13:05:30.829125+0000","last_active":"2024-09-13T13:05:37.920824+0000","last_peered":"2024-09-13T13:05:37.920824+0000","last_clean":"2024-09-13T13:05:37.920824+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:05:37.920824+0000","last_undegraded":"2024-09-13T13:05:37.920824+0000","last_fullsized":"2024-09-13T13:05:37.920824+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:30.828810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:30.828810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:43.724154+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"149'13297","reported_seq":18709,"reported_epoch":149,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T13:05:38.474050+0000","last_change":"2024-09-13T13:05:34.392029+0000","last_active":"2024-09-13T13:05:38.474050+0000","last_peered":"2024-09-13T13:05:38.474050+0000","last_clean":"2024-09-13T13:03:14.460355+0000","last_became_active":"2024-09-13T13:04:03.878596+0000","last_became_peered":"2024-09-13T13:04:03.878596+0000","last_unstale":"2024-09-13T13:05:38.474050+0000","last_undegraded":"2024-09-13T13:05:38.474050+0000","last_fullsized":"2024-09-13T13:04:03.744969+0000","mapping_epoch":119,"log_start":"115'7700","ondisk_log_start":"115'7700","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122880000,"num_objects":1875,"num_object_clones":0,"num_object_copies":5625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13297,"num_write_kb":485504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2632,"num_bytes_recovered":148635648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1875}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":300,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.923994+0000","last_change":"2024-09-13T13:02:39.183839+0000","last_active":"2024-09-13T13:05:37.923994+0000","last_peered":"2024-09-13T13:05:37.923994+0000","last_clean":"2024-09-13T13:05:37.923994+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:05:37.923994+0000","last_undegraded":"2024-09-13T13:05:37.923994+0000","last_fullsized":"2024-09-13T13:05:37.923994+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:02:39.183740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.183740+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":211,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906689+0000","last_change":"2024-09-13T13:04:03.850983+0000","last_active":"2024-09-13T13:05:37.906689+0000","last_peered":"2024-09-13T13:05:37.906689+0000","last_clean":"2024-09-13T13:05:37.906689+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:05:37.906689+0000","last_undegraded":"2024-09-13T13:05:37.906689+0000","last_fullsized":"2024-09-13T13:05:37.906689+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.850898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.850898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":216,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.876160+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:05:36.876160+0000","last_peered":"2024-09-13T13:05:36.876160+0000","last_clean":"2024-09-13T13:05:36.876160+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:05:36.876160+0000","last_undegraded":"2024-09-13T13:05:36.876160+0000","last_fullsized":"2024-09-13T13:05:36.876160+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:26.941315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":191,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.923964+0000","last_change":"2024-09-13T13:02:35.566666+0000","last_active":"2024-09-13T13:05:37.923964+0000","last_peered":"2024-09-13T13:05:37.923964+0000","last_clean":"2024-09-13T13:05:37.923964+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:05:37.923964+0000","last_undegraded":"2024-09-13T13:05:37.923964+0000","last_fullsized":"2024-09-13T13:05:37.923964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:35.566602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:35.566602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":187,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.928753+0000","last_change":"2024-09-13T13:04:01.406226+0000","last_active":"2024-09-13T13:05:37.928753+0000","last_peered":"2024-09-13T13:05:37.928753+0000","last_clean":"2024-09-13T13:05:37.928753+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:05:37.928753+0000","last_undegraded":"2024-09-13T13:05:37.928753+0000","last_fullsized":"2024-09-13T13:05:37.928753+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:01.406167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:01.406167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"149'13217","reported_seq":13359,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:38.473816+0000","last_change":"2024-09-13T13:02:12.380398+0000","last_active":"2024-09-13T13:05:38.473816+0000","last_peered":"2024-09-13T13:05:38.473816+0000","last_clean":"2024-09-13T13:05:38.473816+0000","last_became_active":"2024-09-13T13:02:12.379331+0000","last_became_peered":"2024-09-13T13:02:12.379331+0000","last_unstale":"2024-09-13T13:05:38.473816+0000","last_undegraded":"2024-09-13T13:05:38.473816+0000","last_fullsized":"2024-09-13T13:05:38.473816+0000","mapping_epoch":84,"log_start":"115'7600","ondisk_log_start":"115'7600","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122486784,"num_objects":1869,"num_object_clones":0,"num_object_copies":5607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13217,"num_write_kb":482752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":223,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.919913+0000","last_change":"2024-09-13T13:03:14.271569+0000","last_active":"2024-09-13T13:05:37.919913+0000","last_peered":"2024-09-13T13:05:37.919913+0000","last_clean":"2024-09-13T13:05:37.919913+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:05:37.919913+0000","last_undegraded":"2024-09-13T13:05:37.919913+0000","last_fullsized":"2024-09-13T13:05:37.919913+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:03:14.271480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.271480+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":219,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.919974+0000","last_change":"2024-09-13T13:05:36.692159+0000","last_active":"2024-09-13T13:05:37.919974+0000","last_peered":"2024-09-13T13:05:37.919974+0000","last_clean":"2024-09-13T13:05:37.919974+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:05:37.919974+0000","last_undegraded":"2024-09-13T13:05:37.919974+0000","last_fullsized":"2024-09-13T13:05:37.919974+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:36.692092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:36.692092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:00.057550+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":215,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.917566+0000","last_change":"2024-09-13T13:05:34.321160+0000","last_active":"2024-09-13T13:05:37.917566+0000","last_peered":"2024-09-13T13:05:37.917566+0000","last_clean":"2024-09-13T13:05:37.917566+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:05:37.917566+0000","last_undegraded":"2024-09-13T13:05:37.917566+0000","last_fullsized":"2024-09-13T13:05:37.917566+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:05:34.321081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.321081+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:37.239957+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":192,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.918845+0000","last_change":"2024-09-13T13:03:14.282866+0000","last_active":"2024-09-13T13:05:37.918845+0000","last_peered":"2024-09-13T13:05:37.918845+0000","last_clean":"2024-09-13T13:05:37.918845+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:05:37.918845+0000","last_undegraded":"2024-09-13T13:05:37.918845+0000","last_fullsized":"2024-09-13T13:05:37.918845+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.349860+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":198,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.924239+0000","last_change":"2024-09-13T13:04:02.751582+0000","last_active":"2024-09-13T13:05:37.924239+0000","last_peered":"2024-09-13T13:05:37.924239+0000","last_clean":"2024-09-13T13:05:37.924239+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:05:37.924239+0000","last_undegraded":"2024-09-13T13:05:37.924239+0000","last_fullsized":"2024-09-13T13:05:37.924239+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.994729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.994729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.398308+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"148'13421","reported_seq":13667,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.924106+0000","last_change":"2024-09-13T13:05:35.785111+0000","last_active":"2024-09-13T13:05:37.924106+0000","last_peered":"2024-09-13T13:05:37.924106+0000","last_clean":"2024-09-13T13:05:37.924106+0000","last_became_active":"2024-09-13T13:02:12.379779+0000","last_became_peered":"2024-09-13T13:02:12.379779+0000","last_unstale":"2024-09-13T13:05:37.924106+0000","last_undegraded":"2024-09-13T13:05:37.924106+0000","last_fullsized":"2024-09-13T13:05:37.924106+0000","mapping_epoch":84,"log_start":"116'8200","ondisk_log_start":"116'8200","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'13287","last_scrub_stamp":"2024-09-13T13:05:35.785022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.785022+0000","objects_scrubbed":2122,"log_size":5221,"log_dups_size":0,"ondisk_log_size":5221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:52.011396+0000","scrub_duration":635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128778240,"num_objects":1965,"num_object_clones":0,"num_object_copies":5895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13421,"num_write_kb":492352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":213,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.922471+0000","last_change":"2024-09-13T13:03:18.514436+0000","last_active":"2024-09-13T13:05:37.922471+0000","last_peered":"2024-09-13T13:05:37.922471+0000","last_clean":"2024-09-13T13:05:37.922471+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:05:37.922471+0000","last_undegraded":"2024-09-13T13:05:37.922471+0000","last_fullsized":"2024-09-13T13:05:37.922471+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:03:18.514360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.514360+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":199,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.920333+0000","last_change":"2024-09-13T13:02:37.616313+0000","last_active":"2024-09-13T13:05:37.920333+0000","last_peered":"2024-09-13T13:05:37.920333+0000","last_clean":"2024-09-13T13:05:37.920333+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:05:37.920333+0000","last_undegraded":"2024-09-13T13:05:37.920333+0000","last_fullsized":"2024-09-13T13:05:37.920333+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:37.616224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.616224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":478,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.923767+0000","last_change":"2024-09-13T13:04:56.430717+0000","last_active":"2024-09-13T13:05:37.923767+0000","last_peered":"2024-09-13T13:05:37.923767+0000","last_clean":"2024-09-13T13:05:37.923767+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:05:37.923767+0000","last_undegraded":"2024-09-13T13:05:37.923767+0000","last_fullsized":"2024-09-13T13:05:37.923767+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:04:56.430670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.430670+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:59.639039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":270,"num_read_kb":265,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":189,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.877581+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:05:36.877581+0000","last_peered":"2024-09-13T13:05:36.877581+0000","last_clean":"2024-09-13T13:05:36.877581+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:05:36.877581+0000","last_undegraded":"2024-09-13T13:05:36.877581+0000","last_fullsized":"2024-09-13T13:05:36.877581+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:19.946825+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":188,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.920406+0000","last_change":"2024-09-13T13:02:36.222607+0000","last_active":"2024-09-13T13:05:37.920406+0000","last_peered":"2024-09-13T13:05:37.920406+0000","last_clean":"2024-09-13T13:05:37.920406+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:05:37.920406+0000","last_undegraded":"2024-09-13T13:05:37.920406+0000","last_fullsized":"2024-09-13T13:05:37.920406+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.222329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"149'13204","reported_seq":13345,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:38.442440+0000","last_change":"2024-09-13T13:02:12.379556+0000","last_active":"2024-09-13T13:05:38.442440+0000","last_peered":"2024-09-13T13:05:38.442440+0000","last_clean":"2024-09-13T13:05:38.442440+0000","last_became_active":"2024-09-13T13:02:12.379255+0000","last_became_peered":"2024-09-13T13:02:12.379255+0000","last_unstale":"2024-09-13T13:05:38.442440+0000","last_undegraded":"2024-09-13T13:05:38.442440+0000","last_fullsized":"2024-09-13T13:05:38.442440+0000","mapping_epoch":84,"log_start":"115'7600","ondisk_log_start":"115'7600","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5604,"log_dups_size":0,"ondisk_log_size":5604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125960192,"num_objects":1922,"num_object_clones":0,"num_object_copies":5766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13204,"num_write_kb":484032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":142,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T13:05:39.922377+0000","last_change":"2024-09-13T13:04:02.786042+0000","last_active":"2024-09-13T13:05:39.922377+0000","last_peered":"2024-09-13T13:05:39.922377+0000","last_clean":"2024-09-13T13:05:39.922377+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:05:39.922377+0000","last_undegraded":"2024-09-13T13:05:39.922377+0000","last_fullsized":"2024-09-13T13:05:39.922377+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":210,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.915259+0000","last_change":"2024-09-13T13:04:02.765370+0000","last_active":"2024-09-13T13:05:37.915259+0000","last_peered":"2024-09-13T13:05:37.915259+0000","last_clean":"2024-09-13T13:05:37.915259+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:05:37.915259+0000","last_undegraded":"2024-09-13T13:05:37.915259+0000","last_fullsized":"2024-09-13T13:05:37.915259+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:04.228002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:04.228002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":255,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.923865+0000","last_change":"2024-09-13T13:05:28.373960+0000","last_active":"2024-09-13T13:05:37.923865+0000","last_peered":"2024-09-13T13:05:37.923865+0000","last_clean":"2024-09-13T13:05:37.923865+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:05:37.923865+0000","last_undegraded":"2024-09-13T13:05:37.923865+0000","last_fullsized":"2024-09-13T13:05:37.923865+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:05:28.373902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:05:28.373902+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:33.407719+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":191,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.925754+0000","last_change":"2024-09-13T13:04:51.395273+0000","last_active":"2024-09-13T13:05:37.925754+0000","last_peered":"2024-09-13T13:05:37.925754+0000","last_clean":"2024-09-13T13:05:37.925754+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:05:37.925754+0000","last_undegraded":"2024-09-13T13:05:37.925754+0000","last_fullsized":"2024-09-13T13:05:37.925754+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:51.395204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:51.395204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:56.927380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.877500+0000","last_change":"2024-09-13T13:05:31.564860+0000","last_active":"2024-09-13T13:05:36.877500+0000","last_peered":"2024-09-13T13:05:36.877500+0000","last_clean":"2024-09-13T13:05:36.877500+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:05:36.877500+0000","last_undegraded":"2024-09-13T13:05:36.877500+0000","last_fullsized":"2024-09-13T13:05:36.877500+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:31.564783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:31.564783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:41.885010+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"150'13497","reported_seq":19675,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T13:05:40.835035+0000","last_change":"2024-09-13T13:05:15.349567+0000","last_active":"2024-09-13T13:05:40.835035+0000","last_peered":"2024-09-13T13:05:40.835035+0000","last_clean":"2024-09-13T13:05:40.835035+0000","last_became_active":"2024-09-13T13:05:15.326923+0000","last_became_peered":"2024-09-13T13:05:15.326923+0000","last_unstale":"2024-09-13T13:05:40.835035+0000","last_undegraded":"2024-09-13T13:05:40.835035+0000","last_fullsized":"2024-09-13T13:05:40.835035+0000","mapping_epoch":143,"log_start":"114'7185","ondisk_log_start":"114'7185","created":84,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118226968,"num_objects":1805,"num_object_clones":0,"num_object_copies":5415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1805,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13497,"num_write_kb":489601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3008,"num_bytes_recovered":155058200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":226,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906783+0000","last_change":"2024-09-13T13:04:02.832992+0000","last_active":"2024-09-13T13:05:37.906783+0000","last_peered":"2024-09-13T13:05:37.906783+0000","last_clean":"2024-09-13T13:05:37.906783+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:05:37.906783+0000","last_undegraded":"2024-09-13T13:05:37.906783+0000","last_fullsized":"2024-09-13T13:05:37.906783+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:04:02.832935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.832935+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":200,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906274+0000","last_change":"2024-09-13T13:02:45.015634+0000","last_active":"2024-09-13T13:05:37.906274+0000","last_peered":"2024-09-13T13:05:37.906274+0000","last_clean":"2024-09-13T13:05:37.906274+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:05:37.906274+0000","last_undegraded":"2024-09-13T13:05:37.906274+0000","last_fullsized":"2024-09-13T13:05:37.906274+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.015577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.015577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.095405+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.877418+0000","last_change":"2024-09-13T13:04:47.680651+0000","last_active":"2024-09-13T13:05:36.877418+0000","last_peered":"2024-09-13T13:05:36.877418+0000","last_clean":"2024-09-13T13:05:36.877418+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:05:36.877418+0000","last_undegraded":"2024-09-13T13:05:36.877418+0000","last_fullsized":"2024-09-13T13:05:36.877418+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.680579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.680579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:07.566204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":198,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.913663+0000","last_change":"2024-09-13T13:04:02.757378+0000","last_active":"2024-09-13T13:05:37.913663+0000","last_peered":"2024-09-13T13:05:37.913663+0000","last_clean":"2024-09-13T13:05:37.913663+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:05:37.913663+0000","last_undegraded":"2024-09-13T13:05:37.913663+0000","last_fullsized":"2024-09-13T13:05:37.913663+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.213470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.213470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"149'12746","reported_seq":13013,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:38.083853+0000","last_change":"2024-09-13T13:05:03.355522+0000","last_active":"2024-09-13T13:05:38.083853+0000","last_peered":"2024-09-13T13:05:38.083853+0000","last_clean":"2024-09-13T13:05:38.083853+0000","last_became_active":"2024-09-13T13:02:12.375509+0000","last_became_peered":"2024-09-13T13:02:12.375509+0000","last_unstale":"2024-09-13T13:05:38.083853+0000","last_undegraded":"2024-09-13T13:05:38.083853+0000","last_fullsized":"2024-09-13T13:05:38.083853+0000","mapping_epoch":84,"log_start":"108'5900","ondisk_log_start":"108'5900","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'10840","last_scrub_stamp":"2024-09-13T13:05:03.354789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:03.354789+0000","objects_scrubbed":3912,"log_size":6846,"log_dups_size":0,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.025217+0000","scrub_duration":2101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127926272,"num_objects":1952,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12746,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":232,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906793+0000","last_change":"2024-09-13T13:05:34.166464+0000","last_active":"2024-09-13T13:05:37.906793+0000","last_peered":"2024-09-13T13:05:37.906793+0000","last_clean":"2024-09-13T13:05:37.906793+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:05:37.906793+0000","last_undegraded":"2024-09-13T13:05:37.906793+0000","last_fullsized":"2024-09-13T13:05:37.906793+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:40.189820+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T13:05:39.922279+0000","last_change":"2024-09-13T13:04:02.764657+0000","last_active":"2024-09-13T13:05:39.922279+0000","last_peered":"2024-09-13T13:05:39.922279+0000","last_clean":"2024-09-13T13:05:39.922279+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:05:39.922279+0000","last_undegraded":"2024-09-13T13:05:39.922279+0000","last_fullsized":"2024-09-13T13:05:39.922279+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":184,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.920538+0000","last_change":"2024-09-13T13:01:06.208593+0000","last_active":"2024-09-13T13:05:37.920538+0000","last_peered":"2024-09-13T13:05:37.920538+0000","last_clean":"2024-09-13T13:05:37.920538+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:05:37.920538+0000","last_undegraded":"2024-09-13T13:05:37.920538+0000","last_fullsized":"2024-09-13T13:05:37.920538+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:22.718840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":195,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.918384+0000","last_change":"2024-09-13T13:05:35.310768+0000","last_active":"2024-09-13T13:05:37.918384+0000","last_peered":"2024-09-13T13:05:37.918384+0000","last_clean":"2024-09-13T13:05:37.918384+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:05:37.918384+0000","last_undegraded":"2024-09-13T13:05:37.918384+0000","last_fullsized":"2024-09-13T13:05:37.918384+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:35.310700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.310700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:56.914781+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"149'13222","reported_seq":13441,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:38.456870+0000","last_change":"2024-09-13T13:04:02.904392+0000","last_active":"2024-09-13T13:05:38.456870+0000","last_peered":"2024-09-13T13:05:38.456870+0000","last_clean":"2024-09-13T13:05:38.456870+0000","last_became_active":"2024-09-13T13:02:12.380229+0000","last_became_peered":"2024-09-13T13:02:12.380229+0000","last_unstale":"2024-09-13T13:05:38.456870+0000","last_undegraded":"2024-09-13T13:05:38.456870+0000","last_fullsized":"2024-09-13T13:05:38.456870+0000","mapping_epoch":84,"log_start":"115'7600","ondisk_log_start":"115'7600","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8189","last_scrub_stamp":"2024-09-13T13:04:02.904297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.904297+0000","objects_scrubbed":6969,"log_size":5622,"log_dups_size":0,"ondisk_log_size":5622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126484480,"num_objects":1930,"num_object_clones":0,"num_object_copies":5790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13222,"num_write_kb":484864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":222,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906515+0000","last_change":"2024-09-13T13:04:59.091852+0000","last_active":"2024-09-13T13:05:37.906515+0000","last_peered":"2024-09-13T13:05:37.906515+0000","last_clean":"2024-09-13T13:05:37.906515+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:05:37.906515+0000","last_undegraded":"2024-09-13T13:05:37.906515+0000","last_fullsized":"2024-09-13T13:05:37.906515+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:04:59.091751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.091751+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.396141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":207,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.916889+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:05:37.916889+0000","last_peered":"2024-09-13T13:05:37.916889+0000","last_clean":"2024-09-13T13:05:37.916889+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:05:37.916889+0000","last_undegraded":"2024-09-13T13:05:37.916889+0000","last_fullsized":"2024-09-13T13:05:37.916889+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":264,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.917574+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:05:37.917574+0000","last_peered":"2024-09-13T13:05:37.917574+0000","last_clean":"2024-09-13T13:05:37.917574+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:05:37.917574+0000","last_undegraded":"2024-09-13T13:05:37.917574+0000","last_fullsized":"2024-09-13T13:05:37.917574+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:57.041771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":187,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906423+0000","last_change":"2024-09-13T13:02:33.997780+0000","last_active":"2024-09-13T13:05:37.906423+0000","last_peered":"2024-09-13T13:05:37.906423+0000","last_clean":"2024-09-13T13:05:37.906423+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:05:37.906423+0000","last_undegraded":"2024-09-13T13:05:37.906423+0000","last_fullsized":"2024-09-13T13:05:37.906423+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.997740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.997740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.184374+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"149'13287","reported_seq":13552,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:38.050814+0000","last_change":"2024-09-13T13:05:08.464991+0000","last_active":"2024-09-13T13:05:38.050814+0000","last_peered":"2024-09-13T13:05:38.050814+0000","last_clean":"2024-09-13T13:05:38.050814+0000","last_became_active":"2024-09-13T13:02:12.375310+0000","last_became_peered":"2024-09-13T13:02:12.375310+0000","last_unstale":"2024-09-13T13:05:38.050814+0000","last_undegraded":"2024-09-13T13:05:38.050814+0000","last_fullsized":"2024-09-13T13:05:38.050814+0000","mapping_epoch":84,"log_start":"110'6400","ondisk_log_start":"110'6400","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'11530","last_scrub_stamp":"2024-09-13T13:05:08.464709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:08.464709+0000","objects_scrubbed":3682,"log_size":6887,"log_dups_size":0,"ondisk_log_size":6887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:11.110516+0000","scrub_duration":1226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123797504,"num_objects":1889,"num_object_clones":0,"num_object_copies":5667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13287,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":211,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.920215+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:05:37.920215+0000","last_peered":"2024-09-13T13:05:37.920215+0000","last_clean":"2024-09-13T13:05:37.920215+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:05:37.920215+0000","last_undegraded":"2024-09-13T13:05:37.920215+0000","last_fullsized":"2024-09-13T13:05:37.920215+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":216,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.520655+0000","last_change":"2024-09-13T13:05:37.520655+0000","last_active":"2024-09-13T13:05:37.520655+0000","last_peered":"2024-09-13T13:05:37.520655+0000","last_clean":"2024-09-13T13:05:37.520655+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:05:37.520655+0000","last_undegraded":"2024-09-13T13:05:37.520655+0000","last_fullsized":"2024-09-13T13:05:37.520655+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:37.520603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:37.520603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:56.825070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":200,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.918958+0000","last_change":"2024-09-13T13:04:56.183494+0000","last_active":"2024-09-13T13:05:37.918958+0000","last_peered":"2024-09-13T13:05:37.918958+0000","last_clean":"2024-09-13T13:05:37.918958+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:05:37.918958+0000","last_undegraded":"2024-09-13T13:05:37.918958+0000","last_fullsized":"2024-09-13T13:05:37.918958+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:56.183290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.183290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:07.795607+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":179,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.920295+0000","last_change":"2024-09-13T13:01:08.211229+0000","last_active":"2024-09-13T13:05:37.920295+0000","last_peered":"2024-09-13T13:05:37.920295+0000","last_clean":"2024-09-13T13:05:37.920295+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:05:37.920295+0000","last_undegraded":"2024-09-13T13:05:37.920295+0000","last_fullsized":"2024-09-13T13:05:37.920295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"149'13419","reported_seq":13561,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:38.455339+0000","last_change":"2024-09-13T13:02:12.379755+0000","last_active":"2024-09-13T13:05:38.455339+0000","last_peered":"2024-09-13T13:05:38.455339+0000","last_clean":"2024-09-13T13:05:38.455339+0000","last_became_active":"2024-09-13T13:02:12.379442+0000","last_became_peered":"2024-09-13T13:02:12.379442+0000","last_unstale":"2024-09-13T13:05:38.455339+0000","last_undegraded":"2024-09-13T13:05:38.455339+0000","last_fullsized":"2024-09-13T13:05:38.455339+0000","mapping_epoch":84,"log_start":"115'7800","ondisk_log_start":"115'7800","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5619,"log_dups_size":0,"ondisk_log_size":5619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123666432,"num_objects":1887,"num_object_clones":0,"num_object_copies":5661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13419,"num_write_kb":489792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":189,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T13:05:39.922146+0000","last_change":"2024-09-13T13:04:02.760376+0000","last_active":"2024-09-13T13:05:39.922146+0000","last_peered":"2024-09-13T13:05:39.922146+0000","last_clean":"2024-09-13T13:05:39.922146+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:05:39.922146+0000","last_undegraded":"2024-09-13T13:05:39.922146+0000","last_fullsized":"2024-09-13T13:05:39.922146+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:42.593284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.593284+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":210,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.924417+0000","last_change":"2024-09-13T13:04:02.759538+0000","last_active":"2024-09-13T13:05:37.924417+0000","last_peered":"2024-09-13T13:05:37.924417+0000","last_clean":"2024-09-13T13:05:37.924417+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:05:37.924417+0000","last_undegraded":"2024-09-13T13:05:37.924417+0000","last_fullsized":"2024-09-13T13:05:37.924417+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:29.012312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.012312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:13.429060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":200,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.918016+0000","last_change":"2024-09-13T13:04:47.927556+0000","last_active":"2024-09-13T13:05:37.918016+0000","last_peered":"2024-09-13T13:05:37.918016+0000","last_clean":"2024-09-13T13:05:37.918016+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:05:37.918016+0000","last_undegraded":"2024-09-13T13:05:37.918016+0000","last_fullsized":"2024-09-13T13:05:37.918016+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.927432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.927432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.726893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":187,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.923793+0000","last_change":"2024-09-13T13:05:11.420568+0000","last_active":"2024-09-13T13:05:37.923793+0000","last_peered":"2024-09-13T13:05:37.923793+0000","last_clean":"2024-09-13T13:05:37.923793+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:05:37.923793+0000","last_undegraded":"2024-09-13T13:05:37.923793+0000","last_fullsized":"2024-09-13T13:05:37.923793+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:11.420503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:11.420503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:12.545408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"149'13248","reported_seq":13442,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:38.212560+0000","last_change":"2024-09-13T13:03:45.723433+0000","last_active":"2024-09-13T13:05:38.212560+0000","last_peered":"2024-09-13T13:05:38.212560+0000","last_clean":"2024-09-13T13:05:38.212560+0000","last_became_active":"2024-09-13T13:02:12.376334+0000","last_became_peered":"2024-09-13T13:02:12.376334+0000","last_unstale":"2024-09-13T13:05:38.212560+0000","last_undegraded":"2024-09-13T13:05:38.212560+0000","last_fullsized":"2024-09-13T13:05:38.212560+0000","mapping_epoch":84,"log_start":"122'8600","ondisk_log_start":"122'8600","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":4774,"log_size":4648,"log_dups_size":0,"ondisk_log_size":4648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129236992,"num_objects":1972,"num_object_clones":0,"num_object_copies":5916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13248,"num_write_kb":487040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":219,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906171+0000","last_change":"2024-09-13T13:02:28.990089+0000","last_active":"2024-09-13T13:05:37.906171+0000","last_peered":"2024-09-13T13:05:37.906171+0000","last_clean":"2024-09-13T13:05:37.906171+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:05:37.906171+0000","last_undegraded":"2024-09-13T13:05:37.906171+0000","last_fullsized":"2024-09-13T13:05:37.906171+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:02:28.990005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.990005+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.110320+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":215,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.924699+0000","last_change":"2024-09-13T13:05:29.345863+0000","last_active":"2024-09-13T13:05:37.924699+0000","last_peered":"2024-09-13T13:05:37.924699+0000","last_clean":"2024-09-13T13:05:37.924699+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:05:37.924699+0000","last_undegraded":"2024-09-13T13:05:37.924699+0000","last_fullsized":"2024-09-13T13:05:37.924699+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.345796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.345796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.394429+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":192,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.917556+0000","last_change":"2024-09-13T13:05:29.218178+0000","last_active":"2024-09-13T13:05:37.917556+0000","last_peered":"2024-09-13T13:05:37.917556+0000","last_clean":"2024-09-13T13:05:37.917556+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:05:37.917556+0000","last_undegraded":"2024-09-13T13:05:37.917556+0000","last_fullsized":"2024-09-13T13:05:37.917556+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.218119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.218119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:30.944093+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":187,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.916303+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:05:37.916303+0000","last_peered":"2024-09-13T13:05:37.916303+0000","last_clean":"2024-09-13T13:05:37.916303+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:05:37.916303+0000","last_undegraded":"2024-09-13T13:05:37.916303+0000","last_fullsized":"2024-09-13T13:05:37.916303+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"149'13395","reported_seq":19585,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:38.049924+0000","last_change":"2024-09-13T13:05:29.580927+0000","last_active":"2024-09-13T13:05:38.049924+0000","last_peered":"2024-09-13T13:05:38.049924+0000","last_clean":"2024-09-13T13:05:38.049924+0000","last_became_active":"2024-09-13T13:04:48.765805+0000","last_became_peered":"2024-09-13T13:04:48.765805+0000","last_unstale":"2024-09-13T13:05:38.049924+0000","last_undegraded":"2024-09-13T13:05:38.049924+0000","last_fullsized":"2024-09-13T13:05:38.049924+0000","mapping_epoch":135,"log_start":"117'8300","ondisk_log_start":"117'8300","created":84,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'12879","last_scrub_stamp":"2024-09-13T13:05:29.580736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.580736+0000","objects_scrubbed":2542,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:43.250181+0000","scrub_duration":768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130875392,"num_objects":1997,"num_object_clones":0,"num_object_copies":5991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13395,"num_write_kb":492544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3000,"num_bytes_recovered":156631040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":229,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.875538+0000","last_change":"2024-09-13T13:04:52.666294+0000","last_active":"2024-09-13T13:05:36.875538+0000","last_peered":"2024-09-13T13:05:36.875538+0000","last_clean":"2024-09-13T13:05:36.875538+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:05:36.875538+0000","last_undegraded":"2024-09-13T13:05:36.875538+0000","last_fullsized":"2024-09-13T13:05:36.875538+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:52.666222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:52.666222+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:57.559648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":219,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.924505+0000","last_change":"2024-09-13T13:05:27.422529+0000","last_active":"2024-09-13T13:05:37.924505+0000","last_peered":"2024-09-13T13:05:37.924505+0000","last_clean":"2024-09-13T13:05:37.924505+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:05:37.924505+0000","last_undegraded":"2024-09-13T13:05:37.924505+0000","last_fullsized":"2024-09-13T13:05:37.924505+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:05:27.422451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:27.422451+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.192286+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":272,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.923732+0000","last_change":"2024-09-13T13:04:02.759553+0000","last_active":"2024-09-13T13:05:37.923732+0000","last_peered":"2024-09-13T13:05:37.923732+0000","last_clean":"2024-09-13T13:05:37.923732+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:05:37.923732+0000","last_undegraded":"2024-09-13T13:05:37.923732+0000","last_fullsized":"2024-09-13T13:05:37.923732+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:45.959285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.959285+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.359173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":196,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.875711+0000","last_change":"2024-09-13T13:04:02.757776+0000","last_active":"2024-09-13T13:05:36.875711+0000","last_peered":"2024-09-13T13:05:36.875711+0000","last_clean":"2024-09-13T13:05:36.875711+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:05:36.875711+0000","last_undegraded":"2024-09-13T13:05:36.875711+0000","last_fullsized":"2024-09-13T13:05:36.875711+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:28.529954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.529954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.625121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T13:05:39.922006+0000","last_change":"2024-09-13T13:04:02.756666+0000","last_active":"2024-09-13T13:05:39.922006+0000","last_peered":"2024-09-13T13:05:39.922006+0000","last_clean":"2024-09-13T13:05:39.922006+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:05:39.922006+0000","last_undegraded":"2024-09-13T13:05:39.922006+0000","last_fullsized":"2024-09-13T13:05:39.922006+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":203,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.920022+0000","last_change":"2024-09-13T13:03:14.504339+0000","last_active":"2024-09-13T13:05:37.920022+0000","last_peered":"2024-09-13T13:05:37.920022+0000","last_clean":"2024-09-13T13:05:37.920022+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:05:37.920022+0000","last_undegraded":"2024-09-13T13:05:37.920022+0000","last_fullsized":"2024-09-13T13:05:37.920022+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:03:14.504263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.504263+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":196,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.878509+0000","last_change":"2024-09-13T13:05:27.619960+0000","last_active":"2024-09-13T13:05:36.878509+0000","last_peered":"2024-09-13T13:05:36.878509+0000","last_clean":"2024-09-13T13:05:36.878509+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:05:36.878509+0000","last_undegraded":"2024-09-13T13:05:36.878509+0000","last_fullsized":"2024-09-13T13:05:36.878509+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:27.619884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:27.619884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.490575+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":205,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.924193+0000","last_change":"2024-09-13T13:05:37.197948+0000","last_active":"2024-09-13T13:05:37.924193+0000","last_peered":"2024-09-13T13:05:37.924193+0000","last_clean":"2024-09-13T13:05:37.924193+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:05:37.924193+0000","last_undegraded":"2024-09-13T13:05:37.924193+0000","last_fullsized":"2024-09-13T13:05:37.924193+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:37.197907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:37.197907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:55.791682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":210,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.922588+0000","last_change":"2024-09-13T13:02:42.589951+0000","last_active":"2024-09-13T13:05:37.922588+0000","last_peered":"2024-09-13T13:05:37.922588+0000","last_clean":"2024-09-13T13:05:37.922588+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:05:37.922588+0000","last_undegraded":"2024-09-13T13:05:37.922588+0000","last_fullsized":"2024-09-13T13:05:37.922588+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:42.589866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.589866+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":209,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.924017+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:05:37.924017+0000","last_peered":"2024-09-13T13:05:37.924017+0000","last_clean":"2024-09-13T13:05:37.924017+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:05:37.924017+0000","last_undegraded":"2024-09-13T13:05:37.924017+0000","last_fullsized":"2024-09-13T13:05:37.924017+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.542132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":182,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.919888+0000","last_change":"2024-09-13T13:01:06.210328+0000","last_active":"2024-09-13T13:05:37.919888+0000","last_peered":"2024-09-13T13:05:37.919888+0000","last_clean":"2024-09-13T13:05:37.919888+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:05:37.919888+0000","last_undegraded":"2024-09-13T13:05:37.919888+0000","last_fullsized":"2024-09-13T13:05:37.919888+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":187,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.917689+0000","last_change":"2024-09-13T13:03:21.281950+0000","last_active":"2024-09-13T13:05:37.917689+0000","last_peered":"2024-09-13T13:05:37.917689+0000","last_clean":"2024-09-13T13:05:37.917689+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:05:37.917689+0000","last_undegraded":"2024-09-13T13:05:37.917689+0000","last_fullsized":"2024-09-13T13:05:37.917689+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:21.281870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:21.281870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:42.889031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":208,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.877012+0000","last_change":"2024-09-13T13:04:53.687731+0000","last_active":"2024-09-13T13:05:36.877012+0000","last_peered":"2024-09-13T13:05:36.877012+0000","last_clean":"2024-09-13T13:05:36.877012+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:05:36.877012+0000","last_undegraded":"2024-09-13T13:05:36.877012+0000","last_fullsized":"2024-09-13T13:05:36.877012+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.632415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":198,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.919805+0000","last_change":"2024-09-13T13:02:41.622369+0000","last_active":"2024-09-13T13:05:37.919805+0000","last_peered":"2024-09-13T13:05:37.919805+0000","last_clean":"2024-09-13T13:05:37.919805+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:05:37.919805+0000","last_undegraded":"2024-09-13T13:05:37.919805+0000","last_fullsized":"2024-09-13T13:05:37.919805+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.622292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.622292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":246,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.913887+0000","last_change":"2024-09-13T13:01:06.210251+0000","last_active":"2024-09-13T13:05:37.913887+0000","last_peered":"2024-09-13T13:05:37.913887+0000","last_clean":"2024-09-13T13:05:37.913887+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:05:37.913887+0000","last_undegraded":"2024-09-13T13:05:37.913887+0000","last_fullsized":"2024-09-13T13:05:37.913887+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":187,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906545+0000","last_change":"2024-09-13T13:04:49.927118+0000","last_active":"2024-09-13T13:05:37.906545+0000","last_peered":"2024-09-13T13:05:37.906545+0000","last_clean":"2024-09-13T13:05:37.906545+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:05:37.906545+0000","last_undegraded":"2024-09-13T13:05:37.906545+0000","last_fullsized":"2024-09-13T13:05:37.906545+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:49.927073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:49.927073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:08.714937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":209,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.914735+0000","last_change":"2024-09-13T13:02:37.222400+0000","last_active":"2024-09-13T13:05:37.914735+0000","last_peered":"2024-09-13T13:05:37.914735+0000","last_clean":"2024-09-13T13:05:37.914735+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:05:37.914735+0000","last_undegraded":"2024-09-13T13:05:37.914735+0000","last_fullsized":"2024-09-13T13:05:37.914735+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:37.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.222329+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":198,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.924105+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:05:37.924105+0000","last_peered":"2024-09-13T13:05:37.924105+0000","last_clean":"2024-09-13T13:05:37.924105+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:05:37.924105+0000","last_undegraded":"2024-09-13T13:05:37.924105+0000","last_fullsized":"2024-09-13T13:05:37.924105+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:55.131839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":260,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.876871+0000","last_change":"2024-09-13T13:03:41.676060+0000","last_active":"2024-09-13T13:05:36.876871+0000","last_peered":"2024-09-13T13:05:36.876871+0000","last_clean":"2024-09-13T13:05:36.876871+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:05:36.876871+0000","last_undegraded":"2024-09-13T13:05:36.876871+0000","last_fullsized":"2024-09-13T13:05:36.876871+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:03:41.675978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:41.675978+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":186,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.921040+0000","last_change":"2024-09-13T13:02:40.615749+0000","last_active":"2024-09-13T13:05:37.921040+0000","last_peered":"2024-09-13T13:05:37.921040+0000","last_clean":"2024-09-13T13:05:37.921040+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:05:37.921040+0000","last_undegraded":"2024-09-13T13:05:37.921040+0000","last_fullsized":"2024-09-13T13:05:37.921040+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:40.615674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:40.615674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":233,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906512+0000","last_change":"2024-09-13T13:05:26.186899+0000","last_active":"2024-09-13T13:05:37.906512+0000","last_peered":"2024-09-13T13:05:37.906512+0000","last_clean":"2024-09-13T13:05:37.906512+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:05:37.906512+0000","last_undegraded":"2024-09-13T13:05:37.906512+0000","last_fullsized":"2024-09-13T13:05:37.906512+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:05:26.186830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.186830+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:29.937721+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":218,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.906693+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:05:37.906693+0000","last_peered":"2024-09-13T13:05:37.906693+0000","last_clean":"2024-09-13T13:05:37.906693+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:05:37.906693+0000","last_undegraded":"2024-09-13T13:05:37.906693+0000","last_fullsized":"2024-09-13T13:05:37.906693+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:39.039395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":199,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.920929+0000","last_change":"2024-09-13T13:03:44.467611+0000","last_active":"2024-09-13T13:05:37.920929+0000","last_peered":"2024-09-13T13:05:37.920929+0000","last_clean":"2024-09-13T13:05:37.920929+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:05:37.920929+0000","last_undegraded":"2024-09-13T13:05:37.920929+0000","last_fullsized":"2024-09-13T13:05:37.920929+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:44.467297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:44.467297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":186,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.915163+0000","last_change":"2024-09-13T13:02:42.208021+0000","last_active":"2024-09-13T13:05:37.915163+0000","last_peered":"2024-09-13T13:05:37.915163+0000","last_clean":"2024-09-13T13:05:37.915163+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:05:37.915163+0000","last_undegraded":"2024-09-13T13:05:37.915163+0000","last_fullsized":"2024-09-13T13:05:37.915163+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:42.207956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.207956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":160,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.875594+0000","last_change":"2024-09-13T13:05:35.506093+0000","last_active":"2024-09-13T13:05:36.875594+0000","last_peered":"2024-09-13T13:05:36.875594+0000","last_clean":"2024-09-13T13:05:36.875594+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:05:36.875594+0000","last_undegraded":"2024-09-13T13:05:36.875594+0000","last_fullsized":"2024-09-13T13:05:36.875594+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:05:35.506037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.506037+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:58.281227+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":198,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.924337+0000","last_change":"2024-09-13T13:02:39.921391+0000","last_active":"2024-09-13T13:05:37.924337+0000","last_peered":"2024-09-13T13:05:37.924337+0000","last_clean":"2024-09-13T13:05:37.924337+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:05:37.924337+0000","last_undegraded":"2024-09-13T13:05:37.924337+0000","last_fullsized":"2024-09-13T13:05:37.924337+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.921325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.921325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:20.381317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":182,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.916350+0000","last_change":"2024-09-13T13:01:06.211993+0000","last_active":"2024-09-13T13:05:37.916350+0000","last_peered":"2024-09-13T13:05:37.916350+0000","last_clean":"2024-09-13T13:05:37.916350+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:05:37.916350+0000","last_undegraded":"2024-09-13T13:05:37.916350+0000","last_fullsized":"2024-09-13T13:05:37.916350+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":186,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.919383+0000","last_change":"2024-09-13T13:02:36.586702+0000","last_active":"2024-09-13T13:05:37.919383+0000","last_peered":"2024-09-13T13:05:37.919383+0000","last_clean":"2024-09-13T13:05:37.919383+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:05:37.919383+0000","last_undegraded":"2024-09-13T13:05:37.919383+0000","last_fullsized":"2024-09-13T13:05:37.919383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.586654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.586654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":224,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.915309+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:05:37.915309+0000","last_peered":"2024-09-13T13:05:37.915309+0000","last_clean":"2024-09-13T13:05:37.915309+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:05:37.915309+0000","last_undegraded":"2024-09-13T13:05:37.915309+0000","last_fullsized":"2024-09-13T13:05:37.915309+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":196,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T13:05:36.878596+0000","last_change":"2024-09-13T13:05:12.597696+0000","last_active":"2024-09-13T13:05:36.878596+0000","last_peered":"2024-09-13T13:05:36.878596+0000","last_clean":"2024-09-13T13:05:36.878596+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:05:36.878596+0000","last_undegraded":"2024-09-13T13:05:36.878596+0000","last_fullsized":"2024-09-13T13:05:36.878596+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:12.597637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.597637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:30.070820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":201,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.923579+0000","last_change":"2024-09-13T13:05:08.432874+0000","last_active":"2024-09-13T13:05:37.923579+0000","last_peered":"2024-09-13T13:05:37.923579+0000","last_clean":"2024-09-13T13:05:37.923579+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:05:37.923579+0000","last_undegraded":"2024-09-13T13:05:37.923579+0000","last_fullsized":"2024-09-13T13:05:37.923579+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:08.432816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:08.432816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:09.081986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":189,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.916493+0000","last_change":"2024-09-13T13:04:02.760695+0000","last_active":"2024-09-13T13:05:37.916493+0000","last_peered":"2024-09-13T13:05:37.916493+0000","last_clean":"2024-09-13T13:05:37.916493+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:05:37.916493+0000","last_undegraded":"2024-09-13T13:05:37.916493+0000","last_fullsized":"2024-09-13T13:05:37.916493+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":216,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.916021+0000","last_change":"2024-09-13T13:04:02.755722+0000","last_active":"2024-09-13T13:05:37.916021+0000","last_peered":"2024-09-13T13:05:37.916021+0000","last_clean":"2024-09-13T13:05:37.916021+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:05:37.916021+0000","last_undegraded":"2024-09-13T13:05:37.916021+0000","last_fullsized":"2024-09-13T13:05:37.916021+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":198,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.921563+0000","last_change":"2024-09-13T13:03:15.466909+0000","last_active":"2024-09-13T13:05:37.921563+0000","last_peered":"2024-09-13T13:05:37.921563+0000","last_clean":"2024-09-13T13:05:37.921563+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:05:37.921563+0000","last_undegraded":"2024-09-13T13:05:37.921563+0000","last_fullsized":"2024-09-13T13:05:37.921563+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:15.466859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:15.466859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":86,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T13:05:39.922028+0000","last_change":"2024-09-13T13:04:02.765046+0000","last_active":"2024-09-13T13:05:39.922028+0000","last_peered":"2024-09-13T13:05:39.922028+0000","last_clean":"2024-09-13T13:05:39.922028+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:05:39.922028+0000","last_undegraded":"2024-09-13T13:05:39.922028+0000","last_fullsized":"2024-09-13T13:05:39.922028+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":186,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.923811+0000","last_change":"2024-09-13T13:04:58.468060+0000","last_active":"2024-09-13T13:05:37.923811+0000","last_peered":"2024-09-13T13:05:37.923811+0000","last_clean":"2024-09-13T13:05:37.923811+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:05:37.923811+0000","last_undegraded":"2024-09-13T13:05:37.923811+0000","last_fullsized":"2024-09-13T13:05:37.923811+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:58.468011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:58.468011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.129704+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":187,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.923801+0000","last_change":"2024-09-13T13:02:43.234909+0000","last_active":"2024-09-13T13:05:37.923801+0000","last_peered":"2024-09-13T13:05:37.923801+0000","last_clean":"2024-09-13T13:05:37.923801+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:05:37.923801+0000","last_undegraded":"2024-09-13T13:05:37.923801+0000","last_fullsized":"2024-09-13T13:05:37.923801+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:43.234847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.234847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":198,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.916274+0000","last_change":"2024-09-13T13:02:46.246602+0000","last_active":"2024-09-13T13:05:37.916274+0000","last_peered":"2024-09-13T13:05:37.916274+0000","last_clean":"2024-09-13T13:05:37.916274+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:05:37.916274+0000","last_undegraded":"2024-09-13T13:05:37.916274+0000","last_fullsized":"2024-09-13T13:05:37.916274+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.246540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.246540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":196,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T13:05:39.922476+0000","last_change":"2024-09-13T13:04:02.756287+0000","last_active":"2024-09-13T13:05:39.922476+0000","last_peered":"2024-09-13T13:05:39.922476+0000","last_clean":"2024-09-13T13:05:39.922476+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:05:39.922476+0000","last_undegraded":"2024-09-13T13:05:39.922476+0000","last_fullsized":"2024-09-13T13:05:39.922476+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:02:39.557251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.557251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":191,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T13:05:37.922290+0000","last_change":"2024-09-13T13:02:30.512536+0000","last_active":"2024-09-13T13:05:37.922290+0000","last_peered":"2024-09-13T13:05:37.922290+0000","last_clean":"2024-09-13T13:05:37.922290+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:05:37.922290+0000","last_undegraded":"2024-09-13T13:05:37.922290+0000","last_fullsized":"2024-09-13T13:05:37.922290+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:30.512433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:30.512433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2000486424,"num_objects":30526,"num_object_clones":0,"num_object_copies":91578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211672,"num_write_kb":7750273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14440,"num_bytes_recovered":756088856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6033780736,"data_stored":6033768520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89239,"ondisk_log_size":89239,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":270,"num_read_kb":265,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201338,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":736224,"kb_used_data":637004,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93008928,"statfs":{"total":95995035648,"available":95241142272,"internally_reserved":0,"allocated":652292096,"data_stored":651876707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806140949,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":737516,"kb_used_data":587608,"kb_used_omap":4,"kb_used_meta":149883,"kb_avail":93007636,"statfs":{"total":95995035648,"available":95239819264,"internally_reserved":0,"allocated":601710592,"data_stored":601295116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":153480877},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691910,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":614100,"kb_used_data":514880,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93131052,"statfs":{"total":95995035648,"available":95366197248,"internally_reserved":0,"allocated":527237120,"data_stored":526832315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":732776,"kb_used_data":633556,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93012376,"statfs":{"total":95995035648,"available":95244673024,"internally_reserved":0,"allocated":648761344,"data_stored":648336195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117075,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1294508,"kb_used_data":1128536,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92450644,"statfs":{"total":95995035648,"available":94669459456,"internally_reserved":0,"allocated":1155620864,"data_stored":1155192449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":869508,"kb_used_data":770288,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92875644,"statfs":{"total":95995035648,"available":95104659456,"internally_reserved":0,"allocated":788774912,"data_stored":788385831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":879360,"kb_used_data":757932,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92865792,"statfs":{"total":95995035648,"available":95094571008,"internally_reserved":0,"allocated":776122368,"data_stored":775673178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1159984,"kb_used_data":989984,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":92585168,"statfs":{"total":95995035648,"available":94807212032,"internally_reserved":0,"allocated":1013743616,"data_stored":1013314383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":174062021},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1001197568,"data_stored":1001193496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":764612608,"data_stored":764608536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":766246912,"data_stored":766246912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1134952448,"data_stored":1134952448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":630784000,"data_stored":630784000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":519766016,"data_stored":519766016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":580849664,"data_stored":580845592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":635371520,"data_stored":635371520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:05:44.039 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T13:05:44.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:05:43 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:05:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:43 smithi167 ceph-mon[33178]: 5.17 scrub starts 2024-09-13T13:05:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:43 smithi167 ceph-mon[33178]: 5.17 scrub ok 2024-09-13T13:05:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:43 smithi167 ceph-mon[33178]: from='client.15216 v1:172.21.15.52:0/1013214833' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:43 smithi167 ceph-mon[33178]: 3.10 scrub starts 2024-09-13T13:05:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:43 smithi167 ceph-mon[33178]: 3.10 scrub ok 2024-09-13T13:05:44.126 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:44.130 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:43 smithi052 ceph-mon[36682]: 5.17 scrub starts 2024-09-13T13:05:44.131 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:43 smithi052 ceph-mon[36682]: 5.17 scrub ok 2024-09-13T13:05:44.131 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:43 smithi052 ceph-mon[36682]: from='client.15216 v1:172.21.15.52:0/1013214833' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:44.131 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:43 smithi052 ceph-mon[36682]: 3.10 scrub starts 2024-09-13T13:05:44.131 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:43 smithi052 ceph-mon[36682]: 3.10 scrub ok 2024-09-13T13:05:44.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:43 smithi052 ceph-mon[29471]: 5.17 scrub starts 2024-09-13T13:05:44.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:43 smithi052 ceph-mon[29471]: 5.17 scrub ok 2024-09-13T13:05:44.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:43 smithi052 ceph-mon[29471]: from='client.15216 v1:172.21.15.52:0/1013214833' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:44.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:43 smithi052 ceph-mon[29471]: 3.10 scrub starts 2024-09-13T13:05:44.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:43 smithi052 ceph-mon[29471]: 3.10 scrub ok 2024-09-13T13:05:44.151 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:44.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:44.253 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:44.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:44.316 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:44.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:44.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:44.443 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:44.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:44.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:44.476 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T13:05:44.545 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:44.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:44.640 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:44.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:44.741 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:44.785 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:44.824 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:44.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:44.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:44.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:44.950 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:44.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:44.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:45.052 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:45.102 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:45.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:45.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:45.203 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:45.217 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:44 smithi052 ceph-mon[36682]: pgmap v209: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.48k op/s; 744 KiB/s, 74 objects/s recovering 2024-09-13T13:05:45.217 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:44 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:45.217 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:44 smithi052 ceph-mon[36682]: 4.4 scrub starts 2024-09-13T13:05:45.217 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:44 smithi052 ceph-mon[36682]: 4.4 scrub ok 2024-09-13T13:05:45.217 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:44 smithi052 ceph-mon[36682]: 3.a scrub starts 2024-09-13T13:05:45.217 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:44 smithi052 ceph-mon[36682]: 3.a scrub ok 2024-09-13T13:05:45.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:44 smithi052 ceph-mon[29471]: pgmap v209: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.48k op/s; 744 KiB/s, 74 objects/s recovering 2024-09-13T13:05:45.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:44 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:45.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:44 smithi052 ceph-mon[29471]: 4.4 scrub starts 2024-09-13T13:05:45.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:44 smithi052 ceph-mon[29471]: 4.4 scrub ok 2024-09-13T13:05:45.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:44 smithi052 ceph-mon[29471]: 3.a scrub starts 2024-09-13T13:05:45.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:44 smithi052 ceph-mon[29471]: 3.a scrub ok 2024-09-13T13:05:45.228 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:45.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:45.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:45.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:44 smithi167 ceph-mon[33178]: pgmap v209: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.48k op/s; 744 KiB/s, 74 objects/s recovering 2024-09-13T13:05:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:44 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:44 smithi167 ceph-mon[33178]: 4.4 scrub starts 2024-09-13T13:05:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:44 smithi167 ceph-mon[33178]: 4.4 scrub ok 2024-09-13T13:05:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:44 smithi167 ceph-mon[33178]: 3.a scrub starts 2024-09-13T13:05:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:44 smithi167 ceph-mon[33178]: 3.a scrub ok 2024-09-13T13:05:45.329 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:45.355 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:45.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:45.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:45.457 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:45.463 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:45.475 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:45.520 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:45.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:45.622 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:45.647 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:45.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:45.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:45.749 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:45.805 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:45.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:45.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:45.934 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:45.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:45.976 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:05:46.011 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:05:46.011 INFO:teuthology.orchestra.run.smithi052.stdout:{"fsid":"2b415b38-71cf-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":528,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":153,"num_osds":8,"num_up_osds":8,"osd_up_since":1726232642,"num_in_osds":8,"osd_in_since":1726232429,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":147},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":148,"num_pools":7,"num_objects":27213,"data_bytes":1770920499,"bytes_used":6390910976,"bytes_avail":761569374208,"bytes_total":767960285184,"read_bytes_sec":852,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":688},"fsmap":{"epoch":1,"btime":"2024-09-13T12:55:27:893730+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.52:8443/","prometheus":"http://172.21.15.52:9283/"}},"servicemap":{"epoch":15,"modified":"2024-09-13T13:05:01.658350+0000","services":{"rgw":{"daemons":{"summary":"","14493":{"start_epoch":12,"start_stamp":"2024-09-13T13:01:09.302524+0000","gid":14493,"addr":"172.21.15.52:0/2628990817","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-4917-gb0201b8c (b0201b8c79733293453e7f10a10c7fa43119222b) squid (dev)","ceph_version_short":"19.3.0-4917-gb0201b8c","container_hostname":"smithi052","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi052","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024","kernel_version":"5.14.0-505.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32493496","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"434eb8a4-923f-4e38-a7a5-e2b5e50a0eee","zone_name":"default","zonegroup_id":"5289daf0-e99a-4da1-a9e6-98887649b2bb","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T13:05:46.035 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:46.087 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:46.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:46.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:46.189 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:46.215 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:46.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:46.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:46.224 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[36682]: 6.1d scrub starts 2024-09-13T13:05:46.224 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[36682]: 4.10 scrub starts 2024-09-13T13:05:46.225 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[36682]: 4.10 scrub ok 2024-09-13T13:05:46.225 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[36682]: 6.1d scrub ok 2024-09-13T13:05:46.225 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/584779417' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:05:46.225 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[36682]: from='client.25105 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:05:46.225 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[36682]: 6.14 scrub starts 2024-09-13T13:05:46.225 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[36682]: 6.14 scrub ok 2024-09-13T13:05:46.225 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:05:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:05:46.226 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[29471]: 6.1d scrub starts 2024-09-13T13:05:46.226 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[29471]: 4.10 scrub starts 2024-09-13T13:05:46.226 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[29471]: 4.10 scrub ok 2024-09-13T13:05:46.226 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[29471]: 6.1d scrub ok 2024-09-13T13:05:46.226 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/584779417' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:05:46.226 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[29471]: from='client.25105 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:05:46.226 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[29471]: 6.14 scrub starts 2024-09-13T13:05:46.226 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:45 smithi052 ceph-mon[29471]: 6.14 scrub ok 2024-09-13T13:05:46.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:46.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:45 smithi167 ceph-mon[33178]: 6.1d scrub starts 2024-09-13T13:05:46.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:45 smithi167 ceph-mon[33178]: 4.10 scrub starts 2024-09-13T13:05:46.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:45 smithi167 ceph-mon[33178]: 4.10 scrub ok 2024-09-13T13:05:46.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:45 smithi167 ceph-mon[33178]: 6.1d scrub ok 2024-09-13T13:05:46.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:45 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/584779417' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:05:46.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:45 smithi167 ceph-mon[33178]: from='client.25105 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:05:46.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:45 smithi167 ceph-mon[33178]: 6.14 scrub starts 2024-09-13T13:05:46.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:45 smithi167 ceph-mon[33178]: 6.14 scrub ok 2024-09-13T13:05:46.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:46.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:46.446 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:46.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:46.471 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:46.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:46.573 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:46.624 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:46.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:46.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:46.726 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:46.742 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:05:46.742 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:05:46.751 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:46.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:46.781 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:05:46.781 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:05:46.825 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:05:46.829 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:05:46.853 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:46.864 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:05:46.864 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:05:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:46.890 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:46.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:46.898 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:05:46.898 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:05:46.937 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:05:46.937 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:05:46.968 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:05:46.968 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:05:46.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:47.005 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:05:47.005 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:05:47.018 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:47.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:47.120 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:47.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[36682]: 3.3 scrub starts 2024-09-13T13:05:47.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[36682]: 3.3 scrub ok 2024-09-13T13:05:47.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[36682]: 5.15 scrub starts 2024-09-13T13:05:47.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[36682]: 5.15 scrub ok 2024-09-13T13:05:47.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[36682]: pgmap v210: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 689 op/s 2024-09-13T13:05:47.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[36682]: 6.1b scrub starts 2024-09-13T13:05:47.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[36682]: 6.1b scrub ok 2024-09-13T13:05:47.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:47.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[36682]: from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:05:47.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[36682]: osdmap e153: 8 total, 8 up, 8 in 2024-09-13T13:05:47.150 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3509701453' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T13:05:47.150 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1914151414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:47.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[29471]: 3.3 scrub starts 2024-09-13T13:05:47.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[29471]: 3.3 scrub ok 2024-09-13T13:05:47.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[29471]: 5.15 scrub starts 2024-09-13T13:05:47.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[29471]: 5.15 scrub ok 2024-09-13T13:05:47.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[29471]: pgmap v210: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 689 op/s 2024-09-13T13:05:47.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[29471]: 6.1b scrub starts 2024-09-13T13:05:47.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[29471]: 6.1b scrub ok 2024-09-13T13:05:47.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:47.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[29471]: from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:05:47.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[29471]: osdmap e153: 8 total, 8 up, 8 in 2024-09-13T13:05:47.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3509701453' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T13:05:47.151 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:46 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1914151414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:47.162 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:47.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:47.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:47.264 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:47.289 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:47.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:47.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:46 smithi167 ceph-mon[33178]: 3.3 scrub starts 2024-09-13T13:05:47.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:46 smithi167 ceph-mon[33178]: 3.3 scrub ok 2024-09-13T13:05:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:46 smithi167 ceph-mon[33178]: 5.15 scrub starts 2024-09-13T13:05:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:46 smithi167 ceph-mon[33178]: 5.15 scrub ok 2024-09-13T13:05:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:46 smithi167 ceph-mon[33178]: pgmap v210: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 689 op/s 2024-09-13T13:05:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:46 smithi167 ceph-mon[33178]: 6.1b scrub starts 2024-09-13T13:05:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:46 smithi167 ceph-mon[33178]: 6.1b scrub ok 2024-09-13T13:05:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:46 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:46 smithi167 ceph-mon[33178]: from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:05:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:46 smithi167 ceph-mon[33178]: osdmap e153: 8 total, 8 up, 8 in 2024-09-13T13:05:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:46 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3509701453' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T13:05:47.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:46 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1914151414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:47.393 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:47.433 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:47.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:47.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:47.535 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:47.570 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:47.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:47.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:47.695 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:47.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:47.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:47.797 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:47.834 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:47.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:47.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:47.936 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:47.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:47.979 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:47.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:48.080 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:48.085 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:05:48.085 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:05:48.121 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:48.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:48.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:47 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:05:48.190 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:47 smithi052 ceph-mon[36682]: osdmap e154: 8 total, 8 up, 8 in 2024-09-13T13:05:48.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:47 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:05:48.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:47 smithi052 ceph-mon[29471]: osdmap e154: 8 total, 8 up, 8 in 2024-09-13T13:05:48.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:48.251 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:48.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:48.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:48.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:47 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:05:48.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:47 smithi167 ceph-mon[33178]: osdmap e154: 8 total, 8 up, 8 in 2024-09-13T13:05:48.353 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:48.389 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:48.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:48.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:48.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:48.518 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:48.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:48.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:48.567 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-13T13:05:48.567 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:05:48.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:48.667 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:48.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:48.769 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:48.793 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:48.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:48.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:48.895 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:48.944 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:48.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:49.045 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:49.087 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:49.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:49.187 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:49.201 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:05:49.220 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:48 smithi052 ceph-mon[36682]: pgmap v212: 148 pgs: 148 active+clean; 1.6 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 735 op/s; 0 B/s, 51 objects/s recovering 2024-09-13T13:05:49.220 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:48 smithi052 ceph-mon[36682]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T13:05:49.221 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:48 smithi052 ceph-mon[29471]: pgmap v212: 148 pgs: 148 active+clean; 1.6 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 735 op/s; 0 B/s, 51 objects/s recovering 2024-09-13T13:05:49.221 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:48 smithi052 ceph-mon[29471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T13:05:49.229 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:49.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:49.283 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T13:05:49.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:48 smithi167 ceph-mon[33178]: pgmap v212: 148 pgs: 148 active+clean; 1.6 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 735 op/s; 0 B/s, 51 objects/s recovering 2024-09-13T13:05:49.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:48 smithi167 ceph-mon[33178]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T13:05:49.332 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:49.397 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:49.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:49.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:49.411 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:49.499 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:49.525 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:49.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:49.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:49.627 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:49.686 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:49.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:49.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:49.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:49.802 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:05:49.812 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:49.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:49.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:49.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:49.938 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:49.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:49.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:50.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:50.063 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:50.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:50.148 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:50.165 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:50.178 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:50.238 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:50.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:50.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:50.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:50.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:50.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:50.469 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:50.503 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:50.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:50.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:50.509 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:05:50.511 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:05:50.606 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:50.645 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:50.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:50.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:50.747 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:50.785 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:50.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:50.887 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:50.915 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:50.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:50.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:51.016 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:51.044 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:51.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:51.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:51.146 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:51.192 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:51.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:51.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:50 smithi052 ceph-mon[29471]: pgmap v214: 148 pgs: 148 active+clean; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.24k op/s; 0 B/s, 45 objects/s recovering 2024-09-13T13:05:51.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:50 smithi052 ceph-mon[29471]: from='client.35102 v1:172.21.15.52:0/2483396547' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:51.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:50 smithi052 ceph-mon[36682]: pgmap v214: 148 pgs: 148 active+clean; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.24k op/s; 0 B/s, 45 objects/s recovering 2024-09-13T13:05:51.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:50 smithi052 ceph-mon[36682]: from='client.35102 v1:172.21.15.52:0/2483396547' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:51.294 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:51.319 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:51.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:51.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:51.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:50 smithi167 ceph-mon[33178]: pgmap v214: 148 pgs: 148 active+clean; 1.3 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.24k op/s; 0 B/s, 45 objects/s recovering 2024-09-13T13:05:51.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:50 smithi167 ceph-mon[33178]: from='client.35102 v1:172.21.15.52:0/2483396547' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:51.421 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:51.466 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:51.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:51.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:51.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:51.609 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:51.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:51.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:51.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:51.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:51.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:51.839 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:51.868 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:51.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:51.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:51.970 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:52.016 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:52.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:52.118 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:52.144 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:52.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:52.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:52.246 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:52.251 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-13T13:05:49.674337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1441667635,"num_objects":22189,"num_object_clones":0,"num_object_copies":66567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22189,"num_whiteouts":0,"num_read":1127,"num_read_kb":984,"num_write":220757,"num_write_kb":7751946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14760,"num_bytes_recovered":756679334,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89935,"ondisk_log_size":89935,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5356440,"kb_used_data":4323016,"kb_used_omap":15,"kb_used_meta":1033008,"kb_avail":744604776,"statfs":{"total":767960285184,"available":762475290624,"internally_reserved":0,"allocated":4426768384,"data_stored":4423210801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":1057800985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,0,0,0,0,0,0,16],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-559415296,"num_objects":-8536,"num_object_clones":0,"num_object_copies":-25608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8536,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8536,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.898710"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.786621+0000","last_change":"2024-09-13T13:02:38.651382+0000","last_active":"2024-09-13T13:05:47.786621+0000","last_peered":"2024-09-13T13:05:47.786621+0000","last_clean":"2024-09-13T13:05:47.786621+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:05:47.786621+0000","last_undegraded":"2024-09-13T13:05:47.786621+0000","last_fullsized":"2024-09-13T13:05:47.786621+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:38.651314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:38.651314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":232,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780418+0000","last_change":"2024-09-13T13:04:48.899982+0000","last_active":"2024-09-13T13:05:47.780418+0000","last_peered":"2024-09-13T13:05:47.780418+0000","last_clean":"2024-09-13T13:05:47.780418+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:05:47.780418+0000","last_undegraded":"2024-09-13T13:05:47.780418+0000","last_fullsized":"2024-09-13T13:05:47.780418+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:48.899915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.899915+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:00.227319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.801626+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:05:47.801626+0000","last_peered":"2024-09-13T13:05:47.801626+0000","last_clean":"2024-09-13T13:05:47.801626+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:05:47.801626+0000","last_undegraded":"2024-09-13T13:05:47.801626+0000","last_fullsized":"2024-09-13T13:05:47.801626+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:47.640365+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780536+0000","last_change":"2024-09-13T13:05:01.076346+0000","last_active":"2024-09-13T13:05:47.780536+0000","last_peered":"2024-09-13T13:05:47.780536+0000","last_clean":"2024-09-13T13:05:47.780536+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:05:47.780536+0000","last_undegraded":"2024-09-13T13:05:47.780536+0000","last_fullsized":"2024-09-13T13:05:47.780536+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.076275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.076275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:14.180362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":207,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.789144+0000","last_change":"2024-09-13T13:05:45.805098+0000","last_active":"2024-09-13T13:05:47.789144+0000","last_peered":"2024-09-13T13:05:47.789144+0000","last_clean":"2024-09-13T13:05:47.789144+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:05:47.789144+0000","last_undegraded":"2024-09-13T13:05:47.789144+0000","last_fullsized":"2024-09-13T13:05:47.789144+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:45.800516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.800516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:46.430007+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":220,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.790400+0000","last_change":"2024-09-13T13:04:02.757267+0000","last_active":"2024-09-13T13:05:47.790400+0000","last_peered":"2024-09-13T13:05:47.790400+0000","last_clean":"2024-09-13T13:05:47.790400+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:05:47.790400+0000","last_undegraded":"2024-09-13T13:05:47.790400+0000","last_fullsized":"2024-09-13T13:05:47.790400+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.233259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.233259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":237,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.793872+0000","last_change":"2024-09-13T13:04:02.396436+0000","last_active":"2024-09-13T13:05:47.793872+0000","last_peered":"2024-09-13T13:05:47.793872+0000","last_clean":"2024-09-13T13:05:47.793872+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:05:47.793872+0000","last_undegraded":"2024-09-13T13:05:47.793872+0000","last_fullsized":"2024-09-13T13:05:47.793872+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:02.396342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.396342+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":212,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780339+0000","last_change":"2024-09-13T13:04:02.765671+0000","last_active":"2024-09-13T13:05:47.780339+0000","last_peered":"2024-09-13T13:05:47.780339+0000","last_clean":"2024-09-13T13:05:47.780339+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:05:47.780339+0000","last_undegraded":"2024-09-13T13:05:47.780339+0000","last_fullsized":"2024-09-13T13:05:47.780339+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.021662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.021662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:18.072524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":225,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796562+0000","last_change":"2024-09-13T13:05:00.690387+0000","last_active":"2024-09-13T13:05:47.796562+0000","last_peered":"2024-09-13T13:05:47.796562+0000","last_clean":"2024-09-13T13:05:47.796562+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:05:47.796562+0000","last_undegraded":"2024-09-13T13:05:47.796562+0000","last_fullsized":"2024-09-13T13:05:47.796562+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:05:00.690308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:00.690308+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:03.644084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":221,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780674+0000","last_change":"2024-09-13T13:04:56.066382+0000","last_active":"2024-09-13T13:05:47.780674+0000","last_peered":"2024-09-13T13:05:47.780674+0000","last_clean":"2024-09-13T13:05:47.780674+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:05:47.780674+0000","last_undegraded":"2024-09-13T13:05:47.780674+0000","last_fullsized":"2024-09-13T13:05:47.780674+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:56.066326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.066326+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.781998+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796218+0000","last_change":"2024-09-13T13:05:26.621813+0000","last_active":"2024-09-13T13:05:47.796218+0000","last_peered":"2024-09-13T13:05:47.796218+0000","last_clean":"2024-09-13T13:05:47.796218+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:05:47.796218+0000","last_undegraded":"2024-09-13T13:05:47.796218+0000","last_fullsized":"2024-09-13T13:05:47.796218+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:26.621732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.621732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:48.964955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":196,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.779833+0000","last_change":"2024-09-13T13:02:39.607712+0000","last_active":"2024-09-13T13:05:47.779833+0000","last_peered":"2024-09-13T13:05:47.779833+0000","last_clean":"2024-09-13T13:05:47.779833+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:05:47.779833+0000","last_undegraded":"2024-09-13T13:05:47.779833+0000","last_fullsized":"2024-09-13T13:05:47.779833+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.607646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.607646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":224,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796244+0000","last_change":"2024-09-13T13:05:38.520071+0000","last_active":"2024-09-13T13:05:47.796244+0000","last_peered":"2024-09-13T13:05:47.796244+0000","last_clean":"2024-09-13T13:05:47.796244+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:05:47.796244+0000","last_undegraded":"2024-09-13T13:05:47.796244+0000","last_fullsized":"2024-09-13T13:05:47.796244+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:38.520006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:38.520006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:55.423473+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":223,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780844+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:05:47.780844+0000","last_peered":"2024-09-13T13:05:47.780844+0000","last_clean":"2024-09-13T13:05:47.780844+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:05:47.780844+0000","last_undegraded":"2024-09-13T13:05:47.780844+0000","last_fullsized":"2024-09-13T13:05:47.780844+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:15.096451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":200,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781984+0000","last_change":"2024-09-13T13:05:10.383826+0000","last_active":"2024-09-13T13:05:47.781984+0000","last_peered":"2024-09-13T13:05:47.781984+0000","last_clean":"2024-09-13T13:05:47.781984+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:05:47.781984+0000","last_undegraded":"2024-09-13T13:05:47.781984+0000","last_fullsized":"2024-09-13T13:05:47.781984+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.383756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.383756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:33.805546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":188,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796011+0000","last_change":"2024-09-13T13:01:08.213608+0000","last_active":"2024-09-13T13:05:47.796011+0000","last_peered":"2024-09-13T13:05:47.796011+0000","last_clean":"2024-09-13T13:05:47.796011+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:05:47.796011+0000","last_undegraded":"2024-09-13T13:05:47.796011+0000","last_fullsized":"2024-09-13T13:05:47.796011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.558201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":210,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781282+0000","last_change":"2024-09-13T13:04:54.016136+0000","last_active":"2024-09-13T13:05:47.781282+0000","last_peered":"2024-09-13T13:05:47.781282+0000","last_clean":"2024-09-13T13:05:47.781282+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:05:47.781282+0000","last_undegraded":"2024-09-13T13:05:47.781282+0000","last_fullsized":"2024-09-13T13:05:47.781282+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:04:54.016064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.016064+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.987760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781063+0000","last_change":"2024-09-13T13:02:32.590355+0000","last_active":"2024-09-13T13:05:47.781063+0000","last_peered":"2024-09-13T13:05:47.781063+0000","last_clean":"2024-09-13T13:05:47.781063+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:05:47.781063+0000","last_undegraded":"2024-09-13T13:05:47.781063+0000","last_fullsized":"2024-09-13T13:05:47.781063+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:32.590282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.590282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":246,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.801085+0000","last_change":"2024-09-13T13:05:33.323411+0000","last_active":"2024-09-13T13:05:47.801085+0000","last_peered":"2024-09-13T13:05:47.801085+0000","last_clean":"2024-09-13T13:05:47.801085+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:05:47.801085+0000","last_undegraded":"2024-09-13T13:05:47.801085+0000","last_fullsized":"2024-09-13T13:05:47.801085+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:05:33.323316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.323316+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:36.730491+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":201,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.799199+0000","last_change":"2024-09-13T13:02:45.202865+0000","last_active":"2024-09-13T13:05:47.799199+0000","last_peered":"2024-09-13T13:05:47.799199+0000","last_clean":"2024-09-13T13:05:47.799199+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:05:47.799199+0000","last_undegraded":"2024-09-13T13:05:47.799199+0000","last_fullsized":"2024-09-13T13:05:47.799199+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.202802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.202802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:46.737187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.794257+0000","last_change":"2024-09-13T13:04:03.384955+0000","last_active":"2024-09-13T13:05:47.794257+0000","last_peered":"2024-09-13T13:05:47.794257+0000","last_clean":"2024-09-13T13:05:47.794257+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:05:47.794257+0000","last_undegraded":"2024-09-13T13:05:47.794257+0000","last_fullsized":"2024-09-13T13:05:47.794257+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.384901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.384901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780747+0000","last_change":"2024-09-13T13:04:57.103213+0000","last_active":"2024-09-13T13:05:47.780747+0000","last_peered":"2024-09-13T13:05:47.780747+0000","last_clean":"2024-09-13T13:05:47.780747+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:05:47.780747+0000","last_undegraded":"2024-09-13T13:05:47.780747+0000","last_fullsized":"2024-09-13T13:05:47.780747+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.103167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.103167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.050420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":219,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.788821+0000","last_change":"2024-09-13T13:01:04.195126+0000","last_active":"2024-09-13T13:05:47.788821+0000","last_peered":"2024-09-13T13:05:47.788821+0000","last_clean":"2024-09-13T13:05:47.788821+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:05:47.788821+0000","last_undegraded":"2024-09-13T13:05:47.788821+0000","last_fullsized":"2024-09-13T13:05:47.788821+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781532+0000","last_change":"2024-09-13T13:05:39.185953+0000","last_active":"2024-09-13T13:05:47.781532+0000","last_peered":"2024-09-13T13:05:47.781532+0000","last_clean":"2024-09-13T13:05:47.781532+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:05:47.781532+0000","last_undegraded":"2024-09-13T13:05:47.781532+0000","last_fullsized":"2024-09-13T13:05:47.781532+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:39.185892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:39.185892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:59.347239+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782548+0000","last_change":"2024-09-13T13:05:00.457611+0000","last_active":"2024-09-13T13:05:47.782548+0000","last_peered":"2024-09-13T13:05:47.782548+0000","last_clean":"2024-09-13T13:05:47.782548+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:05:47.782548+0000","last_undegraded":"2024-09-13T13:05:47.782548+0000","last_fullsized":"2024-09-13T13:05:47.782548+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:05:00.457562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:00.457562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:25.611525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":210,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.801962+0000","last_change":"2024-09-13T13:03:12.276224+0000","last_active":"2024-09-13T13:05:47.801962+0000","last_peered":"2024-09-13T13:05:47.801962+0000","last_clean":"2024-09-13T13:05:47.801962+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:05:47.801962+0000","last_undegraded":"2024-09-13T13:05:47.801962+0000","last_fullsized":"2024-09-13T13:05:47.801962+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:12.276136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.276136+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:47.787107+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":220,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.787923+0000","last_change":"2024-09-13T13:02:34.593511+0000","last_active":"2024-09-13T13:05:47.787923+0000","last_peered":"2024-09-13T13:05:47.787923+0000","last_clean":"2024-09-13T13:05:47.787923+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:05:47.787923+0000","last_undegraded":"2024-09-13T13:05:47.787923+0000","last_fullsized":"2024-09-13T13:05:47.787923+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:34.593431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:34.593431+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":211,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T13:05:42.779099+0000","last_change":"2024-09-13T13:04:02.765069+0000","last_active":"2024-09-13T13:05:42.779099+0000","last_peered":"2024-09-13T13:05:42.779099+0000","last_clean":"2024-09-13T13:05:42.779099+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:05:42.779099+0000","last_undegraded":"2024-09-13T13:05:42.779099+0000","last_fullsized":"2024-09-13T13:05:42.779099+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:46.604669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.604669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"152'13544","reported_seq":19465,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T13:05:45.864228+0000","last_change":"2024-09-13T13:04:49.780269+0000","last_active":"2024-09-13T13:05:45.864228+0000","last_peered":"2024-09-13T13:05:45.864228+0000","last_clean":"2024-09-13T13:05:45.864228+0000","last_became_active":"2024-09-13T13:04:49.765876+0000","last_became_peered":"2024-09-13T13:04:49.765876+0000","last_unstale":"2024-09-13T13:05:45.864228+0000","last_undegraded":"2024-09-13T13:05:45.864228+0000","last_fullsized":"2024-09-13T13:05:45.864228+0000","mapping_epoch":136,"log_start":"115'7248","ondisk_log_start":"115'7248","created":84,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":6296,"log_dups_size":0,"ondisk_log_size":6296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96600064,"num_objects":1474,"num_object_clones":0,"num_object_copies":4422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13544,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2909,"num_bytes_recovered":147193856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":239,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796179+0000","last_change":"2024-09-13T13:05:25.691233+0000","last_active":"2024-09-13T13:05:47.796179+0000","last_peered":"2024-09-13T13:05:47.796179+0000","last_clean":"2024-09-13T13:05:47.796179+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:05:47.796179+0000","last_undegraded":"2024-09-13T13:05:47.796179+0000","last_fullsized":"2024-09-13T13:05:47.796179+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:05:25.689487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:25.689487+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:53.347573+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":217,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.791619+0000","last_change":"2024-09-13T13:03:42.443399+0000","last_active":"2024-09-13T13:05:47.791619+0000","last_peered":"2024-09-13T13:05:47.791619+0000","last_clean":"2024-09-13T13:05:47.791619+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:05:47.791619+0000","last_undegraded":"2024-09-13T13:05:47.791619+0000","last_fullsized":"2024-09-13T13:05:47.791619+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:42.443339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:42.443339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":268,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781372+0000","last_change":"2024-09-13T13:02:32.013587+0000","last_active":"2024-09-13T13:05:47.781372+0000","last_peered":"2024-09-13T13:05:47.781372+0000","last_clean":"2024-09-13T13:05:47.781372+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:05:47.781372+0000","last_undegraded":"2024-09-13T13:05:47.781372+0000","last_fullsized":"2024-09-13T13:05:47.781372+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:32.013505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.013505+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:14.086911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796276+0000","last_change":"2024-09-13T13:04:48.684906+0000","last_active":"2024-09-13T13:05:47.796276+0000","last_peered":"2024-09-13T13:05:47.796276+0000","last_clean":"2024-09-13T13:05:47.796276+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:05:47.796276+0000","last_undegraded":"2024-09-13T13:05:47.796276+0000","last_fullsized":"2024-09-13T13:05:47.796276+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:48.684864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.684864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.491555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"154'13759","reported_seq":13988,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:48.245470+0000","last_change":"2024-09-13T13:03:41.936543+0000","last_active":"2024-09-13T13:05:48.245470+0000","last_peered":"2024-09-13T13:05:48.245470+0000","last_clean":"2024-09-13T13:05:48.245470+0000","last_became_active":"2024-09-13T13:02:12.369723+0000","last_became_peered":"2024-09-13T13:02:12.369723+0000","last_unstale":"2024-09-13T13:05:48.245470+0000","last_undegraded":"2024-09-13T13:05:48.245470+0000","last_fullsized":"2024-09-13T13:05:48.245470+0000","mapping_epoch":84,"log_start":"125'9100","ondisk_log_start":"125'9100","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6909","last_scrub_stamp":"2024-09-13T13:03:41.936218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:03:41.936218+0000","objects_scrubbed":6836,"log_size":4659,"log_dups_size":0,"ondisk_log_size":4659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89718784,"num_objects":1369,"num_object_clones":0,"num_object_copies":4107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13759,"num_write_kb":484096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":237,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.791403+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:05:47.791403+0000","last_peered":"2024-09-13T13:05:47.791403+0000","last_clean":"2024-09-13T13:05:47.791403+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:05:47.791403+0000","last_undegraded":"2024-09-13T13:05:47.791403+0000","last_fullsized":"2024-09-13T13:05:47.791403+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":215,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.794642+0000","last_change":"2024-09-13T13:03:18.516327+0000","last_active":"2024-09-13T13:05:47.794642+0000","last_peered":"2024-09-13T13:05:47.794642+0000","last_clean":"2024-09-13T13:05:47.794642+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:05:47.794642+0000","last_undegraded":"2024-09-13T13:05:47.794642+0000","last_fullsized":"2024-09-13T13:05:47.794642+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:18.516251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.516251+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":210,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780891+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:05:47.780891+0000","last_peered":"2024-09-13T13:05:47.780891+0000","last_clean":"2024-09-13T13:05:47.780891+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:05:47.780891+0000","last_undegraded":"2024-09-13T13:05:47.780891+0000","last_fullsized":"2024-09-13T13:05:47.780891+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:40.320454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":198,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780621+0000","last_change":"2024-09-13T13:04:53.006515+0000","last_active":"2024-09-13T13:05:47.780621+0000","last_peered":"2024-09-13T13:05:47.780621+0000","last_clean":"2024-09-13T13:05:47.780621+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:05:47.780621+0000","last_undegraded":"2024-09-13T13:05:47.780621+0000","last_fullsized":"2024-09-13T13:05:47.780621+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:53.006454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.006454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.531241+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"154'13702","reported_seq":13859,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:48.272861+0000","last_change":"2024-09-13T13:03:45.761068+0000","last_active":"2024-09-13T13:05:48.272861+0000","last_peered":"2024-09-13T13:05:48.272861+0000","last_clean":"2024-09-13T13:05:48.272861+0000","last_became_active":"2024-09-13T13:02:12.373492+0000","last_became_peered":"2024-09-13T13:02:12.373492+0000","last_unstale":"2024-09-13T13:05:48.272861+0000","last_undegraded":"2024-09-13T13:05:48.272861+0000","last_fullsized":"2024-09-13T13:05:48.272861+0000","mapping_epoch":84,"log_start":"125'9000","ondisk_log_start":"125'9000","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":299,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88735744,"num_objects":1354,"num_object_clones":0,"num_object_copies":4062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13702,"num_write_kb":481792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":246,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780896+0000","last_change":"2024-09-13T13:05:33.178442+0000","last_active":"2024-09-13T13:05:47.780896+0000","last_peered":"2024-09-13T13:05:47.780896+0000","last_clean":"2024-09-13T13:05:47.780896+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:05:47.780896+0000","last_undegraded":"2024-09-13T13:05:47.780896+0000","last_fullsized":"2024-09-13T13:05:47.780896+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:05:33.178370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.178370+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:57.356161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":210,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.790890+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:05:47.790890+0000","last_peered":"2024-09-13T13:05:47.790890+0000","last_clean":"2024-09-13T13:05:47.790890+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:05:47.790890+0000","last_undegraded":"2024-09-13T13:05:47.790890+0000","last_fullsized":"2024-09-13T13:05:47.790890+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":259,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.791163+0000","last_change":"2024-09-13T13:01:06.211211+0000","last_active":"2024-09-13T13:05:47.791163+0000","last_peered":"2024-09-13T13:05:47.791163+0000","last_clean":"2024-09-13T13:05:47.791163+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:05:47.791163+0000","last_undegraded":"2024-09-13T13:05:47.791163+0000","last_fullsized":"2024-09-13T13:05:47.791163+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:47.764924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.783010+0000","last_change":"2024-09-13T13:04:57.439116+0000","last_active":"2024-09-13T13:05:47.783010+0000","last_peered":"2024-09-13T13:05:47.783010+0000","last_clean":"2024-09-13T13:05:47.783010+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:05:47.783010+0000","last_undegraded":"2024-09-13T13:05:47.783010+0000","last_fullsized":"2024-09-13T13:05:47.783010+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.439053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.439053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:04.992285+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"154'13715","reported_seq":19713,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.908868+0000","last_change":"2024-09-13T13:05:05.014474+0000","last_active":"2024-09-13T13:05:47.908868+0000","last_peered":"2024-09-13T13:05:47.908868+0000","last_clean":"2024-09-13T13:05:47.908868+0000","last_became_active":"2024-09-13T13:04:25.922485+0000","last_became_peered":"2024-09-13T13:04:25.922485+0000","last_unstale":"2024-09-13T13:05:47.908868+0000","last_undegraded":"2024-09-13T13:05:47.908868+0000","last_fullsized":"2024-09-13T13:05:47.908868+0000","mapping_epoch":128,"log_start":"124'8900","ondisk_log_start":"124'8900","created":84,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'11284","last_scrub_stamp":"2024-09-13T13:05:05.014171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:05.014171+0000","objects_scrubbed":3783,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:18.664629+0000","scrub_duration":934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86835200,"num_objects":1325,"num_object_clones":0,"num_object_copies":3975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13715,"num_write_kb":481280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2891,"num_bytes_recovered":148570112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":213,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.801148+0000","last_change":"2024-09-13T13:01:04.199011+0000","last_active":"2024-09-13T13:05:47.801148+0000","last_peered":"2024-09-13T13:05:47.801148+0000","last_clean":"2024-09-13T13:05:47.801148+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:05:47.801148+0000","last_undegraded":"2024-09-13T13:05:47.801148+0000","last_fullsized":"2024-09-13T13:05:47.801148+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.733108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796589+0000","last_change":"2024-09-13T13:05:04.743102+0000","last_active":"2024-09-13T13:05:47.796589+0000","last_peered":"2024-09-13T13:05:47.796589+0000","last_clean":"2024-09-13T13:05:47.796589+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:05:47.796589+0000","last_undegraded":"2024-09-13T13:05:47.796589+0000","last_fullsized":"2024-09-13T13:05:47.796589+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:04.743025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:04.743025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:32.082984+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":273,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780677+0000","last_change":"2024-09-13T13:04:02.764350+0000","last_active":"2024-09-13T13:05:47.780677+0000","last_peered":"2024-09-13T13:05:47.780677+0000","last_clean":"2024-09-13T13:05:47.780677+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:05:47.780677+0000","last_undegraded":"2024-09-13T13:05:47.780677+0000","last_fullsized":"2024-09-13T13:05:47.780677+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780038+0000","last_change":"2024-09-13T13:02:43.558487+0000","last_active":"2024-09-13T13:05:47.780038+0000","last_peered":"2024-09-13T13:05:47.780038+0000","last_clean":"2024-09-13T13:05:47.780038+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:05:47.780038+0000","last_undegraded":"2024-09-13T13:05:47.780038+0000","last_fullsized":"2024-09-13T13:05:47.780038+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:43.558418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.558418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"154'13577","reported_seq":13728,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.901610+0000","last_change":"2024-09-13T13:02:12.379536+0000","last_active":"2024-09-13T13:05:47.901610+0000","last_peered":"2024-09-13T13:05:47.901610+0000","last_clean":"2024-09-13T13:05:47.901610+0000","last_became_active":"2024-09-13T13:02:12.379226+0000","last_became_peered":"2024-09-13T13:02:12.379226+0000","last_unstale":"2024-09-13T13:05:47.901610+0000","last_undegraded":"2024-09-13T13:05:47.901610+0000","last_fullsized":"2024-09-13T13:05:47.901610+0000","mapping_epoch":84,"log_start":"116'7800","ondisk_log_start":"116'7800","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5777,"log_dups_size":0,"ondisk_log_size":5777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.567939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89325568,"num_objects":1363,"num_object_clones":0,"num_object_copies":4089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13577,"num_write_kb":478080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":241,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780857+0000","last_change":"2024-09-13T13:02:29.532879+0000","last_active":"2024-09-13T13:05:47.780857+0000","last_peered":"2024-09-13T13:05:47.780857+0000","last_clean":"2024-09-13T13:05:47.780857+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:05:47.780857+0000","last_undegraded":"2024-09-13T13:05:47.780857+0000","last_fullsized":"2024-09-13T13:05:47.780857+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:29.532779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.532779+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":202,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.803304+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:05:47.803304+0000","last_peered":"2024-09-13T13:05:47.803304+0000","last_clean":"2024-09-13T13:05:47.803304+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:05:47.803304+0000","last_undegraded":"2024-09-13T13:05:47.803304+0000","last_fullsized":"2024-09-13T13:05:47.803304+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:20.738953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":158,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T13:05:42.778599+0000","last_change":"2024-09-13T13:04:02.765838+0000","last_active":"2024-09-13T13:05:42.778599+0000","last_peered":"2024-09-13T13:05:42.778599+0000","last_clean":"2024-09-13T13:05:42.778599+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:05:42.778599+0000","last_undegraded":"2024-09-13T13:05:42.778599+0000","last_fullsized":"2024-09-13T13:05:42.778599+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.563708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.563708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":217,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.801541+0000","last_change":"2024-09-13T13:05:30.829125+0000","last_active":"2024-09-13T13:05:47.801541+0000","last_peered":"2024-09-13T13:05:47.801541+0000","last_clean":"2024-09-13T13:05:47.801541+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:05:47.801541+0000","last_undegraded":"2024-09-13T13:05:47.801541+0000","last_fullsized":"2024-09-13T13:05:47.801541+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:30.828810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:30.828810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:43.724154+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"152'13683","reported_seq":19733,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T13:05:45.881398+0000","last_change":"2024-09-13T13:05:42.806200+0000","last_active":"2024-09-13T13:05:45.881398+0000","last_peered":"2024-09-13T13:05:45.881398+0000","last_clean":"2024-09-13T13:05:45.881398+0000","last_became_active":"2024-09-13T13:05:42.780925+0000","last_became_peered":"2024-09-13T13:05:42.780925+0000","last_unstale":"2024-09-13T13:05:45.881398+0000","last_undegraded":"2024-09-13T13:05:45.881398+0000","last_fullsized":"2024-09-13T13:05:45.881398+0000","mapping_epoch":151,"log_start":"116'7900","ondisk_log_start":"116'7900","created":84,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5783,"log_dups_size":0,"ondisk_log_size":5783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97583104,"num_objects":1489,"num_object_clones":0,"num_object_copies":4467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13683,"num_write_kb":485504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2944,"num_bytes_recovered":148635648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":320,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.803207+0000","last_change":"2024-09-13T13:05:43.787063+0000","last_active":"2024-09-13T13:05:47.803207+0000","last_peered":"2024-09-13T13:05:47.803207+0000","last_clean":"2024-09-13T13:05:47.803207+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:05:47.803207+0000","last_undegraded":"2024-09-13T13:05:47.803207+0000","last_fullsized":"2024-09-13T13:05:47.803207+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:05:43.786273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:43.786273+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:05.440008+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":229,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781077+0000","last_change":"2024-09-13T13:05:45.178086+0000","last_active":"2024-09-13T13:05:47.781077+0000","last_peered":"2024-09-13T13:05:47.781077+0000","last_clean":"2024-09-13T13:05:47.781077+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:05:47.781077+0000","last_undegraded":"2024-09-13T13:05:47.781077+0000","last_fullsized":"2024-09-13T13:05:47.781077+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:45.178025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.178025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:02.693302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":228,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.797875+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:05:47.797875+0000","last_peered":"2024-09-13T13:05:47.797875+0000","last_clean":"2024-09-13T13:05:47.797875+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:05:47.797875+0000","last_undegraded":"2024-09-13T13:05:47.797875+0000","last_fullsized":"2024-09-13T13:05:47.797875+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:26.941315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":201,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.784681+0000","last_change":"2024-09-13T13:02:35.566666+0000","last_active":"2024-09-13T13:05:47.784681+0000","last_peered":"2024-09-13T13:05:47.784681+0000","last_clean":"2024-09-13T13:05:47.784681+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:05:47.784681+0000","last_undegraded":"2024-09-13T13:05:47.784681+0000","last_fullsized":"2024-09-13T13:05:47.784681+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:35.566602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:35.566602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.790053+0000","last_change":"2024-09-13T13:04:01.406226+0000","last_active":"2024-09-13T13:05:47.790053+0000","last_peered":"2024-09-13T13:05:47.790053+0000","last_clean":"2024-09-13T13:05:47.790053+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:05:47.790053+0000","last_undegraded":"2024-09-13T13:05:47.790053+0000","last_fullsized":"2024-09-13T13:05:47.790053+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:01.406167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:01.406167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"154'13798","reported_seq":13950,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:48.481226+0000","last_change":"2024-09-13T13:02:12.380398+0000","last_active":"2024-09-13T13:05:48.481226+0000","last_peered":"2024-09-13T13:05:48.481226+0000","last_clean":"2024-09-13T13:05:48.481226+0000","last_became_active":"2024-09-13T13:02:12.379331+0000","last_became_peered":"2024-09-13T13:02:12.379331+0000","last_unstale":"2024-09-13T13:05:48.481226+0000","last_undegraded":"2024-09-13T13:05:48.481226+0000","last_fullsized":"2024-09-13T13:05:48.481226+0000","mapping_epoch":84,"log_start":"120'8200","ondisk_log_start":"120'8200","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84410368,"num_objects":1288,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13798,"num_write_kb":482752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":233,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796271+0000","last_change":"2024-09-13T13:03:14.271569+0000","last_active":"2024-09-13T13:05:47.796271+0000","last_peered":"2024-09-13T13:05:47.796271+0000","last_clean":"2024-09-13T13:05:47.796271+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:05:47.796271+0000","last_undegraded":"2024-09-13T13:05:47.796271+0000","last_fullsized":"2024-09-13T13:05:47.796271+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:03:14.271480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.271480+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":229,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.804103+0000","last_change":"2024-09-13T13:05:36.692159+0000","last_active":"2024-09-13T13:05:47.804103+0000","last_peered":"2024-09-13T13:05:47.804103+0000","last_clean":"2024-09-13T13:05:47.804103+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:05:47.804103+0000","last_undegraded":"2024-09-13T13:05:47.804103+0000","last_fullsized":"2024-09-13T13:05:47.804103+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:36.692092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:36.692092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:00.057550+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":225,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.800347+0000","last_change":"2024-09-13T13:05:34.321160+0000","last_active":"2024-09-13T13:05:47.800347+0000","last_peered":"2024-09-13T13:05:47.800347+0000","last_clean":"2024-09-13T13:05:47.800347+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:05:47.800347+0000","last_undegraded":"2024-09-13T13:05:47.800347+0000","last_fullsized":"2024-09-13T13:05:47.800347+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:05:34.321081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.321081+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:37.239957+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":202,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.801192+0000","last_change":"2024-09-13T13:03:14.282866+0000","last_active":"2024-09-13T13:05:47.801192+0000","last_peered":"2024-09-13T13:05:47.801192+0000","last_clean":"2024-09-13T13:05:47.801192+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:05:47.801192+0000","last_undegraded":"2024-09-13T13:05:47.801192+0000","last_fullsized":"2024-09-13T13:05:47.801192+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.349860+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782563+0000","last_change":"2024-09-13T13:04:02.751582+0000","last_active":"2024-09-13T13:05:47.782563+0000","last_peered":"2024-09-13T13:05:47.782563+0000","last_clean":"2024-09-13T13:05:47.782563+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:05:47.782563+0000","last_undegraded":"2024-09-13T13:05:47.782563+0000","last_fullsized":"2024-09-13T13:05:47.782563+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.994729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.994729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.398308+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"154'14019","reported_seq":14275,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.924615+0000","last_change":"2024-09-13T13:05:35.785111+0000","last_active":"2024-09-13T13:05:47.924615+0000","last_peered":"2024-09-13T13:05:47.924615+0000","last_clean":"2024-09-13T13:05:47.924615+0000","last_became_active":"2024-09-13T13:02:12.379779+0000","last_became_peered":"2024-09-13T13:02:12.379779+0000","last_unstale":"2024-09-13T13:05:47.924615+0000","last_undegraded":"2024-09-13T13:05:47.924615+0000","last_fullsized":"2024-09-13T13:05:47.924615+0000","mapping_epoch":84,"log_start":"122'8800","ondisk_log_start":"122'8800","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'13287","last_scrub_stamp":"2024-09-13T13:05:35.785022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.785022+0000","objects_scrubbed":2122,"log_size":5219,"log_dups_size":0,"ondisk_log_size":5219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:52.011396+0000","scrub_duration":635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89587712,"num_objects":1367,"num_object_clones":0,"num_object_copies":4101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14019,"num_write_kb":492352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":223,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.788025+0000","last_change":"2024-09-13T13:03:18.514436+0000","last_active":"2024-09-13T13:05:47.788025+0000","last_peered":"2024-09-13T13:05:47.788025+0000","last_clean":"2024-09-13T13:05:47.788025+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:05:47.788025+0000","last_undegraded":"2024-09-13T13:05:47.788025+0000","last_fullsized":"2024-09-13T13:05:47.788025+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:03:18.514360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.514360+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781186+0000","last_change":"2024-09-13T13:02:37.616313+0000","last_active":"2024-09-13T13:05:47.781186+0000","last_peered":"2024-09-13T13:05:47.781186+0000","last_clean":"2024-09-13T13:05:47.781186+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:05:47.781186+0000","last_undegraded":"2024-09-13T13:05:47.781186+0000","last_fullsized":"2024-09-13T13:05:47.781186+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:37.616224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.616224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":498,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782398+0000","last_change":"2024-09-13T13:04:56.430717+0000","last_active":"2024-09-13T13:05:47.782398+0000","last_peered":"2024-09-13T13:05:47.782398+0000","last_clean":"2024-09-13T13:05:47.782398+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:05:47.782398+0000","last_undegraded":"2024-09-13T13:05:47.782398+0000","last_fullsized":"2024-09-13T13:05:47.782398+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:04:56.430670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.430670+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:59.639039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":280,"num_read_kb":275,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":201,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796777+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:05:47.796777+0000","last_peered":"2024-09-13T13:05:47.796777+0000","last_clean":"2024-09-13T13:05:47.796777+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:05:47.796777+0000","last_undegraded":"2024-09-13T13:05:47.796777+0000","last_fullsized":"2024-09-13T13:05:47.796777+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:19.946825+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":206,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.795941+0000","last_change":"2024-09-13T13:05:41.753575+0000","last_active":"2024-09-13T13:05:47.795941+0000","last_peered":"2024-09-13T13:05:47.795941+0000","last_clean":"2024-09-13T13:05:47.795941+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:05:47.795941+0000","last_undegraded":"2024-09-13T13:05:47.795941+0000","last_fullsized":"2024-09-13T13:05:47.795941+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:41.753502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:41.753502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:54.892291+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"154'13791","reported_seq":13943,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:48.481565+0000","last_change":"2024-09-13T13:02:12.379556+0000","last_active":"2024-09-13T13:05:48.481565+0000","last_peered":"2024-09-13T13:05:48.481565+0000","last_clean":"2024-09-13T13:05:48.481565+0000","last_became_active":"2024-09-13T13:02:12.379255+0000","last_became_peered":"2024-09-13T13:02:12.379255+0000","last_unstale":"2024-09-13T13:05:48.481565+0000","last_undegraded":"2024-09-13T13:05:48.481565+0000","last_fullsized":"2024-09-13T13:05:48.481565+0000","mapping_epoch":84,"log_start":"117'8200","ondisk_log_start":"117'8200","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5591,"log_dups_size":0,"ondisk_log_size":5591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87490560,"num_objects":1335,"num_object_clones":0,"num_object_copies":4005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13791,"num_write_kb":484032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":147,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T13:05:42.778874+0000","last_change":"2024-09-13T13:04:02.786042+0000","last_active":"2024-09-13T13:05:42.778874+0000","last_peered":"2024-09-13T13:05:42.778874+0000","last_clean":"2024-09-13T13:05:42.778874+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:05:42.778874+0000","last_undegraded":"2024-09-13T13:05:42.778874+0000","last_fullsized":"2024-09-13T13:05:42.778874+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":220,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.794577+0000","last_change":"2024-09-13T13:04:02.765370+0000","last_active":"2024-09-13T13:05:47.794577+0000","last_peered":"2024-09-13T13:05:47.794577+0000","last_clean":"2024-09-13T13:05:47.794577+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:05:47.794577+0000","last_undegraded":"2024-09-13T13:05:47.794577+0000","last_fullsized":"2024-09-13T13:05:47.794577+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:04.228002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:04.228002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":265,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782584+0000","last_change":"2024-09-13T13:05:28.373960+0000","last_active":"2024-09-13T13:05:47.782584+0000","last_peered":"2024-09-13T13:05:47.782584+0000","last_clean":"2024-09-13T13:05:47.782584+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:05:47.782584+0000","last_undegraded":"2024-09-13T13:05:47.782584+0000","last_fullsized":"2024-09-13T13:05:47.782584+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:05:28.373902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:05:28.373902+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:33.407719+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":201,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782690+0000","last_change":"2024-09-13T13:04:51.395273+0000","last_active":"2024-09-13T13:05:47.782690+0000","last_peered":"2024-09-13T13:05:47.782690+0000","last_clean":"2024-09-13T13:05:47.782690+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:05:47.782690+0000","last_undegraded":"2024-09-13T13:05:47.782690+0000","last_fullsized":"2024-09-13T13:05:47.782690+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:51.395204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:51.395204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:56.927380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":205,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796665+0000","last_change":"2024-09-13T13:05:31.564860+0000","last_active":"2024-09-13T13:05:47.796665+0000","last_peered":"2024-09-13T13:05:47.796665+0000","last_clean":"2024-09-13T13:05:47.796665+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:05:47.796665+0000","last_undegraded":"2024-09-13T13:05:47.796665+0000","last_fullsized":"2024-09-13T13:05:47.796665+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:31.564783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:31.564783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:41.885010+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"152'13760","reported_seq":19942,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T13:05:45.820768+0000","last_change":"2024-09-13T13:05:15.349567+0000","last_active":"2024-09-13T13:05:45.820768+0000","last_peered":"2024-09-13T13:05:45.820768+0000","last_clean":"2024-09-13T13:05:45.820768+0000","last_became_active":"2024-09-13T13:05:15.326923+0000","last_became_peered":"2024-09-13T13:05:15.326923+0000","last_unstale":"2024-09-13T13:05:45.820768+0000","last_undegraded":"2024-09-13T13:05:45.820768+0000","last_fullsized":"2024-09-13T13:05:45.820768+0000","mapping_epoch":143,"log_start":"115'7485","ondisk_log_start":"115'7485","created":84,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100991000,"num_objects":1542,"num_object_clones":0,"num_object_copies":4626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13760,"num_write_kb":489601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3008,"num_bytes_recovered":155058200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":236,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781461+0000","last_change":"2024-09-13T13:04:02.832992+0000","last_active":"2024-09-13T13:05:47.781461+0000","last_peered":"2024-09-13T13:05:47.781461+0000","last_clean":"2024-09-13T13:05:47.781461+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:05:47.781461+0000","last_undegraded":"2024-09-13T13:05:47.781461+0000","last_fullsized":"2024-09-13T13:05:47.781461+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:04:02.832935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.832935+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:47.117698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":210,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781161+0000","last_change":"2024-09-13T13:02:45.015634+0000","last_active":"2024-09-13T13:05:47.781161+0000","last_peered":"2024-09-13T13:05:47.781161+0000","last_clean":"2024-09-13T13:05:47.781161+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:05:47.781161+0000","last_undegraded":"2024-09-13T13:05:47.781161+0000","last_fullsized":"2024-09-13T13:05:47.781161+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.015577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.015577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.095405+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":212,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796531+0000","last_change":"2024-09-13T13:04:47.680651+0000","last_active":"2024-09-13T13:05:47.796531+0000","last_peered":"2024-09-13T13:05:47.796531+0000","last_clean":"2024-09-13T13:05:47.796531+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:05:47.796531+0000","last_undegraded":"2024-09-13T13:05:47.796531+0000","last_fullsized":"2024-09-13T13:05:47.796531+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.680579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.680579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:07.566204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.788756+0000","last_change":"2024-09-13T13:04:02.757378+0000","last_active":"2024-09-13T13:05:47.788756+0000","last_peered":"2024-09-13T13:05:47.788756+0000","last_clean":"2024-09-13T13:05:47.788756+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:05:47.788756+0000","last_undegraded":"2024-09-13T13:05:47.788756+0000","last_fullsized":"2024-09-13T13:05:47.788756+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.213470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.213470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"154'13364","reported_seq":13641,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:48.056115+0000","last_change":"2024-09-13T13:05:03.355522+0000","last_active":"2024-09-13T13:05:48.056115+0000","last_peered":"2024-09-13T13:05:48.056115+0000","last_clean":"2024-09-13T13:05:48.056115+0000","last_became_active":"2024-09-13T13:02:12.375509+0000","last_became_peered":"2024-09-13T13:02:12.375509+0000","last_unstale":"2024-09-13T13:05:48.056115+0000","last_undegraded":"2024-09-13T13:05:48.056115+0000","last_fullsized":"2024-09-13T13:05:48.056115+0000","mapping_epoch":84,"log_start":"111'6500","ondisk_log_start":"111'6500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'10840","last_scrub_stamp":"2024-09-13T13:05:03.354789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:03.354789+0000","objects_scrubbed":3912,"log_size":6864,"log_dups_size":0,"ondisk_log_size":6864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.025217+0000","scrub_duration":2101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87425024,"num_objects":1334,"num_object_clones":0,"num_object_copies":4002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13364,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":242,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781738+0000","last_change":"2024-09-13T13:05:34.166464+0000","last_active":"2024-09-13T13:05:47.781738+0000","last_peered":"2024-09-13T13:05:47.781738+0000","last_clean":"2024-09-13T13:05:47.781738+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:05:47.781738+0000","last_undegraded":"2024-09-13T13:05:47.781738+0000","last_fullsized":"2024-09-13T13:05:47.781738+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:40.189820+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":182,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T13:05:44.119947+0000","last_change":"2024-09-13T13:05:44.119947+0000","last_active":"2024-09-13T13:05:44.119947+0000","last_peered":"2024-09-13T13:05:44.119947+0000","last_clean":"2024-09-13T13:05:44.119947+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:05:44.119947+0000","last_undegraded":"2024-09-13T13:05:44.119947+0000","last_fullsized":"2024-09-13T13:05:44.119947+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:44.119890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.119890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:50.534493+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":194,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796055+0000","last_change":"2024-09-13T13:01:06.208593+0000","last_active":"2024-09-13T13:05:47.796055+0000","last_peered":"2024-09-13T13:05:47.796055+0000","last_clean":"2024-09-13T13:05:47.796055+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:05:47.796055+0000","last_undegraded":"2024-09-13T13:05:47.796055+0000","last_fullsized":"2024-09-13T13:05:47.796055+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:22.718840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":206,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.800707+0000","last_change":"2024-09-13T13:05:35.310768+0000","last_active":"2024-09-13T13:05:47.800707+0000","last_peered":"2024-09-13T13:05:47.800707+0000","last_clean":"2024-09-13T13:05:47.800707+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:05:47.800707+0000","last_undegraded":"2024-09-13T13:05:47.800707+0000","last_fullsized":"2024-09-13T13:05:47.800707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:35.310700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.310700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:56.914781+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"154'13826","reported_seq":14055,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:48.486019+0000","last_change":"2024-09-13T13:04:02.904392+0000","last_active":"2024-09-13T13:05:48.486019+0000","last_peered":"2024-09-13T13:05:48.486019+0000","last_clean":"2024-09-13T13:05:48.486019+0000","last_became_active":"2024-09-13T13:02:12.380229+0000","last_became_peered":"2024-09-13T13:02:12.380229+0000","last_unstale":"2024-09-13T13:05:48.486019+0000","last_undegraded":"2024-09-13T13:05:48.486019+0000","last_fullsized":"2024-09-13T13:05:48.486019+0000","mapping_epoch":84,"log_start":"120'8200","ondisk_log_start":"120'8200","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8189","last_scrub_stamp":"2024-09-13T13:04:02.904297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.904297+0000","objects_scrubbed":6969,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86900736,"num_objects":1326,"num_object_clones":0,"num_object_copies":3978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13826,"num_write_kb":484864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":232,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781552+0000","last_change":"2024-09-13T13:04:59.091852+0000","last_active":"2024-09-13T13:05:47.781552+0000","last_peered":"2024-09-13T13:05:47.781552+0000","last_clean":"2024-09-13T13:05:47.781552+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:05:47.781552+0000","last_undegraded":"2024-09-13T13:05:47.781552+0000","last_fullsized":"2024-09-13T13:05:47.781552+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:04:59.091751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.091751+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.396141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":217,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.789556+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:05:47.789556+0000","last_peered":"2024-09-13T13:05:47.789556+0000","last_clean":"2024-09-13T13:05:47.789556+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:05:47.789556+0000","last_undegraded":"2024-09-13T13:05:47.789556+0000","last_fullsized":"2024-09-13T13:05:47.789556+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":276,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.800614+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:05:47.800614+0000","last_peered":"2024-09-13T13:05:47.800614+0000","last_clean":"2024-09-13T13:05:47.800614+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:05:47.800614+0000","last_undegraded":"2024-09-13T13:05:47.800614+0000","last_fullsized":"2024-09-13T13:05:47.800614+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:57.041771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":198,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781195+0000","last_change":"2024-09-13T13:02:33.997780+0000","last_active":"2024-09-13T13:05:47.781195+0000","last_peered":"2024-09-13T13:05:47.781195+0000","last_clean":"2024-09-13T13:05:47.781195+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:05:47.781195+0000","last_undegraded":"2024-09-13T13:05:47.781195+0000","last_fullsized":"2024-09-13T13:05:47.781195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.997740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.997740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.184374+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"154'13836","reported_seq":14111,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:48.053063+0000","last_change":"2024-09-13T13:05:08.464991+0000","last_active":"2024-09-13T13:05:48.053063+0000","last_peered":"2024-09-13T13:05:48.053063+0000","last_clean":"2024-09-13T13:05:48.053063+0000","last_became_active":"2024-09-13T13:02:12.375310+0000","last_became_peered":"2024-09-13T13:02:12.375310+0000","last_unstale":"2024-09-13T13:05:48.053063+0000","last_undegraded":"2024-09-13T13:05:48.053063+0000","last_fullsized":"2024-09-13T13:05:48.053063+0000","mapping_epoch":84,"log_start":"114'7000","ondisk_log_start":"114'7000","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'11530","last_scrub_stamp":"2024-09-13T13:05:08.464709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:08.464709+0000","objects_scrubbed":3682,"log_size":6836,"log_dups_size":0,"ondisk_log_size":6836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:11.110516+0000","scrub_duration":1226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87818240,"num_objects":1340,"num_object_clones":0,"num_object_copies":4020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13836,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":221,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780771+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:05:47.780771+0000","last_peered":"2024-09-13T13:05:47.780771+0000","last_clean":"2024-09-13T13:05:47.780771+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:05:47.780771+0000","last_undegraded":"2024-09-13T13:05:47.780771+0000","last_fullsized":"2024-09-13T13:05:47.780771+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":228,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796688+0000","last_change":"2024-09-13T13:05:37.520655+0000","last_active":"2024-09-13T13:05:47.796688+0000","last_peered":"2024-09-13T13:05:47.796688+0000","last_clean":"2024-09-13T13:05:47.796688+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:05:47.796688+0000","last_undegraded":"2024-09-13T13:05:47.796688+0000","last_fullsized":"2024-09-13T13:05:47.796688+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:37.520603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:37.520603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:56.825070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":210,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.802606+0000","last_change":"2024-09-13T13:04:56.183494+0000","last_active":"2024-09-13T13:05:47.802606+0000","last_peered":"2024-09-13T13:05:47.802606+0000","last_clean":"2024-09-13T13:05:47.802606+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:05:47.802606+0000","last_undegraded":"2024-09-13T13:05:47.802606+0000","last_fullsized":"2024-09-13T13:05:47.802606+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:56.183290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.183290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:07.795607+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":189,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781090+0000","last_change":"2024-09-13T13:01:08.211229+0000","last_active":"2024-09-13T13:05:47.781090+0000","last_peered":"2024-09-13T13:05:47.781090+0000","last_clean":"2024-09-13T13:05:47.781090+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:05:47.781090+0000","last_undegraded":"2024-09-13T13:05:47.781090+0000","last_fullsized":"2024-09-13T13:05:47.781090+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"154'14011","reported_seq":14163,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:48.448994+0000","last_change":"2024-09-13T13:02:12.379755+0000","last_active":"2024-09-13T13:05:48.448994+0000","last_peered":"2024-09-13T13:05:48.448994+0000","last_clean":"2024-09-13T13:05:48.448994+0000","last_became_active":"2024-09-13T13:02:12.379442+0000","last_became_peered":"2024-09-13T13:02:12.379442+0000","last_unstale":"2024-09-13T13:05:48.448994+0000","last_undegraded":"2024-09-13T13:05:48.448994+0000","last_fullsized":"2024-09-13T13:05:48.448994+0000","mapping_epoch":84,"log_start":"120'8400","ondisk_log_start":"120'8400","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5611,"log_dups_size":0,"ondisk_log_size":5611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84869120,"num_objects":1295,"num_object_clones":0,"num_object_copies":3885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14011,"num_write_kb":489792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":193,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T13:05:42.778612+0000","last_change":"2024-09-13T13:04:02.760376+0000","last_active":"2024-09-13T13:05:42.778612+0000","last_peered":"2024-09-13T13:05:42.778612+0000","last_clean":"2024-09-13T13:05:42.778612+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:05:42.778612+0000","last_undegraded":"2024-09-13T13:05:42.778612+0000","last_fullsized":"2024-09-13T13:05:42.778612+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:42.593284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.593284+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":220,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782874+0000","last_change":"2024-09-13T13:04:02.759538+0000","last_active":"2024-09-13T13:05:47.782874+0000","last_peered":"2024-09-13T13:05:47.782874+0000","last_clean":"2024-09-13T13:05:47.782874+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:05:47.782874+0000","last_undegraded":"2024-09-13T13:05:47.782874+0000","last_fullsized":"2024-09-13T13:05:47.782874+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:29.012312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.012312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:13.429060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":210,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.801447+0000","last_change":"2024-09-13T13:04:47.927556+0000","last_active":"2024-09-13T13:05:47.801447+0000","last_peered":"2024-09-13T13:05:47.801447+0000","last_clean":"2024-09-13T13:05:47.801447+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:05:47.801447+0000","last_undegraded":"2024-09-13T13:05:47.801447+0000","last_fullsized":"2024-09-13T13:05:47.801447+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.927432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.927432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.726893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782491+0000","last_change":"2024-09-13T13:05:11.420568+0000","last_active":"2024-09-13T13:05:47.782491+0000","last_peered":"2024-09-13T13:05:47.782491+0000","last_clean":"2024-09-13T13:05:47.782491+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:05:47.782491+0000","last_undegraded":"2024-09-13T13:05:47.782491+0000","last_fullsized":"2024-09-13T13:05:47.782491+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:11.420503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:11.420503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:12.545408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"154'13842","reported_seq":14046,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:48.262736+0000","last_change":"2024-09-13T13:03:45.723433+0000","last_active":"2024-09-13T13:05:48.262736+0000","last_peered":"2024-09-13T13:05:48.262736+0000","last_clean":"2024-09-13T13:05:48.262736+0000","last_became_active":"2024-09-13T13:02:12.376334+0000","last_became_peered":"2024-09-13T13:02:12.376334+0000","last_unstale":"2024-09-13T13:05:48.262736+0000","last_undegraded":"2024-09-13T13:05:48.262736+0000","last_fullsized":"2024-09-13T13:05:48.262736+0000","mapping_epoch":84,"log_start":"126'9200","ondisk_log_start":"126'9200","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":4774,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90308608,"num_objects":1378,"num_object_clones":0,"num_object_copies":4134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13842,"num_write_kb":487040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":229,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781112+0000","last_change":"2024-09-13T13:02:28.990089+0000","last_active":"2024-09-13T13:05:47.781112+0000","last_peered":"2024-09-13T13:05:47.781112+0000","last_clean":"2024-09-13T13:05:47.781112+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:05:47.781112+0000","last_undegraded":"2024-09-13T13:05:47.781112+0000","last_fullsized":"2024-09-13T13:05:47.781112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:02:28.990005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.990005+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.110320+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":225,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.784914+0000","last_change":"2024-09-13T13:05:29.345863+0000","last_active":"2024-09-13T13:05:47.784914+0000","last_peered":"2024-09-13T13:05:47.784914+0000","last_clean":"2024-09-13T13:05:47.784914+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:05:47.784914+0000","last_undegraded":"2024-09-13T13:05:47.784914+0000","last_fullsized":"2024-09-13T13:05:47.784914+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.345796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.345796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.394429+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":202,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.800242+0000","last_change":"2024-09-13T13:05:29.218178+0000","last_active":"2024-09-13T13:05:47.800242+0000","last_peered":"2024-09-13T13:05:47.800242+0000","last_clean":"2024-09-13T13:05:47.800242+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:05:47.800242+0000","last_undegraded":"2024-09-13T13:05:47.800242+0000","last_fullsized":"2024-09-13T13:05:47.800242+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.218119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.218119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:30.944093+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.789225+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:05:47.789225+0000","last_peered":"2024-09-13T13:05:47.789225+0000","last_clean":"2024-09-13T13:05:47.789225+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:05:47.789225+0000","last_undegraded":"2024-09-13T13:05:47.789225+0000","last_fullsized":"2024-09-13T13:05:47.789225+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"154'13981","reported_seq":20181,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:48.068230+0000","last_change":"2024-09-13T13:05:29.580927+0000","last_active":"2024-09-13T13:05:48.068230+0000","last_peered":"2024-09-13T13:05:48.068230+0000","last_clean":"2024-09-13T13:05:48.068230+0000","last_became_active":"2024-09-13T13:04:48.765805+0000","last_became_peered":"2024-09-13T13:04:48.765805+0000","last_unstale":"2024-09-13T13:05:48.068230+0000","last_undegraded":"2024-09-13T13:05:48.068230+0000","last_fullsized":"2024-09-13T13:05:48.068230+0000","mapping_epoch":135,"log_start":"124'8900","ondisk_log_start":"124'8900","created":84,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'12879","last_scrub_stamp":"2024-09-13T13:05:29.580736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.580736+0000","objects_scrubbed":2542,"log_size":5081,"log_dups_size":0,"ondisk_log_size":5081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:43.250181+0000","scrub_duration":768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92471296,"num_objects":1411,"num_object_clones":0,"num_object_copies":4233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13981,"num_write_kb":492544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3000,"num_bytes_recovered":156631040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":241,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.795990+0000","last_change":"2024-09-13T13:04:52.666294+0000","last_active":"2024-09-13T13:05:47.795990+0000","last_peered":"2024-09-13T13:05:47.795990+0000","last_clean":"2024-09-13T13:05:47.795990+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:05:47.795990+0000","last_undegraded":"2024-09-13T13:05:47.795990+0000","last_fullsized":"2024-09-13T13:05:47.795990+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:52.666222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:52.666222+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:57.559648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":229,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.783656+0000","last_change":"2024-09-13T13:05:27.422529+0000","last_active":"2024-09-13T13:05:47.783656+0000","last_peered":"2024-09-13T13:05:47.783656+0000","last_clean":"2024-09-13T13:05:47.783656+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:05:47.783656+0000","last_undegraded":"2024-09-13T13:05:47.783656+0000","last_fullsized":"2024-09-13T13:05:47.783656+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:05:27.422451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:27.422451+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.192286+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":284,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781951+0000","last_change":"2024-09-13T13:04:02.759553+0000","last_active":"2024-09-13T13:05:47.781951+0000","last_peered":"2024-09-13T13:05:47.781951+0000","last_clean":"2024-09-13T13:05:47.781951+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:05:47.781951+0000","last_undegraded":"2024-09-13T13:05:47.781951+0000","last_fullsized":"2024-09-13T13:05:47.781951+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:45.959285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.959285+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.359173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796172+0000","last_change":"2024-09-13T13:04:02.757776+0000","last_active":"2024-09-13T13:05:47.796172+0000","last_peered":"2024-09-13T13:05:47.796172+0000","last_clean":"2024-09-13T13:05:47.796172+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:05:47.796172+0000","last_undegraded":"2024-09-13T13:05:47.796172+0000","last_fullsized":"2024-09-13T13:05:47.796172+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:28.529954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.529954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.625121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":182,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T13:05:43.098238+0000","last_change":"2024-09-13T13:05:43.098238+0000","last_active":"2024-09-13T13:05:43.098238+0000","last_peered":"2024-09-13T13:05:43.098238+0000","last_clean":"2024-09-13T13:05:43.098238+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:05:43.098238+0000","last_undegraded":"2024-09-13T13:05:43.098238+0000","last_fullsized":"2024-09-13T13:05:43.098238+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:43.098196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:43.098196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:07.803311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":213,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.794500+0000","last_change":"2024-09-13T13:03:14.504339+0000","last_active":"2024-09-13T13:05:47.794500+0000","last_peered":"2024-09-13T13:05:47.794500+0000","last_clean":"2024-09-13T13:05:47.794500+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:05:47.794500+0000","last_undegraded":"2024-09-13T13:05:47.794500+0000","last_fullsized":"2024-09-13T13:05:47.794500+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:03:14.504263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.504263+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796889+0000","last_change":"2024-09-13T13:05:27.619960+0000","last_active":"2024-09-13T13:05:47.796889+0000","last_peered":"2024-09-13T13:05:47.796889+0000","last_clean":"2024-09-13T13:05:47.796889+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:05:47.796889+0000","last_undegraded":"2024-09-13T13:05:47.796889+0000","last_fullsized":"2024-09-13T13:05:47.796889+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:27.619884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:27.619884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.490575+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":215,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782484+0000","last_change":"2024-09-13T13:05:37.197948+0000","last_active":"2024-09-13T13:05:47.782484+0000","last_peered":"2024-09-13T13:05:47.782484+0000","last_clean":"2024-09-13T13:05:47.782484+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:05:47.782484+0000","last_undegraded":"2024-09-13T13:05:47.782484+0000","last_fullsized":"2024-09-13T13:05:47.782484+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:37.197907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:37.197907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:55.791682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":220,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.790788+0000","last_change":"2024-09-13T13:02:42.589951+0000","last_active":"2024-09-13T13:05:47.790788+0000","last_peered":"2024-09-13T13:05:47.790788+0000","last_clean":"2024-09-13T13:05:47.790788+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:05:47.790788+0000","last_undegraded":"2024-09-13T13:05:47.790788+0000","last_fullsized":"2024-09-13T13:05:47.790788+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:42.589866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.589866+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":220,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782144+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:05:47.782144+0000","last_peered":"2024-09-13T13:05:47.782144+0000","last_clean":"2024-09-13T13:05:47.782144+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:05:47.782144+0000","last_undegraded":"2024-09-13T13:05:47.782144+0000","last_fullsized":"2024-09-13T13:05:47.782144+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.542132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":200,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.794359+0000","last_change":"2024-09-13T13:05:42.728927+0000","last_active":"2024-09-13T13:05:47.794359+0000","last_peered":"2024-09-13T13:05:47.794359+0000","last_clean":"2024-09-13T13:05:47.794359+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:05:47.794359+0000","last_undegraded":"2024-09-13T13:05:47.794359+0000","last_fullsized":"2024-09-13T13:05:47.794359+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:42.728881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:42.728881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:06.941341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":205,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.800614+0000","last_change":"2024-09-13T13:05:45.343713+0000","last_active":"2024-09-13T13:05:47.800614+0000","last_peered":"2024-09-13T13:05:47.800614+0000","last_clean":"2024-09-13T13:05:47.800614+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:05:47.800614+0000","last_undegraded":"2024-09-13T13:05:47.800614+0000","last_fullsized":"2024-09-13T13:05:47.800614+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:45.343638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.343638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:52.672377+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":220,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796494+0000","last_change":"2024-09-13T13:04:53.687731+0000","last_active":"2024-09-13T13:05:47.796494+0000","last_peered":"2024-09-13T13:05:47.796494+0000","last_clean":"2024-09-13T13:05:47.796494+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:05:47.796494+0000","last_undegraded":"2024-09-13T13:05:47.796494+0000","last_fullsized":"2024-09-13T13:05:47.796494+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.632415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780410+0000","last_change":"2024-09-13T13:02:41.622369+0000","last_active":"2024-09-13T13:05:47.780410+0000","last_peered":"2024-09-13T13:05:47.780410+0000","last_clean":"2024-09-13T13:05:47.780410+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:05:47.780410+0000","last_undegraded":"2024-09-13T13:05:47.780410+0000","last_fullsized":"2024-09-13T13:05:47.780410+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.622292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.622292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":258,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.788914+0000","last_change":"2024-09-13T13:01:06.210251+0000","last_active":"2024-09-13T13:05:47.788914+0000","last_peered":"2024-09-13T13:05:47.788914+0000","last_clean":"2024-09-13T13:05:47.788914+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:05:47.788914+0000","last_undegraded":"2024-09-13T13:05:47.788914+0000","last_fullsized":"2024-09-13T13:05:47.788914+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781648+0000","last_change":"2024-09-13T13:04:49.927118+0000","last_active":"2024-09-13T13:05:47.781648+0000","last_peered":"2024-09-13T13:05:47.781648+0000","last_clean":"2024-09-13T13:05:47.781648+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:05:47.781648+0000","last_undegraded":"2024-09-13T13:05:47.781648+0000","last_fullsized":"2024-09-13T13:05:47.781648+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:49.927073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:49.927073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:08.714937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":220,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.789302+0000","last_change":"2024-09-13T13:02:37.222400+0000","last_active":"2024-09-13T13:05:47.789302+0000","last_peered":"2024-09-13T13:05:47.789302+0000","last_clean":"2024-09-13T13:05:47.789302+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:05:47.789302+0000","last_undegraded":"2024-09-13T13:05:47.789302+0000","last_fullsized":"2024-09-13T13:05:47.789302+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:37.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.222329+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.783264+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:05:47.783264+0000","last_peered":"2024-09-13T13:05:47.783264+0000","last_clean":"2024-09-13T13:05:47.783264+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:05:47.783264+0000","last_undegraded":"2024-09-13T13:05:47.783264+0000","last_fullsized":"2024-09-13T13:05:47.783264+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:55.131839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":282,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796472+0000","last_change":"2024-09-13T13:05:45.595219+0000","last_active":"2024-09-13T13:05:47.796472+0000","last_peered":"2024-09-13T13:05:47.796472+0000","last_clean":"2024-09-13T13:05:47.796472+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:05:47.796472+0000","last_undegraded":"2024-09-13T13:05:47.796472+0000","last_fullsized":"2024-09-13T13:05:47.796472+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:45.595161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.595161+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:05.452404+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":196,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.786902+0000","last_change":"2024-09-13T13:02:40.615749+0000","last_active":"2024-09-13T13:05:47.786902+0000","last_peered":"2024-09-13T13:05:47.786902+0000","last_clean":"2024-09-13T13:05:47.786902+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:05:47.786902+0000","last_undegraded":"2024-09-13T13:05:47.786902+0000","last_fullsized":"2024-09-13T13:05:47.786902+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:40.615674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:40.615674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":244,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780873+0000","last_change":"2024-09-13T13:05:26.186899+0000","last_active":"2024-09-13T13:05:47.780873+0000","last_peered":"2024-09-13T13:05:47.780873+0000","last_clean":"2024-09-13T13:05:47.780873+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:05:47.780873+0000","last_undegraded":"2024-09-13T13:05:47.780873+0000","last_fullsized":"2024-09-13T13:05:47.780873+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:05:26.186830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.186830+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:29.937721+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":229,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781681+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:05:47.781681+0000","last_peered":"2024-09-13T13:05:47.781681+0000","last_clean":"2024-09-13T13:05:47.781681+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:05:47.781681+0000","last_undegraded":"2024-09-13T13:05:47.781681+0000","last_fullsized":"2024-09-13T13:05:47.781681+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:39.039395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":210,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.801042+0000","last_change":"2024-09-13T13:03:44.467611+0000","last_active":"2024-09-13T13:05:47.801042+0000","last_peered":"2024-09-13T13:05:47.801042+0000","last_clean":"2024-09-13T13:05:47.801042+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:05:47.801042+0000","last_undegraded":"2024-09-13T13:05:47.801042+0000","last_fullsized":"2024-09-13T13:05:47.801042+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:44.467297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:44.467297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.789492+0000","last_change":"2024-09-13T13:02:42.208021+0000","last_active":"2024-09-13T13:05:47.789492+0000","last_peered":"2024-09-13T13:05:47.789492+0000","last_clean":"2024-09-13T13:05:47.789492+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:05:47.789492+0000","last_undegraded":"2024-09-13T13:05:47.789492+0000","last_fullsized":"2024-09-13T13:05:47.789492+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:42.207956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.207956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":172,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796013+0000","last_change":"2024-09-13T13:05:35.506093+0000","last_active":"2024-09-13T13:05:47.796013+0000","last_peered":"2024-09-13T13:05:47.796013+0000","last_clean":"2024-09-13T13:05:47.796013+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:05:47.796013+0000","last_undegraded":"2024-09-13T13:05:47.796013+0000","last_fullsized":"2024-09-13T13:05:47.796013+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:05:35.506037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.506037+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:58.281227+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782742+0000","last_change":"2024-09-13T13:02:39.921391+0000","last_active":"2024-09-13T13:05:47.782742+0000","last_peered":"2024-09-13T13:05:47.782742+0000","last_clean":"2024-09-13T13:05:47.782742+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:05:47.782742+0000","last_undegraded":"2024-09-13T13:05:47.782742+0000","last_fullsized":"2024-09-13T13:05:47.782742+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.921325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.921325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:20.381317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":192,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.789304+0000","last_change":"2024-09-13T13:01:06.211993+0000","last_active":"2024-09-13T13:05:47.789304+0000","last_peered":"2024-09-13T13:05:47.789304+0000","last_clean":"2024-09-13T13:05:47.789304+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:05:47.789304+0000","last_undegraded":"2024-09-13T13:05:47.789304+0000","last_fullsized":"2024-09-13T13:05:47.789304+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781408+0000","last_change":"2024-09-13T13:02:36.586702+0000","last_active":"2024-09-13T13:05:47.781408+0000","last_peered":"2024-09-13T13:05:47.781408+0000","last_clean":"2024-09-13T13:05:47.781408+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:05:47.781408+0000","last_undegraded":"2024-09-13T13:05:47.781408+0000","last_fullsized":"2024-09-13T13:05:47.781408+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.586654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.586654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":235,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.789599+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:05:47.789599+0000","last_peered":"2024-09-13T13:05:47.789599+0000","last_clean":"2024-09-13T13:05:47.789599+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:05:47.789599+0000","last_undegraded":"2024-09-13T13:05:47.789599+0000","last_fullsized":"2024-09-13T13:05:47.789599+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796932+0000","last_change":"2024-09-13T13:05:12.597696+0000","last_active":"2024-09-13T13:05:47.796932+0000","last_peered":"2024-09-13T13:05:47.796932+0000","last_clean":"2024-09-13T13:05:47.796932+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:05:47.796932+0000","last_undegraded":"2024-09-13T13:05:47.796932+0000","last_fullsized":"2024-09-13T13:05:47.796932+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:12.597637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.597637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:30.070820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":212,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782152+0000","last_change":"2024-09-13T13:05:08.432874+0000","last_active":"2024-09-13T13:05:47.782152+0000","last_peered":"2024-09-13T13:05:47.782152+0000","last_clean":"2024-09-13T13:05:47.782152+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:05:47.782152+0000","last_undegraded":"2024-09-13T13:05:47.782152+0000","last_fullsized":"2024-09-13T13:05:47.782152+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:08.432816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:08.432816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:09.081986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":199,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.789471+0000","last_change":"2024-09-13T13:04:02.760695+0000","last_active":"2024-09-13T13:05:47.789471+0000","last_peered":"2024-09-13T13:05:47.789471+0000","last_clean":"2024-09-13T13:05:47.789471+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:05:47.789471+0000","last_undegraded":"2024-09-13T13:05:47.789471+0000","last_fullsized":"2024-09-13T13:05:47.789471+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":235,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.791315+0000","last_change":"2024-09-13T13:05:44.759472+0000","last_active":"2024-09-13T13:05:47.791315+0000","last_peered":"2024-09-13T13:05:47.791315+0000","last_clean":"2024-09-13T13:05:47.791315+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:05:47.791315+0000","last_undegraded":"2024-09-13T13:05:47.791315+0000","last_fullsized":"2024-09-13T13:05:47.791315+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2024-09-13T13:05:44.759233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.759233+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:51.091258+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.786706+0000","last_change":"2024-09-13T13:03:15.466909+0000","last_active":"2024-09-13T13:05:47.786706+0000","last_peered":"2024-09-13T13:05:47.786706+0000","last_clean":"2024-09-13T13:05:47.786706+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:05:47.786706+0000","last_undegraded":"2024-09-13T13:05:47.786706+0000","last_fullsized":"2024-09-13T13:05:47.786706+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:15.466859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:15.466859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":90,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T13:05:42.779087+0000","last_change":"2024-09-13T13:04:02.765046+0000","last_active":"2024-09-13T13:05:42.779087+0000","last_peered":"2024-09-13T13:05:42.779087+0000","last_clean":"2024-09-13T13:05:42.779087+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:05:42.779087+0000","last_undegraded":"2024-09-13T13:05:42.779087+0000","last_fullsized":"2024-09-13T13:05:42.779087+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":196,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781864+0000","last_change":"2024-09-13T13:04:58.468060+0000","last_active":"2024-09-13T13:05:47.781864+0000","last_peered":"2024-09-13T13:05:47.781864+0000","last_clean":"2024-09-13T13:05:47.781864+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:05:47.781864+0000","last_undegraded":"2024-09-13T13:05:47.781864+0000","last_fullsized":"2024-09-13T13:05:47.781864+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:58.468011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:58.468011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.129704+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":205,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796219+0000","last_change":"2024-09-13T13:05:44.759683+0000","last_active":"2024-09-13T13:05:47.796219+0000","last_peered":"2024-09-13T13:05:47.796219+0000","last_clean":"2024-09-13T13:05:47.796219+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:05:47.796219+0000","last_undegraded":"2024-09-13T13:05:47.796219+0000","last_fullsized":"2024-09-13T13:05:47.796219+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:44.759284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.759284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:07.342203+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.791542+0000","last_change":"2024-09-13T13:02:46.246602+0000","last_active":"2024-09-13T13:05:47.791542+0000","last_peered":"2024-09-13T13:05:47.791542+0000","last_clean":"2024-09-13T13:05:47.791542+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:05:47.791542+0000","last_undegraded":"2024-09-13T13:05:47.791542+0000","last_fullsized":"2024-09-13T13:05:47.791542+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.246540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.246540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":200,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T13:05:42.778801+0000","last_change":"2024-09-13T13:04:02.756287+0000","last_active":"2024-09-13T13:05:42.778801+0000","last_peered":"2024-09-13T13:05:42.778801+0000","last_clean":"2024-09-13T13:05:42.778801+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:05:42.778801+0000","last_undegraded":"2024-09-13T13:05:42.778801+0000","last_fullsized":"2024-09-13T13:05:42.778801+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:02:39.557251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.557251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":201,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.787817+0000","last_change":"2024-09-13T13:02:30.512536+0000","last_active":"2024-09-13T13:05:47.787817+0000","last_peered":"2024-09-13T13:05:47.787817+0000","last_clean":"2024-09-13T13:05:47.787817+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:05:47.787817+0000","last_undegraded":"2024-09-13T13:05:47.787817+0000","last_fullsized":"2024-09-13T13:05:47.787817+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:30.512433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:30.512433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1441071128,"num_objects":21990,"num_object_clones":0,"num_object_copies":65970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21990,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220208,"num_write_kb":7750273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14752,"num_bytes_recovered":756088856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4296749056,"data_stored":4296736840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89375,"ondisk_log_size":89375,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":280,"num_read_kb":275,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201340,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":565144,"kb_used_data":453384,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93180008,"statfs":{"total":95995035648,"available":95416328192,"internally_reserved":0,"allocated":464265216,"data_stored":463822702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806140950,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":638464,"kb_used_data":496880,"kb_used_omap":4,"kb_used_meta":141563,"kb_avail":93106688,"statfs":{"total":95995035648,"available":95341248512,"internally_reserved":0,"allocated":508805120,"data_stored":508378753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":144961197},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,16],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691912,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":467664,"kb_used_data":360124,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93277488,"statfs":{"total":95995035648,"available":95516147712,"internally_reserved":0,"allocated":368766976,"data_stored":368335046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560484,"kb_used_data":448784,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93184668,"statfs":{"total":95995035648,"available":95421100032,"internally_reserved":0,"allocated":459554816,"data_stored":459102542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117077,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":960808,"kb_used_data":794964,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92784344,"statfs":{"total":95995035648,"available":95011168256,"internally_reserved":0,"allocated":814043136,"data_stored":813587596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":657276,"kb_used_data":545580,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93087876,"statfs":{"total":95995035648,"available":95321985024,"internally_reserved":0,"allocated":558673920,"data_stored":558257714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":648312,"kb_used_data":530920,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93096840,"statfs":{"total":95995035648,"available":95331164160,"internally_reserved":0,"allocated":543662080,"data_stored":543185765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":858288,"kb_used_data":692380,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92886864,"statfs":{"total":95995035648,"available":95116148736,"internally_reserved":0,"allocated":708997120,"data_stored":708540683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":693047296,"data_stored":693043224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":534712320,"data_stored":534708248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":535232512,"data_stored":535232512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":793182208,"data_stored":793182208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":440664064,"data_stored":440664064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":362020864,"data_stored":362020864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":492507136,"data_stored":492503064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":445382656,"data_stored":445382656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:05:52.290 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:52.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:52.392 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:52.417 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:52.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:52.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:52.543 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:52.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:52.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:52.645 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:52.675 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:52.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:52.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:52.777 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:52.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:52.802 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:52.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:52.903 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:52.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:52.948 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:52.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:53.010 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:05:53.050 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:53.081 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:05:53.081 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:05:53.100 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:53.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:53.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:53.202 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:53.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:52 smithi052 ceph-mon[36682]: pgmap v215: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 742 op/s; 0 B/s, 38 objects/s recovering 2024-09-13T13:05:53.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:52 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/800413675' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:05:53.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:52 smithi052 ceph-mon[29471]: pgmap v215: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 742 op/s; 0 B/s, 38 objects/s recovering 2024-09-13T13:05:53.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:52 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/800413675' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:05:53.240 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:53.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:53.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:52 smithi167 ceph-mon[33178]: pgmap v215: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 742 op/s; 0 B/s, 38 objects/s recovering 2024-09-13T13:05:53.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:52 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/800413675' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:05:53.341 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:53.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:53.382 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:53.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:53.483 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:53.520 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:53.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:53.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:53.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:53.660 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:53.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:53.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:53.714 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:53.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:53.794 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:53.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:53.896 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:53.939 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:53.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:53.996 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:05:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:05:54.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:54.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:54.070 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:54.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:54.171 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:54.197 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:54.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:54.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:53 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:54.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:53 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/800413675' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:05:54.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:53 smithi052 ceph-mon[36682]: osdmap e155: 8 total, 8 up, 8 in 2024-09-13T13:05:54.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:53 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:54.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/800413675' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:05:54.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:53 smithi052 ceph-mon[29471]: osdmap e155: 8 total, 8 up, 8 in 2024-09-13T13:05:54.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:54.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:53 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:05:54.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:53 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/800413675' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:05:54.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:53 smithi167 ceph-mon[33178]: osdmap e155: 8 total, 8 up, 8 in 2024-09-13T13:05:54.324 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:54.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:54.426 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:54.462 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:54.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:54.564 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:54.629 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:54.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:54.721 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T13:05:54.730 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:54.824 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:54.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:54.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:54.951 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:54.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:55.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:55.079 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:55.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:55.181 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:55.204 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:55.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:55.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:55.257 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:05:55.277 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:55 smithi052 ceph-mon[29471]: pgmap v217: 148 pgs: 148 active+clean; 1.0 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-13T13:05:55.278 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:55.278 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:55 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:05:55.278 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:55 smithi052 ceph-mon[36682]: pgmap v217: 148 pgs: 148 active+clean; 1.0 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-13T13:05:55.278 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:55.278 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:55 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:05:55.306 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:55 smithi167 ceph-mon[33178]: pgmap v217: 148 pgs: 148 active+clean; 1.0 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-13T13:05:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:05:55.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:55 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:05:55.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:55.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:55.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:55.364 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:05:55.364 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:05:55.401 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:05:55.402 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:05:55.431 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:55.452 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:05:55.452 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:05:55.455 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:55.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:55.478 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:55.499 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:05:55.499 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:05:55.533 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:05:55.533 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:05:55.556 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:55.567 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:05:55.568 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:05:55.601 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:05:55.601 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:05:55.620 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:55.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:55.641 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:05:55.641 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:05:55.722 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:55.775 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:55.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:55.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:55.877 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:55.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:55.919 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:55.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:56.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:56.044 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:56.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:56.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:56.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:56 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:05:56.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:56 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:05:56.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:56 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:05:56.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:56 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/581408601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:56.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:56 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:05:56.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:56 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:05:56.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:56 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:05:56.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:56 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/581408601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:56.146 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:56.190 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:56.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:56.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:56.213 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:05:56.213 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:05:56.214 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:05:56.214 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:05:56.214 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:05:56.291 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:56.319 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:56.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:56.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:56.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:56 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:05:56.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:56 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:05:56.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:56 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:05:56.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:56 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/581408601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:05:56.359 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:05:56.407 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:05:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:05:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:05:56.421 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:56.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:56.469 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:56.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:56.570 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:56.635 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:56.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:56.737 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:56.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:56.788 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:56.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:56.889 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:56.930 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:56.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:56.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:57.031 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:57.074 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:57.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:57.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:57.146 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T13:05:57.177 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:05:57.259 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:57.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:05:57.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:57.384 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:57.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:57.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:57.486 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:57.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:57 smithi052 ceph-mon[36682]: pgmap v218: 148 pgs: 148 active+clean; 1.0 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-13T13:05:57.497 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:57 smithi052 ceph-mon[29471]: pgmap v218: 148 pgs: 148 active+clean; 1.0 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-13T13:05:57.535 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:57.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:57.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:57.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:57 smithi167 ceph-mon[33178]: pgmap v218: 148 pgs: 148 active+clean; 1.0 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-13T13:05:57.637 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:57.663 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:57.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:57.717 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:57.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:57.788 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:57.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:57.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:57.890 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:57.942 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:57.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:57.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:58.044 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:05:58.069 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:58.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:58.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:05:58.171 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:58.199 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:58.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:58.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:58.301 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:58.346 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:58.354 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:58.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:58.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:58.389 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:05:58.456 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:58.480 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:58.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:58.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:58.582 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:58.621 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:58.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:58.723 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:05:58.747 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:58.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:58.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:05:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:05:58.781 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:05:58.849 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:58.892 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:58.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:58.992 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:59.029 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:59.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:59.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:58 smithi167 ceph-mon[33178]: pgmap v219: 148 pgs: 148 active+clean; 1003 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-09-13T13:05:59.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:58 smithi167 ceph-mon[33178]: osdmap e156: 8 total, 8 up, 8 in 2024-09-13T13:05:59.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:58 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:05:59.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:58 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:05:59.130 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:05:59.171 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:59.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:05:59.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:58 smithi052 ceph-mon[36682]: pgmap v219: 148 pgs: 148 active+clean; 1003 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-09-13T13:05:59.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:58 smithi052 ceph-mon[36682]: osdmap e156: 8 total, 8 up, 8 in 2024-09-13T13:05:59.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:58 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:05:59.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:58 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:05:59.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:58 smithi052 ceph-mon[29471]: pgmap v219: 148 pgs: 148 active+clean; 1003 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-09-13T13:05:59.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:58 smithi052 ceph-mon[29471]: osdmap e156: 8 total, 8 up, 8 in 2024-09-13T13:05:59.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:58 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:05:59.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:58 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:05:59.273 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:59.311 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:59.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:59.413 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:05:59.449 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:59.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:05:59.551 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:59.577 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:59.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:59.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:59.679 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:05:59.679 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:05:59.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:59.732 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:05:59.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:05:59.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:05:59.858 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:59.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:59.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:05:59.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:05:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:05:59.986 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:05:59.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:05:59.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:59 smithi052 ceph-mon[36682]: from='client.25129 v1:172.21.15.52:0/390154424' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:59.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:05:59 smithi052 ceph-mon[36682]: osdmap e157: 8 total, 8 up, 8 in 2024-09-13T13:05:59.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:59 smithi052 ceph-mon[29471]: from='client.25129 v1:172.21.15.52:0/390154424' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:05:59.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:05:59 smithi052 ceph-mon[29471]: osdmap e157: 8 total, 8 up, 8 in 2024-09-13T13:06:00.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:59 smithi167 ceph-mon[33178]: from='client.25129 v1:172.21.15.52:0/390154424' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:06:00.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:05:59 smithi167 ceph-mon[33178]: osdmap e157: 8 total, 8 up, 8 in 2024-09-13T13:06:00.088 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:00.118 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:00.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:00.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:00.246 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:00.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:00.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:00.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:00.377 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:00.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:00.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:00.479 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:00.529 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:00.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:00.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:00.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:00.656 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:00.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:00.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:00.687 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-13T13:05:57.782743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1051990579,"num_objects":16243,"num_object_clones":0,"num_object_copies":48729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16243,"num_whiteouts":0,"num_read":1132,"num_read_kb":989,"num_write":226703,"num_write_kb":7751946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14760,"num_bytes_recovered":756679334,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90581,"ondisk_log_size":90581,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4282884,"kb_used_data":3256192,"kb_used_omap":15,"kb_used_meta":1026096,"kb_avail":745678332,"statfs":{"total":767960285184,"available":763574611968,"internally_reserved":0,"allocated":3334340608,"data_stored":3330728385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":1050723097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-389677056,"num_objects":-5946,"num_object_clones":0,"num_object_copies":-17838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5946,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5946,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.108406"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":211,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.010676+0000","last_change":"2024-09-13T13:02:38.651382+0000","last_active":"2024-09-13T13:05:53.010676+0000","last_peered":"2024-09-13T13:05:53.010676+0000","last_clean":"2024-09-13T13:05:53.010676+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:05:53.010676+0000","last_undegraded":"2024-09-13T13:05:53.010676+0000","last_fullsized":"2024-09-13T13:05:53.010676+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:38.651314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:38.651314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":232,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780418+0000","last_change":"2024-09-13T13:04:48.899982+0000","last_active":"2024-09-13T13:05:47.780418+0000","last_peered":"2024-09-13T13:05:47.780418+0000","last_clean":"2024-09-13T13:05:47.780418+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:05:47.780418+0000","last_undegraded":"2024-09-13T13:05:47.780418+0000","last_fullsized":"2024-09-13T13:05:47.780418+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:48.899915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.899915+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:00.227319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":211,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.025869+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:05:53.025869+0000","last_peered":"2024-09-13T13:05:53.025869+0000","last_clean":"2024-09-13T13:05:53.025869+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:05:53.025869+0000","last_undegraded":"2024-09-13T13:05:53.025869+0000","last_fullsized":"2024-09-13T13:05:53.025869+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:47.640365+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780536+0000","last_change":"2024-09-13T13:05:01.076346+0000","last_active":"2024-09-13T13:05:47.780536+0000","last_peered":"2024-09-13T13:05:47.780536+0000","last_clean":"2024-09-13T13:05:47.780536+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:05:47.780536+0000","last_undegraded":"2024-09-13T13:05:47.780536+0000","last_fullsized":"2024-09-13T13:05:47.780536+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.076275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.076275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:14.180362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.023685+0000","last_change":"2024-09-13T13:05:45.805098+0000","last_active":"2024-09-13T13:05:53.023685+0000","last_peered":"2024-09-13T13:05:53.023685+0000","last_clean":"2024-09-13T13:05:53.023685+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:05:53.023685+0000","last_undegraded":"2024-09-13T13:05:53.023685+0000","last_fullsized":"2024-09-13T13:05:53.023685+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:45.800516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.800516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:46.430007+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":222,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.024510+0000","last_change":"2024-09-13T13:04:02.757267+0000","last_active":"2024-09-13T13:05:53.024510+0000","last_peered":"2024-09-13T13:05:53.024510+0000","last_clean":"2024-09-13T13:05:53.024510+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:05:53.024510+0000","last_undegraded":"2024-09-13T13:05:53.024510+0000","last_fullsized":"2024-09-13T13:05:53.024510+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.233259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.233259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":239,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.021271+0000","last_change":"2024-09-13T13:04:02.396436+0000","last_active":"2024-09-13T13:05:53.021271+0000","last_peered":"2024-09-13T13:05:53.021271+0000","last_clean":"2024-09-13T13:05:53.021271+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:05:53.021271+0000","last_undegraded":"2024-09-13T13:05:53.021271+0000","last_fullsized":"2024-09-13T13:05:53.021271+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:02.396342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.396342+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":212,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780339+0000","last_change":"2024-09-13T13:04:02.765671+0000","last_active":"2024-09-13T13:05:47.780339+0000","last_peered":"2024-09-13T13:05:47.780339+0000","last_clean":"2024-09-13T13:05:47.780339+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:05:47.780339+0000","last_undegraded":"2024-09-13T13:05:47.780339+0000","last_fullsized":"2024-09-13T13:05:47.780339+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.021662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.021662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:18.072524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":225,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796562+0000","last_change":"2024-09-13T13:05:00.690387+0000","last_active":"2024-09-13T13:05:47.796562+0000","last_peered":"2024-09-13T13:05:47.796562+0000","last_clean":"2024-09-13T13:05:47.796562+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:05:47.796562+0000","last_undegraded":"2024-09-13T13:05:47.796562+0000","last_fullsized":"2024-09-13T13:05:47.796562+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:05:00.690308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:00.690308+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:03.644084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":221,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780674+0000","last_change":"2024-09-13T13:04:56.066382+0000","last_active":"2024-09-13T13:05:47.780674+0000","last_peered":"2024-09-13T13:05:47.780674+0000","last_clean":"2024-09-13T13:05:47.780674+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:05:47.780674+0000","last_undegraded":"2024-09-13T13:05:47.780674+0000","last_fullsized":"2024-09-13T13:05:47.780674+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:56.066326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.066326+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.781998+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796218+0000","last_change":"2024-09-13T13:05:26.621813+0000","last_active":"2024-09-13T13:05:47.796218+0000","last_peered":"2024-09-13T13:05:47.796218+0000","last_clean":"2024-09-13T13:05:47.796218+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:05:47.796218+0000","last_undegraded":"2024-09-13T13:05:47.796218+0000","last_fullsized":"2024-09-13T13:05:47.796218+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:26.621732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.621732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:48.964955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":198,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.007531+0000","last_change":"2024-09-13T13:02:39.607712+0000","last_active":"2024-09-13T13:05:53.007531+0000","last_peered":"2024-09-13T13:05:53.007531+0000","last_clean":"2024-09-13T13:05:53.007531+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:05:53.007531+0000","last_undegraded":"2024-09-13T13:05:53.007531+0000","last_fullsized":"2024-09-13T13:05:53.007531+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.607646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.607646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":224,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796244+0000","last_change":"2024-09-13T13:05:38.520071+0000","last_active":"2024-09-13T13:05:47.796244+0000","last_peered":"2024-09-13T13:05:47.796244+0000","last_clean":"2024-09-13T13:05:47.796244+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:05:47.796244+0000","last_undegraded":"2024-09-13T13:05:47.796244+0000","last_fullsized":"2024-09-13T13:05:47.796244+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:38.520006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:38.520006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:55.423473+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":223,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780844+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:05:47.780844+0000","last_peered":"2024-09-13T13:05:47.780844+0000","last_clean":"2024-09-13T13:05:47.780844+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:05:47.780844+0000","last_undegraded":"2024-09-13T13:05:47.780844+0000","last_fullsized":"2024-09-13T13:05:47.780844+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:15.096451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":200,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781984+0000","last_change":"2024-09-13T13:05:10.383826+0000","last_active":"2024-09-13T13:05:47.781984+0000","last_peered":"2024-09-13T13:05:47.781984+0000","last_clean":"2024-09-13T13:05:47.781984+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:05:47.781984+0000","last_undegraded":"2024-09-13T13:05:47.781984+0000","last_fullsized":"2024-09-13T13:05:47.781984+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.383756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.383756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:33.805546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":188,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796011+0000","last_change":"2024-09-13T13:01:08.213608+0000","last_active":"2024-09-13T13:05:47.796011+0000","last_peered":"2024-09-13T13:05:47.796011+0000","last_clean":"2024-09-13T13:05:47.796011+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:05:47.796011+0000","last_undegraded":"2024-09-13T13:05:47.796011+0000","last_fullsized":"2024-09-13T13:05:47.796011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.558201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":210,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781282+0000","last_change":"2024-09-13T13:04:54.016136+0000","last_active":"2024-09-13T13:05:47.781282+0000","last_peered":"2024-09-13T13:05:47.781282+0000","last_clean":"2024-09-13T13:05:47.781282+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:05:47.781282+0000","last_undegraded":"2024-09-13T13:05:47.781282+0000","last_fullsized":"2024-09-13T13:05:47.781282+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:04:54.016064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.016064+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.987760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.009018+0000","last_change":"2024-09-13T13:02:32.590355+0000","last_active":"2024-09-13T13:05:53.009018+0000","last_peered":"2024-09-13T13:05:53.009018+0000","last_clean":"2024-09-13T13:05:53.009018+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:05:53.009018+0000","last_undegraded":"2024-09-13T13:05:53.009018+0000","last_fullsized":"2024-09-13T13:05:53.009018+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:32.590282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.590282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":248,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.020861+0000","last_change":"2024-09-13T13:05:33.323411+0000","last_active":"2024-09-13T13:05:53.020861+0000","last_peered":"2024-09-13T13:05:53.020861+0000","last_clean":"2024-09-13T13:05:53.020861+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:05:53.020861+0000","last_undegraded":"2024-09-13T13:05:53.020861+0000","last_fullsized":"2024-09-13T13:05:53.020861+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:05:33.323316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.323316+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:36.730491+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.025949+0000","last_change":"2024-09-13T13:02:45.202865+0000","last_active":"2024-09-13T13:05:53.025949+0000","last_peered":"2024-09-13T13:05:53.025949+0000","last_clean":"2024-09-13T13:05:53.025949+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:05:53.025949+0000","last_undegraded":"2024-09-13T13:05:53.025949+0000","last_fullsized":"2024-09-13T13:05:53.025949+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.202802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.202802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:46.737187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.022605+0000","last_change":"2024-09-13T13:04:03.384955+0000","last_active":"2024-09-13T13:05:53.022605+0000","last_peered":"2024-09-13T13:05:53.022605+0000","last_clean":"2024-09-13T13:05:53.022605+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:05:53.022605+0000","last_undegraded":"2024-09-13T13:05:53.022605+0000","last_fullsized":"2024-09-13T13:05:53.022605+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.384901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.384901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780747+0000","last_change":"2024-09-13T13:04:57.103213+0000","last_active":"2024-09-13T13:05:47.780747+0000","last_peered":"2024-09-13T13:05:47.780747+0000","last_clean":"2024-09-13T13:05:47.780747+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:05:47.780747+0000","last_undegraded":"2024-09-13T13:05:47.780747+0000","last_fullsized":"2024-09-13T13:05:47.780747+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.103167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.103167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.050420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":221,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.020358+0000","last_change":"2024-09-13T13:01:04.195126+0000","last_active":"2024-09-13T13:05:53.020358+0000","last_peered":"2024-09-13T13:05:53.020358+0000","last_clean":"2024-09-13T13:05:53.020358+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:05:53.020358+0000","last_undegraded":"2024-09-13T13:05:53.020358+0000","last_fullsized":"2024-09-13T13:05:53.020358+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781532+0000","last_change":"2024-09-13T13:05:39.185953+0000","last_active":"2024-09-13T13:05:47.781532+0000","last_peered":"2024-09-13T13:05:47.781532+0000","last_clean":"2024-09-13T13:05:47.781532+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:05:47.781532+0000","last_undegraded":"2024-09-13T13:05:47.781532+0000","last_fullsized":"2024-09-13T13:05:47.781532+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:39.185892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:39.185892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:59.347239+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782548+0000","last_change":"2024-09-13T13:05:00.457611+0000","last_active":"2024-09-13T13:05:47.782548+0000","last_peered":"2024-09-13T13:05:47.782548+0000","last_clean":"2024-09-13T13:05:47.782548+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:05:47.782548+0000","last_undegraded":"2024-09-13T13:05:47.782548+0000","last_fullsized":"2024-09-13T13:05:47.782548+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:05:00.457562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:00.457562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:25.611525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":213,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.026071+0000","last_change":"2024-09-13T13:03:12.276224+0000","last_active":"2024-09-13T13:05:53.026071+0000","last_peered":"2024-09-13T13:05:53.026071+0000","last_clean":"2024-09-13T13:05:53.026071+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:05:53.026071+0000","last_undegraded":"2024-09-13T13:05:53.026071+0000","last_fullsized":"2024-09-13T13:05:53.026071+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:12.276136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.276136+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:47.787107+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":222,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.009516+0000","last_change":"2024-09-13T13:02:34.593511+0000","last_active":"2024-09-13T13:05:53.009516+0000","last_peered":"2024-09-13T13:05:53.009516+0000","last_clean":"2024-09-13T13:05:53.009516+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:05:53.009516+0000","last_undegraded":"2024-09-13T13:05:53.009516+0000","last_fullsized":"2024-09-13T13:05:53.009516+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:34.593431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:34.593431+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":219,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.011893+0000","last_change":"2024-09-13T13:04:02.765069+0000","last_active":"2024-09-13T13:05:53.011893+0000","last_peered":"2024-09-13T13:05:53.011893+0000","last_clean":"2024-09-13T13:05:53.011893+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:05:53.011893+0000","last_undegraded":"2024-09-13T13:05:53.011893+0000","last_fullsized":"2024-09-13T13:05:53.011893+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:46.604669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.604669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"155'14180","reported_seq":20107,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:55.878541+0000","last_change":"2024-09-13T13:04:49.780269+0000","last_active":"2024-09-13T13:05:55.878541+0000","last_peered":"2024-09-13T13:05:55.878541+0000","last_clean":"2024-09-13T13:05:55.878541+0000","last_became_active":"2024-09-13T13:04:49.765876+0000","last_became_peered":"2024-09-13T13:04:49.765876+0000","last_unstale":"2024-09-13T13:05:55.878541+0000","last_undegraded":"2024-09-13T13:05:55.878541+0000","last_fullsized":"2024-09-13T13:05:55.878541+0000","mapping_epoch":136,"log_start":"116'7848","ondisk_log_start":"116'7848","created":84,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":6332,"log_dups_size":0,"ondisk_log_size":6332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:55.075486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54919168,"num_objects":838,"num_object_clones":0,"num_object_copies":2514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14180,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2909,"num_bytes_recovered":147193856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":241,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.018962+0000","last_change":"2024-09-13T13:05:25.691233+0000","last_active":"2024-09-13T13:05:53.018962+0000","last_peered":"2024-09-13T13:05:53.018962+0000","last_clean":"2024-09-13T13:05:53.018962+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:05:53.018962+0000","last_undegraded":"2024-09-13T13:05:53.018962+0000","last_fullsized":"2024-09-13T13:05:53.018962+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:05:25.689487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:25.689487+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:53.347573+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":219,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.025131+0000","last_change":"2024-09-13T13:03:42.443399+0000","last_active":"2024-09-13T13:05:53.025131+0000","last_peered":"2024-09-13T13:05:53.025131+0000","last_clean":"2024-09-13T13:05:53.025131+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:05:53.025131+0000","last_undegraded":"2024-09-13T13:05:53.025131+0000","last_fullsized":"2024-09-13T13:05:53.025131+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:42.443339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:42.443339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":269,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781411+0000","last_change":"2024-09-13T13:02:32.013587+0000","last_active":"2024-09-13T13:05:47.781411+0000","last_peered":"2024-09-13T13:05:47.781411+0000","last_clean":"2024-09-13T13:05:47.781411+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:05:47.781411+0000","last_undegraded":"2024-09-13T13:05:47.781411+0000","last_fullsized":"2024-09-13T13:05:47.781411+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:32.013505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.013505+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:14.086911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796276+0000","last_change":"2024-09-13T13:04:48.684906+0000","last_active":"2024-09-13T13:05:47.796276+0000","last_peered":"2024-09-13T13:05:47.796276+0000","last_clean":"2024-09-13T13:05:47.796276+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:05:47.796276+0000","last_undegraded":"2024-09-13T13:05:47.796276+0000","last_fullsized":"2024-09-13T13:05:47.796276+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:48.684864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.684864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.491555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"155'14067","reported_seq":14298,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.223211+0000","last_change":"2024-09-13T13:03:41.936543+0000","last_active":"2024-09-13T13:05:53.223211+0000","last_peered":"2024-09-13T13:05:53.223211+0000","last_clean":"2024-09-13T13:05:53.223211+0000","last_became_active":"2024-09-13T13:02:12.369723+0000","last_became_peered":"2024-09-13T13:02:12.369723+0000","last_unstale":"2024-09-13T13:05:53.223211+0000","last_undegraded":"2024-09-13T13:05:53.223211+0000","last_fullsized":"2024-09-13T13:05:53.223211+0000","mapping_epoch":84,"log_start":"130'9400","ondisk_log_start":"130'9400","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6909","last_scrub_stamp":"2024-09-13T13:03:41.936218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:03:41.936218+0000","objects_scrubbed":6836,"log_size":4667,"log_dups_size":0,"ondisk_log_size":4667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69533696,"num_objects":1061,"num_object_clones":0,"num_object_copies":3183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14067,"num_write_kb":484096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":239,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.024899+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:05:53.024899+0000","last_peered":"2024-09-13T13:05:53.024899+0000","last_clean":"2024-09-13T13:05:53.024899+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:05:53.024899+0000","last_undegraded":"2024-09-13T13:05:53.024899+0000","last_fullsized":"2024-09-13T13:05:53.024899+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":217,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.023025+0000","last_change":"2024-09-13T13:03:18.516327+0000","last_active":"2024-09-13T13:05:53.023025+0000","last_peered":"2024-09-13T13:05:53.023025+0000","last_clean":"2024-09-13T13:05:53.023025+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:05:53.023025+0000","last_undegraded":"2024-09-13T13:05:53.023025+0000","last_fullsized":"2024-09-13T13:05:53.023025+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:18.516251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.516251+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":210,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780891+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:05:47.780891+0000","last_peered":"2024-09-13T13:05:47.780891+0000","last_clean":"2024-09-13T13:05:47.780891+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:05:47.780891+0000","last_undegraded":"2024-09-13T13:05:47.780891+0000","last_fullsized":"2024-09-13T13:05:47.780891+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:40.320454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":198,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780621+0000","last_change":"2024-09-13T13:04:53.006515+0000","last_active":"2024-09-13T13:05:47.780621+0000","last_peered":"2024-09-13T13:05:47.780621+0000","last_clean":"2024-09-13T13:05:47.780621+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:05:47.780621+0000","last_undegraded":"2024-09-13T13:05:47.780621+0000","last_fullsized":"2024-09-13T13:05:47.780621+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:53.006454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.006454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.531241+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"155'14002","reported_seq":14162,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.261665+0000","last_change":"2024-09-13T13:03:45.761068+0000","last_active":"2024-09-13T13:05:53.261665+0000","last_peered":"2024-09-13T13:05:53.261665+0000","last_clean":"2024-09-13T13:05:53.261665+0000","last_became_active":"2024-09-13T13:02:12.373492+0000","last_became_peered":"2024-09-13T13:02:12.373492+0000","last_unstale":"2024-09-13T13:05:53.261665+0000","last_undegraded":"2024-09-13T13:05:53.261665+0000","last_fullsized":"2024-09-13T13:05:53.261665+0000","mapping_epoch":84,"log_start":"129'9300","ondisk_log_start":"129'9300","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":299,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69074944,"num_objects":1054,"num_object_clones":0,"num_object_copies":3162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14002,"num_write_kb":481792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":246,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780896+0000","last_change":"2024-09-13T13:05:33.178442+0000","last_active":"2024-09-13T13:05:47.780896+0000","last_peered":"2024-09-13T13:05:47.780896+0000","last_clean":"2024-09-13T13:05:47.780896+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:05:47.780896+0000","last_undegraded":"2024-09-13T13:05:47.780896+0000","last_fullsized":"2024-09-13T13:05:47.780896+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:05:33.178370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.178370+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:57.356161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":212,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.009957+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:05:53.009957+0000","last_peered":"2024-09-13T13:05:53.009957+0000","last_clean":"2024-09-13T13:05:53.009957+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:05:53.009957+0000","last_undegraded":"2024-09-13T13:05:53.009957+0000","last_fullsized":"2024-09-13T13:05:53.009957+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":262,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.024677+0000","last_change":"2024-09-13T13:01:06.211211+0000","last_active":"2024-09-13T13:05:53.024677+0000","last_peered":"2024-09-13T13:05:53.024677+0000","last_clean":"2024-09-13T13:05:53.024677+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:05:53.024677+0000","last_undegraded":"2024-09-13T13:05:53.024677+0000","last_fullsized":"2024-09-13T13:05:53.024677+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:47.764924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.783010+0000","last_change":"2024-09-13T13:04:57.439116+0000","last_active":"2024-09-13T13:05:47.783010+0000","last_peered":"2024-09-13T13:05:47.783010+0000","last_clean":"2024-09-13T13:05:47.783010+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:05:47.783010+0000","last_undegraded":"2024-09-13T13:05:47.783010+0000","last_fullsized":"2024-09-13T13:05:47.783010+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.439053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.439053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:04.992285+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"154'14015","reported_seq":20013,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:52.895112+0000","last_change":"2024-09-13T13:05:05.014474+0000","last_active":"2024-09-13T13:05:52.895112+0000","last_peered":"2024-09-13T13:05:52.895112+0000","last_clean":"2024-09-13T13:05:52.895112+0000","last_became_active":"2024-09-13T13:04:25.922485+0000","last_became_peered":"2024-09-13T13:04:25.922485+0000","last_unstale":"2024-09-13T13:05:52.895112+0000","last_undegraded":"2024-09-13T13:05:52.895112+0000","last_fullsized":"2024-09-13T13:05:52.895112+0000","mapping_epoch":128,"log_start":"126'9200","ondisk_log_start":"126'9200","created":84,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'11284","last_scrub_stamp":"2024-09-13T13:05:05.014171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:05.014171+0000","objects_scrubbed":3783,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:18.664629+0000","scrub_duration":934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67174400,"num_objects":1025,"num_object_clones":0,"num_object_copies":3075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14015,"num_write_kb":481280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2891,"num_bytes_recovered":148570112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":215,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.019489+0000","last_change":"2024-09-13T13:01:04.199011+0000","last_active":"2024-09-13T13:05:53.019489+0000","last_peered":"2024-09-13T13:05:53.019489+0000","last_clean":"2024-09-13T13:05:53.019489+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:05:53.019489+0000","last_undegraded":"2024-09-13T13:05:53.019489+0000","last_fullsized":"2024-09-13T13:05:53.019489+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.733108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796589+0000","last_change":"2024-09-13T13:05:04.743102+0000","last_active":"2024-09-13T13:05:47.796589+0000","last_peered":"2024-09-13T13:05:47.796589+0000","last_clean":"2024-09-13T13:05:47.796589+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:05:47.796589+0000","last_undegraded":"2024-09-13T13:05:47.796589+0000","last_fullsized":"2024-09-13T13:05:47.796589+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:04.743025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:04.743025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:32.082984+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":276,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.008801+0000","last_change":"2024-09-13T13:04:02.764350+0000","last_active":"2024-09-13T13:05:53.008801+0000","last_peered":"2024-09-13T13:05:53.008801+0000","last_clean":"2024-09-13T13:05:53.008801+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:05:53.008801+0000","last_undegraded":"2024-09-13T13:05:53.008801+0000","last_fullsized":"2024-09-13T13:05:53.008801+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.140183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":199,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.007719+0000","last_change":"2024-09-13T13:02:43.558487+0000","last_active":"2024-09-13T13:05:53.007719+0000","last_peered":"2024-09-13T13:05:53.007719+0000","last_clean":"2024-09-13T13:05:53.007719+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:05:53.007719+0000","last_undegraded":"2024-09-13T13:05:53.007719+0000","last_fullsized":"2024-09-13T13:05:53.007719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:43.558418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:43.558418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"154'13895","reported_seq":14047,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:52.906563+0000","last_change":"2024-09-13T13:02:12.379536+0000","last_active":"2024-09-13T13:05:52.906563+0000","last_peered":"2024-09-13T13:05:52.906563+0000","last_clean":"2024-09-13T13:05:52.906563+0000","last_became_active":"2024-09-13T13:02:12.379226+0000","last_became_peered":"2024-09-13T13:02:12.379226+0000","last_unstale":"2024-09-13T13:05:52.906563+0000","last_undegraded":"2024-09-13T13:05:52.906563+0000","last_fullsized":"2024-09-13T13:05:52.906563+0000","mapping_epoch":84,"log_start":"120'8100","ondisk_log_start":"120'8100","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5795,"log_dups_size":0,"ondisk_log_size":5795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.567939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68485120,"num_objects":1045,"num_object_clones":0,"num_object_copies":3135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13895,"num_write_kb":478080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":243,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.008899+0000","last_change":"2024-09-13T13:02:29.532879+0000","last_active":"2024-09-13T13:05:53.008899+0000","last_peered":"2024-09-13T13:05:53.008899+0000","last_clean":"2024-09-13T13:05:53.008899+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:05:53.008899+0000","last_undegraded":"2024-09-13T13:05:53.008899+0000","last_fullsized":"2024-09-13T13:05:53.008899+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:29.532779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.532779+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":204,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.022007+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:05:53.022007+0000","last_peered":"2024-09-13T13:05:53.022007+0000","last_clean":"2024-09-13T13:05:53.022007+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:05:53.022007+0000","last_undegraded":"2024-09-13T13:05:53.022007+0000","last_fullsized":"2024-09-13T13:05:53.022007+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:20.738953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":164,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.011416+0000","last_change":"2024-09-13T13:04:02.765838+0000","last_active":"2024-09-13T13:05:53.011416+0000","last_peered":"2024-09-13T13:05:53.011416+0000","last_clean":"2024-09-13T13:05:53.011416+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:05:53.011416+0000","last_undegraded":"2024-09-13T13:05:53.011416+0000","last_fullsized":"2024-09-13T13:05:53.011416+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.563708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.563708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":219,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.022396+0000","last_change":"2024-09-13T13:05:30.829125+0000","last_active":"2024-09-13T13:05:53.022396+0000","last_peered":"2024-09-13T13:05:53.022396+0000","last_clean":"2024-09-13T13:05:53.022396+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:05:53.022396+0000","last_undegraded":"2024-09-13T13:05:53.022396+0000","last_fullsized":"2024-09-13T13:05:53.022396+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:30.828810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:30.828810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:43.724154+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"155'14296","reported_seq":20352,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:55.865478+0000","last_change":"2024-09-13T13:05:42.806200+0000","last_active":"2024-09-13T13:05:55.865478+0000","last_peered":"2024-09-13T13:05:55.865478+0000","last_clean":"2024-09-13T13:05:55.865478+0000","last_became_active":"2024-09-13T13:05:42.780925+0000","last_became_peered":"2024-09-13T13:05:42.780925+0000","last_unstale":"2024-09-13T13:05:55.865478+0000","last_undegraded":"2024-09-13T13:05:55.865478+0000","last_fullsized":"2024-09-13T13:05:55.865478+0000","mapping_epoch":151,"log_start":"116'8000","ondisk_log_start":"116'8000","created":84,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":6296,"log_dups_size":0,"ondisk_log_size":6296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57409536,"num_objects":876,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14296,"num_write_kb":485504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2944,"num_bytes_recovered":148635648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":323,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.021854+0000","last_change":"2024-09-13T13:05:43.787063+0000","last_active":"2024-09-13T13:05:53.021854+0000","last_peered":"2024-09-13T13:05:53.021854+0000","last_clean":"2024-09-13T13:05:53.021854+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:05:53.021854+0000","last_undegraded":"2024-09-13T13:05:53.021854+0000","last_fullsized":"2024-09-13T13:05:53.021854+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:05:43.786273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:43.786273+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:05.440008+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":229,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781077+0000","last_change":"2024-09-13T13:05:45.178086+0000","last_active":"2024-09-13T13:05:47.781077+0000","last_peered":"2024-09-13T13:05:47.781077+0000","last_clean":"2024-09-13T13:05:47.781077+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:05:47.781077+0000","last_undegraded":"2024-09-13T13:05:47.781077+0000","last_fullsized":"2024-09-13T13:05:47.781077+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:45.178025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.178025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:02.693302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":228,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.797875+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:05:47.797875+0000","last_peered":"2024-09-13T13:05:47.797875+0000","last_clean":"2024-09-13T13:05:47.797875+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:05:47.797875+0000","last_undegraded":"2024-09-13T13:05:47.797875+0000","last_fullsized":"2024-09-13T13:05:47.797875+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:26.941315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.009663+0000","last_change":"2024-09-13T13:02:35.566666+0000","last_active":"2024-09-13T13:05:53.009663+0000","last_peered":"2024-09-13T13:05:53.009663+0000","last_clean":"2024-09-13T13:05:53.009663+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:05:53.009663+0000","last_undegraded":"2024-09-13T13:05:53.009663+0000","last_fullsized":"2024-09-13T13:05:53.009663+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:35.566602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:35.566602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":199,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.024278+0000","last_change":"2024-09-13T13:04:01.406226+0000","last_active":"2024-09-13T13:05:53.024278+0000","last_peered":"2024-09-13T13:05:53.024278+0000","last_clean":"2024-09-13T13:05:53.024278+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:05:53.024278+0000","last_undegraded":"2024-09-13T13:05:53.024278+0000","last_fullsized":"2024-09-13T13:05:53.024278+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:01.406167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:01.406167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"155'14107","reported_seq":14261,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.497801+0000","last_change":"2024-09-13T13:02:12.380398+0000","last_active":"2024-09-13T13:05:53.497801+0000","last_peered":"2024-09-13T13:05:53.497801+0000","last_clean":"2024-09-13T13:05:53.497801+0000","last_became_active":"2024-09-13T13:02:12.379331+0000","last_became_peered":"2024-09-13T13:02:12.379331+0000","last_unstale":"2024-09-13T13:05:53.497801+0000","last_undegraded":"2024-09-13T13:05:53.497801+0000","last_fullsized":"2024-09-13T13:05:53.497801+0000","mapping_epoch":84,"log_start":"122'8500","ondisk_log_start":"122'8500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64159744,"num_objects":979,"num_object_clones":0,"num_object_copies":2937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14107,"num_write_kb":482752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":235,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.019053+0000","last_change":"2024-09-13T13:03:14.271569+0000","last_active":"2024-09-13T13:05:53.019053+0000","last_peered":"2024-09-13T13:05:53.019053+0000","last_clean":"2024-09-13T13:05:53.019053+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:05:53.019053+0000","last_undegraded":"2024-09-13T13:05:53.019053+0000","last_fullsized":"2024-09-13T13:05:53.019053+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:03:14.271480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.271480+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:48.789856+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":231,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.022342+0000","last_change":"2024-09-13T13:05:36.692159+0000","last_active":"2024-09-13T13:05:53.022342+0000","last_peered":"2024-09-13T13:05:53.022342+0000","last_clean":"2024-09-13T13:05:53.022342+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:05:53.022342+0000","last_undegraded":"2024-09-13T13:05:53.022342+0000","last_fullsized":"2024-09-13T13:05:53.022342+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:36.692092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:36.692092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:00.057550+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":227,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.021030+0000","last_change":"2024-09-13T13:05:34.321160+0000","last_active":"2024-09-13T13:05:53.021030+0000","last_peered":"2024-09-13T13:05:53.021030+0000","last_clean":"2024-09-13T13:05:53.021030+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:05:53.021030+0000","last_undegraded":"2024-09-13T13:05:53.021030+0000","last_fullsized":"2024-09-13T13:05:53.021030+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:05:34.321081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.321081+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:37.239957+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":204,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.021281+0000","last_change":"2024-09-13T13:03:14.282866+0000","last_active":"2024-09-13T13:05:53.021281+0000","last_peered":"2024-09-13T13:05:53.021281+0000","last_clean":"2024-09-13T13:05:53.021281+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:05:53.021281+0000","last_undegraded":"2024-09-13T13:05:53.021281+0000","last_fullsized":"2024-09-13T13:05:53.021281+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.349860+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782563+0000","last_change":"2024-09-13T13:04:02.751582+0000","last_active":"2024-09-13T13:05:47.782563+0000","last_peered":"2024-09-13T13:05:47.782563+0000","last_clean":"2024-09-13T13:05:47.782563+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:05:47.782563+0000","last_undegraded":"2024-09-13T13:05:47.782563+0000","last_fullsized":"2024-09-13T13:05:47.782563+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.994729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.994729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.398308+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"154'14339","reported_seq":14595,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:52.893589+0000","last_change":"2024-09-13T13:05:35.785111+0000","last_active":"2024-09-13T13:05:52.893589+0000","last_peered":"2024-09-13T13:05:52.893589+0000","last_clean":"2024-09-13T13:05:52.893589+0000","last_became_active":"2024-09-13T13:02:12.379779+0000","last_became_peered":"2024-09-13T13:02:12.379779+0000","last_unstale":"2024-09-13T13:05:52.893589+0000","last_undegraded":"2024-09-13T13:05:52.893589+0000","last_fullsized":"2024-09-13T13:05:52.893589+0000","mapping_epoch":84,"log_start":"124'9100","ondisk_log_start":"124'9100","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'13287","last_scrub_stamp":"2024-09-13T13:05:35.785022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.785022+0000","objects_scrubbed":2122,"log_size":5239,"log_dups_size":0,"ondisk_log_size":5239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:52.011396+0000","scrub_duration":635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68616192,"num_objects":1047,"num_object_clones":0,"num_object_copies":3141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14339,"num_write_kb":492352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":225,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.009612+0000","last_change":"2024-09-13T13:03:18.514436+0000","last_active":"2024-09-13T13:05:53.009612+0000","last_peered":"2024-09-13T13:05:53.009612+0000","last_clean":"2024-09-13T13:05:53.009612+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:05:53.009612+0000","last_undegraded":"2024-09-13T13:05:53.009612+0000","last_fullsized":"2024-09-13T13:05:53.009612+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:03:18.514360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:18.514360+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":211,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.008399+0000","last_change":"2024-09-13T13:02:37.616313+0000","last_active":"2024-09-13T13:05:53.008399+0000","last_peered":"2024-09-13T13:05:53.008399+0000","last_clean":"2024-09-13T13:05:53.008399+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:05:53.008399+0000","last_undegraded":"2024-09-13T13:05:53.008399+0000","last_fullsized":"2024-09-13T13:05:53.008399+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:37.616224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.616224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":503,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:51.767031+0000","last_change":"2024-09-13T13:04:56.430717+0000","last_active":"2024-09-13T13:05:51.767031+0000","last_peered":"2024-09-13T13:05:51.767031+0000","last_clean":"2024-09-13T13:05:51.767031+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:05:51.767031+0000","last_undegraded":"2024-09-13T13:05:51.767031+0000","last_fullsized":"2024-09-13T13:05:51.767031+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:04:56.430670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.430670+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:59.639039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":285,"num_read_kb":280,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":201,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796777+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:05:47.796777+0000","last_peered":"2024-09-13T13:05:47.796777+0000","last_clean":"2024-09-13T13:05:47.796777+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:05:47.796777+0000","last_undegraded":"2024-09-13T13:05:47.796777+0000","last_fullsized":"2024-09-13T13:05:47.796777+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:19.946825+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":208,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.021533+0000","last_change":"2024-09-13T13:05:41.753575+0000","last_active":"2024-09-13T13:05:53.021533+0000","last_peered":"2024-09-13T13:05:53.021533+0000","last_clean":"2024-09-13T13:05:53.021533+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:05:53.021533+0000","last_undegraded":"2024-09-13T13:05:53.021533+0000","last_fullsized":"2024-09-13T13:05:53.021533+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:41.753502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:41.753502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:54.892291+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"155'14139","reported_seq":14292,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.479847+0000","last_change":"2024-09-13T13:02:12.379556+0000","last_active":"2024-09-13T13:05:53.479847+0000","last_peered":"2024-09-13T13:05:53.479847+0000","last_clean":"2024-09-13T13:05:53.479847+0000","last_became_active":"2024-09-13T13:02:12.379255+0000","last_became_peered":"2024-09-13T13:02:12.379255+0000","last_unstale":"2024-09-13T13:05:53.479847+0000","last_undegraded":"2024-09-13T13:05:53.479847+0000","last_fullsized":"2024-09-13T13:05:53.479847+0000","mapping_epoch":84,"log_start":"122'8500","ondisk_log_start":"122'8500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64684032,"num_objects":987,"num_object_clones":0,"num_object_copies":2961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14139,"num_write_kb":484032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":155,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.011638+0000","last_change":"2024-09-13T13:04:02.786042+0000","last_active":"2024-09-13T13:05:53.011638+0000","last_peered":"2024-09-13T13:05:53.011638+0000","last_clean":"2024-09-13T13:05:53.011638+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:05:53.011638+0000","last_undegraded":"2024-09-13T13:05:53.011638+0000","last_fullsized":"2024-09-13T13:05:53.011638+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:01:03.168885+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:51.016610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":222,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.019731+0000","last_change":"2024-09-13T13:04:02.765370+0000","last_active":"2024-09-13T13:05:53.019731+0000","last_peered":"2024-09-13T13:05:53.019731+0000","last_clean":"2024-09-13T13:05:53.019731+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:05:53.019731+0000","last_undegraded":"2024-09-13T13:05:53.019731+0000","last_fullsized":"2024-09-13T13:05:53.019731+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:04.228002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:04.228002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":265,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782584+0000","last_change":"2024-09-13T13:05:28.373960+0000","last_active":"2024-09-13T13:05:47.782584+0000","last_peered":"2024-09-13T13:05:47.782584+0000","last_clean":"2024-09-13T13:05:47.782584+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:05:47.782584+0000","last_undegraded":"2024-09-13T13:05:47.782584+0000","last_fullsized":"2024-09-13T13:05:47.782584+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:05:28.373902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:05:28.373902+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:33.407719+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":201,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782690+0000","last_change":"2024-09-13T13:04:51.395273+0000","last_active":"2024-09-13T13:05:47.782690+0000","last_peered":"2024-09-13T13:05:47.782690+0000","last_clean":"2024-09-13T13:05:47.782690+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:05:47.782690+0000","last_undegraded":"2024-09-13T13:05:47.782690+0000","last_fullsized":"2024-09-13T13:05:47.782690+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:51.395204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:51.395204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:56.927380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":205,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796665+0000","last_change":"2024-09-13T13:05:31.564860+0000","last_active":"2024-09-13T13:05:47.796665+0000","last_peered":"2024-09-13T13:05:47.796665+0000","last_clean":"2024-09-13T13:05:47.796665+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:05:47.796665+0000","last_undegraded":"2024-09-13T13:05:47.796665+0000","last_fullsized":"2024-09-13T13:05:47.796665+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:31.564783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:31.564783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:41.885010+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"155'14389","reported_seq":20577,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:55.828881+0000","last_change":"2024-09-13T13:05:15.349567+0000","last_active":"2024-09-13T13:05:55.828881+0000","last_peered":"2024-09-13T13:05:55.828881+0000","last_clean":"2024-09-13T13:05:55.828881+0000","last_became_active":"2024-09-13T13:05:15.326923+0000","last_became_peered":"2024-09-13T13:05:15.326923+0000","last_unstale":"2024-09-13T13:05:55.828881+0000","last_undegraded":"2024-09-13T13:05:55.828881+0000","last_fullsized":"2024-09-13T13:05:55.828881+0000","mapping_epoch":143,"log_start":"116'8085","ondisk_log_start":"116'8085","created":84,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":6304,"log_dups_size":0,"ondisk_log_size":6304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59768856,"num_objects":913,"num_object_clones":0,"num_object_copies":2739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14389,"num_write_kb":489601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3008,"num_bytes_recovered":155058200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":236,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781461+0000","last_change":"2024-09-13T13:04:02.832992+0000","last_active":"2024-09-13T13:05:47.781461+0000","last_peered":"2024-09-13T13:05:47.781461+0000","last_clean":"2024-09-13T13:05:47.781461+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:05:47.781461+0000","last_undegraded":"2024-09-13T13:05:47.781461+0000","last_fullsized":"2024-09-13T13:05:47.781461+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:04:02.832935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.832935+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:47.117698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":210,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781161+0000","last_change":"2024-09-13T13:02:45.015634+0000","last_active":"2024-09-13T13:05:47.781161+0000","last_peered":"2024-09-13T13:05:47.781161+0000","last_clean":"2024-09-13T13:05:47.781161+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:05:47.781161+0000","last_undegraded":"2024-09-13T13:05:47.781161+0000","last_fullsized":"2024-09-13T13:05:47.781161+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.015577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.015577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.095405+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":212,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796531+0000","last_change":"2024-09-13T13:04:47.680651+0000","last_active":"2024-09-13T13:05:47.796531+0000","last_peered":"2024-09-13T13:05:47.796531+0000","last_clean":"2024-09-13T13:05:47.796531+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:05:47.796531+0000","last_undegraded":"2024-09-13T13:05:47.796531+0000","last_fullsized":"2024-09-13T13:05:47.796531+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.680579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.680579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:07.566204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.022920+0000","last_change":"2024-09-13T13:04:02.757378+0000","last_active":"2024-09-13T13:05:53.022920+0000","last_peered":"2024-09-13T13:05:53.022920+0000","last_clean":"2024-09-13T13:05:53.022920+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:05:53.022920+0000","last_undegraded":"2024-09-13T13:05:53.022920+0000","last_fullsized":"2024-09-13T13:05:53.022920+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.213470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.213470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"155'13685","reported_seq":13964,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.105766+0000","last_change":"2024-09-13T13:05:03.355522+0000","last_active":"2024-09-13T13:05:53.105766+0000","last_peered":"2024-09-13T13:05:53.105766+0000","last_clean":"2024-09-13T13:05:53.105766+0000","last_became_active":"2024-09-13T13:02:12.375509+0000","last_became_peered":"2024-09-13T13:02:12.375509+0000","last_unstale":"2024-09-13T13:05:53.105766+0000","last_undegraded":"2024-09-13T13:05:53.105766+0000","last_fullsized":"2024-09-13T13:05:53.105766+0000","mapping_epoch":84,"log_start":"114'6800","ondisk_log_start":"114'6800","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'10840","last_scrub_stamp":"2024-09-13T13:05:03.354789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:03.354789+0000","objects_scrubbed":3912,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.025217+0000","scrub_duration":2101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66387968,"num_objects":1013,"num_object_clones":0,"num_object_copies":3039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13685,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":242,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781738+0000","last_change":"2024-09-13T13:05:34.166464+0000","last_active":"2024-09-13T13:05:47.781738+0000","last_peered":"2024-09-13T13:05:47.781738+0000","last_clean":"2024-09-13T13:05:47.781738+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:05:47.781738+0000","last_undegraded":"2024-09-13T13:05:47.781738+0000","last_fullsized":"2024-09-13T13:05:47.781738+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:40.189820+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":188,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.011669+0000","last_change":"2024-09-13T13:05:44.119947+0000","last_active":"2024-09-13T13:05:53.011669+0000","last_peered":"2024-09-13T13:05:53.011669+0000","last_clean":"2024-09-13T13:05:53.011669+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:05:53.011669+0000","last_undegraded":"2024-09-13T13:05:53.011669+0000","last_fullsized":"2024-09-13T13:05:53.011669+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:44.119890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.119890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:50.534493+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":196,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.021619+0000","last_change":"2024-09-13T13:01:06.208593+0000","last_active":"2024-09-13T13:05:53.021619+0000","last_peered":"2024-09-13T13:05:53.021619+0000","last_clean":"2024-09-13T13:05:53.021619+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:05:53.021619+0000","last_undegraded":"2024-09-13T13:05:53.021619+0000","last_fullsized":"2024-09-13T13:05:53.021619+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:22.718840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":208,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.020833+0000","last_change":"2024-09-13T13:05:35.310768+0000","last_active":"2024-09-13T13:05:53.020833+0000","last_peered":"2024-09-13T13:05:53.020833+0000","last_clean":"2024-09-13T13:05:53.020833+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:05:53.020833+0000","last_undegraded":"2024-09-13T13:05:53.020833+0000","last_fullsized":"2024-09-13T13:05:53.020833+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:35.310700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.310700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:56.914781+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"155'14121","reported_seq":14350,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.466414+0000","last_change":"2024-09-13T13:04:02.904392+0000","last_active":"2024-09-13T13:05:53.466414+0000","last_peered":"2024-09-13T13:05:53.466414+0000","last_clean":"2024-09-13T13:05:53.466414+0000","last_became_active":"2024-09-13T13:02:12.380229+0000","last_became_peered":"2024-09-13T13:02:12.380229+0000","last_unstale":"2024-09-13T13:05:53.466414+0000","last_undegraded":"2024-09-13T13:05:53.466414+0000","last_fullsized":"2024-09-13T13:05:53.466414+0000","mapping_epoch":84,"log_start":"122'8500","ondisk_log_start":"122'8500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8189","last_scrub_stamp":"2024-09-13T13:04:02.904297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.904297+0000","objects_scrubbed":6969,"log_size":5621,"log_dups_size":0,"ondisk_log_size":5621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67567616,"num_objects":1031,"num_object_clones":0,"num_object_copies":3093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14121,"num_write_kb":484864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":232,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781552+0000","last_change":"2024-09-13T13:04:59.091852+0000","last_active":"2024-09-13T13:05:47.781552+0000","last_peered":"2024-09-13T13:05:47.781552+0000","last_clean":"2024-09-13T13:05:47.781552+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:05:47.781552+0000","last_undegraded":"2024-09-13T13:05:47.781552+0000","last_fullsized":"2024-09-13T13:05:47.781552+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:04:59.091751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.091751+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:17.396141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":219,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.024097+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:05:53.024097+0000","last_peered":"2024-09-13T13:05:53.024097+0000","last_clean":"2024-09-13T13:05:53.024097+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:05:53.024097+0000","last_undegraded":"2024-09-13T13:05:53.024097+0000","last_fullsized":"2024-09-13T13:05:53.024097+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":279,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.027256+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:05:53.027256+0000","last_peered":"2024-09-13T13:05:53.027256+0000","last_clean":"2024-09-13T13:05:53.027256+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:05:53.027256+0000","last_undegraded":"2024-09-13T13:05:53.027256+0000","last_fullsized":"2024-09-13T13:05:53.027256+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:57.041771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":198,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781195+0000","last_change":"2024-09-13T13:02:33.997780+0000","last_active":"2024-09-13T13:05:47.781195+0000","last_peered":"2024-09-13T13:05:47.781195+0000","last_clean":"2024-09-13T13:05:47.781195+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:05:47.781195+0000","last_undegraded":"2024-09-13T13:05:47.781195+0000","last_fullsized":"2024-09-13T13:05:47.781195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.997740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.997740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.184374+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"155'14137","reported_seq":14413,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.109706+0000","last_change":"2024-09-13T13:05:08.464991+0000","last_active":"2024-09-13T13:05:53.109706+0000","last_peered":"2024-09-13T13:05:53.109706+0000","last_clean":"2024-09-13T13:05:53.109706+0000","last_became_active":"2024-09-13T13:02:12.375310+0000","last_became_peered":"2024-09-13T13:02:12.375310+0000","last_unstale":"2024-09-13T13:05:53.109706+0000","last_undegraded":"2024-09-13T13:05:53.109706+0000","last_fullsized":"2024-09-13T13:05:53.109706+0000","mapping_epoch":84,"log_start":"115'7300","ondisk_log_start":"115'7300","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'11530","last_scrub_stamp":"2024-09-13T13:05:08.464709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:08.464709+0000","objects_scrubbed":3682,"log_size":6837,"log_dups_size":0,"ondisk_log_size":6837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:11.110516+0000","scrub_duration":1226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68091904,"num_objects":1039,"num_object_clones":0,"num_object_copies":3117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14137,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":223,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.008843+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:05:53.008843+0000","last_peered":"2024-09-13T13:05:53.008843+0000","last_clean":"2024-09-13T13:05:53.008843+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:05:53.008843+0000","last_undegraded":"2024-09-13T13:05:53.008843+0000","last_fullsized":"2024-09-13T13:05:53.008843+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":228,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796688+0000","last_change":"2024-09-13T13:05:37.520655+0000","last_active":"2024-09-13T13:05:47.796688+0000","last_peered":"2024-09-13T13:05:47.796688+0000","last_clean":"2024-09-13T13:05:47.796688+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:05:47.796688+0000","last_undegraded":"2024-09-13T13:05:47.796688+0000","last_fullsized":"2024-09-13T13:05:47.796688+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:37.520603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:37.520603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:56.825070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":212,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.020595+0000","last_change":"2024-09-13T13:04:56.183494+0000","last_active":"2024-09-13T13:05:53.020595+0000","last_peered":"2024-09-13T13:05:53.020595+0000","last_clean":"2024-09-13T13:05:53.020595+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:05:53.020595+0000","last_undegraded":"2024-09-13T13:05:53.020595+0000","last_fullsized":"2024-09-13T13:05:53.020595+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:56.183290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:56.183290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:07.795607+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":191,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.008316+0000","last_change":"2024-09-13T13:01:08.211229+0000","last_active":"2024-09-13T13:05:53.008316+0000","last_peered":"2024-09-13T13:05:53.008316+0000","last_clean":"2024-09-13T13:05:53.008316+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:05:53.008316+0000","last_undegraded":"2024-09-13T13:05:53.008316+0000","last_fullsized":"2024-09-13T13:05:53.008316+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"155'14314","reported_seq":14468,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.476307+0000","last_change":"2024-09-13T13:02:12.379755+0000","last_active":"2024-09-13T13:05:53.476307+0000","last_peered":"2024-09-13T13:05:53.476307+0000","last_clean":"2024-09-13T13:05:53.476307+0000","last_became_active":"2024-09-13T13:02:12.379442+0000","last_became_peered":"2024-09-13T13:02:12.379442+0000","last_unstale":"2024-09-13T13:05:53.476307+0000","last_undegraded":"2024-09-13T13:05:53.476307+0000","last_fullsized":"2024-09-13T13:05:53.476307+0000","mapping_epoch":84,"log_start":"122'8700","ondisk_log_start":"122'8700","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":0,"log_size":5614,"log_dups_size":0,"ondisk_log_size":5614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65011712,"num_objects":992,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14314,"num_write_kb":489792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":199,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.011444+0000","last_change":"2024-09-13T13:04:02.760376+0000","last_active":"2024-09-13T13:05:53.011444+0000","last_peered":"2024-09-13T13:05:53.011444+0000","last_clean":"2024-09-13T13:05:53.011444+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:05:53.011444+0000","last_undegraded":"2024-09-13T13:05:53.011444+0000","last_fullsized":"2024-09-13T13:05:53.011444+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:02:42.593284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.593284+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":220,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782874+0000","last_change":"2024-09-13T13:04:02.759538+0000","last_active":"2024-09-13T13:05:47.782874+0000","last_peered":"2024-09-13T13:05:47.782874+0000","last_clean":"2024-09-13T13:05:47.782874+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:05:47.782874+0000","last_undegraded":"2024-09-13T13:05:47.782874+0000","last_fullsized":"2024-09-13T13:05:47.782874+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:29.012312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:29.012312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:13.429060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":212,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.021567+0000","last_change":"2024-09-13T13:04:47.927556+0000","last_active":"2024-09-13T13:05:53.021567+0000","last_peered":"2024-09-13T13:05:53.021567+0000","last_clean":"2024-09-13T13:05:53.021567+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:05:53.021567+0000","last_undegraded":"2024-09-13T13:05:53.021567+0000","last_fullsized":"2024-09-13T13:05:53.021567+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.927432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.927432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.726893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782491+0000","last_change":"2024-09-13T13:05:11.420568+0000","last_active":"2024-09-13T13:05:47.782491+0000","last_peered":"2024-09-13T13:05:47.782491+0000","last_clean":"2024-09-13T13:05:47.782491+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:05:47.782491+0000","last_undegraded":"2024-09-13T13:05:47.782491+0000","last_fullsized":"2024-09-13T13:05:47.782491+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:11.420503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:11.420503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:12.545408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"155'14145","reported_seq":14351,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.255381+0000","last_change":"2024-09-13T13:03:45.723433+0000","last_active":"2024-09-13T13:05:53.255381+0000","last_peered":"2024-09-13T13:05:53.255381+0000","last_clean":"2024-09-13T13:05:53.255381+0000","last_became_active":"2024-09-13T13:02:12.376334+0000","last_became_peered":"2024-09-13T13:02:12.376334+0000","last_unstale":"2024-09-13T13:05:53.255381+0000","last_undegraded":"2024-09-13T13:05:53.255381+0000","last_fullsized":"2024-09-13T13:05:53.255381+0000","mapping_epoch":84,"log_start":"130'9500","ondisk_log_start":"130'9500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:02:11.355519+0000","objects_scrubbed":4774,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70451200,"num_objects":1075,"num_object_clones":0,"num_object_copies":3225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14145,"num_write_kb":487040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":229,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781112+0000","last_change":"2024-09-13T13:02:28.990089+0000","last_active":"2024-09-13T13:05:47.781112+0000","last_peered":"2024-09-13T13:05:47.781112+0000","last_clean":"2024-09-13T13:05:47.781112+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:05:47.781112+0000","last_undegraded":"2024-09-13T13:05:47.781112+0000","last_fullsized":"2024-09-13T13:05:47.781112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:02:28.990005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.990005+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.110320+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":225,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.784914+0000","last_change":"2024-09-13T13:05:29.345863+0000","last_active":"2024-09-13T13:05:47.784914+0000","last_peered":"2024-09-13T13:05:47.784914+0000","last_clean":"2024-09-13T13:05:47.784914+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:05:47.784914+0000","last_undegraded":"2024-09-13T13:05:47.784914+0000","last_fullsized":"2024-09-13T13:05:47.784914+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.345796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.345796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.394429+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":204,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.031446+0000","last_change":"2024-09-13T13:05:29.218178+0000","last_active":"2024-09-13T13:05:53.031446+0000","last_peered":"2024-09-13T13:05:53.031446+0000","last_clean":"2024-09-13T13:05:53.031446+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:05:53.031446+0000","last_undegraded":"2024-09-13T13:05:53.031446+0000","last_fullsized":"2024-09-13T13:05:53.031446+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.218119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.218119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:30.944093+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":199,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.023774+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:05:53.023774+0000","last_peered":"2024-09-13T13:05:53.023774+0000","last_clean":"2024-09-13T13:05:53.023774+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:05:53.023774+0000","last_undegraded":"2024-09-13T13:05:53.023774+0000","last_fullsized":"2024-09-13T13:05:53.023774+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"155'14323","reported_seq":20525,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.097852+0000","last_change":"2024-09-13T13:05:29.580927+0000","last_active":"2024-09-13T13:05:53.097852+0000","last_peered":"2024-09-13T13:05:53.097852+0000","last_clean":"2024-09-13T13:05:53.097852+0000","last_became_active":"2024-09-13T13:04:48.765805+0000","last_became_peered":"2024-09-13T13:04:48.765805+0000","last_unstale":"2024-09-13T13:05:53.097852+0000","last_undegraded":"2024-09-13T13:05:53.097852+0000","last_fullsized":"2024-09-13T13:05:53.097852+0000","mapping_epoch":135,"log_start":"126'9300","ondisk_log_start":"126'9300","created":84,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'12879","last_scrub_stamp":"2024-09-13T13:05:29.580736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:02:11.355519+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.580736+0000","objects_scrubbed":2542,"log_size":5023,"log_dups_size":0,"ondisk_log_size":5023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:43.250181+0000","scrub_duration":768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70057984,"num_objects":1069,"num_object_clones":0,"num_object_copies":3207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14323,"num_write_kb":492544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3000,"num_bytes_recovered":156631040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":241,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.795990+0000","last_change":"2024-09-13T13:04:52.666294+0000","last_active":"2024-09-13T13:05:47.795990+0000","last_peered":"2024-09-13T13:05:47.795990+0000","last_clean":"2024-09-13T13:05:47.795990+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:05:47.795990+0000","last_undegraded":"2024-09-13T13:05:47.795990+0000","last_fullsized":"2024-09-13T13:05:47.795990+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:52.666222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:52.666222+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:57.559648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":229,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.783656+0000","last_change":"2024-09-13T13:05:27.422529+0000","last_active":"2024-09-13T13:05:47.783656+0000","last_peered":"2024-09-13T13:05:47.783656+0000","last_clean":"2024-09-13T13:05:47.783656+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:05:47.783656+0000","last_undegraded":"2024-09-13T13:05:47.783656+0000","last_fullsized":"2024-09-13T13:05:47.783656+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:05:27.422451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:27.422451+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.192286+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":285,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782023+0000","last_change":"2024-09-13T13:04:02.759553+0000","last_active":"2024-09-13T13:05:47.782023+0000","last_peered":"2024-09-13T13:05:47.782023+0000","last_clean":"2024-09-13T13:05:47.782023+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:05:47.782023+0000","last_undegraded":"2024-09-13T13:05:47.782023+0000","last_fullsized":"2024-09-13T13:05:47.782023+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:45.959285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.959285+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:21.359173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796172+0000","last_change":"2024-09-13T13:04:02.757776+0000","last_active":"2024-09-13T13:05:47.796172+0000","last_peered":"2024-09-13T13:05:47.796172+0000","last_clean":"2024-09-13T13:05:47.796172+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:05:47.796172+0000","last_undegraded":"2024-09-13T13:05:47.796172+0000","last_fullsized":"2024-09-13T13:05:47.796172+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:28.529954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.529954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.625121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":188,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.011391+0000","last_change":"2024-09-13T13:05:43.098238+0000","last_active":"2024-09-13T13:05:53.011391+0000","last_peered":"2024-09-13T13:05:53.011391+0000","last_clean":"2024-09-13T13:05:53.011391+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:05:53.011391+0000","last_undegraded":"2024-09-13T13:05:53.011391+0000","last_fullsized":"2024-09-13T13:05:53.011391+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:43.098196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:43.098196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:07.803311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":216,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.024366+0000","last_change":"2024-09-13T13:03:14.504339+0000","last_active":"2024-09-13T13:05:53.024366+0000","last_peered":"2024-09-13T13:05:53.024366+0000","last_clean":"2024-09-13T13:05:53.024366+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:05:53.024366+0000","last_undegraded":"2024-09-13T13:05:53.024366+0000","last_fullsized":"2024-09-13T13:05:53.024366+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:03:14.504263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.504263+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":208,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796889+0000","last_change":"2024-09-13T13:05:27.619960+0000","last_active":"2024-09-13T13:05:47.796889+0000","last_peered":"2024-09-13T13:05:47.796889+0000","last_clean":"2024-09-13T13:05:47.796889+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:05:47.796889+0000","last_undegraded":"2024-09-13T13:05:47.796889+0000","last_fullsized":"2024-09-13T13:05:47.796889+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:27.619884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:27.619884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.490575+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":215,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782484+0000","last_change":"2024-09-13T13:05:37.197948+0000","last_active":"2024-09-13T13:05:47.782484+0000","last_peered":"2024-09-13T13:05:47.782484+0000","last_clean":"2024-09-13T13:05:47.782484+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:05:47.782484+0000","last_undegraded":"2024-09-13T13:05:47.782484+0000","last_fullsized":"2024-09-13T13:05:47.782484+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:37.197907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:37.197907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:55.791682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":222,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.009705+0000","last_change":"2024-09-13T13:02:42.589951+0000","last_active":"2024-09-13T13:05:53.009705+0000","last_peered":"2024-09-13T13:05:53.009705+0000","last_clean":"2024-09-13T13:05:53.009705+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:05:53.009705+0000","last_undegraded":"2024-09-13T13:05:53.009705+0000","last_fullsized":"2024-09-13T13:05:53.009705+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:42.589866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.589866+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":220,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782144+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:05:47.782144+0000","last_peered":"2024-09-13T13:05:47.782144+0000","last_clean":"2024-09-13T13:05:47.782144+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:05:47.782144+0000","last_undegraded":"2024-09-13T13:05:47.782144+0000","last_fullsized":"2024-09-13T13:05:47.782144+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:24.542132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.022688+0000","last_change":"2024-09-13T13:05:42.728927+0000","last_active":"2024-09-13T13:05:53.022688+0000","last_peered":"2024-09-13T13:05:53.022688+0000","last_clean":"2024-09-13T13:05:53.022688+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:05:53.022688+0000","last_undegraded":"2024-09-13T13:05:53.022688+0000","last_fullsized":"2024-09-13T13:05:53.022688+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:42.728881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:42.728881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:06.941341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":208,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.020408+0000","last_change":"2024-09-13T13:05:45.343713+0000","last_active":"2024-09-13T13:05:53.020408+0000","last_peered":"2024-09-13T13:05:53.020408+0000","last_clean":"2024-09-13T13:05:53.020408+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:05:53.020408+0000","last_undegraded":"2024-09-13T13:05:53.020408+0000","last_fullsized":"2024-09-13T13:05:53.020408+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:45.343638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.343638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:52.672377+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":221,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796536+0000","last_change":"2024-09-13T13:04:53.687731+0000","last_active":"2024-09-13T13:05:47.796536+0000","last_peered":"2024-09-13T13:05:47.796536+0000","last_clean":"2024-09-13T13:05:47.796536+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:05:47.796536+0000","last_undegraded":"2024-09-13T13:05:47.796536+0000","last_fullsized":"2024-09-13T13:05:47.796536+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:16.632415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":211,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.008486+0000","last_change":"2024-09-13T13:02:41.622369+0000","last_active":"2024-09-13T13:05:53.008486+0000","last_peered":"2024-09-13T13:05:53.008486+0000","last_clean":"2024-09-13T13:05:53.008486+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:05:53.008486+0000","last_undegraded":"2024-09-13T13:05:53.008486+0000","last_fullsized":"2024-09-13T13:05:53.008486+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.622292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.622292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":262,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.023110+0000","last_change":"2024-09-13T13:01:06.210251+0000","last_active":"2024-09-13T13:05:53.023110+0000","last_peered":"2024-09-13T13:05:53.023110+0000","last_clean":"2024-09-13T13:05:53.023110+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:05:53.023110+0000","last_undegraded":"2024-09-13T13:05:53.023110+0000","last_fullsized":"2024-09-13T13:05:53.023110+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781648+0000","last_change":"2024-09-13T13:04:49.927118+0000","last_active":"2024-09-13T13:05:47.781648+0000","last_peered":"2024-09-13T13:05:47.781648+0000","last_clean":"2024-09-13T13:05:47.781648+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:05:47.781648+0000","last_undegraded":"2024-09-13T13:05:47.781648+0000","last_fullsized":"2024-09-13T13:05:47.781648+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:49.927073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:49.927073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:08.714937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":222,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.023362+0000","last_change":"2024-09-13T13:02:37.222400+0000","last_active":"2024-09-13T13:05:53.023362+0000","last_peered":"2024-09-13T13:05:53.023362+0000","last_clean":"2024-09-13T13:05:53.023362+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:05:53.023362+0000","last_undegraded":"2024-09-13T13:05:53.023362+0000","last_fullsized":"2024-09-13T13:05:53.023362+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:37.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.222329+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.783264+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:05:47.783264+0000","last_peered":"2024-09-13T13:05:47.783264+0000","last_clean":"2024-09-13T13:05:47.783264+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:05:47.783264+0000","last_undegraded":"2024-09-13T13:05:47.783264+0000","last_fullsized":"2024-09-13T13:05:47.783264+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:05:55.131839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":283,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796514+0000","last_change":"2024-09-13T13:05:45.595219+0000","last_active":"2024-09-13T13:05:47.796514+0000","last_peered":"2024-09-13T13:05:47.796514+0000","last_clean":"2024-09-13T13:05:47.796514+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:05:47.796514+0000","last_undegraded":"2024-09-13T13:05:47.796514+0000","last_fullsized":"2024-09-13T13:05:47.796514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:45.595161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.595161+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:05.452404+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":198,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.009158+0000","last_change":"2024-09-13T13:02:40.615749+0000","last_active":"2024-09-13T13:05:53.009158+0000","last_peered":"2024-09-13T13:05:53.009158+0000","last_clean":"2024-09-13T13:05:53.009158+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:05:53.009158+0000","last_undegraded":"2024-09-13T13:05:53.009158+0000","last_fullsized":"2024-09-13T13:05:53.009158+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:40.615674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:40.615674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":244,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.780873+0000","last_change":"2024-09-13T13:05:26.186899+0000","last_active":"2024-09-13T13:05:47.780873+0000","last_peered":"2024-09-13T13:05:47.780873+0000","last_clean":"2024-09-13T13:05:47.780873+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:05:47.780873+0000","last_undegraded":"2024-09-13T13:05:47.780873+0000","last_fullsized":"2024-09-13T13:05:47.780873+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:05:26.186830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.186830+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:29.937721+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":229,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781681+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:05:47.781681+0000","last_peered":"2024-09-13T13:05:47.781681+0000","last_clean":"2024-09-13T13:05:47.781681+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:05:47.781681+0000","last_undegraded":"2024-09-13T13:05:47.781681+0000","last_fullsized":"2024-09-13T13:05:47.781681+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:39.039395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":212,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.019324+0000","last_change":"2024-09-13T13:03:44.467611+0000","last_active":"2024-09-13T13:05:53.019324+0000","last_peered":"2024-09-13T13:05:53.019324+0000","last_clean":"2024-09-13T13:05:53.019324+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:05:53.019324+0000","last_undegraded":"2024-09-13T13:05:53.019324+0000","last_fullsized":"2024-09-13T13:05:53.019324+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:44.467297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:44.467297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:50.750321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":199,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.020980+0000","last_change":"2024-09-13T13:02:42.208021+0000","last_active":"2024-09-13T13:05:53.020980+0000","last_peered":"2024-09-13T13:05:53.020980+0000","last_clean":"2024-09-13T13:05:53.020980+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:05:53.020980+0000","last_undegraded":"2024-09-13T13:05:53.020980+0000","last_fullsized":"2024-09-13T13:05:53.020980+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:42.207956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.207956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":172,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796013+0000","last_change":"2024-09-13T13:05:35.506093+0000","last_active":"2024-09-13T13:05:47.796013+0000","last_peered":"2024-09-13T13:05:47.796013+0000","last_clean":"2024-09-13T13:05:47.796013+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:05:47.796013+0000","last_undegraded":"2024-09-13T13:05:47.796013+0000","last_fullsized":"2024-09-13T13:05:47.796013+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:05:35.506037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.506037+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:58.281227+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782773+0000","last_change":"2024-09-13T13:02:39.921391+0000","last_active":"2024-09-13T13:05:47.782773+0000","last_peered":"2024-09-13T13:05:47.782773+0000","last_clean":"2024-09-13T13:05:47.782773+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:05:47.782773+0000","last_undegraded":"2024-09-13T13:05:47.782773+0000","last_fullsized":"2024-09-13T13:05:47.782773+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.921325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.921325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:20.381317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":195,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.023897+0000","last_change":"2024-09-13T13:01:06.211993+0000","last_active":"2024-09-13T13:05:53.023897+0000","last_peered":"2024-09-13T13:05:53.023897+0000","last_clean":"2024-09-13T13:05:53.023897+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:05:53.023897+0000","last_undegraded":"2024-09-13T13:05:53.023897+0000","last_fullsized":"2024-09-13T13:05:53.023897+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":199,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.007372+0000","last_change":"2024-09-13T13:02:36.586702+0000","last_active":"2024-09-13T13:05:53.007372+0000","last_peered":"2024-09-13T13:05:53.007372+0000","last_clean":"2024-09-13T13:05:53.007372+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:05:53.007372+0000","last_undegraded":"2024-09-13T13:05:53.007372+0000","last_fullsized":"2024-09-13T13:05:53.007372+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.586654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.586654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":237,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.021118+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:05:53.021118+0000","last_peered":"2024-09-13T13:05:53.021118+0000","last_clean":"2024-09-13T13:05:53.021118+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:05:53.021118+0000","last_undegraded":"2024-09-13T13:05:53.021118+0000","last_fullsized":"2024-09-13T13:05:53.021118+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":209,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.796970+0000","last_change":"2024-09-13T13:05:12.597696+0000","last_active":"2024-09-13T13:05:47.796970+0000","last_peered":"2024-09-13T13:05:47.796970+0000","last_clean":"2024-09-13T13:05:47.796970+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:05:47.796970+0000","last_undegraded":"2024-09-13T13:05:47.796970+0000","last_fullsized":"2024-09-13T13:05:47.796970+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:12.597637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.597637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:30.070820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":212,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.782152+0000","last_change":"2024-09-13T13:05:08.432874+0000","last_active":"2024-09-13T13:05:47.782152+0000","last_peered":"2024-09-13T13:05:47.782152+0000","last_clean":"2024-09-13T13:05:47.782152+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:05:47.782152+0000","last_undegraded":"2024-09-13T13:05:47.782152+0000","last_fullsized":"2024-09-13T13:05:47.782152+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:08.432816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:08.432816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:09.081986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.024054+0000","last_change":"2024-09-13T13:04:02.760695+0000","last_active":"2024-09-13T13:05:53.024054+0000","last_peered":"2024-09-13T13:05:53.024054+0000","last_clean":"2024-09-13T13:05:53.024054+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:05:53.024054+0000","last_undegraded":"2024-09-13T13:05:53.024054+0000","last_fullsized":"2024-09-13T13:05:53.024054+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:49.780996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":237,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.024815+0000","last_change":"2024-09-13T13:05:44.759472+0000","last_active":"2024-09-13T13:05:53.024815+0000","last_peered":"2024-09-13T13:05:53.024815+0000","last_clean":"2024-09-13T13:05:53.024815+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:05:53.024815+0000","last_undegraded":"2024-09-13T13:05:53.024815+0000","last_fullsized":"2024-09-13T13:05:53.024815+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2024-09-13T13:05:44.759233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.759233+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:51.091258+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":211,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.010797+0000","last_change":"2024-09-13T13:03:15.466909+0000","last_active":"2024-09-13T13:05:53.010797+0000","last_peered":"2024-09-13T13:05:53.010797+0000","last_clean":"2024-09-13T13:05:53.010797+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:05:53.010797+0000","last_undegraded":"2024-09-13T13:05:53.010797+0000","last_fullsized":"2024-09-13T13:05:53.010797+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:15.466859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:15.466859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":96,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.011393+0000","last_change":"2024-09-13T13:04:02.765046+0000","last_active":"2024-09-13T13:05:53.011393+0000","last_peered":"2024-09-13T13:05:53.011393+0000","last_clean":"2024-09-13T13:05:53.011393+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:05:53.011393+0000","last_undegraded":"2024-09-13T13:05:53.011393+0000","last_fullsized":"2024-09-13T13:05:53.011393+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:47.160977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":197,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-13T13:05:47.781916+0000","last_change":"2024-09-13T13:04:58.468060+0000","last_active":"2024-09-13T13:05:47.781916+0000","last_peered":"2024-09-13T13:05:47.781916+0000","last_clean":"2024-09-13T13:05:47.781916+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:05:47.781916+0000","last_undegraded":"2024-09-13T13:05:47.781916+0000","last_fullsized":"2024-09-13T13:05:47.781916+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:58.468011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:58.468011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:06:01.129704+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":207,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.019770+0000","last_change":"2024-09-13T13:05:44.759683+0000","last_active":"2024-09-13T13:05:53.019770+0000","last_peered":"2024-09-13T13:05:53.019770+0000","last_clean":"2024-09-13T13:05:53.019770+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:05:53.019770+0000","last_undegraded":"2024-09-13T13:05:53.019770+0000","last_fullsized":"2024-09-13T13:05:53.019770+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:44.759284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.759284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:07.342203+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.025046+0000","last_change":"2024-09-13T13:02:46.246602+0000","last_active":"2024-09-13T13:05:53.025046+0000","last_peered":"2024-09-13T13:05:53.025046+0000","last_clean":"2024-09-13T13:05:53.025046+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:05:53.025046+0000","last_undegraded":"2024-09-13T13:05:53.025046+0000","last_fullsized":"2024-09-13T13:05:53.025046+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.246540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.246540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":206,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.011794+0000","last_change":"2024-09-13T13:04:02.756287+0000","last_active":"2024-09-13T13:05:53.011794+0000","last_peered":"2024-09-13T13:05:53.011794+0000","last_clean":"2024-09-13T13:05:53.011794+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:05:53.011794+0000","last_undegraded":"2024-09-13T13:05:53.011794+0000","last_fullsized":"2024-09-13T13:05:53.011794+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:02:39.557251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.557251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T13:05:53.009431+0000","last_change":"2024-09-13T13:02:30.512536+0000","last_active":"2024-09-13T13:05:53.009431+0000","last_peered":"2024-09-13T13:05:53.009431+0000","last_clean":"2024-09-13T13:05:53.009431+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:05:53.009431+0000","last_undegraded":"2024-09-13T13:05:53.009431+0000","last_fullsized":"2024-09-13T13:05:53.009431+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:30.512433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:30.512433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1051394072,"num_objects":16044,"num_object_clones":0,"num_object_copies":48132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16044,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226154,"num_write_kb":7750273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14752,"num_bytes_recovered":756088856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3214422016,"data_stored":3214409800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90021,"ondisk_log_size":90021,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":285,"num_read_kb":280,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201341,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":453732,"kb_used_data":350356,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93291420,"statfs":{"total":95995035648,"available":95530414080,"internally_reserved":0,"allocated":358764544,"data_stored":358316566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806140952,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":449188,"kb_used_data":295124,"kb_used_omap":4,"kb_used_meta":154043,"kb_avail":93295964,"statfs":{"total":95995035648,"available":95535067136,"internally_reserved":0,"allocated":302206976,"data_stored":301764217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":157740717},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691913,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":391320,"kb_used_data":279624,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93353832,"statfs":{"total":95995035648,"available":95594323968,"internally_reserved":0,"allocated":286334976,"data_stored":285897582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":450028,"kb_used_data":350812,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93295124,"statfs":{"total":95995035648,"available":95534206976,"internally_reserved":0,"allocated":359231488,"data_stored":358773750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117078,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":791152,"kb_used_data":615584,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92954000,"statfs":{"total":95995035648,"available":95184896000,"internally_reserved":0,"allocated":630358016,"data_stored":629897012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":179763655},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":524872,"kb_used_data":421432,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93220280,"statfs":{"total":95995035648,"available":95457566720,"internally_reserved":0,"allocated":431546368,"data_stored":431124698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":531972,"kb_used_data":410356,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93213180,"statfs":{"total":95995035648,"available":95450296320,"internally_reserved":0,"allocated":420204544,"data_stored":419722765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":690620,"kb_used_data":532904,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93054532,"statfs":{"total":95995035648,"available":95287840768,"internally_reserved":0,"allocated":545693696,"data_stored":545231795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":529600512,"data_stored":529596440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":411897856,"data_stored":411893784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413532160,"data_stored":413532160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":611319808,"data_stored":611319808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":340656128,"data_stored":340656128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":278265856,"data_stored":278265856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":287182848,"data_stored":287178776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":341966848,"data_stored":341966848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:06:00.693 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T13:06:00.693 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:06:00.693 INFO:tasks.thrashosds.thrasher:Setting full ratio to .001 2024-09-13T13:06:00.694 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set-full-ratio .001 2024-09-13T13:06:00.759 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:00.819 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:00.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:00.849 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:06:00.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:00.945 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:00.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:01.047 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:01.074 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:01.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:01.176 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:01.215 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:01.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:01.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:00 smithi052 ceph-mon[36682]: pgmap v222: 148 pgs: 148 active+clean; 752 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-13T13:06:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:00 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3887016079' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:06:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:00 smithi052 ceph-mon[29471]: pgmap v222: 148 pgs: 148 active+clean; 752 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-13T13:06:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:00 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3887016079' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:06:01.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:01.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:00 smithi167 ceph-mon[33178]: pgmap v222: 148 pgs: 148 active+clean; 752 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-13T13:06:01.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:00 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3887016079' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:06:01.346 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:01.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:01.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:01.449 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:01.493 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:01.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:01.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:01.595 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:01.634 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:01.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:01.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:01.736 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:01.767 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:01.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:01.869 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:01.915 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:01.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:02.016 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:02.054 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:02.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:02.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:02.156 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:02.184 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:02.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:02.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:01 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:02.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:01 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3887016079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:06:02.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:01 smithi052 ceph-mon[36682]: osdmap e158: 8 total, 8 up, 8 in 2024-09-13T13:06:02.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:01 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:02.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:01 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3887016079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:06:02.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:01 smithi052 ceph-mon[29471]: osdmap e158: 8 total, 8 up, 8 in 2024-09-13T13:06:02.286 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:02.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:01 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:02.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:01 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3887016079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:06:02.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:01 smithi167 ceph-mon[33178]: osdmap e158: 8 total, 8 up, 8 in 2024-09-13T13:06:02.326 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:02.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:02.428 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:02.466 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:02.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:02.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:02.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:02.608 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:02.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:02.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:02.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:02.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:02.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:02.840 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:02.882 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:02.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:02.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:02.984 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:03.029 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:03.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:03.045 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:02 smithi167 ceph-mon[33178]: pgmap v224: 148 pgs: 148 active+clean; 695 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 982 op/s 2024-09-13T13:06:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:02 smithi167 ceph-mon[33178]: 7.d scrub starts 2024-09-13T13:06:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:02 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:02 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:06:03.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:02 smithi052 ceph-mon[29471]: pgmap v224: 148 pgs: 148 active+clean; 695 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 982 op/s 2024-09-13T13:06:03.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:02 smithi052 ceph-mon[29471]: 7.d scrub starts 2024-09-13T13:06:03.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:02 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:03.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:02 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:06:03.110 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:02 smithi052 ceph-mon[36682]: pgmap v224: 148 pgs: 148 active+clean; 695 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 982 op/s 2024-09-13T13:06:03.110 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:02 smithi052 ceph-mon[36682]: 7.d scrub starts 2024-09-13T13:06:03.111 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:02 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:03.111 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:02 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:06:03.112 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:06:03.112 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:06:03.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:06:03.112 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:06:03.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.112 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83958 7.b 7:d1e60c84:::benchmark_data_smithi052_71865_object83957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:06:03.112 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.524868+0000", 2024-09-13T13:06:03.112 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 168.58187985199999, 2024-09-13T13:06:03.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.019764474, 2024-09-13T13:06:03.112 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.113 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.113 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:06:03.113 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.113 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.113 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.113 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83958 2024-09-13T13:06:03.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.113 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524868+0000", 2024-09-13T13:06:03.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524868+0000", 2024-09-13T13:06:03.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524870+0000", 2024-09-13T13:06:03.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8929999999999999e-06 2024-09-13T13:06:03.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524893+0000", 2024-09-13T13:06:03.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2925000000000002e-05 2024-09-13T13:06:03.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524908+0000", 2024-09-13T13:06:03.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4839e-05 2024-09-13T13:06:03.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524916+0000", 2024-09-13T13:06:03.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0539999999999995e-06 2024-09-13T13:06:03.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525086+0000", 2024-09-13T13:06:03.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016999100000000001 2024-09-13T13:06:03.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525228+0000", 2024-09-13T13:06:03.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014245600000000001 2024-09-13T13:06:03.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:06:03.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525369+0000", 2024-09-13T13:06:03.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014030100000000001 2024-09-13T13:06:03.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.532247+0000", 2024-09-13T13:06:03.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0068779770000000004 2024-09-13T13:06:03.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.532262+0000", 2024-09-13T13:06:03.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5274e-05 2024-09-13T13:06:03.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548202+0000", 2024-09-13T13:06:03.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015939813000000001 2024-09-13T13:06:03.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548214+0000", 2024-09-13T13:06:03.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.24e-05 2024-09-13T13:06:03.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.119 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.119 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.544509+0000", 2024-09-13T13:06:03.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99629462499999999 2024-09-13T13:06:03.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.119 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.119 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.544633+0000", 2024-09-13T13:06:03.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012392600000000001 2024-09-13T13:06:03.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.119 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83959 7.b 7:de17d8b6:::benchmark_data_smithi052_71865_object83958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:06:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.525528+0000", 2024-09-13T13:06:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 168.58122023600001, 2024-09-13T13:06:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0337471460000001, 2024-09-13T13:06:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83959 2024-09-13T13:06:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525528+0000", 2024-09-13T13:06:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525528+0000", 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525529+0000", 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.248e-06 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525551+0000", 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1900999999999999e-05 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525560+0000", 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1810000000000007e-06 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525568+0000", 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7929999999999998e-06 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548247+0000", 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.022678957 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548253+0000", 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.84e-06 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545053+0000", 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.996799993 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545157+0000", 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104693 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5", 2024-09-13T13:06:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545303+0000", 2024-09-13T13:06:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000145434 2024-09-13T13:06:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552563+0000", 2024-09-13T13:06:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0072600859999999998 2024-09-13T13:06:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559142+0000", 2024-09-13T13:06:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0065787509999999999 2024-09-13T13:06:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559190+0000", 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7794999999999999e-05 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559275+0000", 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5474000000000002e-05 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83968 7.b 7:d987446b:::benchmark_data_smithi052_71865_object83967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:06:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.545178+0000", 2024-09-13T13:06:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 168.56156980200001, 2024-09-13T13:06:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.014227582, 2024-09-13T13:06:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83968 2024-09-13T13:06:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545178+0000", 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545178+0000", 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545180+0000", 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.728e-06 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545203+0000", 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3269999999999999e-05 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545212+0000", 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5620000000000004e-06 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545218+0000", 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7599999999999999e-06 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548273+0000", 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0030558460000000001 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548279+0000", 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7989999999999999e-06 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.549411+0000", 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0011321479999999 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.549546+0000", 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013478399999999999 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5", 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.549734+0000", 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000187999 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555255+0000", 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055203860000000004 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559284+0000", 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0040296300000000002 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559328+0000", 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3339e-05 2024-09-13T13:06:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559406+0000", 2024-09-13T13:06:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8331000000000006e-05 2024-09-13T13:06:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83950 7.d 7:b84df720:::benchmark_data_smithi052_71865_object83949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:06:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:06:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 168.55882373399999, 2024-09-13T13:06:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99935902799999998, 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83950 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:06:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547926+0000", 2024-09-13T13:06:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6199999999999999e-06 2024-09-13T13:06:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547961+0000", 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4916999999999998e-05 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547971+0000", 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8789999999999998e-06 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547978+0000", 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6120000000000001e-06 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548150+0000", 2024-09-13T13:06:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017159 2024-09-13T13:06:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548159+0000", 2024-09-13T13:06:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7979999999999995e-06 2024-09-13T13:06:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547139+0000", 2024-09-13T13:06:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99898018700000002 2024-09-13T13:06:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547283+0000", 2024-09-13T13:06:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000144425 2024-09-13T13:06:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83951 7.d 7:bd55e938:::benchmark_data_smithi052_71865_object83950:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:06:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.547994+0000", 2024-09-13T13:06:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 168.55875364600001, 2024-09-13T13:06:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0078699250000001, 2024-09-13T13:06:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:06:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83951 2024-09-13T13:06:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547994+0000", 2024-09-13T13:06:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547994+0000", 2024-09-13T13:06:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547995+0000", 2024-09-13T13:06:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6899999999999996e-07 2024-09-13T13:06:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.141 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:03.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548019+0000", 2024-09-13T13:06:03.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3451999999999998e-05 2024-09-13T13:06:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548026+0000", 2024-09-13T13:06:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1600000000000001e-06 2024-09-13T13:06:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548032+0000", 2024-09-13T13:06:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8359999999999997e-06 2024-09-13T13:06:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548182+0000", 2024-09-13T13:06:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015005300000000001 2024-09-13T13:06:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548188+0000", 2024-09-13T13:06:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7869999999999998e-06 2024-09-13T13:06:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547487+0000", 2024-09-13T13:06:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99929913400000003 2024-09-13T13:06:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for missing object", 2024-09-13T13:06:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547629+0000", 2024-09-13T13:06:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000142106 2024-09-13T13:06:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555757+0000", 2024-09-13T13:06:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0081283139999999993 2024-09-13T13:06:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555864+0000", 2024-09-13T13:06:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010721399999999999 2024-09-13T13:06:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83978 7.d 7:b4a32006:::benchmark_data_smithi052_71865_object83977:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:06:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.562795+0000", 2024-09-13T13:06:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 168.54395352399999, 2024-09-13T13:06:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99338139199999997, 2024-09-13T13:06:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83978 2024-09-13T13:06:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562795+0000", 2024-09-13T13:06:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562795+0000", 2024-09-13T13:06:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562797+0000", 2024-09-13T13:06:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2309999999999999e-06 2024-09-13T13:06:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562824+0000", 2024-09-13T13:06:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6838999999999998e-05 2024-09-13T13:06:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562836+0000", 2024-09-13T13:06:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2812e-05 2024-09-13T13:06:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562845+0000", 2024-09-13T13:06:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2320000000000001e-06 2024-09-13T13:06:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.563048+0000", 2024-09-13T13:06:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020349699999999999 2024-09-13T13:06:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.563058+0000", 2024-09-13T13:06:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5820000000000008e-06 2024-09-13T13:06:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547871+0000", 2024-09-13T13:06:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98481351699999997 2024-09-13T13:06:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548017+0000", 2024-09-13T13:06:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014600999999999999 2024-09-13T13:06:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0", 2024-09-13T13:06:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548480+0000", 2024-09-13T13:06:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00046313600000000001 2024-09-13T13:06:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555942+0000", 2024-09-13T13:06:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0074612120000000001 2024-09-13T13:06:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556034+0000", 2024-09-13T13:06:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2584000000000007e-05 2024-09-13T13:06:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556087+0000", 2024-09-13T13:06:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3118999999999997e-05 2024-09-13T13:06:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556176+0000", 2024-09-13T13:06:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8621000000000004e-05 2024-09-13T13:06:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83979 7.b 7:d79e8e3c:::benchmark_data_smithi052_71865_object83978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:06:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.562864+0000", 2024-09-13T13:06:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 168.54388376099999, 2024-09-13T13:06:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99667409100000004, 2024-09-13T13:06:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83979 2024-09-13T13:06:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562864+0000", 2024-09-13T13:06:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562864+0000", 2024-09-13T13:06:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562865+0000", 2024-09-13T13:06:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0900000000000001e-07 2024-09-13T13:06:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562887+0000", 2024-09-13T13:06:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1977999999999999e-05 2024-09-13T13:06:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562897+0000", 2024-09-13T13:06:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8740000000000007e-06 2024-09-13T13:06:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562902+0000", 2024-09-13T13:06:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2780000000000001e-06 2024-09-13T13:06:03.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562985+0000", 2024-09-13T13:06:03.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3119000000000002e-05 2024-09-13T13:06:03.156 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:03.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562994+0000", 2024-09-13T13:06:03.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1190000000000003e-06 2024-09-13T13:06:03.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552593+0000", 2024-09-13T13:06:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98959876899999999 2024-09-13T13:06:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552700+0000", 2024-09-13T13:06:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010694199999999999 2024-09-13T13:06:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5", 2024-09-13T13:06:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552844+0000", 2024-09-13T13:06:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014343099999999999 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555298+0000", 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024543220000000001 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559413+0000", 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041153700000000001 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559458+0000", 2024-09-13T13:06:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5105999999999998e-05 2024-09-13T13:06:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559538+0000", 2024-09-13T13:06:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0074000000000006e-05 2024-09-13T13:06:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83988 7.b 7:d1d56cdb:::benchmark_data_smithi052_71865_object83987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:06:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.571369+0000", 2024-09-13T13:06:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 168.535379089, 2024-09-13T13:06:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98902711700000001, 2024-09-13T13:06:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83988 2024-09-13T13:06:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571369+0000", 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571369+0000", 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571371+0000", 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0760000000000001e-06 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571391+0000", 2024-09-13T13:06:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9936999999999999e-05 2024-09-13T13:06:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571404+0000", 2024-09-13T13:06:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3241000000000001e-05 2024-09-13T13:06:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571410+0000", 2024-09-13T13:06:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7799999999999997e-06 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571466+0000", 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6030999999999999e-05 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571473+0000", 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8539999999999999e-06 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556198+0000", 2024-09-13T13:06:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98472465899999995 2024-09-13T13:06:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556486+0000", 2024-09-13T13:06:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00028855500000000002 2024-09-13T13:06:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5", 2024-09-13T13:06:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556637+0000", 2024-09-13T13:06:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000151218 2024-09-13T13:06:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559546+0000", 2024-09-13T13:06:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029090740000000002 2024-09-13T13:06:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560209+0000", 2024-09-13T13:06:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00066297199999999995 2024-09-13T13:06:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560301+0000", 2024-09-13T13:06:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1712000000000006e-05 2024-09-13T13:06:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560396+0000", 2024-09-13T13:06:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5007999999999996e-05 2024-09-13T13:06:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:166990 7.b 7:db1a2623:::benchmark_data_smithi052_71865_object45889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.766249+0000", 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 75.340498623000002, 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0195013719999999, 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 166990 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766249+0000", 2024-09-13T13:06:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766249+0000", 2024-09-13T13:06:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766252+0000", 2024-09-13T13:06:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.48e-06 2024-09-13T13:06:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766259+0000", 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0849999999999999e-06 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766272+0000", 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.305e-05 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766281+0000", 2024-09-13T13:06:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5750000000000001e-06 2024-09-13T13:06:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.767413+0000", 2024-09-13T13:06:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011321529999999999 2024-09-13T13:06:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.767559+0000", 2024-09-13T13:06:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000146217 2024-09-13T13:06:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.767706+0000", 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014731799999999999 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.776972+0000", 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0092654350000000007 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.776984+0000", 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2018000000000001e-05 2024-09-13T13:06:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.785665+0000", 2024-09-13T13:06:03.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0086813429999999 2024-09-13T13:06:03.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.785751+0000", 2024-09-13T13:06:03.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5698000000000005e-05 2024-09-13T13:06:03.173 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.173 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.173 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.174 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167004 7.b 7:d0e4af5f:::benchmark_data_smithi052_71865_object45903:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:06:03.174 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.789600+0000", 2024-09-13T13:06:03.174 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 75.317148513000006, 2024-09-13T13:06:03.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0016245429999999, 2024-09-13T13:06:03.174 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.174 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.174 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.174 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.174 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.174 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.174 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167004 2024-09-13T13:06:03.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.175 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789600+0000", 2024-09-13T13:06:03.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789600+0000", 2024-09-13T13:06:03.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789602+0000", 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2230000000000001e-06 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789608+0000", 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.37e-06 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789624+0000", 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6167e-05 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789632+0000", 2024-09-13T13:06:03.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7270000000000007e-06 2024-09-13T13:06:03.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789705+0000", 2024-09-13T13:06:03.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2799999999999994e-05 2024-09-13T13:06:03.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789718+0000", 2024-09-13T13:06:03.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2653999999999999e-05 2024-09-13T13:06:03.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.786269+0000", 2024-09-13T13:06:03.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99655117400000004 2024-09-13T13:06:03.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.786357+0000", 2024-09-13T13:06:03.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7865e-05 2024-09-13T13:06:03.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.786482+0000", 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000125451 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791065+0000", 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045828450000000003 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791120+0000", 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5562000000000001e-05 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791155+0000", 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4326999999999998e-05 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791188+0000", 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2929000000000003e-05 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791224+0000", 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6449000000000003e-05 2024-09-13T13:06:03.180 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167017 7.b 7:d2ad2293:::benchmark_data_smithi052_71865_object45916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.800317+0000", 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 75.306430680999995, 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99742454800000002, 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.181 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167017 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800317+0000", 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800317+0000", 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800320+0000", 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.148e-06 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800326+0000", 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2960000000000004e-06 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800343+0000", 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7079e-05 2024-09-13T13:06:03.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800351+0000", 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1340000000000004e-06 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800433+0000", 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1509999999999997e-05 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800445+0000", 2024-09-13T13:06:03.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2414e-05 2024-09-13T13:06:03.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.788619+0000", 2024-09-13T13:06:03.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98817433300000002 2024-09-13T13:06:03.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.788727+0000", 2024-09-13T13:06:03.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108059 2024-09-13T13:06:03.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:06:03.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.788817+0000", 2024-09-13T13:06:03.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9117999999999996e-05 2024-09-13T13:06:03.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791266+0000", 2024-09-13T13:06:03.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024498599999999999 2024-09-13T13:06:03.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791306+0000", 2024-09-13T13:06:03.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9691000000000003e-05 2024-09-13T13:06:03.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797479+0000", 2024-09-13T13:06:03.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0061724299999999996 2024-09-13T13:06:03.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797695+0000", 2024-09-13T13:06:03.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021655399999999999 2024-09-13T13:06:03.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797742+0000", 2024-09-13T13:06:03.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6922000000000002e-05 2024-09-13T13:06:03.188 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.188 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.188 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.188 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167030 7.b 7:d39cbe03:::benchmark_data_smithi052_71865_object45929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:06:03.188 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.815695+0000", 2024-09-13T13:06:03.188 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 75.291053513999998, 2024-09-13T13:06:03.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98232124600000004, 2024-09-13T13:06:03.188 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.188 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.189 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.189 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.189 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.189 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.189 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167030 2024-09-13T13:06:03.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.189 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815695+0000", 2024-09-13T13:06:03.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815695+0000", 2024-09-13T13:06:03.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815697+0000", 2024-09-13T13:06:03.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8380000000000002e-06 2024-09-13T13:06:03.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815705+0000", 2024-09-13T13:06:03.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1430000000000001e-06 2024-09-13T13:06:03.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815723+0000", 2024-09-13T13:06:03.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.838e-05 2024-09-13T13:06:03.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815732+0000", 2024-09-13T13:06:03.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2010000000000005e-06 2024-09-13T13:06:03.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.192 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.192 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815818+0000", 2024-09-13T13:06:03.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5518000000000004e-05 2024-09-13T13:06:03.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.192 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.192 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815827+0000", 2024-09-13T13:06:03.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6940000000000007e-06 2024-09-13T13:06:03.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.192 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.192 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791337+0000", 2024-09-13T13:06:03.193 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97550924100000003 2024-09-13T13:06:03.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.193 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791455+0000", 2024-09-13T13:06:03.193 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118081 2024-09-13T13:06:03.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.193 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:06:03.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791545+0000", 2024-09-13T13:06:03.193 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0480000000000001e-05 2024-09-13T13:06:03.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797789+0000", 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0062433799999999998 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797800+0000", 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1153e-05 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797829+0000", 2024-09-13T13:06:03.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9527999999999999e-05 2024-09-13T13:06:03.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797977+0000", 2024-09-13T13:06:03.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000148252 2024-09-13T13:06:03.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798016+0000", 2024-09-13T13:06:03.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8356999999999999e-05 2024-09-13T13:06:03.195 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.195 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.196 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.196 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167044 7.b 7:d626642e:::benchmark_data_smithi052_71865_object45943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:06:03.196 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.832092+0000", 2024-09-13T13:06:03.196 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 75.274656385, 2024-09-13T13:06:03.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98043175900000001, 2024-09-13T13:06:03.196 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.196 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.196 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.196 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.196 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.196 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.197 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167044 2024-09-13T13:06:03.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.197 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832092+0000", 2024-09-13T13:06:03.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832092+0000", 2024-09-13T13:06:03.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832094+0000", 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1299999999999999e-06 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832100+0000", 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9349999999999999e-06 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832115+0000", 2024-09-13T13:06:03.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5104e-05 2024-09-13T13:06:03.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.199 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.199 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832122+0000", 2024-09-13T13:06:03.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5769999999999996e-06 2024-09-13T13:06:03.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.199 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.199 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832207+0000", 2024-09-13T13:06:03.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4742999999999998e-05 2024-09-13T13:06:03.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.200 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832217+0000", 2024-09-13T13:06:03.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6069999999999997e-06 2024-09-13T13:06:03.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.200 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.794154+0000", 2024-09-13T13:06:03.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96193714900000005 2024-09-13T13:06:03.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.794275+0000", 2024-09-13T13:06:03.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012122500000000001 2024-09-13T13:06:03.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:06:03.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.794399+0000", 2024-09-13T13:06:03.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012333799999999999 2024-09-13T13:06:03.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798054+0000", 2024-09-13T13:06:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036553319999999999 2024-09-13T13:06:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798090+0000", 2024-09-13T13:06:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6616999999999999e-05 2024-09-13T13:06:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812409+0000", 2024-09-13T13:06:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014318587000000001 2024-09-13T13:06:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.204 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.204 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812475+0000", 2024-09-13T13:06:03.204 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5811999999999999e-05 2024-09-13T13:06:03.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.204 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.204 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812523+0000", 2024-09-13T13:06:03.204 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8603000000000002e-05 2024-09-13T13:06:03.204 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.204 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.204 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.205 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167053 7.b 7:d291c1c0:::benchmark_data_smithi052_71865_object45952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:06:03.205 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.840143+0000", 2024-09-13T13:06:03.205 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 75.266605081999998, 2024-09-13T13:06:03.205 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97266568200000003, 2024-09-13T13:06:03.205 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.205 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.205 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.205 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.205 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.205 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.205 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167053 2024-09-13T13:06:03.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.206 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840143+0000", 2024-09-13T13:06:03.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840143+0000", 2024-09-13T13:06:03.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840145+0000", 2024-09-13T13:06:03.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:06:03.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840153+0000", 2024-09-13T13:06:03.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4080000000000001e-06 2024-09-13T13:06:03.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840184+0000", 2024-09-13T13:06:03.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0988000000000002e-05 2024-09-13T13:06:03.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840408+0000", 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00022425 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840568+0000", 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015967999999999999 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840580+0000", 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.199e-05 2024-09-13T13:06:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.210 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.210 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798117+0000", 2024-09-13T13:06:03.210 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95753740700000001 2024-09-13T13:06:03.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.210 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.210 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798208+0000", 2024-09-13T13:06:03.210 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1225999999999997e-05 2024-09-13T13:06:03.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.210 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:06:03.210 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798311+0000", 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103112 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804034+0000", 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057222940000000002 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804070+0000", 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6854999999999999e-05 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.211 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812721+0000", 2024-09-13T13:06:03.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0086508160000000004 2024-09-13T13:06:03.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812768+0000", 2024-09-13T13:06:03.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6251000000000002e-05 2024-09-13T13:06:03.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812809+0000", 2024-09-13T13:06:03.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1153999999999999e-05 2024-09-13T13:06:03.212 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167055 7.b 7:d9aeaf19:::benchmark_data_smithi052_71865_object45954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:06:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.840429+0000", 2024-09-13T13:06:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 75.266318982000001, 2024-09-13T13:06:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97246315999999999, 2024-09-13T13:06:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.213 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.214 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.214 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167055 2024-09-13T13:06:03.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.214 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.214 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.214 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840429+0000", 2024-09-13T13:06:03.214 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.214 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.214 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840429+0000", 2024-09-13T13:06:03.214 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.215 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840430+0000", 2024-09-13T13:06:03.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0499999999999999e-06 2024-09-13T13:06:03.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.215 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840435+0000", 2024-09-13T13:06:03.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6979999999999997e-06 2024-09-13T13:06:03.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840443+0000", 2024-09-13T13:06:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5709999999999998e-06 2024-09-13T13:06:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840450+0000", 2024-09-13T13:06:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3990000000000001e-06 2024-09-13T13:06:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840607+0000", 2024-09-13T13:06:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015671000000000001 2024-09-13T13:06:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840614+0000", 2024-09-13T13:06:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2459999999999997e-06 2024-09-13T13:06:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.801126+0000", 2024-09-13T13:06:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96051206300000003 2024-09-13T13:06:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.801236+0000", 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110356 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.801360+0000", 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012337399999999999 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804101+0000", 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027418619999999999 2024-09-13T13:06:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806730+0000", 2024-09-13T13:06:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002629022 2024-09-13T13:06:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812818+0000", 2024-09-13T13:06:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0060879970000000004 2024-09-13T13:06:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812858+0000", 2024-09-13T13:06:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9833000000000003e-05 2024-09-13T13:06:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812892+0000", 2024-09-13T13:06:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3979000000000001e-05 2024-09-13T13:06:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167059 7.b 7:d9156452:::benchmark_data_smithi052_71865_object45958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:06:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.843970+0000", 2024-09-13T13:06:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 75.262778036, 2024-09-13T13:06:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96901141300000004, 2024-09-13T13:06:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167059 2024-09-13T13:06:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843970+0000", 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843970+0000", 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843972+0000", 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1660000000000001e-06 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843979+0000", 2024-09-13T13:06:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6490000000000002e-06 2024-09-13T13:06:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843997+0000", 2024-09-13T13:06:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8493999999999999e-05 2024-09-13T13:06:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.844006+0000", 2024-09-13T13:06:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3459999999999994e-06 2024-09-13T13:06:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.844089+0000", 2024-09-13T13:06:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3424999999999995e-05 2024-09-13T13:06:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.844101+0000", 2024-09-13T13:06:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1984000000000001e-05 2024-09-13T13:06:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804127+0000", 2024-09-13T13:06:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96002581099999995 2024-09-13T13:06:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804221+0000", 2024-09-13T13:06:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4267999999999999e-05 2024-09-13T13:06:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:06:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804324+0000", 2024-09-13T13:06:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103071 2024-09-13T13:06:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806768+0000", 2024-09-13T13:06:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024435590000000001 2024-09-13T13:06:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806802+0000", 2024-09-13T13:06:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4023999999999998e-05 2024-09-13T13:06:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812898+0000", 2024-09-13T13:06:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0060959630000000002 2024-09-13T13:06:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812946+0000", 2024-09-13T13:06:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8430000000000002e-05 2024-09-13T13:06:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812981+0000", 2024-09-13T13:06:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5222999999999998e-05 2024-09-13T13:06:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167062 7.b 7:daa2d453:::benchmark_data_smithi052_71865_object45961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:06:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.847548+0000", 2024-09-13T13:06:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 75.259199722999995, 2024-09-13T13:06:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96551024600000002, 2024-09-13T13:06:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167062 2024-09-13T13:06:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847548+0000", 2024-09-13T13:06:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847548+0000", 2024-09-13T13:06:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847551+0000", 2024-09-13T13:06:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.745e-06 2024-09-13T13:06:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847558+0000", 2024-09-13T13:06:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4359999999999999e-06 2024-09-13T13:06:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.231 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847575+0000", 2024-09-13T13:06:03.231 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7405e-05 2024-09-13T13:06:03.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.231 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.231 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.231 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847584+0000", 2024-09-13T13:06:03.231 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2869999999999993e-06 2024-09-13T13:06:03.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.231 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.231 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.231 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847663+0000", 2024-09-13T13:06:03.231 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8288000000000007e-05 2024-09-13T13:06:03.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.232 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.232 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847676+0000", 2024-09-13T13:06:03.232 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.367e-05 2024-09-13T13:06:03.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.232 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.232 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806829+0000", 2024-09-13T13:06:03.232 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95915302099999999 2024-09-13T13:06:03.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.232 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.232 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806951+0000", 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012156599999999999 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.807058+0000", 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010687399999999999 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.809517+0000", 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024595379999999998 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.234 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812643+0000", 2024-09-13T13:06:03.234 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031254239999999999 2024-09-13T13:06:03.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.234 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.234 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812987+0000", 2024-09-13T13:06:03.234 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00034455199999999998 2024-09-13T13:06:03.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.234 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.234 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813026+0000", 2024-09-13T13:06:03.234 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8333e-05 2024-09-13T13:06:03.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.235 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.235 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813059+0000", 2024-09-13T13:06:03.235 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3107e-05 2024-09-13T13:06:03.235 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.235 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.235 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.235 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167076 7.b 7:d73b9ec1:::benchmark_data_smithi052_71865_object45975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:06:03.235 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.861557+0000", 2024-09-13T13:06:03.235 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 75.245191082000005, 2024-09-13T13:06:03.236 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95706732900000002, 2024-09-13T13:06:03.236 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.236 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.236 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.236 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.236 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.236 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.236 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167076 2024-09-13T13:06:03.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.236 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.236 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.236 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.236 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861557+0000", 2024-09-13T13:06:03.237 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.237 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.237 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.237 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861557+0000", 2024-09-13T13:06:03.237 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.237 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.237 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.237 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861559+0000", 2024-09-13T13:06:03.237 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3750000000000001e-06 2024-09-13T13:06:03.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.238 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.238 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861566+0000", 2024-09-13T13:06:03.238 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0990000000000004e-06 2024-09-13T13:06:03.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.238 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.238 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861583+0000", 2024-09-13T13:06:03.238 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7405e-05 2024-09-13T13:06:03.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.238 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.238 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861591+0000", 2024-09-13T13:06:03.239 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1459999999999996e-06 2024-09-13T13:06:03.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.239 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.239 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861653+0000", 2024-09-13T13:06:03.239 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1723000000000005e-05 2024-09-13T13:06:03.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.239 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.239 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861662+0000", 2024-09-13T13:06:03.239 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9879999999999993e-06 2024-09-13T13:06:03.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.240 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.240 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.809546+0000", 2024-09-13T13:06:03.240 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.94788374500000006 2024-09-13T13:06:03.240 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.240 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.240 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.240 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.809649+0000", 2024-09-13T13:06:03.240 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103309 2024-09-13T13:06:03.240 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.240 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.240 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:06:03.240 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.809753+0000", 2024-09-13T13:06:03.240 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104096 2024-09-13T13:06:03.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.241 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.241 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.241 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812707+0000", 2024-09-13T13:06:03.241 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029539829999999999 2024-09-13T13:06:03.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.241 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.241 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.241 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813064+0000", 2024-09-13T13:06:03.241 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035704999999999998 2024-09-13T13:06:03.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.241 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.241 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.242 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818518+0000", 2024-09-13T13:06:03.242 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0054542200000000001 2024-09-13T13:06:03.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.242 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.242 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818577+0000", 2024-09-13T13:06:03.242 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8894000000000002e-05 2024-09-13T13:06:03.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.242 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.244 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818624+0000", 2024-09-13T13:06:03.244 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7296000000000003e-05 2024-09-13T13:06:03.244 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.244 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.244 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.244 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.244 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167129 7.b 7:d5d8d4da:::benchmark_data_smithi052_71865_object46028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:06:03.244 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.929446+0000", 2024-09-13T13:06:03.244 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 75.177302342000004, 2024-09-13T13:06:03.244 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.88935671599999999, 2024-09-13T13:06:03.244 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167129 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929446+0000", 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.245 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.246 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929446+0000", 2024-09-13T13:06:03.246 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.246 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929448+0000", 2024-09-13T13:06:03.246 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.593e-06 2024-09-13T13:06:03.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.246 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929456+0000", 2024-09-13T13:06:03.246 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7800000000000001e-06 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929474+0000", 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7521999999999999e-05 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929481+0000", 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3189999999999997e-06 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929545+0000", 2024-09-13T13:06:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:03.249 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:03.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:03.249 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4216999999999999e-05 2024-09-13T13:06:03.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.250 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.250 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929555+0000", 2024-09-13T13:06:03.250 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8609999999999993e-06 2024-09-13T13:06:03.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.250 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.250 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813093+0000", 2024-09-13T13:06:03.250 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.88353817400000001 2024-09-13T13:06:03.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.250 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.250 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813205+0000", 2024-09-13T13:06:03.251 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000111549 2024-09-13T13:06:03.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.251 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.251 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:06:03.251 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813319+0000", 2024-09-13T13:06:03.251 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114021 2024-09-13T13:06:03.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.251 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.251 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.251 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818667+0000", 2024-09-13T13:06:03.251 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053481539999999999 2024-09-13T13:06:03.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.251 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818682+0000", 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5438e-05 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818723+0000", 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0284000000000003e-05 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818766+0000", 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3798000000000003e-05 2024-09-13T13:06:03.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.253 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.253 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818802+0000", 2024-09-13T13:06:03.253 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6006000000000001e-05 2024-09-13T13:06:03.253 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.253 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.253 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.253 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167134 7.b 7:d3cd8c35:::benchmark_data_smithi052_71865_object46033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:06:03.253 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.936758+0000", 2024-09-13T13:06:03.253 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 75.169989948999998, 2024-09-13T13:06:03.253 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.89043407500000005, 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167134 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936758+0000", 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936758+0000", 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936760+0000", 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2199999999999999e-06 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936768+0000", 2024-09-13T13:06:03.255 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.605e-06 2024-09-13T13:06:03.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.256 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:03.256 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936784+0000", 2024-09-13T13:06:03.256 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5608999999999999e-05 2024-09-13T13:06:03.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.256 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:03.256 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936791+0000", 2024-09-13T13:06:03.256 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:06:03.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.256 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936875+0000", 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3662e-05 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936885+0000", 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0105999999999999e-05 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.815979+0000", 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.87909423799999997 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.816085+0000", 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000106469 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.816202+0000", 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116344 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818689+0000", 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024874559999999999 2024-09-13T13:06:03.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818848+0000", 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015898999999999999 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.827048+0000", 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0081994749999999995 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.827147+0000", 2024-09-13T13:06:03.259 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9359999999999997e-05 2024-09-13T13:06:03.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:03.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:03.260 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:03.260 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.827192+0000", 2024-09-13T13:06:03.260 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5194999999999997e-05 2024-09-13T13:06:03.260 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.260 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.260 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.260 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:03.260 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:03.260 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:06:03.350 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:03.389 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:03.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:03.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:03.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:03.516 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:03.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:03.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:03.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:03.644 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:03.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:03.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:03.745 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:03.774 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:03.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:03.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:03.876 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:03.915 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:03.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:03.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:04.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:04.058 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:04.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:04.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:04.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:03 smithi167 ceph-mon[33178]: 4.1a scrub starts 2024-09-13T13:06:04.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:03 smithi167 ceph-mon[33178]: 4.1a scrub ok 2024-09-13T13:06:04.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:03 smithi167 ceph-mon[33178]: 7.d scrub ok 2024-09-13T13:06:04.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:03 smithi167 ceph-mon[33178]: 7.7 scrub starts 2024-09-13T13:06:04.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:03 smithi167 ceph-mon[33178]: 7.7 scrub ok 2024-09-13T13:06:04.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:06:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:06:04.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:03 smithi052 ceph-mon[36682]: 4.1a scrub starts 2024-09-13T13:06:04.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:03 smithi052 ceph-mon[36682]: 4.1a scrub ok 2024-09-13T13:06:04.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:03 smithi052 ceph-mon[36682]: 7.d scrub ok 2024-09-13T13:06:04.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:03 smithi052 ceph-mon[36682]: 7.7 scrub starts 2024-09-13T13:06:04.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:03 smithi052 ceph-mon[36682]: 7.7 scrub ok 2024-09-13T13:06:04.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:03 smithi052 ceph-mon[29471]: 4.1a scrub starts 2024-09-13T13:06:04.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:03 smithi052 ceph-mon[29471]: 4.1a scrub ok 2024-09-13T13:06:04.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:03 smithi052 ceph-mon[29471]: 7.d scrub ok 2024-09-13T13:06:04.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:03 smithi052 ceph-mon[29471]: 7.7 scrub starts 2024-09-13T13:06:04.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:03 smithi052 ceph-mon[29471]: 7.7 scrub ok 2024-09-13T13:06:04.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:04.191 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:04.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:04.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:04.293 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:04.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:04.326 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:04.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:04.427 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:04.452 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:04.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:04.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:04.579 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:04.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:04.681 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:04.725 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:04.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:04.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:04.853 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:04.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:04.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:04.864 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:06:04.955 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:05.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:05.013 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:05.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:05.039 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:04 smithi052 ceph-mon[36682]: pgmap v225: 148 pgs: 148 active+clean; 455 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-13T13:06:05.039 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:04 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:05.039 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:04 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:06:05.039 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:04 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:06:05.039 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:04 smithi052 ceph-mon[29471]: pgmap v225: 148 pgs: 148 active+clean; 455 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-13T13:06:05.040 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:04 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:05.040 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:04 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:06:05.040 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:04 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:06:05.051 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:06:05.052 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:06:05.099 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:06:05.099 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:06:05.113 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:05.142 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:06:05.143 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:06:05.154 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:05.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:05.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:05.185 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:06:05.186 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:06:05.217 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:06:05.217 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:06:05.252 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:06:05.253 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:06:05.255 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:05.291 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:06:05.291 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:06:05.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:04 smithi167 ceph-mon[33178]: pgmap v225: 148 pgs: 148 active+clean; 455 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-13T13:06:05.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:04 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:05.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:04 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:06:05.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:04 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:06:05.326 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:05.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:05.327 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:06:05.327 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:06:05.428 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:05.471 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T13:06:05.475 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:05.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:05.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:05.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:05.599 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:05.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:05.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:05.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:05.724 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:05.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:05.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:05.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:05.848 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:05.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:05.950 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:06.001 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:06.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:06.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:06.102 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:06.127 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:06.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:06.229 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:06.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:05 smithi052 ceph-mon[36682]: osdmap e159: 8 total, 8 up, 8 in 2024-09-13T13:06:06.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:05 smithi052 ceph-mon[36682]: 5.1f scrub starts 2024-09-13T13:06:06.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:05 smithi052 ceph-mon[36682]: 5.1f scrub ok 2024-09-13T13:06:06.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:05 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/605385346' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:06.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:06:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:06:06.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:05 smithi052 ceph-mon[29471]: osdmap e159: 8 total, 8 up, 8 in 2024-09-13T13:06:06.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:05 smithi052 ceph-mon[29471]: 5.1f scrub starts 2024-09-13T13:06:06.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:05 smithi052 ceph-mon[29471]: 5.1f scrub ok 2024-09-13T13:06:06.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:05 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/605385346' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:06.273 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:06.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:06.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:05 smithi167 ceph-mon[33178]: osdmap e159: 8 total, 8 up, 8 in 2024-09-13T13:06:06.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:05 smithi167 ceph-mon[33178]: 5.1f scrub starts 2024-09-13T13:06:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:05 smithi167 ceph-mon[33178]: 5.1f scrub ok 2024-09-13T13:06:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:05 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/605385346' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:06.375 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:06.429 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:06.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:06.432 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:06.510 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:06.532 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:06.574 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:06.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:06.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:06.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:06.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:06.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:06.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:06.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:06.841 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:06.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:06.920 INFO:teuthology.orchestra.run.smithi052.stderr:osd set-full-ratio 0.001 2024-09-13T13:06:06.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:06.967 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:06.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:06.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:07.069 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:07.097 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:07.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:07.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:07.199 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:07.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:07.227 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:07.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[36682]: 5.4 scrub starts 2024-09-13T13:06:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[36682]: 5.4 scrub ok 2024-09-13T13:06:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[36682]: 6.d scrub starts 2024-09-13T13:06:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[36682]: 6.d scrub ok 2024-09-13T13:06:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[36682]: pgmap v227: 148 pgs: 148 active+clean; 455 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 691 op/s 2024-09-13T13:06:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[36682]: 4.1d scrub starts 2024-09-13T13:06:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[36682]: 4.1d scrub ok 2024-09-13T13:06:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:06:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:06:07.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[36682]: osdmap e160: 8 total, 8 up, 8 in 2024-09-13T13:06:07.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2141395302' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-09-13T13:06:07.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a[29467]: 2024-09-13T13:06:06.904+0000 7effab2fd640 -1 log_channel(cluster) log [ERR] : Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-13T13:06:07.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[29471]: 5.4 scrub starts 2024-09-13T13:06:07.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[29471]: 5.4 scrub ok 2024-09-13T13:06:07.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[29471]: 6.d scrub starts 2024-09-13T13:06:07.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[29471]: 6.d scrub ok 2024-09-13T13:06:07.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[29471]: pgmap v227: 148 pgs: 148 active+clean; 455 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 691 op/s 2024-09-13T13:06:07.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[29471]: 4.1d scrub starts 2024-09-13T13:06:07.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[29471]: 4.1d scrub ok 2024-09-13T13:06:07.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:06:07.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:06:07.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[29471]: osdmap e160: 8 total, 8 up, 8 in 2024-09-13T13:06:07.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:06 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2141395302' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-09-13T13:06:07.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:06 smithi167 ceph-mon[33178]: 5.4 scrub starts 2024-09-13T13:06:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:06 smithi167 ceph-mon[33178]: 5.4 scrub ok 2024-09-13T13:06:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:06 smithi167 ceph-mon[33178]: 6.d scrub starts 2024-09-13T13:06:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:06 smithi167 ceph-mon[33178]: 6.d scrub ok 2024-09-13T13:06:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:06 smithi167 ceph-mon[33178]: pgmap v227: 148 pgs: 148 active+clean; 455 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 691 op/s 2024-09-13T13:06:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:06 smithi167 ceph-mon[33178]: 4.1d scrub starts 2024-09-13T13:06:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:06 smithi167 ceph-mon[33178]: 4.1d scrub ok 2024-09-13T13:06:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:06 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:06:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:06 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:06:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:06 smithi167 ceph-mon[33178]: osdmap e160: 8 total, 8 up, 8 in 2024-09-13T13:06:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:06 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2141395302' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-09-13T13:06:07.328 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:07.353 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:07.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:07.455 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:07.495 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:07.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:07.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:07.597 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:07.637 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:07.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:07.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:07.768 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:07.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:07.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:07.894 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:07.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:07.995 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:08.034 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:08.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:08.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:08.136 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:08.161 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:08.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:08.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:07 smithi052 ceph-mon[36682]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-13T13:06:08.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:07 smithi052 ceph-mon[36682]: 4.a scrub starts 2024-09-13T13:06:08.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:07 smithi052 ceph-mon[36682]: 4.a scrub ok 2024-09-13T13:06:08.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:07 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2141395302' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-09-13T13:06:08.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:07 smithi052 ceph-mon[36682]: osdmap e161: 8 total, 8 up, 8 in 2024-09-13T13:06:08.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:07 smithi052 ceph-mon[36682]: osdmap e162: 8 total, 8 up, 8 in 2024-09-13T13:06:08.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:07 smithi052 ceph-mon[29471]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-13T13:06:08.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:07 smithi052 ceph-mon[29471]: 4.a scrub starts 2024-09-13T13:06:08.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:07 smithi052 ceph-mon[29471]: 4.a scrub ok 2024-09-13T13:06:08.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:07 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2141395302' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-09-13T13:06:08.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:07 smithi052 ceph-mon[29471]: osdmap e161: 8 total, 8 up, 8 in 2024-09-13T13:06:08.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:07 smithi052 ceph-mon[29471]: osdmap e162: 8 total, 8 up, 8 in 2024-09-13T13:06:08.263 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:08.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:08.305 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:08.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:08.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:07 smithi167 ceph-mon[33178]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-13T13:06:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:07 smithi167 ceph-mon[33178]: 4.a scrub starts 2024-09-13T13:06:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:07 smithi167 ceph-mon[33178]: 4.a scrub ok 2024-09-13T13:06:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:07 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2141395302' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-09-13T13:06:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:07 smithi167 ceph-mon[33178]: osdmap e161: 8 total, 8 up, 8 in 2024-09-13T13:06:08.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:07 smithi167 ceph-mon[33178]: osdmap e162: 8 total, 8 up, 8 in 2024-09-13T13:06:08.400 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:06:08.407 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:08.435 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:08.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:08.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:08.537 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:08.594 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:08.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:08.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:08.696 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:08.722 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:08.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:08.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:08.849 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:08.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:08.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:08.951 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:08.997 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:08.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:08.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:09.099 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:09.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:09.147 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:09.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:09.171 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:08 smithi052 ceph-mon[36682]: 2.0 scrub starts 2024-09-13T13:06:09.172 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:08 smithi052 ceph-mon[36682]: 2.0 scrub ok 2024-09-13T13:06:09.172 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:08 smithi052 ceph-mon[36682]: pgmap v230: 148 pgs: 148 active+clean; 395 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 800 op/s 2024-09-13T13:06:09.172 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:08 smithi052 ceph-mon[36682]: 5.a scrub starts 2024-09-13T13:06:09.172 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:08 smithi052 ceph-mon[36682]: 5.a scrub ok 2024-09-13T13:06:09.172 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:08 smithi052 ceph-mon[29471]: 2.0 scrub starts 2024-09-13T13:06:09.172 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:08 smithi052 ceph-mon[29471]: 2.0 scrub ok 2024-09-13T13:06:09.172 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:08 smithi052 ceph-mon[29471]: pgmap v230: 148 pgs: 148 active+clean; 395 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 800 op/s 2024-09-13T13:06:09.172 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:08 smithi052 ceph-mon[29471]: 5.a scrub starts 2024-09-13T13:06:09.173 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:08 smithi052 ceph-mon[29471]: 5.a scrub ok 2024-09-13T13:06:09.248 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:09.286 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:09.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:09.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:09.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:08 smithi167 ceph-mon[33178]: 2.0 scrub starts 2024-09-13T13:06:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:08 smithi167 ceph-mon[33178]: 2.0 scrub ok 2024-09-13T13:06:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:08 smithi167 ceph-mon[33178]: pgmap v230: 148 pgs: 148 active+clean; 395 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 800 op/s 2024-09-13T13:06:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:08 smithi167 ceph-mon[33178]: 5.a scrub starts 2024-09-13T13:06:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:08 smithi167 ceph-mon[33178]: 5.a scrub ok 2024-09-13T13:06:09.387 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:09.425 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:09.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:09.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:09.527 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:09.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:09.582 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:09.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:09.683 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:09.734 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:09.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:09.835 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:09.874 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:06:09.874 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:06:09.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:09.896 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:09.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:09.941 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:06:09.978 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Removed 121098 objects 2024-09-13T13:06:09.979 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Clean up completed and total clean up time :137.8 2024-09-13T13:06:09.997 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:10.020 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:10.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:10.097 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[36682]: 6.12 scrub starts 2024-09-13T13:06:10.098 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[36682]: 6.12 scrub ok 2024-09-13T13:06:10.098 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[36682]: 5.2 scrub starts 2024-09-13T13:06:10.098 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[36682]: 5.2 scrub ok 2024-09-13T13:06:10.098 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1719609493' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:10.098 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[36682]: from='client.35129 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:10.098 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[36682]: 6.3 scrub starts 2024-09-13T13:06:10.098 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[36682]: 6.3 scrub ok 2024-09-13T13:06:10.098 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[29471]: 6.12 scrub starts 2024-09-13T13:06:10.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[29471]: 6.12 scrub ok 2024-09-13T13:06:10.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[29471]: 5.2 scrub starts 2024-09-13T13:06:10.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[29471]: 5.2 scrub ok 2024-09-13T13:06:10.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1719609493' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:10.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[29471]: from='client.35129 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:10.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[29471]: 6.3 scrub starts 2024-09-13T13:06:10.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:09 smithi052 ceph-mon[29471]: 6.3 scrub ok 2024-09-13T13:06:10.123 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:10.145 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:10.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:10.246 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:10.269 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:10.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:10.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:09 smithi167 ceph-mon[33178]: 6.12 scrub starts 2024-09-13T13:06:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:09 smithi167 ceph-mon[33178]: 6.12 scrub ok 2024-09-13T13:06:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:09 smithi167 ceph-mon[33178]: 5.2 scrub starts 2024-09-13T13:06:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:09 smithi167 ceph-mon[33178]: 5.2 scrub ok 2024-09-13T13:06:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:09 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1719609493' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:09 smithi167 ceph-mon[33178]: from='client.35129 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:09 smithi167 ceph-mon[33178]: 6.3 scrub starts 2024-09-13T13:06:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:09 smithi167 ceph-mon[33178]: 6.3 scrub ok 2024-09-13T13:06:10.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:10.392 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:10.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:10.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:10.494 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:10.515 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:10.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:10.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:10.614 INFO:tasks.thrashosds.thrasher:Setting full ratio back to .95 2024-09-13T13:06:10.614 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set-full-ratio .95 2024-09-13T13:06:10.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:10.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:10.639 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:10.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:10.731 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:10.740 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:10.766 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:10.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:10.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:10.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:10.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:10.890 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:10.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:10.983 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[36682]: 6.17 scrub starts 2024-09-13T13:06:10.983 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[36682]: 6.17 scrub ok 2024-09-13T13:06:10.984 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[36682]: pgmap v232: 148 pgs: 148 active+clean; 136 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-09-13T13:06:10.984 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:06:10.984 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[36682]: 4.1f scrub starts 2024-09-13T13:06:10.984 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[36682]: 4.1f scrub ok 2024-09-13T13:06:10.984 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[36682]: from='client.35129 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:06:10.984 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[36682]: osdmap e163: 8 total, 8 up, 8 in 2024-09-13T13:06:10.984 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[36682]: 7.6 scrub starts 2024-09-13T13:06:10.984 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[36682]: 7.6 scrub ok 2024-09-13T13:06:10.984 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[29471]: 6.17 scrub starts 2024-09-13T13:06:10.984 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[29471]: 6.17 scrub ok 2024-09-13T13:06:10.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[29471]: pgmap v232: 148 pgs: 148 active+clean; 136 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-09-13T13:06:10.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:06:10.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[29471]: 4.1f scrub starts 2024-09-13T13:06:10.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[29471]: 4.1f scrub ok 2024-09-13T13:06:10.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[29471]: from='client.35129 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:06:10.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[29471]: osdmap e163: 8 total, 8 up, 8 in 2024-09-13T13:06:10.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[29471]: 7.6 scrub starts 2024-09-13T13:06:10.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:10 smithi052 ceph-mon[29471]: 7.6 scrub ok 2024-09-13T13:06:10.991 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:11.033 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:11.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:11.135 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:11.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:11.158 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:11.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:11.260 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:11.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:11.283 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:11.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:11.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:10 smithi167 ceph-mon[33178]: 6.17 scrub starts 2024-09-13T13:06:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:10 smithi167 ceph-mon[33178]: 6.17 scrub ok 2024-09-13T13:06:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:10 smithi167 ceph-mon[33178]: pgmap v232: 148 pgs: 148 active+clean; 136 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-09-13T13:06:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:10 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:06:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:10 smithi167 ceph-mon[33178]: 4.1f scrub starts 2024-09-13T13:06:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:10 smithi167 ceph-mon[33178]: 4.1f scrub ok 2024-09-13T13:06:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:10 smithi167 ceph-mon[33178]: from='client.35129 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:06:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:10 smithi167 ceph-mon[33178]: osdmap e163: 8 total, 8 up, 8 in 2024-09-13T13:06:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:10 smithi167 ceph-mon[33178]: 7.6 scrub starts 2024-09-13T13:06:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:10 smithi167 ceph-mon[33178]: 7.6 scrub ok 2024-09-13T13:06:11.384 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:11.406 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:11.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:11.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:11.508 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:11.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:11.531 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:11.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:11.537 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:06:11.537 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:06:11.537 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:06:11.538 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:06:11.538 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:06:11.538 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:06:11.538 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:06:11.538 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:06:11.539 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:06:11.539 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:06:11.539 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:06:11.539 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:06:11.539 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:06:11.539 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:06:11.539 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:06:11.540 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:06:11.632 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:11.655 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:11.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:11.756 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:11.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:11.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:11.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:11.900 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:11.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:11.905 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T13:06:11.910 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[29471]: 5.10 scrub starts 2024-09-13T13:06:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[29471]: 5.10 scrub ok 2024-09-13T13:06:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[29471]: 3.1d scrub starts 2024-09-13T13:06:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[29471]: 3.1d scrub ok 2024-09-13T13:06:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[29471]: 3.2 scrub starts 2024-09-13T13:06:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[29471]: 3.2 scrub ok 2024-09-13T13:06:11.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[29471]: 5.5 scrub starts 2024-09-13T13:06:11.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[29471]: 5.5 scrub ok 2024-09-13T13:06:11.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2226710837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:11.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[36682]: 5.10 scrub starts 2024-09-13T13:06:11.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[36682]: 5.10 scrub ok 2024-09-13T13:06:11.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[36682]: 3.1d scrub starts 2024-09-13T13:06:11.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[36682]: 3.1d scrub ok 2024-09-13T13:06:11.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[36682]: 3.2 scrub starts 2024-09-13T13:06:11.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[36682]: 3.2 scrub ok 2024-09-13T13:06:11.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[36682]: 5.5 scrub starts 2024-09-13T13:06:11.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[36682]: 5.5 scrub ok 2024-09-13T13:06:11.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2226710837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:12.002 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:12.024 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:12.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:12.125 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:12.146 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:12.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:12.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:12.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:12.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:12.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:12.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:11 smithi167 ceph-mon[33178]: 5.10 scrub starts 2024-09-13T13:06:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:11 smithi167 ceph-mon[33178]: 5.10 scrub ok 2024-09-13T13:06:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:11 smithi167 ceph-mon[33178]: 3.1d scrub starts 2024-09-13T13:06:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:11 smithi167 ceph-mon[33178]: 3.1d scrub ok 2024-09-13T13:06:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:11 smithi167 ceph-mon[33178]: 3.2 scrub starts 2024-09-13T13:06:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:11 smithi167 ceph-mon[33178]: 3.2 scrub ok 2024-09-13T13:06:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:11 smithi167 ceph-mon[33178]: 5.5 scrub starts 2024-09-13T13:06:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:11 smithi167 ceph-mon[33178]: 5.5 scrub ok 2024-09-13T13:06:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2226710837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:12.369 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:12.371 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:12.394 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:12.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:12.496 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:12.518 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:12.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:12.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:12.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:12.642 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:12.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:12.735 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:06:12.744 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:12.790 INFO:teuthology.orchestra.run.smithi052.stderr:osd set-full-ratio 0.95 2024-09-13T13:06:12.804 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:12.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:12.906 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:12.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:12.931 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:12.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:13.032 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:13.054 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:13.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:13.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:13.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:12 smithi167 ceph-mon[33178]: 5.f scrub starts 2024-09-13T13:06:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:12 smithi167 ceph-mon[33178]: 5.f scrub ok 2024-09-13T13:06:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:12 smithi167 ceph-mon[33178]: pgmap v234: 148 pgs: 148 active+clean; 90 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 884 B/s rd, 0 B/s wr, 843 op/s 2024-09-13T13:06:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:12 smithi167 ceph-mon[33178]: 7.a scrub starts 2024-09-13T13:06:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:12 smithi167 ceph-mon[33178]: 7.a scrub ok 2024-09-13T13:06:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:12 smithi167 ceph-mon[33178]: 7.8 scrub starts 2024-09-13T13:06:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:12 smithi167 ceph-mon[33178]: 7.8 scrub ok 2024-09-13T13:06:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:12 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/385073561' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-13T13:06:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:12 smithi167 ceph-mon[33178]: from='client.35141 ' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-13T13:06:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:12 smithi167 ceph-mon[33178]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-09-13T13:06:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:12 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:06:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:12 smithi167 ceph-mon[33178]: from='client.35141 ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-09-13T13:06:13.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:12 smithi167 ceph-mon[33178]: osdmap e164: 8 total, 8 up, 8 in 2024-09-13T13:06:13.156 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:13.178 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:13.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:13.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:13.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[36682]: 5.f scrub starts 2024-09-13T13:06:13.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[36682]: 5.f scrub ok 2024-09-13T13:06:13.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[36682]: pgmap v234: 148 pgs: 148 active+clean; 90 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 884 B/s rd, 0 B/s wr, 843 op/s 2024-09-13T13:06:13.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[36682]: 7.a scrub starts 2024-09-13T13:06:13.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[36682]: 7.a scrub ok 2024-09-13T13:06:13.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[36682]: 7.8 scrub starts 2024-09-13T13:06:13.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[36682]: 7.8 scrub ok 2024-09-13T13:06:13.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/385073561' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-13T13:06:13.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[36682]: from='client.35141 ' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-13T13:06:13.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[36682]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-09-13T13:06:13.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:06:13.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[36682]: from='client.35141 ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-09-13T13:06:13.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[36682]: osdmap e164: 8 total, 8 up, 8 in 2024-09-13T13:06:13.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[29471]: 5.f scrub starts 2024-09-13T13:06:13.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[29471]: 5.f scrub ok 2024-09-13T13:06:13.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[29471]: pgmap v234: 148 pgs: 148 active+clean; 90 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 884 B/s rd, 0 B/s wr, 843 op/s 2024-09-13T13:06:13.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[29471]: 7.a scrub starts 2024-09-13T13:06:13.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[29471]: 7.a scrub ok 2024-09-13T13:06:13.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[29471]: 7.8 scrub starts 2024-09-13T13:06:13.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[29471]: 7.8 scrub ok 2024-09-13T13:06:13.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/385073561' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-13T13:06:13.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[29471]: from='client.35141 ' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-13T13:06:13.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[29471]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-09-13T13:06:13.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:06:13.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[29471]: from='client.35141 ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-09-13T13:06:13.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:12 smithi052 ceph-mon[29471]: osdmap e164: 8 total, 8 up, 8 in 2024-09-13T13:06:13.280 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:13.304 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:13.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:13.406 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:13.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:13.430 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:13.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:13.527 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:06:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:06:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:06:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:06:13.527 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:06:13.531 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:13.556 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:13.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:13.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:13.658 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:13.680 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:13.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:13.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:13.805 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:13.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:13.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:13.899 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-13T13:06:13.899 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-13T13:06:13.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:13.930 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:13.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:14.032 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:14.056 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:14.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:14.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:06:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:06:14.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:14.182 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:14.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:14.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:14.308 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:14.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:14.410 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:14.432 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:14.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:14.535 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:14.558 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:14.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:14.570 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T13:06:14.605 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:14.661 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:14.686 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:14.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:14.788 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:14.811 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:14.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:14.842 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:06:14.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:14.942 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:14.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:15.043 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:15.067 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:15.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:15.103 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:06:15.103 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:06:15.103 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[36682]: pgmap v236: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-13T13:06:15.103 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:15.103 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[36682]: osdmap e165: 8 total, 8 up, 8 in 2024-09-13T13:06:15.103 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:06:15.103 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:06:15.103 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:06:15.103 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:06:15.103 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[29471]: pgmap v236: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-13T13:06:15.104 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:15.104 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[29471]: osdmap e165: 8 total, 8 up, 8 in 2024-09-13T13:06:15.104 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:06:15.104 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:14 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:06:15.112 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:15.156 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:15.169 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:15.193 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:15.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:15.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:15.295 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:15.317 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:15.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:15.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:14 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:06:15.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:14 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:06:15.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:14 smithi167 ceph-mon[33178]: pgmap v236: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-13T13:06:15.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:14 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:15.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:14 smithi167 ceph-mon[33178]: osdmap e165: 8 total, 8 up, 8 in 2024-09-13T13:06:15.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:14 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:06:15.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:14 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:06:15.418 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:15.440 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:15.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:15.542 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:15.564 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:15.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:15.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:15.692 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:15.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:15.794 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:15.820 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:15.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:15.922 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:15.947 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:15.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:15.956 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'unique_pool_0' removed 2024-09-13T13:06:15.968 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:06:16.049 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:16.072 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:16.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:16.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:16.073 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:06:16.073 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:06:16.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:06:16.074 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:06:16.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.074 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:95502 7.e 7:73e7d3a1:::benchmark_data_smithi052_71865_object95501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:06:16.074 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.170950+0000", 2024-09-13T13:06:16.074 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 169.89955710000001, 2024-09-13T13:06:16.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10520948400000001, 2024-09-13T13:06:16.074 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.074 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.074 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.074 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.075 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.075 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.075 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95502 2024-09-13T13:06:16.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.075 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.075 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.075 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170950+0000", 2024-09-13T13:06:16.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.075 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.075 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170950+0000", 2024-09-13T13:06:16.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170952+0000", 2024-09-13T13:06:16.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.872e-06 2024-09-13T13:06:16.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170974+0000", 2024-09-13T13:06:16.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:06:16.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170982+0000", 2024-09-13T13:06:16.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6349999999999995e-06 2024-09-13T13:06:16.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170989+0000", 2024-09-13T13:06:16.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1619999999999996e-06 2024-09-13T13:06:16.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.171116+0000", 2024-09-13T13:06:16.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000127293 2024-09-13T13:06:16.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.171296+0000", 2024-09-13T13:06:16.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000180278 2024-09-13T13:06:16.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.171419+0000", 2024-09-13T13:06:16.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012266299999999999 2024-09-13T13:06:16.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.175039+0000", 2024-09-13T13:06:16.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036198469999999998 2024-09-13T13:06:16.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.175665+0000", 2024-09-13T13:06:16.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00062585400000000002 2024-09-13T13:06:16.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276069+0000", 2024-09-13T13:06:16.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10040423499999999 2024-09-13T13:06:16.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276119+0000", 2024-09-13T13:06:16.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0290999999999997e-05 2024-09-13T13:06:16.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276160+0000", 2024-09-13T13:06:16.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0516999999999998e-05 2024-09-13T13:06:16.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.081 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.081 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:95513 7.e 7:760c2885:::benchmark_data_smithi052_71865_object95512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:06:16.081 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.177025+0000", 2024-09-13T13:06:16.081 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 169.89348196899999, 2024-09-13T13:06:16.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.099458682000000007, 2024-09-13T13:06:16.081 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.081 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.082 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.082 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95513 2024-09-13T13:06:16.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.082 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177025+0000", 2024-09-13T13:06:16.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177025+0000", 2024-09-13T13:06:16.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177027+0000", 2024-09-13T13:06:16.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8109999999999999e-06 2024-09-13T13:06:16.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177059+0000", 2024-09-13T13:06:16.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2125000000000002e-05 2024-09-13T13:06:16.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177078+0000", 2024-09-13T13:06:16.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9239999999999999e-05 2024-09-13T13:06:16.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177085+0000", 2024-09-13T13:06:16.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9460000000000001e-06 2024-09-13T13:06:16.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177194+0000", 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010910500000000001 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177399+0000", 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000205019 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177525+0000", 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000125969 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276198+0000", 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098672383000000002 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276236+0000", 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7988000000000003e-05 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276248+0000", 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1858000000000001e-05 2024-09-13T13:06:16.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276284+0000", 2024-09-13T13:06:16.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6350000000000003e-05 2024-09-13T13:06:16.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276484+0000", 2024-09-13T13:06:16.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019988800000000001 2024-09-13T13:06:16.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.087 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.088 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95494 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:06:16.088 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.177847+0000", 2024-09-13T13:06:16.088 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 169.89265983499999, 2024-09-13T13:06:16.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098173131999999996, 2024-09-13T13:06:16.088 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.088 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.088 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:06:16.088 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177847+0000", 2024-09-13T13:06:16.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177847+0000", 2024-09-13T13:06:16.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177849+0000", 2024-09-13T13:06:16.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3179999999999999e-06 2024-09-13T13:06:16.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177853+0000", 2024-09-13T13:06:16.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1799999999999998e-06 2024-09-13T13:06:16.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177862+0000", 2024-09-13T13:06:16.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1390000000000001e-06 2024-09-13T13:06:16.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177868+0000", 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4119999999999998e-06 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180221+0000", 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023525130000000001 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180227+0000", 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4930000000000003e-06 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276020+0000", 2024-09-13T13:06:16.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095793077000000004 2024-09-13T13:06:16.092 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.092 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.092 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.092 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95502 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:06:16.092 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.180124+0000", 2024-09-13T13:06:16.092 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 169.89038314699999, 2024-09-13T13:06:16.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.09604087, 2024-09-13T13:06:16.092 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.092 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.093 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:06:16.093 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180124+0000", 2024-09-13T13:06:16.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180124+0000", 2024-09-13T13:06:16.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180125+0000", 2024-09-13T13:06:16.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.311e-06 2024-09-13T13:06:16.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180130+0000", 2024-09-13T13:06:16.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7500000000000003e-06 2024-09-13T13:06:16.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180140+0000", 2024-09-13T13:06:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9310000000000003e-06 2024-09-13T13:06:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180147+0000", 2024-09-13T13:06:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0600000000000002e-06 2024-09-13T13:06:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276055+0000", 2024-09-13T13:06:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095907611000000004 2024-09-13T13:06:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276063+0000", 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9719999999999992e-06 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276165+0000", 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102235 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95513 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:06:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.180192+0000", 2024-09-13T13:06:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 169.89031515799999, 2024-09-13T13:06:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.096011869999999999, 2024-09-13T13:06:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:06:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180192+0000", 2024-09-13T13:06:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180192+0000", 2024-09-13T13:06:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180193+0000", 2024-09-13T13:06:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.032e-06 2024-09-13T13:06:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180197+0000", 2024-09-13T13:06:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7079999999999999e-06 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180204+0000", 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2819999999999999e-06 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180209+0000", 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7330000000000003e-06 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276187+0000", 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095978049999999995 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276193+0000", 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1480000000000003e-06 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276204+0000", 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0917000000000001e-05 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95513 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.180247+0000", 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 169.89025987799999, 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095994805000000002, 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180247+0000", 2024-09-13T13:06:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180247+0000", 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180248+0000", 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.004e-06 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180252+0000", 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7890000000000001e-06 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180261+0000", 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5089999999999993e-06 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180267+0000", 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0079999999999999e-06 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276226+0000", 2024-09-13T13:06:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095959424000000001 2024-09-13T13:06:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276232+0000", 2024-09-13T13:06:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6189999999999999e-06 2024-09-13T13:06:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276242+0000", 2024-09-13T13:06:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0451999999999999e-05 2024-09-13T13:06:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:95658 7.e 7:79f5f817:::benchmark_data_smithi052_71865_object95657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:06:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.394482+0000", 2024-09-13T13:06:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 169.67602548400001, 2024-09-13T13:06:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.081968365000000001, 2024-09-13T13:06:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95658 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394482+0000", 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394482+0000", 2024-09-13T13:06:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394484+0000", 2024-09-13T13:06:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4660000000000002e-06 2024-09-13T13:06:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394512+0000", 2024-09-13T13:06:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7447999999999999e-05 2024-09-13T13:06:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394525+0000", 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3064000000000001e-05 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394533+0000", 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5930000000000006e-06 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.399009+0000", 2024-09-13T13:06:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0044759159999999999 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.399121+0000", 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011148400000000001 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.399244+0000", 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000123192 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.402074+0000", 2024-09-13T13:06:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0028305230000000002 2024-09-13T13:06:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.403012+0000", 2024-09-13T13:06:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00093742399999999996 2024-09-13T13:06:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.475889+0000", 2024-09-13T13:06:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.072877493000000002 2024-09-13T13:06:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.476237+0000", 2024-09-13T13:06:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00034722900000000001 2024-09-13T13:06:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.476450+0000", 2024-09-13T13:06:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021353299999999999 2024-09-13T13:06:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:96740 7.e 7:7b47086e:::benchmark_data_smithi052_71865_object96739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:27.478436+0000", 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 168.59207078899999, 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.082652384999999995, 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 96740 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478436+0000", 2024-09-13T13:06:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478436+0000", 2024-09-13T13:06:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478439+0000", 2024-09-13T13:06:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0990000000000001e-06 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478466+0000", 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7987000000000002e-05 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478477+0000", 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0587999999999999e-05 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478484+0000", 2024-09-13T13:06:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2799999999999998e-06 2024-09-13T13:06:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.479227+0000", 2024-09-13T13:06:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00074284600000000002 2024-09-13T13:06:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.479355+0000", 2024-09-13T13:06:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012828000000000001 2024-09-13T13:06:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.479490+0000", 2024-09-13T13:06:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000134859 2024-09-13T13:06:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.483095+0000", 2024-09-13T13:06:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036047169999999999 2024-09-13T13:06:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.483243+0000", 2024-09-13T13:06:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014764499999999999 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.560910+0000", 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.077667016000000005 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.561046+0000", 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013630099999999999 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.561089+0000", 2024-09-13T13:06:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2766999999999999e-05 2024-09-13T13:06:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:118717 7.e 7:7eac196e:::benchmark_data_smithi052_71865_object118716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:06:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 146.51028851999999, 2024-09-13T13:06:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.120995147, 2024-09-13T13:06:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 118717 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:06:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560220+0000", 2024-09-13T13:06:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4200000000000004e-07 2024-09-13T13:06:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560244+0000", 2024-09-13T13:06:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4354000000000001e-05 2024-09-13T13:06:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560252+0000", 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5189999999999992e-06 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560259+0000", 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.302e-06 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560301+0000", 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2318999999999999e-05 2024-09-13T13:06:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560415+0000", 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114269 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560572+0000", 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000156217 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.564231+0000", 2024-09-13T13:06:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036597489999999999 2024-09-13T13:06:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.565180+0000", 2024-09-13T13:06:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00094882300000000001 2024-09-13T13:06:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.680844+0000", 2024-09-13T13:06:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.115664164 2024-09-13T13:06:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.681157+0000", 2024-09-13T13:06:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00031297600000000002 2024-09-13T13:06:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.681214+0000", 2024-09-13T13:06:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6512999999999999e-05 2024-09-13T13:06:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120031 7.e 7:704b0327:::benchmark_data_smithi052_71865_object120030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 145.04970127000001, 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11824775799999999, 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120031 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020808+0000", 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0439999999999998e-06 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020838+0000", 2024-09-13T13:06:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9791e-05 2024-09-13T13:06:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020856+0000", 2024-09-13T13:06:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8664999999999998e-05 2024-09-13T13:06:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020866+0000", 2024-09-13T13:06:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3670000000000003e-06 2024-09-13T13:06:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021203+0000", 2024-09-13T13:06:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033735200000000002 2024-09-13T13:06:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021578+0000", 2024-09-13T13:06:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00037465399999999998 2024-09-13T13:06:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022015+0000", 2024-09-13T13:06:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000437176 2024-09-13T13:06:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.026523+0000", 2024-09-13T13:06:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045084249999999999 2024-09-13T13:06:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.027199+0000", 2024-09-13T13:06:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00067598400000000002 2024-09-13T13:06:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138915+0000", 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11171563499999999 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139008+0000", 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3329e-05 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139054+0000", 2024-09-13T13:06:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5336000000000001e-05 2024-09-13T13:06:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120050 7.e 7:72296fb6:::benchmark_data_smithi052_71865_object120049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:06:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 145.03765322500001, 2024-09-13T13:06:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12504636999999999, 2024-09-13T13:06:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120050 2024-09-13T13:06:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:06:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:06:16.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032857+0000", 2024-09-13T13:06:16.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1140000000000002e-06 2024-09-13T13:06:16.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032888+0000", 2024-09-13T13:06:16.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1115999999999998e-05 2024-09-13T13:06:16.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032905+0000", 2024-09-13T13:06:16.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6452000000000001e-05 2024-09-13T13:06:16.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032915+0000", 2024-09-13T13:06:16.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8889999999999997e-06 2024-09-13T13:06:16.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033665+0000", 2024-09-13T13:06:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00075015300000000004 2024-09-13T13:06:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033782+0000", 2024-09-13T13:06:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117642 2024-09-13T13:06:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033916+0000", 2024-09-13T13:06:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013400799999999999 2024-09-13T13:06:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.037705+0000", 2024-09-13T13:06:16.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037884289999999998 2024-09-13T13:06:16.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.037719+0000", 2024-09-13T13:06:16.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4034e-05 2024-09-13T13:06:16.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157684+0000", 2024-09-13T13:06:16.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.119964979 2024-09-13T13:06:16.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157805+0000", 2024-09-13T13:06:16.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120715 2024-09-13T13:06:16.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157900+0000", 2024-09-13T13:06:16.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5839000000000002e-05 2024-09-13T13:06:16.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.138 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120054 7.e 7:7dad863d:::benchmark_data_smithi052_71865_object120053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:06:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 145.032137949, 2024-09-13T13:06:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11988120300000001, 2024-09-13T13:06:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120054 2024-09-13T13:06:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:06:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:06:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038370+0000", 2024-09-13T13:06:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.051e-06 2024-09-13T13:06:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038395+0000", 2024-09-13T13:06:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4224999999999999e-05 2024-09-13T13:06:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038403+0000", 2024-09-13T13:06:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3399999999999998e-06 2024-09-13T13:06:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038409+0000", 2024-09-13T13:06:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3160000000000002e-06 2024-09-13T13:06:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038462+0000", 2024-09-13T13:06:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3012e-05 2024-09-13T13:06:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038579+0000", 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116483 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038720+0000", 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014139099999999999 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.042075+0000", 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033552830000000001 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.042095+0000", 2024-09-13T13:06:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9279999999999998e-05 2024-09-13T13:06:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158031+0000", 2024-09-13T13:06:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11593616299999999 2024-09-13T13:06:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158169+0000", 2024-09-13T13:06:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013835799999999999 2024-09-13T13:06:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158250+0000", 2024-09-13T13:06:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1301e-05 2024-09-13T13:06:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:135717 7.e 7:74ed6606:::benchmark_data_smithi052_71865_object14616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:06:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:09.110240+0000", 2024-09-13T13:06:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 126.960266885, 2024-09-13T13:06:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.076080909000000002, 2024-09-13T13:06:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 135717 2024-09-13T13:06:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110240+0000", 2024-09-13T13:06:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110240+0000", 2024-09-13T13:06:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110241+0000", 2024-09-13T13:06:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:06:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110243+0000", 2024-09-13T13:06:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4090000000000001e-06 2024-09-13T13:06:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110248+0000", 2024-09-13T13:06:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1250000000000001e-06 2024-09-13T13:06:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110253+0000", 2024-09-13T13:06:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8350000000000003e-06 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.113974+0000", 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003720737 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.114066+0000", 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1833e-05 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.114179+0000", 2024-09-13T13:06:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112695 2024-09-13T13:06:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.116490+0000", 2024-09-13T13:06:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023109210000000001 2024-09-13T13:06:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.116845+0000", 2024-09-13T13:06:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035593400000000002 2024-09-13T13:06:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.186206+0000", 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.069360755999999996 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.186246+0000", 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9888000000000002e-05 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.186321+0000", 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5136999999999996e-05 2024-09-13T13:06:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151582 7.e 7:7a3dbf39:::benchmark_data_smithi052_71865_object30481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:06:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.170441+0000", 2024-09-13T13:06:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 106.900066165, 2024-09-13T13:06:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.091679105999999996, 2024-09-13T13:06:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151582 2024-09-13T13:06:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170441+0000", 2024-09-13T13:06:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170441+0000", 2024-09-13T13:06:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170442+0000", 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5000000000000002e-07 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170445+0000", 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.88e-06 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170450+0000", 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3419999999999998e-06 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170455+0000", 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.096e-06 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174840+0000", 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0043852609999999997 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174948+0000", 2024-09-13T13:06:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010804 2024-09-13T13:06:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.175063+0000", 2024-09-13T13:06:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0001145 2024-09-13T13:06:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.178533+0000", 2024-09-13T13:06:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003470065 2024-09-13T13:06:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.178591+0000", 2024-09-13T13:06:16.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8149000000000002e-05 2024-09-13T13:06:16.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261725+0000", 2024-09-13T13:06:16.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.083133548000000002 2024-09-13T13:06:16.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261955+0000", 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00023013299999999999 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.262120+0000", 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016544199999999999 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151596 7.e 7:74fc7c8c:::benchmark_data_smithi052_71865_object30495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.187262+0000", 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 106.88324498, 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.084009381999999994, 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151596 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187262+0000", 2024-09-13T13:06:16.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187262+0000", 2024-09-13T13:06:16.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187264+0000", 2024-09-13T13:06:16.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2199999999999999e-06 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187273+0000", 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4500000000000004e-06 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187285+0000", 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2446000000000001e-05 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187293+0000", 2024-09-13T13:06:16.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4000000000000003e-06 2024-09-13T13:06:16.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187360+0000", 2024-09-13T13:06:16.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6908000000000004e-05 2024-09-13T13:06:16.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187487+0000", 2024-09-13T13:06:16.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000126859 2024-09-13T13:06:16.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187631+0000", 2024-09-13T13:06:16.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014448999999999999 2024-09-13T13:06:16.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193046+0000", 2024-09-13T13:06:16.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.005415352 2024-09-13T13:06:16.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193098+0000", 2024-09-13T13:06:16.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1375999999999997e-05 2024-09-13T13:06:16.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.271006+0000", 2024-09-13T13:06:16.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.077907893000000006 2024-09-13T13:06:16.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.271225+0000", 2024-09-13T13:06:16.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021893800000000001 2024-09-13T13:06:16.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.271272+0000", 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7049999999999998e-05 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151600 7.e 7:7fe6ca51:::benchmark_data_smithi052_71865_object30499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.189256+0000", 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 106.881251537, 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.086529871999999994, 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.165 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151600 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189256+0000", 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189256+0000", 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189257+0000", 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7209999999999999e-06 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189263+0000", 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3580000000000002e-06 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189275+0000", 2024-09-13T13:06:16.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1973e-05 2024-09-13T13:06:16.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189283+0000", 2024-09-13T13:06:16.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9039999999999992e-06 2024-09-13T13:06:16.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.190358+0000", 2024-09-13T13:06:16.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010753099999999999 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.190465+0000", 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010736800000000001 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.190557+0000", 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193101+0000", 2024-09-13T13:06:16.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025442569999999999 2024-09-13T13:06:16.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193130+0000", 2024-09-13T13:06:16.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8728999999999999e-05 2024-09-13T13:06:16.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.275647+0000", 2024-09-13T13:06:16.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.082517141000000002 2024-09-13T13:06:16.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.275742+0000", 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5156999999999997e-05 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.275786+0000", 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3047000000000003e-05 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.171 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:153862 7.e 7:797e5c7a:::benchmark_data_smithi052_71865_object32761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:06:16.172 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:06:16.172 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 103.80570047800001, 2024-09-13T13:06:16.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.106314172, 2024-09-13T13:06:16.172 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.172 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.172 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.172 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.172 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.172 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.172 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 153862 2024-09-13T13:06:16.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.172 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264809+0000", 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.148e-06 2024-09-13T13:06:16.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264815+0000", 2024-09-13T13:06:16.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7610000000000004e-06 2024-09-13T13:06:16.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264826+0000", 2024-09-13T13:06:16.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1215e-05 2024-09-13T13:06:16.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264833+0000", 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0090000000000001e-06 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264885+0000", 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1845000000000003e-05 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.265001+0000", 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116795 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.176 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.265104+0000", 2024-09-13T13:06:16.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102979 2024-09-13T13:06:16.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.176 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:16.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.268877+0000", 2024-09-13T13:06:16.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037720779999999999 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.269086+0000", 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000209829 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.370873+0000", 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10178622700000001 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.371063+0000", 2024-09-13T13:06:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000190349 2024-09-13T13:06:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.371121+0000", 2024-09-13T13:06:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7936999999999998e-05 2024-09-13T13:06:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:153884 7.8 7:1006e86e:::benchmark_data_smithi052_71865_object32783:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:06:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:32.288977+0000", 2024-09-13T13:06:16.180 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 103.78152971199999, 2024-09-13T13:06:16.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.086944270000000004, 2024-09-13T13:06:16.180 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.180 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.180 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.180 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.180 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.180 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.180 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 153884 2024-09-13T13:06:16.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.180 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288977+0000", 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288977+0000", 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288979+0000", 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9e-06 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288985+0000", 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4580000000000001e-06 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288997+0000", 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2004999999999999e-05 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.289005+0000", 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0069999999999997e-06 2024-09-13T13:06:16.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.289044+0000", 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9205000000000002e-05 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.289111+0000", 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6952000000000005e-05 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.359719+0000", 2024-09-13T13:06:16.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.070607721999999998 2024-09-13T13:06:16.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.184 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.363436+0000", 2024-09-13T13:06:16.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037171370000000001 2024-09-13T13:06:16.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.184 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375313+0000", 2024-09-13T13:06:16.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011876639 2024-09-13T13:06:16.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375551+0000", 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000238939 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375761+0000", 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021001599999999999 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375922+0000", 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016029 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.185 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:153886 7.e 7:718ea587:::benchmark_data_smithi052_71865_object32785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:32.293022+0000", 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 103.777485349, 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.081446456, 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.186 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 153886 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.293022+0000", 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.293022+0000", 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.293025+0000", 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7580000000000001e-06 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.293032+0000", 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7619999999999995e-06 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.357646+0000", 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.064613241000000002 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.357665+0000", 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9284999999999999e-05 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.358627+0000", 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00096251400000000003 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.358770+0000", 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014308699999999999 2024-09-13T13:06:16.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.358905+0000", 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000134785 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.362681+0000", 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037752800000000002 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.364635+0000", 2024-09-13T13:06:16.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019547779999999999 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.374309+0000", 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0096736170000000007 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.374416+0000", 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107389 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.374468+0000", 2024-09-13T13:06:16.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1959999999999997e-05 2024-09-13T13:06:16.192 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.192 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.192 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.192 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:199092 7.e 7:7925136c:::benchmark_data_smithi052_71865_object77991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T13:06:16.192 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:05:25.233281+0000", 2024-09-13T13:06:16.192 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 50.837226385000001, 2024-09-13T13:06:16.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.076015174000000005, 2024-09-13T13:06:16.192 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:16.192 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:16.192 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 199092 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233281+0000", 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:16.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233281+0000", 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233282+0000", 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2929999999999999e-06 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233287+0000", 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0039999999999999e-06 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233297+0000", 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8509999999999994e-06 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233304+0000", 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6680000000000004e-06 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233364+0000", 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0167999999999997e-05 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233462+0000", 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7813999999999996e-05 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233561+0000", 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9557999999999998e-05 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.236795+0000", 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0032333890000000001 2024-09-13T13:06:16.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.236836+0000", 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1826000000000002e-05 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.309149+0000", 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.072312716999999999 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.309250+0000", 2024-09-13T13:06:16.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010125599999999999 2024-09-13T13:06:16.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:16.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:16.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:16.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.309296+0000", 2024-09-13T13:06:16.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5630000000000002e-05 2024-09-13T13:06:16.198 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.198 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.198 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.198 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:16.198 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:16.198 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:06:16.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:16.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:16.233 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[29471]: 3.d scrub starts 2024-09-13T13:06:16.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[29471]: 3.d scrub ok 2024-09-13T13:06:16.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[29471]: 7.2 scrub starts 2024-09-13T13:06:16.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[29471]: 7.2 scrub ok 2024-09-13T13:06:16.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[29471]: osdmap e166: 8 total, 8 up, 8 in 2024-09-13T13:06:16.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1484184425' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:06:16.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1019288876' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:06:16.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[29471]: from='client.35153 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:06:16.234 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:06:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:06:16.235 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[36682]: 3.d scrub starts 2024-09-13T13:06:16.235 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[36682]: 3.d scrub ok 2024-09-13T13:06:16.235 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[36682]: 7.2 scrub starts 2024-09-13T13:06:16.235 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[36682]: 7.2 scrub ok 2024-09-13T13:06:16.235 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[36682]: osdmap e166: 8 total, 8 up, 8 in 2024-09-13T13:06:16.235 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1484184425' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:06:16.235 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1019288876' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:06:16.235 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:15 smithi052 ceph-mon[36682]: from='client.35153 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:06:16.302 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:16.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:15 smithi167 ceph-mon[33178]: 3.d scrub starts 2024-09-13T13:06:16.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:15 smithi167 ceph-mon[33178]: 3.d scrub ok 2024-09-13T13:06:16.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:15 smithi167 ceph-mon[33178]: 7.2 scrub starts 2024-09-13T13:06:16.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:15 smithi167 ceph-mon[33178]: 7.2 scrub ok 2024-09-13T13:06:16.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:15 smithi167 ceph-mon[33178]: osdmap e166: 8 total, 8 up, 8 in 2024-09-13T13:06:16.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:15 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1484184425' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:06:16.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:15 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1019288876' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:06:16.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:15 smithi167 ceph-mon[33178]: from='client.35153 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:06:16.324 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:16.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:16.426 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:16.449 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:16.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:16.550 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:16.573 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:16.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:16.674 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:16.697 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:16.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:16.799 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:16.822 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:16.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:16.923 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:16.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:16.947 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:16.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:17.048 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:17.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:17.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:17.197 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:17.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:17.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[36682]: 6.8 scrub starts 2024-09-13T13:06:17.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[36682]: 6.8 scrub ok 2024-09-13T13:06:17.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[36682]: pgmap v239: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 888 B/s rd, 0 B/s wr, 378 op/s 2024-09-13T13:06:17.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:17.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1484184425' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:06:17.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[36682]: from='client.35153 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:06:17.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[36682]: osdmap e167: 8 total, 8 up, 8 in 2024-09-13T13:06:17.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[29471]: 6.8 scrub starts 2024-09-13T13:06:17.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[29471]: 6.8 scrub ok 2024-09-13T13:06:17.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[29471]: pgmap v239: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 888 B/s rd, 0 B/s wr, 378 op/s 2024-09-13T13:06:17.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:17.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1484184425' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:06:17.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[29471]: from='client.35153 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:06:17.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:16 smithi052 ceph-mon[29471]: osdmap e167: 8 total, 8 up, 8 in 2024-09-13T13:06:17.300 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:17.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:16 smithi167 ceph-mon[33178]: 6.8 scrub starts 2024-09-13T13:06:17.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:16 smithi167 ceph-mon[33178]: 6.8 scrub ok 2024-09-13T13:06:17.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:16 smithi167 ceph-mon[33178]: pgmap v239: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 888 B/s rd, 0 B/s wr, 378 op/s 2024-09-13T13:06:17.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:16 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:17.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:16 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1484184425' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:06:17.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:16 smithi167 ceph-mon[33178]: from='client.35153 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:06:17.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:16 smithi167 ceph-mon[33178]: osdmap e167: 8 total, 8 up, 8 in 2024-09-13T13:06:17.322 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:17.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:17.424 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:17.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:17.449 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:17.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:17.550 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:17.573 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:17.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:17.675 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:17.698 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:17.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:17.800 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:17.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:17.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:17.916 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T13:06:17.917 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T13:06:17.917 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-13T13:06:17.917 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-13T13:06:17.925 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:17.983 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:17.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:17.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:18.085 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:18.107 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:18.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:18.135 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:18.210 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:18.234 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:18.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:18.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:18.336 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:18.359 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:18.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:18.460 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:18.483 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:18.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:18.570 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T13:06:18.583 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:18.638 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:18.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:18.740 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:18.767 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:18.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:18.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:18.893 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:18.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:18.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:18.995 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:19.021 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:19.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:19.031 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:19.064 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:06:19.064 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:06:19.064 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:18 smithi052 ceph-mon[36682]: pgmap v241: 132 pgs: 132 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-09-13T13:06:19.065 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:18 smithi052 ceph-mon[36682]: osdmap e168: 8 total, 8 up, 8 in 2024-09-13T13:06:19.065 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:18 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:06:19.065 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:18 smithi052 ceph-mon[29471]: pgmap v241: 132 pgs: 132 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-09-13T13:06:19.065 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:18 smithi052 ceph-mon[29471]: osdmap e168: 8 total, 8 up, 8 in 2024-09-13T13:06:19.065 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:18 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:06:19.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:18 smithi167 ceph-mon[33178]: pgmap v241: 132 pgs: 132 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-09-13T13:06:19.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:18 smithi167 ceph-mon[33178]: osdmap e168: 8 total, 8 up, 8 in 2024-09-13T13:06:19.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:18 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:06:19.075 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:19.083 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:06:19.083 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:06:19.100 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:06:19.101 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:06:19.117 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:06:19.117 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:06:19.122 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:19.134 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:06:19.134 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:06:19.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:19.147 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:19.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:19.152 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:06:19.152 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:06:19.169 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:06:19.170 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:06:19.182 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:06:19.183 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T13:06:19.186 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:06:19.187 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:06:19.248 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:19.270 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:19.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:19.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:19.372 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:19.395 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:19.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:19.483 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:06:19.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:19.523 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:19.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:19.625 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:19.652 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:19.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:19.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:19.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:19.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:19.778 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:19.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:19.879 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:19.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:19.902 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:19.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:20.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:20.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:20.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:20.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:20.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:19 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2530916891' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:20.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:20.135 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:19 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2530916891' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:20.136 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:19 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2530916891' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:20.157 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:20.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:20.259 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:20.281 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:20.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:20.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:20.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:20.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:20.410 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:20.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:20.511 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:20.534 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:20.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:20.635 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:20.661 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:20.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:20.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:20.764 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:20.788 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:20.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:20.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:20.801 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'unique_pool_1' created 2024-09-13T13:06:20.889 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:20.943 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:20.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:20.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:20.966 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:06:20.966 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:06:21.046 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:21.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:20 smithi167 ceph-mon[33178]: pgmap v243: 132 pgs: 132 active+clean; 583 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 712 B/s rd, 0 op/s 2024-09-13T13:06:21.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:20 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/638827221' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T13:06:21.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:20 smithi167 ceph-mon[33178]: from='client.35165 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T13:06:21.071 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:21.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:21.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:21.154 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:20 smithi052 ceph-mon[36682]: pgmap v243: 132 pgs: 132 active+clean; 583 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 712 B/s rd, 0 op/s 2024-09-13T13:06:21.154 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:20 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/638827221' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T13:06:21.154 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:20 smithi052 ceph-mon[36682]: from='client.35165 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T13:06:21.155 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:20 smithi052 ceph-mon[29471]: pgmap v243: 132 pgs: 132 active+clean; 583 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 712 B/s rd, 0 op/s 2024-09-13T13:06:21.155 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:20 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/638827221' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T13:06:21.155 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:20 smithi052 ceph-mon[29471]: from='client.35165 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T13:06:21.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:21.199 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:21.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:21.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:21.301 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:21.325 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:21.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:21.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:21.427 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:21.449 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:21.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:21.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:21.552 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:21.574 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:21.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:21.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:21.702 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:21.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:21.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:21.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:21.832 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:21.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:21.874 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-13T13:06:21.934 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:21.959 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:21.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:22.061 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:22.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:21 smithi167 ceph-mon[33178]: from='client.35165 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T13:06:22.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:21 smithi167 ceph-mon[33178]: osdmap e169: 8 total, 8 up, 8 in 2024-09-13T13:06:22.087 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:22.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:22.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:22.189 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:22.211 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:22.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:22.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:22.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:21 smithi052 ceph-mon[36682]: from='client.35165 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T13:06:22.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:21 smithi052 ceph-mon[36682]: osdmap e169: 8 total, 8 up, 8 in 2024-09-13T13:06:22.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:21 smithi052 ceph-mon[29471]: from='client.35165 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T13:06:22.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:21 smithi052 ceph-mon[29471]: osdmap e169: 8 total, 8 up, 8 in 2024-09-13T13:06:22.314 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:22.316 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T13:06:22.340 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:22.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:22.348 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:22.394 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:22.399 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:22.421 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:06:22.442 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:22.487 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:22.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:22.589 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:22.614 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:22.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:22.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:22.715 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:22.741 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:22.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:22.843 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:22.867 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:22.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:22.968 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:22.991 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:22.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:22.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:23.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:22 smithi167 ceph-mon[33178]: pgmap v245: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 583 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-13T13:06:23.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:22 smithi167 ceph-mon[33178]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:06:23.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:22 smithi167 ceph-mon[33178]: osdmap e170: 8 total, 8 up, 8 in 2024-09-13T13:06:23.093 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:23.115 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:23.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:23.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:23.155 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:22 smithi052 ceph-mon[36682]: pgmap v245: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 583 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-13T13:06:23.156 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:22 smithi052 ceph-mon[36682]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:06:23.156 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:22 smithi052 ceph-mon[36682]: osdmap e170: 8 total, 8 up, 8 in 2024-09-13T13:06:23.156 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:22 smithi052 ceph-mon[29471]: pgmap v245: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 583 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-13T13:06:23.156 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:22 smithi052 ceph-mon[29471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:06:23.156 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:22 smithi052 ceph-mon[29471]: osdmap e170: 8 total, 8 up, 8 in 2024-09-13T13:06:23.220 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:23.247 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:23.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:23.349 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:23.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:23.372 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:23.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:23.473 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:23.499 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:23.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:23.601 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:23.625 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:23.627 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:23.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:23.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:23.729 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:23.751 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:23.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:23.817 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T13:06:23.817 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:06:23.854 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:23.880 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:23.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:23.982 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:24.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:24.006 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:24.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:24.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:06:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:06:24.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:23 smithi167 ceph-mon[33178]: osdmap e171: 8 total, 8 up, 8 in 2024-09-13T13:06:24.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:23 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/560641762' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:06:24.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:23 smithi167 ceph-mon[33178]: from='client.35171 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:06:24.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:23 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1929270769' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:24.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:23 smithi167 ceph-mon[33178]: from='client.35177 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:24.107 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:24.129 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:24.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:24.212 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:23 smithi052 ceph-mon[36682]: osdmap e171: 8 total, 8 up, 8 in 2024-09-13T13:06:24.213 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:23 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/560641762' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:06:24.213 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:23 smithi052 ceph-mon[36682]: from='client.35171 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:06:24.213 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:23 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1929270769' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:24.213 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:23 smithi052 ceph-mon[36682]: from='client.35177 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:24.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:23 smithi052 ceph-mon[29471]: osdmap e171: 8 total, 8 up, 8 in 2024-09-13T13:06:24.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:23 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/560641762' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:06:24.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:23 smithi052 ceph-mon[29471]: from='client.35171 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:06:24.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:23 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1929270769' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:24.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:23 smithi052 ceph-mon[29471]: from='client.35177 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:24.231 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:24.253 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:24.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:24.355 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:24.377 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:24.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:24.479 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:24.505 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:24.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:24.521 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:06:24.521 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:06:24.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:24.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:24.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:24.670 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:24.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:24.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:24.761 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:24.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:24.863 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:24.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:24.889 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:24.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:24.990 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:25.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:25.016 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:25.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:25.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:24 smithi167 ceph-mon[33178]: pgmap v248: 148 pgs: 2 creating+peering, 146 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T13:06:25.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:24 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:25.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:24 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:25.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:24 smithi167 ceph-mon[33178]: from='client.35171 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:06:25.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:24 smithi167 ceph-mon[33178]: from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:06:25.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:24 smithi167 ceph-mon[33178]: osdmap e172: 8 total, 8 up, 8 in 2024-09-13T13:06:25.118 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:25.143 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:25.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:25.184 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:24 smithi052 ceph-mon[36682]: pgmap v248: 148 pgs: 2 creating+peering, 146 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T13:06:25.184 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:24 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:25.184 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:24 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:25.184 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:24 smithi052 ceph-mon[36682]: from='client.35171 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:06:25.184 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:24 smithi052 ceph-mon[36682]: from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:06:25.184 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:24 smithi052 ceph-mon[36682]: osdmap e172: 8 total, 8 up, 8 in 2024-09-13T13:06:25.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:24 smithi052 ceph-mon[29471]: pgmap v248: 148 pgs: 2 creating+peering, 146 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T13:06:25.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:24 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:25.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:24 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:25.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:24 smithi052 ceph-mon[29471]: from='client.35171 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:06:25.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:24 smithi052 ceph-mon[29471]: from='client.35177 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:06:25.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:24 smithi052 ceph-mon[29471]: osdmap e172: 8 total, 8 up, 8 in 2024-09-13T13:06:25.221 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T13:06:25.245 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:25.288 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:25.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:25.390 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:25.416 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:25.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:25.518 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:25.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:25.545 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:25.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:25.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:25.674 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:25.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:25.709 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T13:06:25.775 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:25.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:25.802 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:25.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:25.903 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:25.924 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:25.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:26.026 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:26.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:26.054 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:26.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:26.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:26.181 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:26.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:26.283 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:26.309 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:26.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:26.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:26.410 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:26.433 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:26.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:26.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:26.442 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:06:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:06:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:06:26.535 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:26.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:26.564 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:26.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:26.665 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:26.690 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:26.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:26.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:26.792 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:26.815 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:26.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:26.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:26.917 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:26.941 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:26.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:26.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:26.985 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:26 smithi052 ceph-mon[36682]: pgmap v250: 148 pgs: 2 creating+peering, 146 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T13:06:26.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:26 smithi052 ceph-mon[29471]: pgmap v250: 148 pgs: 2 creating+peering, 146 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T13:06:27.044 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:27.069 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:27.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:27.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:26 smithi167 ceph-mon[33178]: pgmap v250: 148 pgs: 2 creating+peering, 146 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T13:06:27.171 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:27.195 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:27.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:27.240 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:06:27.241 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:06:27.258 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:06:27.258 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:06:27.274 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:27.274 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:27.275 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:06:27.276 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:06:27.297 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:27.303 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:06:27.303 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:06:27.321 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:27.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:27.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:27.322 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:06:27.323 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:06:27.339 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:06:27.339 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:06:27.359 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:06:27.359 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:06:27.376 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:06:27.376 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:06:27.422 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:27.446 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:27.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:27.547 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:27.572 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:27.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:27.673 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:27.695 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:27.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:27.797 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:27.821 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:27.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:27.827 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-13T13:06:27.924 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:27.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:27.948 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:27.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:28.024 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:06:28.025 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:06:28.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:06:28.025 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:06:28.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.025 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:52591 7.3 7:c0542627:::benchmark_data_smithi052_71865_object52590:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:06:28.025 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:02:48.958124+0000", 2024-09-13T13:06:28.025 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 219.064504978, 2024-09-13T13:06:28.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.077973423, 2024-09-13T13:06:28.025 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.025 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.025 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.026 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.026 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.026 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.026 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 52591 2024-09-13T13:06:28.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.026 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.026 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.026 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958124+0000", 2024-09-13T13:06:28.026 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.027 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.027 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958124+0000", 2024-09-13T13:06:28.027 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.027 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.027 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958125+0000", 2024-09-13T13:06:28.027 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4640000000000001e-06 2024-09-13T13:06:28.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.027 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.027 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958152+0000", 2024-09-13T13:06:28.027 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6265999999999999e-05 2024-09-13T13:06:28.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.028 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.028 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958166+0000", 2024-09-13T13:06:28.028 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4688999999999999e-05 2024-09-13T13:06:28.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.028 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.029 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:48.958174+0000", 2024-09-13T13:06:28.029 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9049999999999997e-06 2024-09-13T13:06:28.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.029 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.029 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.029 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.028638+0000", 2024-09-13T13:06:28.029 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.070464099000000002 2024-09-13T13:06:28.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.029 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.029 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.029 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.028739+0000", 2024-09-13T13:06:28.029 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000100789 2024-09-13T13:06:28.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.030 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:06:28.030 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.028857+0000", 2024-09-13T13:06:28.030 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117465 2024-09-13T13:06:28.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.030 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.030 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.031772+0000", 2024-09-13T13:06:28.030 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002915071 2024-09-13T13:06:28.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.030 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.031 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.032045+0000", 2024-09-13T13:06:28.031 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027313400000000002 2024-09-13T13:06:28.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.031 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.031 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.035995+0000", 2024-09-13T13:06:28.031 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003950037 2024-09-13T13:06:28.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.032 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.032 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.036051+0000", 2024-09-13T13:06:28.032 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6215000000000003e-05 2024-09-13T13:06:28.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.032 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.032 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:02:49.036097+0000", 2024-09-13T13:06:28.032 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6288999999999997e-05 2024-09-13T13:06:28.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.032 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83963 7.c 7:30d6a006:::benchmark_data_smithi052_71865_object83962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 193.474746842, 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.013163161, 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.033 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83963 2024-09-13T13:06:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:06:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:06:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547884+0000", 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.779e-06 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547907+0000", 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2779999999999999e-05 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547917+0000", 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8439999999999993e-06 2024-09-13T13:06:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:06:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5229999999999998e-06 2024-09-13T13:06:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554417+0000", 2024-09-13T13:06:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064933880000000001 2024-09-13T13:06:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554427+0000", 2024-09-13T13:06:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4280000000000002e-06 2024-09-13T13:06:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.543883+0000", 2024-09-13T13:06:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98945604700000001 2024-09-13T13:06:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545504+0000", 2024-09-13T13:06:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016211660000000001 2024-09-13T13:06:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545747+0000", 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024302199999999999 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552352+0000", 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.006605133 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560880+0000", 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0085281650000000007 2024-09-13T13:06:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560953+0000", 2024-09-13T13:06:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2096999999999998e-05 2024-09-13T13:06:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561045+0000", 2024-09-13T13:06:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2788999999999995e-05 2024-09-13T13:06:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.040 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83964 7.c 7:37783165:::benchmark_data_smithi052_71865_object83963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:06:28.041 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:06:28.041 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 193.47461067500001, 2024-09-13T13:06:28.041 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0132684729999999, 2024-09-13T13:06:28.041 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.041 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.041 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.041 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.041 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.041 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83964 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548019+0000", 2024-09-13T13:06:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0279999999999999e-06 2024-09-13T13:06:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548044+0000", 2024-09-13T13:06:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4581999999999999e-05 2024-09-13T13:06:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548052+0000", 2024-09-13T13:06:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3799999999999994e-06 2024-09-13T13:06:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548059+0000", 2024-09-13T13:06:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8329999999999996e-06 2024-09-13T13:06:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554452+0000", 2024-09-13T13:06:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0063932119999999997 2024-09-13T13:06:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554459+0000", 2024-09-13T13:06:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3319999999999997e-06 2024-09-13T13:06:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552381+0000", 2024-09-13T13:06:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99792229399999999 2024-09-13T13:06:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552485+0000", 2024-09-13T13:06:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010354800000000001 2024-09-13T13:06:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:06:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552626+0000", 2024-09-13T13:06:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014122000000000001 2024-09-13T13:06:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560863+0000", 2024-09-13T13:06:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0082373700000000008 2024-09-13T13:06:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561139+0000", 2024-09-13T13:06:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027546899999999997 2024-09-13T13:06:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561192+0000", 2024-09-13T13:06:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2986999999999999e-05 2024-09-13T13:06:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561287+0000", 2024-09-13T13:06:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5217999999999995e-05 2024-09-13T13:06:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83985 7.c 7:386b3d8b:::benchmark_data_smithi052_71865_object83984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:06:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:06:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 193.45468932099999, 2024-09-13T13:06:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99349958199999999, 2024-09-13T13:06:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83985 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:06:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567942+0000", 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.888e-06 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567971+0000", 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9380999999999999e-05 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568031+0000", 2024-09-13T13:06:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9731000000000001e-05 2024-09-13T13:06:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568039+0000", 2024-09-13T13:06:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8109999999999992e-06 2024-09-13T13:06:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568102+0000", 2024-09-13T13:06:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2271000000000001e-05 2024-09-13T13:06:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568109+0000", 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2570000000000001e-06 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.557963+0000", 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.989854491 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.558066+0000", 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102929 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:06:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.558206+0000", 2024-09-13T13:06:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013928000000000001 2024-09-13T13:06:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561121+0000", 2024-09-13T13:06:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029151849999999998 2024-09-13T13:06:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561301+0000", 2024-09-13T13:06:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018019299999999999 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561349+0000", 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7639999999999998e-05 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561439+0000", 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0525000000000005e-05 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83989 7.c 7:3cf9e082:::benchmark_data_smithi052_71865_object83988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:06:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:06:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 193.44804711800001, 2024-09-13T13:06:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99086929999999995, 2024-09-13T13:06:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83989 2024-09-13T13:06:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:06:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:06:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574584+0000", 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0480000000000001e-06 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574603+0000", 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8913e-05 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574615+0000", 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1678000000000001e-05 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574622+0000", 2024-09-13T13:06:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:06:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574674+0000", 2024-09-13T13:06:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1053000000000001e-05 2024-09-13T13:06:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:06:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574681+0000", 2024-09-13T13:06:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0110000000000003e-06 2024-09-13T13:06:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561482+0000", 2024-09-13T13:06:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98680099200000004 2024-09-13T13:06:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561622+0000", 2024-09-13T13:06:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014003799999999999 2024-09-13T13:06:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:06:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561773+0000", 2024-09-13T13:06:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015180599999999999 2024-09-13T13:06:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.564308+0000", 2024-09-13T13:06:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025348390000000001 2024-09-13T13:06:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565272+0000", 2024-09-13T13:06:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00096360599999999999 2024-09-13T13:06:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565362+0000", 2024-09-13T13:06:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0253000000000005e-05 2024-09-13T13:06:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565451+0000", 2024-09-13T13:06:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9090000000000003e-05 2024-09-13T13:06:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:119477 7.a 7:5b6c862d:::benchmark_data_smithi052_71865_object119476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:50.468842+0000", 2024-09-13T13:06:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 157.553787218, 2024-09-13T13:06:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.090049216000000001, 2024-09-13T13:06:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 119477 2024-09-13T13:06:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.468842+0000", 2024-09-13T13:06:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.468842+0000", 2024-09-13T13:06:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.468844+0000", 2024-09-13T13:06:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.832e-06 2024-09-13T13:06:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.468867+0000", 2024-09-13T13:06:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3852000000000001e-05 2024-09-13T13:06:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.468878+0000", 2024-09-13T13:06:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1004e-05 2024-09-13T13:06:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.468885+0000", 2024-09-13T13:06:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0190000000000001e-06 2024-09-13T13:06:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.469113+0000", 2024-09-13T13:06:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00022774399999999999 2024-09-13T13:06:28.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.469233+0000", 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120247 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,5", 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.469373+0000", 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000139114 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.477078+0000", 2024-09-13T13:06:28.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0077054050000000002 2024-09-13T13:06:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.478765+0000", 2024-09-13T13:06:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001686788 2024-09-13T13:06:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.557874+0000", 2024-09-13T13:06:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.079109309000000003 2024-09-13T13:06:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.558778+0000", 2024-09-13T13:06:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00090368399999999998 2024-09-13T13:06:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.558891+0000", 2024-09-13T13:06:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113218 2024-09-13T13:06:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.068 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:119480 7.2 7:42bd7fbf:::benchmark_data_smithi052_71865_object119479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:50.470021+0000", 2024-09-13T13:06:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 157.55260833099999, 2024-09-13T13:06:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.088160868000000003, 2024-09-13T13:06:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 119480 2024-09-13T13:06:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.470021+0000", 2024-09-13T13:06:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.470021+0000", 2024-09-13T13:06:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.470022+0000", 2024-09-13T13:06:28.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.629e-06 2024-09-13T13:06:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.470045+0000", 2024-09-13T13:06:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2668999999999999e-05 2024-09-13T13:06:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.470055+0000", 2024-09-13T13:06:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0519999999999999e-05 2024-09-13T13:06:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.470062+0000", 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9129999999999997e-06 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.470176+0000", 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011334500000000001 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.470298+0000", 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121858 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:06:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.470442+0000", 2024-09-13T13:06:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000144695 2024-09-13T13:06:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.478744+0000", 2024-09-13T13:06:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0083012469999999994 2024-09-13T13:06:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.478770+0000", 2024-09-13T13:06:28.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6237999999999998e-05 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.557974+0000", 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.079204286999999998 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.558134+0000", 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015960200000000001 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.558182+0000", 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7865000000000003e-05 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:119485 7.2 7:4df3bbb4:::benchmark_data_smithi052_71865_object119484:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:50.479288+0000", 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 157.543340809, 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.082232352999999994, 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.075 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 119485 2024-09-13T13:06:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479288+0000", 2024-09-13T13:06:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479288+0000", 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479290+0000", 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9460000000000001e-06 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479316+0000", 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6163e-05 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479328+0000", 2024-09-13T13:06:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1336e-05 2024-09-13T13:06:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479335+0000", 2024-09-13T13:06:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4900000000000003e-06 2024-09-13T13:06:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479553+0000", 2024-09-13T13:06:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021835800000000001 2024-09-13T13:06:28.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479675+0000", 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121833 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479821+0000", 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014618300000000001 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.558224+0000", 2024-09-13T13:06:28.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.078402594000000006 2024-09-13T13:06:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.558260+0000", 2024-09-13T13:06:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5880000000000002e-05 2024-09-13T13:06:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.561437+0000", 2024-09-13T13:06:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003176651 2024-09-13T13:06:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.561483+0000", 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6171000000000002e-05 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.561521+0000", 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7747999999999998e-05 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:119486 7.2 7:48779d3e:::benchmark_data_smithi052_71865_object119485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:28.081 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:50.479349+0000", 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 157.54327971999999, 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.087574149000000004, 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 119486 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479349+0000", 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479349+0000", 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479350+0000", 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8100000000000001e-07 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479378+0000", 2024-09-13T13:06:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7487999999999998e-05 2024-09-13T13:06:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479385+0000", 2024-09-13T13:06:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4580000000000004e-06 2024-09-13T13:06:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.479391+0000", 2024-09-13T13:06:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1489999999999999e-06 2024-09-13T13:06:28.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.552508+0000", 2024-09-13T13:06:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.073116223999999994 2024-09-13T13:06:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.552646+0000", 2024-09-13T13:06:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000138701 2024-09-13T13:06:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:06:28.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.552798+0000", 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000151557 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.558297+0000", 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0054993719999999998 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.566670+0000", 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0083725150000000005 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.566705+0000", 2024-09-13T13:06:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5119e-05 2024-09-13T13:06:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.566758+0000", 2024-09-13T13:06:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3208000000000003e-05 2024-09-13T13:06:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.566923+0000", 2024-09-13T13:06:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000165377 2024-09-13T13:06:28.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151585 7.c 7:348440bb:::benchmark_data_smithi052_71865_object30484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.174012+0000", 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 118.848617117, 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.092236627000000002, 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.088 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151585 2024-09-13T13:06:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174012+0000", 2024-09-13T13:06:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174012+0000", 2024-09-13T13:06:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174014+0000", 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8929999999999999e-06 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174020+0000", 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8479999999999997e-06 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174031+0000", 2024-09-13T13:06:28.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1034e-05 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174038+0000", 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7260000000000002e-06 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.183763+0000", 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0097243079999999992 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.183904+0000", 2024-09-13T13:06:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000141323 2024-09-13T13:06:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:06:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.184053+0000", 2024-09-13T13:06:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000148722 2024-09-13T13:06:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.186978+0000", 2024-09-13T13:06:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029257300000000001 2024-09-13T13:06:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261696+0000", 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.074717792000000005 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261773+0000", 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7096999999999997e-05 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261885+0000", 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000111301 2024-09-13T13:06:28.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.266248+0000", 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.004363853 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151595 7.2 7:4dd8dfb3:::benchmark_data_smithi052_71865_object30494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.187208+0000", 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 118.835420895, 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.078734839000000001, 2024-09-13T13:06:28.094 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151595 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187208+0000", 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187208+0000", 2024-09-13T13:06:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187210+0000", 2024-09-13T13:06:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6700000000000001e-06 2024-09-13T13:06:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187215+0000", 2024-09-13T13:06:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5999999999999997e-06 2024-09-13T13:06:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187226+0000", 2024-09-13T13:06:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0694e-05 2024-09-13T13:06:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187234+0000", 2024-09-13T13:06:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5070000000000003e-06 2024-09-13T13:06:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187287+0000", 2024-09-13T13:06:28.098 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:28.099 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:27 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1502396931' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:06:28.099 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:27 smithi167 ceph-mon[33178]: from='client.35183 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:06:28.099 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:27 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1418534629' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3304000000000003e-05 2024-09-13T13:06:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.258594+0000", 2024-09-13T13:06:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.071306749000000003 2024-09-13T13:06:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:06:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.258730+0000", 2024-09-13T13:06:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000136484 2024-09-13T13:06:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.264401+0000", 2024-09-13T13:06:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0056712780000000001 2024-09-13T13:06:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.265816+0000", 2024-09-13T13:06:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0014145679999999999 2024-09-13T13:06:28.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.265828+0000", 2024-09-13T13:06:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2437e-05 2024-09-13T13:06:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.265886+0000", 2024-09-13T13:06:28.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7552000000000001e-05 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.265943+0000", 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6996000000000001e-05 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "MOSDPGPushReply(7.c 130/119 [PushReplyOp(7:3590660f:::benchmark_data_smithi052_71865_object104033:head)])", 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.188032+0000", 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 118.83459662200001, 2024-09-13T13:06:28.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.079117181999999994, 2024-09-13T13:06:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:06:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.188032+0000", 2024-09-13T13:06:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.188032+0000", 2024-09-13T13:06:28.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.188034+0000", 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.311e-06 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.188039+0000", 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5060000000000003e-06 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.258478+0000", 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.070438476 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.258497+0000", 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8961e-05 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.266295+0000", 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0077982590000000001 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.267150+0000", 2024-09-13T13:06:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00085466899999999996 2024-09-13T13:06:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:155447 7.3 7:ce0aa4a6:::benchmark_data_smithi052_71865_object34346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T13:06:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:34.093943+0000", 2024-09-13T13:06:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 113.92868604100001, 2024-09-13T13:06:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.085192592999999997, 2024-09-13T13:06:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.107 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 155447 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.093943+0000", 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.093943+0000", 2024-09-13T13:06:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.093945+0000", 2024-09-13T13:06:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9080000000000002e-06 2024-09-13T13:06:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.093949+0000", 2024-09-13T13:06:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5009999999999998e-06 2024-09-13T13:06:28.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.093958+0000", 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8480000000000007e-06 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.093964+0000", 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8259999999999998e-06 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.096183+0000", 2024-09-13T13:06:28.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0022186620000000002 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.096299+0000", 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011603399999999999 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.096416+0000", 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011737 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.108356+0000", 2024-09-13T13:06:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.01193987 2024-09-13T13:06:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.108399+0000", 2024-09-13T13:06:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3118000000000003e-05 2024-09-13T13:06:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.179048+0000", 2024-09-13T13:06:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.070649317000000003 2024-09-13T13:06:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.179093+0000", 2024-09-13T13:06:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4644999999999997e-05 2024-09-13T13:06:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.179136+0000", 2024-09-13T13:06:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2493999999999997e-05 2024-09-13T13:06:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:155453 7.3 7:cca9ded4:::benchmark_data_smithi052_71865_object34352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:34.099424+0000", 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 113.923204707, 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.084085952000000005, 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 155453 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.099424+0000", 2024-09-13T13:06:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.099424+0000", 2024-09-13T13:06:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.099427+0000", 2024-09-13T13:06:28.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3039999999999999e-06 2024-09-13T13:06:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.099432+0000", 2024-09-13T13:06:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7749999999999998e-06 2024-09-13T13:06:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.099443+0000", 2024-09-13T13:06:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1100999999999999e-05 2024-09-13T13:06:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.099451+0000", 2024-09-13T13:06:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6639999999999998e-06 2024-09-13T13:06:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.101259+0000", 2024-09-13T13:06:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018080780000000001 2024-09-13T13:06:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.101363+0000", 2024-09-13T13:06:28.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103935 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.106060+0000", 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0046968929999999997 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.179142+0000", 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.073082313999999995 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.179246+0000", 2024-09-13T13:06:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103309 2024-09-13T13:06:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.183416+0000", 2024-09-13T13:06:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041705739999999998 2024-09-13T13:06:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.183467+0000", 2024-09-13T13:06:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0720000000000002e-05 2024-09-13T13:06:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.183510+0000", 2024-09-13T13:06:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3285000000000003e-05 2024-09-13T13:06:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:155456 7.3 7:c741256a:::benchmark_data_smithi052_71865_object34355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T13:06:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:34.102804+0000", 2024-09-13T13:06:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 113.919825174, 2024-09-13T13:06:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.080821859999999995, 2024-09-13T13:06:28.120 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 155456 2024-09-13T13:06:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.102804+0000", 2024-09-13T13:06:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.102804+0000", 2024-09-13T13:06:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.102806+0000", 2024-09-13T13:06:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9589999999999998e-06 2024-09-13T13:06:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.102811+0000", 2024-09-13T13:06:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9860000000000002e-06 2024-09-13T13:06:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.102820+0000", 2024-09-13T13:06:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2650000000000002e-06 2024-09-13T13:06:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.102827+0000", 2024-09-13T13:06:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6649999999999997e-06 2024-09-13T13:06:28.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.108560+0000", 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057334559999999996 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.108672+0000", 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000111564 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.108783+0000", 2024-09-13T13:06:28.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011092299999999999 2024-09-13T13:06:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.179147+0000", 2024-09-13T13:06:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.070364495999999999 2024-09-13T13:06:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.179635+0000", 2024-09-13T13:06:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00048794799999999998 2024-09-13T13:06:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.183551+0000", 2024-09-13T13:06:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039159329999999999 2024-09-13T13:06:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.183593+0000", 2024-09-13T13:06:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2267e-05 2024-09-13T13:06:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.183626+0000", 2024-09-13T13:06:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2397999999999997e-05 2024-09-13T13:06:28.126 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:155463 7.3 7:ca8be61d:::benchmark_data_smithi052_71865_object34362:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:34.108882+0000", 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 113.913747274, 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.081058903000000002, 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 155463 2024-09-13T13:06:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.108882+0000", 2024-09-13T13:06:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.108882+0000", 2024-09-13T13:06:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.108883+0000", 2024-09-13T13:06:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3300000000000001e-07 2024-09-13T13:06:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.108886+0000", 2024-09-13T13:06:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.303e-06 2024-09-13T13:06:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.108892+0000", 2024-09-13T13:06:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8669999999999999e-06 2024-09-13T13:06:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.108897+0000", 2024-09-13T13:06:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4999999999999999e-06 2024-09-13T13:06:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.112450+0000", 2024-09-13T13:06:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0035528589999999998 2024-09-13T13:06:28.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.176035+0000", 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.063584858999999994 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.176159+0000", 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000124237 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.179154+0000", 2024-09-13T13:06:28.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029943349999999999 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.189724+0000", 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010570632999999999 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.189801+0000", 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7214999999999998e-05 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.189893+0000", 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1285000000000004e-05 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.189941+0000", 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7976999999999998e-05 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:155466 7.c 7:3cfef96a:::benchmark_data_smithi052_71865_object34365:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:34.111666+0000", 2024-09-13T13:06:28.133 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 113.910962571, 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.082416115999999998, 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 155466 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.111666+0000", 2024-09-13T13:06:28.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.111666+0000", 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.111668+0000", 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.66e-06 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.111674+0000", 2024-09-13T13:06:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5940000000000001e-06 2024-09-13T13:06:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.111683+0000", 2024-09-13T13:06:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6490000000000003e-06 2024-09-13T13:06:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.111691+0000", 2024-09-13T13:06:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4150000000000002e-06 2024-09-13T13:06:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.112291+0000", 2024-09-13T13:06:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00060024500000000001 2024-09-13T13:06:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.112417+0000", 2024-09-13T13:06:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012642 2024-09-13T13:06:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:06:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.177576+0000", 2024-09-13T13:06:28.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.065158154999999995 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.193362+0000", 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015786128 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.193405+0000", 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3027e-05 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.193418+0000", 2024-09-13T13:06:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2926e-05 2024-09-13T13:06:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.193464+0000", 2024-09-13T13:06:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6186000000000001e-05 2024-09-13T13:06:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.194083+0000", 2024-09-13T13:06:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00061871099999999998 2024-09-13T13:06:28.139 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:155467 7.c 7:3405f8a6:::benchmark_data_smithi052_71865_object34366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T13:06:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:34.111703+0000", 2024-09-13T13:06:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 113.910925869, 2024-09-13T13:06:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.082863275, 2024-09-13T13:06:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.140 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 155467 2024-09-13T13:06:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.111703+0000", 2024-09-13T13:06:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.111703+0000", 2024-09-13T13:06:28.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.111704+0000", 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3499999999999996e-07 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.111707+0000", 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.79e-06 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.111712+0000", 2024-09-13T13:06:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6409999999999999e-06 2024-09-13T13:06:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.111717+0000", 2024-09-13T13:06:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0440000000000003e-06 2024-09-13T13:06:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.180104+0000", 2024-09-13T13:06:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.068386527000000003 2024-09-13T13:06:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.180228+0000", 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012457000000000001 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.180604+0000", 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00037613499999999999 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.194091+0000", 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.013486178999999999 2024-09-13T13:06:28.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.194136+0000", 2024-09-13T13:06:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4898999999999998e-05 2024-09-13T13:06:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.194177+0000", 2024-09-13T13:06:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1014999999999999e-05 2024-09-13T13:06:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.194224+0000", 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7605999999999998e-05 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:34.194566+0000", 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00034223400000000002 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:165346 7.c 7:3d621664:::benchmark_data_smithi052_71865_object44245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:45.907948+0000", 2024-09-13T13:06:28.146 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 102.11468079700001, 2024-09-13T13:06:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.080342968000000001, 2024-09-13T13:06:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 165346 2024-09-13T13:06:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.907948+0000", 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.907948+0000", 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.907949+0000", 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.09e-07 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.907953+0000", 2024-09-13T13:06:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0029999999999996e-06 2024-09-13T13:06:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.907962+0000", 2024-09-13T13:06:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4999999999999999e-06 2024-09-13T13:06:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.907967+0000", 2024-09-13T13:06:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9379999999999997e-06 2024-09-13T13:06:28.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.966305+0000", 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.058338053000000001 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.966415+0000", 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109205 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.966529+0000", 2024-09-13T13:06:28.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114592 2024-09-13T13:06:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.988048+0000", 2024-09-13T13:06:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.021519002999999998 2024-09-13T13:06:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.988061+0000", 2024-09-13T13:06:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3131e-05 2024-09-13T13:06:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.988109+0000", 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8059999999999997e-05 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.988157+0000", 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7831000000000003e-05 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:45.988291+0000", 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000133843 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.152 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:166869 7.2 7:45b2e8d1:::benchmark_data_smithi052_71865_object45768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.609888+0000", 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 100.41274091, 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.082322062000000001, 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:28.153 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 166869 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.609888+0000", 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.609888+0000", 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:06:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.609889+0000", 2024-09-13T13:06:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.327e-06 2024-09-13T13:06:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:06:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.609894+0000", 2024-09-13T13:06:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5469999999999998e-06 2024-09-13T13:06:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:06:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.609904+0000", 2024-09-13T13:06:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7720000000000006e-06 2024-09-13T13:06:28.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:06:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.609910+0000", 2024-09-13T13:06:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5719999999999999e-06 2024-09-13T13:06:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:06:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.644146+0000", 2024-09-13T13:06:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.034235285999999997 2024-09-13T13:06:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:06:28.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.644275+0000", 2024-09-13T13:06:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012990399999999999 2024-09-13T13:06:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:06:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.644384+0000", 2024-09-13T13:06:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108742 2024-09-13T13:06:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:06:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.647256+0000", 2024-09-13T13:06:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002872206 2024-09-13T13:06:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.650285+0000", 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003028555 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.692120+0000", 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.041834635000000002 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.692171+0000", 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1375000000000002e-05 2024-09-13T13:06:28.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:06:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:06:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:06:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.692210+0000", 2024-09-13T13:06:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9140999999999997e-05 2024-09-13T13:06:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:06:28.159 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:06:28.159 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:06:28.160 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:27 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1502396931' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:06:28.160 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:27 smithi052 ceph-mon[36682]: from='client.35183 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:06:28.160 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:27 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1418534629' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:28.160 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:27 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1502396931' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:06:28.160 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:27 smithi052 ceph-mon[29471]: from='client.35183 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:06:28.160 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:27 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1418534629' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:28.161 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:06:28.183 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:28.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:28.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:28.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:28.310 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:28.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:28.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:28.412 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:28.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:28.441 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:28.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:28.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:28.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:28.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:28.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:28.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:28.695 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:28.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:28.797 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:28.823 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:28.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:28.925 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:28.948 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:28.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:28.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:29.050 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:29.073 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:29.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:29.175 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:29.199 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:29.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:29.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:28 smithi052 ceph-mon[36682]: pgmap v251: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:06:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:28 smithi052 ceph-mon[36682]: from='client.35183 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:06:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:28 smithi052 ceph-mon[36682]: osdmap e173: 8 total, 8 up, 8 in 2024-09-13T13:06:29.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:28 smithi052 ceph-mon[29471]: pgmap v251: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:06:29.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:28 smithi052 ceph-mon[29471]: from='client.35183 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:06:29.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:28 smithi052 ceph-mon[29471]: osdmap e173: 8 total, 8 up, 8 in 2024-09-13T13:06:29.301 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:29.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:28 smithi167 ceph-mon[33178]: pgmap v251: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:06:29.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:28 smithi167 ceph-mon[33178]: from='client.35183 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:06:29.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:28 smithi167 ceph-mon[33178]: osdmap e173: 8 total, 8 up, 8 in 2024-09-13T13:06:29.327 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:29.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:29.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:29.454 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:29.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:29.556 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:29.581 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:29.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:29.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:29.608 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:06:29.683 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:29.710 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:29.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:29.811 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:29.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:29.837 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:29.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:29.880 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:06:29.938 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:29.961 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:29.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:30.001 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:29 smithi167 ceph-mon[33178]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:06:30.062 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:30.084 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:30.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:30.186 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:30.207 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:30.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:30.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:30.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:29 smithi052 ceph-mon[36682]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:06:30.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:29 smithi052 ceph-mon[29471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:06:30.309 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:30.333 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:30.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:30.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:30.435 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:30.459 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:30.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:30.469 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:06:30.469 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:06:30.469 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-09-13T13:06:30.469 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:06:30.561 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:30.584 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:30.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:30.686 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:30.712 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:30.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:30.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:30.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:30.836 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:30.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:30.881 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T13:06:30.938 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:30.962 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:30.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:30.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:30 smithi052 ceph-mon[36682]: pgmap v253: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T13:06:30.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:30 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1335067433' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:06:30.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:30 smithi052 ceph-mon[29471]: pgmap v253: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T13:06:30.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:30 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1335067433' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:06:31.063 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:31.086 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:31.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:31.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:31.142 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:30 smithi167 ceph-mon[33178]: pgmap v253: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T13:06:31.142 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:30 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1335067433' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:06:31.143 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T13:06:31.188 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:31.213 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:31.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:31.314 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:31.339 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:31.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:31.407 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:06:31.441 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:31.465 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:31.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:31.566 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:31.592 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:31.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:31.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:31.704 DEBUG:teuthology.orchestra.run.smithi052:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-13T13:06:31.706 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T13:06:31.706 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T13:06:31.715 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:31.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:31.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:31.759 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:hints = 1 2024-09-13T13:06:31.759 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T13:06:31.759 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Object prefix: benchmark_data_smithi052_101302 2024-09-13T13:06:31.759 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:06:31.759 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 0 0 0 0 0 0 - 0 2024-09-13T13:06:31.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:31.850 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:31.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:31.859 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:31.898 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T13:06:31.952 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:31.977 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:06:31.977 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:06:31.977 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:06:31.977 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:06:31.977 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:06:31.984 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:31.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:31.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:32.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:31 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1335067433' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:06:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:31 smithi167 ceph-mon[33178]: osdmap e174: 8 total, 8 up, 8 in 2024-09-13T13:06:32.086 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:32.142 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:32.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:32.150 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:31 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1335067433' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:06:32.150 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:31 smithi052 ceph-mon[36682]: osdmap e174: 8 total, 8 up, 8 in 2024-09-13T13:06:32.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:31 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1335067433' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:06:32.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:31 smithi052 ceph-mon[29471]: osdmap e174: 8 total, 8 up, 8 in 2024-09-13T13:06:32.244 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:32.277 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:32.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:32.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:32.379 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:32.423 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:32.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:32.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:32.555 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:32.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:32.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:32.567 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:32.657 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:32.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:32.702 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:32.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:32.733 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T13:06:32.759 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 1 16 3070 3054 190.864 190.875 0.0036226 0.00521155 2024-09-13T13:06:32.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:32.842 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:32.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:32.944 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:32.984 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:32.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:32.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:32 smithi052 ceph-mon[29471]: pgmap v255: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-09-13T13:06:32.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:06:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:32 smithi052 ceph-mon[36682]: pgmap v255: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-09-13T13:06:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:06:33.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:32 smithi167 ceph-mon[33178]: pgmap v255: 148 pgs: 148 active+clean; 583 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-09-13T13:06:33.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:06:33.087 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:33.111 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:06:33.112 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:06:33.125 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:33.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:33.227 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:33.276 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:33.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:33.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:33.378 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:33.410 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:33.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:33.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:33.512 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:33.567 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:33.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:33.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:33.585 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:33.670 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:33.705 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:06:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:06:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:06:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:06:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:22467 7.1 7:83ca2fe7:::benchmark_data_smithi052_71865_object22466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-13T13:06:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:31.281822+0000", 2024-09-13T13:06:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 242.420045821, 2024-09-13T13:06:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059630491000000001, 2024-09-13T13:06:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22467 2024-09-13T13:06:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281822+0000", 2024-09-13T13:06:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281822+0000", 2024-09-13T13:06:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281824+0000", 2024-09-13T13:06:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-06 2024-09-13T13:06:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281853+0000", 2024-09-13T13:06:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8864999999999999e-05 2024-09-13T13:06:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281882+0000", 2024-09-13T13:06:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9249000000000001e-05 2024-09-13T13:06:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281890+0000", 2024-09-13T13:06:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4320000000000002e-06 2024-09-13T13:06:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.281939+0000", 2024-09-13T13:06:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9039000000000003e-05 2024-09-13T13:06:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.282052+0000", 2024-09-13T13:06:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113272 2024-09-13T13:06:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:06:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.282179+0000", 2024-09-13T13:06:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012745500000000001 2024-09-13T13:06:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.283337+0000", 2024-09-13T13:06:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011573530000000001 2024-09-13T13:06:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.285171+0000", 2024-09-13T13:06:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001834727 2024-09-13T13:06:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.341389+0000", 2024-09-13T13:06:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056217894999999997 2024-09-13T13:06:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.341424+0000", 2024-09-13T13:06:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5172999999999997e-05 2024-09-13T13:06:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:31.341453+0000", 2024-09-13T13:06:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8441000000000001e-05 2024-09-13T13:06:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:28936 7.1 7:8cc6c498:::benchmark_data_smithi052_71865_object28935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-13T13:06:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:34.434379+0000", 2024-09-13T13:06:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 239.26748919100001, 2024-09-13T13:06:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061581934999999997, 2024-09-13T13:06:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 28936 2024-09-13T13:06:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434379+0000", 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434379+0000", 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434381+0000", 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.933e-06 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434412+0000", 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0580999999999997e-05 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434427+0000", 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5919000000000001e-05 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434435+0000", 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5599999999999996e-06 2024-09-13T13:06:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434513+0000", 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7610999999999999e-05 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434638+0000", 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000125683 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.434781+0000", 2024-09-13T13:06:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014221000000000001 2024-09-13T13:06:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.437744+0000", 2024-09-13T13:06:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029634409999999998 2024-09-13T13:06:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.437765+0000", 2024-09-13T13:06:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0554999999999999e-05 2024-09-13T13:06:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.495871+0000", 2024-09-13T13:06:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058106564999999999 2024-09-13T13:06:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.495912+0000", 2024-09-13T13:06:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1106999999999998e-05 2024-09-13T13:06:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.495961+0000", 2024-09-13T13:06:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8770000000000002e-05 2024-09-13T13:06:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:28940 7.1 7:8444ec31:::benchmark_data_smithi052_71865_object28939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-13T13:06:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:34.436157+0000", 2024-09-13T13:06:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 239.26571099899999, 2024-09-13T13:06:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062002834, 2024-09-13T13:06:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 28940 2024-09-13T13:06:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436157+0000", 2024-09-13T13:06:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436157+0000", 2024-09-13T13:06:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436159+0000", 2024-09-13T13:06:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7880000000000001e-06 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436240+0000", 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0631999999999996e-05 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436250+0000", 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0227000000000001e-05 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436257+0000", 2024-09-13T13:06:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.17e-06 2024-09-13T13:06:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436743+0000", 2024-09-13T13:06:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00048641599999999999 2024-09-13T13:06:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.436875+0000", 2024-09-13T13:06:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013162800000000001 2024-09-13T13:06:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:06:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.437006+0000", 2024-09-13T13:06:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013089500000000001 2024-09-13T13:06:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.437829+0000", 2024-09-13T13:06:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00082255999999999996 2024-09-13T13:06:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.438239+0000", 2024-09-13T13:06:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000410331 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.498083+0000", 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059844616000000003 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.498118+0000", 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4981000000000003e-05 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:34.498160+0000", 2024-09-13T13:06:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.159e-05 2024-09-13T13:06:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:83969 7.1 7:819e28b6:::benchmark_data_smithi052_71865_object83968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:06:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:14.547136+0000", 2024-09-13T13:06:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 199.154732069, 2024-09-13T13:06:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0019013699999999, 2024-09-13T13:06:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 83969 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547136+0000", 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547136+0000", 2024-09-13T13:06:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547138+0000", 2024-09-13T13:06:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2859999999999998e-06 2024-09-13T13:06:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547171+0000", 2024-09-13T13:06:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2586000000000003e-05 2024-09-13T13:06:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547182+0000", 2024-09-13T13:06:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1336e-05 2024-09-13T13:06:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547191+0000", 2024-09-13T13:06:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.67e-06 2024-09-13T13:06:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547351+0000", 2024-09-13T13:06:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000160032 2024-09-13T13:06:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:06:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:14.547364+0000", 2024-09-13T13:06:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3118e-05 2024-09-13T13:06:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.545177+0000", 2024-09-13T13:06:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99781304699999995 2024-09-13T13:06:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.545295+0000", 2024-09-13T13:06:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118134 2024-09-13T13:06:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:06:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.545394+0000", 2024-09-13T13:06:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8840999999999998e-05 2024-09-13T13:06:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.547566+0000", 2024-09-13T13:06:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021715749999999998 2024-09-13T13:06:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.548948+0000", 2024-09-13T13:06:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013822719999999999 2024-09-13T13:06:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.548984+0000", 2024-09-13T13:06:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5725000000000001e-05 2024-09-13T13:06:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.549038+0000", 2024-09-13T13:06:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3748000000000001e-05 2024-09-13T13:06:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14493.0:1218 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94064716857216 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:06:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:15.061508+0000", 2024-09-13T13:06:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 198.640360055, 2024-09-13T13:06:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.48102383999999998, 2024-09-13T13:06:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:06:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14493", 2024-09-13T13:06:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:06:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1218 2024-09-13T13:06:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061508+0000", 2024-09-13T13:06:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061508+0000", 2024-09-13T13:06:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061511+0000", 2024-09-13T13:06:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3750000000000001e-06 2024-09-13T13:06:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061517+0000", 2024-09-13T13:06:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1999999999999999e-06 2024-09-13T13:06:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061528+0000", 2024-09-13T13:06:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1642e-05 2024-09-13T13:06:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061537+0000", 2024-09-13T13:06:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7110000000000001e-06 2024-09-13T13:06:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061610+0000", 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3126000000000004e-05 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061621+0000", 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1112e-05 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.537588+0000", 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.47596680699999999 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.537736+0000", 2024-09-13T13:06:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147887 2024-09-13T13:06:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.542532+0000", 2024-09-13T13:06:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00479598 2024-09-13T13:06:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14493.0:1219 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94064721783040 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:15.061557+0000", 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 198.64031071700001, 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.481215114, 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14493", 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1219 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061557+0000", 2024-09-13T13:06:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061557+0000", 2024-09-13T13:06:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061558+0000", 2024-09-13T13:06:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3499999999999996e-07 2024-09-13T13:06:33.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061561+0000", 2024-09-13T13:06:33.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7240000000000001e-06 2024-09-13T13:06:33.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061567+0000", 2024-09-13T13:06:33.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4459999999999998e-06 2024-09-13T13:06:33.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061572+0000", 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7029999999999997e-06 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061623+0000", 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1078999999999997e-05 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061629+0000", 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4600000000000002e-06 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.539806+0000", 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.47817701200000001 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.539938+0000", 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000132757 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.542773+0000", 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028342979999999999 2024-09-13T13:06:33.746 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.747 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.747 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.747 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:95510 7.1 7:8bdd1589:::benchmark_data_smithi052_71865_object95509:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:06:33.747 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:26.176570+0000", 2024-09-13T13:06:33.747 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.52529865899999, 2024-09-13T13:06:33.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061999859999999997, 2024-09-13T13:06:33.747 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.747 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.747 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.747 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.747 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.748 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.748 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95510 2024-09-13T13:06:33.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.748 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176570+0000", 2024-09-13T13:06:33.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176570+0000", 2024-09-13T13:06:33.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176574+0000", 2024-09-13T13:06:33.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3680000000000004e-06 2024-09-13T13:06:33.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176643+0000", 2024-09-13T13:06:33.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8658999999999994e-05 2024-09-13T13:06:33.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176657+0000", 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4099e-05 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176664+0000", 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6720000000000004e-06 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176716+0000", 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1749999999999997e-05 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.176857+0000", 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000141001 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.177015+0000", 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015754399999999999 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.179003+0000", 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019881690000000001 2024-09-13T13:06:33.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.238473+0000", 2024-09-13T13:06:33.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059470058999999999 2024-09-13T13:06:33.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.238511+0000", 2024-09-13T13:06:33.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8179999999999997e-05 2024-09-13T13:06:33.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.238569+0000", 2024-09-13T13:06:33.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8359000000000001e-05 2024-09-13T13:06:33.753 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.753 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.753 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.753 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:95515 7.1 7:8f0a2bc8:::benchmark_data_smithi052_71865_object95514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:06:33.753 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:26.180498+0000", 2024-09-13T13:06:33.753 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.52136995199999, 2024-09-13T13:06:33.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060428145000000003, 2024-09-13T13:06:33.753 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.753 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.754 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.754 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.754 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.754 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.754 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95515 2024-09-13T13:06:33.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.754 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180498+0000", 2024-09-13T13:06:33.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180498+0000", 2024-09-13T13:06:33.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180500+0000", 2024-09-13T13:06:33.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0679999999999999e-06 2024-09-13T13:06:33.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180554+0000", 2024-09-13T13:06:33.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3749000000000003e-05 2024-09-13T13:06:33.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180565+0000", 2024-09-13T13:06:33.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0614e-05 2024-09-13T13:06:33.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180571+0000", 2024-09-13T13:06:33.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9490000000000001e-06 2024-09-13T13:06:33.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180618+0000", 2024-09-13T13:06:33.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7533e-05 2024-09-13T13:06:33.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180744+0000", 2024-09-13T13:06:33.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012591499999999999 2024-09-13T13:06:33.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:06:33.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.180906+0000", 2024-09-13T13:06:33.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000162292 2024-09-13T13:06:33.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.182700+0000", 2024-09-13T13:06:33.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001793843 2024-09-13T13:06:33.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.240842+0000", 2024-09-13T13:06:33.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058141512999999999 2024-09-13T13:06:33.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.240877+0000", 2024-09-13T13:06:33.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5500000000000002e-05 2024-09-13T13:06:33.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:26.240926+0000", 2024-09-13T13:06:33.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9169000000000003e-05 2024-09-13T13:06:33.763 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.763 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.763 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.763 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:96745 7.1 7:8b8e8d5f:::benchmark_data_smithi052_71865_object96744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:06:33.763 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:27.480362+0000", 2024-09-13T13:06:33.763 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 186.22150610700001, 2024-09-13T13:06:33.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.068948425999999993, 2024-09-13T13:06:33.763 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.764 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.764 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.764 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.764 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.764 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.765 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 96745 2024-09-13T13:06:33.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.765 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480362+0000", 2024-09-13T13:06:33.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480362+0000", 2024-09-13T13:06:33.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480364+0000", 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1500000000000002e-06 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480419+0000", 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4797999999999999e-05 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480433+0000", 2024-09-13T13:06:33.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4304999999999999e-05 2024-09-13T13:06:33.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480441+0000", 2024-09-13T13:06:33.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1799999999999999e-06 2024-09-13T13:06:33.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480495+0000", 2024-09-13T13:06:33.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4583000000000002e-05 2024-09-13T13:06:33.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480636+0000", 2024-09-13T13:06:33.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014137899999999999 2024-09-13T13:06:33.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:06:33.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.480799+0000", 2024-09-13T13:06:33.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016208999999999999 2024-09-13T13:06:33.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.483165+0000", 2024-09-13T13:06:33.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023665209999999999 2024-09-13T13:06:33.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.549136+0000", 2024-09-13T13:06:33.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065970735000000003 2024-09-13T13:06:33.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.549219+0000", 2024-09-13T13:06:33.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2917000000000006e-05 2024-09-13T13:06:33.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.549311+0000", 2024-09-13T13:06:33.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1768000000000006e-05 2024-09-13T13:06:33.770 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.770 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.770 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.770 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:105215 7.1 7:8cc92998:::benchmark_data_smithi052_71865_object105214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-13T13:06:33.770 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:36.018426+0000", 2024-09-13T13:06:33.770 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 177.683442598, 2024-09-13T13:06:33.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069694252999999998, 2024-09-13T13:06:33.770 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.771 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.771 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:33.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:33.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 2 15 5849 5834 182.298 173.75 0.00862457 0.00547422 2024-09-13T13:06:33.773 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.773 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105215 2024-09-13T13:06:33.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.773 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018426+0000", 2024-09-13T13:06:33.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018426+0000", 2024-09-13T13:06:33.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018428+0000", 2024-09-13T13:06:33.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8119999999999999e-06 2024-09-13T13:06:33.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018479+0000", 2024-09-13T13:06:33.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0139999999999998e-05 2024-09-13T13:06:33.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018493+0000", 2024-09-13T13:06:33.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4552999999999999e-05 2024-09-13T13:06:33.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018500+0000", 2024-09-13T13:06:33.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3520000000000001e-06 2024-09-13T13:06:33.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018559+0000", 2024-09-13T13:06:33.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8491e-05 2024-09-13T13:06:33.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018707+0000", 2024-09-13T13:06:33.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147646 2024-09-13T13:06:33.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:06:33.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018875+0000", 2024-09-13T13:06:33.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168195 2024-09-13T13:06:33.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.021096+0000", 2024-09-13T13:06:33.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002221425 2024-09-13T13:06:33.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.087977+0000", 2024-09-13T13:06:33.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066880870999999995 2024-09-13T13:06:33.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.088030+0000", 2024-09-13T13:06:33.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.287e-05 2024-09-13T13:06:33.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.088120+0000", 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9897999999999999e-05 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:105222 7.1 7:8c89bd7c:::benchmark_data_smithi052_71865_object105221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:36.023584+0000", 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 177.67828388699999, 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064762324999999996, 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.779 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.780 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105222 2024-09-13T13:06:33.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.780 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023584+0000", 2024-09-13T13:06:33.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023584+0000", 2024-09-13T13:06:33.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023586+0000", 2024-09-13T13:06:33.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.159e-06 2024-09-13T13:06:33.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023649+0000", 2024-09-13T13:06:33.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2304000000000005e-05 2024-09-13T13:06:33.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023662+0000", 2024-09-13T13:06:33.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3407000000000001e-05 2024-09-13T13:06:33.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023670+0000", 2024-09-13T13:06:33.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:06:33.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023737+0000", 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6997999999999998e-05 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023891+0000", 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015383399999999999 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.024046+0000", 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015552099999999999 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.026536+0000", 2024-09-13T13:06:33.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024894330000000001 2024-09-13T13:06:33.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.088176+0000", 2024-09-13T13:06:33.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061639988 2024-09-13T13:06:33.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.088237+0000", 2024-09-13T13:06:33.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1500000000000004e-05 2024-09-13T13:06:33.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.088347+0000", 2024-09-13T13:06:33.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109422 2024-09-13T13:06:33.785 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.785 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.785 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.785 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:118706 7.1 7:8cfc8949:::benchmark_data_smithi052_71865_object118705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:33.785 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:49.554250+0000", 2024-09-13T13:06:33.785 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 164.14761799499999, 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067355788999999999, 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 118706 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554250+0000", 2024-09-13T13:06:33.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554250+0000", 2024-09-13T13:06:33.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554252+0000", 2024-09-13T13:06:33.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4300000000000001e-06 2024-09-13T13:06:33.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554309+0000", 2024-09-13T13:06:33.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6857999999999997e-05 2024-09-13T13:06:33.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554318+0000", 2024-09-13T13:06:33.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9380000000000004e-06 2024-09-13T13:06:33.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554323+0000", 2024-09-13T13:06:33.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:06:33.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554369+0000", 2024-09-13T13:06:33.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5250999999999998e-05 2024-09-13T13:06:33.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554503+0000", 2024-09-13T13:06:33.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134096 2024-09-13T13:06:33.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:06:33.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.554662+0000", 2024-09-13T13:06:33.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000159674 2024-09-13T13:06:33.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.556745+0000", 2024-09-13T13:06:33.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020823880000000001 2024-09-13T13:06:33.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.621459+0000", 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064714516999999999 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.621509+0000", 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9579000000000001e-05 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.621606+0000", 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7139999999999995e-05 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.791 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.792 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:118712 7.1 7:89036983:::benchmark_data_smithi052_71865_object118711:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:33.792 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:49.557026+0000", 2024-09-13T13:06:33.792 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 164.14484177099999, 2024-09-13T13:06:33.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064706738, 2024-09-13T13:06:33.792 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.792 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.792 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.792 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.792 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.792 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 118712 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557026+0000", 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557026+0000", 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557034+0000", 2024-09-13T13:06:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4270000000000002e-06 2024-09-13T13:06:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557081+0000", 2024-09-13T13:06:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7058000000000002e-05 2024-09-13T13:06:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557090+0000", 2024-09-13T13:06:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.585e-06 2024-09-13T13:06:33.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557095+0000", 2024-09-13T13:06:33.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4870000000000002e-06 2024-09-13T13:06:33.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557130+0000", 2024-09-13T13:06:33.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5389999999999998e-05 2024-09-13T13:06:33.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557220+0000", 2024-09-13T13:06:33.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.975e-05 2024-09-13T13:06:33.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:06:33.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.557310+0000", 2024-09-13T13:06:33.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0183e-05 2024-09-13T13:06:33.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.559010+0000", 2024-09-13T13:06:33.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016999669999999999 2024-09-13T13:06:33.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.621649+0000", 2024-09-13T13:06:33.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062639009999999995 2024-09-13T13:06:33.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.621684+0000", 2024-09-13T13:06:33.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4981999999999998e-05 2024-09-13T13:06:33.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:49.621733+0000", 2024-09-13T13:06:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8899000000000001e-05 2024-09-13T13:06:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:119492 7.1 7:88823659:::benchmark_data_smithi052_71865_object119491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:50.481839+0000", 2024-09-13T13:06:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 163.220029643, 2024-09-13T13:06:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077838544999999995, 2024-09-13T13:06:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 119492 2024-09-13T13:06:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.481839+0000", 2024-09-13T13:06:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.481839+0000", 2024-09-13T13:06:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.481841+0000", 2024-09-13T13:06:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2929999999999999e-06 2024-09-13T13:06:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.481936+0000", 2024-09-13T13:06:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5172000000000003e-05 2024-09-13T13:06:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.481947+0000", 2024-09-13T13:06:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0739e-05 2024-09-13T13:06:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.481953+0000", 2024-09-13T13:06:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5289999999999997e-06 2024-09-13T13:06:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.482005+0000", 2024-09-13T13:06:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1928000000000002e-05 2024-09-13T13:06:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.482123+0000", 2024-09-13T13:06:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011812899999999999 2024-09-13T13:06:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:06:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.482263+0000", 2024-09-13T13:06:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013927899999999999 2024-09-13T13:06:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.484328+0000", 2024-09-13T13:06:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002065356 2024-09-13T13:06:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.559557+0000", 2024-09-13T13:06:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075229375000000001 2024-09-13T13:06:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.559616+0000", 2024-09-13T13:06:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8764000000000002e-05 2024-09-13T13:06:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.559677+0000", 2024-09-13T13:06:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0980999999999998e-05 2024-09-13T13:06:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.806 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:120921 7.1 7:8dbade5d:::benchmark_data_smithi052_71865_object120920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:33.806 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:51.922426+0000", 2024-09-13T13:06:33.806 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 161.77944250499999, 2024-09-13T13:06:33.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077277414000000003, 2024-09-13T13:06:33.806 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.806 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.806 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.806 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.806 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.806 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.807 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 120921 2024-09-13T13:06:33.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.807 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922426+0000", 2024-09-13T13:06:33.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922426+0000", 2024-09-13T13:06:33.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922428+0000", 2024-09-13T13:06:33.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4159999999999998e-06 2024-09-13T13:06:33.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922470+0000", 2024-09-13T13:06:33.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2299000000000003e-05 2024-09-13T13:06:33.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922487+0000", 2024-09-13T13:06:33.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6701000000000002e-05 2024-09-13T13:06:33.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922495+0000", 2024-09-13T13:06:33.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9659999999999996e-06 2024-09-13T13:06:33.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922552+0000", 2024-09-13T13:06:33.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6685999999999999e-05 2024-09-13T13:06:33.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922693+0000", 2024-09-13T13:06:33.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014088800000000001 2024-09-13T13:06:33.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-09-13T13:06:33.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.922871+0000", 2024-09-13T13:06:33.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017871099999999999 2024-09-13T13:06:33.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.924891+0000", 2024-09-13T13:06:33.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020191860000000001 2024-09-13T13:06:33.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.999585+0000", 2024-09-13T13:06:33.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.074694889 2024-09-13T13:06:33.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.999645+0000", 2024-09-13T13:06:33.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9982999999999998e-05 2024-09-13T13:06:33.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:51.999703+0000", 2024-09-13T13:06:33.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7689000000000003e-05 2024-09-13T13:06:33.813 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.813 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.813 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.813 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:136555 7.1 7:8fedf1e6:::benchmark_data_smithi052_71865_object15454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:06:33.813 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:10.092295+0000", 2024-09-13T13:06:33.813 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 143.60957314199999, 2024-09-13T13:06:33.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060400885000000001, 2024-09-13T13:06:33.814 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.814 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.814 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.814 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.814 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.814 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.814 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 136555 2024-09-13T13:06:33.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.814 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092295+0000", 2024-09-13T13:06:33.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092295+0000", 2024-09-13T13:06:33.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092298+0000", 2024-09-13T13:06:33.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6180000000000002e-06 2024-09-13T13:06:33.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092304+0000", 2024-09-13T13:06:33.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1290000000000001e-06 2024-09-13T13:06:33.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092319+0000", 2024-09-13T13:06:33.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4902999999999999e-05 2024-09-13T13:06:33.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092327+0000", 2024-09-13T13:06:33.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9049999999999997e-06 2024-09-13T13:06:33.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.096194+0000", 2024-09-13T13:06:33.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038671959999999998 2024-09-13T13:06:33.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.096319+0000", 2024-09-13T13:06:33.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124831 2024-09-13T13:06:33.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:06:33.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.096437+0000", 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011856100000000001 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.100488+0000", 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040505029999999996 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.105447+0000", 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049591130000000002 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.151544+0000", 2024-09-13T13:06:33.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046097325000000001 2024-09-13T13:06:33.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.151607+0000", 2024-09-13T13:06:33.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2539999999999994e-05 2024-09-13T13:06:33.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.152696+0000", 2024-09-13T13:06:33.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001089261 2024-09-13T13:06:33.820 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.820 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.820 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.820 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:136559 7.1 7:87a02730:::benchmark_data_smithi052_71865_object15458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:06:33.820 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:10.092736+0000", 2024-09-13T13:06:33.820 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 143.609131784, 2024-09-13T13:06:33.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061695030999999997, 2024-09-13T13:06:33.820 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.820 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.820 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 136559 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092736+0000", 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092736+0000", 2024-09-13T13:06:33.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092738+0000", 2024-09-13T13:06:33.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.195e-06 2024-09-13T13:06:33.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092745+0000", 2024-09-13T13:06:33.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.712e-06 2024-09-13T13:06:33.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092755+0000", 2024-09-13T13:06:33.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4360000000000008e-06 2024-09-13T13:06:33.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.092762+0000", 2024-09-13T13:06:33.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7479999999999996e-06 2024-09-13T13:06:33.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.098351+0000", 2024-09-13T13:06:33.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055893660000000001 2024-09-13T13:06:33.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.098435+0000", 2024-09-13T13:06:33.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3751000000000005e-05 2024-09-13T13:06:33.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:06:33.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.098516+0000", 2024-09-13T13:06:33.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1769999999999998e-05 2024-09-13T13:06:33.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.101682+0000", 2024-09-13T13:06:33.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031659079999999998 2024-09-13T13:06:33.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.105502+0000", 2024-09-13T13:06:33.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038194679999999999 2024-09-13T13:06:33.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.153497+0000", 2024-09-13T13:06:33.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047995011999999997 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.153552+0000", 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5538999999999998e-05 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.154431+0000", 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00087912600000000004 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.827 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:136928 7.1 7:86ee2aaa:::benchmark_data_smithi052_71865_object15827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:06:33.827 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:10.611023+0000", 2024-09-13T13:06:33.827 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 143.090845022, 2024-09-13T13:06:33.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059963874, 2024-09-13T13:06:33.827 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.827 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.827 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.827 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.827 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.827 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.827 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 136928 2024-09-13T13:06:33.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.828 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.611023+0000", 2024-09-13T13:06:33.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.611023+0000", 2024-09-13T13:06:33.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.611026+0000", 2024-09-13T13:06:33.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.418e-06 2024-09-13T13:06:33.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.611032+0000", 2024-09-13T13:06:33.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8839999999999997e-06 2024-09-13T13:06:33.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.611048+0000", 2024-09-13T13:06:33.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5136999999999999e-05 2024-09-13T13:06:33.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.611057+0000", 2024-09-13T13:06:33.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2289999999999992e-06 2024-09-13T13:06:33.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.613408+0000", 2024-09-13T13:06:33.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023514320000000001 2024-09-13T13:06:33.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.613505+0000", 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6464000000000003e-05 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.613594+0000", 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9518000000000006e-05 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.621119+0000", 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075246599999999999 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.626948+0000", 2024-09-13T13:06:33.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058287349999999998 2024-09-13T13:06:33.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.670506+0000", 2024-09-13T13:06:33.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043558056999999997 2024-09-13T13:06:33.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.670547+0000", 2024-09-13T13:06:33.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1267999999999999e-05 2024-09-13T13:06:33.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.670987+0000", 2024-09-13T13:06:33.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00044007200000000001 2024-09-13T13:06:33.833 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.833 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.833 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.833 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:136930 7.1 7:89d55d1f:::benchmark_data_smithi052_71865_object15829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:06:33.833 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:10.612698+0000", 2024-09-13T13:06:33.833 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 143.08917046600001, 2024-09-13T13:06:33.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062650329000000005, 2024-09-13T13:06:33.834 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.834 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.834 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.834 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.834 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.834 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.834 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 136930 2024-09-13T13:06:33.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.834 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.612698+0000", 2024-09-13T13:06:33.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.612698+0000", 2024-09-13T13:06:33.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.612700+0000", 2024-09-13T13:06:33.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0509999999999999e-06 2024-09-13T13:06:33.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.612707+0000", 2024-09-13T13:06:33.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5170000000000002e-06 2024-09-13T13:06:33.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.612722+0000", 2024-09-13T13:06:33.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4250000000000001e-05 2024-09-13T13:06:33.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.612730+0000", 2024-09-13T13:06:33.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2179999999999999e-06 2024-09-13T13:06:33.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.616173+0000", 2024-09-13T13:06:33.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034428250000000001 2024-09-13T13:06:33.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.616309+0000", 2024-09-13T13:06:33.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000136227 2024-09-13T13:06:33.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:06:33.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.616446+0000", 2024-09-13T13:06:33.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013688400000000001 2024-09-13T13:06:33.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.626874+0000", 2024-09-13T13:06:33.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010428203 2024-09-13T13:06:33.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.626994+0000", 2024-09-13T13:06:33.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120423 2024-09-13T13:06:33.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.674597+0000", 2024-09-13T13:06:33.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047603148999999997 2024-09-13T13:06:33.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.674657+0000", 2024-09-13T13:06:33.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9169000000000002e-05 2024-09-13T13:06:33.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:10.675348+0000", 2024-09-13T13:06:33.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00069141299999999999 2024-09-13T13:06:33.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.840 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.840 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:155460 7.1 7:83ed900b:::benchmark_data_smithi052_71865_object34359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T13:06:33.840 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:34.105592+0000", 2024-09-13T13:06:33.840 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 119.59627587200001, 2024-09-13T13:06:33.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075149564000000002, 2024-09-13T13:06:33.840 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:33.840 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:33.840 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:33.841 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:33.841 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:33.841 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:33.841 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 155460 2024-09-13T13:06:33.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.841 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:33.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:33.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105592+0000", 2024-09-13T13:06:33.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:33.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105592+0000", 2024-09-13T13:06:33.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:33.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:33.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105595+0000", 2024-09-13T13:06:33.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0759999999999999e-06 2024-09-13T13:06:33.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:33.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105608+0000", 2024-09-13T13:06:33.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2848e-05 2024-09-13T13:06:33.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105630+0000", 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2146000000000001e-05 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105640+0000", 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6860000000000001e-06 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.106344+0000", 2024-09-13T13:06:33.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070417500000000003 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.106500+0000", 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000155468 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.106631+0000", 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131137 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.109754+0000", 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031233480000000002 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.110324+0000", 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000569843 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.180592+0000", 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070267674000000002 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:33.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.180647+0000", 2024-09-13T13:06:33.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5067e-05 2024-09-13T13:06:33.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:33.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:33.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:33.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.180742+0000", 2024-09-13T13:06:33.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5095999999999999e-05 2024-09-13T13:06:33.846 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.846 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.846 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.846 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:33.846 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:33.846 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:06:33.847 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:06:33 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:06:33.872 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:33.912 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:06:33.916 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T13:06:33.916 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:33.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:33.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:33.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:33 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/468069550' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:06:33.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:33 smithi052 ceph-mon[29471]: from='client.25228 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:06:33.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:33 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3911081973' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:06:33.994 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:33 smithi052 ceph-mon[29471]: from='client.25231 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:06:33.994 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:33 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/468069550' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:06:33.994 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:33 smithi052 ceph-mon[36682]: from='client.25228 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:06:33.994 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:33 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3911081973' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:06:33.994 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:33 smithi052 ceph-mon[36682]: from='client.25231 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:06:34.019 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:34.061 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:34.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:34.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:34.163 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:34.205 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:34.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:34.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:34.233 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:06:34.234 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:06:34.272 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:06:34.273 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:06:34.305 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:33 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/468069550' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:06:34.305 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:33 smithi167 ceph-mon[33178]: from='client.25228 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:06:34.305 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:33 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3911081973' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:06:34.305 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:33 smithi167 ceph-mon[33178]: from='client.25231 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:06:34.308 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:34.311 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:06:34.312 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:06:34.342 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:34.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:34.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:34.349 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:06:34.349 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:06:34.382 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:06:34.382 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:06:34.414 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:06:34.414 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:06:34.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:34.450 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:06:34.450 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:06:34.487 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:06:34.487 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:06:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:34.488 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:34.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:34.590 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:34.627 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:34.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:34.729 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:34.759 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 3 16 8717 8701 181.251 179.188 0.00217621 0.00550656 2024-09-13T13:06:34.767 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:34.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:34.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:34.821 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T13:06:34.870 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:34.903 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:34.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:34.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[36682]: pgmap v256: 148 pgs: 148 active+clean; 236 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 471 op/s 2024-09-13T13:06:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[36682]: from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:06:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[36682]: from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:06:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[36682]: osdmap e175: 8 total, 8 up, 8 in 2024-09-13T13:06:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2012167627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:34.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[29471]: pgmap v256: 148 pgs: 148 active+clean; 236 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 471 op/s 2024-09-13T13:06:34.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:34.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:34.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[29471]: from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:06:34.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[29471]: from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:06:34.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[29471]: osdmap e175: 8 total, 8 up, 8 in 2024-09-13T13:06:34.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:34 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2012167627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:35.005 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:35.044 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:35.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:35.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:34 smithi167 ceph-mon[33178]: pgmap v256: 148 pgs: 148 active+clean; 236 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 471 op/s 2024-09-13T13:06:35.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:34 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:34 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:34 smithi167 ceph-mon[33178]: from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:06:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:34 smithi167 ceph-mon[33178]: from='client.25231 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:06:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:34 smithi167 ceph-mon[33178]: osdmap e175: 8 total, 8 up, 8 in 2024-09-13T13:06:35.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:34 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2012167627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:35.145 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:35.184 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:35.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:35.231 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:06:35.284 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T13:06:35.286 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:35.324 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:35.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:35.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:35.426 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:35.455 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:35.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:35.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:35.556 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:35.595 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:35.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:35.697 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:35.741 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:35.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:35.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:35.760 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 4 16 11675 11659 182.153 184.875 0.00707817 0.00548244 2024-09-13T13:06:35.842 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:35.875 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:06:35.876 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:06:35.881 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:35.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:35.983 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:36.014 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:36.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:36.116 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:36.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:36.147 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:36.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:36.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:36.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:36.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:36.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:36.397 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:36.434 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:36.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:36.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:06:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:06:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:06:36.537 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:36.569 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:36.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:36.659 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T13:06:36.671 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:36.710 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:36.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:36.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:36.760 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 5 16 14607 14591 182.369 183.25 0.00981929 0.00547496 2024-09-13T13:06:36.761 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:36.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:36.849 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:36.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:36.951 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:36.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:36 smithi052 ceph-mon[36682]: pgmap v258: 148 pgs: 148 active+clean; 236 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 30 MiB/s wr, 478 op/s 2024-09-13T13:06:36.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:36 smithi052 ceph-mon[29471]: pgmap v258: 148 pgs: 148 active+clean; 236 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 30 MiB/s wr, 478 op/s 2024-09-13T13:06:36.991 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:36.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:37.051 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:06:37.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:37.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:37.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:37.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:37.182 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:36 smithi167 ceph-mon[33178]: pgmap v258: 148 pgs: 148 active+clean; 236 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 30 MiB/s wr, 478 op/s 2024-09-13T13:06:37.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:37.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:37.271 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:37.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:37.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:37.404 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:37.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:37.506 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:37.538 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:37.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:37.640 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:37.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:37.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:37.760 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 6 16 17106 17090 178.003 156.188 0.00321976 0.00561104 2024-09-13T13:06:37.771 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:37.810 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:37.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:37.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:37.912 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:37.943 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:37.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:37.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:38.045 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:38.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:38.087 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:38.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:38.126 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:06:38.126 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:06:38.147 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:06:38.188 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:38.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[36682]: 3.5 scrub starts 2024-09-13T13:06:38.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[36682]: 3.5 scrub ok 2024-09-13T13:06:38.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[36682]: 4.2 scrub starts 2024-09-13T13:06:38.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[36682]: 4.2 scrub ok 2024-09-13T13:06:38.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[36682]: 4.3 scrub starts 2024-09-13T13:06:38.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[36682]: 4.3 scrub ok 2024-09-13T13:06:38.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3128211623' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:06:38.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[36682]: osdmap e176: 8 total, 8 up, 8 in 2024-09-13T13:06:38.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[36682]: from='client.25243 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:06:38.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[29471]: 3.5 scrub starts 2024-09-13T13:06:38.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[29471]: 3.5 scrub ok 2024-09-13T13:06:38.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[29471]: 4.2 scrub starts 2024-09-13T13:06:38.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[29471]: 4.2 scrub ok 2024-09-13T13:06:38.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[29471]: 4.3 scrub starts 2024-09-13T13:06:38.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[29471]: 4.3 scrub ok 2024-09-13T13:06:38.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3128211623' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:06:38.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[29471]: osdmap e176: 8 total, 8 up, 8 in 2024-09-13T13:06:38.191 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:37 smithi052 ceph-mon[29471]: from='client.25243 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:06:38.238 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:38.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:38.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:38.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:37 smithi167 ceph-mon[33178]: 3.5 scrub starts 2024-09-13T13:06:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:37 smithi167 ceph-mon[33178]: 3.5 scrub ok 2024-09-13T13:06:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:37 smithi167 ceph-mon[33178]: 4.2 scrub starts 2024-09-13T13:06:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:37 smithi167 ceph-mon[33178]: 4.2 scrub ok 2024-09-13T13:06:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:37 smithi167 ceph-mon[33178]: 4.3 scrub starts 2024-09-13T13:06:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:37 smithi167 ceph-mon[33178]: 4.3 scrub ok 2024-09-13T13:06:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:37 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3128211623' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:06:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:37 smithi167 ceph-mon[33178]: osdmap e176: 8 total, 8 up, 8 in 2024-09-13T13:06:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:37 smithi167 ceph-mon[33178]: from='client.25243 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:06:38.341 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:38.384 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:38.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:38.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:38.486 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:38.535 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:38.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:38.629 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:38.638 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:38.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:38.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:38.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:38.760 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 7 16 19618 19602 174.997 157 0.00341701 0.005709 2024-09-13T13:06:38.762 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:06:38.770 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:38.806 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T13:06:38.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:38.876 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:38.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:38.877 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T13:06:38.977 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:39.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:39.000 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:39.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:39.102 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:39.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:39.141 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:39.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:39.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[36682]: 6.1e scrub starts 2024-09-13T13:06:39.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[36682]: 6.1e scrub ok 2024-09-13T13:06:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[36682]: pgmap v259: 148 pgs: 148 active+clean; 329 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 657 op/s 2024-09-13T13:06:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[36682]: 5.c scrub starts 2024-09-13T13:06:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[36682]: 5.c scrub ok 2024-09-13T13:06:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:06:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:06:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[36682]: 3.1b scrub starts 2024-09-13T13:06:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[36682]: 3.1b scrub ok 2024-09-13T13:06:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[36682]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:06:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[36682]: osdmap e177: 8 total, 8 up, 8 in 2024-09-13T13:06:39.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[29471]: 6.1e scrub starts 2024-09-13T13:06:39.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[29471]: 6.1e scrub ok 2024-09-13T13:06:39.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[29471]: pgmap v259: 148 pgs: 148 active+clean; 329 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 657 op/s 2024-09-13T13:06:39.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[29471]: 5.c scrub starts 2024-09-13T13:06:39.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[29471]: 5.c scrub ok 2024-09-13T13:06:39.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:06:39.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:06:39.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[29471]: 3.1b scrub starts 2024-09-13T13:06:39.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[29471]: 3.1b scrub ok 2024-09-13T13:06:39.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[29471]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:06:39.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:38 smithi052 ceph-mon[29471]: osdmap e177: 8 total, 8 up, 8 in 2024-09-13T13:06:39.242 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:39.248 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:06:39.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:39.286 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:39.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:38 smithi167 ceph-mon[33178]: 6.1e scrub starts 2024-09-13T13:06:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:38 smithi167 ceph-mon[33178]: 6.1e scrub ok 2024-09-13T13:06:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:38 smithi167 ceph-mon[33178]: pgmap v259: 148 pgs: 148 active+clean; 329 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 657 op/s 2024-09-13T13:06:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:38 smithi167 ceph-mon[33178]: 5.c scrub starts 2024-09-13T13:06:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:38 smithi167 ceph-mon[33178]: 5.c scrub ok 2024-09-13T13:06:39.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:38 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:06:39.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:38 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:06:39.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:38 smithi167 ceph-mon[33178]: 3.1b scrub starts 2024-09-13T13:06:39.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:38 smithi167 ceph-mon[33178]: 3.1b scrub ok 2024-09-13T13:06:39.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:38 smithi167 ceph-mon[33178]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:06:39.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:38 smithi167 ceph-mon[33178]: osdmap e177: 8 total, 8 up, 8 in 2024-09-13T13:06:39.387 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:39.426 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:39.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:39.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:39.511 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:39.529 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:39.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:39.580 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:39.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:39.681 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:39.711 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:39.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:39.760 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 8 16 21805 21789 170.207 136.688 0.00916759 0.00586852 2024-09-13T13:06:39.813 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:39.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:39.856 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:39.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:39.922 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:06:39.922 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:06:39.945 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:06:39.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:39.986 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:39.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:39.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:40.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:39 smithi167 ceph-mon[33178]: 6.1c scrub starts 2024-09-13T13:06:40.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:39 smithi167 ceph-mon[33178]: 6.1c scrub ok 2024-09-13T13:06:40.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:39 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4189795763' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:40.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:39 smithi167 ceph-mon[33178]: from='client.35231 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:40.088 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:40.117 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:40.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:40.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:40.219 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:39 smithi052 ceph-mon[36682]: 6.1c scrub starts 2024-09-13T13:06:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:39 smithi052 ceph-mon[36682]: 6.1c scrub ok 2024-09-13T13:06:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:39 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4189795763' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:39 smithi052 ceph-mon[36682]: from='client.35231 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:40.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:39 smithi052 ceph-mon[29471]: 6.1c scrub starts 2024-09-13T13:06:40.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:39 smithi052 ceph-mon[29471]: 6.1c scrub ok 2024-09-13T13:06:40.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:39 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4189795763' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:40.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:39 smithi052 ceph-mon[29471]: from='client.35231 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:40.250 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:40.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:40.352 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:40.392 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:40.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:40.494 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:40.524 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:40.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:40.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:40.626 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:40.657 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:40.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:40.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:40.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:40.761 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 9 16 24180 24164 167.786 148.438 0.00216236 0.00595326 2024-09-13T13:06:40.796 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:40.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:40.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:40.822 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T13:06:40.898 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:40.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:40.926 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:40.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:41.027 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:41.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:41.066 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:41.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:41.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:41.208 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:41.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:41.212 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:06:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[36682]: 1.0 scrub starts 2024-09-13T13:06:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[36682]: 1.0 scrub ok 2024-09-13T13:06:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[36682]: pgmap v262: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 142 MiB/s wr, 2.28k op/s 2024-09-13T13:06:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:06:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:06:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[36682]: from='client.35231 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:06:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[36682]: osdmap e178: 8 total, 8 up, 8 in 2024-09-13T13:06:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[36682]: 4.1e scrub starts 2024-09-13T13:06:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[36682]: 4.1e scrub ok 2024-09-13T13:06:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[29471]: 1.0 scrub starts 2024-09-13T13:06:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[29471]: 1.0 scrub ok 2024-09-13T13:06:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[29471]: pgmap v262: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 142 MiB/s wr, 2.28k op/s 2024-09-13T13:06:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:06:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:06:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[29471]: from='client.35231 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:06:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[29471]: osdmap e178: 8 total, 8 up, 8 in 2024-09-13T13:06:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[29471]: 4.1e scrub starts 2024-09-13T13:06:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:40 smithi052 ceph-mon[29471]: 4.1e scrub ok 2024-09-13T13:06:41.310 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:41.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:40 smithi167 ceph-mon[33178]: 1.0 scrub starts 2024-09-13T13:06:41.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:40 smithi167 ceph-mon[33178]: 1.0 scrub ok 2024-09-13T13:06:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:40 smithi167 ceph-mon[33178]: pgmap v262: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 142 MiB/s wr, 2.28k op/s 2024-09-13T13:06:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:40 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:06:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:40 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:06:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:40 smithi167 ceph-mon[33178]: from='client.35231 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:06:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:40 smithi167 ceph-mon[33178]: osdmap e178: 8 total, 8 up, 8 in 2024-09-13T13:06:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:40 smithi167 ceph-mon[33178]: 4.1e scrub starts 2024-09-13T13:06:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:40 smithi167 ceph-mon[33178]: 4.1e scrub ok 2024-09-13T13:06:41.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:41.352 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:41.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:41.453 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:41.494 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:41.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:41.596 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:41.617 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T13:06:41.642 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:41.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:41.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:41.744 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:41.760 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 10 15 26178 26163 163.501 124.938 0.00785263 0.00610983 2024-09-13T13:06:41.787 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:41.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:41.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:41.889 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:41.921 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:41.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:42.023 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:42.057 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:06:42.058 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:06:42.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:42.073 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:42.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:42.175 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:42.213 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:42.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:42.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:42.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:41 smithi052 ceph-mon[29471]: 4.c scrub starts 2024-09-13T13:06:42.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:41 smithi052 ceph-mon[29471]: 4.c scrub ok 2024-09-13T13:06:42.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:41 smithi052 ceph-mon[29471]: 3.12 scrub starts 2024-09-13T13:06:42.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:41 smithi052 ceph-mon[29471]: 3.12 scrub ok 2024-09-13T13:06:42.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:41 smithi052 ceph-mon[36682]: 4.c scrub starts 2024-09-13T13:06:42.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:41 smithi052 ceph-mon[36682]: 4.c scrub ok 2024-09-13T13:06:42.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:41 smithi052 ceph-mon[36682]: 3.12 scrub starts 2024-09-13T13:06:42.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:41 smithi052 ceph-mon[36682]: 3.12 scrub ok 2024-09-13T13:06:42.315 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:42.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:41 smithi167 ceph-mon[33178]: 4.c scrub starts 2024-09-13T13:06:42.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:41 smithi167 ceph-mon[33178]: 4.c scrub ok 2024-09-13T13:06:42.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:41 smithi167 ceph-mon[33178]: 3.12 scrub starts 2024-09-13T13:06:42.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:41 smithi167 ceph-mon[33178]: 3.12 scrub ok 2024-09-13T13:06:42.354 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:42.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:42.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:42.456 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:42.494 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:42.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:42.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:42.628 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:42.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:42.730 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:42.760 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:42.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:42.760 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 11 16 28238 28222 160.335 128.688 0.001874 0.00622802 2024-09-13T13:06:42.861 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:42.904 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:42.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:43.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:43.037 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:43.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:43.139 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:43.181 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:43.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:43.183 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:42 smithi052 ceph-mon[36682]: pgmap v264: 148 pgs: 148 active+clean; 1.2 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 159 MiB/s wr, 2.54k op/s 2024-09-13T13:06:43.184 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:42 smithi052 ceph-mon[36682]: 3.1 scrub starts 2024-09-13T13:06:43.184 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:42 smithi052 ceph-mon[36682]: 3.1 scrub ok 2024-09-13T13:06:43.184 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:42 smithi052 ceph-mon[29471]: pgmap v264: 148 pgs: 148 active+clean; 1.2 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 159 MiB/s wr, 2.54k op/s 2024-09-13T13:06:43.184 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:42 smithi052 ceph-mon[29471]: 3.1 scrub starts 2024-09-13T13:06:43.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:42 smithi052 ceph-mon[29471]: 3.1 scrub ok 2024-09-13T13:06:43.282 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:43.315 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:43.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:43.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:43.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:42 smithi167 ceph-mon[33178]: pgmap v264: 148 pgs: 148 active+clean; 1.2 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 159 MiB/s wr, 2.54k op/s 2024-09-13T13:06:43.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:42 smithi167 ceph-mon[33178]: 3.1 scrub starts 2024-09-13T13:06:43.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:42 smithi167 ceph-mon[33178]: 3.1 scrub ok 2024-09-13T13:06:43.343 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:06:43.343 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:06:43.382 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:06:43.383 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:06:43.415 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:06:43.416 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:06:43.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:43.450 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:06:43.450 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:06:43.471 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:43.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:43.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:43.478 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:43.489 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:06:43.489 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:06:43.564 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:06:43.564 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:06:43.573 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:43.598 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:06:43.598 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:06:43.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:43.616 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:43.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:43.633 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:06:43.633 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:06:43.717 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:43.748 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:43.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:43.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:43.761 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 12 16 29954 29938 155.901 107.25 0.00598202 0.00640699 2024-09-13T13:06:43.850 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:43.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:43.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:43.984 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:44.016 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:44.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:43 smithi167 ceph-mon[33178]: 5.8 scrub starts 2024-09-13T13:06:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:43 smithi167 ceph-mon[33178]: 5.8 scrub ok 2024-09-13T13:06:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2010270270' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:44.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:06:43 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:06:44.084 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:43 smithi052 ceph-mon[36682]: 5.8 scrub starts 2024-09-13T13:06:44.085 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:43 smithi052 ceph-mon[36682]: 5.8 scrub ok 2024-09-13T13:06:44.085 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2010270270' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:44.085 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:43 smithi052 ceph-mon[29471]: 5.8 scrub starts 2024-09-13T13:06:44.085 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:43 smithi052 ceph-mon[29471]: 5.8 scrub ok 2024-09-13T13:06:44.085 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2010270270' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:44.118 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:44.160 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:44.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:44.262 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:44.293 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:44.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:44.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:44.395 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:44.426 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:44.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:44.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:44.528 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:44.565 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:44.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:44.667 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:44.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:44.704 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:44.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:44.761 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 13 15 31991 31976 153.706 127.375 0.00872328 0.00649962 2024-09-13T13:06:44.806 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:44.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:44.849 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:44.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:44.936 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:06:44.950 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:45.030 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:45.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:45.103 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T13:06:45.131 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:45.181 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:45.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[29471]: 6.2 scrub starts 2024-09-13T13:06:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[29471]: 6.2 scrub ok 2024-09-13T13:06:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[29471]: pgmap v265: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 241 MiB/s wr, 3.85k op/s 2024-09-13T13:06:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[29471]: 6.1 scrub starts 2024-09-13T13:06:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[29471]: 6.1 scrub ok 2024-09-13T13:06:45.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[29471]: 3.1f scrub starts 2024-09-13T13:06:45.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[29471]: 3.1f scrub ok 2024-09-13T13:06:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[36682]: 6.2 scrub starts 2024-09-13T13:06:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[36682]: 6.2 scrub ok 2024-09-13T13:06:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[36682]: pgmap v265: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 241 MiB/s wr, 3.85k op/s 2024-09-13T13:06:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[36682]: 6.1 scrub starts 2024-09-13T13:06:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[36682]: 6.1 scrub ok 2024-09-13T13:06:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[36682]: 3.1f scrub starts 2024-09-13T13:06:45.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:44 smithi052 ceph-mon[36682]: 3.1f scrub ok 2024-09-13T13:06:45.282 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:45.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:44 smithi167 ceph-mon[33178]: 6.2 scrub starts 2024-09-13T13:06:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:44 smithi167 ceph-mon[33178]: 6.2 scrub ok 2024-09-13T13:06:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:44 smithi167 ceph-mon[33178]: pgmap v265: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 241 MiB/s wr, 3.85k op/s 2024-09-13T13:06:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:44 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:44 smithi167 ceph-mon[33178]: 6.1 scrub starts 2024-09-13T13:06:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:44 smithi167 ceph-mon[33178]: 6.1 scrub ok 2024-09-13T13:06:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:44 smithi167 ceph-mon[33178]: 3.1f scrub starts 2024-09-13T13:06:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:44 smithi167 ceph-mon[33178]: 3.1f scrub ok 2024-09-13T13:06:45.332 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:45.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:45.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:45.423 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:45.434 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:45.469 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:06:45.485 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:45.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:45.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:45.588 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:45.623 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:45.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:45.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:45.748 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:45.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:45.761 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 14 16 33857 33841 151.052 116.562 0.0031669 0.0065982 2024-09-13T13:06:45.851 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:45.892 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:45.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:45.966 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T13:06:45.994 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:46.037 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:46.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:46.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:46.139 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:46.175 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:46.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:46.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:46.238 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:06:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:06:46.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:45 smithi052 ceph-mon[36682]: 4.1c scrub starts 2024-09-13T13:06:46.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:45 smithi052 ceph-mon[36682]: 4.1c scrub ok 2024-09-13T13:06:46.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:45 smithi052 ceph-mon[36682]: 3.7 scrub starts 2024-09-13T13:06:46.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:45 smithi052 ceph-mon[36682]: 3.7 scrub ok 2024-09-13T13:06:46.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:45 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2689801348' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:06:46.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:45 smithi052 ceph-mon[29471]: 4.1c scrub starts 2024-09-13T13:06:46.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:45 smithi052 ceph-mon[29471]: 4.1c scrub ok 2024-09-13T13:06:46.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:45 smithi052 ceph-mon[29471]: 3.7 scrub starts 2024-09-13T13:06:46.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:45 smithi052 ceph-mon[29471]: 3.7 scrub ok 2024-09-13T13:06:46.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:45 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2689801348' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:06:46.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:46.308 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:46.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:46.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:46.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:45 smithi167 ceph-mon[33178]: 4.1c scrub starts 2024-09-13T13:06:46.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:45 smithi167 ceph-mon[33178]: 4.1c scrub ok 2024-09-13T13:06:46.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:45 smithi167 ceph-mon[33178]: 3.7 scrub starts 2024-09-13T13:06:46.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:45 smithi167 ceph-mon[33178]: 3.7 scrub ok 2024-09-13T13:06:46.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:45 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2689801348' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:06:46.410 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:46.439 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:46.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:46.529 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:06:46.540 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:46.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:46.615 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:46.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:46.716 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:46.753 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:46.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:46.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:46.761 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 15 16 35478 35462 147.736 101.312 0.0033059 0.00675537 2024-09-13T13:06:46.855 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:46.888 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:46.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:46.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:47.021 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:47.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:47.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:47.123 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:06:47.123 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:06:47.124 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:06:47.124 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:06:47.124 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:06:47.157 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:47.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:47.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[36682]: 5.1a scrub starts 2024-09-13T13:06:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[36682]: 5.1a scrub ok 2024-09-13T13:06:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[36682]: pgmap v266: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 183 MiB/s wr, 2.92k op/s 2024-09-13T13:06:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2689801348' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:06:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[36682]: osdmap e179: 8 total, 8 up, 8 in 2024-09-13T13:06:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[36682]: 6.0 scrub starts 2024-09-13T13:06:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[36682]: 6.0 scrub ok 2024-09-13T13:06:47.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[29471]: 5.1a scrub starts 2024-09-13T13:06:47.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[29471]: 5.1a scrub ok 2024-09-13T13:06:47.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[29471]: pgmap v266: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 183 MiB/s wr, 2.92k op/s 2024-09-13T13:06:47.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2689801348' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:06:47.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[29471]: osdmap e179: 8 total, 8 up, 8 in 2024-09-13T13:06:47.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[29471]: 6.0 scrub starts 2024-09-13T13:06:47.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:46 smithi052 ceph-mon[29471]: 6.0 scrub ok 2024-09-13T13:06:47.259 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:47.290 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:47.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:47.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:46 smithi167 ceph-mon[33178]: 5.1a scrub starts 2024-09-13T13:06:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:46 smithi167 ceph-mon[33178]: 5.1a scrub ok 2024-09-13T13:06:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:46 smithi167 ceph-mon[33178]: pgmap v266: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 183 MiB/s wr, 2.92k op/s 2024-09-13T13:06:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:46 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2689801348' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:06:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:46 smithi167 ceph-mon[33178]: osdmap e179: 8 total, 8 up, 8 in 2024-09-13T13:06:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:46 smithi167 ceph-mon[33178]: 6.0 scrub starts 2024-09-13T13:06:47.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:46 smithi167 ceph-mon[33178]: 6.0 scrub ok 2024-09-13T13:06:47.392 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:47.425 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:47.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:47.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:47.548 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:47.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:47.649 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:47.673 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:47.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:47.761 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 16 16 37053 37037 144.654 98.4375 0.00769417 0.0069075 2024-09-13T13:06:47.775 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:47.816 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:47.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:47.918 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:47.958 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:47.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:48.059 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:48.082 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:48.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:48.180 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:47 smithi167 ceph-mon[33178]: 6.7 scrub starts 2024-09-13T13:06:48.181 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:47 smithi167 ceph-mon[33178]: 6.7 scrub ok 2024-09-13T13:06:48.181 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:47 smithi167 ceph-mon[33178]: 5.11 scrub starts 2024-09-13T13:06:48.181 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:47 smithi167 ceph-mon[33178]: 5.11 scrub ok 2024-09-13T13:06:48.181 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:47 smithi167 ceph-mon[33178]: 5.1e scrub starts 2024-09-13T13:06:48.181 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:47 smithi167 ceph-mon[33178]: 5.1e scrub ok 2024-09-13T13:06:48.181 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:47 smithi167 ceph-mon[33178]: osdmap e180: 8 total, 8 up, 8 in 2024-09-13T13:06:48.181 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:47 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:48.181 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:47 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:06:48.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:48.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:48.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:48.238 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T13:06:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[36682]: 6.7 scrub starts 2024-09-13T13:06:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[36682]: 6.7 scrub ok 2024-09-13T13:06:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[36682]: 5.11 scrub starts 2024-09-13T13:06:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[36682]: 5.11 scrub ok 2024-09-13T13:06:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[36682]: 5.1e scrub starts 2024-09-13T13:06:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[36682]: 5.1e scrub ok 2024-09-13T13:06:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[36682]: osdmap e180: 8 total, 8 up, 8 in 2024-09-13T13:06:48.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:48.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:06:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[29471]: 6.7 scrub starts 2024-09-13T13:06:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[29471]: 6.7 scrub ok 2024-09-13T13:06:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[29471]: 5.11 scrub starts 2024-09-13T13:06:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[29471]: 5.11 scrub ok 2024-09-13T13:06:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[29471]: 5.1e scrub starts 2024-09-13T13:06:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[29471]: 5.1e scrub ok 2024-09-13T13:06:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[29471]: osdmap e180: 8 total, 8 up, 8 in 2024-09-13T13:06:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:48.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:47 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:06:48.324 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:48.346 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:48.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:48.448 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:48.485 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:48.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:48.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:48.587 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:48.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:48.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:48.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:48.645 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:06:48.745 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:48.762 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 17 15 38746 38731 142.365 105.875 0.00755289 0.00701885 2024-09-13T13:06:48.782 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:48.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:48.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:48.884 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:48.922 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:48.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:48.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:49.024 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:49.050 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:49.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:49.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:49 smithi167 ceph-mon[33178]: pgmap v268: 148 pgs: 148 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-13T13:06:49.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:49 smithi167 ceph-mon[33178]: 5.1d scrub starts 2024-09-13T13:06:49.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:49 smithi167 ceph-mon[33178]: 5.1d scrub ok 2024-09-13T13:06:49.152 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:49.186 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:49.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:49.270 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:06:49.271 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:06:49.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:06:49.271 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:06:49.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.271 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:64964 7.5 7:aa5f6c19:::benchmark_data_smithi052_71865_object64963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T13:06:49.271 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:58.135945+0000", 2024-09-13T13:06:49.271 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 231.1316382, 2024-09-13T13:06:49.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.085435993000000002, 2024-09-13T13:06:49.271 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.271 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.272 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.272 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.272 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.272 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.272 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 64964 2024-09-13T13:06:49.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.272 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.135945+0000", 2024-09-13T13:06:49.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.135945+0000", 2024-09-13T13:06:49.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.135955+0000", 2024-09-13T13:06:49.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9660000000000008e-06 2024-09-13T13:06:49.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.135993+0000", 2024-09-13T13:06:49.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7394000000000001e-05 2024-09-13T13:06:49.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.136005+0000", 2024-09-13T13:06:49.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2658e-05 2024-09-13T13:06:49.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.136013+0000", 2024-09-13T13:06:49.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9049999999999997e-06 2024-09-13T13:06:49.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.136070+0000", 2024-09-13T13:06:49.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7034000000000003e-05 2024-09-13T13:06:49.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.136188+0000", 2024-09-13T13:06:49.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117336 2024-09-13T13:06:49.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.136311+0000", 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012304800000000001 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.138541+0000", 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022307260000000001 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.142318+0000", 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037765210000000001 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.221322+0000", 2024-09-13T13:06:49.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079003844000000004 2024-09-13T13:06:49.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.221354+0000", 2024-09-13T13:06:49.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2539000000000001e-05 2024-09-13T13:06:49.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.221381+0000", 2024-09-13T13:06:49.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7022e-05 2024-09-13T13:06:49.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.278 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.278 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:64972 7.5 7:a118c925:::benchmark_data_smithi052_71865_object64971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T13:06:49.278 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:58.137950+0000", 2024-09-13T13:06:49.278 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 231.129633205, 2024-09-13T13:06:49.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084111670999999999, 2024-09-13T13:06:49.278 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.278 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.278 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.279 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 64972 2024-09-13T13:06:49.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.279 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.137950+0000", 2024-09-13T13:06:49.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.137950+0000", 2024-09-13T13:06:49.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.137952+0000", 2024-09-13T13:06:49.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8920000000000001e-06 2024-09-13T13:06:49.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.138005+0000", 2024-09-13T13:06:49.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2825000000000003e-05 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.138021+0000", 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6279000000000001e-05 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.138031+0000", 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0023e-05 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.138953+0000", 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092176300000000003 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.139097+0000", 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144108 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.139224+0000", 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012709099999999999 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.141497+0000", 2024-09-13T13:06:49.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022726080000000002 2024-09-13T13:06:49.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.142813+0000", 2024-09-13T13:06:49.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013160610000000001 2024-09-13T13:06:49.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.221982+0000", 2024-09-13T13:06:49.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079169541999999996 2024-09-13T13:06:49.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.222037+0000", 2024-09-13T13:06:49.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4175999999999998e-05 2024-09-13T13:06:49.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.222062+0000", 2024-09-13T13:06:49.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5303000000000002e-05 2024-09-13T13:06:49.284 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.284 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.284 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.285 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:64980 7.5 7:a951f5f5:::benchmark_data_smithi052_71865_object64979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-13T13:06:49.285 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:02:58.145381+0000", 2024-09-13T13:06:49.285 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 231.12220223599999, 2024-09-13T13:06:49.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080339815999999994, 2024-09-13T13:06:49.285 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.285 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.285 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.286 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 64980 2024-09-13T13:06:49.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.286 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.145381+0000", 2024-09-13T13:06:49.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.145381+0000", 2024-09-13T13:06:49.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.145384+0000", 2024-09-13T13:06:49.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6910000000000002e-06 2024-09-13T13:06:49.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.145438+0000", 2024-09-13T13:06:49.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3965999999999998e-05 2024-09-13T13:06:49.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.145449+0000", 2024-09-13T13:06:49.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0947e-05 2024-09-13T13:06:49.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.145456+0000", 2024-09-13T13:06:49.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2629999999999997e-06 2024-09-13T13:06:49.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.145510+0000", 2024-09-13T13:06:49.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3671999999999997e-05 2024-09-13T13:06:49.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.145624+0000", 2024-09-13T13:06:49.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114058 2024-09-13T13:06:49.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.145748+0000", 2024-09-13T13:06:49.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012428600000000001 2024-09-13T13:06:49.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.147690+0000", 2024-09-13T13:06:49.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001942254 2024-09-13T13:06:49.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.148945+0000", 2024-09-13T13:06:49.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001255115 2024-09-13T13:06:49.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.225629+0000", 2024-09-13T13:06:49.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076683234000000003 2024-09-13T13:06:49.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.225679+0000", 2024-09-13T13:06:49.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0661000000000002e-05 2024-09-13T13:06:49.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:02:58.225721+0000", 2024-09-13T13:06:49.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1668999999999997e-05 2024-09-13T13:06:49.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.292 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.292 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14493.0:1220 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94064716861824 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:06:49.292 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:15.061504+0000", 2024-09-13T13:06:49.292 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 214.206079636, 2024-09-13T13:06:49.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.48614386900000001, 2024-09-13T13:06:49.292 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.292 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.292 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14493", 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1220 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061504+0000", 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061504+0000", 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061507+0000", 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5599999999999998e-06 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061517+0000", 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3710000000000006e-06 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061530+0000", 2024-09-13T13:06:49.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2943e-05 2024-09-13T13:06:49.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061539+0000", 2024-09-13T13:06:49.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1300000000000007e-06 2024-09-13T13:06:49.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061612+0000", 2024-09-13T13:06:49.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3455999999999996e-05 2024-09-13T13:06:49.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061621+0000", 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9379999999999998e-06 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.540176+0000", 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.47855515700000001 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.540307+0000", 2024-09-13T13:06:49.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000130906 2024-09-13T13:06:49.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.547648+0000", 2024-09-13T13:06:49.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0073404079999999997 2024-09-13T13:06:49.297 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.297 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.297 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:96734 7.5 7:a651fbbc:::benchmark_data_smithi052_71865_object96733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:27.471667+0000", 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 201.79591669600001, 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079756648999999999, 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 96734 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.298 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.471667+0000", 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.471667+0000", 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.471668+0000", 2024-09-13T13:06:49.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6249999999999999e-06 2024-09-13T13:06:49.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.471717+0000", 2024-09-13T13:06:49.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.867e-05 2024-09-13T13:06:49.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.471729+0000", 2024-09-13T13:06:49.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1579e-05 2024-09-13T13:06:49.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.471734+0000", 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.471776+0000", 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1365000000000001e-05 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.471902+0000", 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012615099999999999 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.472067+0000", 2024-09-13T13:06:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016505400000000001 2024-09-13T13:06:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.477073+0000", 2024-09-13T13:06:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050059190000000002 2024-09-13T13:06:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.478931+0000", 2024-09-13T13:06:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001858118 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.551331+0000", 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072400431000000001 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.551392+0000", 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0287000000000001e-05 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:27.551423+0000", 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1814999999999999e-05 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.34781.0:109339 7.6 e113/84 7:6b730d06:::benchmark_data_smithi052_71865_object109338:head v 113'6840, mlcod=113'6837)", 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:40.392340+0000", 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 188.875243619, 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079595092000000006, 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.304 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:40.392340+0000", 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:40.392340+0000", 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:40.392341+0000", 2024-09-13T13:06:49.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0219999999999999e-06 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:40.392387+0000", 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5651e-05 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:40.460614+0000", 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.068227428000000007 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:40.460629+0000", 2024-09-13T13:06:49.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4938e-05 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:40.461255+0000", 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000626025 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:40.461271+0000", 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.628e-05 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:40.471893+0000", 2024-09-13T13:06:49.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01062163 2024-09-13T13:06:49.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:40.471935+0000", 2024-09-13T13:06:49.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2117999999999999e-05 2024-09-13T13:06:49.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.308 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.308 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:119269 7.5 7:ae2627b6:::benchmark_data_smithi052_71865_object119268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:49.308 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:50.224647+0000", 2024-09-13T13:06:49.309 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 179.04293611099999, 2024-09-13T13:06:49.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082874605000000004, 2024-09-13T13:06:49.309 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.309 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.309 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.309 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.309 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.309 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.309 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 119269 2024-09-13T13:06:49.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.309 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.224647+0000", 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.224647+0000", 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.224650+0000", 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.272e-06 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.224696+0000", 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6758999999999997e-05 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.224712+0000", 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5384e-05 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.224719+0000", 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0500000000000003e-06 2024-09-13T13:06:49.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.224770+0000", 2024-09-13T13:06:49.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1056e-05 2024-09-13T13:06:49.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.224900+0000", 2024-09-13T13:06:49.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013030100000000001 2024-09-13T13:06:49.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.225005+0000", 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010447500000000001 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.227170+0000", 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021651909999999999 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.229109+0000", 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019391930000000001 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.307455+0000", 2024-09-13T13:06:49.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.078345974999999998 2024-09-13T13:06:49.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.307488+0000", 2024-09-13T13:06:49.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2605999999999999e-05 2024-09-13T13:06:49.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.307522+0000", 2024-09-13T13:06:49.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4343e-05 2024-09-13T13:06:49.314 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.315 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.315 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.315 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:119278 7.5 7:aa0e54d1:::benchmark_data_smithi052_71865_object119277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:06:49.315 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:50.233668+0000", 2024-09-13T13:06:49.315 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 179.033915044, 2024-09-13T13:06:49.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.081334330999999996, 2024-09-13T13:06:49.315 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.315 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.315 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.315 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.316 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.316 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.316 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 119278 2024-09-13T13:06:49.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.316 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.233668+0000", 2024-09-13T13:06:49.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.233668+0000", 2024-09-13T13:06:49.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.233670+0000", 2024-09-13T13:06:49.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.75e-06 2024-09-13T13:06:49.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.233715+0000", 2024-09-13T13:06:49.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4966000000000003e-05 2024-09-13T13:06:49.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.233731+0000", 2024-09-13T13:06:49.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5432999999999999e-05 2024-09-13T13:06:49.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.233742+0000", 2024-09-13T13:06:49.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1219e-05 2024-09-13T13:06:49.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.236108+0000", 2024-09-13T13:06:49.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023657840000000001 2024-09-13T13:06:49.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.236230+0000", 2024-09-13T13:06:49.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122149 2024-09-13T13:06:49.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.236376+0000", 2024-09-13T13:06:49.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014583099999999999 2024-09-13T13:06:49.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.239039+0000", 2024-09-13T13:06:49.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026638080000000001 2024-09-13T13:06:49.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.240986+0000", 2024-09-13T13:06:49.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001946616 2024-09-13T13:06:49.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.314923+0000", 2024-09-13T13:06:49.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073937462999999995 2024-09-13T13:06:49.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.314963+0000", 2024-09-13T13:06:49.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0006000000000003e-05 2024-09-13T13:06:49.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.315003+0000", 2024-09-13T13:06:49.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9305999999999999e-05 2024-09-13T13:06:49.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:135702 7.5 7:a21ba469:::benchmark_data_smithi052_71865_object14601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:09.098886+0000", 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 160.16869745100001, 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079390235000000003, 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.323 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135702 2024-09-13T13:06:49.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.323 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.098886+0000", 2024-09-13T13:06:49.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.098886+0000", 2024-09-13T13:06:49.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.098888+0000", 2024-09-13T13:06:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1430000000000001e-06 2024-09-13T13:06:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.098895+0000", 2024-09-13T13:06:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9800000000000001e-06 2024-09-13T13:06:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.098910+0000", 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4667999999999999e-05 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.098917+0000", 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0090000000000001e-06 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.098974+0000", 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7294999999999999e-05 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.099111+0000", 2024-09-13T13:06:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000137253 2024-09-13T13:06:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.099246+0000", 2024-09-13T13:06:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013429800000000001 2024-09-13T13:06:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.101497+0000", 2024-09-13T13:06:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002251435 2024-09-13T13:06:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.106757+0000", 2024-09-13T13:06:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0052597319999999996 2024-09-13T13:06:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.178166+0000", 2024-09-13T13:06:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071408801999999993 2024-09-13T13:06:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.178223+0000", 2024-09-13T13:06:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7311000000000001e-05 2024-09-13T13:06:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.178276+0000", 2024-09-13T13:06:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3309000000000001e-05 2024-09-13T13:06:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:135706 7.5 7:a739ed18:::benchmark_data_smithi052_71865_object14605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:06:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:09.100755+0000", 2024-09-13T13:06:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 160.16682833499999, 2024-09-13T13:06:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082058698999999999, 2024-09-13T13:06:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.329 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135706 2024-09-13T13:06:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.100755+0000", 2024-09-13T13:06:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.100755+0000", 2024-09-13T13:06:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.100756+0000", 2024-09-13T13:06:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6799999999999999e-07 2024-09-13T13:06:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.100759+0000", 2024-09-13T13:06:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4759999999999998e-06 2024-09-13T13:06:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.100766+0000", 2024-09-13T13:06:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9240000000000001e-06 2024-09-13T13:06:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.100770+0000", 2024-09-13T13:06:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0659999999999997e-06 2024-09-13T13:06:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.101524+0000", 2024-09-13T13:06:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00075394100000000005 2024-09-13T13:06:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.101653+0000", 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012914400000000001 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.101820+0000", 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016612300000000001 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.104523+0000", 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027031400000000001 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.111271+0000", 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0067480800000000001 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.182707+0000", 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071436161999999997 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.182765+0000", 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8468000000000003e-05 2024-09-13T13:06:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:09.182814+0000", 2024-09-13T13:06:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8406999999999999e-05 2024-09-13T13:06:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:153866 7.5 7:a9811f35:::benchmark_data_smithi052_71865_object32765:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:06:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:32.271645+0000", 2024-09-13T13:06:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.99593859800001, 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.119643267, 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 153866 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.271645+0000", 2024-09-13T13:06:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.271645+0000", 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.271647+0000", 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1280000000000002e-06 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.271653+0000", 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0909999999999998e-06 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.271666+0000", 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2917e-05 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.271673+0000", 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3080000000000002e-06 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.271730+0000", 2024-09-13T13:06:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7170999999999999e-05 2024-09-13T13:06:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.271887+0000", 2024-09-13T13:06:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015653999999999999 2024-09-13T13:06:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.272005+0000", 2024-09-13T13:06:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118139 2024-09-13T13:06:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.274837+0000", 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002831718 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.280805+0000", 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059682829999999996 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.391226+0000", 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.110421175 2024-09-13T13:06:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.391261+0000", 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4258000000000003e-05 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:32.391288+0000", 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7538999999999998e-05 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:155449 7.5 7:ae7e3527:::benchmark_data_smithi052_71865_object34348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T13:06:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:34.094289+0000", 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 135.17329456600001, 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082411852999999993, 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 155449 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.094289+0000", 2024-09-13T13:06:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.094289+0000", 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.094291+0000", 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2120000000000002e-06 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.094298+0000", 2024-09-13T13:06:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7490000000000001e-06 2024-09-13T13:06:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.094312+0000", 2024-09-13T13:06:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4465e-05 2024-09-13T13:06:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.094321+0000", 2024-09-13T13:06:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8699999999999998e-06 2024-09-13T13:06:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.094383+0000", 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1785999999999994e-05 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.094513+0000", 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012987599999999999 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.094628+0000", 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011518099999999999 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.097262+0000", 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026341659999999999 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.101999+0000", 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047365819999999996 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.176618+0000", 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.074619017999999995 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.176663+0000", 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5179000000000002e-05 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.176701+0000", 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7768999999999997e-05 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:155818 7.5 7:a4927c1a:::benchmark_data_smithi052_71865_object34717:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T13:06:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:34.543273+0000", 2024-09-13T13:06:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 134.724310613, 2024-09-13T13:06:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080284409000000001, 2024-09-13T13:06:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 155818 2024-09-13T13:06:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.543273+0000", 2024-09-13T13:06:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.543273+0000", 2024-09-13T13:06:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.543275+0000", 2024-09-13T13:06:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.73e-06 2024-09-13T13:06:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.543280+0000", 2024-09-13T13:06:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8710000000000002e-06 2024-09-13T13:06:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.543292+0000", 2024-09-13T13:06:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1763000000000001e-05 2024-09-13T13:06:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.543299+0000", 2024-09-13T13:06:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9949999999999999e-06 2024-09-13T13:06:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.543338+0000", 2024-09-13T13:06:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8602000000000001e-05 2024-09-13T13:06:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.543417+0000", 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9345999999999996e-05 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.543494+0000", 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6410999999999997e-05 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.545395+0000", 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001901182 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.545901+0000", 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00050659000000000001 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.623457+0000", 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077555631999999999 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.623514+0000", 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6567999999999998e-05 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.623557+0000", 2024-09-13T13:06:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3718999999999999e-05 2024-09-13T13:06:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:199083 7.5 7:ae91a94d:::benchmark_data_smithi052_71865_object77982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T13:06:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:25.224574+0000", 2024-09-13T13:06:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.043009652999999, 2024-09-13T13:06:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11548370299999999, 2024-09-13T13:06:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 199083 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224574+0000", 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224574+0000", 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224576+0000", 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2290000000000002e-06 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224582+0000", 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6890000000000001e-06 2024-09-13T13:06:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224594+0000", 2024-09-13T13:06:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2371000000000001e-05 2024-09-13T13:06:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224616+0000", 2024-09-13T13:06:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1503000000000001e-05 2024-09-13T13:06:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224703+0000", 2024-09-13T13:06:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7394999999999999e-05 2024-09-13T13:06:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224819+0000", 2024-09-13T13:06:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116383 2024-09-13T13:06:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224940+0000", 2024-09-13T13:06:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120133 2024-09-13T13:06:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.231950+0000", 2024-09-13T13:06:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.007010138 2024-09-13T13:06:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.231985+0000", 2024-09-13T13:06:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5679000000000002e-05 2024-09-13T13:06:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.339953+0000", 2024-09-13T13:06:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107967735 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.340010+0000", 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6725000000000003e-05 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.340058+0000", 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7722999999999997e-05 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:199084 7.5 7:ad85f76d:::benchmark_data_smithi052_71865_object77983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T13:06:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:25.224898+0000", 2024-09-13T13:06:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.042685829999996, 2024-09-13T13:06:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.119879877, 2024-09-13T13:06:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 199084 2024-09-13T13:06:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224898+0000", 2024-09-13T13:06:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224898+0000", 2024-09-13T13:06:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224899+0000", 2024-09-13T13:06:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5999999999999999e-06 2024-09-13T13:06:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224905+0000", 2024-09-13T13:06:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.643e-06 2024-09-13T13:06:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224916+0000", 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.116e-05 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.224924+0000", 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9289999999999998e-06 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.227552+0000", 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026279480000000002 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.227659+0000", 2024-09-13T13:06:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010739 2024-09-13T13:06:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.227767+0000", 2024-09-13T13:06:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107822 2024-09-13T13:06:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.231956+0000", 2024-09-13T13:06:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041885430000000003 2024-09-13T13:06:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.232435+0000", 2024-09-13T13:06:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00047897499999999997 2024-09-13T13:06:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.344705+0000", 2024-09-13T13:06:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.112270809 2024-09-13T13:06:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.344737+0000", 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1418999999999999e-05 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.344778+0000", 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0639000000000002e-05 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:199086 7.5 7:abd25e8e:::benchmark_data_smithi052_71865_object77985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T13:06:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:25.227426+0000", 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.040157218999994, 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.117513937, 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 199086 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.227426+0000", 2024-09-13T13:06:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.227426+0000", 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.227428+0000", 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6619999999999999e-06 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.227433+0000", 2024-09-13T13:06:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8810000000000004e-06 2024-09-13T13:06:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.227443+0000", 2024-09-13T13:06:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0023e-05 2024-09-13T13:06:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.227449+0000", 2024-09-13T13:06:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3439999999999997e-06 2024-09-13T13:06:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.229871+0000", 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024216960000000001 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.229938+0000", 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7175999999999995e-05 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.230001+0000", 2024-09-13T13:06:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3436999999999996e-05 2024-09-13T13:06:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.232075+0000", 2024-09-13T13:06:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002073222 2024-09-13T13:06:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.235250+0000", 2024-09-13T13:06:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031754219999999998 2024-09-13T13:06:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.344850+0000", 2024-09-13T13:06:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10959938499999999 2024-09-13T13:06:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.344902+0000", 2024-09-13T13:06:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2358000000000003e-05 2024-09-13T13:06:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.344940+0000", 2024-09-13T13:06:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8331000000000003e-05 2024-09-13T13:06:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:199091 7.5 7:a5581aad:::benchmark_data_smithi052_71865_object77990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T13:06:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:25.232766+0000", 2024-09-13T13:06:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.034817017999998, 2024-09-13T13:06:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.112264724, 2024-09-13T13:06:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 199091 2024-09-13T13:06:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.232766+0000", 2024-09-13T13:06:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.232766+0000", 2024-09-13T13:06:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.232768+0000", 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5889999999999999e-06 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.232773+0000", 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2340000000000002e-06 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.232791+0000", 2024-09-13T13:06:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7785999999999999e-05 2024-09-13T13:06:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.232799+0000", 2024-09-13T13:06:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8280000000000003e-06 2024-09-13T13:06:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.232853+0000", 2024-09-13T13:06:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4045000000000002e-05 2024-09-13T13:06:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.232965+0000", 2024-09-13T13:06:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111995 2024-09-13T13:06:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.233056+0000", 2024-09-13T13:06:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1401000000000001e-05 2024-09-13T13:06:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.235389+0000", 2024-09-13T13:06:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023328390000000002 2024-09-13T13:06:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.236221+0000", 2024-09-13T13:06:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00083191299999999999 2024-09-13T13:06:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.344981+0000", 2024-09-13T13:06:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.108760176 2024-09-13T13:06:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.345010+0000", 2024-09-13T13:06:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8665000000000001e-05 2024-09-13T13:06:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.345031+0000", 2024-09-13T13:06:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1253000000000001e-05 2024-09-13T13:06:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:199095 7.5 7:a6471f00:::benchmark_data_smithi052_71865_object77994:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T13:06:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:25.236687+0000", 2024-09-13T13:06:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.030896661, 2024-09-13T13:06:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.108461706, 2024-09-13T13:06:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 199095 2024-09-13T13:06:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.236687+0000", 2024-09-13T13:06:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.236687+0000", 2024-09-13T13:06:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.236688+0000", 2024-09-13T13:06:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.271e-06 2024-09-13T13:06:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.236693+0000", 2024-09-13T13:06:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0749999999999997e-06 2024-09-13T13:06:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.236703+0000", 2024-09-13T13:06:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7729999999999994e-06 2024-09-13T13:06:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.236708+0000", 2024-09-13T13:06:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1789999999999999e-06 2024-09-13T13:06:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.236744+0000", 2024-09-13T13:06:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6352e-05 2024-09-13T13:06:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.236830+0000", 2024-09-13T13:06:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5984999999999998e-05 2024-09-13T13:06:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.236901+0000", 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0603000000000001e-05 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.238784+0000", 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018826540000000001 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.238807+0000", 2024-09-13T13:06:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2804000000000001e-05 2024-09-13T13:06:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.345068+0000", 2024-09-13T13:06:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10626139 2024-09-13T13:06:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.345115+0000", 2024-09-13T13:06:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6869999999999997e-05 2024-09-13T13:06:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:25.345149+0000", 2024-09-13T13:06:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.375e-05 2024-09-13T13:06:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:201148 7.5 7:acaa5b9f:::benchmark_data_smithi052_71865_object80047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T13:06:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:27.353142+0000", 2024-09-13T13:06:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.914441359999998, 2024-09-13T13:06:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.081219269999999996, 2024-09-13T13:06:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 201148 2024-09-13T13:06:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.353142+0000", 2024-09-13T13:06:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.353142+0000", 2024-09-13T13:06:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.353143+0000", 2024-09-13T13:06:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2899999999999999e-07 2024-09-13T13:06:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.353145+0000", 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1789999999999998e-06 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.353149+0000", 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2570000000000001e-06 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.353153+0000", 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0080000000000004e-06 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.358545+0000", 2024-09-13T13:06:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0053923249999999999 2024-09-13T13:06:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.358669+0000", 2024-09-13T13:06:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123781 2024-09-13T13:06:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.358797+0000", 2024-09-13T13:06:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000128126 2024-09-13T13:06:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.362699+0000", 2024-09-13T13:06:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039019100000000002 2024-09-13T13:06:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.366909+0000", 2024-09-13T13:06:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042094009999999998 2024-09-13T13:06:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.434260+0000", 2024-09-13T13:06:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067351211999999994 2024-09-13T13:06:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.434316+0000", 2024-09-13T13:06:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6434000000000002e-05 2024-09-13T13:06:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.434361+0000", 2024-09-13T13:06:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5207999999999998e-05 2024-09-13T13:06:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:201149 7.5 7:a2d6e33d:::benchmark_data_smithi052_71865_object80048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T13:06:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:27.353157+0000", 2024-09-13T13:06:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.914426243999998, 2024-09-13T13:06:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.081312697000000003, 2024-09-13T13:06:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:06:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:06:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:06:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 201149 2024-09-13T13:06:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.353157+0000", 2024-09-13T13:06:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.353157+0000", 2024-09-13T13:06:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.353158+0000", 2024-09-13T13:06:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7800000000000002e-07 2024-09-13T13:06:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.353160+0000", 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9230000000000001e-06 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.353163+0000", 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8789999999999996e-06 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.353167+0000", 2024-09-13T13:06:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9809999999999997e-06 2024-09-13T13:06:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.360743+0000", 2024-09-13T13:06:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075759080000000001 2024-09-13T13:06:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.360866+0000", 2024-09-13T13:06:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012303300000000001 2024-09-13T13:06:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.360993+0000", 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012624000000000001 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.362778+0000", 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001785308 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.366935+0000", 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041570799999999996 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.434403+0000", 2024-09-13T13:06:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067467540000000006 2024-09-13T13:06:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:06:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.434435+0000", 2024-09-13T13:06:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2515999999999998e-05 2024-09-13T13:06:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:06:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:27.434470+0000", 2024-09-13T13:06:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4910999999999998e-05 2024-09-13T13:06:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:49.404 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:06:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:49.406 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:49.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:49.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:48 smithi052 ceph-mon[36682]: pgmap v268: 148 pgs: 148 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-13T13:06:49.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:48 smithi052 ceph-mon[36682]: 5.1d scrub starts 2024-09-13T13:06:49.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:48 smithi052 ceph-mon[36682]: 5.1d scrub ok 2024-09-13T13:06:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:48 smithi052 ceph-mon[29471]: pgmap v268: 148 pgs: 148 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-13T13:06:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:48 smithi052 ceph-mon[29471]: 5.1d scrub starts 2024-09-13T13:06:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:48 smithi052 ceph-mon[29471]: 5.1d scrub ok 2024-09-13T13:06:49.507 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:49.529 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:49.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:49.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:49.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:49.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:49.666 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:49.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:49.763 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 18 16 40451 40435 140.371 106.5 0.0108358 0.00711833 2024-09-13T13:06:49.769 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:49.800 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:49.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:49.902 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:49.935 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:49.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:49.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:50.037 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:50.076 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:50.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:50.178 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:50.215 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:50.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:50.317 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:50.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:49 smithi167 ceph-mon[33178]: 5.0 scrub starts 2024-09-13T13:06:50.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:49 smithi167 ceph-mon[33178]: 5.0 scrub ok 2024-09-13T13:06:50.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:50.353 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:50.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:50.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T13:06:50.455 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:50.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:49 smithi052 ceph-mon[36682]: 5.0 scrub starts 2024-09-13T13:06:50.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:49 smithi052 ceph-mon[36682]: 5.0 scrub ok 2024-09-13T13:06:50.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:49 smithi052 ceph-mon[29471]: 5.0 scrub starts 2024-09-13T13:06:50.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:49 smithi052 ceph-mon[29471]: 5.0 scrub ok 2024-09-13T13:06:50.494 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:50.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:50.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:50.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:50.636 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:50.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:50.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:50.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:50.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:50.762 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:50.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:50.762 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 19 16 42154 42138 138.585 106.438 0.00249784 0.00719276 2024-09-13T13:06:50.769 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:06:50.862 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:50.899 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:50.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:50.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:51.001 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:51.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:51.042 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:51.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:51.143 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:51.182 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:51.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:51.217 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:51 smithi167 ceph-mon[33178]: pgmap v270: 148 pgs: 148 active+clean; 2.3 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-09-13T13:06:51.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:51 smithi052 ceph-mon[36682]: pgmap v270: 148 pgs: 148 active+clean; 2.3 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-09-13T13:06:51.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:51 smithi052 ceph-mon[29471]: pgmap v270: 148 pgs: 148 active+clean; 2.3 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-09-13T13:06:51.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:51.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:51.326 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:51.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:51.375 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:06:51.375 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:06:51.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.375 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:43159 8.9 8:97ee040e:::benchmark_data_smithi052_101302_object43158:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-13T13:06:51.376 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:51.369203+0000", 2024-09-13T13:06:51.376 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00388369, 2024-09-13T13:06:51.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039170189999999999, 2024-09-13T13:06:51.376 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:51.376 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:51.376 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:06:51.376 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:51.376 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:06:51.377 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:06:51.377 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 43159 2024-09-13T13:06:51.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.377 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:51.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:51.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.369203+0000", 2024-09-13T13:06:51.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:51.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:51.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.369203+0000", 2024-09-13T13:06:51.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:51.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:51.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.369204+0000", 2024-09-13T13:06:51.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.967e-06 2024-09-13T13:06:51.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:51.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.369293+0000", 2024-09-13T13:06:51.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8530999999999997e-05 2024-09-13T13:06:51.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:51.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.369305+0000", 2024-09-13T13:06:51.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1824999999999999e-05 2024-09-13T13:06:51.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:51.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.369314+0000", 2024-09-13T13:06:51.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1589999999999999e-06 2024-09-13T13:06:51.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:51.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.369499+0000", 2024-09-13T13:06:51.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018461000000000001 2024-09-13T13:06:51.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:51.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.369599+0000", 2024-09-13T13:06:51.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010084799999999999 2024-09-13T13:06:51.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:51.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.369692+0000", 2024-09-13T13:06:51.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2054999999999999e-05 2024-09-13T13:06:51.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:51.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.372902+0000", 2024-09-13T13:06:51.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032107300000000002 2024-09-13T13:06:51.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:06:51.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.373005+0000", 2024-09-13T13:06:51.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010304199999999999 2024-09-13T13:06:51.381 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:51.382 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:51.382 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:51.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.382 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:43163 8.9 8:9d2e0e67:::benchmark_data_smithi052_101302_object43162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-13T13:06:51.382 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:51.369950+0000", 2024-09-13T13:06:51.382 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.003136706, 2024-09-13T13:06:51.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003247378, 2024-09-13T13:06:51.382 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:06:51.382 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:06:51.382 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:06:51.382 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:06:51.383 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:06:51.383 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:06:51.383 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 43163 2024-09-13T13:06:51.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.383 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:06:51.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:06:51.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.369950+0000", 2024-09-13T13:06:51.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:51.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:06:51.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.369950+0000", 2024-09-13T13:06:51.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:06:51.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:06:51.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.369951+0000", 2024-09-13T13:06:51.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2300000000000001e-06 2024-09-13T13:06:51.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:06:51.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.370005+0000", 2024-09-13T13:06:51.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4336999999999998e-05 2024-09-13T13:06:51.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:06:51.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.370020+0000", 2024-09-13T13:06:51.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4932e-05 2024-09-13T13:06:51.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:06:51.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.370028+0000", 2024-09-13T13:06:51.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6389999999999992e-06 2024-09-13T13:06:51.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:06:51.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.370999+0000", 2024-09-13T13:06:51.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097094300000000004 2024-09-13T13:06:51.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:06:51.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.371142+0000", 2024-09-13T13:06:51.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000143592 2024-09-13T13:06:51.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:06:51.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.371271+0000", 2024-09-13T13:06:51.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012915800000000001 2024-09-13T13:06:51.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:06:51.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:06:51.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:06:51.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:51.373150+0000", 2024-09-13T13:06:51.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001878691 2024-09-13T13:06:51.387 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:51.387 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:06:51.387 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:51.387 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:06:51.387 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:06:51.387 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 2 2024-09-13T13:06:51.387 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:06:51.427 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:51.464 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:51.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:51.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:51.545 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T13:06:51.554 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:51.565 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:51.615 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:51.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:51.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:51.717 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:51.763 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:06:51.763119+0000 min lat: 0.00118759 max lat: 0.131359 avg lat: 0.00730312 2024-09-13T13:06:51.763 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:06:51.763 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 20 16 43806 43790 136.818 103.25 0.00653251 0.00730312 2024-09-13T13:06:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:51.771 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:51.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:51.872 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:51.897 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:51.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:51.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:51.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:52.013 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:06:52.033 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:52.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:52.134 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T13:06:52.135 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:52.180 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:52.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:52.282 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:52.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[29471]: 2.1 scrub starts 2024-09-13T13:06:52.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[29471]: 2.1 scrub ok 2024-09-13T13:06:52.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[29471]: 5.1b scrub starts 2024-09-13T13:06:52.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[29471]: 5.1b scrub ok 2024-09-13T13:06:52.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3746112342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:06:52.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[29471]: 4.5 scrub starts 2024-09-13T13:06:52.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[29471]: 4.5 scrub ok 2024-09-13T13:06:52.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[36682]: 2.1 scrub starts 2024-09-13T13:06:52.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[36682]: 2.1 scrub ok 2024-09-13T13:06:52.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[36682]: 5.1b scrub starts 2024-09-13T13:06:52.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[36682]: 5.1b scrub ok 2024-09-13T13:06:52.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3746112342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:06:52.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[36682]: 4.5 scrub starts 2024-09-13T13:06:52.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:52 smithi052 ceph-mon[36682]: 4.5 scrub ok 2024-09-13T13:06:52.312 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:52.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:52.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:52 smithi167 ceph-mon[33178]: 2.1 scrub starts 2024-09-13T13:06:52.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:52 smithi167 ceph-mon[33178]: 2.1 scrub ok 2024-09-13T13:06:52.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:52 smithi167 ceph-mon[33178]: 5.1b scrub starts 2024-09-13T13:06:52.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:52 smithi167 ceph-mon[33178]: 5.1b scrub ok 2024-09-13T13:06:52.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:52 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3746112342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:06:52.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:52 smithi167 ceph-mon[33178]: 4.5 scrub starts 2024-09-13T13:06:52.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:52 smithi167 ceph-mon[33178]: 4.5 scrub ok 2024-09-13T13:06:52.327 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:06:52.327 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:06:52.327 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:06:52.327 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:06:52.327 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:06:52.327 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:06:52.362 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:06:52.362 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:06:52.399 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:06:52.400 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:06:52.413 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:52.440 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:06:52.440 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:06:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:52.465 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:52.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:52.477 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:06:52.477 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:06:52.502 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:06:52.514 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:06:52.514 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:06:52.566 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:52.610 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:52.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:52.712 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:52.745 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:52.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:52.763 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 21 16 45418 45402 135.1 100.75 0.00308196 0.00739607 2024-09-13T13:06:52.846 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:52.879 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:52.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:52.981 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:53.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:53.114 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:06:53.114 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:06:53.122 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:53.164 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:53.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:53.198 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:53.266 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:53.309 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:53.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:53.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:53 smithi167 ceph-mon[33178]: 3.f scrub starts 2024-09-13T13:06:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:53 smithi167 ceph-mon[33178]: 3.f scrub ok 2024-09-13T13:06:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:53 smithi167 ceph-mon[33178]: 5.16 scrub starts 2024-09-13T13:06:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:53 smithi167 ceph-mon[33178]: 5.16 scrub ok 2024-09-13T13:06:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:53 smithi167 ceph-mon[33178]: pgmap v271: 148 pgs: 148 active+clean; 2.3 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-13T13:06:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:53 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:53 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3746112342' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:06:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:53 smithi167 ceph-mon[33178]: osdmap e181: 8 total, 8 up, 8 in 2024-09-13T13:06:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:53 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3262525576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:53 smithi167 ceph-mon[33178]: osdmap e182: 8 total, 8 up, 8 in 2024-09-13T13:06:53.412 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:53.441 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:53.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:53.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[36682]: 3.f scrub starts 2024-09-13T13:06:53.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[36682]: 3.f scrub ok 2024-09-13T13:06:53.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[36682]: 5.16 scrub starts 2024-09-13T13:06:53.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[36682]: 5.16 scrub ok 2024-09-13T13:06:53.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[36682]: pgmap v271: 148 pgs: 148 active+clean; 2.3 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-13T13:06:53.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:53.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3746112342' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:06:53.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[36682]: osdmap e181: 8 total, 8 up, 8 in 2024-09-13T13:06:53.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3262525576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:53.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[36682]: osdmap e182: 8 total, 8 up, 8 in 2024-09-13T13:06:53.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[29471]: 3.f scrub starts 2024-09-13T13:06:53.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[29471]: 3.f scrub ok 2024-09-13T13:06:53.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[29471]: 5.16 scrub starts 2024-09-13T13:06:53.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[29471]: 5.16 scrub ok 2024-09-13T13:06:53.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[29471]: pgmap v271: 148 pgs: 148 active+clean; 2.3 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-13T13:06:53.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:53.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3746112342' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:06:53.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[29471]: osdmap e181: 8 total, 8 up, 8 in 2024-09-13T13:06:53.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3262525576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:53.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:53 smithi052 ceph-mon[29471]: osdmap e182: 8 total, 8 up, 8 in 2024-09-13T13:06:53.543 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:53.575 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:53.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:53.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:53.677 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:53.708 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:53.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:53.763 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 22 16 47211 47195 134.053 112.062 0.00342267 0.00745383 2024-09-13T13:06:53.810 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:53.838 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:53.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:53.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:53.896 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T13:06:53.940 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:53.972 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:53.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:54.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:06:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:06:54.074 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:54.113 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:54.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:54.215 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:54.253 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:54.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:54.280 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:06:54.356 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:54.398 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:54.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:54.501 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:54.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:54.546 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:54.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:54.647 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:54.686 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:54.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:54.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:54.763 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 23 16 48813 48797 132.577 100.125 0.00475802 0.00753773 2024-09-13T13:06:54.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:54.823 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:54.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:54.901 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:06:54.901 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:06:54.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:54.957 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:54.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:54.976 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:06:55.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:55.089 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:55.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:55.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:55.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:55.219 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:55.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:55.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:55 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:06:55.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:55 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:06:55.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:55 smithi052 ceph-mon[29471]: pgmap v274: 148 pgs: 148 active+clean; 2.8 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 168 MiB/s wr, 2.70k op/s 2024-09-13T13:06:55.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:55.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:55 smithi052 ceph-mon[29471]: osdmap e183: 8 total, 8 up, 8 in 2024-09-13T13:06:55.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:55 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:06:55.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:55 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:06:55.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:55 smithi052 ceph-mon[36682]: pgmap v274: 148 pgs: 148 active+clean; 2.8 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 168 MiB/s wr, 2.70k op/s 2024-09-13T13:06:55.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:55.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:55 smithi052 ceph-mon[36682]: osdmap e183: 8 total, 8 up, 8 in 2024-09-13T13:06:55.319 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:55 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:06:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:55 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:06:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:55 smithi167 ceph-mon[33178]: pgmap v274: 148 pgs: 148 active+clean; 2.8 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 168 MiB/s wr, 2.70k op/s 2024-09-13T13:06:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:06:55.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:55 smithi167 ceph-mon[33178]: osdmap e183: 8 total, 8 up, 8 in 2024-09-13T13:06:55.373 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:55.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:55.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:55.475 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:55.476 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:55.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:55.529 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:55.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:55.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:55.649 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T13:06:55.664 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:55.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:55.764 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 24 16 50366 50350 131.095 97.0625 0.0100941 0.00762277 2024-09-13T13:06:55.766 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:55.791 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:55.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:55.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:55.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:55.923 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:55.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:56.025 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:56.051 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T13:06:56.053 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:56.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:56.155 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:56.209 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:56.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:56.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:56 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3347218030' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:06:56.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:56 smithi052 ceph-mon[29471]: from='client.25267 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:06:56.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:56 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3347218030' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:06:56.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:56 smithi052 ceph-mon[36682]: from='client.25267 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:06:56.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:06:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:06:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:06:56.311 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:56.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:56 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3347218030' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:06:56.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:56 smithi167 ceph-mon[33178]: from='client.25267 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:06:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:56.343 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:56.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:56.433 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:06:56.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:56.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:56.537 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:56.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:56.638 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:56.686 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:56.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:56.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:56.765 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 25 16 51730 51714 129.255 85.25 0.0107434 0.00773122 2024-09-13T13:06:56.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:56.820 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:56.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:56.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:56.911 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:56.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:56.945 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T13:06:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:56.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:56.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:57.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:57.081 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:57.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:57.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[36682]: pgmap v276: 148 pgs: 148 active+clean; 2.8 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-13T13:06:57.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:06:57.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[36682]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:06:57.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[36682]: osdmap e184: 8 total, 8 up, 8 in 2024-09-13T13:06:57.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:57.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:57.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1450346068' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:57.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[36682]: from='client.35267 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:57.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[29471]: pgmap v276: 148 pgs: 148 active+clean; 2.8 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-13T13:06:57.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:06:57.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[29471]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:06:57.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[29471]: osdmap e184: 8 total, 8 up, 8 in 2024-09-13T13:06:57.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:57.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:57.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1450346068' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:57.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:57 smithi052 ceph-mon[29471]: from='client.35267 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:57.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:06:57.213 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:57.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:06:57.315 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:57.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:57 smithi167 ceph-mon[33178]: pgmap v276: 148 pgs: 148 active+clean; 2.8 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-13T13:06:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:57 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:06:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:57 smithi167 ceph-mon[33178]: from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:06:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:57 smithi167 ceph-mon[33178]: osdmap e184: 8 total, 8 up, 8 in 2024-09-13T13:06:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:57 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:57 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:57 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1450346068' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:57 smithi167 ceph-mon[33178]: from='client.35267 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:06:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:57.353 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:57.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:57.454 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:57.483 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:57.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:57.494 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:06:57.585 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:57.609 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:57.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:57.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:57.622 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:06:57.623 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:06:57.711 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:57.762 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:57.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:57.765 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 26 16 53039 53023 127.43 81.8125 0.00507629 0.00784099 2024-09-13T13:06:57.864 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:06:57.894 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:57.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:57.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:06:57.954 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:06:57.995 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:58.054 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:58.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:58.154 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:06:58.204 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:58.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:58.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:06:58.306 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:58.346 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:58.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:58.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:58.362 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:58 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:58.362 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:58 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:58.362 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:58 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:58.362 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:58 smithi052 ceph-mon[29471]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:06:58.362 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:58 smithi052 ceph-mon[29471]: osdmap e185: 8 total, 8 up, 8 in 2024-09-13T13:06:58.362 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:58 smithi052 ceph-mon[29471]: osdmap e186: 8 total, 8 up, 8 in 2024-09-13T13:06:58.363 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:58 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:58.363 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:58 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:58.363 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:58 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:58.363 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:58 smithi052 ceph-mon[36682]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:06:58.363 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:58 smithi052 ceph-mon[36682]: osdmap e185: 8 total, 8 up, 8 in 2024-09-13T13:06:58.363 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:58 smithi052 ceph-mon[36682]: osdmap e186: 8 total, 8 up, 8 in 2024-09-13T13:06:58.449 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:58.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:58.485 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:58.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:58 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:58.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:58 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:58.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:58 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:06:58.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:58 smithi167 ceph-mon[33178]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:06:58.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:58 smithi167 ceph-mon[33178]: osdmap e185: 8 total, 8 up, 8 in 2024-09-13T13:06:58.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:58 smithi167 ceph-mon[33178]: osdmap e186: 8 total, 8 up, 8 in 2024-09-13T13:06:58.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:06:58.614 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:58.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:06:58.716 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:58.757 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:58.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:58.765 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 27 16 54489 54473 126.067 90.625 0.0269408 0.00792647 2024-09-13T13:06:58.860 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:58.869 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T13:06:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:58.908 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:58.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:59.010 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:59.074 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:59.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:59.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:59.109 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:59 smithi052 ceph-mon[29471]: pgmap v279: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-13T13:06:59.148 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:06:59.148 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:06:59.176 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:59.184 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:06:59.184 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:06:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:59.222 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:59.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:59.227 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:06:59.227 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:06:59.263 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:06:59.264 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:06:59.299 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:06:59.299 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:06:59.323 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:59.334 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:06:59.334 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:06:59.366 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:59.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:59.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:59.368 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:59 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:06:59.368 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:59 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:06:59.368 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:59 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:59.368 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:06:59 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1298517854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:59.368 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:06:59.369 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:06:59.369 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:59 smithi052 ceph-mon[36682]: pgmap v279: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-13T13:06:59.369 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:59 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:06:59.369 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:59 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:06:59.369 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:59 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:59.369 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:06:59 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1298517854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:59.407 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:06:59.408 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:06:59.468 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:06:59.509 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:59.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:06:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:59 smithi167 ceph-mon[33178]: pgmap v279: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-13T13:06:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:59 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:06:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:59 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:06:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:59 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:06:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:06:59 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1298517854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:06:59.611 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:06:59.649 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:59.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:59.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:06:59.751 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:06:59.766 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 28 16 55959 55943 124.839 91.875 0.00890809 0.00800577 2024-09-13T13:06:59.792 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:06:59.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:59.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:06:59.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:06:59.922 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:06:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:06:59.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:06:59.941 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:00.023 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:00.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:00.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:00.158 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:00.197 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:00.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:00.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:00.300 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:00.337 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:00.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:00.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:00.439 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:00.472 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:00.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:00.521 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:00.574 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:00.606 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:00.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:00.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:00.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:00.742 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:00.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:00.767 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 29 16 57544 57528 123.948 99.0625 0.0054179 0.0080628 2024-09-13T13:07:00.843 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:00.868 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:00.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:00.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:00.970 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:01.011 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:01.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:01.114 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:01.151 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:01.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:01.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:01.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:01.287 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:01.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:01.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:01.384 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:01 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:07:01.384 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:01 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:07:01.384 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:01 smithi167 ceph-mon[33178]: pgmap v281: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 898 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-13T13:07:01.384 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:01 smithi167 ceph-mon[33178]: osdmap e187: 8 total, 8 up, 8 in 2024-09-13T13:07:01.390 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:01.426 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:01.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:01.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:01 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:07:01.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:01 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:07:01.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:01 smithi052 ceph-mon[36682]: pgmap v281: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 898 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-13T13:07:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:01 smithi052 ceph-mon[36682]: osdmap e187: 8 total, 8 up, 8 in 2024-09-13T13:07:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:01 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:07:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:01 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:07:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:01 smithi052 ceph-mon[29471]: pgmap v281: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 898 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-13T13:07:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:01 smithi052 ceph-mon[29471]: osdmap e187: 8 total, 8 up, 8 in 2024-09-13T13:07:01.528 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:01.562 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:01.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:01.663 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:01.697 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:01.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:01.767 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 30 16 59046 59030 122.946 93.875 0.00660237 0.00812786 2024-09-13T13:07:01.799 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:01.838 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:01.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:01.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:01.908 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:07:01.942 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:01.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:01.991 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:01.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:02.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:02.119 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:02.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:02.220 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:02.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:02.261 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:02.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:02.363 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:02.400 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:02.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:02.502 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:02.518 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:07:02.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:02.525 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:02.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:02.627 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:02.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:02.658 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:02.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:02.760 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:02.769 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 31 16 60227 60211 121.355 73.8125 0.00450484 0.00823462 2024-09-13T13:07:02.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:02.806 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:02.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:02.811 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T13:07:02.907 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:02.955 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:02.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:03.057 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:03.094 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:03.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:03.196 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:03.234 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:03.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:03.336 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:03.375 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:03.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:03.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:03.477 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[29471]: pgmap v283: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 904 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-09-13T13:07:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2372425105' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:07:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[29471]: from='client.25282 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:07:03.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[29471]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:07:03.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[29471]: osdmap e188: 8 total, 8 up, 8 in 2024-09-13T13:07:03.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:07:03.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:07:03.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[36682]: pgmap v283: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 904 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-09-13T13:07:03.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2372425105' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:07:03.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[36682]: from='client.25282 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:07:03.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[36682]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:07:03.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[36682]: osdmap e188: 8 total, 8 up, 8 in 2024-09-13T13:07:03.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:07:03.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:03 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:07:03.515 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:03.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:03.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:03 smithi167 ceph-mon[33178]: pgmap v283: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 904 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-09-13T13:07:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:03 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2372425105' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:07:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:03 smithi167 ceph-mon[33178]: from='client.25282 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:07:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:03 smithi167 ceph-mon[33178]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:07:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:03 smithi167 ceph-mon[33178]: osdmap e188: 8 total, 8 up, 8 in 2024-09-13T13:07:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:03 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:07:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:03 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:07:03.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:03.640 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:03.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:03.741 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:03.766 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:03.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:03.769 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 32 16 61454 61438 119.959 76.6875 0.00551424 0.00832828 2024-09-13T13:07:03.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:03.901 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:03.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:04.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:04.034 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:04.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:04.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:07:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:07:04.136 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:04.173 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:04.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:04.274 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:04.306 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:04.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:04.408 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:04.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:04.449 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:04.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:04 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:07:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:04 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:07:04.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:04 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:07:04.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:04 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:07:04.551 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:04.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:04 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:07:04.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:04 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:07:04.586 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:04.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:04.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:04.689 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:04.726 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:04.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:04.769 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 33 16 62909 62893 119.079 90.9375 0.00715221 0.00838183 2024-09-13T13:07:04.829 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:04.863 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:04.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:04.965 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:04.992 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:04.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:05.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:05.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:05.131 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:05.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:05.232 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:05.273 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:05.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:05.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:05.375 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:05.403 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:05.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:05.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:05.408 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:05 smithi052 ceph-mon[36682]: pgmap v285: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-13T13:07:05.408 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:05.408 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:05 smithi052 ceph-mon[36682]: osdmap e189: 8 total, 8 up, 8 in 2024-09-13T13:07:05.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:05 smithi052 ceph-mon[29471]: pgmap v285: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-13T13:07:05.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:05.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:05 smithi052 ceph-mon[29471]: osdmap e189: 8 total, 8 up, 8 in 2024-09-13T13:07:05.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T13:07:05.482 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:07:05.482 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:07:05.482 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:07:05.482 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:07:05.482 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:07:05.505 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:05.545 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:05.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:05.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:05 smithi167 ceph-mon[33178]: pgmap v285: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-13T13:07:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:05 smithi167 ceph-mon[33178]: osdmap e189: 8 total, 8 up, 8 in 2024-09-13T13:07:05.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:05.678 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:05.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:05.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 34 16 64326 64310 118.175 88.5625 0.00812888 0.00845682 2024-09-13T13:07:05.780 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:05.825 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:05.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:05.927 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:05.966 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:05.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:05.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:06.067 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:06.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:06.105 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:06.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:06.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:06.245 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:06.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:06.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:06.257 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:06.298 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T13:07:06.310 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:06.347 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:06.400 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:07:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:07:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:07:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:06.410 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:06.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:06.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:06.539 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:06.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:06.641 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:06.693 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:06.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:06.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:06.771 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 35 16 65685 65669 117.225 84.9375 0.0113137 0.0085246 2024-09-13T13:07:06.798 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:06.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:06.838 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:06.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:06.939 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:06.964 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:06.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:07.066 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:07.095 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:07.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:07.197 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:07.249 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:07.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:07.297 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:07:07.298 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:07:07.298 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:07 smithi052 ceph-mon[36682]: pgmap v287: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-13T13:07:07.299 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:07 smithi052 ceph-mon[29471]: pgmap v287: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-13T13:07:07.338 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:07:07.339 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:07:07.350 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:07.385 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:07:07.385 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:07:07.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:07.418 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:07.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:07.428 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:07.430 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:07:07.431 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:07:07.468 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:07:07.468 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:07:07.488 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:07.508 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:07:07.508 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:07:07.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:07.545 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:07:07.545 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:07:07.545 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:07.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:07.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:07 smithi167 ceph-mon[33178]: pgmap v287: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-13T13:07:07.584 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:07:07.584 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:07:07.648 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:07.682 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:07.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:07.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:07.771 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 36 16 66884 66868 116.051 74.9375 0.00603083 0.00861171 2024-09-13T13:07:07.784 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:07.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:07.830 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:07.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:07.931 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:07.977 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:07.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:07.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:08.078 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:08.113 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:08.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:08.180 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:08 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:07:08.182 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:07:08.215 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:08.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:08.272 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:08.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:08.373 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:08.401 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:08.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:08 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:07:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:08 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:07:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:08 smithi052 ceph-mon[36682]: osdmap e190: 8 total, 8 up, 8 in 2024-09-13T13:07:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2336311127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2206758053' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:07:08.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:08 smithi052 ceph-mon[36682]: from='client.25288 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:07:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:08 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:07:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:08 smithi052 ceph-mon[29471]: osdmap e190: 8 total, 8 up, 8 in 2024-09-13T13:07:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2336311127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2206758053' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:07:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:08 smithi052 ceph-mon[29471]: from='client.25288 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:07:08.502 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:08.540 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:08.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:08.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:08 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:07:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:08 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:07:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:08 smithi167 ceph-mon[33178]: osdmap e190: 8 total, 8 up, 8 in 2024-09-13T13:07:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2336311127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:08.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2206758053' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:07:08.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:08 smithi167 ceph-mon[33178]: from='client.25288 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:07:08.642 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:08.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:08.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:08.713 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:07:08.713 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:07:08.770 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:08.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 37 16 68130 68114 115.019 77.875 0.0323343 0.00868888 2024-09-13T13:07:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:08.809 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:08.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:08.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:08.938 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:08.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:09.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:09.077 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:09.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:09.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:09.179 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:09.186 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T13:07:09.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:09.210 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:09.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:09.311 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:09.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:09.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:09.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:09.446 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:09.481 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:09.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[29471]: pgmap v289: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-13T13:07:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:07:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[29471]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:07:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[29471]: osdmap e191: 8 total, 8 up, 8 in 2024-09-13T13:07:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[29471]: 3.8 scrub starts 2024-09-13T13:07:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[29471]: 3.8 scrub ok 2024-09-13T13:07:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/643864713' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:07:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[29471]: from='client.25300 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:07:09.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[36682]: pgmap v289: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-13T13:07:09.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:07:09.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[36682]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:07:09.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[36682]: osdmap e191: 8 total, 8 up, 8 in 2024-09-13T13:07:09.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[36682]: 3.8 scrub starts 2024-09-13T13:07:09.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[36682]: 3.8 scrub ok 2024-09-13T13:07:09.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/643864713' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:07:09.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:09 smithi052 ceph-mon[36682]: from='client.25300 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:07:09.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:09 smithi167 ceph-mon[33178]: pgmap v289: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-13T13:07:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:09 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:07:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:09 smithi167 ceph-mon[33178]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:07:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:09 smithi167 ceph-mon[33178]: osdmap e191: 8 total, 8 up, 8 in 2024-09-13T13:07:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:09 smithi167 ceph-mon[33178]: 3.8 scrub starts 2024-09-13T13:07:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:09 smithi167 ceph-mon[33178]: 3.8 scrub ok 2024-09-13T13:07:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:09 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/643864713' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:07:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:09 smithi167 ceph-mon[33178]: from='client.25300 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:07:09.583 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:09.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:09.626 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:09.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:09.727 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:09.762 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:09.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:09.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:09.771 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 38 16 69560 69544 114.344 89.375 0.0141987 0.00874011 2024-09-13T13:07:09.864 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:09.896 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:09.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:09.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:10.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:10.030 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:10.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:10.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:10.163 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:10.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:10.265 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:10.303 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:10.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:10.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:10.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:10.437 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:10.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[36682]: 5.15 scrub starts 2024-09-13T13:07:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[36682]: 5.15 scrub ok 2024-09-13T13:07:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[36682]: 3.17 scrub starts 2024-09-13T13:07:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[36682]: 3.17 scrub ok 2024-09-13T13:07:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[36682]: from='client.25300 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:07:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[36682]: osdmap e192: 8 total, 8 up, 8 in 2024-09-13T13:07:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[36682]: 4.19 scrub starts 2024-09-13T13:07:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[36682]: 4.19 scrub ok 2024-09-13T13:07:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[29471]: 5.15 scrub starts 2024-09-13T13:07:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[29471]: 5.15 scrub ok 2024-09-13T13:07:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[29471]: 3.17 scrub starts 2024-09-13T13:07:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[29471]: 3.17 scrub ok 2024-09-13T13:07:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[29471]: from='client.25300 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:07:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[29471]: osdmap e192: 8 total, 8 up, 8 in 2024-09-13T13:07:10.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[29471]: 4.19 scrub starts 2024-09-13T13:07:10.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:10 smithi052 ceph-mon[29471]: 4.19 scrub ok 2024-09-13T13:07:10.537 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:10 smithi167 ceph-mon[33178]: 5.15 scrub starts 2024-09-13T13:07:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:10 smithi167 ceph-mon[33178]: 5.15 scrub ok 2024-09-13T13:07:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:10 smithi167 ceph-mon[33178]: 3.17 scrub starts 2024-09-13T13:07:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:10 smithi167 ceph-mon[33178]: 3.17 scrub ok 2024-09-13T13:07:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:10 smithi167 ceph-mon[33178]: from='client.25300 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:07:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:10 smithi167 ceph-mon[33178]: osdmap e192: 8 total, 8 up, 8 in 2024-09-13T13:07:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:10 smithi167 ceph-mon[33178]: 4.19 scrub starts 2024-09-13T13:07:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:10 smithi167 ceph-mon[33178]: 4.19 scrub ok 2024-09-13T13:07:10.574 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:10.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:10.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:10.675 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:10.697 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:10.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:10.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:10.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 39 15 70825 70810 113.441 79.125 0.012989 0.00881024 2024-09-13T13:07:10.798 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:10.832 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:10.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:10.933 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:10.975 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:10.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:11.077 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:11.106 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:11.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:11.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:11.208 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:11.236 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:11.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:11.338 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:11.375 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:11.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:11.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:11.477 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:11 smithi052 ceph-mon[29471]: pgmap v292: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 918 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-09-13T13:07:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:11 smithi052 ceph-mon[29471]: 3.10 scrub starts 2024-09-13T13:07:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:11 smithi052 ceph-mon[29471]: 3.10 scrub ok 2024-09-13T13:07:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:11 smithi052 ceph-mon[29471]: 4.1 scrub starts 2024-09-13T13:07:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:11 smithi052 ceph-mon[29471]: 4.1 scrub ok 2024-09-13T13:07:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:11 smithi052 ceph-mon[36682]: pgmap v292: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 918 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-09-13T13:07:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:11 smithi052 ceph-mon[36682]: 3.10 scrub starts 2024-09-13T13:07:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:11 smithi052 ceph-mon[36682]: 3.10 scrub ok 2024-09-13T13:07:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:11 smithi052 ceph-mon[36682]: 4.1 scrub starts 2024-09-13T13:07:11.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:11 smithi052 ceph-mon[36682]: 4.1 scrub ok 2024-09-13T13:07:11.507 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:11.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:11.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:11 smithi167 ceph-mon[33178]: pgmap v292: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 918 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-09-13T13:07:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:11 smithi167 ceph-mon[33178]: 3.10 scrub starts 2024-09-13T13:07:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:11 smithi167 ceph-mon[33178]: 3.10 scrub ok 2024-09-13T13:07:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:11 smithi167 ceph-mon[33178]: 4.1 scrub starts 2024-09-13T13:07:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:11 smithi167 ceph-mon[33178]: 4.1 scrub ok 2024-09-13T13:07:11.610 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:11.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:11.648 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:11.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:11.667 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:07:11.749 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:11.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:07:11.771941+0000 min lat: 0.00118759 max lat: 0.156296 avg lat: 0.00886654 2024-09-13T13:07:11.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:07:11.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 40 16 72175 72159 112.713 84.3125 0.0138334 0.00886654 2024-09-13T13:07:11.779 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:11.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:11.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:11.880 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:11.927 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:11.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:11.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:12.029 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:12.069 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:12.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:12.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:12.207 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:12.240 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:12.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:12.321 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T13:07:12.321 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:07:12.342 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:12.369 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:12 smithi052 ceph-mon[36682]: 4.6 scrub starts 2024-09-13T13:07:12.370 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:12 smithi052 ceph-mon[36682]: 4.6 scrub ok 2024-09-13T13:07:12.370 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:12 smithi052 ceph-mon[36682]: 5.a scrub starts 2024-09-13T13:07:12.370 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:12 smithi052 ceph-mon[36682]: 5.a scrub ok 2024-09-13T13:07:12.370 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:12 smithi052 ceph-mon[36682]: 6.6 scrub starts 2024-09-13T13:07:12.370 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:12 smithi052 ceph-mon[36682]: 6.6 scrub ok 2024-09-13T13:07:12.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:12 smithi052 ceph-mon[29471]: 4.6 scrub starts 2024-09-13T13:07:12.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:12 smithi052 ceph-mon[29471]: 4.6 scrub ok 2024-09-13T13:07:12.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:12 smithi052 ceph-mon[29471]: 5.a scrub starts 2024-09-13T13:07:12.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:12 smithi052 ceph-mon[29471]: 5.a scrub ok 2024-09-13T13:07:12.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:12 smithi052 ceph-mon[29471]: 6.6 scrub starts 2024-09-13T13:07:12.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:12 smithi052 ceph-mon[29471]: 6.6 scrub ok 2024-09-13T13:07:12.422 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:12.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:12.525 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:12.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:12 smithi167 ceph-mon[33178]: 4.6 scrub starts 2024-09-13T13:07:12.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:12 smithi167 ceph-mon[33178]: 4.6 scrub ok 2024-09-13T13:07:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:12 smithi167 ceph-mon[33178]: 5.a scrub starts 2024-09-13T13:07:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:12 smithi167 ceph-mon[33178]: 5.a scrub ok 2024-09-13T13:07:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:12 smithi167 ceph-mon[33178]: 6.6 scrub starts 2024-09-13T13:07:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:12 smithi167 ceph-mon[33178]: 6.6 scrub ok 2024-09-13T13:07:12.573 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:12.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:12.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:12.675 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:12.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:12.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:12.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 41 16 73423 73407 111.866 78 0.00872781 0.00893372 2024-09-13T13:07:12.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:12.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:12.855 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:12.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:12.877 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T13:07:12.878 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:12.939 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:12.956 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:13.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:13.035 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:13.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:13.136 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:13.168 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:13.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:13.270 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:13.317 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:13.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:13.419 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:13.452 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:13.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:13.481 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:13 smithi052 ceph-mon[36682]: 6.15 scrub starts 2024-09-13T13:07:13.482 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:13 smithi052 ceph-mon[36682]: 6.15 scrub ok 2024-09-13T13:07:13.482 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:13 smithi052 ceph-mon[36682]: pgmap v293: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-13T13:07:13.482 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:13 smithi052 ceph-mon[36682]: 5.1c scrub starts 2024-09-13T13:07:13.482 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:13 smithi052 ceph-mon[36682]: 5.1c scrub ok 2024-09-13T13:07:13.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:13 smithi052 ceph-mon[29471]: 6.15 scrub starts 2024-09-13T13:07:13.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:13 smithi052 ceph-mon[29471]: 6.15 scrub ok 2024-09-13T13:07:13.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:13 smithi052 ceph-mon[29471]: pgmap v293: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-13T13:07:13.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:13 smithi052 ceph-mon[29471]: 5.1c scrub starts 2024-09-13T13:07:13.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:13 smithi052 ceph-mon[29471]: 5.1c scrub ok 2024-09-13T13:07:13.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:13 smithi167 ceph-mon[33178]: 6.15 scrub starts 2024-09-13T13:07:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:13 smithi167 ceph-mon[33178]: 6.15 scrub ok 2024-09-13T13:07:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:13 smithi167 ceph-mon[33178]: pgmap v293: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-13T13:07:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:13 smithi167 ceph-mon[33178]: 5.1c scrub starts 2024-09-13T13:07:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:13 smithi167 ceph-mon[33178]: 5.1c scrub ok 2024-09-13T13:07:13.581 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:13.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:13.683 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:13.711 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:13.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:13.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:13.760 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:07:13.761 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:07:13.773 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 42 16 74597 74581 110.95 73.375 0.0155454 0.00900784 2024-09-13T13:07:13.797 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:07:13.798 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:07:13.812 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:13.839 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:07:13.839 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:07:13.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:13.872 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:13.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:13.878 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:07:13.878 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:07:13.919 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:07:13.919 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:07:13.961 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:07:13.961 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:07:13.973 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:13.998 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:13.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:13.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:14.007 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:07:14.008 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:07:14.029 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:14.047 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:07:14.048 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:07:14.100 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:14.153 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:14.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:14.165 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:07:14.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:07:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:07:14.255 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:14.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:14.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:14.354 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:07:14.355 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:14 smithi052 ceph-mon[36682]: 6.12 scrub starts 2024-09-13T13:07:14.355 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:14 smithi052 ceph-mon[36682]: 6.12 scrub ok 2024-09-13T13:07:14.355 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:14 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2408538791' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:14.358 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:14 smithi052 ceph-mon[29471]: 6.12 scrub starts 2024-09-13T13:07:14.358 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:14 smithi052 ceph-mon[29471]: 6.12 scrub ok 2024-09-13T13:07:14.358 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:14 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2408538791' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:14.372 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:07:14.387 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:14.440 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:14.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:14.541 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:14.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:14 smithi167 ceph-mon[33178]: 6.12 scrub starts 2024-09-13T13:07:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:14 smithi167 ceph-mon[33178]: 6.12 scrub ok 2024-09-13T13:07:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:14 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2408538791' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:14.572 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:14.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:14.673 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:14.751 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:14.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:14.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 43 16 75715 75699 109.995 69.875 0.00845804 0.00908563 2024-09-13T13:07:14.853 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:14.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:14.898 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:14.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:14.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:15.026 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:15.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:15.080 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:07:15.081 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:07:15.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.081 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:76075 8.a 8:5f105e8d:::benchmark_data_smithi052_101302_object76074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T13:07:15.081 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:15.067476+0000", 2024-09-13T13:07:15.081 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.011340868, 2024-09-13T13:07:15.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011383387999999999, 2024-09-13T13:07:15.081 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:15.081 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:15.081 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:07:15.081 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:15.081 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:15.081 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:15.082 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 76075 2024-09-13T13:07:15.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.082 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:15.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:15.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.067476+0000", 2024-09-13T13:07:15.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:15.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.067476+0000", 2024-09-13T13:07:15.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:15.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.067478+0000", 2024-09-13T13:07:15.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.091e-06 2024-09-13T13:07:15.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:15.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.067501+0000", 2024-09-13T13:07:15.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2929999999999999e-05 2024-09-13T13:07:15.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:15.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.067516+0000", 2024-09-13T13:07:15.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4450000000000001e-05 2024-09-13T13:07:15.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:15.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.067523+0000", 2024-09-13T13:07:15.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7959999999999996e-06 2024-09-13T13:07:15.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:15.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.067586+0000", 2024-09-13T13:07:15.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2479999999999998e-05 2024-09-13T13:07:15.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:15.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.067734+0000", 2024-09-13T13:07:15.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000148679 2024-09-13T13:07:15.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,5", 2024-09-13T13:07:15.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.067968+0000", 2024-09-13T13:07:15.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000233561 2024-09-13T13:07:15.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:15.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.077299+0000", 2024-09-13T13:07:15.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0093304410000000001 2024-09-13T13:07:15.086 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.086 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:15.086 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.086 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:76080 8.a 8:526e31d0:::benchmark_data_smithi052_101302_object76079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T13:07:15.086 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:15.069320+0000", 2024-09-13T13:07:15.086 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0094965589999999999, 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0096487429999999996, 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 76080 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.069320+0000", 2024-09-13T13:07:15.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:15.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.069320+0000", 2024-09-13T13:07:15.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:15.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.069322+0000", 2024-09-13T13:07:15.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.562e-06 2024-09-13T13:07:15.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:15.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.069345+0000", 2024-09-13T13:07:15.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2935999999999999e-05 2024-09-13T13:07:15.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:15.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.069356+0000", 2024-09-13T13:07:15.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1482e-05 2024-09-13T13:07:15.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:15.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.069364+0000", 2024-09-13T13:07:15.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0530000000000007e-06 2024-09-13T13:07:15.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:15.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.072813+0000", 2024-09-13T13:07:15.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0034481730000000001 2024-09-13T13:07:15.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:15.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073636+0000", 2024-09-13T13:07:15.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00082357100000000005 2024-09-13T13:07:15.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,5", 2024-09-13T13:07:15.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073844+0000", 2024-09-13T13:07:15.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020748199999999999 2024-09-13T13:07:15.091 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.091 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:15.091 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.092 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:76084 8.3 8:c6a87050:::benchmark_data_smithi052_101302_object76083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T13:07:15.092 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:15.072913+0000", 2024-09-13T13:07:15.092 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0059036920000000003, 2024-09-13T13:07:15.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.006132256, 2024-09-13T13:07:15.092 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:15.092 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:15.092 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:07:15.092 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:15.092 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:15.092 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:15.092 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 76084 2024-09-13T13:07:15.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.092 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:15.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:15.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.072913+0000", 2024-09-13T13:07:15.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:15.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.072913+0000", 2024-09-13T13:07:15.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:15.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.072915+0000", 2024-09-13T13:07:15.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7519999999999999e-06 2024-09-13T13:07:15.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:15.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.072943+0000", 2024-09-13T13:07:15.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8360999999999998e-05 2024-09-13T13:07:15.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:15.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.072954+0000", 2024-09-13T13:07:15.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1083999999999999e-05 2024-09-13T13:07:15.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.072962+0000", 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8860000000000004e-06 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073440+0000", 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00047720299999999999 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073545+0000", 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000105147 2024-09-13T13:07:15.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:07:15.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073675+0000", 2024-09-13T13:07:15.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012998099999999999 2024-09-13T13:07:15.097 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.097 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:15.097 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.097 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:76085 8.a 8:56b3ca3a:::benchmark_data_smithi052_101302_object76084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T13:07:15.097 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:15.073463+0000", 2024-09-13T13:07:15.097 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0053544270000000001, 2024-09-13T13:07:15.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0056540619999999996, 2024-09-13T13:07:15.097 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:15.097 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:15.097 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:07:15.098 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:15.098 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:15.098 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:15.098 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 76085 2024-09-13T13:07:15.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.098 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:15.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:15.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073463+0000", 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073463+0000", 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073464+0000", 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.06e-06 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:15.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073544+0000", 2024-09-13T13:07:15.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0121000000000001e-05 2024-09-13T13:07:15.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:15.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073554+0000", 2024-09-13T13:07:15.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0587e-05 2024-09-13T13:07:15.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:15.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073562+0000", 2024-09-13T13:07:15.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3640000000000001e-06 2024-09-13T13:07:15.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:15.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.077333+0000", 2024-09-13T13:07:15.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003771791 2024-09-13T13:07:15.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:15.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.077446+0000", 2024-09-13T13:07:15.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112477 2024-09-13T13:07:15.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,5", 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.077566+0000", 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120462 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:76087 8.3 8:c77e5e46:::benchmark_data_smithi052_101302_object76086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:15.073572+0000", 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0052447359999999998, 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0056058619999999997, 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:15.102 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:07:15.103 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:15.103 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:15.103 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:15.103 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 76087 2024-09-13T13:07:15.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.103 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:15.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:15.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073572+0000", 2024-09-13T13:07:15.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:15.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073572+0000", 2024-09-13T13:07:15.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:15.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073573+0000", 2024-09-13T13:07:15.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2900000000000002e-07 2024-09-13T13:07:15.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:15.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073595+0000", 2024-09-13T13:07:15.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2410999999999999e-05 2024-09-13T13:07:15.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:15.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073603+0000", 2024-09-13T13:07:15.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5780000000000001e-06 2024-09-13T13:07:15.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:15.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.073611+0000", 2024-09-13T13:07:15.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9109999999999993e-06 2024-09-13T13:07:15.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:15.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.076345+0000", 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027341330000000001 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.076507+0000", 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016196799999999999 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.076797+0000", 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000290458 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:15.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.25222.0:76077 8.0 e192/169 8:049197b6:::benchmark_data_smithi052_101302_object76076:head v 192'4672, mlcod=192'4669)", 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:15.075967+0000", 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0028496900000000002, 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003283345, 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.075967+0000", 2024-09-13T13:07:15.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:15.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.075967+0000", 2024-09-13T13:07:15.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:15.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.075969+0000", 2024-09-13T13:07:15.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.353e-06 2024-09-13T13:07:15.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:15.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.076001+0000", 2024-09-13T13:07:15.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2360999999999997e-05 2024-09-13T13:07:15.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:15.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.076012+0000", 2024-09-13T13:07:15.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1178e-05 2024-09-13T13:07:15.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:15.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.076019+0000", 2024-09-13T13:07:15.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3069999999999997e-06 2024-09-13T13:07:15.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.076081+0000", 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1940999999999995e-05 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.076103+0000", 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1461e-05 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.111 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.25222.0:76089 8.0 e192/169 8:0b99125a:::benchmark_data_smithi052_101302_object76088:head v 192'4673, mlcod=192'4669)", 2024-09-13T13:07:15.111 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:15.078498+0000", 2024-09-13T13:07:15.111 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.00031884699999999998, 2024-09-13T13:07:15.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00079867800000000004, 2024-09-13T13:07:15.111 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:15.111 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:15.111 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:07:15.111 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:15.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:15.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.078498+0000", 2024-09-13T13:07:15.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:15.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.078498+0000", 2024-09-13T13:07:15.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:15.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:15.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.078499+0000", 2024-09-13T13:07:15.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.116e-06 2024-09-13T13:07:15.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:15.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.078522+0000", 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2892999999999999e-05 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.078531+0000", 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7819999999999999e-06 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.078537+0000", 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3090000000000001e-06 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:15.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.078984+0000", 2024-09-13T13:07:15.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000446482 2024-09-13T13:07:15.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:15.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:15.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:15.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:15.079004+0000", 2024-09-13T13:07:15.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9978000000000001e-05 2024-09-13T13:07:15.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.114 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:15.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:15.114 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:07:15.114 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 7 2024-09-13T13:07:15.115 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:07:15.128 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:15.158 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:15.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:15.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:15.259 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:15.289 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:15.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:15.391 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:15.426 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:15.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:15.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:15.528 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:15.546 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:15 smithi052 ceph-mon[29471]: pgmap v294: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 118 MiB/s wr, 1.88k op/s 2024-09-13T13:07:15.547 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:15.547 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:15 smithi052 ceph-mon[29471]: from='client.25312 v1:172.21.15.52:0/3912519509' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:07:15.547 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:15 smithi052 ceph-mon[29471]: 6.1f scrub starts 2024-09-13T13:07:15.547 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:15 smithi052 ceph-mon[29471]: 6.1f scrub ok 2024-09-13T13:07:15.547 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:15 smithi052 ceph-mon[36682]: pgmap v294: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 118 MiB/s wr, 1.88k op/s 2024-09-13T13:07:15.547 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:15.547 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:15 smithi052 ceph-mon[36682]: from='client.25312 v1:172.21.15.52:0/3912519509' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:07:15.547 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:15 smithi052 ceph-mon[36682]: 6.1f scrub starts 2024-09-13T13:07:15.547 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:15 smithi052 ceph-mon[36682]: 6.1f scrub ok 2024-09-13T13:07:15.565 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:15.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:15 smithi167 ceph-mon[33178]: pgmap v294: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 118 MiB/s wr, 1.88k op/s 2024-09-13T13:07:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:15 smithi167 ceph-mon[33178]: from='client.25312 v1:172.21.15.52:0/3912519509' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:07:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:15 smithi167 ceph-mon[33178]: 6.1f scrub starts 2024-09-13T13:07:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:15 smithi167 ceph-mon[33178]: 6.1f scrub ok 2024-09-13T13:07:15.667 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:15.703 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:15.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:15.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 44 16 77009 76993 109.333 80.875 0.00533888 0.00914186 2024-09-13T13:07:15.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:15.851 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:15.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:15.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:15.953 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:15.992 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:15.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:15.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:16.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:16.127 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:16.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:16.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:16.228 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:16.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:16.272 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:16.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:16.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:16 smithi052 ceph-mon[29471]: 4.f scrub starts 2024-09-13T13:07:16.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:16 smithi052 ceph-mon[29471]: 4.f scrub ok 2024-09-13T13:07:16.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:16 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:07:16.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:16 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:07:16.374 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:07:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:07:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:07:16.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:16 smithi052 ceph-mon[36682]: 4.f scrub starts 2024-09-13T13:07:16.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:16 smithi052 ceph-mon[36682]: 4.f scrub ok 2024-09-13T13:07:16.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:16 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:07:16.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:16 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:07:16.374 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:16.432 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:16.434 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-13T13:07:13.712980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4822276635,"num_objects":73772,"num_object_clones":0,"num_object_copies":221316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73772,"num_whiteouts":0,"num_read":1211,"num_read_kb":1068,"num_write":74122,"num_write_kb":4710345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74133,"ondisk_log_size":74133,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15503040,"kb_used_data":14006204,"kb_used_omap":15,"kb_used_meta":1496688,"kb_avail":734458176,"statfs":{"total":767960285184,"available":752085172224,"internally_reserved":0,"allocated":14342352896,"data_stored":14337079929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":1532609305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":807731200,"num_objects":12325,"num_object_clones":0,"num_object_copies":36975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12325,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12325,"num_write_kb":788800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.545557"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.200434+0000","last_change":"2024-09-13T13:06:44.304525+0000","last_active":"2024-09-13T13:07:09.200434+0000","last_peered":"2024-09-13T13:07:09.200434+0000","last_clean":"2024-09-13T13:07:09.200434+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:07:09.200434+0000","last_undegraded":"2024-09-13T13:07:09.200434+0000","last_fullsized":"2024-09-13T13:07:09.200434+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:44.304444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.304444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:59.584889+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":309,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.183957+0000","last_change":"2024-09-13T13:04:48.899982+0000","last_active":"2024-09-13T13:07:09.183957+0000","last_peered":"2024-09-13T13:07:09.183957+0000","last_clean":"2024-09-13T13:07:09.183957+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:07:09.183957+0000","last_undegraded":"2024-09-13T13:07:09.183957+0000","last_fullsized":"2024-09-13T13:07:09.183957+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:48.899915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.899915+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:03.931197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189642+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:07:09.189642+0000","last_peered":"2024-09-13T13:07:09.189642+0000","last_clean":"2024-09-13T13:07:09.189642+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:07:09.189642+0000","last_undegraded":"2024-09-13T13:07:09.189642+0000","last_fullsized":"2024-09-13T13:07:09.189642+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:06.133166+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":274,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185342+0000","last_change":"2024-09-13T13:05:01.076346+0000","last_active":"2024-09-13T13:07:09.185342+0000","last_peered":"2024-09-13T13:07:09.185342+0000","last_clean":"2024-09-13T13:07:09.185342+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:07:09.185342+0000","last_undegraded":"2024-09-13T13:07:09.185342+0000","last_fullsized":"2024-09-13T13:07:09.185342+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.076275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.076275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:30.081244+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.195272+0000","last_change":"2024-09-13T13:05:45.805098+0000","last_active":"2024-09-13T13:07:09.195272+0000","last_peered":"2024-09-13T13:07:09.195272+0000","last_clean":"2024-09-13T13:07:09.195272+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:07:09.195272+0000","last_undegraded":"2024-09-13T13:07:09.195272+0000","last_fullsized":"2024-09-13T13:07:09.195272+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:45.800516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.800516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.201752+0000","last_change":"2024-09-13T13:04:02.757267+0000","last_active":"2024-09-13T13:07:09.201752+0000","last_peered":"2024-09-13T13:07:09.201752+0000","last_clean":"2024-09-13T13:07:09.201752+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:07:09.201752+0000","last_undegraded":"2024-09-13T13:07:09.201752+0000","last_fullsized":"2024-09-13T13:07:09.201752+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.233259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.233259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:23.678677+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":322,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.549946+0000","last_change":"2024-09-13T13:07:09.549946+0000","last_active":"2024-09-13T13:07:09.549946+0000","last_peered":"2024-09-13T13:07:09.549946+0000","last_clean":"2024-09-13T13:07:09.549946+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:07:09.549946+0000","last_undegraded":"2024-09-13T13:07:09.549946+0000","last_fullsized":"2024-09-13T13:07:09.549946+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:07:09.549857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:09.549857+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:27.521844+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":289,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.183849+0000","last_change":"2024-09-13T13:04:02.765671+0000","last_active":"2024-09-13T13:07:09.183849+0000","last_peered":"2024-09-13T13:07:09.183849+0000","last_clean":"2024-09-13T13:07:09.183849+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:07:09.183849+0000","last_undegraded":"2024-09-13T13:07:09.183849+0000","last_fullsized":"2024-09-13T13:07:09.183849+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.021662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.021662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:22.118745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":310,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188427+0000","last_change":"2024-09-13T13:06:10.595426+0000","last_active":"2024-09-13T13:07:09.188427+0000","last_peered":"2024-09-13T13:07:09.188427+0000","last_clean":"2024-09-13T13:07:09.188427+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:07:09.188427+0000","last_undegraded":"2024-09-13T13:07:09.188427+0000","last_fullsized":"2024-09-13T13:07:09.188427+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:06:10.595347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.595347+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:21.918077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":306,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185622+0000","last_change":"2024-09-13T13:06:02.262489+0000","last_active":"2024-09-13T13:07:09.185622+0000","last_peered":"2024-09-13T13:07:09.185622+0000","last_clean":"2024-09-13T13:07:09.185622+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:07:09.185622+0000","last_undegraded":"2024-09-13T13:07:09.185622+0000","last_fullsized":"2024-09-13T13:07:09.185622+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:06:02.262393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:02.262393+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:17.465680+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.190619+0000","last_change":"2024-09-13T13:06:50.665256+0000","last_active":"2024-09-13T13:07:09.190619+0000","last_peered":"2024-09-13T13:07:09.190619+0000","last_clean":"2024-09-13T13:07:09.190619+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:07:09.190619+0000","last_undegraded":"2024-09-13T13:07:09.190619+0000","last_fullsized":"2024-09-13T13:07:09.190619+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:50.665191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.665191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:53.974779+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.198819+0000","last_change":"2024-09-13T13:02:39.607712+0000","last_active":"2024-09-13T13:07:09.198819+0000","last_peered":"2024-09-13T13:07:09.198819+0000","last_clean":"2024-09-13T13:07:09.198819+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:07:09.198819+0000","last_undegraded":"2024-09-13T13:07:09.198819+0000","last_fullsized":"2024-09-13T13:07:09.198819+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.607646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.607646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:21.338399+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":301,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189111+0000","last_change":"2024-09-13T13:05:38.520071+0000","last_active":"2024-09-13T13:07:09.189111+0000","last_peered":"2024-09-13T13:07:09.189111+0000","last_clean":"2024-09-13T13:07:09.189111+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:07:09.189111+0000","last_undegraded":"2024-09-13T13:07:09.189111+0000","last_fullsized":"2024-09-13T13:07:09.189111+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:38.520006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:38.520006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:55.622971+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":300,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184360+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:07:09.184360+0000","last_peered":"2024-09-13T13:07:09.184360+0000","last_clean":"2024-09-13T13:07:09.184360+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:07:09.184360+0000","last_undegraded":"2024-09-13T13:07:09.184360+0000","last_fullsized":"2024-09-13T13:07:09.184360+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:32.042125+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.203258+0000","last_change":"2024-09-13T13:06:45.544015+0000","last_active":"2024-09-13T13:07:09.203258+0000","last_peered":"2024-09-13T13:07:09.203258+0000","last_clean":"2024-09-13T13:07:09.203258+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:07:09.203258+0000","last_undegraded":"2024-09-13T13:07:09.203258+0000","last_fullsized":"2024-09-13T13:07:09.203258+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:45.543961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:45.543961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:50.020558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":265,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188183+0000","last_change":"2024-09-13T13:01:08.213608+0000","last_active":"2024-09-13T13:07:09.188183+0000","last_peered":"2024-09-13T13:07:09.188183+0000","last_clean":"2024-09-13T13:07:09.188183+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:07:09.188183+0000","last_undegraded":"2024-09-13T13:07:09.188183+0000","last_fullsized":"2024-09-13T13:07:09.188183+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:17.609112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":295,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.186013+0000","last_change":"2024-09-13T13:06:37.129744+0000","last_active":"2024-09-13T13:07:09.186013+0000","last_peered":"2024-09-13T13:07:09.186013+0000","last_clean":"2024-09-13T13:07:09.186013+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:07:09.186013+0000","last_undegraded":"2024-09-13T13:07:09.186013+0000","last_fullsized":"2024-09-13T13:07:09.186013+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:06:37.129698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:37.129698+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:46.413310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199533+0000","last_change":"2024-09-13T13:06:38.260486+0000","last_active":"2024-09-13T13:07:09.199533+0000","last_peered":"2024-09-13T13:07:09.199533+0000","last_clean":"2024-09-13T13:07:09.199533+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:07:09.199533+0000","last_undegraded":"2024-09-13T13:07:09.199533+0000","last_fullsized":"2024-09-13T13:07:09.199533+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:38.259822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:38.259822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:06.534345+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":331,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.187546+0000","last_change":"2024-09-13T13:06:44.532305+0000","last_active":"2024-09-13T13:07:09.187546+0000","last_peered":"2024-09-13T13:07:09.187546+0000","last_clean":"2024-09-13T13:07:09.187546+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:07:09.187546+0000","last_undegraded":"2024-09-13T13:07:09.187546+0000","last_fullsized":"2024-09-13T13:07:09.187546+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:06:44.532186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.532186+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:01.016677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.193534+0000","last_change":"2024-09-13T13:06:47.977011+0000","last_active":"2024-09-13T13:07:09.193534+0000","last_peered":"2024-09-13T13:07:09.193534+0000","last_clean":"2024-09-13T13:07:09.193534+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:07:09.193534+0000","last_undegraded":"2024-09-13T13:07:09.193534+0000","last_fullsized":"2024-09-13T13:07:09.193534+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:47.976922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:47.976922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:17.524630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196346+0000","last_change":"2024-09-13T13:04:03.384955+0000","last_active":"2024-09-13T13:07:09.196346+0000","last_peered":"2024-09-13T13:07:09.196346+0000","last_clean":"2024-09-13T13:07:09.196346+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:07:09.196346+0000","last_undegraded":"2024-09-13T13:07:09.196346+0000","last_fullsized":"2024-09-13T13:07:09.196346+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.384901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.384901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185767+0000","last_change":"2024-09-13T13:04:57.103213+0000","last_active":"2024-09-13T13:07:09.185767+0000","last_peered":"2024-09-13T13:07:09.185767+0000","last_clean":"2024-09-13T13:07:09.185767+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:07:09.185767+0000","last_undegraded":"2024-09-13T13:07:09.185767+0000","last_fullsized":"2024-09-13T13:07:09.185767+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.103167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.103167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:53.999219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":304,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.193409+0000","last_change":"2024-09-13T13:06:05.706641+0000","last_active":"2024-09-13T13:07:09.193409+0000","last_peered":"2024-09-13T13:07:09.193409+0000","last_clean":"2024-09-13T13:07:09.193409+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:07:09.193409+0000","last_undegraded":"2024-09-13T13:07:09.193409+0000","last_fullsized":"2024-09-13T13:07:09.193409+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:06:05.706569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:05.706569+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:09.818528+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":294,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.879813+0000","last_change":"2024-09-13T13:07:12.879813+0000","last_active":"2024-09-13T13:07:12.879813+0000","last_peered":"2024-09-13T13:07:12.879813+0000","last_clean":"2024-09-13T13:07:12.879813+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:07:12.879813+0000","last_undegraded":"2024-09-13T13:07:12.879813+0000","last_fullsized":"2024-09-13T13:07:12.879813+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.879727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.879727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:16.423585+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.206440+0000","last_change":"2024-09-13T13:06:38.536219+0000","last_active":"2024-09-13T13:07:09.206440+0000","last_peered":"2024-09-13T13:07:09.206440+0000","last_clean":"2024-09-13T13:07:09.206440+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:07:09.206440+0000","last_undegraded":"2024-09-13T13:07:09.206440+0000","last_fullsized":"2024-09-13T13:07:09.206440+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:06:38.536170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:38.536170+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:39.956585+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":287,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189769+0000","last_change":"2024-09-13T13:03:12.276224+0000","last_active":"2024-09-13T13:07:09.189769+0000","last_peered":"2024-09-13T13:07:09.189769+0000","last_clean":"2024-09-13T13:07:09.189769+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:07:09.189769+0000","last_undegraded":"2024-09-13T13:07:09.189769+0000","last_fullsized":"2024-09-13T13:07:09.189769+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:12.276136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.276136+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:52.950269+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":305,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199724+0000","last_change":"2024-09-13T13:06:40.292804+0000","last_active":"2024-09-13T13:07:09.199724+0000","last_peered":"2024-09-13T13:07:09.199724+0000","last_clean":"2024-09-13T13:07:09.199724+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:07:09.199724+0000","last_undegraded":"2024-09-13T13:07:09.199724+0000","last_fullsized":"2024-09-13T13:07:09.199724+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:06:40.292551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:40.292551+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:01.413088+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":317,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.192632+0000","last_change":"2024-09-13T13:06:04.949976+0000","last_active":"2024-09-13T13:07:09.192632+0000","last_peered":"2024-09-13T13:07:09.192632+0000","last_clean":"2024-09-13T13:07:09.192632+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:07:09.192632+0000","last_undegraded":"2024-09-13T13:07:09.192632+0000","last_fullsized":"2024-09-13T13:07:09.192632+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:06:04.949930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:04.949930+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:33.315989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"192'4689","reported_seq":4746,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.124825+0000","last_change":"2024-09-13T13:06:21.814268+0000","last_active":"2024-09-13T13:07:13.124825+0000","last_peered":"2024-09-13T13:07:13.124825+0000","last_clean":"2024-09-13T13:07:13.124825+0000","last_became_active":"2024-09-13T13:06:21.814002+0000","last_became_peered":"2024-09-13T13:06:21.814002+0000","last_unstale":"2024-09-13T13:07:13.124825+0000","last_undegraded":"2024-09-13T13:07:13.124825+0000","last_fullsized":"2024-09-13T13:07:13.124825+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4689,"log_dups_size":0,"ondisk_log_size":4689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:31.181397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307298304,"num_objects":4689,"num_object_clones":0,"num_object_copies":14067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4689,"num_write_kb":300096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":315,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199241+0000","last_change":"2024-09-13T13:05:25.691233+0000","last_active":"2024-09-13T13:07:09.199241+0000","last_peered":"2024-09-13T13:07:09.199241+0000","last_clean":"2024-09-13T13:07:09.199241+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:07:09.199241+0000","last_undegraded":"2024-09-13T13:07:09.199241+0000","last_fullsized":"2024-09-13T13:07:09.199241+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:05:25.689487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:25.689487+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":301,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.197555+0000","last_change":"2024-09-13T13:07:08.562915+0000","last_active":"2024-09-13T13:07:09.197555+0000","last_peered":"2024-09-13T13:07:09.197555+0000","last_clean":"2024-09-13T13:07:09.197555+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:07:09.197555+0000","last_undegraded":"2024-09-13T13:07:09.197555+0000","last_fullsized":"2024-09-13T13:07:09.197555+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:08.562240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.562240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:17.435791+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":361,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.186104+0000","last_change":"2024-09-13T13:02:32.013587+0000","last_active":"2024-09-13T13:07:09.186104+0000","last_peered":"2024-09-13T13:07:09.186104+0000","last_clean":"2024-09-13T13:07:09.186104+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:07:09.186104+0000","last_undegraded":"2024-09-13T13:07:09.186104+0000","last_fullsized":"2024-09-13T13:07:09.186104+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:32.013505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.013505+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:16.072920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":281,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188176+0000","last_change":"2024-09-13T13:06:05.648648+0000","last_active":"2024-09-13T13:07:09.188176+0000","last_peered":"2024-09-13T13:07:09.188176+0000","last_clean":"2024-09-13T13:07:09.188176+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:07:09.188176+0000","last_undegraded":"2024-09-13T13:07:09.188176+0000","last_fullsized":"2024-09-13T13:07:09.188176+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:05.648589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:05.648589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.628097+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"192'4645","reported_seq":4702,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.638196+0000","last_change":"2024-09-13T13:06:22.000536+0000","last_active":"2024-09-13T13:07:13.638196+0000","last_peered":"2024-09-13T13:07:13.638196+0000","last_clean":"2024-09-13T13:07:13.638196+0000","last_became_active":"2024-09-13T13:06:22.000244+0000","last_became_peered":"2024-09-13T13:06:22.000244+0000","last_unstale":"2024-09-13T13:07:13.638196+0000","last_undegraded":"2024-09-13T13:07:13.638196+0000","last_fullsized":"2024-09-13T13:07:13.638196+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:40.851139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304414720,"num_objects":4645,"num_object_clones":0,"num_object_copies":13935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4645,"num_write_kb":297280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":314,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.197335+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:07:09.197335+0000","last_peered":"2024-09-13T13:07:09.197335+0000","last_clean":"2024-09-13T13:07:09.197335+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:07:09.197335+0000","last_undegraded":"2024-09-13T13:07:09.197335+0000","last_fullsized":"2024-09-13T13:07:09.197335+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:05.558407+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":299,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196790+0000","last_change":"2024-09-13T13:06:44.785235+0000","last_active":"2024-09-13T13:07:09.196790+0000","last_peered":"2024-09-13T13:07:09.196790+0000","last_clean":"2024-09-13T13:07:09.196790+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:07:09.196790+0000","last_undegraded":"2024-09-13T13:07:09.196790+0000","last_fullsized":"2024-09-13T13:07:09.196790+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:06:44.785017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.785017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:00.668665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184932+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:07:09.184932+0000","last_peered":"2024-09-13T13:07:09.184932+0000","last_clean":"2024-09-13T13:07:09.184932+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:07:09.184932+0000","last_undegraded":"2024-09-13T13:07:09.184932+0000","last_fullsized":"2024-09-13T13:07:09.184932+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:04.883415+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185531+0000","last_change":"2024-09-13T13:06:43.138526+0000","last_active":"2024-09-13T13:07:09.185531+0000","last_peered":"2024-09-13T13:07:09.185531+0000","last_clean":"2024-09-13T13:07:09.185531+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:07:09.185531+0000","last_undegraded":"2024-09-13T13:07:09.185531+0000","last_fullsized":"2024-09-13T13:07:09.185531+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.138328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.138328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:54.605867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"192'4562","reported_seq":4619,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.940096+0000","last_change":"2024-09-13T13:06:22.000928+0000","last_active":"2024-09-13T13:07:12.940096+0000","last_peered":"2024-09-13T13:07:12.940096+0000","last_clean":"2024-09-13T13:07:12.940096+0000","last_became_active":"2024-09-13T13:06:22.000610+0000","last_became_peered":"2024-09-13T13:06:22.000610+0000","last_unstale":"2024-09-13T13:07:12.940096+0000","last_undegraded":"2024-09-13T13:07:12.940096+0000","last_fullsized":"2024-09-13T13:07:12.940096+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4562,"log_dups_size":0,"ondisk_log_size":4562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:47.936354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298975232,"num_objects":4562,"num_object_clones":0,"num_object_copies":13686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4562,"num_write_kb":291968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":330,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:10.853834+0000","last_change":"2024-09-13T13:07:10.853834+0000","last_active":"2024-09-13T13:07:10.853834+0000","last_peered":"2024-09-13T13:07:10.853834+0000","last_clean":"2024-09-13T13:07:10.853834+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:07:10.853834+0000","last_undegraded":"2024-09-13T13:07:10.853834+0000","last_fullsized":"2024-09-13T13:07:10.853834+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:07:10.853749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.853749+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:26.687762+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199944+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:07:09.199944+0000","last_peered":"2024-09-13T13:07:09.199944+0000","last_clean":"2024-09-13T13:07:09.199944+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:07:09.199944+0000","last_undegraded":"2024-09-13T13:07:09.199944+0000","last_fullsized":"2024-09-13T13:07:09.199944+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:06.208212+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":361,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.310660+0000","last_change":"2024-09-13T13:06:48.866830+0000","last_active":"2024-09-13T13:07:12.310660+0000","last_peered":"2024-09-13T13:07:12.310660+0000","last_clean":"2024-09-13T13:07:12.310660+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:07:12.310660+0000","last_undegraded":"2024-09-13T13:07:12.310660+0000","last_fullsized":"2024-09-13T13:07:12.310660+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:06:48.866752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:48.866752+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:03.286427+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":292,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.203497+0000","last_change":"2024-09-13T13:06:09.230872+0000","last_active":"2024-09-13T13:07:09.203497+0000","last_peered":"2024-09-13T13:07:09.203497+0000","last_clean":"2024-09-13T13:07:09.203497+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:07:09.203497+0000","last_undegraded":"2024-09-13T13:07:09.203497+0000","last_fullsized":"2024-09-13T13:07:09.203497+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:09.230824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.230824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:19.719773+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"192'4628","reported_seq":4685,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.644755+0000","last_change":"2024-09-13T13:06:21.812511+0000","last_active":"2024-09-13T13:07:13.644755+0000","last_peered":"2024-09-13T13:07:13.644755+0000","last_clean":"2024-09-13T13:07:13.644755+0000","last_became_active":"2024-09-13T13:06:21.812226+0000","last_became_peered":"2024-09-13T13:06:21.812226+0000","last_unstale":"2024-09-13T13:07:13.644755+0000","last_undegraded":"2024-09-13T13:07:13.644755+0000","last_fullsized":"2024-09-13T13:07:13.644755+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:27.069143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303300608,"num_objects":4628,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4628,"num_write_kb":296192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.191532+0000","last_change":"2024-09-13T13:06:36.894642+0000","last_active":"2024-09-13T13:07:09.191532+0000","last_peered":"2024-09-13T13:07:09.191532+0000","last_clean":"2024-09-13T13:07:09.191532+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:07:09.191532+0000","last_undegraded":"2024-09-13T13:07:09.191532+0000","last_fullsized":"2024-09-13T13:07:09.191532+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:06:36.894565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:36.894565+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:53.101599+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188503+0000","last_change":"2024-09-13T13:06:36.645157+0000","last_active":"2024-09-13T13:07:09.188503+0000","last_peered":"2024-09-13T13:07:09.188503+0000","last_clean":"2024-09-13T13:07:09.188503+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:07:09.188503+0000","last_undegraded":"2024-09-13T13:07:09.188503+0000","last_fullsized":"2024-09-13T13:07:09.188503+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:36.645089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:36.645089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:55.978419+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":366,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199209+0000","last_change":"2024-09-13T13:04:02.764350+0000","last_active":"2024-09-13T13:07:09.199209+0000","last_peered":"2024-09-13T13:07:09.199209+0000","last_clean":"2024-09-13T13:07:09.199209+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:07:09.199209+0000","last_undegraded":"2024-09-13T13:07:09.199209+0000","last_fullsized":"2024-09-13T13:07:09.199209+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:52.299393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.200643+0000","last_change":"2024-09-13T13:06:46.288561+0000","last_active":"2024-09-13T13:07:09.200643+0000","last_peered":"2024-09-13T13:07:09.200643+0000","last_clean":"2024-09-13T13:07:09.200643+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:07:09.200643+0000","last_undegraded":"2024-09-13T13:07:09.200643+0000","last_fullsized":"2024-09-13T13:07:09.200643+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:46.288483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:46.288483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:47.729332+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"192'4661","reported_seq":4718,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.302999+0000","last_change":"2024-09-13T13:06:21.812509+0000","last_active":"2024-09-13T13:07:13.302999+0000","last_peered":"2024-09-13T13:07:13.302999+0000","last_clean":"2024-09-13T13:07:13.302999+0000","last_became_active":"2024-09-13T13:06:21.812208+0000","last_became_peered":"2024-09-13T13:06:21.812208+0000","last_unstale":"2024-09-13T13:07:13.302999+0000","last_undegraded":"2024-09-13T13:07:13.302999+0000","last_fullsized":"2024-09-13T13:07:13.302999+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4661,"log_dups_size":0,"ondisk_log_size":4661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:33.753674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305463296,"num_objects":4661,"num_object_clones":0,"num_object_copies":13983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4661,"num_write_kb":298304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":325,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199356+0000","last_change":"2024-09-13T13:06:37.265029+0000","last_active":"2024-09-13T13:07:09.199356+0000","last_peered":"2024-09-13T13:07:09.199356+0000","last_clean":"2024-09-13T13:07:09.199356+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:07:09.199356+0000","last_undegraded":"2024-09-13T13:07:09.199356+0000","last_fullsized":"2024-09-13T13:07:09.199356+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:06:37.264946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:37.264946+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:47.148663+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":278,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.194302+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:07:09.194302+0000","last_peered":"2024-09-13T13:07:09.194302+0000","last_clean":"2024-09-13T13:07:09.194302+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:07:09.194302+0000","last_undegraded":"2024-09-13T13:07:09.194302+0000","last_fullsized":"2024-09-13T13:07:09.194302+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:33.900862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":247,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.190891+0000","last_change":"2024-09-13T13:06:08.941867+0000","last_active":"2024-09-13T13:07:09.190891+0000","last_peered":"2024-09-13T13:07:09.190891+0000","last_clean":"2024-09-13T13:07:09.190891+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:07:09.190891+0000","last_undegraded":"2024-09-13T13:07:09.190891+0000","last_fullsized":"2024-09-13T13:07:09.190891+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:08.941799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:08.941799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:32.827042+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":301,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189561+0000","last_change":"2024-09-13T13:06:43.969251+0000","last_active":"2024-09-13T13:07:09.189561+0000","last_peered":"2024-09-13T13:07:09.189561+0000","last_clean":"2024-09-13T13:07:09.189561+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:07:09.189561+0000","last_undegraded":"2024-09-13T13:07:09.189561+0000","last_fullsized":"2024-09-13T13:07:09.189561+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.969176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.969176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:00.184869+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"192'4505","reported_seq":4562,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.931963+0000","last_change":"2024-09-13T13:06:22.000658+0000","last_active":"2024-09-13T13:07:12.931963+0000","last_peered":"2024-09-13T13:07:12.931963+0000","last_clean":"2024-09-13T13:07:12.931963+0000","last_became_active":"2024-09-13T13:06:22.000261+0000","last_became_peered":"2024-09-13T13:06:22.000261+0000","last_unstale":"2024-09-13T13:07:12.931963+0000","last_undegraded":"2024-09-13T13:07:12.931963+0000","last_fullsized":"2024-09-13T13:07:12.931963+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:41.824631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295239680,"num_objects":4505,"num_object_clones":0,"num_object_copies":13515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4505,"num_write_kb":288320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":413,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.194207+0000","last_change":"2024-09-13T13:05:43.787063+0000","last_active":"2024-09-13T13:07:09.194207+0000","last_peered":"2024-09-13T13:07:09.194207+0000","last_clean":"2024-09-13T13:07:09.194207+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:07:09.194207+0000","last_undegraded":"2024-09-13T13:07:09.194207+0000","last_fullsized":"2024-09-13T13:07:09.194207+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:05:43.786273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:43.786273+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":305,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184731+0000","last_change":"2024-09-13T13:05:45.178086+0000","last_active":"2024-09-13T13:07:09.184731+0000","last_peered":"2024-09-13T13:07:09.184731+0000","last_clean":"2024-09-13T13:07:09.184731+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:07:09.184731+0000","last_undegraded":"2024-09-13T13:07:09.184731+0000","last_fullsized":"2024-09-13T13:07:09.184731+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:45.178025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.178025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":304,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188893+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:07:09.188893+0000","last_peered":"2024-09-13T13:07:09.188893+0000","last_clean":"2024-09-13T13:07:09.188893+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:07:09.188893+0000","last_undegraded":"2024-09-13T13:07:09.188893+0000","last_fullsized":"2024-09-13T13:07:09.188893+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:27.623207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199725+0000","last_change":"2024-09-13T13:06:11.189922+0000","last_active":"2024-09-13T13:07:09.199725+0000","last_peered":"2024-09-13T13:07:09.199725+0000","last_clean":"2024-09-13T13:07:09.199725+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:07:09.199725+0000","last_undegraded":"2024-09-13T13:07:09.199725+0000","last_fullsized":"2024-09-13T13:07:09.199725+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:11.189874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:11.189874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:21.236595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":281,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:11.551545+0000","last_change":"2024-09-13T13:07:11.551545+0000","last_active":"2024-09-13T13:07:11.551545+0000","last_peered":"2024-09-13T13:07:11.551545+0000","last_clean":"2024-09-13T13:07:11.551545+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:07:11.551545+0000","last_undegraded":"2024-09-13T13:07:11.551545+0000","last_fullsized":"2024-09-13T13:07:11.551545+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.551460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.551460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:33.164720+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"192'4595","reported_seq":4652,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.276217+0000","last_change":"2024-09-13T13:06:21.813204+0000","last_active":"2024-09-13T13:07:13.276217+0000","last_peered":"2024-09-13T13:07:13.276217+0000","last_clean":"2024-09-13T13:07:13.276217+0000","last_became_active":"2024-09-13T13:06:21.812869+0000","last_became_peered":"2024-09-13T13:06:21.812869+0000","last_unstale":"2024-09-13T13:07:13.276217+0000","last_undegraded":"2024-09-13T13:07:13.276217+0000","last_fullsized":"2024-09-13T13:07:13.276217+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4595,"log_dups_size":0,"ondisk_log_size":4595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:33.863904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301137920,"num_objects":4595,"num_object_clones":0,"num_object_copies":13785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4595,"num_write_kb":294080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":309,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199342+0000","last_change":"2024-09-13T13:03:14.271569+0000","last_active":"2024-09-13T13:07:09.199342+0000","last_peered":"2024-09-13T13:07:09.199342+0000","last_clean":"2024-09-13T13:07:09.199342+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:07:09.199342+0000","last_undegraded":"2024-09-13T13:07:09.199342+0000","last_fullsized":"2024-09-13T13:07:09.199342+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:03:14.271480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.271480+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:59.122679+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":306,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.192349+0000","last_change":"2024-09-13T13:05:36.692159+0000","last_active":"2024-09-13T13:07:09.192349+0000","last_peered":"2024-09-13T13:07:09.192349+0000","last_clean":"2024-09-13T13:07:09.192349+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:07:09.192349+0000","last_undegraded":"2024-09-13T13:07:09.192349+0000","last_fullsized":"2024-09-13T13:07:09.192349+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:36.692092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:36.692092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":309,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.186216+0000","last_change":"2024-09-13T13:06:50.629016+0000","last_active":"2024-09-13T13:07:09.186216+0000","last_peered":"2024-09-13T13:07:09.186216+0000","last_clean":"2024-09-13T13:07:09.186216+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:07:09.186216+0000","last_undegraded":"2024-09-13T13:07:09.186216+0000","last_fullsized":"2024-09-13T13:07:09.186216+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:06:50.628774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.628774+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:14.442004+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":278,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.192649+0000","last_change":"2024-09-13T13:03:14.282866+0000","last_active":"2024-09-13T13:07:09.192649+0000","last_peered":"2024-09-13T13:07:09.192649+0000","last_clean":"2024-09-13T13:07:09.192649+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:07:09.192649+0000","last_undegraded":"2024-09-13T13:07:09.192649+0000","last_fullsized":"2024-09-13T13:07:09.192649+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:27.402491+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.204674+0000","last_change":"2024-09-13T13:04:02.751582+0000","last_active":"2024-09-13T13:07:09.204674+0000","last_peered":"2024-09-13T13:07:09.204674+0000","last_clean":"2024-09-13T13:07:09.204674+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:07:09.204674+0000","last_undegraded":"2024-09-13T13:07:09.204674+0000","last_fullsized":"2024-09-13T13:07:09.204674+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.994729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.994729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:19.352065+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"192'4593","reported_seq":4650,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.146022+0000","last_change":"2024-09-13T13:06:21.815090+0000","last_active":"2024-09-13T13:07:13.146022+0000","last_peered":"2024-09-13T13:07:13.146022+0000","last_clean":"2024-09-13T13:07:13.146022+0000","last_became_active":"2024-09-13T13:06:21.814772+0000","last_became_peered":"2024-09-13T13:06:21.814772+0000","last_unstale":"2024-09-13T13:07:13.146022+0000","last_undegraded":"2024-09-13T13:07:13.146022+0000","last_fullsized":"2024-09-13T13:07:13.146022+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:41.464053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301006848,"num_objects":4593,"num_object_clones":0,"num_object_copies":13779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":293952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":307,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:11.169884+0000","last_change":"2024-09-13T13:07:11.169884+0000","last_active":"2024-09-13T13:07:11.169884+0000","last_peered":"2024-09-13T13:07:11.169884+0000","last_clean":"2024-09-13T13:07:11.169884+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:07:11.169884+0000","last_undegraded":"2024-09-13T13:07:11.169884+0000","last_fullsized":"2024-09-13T13:07:11.169884+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:07:11.169656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.169656+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:17.059187+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199073+0000","last_change":"2024-09-13T13:06:42.310845+0000","last_active":"2024-09-13T13:07:09.199073+0000","last_peered":"2024-09-13T13:07:09.199073+0000","last_clean":"2024-09-13T13:07:09.199073+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:07:09.199073+0000","last_undegraded":"2024-09-13T13:07:09.199073+0000","last_fullsized":"2024-09-13T13:07:09.199073+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:42.310776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:42.310776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:55.851649+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":668,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:11.867149+0000","last_change":"2024-09-13T13:06:07.290837+0000","last_active":"2024-09-13T13:07:11.867149+0000","last_peered":"2024-09-13T13:07:11.867149+0000","last_clean":"2024-09-13T13:07:11.867149+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:07:11.867149+0000","last_undegraded":"2024-09-13T13:07:11.867149+0000","last_fullsized":"2024-09-13T13:07:11.867149+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:06:07.290717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:07.290717+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:20.222760+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":365,"num_read_kb":360,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":277,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.190230+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:07:09.190230+0000","last_peered":"2024-09-13T13:07:09.190230+0000","last_clean":"2024-09-13T13:07:09.190230+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:07:09.190230+0000","last_undegraded":"2024-09-13T13:07:09.190230+0000","last_fullsized":"2024-09-13T13:07:09.190230+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:25.654670+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.190664+0000","last_change":"2024-09-13T13:05:41.753575+0000","last_active":"2024-09-13T13:07:09.190664+0000","last_peered":"2024-09-13T13:07:09.190664+0000","last_clean":"2024-09-13T13:07:09.190664+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:07:09.190664+0000","last_undegraded":"2024-09-13T13:07:09.190664+0000","last_fullsized":"2024-09-13T13:07:09.190664+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:41.753502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:41.753502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"192'4399","reported_seq":4456,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:10.900334+0000","last_change":"2024-09-13T13:06:21.814495+0000","last_active":"2024-09-13T13:07:10.900334+0000","last_peered":"2024-09-13T13:07:10.900334+0000","last_clean":"2024-09-13T13:07:10.900334+0000","last_became_active":"2024-09-13T13:06:21.814144+0000","last_became_peered":"2024-09-13T13:06:21.814144+0000","last_unstale":"2024-09-13T13:07:10.900334+0000","last_undegraded":"2024-09-13T13:07:10.900334+0000","last_fullsized":"2024-09-13T13:07:10.900334+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4399,"log_dups_size":0,"ondisk_log_size":4399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:21.059266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288292864,"num_objects":4399,"num_object_clones":0,"num_object_copies":13197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4399,"num_write_kb":281536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":253,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.191497+0000","last_change":"2024-09-13T13:06:51.193749+0000","last_active":"2024-09-13T13:07:09.191497+0000","last_peered":"2024-09-13T13:07:09.191497+0000","last_clean":"2024-09-13T13:07:09.191497+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:07:09.191497+0000","last_undegraded":"2024-09-13T13:07:09.191497+0000","last_fullsized":"2024-09-13T13:07:09.191497+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'18","last_scrub_stamp":"2024-09-13T13:06:51.193678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.193678+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:13.330359+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":304,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.193158+0000","last_change":"2024-09-13T13:06:10.715371+0000","last_active":"2024-09-13T13:07:09.193158+0000","last_peered":"2024-09-13T13:07:09.193158+0000","last_clean":"2024-09-13T13:07:09.193158+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:07:09.193158+0000","last_undegraded":"2024-09-13T13:07:09.193158+0000","last_fullsized":"2024-09-13T13:07:09.193158+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:10.715313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.715313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:21.700969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":350,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.203733+0000","last_change":"2024-09-13T13:06:39.564599+0000","last_active":"2024-09-13T13:07:09.203733+0000","last_peered":"2024-09-13T13:07:09.203733+0000","last_clean":"2024-09-13T13:07:09.203733+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:07:09.203733+0000","last_undegraded":"2024-09-13T13:07:09.203733+0000","last_fullsized":"2024-09-13T13:07:09.203733+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:06:39.564549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:06:39.564549+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:03.134331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.206625+0000","last_change":"2024-09-13T13:06:05.308473+0000","last_active":"2024-09-13T13:07:09.206625+0000","last_peered":"2024-09-13T13:07:09.206625+0000","last_clean":"2024-09-13T13:07:09.206625+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:07:09.206625+0000","last_undegraded":"2024-09-13T13:07:09.206625+0000","last_fullsized":"2024-09-13T13:07:09.206625+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:05.308427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:05.308427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:14.135834+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":289,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189794+0000","last_change":"2024-09-13T13:06:46.624345+0000","last_active":"2024-09-13T13:07:09.189794+0000","last_peered":"2024-09-13T13:07:09.189794+0000","last_clean":"2024-09-13T13:07:09.189794+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:07:09.189794+0000","last_undegraded":"2024-09-13T13:07:09.189794+0000","last_fullsized":"2024-09-13T13:07:09.189794+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:46.624238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:46.624238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:10.518686+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"192'4576","reported_seq":4633,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.927778+0000","last_change":"2024-09-13T13:06:21.812371+0000","last_active":"2024-09-13T13:07:12.927778+0000","last_peered":"2024-09-13T13:07:12.927778+0000","last_clean":"2024-09-13T13:07:12.927778+0000","last_became_active":"2024-09-13T13:06:21.812061+0000","last_became_peered":"2024-09-13T13:06:21.812061+0000","last_unstale":"2024-09-13T13:07:12.927778+0000","last_undegraded":"2024-09-13T13:07:12.927778+0000","last_fullsized":"2024-09-13T13:07:12.927778+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4576,"log_dups_size":0,"ondisk_log_size":4576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:23.624139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299892736,"num_objects":4576,"num_object_clones":0,"num_object_copies":13728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4576,"num_write_kb":292864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":312,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.186149+0000","last_change":"2024-09-13T13:04:02.832992+0000","last_active":"2024-09-13T13:07:09.186149+0000","last_peered":"2024-09-13T13:07:09.186149+0000","last_clean":"2024-09-13T13:07:09.186149+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:07:09.186149+0000","last_undegraded":"2024-09-13T13:07:09.186149+0000","last_fullsized":"2024-09-13T13:07:09.186149+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:04:02.832935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.832935+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:52.072763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185797+0000","last_change":"2024-09-13T13:02:45.015634+0000","last_active":"2024-09-13T13:07:09.185797+0000","last_peered":"2024-09-13T13:07:09.185797+0000","last_clean":"2024-09-13T13:07:09.185797+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:07:09.185797+0000","last_undegraded":"2024-09-13T13:07:09.185797+0000","last_fullsized":"2024-09-13T13:07:09.185797+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.015577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.015577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:19.083414+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":296,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189625+0000","last_change":"2024-09-13T13:06:11.637531+0000","last_active":"2024-09-13T13:07:09.189625+0000","last_peered":"2024-09-13T13:07:09.189625+0000","last_clean":"2024-09-13T13:07:09.189625+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:07:09.189625+0000","last_undegraded":"2024-09-13T13:07:09.189625+0000","last_fullsized":"2024-09-13T13:07:09.189625+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:11.637478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:11.637478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:12.457876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196150+0000","last_change":"2024-09-13T13:04:02.757378+0000","last_active":"2024-09-13T13:07:09.196150+0000","last_peered":"2024-09-13T13:07:09.196150+0000","last_clean":"2024-09-13T13:07:09.196150+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:07:09.196150+0000","last_undegraded":"2024-09-13T13:07:09.196150+0000","last_fullsized":"2024-09-13T13:07:09.196150+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.213470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.213470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:22.648573+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"192'4657","reported_seq":4714,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.153458+0000","last_change":"2024-09-13T13:06:21.811878+0000","last_active":"2024-09-13T13:07:13.153458+0000","last_peered":"2024-09-13T13:07:13.153458+0000","last_clean":"2024-09-13T13:07:13.153458+0000","last_became_active":"2024-09-13T13:06:21.811526+0000","last_became_peered":"2024-09-13T13:06:21.811526+0000","last_unstale":"2024-09-13T13:07:13.153458+0000","last_undegraded":"2024-09-13T13:07:13.153458+0000","last_fullsized":"2024-09-13T13:07:13.153458+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4657,"log_dups_size":0,"ondisk_log_size":4657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:41.893735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305201152,"num_objects":4657,"num_object_clones":0,"num_object_copies":13971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4657,"num_write_kb":298048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":318,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184395+0000","last_change":"2024-09-13T13:05:34.166464+0000","last_active":"2024-09-13T13:07:09.184395+0000","last_peered":"2024-09-13T13:07:09.184395+0000","last_clean":"2024-09-13T13:07:09.184395+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:07:09.184395+0000","last_undegraded":"2024-09-13T13:07:09.184395+0000","last_fullsized":"2024-09-13T13:07:09.184395+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:00.016282+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":263,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.191150+0000","last_change":"2024-09-13T13:05:44.119947+0000","last_active":"2024-09-13T13:07:09.191150+0000","last_peered":"2024-09-13T13:07:09.191150+0000","last_clean":"2024-09-13T13:07:09.191150+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:07:09.191150+0000","last_undegraded":"2024-09-13T13:07:09.191150+0000","last_fullsized":"2024-09-13T13:07:09.191150+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:44.119890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.119890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:53.147402+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":278,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.191077+0000","last_change":"2024-09-13T13:06:37.854805+0000","last_active":"2024-09-13T13:07:09.191077+0000","last_peered":"2024-09-13T13:07:09.191077+0000","last_clean":"2024-09-13T13:07:09.191077+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:07:09.191077+0000","last_undegraded":"2024-09-13T13:07:09.191077+0000","last_fullsized":"2024-09-13T13:07:09.191077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:37.854730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:37.854730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:40.674082+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188758+0000","last_change":"2024-09-13T13:05:35.310768+0000","last_active":"2024-09-13T13:07:09.188758+0000","last_peered":"2024-09-13T13:07:09.188758+0000","last_clean":"2024-09-13T13:07:09.188758+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:07:09.188758+0000","last_undegraded":"2024-09-13T13:07:09.188758+0000","last_fullsized":"2024-09-13T13:07:09.188758+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:35.310700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.310700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:02.424290+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"192'4554","reported_seq":4610,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.319449+0000","last_change":"2024-09-13T13:06:21.813258+0000","last_active":"2024-09-13T13:07:13.319449+0000","last_peered":"2024-09-13T13:07:13.319449+0000","last_clean":"2024-09-13T13:07:13.319449+0000","last_became_active":"2024-09-13T13:06:21.812966+0000","last_became_peered":"2024-09-13T13:06:21.812966+0000","last_unstale":"2024-09-13T13:07:13.319449+0000","last_undegraded":"2024-09-13T13:07:13.319449+0000","last_fullsized":"2024-09-13T13:07:13.319449+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4554,"log_dups_size":0,"ondisk_log_size":4554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:49.824314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298450944,"num_objects":4554,"num_object_clones":0,"num_object_copies":13662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4554,"num_write_kb":291456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":316,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.183974+0000","last_change":"2024-09-13T13:06:40.132360+0000","last_active":"2024-09-13T13:07:09.183974+0000","last_peered":"2024-09-13T13:07:09.183974+0000","last_clean":"2024-09-13T13:07:09.183974+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:07:09.183974+0000","last_undegraded":"2024-09-13T13:07:09.183974+0000","last_fullsized":"2024-09-13T13:07:09.183974+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:06:40.132273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:40.132273+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:49.361496+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.195626+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:07:09.195626+0000","last_peered":"2024-09-13T13:07:09.195626+0000","last_clean":"2024-09-13T13:07:09.195626+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:07:09.195626+0000","last_undegraded":"2024-09-13T13:07:09.195626+0000","last_fullsized":"2024-09-13T13:07:09.195626+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:06.582017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":369,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.187805+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:07:09.187805+0000","last_peered":"2024-09-13T13:07:09.187805+0000","last_clean":"2024-09-13T13:07:09.187805+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:07:09.187805+0000","last_undegraded":"2024-09-13T13:07:09.187805+0000","last_fullsized":"2024-09-13T13:07:09.187805+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:05.357132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":274,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185629+0000","last_change":"2024-09-13T13:02:33.997780+0000","last_active":"2024-09-13T13:07:09.185629+0000","last_peered":"2024-09-13T13:07:09.185629+0000","last_clean":"2024-09-13T13:07:09.185629+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:07:09.185629+0000","last_undegraded":"2024-09-13T13:07:09.185629+0000","last_fullsized":"2024-09-13T13:07:09.185629+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.997740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.997740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:27.073331+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"192'4690","reported_seq":4747,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.138761+0000","last_change":"2024-09-13T13:06:21.814825+0000","last_active":"2024-09-13T13:07:13.138761+0000","last_peered":"2024-09-13T13:07:13.138761+0000","last_clean":"2024-09-13T13:07:13.138761+0000","last_became_active":"2024-09-13T13:06:21.814433+0000","last_became_peered":"2024-09-13T13:06:21.814433+0000","last_unstale":"2024-09-13T13:07:13.138761+0000","last_undegraded":"2024-09-13T13:07:13.138761+0000","last_fullsized":"2024-09-13T13:07:13.138761+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:37.466611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307363840,"num_objects":4690,"num_object_clones":0,"num_object_copies":14070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4690,"num_write_kb":300160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199264+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:07:09.199264+0000","last_peered":"2024-09-13T13:07:09.199264+0000","last_clean":"2024-09-13T13:07:09.199264+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:07:09.199264+0000","last_undegraded":"2024-09-13T13:07:09.199264+0000","last_fullsized":"2024-09-13T13:07:09.199264+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:04.211883+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":304,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189710+0000","last_change":"2024-09-13T13:05:37.520655+0000","last_active":"2024-09-13T13:07:09.189710+0000","last_peered":"2024-09-13T13:07:09.189710+0000","last_clean":"2024-09-13T13:07:09.189710+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:07:09.189710+0000","last_undegraded":"2024-09-13T13:07:09.189710+0000","last_fullsized":"2024-09-13T13:07:09.189710+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:37.520603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:37.520603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:57.639968+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":302,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:11.400066+0000","last_change":"2024-09-13T13:07:11.400066+0000","last_active":"2024-09-13T13:07:11.400066+0000","last_peered":"2024-09-13T13:07:11.400066+0000","last_clean":"2024-09-13T13:07:11.400066+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:07:11.400066+0000","last_undegraded":"2024-09-13T13:07:11.400066+0000","last_fullsized":"2024-09-13T13:07:11.400066+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.399840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.399840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:35.796029+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":265,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.198981+0000","last_change":"2024-09-13T13:01:08.211229+0000","last_active":"2024-09-13T13:07:09.198981+0000","last_peered":"2024-09-13T13:07:09.198981+0000","last_clean":"2024-09-13T13:07:09.198981+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:07:09.198981+0000","last_undegraded":"2024-09-13T13:07:09.198981+0000","last_fullsized":"2024-09-13T13:07:09.198981+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:54.303877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"192'4739","reported_seq":4795,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.655088+0000","last_change":"2024-09-13T13:06:22.000224+0000","last_active":"2024-09-13T13:07:13.655088+0000","last_peered":"2024-09-13T13:07:13.655088+0000","last_clean":"2024-09-13T13:07:13.655088+0000","last_became_active":"2024-09-13T13:06:21.999889+0000","last_became_peered":"2024-09-13T13:06:21.999889+0000","last_unstale":"2024-09-13T13:07:13.655088+0000","last_undegraded":"2024-09-13T13:07:13.655088+0000","last_fullsized":"2024-09-13T13:07:13.655088+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:22.321515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310575104,"num_objects":4739,"num_object_clones":0,"num_object_copies":14217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4739,"num_write_kb":303296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.190969+0000","last_change":"2024-09-13T13:06:06.912256+0000","last_active":"2024-09-13T13:07:09.190969+0000","last_peered":"2024-09-13T13:07:09.190969+0000","last_clean":"2024-09-13T13:07:09.190969+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:07:09.190969+0000","last_undegraded":"2024-09-13T13:07:09.190969+0000","last_fullsized":"2024-09-13T13:07:09.190969+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:06:06.912141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:06.912141+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:19.842077+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":304,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.205110+0000","last_change":"2024-09-13T13:06:14.296977+0000","last_active":"2024-09-13T13:07:09.205110+0000","last_peered":"2024-09-13T13:07:09.205110+0000","last_clean":"2024-09-13T13:07:09.205110+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:07:09.205110+0000","last_undegraded":"2024-09-13T13:07:09.205110+0000","last_fullsized":"2024-09-13T13:07:09.205110+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:14.296913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:14.296913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:41.339805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.186402+0000","last_change":"2024-09-13T13:04:47.927556+0000","last_active":"2024-09-13T13:07:09.186402+0000","last_peered":"2024-09-13T13:07:09.186402+0000","last_clean":"2024-09-13T13:07:09.186402+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:07:09.186402+0000","last_undegraded":"2024-09-13T13:07:09.186402+0000","last_fullsized":"2024-09-13T13:07:09.186402+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.927432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.927432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:17.459384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":281,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.203643+0000","last_change":"2024-09-13T13:06:15.283664+0000","last_active":"2024-09-13T13:07:09.203643+0000","last_peered":"2024-09-13T13:07:09.203643+0000","last_clean":"2024-09-13T13:07:09.203643+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:07:09.203643+0000","last_undegraded":"2024-09-13T13:07:09.203643+0000","last_fullsized":"2024-09-13T13:07:09.203643+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:15.283602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:15.283602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:41.496057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"192'4496","reported_seq":4553,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:10.915553+0000","last_change":"2024-09-13T13:06:22.002144+0000","last_active":"2024-09-13T13:07:10.915553+0000","last_peered":"2024-09-13T13:07:10.915553+0000","last_clean":"2024-09-13T13:07:10.915553+0000","last_became_active":"2024-09-13T13:06:22.001404+0000","last_became_peered":"2024-09-13T13:06:22.001404+0000","last_unstale":"2024-09-13T13:07:10.915553+0000","last_undegraded":"2024-09-13T13:07:10.915553+0000","last_fullsized":"2024-09-13T13:07:10.915553+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4496,"log_dups_size":0,"ondisk_log_size":4496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:38.621035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294649856,"num_objects":4496,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":287744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":305,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185587+0000","last_change":"2024-09-13T13:02:28.990089+0000","last_active":"2024-09-13T13:07:09.185587+0000","last_peered":"2024-09-13T13:07:09.185587+0000","last_clean":"2024-09-13T13:07:09.185587+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:07:09.185587+0000","last_undegraded":"2024-09-13T13:07:09.185587+0000","last_fullsized":"2024-09-13T13:07:09.185587+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:02:28.990005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.990005+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:26.096510+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":301,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.204406+0000","last_change":"2024-09-13T13:05:29.345863+0000","last_active":"2024-09-13T13:07:09.204406+0000","last_peered":"2024-09-13T13:07:09.204406+0000","last_clean":"2024-09-13T13:07:09.204406+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:07:09.204406+0000","last_undegraded":"2024-09-13T13:07:09.204406+0000","last_fullsized":"2024-09-13T13:07:09.204406+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.345796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.345796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:03.542269+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.186171+0000","last_change":"2024-09-13T13:06:42.450669+0000","last_active":"2024-09-13T13:07:09.186171+0000","last_peered":"2024-09-13T13:07:09.186171+0000","last_clean":"2024-09-13T13:07:09.186171+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:07:09.186171+0000","last_undegraded":"2024-09-13T13:07:09.186171+0000","last_fullsized":"2024-09-13T13:07:09.186171+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:42.450598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:42.450598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:10.729355+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.195357+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:07:09.195357+0000","last_peered":"2024-09-13T13:07:09.195357+0000","last_clean":"2024-09-13T13:07:09.195357+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:07:09.195357+0000","last_undegraded":"2024-09-13T13:07:09.195357+0000","last_fullsized":"2024-09-13T13:07:09.195357+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:00.622043+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"192'4584","reported_seq":4641,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.921184+0000","last_change":"2024-09-13T13:06:21.816492+0000","last_active":"2024-09-13T13:07:12.921184+0000","last_peered":"2024-09-13T13:07:12.921184+0000","last_clean":"2024-09-13T13:07:12.921184+0000","last_became_active":"2024-09-13T13:06:21.816096+0000","last_became_peered":"2024-09-13T13:06:21.816096+0000","last_unstale":"2024-09-13T13:07:12.921184+0000","last_undegraded":"2024-09-13T13:07:12.921184+0000","last_fullsized":"2024-09-13T13:07:12.921184+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4584,"log_dups_size":0,"ondisk_log_size":4584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.210645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300417024,"num_objects":4584,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4584,"num_write_kb":293376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":317,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.187951+0000","last_change":"2024-09-13T13:04:52.666294+0000","last_active":"2024-09-13T13:07:09.187951+0000","last_peered":"2024-09-13T13:07:09.187951+0000","last_clean":"2024-09-13T13:07:09.187951+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:07:09.187951+0000","last_undegraded":"2024-09-13T13:07:09.187951+0000","last_fullsized":"2024-09-13T13:07:09.187951+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:52.666222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:52.666222+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:02.658270+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":313,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.204184+0000","last_change":"2024-09-13T13:06:51.627764+0000","last_active":"2024-09-13T13:07:09.204184+0000","last_peered":"2024-09-13T13:07:09.204184+0000","last_clean":"2024-09-13T13:07:09.204184+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:07:09.204184+0000","last_undegraded":"2024-09-13T13:07:09.204184+0000","last_fullsized":"2024-09-13T13:07:09.204184+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:06:51.627691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.627691+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:16.124383+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":377,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.202878+0000","last_change":"2024-09-13T13:04:02.759553+0000","last_active":"2024-09-13T13:07:09.202878+0000","last_peered":"2024-09-13T13:07:09.202878+0000","last_clean":"2024-09-13T13:07:09.202878+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:07:09.202878+0000","last_undegraded":"2024-09-13T13:07:09.202878+0000","last_fullsized":"2024-09-13T13:07:09.202878+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:45.959285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.959285+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:28.444715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188662+0000","last_change":"2024-09-13T13:04:02.757776+0000","last_active":"2024-09-13T13:07:09.188662+0000","last_peered":"2024-09-13T13:07:09.188662+0000","last_clean":"2024-09-13T13:07:09.188662+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:07:09.188662+0000","last_undegraded":"2024-09-13T13:07:09.188662+0000","last_fullsized":"2024-09-13T13:07:09.188662+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:28.529954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.529954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.661828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":271,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:10.058616+0000","last_change":"2024-09-13T13:07:10.058616+0000","last_active":"2024-09-13T13:07:10.058616+0000","last_peered":"2024-09-13T13:07:10.058616+0000","last_clean":"2024-09-13T13:07:10.058616+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:07:10.058616+0000","last_undegraded":"2024-09-13T13:07:10.058616+0000","last_fullsized":"2024-09-13T13:07:10.058616+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:10.058555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.058555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:35.057188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":290,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196607+0000","last_change":"2024-09-13T13:03:14.504339+0000","last_active":"2024-09-13T13:07:09.196607+0000","last_peered":"2024-09-13T13:07:09.196607+0000","last_clean":"2024-09-13T13:07:09.196607+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:07:09.196607+0000","last_undegraded":"2024-09-13T13:07:09.196607+0000","last_fullsized":"2024-09-13T13:07:09.196607+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:03:14.504263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.504263+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:29.774489+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189206+0000","last_change":"2024-09-13T13:06:51.640678+0000","last_active":"2024-09-13T13:07:09.189206+0000","last_peered":"2024-09-13T13:07:09.189206+0000","last_clean":"2024-09-13T13:07:09.189206+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:07:09.189206+0000","last_undegraded":"2024-09-13T13:07:09.189206+0000","last_fullsized":"2024-09-13T13:07:09.189206+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:51.640617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.640617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:10.913640+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":300,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:11.584285+0000","last_change":"2024-09-13T13:07:11.584285+0000","last_active":"2024-09-13T13:07:11.584285+0000","last_peered":"2024-09-13T13:07:11.584285+0000","last_clean":"2024-09-13T13:07:11.584285+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:07:11.584285+0000","last_undegraded":"2024-09-13T13:07:11.584285+0000","last_fullsized":"2024-09-13T13:07:11.584285+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.584244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.584244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:24.783280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199855+0000","last_change":"2024-09-13T13:02:42.589951+0000","last_active":"2024-09-13T13:07:09.199855+0000","last_peered":"2024-09-13T13:07:09.199855+0000","last_clean":"2024-09-13T13:07:09.199855+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:07:09.199855+0000","last_undegraded":"2024-09-13T13:07:09.199855+0000","last_fullsized":"2024-09-13T13:07:09.199855+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:42.589866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.589866+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.328386+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":296,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.204554+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:07:09.204554+0000","last_peered":"2024-09-13T13:07:09.204554+0000","last_clean":"2024-09-13T13:07:09.204554+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:07:09.204554+0000","last_undegraded":"2024-09-13T13:07:09.204554+0000","last_fullsized":"2024-09-13T13:07:09.204554+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:31.493583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":277,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196432+0000","last_change":"2024-09-13T13:05:42.728927+0000","last_active":"2024-09-13T13:07:09.196432+0000","last_peered":"2024-09-13T13:07:09.196432+0000","last_clean":"2024-09-13T13:07:09.196432+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:07:09.196432+0000","last_undegraded":"2024-09-13T13:07:09.196432+0000","last_fullsized":"2024-09-13T13:07:09.196432+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:42.728881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:42.728881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.187173+0000","last_change":"2024-09-13T13:05:45.343713+0000","last_active":"2024-09-13T13:07:09.187173+0000","last_peered":"2024-09-13T13:07:09.187173+0000","last_clean":"2024-09-13T13:07:09.187173+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:07:09.187173+0000","last_undegraded":"2024-09-13T13:07:09.187173+0000","last_fullsized":"2024-09-13T13:07:09.187173+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:45.343638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.343638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:54.569396+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188421+0000","last_change":"2024-09-13T13:04:53.687731+0000","last_active":"2024-09-13T13:07:09.188421+0000","last_peered":"2024-09-13T13:07:09.188421+0000","last_clean":"2024-09-13T13:07:09.188421+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:07:09.188421+0000","last_undegraded":"2024-09-13T13:07:09.188421+0000","last_fullsized":"2024-09-13T13:07:09.188421+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:23.655078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199146+0000","last_change":"2024-09-13T13:06:41.308449+0000","last_active":"2024-09-13T13:07:09.199146+0000","last_peered":"2024-09-13T13:07:09.199146+0000","last_clean":"2024-09-13T13:07:09.199146+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:07:09.199146+0000","last_undegraded":"2024-09-13T13:07:09.199146+0000","last_fullsized":"2024-09-13T13:07:09.199146+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:41.308367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:41.308367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:57.315125+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":352,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196396+0000","last_change":"2024-09-13T13:01:06.210251+0000","last_active":"2024-09-13T13:07:09.196396+0000","last_peered":"2024-09-13T13:07:09.196396+0000","last_clean":"2024-09-13T13:07:09.196396+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:07:09.196396+0000","last_undegraded":"2024-09-13T13:07:09.196396+0000","last_fullsized":"2024-09-13T13:07:09.196396+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:01.574770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184167+0000","last_change":"2024-09-13T13:06:09.144962+0000","last_active":"2024-09-13T13:07:09.184167+0000","last_peered":"2024-09-13T13:07:09.184167+0000","last_clean":"2024-09-13T13:07:09.184167+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:07:09.184167+0000","last_undegraded":"2024-09-13T13:07:09.184167+0000","last_fullsized":"2024-09-13T13:07:09.184167+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:09.144898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.144898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:13.405762+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":296,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196608+0000","last_change":"2024-09-13T13:02:37.222400+0000","last_active":"2024-09-13T13:07:09.196608+0000","last_peered":"2024-09-13T13:07:09.196608+0000","last_clean":"2024-09-13T13:07:09.196608+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:07:09.196608+0000","last_undegraded":"2024-09-13T13:07:09.196608+0000","last_fullsized":"2024-09-13T13:07:09.196608+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:37.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.222329+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:20.683338+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.203771+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:07:09.203771+0000","last_peered":"2024-09-13T13:07:09.203771+0000","last_clean":"2024-09-13T13:07:09.203771+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:07:09.203771+0000","last_undegraded":"2024-09-13T13:07:09.203771+0000","last_fullsized":"2024-09-13T13:07:09.203771+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:59.571619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":384,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188365+0000","last_change":"2024-09-13T13:07:08.781005+0000","last_active":"2024-09-13T13:07:09.188365+0000","last_peered":"2024-09-13T13:07:09.188365+0000","last_clean":"2024-09-13T13:07:09.188365+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:07:09.188365+0000","last_undegraded":"2024-09-13T13:07:09.188365+0000","last_fullsized":"2024-09-13T13:07:09.188365+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:07:08.780952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.780952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:38.355148+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199351+0000","last_change":"2024-09-13T13:02:40.615749+0000","last_active":"2024-09-13T13:07:09.199351+0000","last_peered":"2024-09-13T13:07:09.199351+0000","last_clean":"2024-09-13T13:07:09.199351+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:07:09.199351+0000","last_undegraded":"2024-09-13T13:07:09.199351+0000","last_fullsized":"2024-09-13T13:07:09.199351+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:40.615674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:40.615674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:03.206332+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":320,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184450+0000","last_change":"2024-09-13T13:05:26.186899+0000","last_active":"2024-09-13T13:07:09.184450+0000","last_peered":"2024-09-13T13:07:09.184450+0000","last_clean":"2024-09-13T13:07:09.184450+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:07:09.184450+0000","last_undegraded":"2024-09-13T13:07:09.184450+0000","last_fullsized":"2024-09-13T13:07:09.184450+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:05:26.186830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.186830+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:55.950536+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":305,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184170+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:07:09.184170+0000","last_peered":"2024-09-13T13:07:09.184170+0000","last_clean":"2024-09-13T13:07:09.184170+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:07:09.184170+0000","last_undegraded":"2024-09-13T13:07:09.184170+0000","last_fullsized":"2024-09-13T13:07:09.184170+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:56.924269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.191379+0000","last_change":"2024-09-13T13:03:44.467611+0000","last_active":"2024-09-13T13:07:09.191379+0000","last_peered":"2024-09-13T13:07:09.191379+0000","last_clean":"2024-09-13T13:07:09.191379+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:07:09.191379+0000","last_undegraded":"2024-09-13T13:07:09.191379+0000","last_fullsized":"2024-09-13T13:07:09.191379+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:44.467297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:44.467297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:05.125721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.194352+0000","last_change":"2024-09-13T13:02:42.208021+0000","last_active":"2024-09-13T13:07:09.194352+0000","last_peered":"2024-09-13T13:07:09.194352+0000","last_clean":"2024-09-13T13:07:09.194352+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:07:09.194352+0000","last_undegraded":"2024-09-13T13:07:09.194352+0000","last_fullsized":"2024-09-13T13:07:09.194352+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:42.207956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.207956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:18.666832+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":249,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188578+0000","last_change":"2024-09-13T13:05:35.506093+0000","last_active":"2024-09-13T13:07:09.188578+0000","last_peered":"2024-09-13T13:07:09.188578+0000","last_clean":"2024-09-13T13:07:09.188578+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:07:09.188578+0000","last_undegraded":"2024-09-13T13:07:09.188578+0000","last_fullsized":"2024-09-13T13:07:09.188578+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:05:35.506037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.506037+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:07.725091+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.204973+0000","last_change":"2024-09-13T13:02:39.921391+0000","last_active":"2024-09-13T13:07:09.204973+0000","last_peered":"2024-09-13T13:07:09.204973+0000","last_clean":"2024-09-13T13:07:09.204973+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:07:09.204973+0000","last_undegraded":"2024-09-13T13:07:09.204973+0000","last_fullsized":"2024-09-13T13:07:09.204973+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.921325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.921325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:22.433670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":269,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.195431+0000","last_change":"2024-09-13T13:01:06.211993+0000","last_active":"2024-09-13T13:07:09.195431+0000","last_peered":"2024-09-13T13:07:09.195431+0000","last_clean":"2024-09-13T13:07:09.195431+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:07:09.195431+0000","last_undegraded":"2024-09-13T13:07:09.195431+0000","last_fullsized":"2024-09-13T13:07:09.195431+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:57.661293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.198669+0000","last_change":"2024-09-13T13:02:36.586702+0000","last_active":"2024-09-13T13:07:09.198669+0000","last_peered":"2024-09-13T13:07:09.198669+0000","last_clean":"2024-09-13T13:07:09.198669+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:07:09.198669+0000","last_undegraded":"2024-09-13T13:07:09.198669+0000","last_fullsized":"2024-09-13T13:07:09.198669+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.586654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.586654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:22.364490+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":311,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.194481+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:07:09.194481+0000","last_peered":"2024-09-13T13:07:09.194481+0000","last_clean":"2024-09-13T13:07:09.194481+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:07:09.194481+0000","last_undegraded":"2024-09-13T13:07:09.194481+0000","last_fullsized":"2024-09-13T13:07:09.194481+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:02.594342+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189280+0000","last_change":"2024-09-13T13:05:12.597696+0000","last_active":"2024-09-13T13:07:09.189280+0000","last_peered":"2024-09-13T13:07:09.189280+0000","last_clean":"2024-09-13T13:07:09.189280+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:07:09.189280+0000","last_undegraded":"2024-09-13T13:07:09.189280+0000","last_fullsized":"2024-09-13T13:07:09.189280+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:12.597637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.597637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:33.633105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":296,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.202876+0000","last_change":"2024-09-13T13:06:10.265459+0000","last_active":"2024-09-13T13:07:09.202876+0000","last_peered":"2024-09-13T13:07:09.202876+0000","last_clean":"2024-09-13T13:07:09.202876+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:07:09.202876+0000","last_undegraded":"2024-09-13T13:07:09.202876+0000","last_fullsized":"2024-09-13T13:07:09.202876+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:10.265415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.265415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:30.604403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":276,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.195553+0000","last_change":"2024-09-13T13:04:02.760695+0000","last_active":"2024-09-13T13:07:09.195553+0000","last_peered":"2024-09-13T13:07:09.195553+0000","last_clean":"2024-09-13T13:07:09.195553+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:07:09.195553+0000","last_undegraded":"2024-09-13T13:07:09.195553+0000","last_fullsized":"2024-09-13T13:07:09.195553+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:55.662877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":311,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.197247+0000","last_change":"2024-09-13T13:05:44.759472+0000","last_active":"2024-09-13T13:07:09.197247+0000","last_peered":"2024-09-13T13:07:09.197247+0000","last_clean":"2024-09-13T13:07:09.197247+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:07:09.197247+0000","last_undegraded":"2024-09-13T13:07:09.197247+0000","last_fullsized":"2024-09-13T13:07:09.197247+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2024-09-13T13:05:44.759233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.759233+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.200287+0000","last_change":"2024-09-13T13:07:09.132221+0000","last_active":"2024-09-13T13:07:09.200287+0000","last_peered":"2024-09-13T13:07:09.200287+0000","last_clean":"2024-09-13T13:07:09.200287+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:07:09.200287+0000","last_undegraded":"2024-09-13T13:07:09.200287+0000","last_fullsized":"2024-09-13T13:07:09.200287+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:09.132136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:09.132136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:19.094537+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":179,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.191302+0000","last_change":"2024-09-13T13:06:47.181575+0000","last_active":"2024-09-13T13:07:09.191302+0000","last_peered":"2024-09-13T13:07:09.191302+0000","last_clean":"2024-09-13T13:07:09.191302+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:07:09.191302+0000","last_undegraded":"2024-09-13T13:07:09.191302+0000","last_fullsized":"2024-09-13T13:07:09.191302+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:47.181407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:47.181407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:15.131168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":289,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.592938+0000","last_change":"2024-09-13T13:07:12.592938+0000","last_active":"2024-09-13T13:07:12.592938+0000","last_peered":"2024-09-13T13:07:12.592938+0000","last_clean":"2024-09-13T13:07:12.592938+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:07:12.592938+0000","last_undegraded":"2024-09-13T13:07:12.592938+0000","last_fullsized":"2024-09-13T13:07:12.592938+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.592873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.592873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:29.659738+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.193262+0000","last_change":"2024-09-13T13:05:44.759683+0000","last_active":"2024-09-13T13:07:09.193262+0000","last_peered":"2024-09-13T13:07:09.193262+0000","last_clean":"2024-09-13T13:07:09.193262+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:07:09.193262+0000","last_undegraded":"2024-09-13T13:07:09.193262+0000","last_fullsized":"2024-09-13T13:07:09.193262+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:44.759284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.759284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.197479+0000","last_change":"2024-09-13T13:02:46.246602+0000","last_active":"2024-09-13T13:07:09.197479+0000","last_peered":"2024-09-13T13:07:09.197479+0000","last_clean":"2024-09-13T13:07:09.197479+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:07:09.197479+0000","last_undegraded":"2024-09-13T13:07:09.197479+0000","last_fullsized":"2024-09-13T13:07:09.197479+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.246540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.246540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:26.741283+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":289,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.192498+0000","last_change":"2024-09-13T13:06:09.928585+0000","last_active":"2024-09-13T13:07:09.192498+0000","last_peered":"2024-09-13T13:07:09.192498+0000","last_clean":"2024-09-13T13:07:09.192498+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:07:09.192498+0000","last_undegraded":"2024-09-13T13:07:09.192498+0000","last_fullsized":"2024-09-13T13:07:09.192498+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:06:09.928504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.928504+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:39.792862+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199628+0000","last_change":"2024-09-13T13:06:47.291601+0000","last_active":"2024-09-13T13:07:09.199628+0000","last_peered":"2024-09-13T13:07:09.199628+0000","last_clean":"2024-09-13T13:07:09.199628+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:07:09.199628+0000","last_undegraded":"2024-09-13T13:07:09.199628+0000","last_fullsized":"2024-09-13T13:07:09.199628+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:47.291245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:47.291245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:05.181779+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4821680128,"num_objects":73573,"num_object_clones":0,"num_object_copies":220719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73573,"num_write_kb":4708672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14462812160,"data_stored":14462812160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73573,"ondisk_log_size":73573,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":365,"num_read_kb":360,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201357,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1613680,"kb_used_data":1465888,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92131472,"statfs":{"total":95995035648,"available":94342627328,"internally_reserved":0,"allocated":1501069312,"data_stored":1500410848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":151321034},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806140967,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1594084,"kb_used_data":1416400,"kb_used_omap":4,"kb_used_meta":177659,"kb_avail":92151068,"statfs":{"total":95995035648,"available":94362693632,"internally_reserved":0,"allocated":1450393600,"data_stored":1449749891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":181923501},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691929,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2503460,"kb_used_data":2333460,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":91241692,"statfs":{"total":95995035648,"available":93431492608,"internally_reserved":0,"allocated":2389463040,"data_stored":2388815160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":174062021},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920863,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1618040,"kb_used_data":1470248,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92127112,"statfs":{"total":95995035648,"available":94338162688,"internally_reserved":0,"allocated":1505533952,"data_stored":1504865728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117094,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2028476,"kb_used_data":1750188,"kb_used_omap":1,"kb_used_meta":278270,"kb_avail":91716676,"statfs":{"total":95995035648,"available":93917876224,"internally_reserved":0,"allocated":1792192512,"data_stored":1791521022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":284948935},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411433,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1911956,"kb_used_data":1750276,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":91833196,"statfs":{"total":95995035648,"available":94037192704,"internally_reserved":0,"allocated":1792282624,"data_stored":1791650468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165542342},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2219404,"kb_used_data":2043836,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91525748,"statfs":{"total":95995035648,"available":93722365952,"internally_reserved":0,"allocated":2092888064,"data_stored":2092199871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738487,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2013940,"kb_used_data":1775908,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":91731212,"statfs":{"total":95995035648,"available":93932761088,"internally_reserved":0,"allocated":1818529792,"data_stored":1817866941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1832189952,"data_stored":1832189952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2124152832,"data_stored":2124152832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1814036480,"data_stored":1814036480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1812267008,"data_stored":1812267008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1505558528,"data_stored":1505558528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2396585984,"data_stored":2396585984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1470562304,"data_stored":1470562304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1507459072,"data_stored":1507459072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:07:16.441 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:07:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:16.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:16.543 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:16.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:16 smithi167 ceph-mon[33178]: 4.f scrub starts 2024-09-13T13:07:16.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:16 smithi167 ceph-mon[33178]: 4.f scrub ok 2024-09-13T13:07:16.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:16 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:07:16.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:16 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:07:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:16.595 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:16.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:16.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:16.724 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:16.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:16.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:16.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 45 16 78275 78259 108.662 79.125 0.0191457 0.00919717 2024-09-13T13:07:16.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:16.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:16.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:16.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:16.961 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T13:07:16.988 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:16.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:16.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:17.006 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:17.060 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:17.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:17.114 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:17.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:17.216 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:17.275 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:17.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:17.377 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:17.406 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:17.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:17.456 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:17 smithi052 ceph-mon[36682]: pgmap v295: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-13T13:07:17.456 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:17 smithi052 ceph-mon[36682]: osdmap e193: 8 total, 8 up, 8 in 2024-09-13T13:07:17.456 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:17 smithi052 ceph-mon[29471]: pgmap v295: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-13T13:07:17.456 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:17 smithi052 ceph-mon[29471]: osdmap e193: 8 total, 8 up, 8 in 2024-09-13T13:07:17.508 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:17.539 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:17.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:17 smithi167 ceph-mon[33178]: pgmap v295: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-13T13:07:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:17 smithi167 ceph-mon[33178]: osdmap e193: 8 total, 8 up, 8 in 2024-09-13T13:07:17.641 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:17.682 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:17.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:17.747 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:07:17.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 46 16 79260 79244 107.637 61.5625 0.0274301 0.00928451 2024-09-13T13:07:17.785 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:17.858 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:17.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:17.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:17.960 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:18.003 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:18.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:18.104 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:18.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:18.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:18.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:18.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:18.262 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:18.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:18.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:18.363 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:18.388 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:18.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:18.416 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:18.488 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:18.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:07:18.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:07:18.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[36682]: osdmap e194: 8 total, 8 up, 8 in 2024-09-13T13:07:18.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:07:18.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[36682]: 5.e scrub starts 2024-09-13T13:07:18.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[36682]: 5.e scrub ok 2024-09-13T13:07:18.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2527803453' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:07:18.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:07:18.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:07:18.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[29471]: osdmap e194: 8 total, 8 up, 8 in 2024-09-13T13:07:18.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:07:18.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[29471]: 5.e scrub starts 2024-09-13T13:07:18.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[29471]: 5.e scrub ok 2024-09-13T13:07:18.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:18 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2527803453' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:07:18.515 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:18.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:18 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:07:18.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:18 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:07:18.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:18 smithi167 ceph-mon[33178]: osdmap e194: 8 total, 8 up, 8 in 2024-09-13T13:07:18.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:18 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:07:18.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:18 smithi167 ceph-mon[33178]: 5.e scrub starts 2024-09-13T13:07:18.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:18 smithi167 ceph-mon[33178]: 5.e scrub ok 2024-09-13T13:07:18.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:18 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2527803453' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:07:18.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:18.647 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:18.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:18.742 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:18.748 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:18.756 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:07:18.764 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:07:18.773 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 47 16 80072 80056 106.426 50.75 0.0233583 0.00938896 2024-09-13T13:07:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:18.808 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:18.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:18.837 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:07:18.910 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:18.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:18.954 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:18.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:19.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:19.087 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:19.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:19.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:19.189 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:19.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:19.240 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:19.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:19.322 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:07:19.323 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:07:19.323 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:07:19.323 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:07:19.323 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:07:19.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:19.372 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:19.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:19.474 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:19.479 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[36682]: 5.4 scrub starts 2024-09-13T13:07:19.479 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[36682]: 5.4 scrub ok 2024-09-13T13:07:19.480 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[36682]: pgmap v298: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 904 op/s 2024-09-13T13:07:19.480 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[36682]: 5.b scrub starts 2024-09-13T13:07:19.480 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[36682]: 5.b scrub ok 2024-09-13T13:07:19.480 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:07:19.480 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:07:19.480 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2527803453' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:07:19.480 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[36682]: osdmap e195: 8 total, 8 up, 8 in 2024-09-13T13:07:19.480 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[36682]: 3.3 scrub starts 2024-09-13T13:07:19.480 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[36682]: 3.3 scrub ok 2024-09-13T13:07:19.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[29471]: 5.4 scrub starts 2024-09-13T13:07:19.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[29471]: 5.4 scrub ok 2024-09-13T13:07:19.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[29471]: pgmap v298: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 904 op/s 2024-09-13T13:07:19.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[29471]: 5.b scrub starts 2024-09-13T13:07:19.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[29471]: 5.b scrub ok 2024-09-13T13:07:19.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:07:19.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:07:19.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2527803453' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:07:19.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[29471]: osdmap e195: 8 total, 8 up, 8 in 2024-09-13T13:07:19.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[29471]: 3.3 scrub starts 2024-09-13T13:07:19.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:19 smithi052 ceph-mon[29471]: 3.3 scrub ok 2024-09-13T13:07:19.516 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:19.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:19.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:19.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:19 smithi167 ceph-mon[33178]: 5.4 scrub starts 2024-09-13T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:19 smithi167 ceph-mon[33178]: 5.4 scrub ok 2024-09-13T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:19 smithi167 ceph-mon[33178]: pgmap v298: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 904 op/s 2024-09-13T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:19 smithi167 ceph-mon[33178]: 5.b scrub starts 2024-09-13T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:19 smithi167 ceph-mon[33178]: 5.b scrub ok 2024-09-13T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:19 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:19 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:19 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2527803453' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:19 smithi167 ceph-mon[33178]: osdmap e195: 8 total, 8 up, 8 in 2024-09-13T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:19 smithi167 ceph-mon[33178]: 3.3 scrub starts 2024-09-13T13:07:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:19 smithi167 ceph-mon[33178]: 3.3 scrub ok 2024-09-13T13:07:19.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:19.646 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:19.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:19.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:19.748 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:19.773 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 48 16 81182 81166 105.654 69.375 0.00676382 0.00945929 2024-09-13T13:07:19.788 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:19.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:19.890 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:19.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:19.923 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:19.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:20.025 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:20.066 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:20.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:20.168 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:20.204 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:20.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:20.306 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:20.349 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:20.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:20.451 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:20.479 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:20.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:20.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:20.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[29471]: from='client.25321 v1:172.21.15.52:0/599293193' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:07:20.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[29471]: 6.4 scrub starts 2024-09-13T13:07:20.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[29471]: 6.4 scrub ok 2024-09-13T13:07:20.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[29471]: 6.11 scrub starts 2024-09-13T13:07:20.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[29471]: 6.11 scrub ok 2024-09-13T13:07:20.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[29471]: 3.9 scrub starts 2024-09-13T13:07:20.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[29471]: 3.9 scrub ok 2024-09-13T13:07:20.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[36682]: from='client.25321 v1:172.21.15.52:0/599293193' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:07:20.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[36682]: 6.4 scrub starts 2024-09-13T13:07:20.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[36682]: 6.4 scrub ok 2024-09-13T13:07:20.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[36682]: 6.11 scrub starts 2024-09-13T13:07:20.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[36682]: 6.11 scrub ok 2024-09-13T13:07:20.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[36682]: 3.9 scrub starts 2024-09-13T13:07:20.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:20 smithi052 ceph-mon[36682]: 3.9 scrub ok 2024-09-13T13:07:20.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:20 smithi167 ceph-mon[33178]: from='client.25321 v1:172.21.15.52:0/599293193' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:07:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:20 smithi167 ceph-mon[33178]: 6.4 scrub starts 2024-09-13T13:07:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:20 smithi167 ceph-mon[33178]: 6.4 scrub ok 2024-09-13T13:07:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:20 smithi167 ceph-mon[33178]: 6.11 scrub starts 2024-09-13T13:07:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:20 smithi167 ceph-mon[33178]: 6.11 scrub ok 2024-09-13T13:07:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:20 smithi167 ceph-mon[33178]: 3.9 scrub starts 2024-09-13T13:07:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:20 smithi167 ceph-mon[33178]: 3.9 scrub ok 2024-09-13T13:07:20.581 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:20.621 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:20.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:20.723 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:20.751 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:20.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:20.773 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 49 16 82455 82439 105.122 79.5625 0.0226042 0.00950757 2024-09-13T13:07:20.853 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:20.899 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:20.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:20.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:21.001 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:21.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:21.052 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:21.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:21.102 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:07:21.102 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:07:21.102 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:07:21.102 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:07:21.102 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:07:21.102 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:07:21.103 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:07:21.103 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:07:21.103 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:07:21.103 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:07:21.103 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:07:21.103 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:07:21.103 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:07:21.103 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:07:21.107 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:07:21.108 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:07:21.153 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:21.187 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:21.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:21.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:21.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:21.322 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:21.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:21.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:21.389 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[36682]: 6.19 scrub starts 2024-09-13T13:07:21.389 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[36682]: 6.19 scrub ok 2024-09-13T13:07:21.389 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[36682]: pgmap v300: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 954 op/s 2024-09-13T13:07:21.389 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[36682]: 3.0 scrub starts 2024-09-13T13:07:21.389 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[36682]: 3.0 scrub ok 2024-09-13T13:07:21.389 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[36682]: 6.5 scrub starts 2024-09-13T13:07:21.389 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[36682]: 6.5 scrub ok 2024-09-13T13:07:21.389 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/11387833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:21.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[29471]: 6.19 scrub starts 2024-09-13T13:07:21.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[29471]: 6.19 scrub ok 2024-09-13T13:07:21.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[29471]: pgmap v300: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 954 op/s 2024-09-13T13:07:21.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[29471]: 3.0 scrub starts 2024-09-13T13:07:21.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[29471]: 3.0 scrub ok 2024-09-13T13:07:21.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[29471]: 6.5 scrub starts 2024-09-13T13:07:21.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[29471]: 6.5 scrub ok 2024-09-13T13:07:21.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:21 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/11387833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:21.424 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:21.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T13:07:21.460 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:21.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:21.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:21.562 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:21.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:21 smithi167 ceph-mon[33178]: 6.19 scrub starts 2024-09-13T13:07:21.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:21 smithi167 ceph-mon[33178]: 6.19 scrub ok 2024-09-13T13:07:21.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:21 smithi167 ceph-mon[33178]: pgmap v300: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 954 op/s 2024-09-13T13:07:21.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:21 smithi167 ceph-mon[33178]: 3.0 scrub starts 2024-09-13T13:07:21.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:21 smithi167 ceph-mon[33178]: 3.0 scrub ok 2024-09-13T13:07:21.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:21 smithi167 ceph-mon[33178]: 6.5 scrub starts 2024-09-13T13:07:21.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:21 smithi167 ceph-mon[33178]: 6.5 scrub ok 2024-09-13T13:07:21.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:21 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/11387833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:21.592 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:21.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:21.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:21.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:21.723 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:21.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:21.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:21.773 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 50 16 83629 83613 104.487 73.375 0.00681658 0.0095653 2024-09-13T13:07:21.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:21.853 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:21.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:21.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:21.954 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:21.983 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:21.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:22.085 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:22.114 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:22.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:22.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:22.216 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:22.262 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:22.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:22.312 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:22 smithi052 ceph-mon[36682]: 4.14 scrub starts 2024-09-13T13:07:22.312 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:22 smithi052 ceph-mon[29471]: 4.14 scrub starts 2024-09-13T13:07:22.312 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:22 smithi052 ceph-mon[29471]: 4.14 scrub ok 2024-09-13T13:07:22.364 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:22.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:22.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:22.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:22.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:22.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:22.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:22.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:22 smithi167 ceph-mon[33178]: 4.14 scrub starts 2024-09-13T13:07:22.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:22 smithi167 ceph-mon[33178]: 4.14 scrub ok 2024-09-13T13:07:22.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:22 smithi167 ceph-mon[33178]: 6.17 scrub starts 2024-09-13T13:07:22.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:22 smithi167 ceph-mon[33178]: 6.17 scrub ok 2024-09-13T13:07:22.578 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:07:22.619 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:22 smithi052 ceph-mon[29471]: 6.17 scrub starts 2024-09-13T13:07:22.619 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:22 smithi052 ceph-mon[29471]: 6.17 scrub ok 2024-09-13T13:07:22.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:22 smithi052 ceph-mon[36682]: 4.14 scrub ok 2024-09-13T13:07:22.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:22 smithi052 ceph-mon[36682]: 6.17 scrub starts 2024-09-13T13:07:22.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:22 smithi052 ceph-mon[36682]: 6.17 scrub ok 2024-09-13T13:07:22.635 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:22.698 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:22.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:22.773 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 51 16 84908 84892 104.005 79.9375 0.0164691 0.00960966 2024-09-13T13:07:22.800 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:22.836 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:22.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:22.874 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-09-13T13:07:17.715295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4873787931,"num_objects":74558,"num_object_clones":0,"num_object_copies":223674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74558,"num_whiteouts":0,"num_read":1211,"num_read_kb":1068,"num_write":74908,"num_write_kb":4760649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74919,"ondisk_log_size":74919,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15634108,"kb_used_data":14128956,"kb_used_omap":15,"kb_used_meta":1505008,"kb_avail":734327108,"statfs":{"total":767960285184,"available":751950958592,"internally_reserved":0,"allocated":14468050944,"data_stored":14462777977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":1541128985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":474284032,"num_objects":7237,"num_object_clones":0,"num_object_copies":21711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7237,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7237,"num_write_kb":463168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003201"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.200434+0000","last_change":"2024-09-13T13:06:44.304525+0000","last_active":"2024-09-13T13:07:09.200434+0000","last_peered":"2024-09-13T13:07:09.200434+0000","last_clean":"2024-09-13T13:07:09.200434+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:07:09.200434+0000","last_undegraded":"2024-09-13T13:07:09.200434+0000","last_fullsized":"2024-09-13T13:07:09.200434+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:44.304444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.304444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:59.584889+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":309,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.183957+0000","last_change":"2024-09-13T13:04:48.899982+0000","last_active":"2024-09-13T13:07:09.183957+0000","last_peered":"2024-09-13T13:07:09.183957+0000","last_clean":"2024-09-13T13:07:09.183957+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:07:09.183957+0000","last_undegraded":"2024-09-13T13:07:09.183957+0000","last_fullsized":"2024-09-13T13:07:09.183957+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:48.899915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.899915+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:03.931197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189642+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:07:09.189642+0000","last_peered":"2024-09-13T13:07:09.189642+0000","last_clean":"2024-09-13T13:07:09.189642+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:07:09.189642+0000","last_undegraded":"2024-09-13T13:07:09.189642+0000","last_fullsized":"2024-09-13T13:07:09.189642+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:06.133166+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":274,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185342+0000","last_change":"2024-09-13T13:05:01.076346+0000","last_active":"2024-09-13T13:07:09.185342+0000","last_peered":"2024-09-13T13:07:09.185342+0000","last_clean":"2024-09-13T13:07:09.185342+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:07:09.185342+0000","last_undegraded":"2024-09-13T13:07:09.185342+0000","last_fullsized":"2024-09-13T13:07:09.185342+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.076275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.076275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:30.081244+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.195272+0000","last_change":"2024-09-13T13:05:45.805098+0000","last_active":"2024-09-13T13:07:09.195272+0000","last_peered":"2024-09-13T13:07:09.195272+0000","last_clean":"2024-09-13T13:07:09.195272+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:07:09.195272+0000","last_undegraded":"2024-09-13T13:07:09.195272+0000","last_fullsized":"2024-09-13T13:07:09.195272+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:45.800516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.800516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.201752+0000","last_change":"2024-09-13T13:04:02.757267+0000","last_active":"2024-09-13T13:07:09.201752+0000","last_peered":"2024-09-13T13:07:09.201752+0000","last_clean":"2024-09-13T13:07:09.201752+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:07:09.201752+0000","last_undegraded":"2024-09-13T13:07:09.201752+0000","last_fullsized":"2024-09-13T13:07:09.201752+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.233259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.233259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:23.678677+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":322,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.549946+0000","last_change":"2024-09-13T13:07:09.549946+0000","last_active":"2024-09-13T13:07:09.549946+0000","last_peered":"2024-09-13T13:07:09.549946+0000","last_clean":"2024-09-13T13:07:09.549946+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:07:09.549946+0000","last_undegraded":"2024-09-13T13:07:09.549946+0000","last_fullsized":"2024-09-13T13:07:09.549946+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:07:09.549857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:09.549857+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:27.521844+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":289,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.183849+0000","last_change":"2024-09-13T13:04:02.765671+0000","last_active":"2024-09-13T13:07:09.183849+0000","last_peered":"2024-09-13T13:07:09.183849+0000","last_clean":"2024-09-13T13:07:09.183849+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:07:09.183849+0000","last_undegraded":"2024-09-13T13:07:09.183849+0000","last_fullsized":"2024-09-13T13:07:09.183849+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.021662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.021662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:22.118745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":310,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188427+0000","last_change":"2024-09-13T13:06:10.595426+0000","last_active":"2024-09-13T13:07:09.188427+0000","last_peered":"2024-09-13T13:07:09.188427+0000","last_clean":"2024-09-13T13:07:09.188427+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:07:09.188427+0000","last_undegraded":"2024-09-13T13:07:09.188427+0000","last_fullsized":"2024-09-13T13:07:09.188427+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:06:10.595347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.595347+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:21.918077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":306,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185622+0000","last_change":"2024-09-13T13:06:02.262489+0000","last_active":"2024-09-13T13:07:09.185622+0000","last_peered":"2024-09-13T13:07:09.185622+0000","last_clean":"2024-09-13T13:07:09.185622+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:07:09.185622+0000","last_undegraded":"2024-09-13T13:07:09.185622+0000","last_fullsized":"2024-09-13T13:07:09.185622+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:06:02.262393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:02.262393+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:17.465680+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.190619+0000","last_change":"2024-09-13T13:06:50.665256+0000","last_active":"2024-09-13T13:07:09.190619+0000","last_peered":"2024-09-13T13:07:09.190619+0000","last_clean":"2024-09-13T13:07:09.190619+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:07:09.190619+0000","last_undegraded":"2024-09-13T13:07:09.190619+0000","last_fullsized":"2024-09-13T13:07:09.190619+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:50.665191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.665191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:53.974779+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.198819+0000","last_change":"2024-09-13T13:02:39.607712+0000","last_active":"2024-09-13T13:07:09.198819+0000","last_peered":"2024-09-13T13:07:09.198819+0000","last_clean":"2024-09-13T13:07:09.198819+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:07:09.198819+0000","last_undegraded":"2024-09-13T13:07:09.198819+0000","last_fullsized":"2024-09-13T13:07:09.198819+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.607646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.607646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:21.338399+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":301,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189111+0000","last_change":"2024-09-13T13:05:38.520071+0000","last_active":"2024-09-13T13:07:09.189111+0000","last_peered":"2024-09-13T13:07:09.189111+0000","last_clean":"2024-09-13T13:07:09.189111+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:07:09.189111+0000","last_undegraded":"2024-09-13T13:07:09.189111+0000","last_fullsized":"2024-09-13T13:07:09.189111+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:38.520006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:38.520006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:55.622971+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":300,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184360+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:07:09.184360+0000","last_peered":"2024-09-13T13:07:09.184360+0000","last_clean":"2024-09-13T13:07:09.184360+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:07:09.184360+0000","last_undegraded":"2024-09-13T13:07:09.184360+0000","last_fullsized":"2024-09-13T13:07:09.184360+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:32.042125+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.203258+0000","last_change":"2024-09-13T13:06:45.544015+0000","last_active":"2024-09-13T13:07:09.203258+0000","last_peered":"2024-09-13T13:07:09.203258+0000","last_clean":"2024-09-13T13:07:09.203258+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:07:09.203258+0000","last_undegraded":"2024-09-13T13:07:09.203258+0000","last_fullsized":"2024-09-13T13:07:09.203258+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:45.543961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:45.543961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:50.020558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":265,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188183+0000","last_change":"2024-09-13T13:01:08.213608+0000","last_active":"2024-09-13T13:07:09.188183+0000","last_peered":"2024-09-13T13:07:09.188183+0000","last_clean":"2024-09-13T13:07:09.188183+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:07:09.188183+0000","last_undegraded":"2024-09-13T13:07:09.188183+0000","last_fullsized":"2024-09-13T13:07:09.188183+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:17.609112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":295,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.186013+0000","last_change":"2024-09-13T13:06:37.129744+0000","last_active":"2024-09-13T13:07:09.186013+0000","last_peered":"2024-09-13T13:07:09.186013+0000","last_clean":"2024-09-13T13:07:09.186013+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:07:09.186013+0000","last_undegraded":"2024-09-13T13:07:09.186013+0000","last_fullsized":"2024-09-13T13:07:09.186013+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:06:37.129698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:37.129698+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:46.413310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199533+0000","last_change":"2024-09-13T13:06:38.260486+0000","last_active":"2024-09-13T13:07:09.199533+0000","last_peered":"2024-09-13T13:07:09.199533+0000","last_clean":"2024-09-13T13:07:09.199533+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:07:09.199533+0000","last_undegraded":"2024-09-13T13:07:09.199533+0000","last_fullsized":"2024-09-13T13:07:09.199533+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:38.259822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:38.259822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:06.534345+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":331,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.187546+0000","last_change":"2024-09-13T13:06:44.532305+0000","last_active":"2024-09-13T13:07:09.187546+0000","last_peered":"2024-09-13T13:07:09.187546+0000","last_clean":"2024-09-13T13:07:09.187546+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:07:09.187546+0000","last_undegraded":"2024-09-13T13:07:09.187546+0000","last_fullsized":"2024-09-13T13:07:09.187546+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:06:44.532186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.532186+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:01.016677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.193534+0000","last_change":"2024-09-13T13:06:47.977011+0000","last_active":"2024-09-13T13:07:09.193534+0000","last_peered":"2024-09-13T13:07:09.193534+0000","last_clean":"2024-09-13T13:07:09.193534+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:07:09.193534+0000","last_undegraded":"2024-09-13T13:07:09.193534+0000","last_fullsized":"2024-09-13T13:07:09.193534+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:47.976922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:47.976922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:17.524630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196346+0000","last_change":"2024-09-13T13:04:03.384955+0000","last_active":"2024-09-13T13:07:09.196346+0000","last_peered":"2024-09-13T13:07:09.196346+0000","last_clean":"2024-09-13T13:07:09.196346+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:07:09.196346+0000","last_undegraded":"2024-09-13T13:07:09.196346+0000","last_fullsized":"2024-09-13T13:07:09.196346+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:03.384901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:04:03.384901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185767+0000","last_change":"2024-09-13T13:04:57.103213+0000","last_active":"2024-09-13T13:07:09.185767+0000","last_peered":"2024-09-13T13:07:09.185767+0000","last_clean":"2024-09-13T13:07:09.185767+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:07:09.185767+0000","last_undegraded":"2024-09-13T13:07:09.185767+0000","last_fullsized":"2024-09-13T13:07:09.185767+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:57.103167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:57.103167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:53.999219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":304,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.193409+0000","last_change":"2024-09-13T13:06:05.706641+0000","last_active":"2024-09-13T13:07:09.193409+0000","last_peered":"2024-09-13T13:07:09.193409+0000","last_clean":"2024-09-13T13:07:09.193409+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:07:09.193409+0000","last_undegraded":"2024-09-13T13:07:09.193409+0000","last_fullsized":"2024-09-13T13:07:09.193409+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:06:05.706569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:05.706569+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:09.818528+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":294,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.879813+0000","last_change":"2024-09-13T13:07:12.879813+0000","last_active":"2024-09-13T13:07:12.879813+0000","last_peered":"2024-09-13T13:07:12.879813+0000","last_clean":"2024-09-13T13:07:12.879813+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:07:12.879813+0000","last_undegraded":"2024-09-13T13:07:12.879813+0000","last_fullsized":"2024-09-13T13:07:12.879813+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.879727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.879727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:16.423585+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.206440+0000","last_change":"2024-09-13T13:06:38.536219+0000","last_active":"2024-09-13T13:07:09.206440+0000","last_peered":"2024-09-13T13:07:09.206440+0000","last_clean":"2024-09-13T13:07:09.206440+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:07:09.206440+0000","last_undegraded":"2024-09-13T13:07:09.206440+0000","last_fullsized":"2024-09-13T13:07:09.206440+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:06:38.536170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:38.536170+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:39.956585+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":287,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189769+0000","last_change":"2024-09-13T13:03:12.276224+0000","last_active":"2024-09-13T13:07:09.189769+0000","last_peered":"2024-09-13T13:07:09.189769+0000","last_clean":"2024-09-13T13:07:09.189769+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:07:09.189769+0000","last_undegraded":"2024-09-13T13:07:09.189769+0000","last_fullsized":"2024-09-13T13:07:09.189769+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:12.276136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.276136+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:52.950269+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":305,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199724+0000","last_change":"2024-09-13T13:06:40.292804+0000","last_active":"2024-09-13T13:07:09.199724+0000","last_peered":"2024-09-13T13:07:09.199724+0000","last_clean":"2024-09-13T13:07:09.199724+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:07:09.199724+0000","last_undegraded":"2024-09-13T13:07:09.199724+0000","last_fullsized":"2024-09-13T13:07:09.199724+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:06:40.292551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:40.292551+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:01.413088+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":318,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.192669+0000","last_change":"2024-09-13T13:06:04.949976+0000","last_active":"2024-09-13T13:07:09.192669+0000","last_peered":"2024-09-13T13:07:09.192669+0000","last_clean":"2024-09-13T13:07:09.192669+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:07:09.192669+0000","last_undegraded":"2024-09-13T13:07:09.192669+0000","last_fullsized":"2024-09-13T13:07:09.192669+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:06:04.949930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:04.949930+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:33.315989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"192'4689","reported_seq":4746,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.124825+0000","last_change":"2024-09-13T13:06:21.814268+0000","last_active":"2024-09-13T13:07:13.124825+0000","last_peered":"2024-09-13T13:07:13.124825+0000","last_clean":"2024-09-13T13:07:13.124825+0000","last_became_active":"2024-09-13T13:06:21.814002+0000","last_became_peered":"2024-09-13T13:06:21.814002+0000","last_unstale":"2024-09-13T13:07:13.124825+0000","last_undegraded":"2024-09-13T13:07:13.124825+0000","last_fullsized":"2024-09-13T13:07:13.124825+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4689,"log_dups_size":0,"ondisk_log_size":4689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:31.181397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307298304,"num_objects":4689,"num_object_clones":0,"num_object_copies":14067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4689,"num_write_kb":300096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":315,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199241+0000","last_change":"2024-09-13T13:05:25.691233+0000","last_active":"2024-09-13T13:07:09.199241+0000","last_peered":"2024-09-13T13:07:09.199241+0000","last_clean":"2024-09-13T13:07:09.199241+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:07:09.199241+0000","last_undegraded":"2024-09-13T13:07:09.199241+0000","last_fullsized":"2024-09-13T13:07:09.199241+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:05:25.689487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:25.689487+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":301,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.197555+0000","last_change":"2024-09-13T13:07:08.562915+0000","last_active":"2024-09-13T13:07:09.197555+0000","last_peered":"2024-09-13T13:07:09.197555+0000","last_clean":"2024-09-13T13:07:09.197555+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:07:09.197555+0000","last_undegraded":"2024-09-13T13:07:09.197555+0000","last_fullsized":"2024-09-13T13:07:09.197555+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:08.562240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.562240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:17.435791+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":361,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.186104+0000","last_change":"2024-09-13T13:02:32.013587+0000","last_active":"2024-09-13T13:07:09.186104+0000","last_peered":"2024-09-13T13:07:09.186104+0000","last_clean":"2024-09-13T13:07:09.186104+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:07:09.186104+0000","last_undegraded":"2024-09-13T13:07:09.186104+0000","last_fullsized":"2024-09-13T13:07:09.186104+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:32.013505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:32.013505+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:16.072920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":281,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188176+0000","last_change":"2024-09-13T13:06:05.648648+0000","last_active":"2024-09-13T13:07:09.188176+0000","last_peered":"2024-09-13T13:07:09.188176+0000","last_clean":"2024-09-13T13:07:09.188176+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:07:09.188176+0000","last_undegraded":"2024-09-13T13:07:09.188176+0000","last_fullsized":"2024-09-13T13:07:09.188176+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:05.648589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:05.648589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.628097+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"192'4645","reported_seq":4702,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.638196+0000","last_change":"2024-09-13T13:06:22.000536+0000","last_active":"2024-09-13T13:07:13.638196+0000","last_peered":"2024-09-13T13:07:13.638196+0000","last_clean":"2024-09-13T13:07:13.638196+0000","last_became_active":"2024-09-13T13:06:22.000244+0000","last_became_peered":"2024-09-13T13:06:22.000244+0000","last_unstale":"2024-09-13T13:07:13.638196+0000","last_undegraded":"2024-09-13T13:07:13.638196+0000","last_fullsized":"2024-09-13T13:07:13.638196+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:40.851139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304414720,"num_objects":4645,"num_object_clones":0,"num_object_copies":13935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4645,"num_write_kb":297280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":314,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.197335+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:07:09.197335+0000","last_peered":"2024-09-13T13:07:09.197335+0000","last_clean":"2024-09-13T13:07:09.197335+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:07:09.197335+0000","last_undegraded":"2024-09-13T13:07:09.197335+0000","last_fullsized":"2024-09-13T13:07:09.197335+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:05.558407+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":299,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196790+0000","last_change":"2024-09-13T13:06:44.785235+0000","last_active":"2024-09-13T13:07:09.196790+0000","last_peered":"2024-09-13T13:07:09.196790+0000","last_clean":"2024-09-13T13:07:09.196790+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:07:09.196790+0000","last_undegraded":"2024-09-13T13:07:09.196790+0000","last_fullsized":"2024-09-13T13:07:09.196790+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:06:44.785017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.785017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:00.668665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184932+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:07:09.184932+0000","last_peered":"2024-09-13T13:07:09.184932+0000","last_clean":"2024-09-13T13:07:09.184932+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:07:09.184932+0000","last_undegraded":"2024-09-13T13:07:09.184932+0000","last_fullsized":"2024-09-13T13:07:09.184932+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:04.883415+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185531+0000","last_change":"2024-09-13T13:06:43.138526+0000","last_active":"2024-09-13T13:07:09.185531+0000","last_peered":"2024-09-13T13:07:09.185531+0000","last_clean":"2024-09-13T13:07:09.185531+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:07:09.185531+0000","last_undegraded":"2024-09-13T13:07:09.185531+0000","last_fullsized":"2024-09-13T13:07:09.185531+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.138328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.138328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:54.605867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"192'4562","reported_seq":4619,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.940096+0000","last_change":"2024-09-13T13:06:22.000928+0000","last_active":"2024-09-13T13:07:12.940096+0000","last_peered":"2024-09-13T13:07:12.940096+0000","last_clean":"2024-09-13T13:07:12.940096+0000","last_became_active":"2024-09-13T13:06:22.000610+0000","last_became_peered":"2024-09-13T13:06:22.000610+0000","last_unstale":"2024-09-13T13:07:12.940096+0000","last_undegraded":"2024-09-13T13:07:12.940096+0000","last_fullsized":"2024-09-13T13:07:12.940096+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4562,"log_dups_size":0,"ondisk_log_size":4562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:47.936354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298975232,"num_objects":4562,"num_object_clones":0,"num_object_copies":13686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4562,"num_write_kb":291968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":330,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:10.853834+0000","last_change":"2024-09-13T13:07:10.853834+0000","last_active":"2024-09-13T13:07:10.853834+0000","last_peered":"2024-09-13T13:07:10.853834+0000","last_clean":"2024-09-13T13:07:10.853834+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:07:10.853834+0000","last_undegraded":"2024-09-13T13:07:10.853834+0000","last_fullsized":"2024-09-13T13:07:10.853834+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:07:10.853749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.853749+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:26.687762+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199944+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:07:09.199944+0000","last_peered":"2024-09-13T13:07:09.199944+0000","last_clean":"2024-09-13T13:07:09.199944+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:07:09.199944+0000","last_undegraded":"2024-09-13T13:07:09.199944+0000","last_fullsized":"2024-09-13T13:07:09.199944+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:06.208212+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":361,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.310660+0000","last_change":"2024-09-13T13:06:48.866830+0000","last_active":"2024-09-13T13:07:12.310660+0000","last_peered":"2024-09-13T13:07:12.310660+0000","last_clean":"2024-09-13T13:07:12.310660+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:07:12.310660+0000","last_undegraded":"2024-09-13T13:07:12.310660+0000","last_fullsized":"2024-09-13T13:07:12.310660+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:06:48.866752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:48.866752+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:03.286427+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":292,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.203497+0000","last_change":"2024-09-13T13:06:09.230872+0000","last_active":"2024-09-13T13:07:09.203497+0000","last_peered":"2024-09-13T13:07:09.203497+0000","last_clean":"2024-09-13T13:07:09.203497+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:07:09.203497+0000","last_undegraded":"2024-09-13T13:07:09.203497+0000","last_fullsized":"2024-09-13T13:07:09.203497+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:09.230824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.230824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:19.719773+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"192'4628","reported_seq":4685,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.644755+0000","last_change":"2024-09-13T13:06:21.812511+0000","last_active":"2024-09-13T13:07:13.644755+0000","last_peered":"2024-09-13T13:07:13.644755+0000","last_clean":"2024-09-13T13:07:13.644755+0000","last_became_active":"2024-09-13T13:06:21.812226+0000","last_became_peered":"2024-09-13T13:06:21.812226+0000","last_unstale":"2024-09-13T13:07:13.644755+0000","last_undegraded":"2024-09-13T13:07:13.644755+0000","last_fullsized":"2024-09-13T13:07:13.644755+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:27.069143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303300608,"num_objects":4628,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4628,"num_write_kb":296192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.191532+0000","last_change":"2024-09-13T13:06:36.894642+0000","last_active":"2024-09-13T13:07:09.191532+0000","last_peered":"2024-09-13T13:07:09.191532+0000","last_clean":"2024-09-13T13:07:09.191532+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:07:09.191532+0000","last_undegraded":"2024-09-13T13:07:09.191532+0000","last_fullsized":"2024-09-13T13:07:09.191532+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:06:36.894565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:36.894565+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:53.101599+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188503+0000","last_change":"2024-09-13T13:06:36.645157+0000","last_active":"2024-09-13T13:07:09.188503+0000","last_peered":"2024-09-13T13:07:09.188503+0000","last_clean":"2024-09-13T13:07:09.188503+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:07:09.188503+0000","last_undegraded":"2024-09-13T13:07:09.188503+0000","last_fullsized":"2024-09-13T13:07:09.188503+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:36.645089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:36.645089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:55.978419+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":366,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199209+0000","last_change":"2024-09-13T13:04:02.764350+0000","last_active":"2024-09-13T13:07:09.199209+0000","last_peered":"2024-09-13T13:07:09.199209+0000","last_clean":"2024-09-13T13:07:09.199209+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:07:09.199209+0000","last_undegraded":"2024-09-13T13:07:09.199209+0000","last_fullsized":"2024-09-13T13:07:09.199209+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:52.299393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.200643+0000","last_change":"2024-09-13T13:06:46.288561+0000","last_active":"2024-09-13T13:07:09.200643+0000","last_peered":"2024-09-13T13:07:09.200643+0000","last_clean":"2024-09-13T13:07:09.200643+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:07:09.200643+0000","last_undegraded":"2024-09-13T13:07:09.200643+0000","last_fullsized":"2024-09-13T13:07:09.200643+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:46.288483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:46.288483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:47.729332+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"192'4661","reported_seq":4718,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.302999+0000","last_change":"2024-09-13T13:06:21.812509+0000","last_active":"2024-09-13T13:07:13.302999+0000","last_peered":"2024-09-13T13:07:13.302999+0000","last_clean":"2024-09-13T13:07:13.302999+0000","last_became_active":"2024-09-13T13:06:21.812208+0000","last_became_peered":"2024-09-13T13:06:21.812208+0000","last_unstale":"2024-09-13T13:07:13.302999+0000","last_undegraded":"2024-09-13T13:07:13.302999+0000","last_fullsized":"2024-09-13T13:07:13.302999+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4661,"log_dups_size":0,"ondisk_log_size":4661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:33.753674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305463296,"num_objects":4661,"num_object_clones":0,"num_object_copies":13983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4661,"num_write_kb":298304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":325,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199356+0000","last_change":"2024-09-13T13:06:37.265029+0000","last_active":"2024-09-13T13:07:09.199356+0000","last_peered":"2024-09-13T13:07:09.199356+0000","last_clean":"2024-09-13T13:07:09.199356+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:07:09.199356+0000","last_undegraded":"2024-09-13T13:07:09.199356+0000","last_fullsized":"2024-09-13T13:07:09.199356+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:06:37.264946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:37.264946+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:47.148663+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":278,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.194302+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:07:09.194302+0000","last_peered":"2024-09-13T13:07:09.194302+0000","last_clean":"2024-09-13T13:07:09.194302+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:07:09.194302+0000","last_undegraded":"2024-09-13T13:07:09.194302+0000","last_fullsized":"2024-09-13T13:07:09.194302+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:33.900862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":247,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.190891+0000","last_change":"2024-09-13T13:06:08.941867+0000","last_active":"2024-09-13T13:07:09.190891+0000","last_peered":"2024-09-13T13:07:09.190891+0000","last_clean":"2024-09-13T13:07:09.190891+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:07:09.190891+0000","last_undegraded":"2024-09-13T13:07:09.190891+0000","last_fullsized":"2024-09-13T13:07:09.190891+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:08.941799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:08.941799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:32.827042+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":301,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189561+0000","last_change":"2024-09-13T13:06:43.969251+0000","last_active":"2024-09-13T13:07:09.189561+0000","last_peered":"2024-09-13T13:07:09.189561+0000","last_clean":"2024-09-13T13:07:09.189561+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:07:09.189561+0000","last_undegraded":"2024-09-13T13:07:09.189561+0000","last_fullsized":"2024-09-13T13:07:09.189561+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.969176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.969176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:00.184869+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"192'4505","reported_seq":4562,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.931963+0000","last_change":"2024-09-13T13:06:22.000658+0000","last_active":"2024-09-13T13:07:12.931963+0000","last_peered":"2024-09-13T13:07:12.931963+0000","last_clean":"2024-09-13T13:07:12.931963+0000","last_became_active":"2024-09-13T13:06:22.000261+0000","last_became_peered":"2024-09-13T13:06:22.000261+0000","last_unstale":"2024-09-13T13:07:12.931963+0000","last_undegraded":"2024-09-13T13:07:12.931963+0000","last_fullsized":"2024-09-13T13:07:12.931963+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:41.824631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295239680,"num_objects":4505,"num_object_clones":0,"num_object_copies":13515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4505,"num_write_kb":288320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":413,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.194207+0000","last_change":"2024-09-13T13:05:43.787063+0000","last_active":"2024-09-13T13:07:09.194207+0000","last_peered":"2024-09-13T13:07:09.194207+0000","last_clean":"2024-09-13T13:07:09.194207+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:07:09.194207+0000","last_undegraded":"2024-09-13T13:07:09.194207+0000","last_fullsized":"2024-09-13T13:07:09.194207+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:05:43.786273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:43.786273+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":305,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184731+0000","last_change":"2024-09-13T13:05:45.178086+0000","last_active":"2024-09-13T13:07:09.184731+0000","last_peered":"2024-09-13T13:07:09.184731+0000","last_clean":"2024-09-13T13:07:09.184731+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:07:09.184731+0000","last_undegraded":"2024-09-13T13:07:09.184731+0000","last_fullsized":"2024-09-13T13:07:09.184731+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:45.178025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.178025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":304,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188893+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:07:09.188893+0000","last_peered":"2024-09-13T13:07:09.188893+0000","last_clean":"2024-09-13T13:07:09.188893+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:07:09.188893+0000","last_undegraded":"2024-09-13T13:07:09.188893+0000","last_fullsized":"2024-09-13T13:07:09.188893+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:27.623207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199725+0000","last_change":"2024-09-13T13:06:11.189922+0000","last_active":"2024-09-13T13:07:09.199725+0000","last_peered":"2024-09-13T13:07:09.199725+0000","last_clean":"2024-09-13T13:07:09.199725+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:07:09.199725+0000","last_undegraded":"2024-09-13T13:07:09.199725+0000","last_fullsized":"2024-09-13T13:07:09.199725+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:11.189874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:11.189874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:21.236595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":281,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:11.551545+0000","last_change":"2024-09-13T13:07:11.551545+0000","last_active":"2024-09-13T13:07:11.551545+0000","last_peered":"2024-09-13T13:07:11.551545+0000","last_clean":"2024-09-13T13:07:11.551545+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:07:11.551545+0000","last_undegraded":"2024-09-13T13:07:11.551545+0000","last_fullsized":"2024-09-13T13:07:11.551545+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.551460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.551460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:33.164720+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"192'4595","reported_seq":4652,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.276217+0000","last_change":"2024-09-13T13:06:21.813204+0000","last_active":"2024-09-13T13:07:13.276217+0000","last_peered":"2024-09-13T13:07:13.276217+0000","last_clean":"2024-09-13T13:07:13.276217+0000","last_became_active":"2024-09-13T13:06:21.812869+0000","last_became_peered":"2024-09-13T13:06:21.812869+0000","last_unstale":"2024-09-13T13:07:13.276217+0000","last_undegraded":"2024-09-13T13:07:13.276217+0000","last_fullsized":"2024-09-13T13:07:13.276217+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4595,"log_dups_size":0,"ondisk_log_size":4595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:33.863904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301137920,"num_objects":4595,"num_object_clones":0,"num_object_copies":13785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4595,"num_write_kb":294080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":309,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199342+0000","last_change":"2024-09-13T13:03:14.271569+0000","last_active":"2024-09-13T13:07:09.199342+0000","last_peered":"2024-09-13T13:07:09.199342+0000","last_clean":"2024-09-13T13:07:09.199342+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:07:09.199342+0000","last_undegraded":"2024-09-13T13:07:09.199342+0000","last_fullsized":"2024-09-13T13:07:09.199342+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:03:14.271480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.271480+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:59.122679+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":306,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.192349+0000","last_change":"2024-09-13T13:05:36.692159+0000","last_active":"2024-09-13T13:07:09.192349+0000","last_peered":"2024-09-13T13:07:09.192349+0000","last_clean":"2024-09-13T13:07:09.192349+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:07:09.192349+0000","last_undegraded":"2024-09-13T13:07:09.192349+0000","last_fullsized":"2024-09-13T13:07:09.192349+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:36.692092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:36.692092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":309,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.186216+0000","last_change":"2024-09-13T13:06:50.629016+0000","last_active":"2024-09-13T13:07:09.186216+0000","last_peered":"2024-09-13T13:07:09.186216+0000","last_clean":"2024-09-13T13:07:09.186216+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:07:09.186216+0000","last_undegraded":"2024-09-13T13:07:09.186216+0000","last_fullsized":"2024-09-13T13:07:09.186216+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:06:50.628774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.628774+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:14.442004+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":278,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.192649+0000","last_change":"2024-09-13T13:03:14.282866+0000","last_active":"2024-09-13T13:07:09.192649+0000","last_peered":"2024-09-13T13:07:09.192649+0000","last_clean":"2024-09-13T13:07:09.192649+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:07:09.192649+0000","last_undegraded":"2024-09-13T13:07:09.192649+0000","last_fullsized":"2024-09-13T13:07:09.192649+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:27.402491+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.204674+0000","last_change":"2024-09-13T13:04:02.751582+0000","last_active":"2024-09-13T13:07:09.204674+0000","last_peered":"2024-09-13T13:07:09.204674+0000","last_clean":"2024-09-13T13:07:09.204674+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:07:09.204674+0000","last_undegraded":"2024-09-13T13:07:09.204674+0000","last_fullsized":"2024-09-13T13:07:09.204674+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.994729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.994729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:19.352065+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"192'4593","reported_seq":4650,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.146022+0000","last_change":"2024-09-13T13:06:21.815090+0000","last_active":"2024-09-13T13:07:13.146022+0000","last_peered":"2024-09-13T13:07:13.146022+0000","last_clean":"2024-09-13T13:07:13.146022+0000","last_became_active":"2024-09-13T13:06:21.814772+0000","last_became_peered":"2024-09-13T13:06:21.814772+0000","last_unstale":"2024-09-13T13:07:13.146022+0000","last_undegraded":"2024-09-13T13:07:13.146022+0000","last_fullsized":"2024-09-13T13:07:13.146022+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:41.464053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301006848,"num_objects":4593,"num_object_clones":0,"num_object_copies":13779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":293952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":307,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:11.169884+0000","last_change":"2024-09-13T13:07:11.169884+0000","last_active":"2024-09-13T13:07:11.169884+0000","last_peered":"2024-09-13T13:07:11.169884+0000","last_clean":"2024-09-13T13:07:11.169884+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:07:11.169884+0000","last_undegraded":"2024-09-13T13:07:11.169884+0000","last_fullsized":"2024-09-13T13:07:11.169884+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:07:11.169656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.169656+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:17.059187+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199073+0000","last_change":"2024-09-13T13:06:42.310845+0000","last_active":"2024-09-13T13:07:09.199073+0000","last_peered":"2024-09-13T13:07:09.199073+0000","last_clean":"2024-09-13T13:07:09.199073+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:07:09.199073+0000","last_undegraded":"2024-09-13T13:07:09.199073+0000","last_fullsized":"2024-09-13T13:07:09.199073+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:42.310776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:42.310776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:55.851649+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":668,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:11.867149+0000","last_change":"2024-09-13T13:06:07.290837+0000","last_active":"2024-09-13T13:07:11.867149+0000","last_peered":"2024-09-13T13:07:11.867149+0000","last_clean":"2024-09-13T13:07:11.867149+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:07:11.867149+0000","last_undegraded":"2024-09-13T13:07:11.867149+0000","last_fullsized":"2024-09-13T13:07:11.867149+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:06:07.290717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:07.290717+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:20.222760+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":365,"num_read_kb":360,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":277,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.190230+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:07:09.190230+0000","last_peered":"2024-09-13T13:07:09.190230+0000","last_clean":"2024-09-13T13:07:09.190230+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:07:09.190230+0000","last_undegraded":"2024-09-13T13:07:09.190230+0000","last_fullsized":"2024-09-13T13:07:09.190230+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:25.654670+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.190664+0000","last_change":"2024-09-13T13:05:41.753575+0000","last_active":"2024-09-13T13:07:09.190664+0000","last_peered":"2024-09-13T13:07:09.190664+0000","last_clean":"2024-09-13T13:07:09.190664+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:07:09.190664+0000","last_undegraded":"2024-09-13T13:07:09.190664+0000","last_fullsized":"2024-09-13T13:07:09.190664+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:41.753502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:41.753502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"192'4780","reported_seq":4837,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:15.910482+0000","last_change":"2024-09-13T13:06:21.814495+0000","last_active":"2024-09-13T13:07:15.910482+0000","last_peered":"2024-09-13T13:07:15.910482+0000","last_clean":"2024-09-13T13:07:15.910482+0000","last_became_active":"2024-09-13T13:06:21.814144+0000","last_became_peered":"2024-09-13T13:06:21.814144+0000","last_unstale":"2024-09-13T13:07:15.910482+0000","last_undegraded":"2024-09-13T13:07:15.910482+0000","last_fullsized":"2024-09-13T13:07:15.910482+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:21.059266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313262080,"num_objects":4780,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":305920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":254,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.191537+0000","last_change":"2024-09-13T13:06:51.193749+0000","last_active":"2024-09-13T13:07:09.191537+0000","last_peered":"2024-09-13T13:07:09.191537+0000","last_clean":"2024-09-13T13:07:09.191537+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:07:09.191537+0000","last_undegraded":"2024-09-13T13:07:09.191537+0000","last_fullsized":"2024-09-13T13:07:09.191537+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'18","last_scrub_stamp":"2024-09-13T13:06:51.193678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.193678+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:13.330359+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":304,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.193158+0000","last_change":"2024-09-13T13:06:10.715371+0000","last_active":"2024-09-13T13:07:09.193158+0000","last_peered":"2024-09-13T13:07:09.193158+0000","last_clean":"2024-09-13T13:07:09.193158+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:07:09.193158+0000","last_undegraded":"2024-09-13T13:07:09.193158+0000","last_fullsized":"2024-09-13T13:07:09.193158+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:10.715313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.715313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:21.700969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":350,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.203733+0000","last_change":"2024-09-13T13:06:39.564599+0000","last_active":"2024-09-13T13:07:09.203733+0000","last_peered":"2024-09-13T13:07:09.203733+0000","last_clean":"2024-09-13T13:07:09.203733+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:07:09.203733+0000","last_undegraded":"2024-09-13T13:07:09.203733+0000","last_fullsized":"2024-09-13T13:07:09.203733+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:06:39.564549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:06:39.564549+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:03.134331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.206625+0000","last_change":"2024-09-13T13:06:05.308473+0000","last_active":"2024-09-13T13:07:09.206625+0000","last_peered":"2024-09-13T13:07:09.206625+0000","last_clean":"2024-09-13T13:07:09.206625+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:07:09.206625+0000","last_undegraded":"2024-09-13T13:07:09.206625+0000","last_fullsized":"2024-09-13T13:07:09.206625+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:05.308427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:05.308427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:14.135834+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":289,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189794+0000","last_change":"2024-09-13T13:06:46.624345+0000","last_active":"2024-09-13T13:07:09.189794+0000","last_peered":"2024-09-13T13:07:09.189794+0000","last_clean":"2024-09-13T13:07:09.189794+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:07:09.189794+0000","last_undegraded":"2024-09-13T13:07:09.189794+0000","last_fullsized":"2024-09-13T13:07:09.189794+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:46.624238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:46.624238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:10.518686+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"192'4576","reported_seq":4633,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.927778+0000","last_change":"2024-09-13T13:06:21.812371+0000","last_active":"2024-09-13T13:07:12.927778+0000","last_peered":"2024-09-13T13:07:12.927778+0000","last_clean":"2024-09-13T13:07:12.927778+0000","last_became_active":"2024-09-13T13:06:21.812061+0000","last_became_peered":"2024-09-13T13:06:21.812061+0000","last_unstale":"2024-09-13T13:07:12.927778+0000","last_undegraded":"2024-09-13T13:07:12.927778+0000","last_fullsized":"2024-09-13T13:07:12.927778+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4576,"log_dups_size":0,"ondisk_log_size":4576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:23.624139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299892736,"num_objects":4576,"num_object_clones":0,"num_object_copies":13728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4576,"num_write_kb":292864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":312,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.186149+0000","last_change":"2024-09-13T13:04:02.832992+0000","last_active":"2024-09-13T13:07:09.186149+0000","last_peered":"2024-09-13T13:07:09.186149+0000","last_clean":"2024-09-13T13:07:09.186149+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:07:09.186149+0000","last_undegraded":"2024-09-13T13:07:09.186149+0000","last_fullsized":"2024-09-13T13:07:09.186149+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:04:02.832935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:02.832935+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:52.072763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185797+0000","last_change":"2024-09-13T13:02:45.015634+0000","last_active":"2024-09-13T13:07:09.185797+0000","last_peered":"2024-09-13T13:07:09.185797+0000","last_clean":"2024-09-13T13:07:09.185797+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:07:09.185797+0000","last_undegraded":"2024-09-13T13:07:09.185797+0000","last_fullsized":"2024-09-13T13:07:09.185797+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:45.015577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.015577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:19.083414+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":296,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189625+0000","last_change":"2024-09-13T13:06:11.637531+0000","last_active":"2024-09-13T13:07:09.189625+0000","last_peered":"2024-09-13T13:07:09.189625+0000","last_clean":"2024-09-13T13:07:09.189625+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:07:09.189625+0000","last_undegraded":"2024-09-13T13:07:09.189625+0000","last_fullsized":"2024-09-13T13:07:09.189625+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:11.637478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:11.637478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:12.457876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196150+0000","last_change":"2024-09-13T13:04:02.757378+0000","last_active":"2024-09-13T13:07:09.196150+0000","last_peered":"2024-09-13T13:07:09.196150+0000","last_clean":"2024-09-13T13:07:09.196150+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:07:09.196150+0000","last_undegraded":"2024-09-13T13:07:09.196150+0000","last_fullsized":"2024-09-13T13:07:09.196150+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.213470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.213470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:22.648573+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"192'4657","reported_seq":4714,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.153458+0000","last_change":"2024-09-13T13:06:21.811878+0000","last_active":"2024-09-13T13:07:13.153458+0000","last_peered":"2024-09-13T13:07:13.153458+0000","last_clean":"2024-09-13T13:07:13.153458+0000","last_became_active":"2024-09-13T13:06:21.811526+0000","last_became_peered":"2024-09-13T13:06:21.811526+0000","last_unstale":"2024-09-13T13:07:13.153458+0000","last_undegraded":"2024-09-13T13:07:13.153458+0000","last_fullsized":"2024-09-13T13:07:13.153458+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4657,"log_dups_size":0,"ondisk_log_size":4657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:41.893735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305201152,"num_objects":4657,"num_object_clones":0,"num_object_copies":13971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4657,"num_write_kb":298048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":318,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184395+0000","last_change":"2024-09-13T13:05:34.166464+0000","last_active":"2024-09-13T13:07:09.184395+0000","last_peered":"2024-09-13T13:07:09.184395+0000","last_clean":"2024-09-13T13:07:09.184395+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:07:09.184395+0000","last_undegraded":"2024-09-13T13:07:09.184395+0000","last_fullsized":"2024-09-13T13:07:09.184395+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:00.016282+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":263,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.191150+0000","last_change":"2024-09-13T13:05:44.119947+0000","last_active":"2024-09-13T13:07:09.191150+0000","last_peered":"2024-09-13T13:07:09.191150+0000","last_clean":"2024-09-13T13:07:09.191150+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:07:09.191150+0000","last_undegraded":"2024-09-13T13:07:09.191150+0000","last_fullsized":"2024-09-13T13:07:09.191150+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:44.119890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.119890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:53.147402+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":278,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.191077+0000","last_change":"2024-09-13T13:06:37.854805+0000","last_active":"2024-09-13T13:07:09.191077+0000","last_peered":"2024-09-13T13:07:09.191077+0000","last_clean":"2024-09-13T13:07:09.191077+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:07:09.191077+0000","last_undegraded":"2024-09-13T13:07:09.191077+0000","last_fullsized":"2024-09-13T13:07:09.191077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:37.854730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:37.854730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:40.674082+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188758+0000","last_change":"2024-09-13T13:05:35.310768+0000","last_active":"2024-09-13T13:07:09.188758+0000","last_peered":"2024-09-13T13:07:09.188758+0000","last_clean":"2024-09-13T13:07:09.188758+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:07:09.188758+0000","last_undegraded":"2024-09-13T13:07:09.188758+0000","last_fullsized":"2024-09-13T13:07:09.188758+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:35.310700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.310700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:02.424290+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"192'4554","reported_seq":4610,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.319449+0000","last_change":"2024-09-13T13:06:21.813258+0000","last_active":"2024-09-13T13:07:13.319449+0000","last_peered":"2024-09-13T13:07:13.319449+0000","last_clean":"2024-09-13T13:07:13.319449+0000","last_became_active":"2024-09-13T13:06:21.812966+0000","last_became_peered":"2024-09-13T13:06:21.812966+0000","last_unstale":"2024-09-13T13:07:13.319449+0000","last_undegraded":"2024-09-13T13:07:13.319449+0000","last_fullsized":"2024-09-13T13:07:13.319449+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4554,"log_dups_size":0,"ondisk_log_size":4554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:49.824314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298450944,"num_objects":4554,"num_object_clones":0,"num_object_copies":13662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4554,"num_write_kb":291456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":316,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.183974+0000","last_change":"2024-09-13T13:06:40.132360+0000","last_active":"2024-09-13T13:07:09.183974+0000","last_peered":"2024-09-13T13:07:09.183974+0000","last_clean":"2024-09-13T13:07:09.183974+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:07:09.183974+0000","last_undegraded":"2024-09-13T13:07:09.183974+0000","last_fullsized":"2024-09-13T13:07:09.183974+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:06:40.132273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:40.132273+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:49.361496+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.195626+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:07:09.195626+0000","last_peered":"2024-09-13T13:07:09.195626+0000","last_clean":"2024-09-13T13:07:09.195626+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:07:09.195626+0000","last_undegraded":"2024-09-13T13:07:09.195626+0000","last_fullsized":"2024-09-13T13:07:09.195626+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:06.582017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":369,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.187805+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:07:09.187805+0000","last_peered":"2024-09-13T13:07:09.187805+0000","last_clean":"2024-09-13T13:07:09.187805+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:07:09.187805+0000","last_undegraded":"2024-09-13T13:07:09.187805+0000","last_fullsized":"2024-09-13T13:07:09.187805+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:05.357132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":274,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185629+0000","last_change":"2024-09-13T13:02:33.997780+0000","last_active":"2024-09-13T13:07:09.185629+0000","last_peered":"2024-09-13T13:07:09.185629+0000","last_clean":"2024-09-13T13:07:09.185629+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:07:09.185629+0000","last_undegraded":"2024-09-13T13:07:09.185629+0000","last_fullsized":"2024-09-13T13:07:09.185629+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.997740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.997740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:27.073331+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"192'4690","reported_seq":4747,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.138761+0000","last_change":"2024-09-13T13:06:21.814825+0000","last_active":"2024-09-13T13:07:13.138761+0000","last_peered":"2024-09-13T13:07:13.138761+0000","last_clean":"2024-09-13T13:07:13.138761+0000","last_became_active":"2024-09-13T13:06:21.814433+0000","last_became_peered":"2024-09-13T13:06:21.814433+0000","last_unstale":"2024-09-13T13:07:13.138761+0000","last_undegraded":"2024-09-13T13:07:13.138761+0000","last_fullsized":"2024-09-13T13:07:13.138761+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:37.466611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307363840,"num_objects":4690,"num_object_clones":0,"num_object_copies":14070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4690,"num_write_kb":300160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199264+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:07:09.199264+0000","last_peered":"2024-09-13T13:07:09.199264+0000","last_clean":"2024-09-13T13:07:09.199264+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:07:09.199264+0000","last_undegraded":"2024-09-13T13:07:09.199264+0000","last_fullsized":"2024-09-13T13:07:09.199264+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:04.211883+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":304,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189710+0000","last_change":"2024-09-13T13:05:37.520655+0000","last_active":"2024-09-13T13:07:09.189710+0000","last_peered":"2024-09-13T13:07:09.189710+0000","last_clean":"2024-09-13T13:07:09.189710+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:07:09.189710+0000","last_undegraded":"2024-09-13T13:07:09.189710+0000","last_fullsized":"2024-09-13T13:07:09.189710+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:37.520603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:37.520603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:57.639968+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":302,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:11.400066+0000","last_change":"2024-09-13T13:07:11.400066+0000","last_active":"2024-09-13T13:07:11.400066+0000","last_peered":"2024-09-13T13:07:11.400066+0000","last_clean":"2024-09-13T13:07:11.400066+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:07:11.400066+0000","last_undegraded":"2024-09-13T13:07:11.400066+0000","last_fullsized":"2024-09-13T13:07:11.400066+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.399840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.399840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:35.796029+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":265,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.198981+0000","last_change":"2024-09-13T13:01:08.211229+0000","last_active":"2024-09-13T13:07:09.198981+0000","last_peered":"2024-09-13T13:07:09.198981+0000","last_clean":"2024-09-13T13:07:09.198981+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:07:09.198981+0000","last_undegraded":"2024-09-13T13:07:09.198981+0000","last_fullsized":"2024-09-13T13:07:09.198981+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:54.303877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"192'4739","reported_seq":4795,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:13.655088+0000","last_change":"2024-09-13T13:06:22.000224+0000","last_active":"2024-09-13T13:07:13.655088+0000","last_peered":"2024-09-13T13:07:13.655088+0000","last_clean":"2024-09-13T13:07:13.655088+0000","last_became_active":"2024-09-13T13:06:21.999889+0000","last_became_peered":"2024-09-13T13:06:21.999889+0000","last_unstale":"2024-09-13T13:07:13.655088+0000","last_undegraded":"2024-09-13T13:07:13.655088+0000","last_fullsized":"2024-09-13T13:07:13.655088+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:22.321515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310575104,"num_objects":4739,"num_object_clones":0,"num_object_copies":14217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4739,"num_write_kb":303296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.190969+0000","last_change":"2024-09-13T13:06:06.912256+0000","last_active":"2024-09-13T13:07:09.190969+0000","last_peered":"2024-09-13T13:07:09.190969+0000","last_clean":"2024-09-13T13:07:09.190969+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:07:09.190969+0000","last_undegraded":"2024-09-13T13:07:09.190969+0000","last_fullsized":"2024-09-13T13:07:09.190969+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:06:06.912141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:06.912141+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:19.842077+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":304,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.205110+0000","last_change":"2024-09-13T13:06:14.296977+0000","last_active":"2024-09-13T13:07:09.205110+0000","last_peered":"2024-09-13T13:07:09.205110+0000","last_clean":"2024-09-13T13:07:09.205110+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:07:09.205110+0000","last_undegraded":"2024-09-13T13:07:09.205110+0000","last_fullsized":"2024-09-13T13:07:09.205110+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:14.296913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:14.296913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:41.339805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.186402+0000","last_change":"2024-09-13T13:04:47.927556+0000","last_active":"2024-09-13T13:07:09.186402+0000","last_peered":"2024-09-13T13:07:09.186402+0000","last_clean":"2024-09-13T13:07:09.186402+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:07:09.186402+0000","last_undegraded":"2024-09-13T13:07:09.186402+0000","last_fullsized":"2024-09-13T13:07:09.186402+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:47.927432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:47.927432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:17.459384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":281,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.203643+0000","last_change":"2024-09-13T13:06:15.283664+0000","last_active":"2024-09-13T13:07:09.203643+0000","last_peered":"2024-09-13T13:07:09.203643+0000","last_clean":"2024-09-13T13:07:09.203643+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:07:09.203643+0000","last_undegraded":"2024-09-13T13:07:09.203643+0000","last_fullsized":"2024-09-13T13:07:09.203643+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:15.283602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:15.283602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:41.496057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"192'4901","reported_seq":4958,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:15.911102+0000","last_change":"2024-09-13T13:06:22.002144+0000","last_active":"2024-09-13T13:07:15.911102+0000","last_peered":"2024-09-13T13:07:15.911102+0000","last_clean":"2024-09-13T13:07:15.911102+0000","last_became_active":"2024-09-13T13:06:22.001404+0000","last_became_peered":"2024-09-13T13:06:22.001404+0000","last_unstale":"2024-09-13T13:07:15.911102+0000","last_undegraded":"2024-09-13T13:07:15.911102+0000","last_fullsized":"2024-09-13T13:07:15.911102+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4901,"log_dups_size":0,"ondisk_log_size":4901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:38.621035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321191936,"num_objects":4901,"num_object_clones":0,"num_object_copies":14703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":313664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":305,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.185587+0000","last_change":"2024-09-13T13:02:28.990089+0000","last_active":"2024-09-13T13:07:09.185587+0000","last_peered":"2024-09-13T13:07:09.185587+0000","last_clean":"2024-09-13T13:07:09.185587+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:07:09.185587+0000","last_undegraded":"2024-09-13T13:07:09.185587+0000","last_fullsized":"2024-09-13T13:07:09.185587+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:02:28.990005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.990005+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:26.096510+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":301,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.204406+0000","last_change":"2024-09-13T13:05:29.345863+0000","last_active":"2024-09-13T13:07:09.204406+0000","last_peered":"2024-09-13T13:07:09.204406+0000","last_clean":"2024-09-13T13:07:09.204406+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:07:09.204406+0000","last_undegraded":"2024-09-13T13:07:09.204406+0000","last_fullsized":"2024-09-13T13:07:09.204406+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.345796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.345796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:03.542269+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.186171+0000","last_change":"2024-09-13T13:06:42.450669+0000","last_active":"2024-09-13T13:07:09.186171+0000","last_peered":"2024-09-13T13:07:09.186171+0000","last_clean":"2024-09-13T13:07:09.186171+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:07:09.186171+0000","last_undegraded":"2024-09-13T13:07:09.186171+0000","last_fullsized":"2024-09-13T13:07:09.186171+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:42.450598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:42.450598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:10.729355+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.195357+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:07:09.195357+0000","last_peered":"2024-09-13T13:07:09.195357+0000","last_clean":"2024-09-13T13:07:09.195357+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:07:09.195357+0000","last_undegraded":"2024-09-13T13:07:09.195357+0000","last_fullsized":"2024-09-13T13:07:09.195357+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:00.622043+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"192'4584","reported_seq":4641,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.921184+0000","last_change":"2024-09-13T13:06:21.816492+0000","last_active":"2024-09-13T13:07:12.921184+0000","last_peered":"2024-09-13T13:07:12.921184+0000","last_clean":"2024-09-13T13:07:12.921184+0000","last_became_active":"2024-09-13T13:06:21.816096+0000","last_became_peered":"2024-09-13T13:06:21.816096+0000","last_unstale":"2024-09-13T13:07:12.921184+0000","last_undegraded":"2024-09-13T13:07:12.921184+0000","last_fullsized":"2024-09-13T13:07:12.921184+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4584,"log_dups_size":0,"ondisk_log_size":4584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.210645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300417024,"num_objects":4584,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4584,"num_write_kb":293376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":317,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.187951+0000","last_change":"2024-09-13T13:04:52.666294+0000","last_active":"2024-09-13T13:07:09.187951+0000","last_peered":"2024-09-13T13:07:09.187951+0000","last_clean":"2024-09-13T13:07:09.187951+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:07:09.187951+0000","last_undegraded":"2024-09-13T13:07:09.187951+0000","last_fullsized":"2024-09-13T13:07:09.187951+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:04:52.666222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:52.666222+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:02.658270+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":313,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.204184+0000","last_change":"2024-09-13T13:06:51.627764+0000","last_active":"2024-09-13T13:07:09.204184+0000","last_peered":"2024-09-13T13:07:09.204184+0000","last_clean":"2024-09-13T13:07:09.204184+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:07:09.204184+0000","last_undegraded":"2024-09-13T13:07:09.204184+0000","last_fullsized":"2024-09-13T13:07:09.204184+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:06:51.627691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.627691+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:16.124383+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":377,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.202878+0000","last_change":"2024-09-13T13:04:02.759553+0000","last_active":"2024-09-13T13:07:09.202878+0000","last_peered":"2024-09-13T13:07:09.202878+0000","last_clean":"2024-09-13T13:07:09.202878+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:07:09.202878+0000","last_undegraded":"2024-09-13T13:07:09.202878+0000","last_fullsized":"2024-09-13T13:07:09.202878+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:45.959285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.959285+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:28.444715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188662+0000","last_change":"2024-09-13T13:04:02.757776+0000","last_active":"2024-09-13T13:07:09.188662+0000","last_peered":"2024-09-13T13:07:09.188662+0000","last_clean":"2024-09-13T13:07:09.188662+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:07:09.188662+0000","last_undegraded":"2024-09-13T13:07:09.188662+0000","last_fullsized":"2024-09-13T13:07:09.188662+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:28.529954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.529954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.661828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":271,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:10.058616+0000","last_change":"2024-09-13T13:07:10.058616+0000","last_active":"2024-09-13T13:07:10.058616+0000","last_peered":"2024-09-13T13:07:10.058616+0000","last_clean":"2024-09-13T13:07:10.058616+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:07:10.058616+0000","last_undegraded":"2024-09-13T13:07:10.058616+0000","last_fullsized":"2024-09-13T13:07:10.058616+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:10.058555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.058555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:35.057188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":290,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196607+0000","last_change":"2024-09-13T13:03:14.504339+0000","last_active":"2024-09-13T13:07:09.196607+0000","last_peered":"2024-09-13T13:07:09.196607+0000","last_clean":"2024-09-13T13:07:09.196607+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:07:09.196607+0000","last_undegraded":"2024-09-13T13:07:09.196607+0000","last_fullsized":"2024-09-13T13:07:09.196607+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:03:14.504263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.504263+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:29.774489+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189206+0000","last_change":"2024-09-13T13:06:51.640678+0000","last_active":"2024-09-13T13:07:09.189206+0000","last_peered":"2024-09-13T13:07:09.189206+0000","last_clean":"2024-09-13T13:07:09.189206+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:07:09.189206+0000","last_undegraded":"2024-09-13T13:07:09.189206+0000","last_fullsized":"2024-09-13T13:07:09.189206+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:51.640617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.640617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:10.913640+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":300,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:11.584285+0000","last_change":"2024-09-13T13:07:11.584285+0000","last_active":"2024-09-13T13:07:11.584285+0000","last_peered":"2024-09-13T13:07:11.584285+0000","last_clean":"2024-09-13T13:07:11.584285+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:07:11.584285+0000","last_undegraded":"2024-09-13T13:07:11.584285+0000","last_fullsized":"2024-09-13T13:07:11.584285+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.584244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.584244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:24.783280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199855+0000","last_change":"2024-09-13T13:02:42.589951+0000","last_active":"2024-09-13T13:07:09.199855+0000","last_peered":"2024-09-13T13:07:09.199855+0000","last_clean":"2024-09-13T13:07:09.199855+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:07:09.199855+0000","last_undegraded":"2024-09-13T13:07:09.199855+0000","last_fullsized":"2024-09-13T13:07:09.199855+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:42.589866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.589866+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:24.328386+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":296,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.204554+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:07:09.204554+0000","last_peered":"2024-09-13T13:07:09.204554+0000","last_clean":"2024-09-13T13:07:09.204554+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:07:09.204554+0000","last_undegraded":"2024-09-13T13:07:09.204554+0000","last_fullsized":"2024-09-13T13:07:09.204554+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:31.493583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":277,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196432+0000","last_change":"2024-09-13T13:05:42.728927+0000","last_active":"2024-09-13T13:07:09.196432+0000","last_peered":"2024-09-13T13:07:09.196432+0000","last_clean":"2024-09-13T13:07:09.196432+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:07:09.196432+0000","last_undegraded":"2024-09-13T13:07:09.196432+0000","last_fullsized":"2024-09-13T13:07:09.196432+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:42.728881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:42.728881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.187173+0000","last_change":"2024-09-13T13:05:45.343713+0000","last_active":"2024-09-13T13:07:09.187173+0000","last_peered":"2024-09-13T13:07:09.187173+0000","last_clean":"2024-09-13T13:07:09.187173+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:07:09.187173+0000","last_undegraded":"2024-09-13T13:07:09.187173+0000","last_fullsized":"2024-09-13T13:07:09.187173+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:45.343638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:45.343638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:54.569396+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188421+0000","last_change":"2024-09-13T13:04:53.687731+0000","last_active":"2024-09-13T13:07:09.188421+0000","last_peered":"2024-09-13T13:07:09.188421+0000","last_clean":"2024-09-13T13:07:09.188421+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:07:09.188421+0000","last_undegraded":"2024-09-13T13:07:09.188421+0000","last_fullsized":"2024-09-13T13:07:09.188421+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:23.655078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199146+0000","last_change":"2024-09-13T13:06:41.308449+0000","last_active":"2024-09-13T13:07:09.199146+0000","last_peered":"2024-09-13T13:07:09.199146+0000","last_clean":"2024-09-13T13:07:09.199146+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:07:09.199146+0000","last_undegraded":"2024-09-13T13:07:09.199146+0000","last_fullsized":"2024-09-13T13:07:09.199146+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:41.308367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:41.308367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:57.315125+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":352,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196396+0000","last_change":"2024-09-13T13:01:06.210251+0000","last_active":"2024-09-13T13:07:09.196396+0000","last_peered":"2024-09-13T13:07:09.196396+0000","last_clean":"2024-09-13T13:07:09.196396+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:07:09.196396+0000","last_undegraded":"2024-09-13T13:07:09.196396+0000","last_fullsized":"2024-09-13T13:07:09.196396+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:01.574770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184167+0000","last_change":"2024-09-13T13:06:09.144962+0000","last_active":"2024-09-13T13:07:09.184167+0000","last_peered":"2024-09-13T13:07:09.184167+0000","last_clean":"2024-09-13T13:07:09.184167+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:07:09.184167+0000","last_undegraded":"2024-09-13T13:07:09.184167+0000","last_fullsized":"2024-09-13T13:07:09.184167+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:09.144898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.144898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:13.405762+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":296,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.196608+0000","last_change":"2024-09-13T13:02:37.222400+0000","last_active":"2024-09-13T13:07:09.196608+0000","last_peered":"2024-09-13T13:07:09.196608+0000","last_clean":"2024-09-13T13:07:09.196608+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:07:09.196608+0000","last_undegraded":"2024-09-13T13:07:09.196608+0000","last_fullsized":"2024-09-13T13:07:09.196608+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:02:37.222329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:37.222329+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:20.683338+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.203771+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:07:09.203771+0000","last_peered":"2024-09-13T13:07:09.203771+0000","last_clean":"2024-09-13T13:07:09.203771+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:07:09.203771+0000","last_undegraded":"2024-09-13T13:07:09.203771+0000","last_fullsized":"2024-09-13T13:07:09.203771+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:59.571619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":384,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188365+0000","last_change":"2024-09-13T13:07:08.781005+0000","last_active":"2024-09-13T13:07:09.188365+0000","last_peered":"2024-09-13T13:07:09.188365+0000","last_clean":"2024-09-13T13:07:09.188365+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:07:09.188365+0000","last_undegraded":"2024-09-13T13:07:09.188365+0000","last_fullsized":"2024-09-13T13:07:09.188365+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:07:08.780952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.780952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:38.355148+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199351+0000","last_change":"2024-09-13T13:02:40.615749+0000","last_active":"2024-09-13T13:07:09.199351+0000","last_peered":"2024-09-13T13:07:09.199351+0000","last_clean":"2024-09-13T13:07:09.199351+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:07:09.199351+0000","last_undegraded":"2024-09-13T13:07:09.199351+0000","last_fullsized":"2024-09-13T13:07:09.199351+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:40.615674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:40.615674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:03.206332+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":320,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184450+0000","last_change":"2024-09-13T13:05:26.186899+0000","last_active":"2024-09-13T13:07:09.184450+0000","last_peered":"2024-09-13T13:07:09.184450+0000","last_clean":"2024-09-13T13:07:09.184450+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:07:09.184450+0000","last_undegraded":"2024-09-13T13:07:09.184450+0000","last_fullsized":"2024-09-13T13:07:09.184450+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:05:26.186830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.186830+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:55.950536+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":305,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.184170+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:07:09.184170+0000","last_peered":"2024-09-13T13:07:09.184170+0000","last_clean":"2024-09-13T13:07:09.184170+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:07:09.184170+0000","last_undegraded":"2024-09-13T13:07:09.184170+0000","last_fullsized":"2024-09-13T13:07:09.184170+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:56.924269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.191379+0000","last_change":"2024-09-13T13:03:44.467611+0000","last_active":"2024-09-13T13:07:09.191379+0000","last_peered":"2024-09-13T13:07:09.191379+0000","last_clean":"2024-09-13T13:07:09.191379+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:07:09.191379+0000","last_undegraded":"2024-09-13T13:07:09.191379+0000","last_fullsized":"2024-09-13T13:07:09.191379+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:44.467297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:44.467297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:05.125721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.194352+0000","last_change":"2024-09-13T13:02:42.208021+0000","last_active":"2024-09-13T13:07:09.194352+0000","last_peered":"2024-09-13T13:07:09.194352+0000","last_clean":"2024-09-13T13:07:09.194352+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:07:09.194352+0000","last_undegraded":"2024-09-13T13:07:09.194352+0000","last_fullsized":"2024-09-13T13:07:09.194352+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:42.207956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.207956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:18.666832+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":249,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.188578+0000","last_change":"2024-09-13T13:05:35.506093+0000","last_active":"2024-09-13T13:07:09.188578+0000","last_peered":"2024-09-13T13:07:09.188578+0000","last_clean":"2024-09-13T13:07:09.188578+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:07:09.188578+0000","last_undegraded":"2024-09-13T13:07:09.188578+0000","last_fullsized":"2024-09-13T13:07:09.188578+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:05:35.506037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.506037+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:07.725091+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.204973+0000","last_change":"2024-09-13T13:02:39.921391+0000","last_active":"2024-09-13T13:07:09.204973+0000","last_peered":"2024-09-13T13:07:09.204973+0000","last_clean":"2024-09-13T13:07:09.204973+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:07:09.204973+0000","last_undegraded":"2024-09-13T13:07:09.204973+0000","last_fullsized":"2024-09-13T13:07:09.204973+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.921325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.921325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:22.433670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":269,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.195431+0000","last_change":"2024-09-13T13:01:06.211993+0000","last_active":"2024-09-13T13:07:09.195431+0000","last_peered":"2024-09-13T13:07:09.195431+0000","last_clean":"2024-09-13T13:07:09.195431+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:07:09.195431+0000","last_undegraded":"2024-09-13T13:07:09.195431+0000","last_fullsized":"2024-09-13T13:07:09.195431+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:01:05.184528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:57.661293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":273,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.198669+0000","last_change":"2024-09-13T13:02:36.586702+0000","last_active":"2024-09-13T13:07:09.198669+0000","last_peered":"2024-09-13T13:07:09.198669+0000","last_clean":"2024-09-13T13:07:09.198669+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:07:09.198669+0000","last_undegraded":"2024-09-13T13:07:09.198669+0000","last_fullsized":"2024-09-13T13:07:09.198669+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:36.586654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:36.586654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:22.364490+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":311,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.194481+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:07:09.194481+0000","last_peered":"2024-09-13T13:07:09.194481+0000","last_clean":"2024-09-13T13:07:09.194481+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:07:09.194481+0000","last_undegraded":"2024-09-13T13:07:09.194481+0000","last_fullsized":"2024-09-13T13:07:09.194481+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:02.594342+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.189280+0000","last_change":"2024-09-13T13:05:12.597696+0000","last_active":"2024-09-13T13:07:09.189280+0000","last_peered":"2024-09-13T13:07:09.189280+0000","last_clean":"2024-09-13T13:07:09.189280+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:07:09.189280+0000","last_undegraded":"2024-09-13T13:07:09.189280+0000","last_fullsized":"2024-09-13T13:07:09.189280+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:12.597637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.597637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:33.633105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":296,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.202876+0000","last_change":"2024-09-13T13:06:10.265459+0000","last_active":"2024-09-13T13:07:09.202876+0000","last_peered":"2024-09-13T13:07:09.202876+0000","last_clean":"2024-09-13T13:07:09.202876+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:07:09.202876+0000","last_undegraded":"2024-09-13T13:07:09.202876+0000","last_fullsized":"2024-09-13T13:07:09.202876+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:10.265415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.265415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:30.604403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":276,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.195553+0000","last_change":"2024-09-13T13:04:02.760695+0000","last_active":"2024-09-13T13:07:09.195553+0000","last_peered":"2024-09-13T13:07:09.195553+0000","last_clean":"2024-09-13T13:07:09.195553+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:07:09.195553+0000","last_undegraded":"2024-09-13T13:07:09.195553+0000","last_fullsized":"2024-09-13T13:07:09.195553+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:01:07.192321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:55.662877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":311,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.197247+0000","last_change":"2024-09-13T13:05:44.759472+0000","last_active":"2024-09-13T13:07:09.197247+0000","last_peered":"2024-09-13T13:07:09.197247+0000","last_clean":"2024-09-13T13:07:09.197247+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:07:09.197247+0000","last_undegraded":"2024-09-13T13:07:09.197247+0000","last_fullsized":"2024-09-13T13:07:09.197247+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2024-09-13T13:05:44.759233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.759233+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":293,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.200287+0000","last_change":"2024-09-13T13:07:09.132221+0000","last_active":"2024-09-13T13:07:09.200287+0000","last_peered":"2024-09-13T13:07:09.200287+0000","last_clean":"2024-09-13T13:07:09.200287+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:07:09.200287+0000","last_undegraded":"2024-09-13T13:07:09.200287+0000","last_fullsized":"2024-09-13T13:07:09.200287+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:09.132136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:09.132136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:19.094537+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":179,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.191302+0000","last_change":"2024-09-13T13:06:47.181575+0000","last_active":"2024-09-13T13:07:09.191302+0000","last_peered":"2024-09-13T13:07:09.191302+0000","last_clean":"2024-09-13T13:07:09.191302+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:07:09.191302+0000","last_undegraded":"2024-09-13T13:07:09.191302+0000","last_fullsized":"2024-09-13T13:07:09.191302+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:47.181407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:47.181407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:15.131168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":289,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:12.592938+0000","last_change":"2024-09-13T13:07:12.592938+0000","last_active":"2024-09-13T13:07:12.592938+0000","last_peered":"2024-09-13T13:07:12.592938+0000","last_clean":"2024-09-13T13:07:12.592938+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:07:12.592938+0000","last_undegraded":"2024-09-13T13:07:12.592938+0000","last_fullsized":"2024-09-13T13:07:12.592938+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.592873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.592873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:29.659738+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.193262+0000","last_change":"2024-09-13T13:05:44.759683+0000","last_active":"2024-09-13T13:07:09.193262+0000","last_peered":"2024-09-13T13:07:09.193262+0000","last_clean":"2024-09-13T13:07:09.193262+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:07:09.193262+0000","last_undegraded":"2024-09-13T13:07:09.193262+0000","last_fullsized":"2024-09-13T13:07:09.193262+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:44.759284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.759284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.197479+0000","last_change":"2024-09-13T13:02:46.246602+0000","last_active":"2024-09-13T13:07:09.197479+0000","last_peered":"2024-09-13T13:07:09.197479+0000","last_clean":"2024-09-13T13:07:09.197479+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:07:09.197479+0000","last_undegraded":"2024-09-13T13:07:09.197479+0000","last_fullsized":"2024-09-13T13:07:09.197479+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.246540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.246540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:26.741283+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":289,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.192498+0000","last_change":"2024-09-13T13:06:09.928585+0000","last_active":"2024-09-13T13:07:09.192498+0000","last_peered":"2024-09-13T13:07:09.192498+0000","last_clean":"2024-09-13T13:07:09.192498+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:07:09.192498+0000","last_undegraded":"2024-09-13T13:07:09.192498+0000","last_fullsized":"2024-09-13T13:07:09.192498+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:06:09.928504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.928504+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:39.792862+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T13:07:09.199628+0000","last_change":"2024-09-13T13:06:47.291601+0000","last_active":"2024-09-13T13:07:09.199628+0000","last_peered":"2024-09-13T13:07:09.199628+0000","last_clean":"2024-09-13T13:07:09.199628+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:07:09.199628+0000","last_undegraded":"2024-09-13T13:07:09.199628+0000","last_fullsized":"2024-09-13T13:07:09.199628+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:47.291245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:47.291245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:05.181779+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4873191424,"num_objects":74359,"num_object_clones":0,"num_object_copies":223077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74359,"num_write_kb":4758976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14594932736,"data_stored":14594932736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74359,"ondisk_log_size":74359,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":365,"num_read_kb":360,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201357,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1613680,"kb_used_data":1465888,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92131472,"statfs":{"total":95995035648,"available":94342627328,"internally_reserved":0,"allocated":1501069312,"data_stored":1500410848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":151321034},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806140968,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1725152,"kb_used_data":1539152,"kb_used_omap":4,"kb_used_meta":185979,"kb_avail":92020000,"statfs":{"total":95995035648,"available":94228480000,"internally_reserved":0,"allocated":1576091648,"data_stored":1575447939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":190443181},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691929,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2503460,"kb_used_data":2333460,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":91241692,"statfs":{"total":95995035648,"available":93431492608,"internally_reserved":0,"allocated":2389463040,"data_stored":2388815160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":174062021},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920863,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1618040,"kb_used_data":1470248,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92127112,"statfs":{"total":95995035648,"available":94338162688,"internally_reserved":0,"allocated":1505533952,"data_stored":1504865728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117094,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2028476,"kb_used_data":1750188,"kb_used_omap":1,"kb_used_meta":278270,"kb_avail":91716676,"statfs":{"total":95995035648,"available":93917876224,"internally_reserved":0,"allocated":1792192512,"data_stored":1791521022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":284948935},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411433,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1911956,"kb_used_data":1750276,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":91833196,"statfs":{"total":95995035648,"available":94037192704,"internally_reserved":0,"allocated":1792282624,"data_stored":1791650468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165542342},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2219404,"kb_used_data":2043836,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91525748,"statfs":{"total":95995035648,"available":93722365952,"internally_reserved":0,"allocated":2092888064,"data_stored":2092199871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738487,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2013940,"kb_used_data":1775908,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":91731212,"statfs":{"total":95995035648,"available":93932761088,"internally_reserved":0,"allocated":1818529792,"data_stored":1817866941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1832189952,"data_stored":1832189952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2124152832,"data_stored":2124152832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1814036480,"data_stored":1814036480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1812267008,"data_stored":1812267008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1505558528,"data_stored":1505558528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2396585984,"data_stored":2396585984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1602682880,"data_stored":1602682880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1507459072,"data_stored":1507459072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:07:22.880 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T13:07:22.880 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:07:22.881 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-13T13:07:22.881 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-13T13:07:22.938 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:23.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:23.021 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:23.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:23.070 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:23.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:23.128 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:23.147 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:23.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:23.248 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:23.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:23.302 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:23.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:23.403 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:23.433 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:23.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:23.472 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:23 smithi052 ceph-mon[36682]: pgmap v301: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T13:07:23.472 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:23 smithi052 ceph-mon[36682]: 6.1b scrub starts 2024-09-13T13:07:23.472 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:23 smithi052 ceph-mon[36682]: 6.1b scrub ok 2024-09-13T13:07:23.472 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:23 smithi052 ceph-mon[36682]: osdmap e196: 8 total, 8 up, 8 in 2024-09-13T13:07:23.472 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:23 smithi052 ceph-mon[36682]: 5.18 scrub starts 2024-09-13T13:07:23.472 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:23 smithi052 ceph-mon[36682]: 5.18 scrub ok 2024-09-13T13:07:23.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:23 smithi052 ceph-mon[29471]: pgmap v301: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T13:07:23.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:23 smithi052 ceph-mon[29471]: 6.1b scrub starts 2024-09-13T13:07:23.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:23 smithi052 ceph-mon[29471]: 6.1b scrub ok 2024-09-13T13:07:23.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:23 smithi052 ceph-mon[29471]: osdmap e196: 8 total, 8 up, 8 in 2024-09-13T13:07:23.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:23 smithi052 ceph-mon[29471]: 5.18 scrub starts 2024-09-13T13:07:23.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:23 smithi052 ceph-mon[29471]: 5.18 scrub ok 2024-09-13T13:07:23.534 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:23.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:23 smithi167 ceph-mon[33178]: pgmap v301: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T13:07:23.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:23 smithi167 ceph-mon[33178]: 6.1b scrub starts 2024-09-13T13:07:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:23 smithi167 ceph-mon[33178]: 6.1b scrub ok 2024-09-13T13:07:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:23 smithi167 ceph-mon[33178]: osdmap e196: 8 total, 8 up, 8 in 2024-09-13T13:07:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:23 smithi167 ceph-mon[33178]: 5.18 scrub starts 2024-09-13T13:07:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:23 smithi167 ceph-mon[33178]: 5.18 scrub ok 2024-09-13T13:07:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:23.572 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:23.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:23.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:23.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:23.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:23.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:23.773 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 52 16 86016 86000 103.337 69.25 0.0143219 0.00967263 2024-09-13T13:07:23.805 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:23.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:23.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:23.936 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:23.974 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:23.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:23.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:24.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:24.097 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:24.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:24.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:24.198 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:24.230 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:24.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:24.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:24.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:07:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:07:24.333 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:24.369 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:24.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:24.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:24.471 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:24.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:24.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:24.524 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:07:24.592 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:24 smithi052 ceph-mon[36682]: 6.18 scrub starts 2024-09-13T13:07:24.592 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:24 smithi052 ceph-mon[36682]: 6.18 scrub ok 2024-09-13T13:07:24.593 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:24 smithi052 ceph-mon[29471]: 6.18 scrub starts 2024-09-13T13:07:24.593 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:24 smithi052 ceph-mon[29471]: 6.18 scrub ok 2024-09-13T13:07:24.616 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:24.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:24.672 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:24.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:24.773 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:24.774 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 53 16 87242 87226 102.833 76.625 0.0155938 0.00971915 2024-09-13T13:07:24.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:24 smithi167 ceph-mon[33178]: 6.18 scrub starts 2024-09-13T13:07:24.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:24 smithi167 ceph-mon[33178]: 6.18 scrub ok 2024-09-13T13:07:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:24.826 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:24.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:24.927 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:24.958 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:24.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:24.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:25.060 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:25.101 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:25.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:25.203 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:25.247 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:25.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:25.350 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:25.396 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:25.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:25.497 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:25.539 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:25.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:25.642 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:25.679 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:25.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:25 smithi052 ceph-mon[36682]: 5.f scrub starts 2024-09-13T13:07:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:25 smithi052 ceph-mon[36682]: 5.f scrub ok 2024-09-13T13:07:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:25 smithi052 ceph-mon[36682]: pgmap v303: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-13T13:07:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:25 smithi052 ceph-mon[36682]: 4.1a scrub starts 2024-09-13T13:07:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:25 smithi052 ceph-mon[36682]: 4.1a scrub ok 2024-09-13T13:07:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:25 smithi052 ceph-mon[29471]: 5.f scrub starts 2024-09-13T13:07:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:25 smithi052 ceph-mon[29471]: 5.f scrub ok 2024-09-13T13:07:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:25 smithi052 ceph-mon[29471]: pgmap v303: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-13T13:07:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:25 smithi052 ceph-mon[29471]: 4.1a scrub starts 2024-09-13T13:07:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:25 smithi052 ceph-mon[29471]: 4.1a scrub ok 2024-09-13T13:07:25.774 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 54 15 88368 88353 102.233 70.4375 0.022637 0.00977683 2024-09-13T13:07:25.781 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:25.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:25 smithi167 ceph-mon[33178]: 5.f scrub starts 2024-09-13T13:07:25.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:25 smithi167 ceph-mon[33178]: 5.f scrub ok 2024-09-13T13:07:25.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:25 smithi167 ceph-mon[33178]: pgmap v303: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-13T13:07:25.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:25.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:25 smithi167 ceph-mon[33178]: 4.1a scrub starts 2024-09-13T13:07:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:25 smithi167 ceph-mon[33178]: 4.1a scrub ok 2024-09-13T13:07:25.825 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:25.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:25.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:25.927 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:25.959 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:25.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:25.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:26.061 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:26.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:26.085 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:26.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:26.186 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:26.221 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:26.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:26.322 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:26.359 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:26.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:26.453 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:07:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:07:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:07:26.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:26 smithi052 ceph-mon[29471]: 4.4 scrub starts 2024-09-13T13:07:26.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:26 smithi052 ceph-mon[29471]: 4.4 scrub ok 2024-09-13T13:07:26.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:26 smithi052 ceph-mon[29471]: 3.15 scrub starts 2024-09-13T13:07:26.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:26 smithi052 ceph-mon[29471]: 3.15 scrub ok 2024-09-13T13:07:26.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:26 smithi052 ceph-mon[29471]: 8.2 scrub starts 2024-09-13T13:07:26.454 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:26 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/601728565' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:07:26.454 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:26 smithi052 ceph-mon[36682]: 4.4 scrub starts 2024-09-13T13:07:26.454 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:26 smithi052 ceph-mon[36682]: 4.4 scrub ok 2024-09-13T13:07:26.454 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:26 smithi052 ceph-mon[36682]: 3.15 scrub starts 2024-09-13T13:07:26.454 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:26 smithi052 ceph-mon[36682]: 3.15 scrub ok 2024-09-13T13:07:26.454 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:26 smithi052 ceph-mon[36682]: 8.2 scrub starts 2024-09-13T13:07:26.454 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:26 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/601728565' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:07:26.454 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:07:26.460 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:26.495 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:26.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:26.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:26.597 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:26.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:26.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:26.731 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:26.763 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:26.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:26.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:26.774 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 55 16 89486 89470 101.644 69.8125 0.01016 0.00983338 2024-09-13T13:07:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:26 smithi167 ceph-mon[33178]: 4.4 scrub starts 2024-09-13T13:07:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:26 smithi167 ceph-mon[33178]: 4.4 scrub ok 2024-09-13T13:07:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:26 smithi167 ceph-mon[33178]: 3.15 scrub starts 2024-09-13T13:07:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:26 smithi167 ceph-mon[33178]: 3.15 scrub ok 2024-09-13T13:07:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:26 smithi167 ceph-mon[33178]: 8.2 scrub starts 2024-09-13T13:07:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:26 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/601728565' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:07:26.865 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:26.914 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:26.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:26.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:26.982 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:07:26.982 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:07:27.016 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:27.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:27.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:27.152 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:27.184 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:27.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:27.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:27.286 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:27.312 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:27.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:27.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:27.414 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:27.445 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:27.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:27.547 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:27.581 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:27.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:27.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:27.684 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:27.721 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:27.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:27 smithi052 ceph-mon[36682]: pgmap v304: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-13T13:07:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:27 smithi052 ceph-mon[36682]: 6.10 scrub starts 2024-09-13T13:07:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:27 smithi052 ceph-mon[36682]: 6.10 scrub ok 2024-09-13T13:07:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:27 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:07:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:27 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/601728565' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:07:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:27 smithi052 ceph-mon[36682]: osdmap e197: 8 total, 8 up, 8 in 2024-09-13T13:07:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:27 smithi052 ceph-mon[29471]: pgmap v304: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-13T13:07:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:27 smithi052 ceph-mon[29471]: 6.10 scrub starts 2024-09-13T13:07:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:27 smithi052 ceph-mon[29471]: 6.10 scrub ok 2024-09-13T13:07:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:27 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:07:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:27 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/601728565' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:07:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:27 smithi052 ceph-mon[29471]: osdmap e197: 8 total, 8 up, 8 in 2024-09-13T13:07:27.774 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 56 16 90689 90673 101.171 75.1875 0.0117629 0.00987901 2024-09-13T13:07:27.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:27 smithi167 ceph-mon[33178]: pgmap v304: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-13T13:07:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:27 smithi167 ceph-mon[33178]: 6.10 scrub starts 2024-09-13T13:07:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:27 smithi167 ceph-mon[33178]: 6.10 scrub ok 2024-09-13T13:07:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:27 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:07:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:27 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/601728565' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:07:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:27 smithi167 ceph-mon[33178]: osdmap e197: 8 total, 8 up, 8 in 2024-09-13T13:07:27.822 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:27.867 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:27.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:27.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:27.969 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:28.010 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:28.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:28.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:28.112 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:28.143 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:28.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:28.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:28.245 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:28.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:28.293 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:28.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:28.394 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:28.433 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:28.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:28.535 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:28.574 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:28.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:28.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:28.676 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:28.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:28.718 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:28.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:28.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T13:07:28.775 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 57 16 91932 91916 100.758 77.6875 0.0125946 0.00991925 2024-09-13T13:07:28.820 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:28.830 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:28.852 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:28.891 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:28.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:28.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:28 smithi052 ceph-mon[29471]: pgmap v306: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 807 op/s 2024-09-13T13:07:28.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:28 smithi052 ceph-mon[29471]: osdmap e198: 8 total, 8 up, 8 in 2024-09-13T13:07:28.982 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:28 smithi052 ceph-mon[36682]: pgmap v306: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 807 op/s 2024-09-13T13:07:28.982 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:28 smithi052 ceph-mon[36682]: osdmap e198: 8 total, 8 up, 8 in 2024-09-13T13:07:28.993 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:29.040 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:29.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:29.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:29.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:28 smithi167 ceph-mon[33178]: pgmap v306: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 807 op/s 2024-09-13T13:07:29.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:28 smithi167 ceph-mon[33178]: osdmap e198: 8 total, 8 up, 8 in 2024-09-13T13:07:29.142 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:29.170 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:29.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:29.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:29.272 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:29.318 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:29.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:29.420 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:29.471 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:29.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:29.520 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:07:29.520 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:07:29.562 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:07:29.562 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:07:29.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:29.599 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:07:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:07:29.602 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:29.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:29.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:29.637 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:07:29.638 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:07:29.673 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:07:29.673 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:07:29.704 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:29.707 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:07:29.707 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:07:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:29.759 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:29.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:29.760 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:07:29.760 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:07:29.775 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 58 16 92944 92928 100.111 63.25 0.0153266 0.009984 2024-09-13T13:07:29.799 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:07:29.799 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:07:29.860 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:29.899 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:29.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:30.001 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:30.028 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:30.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:30.129 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:30.163 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:29 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2980459771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:30.163 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:29 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2980459771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:30.168 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:30.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:30.252 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:07:30.270 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:30.306 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:30.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:30.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:29 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2980459771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:30.388 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:30.408 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:30.433 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:30.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:30.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:30.535 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:30.583 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:30.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:30.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:30.726 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:30.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:30.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:30.775 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 59 16 94099 94083 99.6379 72.1875 0.0125117 0.0100311 2024-09-13T13:07:30.827 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:30.859 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:30.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:30.961 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:31.009 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:31.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:31.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:31.111 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:31.139 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:31.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:31.221 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:30 smithi167 ceph-mon[33178]: pgmap v308: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-13T13:07:31.221 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:30 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:07:31.221 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:30 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:07:31.221 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:30 smithi167 ceph-mon[33178]: osdmap e199: 8 total, 8 up, 8 in 2024-09-13T13:07:31.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:30 smithi052 ceph-mon[36682]: pgmap v308: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-13T13:07:31.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:30 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:07:31.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:30 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:07:31.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:30 smithi052 ceph-mon[36682]: osdmap e199: 8 total, 8 up, 8 in 2024-09-13T13:07:31.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:30 smithi052 ceph-mon[29471]: pgmap v308: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-13T13:07:31.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:30 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:07:31.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:30 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:07:31.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:30 smithi052 ceph-mon[29471]: osdmap e199: 8 total, 8 up, 8 in 2024-09-13T13:07:31.242 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:31.278 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:31.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:31.350 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-13T13:07:31.380 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:31.451 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:31.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:31.552 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:31.598 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:31.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:31.699 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:31.755 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:31.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:31.775 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:07:31.775037+0000 min lat: 0.00118759 max lat: 0.156296 avg lat: 0.0100802 2024-09-13T13:07:31.775 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:07:31.775 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 60 16 95230 95214 99.1553 70.6875 0.0102836 0.0100802 2024-09-13T13:07:31.857 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:31.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:31.887 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:31.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:31.981 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:31.988 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:32.022 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:07:32.022 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:07:32.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:32.022 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.25222.0:95488 8.e e199/169 8:7d6de5b6:::benchmark_data_smithi052_101302_object95487:head v 199'5960, mlcod=199'5957)", 2024-09-13T13:07:32.022 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:32.015385+0000", 2024-09-13T13:07:32.022 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0026701519999999999, 2024-09-13T13:07:32.022 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027113910000000001, 2024-09-13T13:07:32.022 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:32.022 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:32.015385+0000", 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:32.015385+0000", 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:32.015388+0000", 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0820000000000001e-06 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:32.015415+0000", 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7279000000000001e-05 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:32.015432+0000", 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7581e-05 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:32.015442+0000", 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1689999999999998e-06 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:32.015523+0000", 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1390000000000005e-05 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:32.015544+0000", 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1019e-05 2024-09-13T13:07:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:32.026 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:32.026 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:32.026 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:32.026 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:07:32.026 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 1 2024-09-13T13:07:32.026 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:07:32.041 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:32.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:32.143 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:32.190 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:32.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:32.265 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:32.292 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:32.340 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:32.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:32.442 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:32.481 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:32.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:32.583 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:32.620 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:32.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:32.722 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:32.748 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T13:07:32.769 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:32.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:32.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:32.777 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 61 16 96379 96363 98.7025 71.8125 0.00988743 0.0101266 2024-09-13T13:07:32.872 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:32.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:32.907 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:32.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:32.931 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:07:32.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:32 smithi052 ceph-mon[29471]: pgmap v310: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-13T13:07:32.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:32 smithi052 ceph-mon[29471]: osdmap e200: 8 total, 8 up, 8 in 2024-09-13T13:07:32.987 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:32 smithi052 ceph-mon[36682]: pgmap v310: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-13T13:07:32.987 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:32 smithi052 ceph-mon[36682]: osdmap e200: 8 total, 8 up, 8 in 2024-09-13T13:07:33.009 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:33.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:33.075 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:33.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:33.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:33.201 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:33.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:33.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:33.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:33.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:32 smithi167 ceph-mon[33178]: pgmap v310: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-13T13:07:33.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:32 smithi167 ceph-mon[33178]: osdmap e200: 8 total, 8 up, 8 in 2024-09-13T13:07:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:33.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:33.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:33.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:33.485 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:33.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:33.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:33.587 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:33.623 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:33.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:33.725 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:33.761 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:33.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:33.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:33.778 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 62 15 97499 97484 98.2406 70.0625 0.0121831 0.0101739 2024-09-13T13:07:33.863 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:33.908 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:33.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:34.010 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:34.054 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:34.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:34.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:33 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:07:34.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:07:33 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:07:34.123 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:33 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:07:34.123 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:33 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:07:34.156 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:34.205 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:34.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:34.307 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:34.362 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:34.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:34.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:34.463 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:34.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:34.499 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:34.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:34.600 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:34.649 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:34.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:34.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:34.751 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:34.779 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 63 16 98596 98580 97.7673 68.5 0.0328275 0.010223 2024-09-13T13:07:34.794 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:34.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:34.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:34.799 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:34.897 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:34.945 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:34.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:34.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:35.047 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:35.057 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:35.080 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:35.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:35.181 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:35.193 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:34 smithi052 ceph-mon[29471]: pgmap v312: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-09-13T13:07:35.193 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:34 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:35.193 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:34 smithi052 ceph-mon[36682]: pgmap v312: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-09-13T13:07:35.193 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:34 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:35.228 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:35.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:35.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:35.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:34 smithi167 ceph-mon[33178]: pgmap v312: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-09-13T13:07:35.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:34 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:35.330 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:35.366 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:35.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:35.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:35.502 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:35.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:35.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:35.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:35.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:35.637 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:35.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:35.738 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:35.768 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:35.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:35.778 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 64 16 99803 99787 97.4184 75.4375 0.0142722 0.0102603 2024-09-13T13:07:35.862 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:07:35.870 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:35.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:35.902 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:35.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:36.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:36.039 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:36.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:36.141 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:36.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:35 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1651610221' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:07:36.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:35 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1651610221' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:07:36.176 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:36.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:36.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:36.278 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:36.299 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:36.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:36.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:36.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:35 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1651610221' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:07:36.402 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:36.442 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:36.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:36.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:07:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:07:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:07:36.543 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:36.573 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:36.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:36.674 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:36.717 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-13T13:07:36.718 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:36.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:36.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:36.779 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 65 15 100961 100946 97.034 72.4375 0.00897437 0.0103007 2024-09-13T13:07:36.820 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:36.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:36.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:36.980 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:36.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:36 smithi052 ceph-mon[36682]: pgmap v313: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-13T13:07:36.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:36 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:07:36.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:36 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1651610221' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:07:36.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:36 smithi052 ceph-mon[36682]: osdmap e201: 8 total, 8 up, 8 in 2024-09-13T13:07:36.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:36 smithi052 ceph-mon[29471]: pgmap v313: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-13T13:07:36.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:36 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:07:36.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:36 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1651610221' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:07:36.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:36 smithi052 ceph-mon[29471]: osdmap e201: 8 total, 8 up, 8 in 2024-09-13T13:07:37.007 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:37.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:37.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:37.109 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:37.147 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:37.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:37.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:37.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:37.286 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:37.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:37.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:36 smithi167 ceph-mon[33178]: pgmap v313: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-13T13:07:37.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:36 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:07:37.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:36 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1651610221' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:07:37.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:36 smithi167 ceph-mon[33178]: osdmap e201: 8 total, 8 up, 8 in 2024-09-13T13:07:37.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:37.417 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:37.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:37.519 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:37.564 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:37.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:37.666 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:37.705 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:37.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:37.780 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 66 16 102143 102127 96.6822 73.8125 0.00981807 0.0103384 2024-09-13T13:07:37.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:37.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:37.846 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:37.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:37.947 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:37.990 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:37.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:38.091 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:38.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:38.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:38.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:38.225 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:38.259 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:38.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:38.360 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:38.401 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:38.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:38.503 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:38.534 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:38.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:38.636 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:38.669 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:38.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:38.771 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:38.780 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 67 16 103378 103362 96.3893 77.1875 0.0083859 0.0103697 2024-09-13T13:07:38.807 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:38.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:38.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:38.909 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:38.947 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:38.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:38.987 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:38 smithi052 ceph-mon[36682]: pgmap v315: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 741 B/s rd, 57 MiB/s wr, 907 op/s 2024-09-13T13:07:38.987 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:38 smithi052 ceph-mon[36682]: osdmap e202: 8 total, 8 up, 8 in 2024-09-13T13:07:38.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:38 smithi052 ceph-mon[29471]: pgmap v315: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 741 B/s rd, 57 MiB/s wr, 907 op/s 2024-09-13T13:07:38.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:38 smithi052 ceph-mon[29471]: osdmap e202: 8 total, 8 up, 8 in 2024-09-13T13:07:39.049 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:39.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:38 smithi167 ceph-mon[33178]: pgmap v315: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 741 B/s rd, 57 MiB/s wr, 907 op/s 2024-09-13T13:07:39.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:38 smithi167 ceph-mon[33178]: osdmap e202: 8 total, 8 up, 8 in 2024-09-13T13:07:39.092 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:39.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:39.194 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:39.225 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:39.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:39.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:39.276 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:07:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:07:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:07:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:07:39.276 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:07:39.327 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:39.364 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:39.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:39.466 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:39.496 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:39.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:39.509 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:07:39.509 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:07:39.549 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:07:39.550 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:07:39.583 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:07:39.583 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:07:39.598 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:39.623 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:07:39.624 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:07:39.655 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:39.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:39.656 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:07:39.657 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:07:39.695 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:07:39.695 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:07:39.731 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:07:39.731 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:07:39.757 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:39.773 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:07:39.773 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:07:39.780 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 68 16 104518 104502 96.0196 71.25 0.00714297 0.0104095 2024-09-13T13:07:39.793 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:39.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:39.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:39.895 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:39.933 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:39.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:40.035 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:40.074 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:40.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:40.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:40.094 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:39 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/119032944' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:40.095 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:39 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/119032944' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:40.152 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T13:07:40.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:40.206 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:40.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:40.309 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:40.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:39 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/119032944' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:40.329 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:40.338 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:40.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:40.440 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:40.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:40.502 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:40.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:40.603 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:40.644 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:40.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:40.743 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:07:40.745 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:40.780 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 69 16 105721 105705 95.7176 75.1875 0.020414 0.0104425 2024-09-13T13:07:40.812 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:40.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:40.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:40.915 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:40.940 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:40.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:41.042 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:41.080 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:40 smithi052 ceph-mon[36682]: pgmap v317: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T13:07:41.081 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:40 smithi052 ceph-mon[29471]: pgmap v317: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T13:07:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:41.115 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:41.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:41.215 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:41.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:41.259 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:41.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:41.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:40 smithi167 ceph-mon[33178]: pgmap v317: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T13:07:41.360 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:41.393 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:41.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:41.411 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:41.484 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:41.495 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:41.544 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:41.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:41.646 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:41.675 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:41.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:41.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:41.777 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:41.781 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 70 16 106796 106780 95.3096 67.1875 0.00484 0.0104872 2024-09-13T13:07:41.832 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:41.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:41.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:41.934 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:41.970 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:41.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:42.072 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:42.107 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:42.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:42.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:42.207 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-13T13:07:42.209 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:42.241 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:42.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:42.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:42.373 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:42.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:42.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:42.475 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:42.498 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:42.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:42.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:42.600 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:42.642 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:42.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:42.744 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:42.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 71 16 107961 107945 94.9928 72.8125 0.0087432 0.0105224 2024-09-13T13:07:42.797 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:42.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:42.857 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:07:42.899 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:42.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:42 smithi052 ceph-mon[29471]: pgmap v318: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 826 op/s 2024-09-13T13:07:42.953 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:42 smithi052 ceph-mon[36682]: pgmap v318: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 826 op/s 2024-09-13T13:07:42.969 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:42.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:43.089 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:43.128 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:43.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:43.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:43.230 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:43.264 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:43.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:43.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:42 smithi167 ceph-mon[33178]: pgmap v318: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 826 op/s 2024-09-13T13:07:43.366 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:43.411 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:43.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:43.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:43.534 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:07:43.534 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:07:43.541 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:43.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:43.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:43.643 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:43.682 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:43.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:43.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:43.781 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 72 16 109007 108991 94.5814 65.375 0.00883593 0.0105685 2024-09-13T13:07:43.784 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:43.820 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:43.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:43.869 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:07:43.921 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:43.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:43.963 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:43.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:44.064 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:44.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1652061833' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:07:44.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:07:43 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:07:44.102 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:44.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:44.204 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:44.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:44.240 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:44.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:44.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1652061833' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:07:44.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1652061833' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:07:44.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:44.365 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T13:07:44.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:44.378 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:44.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:44.479 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:44.520 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:44.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:44.537 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:44.621 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:44.650 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:44.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:44.752 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:44.781 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 73 16 110193 110177 94.3012 74.125 0.00693128 0.0105991 2024-09-13T13:07:44.786 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:44.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:44.887 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:44.931 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:44.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:44.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:44 smithi052 ceph-mon[29471]: pgmap v319: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T13:07:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:44 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:44 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:07:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1652061833' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:07:44.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:44 smithi052 ceph-mon[29471]: osdmap e203: 8 total, 8 up, 8 in 2024-09-13T13:07:44.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:44 smithi052 ceph-mon[36682]: pgmap v319: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T13:07:44.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:44 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:44.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:44 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:07:44.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1652061833' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:07:44.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:44 smithi052 ceph-mon[36682]: osdmap e203: 8 total, 8 up, 8 in 2024-09-13T13:07:45.034 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:45.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:45.099 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:45.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:45.200 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:45.241 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:45.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:45.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:44 smithi167 ceph-mon[33178]: pgmap v319: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T13:07:45.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:44 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:44 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:07:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1652061833' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:07:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:44 smithi167 ceph-mon[33178]: osdmap e203: 8 total, 8 up, 8 in 2024-09-13T13:07:45.342 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:45.366 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:45.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:45.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:45.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:45.494 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:45.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:45.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:45.596 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:45.622 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:45.631 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:45.642 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:45.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:45.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:45.744 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:45.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 74 16 111233 111217 93.9034 65 0.00993658 0.0106439 2024-09-13T13:07:45.791 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:45.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:45.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:45.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:45.931 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:45.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:46.033 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:46.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:46.159 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:45 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:07:46.160 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:45 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:07:46.160 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:45 smithi052 ceph-mon[36682]: osdmap e204: 8 total, 8 up, 8 in 2024-09-13T13:07:46.160 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:07:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:07:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:07:46.160 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:45 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:07:46.160 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:45 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:07:46.160 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:45 smithi052 ceph-mon[29471]: osdmap e204: 8 total, 8 up, 8 in 2024-09-13T13:07:46.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:46.192 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:46.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:46.294 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:46.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:45 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:07:46.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:45 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:07:46.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:45 smithi167 ceph-mon[33178]: osdmap e204: 8 total, 8 up, 8 in 2024-09-13T13:07:46.338 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:46.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:46.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:46.440 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:46.467 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:46.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:46.569 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:46.611 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:46.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:46.633 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:07:46.633 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:07:46.711 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:46.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:46.754 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:46.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:46.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 75 16 112217 112201 93.4713 61.5 0.0156009 0.0106925 2024-09-13T13:07:46.855 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:46.889 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:46.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:46 smithi052 ceph-mon[36682]: pgmap v322: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-13T13:07:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:46 smithi052 ceph-mon[36682]: 8.a scrub starts 2024-09-13T13:07:46.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:46 smithi052 ceph-mon[29471]: pgmap v322: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-13T13:07:46.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:46 smithi052 ceph-mon[29471]: 8.a scrub starts 2024-09-13T13:07:46.991 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:47.049 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:47.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:47.053 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:07:47.152 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:47.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:47.206 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:47.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:47.229 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:07:47.229 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:07:47.272 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:07:47.272 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:07:47.307 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:47.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:46 smithi167 ceph-mon[33178]: pgmap v322: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-13T13:07:47.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:46 smithi167 ceph-mon[33178]: 8.a scrub starts 2024-09-13T13:07:47.321 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:07:47.321 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:07:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:47.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:47.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:47.367 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:07:47.367 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:07:47.404 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:07:47.404 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:07:47.438 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:07:47.438 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:07:47.466 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:47.474 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:07:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:07:47.507 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:07:47.507 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:07:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:47.519 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:47.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:47.620 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:47.646 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:47.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:47.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:47.679 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-13T13:07:47.749 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:47.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 76 16 113136 113120 92.9972 57.4375 0.0128978 0.0107482 2024-09-13T13:07:47.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:47.787 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:47.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:47.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:47.921 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:47.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:47.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:48.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:48.057 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:48.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:48.159 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:48.200 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:48.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:47 smithi052 ceph-mon[36682]: 6.1e scrub starts 2024-09-13T13:07:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:47 smithi052 ceph-mon[36682]: 6.1e scrub ok 2024-09-13T13:07:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:47 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2723701882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:48.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:47 smithi052 ceph-mon[29471]: 6.1e scrub starts 2024-09-13T13:07:48.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:47 smithi052 ceph-mon[29471]: 6.1e scrub ok 2024-09-13T13:07:48.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:47 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2723701882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:48.302 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:48.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:47 smithi167 ceph-mon[33178]: 6.1e scrub starts 2024-09-13T13:07:48.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:47 smithi167 ceph-mon[33178]: 6.1e scrub ok 2024-09-13T13:07:48.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:47 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2723701882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:48.340 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T13:07:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:48.360 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:48.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:48.461 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:48.508 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:48.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:48.610 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:48.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:48.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:48.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:48.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 77 16 114193 114177 92.6475 66.0625 0.0326113 0.0107888 2024-09-13T13:07:48.788 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:48.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:48.891 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:48.916 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:48.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:48.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:48.919 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:48.919 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:48.926 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:48.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[29471]: 3.16 scrub starts 2024-09-13T13:07:48.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[29471]: 3.16 scrub ok 2024-09-13T13:07:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[29471]: 6.1c scrub starts 2024-09-13T13:07:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[29471]: 6.1c scrub ok 2024-09-13T13:07:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[29471]: pgmap v323: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 806 op/s 2024-09-13T13:07:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:07:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[29471]: 4.3 scrub starts 2024-09-13T13:07:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[29471]: 4.3 scrub ok 2024-09-13T13:07:48.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[36682]: 3.16 scrub starts 2024-09-13T13:07:48.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[36682]: 3.16 scrub ok 2024-09-13T13:07:48.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[36682]: 6.1c scrub starts 2024-09-13T13:07:48.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[36682]: 6.1c scrub ok 2024-09-13T13:07:48.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[36682]: pgmap v323: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 806 op/s 2024-09-13T13:07:48.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:07:48.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[36682]: 4.3 scrub starts 2024-09-13T13:07:48.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:48 smithi052 ceph-mon[36682]: 4.3 scrub ok 2024-09-13T13:07:49.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:49.054 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:49.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:49.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:49.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:49.195 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:49.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:49.296 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:49.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:48 smithi167 ceph-mon[33178]: 3.16 scrub starts 2024-09-13T13:07:49.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:48 smithi167 ceph-mon[33178]: 3.16 scrub ok 2024-09-13T13:07:49.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:48 smithi167 ceph-mon[33178]: 6.1c scrub starts 2024-09-13T13:07:49.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:48 smithi167 ceph-mon[33178]: 6.1c scrub ok 2024-09-13T13:07:49.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:48 smithi167 ceph-mon[33178]: pgmap v323: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 806 op/s 2024-09-13T13:07:49.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:48 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:07:49.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:48 smithi167 ceph-mon[33178]: 4.3 scrub starts 2024-09-13T13:07:49.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:48 smithi167 ceph-mon[33178]: 4.3 scrub ok 2024-09-13T13:07:49.336 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:49.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:49.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:49.440 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:49.471 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:49.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:49.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:49.600 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:49.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:49.702 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:49.731 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:49.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:49.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 78 16 115253 115237 92.309 66.25 0.00873353 0.0108284 2024-09-13T13:07:49.833 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:49.879 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:49.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:49.940 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:07:49.941 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:07:49.941 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:07:49.941 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:07:49.941 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:07:49.941 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:49 smithi052 ceph-mon[36682]: 3.d scrub starts 2024-09-13T13:07:49.941 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:49 smithi052 ceph-mon[36682]: 3.d scrub ok 2024-09-13T13:07:49.941 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:49 smithi052 ceph-mon[36682]: 8.a scrub ok 2024-09-13T13:07:49.942 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:49 smithi052 ceph-mon[36682]: 6.0 scrub starts 2024-09-13T13:07:49.942 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:49 smithi052 ceph-mon[36682]: 6.0 scrub ok 2024-09-13T13:07:49.942 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:49 smithi052 ceph-mon[36682]: 8.1 scrub starts 2024-09-13T13:07:49.942 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:49 smithi052 ceph-mon[29471]: 3.d scrub starts 2024-09-13T13:07:49.942 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:49 smithi052 ceph-mon[29471]: 3.d scrub ok 2024-09-13T13:07:49.942 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:49 smithi052 ceph-mon[29471]: 8.a scrub ok 2024-09-13T13:07:49.942 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:49 smithi052 ceph-mon[29471]: 6.0 scrub starts 2024-09-13T13:07:49.942 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:49 smithi052 ceph-mon[29471]: 6.0 scrub ok 2024-09-13T13:07:49.942 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:49 smithi052 ceph-mon[29471]: 8.1 scrub starts 2024-09-13T13:07:49.981 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:50.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:50.019 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:50.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:50.120 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:50.152 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:50.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:50.254 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:50.282 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:50.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:50.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:49 smithi167 ceph-mon[33178]: 3.d scrub starts 2024-09-13T13:07:50.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:49 smithi167 ceph-mon[33178]: 3.d scrub ok 2024-09-13T13:07:50.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:49 smithi167 ceph-mon[33178]: 8.a scrub ok 2024-09-13T13:07:50.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:49 smithi167 ceph-mon[33178]: 6.0 scrub starts 2024-09-13T13:07:50.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:49 smithi167 ceph-mon[33178]: 6.0 scrub ok 2024-09-13T13:07:50.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:49 smithi167 ceph-mon[33178]: 8.1 scrub starts 2024-09-13T13:07:50.385 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:50.416 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:50.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:50.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:50.518 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:50.554 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:50.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:50.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:50.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:50.685 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:50.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:50.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:50.714 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T13:07:50.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 79 16 116334 116318 91.9958 67.5625 0.00657808 0.0108644 2024-09-13T13:07:50.787 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:50.811 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:07:50.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:50.858 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:50.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:50.960 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:51.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:51.030 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:51.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:51.132 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:51.159 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:51.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:51.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:50 smithi052 ceph-mon[36682]: 6.8 scrub starts 2024-09-13T13:07:51.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:50 smithi052 ceph-mon[36682]: 6.8 scrub ok 2024-09-13T13:07:51.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:50 smithi052 ceph-mon[36682]: pgmap v324: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-13T13:07:51.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:50 smithi052 ceph-mon[29471]: 6.8 scrub starts 2024-09-13T13:07:51.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:50 smithi052 ceph-mon[29471]: 6.8 scrub ok 2024-09-13T13:07:51.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:50 smithi052 ceph-mon[29471]: pgmap v324: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-13T13:07:51.261 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:51.290 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:51.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:51.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:50 smithi167 ceph-mon[33178]: 6.8 scrub starts 2024-09-13T13:07:51.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:50 smithi167 ceph-mon[33178]: 6.8 scrub ok 2024-09-13T13:07:51.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:50 smithi167 ceph-mon[33178]: pgmap v324: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-13T13:07:51.392 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:51.418 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:51.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:51.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:51.520 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:51.559 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:51.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:51.661 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:51.688 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:51.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:51.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:51.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:07:51.783467+0000 min lat: 0.00118759 max lat: 0.156296 avg lat: 0.0109182 2024-09-13T13:07:51.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:07:51.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 80 16 117235 117219 91.5498 56.3125 0.0102354 0.0109182 2024-09-13T13:07:51.790 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:51.829 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:51.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:51.931 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:51.962 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:51.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:51.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:52.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:52.092 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:52.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:52.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:52.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-13T13:07:52.189 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:07:52.190 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:51 smithi052 ceph-mon[36682]: 5.1a scrub starts 2024-09-13T13:07:52.190 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:51 smithi052 ceph-mon[36682]: 5.1a scrub ok 2024-09-13T13:07:52.190 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:51 smithi052 ceph-mon[36682]: 4.c scrub starts 2024-09-13T13:07:52.190 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:51 smithi052 ceph-mon[36682]: 4.c scrub ok 2024-09-13T13:07:52.190 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:51 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3499885451' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:07:52.191 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:51 smithi052 ceph-mon[29471]: 5.1a scrub starts 2024-09-13T13:07:52.191 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:51 smithi052 ceph-mon[29471]: 5.1a scrub ok 2024-09-13T13:07:52.191 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:51 smithi052 ceph-mon[29471]: 4.c scrub starts 2024-09-13T13:07:52.192 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:51 smithi052 ceph-mon[29471]: 4.c scrub ok 2024-09-13T13:07:52.192 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:51 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3499885451' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:07:52.193 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:52.246 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:52.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:52.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:51 smithi167 ceph-mon[33178]: 5.1a scrub starts 2024-09-13T13:07:52.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:51 smithi167 ceph-mon[33178]: 5.1a scrub ok 2024-09-13T13:07:52.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:51 smithi167 ceph-mon[33178]: 4.c scrub starts 2024-09-13T13:07:52.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:51 smithi167 ceph-mon[33178]: 4.c scrub ok 2024-09-13T13:07:52.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:51 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3499885451' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:07:52.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:52.376 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:52.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:52.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:52.509 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:52.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:52.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:52.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:52.648 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:52.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:52.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:52.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 81 16 118324 118308 91.2598 68.0625 0.0156362 0.010953 2024-09-13T13:07:52.789 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:52.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:52.891 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:52.921 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:52.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:53.023 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:53.053 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:53.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:53.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:53.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:53.183 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:53.184 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:53.185 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:53.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:53.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[29471]: 8.0 scrub starts 2024-09-13T13:07:53.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[29471]: pgmap v325: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 868 op/s 2024-09-13T13:07:53.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:07:53.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:07:53.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3499885451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:07:53.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[29471]: osdmap e205: 8 total, 8 up, 8 in 2024-09-13T13:07:53.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[29471]: 8.1 scrub ok 2024-09-13T13:07:53.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[29471]: 3.1d scrub starts 2024-09-13T13:07:53.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[29471]: 3.1d scrub ok 2024-09-13T13:07:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[36682]: 8.0 scrub starts 2024-09-13T13:07:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[36682]: pgmap v325: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 868 op/s 2024-09-13T13:07:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:07:53.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:07:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3499885451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:07:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[36682]: osdmap e205: 8 total, 8 up, 8 in 2024-09-13T13:07:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[36682]: 8.1 scrub ok 2024-09-13T13:07:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[36682]: 3.1d scrub starts 2024-09-13T13:07:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:52 smithi052 ceph-mon[36682]: 3.1d scrub ok 2024-09-13T13:07:53.285 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:53.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:52 smithi167 ceph-mon[33178]: 8.0 scrub starts 2024-09-13T13:07:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:52 smithi167 ceph-mon[33178]: pgmap v325: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 868 op/s 2024-09-13T13:07:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:52 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:07:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:52 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:07:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:52 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3499885451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:07:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:52 smithi167 ceph-mon[33178]: osdmap e205: 8 total, 8 up, 8 in 2024-09-13T13:07:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:52 smithi167 ceph-mon[33178]: 8.1 scrub ok 2024-09-13T13:07:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:52 smithi167 ceph-mon[33178]: 3.1d scrub starts 2024-09-13T13:07:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:52 smithi167 ceph-mon[33178]: 3.1d scrub ok 2024-09-13T13:07:53.339 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:53.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:53.354 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:07:53.441 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:53.468 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:53.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:53.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:53.569 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:53.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:53.600 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:53.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:53.701 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:53.746 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:53.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:53.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 82 16 119370 119354 90.944 65.375 0.0142981 0.0109903 2024-09-13T13:07:53.847 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:53.883 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:53.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:53.984 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:54.015 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:54.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:54.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:54.156 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:54.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:54.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:54.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:54.283 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:54.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:54.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:54.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:07:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:07:54.385 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:54.429 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:54.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:54.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:54.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:54.560 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:54.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:54.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:54.662 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:54.689 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:54.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:54.740 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:07:54.740 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:07:54.740 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:07:54.740 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:07:54.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.740 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83963 7.c 7:30d6a006:::benchmark_data_smithi052_71865_object83962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:07:54.741 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:07:54.741 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 280.188497111, 2024-09-13T13:07:54.741 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.013163161, 2024-09-13T13:07:54.741 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.741 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.741 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.741 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.741 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:07:54.741 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:07:54.741 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83963 2024-09-13T13:07:54.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.741 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.741 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.742 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.742 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:07:54.742 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.742 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.742 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:07:54.742 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.742 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.742 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547884+0000", 2024-09-13T13:07:54.742 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.779e-06 2024-09-13T13:07:54.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.743 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.743 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547907+0000", 2024-09-13T13:07:54.743 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2779999999999999e-05 2024-09-13T13:07:54.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.743 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.743 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547917+0000", 2024-09-13T13:07:54.743 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8439999999999993e-06 2024-09-13T13:07:54.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.744 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.744 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:07:54.744 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5229999999999998e-06 2024-09-13T13:07:54.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.744 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.744 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554417+0000", 2024-09-13T13:07:54.744 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064933880000000001 2024-09-13T13:07:54.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.744 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:07:54.745 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554427+0000", 2024-09-13T13:07:54.745 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4280000000000002e-06 2024-09-13T13:07:54.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.745 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.745 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.543883+0000", 2024-09-13T13:07:54.745 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98945604700000001 2024-09-13T13:07:54.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.745 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.745 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545504+0000", 2024-09-13T13:07:54.745 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016211660000000001 2024-09-13T13:07:54.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:07:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545747+0000", 2024-09-13T13:07:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024302199999999999 2024-09-13T13:07:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552352+0000", 2024-09-13T13:07:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.006605133 2024-09-13T13:07:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560880+0000", 2024-09-13T13:07:54.747 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0085281650000000007 2024-09-13T13:07:54.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.747 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.747 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560953+0000", 2024-09-13T13:07:54.747 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2096999999999998e-05 2024-09-13T13:07:54.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.747 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.747 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561045+0000", 2024-09-13T13:07:54.747 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2788999999999995e-05 2024-09-13T13:07:54.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.748 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.748 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83964 7.c 7:37783165:::benchmark_data_smithi052_71865_object83963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:07:54.748 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:07:54.748 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 280.18836094400001, 2024-09-13T13:07:54.748 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0132684729999999, 2024-09-13T13:07:54.749 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.749 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.749 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.749 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.749 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:07:54.749 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:07:54.749 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83964 2024-09-13T13:07:54.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.749 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.749 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.749 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.749 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:07:54.749 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.750 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.750 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:07:54.750 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.750 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.750 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548019+0000", 2024-09-13T13:07:54.750 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0279999999999999e-06 2024-09-13T13:07:54.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.750 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548044+0000", 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4581999999999999e-05 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548052+0000", 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3799999999999994e-06 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548059+0000", 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8329999999999996e-06 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.752 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.752 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554452+0000", 2024-09-13T13:07:54.752 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0063932119999999997 2024-09-13T13:07:54.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.752 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.752 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:07:54.752 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554459+0000", 2024-09-13T13:07:54.752 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3319999999999997e-06 2024-09-13T13:07:54.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.752 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.752 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.752 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552381+0000", 2024-09-13T13:07:54.752 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99792229399999999 2024-09-13T13:07:54.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.753 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.753 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.753 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552485+0000", 2024-09-13T13:07:54.753 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010354800000000001 2024-09-13T13:07:54.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.753 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.753 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:07:54.753 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552626+0000", 2024-09-13T13:07:54.753 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014122000000000001 2024-09-13T13:07:54.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.753 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.753 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560863+0000", 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0082373700000000008 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561139+0000", 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027546899999999997 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561192+0000", 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2986999999999999e-05 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.755 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.755 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561287+0000", 2024-09-13T13:07:54.755 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5217999999999995e-05 2024-09-13T13:07:54.755 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.755 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.755 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.756 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83985 7.c 7:386b3d8b:::benchmark_data_smithi052_71865_object83984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:07:54.756 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:07:54.756 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 280.16843958999999, 2024-09-13T13:07:54.756 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99349958199999999, 2024-09-13T13:07:54.756 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.756 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.756 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.756 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.756 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:07:54.756 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:07:54.757 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83985 2024-09-13T13:07:54.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.757 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.757 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.757 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.757 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:07:54.757 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.757 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.757 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.757 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:07:54.757 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567942+0000", 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.888e-06 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567971+0000", 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9380999999999999e-05 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568031+0000", 2024-09-13T13:07:54.758 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9731000000000001e-05 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568039+0000", 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8109999999999992e-06 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568102+0000", 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2271000000000001e-05 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:07:54.759 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568109+0000", 2024-09-13T13:07:54.760 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2570000000000001e-06 2024-09-13T13:07:54.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.760 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.760 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.557963+0000", 2024-09-13T13:07:54.760 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.989854491 2024-09-13T13:07:54.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.760 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.760 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.558066+0000", 2024-09-13T13:07:54.760 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102929 2024-09-13T13:07:54.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.761 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:07:54.761 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.558206+0000", 2024-09-13T13:07:54.761 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013928000000000001 2024-09-13T13:07:54.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.761 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.761 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561121+0000", 2024-09-13T13:07:54.761 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029151849999999998 2024-09-13T13:07:54.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.761 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.761 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561301+0000", 2024-09-13T13:07:54.761 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018019299999999999 2024-09-13T13:07:54.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.762 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.762 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.762 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561349+0000", 2024-09-13T13:07:54.762 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7639999999999998e-05 2024-09-13T13:07:54.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.762 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.762 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.762 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561439+0000", 2024-09-13T13:07:54.762 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0525000000000005e-05 2024-09-13T13:07:54.762 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.762 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.762 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83989 7.c 7:3cf9e082:::benchmark_data_smithi052_71865_object83988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 280.16179738699998, 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99086929999999995, 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83989 2024-09-13T13:07:54.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.764 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.764 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.764 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:07:54.764 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.764 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.764 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:07:54.764 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.764 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574584+0000", 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0480000000000001e-06 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574603+0000", 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8913e-05 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574615+0000", 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1678000000000001e-05 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.766 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.766 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574622+0000", 2024-09-13T13:07:54.766 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:07:54.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.766 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.766 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574674+0000", 2024-09-13T13:07:54.766 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1053000000000001e-05 2024-09-13T13:07:54.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.766 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:07:54.766 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574681+0000", 2024-09-13T13:07:54.766 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0110000000000003e-06 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561482+0000", 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98680099200000004 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561622+0000", 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014003799999999999 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:07:54.767 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561773+0000", 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015180599999999999 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.564308+0000", 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025348390000000001 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565272+0000", 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00096360599999999999 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.768 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.769 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565362+0000", 2024-09-13T13:07:54.769 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0253000000000005e-05 2024-09-13T13:07:54.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.769 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.769 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565451+0000", 2024-09-13T13:07:54.769 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9090000000000003e-05 2024-09-13T13:07:54.769 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.769 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.769 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.769 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:119477 7.a 7:5b6c862d:::benchmark_data_smithi052_71865_object119476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:07:54.770 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:50.468842+0000", 2024-09-13T13:07:54.770 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 244.267537487, 2024-09-13T13:07:54.770 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.090049216000000001, 2024-09-13T13:07:54.770 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.770 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.770 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.770 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.770 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:07:54.770 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:07:54.770 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 119477 2024-09-13T13:07:54.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.770 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.771 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.771 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.468842+0000", 2024-09-13T13:07:54.771 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.771 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.771 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.771 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.468842+0000", 2024-09-13T13:07:54.771 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.771 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.771 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.771 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.468844+0000", 2024-09-13T13:07:54.771 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.832e-06 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.468867+0000", 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3852000000000001e-05 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.468878+0000", 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1004e-05 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.772 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.468885+0000", 2024-09-13T13:07:54.773 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0190000000000001e-06 2024-09-13T13:07:54.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.773 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.773 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.773 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.469113+0000", 2024-09-13T13:07:54.773 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00022774399999999999 2024-09-13T13:07:54.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.773 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.773 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.773 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.469233+0000", 2024-09-13T13:07:54.773 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120247 2024-09-13T13:07:54.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.773 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.774 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,5", 2024-09-13T13:07:54.774 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.469373+0000", 2024-09-13T13:07:54.774 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000139114 2024-09-13T13:07:54.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.774 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.774 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.774 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.477078+0000", 2024-09-13T13:07:54.774 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0077054050000000002 2024-09-13T13:07:54.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.774 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.774 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.774 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.478765+0000", 2024-09-13T13:07:54.774 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001686788 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.557874+0000", 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.079109309000000003 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.558778+0000", 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00090368399999999998 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.775 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:50.558891+0000", 2024-09-13T13:07:54.776 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113218 2024-09-13T13:07:54.776 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.776 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.776 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.776 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.776 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.776 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151585 7.c 7:348440bb:::benchmark_data_smithi052_71865_object30484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:07:54.776 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.174012+0000", 2024-09-13T13:07:54.776 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 205.56236738600001, 2024-09-13T13:07:54.776 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.092236627000000002, 2024-09-13T13:07:54.776 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.776 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.776 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:07:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:07:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151585 2024-09-13T13:07:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174012+0000", 2024-09-13T13:07:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.777 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174012+0000", 2024-09-13T13:07:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174014+0000", 2024-09-13T13:07:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8929999999999999e-06 2024-09-13T13:07:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174020+0000", 2024-09-13T13:07:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8479999999999997e-06 2024-09-13T13:07:54.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.779 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.779 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.779 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174031+0000", 2024-09-13T13:07:54.779 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1034e-05 2024-09-13T13:07:54.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.779 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.779 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.779 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174038+0000", 2024-09-13T13:07:54.779 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7260000000000002e-06 2024-09-13T13:07:54.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.779 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.779 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.779 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.183763+0000", 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0097243079999999992 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.183904+0000", 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000141323 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.184053+0000", 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000148722 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.780 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.781 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.186978+0000", 2024-09-13T13:07:54.781 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029257300000000001 2024-09-13T13:07:54.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.781 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.781 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.781 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261696+0000", 2024-09-13T13:07:54.781 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.074717792000000005 2024-09-13T13:07:54.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.781 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.781 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.781 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261773+0000", 2024-09-13T13:07:54.781 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7096999999999997e-05 2024-09-13T13:07:54.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261885+0000", 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000111301 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.266248+0000", 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.004363853 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.782 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.783 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36744 8.f 8:f94e36b9:::benchmark_data_smithi052_101302_object36743:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:07:54.783 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.446006+0000", 2024-09-13T13:07:54.783 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 67.290373484, 2024-09-13T13:07:54.783 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.128763765, 2024-09-13T13:07:54.783 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.783 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.783 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.783 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.783 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.783 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.783 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36744 2024-09-13T13:07:54.783 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.783 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.784 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.784 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.784 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446006+0000", 2024-09-13T13:07:54.784 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.785 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.785 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.785 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446006+0000", 2024-09-13T13:07:54.785 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.785 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.785 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.785 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446008+0000", 2024-09-13T13:07:54.785 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0949999999999998e-06 2024-09-13T13:07:54.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.785 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.785 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.785 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446034+0000", 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6186999999999998e-05 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446051+0000", 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6549e-05 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446058+0000", 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6480000000000003e-06 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.786 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.787 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.448035+0000", 2024-09-13T13:07:54.787 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019772129999999998 2024-09-13T13:07:54.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.787 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.787 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.787 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.448146+0000", 2024-09-13T13:07:54.787 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011084199999999999 2024-09-13T13:07:54.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.787 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.787 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:07:54.787 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.448268+0000", 2024-09-13T13:07:54.787 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121266 2024-09-13T13:07:54.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451458+0000", 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031904899999999998 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.452999+0000", 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001540823 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.574662+0000", 2024-09-13T13:07:54.788 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.121663207 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.574726+0000", 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4369000000000007e-05 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.574770+0000", 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3075999999999999e-05 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.790 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.790 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36751 8.0 8:0c704d38:::benchmark_data_smithi052_101302_object36750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:07:54.790 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.450874+0000", 2024-09-13T13:07:54.790 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 67.285504809000003, 2024-09-13T13:07:54.790 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12726658099999999, 2024-09-13T13:07:54.790 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.790 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.790 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.790 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.790 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.790 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.790 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36751 2024-09-13T13:07:54.790 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450874+0000", 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450874+0000", 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.791 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450877+0000", 2024-09-13T13:07:54.792 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.159e-06 2024-09-13T13:07:54.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.792 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.792 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.792 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450910+0000", 2024-09-13T13:07:54.792 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.345e-05 2024-09-13T13:07:54.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.792 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.792 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.792 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450922+0000", 2024-09-13T13:07:54.792 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1912e-05 2024-09-13T13:07:54.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450929+0000", 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3180000000000001e-06 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451081+0000", 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015215699999999999 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451225+0000", 2024-09-13T13:07:54.793 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014390400000000001 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451361+0000", 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013605400000000001 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454511+0000", 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031497119999999998 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.794 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454545+0000", 2024-09-13T13:07:54.795 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3543000000000001e-05 2024-09-13T13:07:54.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.795 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.795 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.795 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578050+0000", 2024-09-13T13:07:54.795 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.123505225 2024-09-13T13:07:54.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.795 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.795 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.795 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578103+0000", 2024-09-13T13:07:54.795 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2679999999999997e-05 2024-09-13T13:07:54.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.795 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.796 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.796 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578141+0000", 2024-09-13T13:07:54.796 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8467000000000003e-05 2024-09-13T13:07:54.796 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.796 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.796 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.796 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36752 8.0 8:0ff7cdfb:::benchmark_data_smithi052_101302_object36751:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:07:54.796 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.450957+0000", 2024-09-13T13:07:54.796 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 67.285421936000006, 2024-09-13T13:07:54.796 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12729238900000001, 2024-09-13T13:07:54.796 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.797 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.797 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.797 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.797 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.797 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.797 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36752 2024-09-13T13:07:54.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.797 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.797 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.797 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.797 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450957+0000", 2024-09-13T13:07:54.797 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450957+0000", 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450958+0000", 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1699999999999997e-07 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450980+0000", 2024-09-13T13:07:54.798 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1466000000000001e-05 2024-09-13T13:07:54.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.799 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.799 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.799 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450986+0000", 2024-09-13T13:07:54.799 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.516e-06 2024-09-13T13:07:54.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.799 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.799 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.799 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450991+0000", 2024-09-13T13:07:54.799 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3090000000000003e-06 2024-09-13T13:07:54.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.799 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.800 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.800 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.452820+0000", 2024-09-13T13:07:54.800 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00182898 2024-09-13T13:07:54.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.800 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.800 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.800 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.452929+0000", 2024-09-13T13:07:54.800 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108448 2024-09-13T13:07:54.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.800 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.800 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:07:54.800 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.453078+0000", 2024-09-13T13:07:54.800 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000148663 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455024+0000", 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019467499999999999 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455035+0000", 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.801 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578177+0000", 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.123141319 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578219+0000", 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2348999999999997e-05 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578250+0000", 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0783e-05 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.802 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.803 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.803 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.803 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36753 8.f 8:f05f76d5:::benchmark_data_smithi052_101302_object36752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:07:54.803 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.451293+0000", 2024-09-13T13:07:54.803 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 67.285085902999995, 2024-09-13T13:07:54.803 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.123777199, 2024-09-13T13:07:54.803 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.803 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.803 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.803 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.803 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.803 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.803 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36753 2024-09-13T13:07:54.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.804 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.804 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.804 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.804 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451293+0000", 2024-09-13T13:07:54.804 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.804 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.804 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.804 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451293+0000", 2024-09-13T13:07:54.804 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.804 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451294+0000", 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0699999999999999e-06 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451317+0000", 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2759e-05 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451325+0000", 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1419999999999993e-06 2024-09-13T13:07:54.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451332+0000", 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.63e-06 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451491+0000", 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015913900000000001 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451597+0000", 2024-09-13T13:07:54.806 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010614400000000001 2024-09-13T13:07:54.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.807 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:07:54.807 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451725+0000", 2024-09-13T13:07:54.807 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012755899999999999 2024-09-13T13:07:54.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.807 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.807 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454052+0000", 2024-09-13T13:07:54.807 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002327487 2024-09-13T13:07:54.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.807 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.808 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458004+0000", 2024-09-13T13:07:54.808 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039520670000000001 2024-09-13T13:07:54.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.808 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.808 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.808 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.574824+0000", 2024-09-13T13:07:54.808 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11682004999999999 2024-09-13T13:07:54.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.808 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.808 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.808 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.575030+0000", 2024-09-13T13:07:54.808 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000205658 2024-09-13T13:07:54.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.575071+0000", 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0494000000000002e-05 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36762 8.0 8:04e03c57:::benchmark_data_smithi052_101302_object36761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.454968+0000", 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 67.281411074000005, 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12340187399999999, 2024-09-13T13:07:54.809 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36762 2024-09-13T13:07:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454968+0000", 2024-09-13T13:07:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454968+0000", 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454970+0000", 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.854e-06 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454996+0000", 2024-09-13T13:07:54.811 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5721e-05 2024-09-13T13:07:54.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.812 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.812 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.812 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455007+0000", 2024-09-13T13:07:54.812 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1294000000000001e-05 2024-09-13T13:07:54.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.812 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.812 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.812 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455014+0000", 2024-09-13T13:07:54.812 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7970000000000003e-06 2024-09-13T13:07:54.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.812 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.812 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.813 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455064+0000", 2024-09-13T13:07:54.813 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 83 16 120475 120459 90.6804 69.0625 0.0127108 0.0110226 2024-09-13T13:07:54.813 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0141e-05 2024-09-13T13:07:54.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.814 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.814 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455227+0000", 2024-09-13T13:07:54.814 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016291299999999999 2024-09-13T13:07:54.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.814 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:07:54.814 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455350+0000", 2024-09-13T13:07:54.814 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000122884 2024-09-13T13:07:54.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.814 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.815 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458138+0000", 2024-09-13T13:07:54.815 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027878180000000001 2024-09-13T13:07:54.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.815 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.815 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458328+0000", 2024-09-13T13:07:54.815 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000190352 2024-09-13T13:07:54.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.815 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.815 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578285+0000", 2024-09-13T13:07:54.815 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11995714 2024-09-13T13:07:54.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.816 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.816 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578335+0000", 2024-09-13T13:07:54.816 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9969999999999998e-05 2024-09-13T13:07:54.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.816 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.816 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578370+0000", 2024-09-13T13:07:54.816 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4990000000000002e-05 2024-09-13T13:07:54.816 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.816 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.816 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36765 8.f 8:fc16e397:::benchmark_data_smithi052_101302_object36764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:07:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.455809+0000", 2024-09-13T13:07:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 67.280570105999999, 2024-09-13T13:07:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12131874400000001, 2024-09-13T13:07:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36765 2024-09-13T13:07:54.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.818 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.818 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.818 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455809+0000", 2024-09-13T13:07:54.818 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.818 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.818 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455809+0000", 2024-09-13T13:07:54.818 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.819 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.819 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455811+0000", 2024-09-13T13:07:54.819 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.452e-06 2024-09-13T13:07:54.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.819 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.819 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455840+0000", 2024-09-13T13:07:54.819 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9050000000000001e-05 2024-09-13T13:07:54.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.819 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.819 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455850+0000", 2024-09-13T13:07:54.819 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.013e-05 2024-09-13T13:07:54.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.820 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.820 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455856+0000", 2024-09-13T13:07:54.820 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6630000000000004e-06 2024-09-13T13:07:54.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.820 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.820 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455915+0000", 2024-09-13T13:07:54.820 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8628999999999997e-05 2024-09-13T13:07:54.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.820 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.456126+0000", 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021076100000000001 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.456250+0000", 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012409000000000001 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458014+0000", 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0017640080000000001 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.822 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.822 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458626+0000", 2024-09-13T13:07:54.822 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00061228300000000003 2024-09-13T13:07:54.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.822 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.822 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.822 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.577037+0000", 2024-09-13T13:07:54.822 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.118410709 2024-09-13T13:07:54.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.822 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.822 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.822 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.577084+0000", 2024-09-13T13:07:54.822 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7490000000000001e-05 2024-09-13T13:07:54.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.823 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.823 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.823 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.577128+0000", 2024-09-13T13:07:54.823 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3479000000000002e-05 2024-09-13T13:07:54.823 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.823 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.823 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.823 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.823 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36768 8.b 8:dd28fad3:::benchmark_data_smithi052_101302_object36767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:07:54.823 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.458820+0000", 2024-09-13T13:07:54.823 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 67.277559552, 2024-09-13T13:07:54.824 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.127385796, 2024-09-13T13:07:54.824 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.824 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.824 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.824 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.824 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.824 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.824 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36768 2024-09-13T13:07:54.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.824 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.824 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.824 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.824 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458820+0000", 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458820+0000", 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458821+0000", 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5770000000000001e-06 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.825 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458842+0000", 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1013999999999999e-05 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458853+0000", 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0876000000000001e-05 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458871+0000", 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7371999999999999e-05 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.826 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458943+0000", 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2386999999999997e-05 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459025+0000", 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1864000000000001e-05 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459148+0000", 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012354900000000001 2024-09-13T13:07:54.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.828 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.828 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.828 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.460382+0000", 2024-09-13T13:07:54.828 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012340999999999999 2024-09-13T13:07:54.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.828 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.828 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.828 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.460748+0000", 2024-09-13T13:07:54.828 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00036591599999999998 2024-09-13T13:07:54.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.828 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.828 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.828 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.586104+0000", 2024-09-13T13:07:54.829 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.125355614 2024-09-13T13:07:54.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.829 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.829 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.586163+0000", 2024-09-13T13:07:54.829 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8563999999999997e-05 2024-09-13T13:07:54.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.829 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.829 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.586206+0000", 2024-09-13T13:07:54.829 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2963000000000002e-05 2024-09-13T13:07:54.829 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.829 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.830 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.830 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.830 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36769 8.0 8:0a2fab5a:::benchmark_data_smithi052_101302_object36768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:07:54.830 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.459028+0000", 2024-09-13T13:07:54.830 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 67.277351511999996, 2024-09-13T13:07:54.830 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12155476799999999, 2024-09-13T13:07:54.830 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.830 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.830 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.830 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.830 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.830 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.831 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36769 2024-09-13T13:07:54.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.831 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.831 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.831 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459028+0000", 2024-09-13T13:07:54.831 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.831 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.831 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459028+0000", 2024-09-13T13:07:54.831 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459029+0000", 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.283e-06 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459052+0000", 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3315e-05 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459062+0000", 2024-09-13T13:07:54.832 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2969999999999992e-06 2024-09-13T13:07:54.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.833 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.833 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.833 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459069+0000", 2024-09-13T13:07:54.833 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1690000000000003e-06 2024-09-13T13:07:54.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.833 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.833 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.833 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459120+0000", 2024-09-13T13:07:54.833 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1684999999999997e-05 2024-09-13T13:07:54.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.833 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.833 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459235+0000", 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114246 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459394+0000", 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015965599999999999 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.460679+0000", 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012848759999999999 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.461249+0000", 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00057000599999999996 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.580496+0000", 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.119247045 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.580546+0000", 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9638000000000001e-05 2024-09-13T13:07:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.580583+0000", 2024-09-13T13:07:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6551999999999998e-05 2024-09-13T13:07:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:61266 8.f 8:fd769e11:::benchmark_data_smithi052_101302_object61265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T13:07:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:03.542853+0000", 2024-09-13T13:07:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 51.193525868000002, 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.099333543999999996, 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 61266 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542853+0000", 2024-09-13T13:07:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542853+0000", 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542855+0000", 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9309999999999998e-06 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542879+0000", 2024-09-13T13:07:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4114999999999999e-05 2024-09-13T13:07:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542889+0000", 2024-09-13T13:07:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2229999999999996e-06 2024-09-13T13:07:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542896+0000", 2024-09-13T13:07:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8419999999999999e-06 2024-09-13T13:07:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542967+0000", 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1124999999999994e-05 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.543087+0000", 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012008200000000001 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.543210+0000", 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012324699999999999 2024-09-13T13:07:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.545041+0000", 2024-09-13T13:07:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001831502 2024-09-13T13:07:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.545546+0000", 2024-09-13T13:07:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00050484300000000005 2024-09-13T13:07:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.641903+0000", 2024-09-13T13:07:54.842 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.096356285 2024-09-13T13:07:54.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.842 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:54.843 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.843 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.642135+0000", 2024-09-13T13:07:54.843 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00023198899999999999 2024-09-13T13:07:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.642187+0000", 2024-09-13T13:07:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.236e-05 2024-09-13T13:07:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:61278 8.f 8:f256d1ac:::benchmark_data_smithi052_101302_object61277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T13:07:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:03.551289+0000", 2024-09-13T13:07:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 51.185090504999998, 2024-09-13T13:07:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.091975018000000006, 2024-09-13T13:07:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 61278 2024-09-13T13:07:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.551289+0000", 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.551289+0000", 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.551290+0000", 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6300000000000001e-06 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.551311+0000", 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0607000000000001e-05 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.551321+0000", 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7159999999999998e-06 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.551326+0000", 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4249999999999997e-06 2024-09-13T13:07:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.551377+0000", 2024-09-13T13:07:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1115000000000001e-05 2024-09-13T13:07:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.551466+0000", 2024-09-13T13:07:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8282999999999995e-05 2024-09-13T13:07:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:07:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.551554+0000", 2024-09-13T13:07:54.849 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8085999999999997e-05 2024-09-13T13:07:54.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.849 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.849 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.849 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.553209+0000", 2024-09-13T13:07:54.849 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00165588 2024-09-13T13:07:54.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.849 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.849 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.849 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.553338+0000", 2024-09-13T13:07:54.849 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000128372 2024-09-13T13:07:54.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.849 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.850 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.850 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.643118+0000", 2024-09-13T13:07:54.850 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.089780464000000004 2024-09-13T13:07:54.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.850 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.850 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.643215+0000", 2024-09-13T13:07:54.850 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7140999999999997e-05 2024-09-13T13:07:54.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.850 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.850 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.643264+0000", 2024-09-13T13:07:54.850 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8299e-05 2024-09-13T13:07:54.851 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.851 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.851 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.851 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.851 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:79320 8.b 8:df96c156:::benchmark_data_smithi052_101302_object79319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:07:54.851 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:17.862039+0000", 2024-09-13T13:07:54.851 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 36.874340592999999, 2024-09-13T13:07:54.851 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.093601181000000006, 2024-09-13T13:07:54.851 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.851 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.851 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.851 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 79320 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862039+0000", 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862039+0000", 2024-09-13T13:07:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862039+0000", 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8000000000000005e-07 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862066+0000", 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6327000000000001e-05 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.853 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862074+0000", 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1510000000000004e-06 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862080+0000", 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3289999999999999e-06 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.870063+0000", 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0079826040000000008 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.854 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.870200+0000", 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000137293 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.870353+0000", 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000152483 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.886551+0000", 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.016198371999999999 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.886588+0000", 2024-09-13T13:07:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7477e-05 2024-09-13T13:07:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.955109+0000", 2024-09-13T13:07:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.068520039000000005 2024-09-13T13:07:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.955586+0000", 2024-09-13T13:07:54.856 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000477231 2024-09-13T13:07:54.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.857 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.857 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.857 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.955640+0000", 2024-09-13T13:07:54.857 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4095000000000003e-05 2024-09-13T13:07:54.857 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.857 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.857 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.857 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.857 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:79321 8.b 8:da1818e8:::benchmark_data_smithi052_101302_object79320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:07:54.857 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:17.862090+0000", 2024-09-13T13:07:54.857 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 36.874288970000002, 2024-09-13T13:07:54.858 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.094540208000000001, 2024-09-13T13:07:54.858 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.858 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.858 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.858 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.858 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.858 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.858 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 79321 2024-09-13T13:07:54.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.858 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.858 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.858 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.858 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862090+0000", 2024-09-13T13:07:54.859 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.859 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.859 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862090+0000", 2024-09-13T13:07:54.859 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.859 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.859 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862091+0000", 2024-09-13T13:07:54.859 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:07:54.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.860 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.860 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862124+0000", 2024-09-13T13:07:54.860 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3192999999999999e-05 2024-09-13T13:07:54.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.860 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.860 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.860 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862131+0000", 2024-09-13T13:07:54.860 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6919999999999996e-06 2024-09-13T13:07:54.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.860 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.860 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.860 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862137+0000", 2024-09-13T13:07:54.860 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6899999999999997e-06 2024-09-13T13:07:54.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.861 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.861 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.876441+0000", 2024-09-13T13:07:54.861 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014304292999999999 2024-09-13T13:07:54.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.861 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.861 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.876566+0000", 2024-09-13T13:07:54.861 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012562399999999999 2024-09-13T13:07:54.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.862 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:07:54.862 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.876713+0000", 2024-09-13T13:07:54.862 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000146606 2024-09-13T13:07:54.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.862 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.862 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.862 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.886806+0000", 2024-09-13T13:07:54.862 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010092983999999999 2024-09-13T13:07:54.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.862 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.862 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.862 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.886982+0000", 2024-09-13T13:07:54.862 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017580300000000001 2024-09-13T13:07:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.956512+0000", 2024-09-13T13:07:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.069530442999999997 2024-09-13T13:07:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.956596+0000", 2024-09-13T13:07:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3609000000000005e-05 2024-09-13T13:07:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.864 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.956630+0000", 2024-09-13T13:07:54.864 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4561000000000003e-05 2024-09-13T13:07:54.864 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.864 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.864 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.864 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.864 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:100373 8.f 8:f8437ae4:::benchmark_data_smithi052_101302_object100372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T13:07:54.864 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:36.227085+0000", 2024-09-13T13:07:54.864 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 18.50929391, 2024-09-13T13:07:54.864 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.099780015999999999, 2024-09-13T13:07:54.864 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.865 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.865 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.865 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.865 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.865 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.865 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 100373 2024-09-13T13:07:54.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.865 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.865 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.865 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.865 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227085+0000", 2024-09-13T13:07:54.865 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227085+0000", 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227087+0000", 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.734e-06 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227108+0000", 2024-09-13T13:07:54.866 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0431000000000001e-05 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227120+0000", 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2405999999999999e-05 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227128+0000", 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1100000000000003e-06 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.867 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227196+0000", 2024-09-13T13:07:54.868 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8186000000000001e-05 2024-09-13T13:07:54.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.868 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.868 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.868 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227352+0000", 2024-09-13T13:07:54.868 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015536799999999999 2024-09-13T13:07:54.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.868 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.868 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:07:54.868 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227492+0000", 2024-09-13T13:07:54.868 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014054099999999999 2024-09-13T13:07:54.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.868 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.869 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.869 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.230182+0000", 2024-09-13T13:07:54.869 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002689552 2024-09-13T13:07:54.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.869 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.869 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.869 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.230542+0000", 2024-09-13T13:07:54.869 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000360618 2024-09-13T13:07:54.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.869 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.869 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.869 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.326764+0000", 2024-09-13T13:07:54.869 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.096222012999999995 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.326820+0000", 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6169999999999999e-05 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.326865+0000", 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4886999999999999e-05 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.871 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.871 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:100386 8.f 8:f99c19da:::benchmark_data_smithi052_101302_object100385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T13:07:54.871 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:36.234852+0000", 2024-09-13T13:07:54.871 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 18.501527061000001, 2024-09-13T13:07:54.871 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.092386695000000005, 2024-09-13T13:07:54.871 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:07:54.871 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:07:54.871 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:07:54.871 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:07:54.871 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:07:54.871 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:07:54.871 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 100386 2024-09-13T13:07:54.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.234852+0000", 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.234852+0000", 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:07:54.872 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.234854+0000", 2024-09-13T13:07:54.873 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6440000000000001e-06 2024-09-13T13:07:54.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.873 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:07:54.873 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.234877+0000", 2024-09-13T13:07:54.873 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3054999999999999e-05 2024-09-13T13:07:54.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.873 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:07:54.873 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.234890+0000", 2024-09-13T13:07:54.873 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2848e-05 2024-09-13T13:07:54.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.234897+0000", 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.007e-06 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.234960+0000", 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3152999999999997e-05 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.235051+0000", 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1131000000000006e-05 2024-09-13T13:07:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.235139+0000", 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7793000000000005e-05 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.237416+0000", 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0022767059999999999 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.237620+0000", 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020480699999999999 2024-09-13T13:07:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:07:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.327146+0000", 2024-09-13T13:07:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.089525398000000006 2024-09-13T13:07:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:07:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.327200+0000", 2024-09-13T13:07:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4351999999999997e-05 2024-09-13T13:07:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:07:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:07:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:07:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.327239+0000", 2024-09-13T13:07:54.877 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8800999999999997e-05 2024-09-13T13:07:54.877 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.877 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.877 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.877 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:07:54.877 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:07:54.877 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:07:54.916 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:54.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:54.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:55.018 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:55.029 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:07:55.058 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:55.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:55.103 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:54 smithi052 ceph-mon[36682]: pgmap v327: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T13:07:55.103 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:54 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:55.103 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:54 smithi052 ceph-mon[36682]: 3.19 scrub starts 2024-09-13T13:07:55.103 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:54 smithi052 ceph-mon[36682]: 3.19 scrub ok 2024-09-13T13:07:55.103 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:54 smithi052 ceph-mon[29471]: pgmap v327: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T13:07:55.104 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:54 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:55.104 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:54 smithi052 ceph-mon[29471]: 3.19 scrub starts 2024-09-13T13:07:55.104 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:54 smithi052 ceph-mon[29471]: 3.19 scrub ok 2024-09-13T13:07:55.160 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:55.226 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:55.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:54 smithi167 ceph-mon[33178]: pgmap v327: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T13:07:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:54 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:07:55.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:54 smithi167 ceph-mon[33178]: 3.19 scrub starts 2024-09-13T13:07:55.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:54 smithi167 ceph-mon[33178]: 3.19 scrub ok 2024-09-13T13:07:55.329 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:55.371 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:55.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:55.472 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:55.497 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:55.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:55.599 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:55.634 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:55.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:55.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:55.736 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:55.771 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:55.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:55.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 84 16 121628 121612 90.4587 72.0625 0.00855592 0.0110502 2024-09-13T13:07:55.873 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:55.913 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:55.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:55.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:56.015 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:56.043 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:56.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:56.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:55 smithi052 ceph-mon[29471]: 8.0 scrub ok 2024-09-13T13:07:56.073 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:55 smithi052 ceph-mon[36682]: 8.0 scrub ok 2024-09-13T13:07:56.144 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:56.184 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:56.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:56.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:56.286 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:56.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:56.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:56.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:55 smithi167 ceph-mon[33178]: 8.0 scrub ok 2024-09-13T13:07:56.352 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:07:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:07:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:07:56.405 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-13T13:07:56.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:56.452 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:56.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:56.554 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:56.583 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:56.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:56.685 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:56.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:56.735 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:56.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:56.784 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 85 16 122761 122745 90.2274 70.8125 0.0182697 0.0110786 2024-09-13T13:07:56.836 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:56.880 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:56.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:56.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:07:56.981 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:57.021 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:57.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:57.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:57.123 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:57.149 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:07:57.149 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:07:57.149 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:07:57.149 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:07:57.149 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:07:57.149 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:07:57.150 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:07:57.150 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:07:57.150 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:07:57.150 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:07:57.150 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:07:57.150 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:07:57.150 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:07:57.150 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:07:57.150 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:07:57.150 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:07:57.151 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:56 smithi052 ceph-mon[36682]: pgmap v328: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-13T13:07:57.151 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:56 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2504867151' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:57.151 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:56 smithi052 ceph-mon[29471]: pgmap v328: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-13T13:07:57.151 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:56 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2504867151' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:57.158 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:57.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:57.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:57.261 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:07:57.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:57.308 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:57.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:07:57.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:56 smithi167 ceph-mon[33178]: pgmap v328: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-13T13:07:57.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:56 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2504867151' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:07:57.356 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:07:57.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:07:57.446 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:57.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:07:57.548 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:57.588 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:57.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:57.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:57.690 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:57.728 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:57.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:57.784 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 86 16 123945 123929 90.0387 74 0.0075192 0.0111017 2024-09-13T13:07:57.830 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:57.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:57.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:57.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:57.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:57.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:58.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:57 smithi167 ceph-mon[33178]: 4.2 scrub starts 2024-09-13T13:07:58.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:57 smithi167 ceph-mon[33178]: 4.2 scrub ok 2024-09-13T13:07:58.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:57 smithi167 ceph-mon[33178]: 5.3 scrub starts 2024-09-13T13:07:58.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:57 smithi167 ceph-mon[33178]: 5.3 scrub ok 2024-09-13T13:07:58.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:57 smithi167 ceph-mon[33178]: 3.a scrub starts 2024-09-13T13:07:58.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:57 smithi167 ceph-mon[33178]: 3.a scrub ok 2024-09-13T13:07:58.093 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:58.115 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:58.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:58.216 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:57 smithi052 ceph-mon[36682]: 4.2 scrub starts 2024-09-13T13:07:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:57 smithi052 ceph-mon[36682]: 4.2 scrub ok 2024-09-13T13:07:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:57 smithi052 ceph-mon[36682]: 5.3 scrub starts 2024-09-13T13:07:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:57 smithi052 ceph-mon[36682]: 5.3 scrub ok 2024-09-13T13:07:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:57 smithi052 ceph-mon[36682]: 3.a scrub starts 2024-09-13T13:07:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:57 smithi052 ceph-mon[36682]: 3.a scrub ok 2024-09-13T13:07:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:57 smithi052 ceph-mon[29471]: 4.2 scrub starts 2024-09-13T13:07:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:57 smithi052 ceph-mon[29471]: 4.2 scrub ok 2024-09-13T13:07:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:57 smithi052 ceph-mon[29471]: 5.3 scrub starts 2024-09-13T13:07:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:57 smithi052 ceph-mon[29471]: 5.3 scrub ok 2024-09-13T13:07:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:57 smithi052 ceph-mon[29471]: 3.a scrub starts 2024-09-13T13:07:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:57 smithi052 ceph-mon[29471]: 3.a scrub ok 2024-09-13T13:07:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:58.253 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:58.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:58.354 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:07:58.376 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:58.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:58.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:07:58.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:58.509 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:58.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:58.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:58.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:58.651 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:58.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:58.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:58.703 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:07:58.703 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:07:58.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:07:58.784 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 87 16 124977 124961 89.7448 64.5 0.0117828 0.0111379 2024-09-13T13:07:58.797 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:58.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:07:58.898 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:58.920 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:58.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:59.021 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:59.052 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:59.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:59.154 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:59.191 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:59.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:59.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:58 smithi052 ceph-mon[36682]: 3.1c scrub starts 2024-09-13T13:07:59.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:58 smithi052 ceph-mon[36682]: 3.1c scrub ok 2024-09-13T13:07:59.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:58 smithi052 ceph-mon[36682]: 6.13 scrub starts 2024-09-13T13:07:59.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:58 smithi052 ceph-mon[36682]: 6.13 scrub ok 2024-09-13T13:07:59.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:58 smithi052 ceph-mon[36682]: pgmap v329: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-13T13:07:59.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:58 smithi052 ceph-mon[29471]: 3.1c scrub starts 2024-09-13T13:07:59.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:58 smithi052 ceph-mon[29471]: 3.1c scrub ok 2024-09-13T13:07:59.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:58 smithi052 ceph-mon[29471]: 6.13 scrub starts 2024-09-13T13:07:59.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:58 smithi052 ceph-mon[29471]: 6.13 scrub ok 2024-09-13T13:07:59.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:58 smithi052 ceph-mon[29471]: pgmap v329: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-13T13:07:59.293 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:59.319 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:59.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:59.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:58 smithi167 ceph-mon[33178]: 3.1c scrub starts 2024-09-13T13:07:59.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:58 smithi167 ceph-mon[33178]: 3.1c scrub ok 2024-09-13T13:07:59.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:58 smithi167 ceph-mon[33178]: 6.13 scrub starts 2024-09-13T13:07:59.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:58 smithi167 ceph-mon[33178]: 6.13 scrub ok 2024-09-13T13:07:59.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:58 smithi167 ceph-mon[33178]: pgmap v329: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-13T13:07:59.420 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:07:59.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:59.465 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:59.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:07:59.520 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T13:07:59.566 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:07:59.613 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:07:59.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:59.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:07:59.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:07:59.760 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:59.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:07:59.784 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 88 16 125941 125925 89.4097 60.25 0.00697104 0.0111797 2024-09-13T13:07:59.862 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:07:59.907 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:07:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:07:59.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:07:59.915 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:08:00.009 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:00.054 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:00.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:00.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:00.156 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:00.191 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:00.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:00.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[36682]: 5.13 scrub starts 2024-09-13T13:08:00.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[36682]: 5.13 scrub ok 2024-09-13T13:08:00.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[36682]: 6.14 scrub starts 2024-09-13T13:08:00.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[36682]: 6.14 scrub ok 2024-09-13T13:08:00.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:08:00.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[36682]: 3.c scrub starts 2024-09-13T13:08:00.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[36682]: 3.c scrub ok 2024-09-13T13:08:00.240 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:59 smithi167 ceph-mon[33178]: 5.13 scrub starts 2024-09-13T13:08:00.240 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:59 smithi167 ceph-mon[33178]: 5.13 scrub ok 2024-09-13T13:08:00.240 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:59 smithi167 ceph-mon[33178]: 6.14 scrub starts 2024-09-13T13:08:00.240 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:59 smithi167 ceph-mon[33178]: 6.14 scrub ok 2024-09-13T13:08:00.240 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:59 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:08:00.240 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:59 smithi167 ceph-mon[33178]: 3.c scrub starts 2024-09-13T13:08:00.240 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:07:59 smithi167 ceph-mon[33178]: 3.c scrub ok 2024-09-13T13:08:00.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[29471]: 5.13 scrub starts 2024-09-13T13:08:00.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[29471]: 5.13 scrub ok 2024-09-13T13:08:00.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[29471]: 6.14 scrub starts 2024-09-13T13:08:00.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[29471]: 6.14 scrub ok 2024-09-13T13:08:00.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:08:00.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[29471]: 3.c scrub starts 2024-09-13T13:08:00.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:07:59 smithi052 ceph-mon[29471]: 3.c scrub ok 2024-09-13T13:08:00.293 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:00.326 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:00.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:00.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:00.427 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:00.448 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:00.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:00.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:00.494 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:08:00.494 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:08:00.551 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:00.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:00.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:00.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:00.684 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:00.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:00.716 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:00.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:00.785 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 89 16 126997 126981 89.1466 66 0.0645121 0.0112098 2024-09-13T13:08:00.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:00.848 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:00.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:00.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:00.949 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:00.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:00.991 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:00.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:01.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:01.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:01.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:01.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:01.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T13:08:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:00 smithi052 ceph-mon[29471]: pgmap v330: 148 pgs: 148 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T13:08:01.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:00 smithi052 ceph-mon[36682]: pgmap v330: 148 pgs: 148 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T13:08:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:01.253 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:01.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:01.285 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:00 smithi167 ceph-mon[33178]: pgmap v330: 148 pgs: 148 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T13:08:01.356 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:01.398 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:01.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:01.500 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:01.523 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:01.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:01.596 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:08:01.626 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:01.668 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:01.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:01.769 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:01.785 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Total time run: 90.0091 2024-09-13T13:08:01.785 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Total writes made: 127926 2024-09-13T13:08:01.785 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Write size: 65536 2024-09-13T13:08:01.785 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Object size: 65536 2024-09-13T13:08:01.785 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Bandwidth (MB/sec): 88.8285 2024-09-13T13:08:01.785 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev Bandwidth: 31.4737 2024-09-13T13:08:01.785 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max bandwidth (MB/sec): 190.875 2024-09-13T13:08:01.785 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min bandwidth (MB/sec): 50.75 2024-09-13T13:08:01.785 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Average IOPS: 1421 2024-09-13T13:08:01.785 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev IOPS: 503.58 2024-09-13T13:08:01.785 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max IOPS: 3054 2024-09-13T13:08:01.785 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min IOPS: 812 2024-09-13T13:08:01.786 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Average Latency(s): 0.0112533 2024-09-13T13:08:01.786 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev Latency(s): 0.00843311 2024-09-13T13:08:01.786 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max latency(s): 0.156296 2024-09-13T13:08:01.786 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min latency(s): 0.00118759 2024-09-13T13:08:01.791 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T13:08:01.792 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:01.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:01.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:01.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:01.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:01.929 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:01.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:02.030 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:02.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:02.053 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:02.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:02.154 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:02.195 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:02.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:02.273 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:01 smithi167 ceph-mon[33178]: 6.9 scrub starts 2024-09-13T13:08:02.273 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:01 smithi167 ceph-mon[33178]: 6.9 scrub ok 2024-09-13T13:08:02.273 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:01 smithi167 ceph-mon[33178]: 8.4 scrub starts 2024-09-13T13:08:02.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:02.339 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:02.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:02.407 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:08:02.407 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:08:02.441 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:02.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:02.483 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:02.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:01 smithi052 ceph-mon[36682]: 6.9 scrub starts 2024-09-13T13:08:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:02 smithi052 ceph-mon[36682]: 6.9 scrub ok 2024-09-13T13:08:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:02 smithi052 ceph-mon[36682]: 8.4 scrub starts 2024-09-13T13:08:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:01 smithi052 ceph-mon[29471]: 6.9 scrub starts 2024-09-13T13:08:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:01 smithi052 ceph-mon[29471]: 6.9 scrub ok 2024-09-13T13:08:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:01 smithi052 ceph-mon[29471]: 8.4 scrub starts 2024-09-13T13:08:02.584 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:02.623 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:02.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:02.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:02.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:02.746 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:02.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:02.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:02.849 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:02.889 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:02.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:02.990 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:03.034 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:03.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:03.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:03.119 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T13:08:03.136 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:03.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:03.201 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:03.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:03.303 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:03.335 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:02 smithi167 ceph-mon[33178]: pgmap v331: 148 pgs: 148 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-13T13:08:03.336 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:02 smithi167 ceph-mon[33178]: 4.e scrub starts 2024-09-13T13:08:03.336 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:02 smithi167 ceph-mon[33178]: 4.e scrub ok 2024-09-13T13:08:03.336 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:02 smithi167 ceph-mon[33178]: 3.1 scrub starts 2024-09-13T13:08:03.336 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:02 smithi167 ceph-mon[33178]: 3.1 scrub ok 2024-09-13T13:08:03.336 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:02 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:08:03.336 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:02 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:08:03.381 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:03.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:03.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:03.483 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:03 smithi052 ceph-mon[36682]: pgmap v331: 148 pgs: 148 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-13T13:08:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:03 smithi052 ceph-mon[36682]: 4.e scrub starts 2024-09-13T13:08:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:03 smithi052 ceph-mon[36682]: 4.e scrub ok 2024-09-13T13:08:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:03 smithi052 ceph-mon[36682]: 3.1 scrub starts 2024-09-13T13:08:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:03 smithi052 ceph-mon[36682]: 3.1 scrub ok 2024-09-13T13:08:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:03 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:08:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:03 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:08:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:02 smithi052 ceph-mon[29471]: pgmap v331: 148 pgs: 148 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-13T13:08:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:02 smithi052 ceph-mon[29471]: 4.e scrub starts 2024-09-13T13:08:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:02 smithi052 ceph-mon[29471]: 4.e scrub ok 2024-09-13T13:08:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:02 smithi052 ceph-mon[29471]: 3.1 scrub starts 2024-09-13T13:08:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:02 smithi052 ceph-mon[29471]: 3.1 scrub ok 2024-09-13T13:08:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:02 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:08:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:02 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:08:03.520 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:08:03.521 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:03.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:03.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:03.623 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:03.688 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:03.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:03.691 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:03.731 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:03.789 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:03.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:03.822 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:03.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:03.923 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:03.957 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:03.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:03.996 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:08:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:08:04.060 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:04.107 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:04.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:04.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:04.163 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[36682]: 5.14 scrub starts 2024-09-13T13:08:04.163 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[36682]: 5.14 scrub ok 2024-09-13T13:08:04.164 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[36682]: 6.f scrub starts 2024-09-13T13:08:04.164 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[36682]: 6.f scrub ok 2024-09-13T13:08:04.164 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[36682]: 4.8 scrub starts 2024-09-13T13:08:04.164 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[36682]: 4.8 scrub ok 2024-09-13T13:08:04.164 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:08:04.164 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:08:04.164 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:08:04.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[29471]: 5.14 scrub starts 2024-09-13T13:08:04.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[29471]: 5.14 scrub ok 2024-09-13T13:08:04.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[29471]: 6.f scrub starts 2024-09-13T13:08:04.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[29471]: 6.f scrub ok 2024-09-13T13:08:04.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[29471]: 4.8 scrub starts 2024-09-13T13:08:04.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[29471]: 4.8 scrub ok 2024-09-13T13:08:04.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:08:04.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:08:04.168 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:04 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:08:04.209 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:04.244 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:04.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:04.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:04.299 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:04 smithi167 ceph-mon[33178]: 5.14 scrub starts 2024-09-13T13:08:04.299 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:04 smithi167 ceph-mon[33178]: 5.14 scrub ok 2024-09-13T13:08:04.299 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:04 smithi167 ceph-mon[33178]: 6.f scrub starts 2024-09-13T13:08:04.299 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:04 smithi167 ceph-mon[33178]: 6.f scrub ok 2024-09-13T13:08:04.299 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:04 smithi167 ceph-mon[33178]: 4.8 scrub starts 2024-09-13T13:08:04.299 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:04 smithi167 ceph-mon[33178]: 4.8 scrub ok 2024-09-13T13:08:04.299 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:04 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:08:04.299 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:04 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:08:04.299 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:04 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:08:04.346 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:04.388 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:04.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:04.445 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:08:04.445 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:08:04.490 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:04.522 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:04.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:04.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:04.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:04.649 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:04.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:04.751 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:04.784 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:04.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:04.885 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:04.932 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:04.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:05.034 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:05.067 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:05.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:05.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:05.169 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:05.204 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:05.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:05.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:05.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[36682]: 8.4 scrub ok 2024-09-13T13:08:05.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[36682]: pgmap v332: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 70 MiB/s wr, 1.21k op/s 2024-09-13T13:08:05.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[36682]: 3.1a scrub starts 2024-09-13T13:08:05.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[36682]: 3.1a scrub ok 2024-09-13T13:08:05.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:05.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:08:05.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:08:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[29471]: 8.4 scrub ok 2024-09-13T13:08:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[29471]: pgmap v332: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 70 MiB/s wr, 1.21k op/s 2024-09-13T13:08:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[29471]: 3.1a scrub starts 2024-09-13T13:08:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[29471]: 3.1a scrub ok 2024-09-13T13:08:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:08:05.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:05 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:08:05.306 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:05.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:05 smithi167 ceph-mon[33178]: 8.4 scrub ok 2024-09-13T13:08:05.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:05 smithi167 ceph-mon[33178]: pgmap v332: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 70 MiB/s wr, 1.21k op/s 2024-09-13T13:08:05.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:05 smithi167 ceph-mon[33178]: 3.1a scrub starts 2024-09-13T13:08:05.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:05 smithi167 ceph-mon[33178]: 3.1a scrub ok 2024-09-13T13:08:05.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:05.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:05 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:08:05.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:05 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:08:05.346 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:05.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:05.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:05.447 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:05.488 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:05.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:05.499 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:08:05.524 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T13:08:05.590 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:05.629 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:05.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:05.674 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:08:05.730 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:05.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:05.778 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:05.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:05.879 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:05.896 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:08:05.918 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-13T13:08:05.922 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:05.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:06.025 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:06.100 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:06.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:06.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:06.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[36682]: 4.7 scrub starts 2024-09-13T13:08:06.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[36682]: 4.7 scrub ok 2024-09-13T13:08:06.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/611561845' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:08:06.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[36682]: from='client.25399 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:08:06.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[36682]: 6.16 scrub starts 2024-09-13T13:08:06.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[36682]: 6.16 scrub ok 2024-09-13T13:08:06.142 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[36682]: 6.d scrub starts 2024-09-13T13:08:06.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[36682]: 6.d scrub ok 2024-09-13T13:08:06.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:06.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[36682]: from='client.25399 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:08:06.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[36682]: osdmap e206: 8 total, 8 up, 8 in 2024-09-13T13:08:06.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[29471]: 4.7 scrub starts 2024-09-13T13:08:06.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[29471]: 4.7 scrub ok 2024-09-13T13:08:06.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/611561845' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:08:06.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[29471]: from='client.25399 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:08:06.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[29471]: 6.16 scrub starts 2024-09-13T13:08:06.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[29471]: 6.16 scrub ok 2024-09-13T13:08:06.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[29471]: 6.d scrub starts 2024-09-13T13:08:06.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[29471]: 6.d scrub ok 2024-09-13T13:08:06.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:06.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[29471]: from='client.25399 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:08:06.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-mon[29471]: osdmap e206: 8 total, 8 up, 8 in 2024-09-13T13:08:06.202 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:06.247 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:06.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:06.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:06 smithi167 ceph-mon[33178]: 4.7 scrub starts 2024-09-13T13:08:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:06 smithi167 ceph-mon[33178]: 4.7 scrub ok 2024-09-13T13:08:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:06 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/611561845' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:08:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:06 smithi167 ceph-mon[33178]: from='client.25399 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:08:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:06 smithi167 ceph-mon[33178]: 6.16 scrub starts 2024-09-13T13:08:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:06 smithi167 ceph-mon[33178]: 6.16 scrub ok 2024-09-13T13:08:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:06 smithi167 ceph-mon[33178]: 6.d scrub starts 2024-09-13T13:08:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:06 smithi167 ceph-mon[33178]: 6.d scrub ok 2024-09-13T13:08:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:06 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:06 smithi167 ceph-mon[33178]: from='client.25399 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:08:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:06 smithi167 ceph-mon[33178]: osdmap e206: 8 total, 8 up, 8 in 2024-09-13T13:08:06.349 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:06.391 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:06.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:06.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:06.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:08:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:08:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:08:06.493 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:06.534 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:06.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:06.540 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:08:06.540 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:08:06.637 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:06.671 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:06.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:06.773 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:06.804 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:06.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:06.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:06.873 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:06.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:06.934 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:06.941 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:06.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:07.043 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:07.086 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:07.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:07.188 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:07.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:07.222 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:07.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:07.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:07 smithi167 ceph-mon[33178]: pgmap v334: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 1.06k op/s 2024-09-13T13:08:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:07 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:08:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:07 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:08:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:07 smithi167 ceph-mon[33178]: osdmap e207: 8 total, 8 up, 8 in 2024-09-13T13:08:07.323 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:07.360 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:07.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:07.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:07.462 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:07 smithi052 ceph-mon[36682]: pgmap v334: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 1.06k op/s 2024-09-13T13:08:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:07 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:08:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:07 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:08:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:07 smithi052 ceph-mon[36682]: osdmap e207: 8 total, 8 up, 8 in 2024-09-13T13:08:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:07 smithi052 ceph-mon[29471]: pgmap v334: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 1.06k op/s 2024-09-13T13:08:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:07 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:08:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:07 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:08:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:07 smithi052 ceph-mon[29471]: osdmap e207: 8 total, 8 up, 8 in 2024-09-13T13:08:07.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:07.496 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:07.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:07.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T13:08:07.597 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:07.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:07.636 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:07.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:07.737 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:07.770 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:07.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:07.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:07.872 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:07.902 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:07.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:07.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:07.950 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:08:07.981 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:08:07.981 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:08:08.005 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:08.018 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:08:08.019 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:08:08.043 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:08.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:08.059 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:08:08.059 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:08:08.102 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:08:08.103 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:08:08.138 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:08:08.138 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:08:08.146 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:08.174 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:08:08.174 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:08:08.186 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:08.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:08.209 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:08:08.209 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:08:08.241 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:08:08.241 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:08:08.288 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:08.320 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:08.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:08.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:08.422 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:08.454 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:08.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:08.557 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:08.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:08.598 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:08.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:08.699 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:08.731 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:08.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:08.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:08:08.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:08.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:08.855 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:08.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:08.957 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:08.980 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:08.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:08.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:08.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:08 smithi052 ceph-mon[36682]: pgmap v336: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 767 B/s rd, 30 MiB/s wr, 716 op/s 2024-09-13T13:08:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/956231176' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:08.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:08 smithi052 ceph-mon[29471]: pgmap v336: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 767 B/s rd, 30 MiB/s wr, 716 op/s 2024-09-13T13:08:08.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/956231176' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:09.082 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:09.118 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:09.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:09.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:09.180 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:08 smithi167 ceph-mon[33178]: pgmap v336: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 767 B/s rd, 30 MiB/s wr, 716 op/s 2024-09-13T13:08:09.181 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/956231176' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:09.219 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:09.256 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:09.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:09.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:09.359 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:09.393 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:09.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:09.495 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:09.501 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:08:09.501 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:08:09.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:08:09.502 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:08:09.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.502 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:105216 7.0 7:0c7da74e:::benchmark_data_smithi052_71865_object105215:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-13T13:08:09.502 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:36.018754+0000", 2024-09-13T13:08:09.502 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 273.47696902199999, 2024-09-13T13:08:09.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073909959999999997, 2024-09-13T13:08:09.503 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.503 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.503 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:09.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:09.503 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105216 2024-09-13T13:08:09.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.503 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018754+0000", 2024-09-13T13:08:09.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018754+0000", 2024-09-13T13:08:09.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018756+0000", 2024-09-13T13:08:09.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.711e-06 2024-09-13T13:08:09.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018808+0000", 2024-09-13T13:08:09.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1094000000000002e-05 2024-09-13T13:08:09.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018826+0000", 2024-09-13T13:08:09.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8502e-05 2024-09-13T13:08:09.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018834+0000", 2024-09-13T13:08:09.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8099999999999998e-06 2024-09-13T13:08:09.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018901+0000", 2024-09-13T13:08:09.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6760000000000005e-05 2024-09-13T13:08:09.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.019041+0000", 2024-09-13T13:08:09.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140474 2024-09-13T13:08:09.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:08:09.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.019196+0000", 2024-09-13T13:08:09.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015521 2024-09-13T13:08:09.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.022153+0000", 2024-09-13T13:08:09.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029568210000000001 2024-09-13T13:08:09.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023370+0000", 2024-09-13T13:08:09.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001216658 2024-09-13T13:08:09.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.092552+0000", 2024-09-13T13:08:09.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069182612000000004 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.092613+0000", 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0387999999999998e-05 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.092664+0000", 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.092e-05 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:119489 7.0 7:0001c7f5:::benchmark_data_smithi052_71865_object119488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:50.480010+0000", 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 259.01571318600003, 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076898633999999993, 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 119489 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.510 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480010+0000", 2024-09-13T13:08:09.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480010+0000", 2024-09-13T13:08:09.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480020+0000", 2024-09-13T13:08:09.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.057e-05 2024-09-13T13:08:09.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480071+0000", 2024-09-13T13:08:09.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1051999999999998e-05 2024-09-13T13:08:09.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480083+0000", 2024-09-13T13:08:09.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1678000000000001e-05 2024-09-13T13:08:09.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480099+0000", 2024-09-13T13:08:09.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6127e-05 2024-09-13T13:08:09.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480149+0000", 2024-09-13T13:08:09.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0058000000000001e-05 2024-09-13T13:08:09.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480303+0000", 2024-09-13T13:08:09.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015380799999999999 2024-09-13T13:08:09.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:08:09.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480458+0000", 2024-09-13T13:08:09.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015522299999999999 2024-09-13T13:08:09.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.483021+0000", 2024-09-13T13:08:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025633510000000002 2024-09-13T13:08:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.483702+0000", 2024-09-13T13:08:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00068071600000000005 2024-09-13T13:08:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.556804+0000", 2024-09-13T13:08:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073102271999999996 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.556863+0000", 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8072999999999998e-05 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.556908+0000", 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5705999999999999e-05 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.517 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:35132 8.8 8:18a4857a:::benchmark_data_smithi052_101302_object35131:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:08:09.517 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:46.489370+0000", 2024-09-13T13:08:09.517 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 83.006352731999996, 2024-09-13T13:08:09.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079236983999999996, 2024-09-13T13:08:09.517 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.517 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.517 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.517 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.517 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:08:09.517 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:09.517 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35132 2024-09-13T13:08:09.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.518 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489370+0000", 2024-09-13T13:08:09.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489370+0000", 2024-09-13T13:08:09.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489372+0000", 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4440000000000001e-06 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489417+0000", 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5073e-05 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489432+0000", 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5109999999999999e-05 2024-09-13T13:08:09.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489440+0000", 2024-09-13T13:08:09.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9200000000000004e-06 2024-09-13T13:08:09.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489495+0000", 2024-09-13T13:08:09.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4907999999999997e-05 2024-09-13T13:08:09.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489589+0000", 2024-09-13T13:08:09.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4401000000000006e-05 2024-09-13T13:08:09.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:09.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489799+0000", 2024-09-13T13:08:09.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020981400000000001 2024-09-13T13:08:09.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491453+0000", 2024-09-13T13:08:09.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016542029999999999 2024-09-13T13:08:09.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.495446+0000", 2024-09-13T13:08:09.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039925960000000002 2024-09-13T13:08:09.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.568544+0000", 2024-09-13T13:08:09.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073098053999999996 2024-09-13T13:08:09.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.568578+0000", 2024-09-13T13:08:09.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4788e-05 2024-09-13T13:08:09.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.568607+0000", 2024-09-13T13:08:09.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8673000000000001e-05 2024-09-13T13:08:09.524 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.524 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.524 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.524 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:35138 8.8 8:1549287f:::benchmark_data_smithi052_101302_object35137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:08:09.524 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:46.491520+0000", 2024-09-13T13:08:09.524 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 83.004203188000005, 2024-09-13T13:08:09.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079477935, 2024-09-13T13:08:09.524 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.524 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.525 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.525 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.525 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:08:09.525 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:09.525 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35138 2024-09-13T13:08:09.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.525 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491520+0000", 2024-09-13T13:08:09.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491520+0000", 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491521+0000", 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4640000000000001e-06 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491554+0000", 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2861999999999998e-05 2024-09-13T13:08:09.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491564+0000", 2024-09-13T13:08:09.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0504e-05 2024-09-13T13:08:09.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491572+0000", 2024-09-13T13:08:09.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2389999999999996e-06 2024-09-13T13:08:09.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491614+0000", 2024-09-13T13:08:09.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1876999999999999e-05 2024-09-13T13:08:09.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491727+0000", 2024-09-13T13:08:09.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113761 2024-09-13T13:08:09.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:09.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491860+0000", 2024-09-13T13:08:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000132736 2024-09-13T13:08:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.495482+0000", 2024-09-13T13:08:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036219070000000002 2024-09-13T13:08:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.495565+0000", 2024-09-13T13:08:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3444999999999998e-05 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.570922+0000", 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075356970999999995 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.570958+0000", 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6022999999999997e-05 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.570998+0000", 2024-09-13T13:08:09.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9146000000000001e-05 2024-09-13T13:08:09.531 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.531 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.531 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.531 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:35140 8.8 8:14c93e84:::benchmark_data_smithi052_101302_object35139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:08:09.531 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:46.492075+0000", 2024-09-13T13:08:09.531 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 83.003647942000001, 2024-09-13T13:08:09.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079130470999999994, 2024-09-13T13:08:09.531 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.531 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.531 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.532 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.532 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:08:09.532 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:09.532 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35140 2024-09-13T13:08:09.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.532 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.492075+0000", 2024-09-13T13:08:09.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.492075+0000", 2024-09-13T13:08:09.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.492076+0000", 2024-09-13T13:08:09.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1739999999999999e-06 2024-09-13T13:08:09.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.492177+0000", 2024-09-13T13:08:09.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101215 2024-09-13T13:08:09.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.492187+0000", 2024-09-13T13:08:09.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0064999999999999e-05 2024-09-13T13:08:09.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.492194+0000", 2024-09-13T13:08:09.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1470000000000004e-06 2024-09-13T13:08:09.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.493942+0000", 2024-09-13T13:08:09.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017474929999999999 2024-09-13T13:08:09.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.494068+0000", 2024-09-13T13:08:09.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012629500000000001 2024-09-13T13:08:09.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:09.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.494213+0000", 2024-09-13T13:08:09.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144831 2024-09-13T13:08:09.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.496226+0000", 2024-09-13T13:08:09.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020130259999999998 2024-09-13T13:08:09.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.496557+0000", 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000330534 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.571103+0000", 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.074546442000000004 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.571154+0000", 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0797000000000002e-05 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.571205+0000", 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1452000000000001e-05 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:36962 8.8 8:101fb546:::benchmark_data_smithi052_101302_object36961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:47.664243+0000", 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.831479739000002, 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072421352999999994, 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.538 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 36962 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.664243+0000", 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.664243+0000", 2024-09-13T13:08:09.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.664245+0000", 2024-09-13T13:08:09.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.854e-06 2024-09-13T13:08:09.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.664285+0000", 2024-09-13T13:08:09.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9852999999999999e-05 2024-09-13T13:08:09.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.727378+0000", 2024-09-13T13:08:09.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063093039000000004 2024-09-13T13:08:09.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.727389+0000", 2024-09-13T13:08:09.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1085e-05 2024-09-13T13:08:09.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.727448+0000", 2024-09-13T13:08:09.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8802999999999999e-05 2024-09-13T13:08:09.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.727578+0000", 2024-09-13T13:08:09.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013015100000000001 2024-09-13T13:08:09.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:09.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.727715+0000", 2024-09-13T13:08:09.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000137033 2024-09-13T13:08:09.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.731218+0000", 2024-09-13T13:08:09.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003503277 2024-09-13T13:08:09.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.734833+0000", 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036148880000000001 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.736587+0000", 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017542949999999999 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.736629+0000", 2024-09-13T13:08:09.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2005999999999997e-05 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.736664+0000", 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5068999999999999e-05 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:60517 8.8 8:1266f88d:::benchmark_data_smithi052_101302_object60516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:02.991489+0000", 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.504233442, 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.078013031999999996, 2024-09-13T13:08:09.545 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.546 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.546 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.546 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.546 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:08:09.546 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:09.546 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 60517 2024-09-13T13:08:09.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.546 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991489+0000", 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991489+0000", 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991491+0000", 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.562e-06 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991521+0000", 2024-09-13T13:08:09.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0216999999999999e-05 2024-09-13T13:08:09.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991532+0000", 2024-09-13T13:08:09.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0873000000000001e-05 2024-09-13T13:08:09.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991539+0000", 2024-09-13T13:08:09.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9290000000000001e-06 2024-09-13T13:08:09.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991600+0000", 2024-09-13T13:08:09.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0556000000000001e-05 2024-09-13T13:08:09.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991772+0000", 2024-09-13T13:08:09.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017275400000000001 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991867+0000", 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4965999999999998e-05 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.993734+0000", 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018671370000000001 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.993945+0000", 2024-09-13T13:08:09.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021104000000000001 2024-09-13T13:08:09.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.069398+0000", 2024-09-13T13:08:09.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075452965999999996 2024-09-13T13:08:09.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.069456+0000", 2024-09-13T13:08:09.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7336999999999997e-05 2024-09-13T13:08:09.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.069502+0000", 2024-09-13T13:08:09.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6694999999999999e-05 2024-09-13T13:08:09.553 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.553 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.553 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.553 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:79330 8.8 8:1fb9b2c0:::benchmark_data_smithi052_101302_object79329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:08:09.553 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:17.879136+0000", 2024-09-13T13:08:09.554 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.616586865999999, 2024-09-13T13:08:09.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.07979, 2024-09-13T13:08:09.554 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.554 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.554 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.554 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.554 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:08:09.554 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:09.554 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 79330 2024-09-13T13:08:09.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.555 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879136+0000", 2024-09-13T13:08:09.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879136+0000", 2024-09-13T13:08:09.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879138+0000", 2024-09-13T13:08:09.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0839999999999999e-06 2024-09-13T13:08:09.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879178+0000", 2024-09-13T13:08:09.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0398999999999997e-05 2024-09-13T13:08:09.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879192+0000", 2024-09-13T13:08:09.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3190000000000001e-05 2024-09-13T13:08:09.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879199+0000", 2024-09-13T13:08:09.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3159999999999999e-06 2024-09-13T13:08:09.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879266+0000", 2024-09-13T13:08:09.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7255999999999994e-05 2024-09-13T13:08:09.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879415+0000", 2024-09-13T13:08:09.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014886899999999999 2024-09-13T13:08:09.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:09.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879550+0000", 2024-09-13T13:08:09.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013511900000000001 2024-09-13T13:08:09.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.884904+0000", 2024-09-13T13:08:09.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0053537819999999996 2024-09-13T13:08:09.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.884922+0000", 2024-09-13T13:08:09.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7588000000000002e-05 2024-09-13T13:08:09.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.958827+0000", 2024-09-13T13:08:09.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073905549000000001 2024-09-13T13:08:09.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.958883+0000", 2024-09-13T13:08:09.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5890000000000002e-05 2024-09-13T13:08:09.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.958926+0000", 2024-09-13T13:08:09.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2957999999999998e-05 2024-09-13T13:08:09.562 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.563 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.563 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.563 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:79331 8.8 8:13289dff:::benchmark_data_smithi052_101302_object79330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:08:09.563 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:17.879266+0000", 2024-09-13T13:08:09.563 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.616456462000002, 2024-09-13T13:08:09.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084187697000000006, 2024-09-13T13:08:09.564 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.564 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.564 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.564 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.564 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:08:09.565 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:09.565 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 79331 2024-09-13T13:08:09.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.565 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879266+0000", 2024-09-13T13:08:09.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879266+0000", 2024-09-13T13:08:09.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879268+0000", 2024-09-13T13:08:09.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.454e-06 2024-09-13T13:08:09.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879330+0000", 2024-09-13T13:08:09.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2682999999999996e-05 2024-09-13T13:08:09.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879339+0000", 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6389999999999998e-06 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879344+0000", 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.977e-06 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.882487+0000", 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003142677 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.882606+0000", 2024-09-13T13:08:09.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011966 2024-09-13T13:08:09.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:09.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.882740+0000", 2024-09-13T13:08:09.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013381299999999999 2024-09-13T13:08:09.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.885073+0000", 2024-09-13T13:08:09.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023330849999999999 2024-09-13T13:08:09.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.885588+0000", 2024-09-13T13:08:09.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051493199999999995 2024-09-13T13:08:09.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.963389+0000", 2024-09-13T13:08:09.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077800367999999995 2024-09-13T13:08:09.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.963428+0000", 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9514999999999997e-05 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.963454+0000", 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5894e-05 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.572 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:79349 8.8 8:18091f7f:::benchmark_data_smithi052_101302_object79348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:08:09.573 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:17.895616+0000", 2024-09-13T13:08:09.573 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.600106421, 2024-09-13T13:08:09.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076369714000000005, 2024-09-13T13:08:09.573 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.573 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.573 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.573 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.573 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:08:09.573 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:09.573 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 79349 2024-09-13T13:08:09.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.573 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895616+0000", 2024-09-13T13:08:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895616+0000", 2024-09-13T13:08:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895618+0000", 2024-09-13T13:08:09.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1200000000000001e-06 2024-09-13T13:08:09.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895672+0000", 2024-09-13T13:08:09.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4212000000000002e-05 2024-09-13T13:08:09.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895681+0000", 2024-09-13T13:08:09.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1859999999999998e-06 2024-09-13T13:08:09.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895686+0000", 2024-09-13T13:08:09.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9339999999999997e-06 2024-09-13T13:08:09.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895731+0000", 2024-09-13T13:08:09.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5485999999999998e-05 2024-09-13T13:08:09.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895828+0000", 2024-09-13T13:08:09.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6967999999999997e-05 2024-09-13T13:08:09.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:09.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895946+0000", 2024-09-13T13:08:09.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118051 2024-09-13T13:08:09.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.898138+0000", 2024-09-13T13:08:09.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021917960000000002 2024-09-13T13:08:09.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.898536+0000", 2024-09-13T13:08:09.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000398205 2024-09-13T13:08:09.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.971907+0000", 2024-09-13T13:08:09.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073370608000000004 2024-09-13T13:08:09.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.971943+0000", 2024-09-13T13:08:09.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6480000000000003e-05 2024-09-13T13:08:09.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.971986+0000", 2024-09-13T13:08:09.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2667999999999999e-05 2024-09-13T13:08:09.581 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.581 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.581 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.582 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:79352 8.8 8:126f2653:::benchmark_data_smithi052_101302_object79351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:08:09.582 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:17.899336+0000", 2024-09-13T13:08:09.582 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.596386817999999, 2024-09-13T13:08:09.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072801497000000007, 2024-09-13T13:08:09.582 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.582 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.582 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.582 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.582 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:08:09.582 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:09.582 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 79352 2024-09-13T13:08:09.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.583 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899336+0000", 2024-09-13T13:08:09.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899336+0000", 2024-09-13T13:08:09.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899337+0000", 2024-09-13T13:08:09.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.437e-06 2024-09-13T13:08:09.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899399+0000", 2024-09-13T13:08:09.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1346999999999994e-05 2024-09-13T13:08:09.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899414+0000", 2024-09-13T13:08:09.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5511000000000001e-05 2024-09-13T13:08:09.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899421+0000", 2024-09-13T13:08:09.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7819999999999996e-06 2024-09-13T13:08:09.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899472+0000", 2024-09-13T13:08:09.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1097999999999998e-05 2024-09-13T13:08:09.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899569+0000", 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6386999999999997e-05 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899656+0000", 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7394999999999999e-05 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.902850+0000", 2024-09-13T13:08:09.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031939540000000002 2024-09-13T13:08:09.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:09.590 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:09.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:09.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.902890+0000", 2024-09-13T13:08:09.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0330999999999997e-05 2024-09-13T13:08:09.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.972040+0000", 2024-09-13T13:08:09.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069149585999999999 2024-09-13T13:08:09.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.972097+0000", 2024-09-13T13:08:09.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7182000000000002e-05 2024-09-13T13:08:09.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.972137+0000", 2024-09-13T13:08:09.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0487e-05 2024-09-13T13:08:09.593 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.593 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.593 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.593 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:79353 8.8 8:16cfcad1:::benchmark_data_smithi052_101302_object79352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:08:09.593 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:17.899442+0000", 2024-09-13T13:08:09.593 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.596280991999997, 2024-09-13T13:08:09.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072783257000000004, 2024-09-13T13:08:09.593 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.593 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.594 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.594 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.594 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:08:09.594 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:09.594 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 79353 2024-09-13T13:08:09.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.594 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899442+0000", 2024-09-13T13:08:09.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899442+0000", 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899443+0000", 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9800000000000003e-07 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899472+0000", 2024-09-13T13:08:09.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9139999999999999e-05 2024-09-13T13:08:09.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899478+0000", 2024-09-13T13:08:09.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2639999999999997e-06 2024-09-13T13:08:09.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899482+0000", 2024-09-13T13:08:09.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.602e-06 2024-09-13T13:08:09.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.901362+0000", 2024-09-13T13:08:09.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001880659 2024-09-13T13:08:09.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.901475+0000", 2024-09-13T13:08:09.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112524 2024-09-13T13:08:09.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.901590+0000", 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011524899999999999 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.902996+0000", 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001406399 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.903148+0000", 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015181400000000001 2024-09-13T13:08:09.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.972166+0000", 2024-09-13T13:08:09.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069017700000000001 2024-09-13T13:08:09.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.972195+0000", 2024-09-13T13:08:09.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9385000000000001e-05 2024-09-13T13:08:09.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.972225+0000", 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9723e-05 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135962 8.3 e207/169 8:c288bb32:::benchmark_data_smithi052_101302_object8033:head v 207'8600, mlcod=207'8598)", 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.798736+0000", 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.69698682700000003, 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.214192363, 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.600 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.601 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.601 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798736+0000", 2024-09-13T13:08:09.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798736+0000", 2024-09-13T13:08:09.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798738+0000", 2024-09-13T13:08:09.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.542e-06 2024-09-13T13:08:09.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798746+0000", 2024-09-13T13:08:09.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7299999999999994e-06 2024-09-13T13:08:09.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798754+0000", 2024-09-13T13:08:09.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7770000000000002e-06 2024-09-13T13:08:09.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798760+0000", 2024-09-13T13:08:09.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7769999999999999e-06 2024-09-13T13:08:09.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798809+0000", 2024-09-13T13:08:09.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8714000000000001e-05 2024-09-13T13:08:09.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798822+0000", 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3737000000000001e-05 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012889+0000", 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.21406626500000001 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012928+0000", 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9820999999999997e-05 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.605 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.606 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.606 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:135964 8.8 8:1e05dc47:::benchmark_data_smithi052_101302_object8035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:08:09.606 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.800142+0000", 2024-09-13T13:08:09.606 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.69558073200000003, 2024-09-13T13:08:09.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20392829400000001, 2024-09-13T13:08:09.606 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.606 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.606 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.607 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.607 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:08:09.607 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:09.607 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135964 2024-09-13T13:08:09.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.607 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800142+0000", 2024-09-13T13:08:09.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800142+0000", 2024-09-13T13:08:09.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800144+0000", 2024-09-13T13:08:09.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5719999999999999e-06 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800149+0000", 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2059999999999998e-06 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800159+0000", 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9110000000000005e-06 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800165+0000", 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6640000000000001e-06 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800242+0000", 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6415999999999995e-05 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800369+0000", 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000127238 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:09.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800490+0000", 2024-09-13T13:08:09.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120827 2024-09-13T13:08:09.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803447+0000", 2024-09-13T13:08:09.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029568620000000002 2024-09-13T13:08:09.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803750+0000", 2024-09-13T13:08:09.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030363800000000001 2024-09-13T13:08:09.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003963+0000", 2024-09-13T13:08:09.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200212532 2024-09-13T13:08:09.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004013+0000", 2024-09-13T13:08:09.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9765999999999998e-05 2024-09-13T13:08:09.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004070+0000", 2024-09-13T13:08:09.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7661999999999998e-05 2024-09-13T13:08:09.614 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.614 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.614 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.614 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:135967 8.4 8:2cd9f03c:::benchmark_data_smithi052_101302_object8038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:08:09.614 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.800185+0000", 2024-09-13T13:08:09.614 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.69553739000000003, 2024-09-13T13:08:09.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20402237600000001, 2024-09-13T13:08:09.615 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.615 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.615 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.615 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:09.615 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:08:09.615 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:09.616 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135967 2024-09-13T13:08:09.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.616 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800185+0000", 2024-09-13T13:08:09.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800185+0000", 2024-09-13T13:08:09.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800186+0000", 2024-09-13T13:08:09.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8999999999999996e-07 2024-09-13T13:08:09.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.618 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.618 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800189+0000", 2024-09-13T13:08:09.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.689e-06 2024-09-13T13:08:09.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800194+0000", 2024-09-13T13:08:09.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1309999999999997e-06 2024-09-13T13:08:09.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800199+0000", 2024-09-13T13:08:09.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6959999999999996e-06 2024-09-13T13:08:09.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800250+0000", 2024-09-13T13:08:09.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1402e-05 2024-09-13T13:08:09.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800389+0000", 2024-09-13T13:08:09.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000138596 2024-09-13T13:08:09.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:08:09.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800490+0000", 2024-09-13T13:08:09.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010134899999999999 2024-09-13T13:08:09.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:09.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803436+0000", 2024-09-13T13:08:09.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029464529999999999 2024-09-13T13:08:09.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004082+0000", 2024-09-13T13:08:09.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200645886 2024-09-13T13:08:09.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:09.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004122+0000", 2024-09-13T13:08:09.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9864000000000003e-05 2024-09-13T13:08:09.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004169+0000", 2024-09-13T13:08:09.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7052000000000003e-05 2024-09-13T13:08:09.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.624 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004208+0000", 2024-09-13T13:08:09.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8668000000000003e-05 2024-09-13T13:08:09.624 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.625 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.625 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.625 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135966 8.3 e207/169 8:cd1f5880:::benchmark_data_smithi052_101302_object8037:head v 207'8601, mlcod=207'8598)", 2024-09-13T13:08:09.625 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.802377+0000", 2024-09-13T13:08:09.625 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.69334613499999997, 2024-09-13T13:08:09.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.21058527299999999, 2024-09-13T13:08:09.626 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.626 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.626 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.626 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802377+0000", 2024-09-13T13:08:09.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802377+0000", 2024-09-13T13:08:09.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802379+0000", 2024-09-13T13:08:09.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.827e-06 2024-09-13T13:08:09.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802390+0000", 2024-09-13T13:08:09.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1623999999999999e-05 2024-09-13T13:08:09.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802402+0000", 2024-09-13T13:08:09.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1564e-05 2024-09-13T13:08:09.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802410+0000", 2024-09-13T13:08:09.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2719999999999997e-06 2024-09-13T13:08:09.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802483+0000", 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2840000000000001e-05 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802503+0000", 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0452e-05 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012933+0000", 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.210429965 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012962+0000", 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8728999999999999e-05 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135968 8.2 e207/169 8:4f16015f:::benchmark_data_smithi052_101302_object8039:head v 207'8502, mlcod=207'8500)", 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.803228+0000", 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.69249511799999997, 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20068414800000001, 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.631 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.632 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803228+0000", 2024-09-13T13:08:09.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803228+0000", 2024-09-13T13:08:09.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803229+0000", 2024-09-13T13:08:09.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2399999999999996e-07 2024-09-13T13:08:09.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803234+0000", 2024-09-13T13:08:09.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6400000000000002e-06 2024-09-13T13:08:09.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803244+0000", 2024-09-13T13:08:09.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6770000000000007e-06 2024-09-13T13:08:09.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803251+0000", 2024-09-13T13:08:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0149999999999998e-06 2024-09-13T13:08:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803344+0000", 2024-09-13T13:08:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2656000000000002e-05 2024-09-13T13:08:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803370+0000", 2024-09-13T13:08:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6156000000000002e-05 2024-09-13T13:08:09.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003858+0000", 2024-09-13T13:08:09.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20048846400000001 2024-09-13T13:08:09.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003912+0000", 2024-09-13T13:08:09.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3616000000000003e-05 2024-09-13T13:08:09.636 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.636 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.637 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.637 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.25222.0:135964 8.8 e207/169 ondisk, result = 0)", 2024-09-13T13:08:09.637 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.803563+0000", 2024-09-13T13:08:09.637 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.69215976800000001, 2024-09-13T13:08:09.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200512417, 2024-09-13T13:08:09.637 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.637 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.637 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:08:09.638 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803563+0000", 2024-09-13T13:08:09.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803563+0000", 2024-09-13T13:08:09.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.804856+0000", 2024-09-13T13:08:09.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001293055 2024-09-13T13:08:09.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.804861+0000", 2024-09-13T13:08:09.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5800000000000002e-06 2024-09-13T13:08:09.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003842+0000", 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19898099699999999 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003857+0000", 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5021999999999999e-05 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003949+0000", 2024-09-13T13:08:09.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1824999999999996e-05 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003956+0000", 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9070000000000007e-06 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004075+0000", 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011903100000000001 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.25222.0:135967 8.4 e207/169 ondisk, result = 0)", 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.803807+0000", 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.69191538799999996, 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200283038, 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803807+0000", 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803807+0000", 2024-09-13T13:08:09.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803808+0000", 2024-09-13T13:08:09.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.032e-06 2024-09-13T13:08:09.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803812+0000", 2024-09-13T13:08:09.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0069999999999999e-06 2024-09-13T13:08:09.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803820+0000", 2024-09-13T13:08:09.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5599999999999996e-06 2024-09-13T13:08:09.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003847+0000", 2024-09-13T13:08:09.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20002690300000001 2024-09-13T13:08:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004067+0000", 2024-09-13T13:08:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021978000000000001 2024-09-13T13:08:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004075+0000", 2024-09-13T13:08:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6430000000000001e-06 2024-09-13T13:08:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004090+0000", 2024-09-13T13:08:09.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5113000000000001e-05 2024-09-13T13:08:09.646 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.646 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.646 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.646 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135970 8.3 e207/169 8:ce0cc5da:::benchmark_data_smithi052_101302_object8041:head v 207'8602, mlcod=207'8598)", 2024-09-13T13:08:09.646 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.806702+0000", 2024-09-13T13:08:09.646 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.68902035500000003, 2024-09-13T13:08:09.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20628901199999999, 2024-09-13T13:08:09.646 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:09.646 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806702+0000", 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806702+0000", 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:09.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806703+0000", 2024-09-13T13:08:09.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0109999999999999e-06 2024-09-13T13:08:09.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:09.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806711+0000", 2024-09-13T13:08:09.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.52e-06 2024-09-13T13:08:09.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:09.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003876+0000", 2024-09-13T13:08:09.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19716455899999999 2024-09-13T13:08:09.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:09.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003884+0000", 2024-09-13T13:08:09.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2269999999999993e-06 2024-09-13T13:08:09.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:09.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003930+0000", 2024-09-13T13:08:09.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6368000000000001e-05 2024-09-13T13:08:09.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:09.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003955+0000", 2024-09-13T13:08:09.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4374000000000001e-05 2024-09-13T13:08:09.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012964+0000", 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090092720000000005 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012991+0000", 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7681000000000001e-05 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:09.651 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:08:09.691 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:09.731 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:09.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:09.832 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:09.869 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:09.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:09.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:09.970 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:09.989 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:10.023 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:10.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:10.124 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:10.165 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:10.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:10.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:10.267 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:10.297 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:10.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:10.398 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:10.430 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:10.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:10.532 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:10.571 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:10.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:10.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:10.673 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:10.715 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:10.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:10.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:10.816 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:10.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:10.847 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:10.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:10.949 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:10.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:10.987 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:10.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:11.088 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:11.126 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:11.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:11.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:11.128 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:10 smithi167 ceph-mon[33178]: pgmap v337: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 25 MiB/s wr, 1.28k op/s 2024-09-13T13:08:11.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T13:08:11.228 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:11.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:10 smithi052 ceph-mon[29471]: pgmap v337: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 25 MiB/s wr, 1.28k op/s 2024-09-13T13:08:11.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:10 smithi052 ceph-mon[36682]: pgmap v337: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 25 MiB/s wr, 1.28k op/s 2024-09-13T13:08:11.264 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:11.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:11.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:11.365 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:11.408 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:11.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:11.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:11.510 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:11.542 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:11.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:11.567 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:08:11.644 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:11.681 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:11.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:11.783 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:11.817 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:11.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:11.857 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:08:11.919 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:11.954 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:11.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:11.966 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:08:12.039 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2695418024' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:12.055 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:12.100 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:12.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:12.182 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:08:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:139927 8.c e208/169 8:39571d17:::benchmark_data_smithi052_101302_object11998:head v 208'8668, mlcod=208'8665)", 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:12.180696+0000", 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00071605200000000001, 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00073450399999999997, 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:12.180696+0000", 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:12.180696+0000", 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:12.180698+0000", 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5039999999999999e-06 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:12.180704+0000", 2024-09-13T13:08:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0249999999999999e-06 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:12.180715+0000", 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1354e-05 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:12.180721+0000", 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7359999999999998e-06 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:12.180769+0000", 2024-09-13T13:08:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7874000000000003e-05 2024-09-13T13:08:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:12.180783+0000", 2024-09-13T13:08:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4199000000000001e-05 2024-09-13T13:08:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:08:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-09-13T13:08:12.186 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:08:12.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:12.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:12.238 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:12.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:12.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2695418024' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:12.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2695418024' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:12.340 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:12.382 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:12.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:12.484 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:12.525 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:12.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:12.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:12.627 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:12.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:12.660 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:12.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:12.761 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:12.816 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:12.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:12.858 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:12.917 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:12.948 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:12.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:12.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:12.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:12 smithi052 ceph-mon[29471]: pgmap v338: 148 pgs: 148 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 5.1 MiB/s wr, 941 op/s 2024-09-13T13:08:12.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:12 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:12.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:12 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2695418024' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:08:12.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:12 smithi052 ceph-mon[29471]: osdmap e208: 8 total, 8 up, 8 in 2024-09-13T13:08:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:12 smithi052 ceph-mon[36682]: pgmap v338: 148 pgs: 148 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 5.1 MiB/s wr, 941 op/s 2024-09-13T13:08:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:12 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:12 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2695418024' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:08:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:12 smithi052 ceph-mon[36682]: osdmap e208: 8 total, 8 up, 8 in 2024-09-13T13:08:13.023 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:12 smithi167 ceph-mon[33178]: pgmap v338: 148 pgs: 148 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 5.1 MiB/s wr, 941 op/s 2024-09-13T13:08:13.023 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:12 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:13.023 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:12 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2695418024' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:08:13.023 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:12 smithi167 ceph-mon[33178]: osdmap e208: 8 total, 8 up, 8 in 2024-09-13T13:08:13.052 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:13.072 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T13:08:13.084 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:13.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:13.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:13.186 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:13.223 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:13.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:13.325 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:13.359 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:13.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:13.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:13.458 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:08:13.461 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:13.503 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:13.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:13.556 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:08:13.557 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:08:13.581 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:08:13.581 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:08:13.616 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:08:13.616 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:08:13.655 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:08:13.655 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:08:13.657 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:13.691 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:08:13.691 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:08:13.706 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:13.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:13.726 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:08:13.727 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:08:13.759 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:08:13.759 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:08:13.797 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:08:13.798 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:08:13.808 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:13.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:13.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:13.945 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:13.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:13.984 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:13.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:13.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2554567349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:13.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2554567349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:14.086 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:14.117 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:14.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:14.125 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2554567349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:14.126 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:08:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:08:14.126 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:08:14.126 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:08:14.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:14.255 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:14.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:14.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:14.357 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:14.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:14.395 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:14.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:14.496 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:14.533 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:14.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:14.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:14.589 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:08:14.589 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:08:14.590 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-09-13T13:08:14.590 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-09-13T13:08:14.591 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:08:14.635 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:14.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:14.684 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:14.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:14.732 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:08:14.786 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:14.813 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:14.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:14.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T13:08:14.915 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:14.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:14.968 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:14.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:15.069 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:15.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:15.100 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:15.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:15.115 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:15.201 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:15.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:14 smithi052 ceph-mon[36682]: pgmap v340: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 816 KiB/s wr, 1.34k op/s 2024-09-13T13:08:15.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:14 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:15.241 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:08:15.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:14 smithi052 ceph-mon[29471]: pgmap v340: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 816 KiB/s wr, 1.34k op/s 2024-09-13T13:08:15.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:14 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:15.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:15.243 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:15.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:15.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:14 smithi167 ceph-mon[33178]: pgmap v340: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 816 KiB/s wr, 1.34k op/s 2024-09-13T13:08:15.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:14 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:15.344 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:15.380 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:15.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:15.482 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:15.525 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:15.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:15.627 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:15.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:15.685 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:15.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:15.786 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:15.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:15.836 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:15.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:15.845 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:08:15.845 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:08:15.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:15.965 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:15.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:16.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:16.097 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:16.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:16.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:08:16.155 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:08:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:08:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:08:16.161 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:08:16.199 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:16.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:16.249 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:16.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:16.350 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:16.377 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:16.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:16.479 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:16.515 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:16.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:16.617 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:16.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:08:16.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:16.655 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:16.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:16.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T13:08:16.757 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:16.831 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:16.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:16.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:16.881 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:08:16.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:16.963 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:16.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:16.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:16 smithi052 ceph-mon[29471]: pgmap v341: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-13T13:08:16.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:16 smithi052 ceph-mon[29471]: from='client.25420 v1:172.21.15.52:0/1147110473' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:08:16.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:16 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/100987400' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:08:16.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:16 smithi052 ceph-mon[29471]: from='client.35423 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:08:16.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:16 smithi052 ceph-mon[36682]: pgmap v341: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-13T13:08:16.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:16 smithi052 ceph-mon[36682]: from='client.25420 v1:172.21.15.52:0/1147110473' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:08:16.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:16 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/100987400' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:08:16.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:16 smithi052 ceph-mon[36682]: from='client.35423 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:08:17.016 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:08:17.065 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:17.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:16 smithi167 ceph-mon[33178]: pgmap v341: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-13T13:08:17.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:16 smithi167 ceph-mon[33178]: from='client.25420 v1:172.21.15.52:0/1147110473' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:08:17.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:16 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/100987400' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:08:17.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:16 smithi167 ceph-mon[33178]: from='client.35423 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:08:17.108 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:17.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:17.210 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:17.242 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:17.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:17.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:17.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:17.372 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:17.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:17.473 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:17.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:17.527 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:17.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:17.529 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-09-13T13:08:15.741476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7527864883,"num_objects":115057,"num_object_clones":0,"num_object_copies":345171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115057,"num_whiteouts":0,"num_read":1272,"num_read_kb":1129,"num_write":141545,"num_write_kb":8188938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86056,"ondisk_log_size":86056,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":24070900,"kb_used_data":22201756,"kb_used_omap":15,"kb_used_meta":1868784,"kb_avail":725890316,"statfs":{"total":767960285184,"available":743311683584,"internally_reserved":0,"allocated":22734598144,"data_stored":22731487105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":1913635609},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,6,8],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-731840512,"num_objects":-11167,"num_object_clones":0,"num_object_copies":-33501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11167,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11167,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.239708"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":326,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.880574+0000","last_change":"2024-09-13T13:06:44.304525+0000","last_active":"2024-09-13T13:08:11.880574+0000","last_peered":"2024-09-13T13:08:11.880574+0000","last_clean":"2024-09-13T13:08:11.880574+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:08:11.880574+0000","last_undegraded":"2024-09-13T13:08:11.880574+0000","last_fullsized":"2024-09-13T13:08:11.880574+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:44.304444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.304444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":341,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.858774+0000","last_change":"2024-09-13T13:04:48.899982+0000","last_active":"2024-09-13T13:08:11.858774+0000","last_peered":"2024-09-13T13:08:11.858774+0000","last_clean":"2024-09-13T13:08:11.858774+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:08:11.858774+0000","last_undegraded":"2024-09-13T13:08:11.858774+0000","last_fullsized":"2024-09-13T13:08:11.858774+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:48.899915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:48.899915+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:05.856048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":318,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861199+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:08:11.861199+0000","last_peered":"2024-09-13T13:08:11.861199+0000","last_clean":"2024-09-13T13:08:11.861199+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:08:11.861199+0000","last_undegraded":"2024-09-13T13:08:11.861199+0000","last_fullsized":"2024-09-13T13:08:11.861199+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:07.636691+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":306,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.860612+0000","last_change":"2024-09-13T13:05:01.076346+0000","last_active":"2024-09-13T13:08:11.860612+0000","last_peered":"2024-09-13T13:08:11.860612+0000","last_clean":"2024-09-13T13:08:11.860612+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:08:11.860612+0000","last_undegraded":"2024-09-13T13:08:11.860612+0000","last_fullsized":"2024-09-13T13:08:11.860612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.076275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.076275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:30.838887+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":324,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.865279+0000","last_change":"2024-09-13T13:07:22.605051+0000","last_active":"2024-09-13T13:08:11.865279+0000","last_peered":"2024-09-13T13:08:11.865279+0000","last_clean":"2024-09-13T13:08:11.865279+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:08:11.865279+0000","last_undegraded":"2024-09-13T13:08:11.865279+0000","last_fullsized":"2024-09-13T13:08:11.865279+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:22.604985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:22.604985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:27.869983+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":329,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.865688+0000","last_change":"2024-09-13T13:04:02.757267+0000","last_active":"2024-09-13T13:08:11.865688+0000","last_peered":"2024-09-13T13:08:11.865688+0000","last_clean":"2024-09-13T13:08:11.865688+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:08:11.865688+0000","last_undegraded":"2024-09-13T13:08:11.865688+0000","last_fullsized":"2024-09-13T13:08:11.865688+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:41.233259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:41.233259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:27.669426+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":354,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.874245+0000","last_change":"2024-09-13T13:07:09.549946+0000","last_active":"2024-09-13T13:08:11.874245+0000","last_peered":"2024-09-13T13:08:11.874245+0000","last_clean":"2024-09-13T13:08:11.874245+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:08:11.874245+0000","last_undegraded":"2024-09-13T13:08:11.874245+0000","last_fullsized":"2024-09-13T13:08:11.874245+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:07:09.549857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:09.549857+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:27.521844+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":329,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.858688+0000","last_change":"2024-09-13T13:07:22.928038+0000","last_active":"2024-09-13T13:08:11.858688+0000","last_peered":"2024-09-13T13:08:11.858688+0000","last_clean":"2024-09-13T13:08:11.858688+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:08:11.858688+0000","last_undegraded":"2024-09-13T13:08:11.858688+0000","last_fullsized":"2024-09-13T13:08:11.858688+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:22.927972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:22.927972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:40.607997+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":350,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.864433+0000","last_change":"2024-09-13T13:07:52.373558+0000","last_active":"2024-09-13T13:08:11.864433+0000","last_peered":"2024-09-13T13:08:11.864433+0000","last_clean":"2024-09-13T13:08:11.864433+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:08:11.864433+0000","last_undegraded":"2024-09-13T13:08:11.864433+0000","last_fullsized":"2024-09-13T13:08:11.864433+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:07:52.373485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:52.373485+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:58.189353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":346,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861699+0000","last_change":"2024-09-13T13:07:24.894436+0000","last_active":"2024-09-13T13:08:11.861699+0000","last_peered":"2024-09-13T13:08:11.861699+0000","last_clean":"2024-09-13T13:08:11.861699+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:08:11.861699+0000","last_undegraded":"2024-09-13T13:08:11.861699+0000","last_fullsized":"2024-09-13T13:08:11.861699+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:07:24.894360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:24.894360+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:48.354710+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":325,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.863164+0000","last_change":"2024-09-13T13:06:50.665256+0000","last_active":"2024-09-13T13:08:11.863164+0000","last_peered":"2024-09-13T13:08:11.863164+0000","last_clean":"2024-09-13T13:08:11.863164+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:08:11.863164+0000","last_undegraded":"2024-09-13T13:08:11.863164+0000","last_fullsized":"2024-09-13T13:08:11.863164+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:50.665191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.665191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":313,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.878346+0000","last_change":"2024-09-13T13:07:23.222566+0000","last_active":"2024-09-13T13:08:11.878346+0000","last_peered":"2024-09-13T13:08:11.878346+0000","last_clean":"2024-09-13T13:08:11.878346+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:08:11.878346+0000","last_undegraded":"2024-09-13T13:08:11.878346+0000","last_fullsized":"2024-09-13T13:08:11.878346+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:23.222487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:23.222487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:49.163807+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":341,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.863422+0000","last_change":"2024-09-13T13:07:57.425510+0000","last_active":"2024-09-13T13:08:11.863422+0000","last_peered":"2024-09-13T13:08:11.863422+0000","last_clean":"2024-09-13T13:08:11.863422+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:08:11.863422+0000","last_undegraded":"2024-09-13T13:08:11.863422+0000","last_fullsized":"2024-09-13T13:08:11.863422+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.425436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.425436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:08.741221+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":332,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.859028+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:08:11.859028+0000","last_peered":"2024-09-13T13:08:11.859028+0000","last_clean":"2024-09-13T13:08:11.859028+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:08:11.859028+0000","last_undegraded":"2024-09-13T13:08:11.859028+0000","last_fullsized":"2024-09-13T13:08:11.859028+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:36.785330+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":325,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866492+0000","last_change":"2024-09-13T13:07:50.530956+0000","last_active":"2024-09-13T13:08:11.866492+0000","last_peered":"2024-09-13T13:08:11.866492+0000","last_clean":"2024-09-13T13:08:11.866492+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:08:11.866492+0000","last_undegraded":"2024-09-13T13:08:11.866492+0000","last_fullsized":"2024-09-13T13:08:11.866492+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:50.530904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:50.530904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:13.322916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":305,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.863318+0000","last_change":"2024-09-13T13:07:19.641119+0000","last_active":"2024-09-13T13:08:11.863318+0000","last_peered":"2024-09-13T13:08:11.863318+0000","last_clean":"2024-09-13T13:08:11.863318+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:08:11.863318+0000","last_undegraded":"2024-09-13T13:08:11.863318+0000","last_fullsized":"2024-09-13T13:08:11.863318+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:19.640697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:19.640697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:34.249092+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":335,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861411+0000","last_change":"2024-09-13T13:07:46.772159+0000","last_active":"2024-09-13T13:08:11.861411+0000","last_peered":"2024-09-13T13:08:11.861411+0000","last_clean":"2024-09-13T13:08:11.861411+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:08:11.861411+0000","last_undegraded":"2024-09-13T13:08:11.861411+0000","last_fullsized":"2024-09-13T13:08:11.861411+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:07:46.772111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:46.772111+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:08.257276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":325,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.880002+0000","last_change":"2024-09-13T13:06:38.260486+0000","last_active":"2024-09-13T13:08:11.880002+0000","last_peered":"2024-09-13T13:08:11.880002+0000","last_clean":"2024-09-13T13:08:11.880002+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:08:11.880002+0000","last_undegraded":"2024-09-13T13:08:11.880002+0000","last_fullsized":"2024-09-13T13:08:11.880002+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:38.259822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:38.259822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":363,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.870435+0000","last_change":"2024-09-13T13:06:44.532305+0000","last_active":"2024-09-13T13:08:11.870435+0000","last_peered":"2024-09-13T13:08:11.870435+0000","last_clean":"2024-09-13T13:08:11.870435+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:08:11.870435+0000","last_undegraded":"2024-09-13T13:08:11.870435+0000","last_fullsized":"2024-09-13T13:08:11.870435+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:06:44.532186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.532186+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:10.017851+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":318,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.864491+0000","last_change":"2024-09-13T13:06:47.977011+0000","last_active":"2024-09-13T13:08:11.864491+0000","last_peered":"2024-09-13T13:08:11.864491+0000","last_clean":"2024-09-13T13:08:11.864491+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:08:11.864491+0000","last_undegraded":"2024-09-13T13:08:11.864491+0000","last_fullsized":"2024-09-13T13:08:11.864491+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:47.976922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:47.976922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:17.524630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":325,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.869314+0000","last_change":"2024-09-13T13:07:14.506322+0000","last_active":"2024-09-13T13:08:11.869314+0000","last_peered":"2024-09-13T13:08:11.869314+0000","last_clean":"2024-09-13T13:08:11.869314+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:08:11.869314+0000","last_undegraded":"2024-09-13T13:08:11.869314+0000","last_fullsized":"2024-09-13T13:08:11.869314+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:14.505866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:14.505866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:41.654897+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":326,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861760+0000","last_change":"2024-09-13T13:08:03.840282+0000","last_active":"2024-09-13T13:08:11.861760+0000","last_peered":"2024-09-13T13:08:11.861760+0000","last_clean":"2024-09-13T13:08:11.861760+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:08:11.861760+0000","last_undegraded":"2024-09-13T13:08:11.861760+0000","last_fullsized":"2024-09-13T13:08:11.861760+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:03.840225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.840225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:13.521064+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":336,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.865217+0000","last_change":"2024-09-13T13:06:05.706641+0000","last_active":"2024-09-13T13:08:11.865217+0000","last_peered":"2024-09-13T13:08:11.865217+0000","last_clean":"2024-09-13T13:08:11.865217+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:08:11.865217+0000","last_undegraded":"2024-09-13T13:08:11.865217+0000","last_fullsized":"2024-09-13T13:08:11.865217+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:06:05.706569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:05.706569+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:35.590112+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":326,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861783+0000","last_change":"2024-09-13T13:07:12.879813+0000","last_active":"2024-09-13T13:08:11.861783+0000","last_peered":"2024-09-13T13:08:11.861783+0000","last_clean":"2024-09-13T13:08:11.861783+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:08:11.861783+0000","last_undegraded":"2024-09-13T13:08:11.861783+0000","last_fullsized":"2024-09-13T13:08:11.861783+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.879727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.879727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:16.423585+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":322,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866336+0000","last_change":"2024-09-13T13:07:47.597872+0000","last_active":"2024-09-13T13:08:11.866336+0000","last_peered":"2024-09-13T13:08:11.866336+0000","last_clean":"2024-09-13T13:08:11.866336+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:08:11.866336+0000","last_undegraded":"2024-09-13T13:08:11.866336+0000","last_fullsized":"2024-09-13T13:08:11.866336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:07:47.597802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:47.597802+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:50.877656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":327,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.865823+0000","last_change":"2024-09-13T13:07:54.442276+0000","last_active":"2024-09-13T13:08:11.865823+0000","last_peered":"2024-09-13T13:08:11.865823+0000","last_clean":"2024-09-13T13:08:11.865823+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:08:11.865823+0000","last_undegraded":"2024-09-13T13:08:11.865823+0000","last_fullsized":"2024-09-13T13:08:11.865823+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:07:54.439269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:54.439269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:08.772699+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":337,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.881815+0000","last_change":"2024-09-13T13:06:40.292804+0000","last_active":"2024-09-13T13:08:11.881815+0000","last_peered":"2024-09-13T13:08:11.881815+0000","last_clean":"2024-09-13T13:08:11.881815+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:08:11.881815+0000","last_undegraded":"2024-09-13T13:08:11.881815+0000","last_fullsized":"2024-09-13T13:08:11.881815+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:06:40.292551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:40.292551+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":360,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T13:08:07.322372+0000","last_change":"2024-09-13T13:06:04.949976+0000","last_active":"2024-09-13T13:08:07.322372+0000","last_peered":"2024-09-13T13:08:07.322372+0000","last_clean":"2024-09-13T13:08:07.322372+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:08:07.322372+0000","last_undegraded":"2024-09-13T13:08:07.322372+0000","last_fullsized":"2024-09-13T13:08:07.322372+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:06:04.949930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:04.949930+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:37.955065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"208'8939","reported_seq":9028,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:13.266620+0000","last_change":"2024-09-13T13:06:21.814268+0000","last_active":"2024-09-13T13:08:13.266620+0000","last_peered":"2024-09-13T13:08:13.266620+0000","last_clean":"2024-09-13T13:08:13.266620+0000","last_became_active":"2024-09-13T13:06:21.814002+0000","last_became_peered":"2024-09-13T13:06:21.814002+0000","last_unstale":"2024-09-13T13:08:13.266620+0000","last_undegraded":"2024-09-13T13:08:13.266620+0000","last_fullsized":"2024-09-13T13:08:13.266620+0000","mapping_epoch":169,"log_start":"189'4000","ondisk_log_start":"189'4000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:33.466182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471793664,"num_objects":7199,"num_object_clones":0,"num_object_copies":21597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8939,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":356,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.867565+0000","last_change":"2024-09-13T13:07:15.243958+0000","last_active":"2024-09-13T13:08:11.867565+0000","last_peered":"2024-09-13T13:08:11.867565+0000","last_clean":"2024-09-13T13:08:11.867565+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:08:11.867565+0000","last_undegraded":"2024-09-13T13:08:11.867565+0000","last_fullsized":"2024-09-13T13:08:11.867565+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:07:15.243715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:15.243715+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:38.551933+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":334,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866593+0000","last_change":"2024-09-13T13:07:08.562915+0000","last_active":"2024-09-13T13:08:11.866593+0000","last_peered":"2024-09-13T13:08:11.866593+0000","last_clean":"2024-09-13T13:08:11.866593+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:08:11.866593+0000","last_undegraded":"2024-09-13T13:08:11.866593+0000","last_fullsized":"2024-09-13T13:08:11.866593+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:08.562240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.562240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:17.435791+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":414,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861536+0000","last_change":"2024-09-13T13:07:17.960079+0000","last_active":"2024-09-13T13:08:11.861536+0000","last_peered":"2024-09-13T13:08:11.861536+0000","last_clean":"2024-09-13T13:08:11.861536+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:08:11.861536+0000","last_undegraded":"2024-09-13T13:08:11.861536+0000","last_fullsized":"2024-09-13T13:08:11.861536+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:07:17.960013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:17.960013+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:34.175001+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":322,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.863580+0000","last_change":"2024-09-13T13:08:05.356744+0000","last_active":"2024-09-13T13:08:11.863580+0000","last_peered":"2024-09-13T13:08:11.863580+0000","last_clean":"2024-09-13T13:08:11.863580+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:08:11.863580+0000","last_undegraded":"2024-09-13T13:08:11.863580+0000","last_fullsized":"2024-09-13T13:08:11.863580+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:05.356688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:05.356688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:25.974850+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"208'8800","reported_seq":8889,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:13.769386+0000","last_change":"2024-09-13T13:06:22.000536+0000","last_active":"2024-09-13T13:08:13.769386+0000","last_peered":"2024-09-13T13:08:13.769386+0000","last_clean":"2024-09-13T13:08:13.769386+0000","last_became_active":"2024-09-13T13:06:22.000244+0000","last_became_peered":"2024-09-13T13:06:22.000244+0000","last_unstale":"2024-09-13T13:08:13.769386+0000","last_undegraded":"2024-09-13T13:08:13.769386+0000","last_fullsized":"2024-09-13T13:08:13.769386+0000","mapping_epoch":169,"log_start":"182'3000","ondisk_log_start":"182'3000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:42.077147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459276288,"num_objects":7008,"num_object_clones":0,"num_object_copies":21024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8800,"num_write_kb":505856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":346,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.867114+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:08:11.867114+0000","last_peered":"2024-09-13T13:08:11.867114+0000","last_clean":"2024-09-13T13:08:11.867114+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:08:11.867114+0000","last_undegraded":"2024-09-13T13:08:11.867114+0000","last_fullsized":"2024-09-13T13:08:11.867114+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:10.607736+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":332,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.869819+0000","last_change":"2024-09-13T13:06:44.785235+0000","last_active":"2024-09-13T13:08:11.869819+0000","last_peered":"2024-09-13T13:08:11.869819+0000","last_clean":"2024-09-13T13:08:11.869819+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:08:11.869819+0000","last_undegraded":"2024-09-13T13:08:11.869819+0000","last_fullsized":"2024-09-13T13:08:11.869819+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:06:44.785017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.785017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":319,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.858941+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:08:11.858941+0000","last_peered":"2024-09-13T13:08:11.858941+0000","last_clean":"2024-09-13T13:08:11.858941+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:08:11.858941+0000","last_undegraded":"2024-09-13T13:08:11.858941+0000","last_fullsized":"2024-09-13T13:08:11.858941+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":315,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.860773+0000","last_change":"2024-09-13T13:06:43.138526+0000","last_active":"2024-09-13T13:08:11.860773+0000","last_peered":"2024-09-13T13:08:11.860773+0000","last_clean":"2024-09-13T13:08:11.860773+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:08:11.860773+0000","last_undegraded":"2024-09-13T13:08:11.860773+0000","last_fullsized":"2024-09-13T13:08:11.860773+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.138328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.138328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"208'8797","reported_seq":8887,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:12.985698+0000","last_change":"2024-09-13T13:06:22.000928+0000","last_active":"2024-09-13T13:08:12.985698+0000","last_peered":"2024-09-13T13:08:12.985698+0000","last_clean":"2024-09-13T13:08:12.985698+0000","last_became_active":"2024-09-13T13:06:22.000610+0000","last_became_peered":"2024-09-13T13:06:22.000610+0000","last_unstale":"2024-09-13T13:08:12.985698+0000","last_undegraded":"2024-09-13T13:08:12.985698+0000","last_fullsized":"2024-09-13T13:08:12.985698+0000","mapping_epoch":169,"log_start":"186'3400","ondisk_log_start":"186'3400","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467664920,"num_objects":7137,"num_object_clones":0,"num_object_copies":21411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7137,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8797,"num_write_kb":509825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":363,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.859027+0000","last_change":"2024-09-13T13:07:10.853834+0000","last_active":"2024-09-13T13:08:11.859027+0000","last_peered":"2024-09-13T13:08:11.859027+0000","last_clean":"2024-09-13T13:08:11.859027+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:08:11.859027+0000","last_undegraded":"2024-09-13T13:08:11.859027+0000","last_fullsized":"2024-09-13T13:08:11.859027+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:07:10.853749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.853749+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:26.687762+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":319,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.881977+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:08:11.881977+0000","last_peered":"2024-09-13T13:08:11.881977+0000","last_clean":"2024-09-13T13:08:11.881977+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:08:11.881977+0000","last_undegraded":"2024-09-13T13:08:11.881977+0000","last_fullsized":"2024-09-13T13:08:11.881977+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":405,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866162+0000","last_change":"2024-09-13T13:06:48.866830+0000","last_active":"2024-09-13T13:08:11.866162+0000","last_peered":"2024-09-13T13:08:11.866162+0000","last_clean":"2024-09-13T13:08:11.866162+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:08:11.866162+0000","last_undegraded":"2024-09-13T13:08:11.866162+0000","last_fullsized":"2024-09-13T13:08:11.866162+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:06:48.866752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:48.866752+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":325,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866751+0000","last_change":"2024-09-13T13:06:09.230872+0000","last_active":"2024-09-13T13:08:11.866751+0000","last_peered":"2024-09-13T13:08:11.866751+0000","last_clean":"2024-09-13T13:08:11.866751+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:08:11.866751+0000","last_undegraded":"2024-09-13T13:08:11.866751+0000","last_fullsized":"2024-09-13T13:08:11.866751+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:09.230824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.230824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:36.446120+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"208'8765","reported_seq":8852,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:13.779112+0000","last_change":"2024-09-13T13:06:21.812511+0000","last_active":"2024-09-13T13:08:13.779112+0000","last_peered":"2024-09-13T13:08:13.779112+0000","last_clean":"2024-09-13T13:08:13.779112+0000","last_became_active":"2024-09-13T13:06:21.812226+0000","last_became_peered":"2024-09-13T13:06:21.812226+0000","last_unstale":"2024-09-13T13:08:13.779112+0000","last_undegraded":"2024-09-13T13:08:13.779112+0000","last_fullsized":"2024-09-13T13:08:13.779112+0000","mapping_epoch":169,"log_start":"182'2900","ondisk_log_start":"182'2900","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5865,"log_dups_size":0,"ondisk_log_size":5865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:32.174692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466288640,"num_objects":7115,"num_object_clones":0,"num_object_copies":21345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8765,"num_write_kb":508160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":338,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861432+0000","last_change":"2024-09-13T13:07:56.531041+0000","last_active":"2024-09-13T13:08:11.861432+0000","last_peered":"2024-09-13T13:08:11.861432+0000","last_clean":"2024-09-13T13:08:11.861432+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:08:11.861432+0000","last_undegraded":"2024-09-13T13:08:11.861432+0000","last_fullsized":"2024-09-13T13:08:11.861432+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:07:56.530955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:56.530955+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:20.409897+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":326,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.864498+0000","last_change":"2024-09-13T13:06:36.645157+0000","last_active":"2024-09-13T13:08:11.864498+0000","last_peered":"2024-09-13T13:08:11.864498+0000","last_clean":"2024-09-13T13:08:11.864498+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:08:11.864498+0000","last_undegraded":"2024-09-13T13:08:11.864498+0000","last_fullsized":"2024-09-13T13:08:11.864498+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:36.645089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:36.645089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":419,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.878402+0000","last_change":"2024-09-13T13:07:57.028959+0000","last_active":"2024-09-13T13:08:11.878402+0000","last_peered":"2024-09-13T13:08:11.878402+0000","last_clean":"2024-09-13T13:08:11.878402+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:08:11.878402+0000","last_undegraded":"2024-09-13T13:08:11.878402+0000","last_fullsized":"2024-09-13T13:08:11.878402+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:07:57.028875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.028875+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:08.874427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":322,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.878663+0000","last_change":"2024-09-13T13:07:49.203744+0000","last_active":"2024-09-13T13:08:11.878663+0000","last_peered":"2024-09-13T13:08:11.878663+0000","last_clean":"2024-09-13T13:08:11.878663+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:08:11.878663+0000","last_undegraded":"2024-09-13T13:08:11.878663+0000","last_fullsized":"2024-09-13T13:08:11.878663+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:49.203673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:49.203673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:50.703119+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"208'8841","reported_seq":8929,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:13.392068+0000","last_change":"2024-09-13T13:06:21.812509+0000","last_active":"2024-09-13T13:08:13.392068+0000","last_peered":"2024-09-13T13:08:13.392068+0000","last_clean":"2024-09-13T13:08:13.392068+0000","last_became_active":"2024-09-13T13:06:21.812208+0000","last_became_peered":"2024-09-13T13:06:21.812208+0000","last_unstale":"2024-09-13T13:08:13.392068+0000","last_undegraded":"2024-09-13T13:08:13.392068+0000","last_fullsized":"2024-09-13T13:08:13.392068+0000","mapping_epoch":169,"log_start":"189'4000","ondisk_log_start":"189'4000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:37.648714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467206144,"num_objects":7129,"num_object_clones":0,"num_object_copies":21387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8841,"num_write_kb":511040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":366,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.878607+0000","last_change":"2024-09-13T13:07:48.167973+0000","last_active":"2024-09-13T13:08:11.878607+0000","last_peered":"2024-09-13T13:08:11.878607+0000","last_clean":"2024-09-13T13:08:11.878607+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:08:11.878607+0000","last_undegraded":"2024-09-13T13:08:11.878607+0000","last_fullsized":"2024-09-13T13:08:11.878607+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:07:48.167863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:48.167863+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:04.628141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":311,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861509+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:08:11.861509+0000","last_peered":"2024-09-13T13:08:11.861509+0000","last_clean":"2024-09-13T13:08:11.861509+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:08:11.861509+0000","last_undegraded":"2024-09-13T13:08:11.861509+0000","last_fullsized":"2024-09-13T13:08:11.861509+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:36.410134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":278,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T13:08:06.522908+0000","last_change":"2024-09-13T13:06:08.941867+0000","last_active":"2024-09-13T13:08:06.522908+0000","last_peered":"2024-09-13T13:08:06.522908+0000","last_clean":"2024-09-13T13:08:06.522908+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:08:06.522908+0000","last_undegraded":"2024-09-13T13:08:06.522908+0000","last_fullsized":"2024-09-13T13:08:06.522908+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:08.941799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:08.941799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:36.966167+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":334,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861113+0000","last_change":"2024-09-13T13:06:43.969251+0000","last_active":"2024-09-13T13:08:11.861113+0000","last_peered":"2024-09-13T13:08:11.861113+0000","last_clean":"2024-09-13T13:08:11.861113+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:08:11.861113+0000","last_undegraded":"2024-09-13T13:08:11.861113+0000","last_fullsized":"2024-09-13T13:08:11.861113+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.969176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.969176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:11.554952+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"208'8697","reported_seq":8786,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:13.017020+0000","last_change":"2024-09-13T13:06:22.000658+0000","last_active":"2024-09-13T13:08:13.017020+0000","last_peered":"2024-09-13T13:08:13.017020+0000","last_clean":"2024-09-13T13:08:13.017020+0000","last_became_active":"2024-09-13T13:06:22.000261+0000","last_became_peered":"2024-09-13T13:06:22.000261+0000","last_unstale":"2024-09-13T13:08:13.017020+0000","last_undegraded":"2024-09-13T13:08:13.017020+0000","last_fullsized":"2024-09-13T13:08:13.017020+0000","mapping_epoch":169,"log_start":"186'3300","ondisk_log_start":"186'3300","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:12.319265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464060416,"num_objects":7081,"num_object_clones":0,"num_object_copies":21243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8697,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":466,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861411+0000","last_change":"2024-09-13T13:07:25.438610+0000","last_active":"2024-09-13T13:08:11.861411+0000","last_peered":"2024-09-13T13:08:11.861411+0000","last_clean":"2024-09-13T13:08:11.861411+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:08:11.861411+0000","last_undegraded":"2024-09-13T13:08:11.861411+0000","last_fullsized":"2024-09-13T13:08:11.861411+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:07:25.438528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:25.438528+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:42.664791+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":346,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.859412+0000","last_change":"2024-09-13T13:07:18.888103+0000","last_active":"2024-09-13T13:08:11.859412+0000","last_peered":"2024-09-13T13:08:11.859412+0000","last_clean":"2024-09-13T13:08:11.859412+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:08:11.859412+0000","last_undegraded":"2024-09-13T13:08:11.859412+0000","last_fullsized":"2024-09-13T13:08:11.859412+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:18.888060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:18.888060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:22.169522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":337,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866347+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:08:11.866347+0000","last_peered":"2024-09-13T13:08:11.866347+0000","last_clean":"2024-09-13T13:08:11.866347+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:08:11.866347+0000","last_undegraded":"2024-09-13T13:08:11.866347+0000","last_fullsized":"2024-09-13T13:08:11.866347+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:37.479095+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":318,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.879661+0000","last_change":"2024-09-13T13:06:11.189922+0000","last_active":"2024-09-13T13:08:11.879661+0000","last_peered":"2024-09-13T13:08:11.879661+0000","last_clean":"2024-09-13T13:08:11.879661+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:08:11.879661+0000","last_undegraded":"2024-09-13T13:08:11.879661+0000","last_fullsized":"2024-09-13T13:08:11.879661+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:11.189874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:11.189874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:29.160817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":314,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.865455+0000","last_change":"2024-09-13T13:07:11.551545+0000","last_active":"2024-09-13T13:08:11.865455+0000","last_peered":"2024-09-13T13:08:11.865455+0000","last_clean":"2024-09-13T13:08:11.865455+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:08:11.865455+0000","last_undegraded":"2024-09-13T13:08:11.865455+0000","last_fullsized":"2024-09-13T13:08:11.865455+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.551460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.551460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:33.164720+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"208'8777","reported_seq":8866,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:13.381214+0000","last_change":"2024-09-13T13:06:21.813204+0000","last_active":"2024-09-13T13:08:13.381214+0000","last_peered":"2024-09-13T13:08:13.381214+0000","last_clean":"2024-09-13T13:08:13.381214+0000","last_became_active":"2024-09-13T13:06:21.812869+0000","last_became_peered":"2024-09-13T13:06:21.812869+0000","last_unstale":"2024-09-13T13:08:13.381214+0000","last_undegraded":"2024-09-13T13:08:13.381214+0000","last_fullsized":"2024-09-13T13:08:13.381214+0000","mapping_epoch":169,"log_start":"189'3900","ondisk_log_start":"189'3900","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4877,"log_dups_size":0,"ondisk_log_size":4877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:40.646359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465895424,"num_objects":7109,"num_object_clones":0,"num_object_copies":21327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":508352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":350,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.867662+0000","last_change":"2024-09-13T13:08:04.680025+0000","last_active":"2024-09-13T13:08:11.867662+0000","last_peered":"2024-09-13T13:08:11.867662+0000","last_clean":"2024-09-13T13:08:11.867662+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:08:11.867662+0000","last_undegraded":"2024-09-13T13:08:11.867662+0000","last_fullsized":"2024-09-13T13:08:11.867662+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:08:04.679766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:04.679766+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:13.571663+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":346,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.862298+0000","last_change":"2024-09-13T13:07:20.294836+0000","last_active":"2024-09-13T13:08:11.862298+0000","last_peered":"2024-09-13T13:08:11.862298+0000","last_clean":"2024-09-13T13:08:11.862298+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:08:11.862298+0000","last_undegraded":"2024-09-13T13:08:11.862298+0000","last_fullsized":"2024-09-13T13:08:11.862298+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:20.292616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:20.292616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:44.913353+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":342,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.869289+0000","last_change":"2024-09-13T13:06:50.629016+0000","last_active":"2024-09-13T13:08:11.869289+0000","last_peered":"2024-09-13T13:08:11.869289+0000","last_clean":"2024-09-13T13:08:11.869289+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:08:11.869289+0000","last_undegraded":"2024-09-13T13:08:11.869289+0000","last_fullsized":"2024-09-13T13:08:11.869289+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:06:50.628774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.628774+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:14.442004+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":311,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.870337+0000","last_change":"2024-09-13T13:03:14.282866+0000","last_active":"2024-09-13T13:08:11.870337+0000","last_peered":"2024-09-13T13:08:11.870337+0000","last_clean":"2024-09-13T13:08:11.870337+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:08:11.870337+0000","last_undegraded":"2024-09-13T13:08:11.870337+0000","last_fullsized":"2024-09-13T13:08:11.870337+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:28.229862+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":325,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.867923+0000","last_change":"2024-09-13T13:07:20.623165+0000","last_active":"2024-09-13T13:08:11.867923+0000","last_peered":"2024-09-13T13:08:11.867923+0000","last_clean":"2024-09-13T13:08:11.867923+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:08:11.867923+0000","last_undegraded":"2024-09-13T13:08:11.867923+0000","last_fullsized":"2024-09-13T13:08:11.867923+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:20.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:20.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:34.843959+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"208'8728","reported_seq":8897,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:13.224645+0000","last_change":"2024-09-13T13:07:49.115654+0000","last_active":"2024-09-13T13:08:13.224645+0000","last_peered":"2024-09-13T13:08:13.224645+0000","last_clean":"2024-09-13T13:08:13.224645+0000","last_became_active":"2024-09-13T13:06:21.814772+0000","last_became_peered":"2024-09-13T13:06:21.814772+0000","last_unstale":"2024-09-13T13:08:13.224645+0000","last_undegraded":"2024-09-13T13:08:13.224645+0000","last_fullsized":"2024-09-13T13:08:13.224645+0000","mapping_epoch":169,"log_start":"188'3800","ondisk_log_start":"188'3800","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'7112","last_scrub_stamp":"2024-09-13T13:07:49.115526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:07:49.115526+0000","objects_scrubbed":7029,"log_size":4928,"log_dups_size":0,"ondisk_log_size":4928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:12.393347+0000","scrub_duration":2765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466223104,"num_objects":7114,"num_object_clones":0,"num_object_copies":21342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8728,"num_write_kb":506944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":340,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.881869+0000","last_change":"2024-09-13T13:07:11.169884+0000","last_active":"2024-09-13T13:08:11.881869+0000","last_peered":"2024-09-13T13:08:11.881869+0000","last_clean":"2024-09-13T13:08:11.881869+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:08:11.881869+0000","last_undegraded":"2024-09-13T13:08:11.881869+0000","last_fullsized":"2024-09-13T13:08:11.881869+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:07:11.169656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.169656+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:17.059187+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":334,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.876188+0000","last_change":"2024-09-13T13:08:02.152950+0000","last_active":"2024-09-13T13:08:11.876188+0000","last_peered":"2024-09-13T13:08:11.876188+0000","last_clean":"2024-09-13T13:08:11.876188+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:08:11.876188+0000","last_undegraded":"2024-09-13T13:08:11.876188+0000","last_fullsized":"2024-09-13T13:08:11.876188+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:02.152859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:02.152859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:11.967275+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":760,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.966973+0000","last_change":"2024-09-13T13:06:07.290837+0000","last_active":"2024-09-13T13:08:11.966973+0000","last_peered":"2024-09-13T13:08:11.966973+0000","last_clean":"2024-09-13T13:08:11.966973+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:08:11.966973+0000","last_undegraded":"2024-09-13T13:08:11.966973+0000","last_fullsized":"2024-09-13T13:08:11.966973+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:06:07.290717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:07.290717+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:37.493750+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":425,"num_read_kb":420,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":310,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.863794+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:08:11.863794+0000","last_peered":"2024-09-13T13:08:11.863794+0000","last_clean":"2024-09-13T13:08:11.863794+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:08:11.863794+0000","last_undegraded":"2024-09-13T13:08:11.863794+0000","last_fullsized":"2024-09-13T13:08:11.863794+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:36.517148+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":323,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861073+0000","last_change":"2024-09-13T13:07:19.253424+0000","last_active":"2024-09-13T13:08:11.861073+0000","last_peered":"2024-09-13T13:08:11.861073+0000","last_clean":"2024-09-13T13:08:11.861073+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:08:11.861073+0000","last_undegraded":"2024-09-13T13:08:11.861073+0000","last_fullsized":"2024-09-13T13:08:11.861073+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:19.253354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:19.253354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:43.576206+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"207'8562","reported_seq":8649,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T13:08:10.910274+0000","last_change":"2024-09-13T13:06:21.814495+0000","last_active":"2024-09-13T13:08:10.910274+0000","last_peered":"2024-09-13T13:08:10.910274+0000","last_clean":"2024-09-13T13:08:10.910274+0000","last_became_active":"2024-09-13T13:06:21.814144+0000","last_became_peered":"2024-09-13T13:06:21.814144+0000","last_unstale":"2024-09-13T13:08:10.910274+0000","last_undegraded":"2024-09-13T13:08:10.910274+0000","last_fullsized":"2024-09-13T13:08:10.910274+0000","mapping_epoch":169,"log_start":"180'2300","ondisk_log_start":"180'2300","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:29.012009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475136000,"num_objects":7250,"num_object_clones":0,"num_object_copies":21750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8562,"num_write_kb":505984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":296,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T13:08:07.322136+0000","last_change":"2024-09-13T13:06:51.193749+0000","last_active":"2024-09-13T13:08:07.322136+0000","last_peered":"2024-09-13T13:08:07.322136+0000","last_clean":"2024-09-13T13:08:07.322136+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:08:07.322136+0000","last_undegraded":"2024-09-13T13:08:07.322136+0000","last_fullsized":"2024-09-13T13:08:07.322136+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'18","last_scrub_stamp":"2024-09-13T13:06:51.193678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.193678+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:13.330359+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":337,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.865002+0000","last_change":"2024-09-13T13:06:10.715371+0000","last_active":"2024-09-13T13:08:11.865002+0000","last_peered":"2024-09-13T13:08:11.865002+0000","last_clean":"2024-09-13T13:08:11.865002+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:08:11.865002+0000","last_undegraded":"2024-09-13T13:08:11.865002+0000","last_fullsized":"2024-09-13T13:08:11.865002+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:10.715313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.715313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:36.607325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":382,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866185+0000","last_change":"2024-09-13T13:06:39.564599+0000","last_active":"2024-09-13T13:08:11.866185+0000","last_peered":"2024-09-13T13:08:11.866185+0000","last_clean":"2024-09-13T13:08:11.866185+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:08:11.866185+0000","last_undegraded":"2024-09-13T13:08:11.866185+0000","last_fullsized":"2024-09-13T13:08:11.866185+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:06:39.564549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:06:39.564549+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":326,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866521+0000","last_change":"2024-09-13T13:07:17.546556+0000","last_active":"2024-09-13T13:08:11.866521+0000","last_peered":"2024-09-13T13:08:11.866521+0000","last_clean":"2024-09-13T13:08:11.866521+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:08:11.866521+0000","last_undegraded":"2024-09-13T13:08:11.866521+0000","last_fullsized":"2024-09-13T13:08:11.866521+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:17.546486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:17.546486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:23.893461+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":322,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.863683+0000","last_change":"2024-09-13T13:06:46.624345+0000","last_active":"2024-09-13T13:08:11.863683+0000","last_peered":"2024-09-13T13:08:11.863683+0000","last_clean":"2024-09-13T13:08:11.863683+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:08:11.863683+0000","last_undegraded":"2024-09-13T13:08:11.863683+0000","last_fullsized":"2024-09-13T13:08:11.863683+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:46.624238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:46.624238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"208'8802","reported_seq":8911,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:12.958395+0000","last_change":"2024-09-13T13:07:26.381412+0000","last_active":"2024-09-13T13:08:12.958395+0000","last_peered":"2024-09-13T13:08:12.958395+0000","last_clean":"2024-09-13T13:08:12.958395+0000","last_became_active":"2024-09-13T13:06:21.812061+0000","last_became_peered":"2024-09-13T13:06:21.812061+0000","last_unstale":"2024-09-13T13:08:12.958395+0000","last_undegraded":"2024-09-13T13:08:12.958395+0000","last_fullsized":"2024-09-13T13:08:12.958395+0000","mapping_epoch":169,"log_start":"189'4000","ondisk_log_start":"189'4000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":1649,"log_size":4802,"log_dups_size":0,"ondisk_log_size":4802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473038848,"num_objects":7218,"num_object_clones":0,"num_object_copies":21654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8802,"num_write_kb":512640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":353,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861581+0000","last_change":"2024-09-13T13:08:01.790332+0000","last_active":"2024-09-13T13:08:11.861581+0000","last_peered":"2024-09-13T13:08:11.861581+0000","last_clean":"2024-09-13T13:08:11.861581+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:08:11.861581+0000","last_undegraded":"2024-09-13T13:08:11.861581+0000","last_fullsized":"2024-09-13T13:08:11.861581+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:08:01.790280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:01.790280+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:11.564007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":327,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.859372+0000","last_change":"2024-09-13T13:07:19.932836+0000","last_active":"2024-09-13T13:08:11.859372+0000","last_peered":"2024-09-13T13:08:11.859372+0000","last_clean":"2024-09-13T13:08:11.859372+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:08:11.859372+0000","last_undegraded":"2024-09-13T13:08:11.859372+0000","last_fullsized":"2024-09-13T13:08:11.859372+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:19.932761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:19.932761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:31.526197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":337,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.863535+0000","last_change":"2024-09-13T13:07:23.602111+0000","last_active":"2024-09-13T13:08:11.863535+0000","last_peered":"2024-09-13T13:08:11.863535+0000","last_clean":"2024-09-13T13:08:11.863535+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:08:11.863535+0000","last_undegraded":"2024-09-13T13:08:11.863535+0000","last_fullsized":"2024-09-13T13:08:11.863535+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:23.602052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:23.602052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:40.057397+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":317,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866455+0000","last_change":"2024-09-13T13:04:02.757378+0000","last_active":"2024-09-13T13:08:11.866455+0000","last_peered":"2024-09-13T13:08:11.866455+0000","last_clean":"2024-09-13T13:08:11.866455+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:08:11.866455+0000","last_undegraded":"2024-09-13T13:08:11.866455+0000","last_fullsized":"2024-09-13T13:08:11.866455+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:39.213470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:39.213470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:26.664496+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"208'8908","reported_seq":9078,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:13.273812+0000","last_change":"2024-09-13T13:07:52.135686+0000","last_active":"2024-09-13T13:08:13.273812+0000","last_peered":"2024-09-13T13:08:13.273812+0000","last_clean":"2024-09-13T13:08:13.273812+0000","last_became_active":"2024-09-13T13:06:21.811526+0000","last_became_peered":"2024-09-13T13:06:21.811526+0000","last_unstale":"2024-09-13T13:08:13.273812+0000","last_undegraded":"2024-09-13T13:08:13.273812+0000","last_fullsized":"2024-09-13T13:08:13.273812+0000","mapping_epoch":169,"log_start":"188'3900","ondisk_log_start":"188'3900","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'7389","last_scrub_stamp":"2024-09-13T13:07:52.135527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:07:52.135527+0000","objects_scrubbed":7297,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:10.792093+0000","scrub_duration":2761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473038848,"num_objects":7218,"num_object_clones":0,"num_object_copies":21654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8908,"num_write_kb":516032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":351,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861327+0000","last_change":"2024-09-13T13:05:34.166464+0000","last_active":"2024-09-13T13:08:11.861327+0000","last_peered":"2024-09-13T13:08:11.861327+0000","last_clean":"2024-09-13T13:08:11.861327+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:08:11.861327+0000","last_undegraded":"2024-09-13T13:08:11.861327+0000","last_fullsized":"2024-09-13T13:08:11.861327+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:10.773066+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":301,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T13:08:06.523150+0000","last_change":"2024-09-13T13:07:57.077496+0000","last_active":"2024-09-13T13:08:06.523150+0000","last_peered":"2024-09-13T13:08:06.523150+0000","last_clean":"2024-09-13T13:08:06.523150+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:08:06.523150+0000","last_undegraded":"2024-09-13T13:08:06.523150+0000","last_fullsized":"2024-09-13T13:08:06.523150+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.077428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.077428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:03.677437+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":311,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861173+0000","last_change":"2024-09-13T13:06:37.854805+0000","last_active":"2024-09-13T13:08:11.861173+0000","last_peered":"2024-09-13T13:08:11.861173+0000","last_clean":"2024-09-13T13:08:11.861173+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:08:11.861173+0000","last_undegraded":"2024-09-13T13:08:11.861173+0000","last_fullsized":"2024-09-13T13:08:11.861173+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:37.854730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:37.854730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:43.415646+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":323,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.869674+0000","last_change":"2024-09-13T13:08:03.015302+0000","last_active":"2024-09-13T13:08:11.869674+0000","last_peered":"2024-09-13T13:08:11.869674+0000","last_clean":"2024-09-13T13:08:11.869674+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:08:11.869674+0000","last_undegraded":"2024-09-13T13:08:11.869674+0000","last_fullsized":"2024-09-13T13:08:11.869674+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:03.015233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.015233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:21.081201+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"208'8751","reported_seq":8923,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:13.396610+0000","last_change":"2024-09-13T13:07:54.357736+0000","last_active":"2024-09-13T13:08:13.396610+0000","last_peered":"2024-09-13T13:08:13.396610+0000","last_clean":"2024-09-13T13:08:13.396610+0000","last_became_active":"2024-09-13T13:06:21.812966+0000","last_became_peered":"2024-09-13T13:06:21.812966+0000","last_unstale":"2024-09-13T13:08:13.396610+0000","last_undegraded":"2024-09-13T13:08:13.396610+0000","last_fullsized":"2024-09-13T13:08:13.396610+0000","mapping_epoch":169,"log_start":"189'3900","ondisk_log_start":"189'3900","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'7426","last_scrub_stamp":"2024-09-13T13:07:54.357373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:07:54.357373+0000","objects_scrubbed":7324,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:03.683352+0000","scrub_duration":2702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463405056,"num_objects":7071,"num_object_clones":0,"num_object_copies":21213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8751,"num_write_kb":506304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":357,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861869+0000","last_change":"2024-09-13T13:07:50.760850+0000","last_active":"2024-09-13T13:08:11.861869+0000","last_peered":"2024-09-13T13:08:11.861869+0000","last_clean":"2024-09-13T13:08:11.861869+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:08:11.861869+0000","last_undegraded":"2024-09-13T13:08:11.861869+0000","last_fullsized":"2024-09-13T13:08:11.861869+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:07:50.760777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:50.760777+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:04.058171+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":326,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866039+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:08:11.866039+0000","last_peered":"2024-09-13T13:08:11.866039+0000","last_clean":"2024-09-13T13:08:11.866039+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:08:11.866039+0000","last_undegraded":"2024-09-13T13:08:11.866039+0000","last_fullsized":"2024-09-13T13:08:11.866039+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":414,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.872826+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:08:11.872826+0000","last_peered":"2024-09-13T13:08:11.872826+0000","last_clean":"2024-09-13T13:08:11.872826+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:08:11.872826+0000","last_undegraded":"2024-09-13T13:08:11.872826+0000","last_fullsized":"2024-09-13T13:08:11.872826+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:09.037942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":307,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861325+0000","last_change":"2024-09-13T13:02:33.997780+0000","last_active":"2024-09-13T13:08:11.861325+0000","last_peered":"2024-09-13T13:08:11.861325+0000","last_clean":"2024-09-13T13:08:11.861325+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:08:11.861325+0000","last_undegraded":"2024-09-13T13:08:11.861325+0000","last_fullsized":"2024-09-13T13:08:11.861325+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:33.997740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:33.997740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:28.905371+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"208'8985","reported_seq":9074,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:13.266558+0000","last_change":"2024-09-13T13:06:21.814825+0000","last_active":"2024-09-13T13:08:13.266558+0000","last_peered":"2024-09-13T13:08:13.266558+0000","last_clean":"2024-09-13T13:08:13.266558+0000","last_became_active":"2024-09-13T13:06:21.814433+0000","last_became_peered":"2024-09-13T13:06:21.814433+0000","last_unstale":"2024-09-13T13:08:13.266558+0000","last_undegraded":"2024-09-13T13:08:13.266558+0000","last_fullsized":"2024-09-13T13:08:13.266558+0000","mapping_epoch":169,"log_start":"189'4000","ondisk_log_start":"189'4000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4985,"log_dups_size":0,"ondisk_log_size":4985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:41.466835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478478336,"num_objects":7301,"num_object_clones":0,"num_object_copies":21903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8985,"num_write_kb":521152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":330,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.878451+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:08:11.878451+0000","last_peered":"2024-09-13T13:08:11.878451+0000","last_clean":"2024-09-13T13:08:11.878451+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:08:11.878451+0000","last_undegraded":"2024-09-13T13:08:11.878451+0000","last_fullsized":"2024-09-13T13:08:11.878451+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:09.178675+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":345,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.863508+0000","last_change":"2024-09-13T13:07:59.376928+0000","last_active":"2024-09-13T13:08:11.863508+0000","last_peered":"2024-09-13T13:08:11.863508+0000","last_clean":"2024-09-13T13:08:11.863508+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:08:11.863508+0000","last_undegraded":"2024-09-13T13:08:11.863508+0000","last_fullsized":"2024-09-13T13:08:11.863508+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:59.376860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:59.376860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:05.880019+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":335,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.871312+0000","last_change":"2024-09-13T13:07:11.400066+0000","last_active":"2024-09-13T13:08:11.871312+0000","last_peered":"2024-09-13T13:08:11.871312+0000","last_clean":"2024-09-13T13:08:11.871312+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:08:11.871312+0000","last_undegraded":"2024-09-13T13:08:11.871312+0000","last_fullsized":"2024-09-13T13:08:11.871312+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.399840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.399840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:35.796029+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":306,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.876104+0000","last_change":"2024-09-13T13:08:01.148572+0000","last_active":"2024-09-13T13:08:11.876104+0000","last_peered":"2024-09-13T13:08:11.876104+0000","last_clean":"2024-09-13T13:08:11.876104+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:08:11.876104+0000","last_undegraded":"2024-09-13T13:08:11.876104+0000","last_fullsized":"2024-09-13T13:08:11.876104+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:01.148522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:01.148522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:28.436190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"208'9006","reported_seq":9095,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:13.766958+0000","last_change":"2024-09-13T13:06:22.000224+0000","last_active":"2024-09-13T13:08:13.766958+0000","last_peered":"2024-09-13T13:08:13.766958+0000","last_clean":"2024-09-13T13:08:13.766958+0000","last_became_active":"2024-09-13T13:06:21.999889+0000","last_became_peered":"2024-09-13T13:06:21.999889+0000","last_unstale":"2024-09-13T13:08:13.766958+0000","last_undegraded":"2024-09-13T13:08:13.766958+0000","last_fullsized":"2024-09-13T13:08:13.766958+0000","mapping_epoch":169,"log_start":"183'3200","ondisk_log_start":"183'3200","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5806,"log_dups_size":0,"ondisk_log_size":5806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:30.143793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473825280,"num_objects":7230,"num_object_clones":0,"num_object_copies":21690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9006,"num_write_kb":519552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":312,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T13:08:06.522958+0000","last_change":"2024-09-13T13:06:06.912256+0000","last_active":"2024-09-13T13:08:06.522958+0000","last_peered":"2024-09-13T13:08:06.522958+0000","last_clean":"2024-09-13T13:08:06.522958+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:08:06.522958+0000","last_undegraded":"2024-09-13T13:08:06.522958+0000","last_fullsized":"2024-09-13T13:08:06.522958+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:06:06.912141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:06.912141+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:28.032751+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":345,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.868154+0000","last_change":"2024-09-13T13:07:48.579717+0000","last_active":"2024-09-13T13:08:11.868154+0000","last_peered":"2024-09-13T13:08:11.868154+0000","last_clean":"2024-09-13T13:08:11.868154+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:08:11.868154+0000","last_undegraded":"2024-09-13T13:08:11.868154+0000","last_fullsized":"2024-09-13T13:08:11.868154+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:48.579658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:48.579658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:04.921360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":327,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.869360+0000","last_change":"2024-09-13T13:07:18.355984+0000","last_active":"2024-09-13T13:08:11.869360+0000","last_peered":"2024-09-13T13:08:11.869360+0000","last_clean":"2024-09-13T13:08:11.869360+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:08:11.869360+0000","last_undegraded":"2024-09-13T13:08:11.869360+0000","last_fullsized":"2024-09-13T13:08:11.869360+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:18.355901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:18.355901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:25.490452+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":322,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866107+0000","last_change":"2024-09-13T13:07:49.584576+0000","last_active":"2024-09-13T13:08:11.866107+0000","last_peered":"2024-09-13T13:08:11.866107+0000","last_clean":"2024-09-13T13:08:11.866107+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:08:11.866107+0000","last_undegraded":"2024-09-13T13:08:11.866107+0000","last_fullsized":"2024-09-13T13:08:11.866107+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:49.584505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:49.584505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:13.944016+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"207'8813","reported_seq":8900,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T13:08:10.909829+0000","last_change":"2024-09-13T13:06:22.002144+0000","last_active":"2024-09-13T13:08:10.909829+0000","last_peered":"2024-09-13T13:08:10.909829+0000","last_clean":"2024-09-13T13:08:10.909829+0000","last_became_active":"2024-09-13T13:06:22.001404+0000","last_became_peered":"2024-09-13T13:06:22.001404+0000","last_unstale":"2024-09-13T13:08:10.909829+0000","last_undegraded":"2024-09-13T13:08:10.909829+0000","last_fullsized":"2024-09-13T13:08:10.909829+0000","mapping_epoch":169,"log_start":"180'2500","ondisk_log_start":"180'2500","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:38.964208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490930176,"num_objects":7491,"num_object_clones":0,"num_object_copies":22473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8813,"num_write_kb":521728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":338,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.859231+0000","last_change":"2024-09-13T13:02:28.990089+0000","last_active":"2024-09-13T13:08:11.859231+0000","last_peered":"2024-09-13T13:08:11.859231+0000","last_clean":"2024-09-13T13:08:11.859231+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:08:11.859231+0000","last_undegraded":"2024-09-13T13:08:11.859231+0000","last_fullsized":"2024-09-13T13:08:11.859231+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:02:28.990005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.990005+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:27.871541+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":334,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.868085+0000","last_change":"2024-09-13T13:05:29.345863+0000","last_active":"2024-09-13T13:08:11.868085+0000","last_peered":"2024-09-13T13:08:11.868085+0000","last_clean":"2024-09-13T13:08:11.868085+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:08:11.868085+0000","last_undegraded":"2024-09-13T13:08:11.868085+0000","last_fullsized":"2024-09-13T13:08:11.868085+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.345796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.345796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":319,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.868911+0000","last_change":"2024-09-13T13:06:42.450669+0000","last_active":"2024-09-13T13:08:11.868911+0000","last_peered":"2024-09-13T13:08:11.868911+0000","last_clean":"2024-09-13T13:08:11.868911+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:08:11.868911+0000","last_undegraded":"2024-09-13T13:08:11.868911+0000","last_fullsized":"2024-09-13T13:08:11.868911+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:42.450598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:42.450598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:11.062883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":306,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.865377+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:08:11.865377+0000","last_peered":"2024-09-13T13:08:11.865377+0000","last_clean":"2024-09-13T13:08:11.865377+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:08:11.865377+0000","last_undegraded":"2024-09-13T13:08:11.865377+0000","last_fullsized":"2024-09-13T13:08:11.865377+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:05.648446+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"208'8825","reported_seq":9003,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:13.009778+0000","last_change":"2024-09-13T13:08:03.659578+0000","last_active":"2024-09-13T13:08:13.009778+0000","last_peered":"2024-09-13T13:08:13.009778+0000","last_clean":"2024-09-13T13:08:13.009778+0000","last_became_active":"2024-09-13T13:06:21.816096+0000","last_became_peered":"2024-09-13T13:06:21.816096+0000","last_unstale":"2024-09-13T13:08:13.009778+0000","last_undegraded":"2024-09-13T13:08:13.009778+0000","last_fullsized":"2024-09-13T13:08:13.009778+0000","mapping_epoch":169,"log_start":"186'3400","ondisk_log_start":"186'3400","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'8119","last_scrub_stamp":"2024-09-13T13:08:03.659418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.659418+0000","objects_scrubbed":7960,"log_size":5425,"log_dups_size":0,"ondisk_log_size":5425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:21.490691+0000","scrub_duration":2349,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471007232,"num_objects":7187,"num_object_clones":0,"num_object_copies":21561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8825,"num_write_kb":512384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":358,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.863027+0000","last_change":"2024-09-13T13:08:03.369311+0000","last_active":"2024-09-13T13:08:11.863027+0000","last_peered":"2024-09-13T13:08:11.863027+0000","last_clean":"2024-09-13T13:08:11.863027+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:08:11.863027+0000","last_undegraded":"2024-09-13T13:08:11.863027+0000","last_fullsized":"2024-09-13T13:08:11.863027+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:08:03.369143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.369143+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:10.624423+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":346,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.868202+0000","last_change":"2024-09-13T13:06:51.627764+0000","last_active":"2024-09-13T13:08:11.868202+0000","last_peered":"2024-09-13T13:08:11.868202+0000","last_clean":"2024-09-13T13:08:11.868202+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:08:11.868202+0000","last_undegraded":"2024-09-13T13:08:11.868202+0000","last_fullsized":"2024-09-13T13:08:11.868202+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:06:51.627691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.627691+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:16.124383+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":422,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866047+0000","last_change":"2024-09-13T13:04:02.759553+0000","last_active":"2024-09-13T13:08:11.866047+0000","last_peered":"2024-09-13T13:08:11.866047+0000","last_clean":"2024-09-13T13:08:11.866047+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:08:11.866047+0000","last_undegraded":"2024-09-13T13:08:11.866047+0000","last_fullsized":"2024-09-13T13:08:11.866047+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:45.959285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.959285+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:33.559341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":317,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866052+0000","last_change":"2024-09-13T13:04:02.757776+0000","last_active":"2024-09-13T13:08:11.866052+0000","last_peered":"2024-09-13T13:08:11.866052+0000","last_clean":"2024-09-13T13:08:11.866052+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:08:11.866052+0000","last_undegraded":"2024-09-13T13:08:11.866052+0000","last_fullsized":"2024-09-13T13:08:11.866052+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:28.529954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:02:28.529954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:29.618485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":301,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T13:08:06.522612+0000","last_change":"2024-09-13T13:07:10.058616+0000","last_active":"2024-09-13T13:08:06.522612+0000","last_peered":"2024-09-13T13:08:06.522612+0000","last_clean":"2024-09-13T13:08:06.522612+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:08:06.522612+0000","last_undegraded":"2024-09-13T13:08:06.522612+0000","last_fullsized":"2024-09-13T13:08:06.522612+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:10.058555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.058555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:35.057188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":322,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.869563+0000","last_change":"2024-09-13T13:03:14.504339+0000","last_active":"2024-09-13T13:08:11.869563+0000","last_peered":"2024-09-13T13:08:11.869563+0000","last_clean":"2024-09-13T13:08:11.869563+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:08:11.869563+0000","last_undegraded":"2024-09-13T13:08:11.869563+0000","last_fullsized":"2024-09-13T13:08:11.869563+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:03:14.504263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.504263+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:30.692655+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":325,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.863355+0000","last_change":"2024-09-13T13:06:51.640678+0000","last_active":"2024-09-13T13:08:11.863355+0000","last_peered":"2024-09-13T13:08:11.863355+0000","last_clean":"2024-09-13T13:08:11.863355+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:08:11.863355+0000","last_undegraded":"2024-09-13T13:08:11.863355+0000","last_fullsized":"2024-09-13T13:08:11.863355+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:51.640617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.640617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":332,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.867856+0000","last_change":"2024-09-13T13:07:11.584285+0000","last_active":"2024-09-13T13:08:11.867856+0000","last_peered":"2024-09-13T13:08:11.867856+0000","last_clean":"2024-09-13T13:08:11.867856+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:08:11.867856+0000","last_undegraded":"2024-09-13T13:08:11.867856+0000","last_fullsized":"2024-09-13T13:08:11.867856+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.584244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.584244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:24.783280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":329,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.881914+0000","last_change":"2024-09-13T13:02:42.589951+0000","last_active":"2024-09-13T13:08:11.881914+0000","last_peered":"2024-09-13T13:08:11.881914+0000","last_clean":"2024-09-13T13:08:11.881914+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:08:11.881914+0000","last_undegraded":"2024-09-13T13:08:11.881914+0000","last_fullsized":"2024-09-13T13:08:11.881914+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:02:42.589866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:02:42.589866+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:28.186553+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":329,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866271+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:08:11.866271+0000","last_peered":"2024-09-13T13:08:11.866271+0000","last_clean":"2024-09-13T13:08:11.866271+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:08:11.866271+0000","last_undegraded":"2024-09-13T13:08:11.866271+0000","last_fullsized":"2024-09-13T13:08:11.866271+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:34.515692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":309,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.869406+0000","last_change":"2024-09-13T13:05:42.728927+0000","last_active":"2024-09-13T13:08:11.869406+0000","last_peered":"2024-09-13T13:08:11.869406+0000","last_clean":"2024-09-13T13:08:11.869406+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:08:11.869406+0000","last_undegraded":"2024-09-13T13:08:11.869406+0000","last_fullsized":"2024-09-13T13:08:11.869406+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:42.728881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:42.728881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:33.588100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":323,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.869879+0000","last_change":"2024-09-13T13:07:59.006706+0000","last_active":"2024-09-13T13:08:11.869879+0000","last_peered":"2024-09-13T13:08:11.869879+0000","last_clean":"2024-09-13T13:08:11.869879+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:08:11.869879+0000","last_undegraded":"2024-09-13T13:08:11.869879+0000","last_fullsized":"2024-09-13T13:08:11.869879+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:59.006291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:59.006291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:07.302204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":329,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.864410+0000","last_change":"2024-09-13T13:04:53.687731+0000","last_active":"2024-09-13T13:08:11.864410+0000","last_peered":"2024-09-13T13:08:11.864410+0000","last_clean":"2024-09-13T13:08:11.864410+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:08:11.864410+0000","last_undegraded":"2024-09-13T13:08:11.864410+0000","last_fullsized":"2024-09-13T13:08:11.864410+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:34.559702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":326,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.878093+0000","last_change":"2024-09-13T13:06:41.308449+0000","last_active":"2024-09-13T13:08:11.878093+0000","last_peered":"2024-09-13T13:08:11.878093+0000","last_clean":"2024-09-13T13:08:11.878093+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:08:11.878093+0000","last_undegraded":"2024-09-13T13:08:11.878093+0000","last_fullsized":"2024-09-13T13:08:11.878093+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:41.308367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:41.308367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":405,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:12.321891+0000","last_change":"2024-09-13T13:08:02.779173+0000","last_active":"2024-09-13T13:08:12.321891+0000","last_peered":"2024-09-13T13:08:12.321891+0000","last_clean":"2024-09-13T13:08:12.321891+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:08:12.321891+0000","last_undegraded":"2024-09-13T13:08:12.321891+0000","last_fullsized":"2024-09-13T13:08:12.321891+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:08:02.778891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:02.778891+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:21.610126+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":322,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861416+0000","last_change":"2024-09-13T13:07:21.958114+0000","last_active":"2024-09-13T13:08:11.861416+0000","last_peered":"2024-09-13T13:08:11.861416+0000","last_clean":"2024-09-13T13:08:11.861416+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:08:11.861416+0000","last_undegraded":"2024-09-13T13:08:11.861416+0000","last_fullsized":"2024-09-13T13:08:11.861416+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:21.958030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:21.958030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:23.794528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":337,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866909+0000","last_change":"2024-09-13T13:07:21.588629+0000","last_active":"2024-09-13T13:08:11.866909+0000","last_peered":"2024-09-13T13:08:11.866909+0000","last_clean":"2024-09-13T13:08:11.866909+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:08:11.866909+0000","last_undegraded":"2024-09-13T13:08:11.866909+0000","last_fullsized":"2024-09-13T13:08:11.866909+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:07:21.588544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:21.588544+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:36.180777+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":318,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866901+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:08:11.866901+0000","last_peered":"2024-09-13T13:08:11.866901+0000","last_clean":"2024-09-13T13:08:11.866901+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:08:11.866901+0000","last_undegraded":"2024-09-13T13:08:11.866901+0000","last_fullsized":"2024-09-13T13:08:11.866901+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:08.601342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":428,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.864382+0000","last_change":"2024-09-13T13:07:08.781005+0000","last_active":"2024-09-13T13:08:11.864382+0000","last_peered":"2024-09-13T13:08:11.864382+0000","last_clean":"2024-09-13T13:08:11.864382+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:08:11.864382+0000","last_undegraded":"2024-09-13T13:08:11.864382+0000","last_fullsized":"2024-09-13T13:08:11.864382+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:07:08.780952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.780952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:38.355148+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":313,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.881482+0000","last_change":"2024-09-13T13:08:05.181441+0000","last_active":"2024-09-13T13:08:11.881482+0000","last_peered":"2024-09-13T13:08:11.881482+0000","last_clean":"2024-09-13T13:08:11.881482+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:08:11.881482+0000","last_undegraded":"2024-09-13T13:08:11.881482+0000","last_fullsized":"2024-09-13T13:08:11.881482+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:05.181374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:05.181374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:31.869352+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":353,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.859127+0000","last_change":"2024-09-13T13:05:26.186899+0000","last_active":"2024-09-13T13:08:11.859127+0000","last_peered":"2024-09-13T13:08:11.859127+0000","last_clean":"2024-09-13T13:08:11.859127+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:08:11.859127+0000","last_undegraded":"2024-09-13T13:08:11.859127+0000","last_fullsized":"2024-09-13T13:08:11.859127+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:05:26.186830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.186830+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:07.825572+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":338,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.862090+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:08:11.862090+0000","last_peered":"2024-09-13T13:08:11.862090+0000","last_clean":"2024-09-13T13:08:11.862090+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:08:11.862090+0000","last_undegraded":"2024-09-13T13:08:11.862090+0000","last_fullsized":"2024-09-13T13:08:11.862090+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:09.808815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":319,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.861262+0000","last_change":"2024-09-13T13:03:44.467611+0000","last_active":"2024-09-13T13:08:11.861262+0000","last_peered":"2024-09-13T13:08:11.861262+0000","last_clean":"2024-09-13T13:08:11.861262+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:08:11.861262+0000","last_undegraded":"2024-09-13T13:08:11.861262+0000","last_fullsized":"2024-09-13T13:08:11.861262+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:44.467297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:44.467297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:05.702174+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":314,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.873920+0000","last_change":"2024-09-13T13:07:19.629237+0000","last_active":"2024-09-13T13:08:11.873920+0000","last_peered":"2024-09-13T13:08:11.873920+0000","last_clean":"2024-09-13T13:08:11.873920+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:08:11.873920+0000","last_undegraded":"2024-09-13T13:08:11.873920+0000","last_fullsized":"2024-09-13T13:08:11.873920+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:19.629175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:19.629175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:29.740219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":281,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.863126+0000","last_change":"2024-09-13T13:05:35.506093+0000","last_active":"2024-09-13T13:08:11.863126+0000","last_peered":"2024-09-13T13:08:11.863126+0000","last_clean":"2024-09-13T13:08:11.863126+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:08:11.863126+0000","last_undegraded":"2024-09-13T13:08:11.863126+0000","last_fullsized":"2024-09-13T13:08:11.863126+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:05:35.506037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.506037+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:09.399751+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":325,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.868033+0000","last_change":"2024-09-13T13:07:25.575643+0000","last_active":"2024-09-13T13:08:11.868033+0000","last_peered":"2024-09-13T13:08:11.868033+0000","last_clean":"2024-09-13T13:08:11.868033+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:08:11.868033+0000","last_undegraded":"2024-09-13T13:08:11.868033+0000","last_fullsized":"2024-09-13T13:08:11.868033+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:25.575583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:25.575583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:51.900874+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":310,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.865508+0000","last_change":"2024-09-13T13:07:58.637694+0000","last_active":"2024-09-13T13:08:11.865508+0000","last_peered":"2024-09-13T13:08:11.865508+0000","last_clean":"2024-09-13T13:08:11.865508+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:08:11.865508+0000","last_undegraded":"2024-09-13T13:08:11.865508+0000","last_fullsized":"2024-09-13T13:08:11.865508+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:58.637455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:58.637455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:59.169711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":314,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.877951+0000","last_change":"2024-09-13T13:07:26.172470+0000","last_active":"2024-09-13T13:08:11.877951+0000","last_peered":"2024-09-13T13:08:11.877951+0000","last_clean":"2024-09-13T13:08:11.877951+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:08:11.877951+0000","last_undegraded":"2024-09-13T13:08:11.877951+0000","last_fullsized":"2024-09-13T13:08:11.877951+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:26.172403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:26.172403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:54.732500+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":344,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.874081+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:08:11.874081+0000","last_peered":"2024-09-13T13:08:11.874081+0000","last_clean":"2024-09-13T13:08:11.874081+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:08:11.874081+0000","last_undegraded":"2024-09-13T13:08:11.874081+0000","last_fullsized":"2024-09-13T13:08:11.874081+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:06.638780+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":325,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.863453+0000","last_change":"2024-09-13T13:07:47.481391+0000","last_active":"2024-09-13T13:08:11.863453+0000","last_peered":"2024-09-13T13:08:11.863453+0000","last_clean":"2024-09-13T13:08:11.863453+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:08:11.863453+0000","last_undegraded":"2024-09-13T13:08:11.863453+0000","last_fullsized":"2024-09-13T13:08:11.863453+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:47.481330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:47.481330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:49.987610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":329,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.865725+0000","last_change":"2024-09-13T13:06:10.265459+0000","last_active":"2024-09-13T13:08:11.865725+0000","last_peered":"2024-09-13T13:08:11.865725+0000","last_clean":"2024-09-13T13:08:11.865725+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:08:11.865725+0000","last_undegraded":"2024-09-13T13:08:11.865725+0000","last_fullsized":"2024-09-13T13:08:11.865725+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:10.265415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.265415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:43.541482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":317,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.865664+0000","last_change":"2024-09-13T13:07:57.596272+0000","last_active":"2024-09-13T13:08:11.865664+0000","last_peered":"2024-09-13T13:08:11.865664+0000","last_clean":"2024-09-13T13:08:11.865664+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:08:11.865664+0000","last_undegraded":"2024-09-13T13:08:11.865664+0000","last_fullsized":"2024-09-13T13:08:11.865664+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.596097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:18.920930+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":344,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866210+0000","last_change":"2024-09-13T13:05:44.759472+0000","last_active":"2024-09-13T13:08:11.866210+0000","last_peered":"2024-09-13T13:08:11.866210+0000","last_clean":"2024-09-13T13:08:11.866210+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:08:11.866210+0000","last_undegraded":"2024-09-13T13:08:11.866210+0000","last_fullsized":"2024-09-13T13:08:11.866210+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2024-09-13T13:05:44.759233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.759233+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:32.615650+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":325,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.880559+0000","last_change":"2024-09-13T13:07:09.132221+0000","last_active":"2024-09-13T13:08:11.880559+0000","last_peered":"2024-09-13T13:08:11.880559+0000","last_clean":"2024-09-13T13:08:11.880559+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:08:11.880559+0000","last_undegraded":"2024-09-13T13:08:11.880559+0000","last_fullsized":"2024-09-13T13:08:11.880559+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:09.132136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:09.132136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:19.094537+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":209,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T13:08:06.523046+0000","last_change":"2024-09-13T13:06:47.181575+0000","last_active":"2024-09-13T13:08:06.523046+0000","last_peered":"2024-09-13T13:08:06.523046+0000","last_clean":"2024-09-13T13:08:06.523046+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:08:06.523046+0000","last_undegraded":"2024-09-13T13:08:06.523046+0000","last_fullsized":"2024-09-13T13:08:06.523046+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:47.181407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:47.181407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:15.131168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":322,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866451+0000","last_change":"2024-09-13T13:07:12.592938+0000","last_active":"2024-09-13T13:08:11.866451+0000","last_peered":"2024-09-13T13:08:11.866451+0000","last_clean":"2024-09-13T13:08:11.866451+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:08:11.866451+0000","last_undegraded":"2024-09-13T13:08:11.866451+0000","last_fullsized":"2024-09-13T13:08:11.866451+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.592873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.592873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:29.659738+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":314,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.864185+0000","last_change":"2024-09-13T13:05:44.759683+0000","last_active":"2024-09-13T13:08:11.864185+0000","last_peered":"2024-09-13T13:08:11.864185+0000","last_clean":"2024-09-13T13:08:11.864185+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:08:11.864185+0000","last_undegraded":"2024-09-13T13:08:11.864185+0000","last_fullsized":"2024-09-13T13:08:11.864185+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:44.759284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.759284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:26.402020+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":317,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.866516+0000","last_change":"2024-09-13T13:02:46.246602+0000","last_active":"2024-09-13T13:08:11.866516+0000","last_peered":"2024-09-13T13:08:11.866516+0000","last_clean":"2024-09-13T13:08:11.866516+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:08:11.866516+0000","last_undegraded":"2024-09-13T13:08:11.866516+0000","last_fullsized":"2024-09-13T13:08:11.866516+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:02:46.246540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:02:46.246540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:29.724650+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":320,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-13T13:08:06.527545+0000","last_change":"2024-09-13T13:06:09.928585+0000","last_active":"2024-09-13T13:08:06.527545+0000","last_peered":"2024-09-13T13:08:06.527545+0000","last_clean":"2024-09-13T13:08:06.527545+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:08:06.527545+0000","last_undegraded":"2024-09-13T13:08:06.527545+0000","last_fullsized":"2024-09-13T13:08:06.527545+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:06:09.928504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.928504+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:08:39.964724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":318,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-13T13:08:11.881726+0000","last_change":"2024-09-13T13:06:47.291601+0000","last_active":"2024-09-13T13:08:11.881726+0000","last_peered":"2024-09-13T13:08:11.881726+0000","last_clean":"2024-09-13T13:08:11.881726+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:08:11.881726+0000","last_undegraded":"2024-09-13T13:08:11.881726+0000","last_fullsized":"2024-09-13T13:08:11.881726+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:47.291245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:47.291245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":7527268376,"num_objects":114858,"num_object_clones":0,"num_object_copies":344574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140996,"num_write_kb":8187265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22591057920,"data_stored":22591045704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85496,"ondisk_log_size":85496,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":425,"num_read_kb":420,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201369,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2475796,"kb_used_data":2309952,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91269356,"statfs":{"total":95995035648,"available":93459820544,"internally_reserved":0,"allocated":2365390848,"data_stored":2364999397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":169802186},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806140979,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2619440,"kb_used_data":2393184,"kb_used_omap":4,"kb_used_meta":226235,"kb_avail":91125712,"statfs":{"total":95995035648,"available":93312729088,"internally_reserved":0,"allocated":2450620416,"data_stored":2450253256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":231665325},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691941,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3921352,"kb_used_data":3687476,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":89823800,"statfs":{"total":95995035648,"available":91979571200,"internally_reserved":0,"allocated":3775975424,"data_stored":3775594557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920875,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2480020,"kb_used_data":2314180,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91265132,"statfs":{"total":95995035648,"available":93455495168,"internally_reserved":0,"allocated":2369720320,"data_stored":2369323181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169802182},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117106,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3056604,"kb_used_data":2754632,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":90688548,"statfs":{"total":95995035648,"available":92865073152,"internally_reserved":0,"allocated":2820743168,"data_stored":2820342763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":309131719},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411445,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2998456,"kb_used_data":2754788,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":90746696,"statfs":{"total":95995035648,"available":92924616704,"internally_reserved":0,"allocated":2820902912,"data_stored":2820537769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":249428422},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574972,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3501164,"kb_used_data":3235352,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":90243988,"statfs":{"total":95995035648,"available":92409843712,"internally_reserved":0,"allocated":3313000448,"data_stored":3312583340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738499,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3018068,"kb_used_data":2752192,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":90727084,"statfs":{"total":95995035648,"available":92904534016,"internally_reserved":0,"allocated":2818244608,"data_stored":2817852842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2799763456,"data_stored":2799763456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3288006656,"data_stored":3288006656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2813988864,"data_stored":2813984792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2797076480,"data_stored":2797076480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2364014592,"data_stored":2364014592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3747287040,"data_stored":3747282968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2427322368,"data_stored":2427322368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2353598464,"data_stored":2353594392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:08:17.535 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-13T13:08:17.628 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:17.659 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:17.659 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:08:17.659 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:08:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:17.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:17.761 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:17.794 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:17.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:17.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:17.896 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:17.930 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:17.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:17.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:18.007 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:18.007 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[36682]: from='client.35423 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:08:18.007 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[36682]: osdmap e209: 8 total, 8 up, 8 in 2024-09-13T13:08:18.007 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[36682]: 5.11 scrub starts 2024-09-13T13:08:18.007 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[36682]: 5.11 scrub ok 2024-09-13T13:08:18.007 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[36682]: 3.5 scrub starts 2024-09-13T13:08:18.007 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[36682]: 3.5 scrub ok 2024-09-13T13:08:18.008 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:18.008 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[29471]: from='client.35423 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:08:18.008 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[29471]: osdmap e209: 8 total, 8 up, 8 in 2024-09-13T13:08:18.008 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[29471]: 5.11 scrub starts 2024-09-13T13:08:18.008 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[29471]: 5.11 scrub ok 2024-09-13T13:08:18.008 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[29471]: 3.5 scrub starts 2024-09-13T13:08:18.008 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:17 smithi052 ceph-mon[29471]: 3.5 scrub ok 2024-09-13T13:08:18.032 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:18.090 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:18.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:18.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:18.140 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:18.192 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:18.236 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:18.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:18.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:18.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:17 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:18.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:17 smithi167 ceph-mon[33178]: from='client.35423 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:08:18.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:17 smithi167 ceph-mon[33178]: osdmap e209: 8 total, 8 up, 8 in 2024-09-13T13:08:18.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:17 smithi167 ceph-mon[33178]: 5.11 scrub starts 2024-09-13T13:08:18.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:17 smithi167 ceph-mon[33178]: 5.11 scrub ok 2024-09-13T13:08:18.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:17 smithi167 ceph-mon[33178]: 3.5 scrub starts 2024-09-13T13:08:18.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:17 smithi167 ceph-mon[33178]: 3.5 scrub ok 2024-09-13T13:08:18.338 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:18.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:18.381 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:18.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:18.482 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:18.527 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:18.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:18.529 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T13:08:18.628 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:18.660 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:18.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:18.762 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:18.795 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:08:18.796 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:08:18.802 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:18.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:18.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:18.838 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:08:18.839 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:08:18.882 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:08:18.882 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:08:18.904 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:18.916 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:08:18.927 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:08:18.927 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:08:18.943 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:18.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:18.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:18.961 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:08:18.961 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:08:18.997 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:08:18.997 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:08:19.033 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:08:19.034 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:08:19.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:19.066 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:08:19.067 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:08:19.072 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:19.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:19.174 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:19.207 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:19.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:19.213 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:18 smithi167 ceph-mon[33178]: 4.1e scrub starts 2024-09-13T13:08:19.213 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:18 smithi167 ceph-mon[33178]: 4.1e scrub ok 2024-09-13T13:08:19.213 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:18 smithi167 ceph-mon[33178]: pgmap v343: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 840 op/s 2024-09-13T13:08:19.213 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:18 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:08:19.213 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:18 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:08:19.213 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:18 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2895826576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:18 smithi052 ceph-mon[36682]: 4.1e scrub starts 2024-09-13T13:08:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:18 smithi052 ceph-mon[36682]: 4.1e scrub ok 2024-09-13T13:08:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:18 smithi052 ceph-mon[36682]: pgmap v343: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 840 op/s 2024-09-13T13:08:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:18 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:08:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:18 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:08:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:18 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2895826576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:18 smithi052 ceph-mon[29471]: 4.1e scrub starts 2024-09-13T13:08:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:18 smithi052 ceph-mon[29471]: 4.1e scrub ok 2024-09-13T13:08:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:18 smithi052 ceph-mon[29471]: pgmap v343: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 840 op/s 2024-09-13T13:08:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:18 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:08:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:18 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:08:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:18 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2895826576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:19.309 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:19.351 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:19.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:19.452 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:19.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:19.490 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:19.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:19.563 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:08:19.563 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:08:19.563 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:08:19.563 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:08:19.563 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:08:19.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:19.624 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:19.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:19.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:19.726 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:19.771 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:19.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:19.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:19.873 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:19.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:19.916 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:19.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:20.016 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:20.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:20.053 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:20.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:20.060 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:08:20.154 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:20.183 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:20.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:20.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:20.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:19 smithi167 ceph-mon[33178]: 5.1e scrub starts 2024-09-13T13:08:20.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:19 smithi167 ceph-mon[33178]: 5.1e scrub ok 2024-09-13T13:08:20.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:19 smithi167 ceph-mon[33178]: 5.1d scrub starts 2024-09-13T13:08:20.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:19 smithi167 ceph-mon[33178]: 5.1d scrub ok 2024-09-13T13:08:20.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:19 smithi167 ceph-mon[33178]: 2.1 scrub starts 2024-09-13T13:08:20.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:19 smithi167 ceph-mon[33178]: 2.1 scrub ok 2024-09-13T13:08:20.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:19 smithi052 ceph-mon[36682]: 5.1e scrub starts 2024-09-13T13:08:20.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:19 smithi052 ceph-mon[36682]: 5.1e scrub ok 2024-09-13T13:08:20.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:19 smithi052 ceph-mon[36682]: 5.1d scrub starts 2024-09-13T13:08:20.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:19 smithi052 ceph-mon[36682]: 5.1d scrub ok 2024-09-13T13:08:20.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:19 smithi052 ceph-mon[36682]: 2.1 scrub starts 2024-09-13T13:08:20.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:19 smithi052 ceph-mon[36682]: 2.1 scrub ok 2024-09-13T13:08:20.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:19 smithi052 ceph-mon[29471]: 5.1e scrub starts 2024-09-13T13:08:20.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:19 smithi052 ceph-mon[29471]: 5.1e scrub ok 2024-09-13T13:08:20.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:19 smithi052 ceph-mon[29471]: 5.1d scrub starts 2024-09-13T13:08:20.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:19 smithi052 ceph-mon[29471]: 5.1d scrub ok 2024-09-13T13:08:20.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:19 smithi052 ceph-mon[29471]: 2.1 scrub starts 2024-09-13T13:08:20.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:19 smithi052 ceph-mon[29471]: 2.1 scrub ok 2024-09-13T13:08:20.285 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:20.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:20.337 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:20.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:20.438 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:20.483 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:20.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:20.562 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:20.584 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:20.627 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:20.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:20.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:20.695 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T13:08:20.728 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:20.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:20.766 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:20.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:20.867 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:20.909 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:20.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:20.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[36682]: 3.1b scrub starts 2024-09-13T13:08:20.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[36682]: 3.1b scrub ok 2024-09-13T13:08:20.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[36682]: 3.8 scrub starts 2024-09-13T13:08:20.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[36682]: 3.8 scrub ok 2024-09-13T13:08:20.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[36682]: pgmap v344: 148 pgs: 148 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-13T13:08:20.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[36682]: 6.7 scrub starts 2024-09-13T13:08:20.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[36682]: 6.7 scrub ok 2024-09-13T13:08:20.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[29471]: 3.1b scrub starts 2024-09-13T13:08:20.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[29471]: 3.1b scrub ok 2024-09-13T13:08:20.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[29471]: 3.8 scrub starts 2024-09-13T13:08:20.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[29471]: 3.8 scrub ok 2024-09-13T13:08:20.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[29471]: pgmap v344: 148 pgs: 148 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-13T13:08:20.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[29471]: 6.7 scrub starts 2024-09-13T13:08:20.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:20 smithi052 ceph-mon[29471]: 6.7 scrub ok 2024-09-13T13:08:21.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:21.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:21.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:21.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:21.116 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:08:21.157 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:21.203 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:21.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:21.245 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:20 smithi167 ceph-mon[33178]: 3.1b scrub starts 2024-09-13T13:08:21.245 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:20 smithi167 ceph-mon[33178]: 3.1b scrub ok 2024-09-13T13:08:21.245 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:20 smithi167 ceph-mon[33178]: 3.8 scrub starts 2024-09-13T13:08:21.245 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:20 smithi167 ceph-mon[33178]: 3.8 scrub ok 2024-09-13T13:08:21.246 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:20 smithi167 ceph-mon[33178]: pgmap v344: 148 pgs: 148 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-13T13:08:21.246 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:20 smithi167 ceph-mon[33178]: 6.7 scrub starts 2024-09-13T13:08:21.246 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:20 smithi167 ceph-mon[33178]: 6.7 scrub ok 2024-09-13T13:08:21.304 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:21.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:21.344 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:21.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:21.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:21.477 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:21.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:21.578 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:21.624 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:21.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:21.726 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:21.761 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:21.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:21.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:21.863 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:21.905 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:21.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:21.939 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:08:21.948 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:08:21.949 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:08:21.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:08:21.949 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:08:21.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.949 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167255 7.c 7:372d90c4:::benchmark_data_smithi052_71865_object46154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:08:21.949 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.812976+0000", 2024-09-13T13:08:21.949 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 213.132245041, 2024-09-13T13:08:21.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.970962349, 2024-09-13T13:08:21.949 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:21.949 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:21.949 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:21.949 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:21.950 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:21.950 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:21.950 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167255 2024-09-13T13:08:21.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.950 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:21.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:21.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812976+0000", 2024-09-13T13:08:21.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:21.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:21.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812976+0000", 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812978+0000", 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.547e-06 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812985+0000", 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1389999999999998e-06 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:21.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812994+0000", 2024-09-13T13:08:21.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0580000000000004e-06 2024-09-13T13:08:21.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:21.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812999+0000", 2024-09-13T13:08:21.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1719999999999998e-06 2024-09-13T13:08:21.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:21.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.813060+0000", 2024-09-13T13:08:21.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0238000000000002e-05 2024-09-13T13:08:21.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:21.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.813067+0000", 2024-09-13T13:08:21.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1019999999999999e-06 2024-09-13T13:08:21.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:21.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.778497+0000", 2024-09-13T13:08:21.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96543056400000005 2024-09-13T13:08:21.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:21.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.778584+0000", 2024-09-13T13:08:21.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6466999999999997e-05 2024-09-13T13:08:21.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:21.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.778664+0000", 2024-09-13T13:08:21.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0111000000000006e-05 2024-09-13T13:08:21.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:21.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783699+0000", 2024-09-13T13:08:21.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050350940000000004 2024-09-13T13:08:21.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:21.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783753+0000", 2024-09-13T13:08:21.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4301000000000001e-05 2024-09-13T13:08:21.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:21.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783880+0000", 2024-09-13T13:08:21.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012635700000000001 2024-09-13T13:08:21.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:21.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783912+0000", 2024-09-13T13:08:21.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2736e-05 2024-09-13T13:08:21.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:21.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783939+0000", 2024-09-13T13:08:21.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6463000000000001e-05 2024-09-13T13:08:21.956 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:21.956 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:21.957 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:21.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.957 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167302 7.c 7:3bad7f06:::benchmark_data_smithi052_71865_object46201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:08:21.957 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.859143+0000", 2024-09-13T13:08:21.957 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 213.086078294, 2024-09-13T13:08:21.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92766838399999996, 2024-09-13T13:08:21.957 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:21.957 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:21.957 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:21.957 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:21.957 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:21.957 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:21.958 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167302 2024-09-13T13:08:21.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.958 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:21.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:21.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859143+0000", 2024-09-13T13:08:21.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:21.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:21.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859143+0000", 2024-09-13T13:08:21.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:21.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:21.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859145+0000", 2024-09-13T13:08:21.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-09-13T13:08:21.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:21.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859153+0000", 2024-09-13T13:08:21.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4540000000000007e-06 2024-09-13T13:08:21.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859165+0000", 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1246e-05 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859170+0000", 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859231+0000", 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1163999999999999e-05 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:21.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859244+0000", 2024-09-13T13:08:21.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2539e-05 2024-09-13T13:08:21.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:21.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.780456+0000", 2024-09-13T13:08:21.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92121202899999999 2024-09-13T13:08:21.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:21.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.780530+0000", 2024-09-13T13:08:21.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4692999999999998e-05 2024-09-13T13:08:21.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:21.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.780605+0000", 2024-09-13T13:08:21.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4281000000000003e-05 2024-09-13T13:08:21.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:21.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783805+0000", 2024-09-13T13:08:21.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031999709999999998 2024-09-13T13:08:21.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783844+0000", 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9496000000000003e-05 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786656+0000", 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028116249999999999 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786765+0000", 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109205 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.964 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:08:21.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:21.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786812+0000", 2024-09-13T13:08:21.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6440999999999998e-05 2024-09-13T13:08:21.965 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:21.965 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:21.965 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:21.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.966 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167310 7.c 7:3e43ee27:::benchmark_data_smithi052_71865_object46209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:08:21.966 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.867904+0000", 2024-09-13T13:08:21.966 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 213.07731766800001, 2024-09-13T13:08:21.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91916685399999998, 2024-09-13T13:08:21.966 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:21.966 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:21.966 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:21.966 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:21.966 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:21.966 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:21.966 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167310 2024-09-13T13:08:21.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.966 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867904+0000", 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867904+0000", 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867906+0000", 2024-09-13T13:08:21.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.827e-06 2024-09-13T13:08:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867911+0000", 2024-09-13T13:08:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.733e-06 2024-09-13T13:08:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867925+0000", 2024-09-13T13:08:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3701e-05 2024-09-13T13:08:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867932+0000", 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8689999999999998e-06 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867995+0000", 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3208999999999997e-05 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.868005+0000", 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0379e-05 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:21.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.782379+0000", 2024-09-13T13:08:21.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91437379500000004 2024-09-13T13:08:21.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:21.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.782458+0000", 2024-09-13T13:08:21.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8485999999999994e-05 2024-09-13T13:08:21.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:21.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.782522+0000", 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4598999999999996e-05 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786890+0000", 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0043677120000000002 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786952+0000", 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1827000000000003e-05 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786989+0000", 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6674999999999997e-05 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787034+0000", 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5003000000000003e-05 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787071+0000", 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7039000000000002e-05 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:21.972 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:21.973 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:21.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.973 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167324 7.c 7:3d57ad43:::benchmark_data_smithi052_71865_object46223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:08:21.973 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.880402+0000", 2024-09-13T13:08:21.973 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 213.06481986599999, 2024-09-13T13:08:21.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91706507599999998, 2024-09-13T13:08:21.973 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:21.973 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:21.973 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:21.973 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167324 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880402+0000", 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880402+0000", 2024-09-13T13:08:21.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880404+0000", 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3329999999999999e-06 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880410+0000", 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3199999999999996e-06 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880424+0000", 2024-09-13T13:08:21.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3396e-05 2024-09-13T13:08:21.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:21.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880431+0000", 2024-09-13T13:08:21.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8010000000000004e-06 2024-09-13T13:08:21.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:21.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880504+0000", 2024-09-13T13:08:21.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2482000000000002e-05 2024-09-13T13:08:21.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880515+0000", 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0745e-05 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783970+0000", 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.90345520499999998 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.784030+0000", 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0668000000000003e-05 2024-09-13T13:08:21.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:21.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.784102+0000", 2024-09-13T13:08:21.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1852000000000003e-05 2024-09-13T13:08:21.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:21.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.788798+0000", 2024-09-13T13:08:21.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004696034 2024-09-13T13:08:21.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:21.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.790728+0000", 2024-09-13T13:08:21.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001929828 2024-09-13T13:08:21.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:21.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797366+0000", 2024-09-13T13:08:21.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066377340000000002 2024-09-13T13:08:21.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:21.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797415+0000", 2024-09-13T13:08:21.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9502000000000001e-05 2024-09-13T13:08:21.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:21.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797467+0000", 2024-09-13T13:08:21.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1175999999999999e-05 2024-09-13T13:08:21.980 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:21.980 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:21.980 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:21.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.980 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167331 7.c 7:314bd5a0:::benchmark_data_smithi052_71865_object46230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:08:21.980 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.885575+0000", 2024-09-13T13:08:21.980 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 213.05964645399999, 2024-09-13T13:08:21.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91216846600000001, 2024-09-13T13:08:21.981 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:21.981 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:21.981 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:21.981 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:21.981 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:21.981 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:21.981 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167331 2024-09-13T13:08:21.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.981 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:21.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:21.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885575+0000", 2024-09-13T13:08:21.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:21.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:21.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885575+0000", 2024-09-13T13:08:21.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:21.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:21.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885577+0000", 2024-09-13T13:08:21.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.446e-06 2024-09-13T13:08:21.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885583+0000", 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0759999999999999e-06 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885597+0000", 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3909e-05 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885610+0000", 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2288e-05 2024-09-13T13:08:21.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:21.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885692+0000", 2024-09-13T13:08:21.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2060999999999999e-05 2024-09-13T13:08:21.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:21.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885703+0000", 2024-09-13T13:08:21.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1331e-05 2024-09-13T13:08:21.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:21.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.784949+0000", 2024-09-13T13:08:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.89924553699999998 2024-09-13T13:08:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.785047+0000", 2024-09-13T13:08:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8639000000000002e-05 2024-09-13T13:08:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.785146+0000", 2024-09-13T13:08:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8556999999999998e-05 2024-09-13T13:08:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.788803+0000", 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003657637 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.790769+0000", 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019660250000000002 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797662+0000", 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068928160000000004 2024-09-13T13:08:21.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:21.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797708+0000", 2024-09-13T13:08:21.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6038999999999997e-05 2024-09-13T13:08:21.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:21.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797743+0000", 2024-09-13T13:08:21.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5104999999999997e-05 2024-09-13T13:08:21.987 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:21.987 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:21.987 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:21.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.988 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167336 7.c 7:3d629c66:::benchmark_data_smithi052_71865_object46235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:08:21.988 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.887959+0000", 2024-09-13T13:08:21.988 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 213.05726265199999, 2024-09-13T13:08:21.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.90987360799999994, 2024-09-13T13:08:21.988 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:21.988 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:21.988 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:21.988 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:21.988 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:21.988 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:21.988 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167336 2024-09-13T13:08:21.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.989 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:21.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:21.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887959+0000", 2024-09-13T13:08:21.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:21.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:21.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887959+0000", 2024-09-13T13:08:21.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:21.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:21.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887960+0000", 2024-09-13T13:08:21.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.728e-06 2024-09-13T13:08:21.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:21.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887965+0000", 2024-09-13T13:08:21.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7319999999999998e-06 2024-09-13T13:08:21.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:21.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887978+0000", 2024-09-13T13:08:21.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2702e-05 2024-09-13T13:08:21.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:21.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887985+0000", 2024-09-13T13:08:21.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.097e-06 2024-09-13T13:08:21.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:21.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.888051+0000", 2024-09-13T13:08:21.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6277000000000003e-05 2024-09-13T13:08:21.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:21.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.888062+0000", 2024-09-13T13:08:21.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049e-05 2024-09-13T13:08:21.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:21.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787109+0000", 2024-09-13T13:08:21.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.89904651199999996 2024-09-13T13:08:21.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:21.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787218+0000", 2024-09-13T13:08:21.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010893200000000001 2024-09-13T13:08:21.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:21.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787319+0000", 2024-09-13T13:08:21.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101605 2024-09-13T13:08:21.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:21.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.792171+0000", 2024-09-13T13:08:21.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048516380000000001 2024-09-13T13:08:21.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:21.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.794176+0000", 2024-09-13T13:08:21.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020051280000000001 2024-09-13T13:08:21.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:21.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797764+0000", 2024-09-13T13:08:21.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003588277 2024-09-13T13:08:21.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:21.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797807+0000", 2024-09-13T13:08:21.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3000999999999997e-05 2024-09-13T13:08:21.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:21.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797832+0000", 2024-09-13T13:08:21.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.493e-05 2024-09-13T13:08:21.998 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:21.998 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:21.999 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:21.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:21.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:21.999 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189427 7.d 7:b2417336:::benchmark_data_smithi052_71865_object68326:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:21.999 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.325139+0000", 2024-09-13T13:08:21.999 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.620082252, 2024-09-13T13:08:21.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0242147180000001, 2024-09-13T13:08:21.999 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:21.999 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:21.999 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:08:21.999 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:21.999 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:21.999 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.000 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189427 2024-09-13T13:08:22.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.000 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325139+0000", 2024-09-13T13:08:22.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325139+0000", 2024-09-13T13:08:22.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325142+0000", 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.526e-06 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325151+0000", 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4080000000000004e-06 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325164+0000", 2024-09-13T13:08:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2676e-05 2024-09-13T13:08:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325173+0000", 2024-09-13T13:08:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0070000000000003e-06 2024-09-13T13:08:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.327118+0000", 2024-09-13T13:08:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001944723 2024-09-13T13:08:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.327125+0000", 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7729999999999999e-06 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.333636+0000", 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065108279999999998 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.333645+0000", 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5639999999999997e-06 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349274+0000", 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0156289759999999 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349354+0000", 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0236999999999998e-05 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189429 7.d 7:b1726e7a:::benchmark_data_smithi052_71865_object68328:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:22.005 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.325193+0000", 2024-09-13T13:08:22.005 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.620028582, 2024-09-13T13:08:22.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0246123009999999, 2024-09-13T13:08:22.005 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:22.005 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:22.005 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:08:22.005 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:22.005 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:22.005 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.005 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189429 2024-09-13T13:08:22.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.006 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325193+0000", 2024-09-13T13:08:22.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325193+0000", 2024-09-13T13:08:22.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325194+0000", 2024-09-13T13:08:22.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:08:22.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325197+0000", 2024-09-13T13:08:22.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2200000000000001e-06 2024-09-13T13:08:22.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325204+0000", 2024-09-13T13:08:22.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6989999999999997e-06 2024-09-13T13:08:22.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325209+0000", 2024-09-13T13:08:22.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.118e-06 2024-09-13T13:08:22.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.327226+0000", 2024-09-13T13:08:22.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020175979999999998 2024-09-13T13:08:22.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:08:22.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.327232+0000", 2024-09-13T13:08:22.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9789999999999998e-06 2024-09-13T13:08:22.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.333665+0000", 2024-09-13T13:08:22.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064329620000000004 2024-09-13T13:08:22.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.333671+0000", 2024-09-13T13:08:22.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0599999999999996e-06 2024-09-13T13:08:22.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349742+0000", 2024-09-13T13:08:22.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.016070799 2024-09-13T13:08:22.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349805+0000", 2024-09-13T13:08:22.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3083000000000006e-05 2024-09-13T13:08:22.011 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.011 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.011 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.012 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189477 7.d 7:bf75cb66:::benchmark_data_smithi052_71865_object68376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:22.012 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.372477+0000", 2024-09-13T13:08:22.012 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.57274478400001, 2024-09-13T13:08:22.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98440433000000005, 2024-09-13T13:08:22.012 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:22.012 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:22.012 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:22.012 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:22.012 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:22.012 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.012 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189477 2024-09-13T13:08:22.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.013 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372477+0000", 2024-09-13T13:08:22.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372477+0000", 2024-09-13T13:08:22.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:22.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372479+0000", 2024-09-13T13:08:22.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8950000000000001e-06 2024-09-13T13:08:22.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372485+0000", 2024-09-13T13:08:22.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9900000000000002e-06 2024-09-13T13:08:22.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372494+0000", 2024-09-13T13:08:22.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5689999999999994e-06 2024-09-13T13:08:22.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372499+0000", 2024-09-13T13:08:22.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3310000000000003e-06 2024-09-13T13:08:22.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372566+0000", 2024-09-13T13:08:22.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6716000000000003e-05 2024-09-13T13:08:22.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372586+0000", 2024-09-13T13:08:22.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9987e-05 2024-09-13T13:08:22.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349836+0000", 2024-09-13T13:08:22.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97724946000000001 2024-09-13T13:08:22.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:22.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349909+0000", 2024-09-13T13:08:22.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3281999999999994e-05 2024-09-13T13:08:22.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:08:22.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349980+0000", 2024-09-13T13:08:22.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0748999999999996e-05 2024-09-13T13:08:22.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356657+0000", 2024-09-13T13:08:22.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066769580000000002 2024-09-13T13:08:22.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356719+0000", 2024-09-13T13:08:22.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2872000000000004e-05 2024-09-13T13:08:22.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:22.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356776+0000", 2024-09-13T13:08:22.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6314000000000003e-05 2024-09-13T13:08:22.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:22.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356840+0000", 2024-09-13T13:08:22.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4484000000000001e-05 2024-09-13T13:08:22.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356881+0000", 2024-09-13T13:08:22.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0723000000000003e-05 2024-09-13T13:08:22.022 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.022 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.022 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.022 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189510 7.d 7:b635578a:::benchmark_data_smithi052_71865_object68409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:22.022 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.399587+0000", 2024-09-13T13:08:22.023 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.545634339, 2024-09-13T13:08:22.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96507304999999999, 2024-09-13T13:08:22.023 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:22.023 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:22.023 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:22.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:22.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:22.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.023 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189510 2024-09-13T13:08:22.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.024 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399587+0000", 2024-09-13T13:08:22.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399587+0000", 2024-09-13T13:08:22.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399589+0000", 2024-09-13T13:08:22.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2759999999999999e-06 2024-09-13T13:08:22.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399599+0000", 2024-09-13T13:08:22.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4730000000000006e-06 2024-09-13T13:08:22.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399613+0000", 2024-09-13T13:08:22.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.403e-05 2024-09-13T13:08:22.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399621+0000", 2024-09-13T13:08:22.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2609999999999993e-06 2024-09-13T13:08:22.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399690+0000", 2024-09-13T13:08:22.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8448000000000006e-05 2024-09-13T13:08:22.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399700+0000", 2024-09-13T13:08:22.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9490000000000008e-06 2024-09-13T13:08:22.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.353851+0000", 2024-09-13T13:08:22.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95415173900000005 2024-09-13T13:08:22.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:22.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.353952+0000", 2024-09-13T13:08:22.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101093 2024-09-13T13:08:22.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:08:22.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.354065+0000", 2024-09-13T13:08:22.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112314 2024-09-13T13:08:22.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:22.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.362390+0000", 2024-09-13T13:08:22.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0083252619999999999 2024-09-13T13:08:22.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364495+0000", 2024-09-13T13:08:22.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021055179999999998 2024-09-13T13:08:22.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364585+0000", 2024-09-13T13:08:22.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9598000000000005e-05 2024-09-13T13:08:22.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:22.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364629+0000", 2024-09-13T13:08:22.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3711000000000002e-05 2024-09-13T13:08:22.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364660+0000", 2024-09-13T13:08:22.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1377999999999997e-05 2024-09-13T13:08:22.030 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.030 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.030 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.031 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189512 7.d 7:b7cfc9e5:::benchmark_data_smithi052_71865_object68411:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:22.031 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.400157+0000", 2024-09-13T13:08:22.031 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.545063941, 2024-09-13T13:08:22.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96465484199999996, 2024-09-13T13:08:22.031 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:22.031 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:22.031 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:22.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:22.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:22.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.031 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189512 2024-09-13T13:08:22.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.032 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400157+0000", 2024-09-13T13:08:22.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400157+0000", 2024-09-13T13:08:22.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400158+0000", 2024-09-13T13:08:22.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.16e-07 2024-09-13T13:08:22.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400162+0000", 2024-09-13T13:08:22.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.304e-06 2024-09-13T13:08:22.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400170+0000", 2024-09-13T13:08:22.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:08:22.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400182+0000", 2024-09-13T13:08:22.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2254e-05 2024-09-13T13:08:22.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400234+0000", 2024-09-13T13:08:22.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.198e-05 2024-09-13T13:08:22.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400244+0000", 2024-09-13T13:08:22.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0036e-05 2024-09-13T13:08:22.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356925+0000", 2024-09-13T13:08:22.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95668097299999999 2024-09-13T13:08:22.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:22.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.357056+0000", 2024-09-13T13:08:22.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013119499999999999 2024-09-13T13:08:22.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:08:22.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.357181+0000", 2024-09-13T13:08:22.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012478000000000001 2024-09-13T13:08:22.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:22.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364436+0000", 2024-09-13T13:08:22.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072547779999999999 2024-09-13T13:08:22.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364542+0000", 2024-09-13T13:08:22.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010571 2024-09-13T13:08:22.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364710+0000", 2024-09-13T13:08:22.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168486 2024-09-13T13:08:22.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:22.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364754+0000", 2024-09-13T13:08:22.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4141000000000003e-05 2024-09-13T13:08:22.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364812+0000", 2024-09-13T13:08:22.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8180000000000002e-05 2024-09-13T13:08:22.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.039 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.039 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189519 7.d 7:b156376f:::benchmark_data_smithi052_71865_object68418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:22.039 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.408717+0000", 2024-09-13T13:08:22.039 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.536504751, 2024-09-13T13:08:22.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97666052999999997, 2024-09-13T13:08:22.039 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:22.039 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:22.039 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:22.040 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:22.040 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:22.040 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.040 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189519 2024-09-13T13:08:22.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.040 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408717+0000", 2024-09-13T13:08:22.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408717+0000", 2024-09-13T13:08:22.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408718+0000", 2024-09-13T13:08:22.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3939999999999999e-06 2024-09-13T13:08:22.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408722+0000", 2024-09-13T13:08:22.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2740000000000001e-06 2024-09-13T13:08:22.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408732+0000", 2024-09-13T13:08:22.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9129999999999998e-06 2024-09-13T13:08:22.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408739+0000", 2024-09-13T13:08:22.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7789999999999998e-06 2024-09-13T13:08:22.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408803+0000", 2024-09-13T13:08:22.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3750000000000005e-05 2024-09-13T13:08:22.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408811+0000", 2024-09-13T13:08:22.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4079999999999998e-06 2024-09-13T13:08:22.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.359633+0000", 2024-09-13T13:08:22.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95082174900000005 2024-09-13T13:08:22.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:22.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.359882+0000", 2024-09-13T13:08:22.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024927 2024-09-13T13:08:22.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:08:22.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.359990+0000", 2024-09-13T13:08:22.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010738099999999999 2024-09-13T13:08:22.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:22.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364551+0000", 2024-09-13T13:08:22.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045615020000000003 2024-09-13T13:08:22.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371210+0000", 2024-09-13T13:08:22.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066585400000000001 2024-09-13T13:08:22.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.385259+0000", 2024-09-13T13:08:22.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014048932 2024-09-13T13:08:22.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:22.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.385324+0000", 2024-09-13T13:08:22.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5908999999999995e-05 2024-09-13T13:08:22.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.385377+0000", 2024-09-13T13:08:22.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2729000000000003e-05 2024-09-13T13:08:22.047 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.048 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.048 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189523 7.d 7:ba812c0f:::benchmark_data_smithi052_71865_object68422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:22.048 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.410273+0000", 2024-09-13T13:08:22.048 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.53494866400001, 2024-09-13T13:08:22.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98121212400000002, 2024-09-13T13:08:22.048 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:22.048 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:22.048 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:22.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:22.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:22.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.049 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189523 2024-09-13T13:08:22.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.049 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410273+0000", 2024-09-13T13:08:22.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410273+0000", 2024-09-13T13:08:22.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410274+0000", 2024-09-13T13:08:22.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3179999999999999e-06 2024-09-13T13:08:22.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410281+0000", 2024-09-13T13:08:22.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8700000000000003e-06 2024-09-13T13:08:22.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410293+0000", 2024-09-13T13:08:22.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.225e-05 2024-09-13T13:08:22.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410300+0000", 2024-09-13T13:08:22.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6529999999999997e-06 2024-09-13T13:08:22.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410356+0000", 2024-09-13T13:08:22.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6366000000000002e-05 2024-09-13T13:08:22.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410366+0000", 2024-09-13T13:08:22.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0166999999999999e-05 2024-09-13T13:08:22.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.362418+0000", 2024-09-13T13:08:22.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95205126100000004 2024-09-13T13:08:22.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:22.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.362525+0000", 2024-09-13T13:08:22.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106965 2024-09-13T13:08:22.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:08:22.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.362594+0000", 2024-09-13T13:08:22.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8967999999999994e-05 2024-09-13T13:08:22.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:22.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.368931+0000", 2024-09-13T13:08:22.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063376480000000004 2024-09-13T13:08:22.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.387384+0000", 2024-09-13T13:08:22.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018452995999999999 2024-09-13T13:08:22.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391424+0000", 2024-09-13T13:08:22.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040394380000000002 2024-09-13T13:08:22.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:22.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391455+0000", 2024-09-13T13:08:22.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1152999999999998e-05 2024-09-13T13:08:22.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391485+0000", 2024-09-13T13:08:22.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0071000000000001e-05 2024-09-13T13:08:22.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.056 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.057 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189527 7.d 7:b61412a6:::benchmark_data_smithi052_71865_object68426:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:22.057 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.414261+0000", 2024-09-13T13:08:22.057 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.53096037500001, 2024-09-13T13:08:22.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97733418299999997, 2024-09-13T13:08:22.057 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:22.057 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:22.057 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:22.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:22.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:22.058 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.058 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189527 2024-09-13T13:08:22.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.058 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414261+0000", 2024-09-13T13:08:22.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414261+0000", 2024-09-13T13:08:22.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414262+0000", 2024-09-13T13:08:22.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6599999999999995e-07 2024-09-13T13:08:22.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414264+0000", 2024-09-13T13:08:22.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4830000000000002e-06 2024-09-13T13:08:22.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414273+0000", 2024-09-13T13:08:22.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5669999999999995e-06 2024-09-13T13:08:22.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414277+0000", 2024-09-13T13:08:22.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9530000000000001e-06 2024-09-13T13:08:22.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414327+0000", 2024-09-13T13:08:22.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9759999999999998e-05 2024-09-13T13:08:22.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414336+0000", 2024-09-13T13:08:22.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3330000000000003e-06 2024-09-13T13:08:22.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364845+0000", 2024-09-13T13:08:22.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95050922199999999 2024-09-13T13:08:22.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:22.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364910+0000", 2024-09-13T13:08:22.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5036999999999994e-05 2024-09-13T13:08:22.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:08:22.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364972+0000", 2024-09-13T13:08:22.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1741000000000004e-05 2024-09-13T13:08:22.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:22.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371169+0000", 2024-09-13T13:08:22.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061969069999999998 2024-09-13T13:08:22.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.387420+0000", 2024-09-13T13:08:22.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016250882000000001 2024-09-13T13:08:22.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391522+0000", 2024-09-13T13:08:22.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041024420000000004 2024-09-13T13:08:22.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:22.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391561+0000", 2024-09-13T13:08:22.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9026999999999998e-05 2024-09-13T13:08:22.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391595+0000", 2024-09-13T13:08:22.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4063000000000002e-05 2024-09-13T13:08:22.065 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.065 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.065 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.065 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189540 7.d 7:bd5b765e:::benchmark_data_smithi052_71865_object68439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:22.065 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.427873+0000", 2024-09-13T13:08:22.065 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.51734888999999, 2024-09-13T13:08:22.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96380134799999995, 2024-09-13T13:08:22.065 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:22.066 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:22.066 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:22.066 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:22.066 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:22.066 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.066 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189540 2024-09-13T13:08:22.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.066 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427873+0000", 2024-09-13T13:08:22.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427873+0000", 2024-09-13T13:08:22.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427874+0000", 2024-09-13T13:08:22.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.765e-06 2024-09-13T13:08:22.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427880+0000", 2024-09-13T13:08:22.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2299999999999999e-06 2024-09-13T13:08:22.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427892+0000", 2024-09-13T13:08:22.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2578999999999999e-05 2024-09-13T13:08:22.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427901+0000", 2024-09-13T13:08:22.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8480000000000007e-06 2024-09-13T13:08:22.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427976+0000", 2024-09-13T13:08:22.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4636999999999997e-05 2024-09-13T13:08:22.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427982+0000", 2024-09-13T13:08:22.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:08:22.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.366695+0000", 2024-09-13T13:08:22.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93871229899999997 2024-09-13T13:08:22.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:22.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.366759+0000", 2024-09-13T13:08:22.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4239000000000006e-05 2024-09-13T13:08:22.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:08:22.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.366861+0000", 2024-09-13T13:08:22.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102506 2024-09-13T13:08:22.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:22.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.372748+0000", 2024-09-13T13:08:22.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058863960000000003 2024-09-13T13:08:22.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.387463+0000", 2024-09-13T13:08:22.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014715045 2024-09-13T13:08:22.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391626+0000", 2024-09-13T13:08:22.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041627900000000004 2024-09-13T13:08:22.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:22.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391651+0000", 2024-09-13T13:08:22.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5678e-05 2024-09-13T13:08:22.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391674+0000", 2024-09-13T13:08:22.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2586e-05 2024-09-13T13:08:22.073 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.073 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.074 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.074 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189552 7.d 7:b0ca6a67:::benchmark_data_smithi052_71865_object68451:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:22.074 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.438326+0000", 2024-09-13T13:08:22.074 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.50689521699999, 2024-09-13T13:08:22.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96655187600000003, 2024-09-13T13:08:22.074 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:22.074 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:22.074 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:22.074 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:22.074 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:22.075 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.075 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189552 2024-09-13T13:08:22.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.075 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438326+0000", 2024-09-13T13:08:22.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438326+0000", 2024-09-13T13:08:22.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438328+0000", 2024-09-13T13:08:22.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.587e-06 2024-09-13T13:08:22.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438337+0000", 2024-09-13T13:08:22.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7280000000000001e-06 2024-09-13T13:08:22.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438349+0000", 2024-09-13T13:08:22.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2944e-05 2024-09-13T13:08:22.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438356+0000", 2024-09-13T13:08:22.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3369999999999996e-06 2024-09-13T13:08:22.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438420+0000", 2024-09-13T13:08:22.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4082000000000001e-05 2024-09-13T13:08:22.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438430+0000", 2024-09-13T13:08:22.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0356e-05 2024-09-13T13:08:22.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.368958+0000", 2024-09-13T13:08:22.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93052786899999995 2024-09-13T13:08:22.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:22.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.369054+0000", 2024-09-13T13:08:22.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5421999999999995e-05 2024-09-13T13:08:22.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:08:22.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.369151+0000", 2024-09-13T13:08:22.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7264999999999998e-05 2024-09-13T13:08:22.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:22.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.375369+0000", 2024-09-13T13:08:22.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0062185900000000004 2024-09-13T13:08:22.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391714+0000", 2024-09-13T13:08:22.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016344250000000001 2024-09-13T13:08:22.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404746+0000", 2024-09-13T13:08:22.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013032871999999999 2024-09-13T13:08:22.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:22.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404838+0000", 2024-09-13T13:08:22.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1419e-05 2024-09-13T13:08:22.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404878+0000", 2024-09-13T13:08:22.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0154999999999998e-05 2024-09-13T13:08:22.083 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.083 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.083 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.084 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189574 7.d 7:bbe21a3a:::benchmark_data_smithi052_71865_object68473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:22.084 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.459595+0000", 2024-09-13T13:08:22.084 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.48562625400001, 2024-09-13T13:08:22.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94540613600000001, 2024-09-13T13:08:22.084 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:22.084 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:22.086 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:22.086 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:22.086 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:22.086 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.086 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189574 2024-09-13T13:08:22.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.086 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459595+0000", 2024-09-13T13:08:22.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459595+0000", 2024-09-13T13:08:22.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459597+0000", 2024-09-13T13:08:22.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9920000000000002e-06 2024-09-13T13:08:22.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459603+0000", 2024-09-13T13:08:22.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7629999999999997e-06 2024-09-13T13:08:22.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459615+0000", 2024-09-13T13:08:22.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1909999999999999e-05 2024-09-13T13:08:22.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459621+0000", 2024-09-13T13:08:22.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8139999999999997e-06 2024-09-13T13:08:22.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459683+0000", 2024-09-13T13:08:22.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2174000000000005e-05 2024-09-13T13:08:22.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459694+0000", 2024-09-13T13:08:22.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1297e-05 2024-09-13T13:08:22.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371239+0000", 2024-09-13T13:08:22.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.911544507 2024-09-13T13:08:22.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:22.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371305+0000", 2024-09-13T13:08:22.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6738999999999999e-05 2024-09-13T13:08:22.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:08:22.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371370+0000", 2024-09-13T13:08:22.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4368000000000004e-05 2024-09-13T13:08:22.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:22.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.380489+0000", 2024-09-13T13:08:22.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091196799999999998 2024-09-13T13:08:22.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391735+0000", 2024-09-13T13:08:22.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011245965 2024-09-13T13:08:22.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404921+0000", 2024-09-13T13:08:22.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013185357 2024-09-13T13:08:22.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:22.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404969+0000", 2024-09-13T13:08:22.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8319999999999998e-05 2024-09-13T13:08:22.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405001+0000", 2024-09-13T13:08:22.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2249999999999998e-05 2024-09-13T13:08:22.095 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.095 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.095 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.096 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189587 7.d 7:bb16c9f3:::benchmark_data_smithi052_71865_object68486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:22.096 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.470029+0000", 2024-09-13T13:08:22.096 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.47519254299999, 2024-09-13T13:08:22.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93511254399999999, 2024-09-13T13:08:22.096 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:22.096 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:22.096 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:22.096 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:22.097 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:22.097 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.097 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189587 2024-09-13T13:08:22.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.097 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470029+0000", 2024-09-13T13:08:22.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470029+0000", 2024-09-13T13:08:22.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470030+0000", 2024-09-13T13:08:22.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.42e-06 2024-09-13T13:08:22.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470034+0000", 2024-09-13T13:08:22.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1919999999999998e-06 2024-09-13T13:08:22.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470045+0000", 2024-09-13T13:08:22.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0239e-05 2024-09-13T13:08:22.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470050+0000", 2024-09-13T13:08:22.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2569999999999998e-06 2024-09-13T13:08:22.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470113+0000", 2024-09-13T13:08:22.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3417999999999995e-05 2024-09-13T13:08:22.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470123+0000", 2024-09-13T13:08:22.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3379999999999994e-06 2024-09-13T13:08:22.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.372770+0000", 2024-09-13T13:08:22.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.90264689300000001 2024-09-13T13:08:22.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:22.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.372854+0000", 2024-09-13T13:08:22.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4221000000000006e-05 2024-09-13T13:08:22.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:08:22.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.372914+0000", 2024-09-13T13:08:22.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0451000000000001e-05 2024-09-13T13:08:22.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:22.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.380494+0000", 2024-09-13T13:08:22.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075800609999999999 2024-09-13T13:08:22.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.403571+0000", 2024-09-13T13:08:22.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02307698 2024-09-13T13:08:22.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405053+0000", 2024-09-13T13:08:22.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014813879999999999 2024-09-13T13:08:22.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:22.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405106+0000", 2024-09-13T13:08:22.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2904e-05 2024-09-13T13:08:22.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405141+0000", 2024-09-13T13:08:22.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5781999999999997e-05 2024-09-13T13:08:22.104 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.104 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.104 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.104 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189600 7.d 7:b65df3b7:::benchmark_data_smithi052_71865_object68499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:22.105 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.492940+0000", 2024-09-13T13:08:22.105 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.45228140500001, 2024-09-13T13:08:22.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91826516899999999, 2024-09-13T13:08:22.105 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:22.105 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:22.105 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:22.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:22.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:22.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.105 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189600 2024-09-13T13:08:22.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.106 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492940+0000", 2024-09-13T13:08:22.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492940+0000", 2024-09-13T13:08:22.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492942+0000", 2024-09-13T13:08:22.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3539999999999998e-06 2024-09-13T13:08:22.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492952+0000", 2024-09-13T13:08:22.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.448e-06 2024-09-13T13:08:22.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492967+0000", 2024-09-13T13:08:22.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.554e-05 2024-09-13T13:08:22.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492975+0000", 2024-09-13T13:08:22.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:08:22.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.493051+0000", 2024-09-13T13:08:22.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6112999999999994e-05 2024-09-13T13:08:22.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.493064+0000", 2024-09-13T13:08:22.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2235000000000001e-05 2024-09-13T13:08:22.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.374296+0000", 2024-09-13T13:08:22.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.88123260599999997 2024-09-13T13:08:22.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:22.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.374360+0000", 2024-09-13T13:08:22.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4077000000000003e-05 2024-09-13T13:08:22.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:08:22.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.374430+0000", 2024-09-13T13:08:22.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9789999999999994e-05 2024-09-13T13:08:22.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:22.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.380501+0000", 2024-09-13T13:08:22.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060705790000000004 2024-09-13T13:08:22.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405185+0000", 2024-09-13T13:08:22.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024683738 2024-09-13T13:08:22.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411130+0000", 2024-09-13T13:08:22.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059459279999999996 2024-09-13T13:08:22.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:22.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411163+0000", 2024-09-13T13:08:22.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2898000000000002e-05 2024-09-13T13:08:22.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411205+0000", 2024-09-13T13:08:22.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1841000000000001e-05 2024-09-13T13:08:22.113 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.113 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.113 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.113 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189604 7.d 7:b85ca2e8:::benchmark_data_smithi052_71865_object68503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:08:22.113 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.496890+0000", 2024-09-13T13:08:22.113 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.44833122, 2024-09-13T13:08:22.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91447940100000003, 2024-09-13T13:08:22.114 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:08:22.114 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:08:22.114 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:22.114 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:08:22.114 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:08:22.114 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:22.114 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189604 2024-09-13T13:08:22.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.114 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:08:22.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:08:22.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496890+0000", 2024-09-13T13:08:22.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:08:22.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496890+0000", 2024-09-13T13:08:22.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:08:22.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:08:22.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496892+0000", 2024-09-13T13:08:22.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5149999999999999e-06 2024-09-13T13:08:22.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:08:22.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496896+0000", 2024-09-13T13:08:22.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8569999999999997e-06 2024-09-13T13:08:22.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:08:22.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496905+0000", 2024-09-13T13:08:22.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4280000000000002e-06 2024-09-13T13:08:22.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:08:22.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496910+0000", 2024-09-13T13:08:22.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3499999999999996e-06 2024-09-13T13:08:22.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496968+0000", 2024-09-13T13:08:22.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7939000000000002e-05 2024-09-13T13:08:22.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:08:22.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496978+0000", 2024-09-13T13:08:22.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0209e-05 2024-09-13T13:08:22.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:08:22.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.375402+0000", 2024-09-13T13:08:22.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87842360799999997 2024-09-13T13:08:22.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:08:22.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.375497+0000", 2024-09-13T13:08:22.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4949000000000002e-05 2024-09-13T13:08:22.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:08:22.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.375606+0000", 2024-09-13T13:08:22.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108652 2024-09-13T13:08:22.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:08:22.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.380505+0000", 2024-09-13T13:08:22.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048993880000000002 2024-09-13T13:08:22.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405223+0000", 2024-09-13T13:08:22.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024717682000000001 2024-09-13T13:08:22.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:22.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411282+0000", 2024-09-13T13:08:22.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060587799999999997 2024-09-13T13:08:22.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:08:22.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411332+0000", 2024-09-13T13:08:22.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0365000000000003e-05 2024-09-13T13:08:22.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:08:22.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:08:22.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:08:22.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411370+0000", 2024-09-13T13:08:22.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7679000000000003e-05 2024-09-13T13:08:22.121 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.121 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.121 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.121 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:08:22.121 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:08:22.121 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:08:22.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:22.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:22.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:22.224 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:22.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:08:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:08:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[36682]: osdmap e210: 8 total, 8 up, 8 in 2024-09-13T13:08:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[36682]: 4.6 scrub starts 2024-09-13T13:08:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[36682]: 4.6 scrub ok 2024-09-13T13:08:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[36682]: 5.16 scrub starts 2024-09-13T13:08:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[36682]: 5.16 scrub ok 2024-09-13T13:08:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2316568992' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[36682]: from='client.25435 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:22.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:08:22.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:08:22.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[29471]: osdmap e210: 8 total, 8 up, 8 in 2024-09-13T13:08:22.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[29471]: 4.6 scrub starts 2024-09-13T13:08:22.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[29471]: 4.6 scrub ok 2024-09-13T13:08:22.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[29471]: 5.16 scrub starts 2024-09-13T13:08:22.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[29471]: 5.16 scrub ok 2024-09-13T13:08:22.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2316568992' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:22.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:21 smithi052 ceph-mon[29471]: from='client.25435 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:22.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:22.278 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:22.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:22.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:21 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:08:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:21 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:08:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:21 smithi167 ceph-mon[33178]: osdmap e210: 8 total, 8 up, 8 in 2024-09-13T13:08:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:21 smithi167 ceph-mon[33178]: 4.6 scrub starts 2024-09-13T13:08:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:21 smithi167 ceph-mon[33178]: 4.6 scrub ok 2024-09-13T13:08:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:21 smithi167 ceph-mon[33178]: 5.16 scrub starts 2024-09-13T13:08:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:21 smithi167 ceph-mon[33178]: 5.16 scrub ok 2024-09-13T13:08:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:21 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2316568992' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:21 smithi167 ceph-mon[33178]: from='client.25435 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:22.379 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:22.410 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:22.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:22.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:22.513 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:22.536 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:08:22.536 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:08:22.536 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-13T13:08:22.536 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T13:08:22.537 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T13:08:22.537 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-13T13:08:22.549 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:22.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:22.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:22.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:22.688 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:22.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:22.790 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:22.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:22.834 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:22.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:22.898 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:22.935 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:22.970 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:22.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:22.988 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T13:08:22.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:22 smithi052 ceph-mon[36682]: pgmap v346: 148 pgs: 148 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 937 op/s 2024-09-13T13:08:22.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:22 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:08:22.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:22 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:08:22.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:22 smithi052 ceph-mon[36682]: from='client.25435 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:08:22.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:22 smithi052 ceph-mon[36682]: osdmap e211: 8 total, 8 up, 8 in 2024-09-13T13:08:22.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:22 smithi052 ceph-mon[29471]: pgmap v346: 148 pgs: 148 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 937 op/s 2024-09-13T13:08:22.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:22 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:08:22.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:22 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:08:22.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:22 smithi052 ceph-mon[29471]: from='client.25435 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:08:22.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:22 smithi052 ceph-mon[29471]: osdmap e211: 8 total, 8 up, 8 in 2024-09-13T13:08:23.067 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:08:23.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:23.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:23.110 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:23.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:23.204 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:22 smithi167 ceph-mon[33178]: pgmap v346: 148 pgs: 148 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 937 op/s 2024-09-13T13:08:23.204 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:22 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:08:23.204 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:22 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:08:23.204 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:22 smithi167 ceph-mon[33178]: from='client.25435 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:08:23.205 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:22 smithi167 ceph-mon[33178]: osdmap e211: 8 total, 8 up, 8 in 2024-09-13T13:08:23.211 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:23.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:23.253 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:23.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:23.354 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:23.395 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:23.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:23.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:23.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:23.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:23.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:23.636 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:23.670 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:23.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:23.713 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:08:23.713 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T13:08:23.713 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:08:23.772 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:23.812 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:23.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:23.914 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:23.960 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:23.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:23.972 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:23 smithi052 ceph-mon[36682]: 6.2 scrub starts 2024-09-13T13:08:23.972 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:23 smithi052 ceph-mon[36682]: 6.2 scrub ok 2024-09-13T13:08:23.972 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:23 smithi052 ceph-mon[36682]: 3.17 scrub starts 2024-09-13T13:08:23.972 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:23 smithi052 ceph-mon[36682]: 3.17 scrub ok 2024-09-13T13:08:23.972 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:23 smithi052 ceph-mon[29471]: 6.2 scrub starts 2024-09-13T13:08:23.972 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:23 smithi052 ceph-mon[29471]: 6.2 scrub ok 2024-09-13T13:08:23.972 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:23 smithi052 ceph-mon[29471]: 3.17 scrub starts 2024-09-13T13:08:23.973 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:23 smithi052 ceph-mon[29471]: 3.17 scrub ok 2024-09-13T13:08:24.000 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:08:24.001 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:08:24.037 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:08:24.037 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:08:24.062 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:24.069 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:24.077 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:08:24.077 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:08:24.111 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:08:24.111 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:08:24.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:24.121 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:24.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:24.144 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:08:24.145 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:08:24.182 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:08:24.182 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:08:24.213 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:08:24.213 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:08:24.222 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:24.249 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:08:24.249 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:08:24.256 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:24.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:24.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:24.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:08:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:08:24.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:23 smithi167 ceph-mon[33178]: 6.2 scrub starts 2024-09-13T13:08:24.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:23 smithi167 ceph-mon[33178]: 6.2 scrub ok 2024-09-13T13:08:24.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:23 smithi167 ceph-mon[33178]: 3.17 scrub starts 2024-09-13T13:08:24.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:23 smithi167 ceph-mon[33178]: 3.17 scrub ok 2024-09-13T13:08:24.358 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:24.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:24.411 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:24.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:24.512 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:24.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:24.555 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:24.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:24.656 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:24.689 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:24.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:24.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:24.765 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:08:24.791 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:24.849 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:24.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:24.950 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:24.994 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:24.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:24.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:25.095 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:25.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:25.138 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:25.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:25.238 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:25.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:24 smithi052 ceph-mon[36682]: pgmap v348: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-09-13T13:08:25.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:24 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:25.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:24 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1064268940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:25.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:24 smithi052 ceph-mon[29471]: pgmap v348: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-09-13T13:08:25.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:24 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:25.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:24 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1064268940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:25.281 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:25.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:25.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:24 smithi167 ceph-mon[33178]: pgmap v348: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-09-13T13:08:25.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:24 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:25.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:24 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1064268940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:25.382 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:25.422 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:25.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:25.500 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:08:25.501 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:08:25.524 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:25.569 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:25.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:25.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:25.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:25.705 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:25.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:25.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:25.836 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:25.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:25.938 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:25.975 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:25.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:26.076 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:26.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:26.112 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:26.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:26.213 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:26.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:08:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:08:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:08:26.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:25 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:08:26.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:25 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:08:26.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:25 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:08:26.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:25 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:08:26.252 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:26.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:26.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:25 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:08:26.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:25 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:08:26.353 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:26.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:26.398 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:26.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:26.499 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:26.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:26.542 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:26.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:26.552 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T13:08:26.589 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:26.643 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:26.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:26.692 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:26.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:26.793 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:26.819 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:26.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:26.921 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:26.976 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:26.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:27.078 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:27.086 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:26 smithi052 ceph-mon[36682]: 5.4 scrub starts 2024-09-13T13:08:27.086 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:26 smithi052 ceph-mon[36682]: 5.4 scrub ok 2024-09-13T13:08:27.086 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:26 smithi052 ceph-mon[36682]: pgmap v349: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-13T13:08:27.086 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:26 smithi052 ceph-mon[36682]: osdmap e212: 8 total, 8 up, 8 in 2024-09-13T13:08:27.087 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:26 smithi052 ceph-mon[36682]: 5.b scrub starts 2024-09-13T13:08:27.087 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:26 smithi052 ceph-mon[36682]: 5.b scrub ok 2024-09-13T13:08:27.088 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:26 smithi052 ceph-mon[29471]: 5.4 scrub starts 2024-09-13T13:08:27.088 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:26 smithi052 ceph-mon[29471]: 5.4 scrub ok 2024-09-13T13:08:27.088 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:26 smithi052 ceph-mon[29471]: pgmap v349: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-13T13:08:27.088 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:26 smithi052 ceph-mon[29471]: osdmap e212: 8 total, 8 up, 8 in 2024-09-13T13:08:27.088 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:26 smithi052 ceph-mon[29471]: 5.b scrub starts 2024-09-13T13:08:27.088 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:26 smithi052 ceph-mon[29471]: 5.b scrub ok 2024-09-13T13:08:27.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:27.122 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:27.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:27.223 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:27.254 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:27.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:27.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:27.318 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:27.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:26 smithi167 ceph-mon[33178]: 5.4 scrub starts 2024-09-13T13:08:27.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:26 smithi167 ceph-mon[33178]: 5.4 scrub ok 2024-09-13T13:08:27.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:26 smithi167 ceph-mon[33178]: pgmap v349: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-13T13:08:27.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:26 smithi167 ceph-mon[33178]: osdmap e212: 8 total, 8 up, 8 in 2024-09-13T13:08:27.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:26 smithi167 ceph-mon[33178]: 5.b scrub starts 2024-09-13T13:08:27.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:26 smithi167 ceph-mon[33178]: 5.b scrub ok 2024-09-13T13:08:27.356 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:27.403 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:08:27.413 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:27.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:27.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:27.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:27.545 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:27.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:27.646 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:27.673 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:27.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:27.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:27.775 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:27.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:27.805 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:27.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:27.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:27.931 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:27.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:27.969 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:08:27.983 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:08:27.984 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:08:28.033 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:28.089 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:28.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:28.190 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:28.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:28.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:28.228 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:27 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2459675766' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:08:28.228 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:27 smithi052 ceph-mon[36682]: from='client.25441 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:08:28.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:27 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2459675766' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:08:28.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:27 smithi052 ceph-mon[29471]: from='client.25441 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:08:28.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:27 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2459675766' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:08:28.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:27 smithi167 ceph-mon[33178]: from='client.25441 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:08:28.322 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:28.358 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:28.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:28.460 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:28.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:28.499 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:28.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:28.600 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:28.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:28.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:28.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:28.731 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:28.752 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:28.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:28.854 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:28.884 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:28.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:28.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:29.012 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:29.047 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:29.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:29.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:29.149 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:29.187 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:29.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:29.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[36682]: 6.c scrub starts 2024-09-13T13:08:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[36682]: 6.c scrub ok 2024-09-13T13:08:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[36682]: 6.1d scrub starts 2024-09-13T13:08:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[36682]: 6.1d scrub ok 2024-09-13T13:08:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[36682]: pgmap v351: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 750 B/s rd, 0 B/s wr, 990 op/s 2024-09-13T13:08:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[36682]: from='client.25441 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:08:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[36682]: osdmap e213: 8 total, 8 up, 8 in 2024-09-13T13:08:29.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[29471]: 6.c scrub starts 2024-09-13T13:08:29.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[29471]: 6.c scrub ok 2024-09-13T13:08:29.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[29471]: 6.1d scrub starts 2024-09-13T13:08:29.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[29471]: 6.1d scrub ok 2024-09-13T13:08:29.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[29471]: pgmap v351: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 750 B/s rd, 0 B/s wr, 990 op/s 2024-09-13T13:08:29.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:29.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[29471]: from='client.25441 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:08:29.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:28 smithi052 ceph-mon[29471]: osdmap e213: 8 total, 8 up, 8 in 2024-09-13T13:08:29.289 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:29.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:28 smithi167 ceph-mon[33178]: 6.c scrub starts 2024-09-13T13:08:29.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:28 smithi167 ceph-mon[33178]: 6.c scrub ok 2024-09-13T13:08:29.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:28 smithi167 ceph-mon[33178]: 6.1d scrub starts 2024-09-13T13:08:29.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:28 smithi167 ceph-mon[33178]: 6.1d scrub ok 2024-09-13T13:08:29.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:28 smithi167 ceph-mon[33178]: pgmap v351: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 750 B/s rd, 0 B/s wr, 990 op/s 2024-09-13T13:08:29.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:28 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:29.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:28 smithi167 ceph-mon[33178]: from='client.25441 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:08:29.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:28 smithi167 ceph-mon[33178]: osdmap e213: 8 total, 8 up, 8 in 2024-09-13T13:08:29.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:29.332 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:29.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:29.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:29.463 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:29.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:29.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:29.521 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:08:29.521 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:08:29.521 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-13T13:08:29.521 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T13:08:29.521 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T13:08:29.522 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-13T13:08:29.565 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:29.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:29.605 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:29.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:29.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:29.738 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:29.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:29.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:29.840 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:29.885 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:29.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:29.986 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:30.016 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:30.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:30.118 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:30.143 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:30.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:30.244 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:30.276 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:30.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:30.378 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:30.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:30.421 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:30.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:30.522 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:30.549 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:30.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:30.651 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:30.690 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:30.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:30.791 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:30.813 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:30.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:30.915 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:30.944 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:30.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:31.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:31.074 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:31.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:31.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:31.176 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:31.201 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:31.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:31.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:31.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:31.313 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:30 smithi052 ceph-mon[36682]: pgmap v353: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-13T13:08:31.313 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:30 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:08:31.313 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:30 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:08:31.314 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:30 smithi052 ceph-mon[29471]: pgmap v353: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-13T13:08:31.314 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:30 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:08:31.314 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:30 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:08:31.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:30 smithi167 ceph-mon[33178]: pgmap v353: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-13T13:08:31.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:30 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:08:31.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:30 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:08:31.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:31.334 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:31.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:31.435 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:31.456 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:31.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:31.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:31.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:31.593 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:31.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:31.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:31.695 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:31.732 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:31.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:31.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:31.834 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:31.873 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:31.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:31.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:31.975 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:32.015 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:32.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:32.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:32 smithi167 ceph-mon[33178]: osdmap e214: 8 total, 8 up, 8 in 2024-09-13T13:08:32.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:32.140 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:32.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:32.242 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:32.277 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:32.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:32.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:32.379 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:32.418 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:32.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:32.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:32.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:32 smithi052 ceph-mon[36682]: osdmap e214: 8 total, 8 up, 8 in 2024-09-13T13:08:32.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:32 smithi052 ceph-mon[29471]: osdmap e214: 8 total, 8 up, 8 in 2024-09-13T13:08:32.520 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:32.563 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:32.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:32.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:32.665 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:32.693 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:32.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:32.795 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:32.823 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:32.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:32.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:32.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:32.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:32.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:33.062 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:33.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:33 smithi167 ceph-mon[33178]: pgmap v355: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-13T13:08:33.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:33.094 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:33.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:33.195 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:33.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:33.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:33.321 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:33.360 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:33.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:33.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:33.462 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:33.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:33 smithi052 ceph-mon[36682]: pgmap v355: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-13T13:08:33.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:33 smithi052 ceph-mon[29471]: pgmap v355: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-13T13:08:33.491 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:33.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:33.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:33.593 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:33.615 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:33.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:33.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:33.717 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:33.753 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:33.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:33.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:33.855 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:33.894 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:33.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:33.997 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:34.034 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:34.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:34.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:34.136 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:34.178 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:34.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:34.280 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:34.304 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:34.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:34.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:34 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:08:34.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:08:33 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:08:34.406 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:34.443 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:34.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:34.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:34.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:34 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:08:34.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:34 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:08:34.544 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:34.580 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:34.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:34.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:34.682 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:34.711 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:34.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:34.813 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:34.850 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:34.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:34.952 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:34.991 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:34.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:35.092 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:35.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:35.133 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:35.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:35.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:35.259 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:35.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:35.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:35.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:35 smithi167 ceph-mon[33178]: pgmap v356: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-13T13:08:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:35.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:35.397 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:35.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:35.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:35 smithi052 ceph-mon[36682]: pgmap v356: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-13T13:08:35.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:35.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:35 smithi052 ceph-mon[29471]: pgmap v356: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-13T13:08:35.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:35.499 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:35.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:35.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:35.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:35.546 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:35.591 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T13:08:35.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:35.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:35.664 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:35.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:35.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:35.792 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:35.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:35.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:35.920 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:35.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:36.021 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:36.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:36.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:36.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:36.151 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:36.206 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:36.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:36.215 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:08:36.216 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:08:36.217 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:08:36.261 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:08:36.261 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:08:36.290 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:36.298 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:08:36.299 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:08:36.307 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:36.318 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:36.333 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:36.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:36.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:36.341 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:08:36.342 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:08:36.372 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:08:36.373 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:08:36.410 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:08:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:08:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:08:36.410 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:08:36.410 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:08:36.435 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:36.443 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:08:36.443 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:08:36.465 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:36.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:36.481 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:08:36.482 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:08:36.567 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:36.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:36.623 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:36.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:36.724 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:36.759 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:36.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:36.860 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:36.919 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:36.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:36.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:37.021 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:37.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:37.054 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:37.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:37.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:37 smithi052 ceph-mon[36682]: pgmap v357: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-13T13:08:37.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:37 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/324883615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:37.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:37 smithi052 ceph-mon[29471]: pgmap v357: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-13T13:08:37.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:37 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/324883615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:37.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:37.181 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:37.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:37.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:08:37.210 INFO:teuthology.orchestra.run.smithi052.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T13:08:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:08:37.282 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:37.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:37.316 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:37.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:37.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:37 smithi167 ceph-mon[33178]: pgmap v357: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-13T13:08:37.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:37 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/324883615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:37.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:37.457 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:37.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:37.559 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:37.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:37.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:37.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:37.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:37.724 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:37.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:37.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:37.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:37.858 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:37.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:37.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:37.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:37.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:38.092 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:38.132 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:38.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:38.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:38.233 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:38.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:38.274 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:38.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:38.302 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:38 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:08:38.302 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:38 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:08:38.303 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:38 smithi052 ceph-mon[36682]: osdmap e215: 8 total, 8 up, 8 in 2024-09-13T13:08:38.303 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:38 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:08:38.303 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:38 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:08:38.303 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:38 smithi052 ceph-mon[29471]: osdmap e215: 8 total, 8 up, 8 in 2024-09-13T13:08:38.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:38 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:08:38.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:38 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:08:38.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:38 smithi167 ceph-mon[33178]: osdmap e215: 8 total, 8 up, 8 in 2024-09-13T13:08:38.375 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:38.417 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:38.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:38.420 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:38.479 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:08:38.479 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:08:38.479 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:08:38.479 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:08:38.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.479 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83976 7.7 7:ec00590c:::benchmark_data_smithi052_71865_object83975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:08:38.480 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.558362+0000", 2024-09-13T13:08:38.480 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 323.91574314799999, 2024-09-13T13:08:38.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99679333000000003, 2024-09-13T13:08:38.480 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.480 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.480 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.480 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.480 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.480 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.480 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83976 2024-09-13T13:08:38.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.481 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.481 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.481 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558362+0000", 2024-09-13T13:08:38.481 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.481 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.481 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558362+0000", 2024-09-13T13:08:38.481 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.482 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.482 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558364+0000", 2024-09-13T13:08:38.482 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0870000000000001e-06 2024-09-13T13:08:38.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.482 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.482 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558389+0000", 2024-09-13T13:08:38.482 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5051000000000001e-05 2024-09-13T13:08:38.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.482 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.482 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558401+0000", 2024-09-13T13:08:38.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1943000000000001e-05 2024-09-13T13:08:38.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558409+0000", 2024-09-13T13:08:38.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1580000000000005e-06 2024-09-13T13:08:38.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558524+0000", 2024-09-13T13:08:38.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114991 2024-09-13T13:08:38.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:08:38.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558532+0000", 2024-09-13T13:08:38.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0450000000000001e-06 2024-09-13T13:08:38.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.542003+0000", 2024-09-13T13:08:38.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98347105700000004 2024-09-13T13:08:38.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.544289+0000", 2024-09-13T13:08:38.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002285493 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.544471+0000", 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018247200000000001 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555006+0000", 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010534559000000001 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555018+0000", 2024-09-13T13:08:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1756999999999999e-05 2024-09-13T13:08:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555068+0000", 2024-09-13T13:08:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9982999999999999e-05 2024-09-13T13:08:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555155+0000", 2024-09-13T13:08:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7733999999999997e-05 2024-09-13T13:08:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.487 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.487 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83980 7.7 7:e454f641:::benchmark_data_smithi052_71865_object83979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:08:38.487 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.562536+0000", 2024-09-13T13:08:38.487 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 323.911568768, 2024-09-13T13:08:38.487 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99324992999999995, 2024-09-13T13:08:38.487 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.487 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.487 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.487 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.487 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.487 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.488 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83980 2024-09-13T13:08:38.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.488 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.488 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562536+0000", 2024-09-13T13:08:38.488 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.488 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562536+0000", 2024-09-13T13:08:38.488 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.489 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.489 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562538+0000", 2024-09-13T13:08:38.489 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9460000000000001e-06 2024-09-13T13:08:38.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.489 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.489 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562564+0000", 2024-09-13T13:08:38.489 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5812e-05 2024-09-13T13:08:38.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562579+0000", 2024-09-13T13:08:38.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4515000000000001e-05 2024-09-13T13:08:38.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562587+0000", 2024-09-13T13:08:38.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1529999999999997e-06 2024-09-13T13:08:38.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562654+0000", 2024-09-13T13:08:38.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7094000000000005e-05 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562662+0000", 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2870000000000004e-06 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548143+0000", 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98548045299999998 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548261+0000", 2024-09-13T13:08:38.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118776 2024-09-13T13:08:38.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.492 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:08:38.492 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548419+0000", 2024-09-13T13:08:38.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015719799999999999 2024-09-13T13:08:38.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.492 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.492 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555169+0000", 2024-09-13T13:08:38.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0067500839999999999 2024-09-13T13:08:38.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.493 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.493 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555200+0000", 2024-09-13T13:08:38.493 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1285999999999998e-05 2024-09-13T13:08:38.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.493 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.493 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555285+0000", 2024-09-13T13:08:38.493 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4899000000000001e-05 2024-09-13T13:08:38.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555786+0000", 2024-09-13T13:08:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00050142700000000004 2024-09-13T13:08:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83984 7.7 7:e1036ce8:::benchmark_data_smithi052_71865_object83983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:08:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.567703+0000", 2024-09-13T13:08:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 323.906401652, 2024-09-13T13:08:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99284181800000004, 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83984 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567703+0000", 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567703+0000", 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567706+0000", 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3889999999999999e-06 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567733+0000", 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.69e-05 2024-09-13T13:08:38.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567754+0000", 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1772000000000001e-05 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567765+0000", 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0553999999999999e-05 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567856+0000", 2024-09-13T13:08:38.497 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0926000000000003e-05 2024-09-13T13:08:38.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:08:38.498 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567874+0000", 2024-09-13T13:08:38.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8481000000000001e-05 2024-09-13T13:08:38.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.498 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552014+0000", 2024-09-13T13:08:38.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98413990600000001 2024-09-13T13:08:38.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.499 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552117+0000", 2024-09-13T13:08:38.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010265200000000001 2024-09-13T13:08:38.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.499 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:08:38.499 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552243+0000", 2024-09-13T13:08:38.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012594900000000001 2024-09-13T13:08:38.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.499 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.499 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560036+0000", 2024-09-13T13:08:38.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0077931570000000002 2024-09-13T13:08:38.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.500 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.500 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560388+0000", 2024-09-13T13:08:38.500 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035232500000000001 2024-09-13T13:08:38.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.501 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.501 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560444+0000", 2024-09-13T13:08:38.501 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5460000000000001e-05 2024-09-13T13:08:38.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.501 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.501 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560545+0000", 2024-09-13T13:08:38.501 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101347 2024-09-13T13:08:38.501 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.501 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.501 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.502 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.14493.0:1217 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94064716851456 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:08:38.502 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:15.061486+0000", 2024-09-13T13:08:38.502 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 323.41261949900002, 2024-09-13T13:08:38.502 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.48025142399999998, 2024-09-13T13:08:38.502 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.502 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.502 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:08:38.502 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.502 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.14493", 2024-09-13T13:08:38.503 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:08:38.503 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 1217 2024-09-13T13:08:38.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.503 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.503 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.503 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061486+0000", 2024-09-13T13:08:38.503 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.503 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.503 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061486+0000", 2024-09-13T13:08:38.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.504 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061489+0000", 2024-09-13T13:08:38.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3249999999999999e-06 2024-09-13T13:08:38.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.504 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061499+0000", 2024-09-13T13:08:38.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0463e-05 2024-09-13T13:08:38.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.505 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.505 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061513+0000", 2024-09-13T13:08:38.505 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3269e-05 2024-09-13T13:08:38.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.505 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.505 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061521+0000", 2024-09-13T13:08:38.505 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.909e-06 2024-09-13T13:08:38.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.505 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.506 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061582+0000", 2024-09-13T13:08:38.506 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1568000000000004e-05 2024-09-13T13:08:38.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.506 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:08:38.506 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061589+0000", 2024-09-13T13:08:38.506 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2060000000000001e-06 2024-09-13T13:08:38.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.506 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.506 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.537301+0000", 2024-09-13T13:08:38.506 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.47571140000000001 2024-09-13T13:08:38.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.537446+0000", 2024-09-13T13:08:38.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000145339 2024-09-13T13:08:38.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.541737+0000", 2024-09-13T13:08:38.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.004290945 2024-09-13T13:08:38.507 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.507 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.507 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.508 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:118708 7.d e115/103 7:b430a861:::benchmark_data_smithi052_71865_object118707:head v 115'7520, mlcod=115'7517)", 2024-09-13T13:08:38.508 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:49.561615+0000", 2024-09-13T13:08:38.508 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 288.912489687, 2024-09-13T13:08:38.508 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.115160417, 2024-09-13T13:08:38.508 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.508 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.508 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.508 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.508 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.508 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.561615+0000", 2024-09-13T13:08:38.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.561615+0000", 2024-09-13T13:08:38.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.561617+0000", 2024-09-13T13:08:38.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.409e-06 2024-09-13T13:08:38.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.561641+0000", 2024-09-13T13:08:38.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4428000000000001e-05 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.561652+0000", 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0968000000000001e-05 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.561660+0000", 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3769999999999998e-06 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.562187+0000", 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00052744300000000005 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.562209+0000", 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2456e-05 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.673867+0000", 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.111657149 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.512 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.676776+0000", 2024-09-13T13:08:38.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029091870000000001 2024-09-13T13:08:38.513 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.513 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.513 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.513 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:118713 7.d e115/103 7:b0c4bc72:::benchmark_data_smithi052_71865_object118712:head v 115'7521, mlcod=115'7517)", 2024-09-13T13:08:38.513 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:49.563909+0000", 2024-09-13T13:08:38.513 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 288.910195655, 2024-09-13T13:08:38.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.118718926, 2024-09-13T13:08:38.513 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.513 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.563909+0000", 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.563909+0000", 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.563910+0000", 2024-09-13T13:08:38.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.02e-07 2024-09-13T13:08:38.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.515 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.563942+0000", 2024-09-13T13:08:38.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1276999999999999e-05 2024-09-13T13:08:38.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.563949+0000", 2024-09-13T13:08:38.516 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8709999999999997e-06 2024-09-13T13:08:38.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.563955+0000", 2024-09-13T13:08:38.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7919999999999998e-06 2024-09-13T13:08:38.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.671251+0000", 2024-09-13T13:08:38.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.107295289 2024-09-13T13:08:38.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.671270+0000", 2024-09-13T13:08:38.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9649000000000001e-05 2024-09-13T13:08:38.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.682606+0000", 2024-09-13T13:08:38.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011335567 2024-09-13T13:08:38.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.682628+0000", 2024-09-13T13:08:38.519 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2578999999999998e-05 2024-09-13T13:08:38.519 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.519 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.519 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.519 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:118721 7.7 7:e1e4eb83:::benchmark_data_smithi052_71865_object118720:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:08:38.519 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:49.564255+0000", 2024-09-13T13:08:38.519 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 288.90984973000002, 2024-09-13T13:08:38.519 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11708737399999999, 2024-09-13T13:08:38.519 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.520 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.520 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.520 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.520 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.520 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.520 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 118721 2024-09-13T13:08:38.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.520 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.520 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.564255+0000", 2024-09-13T13:08:38.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.564255+0000", 2024-09-13T13:08:38.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.564256+0000", 2024-09-13T13:08:38.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9999999999999997e-07 2024-09-13T13:08:38.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.522 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.564277+0000", 2024-09-13T13:08:38.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1061e-05 2024-09-13T13:08:38.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.564283+0000", 2024-09-13T13:08:38.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3969999999999999e-06 2024-09-13T13:08:38.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.564289+0000", 2024-09-13T13:08:38.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6500000000000001e-06 2024-09-13T13:08:38.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.564921+0000", 2024-09-13T13:08:38.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00063224500000000003 2024-09-13T13:08:38.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.565041+0000", 2024-09-13T13:08:38.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011938 2024-09-13T13:08:38.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:08:38.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.671974+0000", 2024-09-13T13:08:38.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10693373 2024-09-13T13:08:38.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.677884+0000", 2024-09-13T13:08:38.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0059090480000000001 2024-09-13T13:08:38.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.681076+0000", 2024-09-13T13:08:38.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031923379999999999 2024-09-13T13:08:38.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.681247+0000", 2024-09-13T13:08:38.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017127599999999999 2024-09-13T13:08:38.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.681343+0000", 2024-09-13T13:08:38.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5549000000000003e-05 2024-09-13T13:08:38.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.526 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120034 7.7 7:e04aff1b:::benchmark_data_smithi052_71865_object120033:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.021565+0000", 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 287.45253994799998, 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.117078261, 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120034 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.527 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021565+0000", 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021565+0000", 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021566+0000", 2024-09-13T13:08:38.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.062e-06 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021588+0000", 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1614999999999999e-05 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021596+0000", 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2190000000000004e-06 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021603+0000", 2024-09-13T13:08:38.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7719999999999997e-06 2024-09-13T13:08:38.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022220+0000", 2024-09-13T13:08:38.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00061740999999999999 2024-09-13T13:08:38.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022327+0000", 2024-09-13T13:08:38.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000106872 2024-09-13T13:08:38.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:08:38.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022465+0000", 2024-09-13T13:08:38.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000137992 2024-09-13T13:08:38.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.029664+0000", 2024-09-13T13:08:38.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0071989549999999999 2024-09-13T13:08:38.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138504+0000", 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.108840177 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138557+0000", 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3186000000000002e-05 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138643+0000", 2024-09-13T13:08:38.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6001000000000006e-05 2024-09-13T13:08:38.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.533 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.533 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120036 7.6 7:6992d7bb:::benchmark_data_smithi052_71865_object120035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:08:38.533 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.022077+0000", 2024-09-13T13:08:38.533 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 287.45202829099998, 2024-09-13T13:08:38.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.116773496, 2024-09-13T13:08:38.533 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.533 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.533 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.534 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.534 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.534 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.534 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120036 2024-09-13T13:08:38.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.534 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022077+0000", 2024-09-13T13:08:38.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022077+0000", 2024-09-13T13:08:38.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022078+0000", 2024-09-13T13:08:38.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3430000000000001e-06 2024-09-13T13:08:38.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022108+0000", 2024-09-13T13:08:38.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9493e-05 2024-09-13T13:08:38.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022118+0000", 2024-09-13T13:08:38.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0152e-05 2024-09-13T13:08:38.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022124+0000", 2024-09-13T13:08:38.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7159999999999997e-06 2024-09-13T13:08:38.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022303+0000", 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017860099999999999 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022414+0000", 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000111276 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022541+0000", 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012691800000000001 2024-09-13T13:08:38.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032820+0000", 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010279039 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039742+0000", 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0069220109999999996 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138709+0000", 2024-09-13T13:08:38.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098966328000000006 2024-09-13T13:08:38.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.539 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138810+0000", 2024-09-13T13:08:38.539 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101604 2024-09-13T13:08:38.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.539 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138850+0000", 2024-09-13T13:08:38.539 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0015000000000002e-05 2024-09-13T13:08:38.539 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.539 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.539 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120037 7.6 7:6874521c:::benchmark_data_smithi052_71865_object120036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.022235+0000", 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 287.45186981400002, 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11786221199999999, 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120037 2024-09-13T13:08:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022235+0000", 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022235+0000", 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022236+0000", 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1769999999999999e-06 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022256+0000", 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9351000000000002e-05 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022264+0000", 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2700000000000004e-06 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022270+0000", 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6380000000000001e-06 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028622+0000", 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0063519969999999999 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028742+0000", 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120711 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:38.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028872+0000", 2024-09-13T13:08:38.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000129338 2024-09-13T13:08:38.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032859+0000", 2024-09-13T13:08:38.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039876989999999999 2024-09-13T13:08:38.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138897+0000", 2024-09-13T13:08:38.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.106037824 2024-09-13T13:08:38.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139029+0000", 2024-09-13T13:08:38.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013220800000000001 2024-09-13T13:08:38.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139083+0000", 2024-09-13T13:08:38.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3971999999999998e-05 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.140097+0000", 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001014027 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120038 7.7 7:e43b7369:::benchmark_data_smithi052_71865_object120037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.023804+0000", 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 287.45030113500002, 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.115066788, 2024-09-13T13:08:38.546 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.547 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.547 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.547 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.547 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.547 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.547 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120038 2024-09-13T13:08:38.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.547 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.023804+0000", 2024-09-13T13:08:38.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.023804+0000", 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.023806+0000", 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7719999999999999e-06 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.023853+0000", 2024-09-13T13:08:38.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7648000000000002e-05 2024-09-13T13:08:38.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.023864+0000", 2024-09-13T13:08:38.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0586e-05 2024-09-13T13:08:38.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.023871+0000", 2024-09-13T13:08:38.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4569999999999999e-06 2024-09-13T13:08:38.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.025868+0000", 2024-09-13T13:08:38.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019967930000000002 2024-09-13T13:08:38.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.025991+0000", 2024-09-13T13:08:38.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012252699999999999 2024-09-13T13:08:38.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:08:38.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.026125+0000", 2024-09-13T13:08:38.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013470600000000001 2024-09-13T13:08:38.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032642+0000", 2024-09-13T13:08:38.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0065167289999999998 2024-09-13T13:08:38.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138726+0000", 2024-09-13T13:08:38.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.106083372 2024-09-13T13:08:38.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138778+0000", 2024-09-13T13:08:38.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2120999999999998e-05 2024-09-13T13:08:38.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138871+0000", 2024-09-13T13:08:38.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3077000000000003e-05 2024-09-13T13:08:38.552 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.552 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.552 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.553 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120039 7.6 7:6fe353e4:::benchmark_data_smithi052_71865_object120038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:08:38.553 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.024096+0000", 2024-09-13T13:08:38.553 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 287.45000884699999, 2024-09-13T13:08:38.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.121330047, 2024-09-13T13:08:38.553 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.553 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.553 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.553 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.553 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.553 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.553 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120039 2024-09-13T13:08:38.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.553 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024096+0000", 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024096+0000", 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024098+0000", 2024-09-13T13:08:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3459999999999999e-06 2024-09-13T13:08:38.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024118+0000", 2024-09-13T13:08:38.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0363000000000001e-05 2024-09-13T13:08:38.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024129+0000", 2024-09-13T13:08:38.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1396000000000001e-05 2024-09-13T13:08:38.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024136+0000", 2024-09-13T13:08:38.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8369999999999999e-06 2024-09-13T13:08:38.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032888+0000", 2024-09-13T13:08:38.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0087514399999999992 2024-09-13T13:08:38.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.557 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:38.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032997+0000", 2024-09-13T13:08:38.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109399 2024-09-13T13:08:38.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:38.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033123+0000", 2024-09-13T13:08:38.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012575100000000001 2024-09-13T13:08:38.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138649+0000", 2024-09-13T13:08:38.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10552584600000001 2024-09-13T13:08:38.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.140107+0000", 2024-09-13T13:08:38.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0014585620000000001 2024-09-13T13:08:38.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145128+0000", 2024-09-13T13:08:38.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050206909999999999 2024-09-13T13:08:38.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145376+0000", 2024-09-13T13:08:38.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024774999999999998 2024-09-13T13:08:38.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145426+0000", 2024-09-13T13:08:38.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0665999999999999e-05 2024-09-13T13:08:38.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.561 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.561 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120041 7.7 7:e939ce74:::benchmark_data_smithi052_71865_object120040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:08:38.561 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.024471+0000", 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 287.44963395399998, 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.114681952, 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120041 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024471+0000", 2024-09-13T13:08:38.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024471+0000", 2024-09-13T13:08:38.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024472+0000", 2024-09-13T13:08:38.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.184e-06 2024-09-13T13:08:38.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024493+0000", 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0216000000000001e-05 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024502+0000", 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2110000000000004e-06 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024508+0000", 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4479999999999999e-06 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.029696+0000", 2024-09-13T13:08:38.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051878699999999998 2024-09-13T13:08:38.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.029809+0000", 2024-09-13T13:08:38.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112899 2024-09-13T13:08:38.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:08:38.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.029949+0000", 2024-09-13T13:08:38.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013980199999999999 2024-09-13T13:08:38.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138400+0000", 2024-09-13T13:08:38.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.108450824 2024-09-13T13:08:38.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138921+0000", 2024-09-13T13:08:38.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00052132800000000005 2024-09-13T13:08:38.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139047+0000", 2024-09-13T13:08:38.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012600999999999999 2024-09-13T13:08:38.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139153+0000", 2024-09-13T13:08:38.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010616 2024-09-13T13:08:38.567 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.568 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.568 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.568 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120046 7.6 7:64955858:::benchmark_data_smithi052_71865_object120045:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:08:38.568 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.028119+0000", 2024-09-13T13:08:38.568 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 287.44598593400002, 2024-09-13T13:08:38.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12583912, 2024-09-13T13:08:38.568 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.568 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.568 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.568 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.568 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.569 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.569 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120046 2024-09-13T13:08:38.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.569 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028119+0000", 2024-09-13T13:08:38.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028119+0000", 2024-09-13T13:08:38.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.570 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.570 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028120+0000", 2024-09-13T13:08:38.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2999999999999999e-07 2024-09-13T13:08:38.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.570 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.570 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028143+0000", 2024-09-13T13:08:38.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3104000000000001e-05 2024-09-13T13:08:38.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028151+0000", 2024-09-13T13:08:38.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9850000000000006e-06 2024-09-13T13:08:38.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028158+0000", 2024-09-13T13:08:38.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5969999999999997e-06 2024-09-13T13:08:38.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036251+0000", 2024-09-13T13:08:38.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0080934660000000005 2024-09-13T13:08:38.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036367+0000", 2024-09-13T13:08:38.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000115832 2024-09-13T13:08:38.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:38.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036497+0000", 2024-09-13T13:08:38.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012981399999999999 2024-09-13T13:08:38.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145466+0000", 2024-09-13T13:08:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10896932099999999 2024-09-13T13:08:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145645+0000", 2024-09-13T13:08:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017834400000000001 2024-09-13T13:08:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153238+0000", 2024-09-13T13:08:38.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0075929919999999998 2024-09-13T13:08:38.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153469+0000", 2024-09-13T13:08:38.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00023148999999999999 2024-09-13T13:08:38.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153958+0000", 2024-09-13T13:08:38.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00048924500000000002 2024-09-13T13:08:38.574 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.574 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.575 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.575 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120049 7.6 7:6e13283c:::benchmark_data_smithi052_71865_object120048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:08:38.575 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.031872+0000", 2024-09-13T13:08:38.575 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 287.44223345400002, 2024-09-13T13:08:38.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.125660096, 2024-09-13T13:08:38.575 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.575 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.575 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.575 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.575 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.575 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.576 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120049 2024-09-13T13:08:38.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.576 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031872+0000", 2024-09-13T13:08:38.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031872+0000", 2024-09-13T13:08:38.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031873+0000", 2024-09-13T13:08:38.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0130000000000001e-06 2024-09-13T13:08:38.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031889+0000", 2024-09-13T13:08:38.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6218999999999998e-05 2024-09-13T13:08:38.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031895+0000", 2024-09-13T13:08:38.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3300000000000004e-06 2024-09-13T13:08:38.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031900+0000", 2024-09-13T13:08:38.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4129999999999999e-06 2024-09-13T13:08:38.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145674+0000", 2024-09-13T13:08:38.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.113774598 2024-09-13T13:08:38.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145885+0000", 2024-09-13T13:08:38.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000210929 2024-09-13T13:08:38.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:38.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.146035+0000", 2024-09-13T13:08:38.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015038000000000001 2024-09-13T13:08:38.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.149376+0000", 2024-09-13T13:08:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033409690000000001 2024-09-13T13:08:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153999+0000", 2024-09-13T13:08:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0046225670000000002 2024-09-13T13:08:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157357+0000", 2024-09-13T13:08:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033581800000000001 2024-09-13T13:08:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157485+0000", 2024-09-13T13:08:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012788599999999999 2024-09-13T13:08:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157532+0000", 2024-09-13T13:08:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6612e-05 2024-09-13T13:08:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:120050 7.e e115/84 7:72296fb6:::benchmark_data_smithi052_71865_object120049:head v 115'7271, mlcod=115'7268)", 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.036753+0000", 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 287.43735216099998, 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.120595173, 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036753+0000", 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036753+0000", 2024-09-13T13:08:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036754+0000", 2024-09-13T13:08:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5099999999999998e-07 2024-09-13T13:08:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036779+0000", 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5122000000000001e-05 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036786+0000", 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9879999999999998e-06 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036790+0000", 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6e-06 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138925+0000", 2024-09-13T13:08:38.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.102134962 2024-09-13T13:08:38.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138968+0000", 2024-09-13T13:08:38.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2760999999999999e-05 2024-09-13T13:08:38.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157292+0000", 2024-09-13T13:08:38.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.018324258 2024-09-13T13:08:38.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157348+0000", 2024-09-13T13:08:38.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5631000000000003e-05 2024-09-13T13:08:38.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.586 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:120054 7.e e115/84 7:7dad863d:::benchmark_data_smithi052_71865_object120053:head v 115'7272, mlcod=115'7268)", 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.038814+0000", 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 287.43529070900001, 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.118752855, 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038814+0000", 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038814+0000", 2024-09-13T13:08:38.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038815+0000", 2024-09-13T13:08:38.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0640000000000001e-06 2024-09-13T13:08:38.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038839+0000", 2024-09-13T13:08:38.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3828999999999998e-05 2024-09-13T13:08:38.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038847+0000", 2024-09-13T13:08:38.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9990000000000008e-06 2024-09-13T13:08:38.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038853+0000", 2024-09-13T13:08:38.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8329999999999999e-06 2024-09-13T13:08:38.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145132+0000", 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.106279179 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145152+0000", 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9264999999999999e-05 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157535+0000", 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.012383939 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.591 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157567+0000", 2024-09-13T13:08:38.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1746999999999999e-05 2024-09-13T13:08:38.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.591 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.591 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120055 7.6 7:6a228e92:::benchmark_data_smithi052_71865_object120054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:08:38.591 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.039067+0000", 2024-09-13T13:08:38.591 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 287.435038443, 2024-09-13T13:08:38.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.124460581, 2024-09-13T13:08:38.591 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.591 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.592 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.592 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.592 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.592 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.592 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120055 2024-09-13T13:08:38.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.592 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.592 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.592 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039067+0000", 2024-09-13T13:08:38.592 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039067+0000", 2024-09-13T13:08:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039068+0000", 2024-09-13T13:08:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7859999999999999e-06 2024-09-13T13:08:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.593 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039092+0000", 2024-09-13T13:08:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3323999999999999e-05 2024-09-13T13:08:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039103+0000", 2024-09-13T13:08:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1408e-05 2024-09-13T13:08:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039110+0000", 2024-09-13T13:08:38.594 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0700000000000001e-06 2024-09-13T13:08:38.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.595 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.595 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.149407+0000", 2024-09-13T13:08:38.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.110296832 2024-09-13T13:08:38.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.595 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.595 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.149533+0000", 2024-09-13T13:08:38.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012555699999999999 2024-09-13T13:08:38.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.596 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:08:38.596 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.149681+0000", 2024-09-13T13:08:38.596 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014797000000000001 2024-09-13T13:08:38.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.596 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.596 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153150+0000", 2024-09-13T13:08:38.596 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0034697669999999999 2024-09-13T13:08:38.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.596 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.597 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157574+0000", 2024-09-13T13:08:38.597 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.004423382 2024-09-13T13:08:38.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.597 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.597 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.163426+0000", 2024-09-13T13:08:38.597 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058521060000000001 2024-09-13T13:08:38.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.597 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.597 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.163482+0000", 2024-09-13T13:08:38.597 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6328e-05 2024-09-13T13:08:38.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.598 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.598 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.598 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.163527+0000", 2024-09-13T13:08:38.598 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5050999999999999e-05 2024-09-13T13:08:38.598 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.598 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.598 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.598 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.598 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151591 7.7 7:e6ed771a:::benchmark_data_smithi052_71865_object30490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:08:38.598 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.181773+0000", 2024-09-13T13:08:38.598 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 249.292331808, 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.115792996, 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151591 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.181773+0000", 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.600 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.600 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.181773+0000", 2024-09-13T13:08:38.600 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.600 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.600 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.181775+0000", 2024-09-13T13:08:38.600 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.64e-06 2024-09-13T13:08:38.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.181780+0000", 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3199999999999999e-06 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.181791+0000", 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1178e-05 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.181799+0000", 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.52e-06 2024-09-13T13:08:38.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.602 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.602 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.267565+0000", 2024-09-13T13:08:38.602 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.085766133999999994 2024-09-13T13:08:38.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.602 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.602 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.267685+0000", 2024-09-13T13:08:38.602 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120227 2024-09-13T13:08:38.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.602 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:08:38.602 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.267814+0000", 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000128837 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.276650+0000", 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0088356479999999998 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.295432+0000", 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.018782107999999999 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.603 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.295928+0000", 2024-09-13T13:08:38.604 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00049640600000000002 2024-09-13T13:08:38.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.604 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.604 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.604 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.296077+0000", 2024-09-13T13:08:38.604 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014841299999999999 2024-09-13T13:08:38.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.604 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.604 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.604 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.297566+0000", 2024-09-13T13:08:38.604 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001489565 2024-09-13T13:08:38.604 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.604 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.605 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.605 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.605 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:199048 7.7 7:ef6de7c0:::benchmark_data_smithi052_71865_object77947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T13:08:38.605 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:05:25.185200+0000", 2024-09-13T13:08:38.605 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 193.28890465699999, 2024-09-13T13:08:38.605 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.127661889, 2024-09-13T13:08:38.605 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:38.605 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:38.605 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:08:38.605 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:38.605 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:08:38.605 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 199048 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185200+0000", 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185200+0000", 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.606 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185201+0000", 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3900000000000004e-07 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185205+0000", 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.455e-06 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185358+0000", 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000152919 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.608 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:38.608 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185368+0000", 2024-09-13T13:08:38.608 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0465000000000001e-05 2024-09-13T13:08:38.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.608 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.608 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:38.608 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.201869+0000", 2024-09-13T13:08:38.608 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.016500959999999999 2024-09-13T13:08:38.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.608 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.608 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:38.608 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.202064+0000", 2024-09-13T13:08:38.608 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019464999999999999 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.202323+0000", 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00025961499999999999 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.210108+0000", 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.007784549 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.219023+0000", 2024-09-13T13:08:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0089156979999999997 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.312149+0000", 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.093125873999999997 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.312287+0000", 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013777700000000001 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.312862+0000", 2024-09-13T13:08:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00057518800000000002 2024-09-13T13:08:38.611 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.611 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.611 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.611 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:38.611 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:38.611 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:08:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:38.649 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:38.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:38.750 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:38.786 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:38.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:38.888 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:38.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:38.933 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:38.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:39.034 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:39.064 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:39.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:39.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:39.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:39.196 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:39.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:39.298 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:39 smithi167 ceph-mon[33178]: pgmap v359: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 733 op/s 2024-09-13T13:08:39.342 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:39.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:39.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:39.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:39.482 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:39.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:39.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:39.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:39 smithi052 ceph-mon[36682]: pgmap v359: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 733 op/s 2024-09-13T13:08:39.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:39 smithi052 ceph-mon[29471]: pgmap v359: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 733 op/s 2024-09-13T13:08:39.584 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:39.618 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:39.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:39.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:39.720 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:39.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:39.756 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:39.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:39.858 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:39.898 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:39.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:39.958 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T13:08:39.979 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:08:40.000 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:40.058 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:40.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:40.159 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:40.246 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:40.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:40.347 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:40.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:40.412 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:40.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:40.513 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:40.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:40.543 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:40.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:40.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:40.678 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:40.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:40.780 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:40.830 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:40.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:40.932 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:40.996 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:40.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:40.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:41.004 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:41.056 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:41.098 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:41.154 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:41.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:41.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:41.288 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:41.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:41.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:41 smithi167 ceph-mon[33178]: pgmap v360: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T13:08:41.361 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:41 smithi052 ceph-mon[36682]: pgmap v360: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T13:08:41.361 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:41 smithi052 ceph-mon[29471]: pgmap v360: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T13:08:41.389 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:41.436 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:41.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:41.537 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:41.572 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:41.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:41.674 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:41.709 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:41.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:41.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:41.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:41.842 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:41.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:41.898 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:08:41.899 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:08:41.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.899 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:173881 8.1 8:82407ef8:::benchmark_data_smithi052_101302_object45952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T13:08:41.899 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:41.892075+0000", 2024-09-13T13:08:41.899 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0049006259999999999, 2024-09-13T13:08:41.899 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049278760000000003, 2024-09-13T13:08:41.899 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:41.899 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:41.899 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:08:41.899 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:41.899 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:08:41.899 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:41.900 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 173881 2024-09-13T13:08:41.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.900 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:41.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.900 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:41.900 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.892075+0000", 2024-09-13T13:08:41.900 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:41.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.900 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:41.900 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.892075+0000", 2024-09-13T13:08:41.900 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:41.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.901 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.901 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:41.901 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.892077+0000", 2024-09-13T13:08:41.901 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9709999999999998e-06 2024-09-13T13:08:41.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.901 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.901 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:41.901 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.892113+0000", 2024-09-13T13:08:41.901 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5410000000000001e-05 2024-09-13T13:08:41.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.901 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.901 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:41.902 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.892161+0000", 2024-09-13T13:08:41.902 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8174000000000003e-05 2024-09-13T13:08:41.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.902 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.902 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:41.902 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.892170+0000", 2024-09-13T13:08:41.902 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0999999999999993e-06 2024-09-13T13:08:41.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.902 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.902 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:41.902 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.893980+0000", 2024-09-13T13:08:41.902 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001810435 2024-09-13T13:08:41.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.894089+0000", 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0001087 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,2", 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.894191+0000", 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101944 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.904 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:173875 8.a 8:5f30b690:::benchmark_data_smithi052_101302_object45946:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T13:08:41.904 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:41.886691+0000", 2024-09-13T13:08:41.904 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.010285069000000001, 2024-09-13T13:08:41.904 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010385718, 2024-09-13T13:08:41.904 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:41.904 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:41.904 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:08:41.904 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:41.904 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:08:41.904 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:41.905 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 173875 2024-09-13T13:08:41.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.905 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:41.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.905 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:41.905 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.886691+0000", 2024-09-13T13:08:41.905 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:41.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.905 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:41.905 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.886691+0000", 2024-09-13T13:08:41.905 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:41.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.906 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:41.906 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.886692+0000", 2024-09-13T13:08:41.906 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5120000000000001e-06 2024-09-13T13:08:41.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.906 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:41.906 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.886697+0000", 2024-09-13T13:08:41.906 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9370000000000003e-06 2024-09-13T13:08:41.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.906 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:41.906 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.886706+0000", 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0310000000000004e-06 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.886713+0000", 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5799999999999997e-06 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.889604+0000", 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0028908610000000002 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.907 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.889705+0000", 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010110799999999999 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,5", 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.889807+0000", 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102027 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.895123+0000", 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053162499999999998 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:41.908 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:173877 8.7 8:ee810c3b:::benchmark_data_smithi052_101302_object45948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:41.887050+0000", 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0099256090000000002, 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010092459, 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:08:41.909 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 173877 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.887050+0000", 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.887050+0000", 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.910 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.887052+0000", 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.156e-06 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.887056+0000", 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4889999999999997e-06 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.887065+0000", 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9609999999999994e-06 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.911 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:41.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.887072+0000", 2024-09-13T13:08:41.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7240000000000003e-06 2024-09-13T13:08:41.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:41.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.887118+0000", 2024-09-13T13:08:41.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6100999999999998e-05 2024-09-13T13:08:41.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:41.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.887279+0000", 2024-09-13T13:08:41.913 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016161699999999999 2024-09-13T13:08:41.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:08:41.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.889042+0000", 2024-09-13T13:08:41.913 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001762516 2024-09-13T13:08:41.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:08:41.914 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.892740+0000", 2024-09-13T13:08:41.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036983319999999999 2024-09-13T13:08:41.914 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:41.914 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:41.914 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:41.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.914 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.25222.0:173874 8.0 e215/169 8:0cfa912f:::benchmark_data_smithi052_101302_object45945:head v 215'10711, mlcod=215'10706)", 2024-09-13T13:08:41.914 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:41.891985+0000", 2024-09-13T13:08:41.914 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0049907689999999999, 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0052241680000000004, 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.891985+0000", 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.891985+0000", 2024-09-13T13:08:41.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:41.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.916 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.916 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:41.916 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.891986+0000", 2024-09-13T13:08:41.916 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0400000000000005e-07 2024-09-13T13:08:41.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.916 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.916 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:41.916 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.891992+0000", 2024-09-13T13:08:41.916 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6890000000000001e-06 2024-09-13T13:08:41.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.916 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.916 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.891998+0000", 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2090000000000002e-06 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.892003+0000", 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7720000000000002e-06 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.896390+0000", 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.004387419 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:41.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:41.918 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:41.918 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:41.896410+0000", 2024-09-13T13:08:41.918 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9621000000000001e-05 2024-09-13T13:08:41.918 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:41.918 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:41.918 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:41.918 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:41.918 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:08:41.918 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 4 2024-09-13T13:08:41.918 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:08:41.944 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:41.971 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:41.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:41.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:42.073 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:42.076 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:08:42.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:42.102 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:42.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:42.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:42 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1611685655' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:42.149 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:42 smithi052 ceph-mon[36682]: from='client.35444 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:42.149 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1611685655' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:42.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:42 smithi052 ceph-mon[29471]: from='client.35444 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:42.203 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:42.245 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:42.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:42.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:42.347 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:42.378 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:42.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:42.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:42.480 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:42.514 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:42.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:42.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:42.569 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:08:42.570 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:08:42.570 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:08:42.570 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:08:42.570 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:08:42.570 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:08:42.570 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:08:42.570 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:08:42.571 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:08:42.571 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:08:42.571 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:08:42.571 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:08:42.571 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:08:42.571 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:08:42.571 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:08:42.571 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:08:42.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:42 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1611685655' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:42.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:42 smithi167 ceph-mon[33178]: from='client.35444 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:42.616 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:42.653 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:42.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:42.754 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:42.792 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:42.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:42.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:42.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:42.924 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:42.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:43.026 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:43.055 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:43.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:43.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:43.157 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:43.192 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:43.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:43.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:43 smithi052 ceph-mon[36682]: pgmap v361: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-13T13:08:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:43 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:43 smithi052 ceph-mon[36682]: from='client.35444 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:08:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:43 smithi052 ceph-mon[36682]: osdmap e216: 8 total, 8 up, 8 in 2024-09-13T13:08:43.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2615543825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:43.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:43 smithi052 ceph-mon[29471]: pgmap v361: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-13T13:08:43.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:43 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:43.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:43 smithi052 ceph-mon[29471]: from='client.35444 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:08:43.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:43 smithi052 ceph-mon[29471]: osdmap e216: 8 total, 8 up, 8 in 2024-09-13T13:08:43.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2615543825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:43.294 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:43.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:43 smithi167 ceph-mon[33178]: pgmap v361: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-13T13:08:43.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:43 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:43.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:43 smithi167 ceph-mon[33178]: from='client.35444 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:08:43.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:43 smithi167 ceph-mon[33178]: osdmap e216: 8 total, 8 up, 8 in 2024-09-13T13:08:43.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2615543825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:43.337 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:43.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:43.438 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:43.478 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:43.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:43.580 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:43.615 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:43.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:43.717 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:43.762 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:43.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:43.863 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:43.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T13:08:43.903 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:43.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:44.005 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:44.052 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:44.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:44.123 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:08:44.124 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T13:08:44.155 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:44.228 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:44.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:44.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:44.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:44 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:08:44.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:44 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:08:44.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:44 smithi167 ceph-mon[33178]: osdmap e217: 8 total, 8 up, 8 in 2024-09-13T13:08:44.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:08:43 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:08:44.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:44.391 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:44 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:08:44.391 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:44 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:08:44.391 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:44 smithi052 ceph-mon[29471]: osdmap e217: 8 total, 8 up, 8 in 2024-09-13T13:08:44.392 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:44.393 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:44 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:08:44.393 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:44 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:08:44.393 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:44 smithi052 ceph-mon[36682]: osdmap e217: 8 total, 8 up, 8 in 2024-09-13T13:08:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:44.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:44.495 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:44.510 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:44.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:44.550 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:44.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:44.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:44.685 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:44.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:44.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:44.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:44.820 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:44.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:44.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:44.922 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:44.954 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:44.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:44.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:45.056 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:45.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:45.090 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:45.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:45.133 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:45.194 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:08:45.194 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:08:45.194 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:45.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:45.245 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:45.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:45.276 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:45 smithi052 ceph-mon[36682]: pgmap v364: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-13T13:08:45.276 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:45 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:45.276 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:45 smithi052 ceph-mon[29471]: pgmap v364: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-13T13:08:45.277 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:45 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:45.346 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:45.375 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:45.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:45.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:45.411 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:08:45.477 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:45.527 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:45.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:45.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:45.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:45 smithi167 ceph-mon[33178]: pgmap v364: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-13T13:08:45.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:45 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:45.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:45.662 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:45.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:45.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:45.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:45.795 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:45.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:45.897 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:45.937 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:45.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:46.039 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:46.066 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:46.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:46.167 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:46.204 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:46.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:46.306 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:46.341 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:46.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:46.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:46.394 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:08:46.442 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:08:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:08:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:08:46.443 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:46.450 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T13:08:46.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:46.484 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:46.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:46.581 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:46.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:46.613 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:46.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:46.714 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:46.763 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:46.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:46.865 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:46.895 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:46.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:46.997 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:47.055 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:47.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:47.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:47.123 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T13:08:47.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:47.184 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:47.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:47.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:47.286 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:47.312 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:47.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:47.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:47.413 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:47.463 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:47.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:47.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:47.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:47 smithi052 ceph-mon[29471]: pgmap v365: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-13T13:08:47.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:47 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4042144597' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:08:47.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:47 smithi052 ceph-mon[29471]: from='client.35453 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:08:47.477 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:47 smithi052 ceph-mon[36682]: pgmap v365: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-13T13:08:47.477 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:47 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4042144597' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:08:47.477 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:47 smithi052 ceph-mon[36682]: from='client.35453 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:08:47.565 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:47.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:47 smithi167 ceph-mon[33178]: pgmap v365: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-13T13:08:47.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:47 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4042144597' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:08:47.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:47 smithi167 ceph-mon[33178]: from='client.35453 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:08:47.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:47.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:47.690 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:47.731 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:47.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:47.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:47.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:47.862 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:47.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:47.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:47.940 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T13:08:47.965 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:47.978 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:47.980 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:48.043 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:48.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:48.140 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:08:48.145 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:48.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:48.215 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:48.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:48.316 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:48.359 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:48.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:48.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:48.461 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:48.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:48 smithi052 ceph-mon[29471]: from='client.35453 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:08:48.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:48 smithi052 ceph-mon[29471]: osdmap e218: 8 total, 8 up, 8 in 2024-09-13T13:08:48.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:48 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2990075893' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:08:48.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:48 smithi052 ceph-mon[29471]: from='client.25459 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:08:48.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:48 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:08:48.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:48 smithi052 ceph-mon[36682]: from='client.35453 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:08:48.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:48 smithi052 ceph-mon[36682]: osdmap e218: 8 total, 8 up, 8 in 2024-09-13T13:08:48.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:48 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2990075893' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:08:48.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:48 smithi052 ceph-mon[36682]: from='client.25459 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:08:48.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:48 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:08:48.493 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:48.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:48.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:48.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:48 smithi167 ceph-mon[33178]: from='client.35453 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:08:48.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:48 smithi167 ceph-mon[33178]: osdmap e218: 8 total, 8 up, 8 in 2024-09-13T13:08:48.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:48 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2990075893' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:08:48.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:48 smithi167 ceph-mon[33178]: from='client.25459 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:08:48.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:48 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:08:48.595 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:48.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:48.632 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:48.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:48.733 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:48.757 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:48.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:48.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:48.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:48.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:48.890 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:48.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:48.992 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:49.034 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:49.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:49.135 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:49.169 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:49.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:49.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:49.271 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:49.300 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:49.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:49.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:49 smithi052 ceph-mon[29471]: pgmap v367: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 989 op/s 2024-09-13T13:08:49.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:49 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:49.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:49 smithi052 ceph-mon[29471]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:08:49.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:49 smithi052 ceph-mon[29471]: osdmap e219: 8 total, 8 up, 8 in 2024-09-13T13:08:49.343 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:49 smithi052 ceph-mon[36682]: pgmap v367: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 989 op/s 2024-09-13T13:08:49.343 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:49 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:49.343 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:49 smithi052 ceph-mon[36682]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:08:49.343 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:49 smithi052 ceph-mon[36682]: osdmap e219: 8 total, 8 up, 8 in 2024-09-13T13:08:49.401 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:49.429 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:49.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:49.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:49.562 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:49.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:49.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:49.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:49 smithi167 ceph-mon[33178]: pgmap v367: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 989 op/s 2024-09-13T13:08:49.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:49 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:08:49.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:49 smithi167 ceph-mon[33178]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:08:49.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:49 smithi167 ceph-mon[33178]: osdmap e219: 8 total, 8 up, 8 in 2024-09-13T13:08:49.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:49.701 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:49.701 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:08:49.701 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:08:49.718 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:49.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:49.820 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:49.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:49.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:49.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:49.994 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:49.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:50.097 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:50.129 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:50.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:50.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:50.263 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:50.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:50.365 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:50.391 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:50 smithi052 ceph-mon[36682]: 2.0 scrub starts 2024-09-13T13:08:50.391 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:50 smithi052 ceph-mon[36682]: 2.0 scrub ok 2024-09-13T13:08:50.391 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:50 smithi052 ceph-mon[36682]: 6.18 scrub starts 2024-09-13T13:08:50.391 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:50 smithi052 ceph-mon[36682]: 6.18 scrub ok 2024-09-13T13:08:50.391 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:50 smithi052 ceph-mon[29471]: 2.0 scrub starts 2024-09-13T13:08:50.392 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:50 smithi052 ceph-mon[29471]: 2.0 scrub ok 2024-09-13T13:08:50.392 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:50 smithi052 ceph-mon[29471]: 6.18 scrub starts 2024-09-13T13:08:50.392 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:50 smithi052 ceph-mon[29471]: 6.18 scrub ok 2024-09-13T13:08:50.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:50.418 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:50.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:50.519 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:50.560 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:50.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:50 smithi167 ceph-mon[33178]: 2.0 scrub starts 2024-09-13T13:08:50.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:50 smithi167 ceph-mon[33178]: 2.0 scrub ok 2024-09-13T13:08:50.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:50 smithi167 ceph-mon[33178]: 6.18 scrub starts 2024-09-13T13:08:50.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:50 smithi167 ceph-mon[33178]: 6.18 scrub ok 2024-09-13T13:08:50.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:50.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:50.677 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:50.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:50.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:50.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:50.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:50.849 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:50.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:50.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:50.876 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:08:50.876 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:08:50.914 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:08:50.914 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:08:50.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:50.957 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:08:50.957 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:08:50.985 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:50.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:50.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:50.996 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:08:50.997 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:08:51.033 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:08:51.033 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:08:51.065 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:08:51.066 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:08:51.087 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:51.100 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:08:51.101 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:08:51.106 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:08:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:51.139 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:51.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:51.142 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:08:51.142 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:08:51.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:51 smithi052 ceph-mon[36682]: 8.f scrub starts 2024-09-13T13:08:51.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:51 smithi052 ceph-mon[36682]: pgmap v369: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-13T13:08:51.214 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:51 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3605385197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:51.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:51 smithi052 ceph-mon[29471]: 8.f scrub starts 2024-09-13T13:08:51.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:51 smithi052 ceph-mon[29471]: pgmap v369: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-13T13:08:51.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:51 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3605385197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:51.239 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:51.246 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T13:08:51.265 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:51.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:51.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:51.367 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:51.427 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:51.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:51.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:51.528 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:51.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:51.557 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:51.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:51.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:51 smithi167 ceph-mon[33178]: 8.f scrub starts 2024-09-13T13:08:51.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:51 smithi167 ceph-mon[33178]: pgmap v369: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-13T13:08:51.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:51 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3605385197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:51.659 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:51.703 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:51.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:51.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:51.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:51.842 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:51.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:51.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:51.944 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:51.973 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:51.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:51.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:52.075 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:52.106 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:52.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:52.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:52.208 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:52.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:52.234 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:52.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:52.243 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:52.334 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:52.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:52.360 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:52.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:52.422 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:52 smithi052 ceph-mon[29471]: 8.b scrub starts 2024-09-13T13:08:52.422 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:52 smithi052 ceph-mon[29471]: 3.16 scrub starts 2024-09-13T13:08:52.422 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:52 smithi052 ceph-mon[29471]: 3.16 scrub ok 2024-09-13T13:08:52.422 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:52 smithi052 ceph-mon[29471]: 8.f scrub ok 2024-09-13T13:08:52.422 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:52 smithi052 ceph-mon[29471]: 3.9 scrub starts 2024-09-13T13:08:52.422 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:52 smithi052 ceph-mon[29471]: 3.9 scrub ok 2024-09-13T13:08:52.422 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:52 smithi052 ceph-mon[36682]: 8.b scrub starts 2024-09-13T13:08:52.423 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:52 smithi052 ceph-mon[36682]: 3.16 scrub starts 2024-09-13T13:08:52.423 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:52 smithi052 ceph-mon[36682]: 3.16 scrub ok 2024-09-13T13:08:52.423 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:52 smithi052 ceph-mon[36682]: 8.f scrub ok 2024-09-13T13:08:52.423 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:52 smithi052 ceph-mon[36682]: 3.9 scrub starts 2024-09-13T13:08:52.423 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:52 smithi052 ceph-mon[36682]: 3.9 scrub ok 2024-09-13T13:08:52.462 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:52.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:52.515 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:52.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:52.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:52 smithi167 ceph-mon[33178]: 8.b scrub starts 2024-09-13T13:08:52.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:52 smithi167 ceph-mon[33178]: 3.16 scrub starts 2024-09-13T13:08:52.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:52 smithi167 ceph-mon[33178]: 3.16 scrub ok 2024-09-13T13:08:52.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:52 smithi167 ceph-mon[33178]: 8.f scrub ok 2024-09-13T13:08:52.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:52 smithi167 ceph-mon[33178]: 3.9 scrub starts 2024-09-13T13:08:52.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:52 smithi167 ceph-mon[33178]: 3.9 scrub ok 2024-09-13T13:08:52.615 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:52.656 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:52.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:52.757 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:52.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:52.786 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:52.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:52.887 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:52.913 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:52.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:53.015 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:53.048 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:53.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:53.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:53.179 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:53.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:53.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:53.190 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T13:08:53.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:53 smithi052 ceph-mon[29471]: pgmap v370: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 922 op/s 2024-09-13T13:08:53.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:53 smithi052 ceph-mon[29471]: 6.0 scrub starts 2024-09-13T13:08:53.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:53 smithi052 ceph-mon[29471]: 6.0 scrub ok 2024-09-13T13:08:53.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/436025293' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:08:53.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:53 smithi052 ceph-mon[29471]: from='client.35468 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:08:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:53 smithi052 ceph-mon[36682]: pgmap v370: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 922 op/s 2024-09-13T13:08:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:53 smithi052 ceph-mon[36682]: 6.0 scrub starts 2024-09-13T13:08:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:53 smithi052 ceph-mon[36682]: 6.0 scrub ok 2024-09-13T13:08:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:53 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/436025293' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:08:53.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:53 smithi052 ceph-mon[36682]: from='client.35468 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:08:53.281 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:53.326 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:53.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:53.427 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:53.467 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:53.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:53.569 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:53 smithi167 ceph-mon[33178]: pgmap v370: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 922 op/s 2024-09-13T13:08:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:53 smithi167 ceph-mon[33178]: 6.0 scrub starts 2024-09-13T13:08:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:53 smithi167 ceph-mon[33178]: 6.0 scrub ok 2024-09-13T13:08:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:53 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/436025293' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:08:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:53 smithi167 ceph-mon[33178]: from='client.35468 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:08:53.578 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:08:53.578 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:08:53.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.25222.0:186684 8.1 e220/169 8:8409ecb9:::benchmark_data_smithi052_101302_object58755:head v 220'11804, mlcod=220'11802)", 2024-09-13T13:08:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:53.413691+0000", 2024-09-13T13:08:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0034005160000000001, 2024-09-13T13:08:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0034278999999999998, 2024-09-13T13:08:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:08:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.413691+0000", 2024-09-13T13:08:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.413691+0000", 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.413692+0000", 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.48e-06 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.413700+0000", 2024-09-13T13:08:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5120000000000002e-06 2024-09-13T13:08:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.413710+0000", 2024-09-13T13:08:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0372e-05 2024-09-13T13:08:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.413718+0000", 2024-09-13T13:08:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9999999999999996e-06 2024-09-13T13:08:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.413821+0000", 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103087 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.413840+0000", 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8512000000000002e-05 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.25222.0:186681 8.a e220/169 8:59fdcf1f:::benchmark_data_smithi052_101302_object58752:head v 220'11569, mlcod=220'11567)", 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:53.416824+0000", 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.00026709900000000003, 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035463399999999999, 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.416824+0000", 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.416824+0000", 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.416825+0000", 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.248e-06 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.416832+0000", 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1539999999999996e-06 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.416847+0000", 2024-09-13T13:08:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4397e-05 2024-09-13T13:08:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.416853+0000", 2024-09-13T13:08:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5259999999999999e-06 2024-09-13T13:08:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:08:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.417048+0000", 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019471000000000001 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.417112+0000", 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4017999999999996e-05 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.25222.0:186683 8.e e220/169)", 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:53.417093+0000", 2024-09-13T13:08:53.586 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 4294967295.9999976, 2024-09-13T13:08:53.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013297499999999999, 2024-09-13T13:08:53.587 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:08:53.587 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:08:53.587 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:08:53.587 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:08:53.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:08:53.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.417093+0000", 2024-09-13T13:08:53.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:53.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:08:53.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.417093+0000", 2024-09-13T13:08:53.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:08:53.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:08:53.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.417095+0000", 2024-09-13T13:08:53.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.333e-06 2024-09-13T13:08:53.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:08:53.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.417102+0000", 2024-09-13T13:08:53.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1430000000000001e-06 2024-09-13T13:08:53.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:08:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.417115+0000", 2024-09-13T13:08:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2658e-05 2024-09-13T13:08:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:08:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:08:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:08:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:53.417122+0000", 2024-09-13T13:08:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5689999999999999e-06 2024-09-13T13:08:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:08:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:08:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:08:53.590 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 3 2024-09-13T13:08:53.590 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:08:53.607 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:53.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:53.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:53.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:53.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:53.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:53.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:53.870 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:53.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:53.933 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:08:53.972 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:54.006 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:54.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:54.108 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:54.142 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:54.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:54.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:54.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:08:53 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:08:54.202 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:08:54.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[36682]: 8.b scrub ok 2024-09-13T13:08:54.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[36682]: from='client.35468 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:08:54.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[36682]: osdmap e220: 8 total, 8 up, 8 in 2024-09-13T13:08:54.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[36682]: 5.10 scrub starts 2024-09-13T13:08:54.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[36682]: 5.10 scrub ok 2024-09-13T13:08:54.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[36682]: 5.e scrub starts 2024-09-13T13:08:54.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[36682]: 5.e scrub ok 2024-09-13T13:08:54.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/323555303' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:54.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[36682]: from='client.25468 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:54.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[29471]: 8.b scrub ok 2024-09-13T13:08:54.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[29471]: from='client.35468 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:08:54.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[29471]: osdmap e220: 8 total, 8 up, 8 in 2024-09-13T13:08:54.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[29471]: 5.10 scrub starts 2024-09-13T13:08:54.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[29471]: 5.10 scrub ok 2024-09-13T13:08:54.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[29471]: 5.e scrub starts 2024-09-13T13:08:54.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[29471]: 5.e scrub ok 2024-09-13T13:08:54.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/323555303' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:54.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:54 smithi052 ceph-mon[29471]: from='client.25468 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:54.244 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:54.288 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:54.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:54.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:54.390 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:54.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:54.423 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:54.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:54.524 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:54.555 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:54.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:54.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:54 smithi167 ceph-mon[33178]: 8.b scrub ok 2024-09-13T13:08:54.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:54 smithi167 ceph-mon[33178]: from='client.35468 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:08:54.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:54 smithi167 ceph-mon[33178]: osdmap e220: 8 total, 8 up, 8 in 2024-09-13T13:08:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:54 smithi167 ceph-mon[33178]: 5.10 scrub starts 2024-09-13T13:08:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:54 smithi167 ceph-mon[33178]: 5.10 scrub ok 2024-09-13T13:08:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:54 smithi167 ceph-mon[33178]: 5.e scrub starts 2024-09-13T13:08:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:54 smithi167 ceph-mon[33178]: 5.e scrub ok 2024-09-13T13:08:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:54 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/323555303' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:54.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:54 smithi167 ceph-mon[33178]: from='client.25468 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:08:54.657 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:54.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:54.701 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:54.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:54.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:54.841 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:54.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:54.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:54.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:54.975 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:54.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:55.076 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:55.115 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:55.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:55.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:55.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:55.245 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:55.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:55.346 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:55.374 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:55.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:55.476 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[29471]: 4.19 scrub starts 2024-09-13T13:08:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[29471]: 4.19 scrub ok 2024-09-13T13:08:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[29471]: 5.c scrub starts 2024-09-13T13:08:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[29471]: 5.c scrub ok 2024-09-13T13:08:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[29471]: pgmap v372: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-13T13:08:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:08:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:08:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[29471]: from='client.25468 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:08:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[29471]: osdmap e221: 8 total, 8 up, 8 in 2024-09-13T13:08:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[29471]: 6.5 scrub starts 2024-09-13T13:08:55.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[29471]: 6.5 scrub ok 2024-09-13T13:08:55.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[36682]: 4.19 scrub starts 2024-09-13T13:08:55.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[36682]: 4.19 scrub ok 2024-09-13T13:08:55.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[36682]: 5.c scrub starts 2024-09-13T13:08:55.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[36682]: 5.c scrub ok 2024-09-13T13:08:55.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[36682]: pgmap v372: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-13T13:08:55.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:55.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:08:55.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:08:55.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[36682]: from='client.25468 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:08:55.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[36682]: osdmap e221: 8 total, 8 up, 8 in 2024-09-13T13:08:55.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[36682]: 6.5 scrub starts 2024-09-13T13:08:55.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:55 smithi052 ceph-mon[36682]: 6.5 scrub ok 2024-09-13T13:08:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:55.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:55.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:55 smithi167 ceph-mon[33178]: 4.19 scrub starts 2024-09-13T13:08:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:55 smithi167 ceph-mon[33178]: 4.19 scrub ok 2024-09-13T13:08:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:55 smithi167 ceph-mon[33178]: 5.c scrub starts 2024-09-13T13:08:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:55 smithi167 ceph-mon[33178]: 5.c scrub ok 2024-09-13T13:08:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:55 smithi167 ceph-mon[33178]: pgmap v372: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-13T13:08:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:08:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:55 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:08:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:55 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:08:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:55 smithi167 ceph-mon[33178]: from='client.25468 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:08:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:55 smithi167 ceph-mon[33178]: osdmap e221: 8 total, 8 up, 8 in 2024-09-13T13:08:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:55 smithi167 ceph-mon[33178]: 6.5 scrub starts 2024-09-13T13:08:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:55 smithi167 ceph-mon[33178]: 6.5 scrub ok 2024-09-13T13:08:55.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:55.651 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:55.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:55.753 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:55.782 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:55.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:55.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:55.884 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:55.922 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:55.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:56.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:56.052 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:56.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:56.154 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:56.188 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:56.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:56.290 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:56.311 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:56.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:56.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:56.413 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:56.451 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:56.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:56.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:56 smithi052 ceph-mon[29471]: 5.18 scrub starts 2024-09-13T13:08:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:56 smithi052 ceph-mon[29471]: 5.18 scrub ok 2024-09-13T13:08:56.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:08:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:08:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:08:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:56 smithi052 ceph-mon[36682]: 5.18 scrub starts 2024-09-13T13:08:56.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:56 smithi052 ceph-mon[36682]: 5.18 scrub ok 2024-09-13T13:08:56.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:56.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:56 smithi167 ceph-mon[33178]: 5.18 scrub starts 2024-09-13T13:08:56.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:56 smithi167 ceph-mon[33178]: 5.18 scrub ok 2024-09-13T13:08:56.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:56.589 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:56.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:56.691 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:56.728 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:56.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:56.829 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:56.862 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:56.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:56.881 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T13:08:56.964 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:57.009 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:57.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:57.111 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:57.162 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:57.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:57.264 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:57.293 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:57.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:57.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:57.395 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:57.439 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:57.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:57.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:57.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[36682]: 6.1b scrub starts 2024-09-13T13:08:57.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[36682]: 6.1b scrub ok 2024-09-13T13:08:57.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[36682]: 6.4 scrub starts 2024-09-13T13:08:57.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[36682]: 6.4 scrub ok 2024-09-13T13:08:57.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[36682]: pgmap v374: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 671 B/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T13:08:57.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[36682]: 6.10 scrub starts 2024-09-13T13:08:57.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[36682]: 6.10 scrub ok 2024-09-13T13:08:57.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[29471]: 6.1b scrub starts 2024-09-13T13:08:57.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[29471]: 6.1b scrub ok 2024-09-13T13:08:57.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[29471]: 6.4 scrub starts 2024-09-13T13:08:57.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[29471]: 6.4 scrub ok 2024-09-13T13:08:57.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[29471]: pgmap v374: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 671 B/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T13:08:57.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[29471]: 6.10 scrub starts 2024-09-13T13:08:57.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:57 smithi052 ceph-mon[29471]: 6.10 scrub ok 2024-09-13T13:08:57.541 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:08:57.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:57.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:57.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:08:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:57 smithi167 ceph-mon[33178]: 6.1b scrub starts 2024-09-13T13:08:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:57 smithi167 ceph-mon[33178]: 6.1b scrub ok 2024-09-13T13:08:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:57 smithi167 ceph-mon[33178]: 6.4 scrub starts 2024-09-13T13:08:57.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:57 smithi167 ceph-mon[33178]: 6.4 scrub ok 2024-09-13T13:08:57.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:57 smithi167 ceph-mon[33178]: pgmap v374: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 671 B/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T13:08:57.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:57 smithi167 ceph-mon[33178]: 6.10 scrub starts 2024-09-13T13:08:57.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:57 smithi167 ceph-mon[33178]: 6.10 scrub ok 2024-09-13T13:08:57.672 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:57.714 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:57.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:57.815 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:57.849 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:57.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:57.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:57.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:08:57.980 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:57.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:08:58.082 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:58.088 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T13:08:58.106 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:58.132 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:58.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:58.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:58.178 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:58.234 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:58.281 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:58.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:58.315 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:58 smithi052 ceph-mon[36682]: 3.0 scrub starts 2024-09-13T13:08:58.315 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:58 smithi052 ceph-mon[36682]: 3.0 scrub ok 2024-09-13T13:08:58.316 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:58 smithi052 ceph-mon[29471]: 3.0 scrub starts 2024-09-13T13:08:58.316 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:58 smithi052 ceph-mon[29471]: 3.0 scrub ok 2024-09-13T13:08:58.382 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:58.440 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:58.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:58.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:58.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:58 smithi167 ceph-mon[33178]: 3.0 scrub starts 2024-09-13T13:08:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:58 smithi167 ceph-mon[33178]: 3.0 scrub ok 2024-09-13T13:08:58.575 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:58.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:58.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:58.677 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:58.707 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:58.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:58.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:58.809 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:58.842 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:58.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:58.943 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:58.982 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:58.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:58.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:59.061 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:08:59.061 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:08:59.084 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:08:59.097 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:08:59.097 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:08:59.129 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:59.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:59.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:08:59.133 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:08:59.133 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:08:59.177 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:08:59.177 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:08:59.210 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:08:59.210 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:08:59.231 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:08:59.245 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:08:59.246 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:08:59.256 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:59.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:08:59.269 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:08:59.269 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:08:59.282 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:08:59.282 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:08:59.290 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:08:59.318 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:08:59.319 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:08:59.358 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:08:59.399 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:59.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:59.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:08:59.484 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:59 smithi052 ceph-mon[36682]: pgmap v375: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 773 op/s 2024-09-13T13:08:59.484 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:08:59 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3829124201' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:59.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:59 smithi052 ceph-mon[29471]: pgmap v375: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 773 op/s 2024-09-13T13:08:59.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:08:59 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3829124201' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:59.502 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:08:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:59.534 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:08:59.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:08:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:59 smithi167 ceph-mon[33178]: pgmap v375: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 773 op/s 2024-09-13T13:08:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:08:59 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3829124201' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:08:59.596 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:08:59.635 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:08:59.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:59.721 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:59.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:08:59.822 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:08:59.861 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:08:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:08:59.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:08:59.963 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:00.003 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:00.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:00.104 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:00.148 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:00.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:00.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:00.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:00.278 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:00.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:00.380 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:00.412 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:00.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:00.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:00.514 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:00.548 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:00.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:00.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:00 smithi167 ceph-mon[33178]: 6.1c scrub starts 2024-09-13T13:09:00.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:00 smithi167 ceph-mon[33178]: 6.1c scrub ok 2024-09-13T13:09:00.650 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:00.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:00.685 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:00.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:00.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:00 smithi052 ceph-mon[36682]: 6.1c scrub starts 2024-09-13T13:09:00.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:00 smithi052 ceph-mon[36682]: 6.1c scrub ok 2024-09-13T13:09:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:00 smithi052 ceph-mon[29471]: 6.1c scrub starts 2024-09-13T13:09:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:00 smithi052 ceph-mon[29471]: 6.1c scrub ok 2024-09-13T13:09:00.786 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:00.823 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:00.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:00.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:00.957 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:00.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:00.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:01.059 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:01.087 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:01.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:01.189 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:01.215 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:01.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:01.303 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[29471]: 6.11 scrub starts 2024-09-13T13:09:01.303 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[29471]: 6.11 scrub ok 2024-09-13T13:09:01.304 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[29471]: 3.1d scrub starts 2024-09-13T13:09:01.304 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[29471]: 3.1d scrub ok 2024-09-13T13:09:01.304 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[29471]: pgmap v376: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-13T13:09:01.304 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[29471]: 4.1a scrub starts 2024-09-13T13:09:01.304 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[29471]: 4.1a scrub ok 2024-09-13T13:09:01.304 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[36682]: 6.11 scrub starts 2024-09-13T13:09:01.304 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[36682]: 6.11 scrub ok 2024-09-13T13:09:01.304 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[36682]: 3.1d scrub starts 2024-09-13T13:09:01.304 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[36682]: 3.1d scrub ok 2024-09-13T13:09:01.304 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[36682]: pgmap v376: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-13T13:09:01.305 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[36682]: 4.1a scrub starts 2024-09-13T13:09:01.305 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:01 smithi052 ceph-mon[36682]: 4.1a scrub ok 2024-09-13T13:09:01.317 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:01.369 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:01.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:01.471 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:01.512 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:01.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:01 smithi167 ceph-mon[33178]: 6.11 scrub starts 2024-09-13T13:09:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:01 smithi167 ceph-mon[33178]: 6.11 scrub ok 2024-09-13T13:09:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:01 smithi167 ceph-mon[33178]: 3.1d scrub starts 2024-09-13T13:09:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:01 smithi167 ceph-mon[33178]: 3.1d scrub ok 2024-09-13T13:09:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:01 smithi167 ceph-mon[33178]: pgmap v376: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-13T13:09:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:01 smithi167 ceph-mon[33178]: 4.1a scrub starts 2024-09-13T13:09:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:01 smithi167 ceph-mon[33178]: 4.1a scrub ok 2024-09-13T13:09:01.614 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:01.654 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:01.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:01.756 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:01.794 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:01.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:01.895 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:01.933 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:01.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:01.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:02.035 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:02.072 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:02.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:02.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:02.214 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:02.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:02.250 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T13:09:02.315 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:02.353 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:02.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:02.455 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:02.487 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:02.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:02.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:02.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:02 smithi167 ceph-mon[33178]: 6.6 scrub starts 2024-09-13T13:09:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:02 smithi167 ceph-mon[33178]: 6.6 scrub ok 2024-09-13T13:09:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:02 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3722292257' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:09:02.589 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:02.629 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:02.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:02.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:02.731 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:02.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:02 smithi052 ceph-mon[36682]: 6.6 scrub starts 2024-09-13T13:09:02.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:02 smithi052 ceph-mon[36682]: 6.6 scrub ok 2024-09-13T13:09:02.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:02 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3722292257' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:09:02.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:02 smithi052 ceph-mon[29471]: 6.6 scrub starts 2024-09-13T13:09:02.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:02 smithi052 ceph-mon[29471]: 6.6 scrub ok 2024-09-13T13:09:02.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:02 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3722292257' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:09:02.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:02.767 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:02.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:02.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:02.903 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:02.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:02.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:03.005 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:03.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:03.042 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:03.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:03.075 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:09:03.143 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:03.164 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:03.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:03.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:03.293 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:03.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:03.395 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:03.421 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:03.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:03.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:03 smithi052 ceph-mon[36682]: pgmap v377: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 596 B/s rd, 0 B/s wr, 1.19k op/s 2024-09-13T13:09:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:03 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3722292257' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:09:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:03 smithi052 ceph-mon[36682]: osdmap e222: 8 total, 8 up, 8 in 2024-09-13T13:09:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:03 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:09:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:03 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:09:03.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:03 smithi052 ceph-mon[29471]: pgmap v377: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 596 B/s rd, 0 B/s wr, 1.19k op/s 2024-09-13T13:09:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:03 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3722292257' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:09:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:03 smithi052 ceph-mon[29471]: osdmap e222: 8 total, 8 up, 8 in 2024-09-13T13:09:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:03 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:09:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:03 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:09:03.522 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:03.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:03.564 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:03.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:03 smithi167 ceph-mon[33178]: pgmap v377: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 596 B/s rd, 0 B/s wr, 1.19k op/s 2024-09-13T13:09:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:03 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3722292257' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:09:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:03 smithi167 ceph-mon[33178]: osdmap e222: 8 total, 8 up, 8 in 2024-09-13T13:09:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:03 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:09:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:03 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:09:03.665 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:03.703 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:03.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:03.804 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:03.826 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:03.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:03.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:03.928 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:03.961 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:03.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:03.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:04.063 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:04.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:04.101 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:04.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:04.202 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:04.242 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:04.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:04.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:09:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:09:04.344 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:04.366 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:04.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:04.468 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:04.511 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:04.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:04.612 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:04.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:04.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:04.735 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:04.775 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:04.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:04.877 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:04.917 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:04.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:05.018 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:05.056 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:05.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:05.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:05.196 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:05.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:05.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:05.297 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:05.315 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T13:09:05.341 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:05.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:05.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:05.444 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:05.472 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:05.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:05 smithi167 ceph-mon[33178]: pgmap v379: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-09-13T13:09:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:05 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:09:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:05 smithi167 ceph-mon[33178]: 4.14 scrub starts 2024-09-13T13:09:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:05 smithi167 ceph-mon[33178]: 4.14 scrub ok 2024-09-13T13:09:05.574 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:05.640 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:05.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:05.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:05.670 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:05.682 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:05.742 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:05.743 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:05 smithi052 ceph-mon[36682]: pgmap v379: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-09-13T13:09:05.743 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:05.743 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:05 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:09:05.743 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:05 smithi052 ceph-mon[36682]: 4.14 scrub starts 2024-09-13T13:09:05.743 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:05 smithi052 ceph-mon[36682]: 4.14 scrub ok 2024-09-13T13:09:05.744 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:05 smithi052 ceph-mon[29471]: pgmap v379: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-09-13T13:09:05.744 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:05.744 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:05 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:09:05.744 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:05 smithi052 ceph-mon[29471]: 4.14 scrub starts 2024-09-13T13:09:05.744 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:05 smithi052 ceph-mon[29471]: 4.14 scrub ok 2024-09-13T13:09:05.809 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:05.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:05.911 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:05.956 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:05.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:06.058 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:06.101 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:06.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:06.203 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:06.226 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T13:09:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:06.249 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:06.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:06.350 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:06.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:06.379 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:06.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:06.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:06.487 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:09:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:09:06.487 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[36682]: 3.a scrub starts 2024-09-13T13:09:06.487 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[36682]: 3.a scrub ok 2024-09-13T13:09:06.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[36682]: 4.3 scrub starts 2024-09-13T13:09:06.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[36682]: 4.3 scrub ok 2024-09-13T13:09:06.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[36682]: 4.c scrub starts 2024-09-13T13:09:06.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[36682]: 4.c scrub ok 2024-09-13T13:09:06.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:09:06.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:09:06.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:09:06.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[29471]: 3.a scrub starts 2024-09-13T13:09:06.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[29471]: 3.a scrub ok 2024-09-13T13:09:06.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[29471]: 4.3 scrub starts 2024-09-13T13:09:06.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[29471]: 4.3 scrub ok 2024-09-13T13:09:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[29471]: 4.c scrub starts 2024-09-13T13:09:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[29471]: 4.c scrub ok 2024-09-13T13:09:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:09:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:09:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:06 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:09:06.505 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:06.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:06.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:06 smithi167 ceph-mon[33178]: 3.a scrub starts 2024-09-13T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:06 smithi167 ceph-mon[33178]: 3.a scrub ok 2024-09-13T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:06 smithi167 ceph-mon[33178]: 4.3 scrub starts 2024-09-13T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:06 smithi167 ceph-mon[33178]: 4.3 scrub ok 2024-09-13T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:06 smithi167 ceph-mon[33178]: 4.c scrub starts 2024-09-13T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:06 smithi167 ceph-mon[33178]: 4.c scrub ok 2024-09-13T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:06 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:06 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:09:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:06 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:09:06.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:06.650 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:06.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:06.751 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:06.793 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:06.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:06.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:06.895 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:06.924 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:06.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:07.026 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:07.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:07.074 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:07.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:07.175 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:07.222 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:07.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:07.230 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:09:07.230 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:09:07.230 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:09:07.231 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:09:07.272 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:09:07.272 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:09:07.318 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:09:07.319 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:09:07.324 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:07.332 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:07.347 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:07.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:07.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:07.353 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:07.355 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:09:07.356 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:09:07.356 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:07 smithi052 ceph-mon[36682]: pgmap v380: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-13T13:09:07.356 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:07 smithi052 ceph-mon[36682]: 5.12 scrub starts 2024-09-13T13:09:07.356 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:07 smithi052 ceph-mon[36682]: 5.12 scrub ok 2024-09-13T13:09:07.356 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:07 smithi052 ceph-mon[36682]: 4.18 scrub starts 2024-09-13T13:09:07.356 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:07 smithi052 ceph-mon[36682]: 4.18 scrub ok 2024-09-13T13:09:07.356 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:07 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/75351232' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:07.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:07 smithi052 ceph-mon[29471]: pgmap v380: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-13T13:09:07.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:07 smithi052 ceph-mon[29471]: 5.12 scrub starts 2024-09-13T13:09:07.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:07 smithi052 ceph-mon[29471]: 5.12 scrub ok 2024-09-13T13:09:07.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:07 smithi052 ceph-mon[29471]: 4.18 scrub starts 2024-09-13T13:09:07.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:07 smithi052 ceph-mon[29471]: 4.18 scrub ok 2024-09-13T13:09:07.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:07 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/75351232' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:07.398 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:09:07.398 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:09:07.447 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:09:07.447 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:09:07.448 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:07.491 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:09:07.491 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:09:07.496 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:07.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:07.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:07 smithi167 ceph-mon[33178]: pgmap v380: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-13T13:09:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:07 smithi167 ceph-mon[33178]: 5.12 scrub starts 2024-09-13T13:09:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:07 smithi167 ceph-mon[33178]: 5.12 scrub ok 2024-09-13T13:09:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:07 smithi167 ceph-mon[33178]: 4.18 scrub starts 2024-09-13T13:09:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:07 smithi167 ceph-mon[33178]: 4.18 scrub ok 2024-09-13T13:09:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:07 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/75351232' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:07.597 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:07.641 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:07.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:07.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:07.773 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:07.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:07.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:07.878 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:07.913 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:07.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:07.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:08.015 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:08.055 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:08.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:08.156 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:08.187 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:08.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:08.289 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:08.290 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:09:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:08.334 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:08.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:08.436 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:08.475 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:08.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:08.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:08.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:08 smithi167 ceph-mon[33178]: 3.c scrub starts 2024-09-13T13:09:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:08 smithi167 ceph-mon[33178]: 3.c scrub ok 2024-09-13T13:09:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:08 smithi167 ceph-mon[33178]: 3.15 scrub starts 2024-09-13T13:09:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:08 smithi167 ceph-mon[33178]: 3.15 scrub ok 2024-09-13T13:09:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:08 smithi167 ceph-mon[33178]: 6.14 scrub starts 2024-09-13T13:09:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:08 smithi167 ceph-mon[33178]: 6.14 scrub ok 2024-09-13T13:09:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2099460108' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:09:08.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:08.611 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:08.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:08.712 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:08.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[36682]: 3.c scrub starts 2024-09-13T13:09:08.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[36682]: 3.c scrub ok 2024-09-13T13:09:08.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[36682]: 3.15 scrub starts 2024-09-13T13:09:08.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[36682]: 3.15 scrub ok 2024-09-13T13:09:08.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[36682]: 6.14 scrub starts 2024-09-13T13:09:08.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[36682]: 6.14 scrub ok 2024-09-13T13:09:08.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2099460108' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:09:08.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[29471]: 3.c scrub starts 2024-09-13T13:09:08.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[29471]: 3.c scrub ok 2024-09-13T13:09:08.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[29471]: 3.15 scrub starts 2024-09-13T13:09:08.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[29471]: 3.15 scrub ok 2024-09-13T13:09:08.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[29471]: 6.14 scrub starts 2024-09-13T13:09:08.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[29471]: 6.14 scrub ok 2024-09-13T13:09:08.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2099460108' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:09:08.745 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:08.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:08.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:08.847 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:08.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:08.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:08.986 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:09.016 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:09.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:09.121 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:09.153 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:09.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:09.254 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:09.297 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:09.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:09.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:09.399 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:09.438 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:09.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:09.540 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:09.569 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:09.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:09 smithi167 ceph-mon[33178]: pgmap v381: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T13:09:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:09 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:09 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2099460108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:09:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:09 smithi167 ceph-mon[33178]: osdmap e223: 8 total, 8 up, 8 in 2024-09-13T13:09:09.609 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:09 smithi052 ceph-mon[29471]: pgmap v381: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T13:09:09.609 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:09 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:09.609 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:09 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2099460108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:09:09.609 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:09 smithi052 ceph-mon[29471]: osdmap e223: 8 total, 8 up, 8 in 2024-09-13T13:09:09.609 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:09 smithi052 ceph-mon[36682]: pgmap v381: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s 2024-09-13T13:09:09.609 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:09 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:09.609 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:09 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2099460108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:09:09.609 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:09 smithi052 ceph-mon[36682]: osdmap e223: 8 total, 8 up, 8 in 2024-09-13T13:09:09.648 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:09:09.648 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:09:09.648 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:09:09.648 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:09:09.648 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.649 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:95502 7.e 7:73e7d3a1:::benchmark_data_smithi052_71865_object95501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:09:09.649 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.170950+0000", 2024-09-13T13:09:09.649 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 343.43370556000002, 2024-09-13T13:09:09.649 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10520948400000001, 2024-09-13T13:09:09.649 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.649 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.649 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.649 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.649 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.649 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.649 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95502 2024-09-13T13:09:09.649 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.650 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.650 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.650 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.650 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170950+0000", 2024-09-13T13:09:09.650 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.650 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.650 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.650 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170950+0000", 2024-09-13T13:09:09.650 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.650 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.650 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.651 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170952+0000", 2024-09-13T13:09:09.651 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.872e-06 2024-09-13T13:09:09.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.651 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.651 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.651 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170974+0000", 2024-09-13T13:09:09.651 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:09:09.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.651 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.651 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.651 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170982+0000", 2024-09-13T13:09:09.651 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6349999999999995e-06 2024-09-13T13:09:09.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.652 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.652 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.652 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170989+0000", 2024-09-13T13:09:09.652 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1619999999999996e-06 2024-09-13T13:09:09.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.652 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.652 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.652 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.171116+0000", 2024-09-13T13:09:09.652 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000127293 2024-09-13T13:09:09.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.652 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.652 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.653 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.171296+0000", 2024-09-13T13:09:09.653 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000180278 2024-09-13T13:09:09.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.653 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.653 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.171419+0000", 2024-09-13T13:09:09.653 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012266299999999999 2024-09-13T13:09:09.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.653 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.653 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.175039+0000", 2024-09-13T13:09:09.653 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036198469999999998 2024-09-13T13:09:09.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.654 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.654 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.175665+0000", 2024-09-13T13:09:09.654 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00062585400000000002 2024-09-13T13:09:09.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.654 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.654 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276069+0000", 2024-09-13T13:09:09.654 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10040423499999999 2024-09-13T13:09:09.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.654 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.654 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276119+0000", 2024-09-13T13:09:09.655 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0290999999999997e-05 2024-09-13T13:09:09.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.655 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.655 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.655 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276160+0000", 2024-09-13T13:09:09.655 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0516999999999998e-05 2024-09-13T13:09:09.655 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.655 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.655 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.655 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.655 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:95513 7.e 7:760c2885:::benchmark_data_smithi052_71865_object95512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:09:09.655 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.177025+0000", 2024-09-13T13:09:09.656 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 343.42763042899998, 2024-09-13T13:09:09.656 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.099458682000000007, 2024-09-13T13:09:09.656 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.656 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.656 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.656 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.656 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.656 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.656 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95513 2024-09-13T13:09:09.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.656 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.656 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.656 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.657 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177025+0000", 2024-09-13T13:09:09.657 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.657 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.657 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177025+0000", 2024-09-13T13:09:09.657 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.657 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.657 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177027+0000", 2024-09-13T13:09:09.657 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8109999999999999e-06 2024-09-13T13:09:09.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177059+0000", 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2125000000000002e-05 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177078+0000", 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9239999999999999e-05 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177085+0000", 2024-09-13T13:09:09.658 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9460000000000001e-06 2024-09-13T13:09:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177194+0000", 2024-09-13T13:09:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010910500000000001 2024-09-13T13:09:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177399+0000", 2024-09-13T13:09:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000205019 2024-09-13T13:09:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.660 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177525+0000", 2024-09-13T13:09:09.660 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000125969 2024-09-13T13:09:09.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.660 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.660 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276198+0000", 2024-09-13T13:09:09.660 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098672383000000002 2024-09-13T13:09:09.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.660 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.660 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276236+0000", 2024-09-13T13:09:09.660 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7988000000000003e-05 2024-09-13T13:09:09.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276248+0000", 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1858000000000001e-05 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276284+0000", 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6350000000000003e-05 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276484+0000", 2024-09-13T13:09:09.661 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019988800000000001 2024-09-13T13:09:09.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.662 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.662 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.662 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95494 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:09:09.662 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.177847+0000", 2024-09-13T13:09:09.662 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 343.426808295, 2024-09-13T13:09:09.662 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098173131999999996, 2024-09-13T13:09:09.662 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.662 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.662 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:09:09.662 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.663 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.663 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.663 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177847+0000", 2024-09-13T13:09:09.663 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.663 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.663 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.663 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.663 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177847+0000", 2024-09-13T13:09:09.663 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.663 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.663 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.663 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.663 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177849+0000", 2024-09-13T13:09:09.664 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3179999999999999e-06 2024-09-13T13:09:09.664 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.664 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.664 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.664 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177853+0000", 2024-09-13T13:09:09.664 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1799999999999998e-06 2024-09-13T13:09:09.664 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.664 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.664 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.664 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177862+0000", 2024-09-13T13:09:09.664 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1390000000000001e-06 2024-09-13T13:09:09.664 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.664 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177868+0000", 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4119999999999998e-06 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180221+0000", 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023525130000000001 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180227+0000", 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4930000000000003e-06 2024-09-13T13:09:09.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276020+0000", 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095793077000000004 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95502 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.180124+0000", 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 343.42453160700001, 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.09604087, 2024-09-13T13:09:09.666 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180124+0000", 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180124+0000", 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.667 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.668 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.668 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180125+0000", 2024-09-13T13:09:09.668 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.311e-06 2024-09-13T13:09:09.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.668 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.668 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.668 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180130+0000", 2024-09-13T13:09:09.668 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7500000000000003e-06 2024-09-13T13:09:09.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.668 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.668 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.668 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180140+0000", 2024-09-13T13:09:09.668 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9310000000000003e-06 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180147+0000", 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0600000000000002e-06 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276055+0000", 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095907611000000004 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276063+0000", 2024-09-13T13:09:09.669 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9719999999999992e-06 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276165+0000", 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102235 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95513 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.180192+0000", 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 343.424463618, 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.096011869999999999, 2024-09-13T13:09:09.670 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180192+0000", 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180192+0000", 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.671 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180193+0000", 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.032e-06 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180197+0000", 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7079999999999999e-06 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180204+0000", 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2819999999999999e-06 2024-09-13T13:09:09.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.673 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.673 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180209+0000", 2024-09-13T13:09:09.673 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7330000000000003e-06 2024-09-13T13:09:09.673 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.673 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.673 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276187+0000", 2024-09-13T13:09:09.673 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095978049999999995 2024-09-13T13:09:09.673 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.673 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.673 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276193+0000", 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1480000000000003e-06 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276204+0000", 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0917000000000001e-05 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95513 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.180247+0000", 2024-09-13T13:09:09.674 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 343.42440833799998, 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095994805000000002, 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180247+0000", 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180247+0000", 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.675 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.676 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.676 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180248+0000", 2024-09-13T13:09:09.676 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.004e-06 2024-09-13T13:09:09.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.676 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.676 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180252+0000", 2024-09-13T13:09:09.676 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7890000000000001e-06 2024-09-13T13:09:09.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.676 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.676 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180261+0000", 2024-09-13T13:09:09.676 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5089999999999993e-06 2024-09-13T13:09:09.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.677 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.677 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.677 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180267+0000", 2024-09-13T13:09:09.677 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0079999999999999e-06 2024-09-13T13:09:09.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.677 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.677 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.677 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276226+0000", 2024-09-13T13:09:09.678 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095959424000000001 2024-09-13T13:09:09.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.678 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.678 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.678 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276232+0000", 2024-09-13T13:09:09.678 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6189999999999999e-06 2024-09-13T13:09:09.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.678 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.678 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.678 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276242+0000", 2024-09-13T13:09:09.678 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0451999999999999e-05 2024-09-13T13:09:09.678 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.678 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.679 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.679 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.679 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:95658 7.e 7:79f5f817:::benchmark_data_smithi052_71865_object95657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:09:09.679 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.394482+0000", 2024-09-13T13:09:09.679 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 343.21017394400002, 2024-09-13T13:09:09.679 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.081968365000000001, 2024-09-13T13:09:09.679 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.679 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.679 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.679 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.679 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.679 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.680 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95658 2024-09-13T13:09:09.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.680 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.680 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.680 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.680 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394482+0000", 2024-09-13T13:09:09.680 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.680 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.680 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.680 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394482+0000", 2024-09-13T13:09:09.680 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394484+0000", 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4660000000000002e-06 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394512+0000", 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7447999999999999e-05 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394525+0000", 2024-09-13T13:09:09.681 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3064000000000001e-05 2024-09-13T13:09:09.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.682 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.682 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.682 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394533+0000", 2024-09-13T13:09:09.682 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5930000000000006e-06 2024-09-13T13:09:09.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.682 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.682 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.682 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.399009+0000", 2024-09-13T13:09:09.682 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0044759159999999999 2024-09-13T13:09:09.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.682 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.399121+0000", 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011148400000000001 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.399244+0000", 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000123192 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.402074+0000", 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0028305230000000002 2024-09-13T13:09:09.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.403012+0000", 2024-09-13T13:09:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00093742399999999996 2024-09-13T13:09:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.475889+0000", 2024-09-13T13:09:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.072877493000000002 2024-09-13T13:09:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.684 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.476237+0000", 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00034722900000000001 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.476450+0000", 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021353299999999999 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:96740 7.e 7:7b47086e:::benchmark_data_smithi052_71865_object96739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:27.478436+0000", 2024-09-13T13:09:09.685 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 342.12621924899997, 2024-09-13T13:09:09.686 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.082652384999999995, 2024-09-13T13:09:09.686 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.686 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.686 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.686 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.686 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.686 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.686 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 96740 2024-09-13T13:09:09.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.686 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.686 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.686 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.686 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478436+0000", 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478436+0000", 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478439+0000", 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0990000000000001e-06 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.687 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.688 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478466+0000", 2024-09-13T13:09:09.688 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7987000000000002e-05 2024-09-13T13:09:09.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.688 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.688 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.688 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478477+0000", 2024-09-13T13:09:09.688 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0587999999999999e-05 2024-09-13T13:09:09.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.688 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.688 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.688 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478484+0000", 2024-09-13T13:09:09.688 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2799999999999998e-06 2024-09-13T13:09:09.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.479227+0000", 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00074284600000000002 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.479355+0000", 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012828000000000001 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.479490+0000", 2024-09-13T13:09:09.689 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000134859 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.483095+0000", 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036047169999999999 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.483243+0000", 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014764499999999999 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.560910+0000", 2024-09-13T13:09:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.077667016000000005 2024-09-13T13:09:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.561046+0000", 2024-09-13T13:09:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013630099999999999 2024-09-13T13:09:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.561089+0000", 2024-09-13T13:09:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2766999999999999e-05 2024-09-13T13:09:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:118717 7.e 7:7eac196e:::benchmark_data_smithi052_71865_object118716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 320.04443698, 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.120995147, 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.692 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 118717 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.693 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560220+0000", 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4200000000000004e-07 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560244+0000", 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4354000000000001e-05 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560252+0000", 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5189999999999992e-06 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560259+0000", 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.302e-06 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560301+0000", 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2318999999999999e-05 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560415+0000", 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114269 2024-09-13T13:09:09.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.696 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.696 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560572+0000", 2024-09-13T13:09:09.696 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000156217 2024-09-13T13:09:09.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.696 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.696 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.564231+0000", 2024-09-13T13:09:09.696 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036597489999999999 2024-09-13T13:09:09.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.696 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.696 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.565180+0000", 2024-09-13T13:09:09.697 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00094882300000000001 2024-09-13T13:09:09.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.697 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.697 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.680844+0000", 2024-09-13T13:09:09.697 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.115664164 2024-09-13T13:09:09.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.697 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.697 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.681157+0000", 2024-09-13T13:09:09.697 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00031297600000000002 2024-09-13T13:09:09.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.698 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.698 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.681214+0000", 2024-09-13T13:09:09.698 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6512999999999999e-05 2024-09-13T13:09:09.698 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.698 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.698 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.698 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.698 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.698 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120031 7.e 7:704b0327:::benchmark_data_smithi052_71865_object120030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:09:09.698 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:09:09.698 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 318.58384973, 2024-09-13T13:09:09.698 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11824775799999999, 2024-09-13T13:09:09.698 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120031 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:09:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020808+0000", 2024-09-13T13:09:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0439999999999998e-06 2024-09-13T13:09:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020838+0000", 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9791e-05 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020856+0000", 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8664999999999998e-05 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020866+0000", 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3670000000000003e-06 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021203+0000", 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033735200000000002 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021578+0000", 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00037465399999999998 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022015+0000", 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000437176 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.703 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.703 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.026523+0000", 2024-09-13T13:09:09.703 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045084249999999999 2024-09-13T13:09:09.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.703 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.703 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.703 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.027199+0000", 2024-09-13T13:09:09.703 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00067598400000000002 2024-09-13T13:09:09.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138915+0000", 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11171563499999999 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139008+0000", 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3329e-05 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139054+0000", 2024-09-13T13:09:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5336000000000001e-05 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120050 7.e 7:72296fb6:::benchmark_data_smithi052_71865_object120049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 318.57180168500003, 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12504636999999999, 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120050 2024-09-13T13:09:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:09:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:09:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032857+0000", 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1140000000000002e-06 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032888+0000", 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1115999999999998e-05 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032905+0000", 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6452000000000001e-05 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032915+0000", 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8889999999999997e-06 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033665+0000", 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00075015300000000004 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033782+0000", 2024-09-13T13:09:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117642 2024-09-13T13:09:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033916+0000", 2024-09-13T13:09:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013400799999999999 2024-09-13T13:09:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.037705+0000", 2024-09-13T13:09:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037884289999999998 2024-09-13T13:09:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.037719+0000", 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4034e-05 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157684+0000", 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.119964979 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157805+0000", 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120715 2024-09-13T13:09:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157900+0000", 2024-09-13T13:09:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5839000000000002e-05 2024-09-13T13:09:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120054 7.e 7:7dad863d:::benchmark_data_smithi052_71865_object120053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:09:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:09:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 318.56628640899999, 2024-09-13T13:09:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11988120300000001, 2024-09-13T13:09:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120054 2024-09-13T13:09:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:09:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:09:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038370+0000", 2024-09-13T13:09:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.051e-06 2024-09-13T13:09:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.714 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038395+0000", 2024-09-13T13:09:09.714 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4224999999999999e-05 2024-09-13T13:09:09.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.714 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.714 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.714 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038403+0000", 2024-09-13T13:09:09.714 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3399999999999998e-06 2024-09-13T13:09:09.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.714 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.714 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.714 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038409+0000", 2024-09-13T13:09:09.714 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3160000000000002e-06 2024-09-13T13:09:09.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038462+0000", 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3012e-05 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038579+0000", 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116483 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038720+0000", 2024-09-13T13:09:09.715 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014139099999999999 2024-09-13T13:09:09.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.716 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.716 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.716 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.042075+0000", 2024-09-13T13:09:09.716 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033552830000000001 2024-09-13T13:09:09.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.716 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.716 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.716 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.042095+0000", 2024-09-13T13:09:09.716 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9279999999999998e-05 2024-09-13T13:09:09.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.716 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.716 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158031+0000", 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11593616299999999 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158169+0000", 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013835799999999999 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158250+0000", 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1301e-05 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.717 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.718 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.718 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.718 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.718 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:135717 7.e 7:74ed6606:::benchmark_data_smithi052_71865_object14616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:09:09.718 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:09.110240+0000", 2024-09-13T13:09:09.718 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 300.49441534499999, 2024-09-13T13:09:09.718 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.076080909000000002, 2024-09-13T13:09:09.718 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.718 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.718 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.718 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.718 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.718 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 135717 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110240+0000", 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110240+0000", 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.719 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.720 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:09.721 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110241+0000", 2024-09-13T13:09:09.721 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:09:09.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.721 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110243+0000", 2024-09-13T13:09:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4090000000000001e-06 2024-09-13T13:09:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110248+0000", 2024-09-13T13:09:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1250000000000001e-06 2024-09-13T13:09:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110253+0000", 2024-09-13T13:09:09.722 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8350000000000003e-06 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.113974+0000", 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003720737 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.114066+0000", 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1833e-05 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.723 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.114179+0000", 2024-09-13T13:09:09.724 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112695 2024-09-13T13:09:09.724 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.724 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.724 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.724 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.116490+0000", 2024-09-13T13:09:09.724 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023109210000000001 2024-09-13T13:09:09.724 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.724 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.724 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.724 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.116845+0000", 2024-09-13T13:09:09.724 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035593400000000002 2024-09-13T13:09:09.724 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.724 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.186206+0000", 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.069360755999999996 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.186246+0000", 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9888000000000002e-05 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.186321+0000", 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5136999999999996e-05 2024-09-13T13:09:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.726 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.726 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.726 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.726 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151582 7.e 7:7a3dbf39:::benchmark_data_smithi052_71865_object30481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:09:09.726 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.170441+0000", 2024-09-13T13:09:09.726 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 280.43421462499998, 2024-09-13T13:09:09.726 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.091679105999999996, 2024-09-13T13:09:09.726 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.726 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.726 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.726 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.726 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151582 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170441+0000", 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170441+0000", 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170442+0000", 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5000000000000002e-07 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170445+0000", 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.88e-06 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170450+0000", 2024-09-13T13:09:09.728 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3419999999999998e-06 2024-09-13T13:09:09.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.729 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.729 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.729 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170455+0000", 2024-09-13T13:09:09.729 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.096e-06 2024-09-13T13:09:09.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.729 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.729 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.729 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174840+0000", 2024-09-13T13:09:09.729 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0043852609999999997 2024-09-13T13:09:09.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.729 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.729 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174948+0000", 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010804 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.175063+0000", 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0001145 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.178533+0000", 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003470065 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.730 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.178591+0000", 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8149000000000002e-05 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261725+0000", 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.083133548000000002 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261955+0000", 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00023013299999999999 2024-09-13T13:09:09.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.732 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.732 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.732 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.262120+0000", 2024-09-13T13:09:09.732 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016544199999999999 2024-09-13T13:09:09.732 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.732 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.732 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.732 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.732 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151596 7.e 7:74fc7c8c:::benchmark_data_smithi052_71865_object30495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:09:09.732 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.187262+0000", 2024-09-13T13:09:09.732 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 280.41739344000001, 2024-09-13T13:09:09.732 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.084009381999999994, 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151596 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187262+0000", 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187262+0000", 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187264+0000", 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2199999999999999e-06 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187273+0000", 2024-09-13T13:09:09.734 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4500000000000004e-06 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187285+0000", 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2446000000000001e-05 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187293+0000", 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4000000000000003e-06 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.735 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187360+0000", 2024-09-13T13:09:09.736 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6908000000000004e-05 2024-09-13T13:09:09.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.736 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.736 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.736 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187487+0000", 2024-09-13T13:09:09.736 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000126859 2024-09-13T13:09:09.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.736 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.736 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.736 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187631+0000", 2024-09-13T13:09:09.736 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014448999999999999 2024-09-13T13:09:09.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.736 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.737 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.737 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193046+0000", 2024-09-13T13:09:09.737 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.005415352 2024-09-13T13:09:09.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.737 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.737 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.737 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193098+0000", 2024-09-13T13:09:09.737 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1375999999999997e-05 2024-09-13T13:09:09.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.737 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.737 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.737 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.271006+0000", 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.077907893000000006 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.271225+0000", 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021893800000000001 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.271272+0000", 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7049999999999998e-05 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.738 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.739 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151600 7.e 7:7fe6ca51:::benchmark_data_smithi052_71865_object30499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:09:09.739 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.189256+0000", 2024-09-13T13:09:09.739 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 280.41539999700001, 2024-09-13T13:09:09.739 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.086529871999999994, 2024-09-13T13:09:09.739 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.739 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.739 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.739 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.739 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.739 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.739 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151600 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189256+0000", 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189256+0000", 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.740 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.741 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189257+0000", 2024-09-13T13:09:09.741 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7209999999999999e-06 2024-09-13T13:09:09.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.741 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.741 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.741 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189263+0000", 2024-09-13T13:09:09.741 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3580000000000002e-06 2024-09-13T13:09:09.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.741 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.741 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.741 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189275+0000", 2024-09-13T13:09:09.741 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1973e-05 2024-09-13T13:09:09.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189283+0000", 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9039999999999992e-06 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.190358+0000", 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010753099999999999 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.190465+0000", 2024-09-13T13:09:09.742 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010736800000000001 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.190557+0000", 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193101+0000", 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025442569999999999 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.743 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193130+0000", 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8728999999999999e-05 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.275647+0000", 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.082517141000000002 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.275742+0000", 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5156999999999997e-05 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.744 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.745 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.275786+0000", 2024-09-13T13:09:09.745 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3047000000000003e-05 2024-09-13T13:09:09.745 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.745 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.745 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.745 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:153862 7.e 7:797e5c7a:::benchmark_data_smithi052_71865_object32761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:09:09.745 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:09:09.745 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 277.33984893799999, 2024-09-13T13:09:09.745 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.106314172, 2024-09-13T13:09:09.745 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.745 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 153862 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.746 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264809+0000", 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.148e-06 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264815+0000", 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7610000000000004e-06 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.747 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264826+0000", 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1215e-05 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264833+0000", 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0090000000000001e-06 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264885+0000", 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1845000000000003e-05 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.265001+0000", 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116795 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.265104+0000", 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102979 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.268877+0000", 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037720779999999999 2024-09-13T13:09:09.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.750 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.750 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.269086+0000", 2024-09-13T13:09:09.750 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000209829 2024-09-13T13:09:09.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.750 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.750 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.370873+0000", 2024-09-13T13:09:09.750 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10178622700000001 2024-09-13T13:09:09.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.750 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.750 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.371063+0000", 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000190349 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.371121+0000", 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7936999999999998e-05 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:153884 7.8 7:1006e86e:::benchmark_data_smithi052_71865_object32783:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:32.288977+0000", 2024-09-13T13:09:09.751 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 277.31567817199999, 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.086944270000000004, 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 153884 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288977+0000", 2024-09-13T13:09:09.752 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.753 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.753 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.753 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288977+0000", 2024-09-13T13:09:09.753 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.753 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.753 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.753 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288979+0000", 2024-09-13T13:09:09.753 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9e-06 2024-09-13T13:09:09.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.753 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.753 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288985+0000", 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4580000000000001e-06 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288997+0000", 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2004999999999999e-05 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.289005+0000", 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0069999999999997e-06 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.754 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.289044+0000", 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9205000000000002e-05 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.289111+0000", 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6952000000000005e-05 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.359719+0000", 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.070607721999999998 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.756 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.756 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.363436+0000", 2024-09-13T13:09:09.756 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037171370000000001 2024-09-13T13:09:09.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.756 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.756 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.756 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375313+0000", 2024-09-13T13:09:09.756 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011876639 2024-09-13T13:09:09.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.756 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.756 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.756 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375551+0000", 2024-09-13T13:09:09.756 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000238939 2024-09-13T13:09:09.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.757 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.757 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.757 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375761+0000", 2024-09-13T13:09:09.757 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021001599999999999 2024-09-13T13:09:09.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.757 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.757 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.757 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375922+0000", 2024-09-13T13:09:09.757 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016029 2024-09-13T13:09:09.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.757 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:153886 7.e 7:718ea587:::benchmark_data_smithi052_71865_object32785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:32.293022+0000", 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 277.311633809, 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.081446456, 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 153886 2024-09-13T13:09:09.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.293022+0000", 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.293022+0000", 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.759 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.293025+0000", 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7580000000000001e-06 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.293032+0000", 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7619999999999995e-06 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.357646+0000", 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.064613241000000002 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.760 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.761 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.357665+0000", 2024-09-13T13:09:09.761 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9284999999999999e-05 2024-09-13T13:09:09.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.761 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.761 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.358627+0000", 2024-09-13T13:09:09.761 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00096251400000000003 2024-09-13T13:09:09.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.761 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.761 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.358770+0000", 2024-09-13T13:09:09.761 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014308699999999999 2024-09-13T13:09:09.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.358905+0000", 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000134785 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.362681+0000", 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037752800000000002 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.364635+0000", 2024-09-13T13:09:09.762 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019547779999999999 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.374309+0000", 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0096736170000000007 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.374416+0000", 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107389 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.763 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.374468+0000", 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1959999999999997e-05 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:199092 7.e 7:7925136c:::benchmark_data_smithi052_71865_object77991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:05:25.233281+0000", 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 224.37137484499999, 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.076015174000000005, 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:09.764 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:09.765 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:09.765 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:09.765 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 199092 2024-09-13T13:09:09.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.765 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:09.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.765 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:09.765 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233281+0000", 2024-09-13T13:09:09.765 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.765 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:09.765 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233281+0000", 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233282+0000", 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2929999999999999e-06 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233287+0000", 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0039999999999999e-06 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.766 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233297+0000", 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8509999999999994e-06 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233304+0000", 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6680000000000004e-06 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233364+0000", 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0167999999999997e-05 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233462+0000", 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7813999999999996e-05 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233561+0000", 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9557999999999998e-05 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.236795+0000", 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0032333890000000001 2024-09-13T13:09:09.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.236836+0000", 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1826000000000002e-05 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.309149+0000", 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.072312716999999999 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.309250+0000", 2024-09-13T13:09:09.769 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010125599999999999 2024-09-13T13:09:09.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:09.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:09.770 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:09.770 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.309296+0000", 2024-09-13T13:09:09.770 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5630000000000002e-05 2024-09-13T13:09:09.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.770 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:09.770 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:09.770 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:09:09.811 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:09.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:09.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:09.943 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:09.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:09.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:10.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:10.079 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:10.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:10.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:10.181 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:10.215 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:10.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:10.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:10.317 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:10.327 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T13:09:10.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:10.367 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:10.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:10.403 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:09:10.468 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:10.519 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:10.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:10.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:10 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1029170712' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:09:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:10 smithi167 ceph-mon[33178]: from='client.35492 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:09:10.621 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:10.650 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:10.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:10.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:10.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:10 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1029170712' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:09:10.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:10 smithi052 ceph-mon[36682]: from='client.35492 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:09:10.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:10 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1029170712' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:09:10.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:10 smithi052 ceph-mon[29471]: from='client.35492 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:09:10.751 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:10.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:10.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:10.880 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:10.918 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T13:09:10.927 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:10.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:10.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:11.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:11.054 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:11.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:11.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:11.098 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:09:11.156 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:11.245 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:11.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:11.346 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:11.393 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:11.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:11.452 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:11 smithi052 ceph-mon[29471]: pgmap v383: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-13T13:09:11.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:11 smithi052 ceph-mon[29471]: from='client.35492 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:09:11.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:11 smithi052 ceph-mon[29471]: osdmap e224: 8 total, 8 up, 8 in 2024-09-13T13:09:11.453 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:11 smithi052 ceph-mon[36682]: pgmap v383: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-13T13:09:11.453 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:11 smithi052 ceph-mon[36682]: from='client.35492 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:09:11.453 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:11 smithi052 ceph-mon[36682]: osdmap e224: 8 total, 8 up, 8 in 2024-09-13T13:09:11.496 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:11.528 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:11.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:11.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:11.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:11.663 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:11.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:11.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:11.797 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:11.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:11.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:11.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:11 smithi167 ceph-mon[33178]: pgmap v383: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-13T13:09:11.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:11 smithi167 ceph-mon[33178]: from='client.35492 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:09:11.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:11 smithi167 ceph-mon[33178]: osdmap e224: 8 total, 8 up, 8 in 2024-09-13T13:09:11.899 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:11.941 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:11.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:11.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:12.043 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T13:09:12.043 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:12.101 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:12.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:12.201 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:12.202 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:12.244 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:12.302 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:12.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:12.404 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:12.452 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:12.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:12.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:12.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:12.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:12.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:12.689 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:12.720 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:12.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:12.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:12.822 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:12.862 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:12.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:12.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:12.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:12.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:12.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:13.092 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:13.135 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:13.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:13.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:13.237 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:13.242 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:09:13.242 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:09:13.281 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:13.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:13.283 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:09:13.283 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:09:13.299 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:09:13.299 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:09:13.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.299 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.25222.0:209380 8.8 e224/169 8:1c316b4e:::benchmark_data_smithi052_101302_object81451:head v 224'12897, mlcod=224'12895)", 2024-09-13T13:09:13.299 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:13.106325+0000", 2024-09-13T13:09:13.299 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.011321365999999999, 2024-09-13T13:09:13.299 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011349069999999999, 2024-09-13T13:09:13.299 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:13.299 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:13.299 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:09:13.299 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:13.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.300 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:13.300 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106325+0000", 2024-09-13T13:09:13.300 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.300 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:13.300 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106325+0000", 2024-09-13T13:09:13.300 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.300 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:13.301 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106326+0000", 2024-09-13T13:09:13.301 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6789999999999999e-06 2024-09-13T13:09:13.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.301 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.301 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:13.301 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106334+0000", 2024-09-13T13:09:13.301 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.452e-06 2024-09-13T13:09:13.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.301 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.301 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:13.301 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106344+0000", 2024-09-13T13:09:13.301 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9890000000000004e-06 2024-09-13T13:09:13.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.302 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.302 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:13.302 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106351+0000", 2024-09-13T13:09:13.302 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5889999999999997e-06 2024-09-13T13:09:13.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.302 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.302 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:13.302 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106446+0000", 2024-09-13T13:09:13.302 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4689000000000001e-05 2024-09-13T13:09:13.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.302 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.303 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:13.303 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106471+0000", 2024-09-13T13:09:13.303 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4797999999999999e-05 2024-09-13T13:09:13.303 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.303 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:13.303 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.303 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.303 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:209384 8.b 8:dca031a4:::benchmark_data_smithi052_101302_object81455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T13:09:13.303 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:13.106600+0000", 2024-09-13T13:09:13.303 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.011045732000000001, 2024-09-13T13:09:13.303 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011136559000000001, 2024-09-13T13:09:13.303 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:13.304 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:13.304 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:09:13.304 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:13.304 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:13.304 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:13.304 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 209384 2024-09-13T13:09:13.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.304 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:13.304 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.304 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:13.304 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106600+0000", 2024-09-13T13:09:13.304 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.305 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:13.305 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106600+0000", 2024-09-13T13:09:13.305 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.305 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:13.305 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106602+0000", 2024-09-13T13:09:13.305 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.539e-06 2024-09-13T13:09:13.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.306 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:13.306 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106606+0000", 2024-09-13T13:09:13.306 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3710000000000002e-06 2024-09-13T13:09:13.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.306 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:13.306 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106613+0000", 2024-09-13T13:09:13.306 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9920000000000001e-06 2024-09-13T13:09:13.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.306 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:13.306 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106618+0000", 2024-09-13T13:09:13.306 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0969999999999996e-06 2024-09-13T13:09:13.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.307 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:13.307 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.106972+0000", 2024-09-13T13:09:13.307 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035347200000000002 2024-09-13T13:09:13.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.307 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:13.307 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.107120+0000", 2024-09-13T13:09:13.307 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000148386 2024-09-13T13:09:13.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.307 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.107339+0000", 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000219328 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.115576+0000", 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0082360679999999992 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.115618+0000", 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2188999999999999e-05 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:13.309 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.309 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.25222.0:209382 8.8 e224/169)", 2024-09-13T13:09:13.309 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:13.112240+0000", 2024-09-13T13:09:13.309 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.005405979, 2024-09-13T13:09:13.309 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055718859999999999, 2024-09-13T13:09:13.309 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:13.309 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:13.309 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:09:13.309 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:13.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.310 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:13.310 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.112240+0000", 2024-09-13T13:09:13.310 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.310 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.310 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:13.310 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.112240+0000", 2024-09-13T13:09:13.310 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.310 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.310 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:13.310 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.112242+0000", 2024-09-13T13:09:13.310 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3489999999999999e-06 2024-09-13T13:09:13.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.311 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:13.311 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.112252+0000", 2024-09-13T13:09:13.311 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1849999999999993e-06 2024-09-13T13:09:13.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.311 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:13.311 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.112265+0000", 2024-09-13T13:09:13.311 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2944e-05 2024-09-13T13:09:13.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.312 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.312 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:13.312 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.112275+0000", 2024-09-13T13:09:13.312 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0321e-05 2024-09-13T13:09:13.312 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.312 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:13.312 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.312 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.312 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:209389 8.0 8.cafdb560 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T13:09:13.312 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:13.113146+0000", 2024-09-13T13:09:13.312 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0044999560000000003, 2024-09-13T13:09:13.313 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047058999999999998, 2024-09-13T13:09:13.313 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:13.313 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:13.313 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:09:13.313 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:13.316 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:13.316 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:13.316 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 209389 2024-09-13T13:09:13.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.317 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:13.317 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.317 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:13.317 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.113146+0000", 2024-09-13T13:09:13.317 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.317 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.317 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:13.317 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.113146+0000", 2024-09-13T13:09:13.317 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.317 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.318 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:13.318 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.113148+0000", 2024-09-13T13:09:13.318 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1160000000000002e-06 2024-09-13T13:09:13.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.318 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.318 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:13.318 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.113154+0000", 2024-09-13T13:09:13.318 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7599999999999999e-06 2024-09-13T13:09:13.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.318 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.318 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:13.318 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.113165+0000", 2024-09-13T13:09:13.318 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:09:13.319 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.319 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.319 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:13.319 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.113172+0000", 2024-09-13T13:09:13.319 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2910000000000002e-06 2024-09-13T13:09:13.319 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.319 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:13.319 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.319 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.319 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.319 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:209390 8.b 8:d8feb52f:::benchmark_data_smithi052_101302_object81461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T13:09:13.319 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:13.113186+0000", 2024-09-13T13:09:13.319 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.004460505, 2024-09-13T13:09:13.320 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047186670000000002, 2024-09-13T13:09:13.320 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:13.320 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:13.320 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:09:13.320 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:13.320 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:13.320 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:13.320 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 209390 2024-09-13T13:09:13.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.320 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:13.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.320 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:13.321 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.113186+0000", 2024-09-13T13:09:13.321 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.321 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.321 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:13.321 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.113186+0000", 2024-09-13T13:09:13.321 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.321 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.321 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:13.321 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.113186+0000", 2024-09-13T13:09:13.321 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1299999999999999e-07 2024-09-13T13:09:13.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.113190+0000", 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3390000000000001e-06 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.113196+0000", 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4330000000000001e-06 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.113201+0000", 2024-09-13T13:09:13.322 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4360000000000001e-06 2024-09-13T13:09:13.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.323 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:13.323 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.115650+0000", 2024-09-13T13:09:13.323 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024487469999999998 2024-09-13T13:09:13.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.323 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:13.323 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.115770+0000", 2024-09-13T13:09:13.323 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011950799999999999 2024-09-13T13:09:13.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.324 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:09:13.324 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.115891+0000", 2024-09-13T13:09:13.324 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120912 2024-09-13T13:09:13.324 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.324 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:13.324 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.324 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:209395 8.f 8:feb7fb67:::benchmark_data_smithi052_101302_object81466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T13:09:13.324 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:13.117618+0000", 2024-09-13T13:09:13.325 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 2.7614e-05, 2024-09-13T13:09:13.325 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035539000000000002, 2024-09-13T13:09:13.325 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:13.325 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:13.325 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:09:13.325 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:13.325 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:13.325 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:13.325 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 209395 2024-09-13T13:09:13.325 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.325 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:13.325 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.325 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117618+0000", 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117618+0000", 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117621+0000", 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5210000000000001e-06 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.326 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:13.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117629+0000", 2024-09-13T13:09:13.327 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8199999999999997e-06 2024-09-13T13:09:13.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.327 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:13.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117648+0000", 2024-09-13T13:09:13.327 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8953999999999998e-05 2024-09-13T13:09:13.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.327 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:13.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117658+0000", 2024-09-13T13:09:13.327 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0156000000000001e-05 2024-09-13T13:09:13.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.328 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:13.328 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117808+0000", 2024-09-13T13:09:13.328 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014974899999999999 2024-09-13T13:09:13.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.328 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:13.328 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117924+0000", 2024-09-13T13:09:13.328 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011667099999999999 2024-09-13T13:09:13.328 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.328 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:13.328 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.329 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:209398 8.b 8.5975693b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-13T13:09:13.329 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:13.117681+0000", 2024-09-13T13:09:13.329 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 4294967295.9999647, 2024-09-13T13:09:13.329 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00034594200000000002, 2024-09-13T13:09:13.329 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:13.329 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:13.329 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:09:13.329 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:13.329 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:13.329 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:13.329 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 209398 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117681+0000", 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117681+0000", 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.330 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117683+0000", 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0559999999999999e-06 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117686+0000", 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6260000000000001e-06 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117694+0000", 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9899999999999997e-06 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:13.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:13.332 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:13.332 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:13.117700+0000", 2024-09-13T13:09:13.332 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8780000000000003e-06 2024-09-13T13:09:13.332 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.332 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:13.332 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.332 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:13.332 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:09:13.332 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 7 2024-09-13T13:09:13.332 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:09:13.333 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:09:13.333 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:09:13.365 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:09:13.365 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:09:13.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:13.402 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:09:13.402 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:09:13.410 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:13.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:13.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:13.437 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:09:13.437 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:09:13.467 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:09:13.468 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:09:13.503 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:09:13.504 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:09:13.504 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:13 smithi052 ceph-mon[36682]: pgmap v385: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 904 op/s 2024-09-13T13:09:13.504 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1776585599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:13.504 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:13 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:09:13.504 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:13 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:09:13.504 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:13 smithi052 ceph-mon[29471]: pgmap v385: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 904 op/s 2024-09-13T13:09:13.505 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1776585599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:13.505 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:13 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:09:13.505 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:13 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:09:13.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:13.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:13.545 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:13.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:13.646 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:13.677 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:13.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:13.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:13.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:13.809 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:13.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:13.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:13.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:13 smithi167 ceph-mon[33178]: pgmap v385: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 904 op/s 2024-09-13T13:09:13.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1776585599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:13.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:13 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:09:13.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:13 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:09:13.911 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:13.938 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:13.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:13.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:14.040 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:14.075 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:14.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:14.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:14.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:14.205 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:14.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:14.307 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:14.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:09:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:09:14.346 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:14.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:14.445 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:14.448 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:14.471 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:14.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:14.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:14.612 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:14.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:14.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:14.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:14 smithi052 ceph-mon[29471]: osdmap e225: 8 total, 8 up, 8 in 2024-09-13T13:09:14.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:14 smithi052 ceph-mon[36682]: osdmap e225: 8 total, 8 up, 8 in 2024-09-13T13:09:14.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:14.746 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:14.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:14.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:14 smithi167 ceph-mon[33178]: osdmap e225: 8 total, 8 up, 8 in 2024-09-13T13:09:14.847 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:14.881 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:14.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:14.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:14.913 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T13:09:14.965 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:15.010 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:15.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:15.065 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:15.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:15.166 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:15.262 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:15.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:15.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:15.363 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:15.391 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:15.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:15.452 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:15 smithi052 ceph-mon[36682]: pgmap v387: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-09-13T13:09:15.452 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:15.454 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:15 smithi052 ceph-mon[29471]: pgmap v387: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-09-13T13:09:15.454 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:15.492 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:15.519 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:15.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:15.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:15.621 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:15.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:15.769 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:15.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:15.816 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:15.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:15.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:15 smithi167 ceph-mon[33178]: pgmap v387: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-09-13T13:09:15.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:15.917 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:15.959 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:15.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:15.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:16.062 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:16.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:16.099 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:16.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:16.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:16.229 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:16.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:16.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:16.238 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:09:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:09:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:09:16.331 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:16.372 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:16.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:16.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:16.389 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:09:16.474 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:16.490 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:09:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:16.516 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:16.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:16.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:16.649 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:16.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:16 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3965271689' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:16 smithi052 ceph-mon[36682]: from='client.25504 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:16 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3965271689' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:16 smithi052 ceph-mon[29471]: from='client.25504 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:16.750 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:16.818 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:16.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:16.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:16.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:16 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3965271689' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:16.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:16 smithi167 ceph-mon[33178]: from='client.25504 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:16.920 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:16.970 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:16.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:17.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:17.100 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:17.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:17.205 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:17.231 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:17.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:17.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:17.333 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:17.371 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:17.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:17.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:17.473 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:17.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:17 smithi052 ceph-mon[29471]: pgmap v388: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 685 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-13T13:09:17.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:17 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:17.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:17 smithi052 ceph-mon[29471]: from='client.25504 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:09:17.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:17 smithi052 ceph-mon[29471]: osdmap e226: 8 total, 8 up, 8 in 2024-09-13T13:09:17.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:17 smithi052 ceph-mon[36682]: pgmap v388: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 685 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-13T13:09:17.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:17 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:17.493 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:17 smithi052 ceph-mon[36682]: from='client.25504 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:09:17.493 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:17 smithi052 ceph-mon[36682]: osdmap e226: 8 total, 8 up, 8 in 2024-09-13T13:09:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:17.512 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:17.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:17.616 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:17.662 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:17.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:17.763 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:17.790 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:17.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:17 smithi167 ceph-mon[33178]: pgmap v388: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 685 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-13T13:09:17.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:17 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:17.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:17 smithi167 ceph-mon[33178]: from='client.25504 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:09:17.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:17 smithi167 ceph-mon[33178]: osdmap e226: 8 total, 8 up, 8 in 2024-09-13T13:09:17.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:17.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:17.927 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:17.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:18.028 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:18.063 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:18.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:18.165 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:18.196 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:18.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:18.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:18.298 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:18.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:18.338 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:18.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:18.380 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T13:09:18.439 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:18.482 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:18.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:18.583 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:18.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:18.624 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:18.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:18.665 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:18 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2767510929' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:09:18.665 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:18 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:09:18.665 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:18 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:09:18.666 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:18 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2767510929' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:09:18.666 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:18 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:09:18.666 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:18 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:09:18.725 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:18.766 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:18.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:18.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:18 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2767510929' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:09:18.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:18 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:09:18.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:18 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:09:18.868 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:18.900 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:18.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:18.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:19.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:19.038 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:19.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:19.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:19.140 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:19.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:19.178 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:19.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:19.279 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:19.307 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:19.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:19.381 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:19 smithi052 ceph-mon[29471]: pgmap v390: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 687 B/s rd, 0 B/s wr, 886 op/s 2024-09-13T13:09:19.381 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:19 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2767510929' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:09:19.381 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:19 smithi052 ceph-mon[29471]: osdmap e227: 8 total, 8 up, 8 in 2024-09-13T13:09:19.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:19.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:19.457 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:19.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:19.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:19.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:19.594 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:19.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:19.695 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:19.735 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:19.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:19.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:19.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:19 smithi052 ceph-mon[36682]: pgmap v390: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 687 B/s rd, 0 B/s wr, 886 op/s 2024-09-13T13:09:19.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:19 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2767510929' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:09:19.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:19 smithi052 ceph-mon[36682]: osdmap e227: 8 total, 8 up, 8 in 2024-09-13T13:09:19.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:19 smithi167 ceph-mon[33178]: pgmap v390: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 687 B/s rd, 0 B/s wr, 886 op/s 2024-09-13T13:09:19.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:19 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2767510929' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:09:19.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:19 smithi167 ceph-mon[33178]: osdmap e227: 8 total, 8 up, 8 in 2024-09-13T13:09:19.837 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:19.869 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:19.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:19.971 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:19.985 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T13:09:20.002 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:20.018 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:20.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:20.036 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:20.120 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:20.174 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:20.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:20.276 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:20.300 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:20.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:20.402 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:20.457 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:20.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:20.559 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:20.591 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:20.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:20.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:20.694 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:20.714 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:09:20.714 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:09:20.746 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:20.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:20.753 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:09:20.753 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:09:20.796 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:09:20.797 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:09:20.829 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:09:20.829 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:09:20.848 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:20.861 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:09:20.861 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:09:20.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:20.887 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:20.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:20.899 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:09:20.899 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:09:20.933 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:09:20.933 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:09:20.970 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:09:20.971 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:09:20.988 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:21.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:21.019 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:21.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:21.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:09:21.119 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:21.151 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:21.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:21.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:21.281 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:21.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:21.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:21.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:21.407 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:21.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:21.506 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:09:21.506 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:09:21.506 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:09:21.506 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:09:21.506 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:09:21.509 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:21.550 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:21.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:21.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:21.652 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:21.691 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:21.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:21.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:21.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:21 smithi052 ceph-mon[29471]: pgmap v392: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-13T13:09:21.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:21 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/16188614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:21.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:21 smithi052 ceph-mon[36682]: pgmap v392: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-13T13:09:21.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:21 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/16188614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:21.793 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:21.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:21 smithi167 ceph-mon[33178]: pgmap v392: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-13T13:09:21.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:21 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/16188614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:21.821 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:21.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:21.923 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:21.959 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:21.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:22.061 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:22.102 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:22.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:22.204 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:22.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:22.241 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:22.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:22.342 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:22.373 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:22.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:22.475 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:22.516 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:22.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:22.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:22.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:22.652 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:22.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:22.754 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:22.784 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:22.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:22.885 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:22.916 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:22.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:22.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:23.018 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:23.024 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T13:09:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:23.056 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:23.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:23.158 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:23.184 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:23.210 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:23.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:23.273 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:23.312 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:23.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:23.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:23.445 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:23.489 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:23.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:23.590 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:23.635 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:23.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:23.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:23.644 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:23 smithi052 ceph-mon[36682]: pgmap v393: 148 pgs: 148 active+clean; 2.5 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 925 op/s 2024-09-13T13:09:23.644 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:23 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:09:23.644 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:23 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:09:23.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:23 smithi052 ceph-mon[29471]: pgmap v393: 148 pgs: 148 active+clean; 2.5 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 925 op/s 2024-09-13T13:09:23.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:23 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:09:23.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:23 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:09:23.737 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:23.774 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:23.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:23 smithi167 ceph-mon[33178]: pgmap v393: 148 pgs: 148 active+clean; 2.5 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 925 op/s 2024-09-13T13:09:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:23 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:09:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:23 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:09:23.875 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:23.920 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:23.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:24.022 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:24.069 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:24.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:24.171 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:24.195 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:24.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:24.297 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:24.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:09:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:09:24.331 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:24.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:24.433 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:24.495 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:24.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:24.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:24.513 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:24.597 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:24.639 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:24.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:24.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:24.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:24 smithi052 ceph-mon[29471]: osdmap e228: 8 total, 8 up, 8 in 2024-09-13T13:09:24.662 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:24 smithi052 ceph-mon[36682]: osdmap e228: 8 total, 8 up, 8 in 2024-09-13T13:09:24.741 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:24.786 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:24.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:24.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:24 smithi167 ceph-mon[33178]: osdmap e228: 8 total, 8 up, 8 in 2024-09-13T13:09:24.888 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:24.948 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:24.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:25.050 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:25.088 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:25.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:25.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:25.153 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:09:25.153 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:09:25.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:09:25.153 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:09:25.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.153 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83963 7.c 7:30d6a006:::benchmark_data_smithi052_71865_object83962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:09:25.154 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:09:25.154 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 370.60144766600001, 2024-09-13T13:09:25.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.013163161, 2024-09-13T13:09:25.154 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.154 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.154 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83963 2024-09-13T13:09:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:09:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:09:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547884+0000", 2024-09-13T13:09:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.779e-06 2024-09-13T13:09:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547907+0000", 2024-09-13T13:09:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2779999999999999e-05 2024-09-13T13:09:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547917+0000", 2024-09-13T13:09:25.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8439999999999993e-06 2024-09-13T13:09:25.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:09:25.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5229999999999998e-06 2024-09-13T13:09:25.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554417+0000", 2024-09-13T13:09:25.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064933880000000001 2024-09-13T13:09:25.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:09:25.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554427+0000", 2024-09-13T13:09:25.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4280000000000002e-06 2024-09-13T13:09:25.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.543883+0000", 2024-09-13T13:09:25.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98945604700000001 2024-09-13T13:09:25.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545504+0000", 2024-09-13T13:09:25.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016211660000000001 2024-09-13T13:09:25.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:09:25.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545747+0000", 2024-09-13T13:09:25.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024302199999999999 2024-09-13T13:09:25.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552352+0000", 2024-09-13T13:09:25.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.006605133 2024-09-13T13:09:25.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560880+0000", 2024-09-13T13:09:25.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0085281650000000007 2024-09-13T13:09:25.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560953+0000", 2024-09-13T13:09:25.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2096999999999998e-05 2024-09-13T13:09:25.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561045+0000", 2024-09-13T13:09:25.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2788999999999995e-05 2024-09-13T13:09:25.162 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.162 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.162 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.162 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83964 7.c 7:37783165:::benchmark_data_smithi052_71865_object83963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:09:25.162 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:09:25.162 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 370.60131149900002, 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0132684729999999, 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83964 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:09:25.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548019+0000", 2024-09-13T13:09:25.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0279999999999999e-06 2024-09-13T13:09:25.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548044+0000", 2024-09-13T13:09:25.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4581999999999999e-05 2024-09-13T13:09:25.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548052+0000", 2024-09-13T13:09:25.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3799999999999994e-06 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548059+0000", 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8329999999999996e-06 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554452+0000", 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0063932119999999997 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:09:25.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554459+0000", 2024-09-13T13:09:25.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3319999999999997e-06 2024-09-13T13:09:25.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552381+0000", 2024-09-13T13:09:25.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99792229399999999 2024-09-13T13:09:25.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552485+0000", 2024-09-13T13:09:25.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010354800000000001 2024-09-13T13:09:25.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552626+0000", 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014122000000000001 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560863+0000", 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0082373700000000008 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561139+0000", 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027546899999999997 2024-09-13T13:09:25.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561192+0000", 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2986999999999999e-05 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561287+0000", 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5217999999999995e-05 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.170 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83985 7.c 7:386b3d8b:::benchmark_data_smithi052_71865_object83984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:09:25.170 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:09:25.170 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 370.581390145, 2024-09-13T13:09:25.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99349958199999999, 2024-09-13T13:09:25.170 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.171 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.171 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.171 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.171 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:25.171 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:25.171 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83985 2024-09-13T13:09:25.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.171 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:09:25.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:09:25.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567942+0000", 2024-09-13T13:09:25.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.888e-06 2024-09-13T13:09:25.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567971+0000", 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9380999999999999e-05 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568031+0000", 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9731000000000001e-05 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568039+0000", 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8109999999999992e-06 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568102+0000", 2024-09-13T13:09:25.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2271000000000001e-05 2024-09-13T13:09:25.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:09:25.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568109+0000", 2024-09-13T13:09:25.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2570000000000001e-06 2024-09-13T13:09:25.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.557963+0000", 2024-09-13T13:09:25.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.989854491 2024-09-13T13:09:25.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.558066+0000", 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102929 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.558206+0000", 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013928000000000001 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561121+0000", 2024-09-13T13:09:25.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029151849999999998 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561301+0000", 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018019299999999999 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561349+0000", 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7639999999999998e-05 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.176 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561439+0000", 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0525000000000005e-05 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83989 7.c 7:3cf9e082:::benchmark_data_smithi052_71865_object83988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 370.57474794199999, 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99086929999999995, 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.177 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83989 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:09:25.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574584+0000", 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0480000000000001e-06 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574603+0000", 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8913e-05 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574615+0000", 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1678000000000001e-05 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574622+0000", 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574674+0000", 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1053000000000001e-05 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574681+0000", 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0110000000000003e-06 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561482+0000", 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98680099200000004 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561622+0000", 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014003799999999999 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561773+0000", 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015180599999999999 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.564308+0000", 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025348390000000001 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565272+0000", 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00096360599999999999 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565362+0000", 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0253000000000005e-05 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565451+0000", 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9090000000000003e-05 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36744 8.f 8:f94e36b9:::benchmark_data_smithi052_101302_object36743:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:09:25.184 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.446006+0000", 2024-09-13T13:09:25.184 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 157.70332403899999, 2024-09-13T13:09:25.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.128763765, 2024-09-13T13:09:25.184 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.184 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.184 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.184 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.184 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.184 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.184 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36744 2024-09-13T13:09:25.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.184 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446006+0000", 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446006+0000", 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446008+0000", 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0949999999999998e-06 2024-09-13T13:09:25.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446034+0000", 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6186999999999998e-05 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446051+0000", 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6549e-05 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446058+0000", 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6480000000000003e-06 2024-09-13T13:09:25.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.448035+0000", 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019772129999999998 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.448146+0000", 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011084199999999999 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.448268+0000", 2024-09-13T13:09:25.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121266 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451458+0000", 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031904899999999998 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.452999+0000", 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001540823 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.574662+0000", 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.121663207 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.574726+0000", 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4369000000000007e-05 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.574770+0000", 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3075999999999999e-05 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.189 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36751 8.0 8:0c704d38:::benchmark_data_smithi052_101302_object36750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.450874+0000", 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 157.69845536400001, 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12726658099999999, 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36751 2024-09-13T13:09:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450874+0000", 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450874+0000", 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450877+0000", 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.159e-06 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450910+0000", 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.345e-05 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450922+0000", 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1912e-05 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450929+0000", 2024-09-13T13:09:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3180000000000001e-06 2024-09-13T13:09:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451081+0000", 2024-09-13T13:09:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015215699999999999 2024-09-13T13:09:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451225+0000", 2024-09-13T13:09:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014390400000000001 2024-09-13T13:09:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:09:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451361+0000", 2024-09-13T13:09:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013605400000000001 2024-09-13T13:09:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454511+0000", 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031497119999999998 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454545+0000", 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3543000000000001e-05 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578050+0000", 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.123505225 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578103+0000", 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2679999999999997e-05 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578141+0000", 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8467000000000003e-05 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36752 8.0 8:0ff7cdfb:::benchmark_data_smithi052_101302_object36751:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:09:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.450957+0000", 2024-09-13T13:09:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 157.69837249099999, 2024-09-13T13:09:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12729238900000001, 2024-09-13T13:09:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36752 2024-09-13T13:09:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.198 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450957+0000", 2024-09-13T13:09:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450957+0000", 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450958+0000", 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1699999999999997e-07 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450980+0000", 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1466000000000001e-05 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450986+0000", 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.516e-06 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450991+0000", 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3090000000000003e-06 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.452820+0000", 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00182898 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.452929+0000", 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108448 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.453078+0000", 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000148663 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455024+0000", 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019467499999999999 2024-09-13T13:09:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455035+0000", 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578177+0000", 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.123141319 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578219+0000", 2024-09-13T13:09:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2348999999999997e-05 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578250+0000", 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0783e-05 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36753 8.f 8:f05f76d5:::benchmark_data_smithi052_101302_object36752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.451293+0000", 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 157.69803645799999, 2024-09-13T13:09:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.123777199, 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36753 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451293+0000", 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451293+0000", 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451294+0000", 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0699999999999999e-06 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451317+0000", 2024-09-13T13:09:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2759e-05 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451325+0000", 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1419999999999993e-06 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451332+0000", 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.63e-06 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451491+0000", 2024-09-13T13:09:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015913900000000001 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451597+0000", 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010614400000000001 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451725+0000", 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012755899999999999 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454052+0000", 2024-09-13T13:09:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002327487 2024-09-13T13:09:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458004+0000", 2024-09-13T13:09:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039520670000000001 2024-09-13T13:09:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.574824+0000", 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11682004999999999 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.575030+0000", 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000205658 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.575071+0000", 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0494000000000002e-05 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36762 8.0 8:04e03c57:::benchmark_data_smithi052_101302_object36761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:09:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.454968+0000", 2024-09-13T13:09:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 157.69436162900001, 2024-09-13T13:09:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12340187399999999, 2024-09-13T13:09:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36762 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454968+0000", 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454968+0000", 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454970+0000", 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.854e-06 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454996+0000", 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5721e-05 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455007+0000", 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1294000000000001e-05 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455014+0000", 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7970000000000003e-06 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455064+0000", 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0141e-05 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455227+0000", 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016291299999999999 2024-09-13T13:09:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455350+0000", 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000122884 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458138+0000", 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027878180000000001 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458328+0000", 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000190352 2024-09-13T13:09:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578285+0000", 2024-09-13T13:09:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11995714 2024-09-13T13:09:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578335+0000", 2024-09-13T13:09:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9969999999999998e-05 2024-09-13T13:09:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578370+0000", 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4990000000000002e-05 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36765 8.f 8:fc16e397:::benchmark_data_smithi052_101302_object36764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.455809+0000", 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 157.69352066100001, 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12131874400000001, 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36765 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455809+0000", 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455809+0000", 2024-09-13T13:09:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455811+0000", 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.452e-06 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455840+0000", 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9050000000000001e-05 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455850+0000", 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.013e-05 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455856+0000", 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6630000000000004e-06 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455915+0000", 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8628999999999997e-05 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.456126+0000", 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021076100000000001 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.456250+0000", 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012409000000000001 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458014+0000", 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0017640080000000001 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458626+0000", 2024-09-13T13:09:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00061228300000000003 2024-09-13T13:09:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.577037+0000", 2024-09-13T13:09:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.118410709 2024-09-13T13:09:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.577084+0000", 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7490000000000001e-05 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.577128+0000", 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3479000000000002e-05 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36768 8.b 8:dd28fad3:::benchmark_data_smithi052_101302_object36767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.458820+0000", 2024-09-13T13:09:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 157.69051010699999, 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.127385796, 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36768 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458820+0000", 2024-09-13T13:09:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458820+0000", 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458821+0000", 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5770000000000001e-06 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458842+0000", 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1013999999999999e-05 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458853+0000", 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0876000000000001e-05 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458871+0000", 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7371999999999999e-05 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458943+0000", 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2386999999999997e-05 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459025+0000", 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1864000000000001e-05 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459148+0000", 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012354900000000001 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.460382+0000", 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012340999999999999 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.460748+0000", 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00036591599999999998 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.586104+0000", 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.125355614 2024-09-13T13:09:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.586163+0000", 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8563999999999997e-05 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.586206+0000", 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2963000000000002e-05 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36769 8.0 8:0a2fab5a:::benchmark_data_smithi052_101302_object36768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.459028+0000", 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 157.690302067, 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12155476799999999, 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36769 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459028+0000", 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459028+0000", 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459029+0000", 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.283e-06 2024-09-13T13:09:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459052+0000", 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3315e-05 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459062+0000", 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2969999999999992e-06 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459069+0000", 2024-09-13T13:09:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1690000000000003e-06 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459120+0000", 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1684999999999997e-05 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459235+0000", 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114246 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:09:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459394+0000", 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015965599999999999 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.460679+0000", 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012848759999999999 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.461249+0000", 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00057000599999999996 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.580496+0000", 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.119247045 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.580546+0000", 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9638000000000001e-05 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.580583+0000", 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6551999999999998e-05 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:61266 8.f 8:fd769e11:::benchmark_data_smithi052_101302_object61265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:03.542853+0000", 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 141.606476423, 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.099333543999999996, 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 61266 2024-09-13T13:09:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542853+0000", 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542853+0000", 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542855+0000", 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9309999999999998e-06 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542879+0000", 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4114999999999999e-05 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542889+0000", 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2229999999999996e-06 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542896+0000", 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8419999999999999e-06 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.542967+0000", 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1124999999999994e-05 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.543087+0000", 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012008200000000001 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:09:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.543210+0000", 2024-09-13T13:09:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012324699999999999 2024-09-13T13:09:25.240 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:09:25.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.241 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.241 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.545041+0000", 2024-09-13T13:09:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001831502 2024-09-13T13:09:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.545546+0000", 2024-09-13T13:09:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00050484300000000005 2024-09-13T13:09:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.641903+0000", 2024-09-13T13:09:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.096356285 2024-09-13T13:09:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.642135+0000", 2024-09-13T13:09:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00023198899999999999 2024-09-13T13:09:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:03.642187+0000", 2024-09-13T13:09:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.236e-05 2024-09-13T13:09:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:79320 8.b 8:df96c156:::benchmark_data_smithi052_101302_object79319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:17.862039+0000", 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 127.28729114799999, 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.093601181000000006, 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 79320 2024-09-13T13:09:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862039+0000", 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862039+0000", 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862039+0000", 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8000000000000005e-07 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862066+0000", 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6327000000000001e-05 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862074+0000", 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1510000000000004e-06 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862080+0000", 2024-09-13T13:09:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3289999999999999e-06 2024-09-13T13:09:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.870063+0000", 2024-09-13T13:09:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0079826040000000008 2024-09-13T13:09:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.870200+0000", 2024-09-13T13:09:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000137293 2024-09-13T13:09:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.870353+0000", 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000152483 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.886551+0000", 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.016198371999999999 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.886588+0000", 2024-09-13T13:09:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7477e-05 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.955109+0000", 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.068520039000000005 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.955586+0000", 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000477231 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.955640+0000", 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4095000000000003e-05 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:79321 8.b 8:da1818e8:::benchmark_data_smithi052_101302_object79320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:17.862090+0000", 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 127.287239525, 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.094540208000000001, 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 79321 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862090+0000", 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862090+0000", 2024-09-13T13:09:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862091+0000", 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862124+0000", 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3192999999999999e-05 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862131+0000", 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6919999999999996e-06 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.862137+0000", 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6899999999999997e-06 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.876441+0000", 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014304292999999999 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.876566+0000", 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012562399999999999 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.876713+0000", 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000146606 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.886806+0000", 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010092983999999999 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.886982+0000", 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017580300000000001 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.956512+0000", 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.069530442999999997 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.956596+0000", 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3609000000000005e-05 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:17.956630+0000", 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4561000000000003e-05 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:100373 8.f 8:f8437ae4:::benchmark_data_smithi052_101302_object100372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:07:36.227085+0000", 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 108.92224446500001, 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.099780015999999999, 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 100373 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227085+0000", 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227085+0000", 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227087+0000", 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.734e-06 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227108+0000", 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0431000000000001e-05 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227120+0000", 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2405999999999999e-05 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227128+0000", 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1100000000000003e-06 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227196+0000", 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8186000000000001e-05 2024-09-13T13:09:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227352+0000", 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015536799999999999 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.227492+0000", 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014054099999999999 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.230182+0000", 2024-09-13T13:09:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002689552 2024-09-13T13:09:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.230542+0000", 2024-09-13T13:09:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000360618 2024-09-13T13:09:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.326764+0000", 2024-09-13T13:09:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.096222012999999995 2024-09-13T13:09:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.326820+0000", 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6169999999999999e-05 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:07:36.326865+0000", 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4886999999999999e-05 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:135978 8.0 8:077a9a64:::benchmark_data_smithi052_101302_object8049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:08.813985+0000", 2024-09-13T13:09:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 76.335344399999997, 2024-09-13T13:09:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19821777300000001, 2024-09-13T13:09:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 135978 2024-09-13T13:09:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.813985+0000", 2024-09-13T13:09:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.813985+0000", 2024-09-13T13:09:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.813987+0000", 2024-09-13T13:09:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7489999999999999e-06 2024-09-13T13:09:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.813992+0000", 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0679999999999996e-06 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814245+0000", 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00025265800000000001 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814253+0000", 2024-09-13T13:09:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7039999999999994e-06 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814370+0000", 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117674 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814476+0000", 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000105457 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:09:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814578+0000", 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102204 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.817977+0000", 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033988999999999998 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819611+0000", 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016344199999999999 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.012080+0000", 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.192468369 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.012156+0000", 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6459999999999996e-05 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.012203+0000", 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7110000000000001e-05 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:135984 8.0 8:0135d23d:::benchmark_data_smithi052_101302_object8055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:08.819395+0000", 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 76.329934395999999, 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.197854312, 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 135984 2024-09-13T13:09:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819395+0000", 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819395+0000", 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819397+0000", 2024-09-13T13:09:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.945e-06 2024-09-13T13:09:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819403+0000", 2024-09-13T13:09:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0739999999999998e-06 2024-09-13T13:09:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819419+0000", 2024-09-13T13:09:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5481999999999998e-05 2024-09-13T13:09:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819427+0000", 2024-09-13T13:09:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7300000000000005e-06 2024-09-13T13:09:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819641+0000", 2024-09-13T13:09:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000214606 2024-09-13T13:09:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819763+0000", 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121311 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819881+0000", 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118528 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.822831+0000", 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002949587 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.822973+0000", 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014218999999999999 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.016953+0000", 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19398000800000001 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.017192+0000", 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000238879 2024-09-13T13:09:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.017250+0000", 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7972e-05 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:135989 8.0 8:0d783973:::benchmark_data_smithi052_101302_object8060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:08.824790+0000", 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 76.324540299000006, 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.204002616, 2024-09-13T13:09:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 135989 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824790+0000", 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824790+0000", 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824792+0000", 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.04e-06 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824797+0000", 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6840000000000001e-06 2024-09-13T13:09:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824813+0000", 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5529999999999999e-05 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824821+0000", 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.259e-06 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824872+0000", 2024-09-13T13:09:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1159999999999998e-05 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824997+0000", 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012481499999999999 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.825118+0000", 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120907 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.828631+0000", 2024-09-13T13:09:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0035128339999999998 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.828649+0000", 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8048000000000001e-05 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.027817+0000", 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19916857399999999 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.028750+0000", 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00093269499999999999 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.028792+0000", 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2070000000000002e-05 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:188581 8.f 8:f81f445b:::benchmark_data_smithi052_101302_object60652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:55.002220+0000", 2024-09-13T13:09:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 30.147109617000002, 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.106845394, 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 188581 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.002220+0000", 2024-09-13T13:09:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.002220+0000", 2024-09-13T13:09:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.002223+0000", 2024-09-13T13:09:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.356e-06 2024-09-13T13:09:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.002229+0000", 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.528e-06 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.002242+0000", 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2582000000000001e-05 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.002250+0000", 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8270000000000004e-06 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.002302+0000", 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1032999999999997e-05 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.002377+0000", 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5746000000000003e-05 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.002459+0000", 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2028999999999997e-05 2024-09-13T13:09:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.004962+0000", 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002502981 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.005026+0000", 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3792000000000002e-05 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.108982+0000", 2024-09-13T13:09:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.103955592 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.109026+0000", 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3955000000000002e-05 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:55.109066+0000", 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9972999999999998e-05 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:25.289 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:09:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:25.289 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:25.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:25.391 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:25.418 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T13:09:25.431 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:09:25.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:25.432 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:25.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:25.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:25 smithi052 ceph-mon[36682]: pgmap v395: 148 pgs: 148 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-13T13:09:25.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:25.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:25 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4148905862' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:09:25.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:25 smithi052 ceph-mon[36682]: from='client.35522 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:09:25.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:25 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3445837525' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:09:25.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:25 smithi052 ceph-mon[29471]: pgmap v395: 148 pgs: 148 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-13T13:09:25.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:25.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:25 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4148905862' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:09:25.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:25 smithi052 ceph-mon[29471]: from='client.35522 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:09:25.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:25 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3445837525' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:09:25.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:25.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:25.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:25.655 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:25.686 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:25.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:25.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:25.788 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:25.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:25 smithi167 ceph-mon[33178]: pgmap v395: 148 pgs: 148 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-13T13:09:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:25 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4148905862' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:09:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:25 smithi167 ceph-mon[33178]: from='client.35522 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:09:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:25 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3445837525' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:09:25.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:25.827 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:25.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:25.928 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:25.964 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:25.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:26.066 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:26.097 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:26.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:26.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:26.199 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:26.234 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:26.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:26.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:26.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:26.368 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:26.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:26.442 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:09:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:09:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:09:26.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:26.505 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:26.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:26.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:26.633 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:26.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:26.734 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:26 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:26 smithi052 ceph-mon[29471]: from='client.35522 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:09:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:26 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3445837525' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:09:26.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:26 smithi052 ceph-mon[29471]: osdmap e229: 8 total, 8 up, 8 in 2024-09-13T13:09:26.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:26 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:26.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:26 smithi052 ceph-mon[36682]: from='client.35522 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:09:26.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:26 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3445837525' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:09:26.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:26 smithi052 ceph-mon[36682]: osdmap e229: 8 total, 8 up, 8 in 2024-09-13T13:09:26.775 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:26.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:26.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:26.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:26 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:26.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:26 smithi167 ceph-mon[33178]: from='client.35522 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:09:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:26 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3445837525' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:09:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:26 smithi167 ceph-mon[33178]: osdmap e229: 8 total, 8 up, 8 in 2024-09-13T13:09:26.877 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:26.910 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:26.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:27.012 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:27.052 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:27.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:27.153 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:27.186 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:27.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:27.288 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:27.323 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:27.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:27.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:27.392 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:09:27.425 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:27.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:27.483 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:27.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:27.584 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:27.622 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:27.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:27.628 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:27 smithi052 ceph-mon[29471]: pgmap v397: 148 pgs: 148 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 692 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-13T13:09:27.628 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:27 smithi052 ceph-mon[36682]: pgmap v397: 148 pgs: 148 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 692 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-13T13:09:27.724 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:27.770 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:27.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:27.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:27.798 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:09:27.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:27 smithi167 ceph-mon[33178]: pgmap v397: 148 pgs: 148 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 692 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-13T13:09:27.874 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:27.909 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:27.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:28.011 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:28.047 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:28.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:28.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:28.149 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:28.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:28.184 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:28.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:28.229 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T13:09:28.286 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:28.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:28.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:28.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:28.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:28 smithi052 ceph-mon[29471]: 5.11 scrub starts 2024-09-13T13:09:28.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:28 smithi052 ceph-mon[29471]: 5.11 scrub ok 2024-09-13T13:09:28.465 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:28.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:28.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:28.551 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:09:28.551 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:09:28.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.551 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:227330 8.5 e229/169 8:ae02ece4:::benchmark_data_smithi052_101302_object99401:head v 229'14454, mlcod=229'14451)", 2024-09-13T13:09:28.551 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:09:28.543729+0000", 2024-09-13T13:09:28.552 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0056794890000000002, 2024-09-13T13:09:28.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005705636, 2024-09-13T13:09:28.552 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:28.552 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:28.552 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:09:28.552 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:28.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:28.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543729+0000", 2024-09-13T13:09:28.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:28.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:28.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543729+0000", 2024-09-13T13:09:28.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:28.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:28.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543730+0000", 2024-09-13T13:09:28.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3200000000000004e-07 2024-09-13T13:09:28.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:28.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543735+0000", 2024-09-13T13:09:28.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4989999999999996e-06 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543742+0000", 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5159999999999997e-06 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543747+0000", 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3600000000000004e-06 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:28.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543834+0000", 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6776999999999999e-05 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543854+0000", 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9772e-05 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:227335 8.5 e229/169 8:a1c679bc:::benchmark_data_smithi052_101302_object99406:head v 229'14455, mlcod=229'14451)", 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:09:28.547397+0000", 2024-09-13T13:09:28.555 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.002011536, 2024-09-13T13:09:28.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002074312, 2024-09-13T13:09:28.556 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:28.556 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:28.556 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:09:28.556 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:28.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:28.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.547397+0000", 2024-09-13T13:09:28.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:28.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:28.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.547397+0000", 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.547398+0000", 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0470000000000001e-06 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.547404+0000", 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5729999999999998e-06 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.547412+0000", 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4290000000000001e-06 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.547418+0000", 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9150000000000001e-06 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.547490+0000", 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1454999999999999e-05 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.547509+0000", 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9517e-05 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:227333 8.2 8:4151e091:::benchmark_data_smithi052_101302_object99404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:09:28.543034+0000", 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0063745870000000001, 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064918930000000003, 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:28.559 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 227333 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543034+0000", 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543034+0000", 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543036+0000", 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2900000000000001e-06 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543043+0000", 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3400000000000003e-06 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543060+0000", 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6806000000000001e-05 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543068+0000", 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1030000000000002e-06 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543124+0000", 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.626e-05 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543228+0000", 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103745 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.543330+0000", 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010197 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.546593+0000", 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032629769999999998 2024-09-13T13:09:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:28.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:28.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:28.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:28.546773+0000", 2024-09-13T13:09:28.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018033799999999999 2024-09-13T13:09:28.564 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:28.564 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:28.564 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:28.564 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:28.564 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:09:28.564 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 3 2024-09-13T13:09:28.564 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:09:28.567 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:28.581 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:28 smithi167 ceph-mon[33178]: 5.11 scrub starts 2024-09-13T13:09:28.581 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:28 smithi167 ceph-mon[33178]: 5.11 scrub ok 2024-09-13T13:09:28.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:28.605 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:28.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:28.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:28.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:28 smithi052 ceph-mon[36682]: 5.11 scrub starts 2024-09-13T13:09:28.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:28 smithi052 ceph-mon[36682]: 5.11 scrub ok 2024-09-13T13:09:28.745 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:28.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:28.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:28.846 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:28.888 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:28.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:28.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:28.961 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:28.977 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:28.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:29.026 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:29.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:29.127 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:29.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:29.174 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:29.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:29.275 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:29.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:29.318 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:29.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:29.419 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:29.457 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:29.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:29.535 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:29 smithi167 ceph-mon[33178]: pgmap v398: 148 pgs: 148 active+clean; 2.1 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 837 op/s 2024-09-13T13:09:29.535 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:29 smithi167 ceph-mon[33178]: 4.2 scrub starts 2024-09-13T13:09:29.535 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:29 smithi167 ceph-mon[33178]: 4.2 scrub ok 2024-09-13T13:09:29.535 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:29 smithi167 ceph-mon[33178]: 4.1e scrub starts 2024-09-13T13:09:29.535 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:29 smithi167 ceph-mon[33178]: 4.1e scrub ok 2024-09-13T13:09:29.535 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:29 smithi167 ceph-mon[33178]: 5.0 scrub starts 2024-09-13T13:09:29.535 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:29 smithi167 ceph-mon[33178]: 5.0 scrub ok 2024-09-13T13:09:29.536 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:29 smithi167 ceph-mon[33178]: 3.1a scrub starts 2024-09-13T13:09:29.536 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:29 smithi167 ceph-mon[33178]: 3.1a scrub ok 2024-09-13T13:09:29.537 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T13:09:29.558 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:29.595 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:29.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:29.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[36682]: pgmap v398: 148 pgs: 148 active+clean; 2.1 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 837 op/s 2024-09-13T13:09:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[36682]: 4.2 scrub starts 2024-09-13T13:09:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[36682]: 4.2 scrub ok 2024-09-13T13:09:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[36682]: 4.1e scrub starts 2024-09-13T13:09:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[36682]: 4.1e scrub ok 2024-09-13T13:09:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[36682]: 5.0 scrub starts 2024-09-13T13:09:29.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[36682]: 5.0 scrub ok 2024-09-13T13:09:29.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[36682]: 3.1a scrub starts 2024-09-13T13:09:29.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[36682]: 3.1a scrub ok 2024-09-13T13:09:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[29471]: pgmap v398: 148 pgs: 148 active+clean; 2.1 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 837 op/s 2024-09-13T13:09:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[29471]: 4.2 scrub starts 2024-09-13T13:09:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[29471]: 4.2 scrub ok 2024-09-13T13:09:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[29471]: 4.1e scrub starts 2024-09-13T13:09:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[29471]: 4.1e scrub ok 2024-09-13T13:09:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[29471]: 5.0 scrub starts 2024-09-13T13:09:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[29471]: 5.0 scrub ok 2024-09-13T13:09:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[29471]: 3.1a scrub starts 2024-09-13T13:09:29.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:29 smithi052 ceph-mon[29471]: 3.1a scrub ok 2024-09-13T13:09:29.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:29.748 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:29.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:29.849 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:29.871 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:29.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:29.894 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:09:29.972 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:30.009 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:30.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:30.111 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:30.152 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:30.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:30.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:30.254 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:30.291 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:30.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:30.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:30.393 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:30.431 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:30.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:30.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:30.487 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:09:30.488 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:09:30.508 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:09:30.508 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:09:30.508 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:09:30.508 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:09:30.508 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:09:30.528 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:09:30.528 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:09:30.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:30.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:30 smithi052 ceph-mon[29471]: 6.9 scrub starts 2024-09-13T13:09:30.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:30 smithi052 ceph-mon[29471]: 6.9 scrub ok 2024-09-13T13:09:30.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:30 smithi052 ceph-mon[29471]: 6.1f scrub starts 2024-09-13T13:09:30.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:30 smithi052 ceph-mon[29471]: 6.1f scrub ok 2024-09-13T13:09:30.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:30 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4061971903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:30.562 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:30 smithi052 ceph-mon[36682]: 6.9 scrub starts 2024-09-13T13:09:30.562 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:30 smithi052 ceph-mon[36682]: 6.9 scrub ok 2024-09-13T13:09:30.562 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:30 smithi052 ceph-mon[36682]: 6.1f scrub starts 2024-09-13T13:09:30.562 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:30 smithi052 ceph-mon[36682]: 6.1f scrub ok 2024-09-13T13:09:30.562 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:30 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4061971903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:30.563 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:09:30.563 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:09:30.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:30.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:30.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:30.611 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:09:30.611 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:09:30.642 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:09:30.642 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:09:30.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:30.674 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:09:30.675 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:09:30.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:30.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:30.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:30.710 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:09:30.710 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:09:30.742 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:09:30.742 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:09:30.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:30.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:30 smithi167 ceph-mon[33178]: 6.9 scrub starts 2024-09-13T13:09:30.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:30 smithi167 ceph-mon[33178]: 6.9 scrub ok 2024-09-13T13:09:30.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:30 smithi167 ceph-mon[33178]: 6.1f scrub starts 2024-09-13T13:09:30.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:30 smithi167 ceph-mon[33178]: 6.1f scrub ok 2024-09-13T13:09:30.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:30 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4061971903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:30.835 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:09:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:30.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:30.957 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:31.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:31.008 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:31.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:31.109 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:31.133 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:31.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:31.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:31.235 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:31.271 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:31.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:31.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:31.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T13:09:31.374 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:31.413 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:31.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:31.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:31.487 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T13:09:31.516 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:31.553 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:31.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:31.654 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:31.689 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:31.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:31.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[29471]: pgmap v399: 148 pgs: 148 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-13T13:09:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[29471]: 3.17 scrub starts 2024-09-13T13:09:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[29471]: 3.17 scrub ok 2024-09-13T13:09:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[29471]: 4.9 scrub starts 2024-09-13T13:09:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[29471]: 4.9 scrub ok 2024-09-13T13:09:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[29471]: 6.a scrub starts 2024-09-13T13:09:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[29471]: 6.a scrub ok 2024-09-13T13:09:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[29471]: 6.15 scrub starts 2024-09-13T13:09:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[29471]: 6.15 scrub ok 2024-09-13T13:09:31.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3851381319' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:09:31.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[36682]: pgmap v399: 148 pgs: 148 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-13T13:09:31.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[36682]: 3.17 scrub starts 2024-09-13T13:09:31.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[36682]: 3.17 scrub ok 2024-09-13T13:09:31.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[36682]: 4.9 scrub starts 2024-09-13T13:09:31.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[36682]: 4.9 scrub ok 2024-09-13T13:09:31.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[36682]: 6.a scrub starts 2024-09-13T13:09:31.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[36682]: 6.a scrub ok 2024-09-13T13:09:31.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[36682]: 6.15 scrub starts 2024-09-13T13:09:31.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[36682]: 6.15 scrub ok 2024-09-13T13:09:31.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:31 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3851381319' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:09:31.746 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:09:31.792 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:31.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:31 smithi167 ceph-mon[33178]: pgmap v399: 148 pgs: 148 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-13T13:09:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:31 smithi167 ceph-mon[33178]: 3.17 scrub starts 2024-09-13T13:09:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:31 smithi167 ceph-mon[33178]: 3.17 scrub ok 2024-09-13T13:09:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:31 smithi167 ceph-mon[33178]: 4.9 scrub starts 2024-09-13T13:09:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:31 smithi167 ceph-mon[33178]: 4.9 scrub ok 2024-09-13T13:09:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:31 smithi167 ceph-mon[33178]: 6.a scrub starts 2024-09-13T13:09:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:31 smithi167 ceph-mon[33178]: 6.a scrub ok 2024-09-13T13:09:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:31 smithi167 ceph-mon[33178]: 6.15 scrub starts 2024-09-13T13:09:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:31 smithi167 ceph-mon[33178]: 6.15 scrub ok 2024-09-13T13:09:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:31 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3851381319' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:09:31.827 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:31.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:31.929 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:31.959 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:31.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:31.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:32.061 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:32.104 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:32.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:32.205 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:32.241 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:32.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:32.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:32.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:32.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:32.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:32.385 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:09:32.385 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:09:32.482 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:32.528 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:32.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:32.630 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:32.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:32.674 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:32.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:32.704 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:32.705 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T13:09:32.775 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:32.796 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:09:32.796 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:09:32.796 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3851381319' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:09:32.796 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[29471]: 3.1e scrub starts 2024-09-13T13:09:32.796 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[29471]: osdmap e230: 8 total, 8 up, 8 in 2024-09-13T13:09:32.796 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[29471]: 3.1e scrub ok 2024-09-13T13:09:32.797 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[29471]: 6.12 scrub starts 2024-09-13T13:09:32.797 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[29471]: 6.12 scrub ok 2024-09-13T13:09:32.797 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:09:32.797 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:09:32.797 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3851381319' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:09:32.797 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[36682]: 3.1e scrub starts 2024-09-13T13:09:32.797 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[36682]: osdmap e230: 8 total, 8 up, 8 in 2024-09-13T13:09:32.797 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[36682]: 3.1e scrub ok 2024-09-13T13:09:32.797 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[36682]: 6.12 scrub starts 2024-09-13T13:09:32.797 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:32 smithi052 ceph-mon[36682]: 6.12 scrub ok 2024-09-13T13:09:32.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:32.806 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:32.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:32 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:09:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:32 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:09:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:32 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3851381319' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:09:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:32 smithi167 ceph-mon[33178]: 3.1e scrub starts 2024-09-13T13:09:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:32 smithi167 ceph-mon[33178]: osdmap e230: 8 total, 8 up, 8 in 2024-09-13T13:09:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:32 smithi167 ceph-mon[33178]: 3.1e scrub ok 2024-09-13T13:09:32.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:32 smithi167 ceph-mon[33178]: 6.12 scrub starts 2024-09-13T13:09:32.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:32 smithi167 ceph-mon[33178]: 6.12 scrub ok 2024-09-13T13:09:32.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:32.939 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:32.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:32.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:33.041 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:33.092 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:33.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:33.193 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:33.212 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T13:09:33.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:33.227 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:33.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:33.328 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:33.379 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:33.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:33.462 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:33.481 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:33.505 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:33 smithi167 ceph-mon[33178]: pgmap v401: 148 pgs: 148 active+clean; 1.7 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 958 op/s 2024-09-13T13:09:33.505 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:33 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:09:33.505 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:33 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:09:33.505 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:33 smithi167 ceph-mon[33178]: 5.13 scrub starts 2024-09-13T13:09:33.506 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:33 smithi167 ceph-mon[33178]: 5.13 scrub ok 2024-09-13T13:09:33.506 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:33 smithi167 ceph-mon[33178]: osdmap e231: 8 total, 8 up, 8 in 2024-09-13T13:09:33.506 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:33 smithi167 ceph-mon[33178]: 4.a scrub starts 2024-09-13T13:09:33.506 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:33 smithi167 ceph-mon[33178]: 4.a scrub ok 2024-09-13T13:09:33.506 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:33 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:09:33.518 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:33.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:33.599 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:09:33.620 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:33.659 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:09:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:33.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:33.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:33.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[29471]: pgmap v401: 148 pgs: 148 active+clean; 1.7 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 958 op/s 2024-09-13T13:09:33.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:09:33.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:09:33.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[29471]: 5.13 scrub starts 2024-09-13T13:09:33.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[29471]: 5.13 scrub ok 2024-09-13T13:09:33.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[29471]: osdmap e231: 8 total, 8 up, 8 in 2024-09-13T13:09:33.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[29471]: 4.a scrub starts 2024-09-13T13:09:33.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[29471]: 4.a scrub ok 2024-09-13T13:09:33.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:09:33.759 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[36682]: pgmap v401: 148 pgs: 148 active+clean; 1.7 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 958 op/s 2024-09-13T13:09:33.759 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:09:33.759 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:09:33.759 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[36682]: 5.13 scrub starts 2024-09-13T13:09:33.759 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[36682]: 5.13 scrub ok 2024-09-13T13:09:33.759 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[36682]: osdmap e231: 8 total, 8 up, 8 in 2024-09-13T13:09:33.759 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[36682]: 4.a scrub starts 2024-09-13T13:09:33.759 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[36682]: 4.a scrub ok 2024-09-13T13:09:33.759 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:33 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:09:33.766 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:33.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:33.815 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:33.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:33.916 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:33.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:33.969 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:33.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:34.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:34.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:09:33 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:09:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:34.101 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:34.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:34.198 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:34.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:34.235 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:34.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:34.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:34.241 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:09:34.241 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:09:34.337 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:34.368 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:34.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:34.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:34.501 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:34.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:34.521 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T13:09:34.528 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:09:34.590 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:34 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2565977989' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:34.590 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:34 smithi052 ceph-mon[36682]: from='client.35546 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:34.590 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:34 smithi052 ceph-mon[36682]: 4.8 scrub starts 2024-09-13T13:09:34.591 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:34 smithi052 ceph-mon[36682]: 4.8 scrub ok 2024-09-13T13:09:34.591 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:34 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1596522462' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:09:34.591 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:34 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2565977989' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:34.591 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:34 smithi052 ceph-mon[29471]: from='client.35546 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:34.591 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:34 smithi052 ceph-mon[29471]: 4.8 scrub starts 2024-09-13T13:09:34.591 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:34 smithi052 ceph-mon[29471]: 4.8 scrub ok 2024-09-13T13:09:34.591 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:34 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1596522462' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:09:34.602 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:34.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:34.733 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:34.777 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:34.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:34.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:34.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:34 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2565977989' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:34.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:34 smithi167 ceph-mon[33178]: from='client.35546 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:34.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:34 smithi167 ceph-mon[33178]: 4.8 scrub starts 2024-09-13T13:09:34.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:34 smithi167 ceph-mon[33178]: 4.8 scrub ok 2024-09-13T13:09:34.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:34 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1596522462' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:09:34.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:34.913 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:34.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:35.001 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:09:35.001 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:09:35.001 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:09:35.001 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:09:35.002 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:09:35.002 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:09:35.002 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:09:35.002 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:09:35.002 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:09:35.002 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:09:35.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T13:09:35.014 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:35.021 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:09:35.021 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:09:35.051 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:35.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:35.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:35.055 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:09:35.056 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:09:35.092 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:09:35.092 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:09:35.153 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:35.192 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:35.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:35.294 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:35.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:35.334 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:35.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:35.399 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:09:35.435 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:35.471 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:35.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:35.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:35.573 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:35.611 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:35.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:35.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:35.713 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:35.747 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:35.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:35.765 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:35 smithi167 ceph-mon[33178]: pgmap v403: 148 pgs: 148 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-13T13:09:35.765 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:35 smithi167 ceph-mon[33178]: 8.3 scrub starts 2024-09-13T13:09:35.765 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:35.765 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:35 smithi167 ceph-mon[33178]: 8.3 scrub ok 2024-09-13T13:09:35.765 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:35 smithi167 ceph-mon[33178]: 4.16 scrub starts 2024-09-13T13:09:35.765 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:35 smithi167 ceph-mon[33178]: 4.16 scrub ok 2024-09-13T13:09:35.766 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:35 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:09:35.766 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:35 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:09:35.766 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:35 smithi167 ceph-mon[33178]: from='client.35546 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:09:35.766 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:35 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1596522462' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:09:35.766 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:35 smithi167 ceph-mon[33178]: osdmap e232: 8 total, 8 up, 8 in 2024-09-13T13:09:35.766 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:35 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/812742417' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:35.773 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[29471]: pgmap v403: 148 pgs: 148 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-13T13:09:35.773 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[29471]: 8.3 scrub starts 2024-09-13T13:09:35.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:35.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[29471]: 8.3 scrub ok 2024-09-13T13:09:35.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[29471]: 4.16 scrub starts 2024-09-13T13:09:35.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[29471]: 4.16 scrub ok 2024-09-13T13:09:35.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:09:35.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:09:35.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[29471]: from='client.35546 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:09:35.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1596522462' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:09:35.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[29471]: osdmap e232: 8 total, 8 up, 8 in 2024-09-13T13:09:35.774 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/812742417' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:35.774 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[36682]: pgmap v403: 148 pgs: 148 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-13T13:09:35.774 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[36682]: 8.3 scrub starts 2024-09-13T13:09:35.775 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:35.775 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[36682]: 8.3 scrub ok 2024-09-13T13:09:35.775 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[36682]: 4.16 scrub starts 2024-09-13T13:09:35.775 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[36682]: 4.16 scrub ok 2024-09-13T13:09:35.775 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:09:35.775 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:09:35.775 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[36682]: from='client.35546 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:09:35.775 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1596522462' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:09:35.775 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[36682]: osdmap e232: 8 total, 8 up, 8 in 2024-09-13T13:09:35.775 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:35 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/812742417' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:35.820 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T13:09:35.821 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:09:35.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:35.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:35.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:35.983 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:36.030 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:36.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:36.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:36.159 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:36.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:36.260 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:36.309 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:36.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:36.411 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:36.447 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:36.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:36.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:36.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:09:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:09:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:09:36.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:36.575 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:36.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:36.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:36.677 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:36.691 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:09:36.691 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:09:36.692 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:09:36.692 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:09:36.692 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:09:36.709 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:36.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:36.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:36.810 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:36.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:36 smithi167 ceph-mon[33178]: 3.18 scrub starts 2024-09-13T13:09:36.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:36 smithi167 ceph-mon[33178]: 3.18 scrub ok 2024-09-13T13:09:36.848 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:36.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:36.892 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:36 smithi052 ceph-mon[36682]: 3.18 scrub starts 2024-09-13T13:09:36.892 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:36 smithi052 ceph-mon[36682]: 3.18 scrub ok 2024-09-13T13:09:36.893 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:36 smithi052 ceph-mon[29471]: 3.18 scrub starts 2024-09-13T13:09:36.893 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:36 smithi052 ceph-mon[29471]: 3.18 scrub ok 2024-09-13T13:09:36.950 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:36.979 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:36.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:36.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:37.074 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:37.081 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:37.116 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:37.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:37.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:37.247 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:37.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:37.348 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:37.387 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:37.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:37.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T13:09:37.488 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:37.515 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:37.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:37.616 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:37.660 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:37.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:37.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:37.762 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:37.784 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:37.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:37.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:37.793 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:09:37.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:37 smithi167 ceph-mon[33178]: pgmap v405: 148 pgs: 148 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 914 op/s 2024-09-13T13:09:37.823 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:37 smithi167 ceph-mon[33178]: 6.16 scrub starts 2024-09-13T13:09:37.823 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:37 smithi167 ceph-mon[33178]: 6.16 scrub ok 2024-09-13T13:09:37.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:37.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:37.921 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:37.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:37.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:37 smithi052 ceph-mon[36682]: pgmap v405: 148 pgs: 148 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 914 op/s 2024-09-13T13:09:37.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:37 smithi052 ceph-mon[36682]: 6.16 scrub starts 2024-09-13T13:09:37.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:37 smithi052 ceph-mon[36682]: 6.16 scrub ok 2024-09-13T13:09:37.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:37 smithi052 ceph-mon[29471]: pgmap v405: 148 pgs: 148 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 914 op/s 2024-09-13T13:09:37.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:37 smithi052 ceph-mon[29471]: 6.16 scrub starts 2024-09-13T13:09:37.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:37 smithi052 ceph-mon[29471]: 6.16 scrub ok 2024-09-13T13:09:38.023 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:38.061 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:38.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:38.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:38.184 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:38.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:38.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:38.263 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:09:38.286 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:38.354 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:38.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:38.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:38.434 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:09:38.435 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:09:38.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:09:38.435 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:09:38.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.435 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:105216 7.0 7:0c7da74e:::benchmark_data_smithi052_71865_object105215:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-13T13:09:38.435 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:36.018754+0000", 2024-09-13T13:09:38.435 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 362.413597944, 2024-09-13T13:09:38.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073909959999999997, 2024-09-13T13:09:38.435 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.435 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.435 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.435 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.436 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:38.436 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:38.436 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105216 2024-09-13T13:09:38.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.436 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018754+0000", 2024-09-13T13:09:38.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018754+0000", 2024-09-13T13:09:38.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018756+0000", 2024-09-13T13:09:38.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.711e-06 2024-09-13T13:09:38.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018808+0000", 2024-09-13T13:09:38.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1094000000000002e-05 2024-09-13T13:09:38.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018826+0000", 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8502e-05 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018834+0000", 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8099999999999998e-06 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.018901+0000", 2024-09-13T13:09:38.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6760000000000005e-05 2024-09-13T13:09:38.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.019041+0000", 2024-09-13T13:09:38.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140474 2024-09-13T13:09:38.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:09:38.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.019196+0000", 2024-09-13T13:09:38.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015521 2024-09-13T13:09:38.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.022153+0000", 2024-09-13T13:09:38.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029568210000000001 2024-09-13T13:09:38.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.023370+0000", 2024-09-13T13:09:38.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001216658 2024-09-13T13:09:38.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.092552+0000", 2024-09-13T13:09:38.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069182612000000004 2024-09-13T13:09:38.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.092613+0000", 2024-09-13T13:09:38.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0387999999999998e-05 2024-09-13T13:09:38.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:36.092664+0000", 2024-09-13T13:09:38.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.092e-05 2024-09-13T13:09:38.441 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.441 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.441 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.442 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:119489 7.0 7:0001c7f5:::benchmark_data_smithi052_71865_object119488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:09:38.442 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:50.480010+0000", 2024-09-13T13:09:38.442 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 347.95234210799998, 2024-09-13T13:09:38.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076898633999999993, 2024-09-13T13:09:38.442 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.442 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.442 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.442 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.443 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:38.443 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:38.443 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 119489 2024-09-13T13:09:38.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.443 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480010+0000", 2024-09-13T13:09:38.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480010+0000", 2024-09-13T13:09:38.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480020+0000", 2024-09-13T13:09:38.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.057e-05 2024-09-13T13:09:38.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480071+0000", 2024-09-13T13:09:38.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1051999999999998e-05 2024-09-13T13:09:38.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480083+0000", 2024-09-13T13:09:38.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1678000000000001e-05 2024-09-13T13:09:38.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480099+0000", 2024-09-13T13:09:38.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6127e-05 2024-09-13T13:09:38.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480149+0000", 2024-09-13T13:09:38.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0058000000000001e-05 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480303+0000", 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015380799999999999 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.480458+0000", 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015522299999999999 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.483021+0000", 2024-09-13T13:09:38.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025633510000000002 2024-09-13T13:09:38.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.483702+0000", 2024-09-13T13:09:38.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00068071600000000005 2024-09-13T13:09:38.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.556804+0000", 2024-09-13T13:09:38.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073102271999999996 2024-09-13T13:09:38.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.556863+0000", 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8072999999999998e-05 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:50.556908+0000", 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5705999999999999e-05 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:35132 8.8 8:18a4857a:::benchmark_data_smithi052_101302_object35131:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:09:38.448 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:46.489370+0000", 2024-09-13T13:09:38.449 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 171.94298165399999, 2024-09-13T13:09:38.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079236983999999996, 2024-09-13T13:09:38.449 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.449 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.449 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.449 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.449 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:38.449 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:38.449 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35132 2024-09-13T13:09:38.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.449 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489370+0000", 2024-09-13T13:09:38.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489370+0000", 2024-09-13T13:09:38.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489372+0000", 2024-09-13T13:09:38.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4440000000000001e-06 2024-09-13T13:09:38.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489417+0000", 2024-09-13T13:09:38.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5073e-05 2024-09-13T13:09:38.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489432+0000", 2024-09-13T13:09:38.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5109999999999999e-05 2024-09-13T13:09:38.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489440+0000", 2024-09-13T13:09:38.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9200000000000004e-06 2024-09-13T13:09:38.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489495+0000", 2024-09-13T13:09:38.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4907999999999997e-05 2024-09-13T13:09:38.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489589+0000", 2024-09-13T13:09:38.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4401000000000006e-05 2024-09-13T13:09:38.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.489799+0000", 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020981400000000001 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491453+0000", 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016542029999999999 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.495446+0000", 2024-09-13T13:09:38.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039925960000000002 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.568544+0000", 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073098053999999996 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.568578+0000", 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4788e-05 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.568607+0000", 2024-09-13T13:09:38.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8673000000000001e-05 2024-09-13T13:09:38.455 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.455 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.455 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.455 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:35138 8.8 8:1549287f:::benchmark_data_smithi052_101302_object35137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:09:38.455 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:46.491520+0000", 2024-09-13T13:09:38.456 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 171.94083211, 2024-09-13T13:09:38.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079477935, 2024-09-13T13:09:38.456 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.456 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.456 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.456 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.456 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:38.456 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:38.456 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35138 2024-09-13T13:09:38.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491520+0000", 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491520+0000", 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491521+0000", 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4640000000000001e-06 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491554+0000", 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2861999999999998e-05 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491564+0000", 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0504e-05 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491572+0000", 2024-09-13T13:09:38.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2389999999999996e-06 2024-09-13T13:09:38.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491614+0000", 2024-09-13T13:09:38.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1876999999999999e-05 2024-09-13T13:09:38.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491727+0000", 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113761 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.491860+0000", 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000132736 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.495482+0000", 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036219070000000002 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.495565+0000", 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3444999999999998e-05 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.570922+0000", 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075356970999999995 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.570958+0000", 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6022999999999997e-05 2024-09-13T13:09:38.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.570998+0000", 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9146000000000001e-05 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:35140 8.8 8:14c93e84:::benchmark_data_smithi052_101302_object35139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:46.492075+0000", 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 171.940276864, 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079130470999999994, 2024-09-13T13:09:38.462 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35140 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.492075+0000", 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.492075+0000", 2024-09-13T13:09:38.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.464 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:38.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.492076+0000", 2024-09-13T13:09:38.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1739999999999999e-06 2024-09-13T13:09:38.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.492177+0000", 2024-09-13T13:09:38.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101215 2024-09-13T13:09:38.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.492187+0000", 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0064999999999999e-05 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.492194+0000", 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1470000000000004e-06 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.493942+0000", 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017474929999999999 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.494068+0000", 2024-09-13T13:09:38.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012629500000000001 2024-09-13T13:09:38.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:38.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.494213+0000", 2024-09-13T13:09:38.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144831 2024-09-13T13:09:38.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.496226+0000", 2024-09-13T13:09:38.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020130259999999998 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.496557+0000", 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000330534 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.571103+0000", 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.074546442000000004 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.571154+0000", 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0797000000000002e-05 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:46.571205+0000", 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1452000000000001e-05 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:36962 8.8 8:101fb546:::benchmark_data_smithi052_101302_object36961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:47.664243+0000", 2024-09-13T13:09:38.470 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 170.76810866100001, 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072421352999999994, 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 36962 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.664243+0000", 2024-09-13T13:09:38.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.664243+0000", 2024-09-13T13:09:38.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.664245+0000", 2024-09-13T13:09:38.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.854e-06 2024-09-13T13:09:38.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.664285+0000", 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9852999999999999e-05 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.727378+0000", 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063093039000000004 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.727389+0000", 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1085e-05 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.727448+0000", 2024-09-13T13:09:38.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8802999999999999e-05 2024-09-13T13:09:38.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.727578+0000", 2024-09-13T13:09:38.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013015100000000001 2024-09-13T13:09:38.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:38.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.727715+0000", 2024-09-13T13:09:38.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000137033 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.731218+0000", 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003503277 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.734833+0000", 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036148880000000001 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.736587+0000", 2024-09-13T13:09:38.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017542949999999999 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.736629+0000", 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2005999999999997e-05 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.736664+0000", 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5068999999999999e-05 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:60517 8.8 8:1266f88d:::benchmark_data_smithi052_101302_object60516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:02.991489+0000", 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 155.440862364, 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.078013031999999996, 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 60517 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.477 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991489+0000", 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991489+0000", 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991491+0000", 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.562e-06 2024-09-13T13:09:38.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991521+0000", 2024-09-13T13:09:38.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0216999999999999e-05 2024-09-13T13:09:38.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991532+0000", 2024-09-13T13:09:38.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0873000000000001e-05 2024-09-13T13:09:38.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991539+0000", 2024-09-13T13:09:38.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9290000000000001e-06 2024-09-13T13:09:38.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991600+0000", 2024-09-13T13:09:38.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0556000000000001e-05 2024-09-13T13:09:38.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991772+0000", 2024-09-13T13:09:38.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017275400000000001 2024-09-13T13:09:38.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.991867+0000", 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4965999999999998e-05 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.993734+0000", 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018671370000000001 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.993945+0000", 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021104000000000001 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.069398+0000", 2024-09-13T13:09:38.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075452965999999996 2024-09-13T13:09:38.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.069456+0000", 2024-09-13T13:09:38.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7336999999999997e-05 2024-09-13T13:09:38.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.069502+0000", 2024-09-13T13:09:38.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6694999999999999e-05 2024-09-13T13:09:38.483 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.483 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.483 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.483 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:79330 8.8 8:1fb9b2c0:::benchmark_data_smithi052_101302_object79329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:09:38.483 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:17.879136+0000", 2024-09-13T13:09:38.483 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.55321578799999, 2024-09-13T13:09:38.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.07979, 2024-09-13T13:09:38.483 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.483 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.483 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.483 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.484 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:38.484 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:38.484 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 79330 2024-09-13T13:09:38.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.484 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879136+0000", 2024-09-13T13:09:38.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879136+0000", 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879138+0000", 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0839999999999999e-06 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879178+0000", 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0398999999999997e-05 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879192+0000", 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3190000000000001e-05 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879199+0000", 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3159999999999999e-06 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879266+0000", 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7255999999999994e-05 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879415+0000", 2024-09-13T13:09:38.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014886899999999999 2024-09-13T13:09:38.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:38.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879550+0000", 2024-09-13T13:09:38.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013511900000000001 2024-09-13T13:09:38.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.884904+0000", 2024-09-13T13:09:38.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0053537819999999996 2024-09-13T13:09:38.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.884922+0000", 2024-09-13T13:09:38.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7588000000000002e-05 2024-09-13T13:09:38.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.958827+0000", 2024-09-13T13:09:38.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073905549000000001 2024-09-13T13:09:38.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.958883+0000", 2024-09-13T13:09:38.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5890000000000002e-05 2024-09-13T13:09:38.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.958926+0000", 2024-09-13T13:09:38.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2957999999999998e-05 2024-09-13T13:09:38.489 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.489 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.489 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.490 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:79331 8.8 8:13289dff:::benchmark_data_smithi052_101302_object79330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:09:38.490 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:17.879266+0000", 2024-09-13T13:09:38.490 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.55308538400001, 2024-09-13T13:09:38.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084187697000000006, 2024-09-13T13:09:38.490 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.490 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.490 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.490 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.490 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:38.490 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 79331 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879266+0000", 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879266+0000", 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879268+0000", 2024-09-13T13:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.454e-06 2024-09-13T13:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879330+0000", 2024-09-13T13:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2682999999999996e-05 2024-09-13T13:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879339+0000", 2024-09-13T13:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6389999999999998e-06 2024-09-13T13:09:38.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879344+0000", 2024-09-13T13:09:38.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.977e-06 2024-09-13T13:09:38.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.882487+0000", 2024-09-13T13:09:38.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003142677 2024-09-13T13:09:38.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.882606+0000", 2024-09-13T13:09:38.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011966 2024-09-13T13:09:38.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:38.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.882740+0000", 2024-09-13T13:09:38.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013381299999999999 2024-09-13T13:09:38.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.885073+0000", 2024-09-13T13:09:38.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023330849999999999 2024-09-13T13:09:38.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.885588+0000", 2024-09-13T13:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051493199999999995 2024-09-13T13:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.963389+0000", 2024-09-13T13:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077800367999999995 2024-09-13T13:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.963428+0000", 2024-09-13T13:09:38.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9514999999999997e-05 2024-09-13T13:09:38.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.963454+0000", 2024-09-13T13:09:38.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5894e-05 2024-09-13T13:09:38.496 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.496 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.496 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.496 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:79349 8.8 8:18091f7f:::benchmark_data_smithi052_101302_object79348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:09:38.496 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:17.895616+0000", 2024-09-13T13:09:38.497 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.536735343, 2024-09-13T13:09:38.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076369714000000005, 2024-09-13T13:09:38.497 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.497 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.497 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.497 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.497 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:38.497 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:38.497 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 79349 2024-09-13T13:09:38.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.497 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895616+0000", 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895616+0000", 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895618+0000", 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1200000000000001e-06 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895672+0000", 2024-09-13T13:09:38.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4212000000000002e-05 2024-09-13T13:09:38.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895681+0000", 2024-09-13T13:09:38.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1859999999999998e-06 2024-09-13T13:09:38.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895686+0000", 2024-09-13T13:09:38.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9339999999999997e-06 2024-09-13T13:09:38.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895731+0000", 2024-09-13T13:09:38.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5485999999999998e-05 2024-09-13T13:09:38.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895828+0000", 2024-09-13T13:09:38.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6967999999999997e-05 2024-09-13T13:09:38.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:38.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.895946+0000", 2024-09-13T13:09:38.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118051 2024-09-13T13:09:38.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.898138+0000", 2024-09-13T13:09:38.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021917960000000002 2024-09-13T13:09:38.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.898536+0000", 2024-09-13T13:09:38.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000398205 2024-09-13T13:09:38.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.971907+0000", 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073370608000000004 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.971943+0000", 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6480000000000003e-05 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.971986+0000", 2024-09-13T13:09:38.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2667999999999999e-05 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:79352 8.8 8:126f2653:::benchmark_data_smithi052_101302_object79351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:17.899336+0000", 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.53301574, 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072801497000000007, 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 79352 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899336+0000", 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899336+0000", 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899337+0000", 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.437e-06 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899399+0000", 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1346999999999994e-05 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899414+0000", 2024-09-13T13:09:38.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5511000000000001e-05 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899421+0000", 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7819999999999996e-06 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899472+0000", 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1097999999999998e-05 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899569+0000", 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6386999999999997e-05 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899656+0000", 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7394999999999999e-05 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.902850+0000", 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031939540000000002 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.902890+0000", 2024-09-13T13:09:38.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0330999999999997e-05 2024-09-13T13:09:38.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.972040+0000", 2024-09-13T13:09:38.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069149585999999999 2024-09-13T13:09:38.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.972097+0000", 2024-09-13T13:09:38.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7182000000000002e-05 2024-09-13T13:09:38.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.972137+0000", 2024-09-13T13:09:38.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0487e-05 2024-09-13T13:09:38.509 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.509 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.509 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.509 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:79353 8.8 8:16cfcad1:::benchmark_data_smithi052_101302_object79352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:09:38.509 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:17.899442+0000", 2024-09-13T13:09:38.509 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.53290991399999, 2024-09-13T13:09:38.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072783257000000004, 2024-09-13T13:09:38.509 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 79353 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899442+0000", 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899442+0000", 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899443+0000", 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9800000000000003e-07 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899472+0000", 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9139999999999999e-05 2024-09-13T13:09:38.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899478+0000", 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2639999999999997e-06 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.899482+0000", 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.602e-06 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.901362+0000", 2024-09-13T13:09:38.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001880659 2024-09-13T13:09:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.901475+0000", 2024-09-13T13:09:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112524 2024-09-13T13:09:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.901590+0000", 2024-09-13T13:09:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011524899999999999 2024-09-13T13:09:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.902996+0000", 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001406399 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.903148+0000", 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015181400000000001 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.972166+0000", 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069017700000000001 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.972195+0000", 2024-09-13T13:09:38.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9385000000000001e-05 2024-09-13T13:09:38.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.972225+0000", 2024-09-13T13:09:38.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9723e-05 2024-09-13T13:09:38.515 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.515 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.515 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.516 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135962 8.3 e207/169 8:c288bb32:::benchmark_data_smithi052_101302_object8033:head v 207'8600, mlcod=207'8598)", 2024-09-13T13:09:38.516 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.798736+0000", 2024-09-13T13:09:38.516 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.633615749000001, 2024-09-13T13:09:38.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.214192363, 2024-09-13T13:09:38.516 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.516 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.516 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.516 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798736+0000", 2024-09-13T13:09:38.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798736+0000", 2024-09-13T13:09:38.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798738+0000", 2024-09-13T13:09:38.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.542e-06 2024-09-13T13:09:38.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798746+0000", 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7299999999999994e-06 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798754+0000", 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7770000000000002e-06 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798760+0000", 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7769999999999999e-06 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798809+0000", 2024-09-13T13:09:38.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8714000000000001e-05 2024-09-13T13:09:38.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798822+0000", 2024-09-13T13:09:38.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3737000000000001e-05 2024-09-13T13:09:38.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012889+0000", 2024-09-13T13:09:38.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.21406626500000001 2024-09-13T13:09:38.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012928+0000", 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9820999999999997e-05 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:135964 8.8 8:1e05dc47:::benchmark_data_smithi052_101302_object8035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.800142+0000", 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.632209653999993, 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20392829400000001, 2024-09-13T13:09:38.520 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135964 2024-09-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800142+0000", 2024-09-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800142+0000", 2024-09-13T13:09:38.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800144+0000", 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5719999999999999e-06 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800149+0000", 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2059999999999998e-06 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800159+0000", 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9110000000000005e-06 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800165+0000", 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6640000000000001e-06 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800242+0000", 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6415999999999995e-05 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800369+0000", 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000127238 2024-09-13T13:09:38.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800490+0000", 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120827 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803447+0000", 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029568620000000002 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803750+0000", 2024-09-13T13:09:38.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030363800000000001 2024-09-13T13:09:38.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003963+0000", 2024-09-13T13:09:38.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200212532 2024-09-13T13:09:38.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004013+0000", 2024-09-13T13:09:38.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9765999999999998e-05 2024-09-13T13:09:38.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004070+0000", 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7661999999999998e-05 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:135967 8.4 8:2cd9f03c:::benchmark_data_smithi052_101302_object8038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.800185+0000", 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.632166311999995, 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20402237600000001, 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135967 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800185+0000", 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800185+0000", 2024-09-13T13:09:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800186+0000", 2024-09-13T13:09:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8999999999999996e-07 2024-09-13T13:09:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800189+0000", 2024-09-13T13:09:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.689e-06 2024-09-13T13:09:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800194+0000", 2024-09-13T13:09:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1309999999999997e-06 2024-09-13T13:09:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800199+0000", 2024-09-13T13:09:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6959999999999996e-06 2024-09-13T13:09:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800250+0000", 2024-09-13T13:09:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1402e-05 2024-09-13T13:09:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800389+0000", 2024-09-13T13:09:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000138596 2024-09-13T13:09:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:09:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800490+0000", 2024-09-13T13:09:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010134899999999999 2024-09-13T13:09:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803436+0000", 2024-09-13T13:09:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029464529999999999 2024-09-13T13:09:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004082+0000", 2024-09-13T13:09:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200645886 2024-09-13T13:09:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004122+0000", 2024-09-13T13:09:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9864000000000003e-05 2024-09-13T13:09:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004169+0000", 2024-09-13T13:09:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7052000000000003e-05 2024-09-13T13:09:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004208+0000", 2024-09-13T13:09:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8668000000000003e-05 2024-09-13T13:09:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135966 8.3 e207/169 8:cd1f5880:::benchmark_data_smithi052_101302_object8037:head v 207'8601, mlcod=207'8598)", 2024-09-13T13:09:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.802377+0000", 2024-09-13T13:09:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.629975056999996, 2024-09-13T13:09:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.21058527299999999, 2024-09-13T13:09:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802377+0000", 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802377+0000", 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802379+0000", 2024-09-13T13:09:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.827e-06 2024-09-13T13:09:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802390+0000", 2024-09-13T13:09:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1623999999999999e-05 2024-09-13T13:09:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802402+0000", 2024-09-13T13:09:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1564e-05 2024-09-13T13:09:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802410+0000", 2024-09-13T13:09:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2719999999999997e-06 2024-09-13T13:09:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802483+0000", 2024-09-13T13:09:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2840000000000001e-05 2024-09-13T13:09:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802503+0000", 2024-09-13T13:09:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0452e-05 2024-09-13T13:09:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012933+0000", 2024-09-13T13:09:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.210429965 2024-09-13T13:09:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012962+0000", 2024-09-13T13:09:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8728999999999999e-05 2024-09-13T13:09:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135968 8.2 e207/169 8:4f16015f:::benchmark_data_smithi052_101302_object8039:head v 207'8502, mlcod=207'8500)", 2024-09-13T13:09:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.803228+0000", 2024-09-13T13:09:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.629124039999994, 2024-09-13T13:09:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20068414800000001, 2024-09-13T13:09:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803228+0000", 2024-09-13T13:09:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803228+0000", 2024-09-13T13:09:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803229+0000", 2024-09-13T13:09:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2399999999999996e-07 2024-09-13T13:09:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803234+0000", 2024-09-13T13:09:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6400000000000002e-06 2024-09-13T13:09:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803244+0000", 2024-09-13T13:09:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6770000000000007e-06 2024-09-13T13:09:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803251+0000", 2024-09-13T13:09:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0149999999999998e-06 2024-09-13T13:09:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803344+0000", 2024-09-13T13:09:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2656000000000002e-05 2024-09-13T13:09:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803370+0000", 2024-09-13T13:09:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6156000000000002e-05 2024-09-13T13:09:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003858+0000", 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20048846400000001 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003912+0000", 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3616000000000003e-05 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.25222.0:135964 8.8 e207/169 ondisk, result = 0)", 2024-09-13T13:09:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.803563+0000", 2024-09-13T13:09:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.628788689999993, 2024-09-13T13:09:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200512417, 2024-09-13T13:09:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:09:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803563+0000", 2024-09-13T13:09:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803563+0000", 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.804856+0000", 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001293055 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.804861+0000", 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5800000000000002e-06 2024-09-13T13:09:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003842+0000", 2024-09-13T13:09:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19898099699999999 2024-09-13T13:09:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003857+0000", 2024-09-13T13:09:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5021999999999999e-05 2024-09-13T13:09:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003949+0000", 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1824999999999996e-05 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003956+0000", 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9070000000000007e-06 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004075+0000", 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011903100000000001 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.25222.0:135967 8.4 e207/169 ondisk, result = 0)", 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.803807+0000", 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.628544309999995, 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200283038, 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803807+0000", 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803807+0000", 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803808+0000", 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.032e-06 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803812+0000", 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0069999999999999e-06 2024-09-13T13:09:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803820+0000", 2024-09-13T13:09:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5599999999999996e-06 2024-09-13T13:09:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003847+0000", 2024-09-13T13:09:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20002690300000001 2024-09-13T13:09:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004067+0000", 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021978000000000001 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004075+0000", 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6430000000000001e-06 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004090+0000", 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5113000000000001e-05 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135970 8.3 e207/169 8:ce0cc5da:::benchmark_data_smithi052_101302_object8041:head v 207'8602, mlcod=207'8598)", 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.806702+0000", 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.625649276999994, 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20628901199999999, 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806702+0000", 2024-09-13T13:09:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806702+0000", 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806703+0000", 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0109999999999999e-06 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806711+0000", 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.52e-06 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003876+0000", 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19716455899999999 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003884+0000", 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2269999999999993e-06 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003930+0000", 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6368000000000001e-05 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003955+0000", 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4374000000000001e-05 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012964+0000", 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090092720000000005 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012991+0000", 2024-09-13T13:09:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7681000000000001e-05 2024-09-13T13:09:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:38.559 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:09:38.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:38.560 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:38.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:38.661 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:38.700 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:38.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:38.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:38.836 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:38.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:38.938 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:38.940 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:09:38.967 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:38.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:39.069 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:39.099 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:39.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:39.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:39.201 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:39.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:38 smithi052 ceph-mon[29471]: 4.17 scrub starts 2024-09-13T13:09:39.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:38 smithi052 ceph-mon[29471]: 4.17 scrub ok 2024-09-13T13:09:39.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:38 smithi052 ceph-mon[29471]: 6.e scrub starts 2024-09-13T13:09:39.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:38 smithi052 ceph-mon[29471]: pgmap v406: 148 pgs: 148 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 877 op/s 2024-09-13T13:09:39.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:38 smithi052 ceph-mon[29471]: 6.e scrub ok 2024-09-13T13:09:39.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:38 smithi052 ceph-mon[29471]: 8.9 scrub starts 2024-09-13T13:09:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:38 smithi052 ceph-mon[36682]: 4.17 scrub starts 2024-09-13T13:09:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:38 smithi052 ceph-mon[36682]: 4.17 scrub ok 2024-09-13T13:09:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:38 smithi052 ceph-mon[36682]: 6.e scrub starts 2024-09-13T13:09:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:38 smithi052 ceph-mon[36682]: pgmap v406: 148 pgs: 148 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 877 op/s 2024-09-13T13:09:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:38 smithi052 ceph-mon[36682]: 6.e scrub ok 2024-09-13T13:09:39.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:38 smithi052 ceph-mon[36682]: 8.9 scrub starts 2024-09-13T13:09:39.242 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:39.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:38 smithi167 ceph-mon[33178]: 4.17 scrub starts 2024-09-13T13:09:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:38 smithi167 ceph-mon[33178]: 4.17 scrub ok 2024-09-13T13:09:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:38 smithi167 ceph-mon[33178]: 6.e scrub starts 2024-09-13T13:09:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:38 smithi167 ceph-mon[33178]: pgmap v406: 148 pgs: 148 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 877 op/s 2024-09-13T13:09:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:38 smithi167 ceph-mon[33178]: 6.e scrub ok 2024-09-13T13:09:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:38 smithi167 ceph-mon[33178]: 8.9 scrub starts 2024-09-13T13:09:39.344 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:39.370 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:39.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:39.441 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T13:09:39.472 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:39.509 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:39.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:39.610 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:39.659 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:39.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:39.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:39.722 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:39.762 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:39.797 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:09:39.827 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:39.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:39.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:39.878 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[36682]: 5.1e scrub starts 2024-09-13T13:09:39.878 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[36682]: 5.1e scrub ok 2024-09-13T13:09:39.878 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[36682]: 8.9 scrub ok 2024-09-13T13:09:39.879 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[29471]: 5.1e scrub starts 2024-09-13T13:09:39.879 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[29471]: 5.1e scrub ok 2024-09-13T13:09:39.879 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[29471]: 8.9 scrub ok 2024-09-13T13:09:39.879 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[29471]: 5.2 scrub starts 2024-09-13T13:09:39.880 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[29471]: 5.2 scrub ok 2024-09-13T13:09:39.880 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[29471]: 3.4 scrub starts 2024-09-13T13:09:39.880 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[29471]: 3.4 scrub ok 2024-09-13T13:09:39.929 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:39.959 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:39.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:39.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:39.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:39 smithi167 ceph-mon[33178]: 5.1e scrub starts 2024-09-13T13:09:39.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:39 smithi167 ceph-mon[33178]: 5.1e scrub ok 2024-09-13T13:09:39.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:39 smithi167 ceph-mon[33178]: 8.9 scrub ok 2024-09-13T13:09:39.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:39 smithi167 ceph-mon[33178]: 5.2 scrub starts 2024-09-13T13:09:39.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:39 smithi167 ceph-mon[33178]: 5.2 scrub ok 2024-09-13T13:09:39.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:39 smithi167 ceph-mon[33178]: 3.4 scrub starts 2024-09-13T13:09:39.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:39 smithi167 ceph-mon[33178]: 3.4 scrub ok 2024-09-13T13:09:40.060 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:40.061 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:40.078 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:09:40.081 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:09:40.100 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:40.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:40.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:40.202 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:40.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:40.232 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:40.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[36682]: 5.2 scrub starts 2024-09-13T13:09:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[36682]: 5.2 scrub ok 2024-09-13T13:09:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[36682]: 3.4 scrub starts 2024-09-13T13:09:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:39 smithi052 ceph-mon[36682]: 3.4 scrub ok 2024-09-13T13:09:40.334 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:40.368 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:40.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:40.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:40.453 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:09:40.454 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:09:40.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:40.454 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:240439 8.6 e232/169 8:63151b2f:::benchmark_data_smithi052_101302_object112510:head v 232'15219, mlcod=232'15217)", 2024-09-13T13:09:40.454 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:09:40.413699+0000", 2024-09-13T13:09:40.454 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0037693879999999998, 2024-09-13T13:09:40.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037959059999999999, 2024-09-13T13:09:40.454 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:40.454 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:40.454 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:09:40.454 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:40.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:40.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:40.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:40.413699+0000", 2024-09-13T13:09:40.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:40.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:40.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:40.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:40.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:40.413699+0000", 2024-09-13T13:09:40.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:40.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:40.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:40.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:40.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:40.413700+0000", 2024-09-13T13:09:40.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1909999999999999e-06 2024-09-13T13:09:40.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:40.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:40.413706+0000", 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5130000000000002e-06 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:40.413716+0000", 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3859999999999996e-06 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:40.413720+0000", 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3800000000000004e-06 2024-09-13T13:09:40.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:40.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:40.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:40.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:40.413764+0000", 2024-09-13T13:09:40.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3618999999999997e-05 2024-09-13T13:09:40.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:40.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:40.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:40.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:40.413783+0000", 2024-09-13T13:09:40.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8856e-05 2024-09-13T13:09:40.457 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:40.457 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:40.457 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:40.457 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:40.458 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:09:40.458 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-09-13T13:09:40.458 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:09:40.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:40.503 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:40.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:40.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:40.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:40.633 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:40.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:40.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:40.735 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:40.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:40.779 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:40.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:40.880 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:40.925 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:40.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:40.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:41.027 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:41.043 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:09:41.043 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:09:41.073 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:41.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:41.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:41.082 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:09:41.082 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:09:41.106 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:40 smithi052 ceph-mon[36682]: 4.10 scrub starts 2024-09-13T13:09:41.107 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:40 smithi052 ceph-mon[36682]: 4.10 scrub ok 2024-09-13T13:09:41.107 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:40 smithi052 ceph-mon[36682]: pgmap v407: 148 pgs: 148 active+clean; 1.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-13T13:09:41.107 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:40 smithi052 ceph-mon[36682]: 5.1f scrub starts 2024-09-13T13:09:41.107 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:40 smithi052 ceph-mon[36682]: 5.1f scrub ok 2024-09-13T13:09:41.107 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:40 smithi052 ceph-mon[36682]: from='client.15687 v1:172.21.15.52:0/1099469095' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:09:41.107 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:40 smithi052 ceph-mon[29471]: 4.10 scrub starts 2024-09-13T13:09:41.107 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:40 smithi052 ceph-mon[29471]: 4.10 scrub ok 2024-09-13T13:09:41.107 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:40 smithi052 ceph-mon[29471]: pgmap v407: 148 pgs: 148 active+clean; 1.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-13T13:09:41.108 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:40 smithi052 ceph-mon[29471]: 5.1f scrub starts 2024-09-13T13:09:41.108 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:40 smithi052 ceph-mon[29471]: 5.1f scrub ok 2024-09-13T13:09:41.108 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:40 smithi052 ceph-mon[29471]: from='client.15687 v1:172.21.15.52:0/1099469095' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:09:41.120 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:09:41.120 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:09:41.156 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:09:41.156 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:09:41.174 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:41.189 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:09:41.189 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:09:41.223 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:09:41.223 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:09:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:41.225 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:41.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:41.258 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:09:41.259 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:09:41.292 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:09:41.292 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:09:41.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:40 smithi167 ceph-mon[33178]: 4.10 scrub starts 2024-09-13T13:09:41.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:40 smithi167 ceph-mon[33178]: 4.10 scrub ok 2024-09-13T13:09:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:40 smithi167 ceph-mon[33178]: pgmap v407: 148 pgs: 148 active+clean; 1.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-13T13:09:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:40 smithi167 ceph-mon[33178]: 5.1f scrub starts 2024-09-13T13:09:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:40 smithi167 ceph-mon[33178]: 5.1f scrub ok 2024-09-13T13:09:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:40 smithi167 ceph-mon[33178]: from='client.15687 v1:172.21.15.52:0/1099469095' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:09:41.326 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:41.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:41.370 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:41.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:41.472 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:41.486 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T13:09:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:41.508 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:41.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:41.609 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:41.653 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:41.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:41.754 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:41.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:41.808 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:41.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:41.880 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:09:41.909 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:41.953 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:41.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:42.044 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-09-13T13:09:39.778530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1176705587,"num_objects":18146,"num_object_clones":0,"num_object_copies":54438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18146,"num_whiteouts":0,"num_read":1356,"num_read_kb":1213,"num_write":238456,"num_write_kb":8188938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86367,"ondisk_log_size":86367,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5622764,"kb_used_data":3520232,"kb_used_omap":15,"kb_used_meta":2102064,"kb_avail":744338452,"statfs":{"total":767960285184,"available":762202574848,"internally_reserved":0,"allocated":3604717568,"data_stored":3601428281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":2152514329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,2,3,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-675872768,"num_objects":-10313,"num_object_clones":0,"num_object_copies":-30939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10313,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10313,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006383"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":375,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531661+0000","last_change":"2024-09-13T13:06:44.304525+0000","last_active":"2024-09-13T13:09:34.531661+0000","last_peered":"2024-09-13T13:09:34.531661+0000","last_clean":"2024-09-13T13:09:34.531661+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:09:34.531661+0000","last_undegraded":"2024-09-13T13:09:34.531661+0000","last_fullsized":"2024-09-13T13:09:34.531661+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:44.304444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.304444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.250196+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":398,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520524+0000","last_change":"2024-09-13T13:09:06.567774+0000","last_active":"2024-09-13T13:09:34.520524+0000","last_peered":"2024-09-13T13:09:34.520524+0000","last_clean":"2024-09-13T13:09:34.520524+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:09:34.520524+0000","last_undegraded":"2024-09-13T13:09:34.520524+0000","last_fullsized":"2024-09-13T13:09:34.520524+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:06.567719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:06.567719+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:30.867489+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":367,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.548252+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:09:34.548252+0000","last_peered":"2024-09-13T13:09:34.548252+0000","last_clean":"2024-09-13T13:09:34.548252+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:09:34.548252+0000","last_undegraded":"2024-09-13T13:09:34.548252+0000","last_fullsized":"2024-09-13T13:09:34.548252+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:08.843909+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":355,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520719+0000","last_change":"2024-09-13T13:05:01.076346+0000","last_active":"2024-09-13T13:09:34.520719+0000","last_peered":"2024-09-13T13:09:34.520719+0000","last_clean":"2024-09-13T13:09:34.520719+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:09:34.520719+0000","last_undegraded":"2024-09-13T13:09:34.520719+0000","last_fullsized":"2024-09-13T13:09:34.520719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.076275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.076275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:35.591462+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":381,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.534767+0000","last_change":"2024-09-13T13:08:55.504675+0000","last_active":"2024-09-13T13:09:34.534767+0000","last_peered":"2024-09-13T13:09:34.534767+0000","last_clean":"2024-09-13T13:09:34.534767+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:09:34.534767+0000","last_undegraded":"2024-09-13T13:09:34.534767+0000","last_fullsized":"2024-09-13T13:09:34.534767+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.504475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.504475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.958611+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":386,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.530681+0000","last_change":"2024-09-13T13:09:31.491481+0000","last_active":"2024-09-13T13:09:34.530681+0000","last_peered":"2024-09-13T13:09:34.530681+0000","last_clean":"2024-09-13T13:09:34.530681+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:09:34.530681+0000","last_undegraded":"2024-09-13T13:09:34.530681+0000","last_fullsized":"2024-09-13T13:09:34.530681+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:31.491404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:31.491404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.774257+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":411,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.529311+0000","last_change":"2024-09-13T13:08:53.510436+0000","last_active":"2024-09-13T13:09:34.529311+0000","last_peered":"2024-09-13T13:09:34.529311+0000","last_clean":"2024-09-13T13:09:34.529311+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:09:34.529311+0000","last_undegraded":"2024-09-13T13:09:34.529311+0000","last_fullsized":"2024-09-13T13:09:34.529311+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:08:53.510335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.510335+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.247321+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":386,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520492+0000","last_change":"2024-09-13T13:08:55.513716+0000","last_active":"2024-09-13T13:09:34.520492+0000","last_peered":"2024-09-13T13:09:34.520492+0000","last_clean":"2024-09-13T13:09:34.520492+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:09:34.520492+0000","last_undegraded":"2024-09-13T13:09:34.520492+0000","last_fullsized":"2024-09-13T13:09:34.520492+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.513667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.513667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.654393+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":407,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526274+0000","last_change":"2024-09-13T13:08:59.654555+0000","last_active":"2024-09-13T13:09:34.526274+0000","last_peered":"2024-09-13T13:09:34.526274+0000","last_clean":"2024-09-13T13:09:34.526274+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:09:34.526274+0000","last_undegraded":"2024-09-13T13:09:34.526274+0000","last_fullsized":"2024-09-13T13:09:34.526274+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:08:59.654388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.654388+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.560420+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":403,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520959+0000","last_change":"2024-09-13T13:09:00.426054+0000","last_active":"2024-09-13T13:09:34.520959+0000","last_peered":"2024-09-13T13:09:34.520959+0000","last_clean":"2024-09-13T13:09:34.520959+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:09:34.520959+0000","last_undegraded":"2024-09-13T13:09:34.520959+0000","last_fullsized":"2024-09-13T13:09:34.520959+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:00.425965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:00.425965+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:24.968404+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":374,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.529949+0000","last_change":"2024-09-13T13:06:50.665256+0000","last_active":"2024-09-13T13:09:34.529949+0000","last_peered":"2024-09-13T13:09:34.529949+0000","last_clean":"2024-09-13T13:09:34.529949+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:09:34.529949+0000","last_undegraded":"2024-09-13T13:09:34.529949+0000","last_fullsized":"2024-09-13T13:09:34.529949+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:50.665191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.665191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.666378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":370,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.524424+0000","last_change":"2024-09-13T13:08:49.280146+0000","last_active":"2024-09-13T13:09:34.524424+0000","last_peered":"2024-09-13T13:09:34.524424+0000","last_clean":"2024-09-13T13:09:34.524424+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:09:34.524424+0000","last_undegraded":"2024-09-13T13:09:34.524424+0000","last_fullsized":"2024-09-13T13:09:34.524424+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:49.280073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:49.280073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.252935+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":390,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526514+0000","last_change":"2024-09-13T13:07:57.425510+0000","last_active":"2024-09-13T13:09:34.526514+0000","last_peered":"2024-09-13T13:09:34.526514+0000","last_clean":"2024-09-13T13:09:34.526514+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:09:34.526514+0000","last_undegraded":"2024-09-13T13:09:34.526514+0000","last_fullsized":"2024-09-13T13:09:34.526514+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.425436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.425436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:24.739469+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":381,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520607+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:09:34.520607+0000","last_peered":"2024-09-13T13:09:34.520607+0000","last_clean":"2024-09-13T13:09:34.520607+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:09:34.520607+0000","last_undegraded":"2024-09-13T13:09:34.520607+0000","last_fullsized":"2024-09-13T13:09:34.520607+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:40.626212+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":374,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.529073+0000","last_change":"2024-09-13T13:07:50.530956+0000","last_active":"2024-09-13T13:09:34.529073+0000","last_peered":"2024-09-13T13:09:34.529073+0000","last_clean":"2024-09-13T13:09:34.529073+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:09:34.529073+0000","last_undegraded":"2024-09-13T13:09:34.529073+0000","last_fullsized":"2024-09-13T13:09:34.529073+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:50.530904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:50.530904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.091714+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":354,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.525948+0000","last_change":"2024-09-13T13:07:19.641119+0000","last_active":"2024-09-13T13:09:34.525948+0000","last_peered":"2024-09-13T13:09:34.525948+0000","last_clean":"2024-09-13T13:09:34.525948+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:09:34.525948+0000","last_undegraded":"2024-09-13T13:09:34.525948+0000","last_fullsized":"2024-09-13T13:09:34.525948+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:19.640697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:19.640697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:40.384060+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":384,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520150+0000","last_change":"2024-09-13T13:07:46.772159+0000","last_active":"2024-09-13T13:09:34.520150+0000","last_peered":"2024-09-13T13:09:34.520150+0000","last_clean":"2024-09-13T13:09:34.520150+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:09:34.520150+0000","last_undegraded":"2024-09-13T13:09:34.520150+0000","last_fullsized":"2024-09-13T13:09:34.520150+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:07:46.772111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:46.772111+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:22.644441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":382,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526960+0000","last_change":"2024-09-13T13:08:19.181017+0000","last_active":"2024-09-13T13:09:34.526960+0000","last_peered":"2024-09-13T13:09:34.526960+0000","last_clean":"2024-09-13T13:09:34.526960+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:09:34.526960+0000","last_undegraded":"2024-09-13T13:09:34.526960+0000","last_fullsized":"2024-09-13T13:09:34.526960+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:19.180948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.180948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:43.499202+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":412,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526225+0000","last_change":"2024-09-13T13:06:44.532305+0000","last_active":"2024-09-13T13:09:34.526225+0000","last_peered":"2024-09-13T13:09:34.526225+0000","last_clean":"2024-09-13T13:09:34.526225+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:09:34.526225+0000","last_undegraded":"2024-09-13T13:09:34.526225+0000","last_fullsized":"2024-09-13T13:09:34.526225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:06:44.532186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.532186+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.375109+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":375,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.538503+0000","last_change":"2024-09-13T13:08:18.639588+0000","last_active":"2024-09-13T13:09:34.538503+0000","last_peered":"2024-09-13T13:09:34.538503+0000","last_clean":"2024-09-13T13:09:34.538503+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:09:34.538503+0000","last_undegraded":"2024-09-13T13:09:34.538503+0000","last_fullsized":"2024-09-13T13:09:34.538503+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:18.639517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:18.639517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:41.488821+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":382,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.530970+0000","last_change":"2024-09-13T13:09:29.517189+0000","last_active":"2024-09-13T13:09:34.530970+0000","last_peered":"2024-09-13T13:09:34.530970+0000","last_clean":"2024-09-13T13:09:34.530970+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:09:34.530970+0000","last_undegraded":"2024-09-13T13:09:34.530970+0000","last_fullsized":"2024-09-13T13:09:34.530970+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:29.517034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:29.517034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:34.784832+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":383,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.521080+0000","last_change":"2024-09-13T13:09:28.614216+0000","last_active":"2024-09-13T13:09:34.521080+0000","last_peered":"2024-09-13T13:09:34.521080+0000","last_clean":"2024-09-13T13:09:34.521080+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:09:34.521080+0000","last_undegraded":"2024-09-13T13:09:34.521080+0000","last_fullsized":"2024-09-13T13:09:34.521080+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:28.614027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.614027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:31.496891+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":385,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.530671+0000","last_change":"2024-09-13T13:06:05.706641+0000","last_active":"2024-09-13T13:09:34.530671+0000","last_peered":"2024-09-13T13:09:34.530671+0000","last_clean":"2024-09-13T13:09:34.530671+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:09:34.530671+0000","last_undegraded":"2024-09-13T13:09:34.530671+0000","last_fullsized":"2024-09-13T13:09:34.530671+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:06:05.706569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:05.706569+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:41.423833+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":375,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520591+0000","last_change":"2024-09-13T13:07:12.879813+0000","last_active":"2024-09-13T13:09:34.520591+0000","last_peered":"2024-09-13T13:09:34.520591+0000","last_clean":"2024-09-13T13:09:34.520591+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:09:34.520591+0000","last_undegraded":"2024-09-13T13:09:34.520591+0000","last_fullsized":"2024-09-13T13:09:34.520591+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.879727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.879727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:41.576645+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":379,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.530636+0000","last_change":"2024-09-13T13:08:59.165835+0000","last_active":"2024-09-13T13:09:34.530636+0000","last_peered":"2024-09-13T13:09:34.530636+0000","last_clean":"2024-09-13T13:09:34.530636+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:09:34.530636+0000","last_undegraded":"2024-09-13T13:09:34.530636+0000","last_fullsized":"2024-09-13T13:09:34.530636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:08:59.165754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.165754+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.048878+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":376,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.548389+0000","last_change":"2024-09-13T13:07:54.442276+0000","last_active":"2024-09-13T13:09:34.548389+0000","last_peered":"2024-09-13T13:09:34.548389+0000","last_clean":"2024-09-13T13:09:34.548389+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:09:34.548389+0000","last_undegraded":"2024-09-13T13:09:34.548389+0000","last_fullsized":"2024-09-13T13:09:34.548389+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:07:54.439269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:54.439269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.738173+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":402,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531537+0000","last_change":"2024-09-13T13:09:28.434007+0000","last_active":"2024-09-13T13:09:34.531537+0000","last_peered":"2024-09-13T13:09:34.531537+0000","last_clean":"2024-09-13T13:09:34.531537+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:09:34.531537+0000","last_undegraded":"2024-09-13T13:09:34.531537+0000","last_fullsized":"2024-09-13T13:09:34.531537+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:28.433899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.433899+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.061308+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":427,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528294+0000","last_change":"2024-09-13T13:06:04.949976+0000","last_active":"2024-09-13T13:09:34.528294+0000","last_peered":"2024-09-13T13:09:34.528294+0000","last_clean":"2024-09-13T13:09:34.528294+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:09:34.528294+0000","last_undegraded":"2024-09-13T13:09:34.528294+0000","last_fullsized":"2024-09-13T13:09:34.528294+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:06:04.949930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:04.949930+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:39.043496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"232'15064","reported_seq":15225,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.478415+0000","last_change":"2024-09-13T13:09:34.404950+0000","last_active":"2024-09-13T13:09:38.478415+0000","last_peered":"2024-09-13T13:09:38.478415+0000","last_clean":"2024-09-13T13:09:38.478415+0000","last_became_active":"2024-09-13T13:06:21.814002+0000","last_became_peered":"2024-09-13T13:06:21.814002+0000","last_unstale":"2024-09-13T13:09:38.478415+0000","last_undegraded":"2024-09-13T13:09:38.478415+0000","last_fullsized":"2024-09-13T13:09:38.478415+0000","mapping_epoch":169,"log_start":"213'10100","ondisk_log_start":"213'10100","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'14788","last_scrub_stamp":"2024-09-13T13:09:34.404844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:09:34.404844+0000","objects_scrubbed":1367,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:36.235023+0000","scrub_duration":561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70385664,"num_objects":1074,"num_object_clones":0,"num_object_copies":3222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15064,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":404,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.539719+0000","last_change":"2024-09-13T13:07:15.243958+0000","last_active":"2024-09-13T13:09:34.539719+0000","last_peered":"2024-09-13T13:09:34.539719+0000","last_clean":"2024-09-13T13:09:34.539719+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:09:34.539719+0000","last_undegraded":"2024-09-13T13:09:34.539719+0000","last_fullsized":"2024-09-13T13:09:34.539719+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:07:15.243715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:15.243715+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:40.693139+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":390,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.530177+0000","last_change":"2024-09-13T13:08:19.560359+0000","last_active":"2024-09-13T13:09:34.530177+0000","last_peered":"2024-09-13T13:09:34.530177+0000","last_clean":"2024-09-13T13:09:34.530177+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:09:34.530177+0000","last_undegraded":"2024-09-13T13:09:34.530177+0000","last_fullsized":"2024-09-13T13:09:34.530177+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:19.560157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.560157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":487,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520366+0000","last_change":"2024-09-13T13:08:53.508763+0000","last_active":"2024-09-13T13:09:34.520366+0000","last_peered":"2024-09-13T13:09:34.520366+0000","last_clean":"2024-09-13T13:09:34.520366+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:09:34.520366+0000","last_undegraded":"2024-09-13T13:09:34.520366+0000","last_fullsized":"2024-09-13T13:09:34.520366+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:08:53.508690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.508690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:06.126931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":370,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.525986+0000","last_change":"2024-09-13T13:08:05.356744+0000","last_active":"2024-09-13T13:09:34.525986+0000","last_peered":"2024-09-13T13:09:34.525986+0000","last_clean":"2024-09-13T13:09:34.525986+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:09:34.525986+0000","last_undegraded":"2024-09-13T13:09:34.525986+0000","last_fullsized":"2024-09-13T13:09:34.525986+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:05.356688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:05.356688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"232'14839","reported_seq":14977,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.916181+0000","last_change":"2024-09-13T13:06:22.000536+0000","last_active":"2024-09-13T13:09:38.916181+0000","last_peered":"2024-09-13T13:09:38.916181+0000","last_clean":"2024-09-13T13:09:38.916181+0000","last_became_active":"2024-09-13T13:06:22.000244+0000","last_became_peered":"2024-09-13T13:06:22.000244+0000","last_unstale":"2024-09-13T13:09:38.916181+0000","last_undegraded":"2024-09-13T13:09:38.916181+0000","last_fullsized":"2024-09-13T13:09:38.916181+0000","mapping_epoch":169,"log_start":"208'9000","ondisk_log_start":"208'9000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:47.271652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63504384,"num_objects":969,"num_object_clones":0,"num_object_copies":2907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14839,"num_write_kb":505856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":395,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.529957+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:09:34.529957+0000","last_peered":"2024-09-13T13:09:34.529957+0000","last_clean":"2024-09-13T13:09:34.529957+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:09:34.529957+0000","last_undegraded":"2024-09-13T13:09:34.529957+0000","last_fullsized":"2024-09-13T13:09:34.529957+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.546922+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":380,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531330+0000","last_change":"2024-09-13T13:06:44.785235+0000","last_active":"2024-09-13T13:09:34.531330+0000","last_peered":"2024-09-13T13:09:34.531330+0000","last_clean":"2024-09-13T13:09:34.531330+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:09:34.531330+0000","last_undegraded":"2024-09-13T13:09:34.531330+0000","last_fullsized":"2024-09-13T13:09:34.531330+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:06:44.785017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.785017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.624176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":368,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520387+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:09:34.520387+0000","last_peered":"2024-09-13T13:09:34.520387+0000","last_clean":"2024-09-13T13:09:34.520387+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:09:34.520387+0000","last_undegraded":"2024-09-13T13:09:34.520387+0000","last_fullsized":"2024-09-13T13:09:34.520387+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.668051+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":372,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520927+0000","last_change":"2024-09-13T13:08:22.683148+0000","last_active":"2024-09-13T13:09:34.520927+0000","last_peered":"2024-09-13T13:09:34.520927+0000","last_clean":"2024-09-13T13:09:34.520927+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:09:34.520927+0000","last_undegraded":"2024-09-13T13:09:34.520927+0000","last_fullsized":"2024-09-13T13:09:34.520927+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:22.683019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:22.683019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:37.253075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"232'14748","reported_seq":14887,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.079733+0000","last_change":"2024-09-13T13:06:22.000928+0000","last_active":"2024-09-13T13:09:38.079733+0000","last_peered":"2024-09-13T13:09:38.079733+0000","last_clean":"2024-09-13T13:09:38.079733+0000","last_became_active":"2024-09-13T13:06:22.000610+0000","last_became_peered":"2024-09-13T13:06:22.000610+0000","last_unstale":"2024-09-13T13:09:38.079733+0000","last_undegraded":"2024-09-13T13:09:38.079733+0000","last_fullsized":"2024-09-13T13:09:38.079733+0000","mapping_epoch":169,"log_start":"209'9300","ondisk_log_start":"209'9300","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:18.686758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77660184,"num_objects":1186,"num_object_clones":0,"num_object_copies":3558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14748,"num_write_kb":509825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":412,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520871+0000","last_change":"2024-09-13T13:07:10.853834+0000","last_active":"2024-09-13T13:09:34.520871+0000","last_peered":"2024-09-13T13:09:34.520871+0000","last_clean":"2024-09-13T13:09:34.520871+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:09:34.520871+0000","last_undegraded":"2024-09-13T13:09:34.520871+0000","last_fullsized":"2024-09-13T13:09:34.520871+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:07:10.853749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.853749+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:45.529858+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":367,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.532963+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:09:34.532963+0000","last_peered":"2024-09-13T13:09:34.532963+0000","last_clean":"2024-09-13T13:09:34.532963+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:09:34.532963+0000","last_undegraded":"2024-09-13T13:09:34.532963+0000","last_fullsized":"2024-09-13T13:09:34.532963+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.235292+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":479,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.529820+0000","last_change":"2024-09-13T13:09:28.477414+0000","last_active":"2024-09-13T13:09:34.529820+0000","last_peered":"2024-09-13T13:09:34.529820+0000","last_clean":"2024-09-13T13:09:34.529820+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:09:34.529820+0000","last_undegraded":"2024-09-13T13:09:34.529820+0000","last_fullsized":"2024-09-13T13:09:34.529820+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:09:28.477338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.477338+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:42.900990+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":374,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.529325+0000","last_change":"2024-09-13T13:06:09.230872+0000","last_active":"2024-09-13T13:09:34.529325+0000","last_peered":"2024-09-13T13:09:34.529325+0000","last_clean":"2024-09-13T13:09:34.529325+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:09:34.529325+0000","last_undegraded":"2024-09-13T13:09:34.529325+0000","last_fullsized":"2024-09-13T13:09:34.529325+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:09.230824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.230824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:43.287111+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"232'14839","reported_seq":14977,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.926622+0000","last_change":"2024-09-13T13:06:21.812511+0000","last_active":"2024-09-13T13:09:38.926622+0000","last_peered":"2024-09-13T13:09:38.926622+0000","last_clean":"2024-09-13T13:09:38.926622+0000","last_became_active":"2024-09-13T13:06:21.812226+0000","last_became_peered":"2024-09-13T13:06:21.812226+0000","last_unstale":"2024-09-13T13:09:38.926622+0000","last_undegraded":"2024-09-13T13:09:38.926622+0000","last_fullsized":"2024-09-13T13:09:38.926622+0000","mapping_epoch":169,"log_start":"209'9000","ondisk_log_start":"209'9000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:41.321590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68222976,"num_objects":1041,"num_object_clones":0,"num_object_copies":3123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14839,"num_write_kb":508160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":395,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.534040+0000","last_change":"2024-09-13T13:09:27.854186+0000","last_active":"2024-09-13T13:09:34.534040+0000","last_peered":"2024-09-13T13:09:34.534040+0000","last_clean":"2024-09-13T13:09:34.534040+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:09:34.534040+0000","last_undegraded":"2024-09-13T13:09:34.534040+0000","last_fullsized":"2024-09-13T13:09:34.534040+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:27.854114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:27.854114+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:39.809612+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":383,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526756+0000","last_change":"2024-09-13T13:08:17.298581+0000","last_active":"2024-09-13T13:09:34.526756+0000","last_peered":"2024-09-13T13:09:34.526756+0000","last_clean":"2024-09-13T13:09:34.526756+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:09:34.526756+0000","last_undegraded":"2024-09-13T13:09:34.526756+0000","last_fullsized":"2024-09-13T13:09:34.526756+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:17.298430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:17.298430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":485,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526796+0000","last_change":"2024-09-13T13:07:57.028959+0000","last_active":"2024-09-13T13:09:34.526796+0000","last_peered":"2024-09-13T13:09:34.526796+0000","last_clean":"2024-09-13T13:09:34.526796+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:09:34.526796+0000","last_undegraded":"2024-09-13T13:09:34.526796+0000","last_fullsized":"2024-09-13T13:09:34.526796+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:07:57.028875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.028875+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.174262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":379,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526458+0000","last_change":"2024-09-13T13:08:52.207903+0000","last_active":"2024-09-13T13:09:34.526458+0000","last_peered":"2024-09-13T13:09:34.526458+0000","last_clean":"2024-09-13T13:09:34.526458+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:09:34.526458+0000","last_undegraded":"2024-09-13T13:09:34.526458+0000","last_fullsized":"2024-09-13T13:09:34.526458+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:52.207841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:52.207841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.868681+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"232'14870","reported_seq":15060,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.547379+0000","last_change":"2024-09-13T13:08:51.116458+0000","last_active":"2024-09-13T13:09:38.547379+0000","last_peered":"2024-09-13T13:09:38.547379+0000","last_clean":"2024-09-13T13:09:38.547379+0000","last_became_active":"2024-09-13T13:06:21.812208+0000","last_became_peered":"2024-09-13T13:06:21.812208+0000","last_unstale":"2024-09-13T13:09:38.547379+0000","last_undegraded":"2024-09-13T13:09:38.547379+0000","last_fullsized":"2024-09-13T13:09:38.547379+0000","mapping_epoch":169,"log_start":"213'10000","ondisk_log_start":"213'10000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'11587","last_scrub_stamp":"2024-09-13T13:08:51.116305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:08:51.116305+0000","objects_scrubbed":4435,"log_size":4870,"log_dups_size":0,"ondisk_log_size":4870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.050196+0000","scrub_duration":1647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72089600,"num_objects":1100,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14870,"num_write_kb":511040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":423,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526850+0000","last_change":"2024-09-13T13:09:05.142515+0000","last_active":"2024-09-13T13:09:34.526850+0000","last_peered":"2024-09-13T13:09:34.526850+0000","last_clean":"2024-09-13T13:09:34.526850+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:09:34.526850+0000","last_undegraded":"2024-09-13T13:09:34.526850+0000","last_fullsized":"2024-09-13T13:09:34.526850+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:09:05.142427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.142427+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:06.191632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":360,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.538135+0000","last_change":"2024-09-13T13:01:02.182121+0000","last_active":"2024-09-13T13:09:34.538135+0000","last_peered":"2024-09-13T13:09:34.538135+0000","last_clean":"2024-09-13T13:09:34.538135+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:09:34.538135+0000","last_undegraded":"2024-09-13T13:09:34.538135+0000","last_fullsized":"2024-09-13T13:09:34.538135+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:01:01.155841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:36.648956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":328,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.527739+0000","last_change":"2024-09-13T13:06:08.941867+0000","last_active":"2024-09-13T13:09:34.527739+0000","last_peered":"2024-09-13T13:09:34.527739+0000","last_clean":"2024-09-13T13:09:34.527739+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:09:34.527739+0000","last_undegraded":"2024-09-13T13:09:34.527739+0000","last_fullsized":"2024-09-13T13:09:34.527739+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:08.941799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:08.941799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:37.020191+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":383,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.548168+0000","last_change":"2024-09-13T13:06:43.969251+0000","last_active":"2024-09-13T13:09:34.548168+0000","last_peered":"2024-09-13T13:09:34.548168+0000","last_clean":"2024-09-13T13:09:34.548168+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:09:34.548168+0000","last_undegraded":"2024-09-13T13:09:34.548168+0000","last_fullsized":"2024-09-13T13:09:34.548168+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.969176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.969176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.710958+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"232'14650","reported_seq":14788,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.090695+0000","last_change":"2024-09-13T13:06:22.000658+0000","last_active":"2024-09-13T13:09:38.090695+0000","last_peered":"2024-09-13T13:09:38.090695+0000","last_clean":"2024-09-13T13:09:38.090695+0000","last_became_active":"2024-09-13T13:06:22.000261+0000","last_became_peered":"2024-09-13T13:06:22.000261+0000","last_unstale":"2024-09-13T13:09:38.090695+0000","last_undegraded":"2024-09-13T13:09:38.090695+0000","last_fullsized":"2024-09-13T13:09:38.090695+0000","mapping_epoch":169,"log_start":"209'9200","ondisk_log_start":"209'9200","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5450,"log_dups_size":0,"ondisk_log_size":5450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.720130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73924608,"num_objects":1128,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14650,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":532,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.538054+0000","last_change":"2024-09-13T13:07:25.438610+0000","last_active":"2024-09-13T13:09:34.538054+0000","last_peered":"2024-09-13T13:09:34.538054+0000","last_clean":"2024-09-13T13:09:34.538054+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:09:34.538054+0000","last_undegraded":"2024-09-13T13:09:34.538054+0000","last_fullsized":"2024-09-13T13:09:34.538054+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:07:25.438528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:25.438528+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:42.700793+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":395,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.522127+0000","last_change":"2024-09-13T13:07:18.888103+0000","last_active":"2024-09-13T13:09:34.522127+0000","last_peered":"2024-09-13T13:09:34.522127+0000","last_clean":"2024-09-13T13:09:34.522127+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:09:34.522127+0000","last_undegraded":"2024-09-13T13:09:34.522127+0000","last_fullsized":"2024-09-13T13:09:34.522127+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:18.888060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:18.888060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:42.559510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":386,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.532006+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:09:34.532006+0000","last_peered":"2024-09-13T13:09:34.532006+0000","last_clean":"2024-09-13T13:09:34.532006+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:09:34.532006+0000","last_undegraded":"2024-09-13T13:09:34.532006+0000","last_fullsized":"2024-09-13T13:09:34.532006+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:38.410879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":367,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.527094+0000","last_change":"2024-09-13T13:06:11.189922+0000","last_active":"2024-09-13T13:09:34.527094+0000","last_peered":"2024-09-13T13:09:34.527094+0000","last_clean":"2024-09-13T13:09:34.527094+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:09:34.527094+0000","last_undegraded":"2024-09-13T13:09:34.527094+0000","last_fullsized":"2024-09-13T13:09:34.527094+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:11.189874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:11.189874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:38.354895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":370,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.530392+0000","last_change":"2024-09-13T13:09:00.570589+0000","last_active":"2024-09-13T13:09:34.530392+0000","last_peered":"2024-09-13T13:09:34.530392+0000","last_clean":"2024-09-13T13:09:34.530392+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:09:34.530392+0000","last_undegraded":"2024-09-13T13:09:34.530392+0000","last_fullsized":"2024-09-13T13:09:34.530392+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:00.570518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:00.570518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.670665+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"232'14781","reported_seq":14971,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.557701+0000","last_change":"2024-09-13T13:08:52.266749+0000","last_active":"2024-09-13T13:09:38.557701+0000","last_peered":"2024-09-13T13:09:38.557701+0000","last_clean":"2024-09-13T13:09:38.557701+0000","last_became_active":"2024-09-13T13:06:21.812869+0000","last_became_peered":"2024-09-13T13:06:21.812869+0000","last_unstale":"2024-09-13T13:09:38.557701+0000","last_undegraded":"2024-09-13T13:09:38.557701+0000","last_fullsized":"2024-09-13T13:09:38.557701+0000","mapping_epoch":169,"log_start":"213'9900","ondisk_log_start":"213'9900","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'11526","last_scrub_stamp":"2024-09-13T13:08:52.266659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:08:52.266659+0000","objects_scrubbed":4408,"log_size":4881,"log_dups_size":0,"ondisk_log_size":4881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.678811+0000","scrub_duration":1778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72417280,"num_objects":1105,"num_object_clones":0,"num_object_copies":3315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14781,"num_write_kb":508352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":398,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.539809+0000","last_change":"2024-09-13T13:08:04.680025+0000","last_active":"2024-09-13T13:09:34.539809+0000","last_peered":"2024-09-13T13:09:34.539809+0000","last_clean":"2024-09-13T13:09:34.539809+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:09:34.539809+0000","last_undegraded":"2024-09-13T13:09:34.539809+0000","last_fullsized":"2024-09-13T13:09:34.539809+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:08:04.679766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:04.679766+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:22.837118+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":403,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.535234+0000","last_change":"2024-09-13T13:08:56.771534+0000","last_active":"2024-09-13T13:09:34.535234+0000","last_peered":"2024-09-13T13:09:34.535234+0000","last_clean":"2024-09-13T13:09:34.535234+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:09:34.535234+0000","last_undegraded":"2024-09-13T13:09:34.535234+0000","last_fullsized":"2024-09-13T13:09:34.535234+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:56.771432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:56.771432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.588929+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":399,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528635+0000","last_change":"2024-09-13T13:08:19.038471+0000","last_active":"2024-09-13T13:09:34.528635+0000","last_peered":"2024-09-13T13:09:34.528635+0000","last_clean":"2024-09-13T13:09:34.528635+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:09:34.528635+0000","last_undegraded":"2024-09-13T13:09:34.528635+0000","last_fullsized":"2024-09-13T13:09:34.528635+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:08:19.038393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.038393+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:25.295684+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":359,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526277+0000","last_change":"2024-09-13T13:03:14.282866+0000","last_active":"2024-09-13T13:09:34.526277+0000","last_peered":"2024-09-13T13:09:34.526277+0000","last_clean":"2024-09-13T13:09:34.526277+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:09:34.526277+0000","last_undegraded":"2024-09-13T13:09:34.526277+0000","last_fullsized":"2024-09-13T13:09:34.526277+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:41.328750+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":382,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.530924+0000","last_change":"2024-09-13T13:08:54.232191+0000","last_active":"2024-09-13T13:09:34.530924+0000","last_peered":"2024-09-13T13:09:34.530924+0000","last_clean":"2024-09-13T13:09:34.530924+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:09:34.530924+0000","last_undegraded":"2024-09-13T13:09:34.530924+0000","last_fullsized":"2024-09-13T13:09:34.530924+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:54.232130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:54.232130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.097713+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"232'14775","reported_seq":14992,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.479383+0000","last_change":"2024-09-13T13:07:49.115654+0000","last_active":"2024-09-13T13:09:38.479383+0000","last_peered":"2024-09-13T13:09:38.479383+0000","last_clean":"2024-09-13T13:09:38.479383+0000","last_became_active":"2024-09-13T13:06:21.814772+0000","last_became_peered":"2024-09-13T13:06:21.814772+0000","last_unstale":"2024-09-13T13:09:38.479383+0000","last_undegraded":"2024-09-13T13:09:38.479383+0000","last_fullsized":"2024-09-13T13:09:38.479383+0000","mapping_epoch":169,"log_start":"212'9800","ondisk_log_start":"212'9800","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'7112","last_scrub_stamp":"2024-09-13T13:07:49.115526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:07:49.115526+0000","objects_scrubbed":7029,"log_size":4975,"log_dups_size":0,"ondisk_log_size":4975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.794726+0000","scrub_duration":2765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69926912,"num_objects":1067,"num_object_clones":0,"num_object_copies":3201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14775,"num_write_kb":506944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":396,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531619+0000","last_change":"2024-09-13T13:08:21.219398+0000","last_active":"2024-09-13T13:09:34.531619+0000","last_peered":"2024-09-13T13:09:34.531619+0000","last_clean":"2024-09-13T13:09:34.531619+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:09:34.531619+0000","last_undegraded":"2024-09-13T13:09:34.531619+0000","last_fullsized":"2024-09-13T13:09:34.531619+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:08:21.219300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:21.219300+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:38.637760+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":383,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526234+0000","last_change":"2024-09-13T13:08:02.152950+0000","last_active":"2024-09-13T13:09:34.526234+0000","last_peered":"2024-09-13T13:09:34.526234+0000","last_clean":"2024-09-13T13:09:34.526234+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:09:34.526234+0000","last_undegraded":"2024-09-13T13:09:34.526234+0000","last_fullsized":"2024-09-13T13:09:34.526234+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:02.152859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:02.152859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:25.356503+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":901,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:36.087826+0000","last_change":"2024-09-13T13:08:49.264831+0000","last_active":"2024-09-13T13:09:36.087826+0000","last_peered":"2024-09-13T13:09:36.087826+0000","last_clean":"2024-09-13T13:09:36.087826+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:09:36.087826+0000","last_undegraded":"2024-09-13T13:09:36.087826+0000","last_fullsized":"2024-09-13T13:09:36.087826+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:08:49.264775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:49.264775+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.716838+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":509,"num_read_kb":504,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":358,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526688+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:09:34.526688+0000","last_peered":"2024-09-13T13:09:34.526688+0000","last_clean":"2024-09-13T13:09:34.526688+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:09:34.526688+0000","last_undegraded":"2024-09-13T13:09:34.526688+0000","last_fullsized":"2024-09-13T13:09:34.526688+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:37.454063+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":380,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.537734+0000","last_change":"2024-09-13T13:08:55.738138+0000","last_active":"2024-09-13T13:09:34.537734+0000","last_peered":"2024-09-13T13:09:34.537734+0000","last_clean":"2024-09-13T13:09:34.537734+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:09:34.537734+0000","last_undegraded":"2024-09-13T13:09:34.537734+0000","last_fullsized":"2024-09-13T13:09:34.537734+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.738069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.738069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:25.032322+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"232'14502","reported_seq":14640,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:35.933733+0000","last_change":"2024-09-13T13:06:21.814495+0000","last_active":"2024-09-13T13:09:35.933733+0000","last_peered":"2024-09-13T13:09:35.933733+0000","last_clean":"2024-09-13T13:09:35.933733+0000","last_became_active":"2024-09-13T13:06:21.814144+0000","last_became_peered":"2024-09-13T13:06:21.814144+0000","last_unstale":"2024-09-13T13:09:35.933733+0000","last_undegraded":"2024-09-13T13:09:35.933733+0000","last_fullsized":"2024-09-13T13:09:35.933733+0000","mapping_epoch":169,"log_start":"206'8200","ondisk_log_start":"206'8200","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":6302,"log_dups_size":0,"ondisk_log_size":6302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85852160,"num_objects":1310,"num_object_clones":0,"num_object_copies":3930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14502,"num_write_kb":505984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":363,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528524+0000","last_change":"2024-09-13T13:06:51.193749+0000","last_active":"2024-09-13T13:09:34.528524+0000","last_peered":"2024-09-13T13:09:34.528524+0000","last_clean":"2024-09-13T13:09:34.528524+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:09:34.528524+0000","last_undegraded":"2024-09-13T13:09:34.528524+0000","last_fullsized":"2024-09-13T13:09:34.528524+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'18","last_scrub_stamp":"2024-09-13T13:06:51.193678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.193678+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.989653+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":386,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.530528+0000","last_change":"2024-09-13T13:06:10.715371+0000","last_active":"2024-09-13T13:09:34.530528+0000","last_peered":"2024-09-13T13:09:34.530528+0000","last_clean":"2024-09-13T13:09:34.530528+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:09:34.530528+0000","last_undegraded":"2024-09-13T13:09:34.530528+0000","last_fullsized":"2024-09-13T13:09:34.530528+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:10.715313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.715313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:46.404429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":431,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528677+0000","last_change":"2024-09-13T13:06:39.564599+0000","last_active":"2024-09-13T13:09:34.528677+0000","last_peered":"2024-09-13T13:09:34.528677+0000","last_clean":"2024-09-13T13:09:34.528677+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:09:34.528677+0000","last_undegraded":"2024-09-13T13:09:34.528677+0000","last_fullsized":"2024-09-13T13:09:34.528677+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:06:39.564549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:06:39.564549+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:15.102832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":383,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531579+0000","last_change":"2024-09-13T13:08:25.500731+0000","last_active":"2024-09-13T13:09:34.531579+0000","last_peered":"2024-09-13T13:09:34.531579+0000","last_clean":"2024-09-13T13:09:34.531579+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:09:34.531579+0000","last_undegraded":"2024-09-13T13:09:34.531579+0000","last_fullsized":"2024-09-13T13:09:34.531579+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:25.500660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:25.500660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:47.582413+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":378,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526549+0000","last_change":"2024-09-13T13:08:20.287277+0000","last_active":"2024-09-13T13:09:34.526549+0000","last_peered":"2024-09-13T13:09:34.526549+0000","last_clean":"2024-09-13T13:09:34.526549+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:09:34.526549+0000","last_undegraded":"2024-09-13T13:09:34.526549+0000","last_fullsized":"2024-09-13T13:09:34.526549+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:20.287215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:20.287215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:38.617880+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"232'14900","reported_seq":15057,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.008446+0000","last_change":"2024-09-13T13:07:26.381412+0000","last_active":"2024-09-13T13:09:38.008446+0000","last_peered":"2024-09-13T13:09:38.008446+0000","last_clean":"2024-09-13T13:09:38.008446+0000","last_became_active":"2024-09-13T13:06:21.812061+0000","last_became_peered":"2024-09-13T13:06:21.812061+0000","last_unstale":"2024-09-13T13:09:38.008446+0000","last_undegraded":"2024-09-13T13:09:38.008446+0000","last_fullsized":"2024-09-13T13:09:38.008446+0000","mapping_epoch":169,"log_start":"213'10100","ondisk_log_start":"213'10100","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":1649,"log_size":4800,"log_dups_size":0,"ondisk_log_size":4800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.683639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73400320,"num_objects":1120,"num_object_clones":0,"num_object_copies":3360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14900,"num_write_kb":512640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":401,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520415+0000","last_change":"2024-09-13T13:08:01.790332+0000","last_active":"2024-09-13T13:09:34.520415+0000","last_peered":"2024-09-13T13:09:34.520415+0000","last_clean":"2024-09-13T13:09:34.520415+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:09:34.520415+0000","last_undegraded":"2024-09-13T13:09:34.520415+0000","last_fullsized":"2024-09-13T13:09:34.520415+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:08:01.790280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:01.790280+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:24.650459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":383,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520795+0000","last_change":"2024-09-13T13:08:51.546064+0000","last_active":"2024-09-13T13:09:34.520795+0000","last_peered":"2024-09-13T13:09:34.520795+0000","last_clean":"2024-09-13T13:09:34.520795+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:09:34.520795+0000","last_undegraded":"2024-09-13T13:09:34.520795+0000","last_fullsized":"2024-09-13T13:09:34.520795+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:51.546003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:51.546003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:59.798244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":385,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526407+0000","last_change":"2024-09-13T13:07:23.602111+0000","last_active":"2024-09-13T13:09:34.526407+0000","last_peered":"2024-09-13T13:09:34.526407+0000","last_clean":"2024-09-13T13:09:34.526407+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:09:34.526407+0000","last_undegraded":"2024-09-13T13:09:34.526407+0000","last_fullsized":"2024-09-13T13:09:34.526407+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:23.602052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:23.602052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:43.434576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":373,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.530645+0000","last_change":"2024-09-13T13:08:27.451600+0000","last_active":"2024-09-13T13:09:34.530645+0000","last_peered":"2024-09-13T13:09:34.530645+0000","last_clean":"2024-09-13T13:09:34.530645+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:09:34.530645+0000","last_undegraded":"2024-09-13T13:09:34.530645+0000","last_fullsized":"2024-09-13T13:09:34.530645+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:27.448636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:27.448636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:45.909228+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"232'15002","reported_seq":15221,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.481767+0000","last_change":"2024-09-13T13:07:52.135686+0000","last_active":"2024-09-13T13:09:38.481767+0000","last_peered":"2024-09-13T13:09:38.481767+0000","last_clean":"2024-09-13T13:09:38.481767+0000","last_became_active":"2024-09-13T13:06:21.811526+0000","last_became_peered":"2024-09-13T13:06:21.811526+0000","last_unstale":"2024-09-13T13:09:38.481767+0000","last_undegraded":"2024-09-13T13:09:38.481767+0000","last_fullsized":"2024-09-13T13:09:38.481767+0000","mapping_epoch":169,"log_start":"213'10000","ondisk_log_start":"213'10000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'7389","last_scrub_stamp":"2024-09-13T13:07:52.135527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:07:52.135527+0000","objects_scrubbed":7297,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.778806+0000","scrub_duration":2761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73662464,"num_objects":1124,"num_object_clones":0,"num_object_copies":3372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15002,"num_write_kb":516032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":399,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520812+0000","last_change":"2024-09-13T13:05:34.166464+0000","last_active":"2024-09-13T13:09:34.520812+0000","last_peered":"2024-09-13T13:09:34.520812+0000","last_clean":"2024-09-13T13:09:34.520812+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:09:34.520812+0000","last_undegraded":"2024-09-13T13:09:34.520812+0000","last_fullsized":"2024-09-13T13:09:34.520812+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.686209+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":360,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528019+0000","last_change":"2024-09-13T13:09:05.049507+0000","last_active":"2024-09-13T13:09:34.528019+0000","last_peered":"2024-09-13T13:09:34.528019+0000","last_clean":"2024-09-13T13:09:34.528019+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:09:34.528019+0000","last_undegraded":"2024-09-13T13:09:34.528019+0000","last_fullsized":"2024-09-13T13:09:34.528019+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:05.049446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.049446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.951824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":367,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.537831+0000","last_change":"2024-09-13T13:08:53.747370+0000","last_active":"2024-09-13T13:09:34.537831+0000","last_peered":"2024-09-13T13:09:34.537831+0000","last_clean":"2024-09-13T13:09:34.537831+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:09:34.537831+0000","last_undegraded":"2024-09-13T13:09:34.537831+0000","last_fullsized":"2024-09-13T13:09:34.537831+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.747302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.747302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.330071+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":371,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526266+0000","last_change":"2024-09-13T13:08:03.015302+0000","last_active":"2024-09-13T13:09:34.526266+0000","last_peered":"2024-09-13T13:09:34.526266+0000","last_clean":"2024-09-13T13:09:34.526266+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:09:34.526266+0000","last_undegraded":"2024-09-13T13:09:34.526266+0000","last_fullsized":"2024-09-13T13:09:34.526266+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:03.015233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.015233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:23.253063+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"232'14725","reported_seq":14945,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.556513+0000","last_change":"2024-09-13T13:07:54.357736+0000","last_active":"2024-09-13T13:09:38.556513+0000","last_peered":"2024-09-13T13:09:38.556513+0000","last_clean":"2024-09-13T13:09:38.556513+0000","last_became_active":"2024-09-13T13:06:21.812966+0000","last_became_peered":"2024-09-13T13:06:21.812966+0000","last_unstale":"2024-09-13T13:09:38.556513+0000","last_undegraded":"2024-09-13T13:09:38.556513+0000","last_fullsized":"2024-09-13T13:09:38.556513+0000","mapping_epoch":169,"log_start":"213'9800","ondisk_log_start":"213'9800","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'7426","last_scrub_stamp":"2024-09-13T13:07:54.357373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:07:54.357373+0000","objects_scrubbed":7324,"log_size":4925,"log_dups_size":0,"ondisk_log_size":4925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71892992,"num_objects":1097,"num_object_clones":0,"num_object_copies":3291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14725,"num_write_kb":506304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":413,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520897+0000","last_change":"2024-09-13T13:09:05.619225+0000","last_active":"2024-09-13T13:09:34.520897+0000","last_peered":"2024-09-13T13:09:34.520897+0000","last_clean":"2024-09-13T13:09:34.520897+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:09:34.520897+0000","last_undegraded":"2024-09-13T13:09:34.520897+0000","last_fullsized":"2024-09-13T13:09:34.520897+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:05.619171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.619171+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:07.343034+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":374,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.535134+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:09:34.535134+0000","last_peered":"2024-09-13T13:09:34.535134+0000","last_clean":"2024-09-13T13:09:34.535134+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:09:34.535134+0000","last_undegraded":"2024-09-13T13:09:34.535134+0000","last_fullsized":"2024-09-13T13:09:34.535134+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:15.617560+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":479,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.527449+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:09:34.527449+0000","last_peered":"2024-09-13T13:09:34.527449+0000","last_clean":"2024-09-13T13:09:34.527449+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:09:34.527449+0000","last_undegraded":"2024-09-13T13:09:34.527449+0000","last_fullsized":"2024-09-13T13:09:34.527449+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.422585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":363,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:37.812544+0000","last_change":"2024-09-13T13:09:37.812544+0000","last_active":"2024-09-13T13:09:37.812544+0000","last_peered":"2024-09-13T13:09:37.812544+0000","last_clean":"2024-09-13T13:09:37.812544+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:09:37.812544+0000","last_undegraded":"2024-09-13T13:09:37.812544+0000","last_fullsized":"2024-09-13T13:09:37.812544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:37.812451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:37.812451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:00.130077+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"232'15190","reported_seq":15328,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.466877+0000","last_change":"2024-09-13T13:06:21.814825+0000","last_active":"2024-09-13T13:09:38.466877+0000","last_peered":"2024-09-13T13:09:38.466877+0000","last_clean":"2024-09-13T13:09:38.466877+0000","last_became_active":"2024-09-13T13:06:21.814433+0000","last_became_peered":"2024-09-13T13:06:21.814433+0000","last_unstale":"2024-09-13T13:09:38.466877+0000","last_undegraded":"2024-09-13T13:09:38.466877+0000","last_fullsized":"2024-09-13T13:09:38.466877+0000","mapping_epoch":169,"log_start":"213'10200","ondisk_log_start":"213'10200","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":4990,"log_dups_size":0,"ondisk_log_size":4990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:41.690088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71827456,"num_objects":1096,"num_object_clones":0,"num_object_copies":3288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15190,"num_write_kb":521152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":378,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526803+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:09:34.526803+0000","last_peered":"2024-09-13T13:09:34.526803+0000","last_clean":"2024-09-13T13:09:34.526803+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:09:34.526803+0000","last_undegraded":"2024-09-13T13:09:34.526803+0000","last_fullsized":"2024-09-13T13:09:34.526803+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.254641+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":401,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526641+0000","last_change":"2024-09-13T13:09:06.668526+0000","last_active":"2024-09-13T13:09:34.526641+0000","last_peered":"2024-09-13T13:09:34.526641+0000","last_clean":"2024-09-13T13:09:34.526641+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:09:34.526641+0000","last_undegraded":"2024-09-13T13:09:34.526641+0000","last_fullsized":"2024-09-13T13:09:34.526641+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:06.668478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:06.668478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.164862+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":383,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.525914+0000","last_change":"2024-09-13T13:07:11.400066+0000","last_active":"2024-09-13T13:09:34.525914+0000","last_peered":"2024-09-13T13:09:34.525914+0000","last_clean":"2024-09-13T13:09:34.525914+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:09:34.525914+0000","last_undegraded":"2024-09-13T13:09:34.525914+0000","last_fullsized":"2024-09-13T13:09:34.525914+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.399840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.399840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:43.349392+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":362,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526142+0000","last_change":"2024-09-13T13:09:29.473480+0000","last_active":"2024-09-13T13:09:34.526142+0000","last_peered":"2024-09-13T13:09:34.526142+0000","last_clean":"2024-09-13T13:09:34.526142+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:09:34.526142+0000","last_undegraded":"2024-09-13T13:09:34.526142+0000","last_fullsized":"2024-09-13T13:09:34.526142+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:29.473418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:29.473418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:55.955954+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"232'15111","reported_seq":15249,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.901030+0000","last_change":"2024-09-13T13:06:22.000224+0000","last_active":"2024-09-13T13:09:38.901030+0000","last_peered":"2024-09-13T13:09:38.901030+0000","last_clean":"2024-09-13T13:09:38.901030+0000","last_became_active":"2024-09-13T13:06:21.999889+0000","last_became_peered":"2024-09-13T13:06:21.999889+0000","last_unstale":"2024-09-13T13:09:38.901030+0000","last_undegraded":"2024-09-13T13:09:38.901030+0000","last_fullsized":"2024-09-13T13:09:38.901030+0000","mapping_epoch":169,"log_start":"209'9300","ondisk_log_start":"209'9300","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:40.313069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73728000,"num_objects":1125,"num_object_clones":0,"num_object_copies":3375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15111,"num_write_kb":519552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":371,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.527834+0000","last_change":"2024-09-13T13:09:32.849360+0000","last_active":"2024-09-13T13:09:34.527834+0000","last_peered":"2024-09-13T13:09:34.527834+0000","last_clean":"2024-09-13T13:09:34.527834+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:09:34.527834+0000","last_undegraded":"2024-09-13T13:09:34.527834+0000","last_fullsized":"2024-09-13T13:09:34.527834+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:09:32.849277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:32.849277+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:00.662384+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":393,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531219+0000","last_change":"2024-09-13T13:07:48.579717+0000","last_active":"2024-09-13T13:09:34.531219+0000","last_peered":"2024-09-13T13:09:34.531219+0000","last_clean":"2024-09-13T13:09:34.531219+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:09:34.531219+0000","last_undegraded":"2024-09-13T13:09:34.531219+0000","last_fullsized":"2024-09-13T13:09:34.531219+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:48.579658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:48.579658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.184890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":383,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.525528+0000","last_change":"2024-09-13T13:08:26.033729+0000","last_active":"2024-09-13T13:09:34.525528+0000","last_peered":"2024-09-13T13:09:34.525528+0000","last_clean":"2024-09-13T13:09:34.525528+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:09:34.525528+0000","last_undegraded":"2024-09-13T13:09:34.525528+0000","last_fullsized":"2024-09-13T13:09:34.525528+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:26.033663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:26.033663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:37.040385+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":370,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528625+0000","last_change":"2024-09-13T13:07:49.584576+0000","last_active":"2024-09-13T13:09:34.528625+0000","last_peered":"2024-09-13T13:09:34.528625+0000","last_clean":"2024-09-13T13:09:34.528625+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:09:34.528625+0000","last_undegraded":"2024-09-13T13:09:34.528625+0000","last_fullsized":"2024-09-13T13:09:34.528625+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:49.584505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:49.584505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.019080+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"232'15012","reported_seq":15150,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:35.962057+0000","last_change":"2024-09-13T13:06:22.002144+0000","last_active":"2024-09-13T13:09:35.962057+0000","last_peered":"2024-09-13T13:09:35.962057+0000","last_clean":"2024-09-13T13:09:35.962057+0000","last_became_active":"2024-09-13T13:06:22.001404+0000","last_became_peered":"2024-09-13T13:06:22.001404+0000","last_unstale":"2024-09-13T13:09:35.962057+0000","last_undegraded":"2024-09-13T13:09:35.962057+0000","last_fullsized":"2024-09-13T13:09:35.962057+0000","mapping_epoch":169,"log_start":"207'8700","ondisk_log_start":"207'8700","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:40.004617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84672512,"num_objects":1292,"num_object_clones":0,"num_object_copies":3876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15012,"num_write_kb":521728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":394,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520638+0000","last_change":"2024-09-13T13:09:30.614842+0000","last_active":"2024-09-13T13:09:34.520638+0000","last_peered":"2024-09-13T13:09:34.520638+0000","last_clean":"2024-09-13T13:09:34.520638+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:09:34.520638+0000","last_undegraded":"2024-09-13T13:09:34.520638+0000","last_fullsized":"2024-09-13T13:09:34.520638+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:09:30.614748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.614748+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:58.916392+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":382,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.529749+0000","last_change":"2024-09-13T13:05:29.345863+0000","last_active":"2024-09-13T13:09:34.529749+0000","last_peered":"2024-09-13T13:09:34.529749+0000","last_clean":"2024-09-13T13:09:34.529749+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:09:34.529749+0000","last_undegraded":"2024-09-13T13:09:34.529749+0000","last_fullsized":"2024-09-13T13:09:34.529749+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.345796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.345796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:14.146169+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":367,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.525057+0000","last_change":"2024-09-13T13:06:42.450669+0000","last_active":"2024-09-13T13:09:34.525057+0000","last_peered":"2024-09-13T13:09:34.525057+0000","last_clean":"2024-09-13T13:09:34.525057+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:09:34.525057+0000","last_undegraded":"2024-09-13T13:09:34.525057+0000","last_fullsized":"2024-09-13T13:09:34.525057+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:42.450598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:42.450598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:14.400638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":354,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.534847+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:09:34.534847+0000","last_peered":"2024-09-13T13:09:34.534847+0000","last_clean":"2024-09-13T13:09:34.534847+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:09:34.534847+0000","last_undegraded":"2024-09-13T13:09:34.534847+0000","last_fullsized":"2024-09-13T13:09:34.534847+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:10.602171+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"232'14899","reported_seq":15125,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.076711+0000","last_change":"2024-09-13T13:08:03.659578+0000","last_active":"2024-09-13T13:09:38.076711+0000","last_peered":"2024-09-13T13:09:38.076711+0000","last_clean":"2024-09-13T13:09:38.076711+0000","last_became_active":"2024-09-13T13:06:21.816096+0000","last_became_peered":"2024-09-13T13:06:21.816096+0000","last_unstale":"2024-09-13T13:09:38.076711+0000","last_undegraded":"2024-09-13T13:09:38.076711+0000","last_fullsized":"2024-09-13T13:09:38.076711+0000","mapping_epoch":169,"log_start":"211'9500","ondisk_log_start":"211'9500","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'8119","last_scrub_stamp":"2024-09-13T13:08:03.659418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.659418+0000","objects_scrubbed":7960,"log_size":5399,"log_dups_size":0,"ondisk_log_size":5399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2349,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72941568,"num_objects":1113,"num_object_clones":0,"num_object_copies":3339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14899,"num_write_kb":512384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":414,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.525948+0000","last_change":"2024-09-13T13:09:33.843654+0000","last_active":"2024-09-13T13:09:34.525948+0000","last_peered":"2024-09-13T13:09:34.525948+0000","last_clean":"2024-09-13T13:09:34.525948+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:09:34.525948+0000","last_undegraded":"2024-09-13T13:09:34.525948+0000","last_fullsized":"2024-09-13T13:09:34.525948+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:09:33.843579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:33.843579+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:40.459268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":394,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.529953+0000","last_change":"2024-09-13T13:06:51.627764+0000","last_active":"2024-09-13T13:09:34.529953+0000","last_peered":"2024-09-13T13:09:34.529953+0000","last_clean":"2024-09-13T13:09:34.529953+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:09:34.529953+0000","last_undegraded":"2024-09-13T13:09:34.529953+0000","last_fullsized":"2024-09-13T13:09:34.529953+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:06:51.627691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.627691+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.064623+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":487,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.530114+0000","last_change":"2024-09-13T13:04:02.759553+0000","last_active":"2024-09-13T13:09:34.530114+0000","last_peered":"2024-09-13T13:09:34.530114+0000","last_clean":"2024-09-13T13:09:34.530114+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:09:34.530114+0000","last_undegraded":"2024-09-13T13:09:34.530114+0000","last_fullsized":"2024-09-13T13:09:34.530114+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:02:45.959285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:02:45.959285+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:37.358597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":373,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.527458+0000","last_change":"2024-09-13T13:09:30.836048+0000","last_active":"2024-09-13T13:09:34.527458+0000","last_peered":"2024-09-13T13:09:34.527458+0000","last_clean":"2024-09-13T13:09:34.527458+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:09:34.527458+0000","last_undegraded":"2024-09-13T13:09:34.527458+0000","last_fullsized":"2024-09-13T13:09:34.527458+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.836006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.836006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:38.200874+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":352,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.527352+0000","last_change":"2024-09-13T13:07:10.058616+0000","last_active":"2024-09-13T13:09:34.527352+0000","last_peered":"2024-09-13T13:09:34.527352+0000","last_clean":"2024-09-13T13:09:34.527352+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:09:34.527352+0000","last_undegraded":"2024-09-13T13:09:34.527352+0000","last_fullsized":"2024-09-13T13:09:34.527352+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:10.058555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.058555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:44.072613+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":379,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:37.358795+0000","last_change":"2024-09-13T13:09:37.358795+0000","last_active":"2024-09-13T13:09:37.358795+0000","last_peered":"2024-09-13T13:09:37.358795+0000","last_clean":"2024-09-13T13:09:37.358795+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:09:37.358795+0000","last_undegraded":"2024-09-13T13:09:37.358795+0000","last_fullsized":"2024-09-13T13:09:37.358795+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:09:37.358710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:37.358710+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:38.147999+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":382,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526184+0000","last_change":"2024-09-13T13:08:21.310761+0000","last_active":"2024-09-13T13:09:34.526184+0000","last_peered":"2024-09-13T13:09:34.526184+0000","last_clean":"2024-09-13T13:09:34.526184+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:09:34.526184+0000","last_undegraded":"2024-09-13T13:09:34.526184+0000","last_fullsized":"2024-09-13T13:09:34.526184+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:21.310681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:21.310681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:44.757345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":389,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.530837+0000","last_change":"2024-09-13T13:09:30.959919+0000","last_active":"2024-09-13T13:09:34.530837+0000","last_peered":"2024-09-13T13:09:34.530837+0000","last_clean":"2024-09-13T13:09:34.530837+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:09:34.530837+0000","last_undegraded":"2024-09-13T13:09:34.530837+0000","last_fullsized":"2024-09-13T13:09:34.530837+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.959870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.959870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:45.945701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":386,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531702+0000","last_change":"2024-09-13T13:09:34.426362+0000","last_active":"2024-09-13T13:09:34.531702+0000","last_peered":"2024-09-13T13:09:34.531702+0000","last_clean":"2024-09-13T13:09:34.531702+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:09:34.531702+0000","last_undegraded":"2024-09-13T13:09:34.531702+0000","last_fullsized":"2024-09-13T13:09:34.531702+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:34.426272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:34.426272+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.871047+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":377,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.530311+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:09:34.530311+0000","last_peered":"2024-09-13T13:09:34.530311+0000","last_clean":"2024-09-13T13:09:34.530311+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:09:34.530311+0000","last_undegraded":"2024-09-13T13:09:34.530311+0000","last_fullsized":"2024-09-13T13:09:34.530311+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:40.294134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":358,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531041+0000","last_change":"2024-09-13T13:05:42.728927+0000","last_active":"2024-09-13T13:09:34.531041+0000","last_peered":"2024-09-13T13:09:34.531041+0000","last_clean":"2024-09-13T13:09:34.531041+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:09:34.531041+0000","last_undegraded":"2024-09-13T13:09:34.531041+0000","last_fullsized":"2024-09-13T13:09:34.531041+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:42.728881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:42.728881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:38.377382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":379,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.525468+0000","last_change":"2024-09-13T13:09:07.378142+0000","last_active":"2024-09-13T13:09:34.525468+0000","last_peered":"2024-09-13T13:09:34.525468+0000","last_clean":"2024-09-13T13:09:34.525468+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:09:34.525468+0000","last_undegraded":"2024-09-13T13:09:34.525468+0000","last_fullsized":"2024-09-13T13:09:34.525468+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:07.378066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:07.378066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:33.759039+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":378,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526249+0000","last_change":"2024-09-13T13:04:53.687731+0000","last_active":"2024-09-13T13:09:34.526249+0000","last_peered":"2024-09-13T13:09:34.526249+0000","last_clean":"2024-09-13T13:09:34.526249+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:09:34.526249+0000","last_undegraded":"2024-09-13T13:09:34.526249+0000","last_fullsized":"2024-09-13T13:09:34.526249+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:35.367886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":374,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.525152+0000","last_change":"2024-09-13T13:06:41.308449+0000","last_active":"2024-09-13T13:09:34.525152+0000","last_peered":"2024-09-13T13:09:34.525152+0000","last_clean":"2024-09-13T13:09:34.525152+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:09:34.525152+0000","last_undegraded":"2024-09-13T13:09:34.525152+0000","last_fullsized":"2024-09-13T13:09:34.525152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:41.308367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:41.308367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.223962+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":470,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.530866+0000","last_change":"2024-09-13T13:08:02.779173+0000","last_active":"2024-09-13T13:09:34.530866+0000","last_peered":"2024-09-13T13:09:34.530866+0000","last_clean":"2024-09-13T13:09:34.530866+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:09:34.530866+0000","last_undegraded":"2024-09-13T13:09:34.530866+0000","last_fullsized":"2024-09-13T13:09:34.530866+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:08:02.778891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:02.778891+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":371,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520146+0000","last_change":"2024-09-13T13:07:21.958114+0000","last_active":"2024-09-13T13:09:34.520146+0000","last_peered":"2024-09-13T13:09:34.520146+0000","last_clean":"2024-09-13T13:09:34.520146+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:09:34.520146+0000","last_undegraded":"2024-09-13T13:09:34.520146+0000","last_fullsized":"2024-09-13T13:09:34.520146+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:21.958030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:21.958030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:44.550122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":393,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.532007+0000","last_change":"2024-09-13T13:09:04.635198+0000","last_active":"2024-09-13T13:09:34.532007+0000","last_peered":"2024-09-13T13:09:34.532007+0000","last_clean":"2024-09-13T13:09:34.532007+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:09:34.532007+0000","last_undegraded":"2024-09-13T13:09:34.532007+0000","last_fullsized":"2024-09-13T13:09:34.532007+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:04.599040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:04.599040+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:30.955987+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":366,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.529627+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:09:34.529627+0000","last_peered":"2024-09-13T13:09:34.529627+0000","last_clean":"2024-09-13T13:09:34.529627+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:09:34.529627+0000","last_undegraded":"2024-09-13T13:09:34.529627+0000","last_fullsized":"2024-09-13T13:09:34.529627+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:11.161132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":494,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526149+0000","last_change":"2024-09-13T13:07:08.781005+0000","last_active":"2024-09-13T13:09:34.526149+0000","last_peered":"2024-09-13T13:09:34.526149+0000","last_clean":"2024-09-13T13:09:34.526149+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:09:34.526149+0000","last_undegraded":"2024-09-13T13:09:34.526149+0000","last_fullsized":"2024-09-13T13:09:34.526149+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:07:08.780952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.780952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:41.374850+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":370,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:36.414394+0000","last_change":"2024-09-13T13:09:36.414394+0000","last_active":"2024-09-13T13:09:36.414394+0000","last_peered":"2024-09-13T13:09:36.414394+0000","last_clean":"2024-09-13T13:09:36.414394+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:09:36.414394+0000","last_undegraded":"2024-09-13T13:09:36.414394+0000","last_fullsized":"2024-09-13T13:09:36.414394+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:36.414322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:36.414322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:39.835131+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":401,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520721+0000","last_change":"2024-09-13T13:05:26.186899+0000","last_active":"2024-09-13T13:09:34.520721+0000","last_peered":"2024-09-13T13:09:34.520721+0000","last_clean":"2024-09-13T13:09:34.520721+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:09:34.520721+0000","last_undegraded":"2024-09-13T13:09:34.520721+0000","last_fullsized":"2024-09-13T13:09:34.520721+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:05:26.186830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.186830+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:14.738553+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":386,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.520827+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:09:34.520827+0000","last_peered":"2024-09-13T13:09:34.520827+0000","last_clean":"2024-09-13T13:09:34.520827+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:09:34.520827+0000","last_undegraded":"2024-09-13T13:09:34.520827+0000","last_fullsized":"2024-09-13T13:09:34.520827+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.696749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":375,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.533897+0000","last_change":"2024-09-13T13:09:05.882862+0000","last_active":"2024-09-13T13:09:34.533897+0000","last_peered":"2024-09-13T13:09:34.533897+0000","last_clean":"2024-09-13T13:09:34.533897+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:09:34.533897+0000","last_undegraded":"2024-09-13T13:09:34.533897+0000","last_fullsized":"2024-09-13T13:09:34.533897+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:05.882769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.882769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.443670+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":370,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.529033+0000","last_change":"2024-09-13T13:08:59.585721+0000","last_active":"2024-09-13T13:09:34.529033+0000","last_peered":"2024-09-13T13:09:34.529033+0000","last_clean":"2024-09-13T13:09:34.529033+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:09:34.529033+0000","last_undegraded":"2024-09-13T13:09:34.529033+0000","last_fullsized":"2024-09-13T13:09:34.529033+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:59.585646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.585646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.171797+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":330,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.527251+0000","last_change":"2024-09-13T13:05:35.506093+0000","last_active":"2024-09-13T13:09:34.527251+0000","last_peered":"2024-09-13T13:09:34.527251+0000","last_clean":"2024-09-13T13:09:34.527251+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:09:34.527251+0000","last_undegraded":"2024-09-13T13:09:34.527251+0000","last_fullsized":"2024-09-13T13:09:34.527251+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:05:35.506037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.506037+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:10.670571+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":382,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531099+0000","last_change":"2024-09-13T13:09:07.118545+0000","last_active":"2024-09-13T13:09:34.531099+0000","last_peered":"2024-09-13T13:09:34.531099+0000","last_clean":"2024-09-13T13:09:34.531099+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:09:34.531099+0000","last_undegraded":"2024-09-13T13:09:34.531099+0000","last_fullsized":"2024-09-13T13:09:34.531099+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:07.118467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:07.118467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.947652+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":366,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.534923+0000","last_change":"2024-09-13T13:09:32.467362+0000","last_active":"2024-09-13T13:09:34.534923+0000","last_peered":"2024-09-13T13:09:34.534923+0000","last_clean":"2024-09-13T13:09:34.534923+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:09:34.534923+0000","last_undegraded":"2024-09-13T13:09:34.534923+0000","last_fullsized":"2024-09-13T13:09:34.534923+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:32.467151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:32.467151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:55.111614+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":370,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.524263+0000","last_change":"2024-09-13T13:08:56.227109+0000","last_active":"2024-09-13T13:09:34.524263+0000","last_peered":"2024-09-13T13:09:34.524263+0000","last_clean":"2024-09-13T13:09:34.524263+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:09:34.524263+0000","last_undegraded":"2024-09-13T13:09:34.524263+0000","last_fullsized":"2024-09-13T13:09:34.524263+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:56.227027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:56.227027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:14.111530+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":392,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.529173+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:09:34.529173+0000","last_peered":"2024-09-13T13:09:34.529173+0000","last_clean":"2024-09-13T13:09:34.529173+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:09:34.529173+0000","last_undegraded":"2024-09-13T13:09:34.529173+0000","last_fullsized":"2024-09-13T13:09:34.529173+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:11.582994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":382,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526280+0000","last_change":"2024-09-13T13:08:50.517587+0000","last_active":"2024-09-13T13:09:34.526280+0000","last_peered":"2024-09-13T13:09:34.526280+0000","last_clean":"2024-09-13T13:09:34.526280+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:09:34.526280+0000","last_undegraded":"2024-09-13T13:09:34.526280+0000","last_fullsized":"2024-09-13T13:09:34.526280+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:50.517526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:50.517526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:03.602123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":385,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528369+0000","last_change":"2024-09-13T13:08:53.215325+0000","last_active":"2024-09-13T13:09:34.528369+0000","last_peered":"2024-09-13T13:09:34.528369+0000","last_clean":"2024-09-13T13:09:34.528369+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:09:34.528369+0000","last_undegraded":"2024-09-13T13:09:34.528369+0000","last_fullsized":"2024-09-13T13:09:34.528369+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.215263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.215263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:10.458784+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":365,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.535056+0000","last_change":"2024-09-13T13:07:57.596272+0000","last_active":"2024-09-13T13:09:34.535056+0000","last_peered":"2024-09-13T13:09:34.535056+0000","last_clean":"2024-09-13T13:09:34.535056+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:09:34.535056+0000","last_undegraded":"2024-09-13T13:09:34.535056+0000","last_fullsized":"2024-09-13T13:09:34.535056+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.596097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":392,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.529864+0000","last_change":"2024-09-13T13:05:44.759472+0000","last_active":"2024-09-13T13:09:34.529864+0000","last_peered":"2024-09-13T13:09:34.529864+0000","last_clean":"2024-09-13T13:09:34.529864+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:09:34.529864+0000","last_undegraded":"2024-09-13T13:09:34.529864+0000","last_fullsized":"2024-09-13T13:09:34.529864+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2024-09-13T13:05:44.759233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:44.759233+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:37.352712+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":390,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531761+0000","last_change":"2024-09-13T13:09:30.487486+0000","last_active":"2024-09-13T13:09:34.531761+0000","last_peered":"2024-09-13T13:09:34.531761+0000","last_clean":"2024-09-13T13:09:34.531761+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:09:34.531761+0000","last_undegraded":"2024-09-13T13:09:34.531761+0000","last_fullsized":"2024-09-13T13:09:34.531761+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.487413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.487413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:53.159481+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":276,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528145+0000","last_change":"2024-09-13T13:09:27.844085+0000","last_active":"2024-09-13T13:09:34.528145+0000","last_peered":"2024-09-13T13:09:34.528145+0000","last_clean":"2024-09-13T13:09:34.528145+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:09:34.528145+0000","last_undegraded":"2024-09-13T13:09:34.528145+0000","last_fullsized":"2024-09-13T13:09:34.528145+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:27.844003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:27.844003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:31.056528+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":378,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528976+0000","last_change":"2024-09-13T13:09:31.943644+0000","last_active":"2024-09-13T13:09:34.528976+0000","last_peered":"2024-09-13T13:09:34.528976+0000","last_clean":"2024-09-13T13:09:34.528976+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:09:34.528976+0000","last_undegraded":"2024-09-13T13:09:34.528976+0000","last_fullsized":"2024-09-13T13:09:34.528976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:31.943568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:31.943568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.734605+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":371,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.538239+0000","last_change":"2024-09-13T13:08:27.641188+0000","last_active":"2024-09-13T13:09:34.538239+0000","last_peered":"2024-09-13T13:09:34.538239+0000","last_clean":"2024-09-13T13:09:34.538239+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:09:34.538239+0000","last_undegraded":"2024-09-13T13:09:34.538239+0000","last_fullsized":"2024-09-13T13:09:34.538239+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:27.641105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:27.641105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:53.127406+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":374,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:35.355561+0000","last_change":"2024-09-13T13:09:35.355561+0000","last_active":"2024-09-13T13:09:35.355561+0000","last_peered":"2024-09-13T13:09:35.355561+0000","last_clean":"2024-09-13T13:09:35.355561+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:09:35.355561+0000","last_undegraded":"2024-09-13T13:09:35.355561+0000","last_fullsized":"2024-09-13T13:09:35.355561+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:35.355369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:35.355369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:52.921712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":370,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528167+0000","last_change":"2024-09-13T13:06:09.928585+0000","last_active":"2024-09-13T13:09:34.528167+0000","last_peered":"2024-09-13T13:09:34.528167+0000","last_clean":"2024-09-13T13:09:34.528167+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:09:34.528167+0000","last_undegraded":"2024-09-13T13:09:34.528167+0000","last_fullsized":"2024-09-13T13:09:34.528167+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:06:09.928504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.928504+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:42.064985+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":383,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.337230+0000","last_change":"2024-09-13T13:09:38.337230+0000","last_active":"2024-09-13T13:09:38.337230+0000","last_peered":"2024-09-13T13:09:38.337230+0000","last_clean":"2024-09-13T13:09:38.337230+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:09:38.337230+0000","last_undegraded":"2024-09-13T13:09:38.337230+0000","last_fullsized":"2024-09-13T13:09:38.337230+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.337156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.337156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:40.248177+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1176109080,"num_objects":17947,"num_object_clones":0,"num_object_copies":53841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237907,"num_write_kb":8187265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3533320192,"data_stored":3533307976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85807,"ondisk_log_size":85807,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":509,"num_read_kb":504,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201386,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":555328,"kb_used_data":365808,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93189824,"statfs":{"total":95995035648,"available":95426379776,"internally_reserved":0,"allocated":374587392,"data_stored":374174343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":193984970},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806140996,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":665520,"kb_used_data":415580,"kb_used_omap":4,"kb_used_meta":249851,"kb_avail":93079632,"statfs":{"total":95995035648,"available":95313543168,"internally_reserved":0,"allocated":425553920,"data_stored":425159698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":255848109},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691958,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":856372,"kb_used_data":586340,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92888780,"statfs":{"total":95995035648,"available":95118110720,"internally_reserved":0,"allocated":600412160,"data_stored":600009695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920892,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":553540,"kb_used_data":364084,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93191612,"statfs":{"total":95995035648,"available":95428210688,"internally_reserved":0,"allocated":372822016,"data_stored":372403279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":193984966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117123,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":760136,"kb_used_data":422008,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92985016,"statfs":{"total":95995035648,"available":95216656384,"internally_reserved":0,"allocated":432136192,"data_stored":431714189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":346094023},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411462,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":697636,"kb_used_data":431828,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":93047516,"statfs":{"total":95995035648,"available":95280656384,"internally_reserved":0,"allocated":442191872,"data_stored":441805131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":272169414},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574989,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808856,"kb_used_data":506952,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92936296,"statfs":{"total":95995035648,"available":95166767104,"internally_reserved":0,"allocated":519118848,"data_stored":518680142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738516,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":725376,"kb_used_data":427632,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93019776,"statfs":{"total":95995035648,"available":95252250624,"internally_reserved":0,"allocated":437895168,"data_stored":437481804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":418906112,"data_stored":418906112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":503119872,"data_stored":503119872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":437719040,"data_stored":437714968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":425852928,"data_stored":425852928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":366346240,"data_stored":366346240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":589107200,"data_stored":589103128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":420937728,"data_stored":420937728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":371331072,"data_stored":371327000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:09:42.051 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:09:42.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:42.082 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:42.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:42.184 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:42.205 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[36682]: 5.5 scrub starts 2024-09-13T13:09:42.205 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[36682]: 5.5 scrub ok 2024-09-13T13:09:42.205 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[36682]: 5.17 scrub starts 2024-09-13T13:09:42.205 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[36682]: 5.17 scrub ok 2024-09-13T13:09:42.205 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2838480350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:42.206 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[36682]: 5.9 scrub starts 2024-09-13T13:09:42.206 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[36682]: 5.9 scrub ok 2024-09-13T13:09:42.207 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[29471]: 5.5 scrub starts 2024-09-13T13:09:42.207 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[29471]: 5.5 scrub ok 2024-09-13T13:09:42.208 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[29471]: 5.17 scrub starts 2024-09-13T13:09:42.208 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[29471]: 5.17 scrub ok 2024-09-13T13:09:42.208 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2838480350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:42.208 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[29471]: 5.9 scrub starts 2024-09-13T13:09:42.208 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:41 smithi052 ceph-mon[29471]: 5.9 scrub ok 2024-09-13T13:09:42.242 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:42.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:42.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:42.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:41 smithi167 ceph-mon[33178]: 5.5 scrub starts 2024-09-13T13:09:42.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:41 smithi167 ceph-mon[33178]: 5.5 scrub ok 2024-09-13T13:09:42.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:41 smithi167 ceph-mon[33178]: 5.17 scrub starts 2024-09-13T13:09:42.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:41 smithi167 ceph-mon[33178]: 5.17 scrub ok 2024-09-13T13:09:42.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:41 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2838480350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:42.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:41 smithi167 ceph-mon[33178]: 5.9 scrub starts 2024-09-13T13:09:42.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:41 smithi167 ceph-mon[33178]: 5.9 scrub ok 2024-09-13T13:09:42.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:42.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:42.366 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:42.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:42.467 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:42.507 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:42.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:42.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:42.609 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:42.645 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:42.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:42.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:42.747 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:42.776 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:42.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:42.878 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:42.909 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:42.912 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:42.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:42.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:42.921 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:09:43.013 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:43.042 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:43.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:43.144 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:43.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:43.195 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:43.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:42 smithi052 ceph-mon[36682]: pgmap v408: 148 pgs: 148 active+clean; 1.1 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-13T13:09:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:42 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3168583316' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:09:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:42 smithi052 ceph-mon[36682]: from='client.35573 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:09:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:42 smithi052 ceph-mon[29471]: pgmap v408: 148 pgs: 148 active+clean; 1.1 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-13T13:09:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3168583316' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:09:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:42 smithi052 ceph-mon[29471]: from='client.35573 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:09:43.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:43.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:42 smithi167 ceph-mon[33178]: pgmap v408: 148 pgs: 148 active+clean; 1.1 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-13T13:09:43.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:42 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3168583316' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:09:43.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:42 smithi167 ceph-mon[33178]: from='client.35573 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:09:43.331 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:43.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:43.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:43.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:43.465 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:43.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:43.566 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:43.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:43.595 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:43.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:43.696 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:43.733 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:43.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:43.736 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:09:43.736 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:09:43.835 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:43.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:43.874 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:43.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:43.975 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:44.002 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:44.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:44.012 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:43 smithi167 ceph-mon[33178]: 4.1d scrub starts 2024-09-13T13:09:44.012 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:43 smithi167 ceph-mon[33178]: 4.1d scrub ok 2024-09-13T13:09:44.012 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:43 smithi167 ceph-mon[33178]: 8.5 scrub starts 2024-09-13T13:09:44.012 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:43 smithi167 ceph-mon[33178]: 6.1a scrub starts 2024-09-13T13:09:44.012 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:43 smithi167 ceph-mon[33178]: 6.1a scrub ok 2024-09-13T13:09:44.012 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:43 smithi167 ceph-mon[33178]: 8.7 scrub starts 2024-09-13T13:09:44.012 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:43 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:09:44.012 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:43 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:09:44.012 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:43 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:44.012 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:43 smithi167 ceph-mon[33178]: from='client.35573 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:09:44.012 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:43 smithi167 ceph-mon[33178]: osdmap e233: 8 total, 8 up, 8 in 2024-09-13T13:09:44.013 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:43 smithi167 ceph-mon[33178]: 4.15 scrub starts 2024-09-13T13:09:44.105 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:44.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:44.141 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:44.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:44.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[36682]: 4.1d scrub starts 2024-09-13T13:09:44.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[36682]: 4.1d scrub ok 2024-09-13T13:09:44.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[36682]: 8.5 scrub starts 2024-09-13T13:09:44.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[36682]: 6.1a scrub starts 2024-09-13T13:09:44.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[36682]: 6.1a scrub ok 2024-09-13T13:09:44.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[36682]: 8.7 scrub starts 2024-09-13T13:09:44.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:09:44.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:09:44.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:44.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[36682]: from='client.35573 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:09:44.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[36682]: osdmap e233: 8 total, 8 up, 8 in 2024-09-13T13:09:44.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[36682]: 4.15 scrub starts 2024-09-13T13:09:44.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[29471]: 4.1d scrub starts 2024-09-13T13:09:44.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[29471]: 4.1d scrub ok 2024-09-13T13:09:44.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[29471]: 8.5 scrub starts 2024-09-13T13:09:44.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[29471]: 6.1a scrub starts 2024-09-13T13:09:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[29471]: 6.1a scrub ok 2024-09-13T13:09:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[29471]: 8.7 scrub starts 2024-09-13T13:09:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:09:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:09:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[29471]: from='client.35573 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:09:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[29471]: osdmap e233: 8 total, 8 up, 8 in 2024-09-13T13:09:44.243 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:43 smithi052 ceph-mon[29471]: 4.15 scrub starts 2024-09-13T13:09:44.243 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:44.276 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:44.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:44.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:44.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:09:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:09:44.378 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:44.413 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:44.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:44.516 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:44.558 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:44.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:44.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:44.660 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:44.696 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:44.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:44.780 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:09:44.785 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T13:09:44.798 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:44.822 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:09:44.826 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:09:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:44.872 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:44.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:44.972 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:45.008 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:45.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:45.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:44 smithi167 ceph-mon[33178]: pgmap v410: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 830 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-09-13T13:09:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:44 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:45.110 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:45.152 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:45.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:45.176 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:09:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:44 smithi052 ceph-mon[36682]: pgmap v410: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 830 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-09-13T13:09:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:44 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:45.243 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:44 smithi052 ceph-mon[29471]: pgmap v410: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 830 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-09-13T13:09:45.243 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:44 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:45.254 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:45.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:45.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:45.389 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:45.426 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:45.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:45.528 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:45.587 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:45.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:45.689 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:45.727 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:45.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:45.796 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:09:45.797 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:09:45.828 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:45.836 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-09-13T13:09:43.781739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":870652467,"num_objects":13476,"num_object_clones":0,"num_object_copies":40428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13476,"num_whiteouts":0,"num_read":1361,"num_read_kb":1218,"num_write":243126,"num_write_kb":8188938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86137,"ondisk_log_size":86137,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4744776,"kb_used_data":2600656,"kb_used_omap":15,"kb_used_meta":2143664,"kb_avail":745216440,"statfs":{"total":767960285184,"available":763101634560,"internally_reserved":0,"allocated":2663071744,"data_stored":2659880075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":2195112729},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,4,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-668598272,"num_objects":-10202,"num_object_clones":0,"num_object_copies":-30606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10202,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10202,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.258256"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":375,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531661+0000","last_change":"2024-09-13T13:06:44.304525+0000","last_active":"2024-09-13T13:09:34.531661+0000","last_peered":"2024-09-13T13:09:34.531661+0000","last_clean":"2024-09-13T13:09:34.531661+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:09:34.531661+0000","last_undegraded":"2024-09-13T13:09:34.531661+0000","last_fullsized":"2024-09-13T13:09:34.531661+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:44.304444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.304444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.250196+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":400,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.916019+0000","last_change":"2024-09-13T13:09:06.567774+0000","last_active":"2024-09-13T13:09:42.916019+0000","last_peered":"2024-09-13T13:09:42.916019+0000","last_clean":"2024-09-13T13:09:42.916019+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:09:42.916019+0000","last_undegraded":"2024-09-13T13:09:42.916019+0000","last_fullsized":"2024-09-13T13:09:42.916019+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:06.567719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:06.567719+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:30.867489+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":369,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.923168+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:09:42.923168+0000","last_peered":"2024-09-13T13:09:42.923168+0000","last_clean":"2024-09-13T13:09:42.923168+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:09:42.923168+0000","last_undegraded":"2024-09-13T13:09:42.923168+0000","last_fullsized":"2024-09-13T13:09:42.923168+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:08.843909+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":365,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.916073+0000","last_change":"2024-09-13T13:09:42.788562+0000","last_active":"2024-09-13T13:09:42.916073+0000","last_peered":"2024-09-13T13:09:42.916073+0000","last_clean":"2024-09-13T13:09:42.916073+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:09:42.916073+0000","last_undegraded":"2024-09-13T13:09:42.916073+0000","last_fullsized":"2024-09-13T13:09:42.916073+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:42.787492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:42.787492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:47.723359+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":383,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.927586+0000","last_change":"2024-09-13T13:08:55.504675+0000","last_active":"2024-09-13T13:09:42.927586+0000","last_peered":"2024-09-13T13:09:42.927586+0000","last_clean":"2024-09-13T13:09:42.927586+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:09:42.927586+0000","last_undegraded":"2024-09-13T13:09:42.927586+0000","last_fullsized":"2024-09-13T13:09:42.927586+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.504475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.504475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.958611+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":388,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.930287+0000","last_change":"2024-09-13T13:09:31.491481+0000","last_active":"2024-09-13T13:09:42.930287+0000","last_peered":"2024-09-13T13:09:42.930287+0000","last_clean":"2024-09-13T13:09:42.930287+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:09:42.930287+0000","last_undegraded":"2024-09-13T13:09:42.930287+0000","last_fullsized":"2024-09-13T13:09:42.930287+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:31.491404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:31.491404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.774257+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":413,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.926655+0000","last_change":"2024-09-13T13:08:53.510436+0000","last_active":"2024-09-13T13:09:42.926655+0000","last_peered":"2024-09-13T13:09:42.926655+0000","last_clean":"2024-09-13T13:09:42.926655+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:09:42.926655+0000","last_undegraded":"2024-09-13T13:09:42.926655+0000","last_fullsized":"2024-09-13T13:09:42.926655+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:08:53.510335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.510335+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.247321+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":388,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.915900+0000","last_change":"2024-09-13T13:08:55.513716+0000","last_active":"2024-09-13T13:09:42.915900+0000","last_peered":"2024-09-13T13:09:42.915900+0000","last_clean":"2024-09-13T13:09:42.915900+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:09:42.915900+0000","last_undegraded":"2024-09-13T13:09:42.915900+0000","last_fullsized":"2024-09-13T13:09:42.915900+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.513667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.513667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.654393+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":409,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.925308+0000","last_change":"2024-09-13T13:08:59.654555+0000","last_active":"2024-09-13T13:09:42.925308+0000","last_peered":"2024-09-13T13:09:42.925308+0000","last_clean":"2024-09-13T13:09:42.925308+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:09:42.925308+0000","last_undegraded":"2024-09-13T13:09:42.925308+0000","last_fullsized":"2024-09-13T13:09:42.925308+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:08:59.654388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.654388+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.560420+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":405,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.916379+0000","last_change":"2024-09-13T13:09:00.426054+0000","last_active":"2024-09-13T13:09:42.916379+0000","last_peered":"2024-09-13T13:09:42.916379+0000","last_clean":"2024-09-13T13:09:42.916379+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:09:42.916379+0000","last_undegraded":"2024-09-13T13:09:42.916379+0000","last_fullsized":"2024-09-13T13:09:42.916379+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:00.425965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:00.425965+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:24.968404+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":376,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.927795+0000","last_change":"2024-09-13T13:06:50.665256+0000","last_active":"2024-09-13T13:09:42.927795+0000","last_peered":"2024-09-13T13:09:42.927795+0000","last_clean":"2024-09-13T13:09:42.927795+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:09:42.927795+0000","last_undegraded":"2024-09-13T13:09:42.927795+0000","last_fullsized":"2024-09-13T13:09:42.927795+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:50.665191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.665191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.666378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":370,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.524424+0000","last_change":"2024-09-13T13:08:49.280146+0000","last_active":"2024-09-13T13:09:34.524424+0000","last_peered":"2024-09-13T13:09:34.524424+0000","last_clean":"2024-09-13T13:09:34.524424+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:09:34.524424+0000","last_undegraded":"2024-09-13T13:09:34.524424+0000","last_fullsized":"2024-09-13T13:09:34.524424+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:49.280073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:49.280073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.252935+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":392,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.930552+0000","last_change":"2024-09-13T13:07:57.425510+0000","last_active":"2024-09-13T13:09:42.930552+0000","last_peered":"2024-09-13T13:09:42.930552+0000","last_clean":"2024-09-13T13:09:42.930552+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:09:42.930552+0000","last_undegraded":"2024-09-13T13:09:42.930552+0000","last_fullsized":"2024-09-13T13:09:42.930552+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.425436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.425436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:24.739469+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":383,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.916243+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:09:42.916243+0000","last_peered":"2024-09-13T13:09:42.916243+0000","last_clean":"2024-09-13T13:09:42.916243+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:09:42.916243+0000","last_undegraded":"2024-09-13T13:09:42.916243+0000","last_fullsized":"2024-09-13T13:09:42.916243+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":376,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929702+0000","last_change":"2024-09-13T13:07:50.530956+0000","last_active":"2024-09-13T13:09:42.929702+0000","last_peered":"2024-09-13T13:09:42.929702+0000","last_clean":"2024-09-13T13:09:42.929702+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:09:42.929702+0000","last_undegraded":"2024-09-13T13:09:42.929702+0000","last_fullsized":"2024-09-13T13:09:42.929702+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:50.530904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:50.530904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.091714+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":356,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.925266+0000","last_change":"2024-09-13T13:07:19.641119+0000","last_active":"2024-09-13T13:09:42.925266+0000","last_peered":"2024-09-13T13:09:42.925266+0000","last_clean":"2024-09-13T13:09:42.925266+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:09:42.925266+0000","last_undegraded":"2024-09-13T13:09:42.925266+0000","last_fullsized":"2024-09-13T13:09:42.925266+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:19.640697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:19.640697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":386,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.917441+0000","last_change":"2024-09-13T13:07:46.772159+0000","last_active":"2024-09-13T13:09:42.917441+0000","last_peered":"2024-09-13T13:09:42.917441+0000","last_clean":"2024-09-13T13:09:42.917441+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:09:42.917441+0000","last_undegraded":"2024-09-13T13:09:42.917441+0000","last_fullsized":"2024-09-13T13:09:42.917441+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:07:46.772111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:46.772111+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:22.644441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":382,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526960+0000","last_change":"2024-09-13T13:08:19.181017+0000","last_active":"2024-09-13T13:09:34.526960+0000","last_peered":"2024-09-13T13:09:34.526960+0000","last_clean":"2024-09-13T13:09:34.526960+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:09:34.526960+0000","last_undegraded":"2024-09-13T13:09:34.526960+0000","last_fullsized":"2024-09-13T13:09:34.526960+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:19.180948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.180948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:43.499202+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":414,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.932306+0000","last_change":"2024-09-13T13:06:44.532305+0000","last_active":"2024-09-13T13:09:42.932306+0000","last_peered":"2024-09-13T13:09:42.932306+0000","last_clean":"2024-09-13T13:09:42.932306+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:09:42.932306+0000","last_undegraded":"2024-09-13T13:09:42.932306+0000","last_fullsized":"2024-09-13T13:09:42.932306+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:06:44.532186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.532186+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.375109+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":377,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.922260+0000","last_change":"2024-09-13T13:08:18.639588+0000","last_active":"2024-09-13T13:09:42.922260+0000","last_peered":"2024-09-13T13:09:42.922260+0000","last_clean":"2024-09-13T13:09:42.922260+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:09:42.922260+0000","last_undegraded":"2024-09-13T13:09:42.922260+0000","last_fullsized":"2024-09-13T13:09:42.922260+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:18.639517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:18.639517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":384,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.927521+0000","last_change":"2024-09-13T13:09:29.517189+0000","last_active":"2024-09-13T13:09:42.927521+0000","last_peered":"2024-09-13T13:09:42.927521+0000","last_clean":"2024-09-13T13:09:42.927521+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:09:42.927521+0000","last_undegraded":"2024-09-13T13:09:42.927521+0000","last_fullsized":"2024-09-13T13:09:42.927521+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:29.517034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:29.517034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:34.784832+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":385,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.916514+0000","last_change":"2024-09-13T13:09:28.614216+0000","last_active":"2024-09-13T13:09:42.916514+0000","last_peered":"2024-09-13T13:09:42.916514+0000","last_clean":"2024-09-13T13:09:42.916514+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:09:42.916514+0000","last_undegraded":"2024-09-13T13:09:42.916514+0000","last_fullsized":"2024-09-13T13:09:42.916514+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:28.614027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.614027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:31.496891+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":395,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.928802+0000","last_change":"2024-09-13T13:09:42.310274+0000","last_active":"2024-09-13T13:09:42.928802+0000","last_peered":"2024-09-13T13:09:42.928802+0000","last_clean":"2024-09-13T13:09:42.928802+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:09:42.928802+0000","last_undegraded":"2024-09-13T13:09:42.928802+0000","last_fullsized":"2024-09-13T13:09:42.928802+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:09:42.310185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:42.310185+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:09.881670+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":377,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.916913+0000","last_change":"2024-09-13T13:07:12.879813+0000","last_active":"2024-09-13T13:09:42.916913+0000","last_peered":"2024-09-13T13:09:42.916913+0000","last_clean":"2024-09-13T13:09:42.916913+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:09:42.916913+0000","last_undegraded":"2024-09-13T13:09:42.916913+0000","last_fullsized":"2024-09-13T13:09:42.916913+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.879727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.879727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":381,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929897+0000","last_change":"2024-09-13T13:08:59.165835+0000","last_active":"2024-09-13T13:09:42.929897+0000","last_peered":"2024-09-13T13:09:42.929897+0000","last_clean":"2024-09-13T13:09:42.929897+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:09:42.929897+0000","last_undegraded":"2024-09-13T13:09:42.929897+0000","last_fullsized":"2024-09-13T13:09:42.929897+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:08:59.165754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.165754+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.048878+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":378,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.923306+0000","last_change":"2024-09-13T13:07:54.442276+0000","last_active":"2024-09-13T13:09:42.923306+0000","last_peered":"2024-09-13T13:09:42.923306+0000","last_clean":"2024-09-13T13:09:42.923306+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:09:42.923306+0000","last_undegraded":"2024-09-13T13:09:42.923306+0000","last_fullsized":"2024-09-13T13:09:42.923306+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:07:54.439269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:54.439269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.738173+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":402,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531537+0000","last_change":"2024-09-13T13:09:28.434007+0000","last_active":"2024-09-13T13:09:34.531537+0000","last_peered":"2024-09-13T13:09:34.531537+0000","last_clean":"2024-09-13T13:09:34.531537+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:09:34.531537+0000","last_undegraded":"2024-09-13T13:09:34.531537+0000","last_fullsized":"2024-09-13T13:09:34.531537+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:28.433899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.433899+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.061308+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":436,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:39.831623+0000","last_change":"2024-09-13T13:09:39.831623+0000","last_active":"2024-09-13T13:09:39.831623+0000","last_peered":"2024-09-13T13:09:39.831623+0000","last_clean":"2024-09-13T13:09:39.831623+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:09:39.831623+0000","last_undegraded":"2024-09-13T13:09:39.831623+0000","last_fullsized":"2024-09-13T13:09:39.831623+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:09:39.831547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:39.831547+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.629532+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"233'15410","reported_seq":15573,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:43.456870+0000","last_change":"2024-09-13T13:09:34.404950+0000","last_active":"2024-09-13T13:09:43.456870+0000","last_peered":"2024-09-13T13:09:43.456870+0000","last_clean":"2024-09-13T13:09:43.456870+0000","last_became_active":"2024-09-13T13:06:21.814002+0000","last_became_peered":"2024-09-13T13:06:21.814002+0000","last_unstale":"2024-09-13T13:09:43.456870+0000","last_undegraded":"2024-09-13T13:09:43.456870+0000","last_fullsized":"2024-09-13T13:09:43.456870+0000","mapping_epoch":169,"log_start":"214'10400","ondisk_log_start":"214'10400","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'14788","last_scrub_stamp":"2024-09-13T13:09:34.404844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:09:34.404844+0000","objects_scrubbed":1367,"log_size":5010,"log_dups_size":0,"ondisk_log_size":5010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:36.235023+0000","scrub_duration":561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47710208,"num_objects":728,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15410,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":406,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.923694+0000","last_change":"2024-09-13T13:07:15.243958+0000","last_active":"2024-09-13T13:09:42.923694+0000","last_peered":"2024-09-13T13:09:42.923694+0000","last_clean":"2024-09-13T13:09:42.923694+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:09:42.923694+0000","last_undegraded":"2024-09-13T13:09:42.923694+0000","last_fullsized":"2024-09-13T13:09:42.923694+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:07:15.243715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:15.243715+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":393,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.927042+0000","last_change":"2024-09-13T13:08:19.560359+0000","last_active":"2024-09-13T13:09:42.927042+0000","last_peered":"2024-09-13T13:09:42.927042+0000","last_clean":"2024-09-13T13:09:42.927042+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:09:42.927042+0000","last_undegraded":"2024-09-13T13:09:42.927042+0000","last_fullsized":"2024-09-13T13:09:42.927042+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:19.560157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.560157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":490,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.917523+0000","last_change":"2024-09-13T13:08:53.508763+0000","last_active":"2024-09-13T13:09:42.917523+0000","last_peered":"2024-09-13T13:09:42.917523+0000","last_clean":"2024-09-13T13:09:42.917523+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:09:42.917523+0000","last_undegraded":"2024-09-13T13:09:42.917523+0000","last_fullsized":"2024-09-13T13:09:42.917523+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:08:53.508690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.508690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:06.126931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":372,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.924860+0000","last_change":"2024-09-13T13:08:05.356744+0000","last_active":"2024-09-13T13:09:42.924860+0000","last_peered":"2024-09-13T13:09:42.924860+0000","last_clean":"2024-09-13T13:09:42.924860+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:09:42.924860+0000","last_undegraded":"2024-09-13T13:09:42.924860+0000","last_fullsized":"2024-09-13T13:09:42.924860+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:05.356688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:05.356688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"232'14839","reported_seq":14977,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.916181+0000","last_change":"2024-09-13T13:06:22.000536+0000","last_active":"2024-09-13T13:09:38.916181+0000","last_peered":"2024-09-13T13:09:38.916181+0000","last_clean":"2024-09-13T13:09:38.916181+0000","last_became_active":"2024-09-13T13:06:22.000244+0000","last_became_peered":"2024-09-13T13:06:22.000244+0000","last_unstale":"2024-09-13T13:09:38.916181+0000","last_undegraded":"2024-09-13T13:09:38.916181+0000","last_fullsized":"2024-09-13T13:09:38.916181+0000","mapping_epoch":169,"log_start":"208'9000","ondisk_log_start":"208'9000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:47.271652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63504384,"num_objects":969,"num_object_clones":0,"num_object_copies":2907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14839,"num_write_kb":505856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":397,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.926256+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:09:42.926256+0000","last_peered":"2024-09-13T13:09:42.926256+0000","last_clean":"2024-09-13T13:09:42.926256+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:09:42.926256+0000","last_undegraded":"2024-09-13T13:09:42.926256+0000","last_fullsized":"2024-09-13T13:09:42.926256+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.546922+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":383,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.927926+0000","last_change":"2024-09-13T13:06:44.785235+0000","last_active":"2024-09-13T13:09:42.927926+0000","last_peered":"2024-09-13T13:09:42.927926+0000","last_clean":"2024-09-13T13:09:42.927926+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:09:42.927926+0000","last_undegraded":"2024-09-13T13:09:42.927926+0000","last_fullsized":"2024-09-13T13:09:42.927926+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:06:44.785017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.785017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.624176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":370,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.916039+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:09:42.916039+0000","last_peered":"2024-09-13T13:09:42.916039+0000","last_clean":"2024-09-13T13:09:42.916039+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:09:42.916039+0000","last_undegraded":"2024-09-13T13:09:42.916039+0000","last_fullsized":"2024-09-13T13:09:42.916039+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.668051+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":374,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.916293+0000","last_change":"2024-09-13T13:08:22.683148+0000","last_active":"2024-09-13T13:09:42.916293+0000","last_peered":"2024-09-13T13:09:42.916293+0000","last_clean":"2024-09-13T13:09:42.916293+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:09:42.916293+0000","last_undegraded":"2024-09-13T13:09:42.916293+0000","last_fullsized":"2024-09-13T13:09:42.916293+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:22.683019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:22.683019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"233'15117","reported_seq":15258,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:43.091939+0000","last_change":"2024-09-13T13:06:22.000928+0000","last_active":"2024-09-13T13:09:43.091939+0000","last_peered":"2024-09-13T13:09:43.091939+0000","last_clean":"2024-09-13T13:09:43.091939+0000","last_became_active":"2024-09-13T13:06:22.000610+0000","last_became_peered":"2024-09-13T13:06:22.000610+0000","last_unstale":"2024-09-13T13:09:43.091939+0000","last_undegraded":"2024-09-13T13:09:43.091939+0000","last_fullsized":"2024-09-13T13:09:43.091939+0000","mapping_epoch":169,"log_start":"211'9700","ondisk_log_start":"211'9700","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:18.686758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53477400,"num_objects":817,"num_object_clones":0,"num_object_copies":2451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15117,"num_write_kb":509825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":414,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.916126+0000","last_change":"2024-09-13T13:07:10.853834+0000","last_active":"2024-09-13T13:09:42.916126+0000","last_peered":"2024-09-13T13:09:42.916126+0000","last_clean":"2024-09-13T13:09:42.916126+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:09:42.916126+0000","last_undegraded":"2024-09-13T13:09:42.916126+0000","last_fullsized":"2024-09-13T13:09:42.916126+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:07:10.853749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.853749+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:45.529858+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":367,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.532963+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:09:34.532963+0000","last_peered":"2024-09-13T13:09:34.532963+0000","last_clean":"2024-09-13T13:09:34.532963+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:09:34.532963+0000","last_undegraded":"2024-09-13T13:09:34.532963+0000","last_fullsized":"2024-09-13T13:09:34.532963+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.235292+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":482,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.925906+0000","last_change":"2024-09-13T13:09:28.477414+0000","last_active":"2024-09-13T13:09:42.925906+0000","last_peered":"2024-09-13T13:09:42.925906+0000","last_clean":"2024-09-13T13:09:42.925906+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:09:42.925906+0000","last_undegraded":"2024-09-13T13:09:42.925906+0000","last_fullsized":"2024-09-13T13:09:42.925906+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:09:28.477338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.477338+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:42.900990+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":376,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929084+0000","last_change":"2024-09-13T13:06:09.230872+0000","last_active":"2024-09-13T13:09:42.929084+0000","last_peered":"2024-09-13T13:09:42.929084+0000","last_clean":"2024-09-13T13:09:42.929084+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:09:42.929084+0000","last_undegraded":"2024-09-13T13:09:42.929084+0000","last_fullsized":"2024-09-13T13:09:42.929084+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:09.230824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.230824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:43.287111+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"232'14839","reported_seq":14977,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.926622+0000","last_change":"2024-09-13T13:06:21.812511+0000","last_active":"2024-09-13T13:09:38.926622+0000","last_peered":"2024-09-13T13:09:38.926622+0000","last_clean":"2024-09-13T13:09:38.926622+0000","last_became_active":"2024-09-13T13:06:21.812226+0000","last_became_peered":"2024-09-13T13:06:21.812226+0000","last_unstale":"2024-09-13T13:09:38.926622+0000","last_undegraded":"2024-09-13T13:09:38.926622+0000","last_fullsized":"2024-09-13T13:09:38.926622+0000","mapping_epoch":169,"log_start":"209'9000","ondisk_log_start":"209'9000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:41.321590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68222976,"num_objects":1041,"num_object_clones":0,"num_object_copies":3123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14839,"num_write_kb":508160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":397,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.922954+0000","last_change":"2024-09-13T13:09:27.854186+0000","last_active":"2024-09-13T13:09:42.922954+0000","last_peered":"2024-09-13T13:09:42.922954+0000","last_clean":"2024-09-13T13:09:42.922954+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:09:42.922954+0000","last_undegraded":"2024-09-13T13:09:42.922954+0000","last_fullsized":"2024-09-13T13:09:42.922954+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:27.854114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:27.854114+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:39.809612+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":385,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.925402+0000","last_change":"2024-09-13T13:08:17.298581+0000","last_active":"2024-09-13T13:09:42.925402+0000","last_peered":"2024-09-13T13:09:42.925402+0000","last_clean":"2024-09-13T13:09:42.925402+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:09:42.925402+0000","last_undegraded":"2024-09-13T13:09:42.925402+0000","last_fullsized":"2024-09-13T13:09:42.925402+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:17.298430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:17.298430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":485,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526796+0000","last_change":"2024-09-13T13:07:57.028959+0000","last_active":"2024-09-13T13:09:34.526796+0000","last_peered":"2024-09-13T13:09:34.526796+0000","last_clean":"2024-09-13T13:09:34.526796+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:09:34.526796+0000","last_undegraded":"2024-09-13T13:09:34.526796+0000","last_fullsized":"2024-09-13T13:09:34.526796+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:07:57.028875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.028875+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.174262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":379,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526458+0000","last_change":"2024-09-13T13:08:52.207903+0000","last_active":"2024-09-13T13:09:34.526458+0000","last_peered":"2024-09-13T13:09:34.526458+0000","last_clean":"2024-09-13T13:09:34.526458+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:09:34.526458+0000","last_undegraded":"2024-09-13T13:09:34.526458+0000","last_fullsized":"2024-09-13T13:09:34.526458+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:52.207841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:52.207841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.868681+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"233'15252","reported_seq":15444,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:43.527801+0000","last_change":"2024-09-13T13:08:51.116458+0000","last_active":"2024-09-13T13:09:43.527801+0000","last_peered":"2024-09-13T13:09:43.527801+0000","last_clean":"2024-09-13T13:09:43.527801+0000","last_became_active":"2024-09-13T13:06:21.812208+0000","last_became_peered":"2024-09-13T13:06:21.812208+0000","last_unstale":"2024-09-13T13:09:43.527801+0000","last_undegraded":"2024-09-13T13:09:43.527801+0000","last_fullsized":"2024-09-13T13:09:43.527801+0000","mapping_epoch":169,"log_start":"214'10400","ondisk_log_start":"214'10400","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'11587","last_scrub_stamp":"2024-09-13T13:08:51.116305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:08:51.116305+0000","objects_scrubbed":4435,"log_size":4852,"log_dups_size":0,"ondisk_log_size":4852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.050196+0000","scrub_duration":1647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47054848,"num_objects":718,"num_object_clones":0,"num_object_copies":2154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15252,"num_write_kb":511040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":423,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526850+0000","last_change":"2024-09-13T13:09:05.142515+0000","last_active":"2024-09-13T13:09:34.526850+0000","last_peered":"2024-09-13T13:09:34.526850+0000","last_clean":"2024-09-13T13:09:34.526850+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:09:34.526850+0000","last_undegraded":"2024-09-13T13:09:34.526850+0000","last_fullsized":"2024-09-13T13:09:34.526850+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:09:05.142427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.142427+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:06.191632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":370,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.922396+0000","last_change":"2024-09-13T13:09:38.867734+0000","last_active":"2024-09-13T13:09:42.922396+0000","last_peered":"2024-09-13T13:09:42.922396+0000","last_clean":"2024-09-13T13:09:42.922396+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:09:42.922396+0000","last_undegraded":"2024-09-13T13:09:42.922396+0000","last_fullsized":"2024-09-13T13:09:42.922396+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.867688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.867688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:54.044585+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":336,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.854654+0000","last_change":"2024-09-13T13:09:38.854654+0000","last_active":"2024-09-13T13:09:38.854654+0000","last_peered":"2024-09-13T13:09:38.854654+0000","last_clean":"2024-09-13T13:09:38.854654+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:09:38.854654+0000","last_undegraded":"2024-09-13T13:09:38.854654+0000","last_fullsized":"2024-09-13T13:09:38.854654+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.854581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.854581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:50.922952+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":385,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.923082+0000","last_change":"2024-09-13T13:06:43.969251+0000","last_active":"2024-09-13T13:09:42.923082+0000","last_peered":"2024-09-13T13:09:42.923082+0000","last_clean":"2024-09-13T13:09:42.923082+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:09:42.923082+0000","last_undegraded":"2024-09-13T13:09:42.923082+0000","last_fullsized":"2024-09-13T13:09:42.923082+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.969176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.969176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.710958+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"233'14990","reported_seq":15130,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:43.081702+0000","last_change":"2024-09-13T13:06:22.000658+0000","last_active":"2024-09-13T13:09:43.081702+0000","last_peered":"2024-09-13T13:09:43.081702+0000","last_clean":"2024-09-13T13:09:43.081702+0000","last_became_active":"2024-09-13T13:06:22.000261+0000","last_became_peered":"2024-09-13T13:06:22.000261+0000","last_unstale":"2024-09-13T13:09:43.081702+0000","last_undegraded":"2024-09-13T13:09:43.081702+0000","last_fullsized":"2024-09-13T13:09:43.081702+0000","mapping_epoch":169,"log_start":"212'9600","ondisk_log_start":"212'9600","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5390,"log_dups_size":0,"ondisk_log_size":5390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.720130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51642368,"num_objects":788,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14990,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":535,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.922256+0000","last_change":"2024-09-13T13:07:25.438610+0000","last_active":"2024-09-13T13:09:42.922256+0000","last_peered":"2024-09-13T13:09:42.922256+0000","last_clean":"2024-09-13T13:09:42.922256+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:09:42.922256+0000","last_undegraded":"2024-09-13T13:09:42.922256+0000","last_fullsized":"2024-09-13T13:09:42.922256+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:07:25.438528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:25.438528+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":397,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.917177+0000","last_change":"2024-09-13T13:07:18.888103+0000","last_active":"2024-09-13T13:09:42.917177+0000","last_peered":"2024-09-13T13:09:42.917177+0000","last_clean":"2024-09-13T13:09:42.917177+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:09:42.917177+0000","last_undegraded":"2024-09-13T13:09:42.917177+0000","last_fullsized":"2024-09-13T13:09:42.917177+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:18.888060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:18.888060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":388,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.930648+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:09:42.930648+0000","last_peered":"2024-09-13T13:09:42.930648+0000","last_clean":"2024-09-13T13:09:42.930648+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:09:42.930648+0000","last_undegraded":"2024-09-13T13:09:42.930648+0000","last_fullsized":"2024-09-13T13:09:42.930648+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":367,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.527094+0000","last_change":"2024-09-13T13:06:11.189922+0000","last_active":"2024-09-13T13:09:34.527094+0000","last_peered":"2024-09-13T13:09:34.527094+0000","last_clean":"2024-09-13T13:09:34.527094+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:09:34.527094+0000","last_undegraded":"2024-09-13T13:09:34.527094+0000","last_fullsized":"2024-09-13T13:09:34.527094+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:11.189874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:11.189874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:38.354895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":372,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929788+0000","last_change":"2024-09-13T13:09:00.570589+0000","last_active":"2024-09-13T13:09:42.929788+0000","last_peered":"2024-09-13T13:09:42.929788+0000","last_clean":"2024-09-13T13:09:42.929788+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:09:42.929788+0000","last_undegraded":"2024-09-13T13:09:42.929788+0000","last_fullsized":"2024-09-13T13:09:42.929788+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:00.570518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:00.570518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.670665+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"233'15148","reported_seq":15341,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:43.540374+0000","last_change":"2024-09-13T13:08:52.266749+0000","last_active":"2024-09-13T13:09:43.540374+0000","last_peered":"2024-09-13T13:09:43.540374+0000","last_clean":"2024-09-13T13:09:43.540374+0000","last_became_active":"2024-09-13T13:06:21.812869+0000","last_became_peered":"2024-09-13T13:06:21.812869+0000","last_unstale":"2024-09-13T13:09:43.540374+0000","last_undegraded":"2024-09-13T13:09:43.540374+0000","last_fullsized":"2024-09-13T13:09:43.540374+0000","mapping_epoch":169,"log_start":"214'10300","ondisk_log_start":"214'10300","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'11526","last_scrub_stamp":"2024-09-13T13:08:52.266659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:08:52.266659+0000","objects_scrubbed":4408,"log_size":4848,"log_dups_size":0,"ondisk_log_size":4848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.678811+0000","scrub_duration":1778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48365568,"num_objects":738,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15148,"num_write_kb":508352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":400,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.923715+0000","last_change":"2024-09-13T13:08:04.680025+0000","last_active":"2024-09-13T13:09:42.923715+0000","last_peered":"2024-09-13T13:09:42.923715+0000","last_clean":"2024-09-13T13:09:42.923715+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:09:42.923715+0000","last_undegraded":"2024-09-13T13:09:42.923715+0000","last_fullsized":"2024-09-13T13:09:42.923715+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:08:04.679766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:04.679766+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:22.837118+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":405,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.922558+0000","last_change":"2024-09-13T13:08:56.771534+0000","last_active":"2024-09-13T13:09:42.922558+0000","last_peered":"2024-09-13T13:09:42.922558+0000","last_clean":"2024-09-13T13:09:42.922558+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:09:42.922558+0000","last_undegraded":"2024-09-13T13:09:42.922558+0000","last_fullsized":"2024-09-13T13:09:42.922558+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:56.771432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:56.771432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.588929+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":401,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.930155+0000","last_change":"2024-09-13T13:08:19.038471+0000","last_active":"2024-09-13T13:09:42.930155+0000","last_peered":"2024-09-13T13:09:42.930155+0000","last_clean":"2024-09-13T13:09:42.930155+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:09:42.930155+0000","last_undegraded":"2024-09-13T13:09:42.930155+0000","last_fullsized":"2024-09-13T13:09:42.930155+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:08:19.038393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.038393+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:25.295684+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":362,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.935214+0000","last_change":"2024-09-13T13:03:14.282866+0000","last_active":"2024-09-13T13:09:42.935214+0000","last_peered":"2024-09-13T13:09:42.935214+0000","last_clean":"2024-09-13T13:09:42.935214+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:09:42.935214+0000","last_undegraded":"2024-09-13T13:09:42.935214+0000","last_fullsized":"2024-09-13T13:09:42.935214+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":384,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929227+0000","last_change":"2024-09-13T13:08:54.232191+0000","last_active":"2024-09-13T13:09:42.929227+0000","last_peered":"2024-09-13T13:09:42.929227+0000","last_clean":"2024-09-13T13:09:42.929227+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:09:42.929227+0000","last_undegraded":"2024-09-13T13:09:42.929227+0000","last_fullsized":"2024-09-13T13:09:42.929227+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:54.232130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:54.232130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.097713+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"233'15124","reported_seq":15343,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:43.454010+0000","last_change":"2024-09-13T13:07:49.115654+0000","last_active":"2024-09-13T13:09:43.454010+0000","last_peered":"2024-09-13T13:09:43.454010+0000","last_clean":"2024-09-13T13:09:43.454010+0000","last_became_active":"2024-09-13T13:06:21.814772+0000","last_became_peered":"2024-09-13T13:06:21.814772+0000","last_unstale":"2024-09-13T13:09:43.454010+0000","last_undegraded":"2024-09-13T13:09:43.454010+0000","last_fullsized":"2024-09-13T13:09:43.454010+0000","mapping_epoch":169,"log_start":"214'10200","ondisk_log_start":"214'10200","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'7112","last_scrub_stamp":"2024-09-13T13:07:49.115526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:07:49.115526+0000","objects_scrubbed":7029,"log_size":4924,"log_dups_size":0,"ondisk_log_size":4924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.794726+0000","scrub_duration":2765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47054848,"num_objects":718,"num_object_clones":0,"num_object_copies":2154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15124,"num_write_kb":506944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":396,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531619+0000","last_change":"2024-09-13T13:08:21.219398+0000","last_active":"2024-09-13T13:09:34.531619+0000","last_peered":"2024-09-13T13:09:34.531619+0000","last_clean":"2024-09-13T13:09:34.531619+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:09:34.531619+0000","last_undegraded":"2024-09-13T13:09:34.531619+0000","last_fullsized":"2024-09-13T13:09:34.531619+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:08:21.219300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:21.219300+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:38.637760+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":383,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526234+0000","last_change":"2024-09-13T13:08:02.152950+0000","last_active":"2024-09-13T13:09:34.526234+0000","last_peered":"2024-09-13T13:09:34.526234+0000","last_clean":"2024-09-13T13:09:34.526234+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:09:34.526234+0000","last_undegraded":"2024-09-13T13:09:34.526234+0000","last_fullsized":"2024-09-13T13:09:34.526234+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:02.152859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:02.152859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:25.356503+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":908,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929493+0000","last_change":"2024-09-13T13:08:49.264831+0000","last_active":"2024-09-13T13:09:42.929493+0000","last_peered":"2024-09-13T13:09:42.929493+0000","last_clean":"2024-09-13T13:09:42.929493+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:09:42.929493+0000","last_undegraded":"2024-09-13T13:09:42.929493+0000","last_fullsized":"2024-09-13T13:09:42.929493+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:08:49.264775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:49.264775+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.716838+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":514,"num_read_kb":509,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":360,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.925561+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:09:42.925561+0000","last_peered":"2024-09-13T13:09:42.925561+0000","last_clean":"2024-09-13T13:09:42.925561+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:09:42.925561+0000","last_undegraded":"2024-09-13T13:09:42.925561+0000","last_fullsized":"2024-09-13T13:09:42.925561+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":382,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.921865+0000","last_change":"2024-09-13T13:08:55.738138+0000","last_active":"2024-09-13T13:09:42.921865+0000","last_peered":"2024-09-13T13:09:42.921865+0000","last_clean":"2024-09-13T13:09:42.921865+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:09:42.921865+0000","last_undegraded":"2024-09-13T13:09:42.921865+0000","last_fullsized":"2024-09-13T13:09:42.921865+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.738069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.738069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:25.032322+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"232'14843","reported_seq":15002,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:40.956206+0000","last_change":"2024-09-13T13:09:38.386284+0000","last_active":"2024-09-13T13:09:40.956206+0000","last_peered":"2024-09-13T13:09:40.956206+0000","last_clean":"2024-09-13T13:09:40.956206+0000","last_became_active":"2024-09-13T13:06:21.814144+0000","last_became_peered":"2024-09-13T13:06:21.814144+0000","last_unstale":"2024-09-13T13:09:40.956206+0000","last_undegraded":"2024-09-13T13:09:40.956206+0000","last_fullsized":"2024-09-13T13:09:40.956206+0000","mapping_epoch":169,"log_start":"207'8500","ondisk_log_start":"207'8500","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'14664","last_scrub_stamp":"2024-09-13T13:09:38.386024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.386024+0000","objects_scrubbed":1158,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:01.089959+0000","scrub_duration":546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63504384,"num_objects":969,"num_object_clones":0,"num_object_copies":2907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14843,"num_write_kb":505984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":364,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528561+0000","last_change":"2024-09-13T13:06:51.193749+0000","last_active":"2024-09-13T13:09:34.528561+0000","last_peered":"2024-09-13T13:09:34.528561+0000","last_clean":"2024-09-13T13:09:34.528561+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:09:34.528561+0000","last_undegraded":"2024-09-13T13:09:34.528561+0000","last_fullsized":"2024-09-13T13:09:34.528561+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'18","last_scrub_stamp":"2024-09-13T13:06:51.193678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.193678+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.989653+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":388,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929500+0000","last_change":"2024-09-13T13:06:10.715371+0000","last_active":"2024-09-13T13:09:42.929500+0000","last_peered":"2024-09-13T13:09:42.929500+0000","last_clean":"2024-09-13T13:09:42.929500+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:09:42.929500+0000","last_undegraded":"2024-09-13T13:09:42.929500+0000","last_fullsized":"2024-09-13T13:09:42.929500+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:10.715313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.715313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:46.404429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":433,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929706+0000","last_change":"2024-09-13T13:06:39.564599+0000","last_active":"2024-09-13T13:09:42.929706+0000","last_peered":"2024-09-13T13:09:42.929706+0000","last_clean":"2024-09-13T13:09:42.929706+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:09:42.929706+0000","last_undegraded":"2024-09-13T13:09:42.929706+0000","last_fullsized":"2024-09-13T13:09:42.929706+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:06:39.564549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:06:39.564549+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:15.102832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":385,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.930061+0000","last_change":"2024-09-13T13:08:25.500731+0000","last_active":"2024-09-13T13:09:42.930061+0000","last_peered":"2024-09-13T13:09:42.930061+0000","last_clean":"2024-09-13T13:09:42.930061+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:09:42.930061+0000","last_undegraded":"2024-09-13T13:09:42.930061+0000","last_fullsized":"2024-09-13T13:09:42.930061+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:25.500660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:25.500660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:47.582413+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":380,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.925425+0000","last_change":"2024-09-13T13:08:20.287277+0000","last_active":"2024-09-13T13:09:42.925425+0000","last_peered":"2024-09-13T13:09:42.925425+0000","last_clean":"2024-09-13T13:09:42.925425+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:09:42.925425+0000","last_undegraded":"2024-09-13T13:09:42.925425+0000","last_fullsized":"2024-09-13T13:09:42.925425+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:20.287215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:20.287215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"233'15273","reported_seq":15433,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.964135+0000","last_change":"2024-09-13T13:07:26.381412+0000","last_active":"2024-09-13T13:09:42.964135+0000","last_peered":"2024-09-13T13:09:42.964135+0000","last_clean":"2024-09-13T13:09:42.964135+0000","last_became_active":"2024-09-13T13:06:21.812061+0000","last_became_peered":"2024-09-13T13:06:21.812061+0000","last_unstale":"2024-09-13T13:09:42.964135+0000","last_undegraded":"2024-09-13T13:09:42.964135+0000","last_fullsized":"2024-09-13T13:09:42.964135+0000","mapping_epoch":169,"log_start":"215'10500","ondisk_log_start":"215'10500","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":1649,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.683639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48955392,"num_objects":747,"num_object_clones":0,"num_object_copies":2241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15273,"num_write_kb":512640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":403,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.917604+0000","last_change":"2024-09-13T13:08:01.790332+0000","last_active":"2024-09-13T13:09:42.917604+0000","last_peered":"2024-09-13T13:09:42.917604+0000","last_clean":"2024-09-13T13:09:42.917604+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:09:42.917604+0000","last_undegraded":"2024-09-13T13:09:42.917604+0000","last_fullsized":"2024-09-13T13:09:42.917604+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:08:01.790280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:01.790280+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:24.650459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":386,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.916524+0000","last_change":"2024-09-13T13:08:51.546064+0000","last_active":"2024-09-13T13:09:42.916524+0000","last_peered":"2024-09-13T13:09:42.916524+0000","last_clean":"2024-09-13T13:09:42.916524+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:09:42.916524+0000","last_undegraded":"2024-09-13T13:09:42.916524+0000","last_fullsized":"2024-09-13T13:09:42.916524+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:51.546003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:51.546003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:59.798244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":387,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.925279+0000","last_change":"2024-09-13T13:07:23.602111+0000","last_active":"2024-09-13T13:09:42.925279+0000","last_peered":"2024-09-13T13:09:42.925279+0000","last_clean":"2024-09-13T13:09:42.925279+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:09:42.925279+0000","last_undegraded":"2024-09-13T13:09:42.925279+0000","last_fullsized":"2024-09-13T13:09:42.925279+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:23.602052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:23.602052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:43.434576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":375,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.927016+0000","last_change":"2024-09-13T13:08:27.451600+0000","last_active":"2024-09-13T13:09:42.927016+0000","last_peered":"2024-09-13T13:09:42.927016+0000","last_clean":"2024-09-13T13:09:42.927016+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:09:42.927016+0000","last_undegraded":"2024-09-13T13:09:42.927016+0000","last_fullsized":"2024-09-13T13:09:42.927016+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:27.448636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:27.448636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:45.909228+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"233'15356","reported_seq":15576,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:43.487124+0000","last_change":"2024-09-13T13:07:52.135686+0000","last_active":"2024-09-13T13:09:43.487124+0000","last_peered":"2024-09-13T13:09:43.487124+0000","last_clean":"2024-09-13T13:09:43.487124+0000","last_became_active":"2024-09-13T13:06:21.811526+0000","last_became_peered":"2024-09-13T13:06:21.811526+0000","last_unstale":"2024-09-13T13:09:43.487124+0000","last_undegraded":"2024-09-13T13:09:43.487124+0000","last_fullsized":"2024-09-13T13:09:43.487124+0000","mapping_epoch":169,"log_start":"214'10400","ondisk_log_start":"214'10400","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'7389","last_scrub_stamp":"2024-09-13T13:07:52.135527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:07:52.135527+0000","objects_scrubbed":7297,"log_size":4956,"log_dups_size":0,"ondisk_log_size":4956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.778806+0000","scrub_duration":2761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50462720,"num_objects":770,"num_object_clones":0,"num_object_copies":2310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15356,"num_write_kb":516032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":401,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.920711+0000","last_change":"2024-09-13T13:05:34.166464+0000","last_active":"2024-09-13T13:09:42.920711+0000","last_peered":"2024-09-13T13:09:42.920711+0000","last_clean":"2024-09-13T13:09:42.920711+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:09:42.920711+0000","last_undegraded":"2024-09-13T13:09:42.920711+0000","last_fullsized":"2024-09-13T13:09:42.920711+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.686209+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":360,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528019+0000","last_change":"2024-09-13T13:09:05.049507+0000","last_active":"2024-09-13T13:09:34.528019+0000","last_peered":"2024-09-13T13:09:34.528019+0000","last_clean":"2024-09-13T13:09:34.528019+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:09:34.528019+0000","last_undegraded":"2024-09-13T13:09:34.528019+0000","last_fullsized":"2024-09-13T13:09:34.528019+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:05.049446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.049446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.951824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":369,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.922073+0000","last_change":"2024-09-13T13:08:53.747370+0000","last_active":"2024-09-13T13:09:42.922073+0000","last_peered":"2024-09-13T13:09:42.922073+0000","last_clean":"2024-09-13T13:09:42.922073+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:09:42.922073+0000","last_undegraded":"2024-09-13T13:09:42.922073+0000","last_fullsized":"2024-09-13T13:09:42.922073+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.747302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.747302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.330071+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":373,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.935330+0000","last_change":"2024-09-13T13:08:03.015302+0000","last_active":"2024-09-13T13:09:42.935330+0000","last_peered":"2024-09-13T13:09:42.935330+0000","last_clean":"2024-09-13T13:09:42.935330+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:09:42.935330+0000","last_undegraded":"2024-09-13T13:09:42.935330+0000","last_fullsized":"2024-09-13T13:09:42.935330+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:03.015233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.015233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:23.253063+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"233'15088","reported_seq":15310,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:43.545564+0000","last_change":"2024-09-13T13:07:54.357736+0000","last_active":"2024-09-13T13:09:43.545564+0000","last_peered":"2024-09-13T13:09:43.545564+0000","last_clean":"2024-09-13T13:09:43.545564+0000","last_became_active":"2024-09-13T13:06:21.812966+0000","last_became_peered":"2024-09-13T13:06:21.812966+0000","last_unstale":"2024-09-13T13:09:43.545564+0000","last_undegraded":"2024-09-13T13:09:43.545564+0000","last_fullsized":"2024-09-13T13:09:43.545564+0000","mapping_epoch":169,"log_start":"214'10200","ondisk_log_start":"214'10200","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'7426","last_scrub_stamp":"2024-09-13T13:07:54.357373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:07:54.357373+0000","objects_scrubbed":7324,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48103424,"num_objects":734,"num_object_clones":0,"num_object_copies":2202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15088,"num_write_kb":506304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":415,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.917046+0000","last_change":"2024-09-13T13:09:05.619225+0000","last_active":"2024-09-13T13:09:42.917046+0000","last_peered":"2024-09-13T13:09:42.917046+0000","last_clean":"2024-09-13T13:09:42.917046+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:09:42.917046+0000","last_undegraded":"2024-09-13T13:09:42.917046+0000","last_fullsized":"2024-09-13T13:09:42.917046+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:05.619171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.619171+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:07.343034+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":376,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.928190+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:09:42.928190+0000","last_peered":"2024-09-13T13:09:42.928190+0000","last_clean":"2024-09-13T13:09:42.928190+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:09:42.928190+0000","last_undegraded":"2024-09-13T13:09:42.928190+0000","last_fullsized":"2024-09-13T13:09:42.928190+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:15.617560+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":482,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.931183+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:09:42.931183+0000","last_peered":"2024-09-13T13:09:42.931183+0000","last_clean":"2024-09-13T13:09:42.931183+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:09:42.931183+0000","last_undegraded":"2024-09-13T13:09:42.931183+0000","last_fullsized":"2024-09-13T13:09:42.931183+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.422585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":365,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.917353+0000","last_change":"2024-09-13T13:09:37.812544+0000","last_active":"2024-09-13T13:09:42.917353+0000","last_peered":"2024-09-13T13:09:42.917353+0000","last_clean":"2024-09-13T13:09:42.917353+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:09:42.917353+0000","last_undegraded":"2024-09-13T13:09:42.917353+0000","last_fullsized":"2024-09-13T13:09:42.917353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:37.812451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:37.812451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:00.130077+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"233'15546","reported_seq":15690,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:43.486544+0000","last_change":"2024-09-13T13:09:42.955518+0000","last_active":"2024-09-13T13:09:43.486544+0000","last_peered":"2024-09-13T13:09:43.486544+0000","last_clean":"2024-09-13T13:09:43.486544+0000","last_became_active":"2024-09-13T13:06:21.814433+0000","last_became_peered":"2024-09-13T13:06:21.814433+0000","last_unstale":"2024-09-13T13:09:43.486544+0000","last_undegraded":"2024-09-13T13:09:43.486544+0000","last_fullsized":"2024-09-13T13:09:43.486544+0000","mapping_epoch":169,"log_start":"214'10600","ondisk_log_start":"214'10600","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":99,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.924104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":740,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15546,"num_write_kb":521152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":378,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526803+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:09:34.526803+0000","last_peered":"2024-09-13T13:09:34.526803+0000","last_clean":"2024-09-13T13:09:34.526803+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:09:34.526803+0000","last_undegraded":"2024-09-13T13:09:34.526803+0000","last_fullsized":"2024-09-13T13:09:34.526803+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.254641+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":403,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.930676+0000","last_change":"2024-09-13T13:09:06.668526+0000","last_active":"2024-09-13T13:09:42.930676+0000","last_peered":"2024-09-13T13:09:42.930676+0000","last_clean":"2024-09-13T13:09:42.930676+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:09:42.930676+0000","last_undegraded":"2024-09-13T13:09:42.930676+0000","last_fullsized":"2024-09-13T13:09:42.930676+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:06.668478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:06.668478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.164862+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":385,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.930543+0000","last_change":"2024-09-13T13:07:11.400066+0000","last_active":"2024-09-13T13:09:42.930543+0000","last_peered":"2024-09-13T13:09:42.930543+0000","last_clean":"2024-09-13T13:09:42.930543+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:09:42.930543+0000","last_undegraded":"2024-09-13T13:09:42.930543+0000","last_fullsized":"2024-09-13T13:09:42.930543+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.399840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.399840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:43.349392+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":362,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.526142+0000","last_change":"2024-09-13T13:09:29.473480+0000","last_active":"2024-09-13T13:09:34.526142+0000","last_peered":"2024-09-13T13:09:34.526142+0000","last_clean":"2024-09-13T13:09:34.526142+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:09:34.526142+0000","last_undegraded":"2024-09-13T13:09:34.526142+0000","last_fullsized":"2024-09-13T13:09:34.526142+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:29.473418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:29.473418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:55.955954+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"232'15111","reported_seq":15249,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.901030+0000","last_change":"2024-09-13T13:06:22.000224+0000","last_active":"2024-09-13T13:09:38.901030+0000","last_peered":"2024-09-13T13:09:38.901030+0000","last_clean":"2024-09-13T13:09:38.901030+0000","last_became_active":"2024-09-13T13:06:21.999889+0000","last_became_peered":"2024-09-13T13:06:21.999889+0000","last_unstale":"2024-09-13T13:09:38.901030+0000","last_undegraded":"2024-09-13T13:09:38.901030+0000","last_fullsized":"2024-09-13T13:09:38.901030+0000","mapping_epoch":169,"log_start":"209'9300","ondisk_log_start":"209'9300","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:40.313069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73728000,"num_objects":1125,"num_object_clones":0,"num_object_copies":3375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15111,"num_write_kb":519552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":371,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.527834+0000","last_change":"2024-09-13T13:09:32.849360+0000","last_active":"2024-09-13T13:09:34.527834+0000","last_peered":"2024-09-13T13:09:34.527834+0000","last_clean":"2024-09-13T13:09:34.527834+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:09:34.527834+0000","last_undegraded":"2024-09-13T13:09:34.527834+0000","last_fullsized":"2024-09-13T13:09:34.527834+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:09:32.849277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:32.849277+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:00.662384+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":395,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929600+0000","last_change":"2024-09-13T13:07:48.579717+0000","last_active":"2024-09-13T13:09:42.929600+0000","last_peered":"2024-09-13T13:09:42.929600+0000","last_clean":"2024-09-13T13:09:42.929600+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:09:42.929600+0000","last_undegraded":"2024-09-13T13:09:42.929600+0000","last_fullsized":"2024-09-13T13:09:42.929600+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:48.579658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:48.579658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.184890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":385,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.930143+0000","last_change":"2024-09-13T13:08:26.033729+0000","last_active":"2024-09-13T13:09:42.930143+0000","last_peered":"2024-09-13T13:09:42.930143+0000","last_clean":"2024-09-13T13:09:42.930143+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:09:42.930143+0000","last_undegraded":"2024-09-13T13:09:42.930143+0000","last_fullsized":"2024-09-13T13:09:42.930143+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:26.033663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:26.033663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":372,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929612+0000","last_change":"2024-09-13T13:07:49.584576+0000","last_active":"2024-09-13T13:09:42.929612+0000","last_peered":"2024-09-13T13:09:42.929612+0000","last_clean":"2024-09-13T13:09:42.929612+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:09:42.929612+0000","last_undegraded":"2024-09-13T13:09:42.929612+0000","last_fullsized":"2024-09-13T13:09:42.929612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:49.584505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:49.584505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.019080+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"232'15411","reported_seq":15549,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:40.979577+0000","last_change":"2024-09-13T13:06:22.002144+0000","last_active":"2024-09-13T13:09:40.979577+0000","last_peered":"2024-09-13T13:09:40.979577+0000","last_clean":"2024-09-13T13:09:40.979577+0000","last_became_active":"2024-09-13T13:06:22.001404+0000","last_became_peered":"2024-09-13T13:06:22.001404+0000","last_unstale":"2024-09-13T13:09:40.979577+0000","last_undegraded":"2024-09-13T13:09:40.979577+0000","last_fullsized":"2024-09-13T13:09:40.979577+0000","mapping_epoch":169,"log_start":"208'9100","ondisk_log_start":"208'9100","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":6311,"log_dups_size":0,"ondisk_log_size":6311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58523648,"num_objects":893,"num_object_clones":0,"num_object_copies":2679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15411,"num_write_kb":521728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":396,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.916345+0000","last_change":"2024-09-13T13:09:30.614842+0000","last_active":"2024-09-13T13:09:42.916345+0000","last_peered":"2024-09-13T13:09:42.916345+0000","last_clean":"2024-09-13T13:09:42.916345+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:09:42.916345+0000","last_undegraded":"2024-09-13T13:09:42.916345+0000","last_fullsized":"2024-09-13T13:09:42.916345+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:09:30.614748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.614748+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:58.916392+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":384,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.930824+0000","last_change":"2024-09-13T13:05:29.345863+0000","last_active":"2024-09-13T13:09:42.930824+0000","last_peered":"2024-09-13T13:09:42.930824+0000","last_clean":"2024-09-13T13:09:42.930824+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:09:42.930824+0000","last_undegraded":"2024-09-13T13:09:42.930824+0000","last_fullsized":"2024-09-13T13:09:42.930824+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.345796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.345796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:14.146169+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":369,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.926148+0000","last_change":"2024-09-13T13:06:42.450669+0000","last_active":"2024-09-13T13:09:42.926148+0000","last_peered":"2024-09-13T13:09:42.926148+0000","last_clean":"2024-09-13T13:09:42.926148+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:09:42.926148+0000","last_undegraded":"2024-09-13T13:09:42.926148+0000","last_fullsized":"2024-09-13T13:09:42.926148+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:42.450598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:42.450598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:14.400638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":356,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.927682+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:09:42.927682+0000","last_peered":"2024-09-13T13:09:42.927682+0000","last_clean":"2024-09-13T13:09:42.927682+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:09:42.927682+0000","last_undegraded":"2024-09-13T13:09:42.927682+0000","last_fullsized":"2024-09-13T13:09:42.927682+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:10.602171+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"233'15230","reported_seq":15458,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:43.082399+0000","last_change":"2024-09-13T13:08:03.659578+0000","last_active":"2024-09-13T13:09:43.082399+0000","last_peered":"2024-09-13T13:09:43.082399+0000","last_clean":"2024-09-13T13:09:43.082399+0000","last_became_active":"2024-09-13T13:06:21.816096+0000","last_became_peered":"2024-09-13T13:06:21.816096+0000","last_unstale":"2024-09-13T13:09:43.082399+0000","last_undegraded":"2024-09-13T13:09:43.082399+0000","last_fullsized":"2024-09-13T13:09:43.082399+0000","mapping_epoch":169,"log_start":"212'9800","ondisk_log_start":"212'9800","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'8119","last_scrub_stamp":"2024-09-13T13:08:03.659418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.659418+0000","objects_scrubbed":7960,"log_size":5430,"log_dups_size":0,"ondisk_log_size":5430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2349,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51249152,"num_objects":782,"num_object_clones":0,"num_object_copies":2346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15230,"num_write_kb":512384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":417,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.927530+0000","last_change":"2024-09-13T13:09:33.843654+0000","last_active":"2024-09-13T13:09:42.927530+0000","last_peered":"2024-09-13T13:09:42.927530+0000","last_clean":"2024-09-13T13:09:42.927530+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:09:42.927530+0000","last_undegraded":"2024-09-13T13:09:42.927530+0000","last_fullsized":"2024-09-13T13:09:42.927530+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:09:33.843579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:33.843579+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:40.459268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":396,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.930130+0000","last_change":"2024-09-13T13:06:51.627764+0000","last_active":"2024-09-13T13:09:42.930130+0000","last_peered":"2024-09-13T13:09:42.930130+0000","last_clean":"2024-09-13T13:09:42.930130+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:09:42.930130+0000","last_undegraded":"2024-09-13T13:09:42.930130+0000","last_fullsized":"2024-09-13T13:09:42.930130+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:06:51.627691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.627691+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.064623+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":498,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929192+0000","last_change":"2024-09-13T13:09:41.018928+0000","last_active":"2024-09-13T13:09:42.929192+0000","last_peered":"2024-09-13T13:09:42.929192+0000","last_clean":"2024-09-13T13:09:42.929192+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:09:42.929192+0000","last_undegraded":"2024-09-13T13:09:42.929192+0000","last_fullsized":"2024-09-13T13:09:42.929192+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:09:41.018848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:41.018848+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:56.940353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":375,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.930287+0000","last_change":"2024-09-13T13:09:30.836048+0000","last_active":"2024-09-13T13:09:42.930287+0000","last_peered":"2024-09-13T13:09:42.930287+0000","last_clean":"2024-09-13T13:09:42.930287+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:09:42.930287+0000","last_undegraded":"2024-09-13T13:09:42.930287+0000","last_fullsized":"2024-09-13T13:09:42.930287+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.836006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.836006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:38.200874+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":352,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.527352+0000","last_change":"2024-09-13T13:07:10.058616+0000","last_active":"2024-09-13T13:09:34.527352+0000","last_peered":"2024-09-13T13:09:34.527352+0000","last_clean":"2024-09-13T13:09:34.527352+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:09:34.527352+0000","last_undegraded":"2024-09-13T13:09:34.527352+0000","last_fullsized":"2024-09-13T13:09:34.527352+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:10.058555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.058555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:44.072613+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":381,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.927715+0000","last_change":"2024-09-13T13:09:37.358795+0000","last_active":"2024-09-13T13:09:42.927715+0000","last_peered":"2024-09-13T13:09:42.927715+0000","last_clean":"2024-09-13T13:09:42.927715+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:09:42.927715+0000","last_undegraded":"2024-09-13T13:09:42.927715+0000","last_fullsized":"2024-09-13T13:09:42.927715+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:09:37.358710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:37.358710+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:38.147999+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":384,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.924933+0000","last_change":"2024-09-13T13:08:21.310761+0000","last_active":"2024-09-13T13:09:42.924933+0000","last_peered":"2024-09-13T13:09:42.924933+0000","last_clean":"2024-09-13T13:09:42.924933+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:09:42.924933+0000","last_undegraded":"2024-09-13T13:09:42.924933+0000","last_fullsized":"2024-09-13T13:09:42.924933+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:21.310681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:21.310681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:44.757345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":391,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929134+0000","last_change":"2024-09-13T13:09:30.959919+0000","last_active":"2024-09-13T13:09:42.929134+0000","last_peered":"2024-09-13T13:09:42.929134+0000","last_clean":"2024-09-13T13:09:42.929134+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:09:42.929134+0000","last_undegraded":"2024-09-13T13:09:42.929134+0000","last_fullsized":"2024-09-13T13:09:42.929134+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.959870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.959870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:45.945701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":386,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531702+0000","last_change":"2024-09-13T13:09:34.426362+0000","last_active":"2024-09-13T13:09:34.531702+0000","last_peered":"2024-09-13T13:09:34.531702+0000","last_clean":"2024-09-13T13:09:34.531702+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:09:34.531702+0000","last_undegraded":"2024-09-13T13:09:34.531702+0000","last_fullsized":"2024-09-13T13:09:34.531702+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:34.426272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:34.426272+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.871047+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":379,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929461+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:09:42.929461+0000","last_peered":"2024-09-13T13:09:42.929461+0000","last_clean":"2024-09-13T13:09:42.929461+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:09:42.929461+0000","last_undegraded":"2024-09-13T13:09:42.929461+0000","last_fullsized":"2024-09-13T13:09:42.929461+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":368,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.927618+0000","last_change":"2024-09-13T13:09:40.333896+0000","last_active":"2024-09-13T13:09:42.927618+0000","last_peered":"2024-09-13T13:09:42.927618+0000","last_clean":"2024-09-13T13:09:42.927618+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:09:42.927618+0000","last_undegraded":"2024-09-13T13:09:42.927618+0000","last_fullsized":"2024-09-13T13:09:42.927618+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:40.333836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:40.333836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:03.786971+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":381,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.931720+0000","last_change":"2024-09-13T13:09:07.378142+0000","last_active":"2024-09-13T13:09:42.931720+0000","last_peered":"2024-09-13T13:09:42.931720+0000","last_clean":"2024-09-13T13:09:42.931720+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:09:42.931720+0000","last_undegraded":"2024-09-13T13:09:42.931720+0000","last_fullsized":"2024-09-13T13:09:42.931720+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:07.378066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:07.378066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:33.759039+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":383,"reported_epoch":233,"state":"active+clean+scrubbing","last_fresh":"2024-09-13T13:09:42.928337+0000","last_change":"2024-09-13T13:09:42.914583+0000","last_active":"2024-09-13T13:09:42.928337+0000","last_peered":"2024-09-13T13:09:42.928337+0000","last_clean":"2024-09-13T13:09:42.928337+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:09:42.928337+0000","last_undegraded":"2024-09-13T13:09:42.928337+0000","last_fullsized":"2024-09-13T13:09:42.928337+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":374,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.525152+0000","last_change":"2024-09-13T13:06:41.308449+0000","last_active":"2024-09-13T13:09:34.525152+0000","last_peered":"2024-09-13T13:09:34.525152+0000","last_clean":"2024-09-13T13:09:34.525152+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:09:34.525152+0000","last_undegraded":"2024-09-13T13:09:34.525152+0000","last_fullsized":"2024-09-13T13:09:34.525152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:41.308367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:41.308367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.223962+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":473,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.927616+0000","last_change":"2024-09-13T13:08:02.779173+0000","last_active":"2024-09-13T13:09:42.927616+0000","last_peered":"2024-09-13T13:09:42.927616+0000","last_clean":"2024-09-13T13:09:42.927616+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:09:42.927616+0000","last_undegraded":"2024-09-13T13:09:42.927616+0000","last_fullsized":"2024-09-13T13:09:42.927616+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:08:02.778891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:02.778891+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":373,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.916692+0000","last_change":"2024-09-13T13:07:21.958114+0000","last_active":"2024-09-13T13:09:42.916692+0000","last_peered":"2024-09-13T13:09:42.916692+0000","last_clean":"2024-09-13T13:09:42.916692+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:09:42.916692+0000","last_undegraded":"2024-09-13T13:09:42.916692+0000","last_fullsized":"2024-09-13T13:09:42.916692+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:21.958030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:21.958030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:44.550122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":395,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.928127+0000","last_change":"2024-09-13T13:09:04.635198+0000","last_active":"2024-09-13T13:09:42.928127+0000","last_peered":"2024-09-13T13:09:42.928127+0000","last_clean":"2024-09-13T13:09:42.928127+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:09:42.928127+0000","last_undegraded":"2024-09-13T13:09:42.928127+0000","last_fullsized":"2024-09-13T13:09:42.928127+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:04.599040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:04.599040+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:30.955987+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":368,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929417+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:09:42.929417+0000","last_peered":"2024-09-13T13:09:42.929417+0000","last_clean":"2024-09-13T13:09:42.929417+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:09:42.929417+0000","last_undegraded":"2024-09-13T13:09:42.929417+0000","last_fullsized":"2024-09-13T13:09:42.929417+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:11.161132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":497,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.925018+0000","last_change":"2024-09-13T13:07:08.781005+0000","last_active":"2024-09-13T13:09:42.925018+0000","last_peered":"2024-09-13T13:09:42.925018+0000","last_clean":"2024-09-13T13:09:42.925018+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:09:42.925018+0000","last_undegraded":"2024-09-13T13:09:42.925018+0000","last_fullsized":"2024-09-13T13:09:42.925018+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:07:08.780952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.780952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":370,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:36.414394+0000","last_change":"2024-09-13T13:09:36.414394+0000","last_active":"2024-09-13T13:09:36.414394+0000","last_peered":"2024-09-13T13:09:36.414394+0000","last_clean":"2024-09-13T13:09:36.414394+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:09:36.414394+0000","last_undegraded":"2024-09-13T13:09:36.414394+0000","last_fullsized":"2024-09-13T13:09:36.414394+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:36.414322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:36.414322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:39.835131+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":403,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.916331+0000","last_change":"2024-09-13T13:05:26.186899+0000","last_active":"2024-09-13T13:09:42.916331+0000","last_peered":"2024-09-13T13:09:42.916331+0000","last_clean":"2024-09-13T13:09:42.916331+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:09:42.916331+0000","last_undegraded":"2024-09-13T13:09:42.916331+0000","last_fullsized":"2024-09-13T13:09:42.916331+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:05:26.186830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.186830+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:14.738553+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":388,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.917232+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:09:42.917232+0000","last_peered":"2024-09-13T13:09:42.917232+0000","last_clean":"2024-09-13T13:09:42.917232+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:09:42.917232+0000","last_undegraded":"2024-09-13T13:09:42.917232+0000","last_fullsized":"2024-09-13T13:09:42.917232+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.696749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":377,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.922860+0000","last_change":"2024-09-13T13:09:05.882862+0000","last_active":"2024-09-13T13:09:42.922860+0000","last_peered":"2024-09-13T13:09:42.922860+0000","last_clean":"2024-09-13T13:09:42.922860+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:09:42.922860+0000","last_undegraded":"2024-09-13T13:09:42.922860+0000","last_fullsized":"2024-09-13T13:09:42.922860+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:05.882769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.882769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.443670+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":372,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.925741+0000","last_change":"2024-09-13T13:08:59.585721+0000","last_active":"2024-09-13T13:09:42.925741+0000","last_peered":"2024-09-13T13:09:42.925741+0000","last_clean":"2024-09-13T13:09:42.925741+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:09:42.925741+0000","last_undegraded":"2024-09-13T13:09:42.925741+0000","last_fullsized":"2024-09-13T13:09:42.925741+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:59.585646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.585646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.171797+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":332,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.930053+0000","last_change":"2024-09-13T13:05:35.506093+0000","last_active":"2024-09-13T13:09:42.930053+0000","last_peered":"2024-09-13T13:09:42.930053+0000","last_clean":"2024-09-13T13:09:42.930053+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:09:42.930053+0000","last_undegraded":"2024-09-13T13:09:42.930053+0000","last_fullsized":"2024-09-13T13:09:42.930053+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:05:35.506037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.506037+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:10.670571+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":384,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929430+0000","last_change":"2024-09-13T13:09:07.118545+0000","last_active":"2024-09-13T13:09:42.929430+0000","last_peered":"2024-09-13T13:09:42.929430+0000","last_clean":"2024-09-13T13:09:42.929430+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:09:42.929430+0000","last_undegraded":"2024-09-13T13:09:42.929430+0000","last_fullsized":"2024-09-13T13:09:42.929430+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:07.118467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:07.118467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.947652+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":368,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.927771+0000","last_change":"2024-09-13T13:09:32.467362+0000","last_active":"2024-09-13T13:09:42.927771+0000","last_peered":"2024-09-13T13:09:42.927771+0000","last_clean":"2024-09-13T13:09:42.927771+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:09:42.927771+0000","last_undegraded":"2024-09-13T13:09:42.927771+0000","last_fullsized":"2024-09-13T13:09:42.927771+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:32.467151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:32.467151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:55.111614+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":370,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.524263+0000","last_change":"2024-09-13T13:08:56.227109+0000","last_active":"2024-09-13T13:09:34.524263+0000","last_peered":"2024-09-13T13:09:34.524263+0000","last_clean":"2024-09-13T13:09:34.524263+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:09:34.524263+0000","last_undegraded":"2024-09-13T13:09:34.524263+0000","last_fullsized":"2024-09-13T13:09:34.524263+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:56.227027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:56.227027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:14.111530+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":394,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.925890+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:09:42.925890+0000","last_peered":"2024-09-13T13:09:42.925890+0000","last_clean":"2024-09-13T13:09:42.925890+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:09:42.925890+0000","last_undegraded":"2024-09-13T13:09:42.925890+0000","last_fullsized":"2024-09-13T13:09:42.925890+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:11.582994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":384,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.925020+0000","last_change":"2024-09-13T13:08:50.517587+0000","last_active":"2024-09-13T13:09:42.925020+0000","last_peered":"2024-09-13T13:09:42.925020+0000","last_clean":"2024-09-13T13:09:42.925020+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:09:42.925020+0000","last_undegraded":"2024-09-13T13:09:42.925020+0000","last_fullsized":"2024-09-13T13:09:42.925020+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:50.517526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:50.517526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:03.602123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":387,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929149+0000","last_change":"2024-09-13T13:08:53.215325+0000","last_active":"2024-09-13T13:09:42.929149+0000","last_peered":"2024-09-13T13:09:42.929149+0000","last_clean":"2024-09-13T13:09:42.929149+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:09:42.929149+0000","last_undegraded":"2024-09-13T13:09:42.929149+0000","last_fullsized":"2024-09-13T13:09:42.929149+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.215263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.215263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:10.458784+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":367,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.928093+0000","last_change":"2024-09-13T13:07:57.596272+0000","last_active":"2024-09-13T13:09:42.928093+0000","last_peered":"2024-09-13T13:09:42.928093+0000","last_clean":"2024-09-13T13:09:42.928093+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:09:42.928093+0000","last_undegraded":"2024-09-13T13:09:42.928093+0000","last_fullsized":"2024-09-13T13:09:42.928093+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.596097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":402,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.926146+0000","last_change":"2024-09-13T13:09:39.368581+0000","last_active":"2024-09-13T13:09:42.926146+0000","last_peered":"2024-09-13T13:09:42.926146+0000","last_clean":"2024-09-13T13:09:42.926146+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:09:42.926146+0000","last_undegraded":"2024-09-13T13:09:42.926146+0000","last_fullsized":"2024-09-13T13:09:42.926146+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2024-09-13T13:09:39.368381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:39.368381+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.235364+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":390,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.531761+0000","last_change":"2024-09-13T13:09:30.487486+0000","last_active":"2024-09-13T13:09:34.531761+0000","last_peered":"2024-09-13T13:09:34.531761+0000","last_clean":"2024-09-13T13:09:34.531761+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:09:34.531761+0000","last_undegraded":"2024-09-13T13:09:34.531761+0000","last_fullsized":"2024-09-13T13:09:34.531761+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.487413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.487413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:53.159481+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":276,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528145+0000","last_change":"2024-09-13T13:09:27.844085+0000","last_active":"2024-09-13T13:09:34.528145+0000","last_peered":"2024-09-13T13:09:34.528145+0000","last_clean":"2024-09-13T13:09:34.528145+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:09:34.528145+0000","last_undegraded":"2024-09-13T13:09:34.528145+0000","last_fullsized":"2024-09-13T13:09:34.528145+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:27.844003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:27.844003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:31.056528+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":380,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.929616+0000","last_change":"2024-09-13T13:09:31.943644+0000","last_active":"2024-09-13T13:09:42.929616+0000","last_peered":"2024-09-13T13:09:42.929616+0000","last_clean":"2024-09-13T13:09:42.929616+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:09:42.929616+0000","last_undegraded":"2024-09-13T13:09:42.929616+0000","last_fullsized":"2024-09-13T13:09:42.929616+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:31.943568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:31.943568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.734605+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":373,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.921862+0000","last_change":"2024-09-13T13:08:27.641188+0000","last_active":"2024-09-13T13:09:42.921862+0000","last_peered":"2024-09-13T13:09:42.921862+0000","last_clean":"2024-09-13T13:09:42.921862+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:09:42.921862+0000","last_undegraded":"2024-09-13T13:09:42.921862+0000","last_fullsized":"2024-09-13T13:09:42.921862+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:27.641105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:27.641105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:53.127406+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":376,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T13:09:42.926431+0000","last_change":"2024-09-13T13:09:35.355561+0000","last_active":"2024-09-13T13:09:42.926431+0000","last_peered":"2024-09-13T13:09:42.926431+0000","last_clean":"2024-09-13T13:09:42.926431+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:09:42.926431+0000","last_undegraded":"2024-09-13T13:09:42.926431+0000","last_fullsized":"2024-09-13T13:09:42.926431+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:35.355369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:35.355369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:52.921712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":370,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:34.528167+0000","last_change":"2024-09-13T13:06:09.928585+0000","last_active":"2024-09-13T13:09:34.528167+0000","last_peered":"2024-09-13T13:09:34.528167+0000","last_clean":"2024-09-13T13:09:34.528167+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:09:34.528167+0000","last_undegraded":"2024-09-13T13:09:34.528167+0000","last_fullsized":"2024-09-13T13:09:34.528167+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:06:09.928504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.928504+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:42.064985+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":383,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T13:09:38.337230+0000","last_change":"2024-09-13T13:09:38.337230+0000","last_active":"2024-09-13T13:09:38.337230+0000","last_peered":"2024-09-13T13:09:38.337230+0000","last_clean":"2024-09-13T13:09:38.337230+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:09:38.337230+0000","last_undegraded":"2024-09-13T13:09:38.337230+0000","last_fullsized":"2024-09-13T13:09:38.337230+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.337156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.337156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:40.248177+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":870055960,"num_objects":13277,"num_object_clones":0,"num_object_copies":39831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13277,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242577,"num_write_kb":8187265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2558144512,"data_stored":2558132296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85577,"ondisk_log_size":85577,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":514,"num_read_kb":509,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201387,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":466112,"kb_used_data":272496,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93279040,"statfs":{"total":95995035648,"available":95517736960,"internally_reserved":0,"allocated":279035904,"data_stored":278622855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":198244810},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806140997,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":562732,"kb_used_data":304540,"kb_used_omap":4,"kb_used_meta":258171,"kb_avail":93182420,"statfs":{"total":95995035648,"available":95418798080,"internally_reserved":0,"allocated":311848960,"data_stored":311454738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":264367789},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691959,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671156,"kb_used_data":405284,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":93073996,"statfs":{"total":95995035648,"available":95307771904,"internally_reserved":0,"allocated":415010816,"data_stored":414608351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920893,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":448644,"kb_used_data":250868,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93296508,"statfs":{"total":95995035648,"available":95535624192,"internally_reserved":0,"allocated":256888832,"data_stored":256470095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":202504646},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117124,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":635160,"kb_used_data":288716,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93109992,"statfs":{"total":95995035648,"available":95344631808,"internally_reserved":0,"allocated":295645184,"data_stored":295271990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":354613703},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411463,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571188,"kb_used_data":296936,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93173964,"statfs":{"total":95995035648,"available":95410139136,"internally_reserved":0,"allocated":304062464,"data_stored":303724532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574990,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":664408,"kb_used_data":354184,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":93080744,"statfs":{"total":95995035648,"available":95314681856,"internally_reserved":0,"allocated":362684416,"data_stored":362245710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738516,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":725376,"kb_used_data":427632,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93019776,"statfs":{"total":95995035648,"available":95252250624,"internally_reserved":0,"allocated":437895168,"data_stored":437481804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":418906112,"data_stored":418906112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":338165760,"data_stored":338165760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":297078784,"data_stored":297074712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285409280,"data_stored":285409280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":251002880,"data_stored":251002880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":407113728,"data_stored":407109656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":305004544,"data_stored":305004544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":255463424,"data_stored":255459352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:09:45.842 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T13:09:45.843 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:09:45.843 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T13:09:45.843 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 1.000000 2024-09-13T13:09:45.843 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-13T13:09:45.859 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:45.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:45.960 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:45.973 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:46.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:46.003 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:46.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:46.104 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:46.120 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:45 smithi052 ceph-mon[29471]: from='client.25558 v1:172.21.15.52:0/994425344' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:09:46.121 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:45 smithi052 ceph-mon[36682]: from='client.25558 v1:172.21.15.52:0/994425344' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:09:46.170 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:46.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:46.272 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:46.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:46.304 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:46.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:46.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:45 smithi167 ceph-mon[33178]: from='client.25558 v1:172.21.15.52:0/994425344' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:09:46.405 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:46.430 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:46.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:46.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:46.439 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:09:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:09:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:09:46.532 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:46.553 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:46.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:46.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:46.655 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:46.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:46.694 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:46.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:46.795 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:46.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:46.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:46.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:46.927 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:46.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:46.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:47.003 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:09:47.041 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:46 smithi052 ceph-mon[29471]: pgmap v411: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 765 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-13T13:09:47.043 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:46 smithi052 ceph-mon[36682]: pgmap v411: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 765 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-13T13:09:47.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:47.082 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:47.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:47.184 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:47.208 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:47.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:47.301 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:46 smithi167 ceph-mon[33178]: pgmap v411: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 765 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-13T13:09:47.310 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:47.346 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T13:09:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:47.363 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:47.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:47.432 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:09:47.432 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:09:47.455 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:47.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:47.477 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:09:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:09:47.493 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:47.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:47.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:47.520 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:09:47.520 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:09:47.526 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:47.556 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:09:47.557 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:09:47.586 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:09:47.587 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:09:47.595 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:47.620 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:09:47.621 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:09:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:47.652 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:47.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:47.656 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:09:47.656 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:09:47.694 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:09:47.696 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:09:47.696 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:09:47.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:47.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:47.796 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:47.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:47.897 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:47.941 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:47.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:47.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:48.043 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:48.086 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:48.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:48.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:48.156 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:47 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3678932666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:48.188 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:48.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:47 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3678932666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:48.217 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:47 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3678932666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:48.229 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:48.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:48.297 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:09:48.297 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:09:48.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:48.297 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:249148 8.5 8:a53d3279:::benchmark_data_smithi052_101302_object121219:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T13:09:48.297 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:09:48.288802+0000", 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0060096919999999996, 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060399319999999996, 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 249148 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:48.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:48.288802+0000", 2024-09-13T13:09:48.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:48.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:48.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:48.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:48.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:48.288802+0000", 2024-09-13T13:09:48.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:48.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:48.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:48.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:48.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:48.288804+0000", 2024-09-13T13:09:48.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:09:48.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:48.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:48.288809+0000", 2024-09-13T13:09:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4190000000000001e-06 2024-09-13T13:09:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:48.288845+0000", 2024-09-13T13:09:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6000000000000001e-05 2024-09-13T13:09:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:48.288853+0000", 2024-09-13T13:09:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4090000000000003e-06 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:48.289055+0000", 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020118600000000001 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:48.289153+0000", 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8507999999999999e-05 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:09:48.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:48.289250+0000", 2024-09-13T13:09:48.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6642000000000001e-05 2024-09-13T13:09:48.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:48.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:48.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:48.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:48.292853+0000", 2024-09-13T13:09:48.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036036280000000002 2024-09-13T13:09:48.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:48.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:48.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:48.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:48.294527+0000", 2024-09-13T13:09:48.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016738289999999999 2024-09-13T13:09:48.302 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:48.302 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:48.303 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:48.303 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:48.303 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:09:48.303 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-09-13T13:09:48.303 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:09:48.332 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:48.380 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:48.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:48.481 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:48.523 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:48.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:48.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:48.625 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:48.665 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:48.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:48.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:48.767 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:48.802 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:48.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:48.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:48.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:48.938 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:48.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:48.980 INFO:teuthology.orchestra.run.smithi052.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-13T13:09:49.023 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T13:09:49.039 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:49.075 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:49.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:49.177 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:49.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:49.213 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:49.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:49.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:48 smithi052 ceph-mon[29471]: pgmap v412: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 718 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-13T13:09:49.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:48 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:09:49.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:48 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/880661741' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:09:49.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:48 smithi052 ceph-mon[36682]: pgmap v412: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 718 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-13T13:09:49.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:48 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:09:49.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:48 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/880661741' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:09:49.315 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:49.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:48 smithi167 ceph-mon[33178]: pgmap v412: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 718 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-13T13:09:49.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:48 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:09:49.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:48 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/880661741' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:09:49.352 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:49.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:49.405 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:09:49.454 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:49.495 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:49.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:49.597 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:49.639 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:49.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:49.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:49.741 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:49.778 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:49.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:49.880 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:49.923 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:49.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:50.025 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:50.060 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:50.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:50.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:50.192 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:49 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/880661741' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:09:50.192 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:49 smithi167 ceph-mon[33178]: osdmap e234: 8 total, 8 up, 8 in 2024-09-13T13:09:50.197 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:50.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:50.242 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:09:50.242 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:09:50.242 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:09:50.242 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:09:50.242 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:09:50.299 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:50.320 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:50.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:50.422 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:50.457 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:50.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:50.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:50.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:49 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/880661741' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:09:50.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:49 smithi052 ceph-mon[36682]: osdmap e234: 8 total, 8 up, 8 in 2024-09-13T13:09:50.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:49 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/880661741' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:09:50.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:49 smithi052 ceph-mon[29471]: osdmap e234: 8 total, 8 up, 8 in 2024-09-13T13:09:50.560 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:50.591 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:50.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:50.693 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:50.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:50.732 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:50.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:50.833 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:50.876 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:50.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:50.891 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:09:50.949 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T13:09:50.977 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:51.004 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:09:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:51.032 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:51.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:51.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:51.165 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:51.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:51.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:51.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:51 smithi052 ceph-mon[36682]: pgmap v414: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 424 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-13T13:09:51.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:51 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4033581509' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:51.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:51 smithi052 ceph-mon[36682]: from='client.35591 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:51.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:51 smithi052 ceph-mon[29471]: pgmap v414: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 424 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-13T13:09:51.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:51 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4033581509' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:51.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:51 smithi052 ceph-mon[29471]: from='client.35591 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:51.267 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:51.314 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:51.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:51.318 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:09:51.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:51 smithi167 ceph-mon[33178]: pgmap v414: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 424 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-13T13:09:51.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:51 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4033581509' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:51.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:51 smithi167 ceph-mon[33178]: from='client.35591 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:09:51.416 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:51.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:51.454 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:51.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:51.555 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:51.586 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:51.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:51.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:51.689 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:51.723 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:51.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:51.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:51.814 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:51.825 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:51.869 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:51.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:51.970 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:52.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:52.012 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:52.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:52.113 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:52.155 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:52.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:52.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:52.220 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:09:52.220 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:09:52.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:09:52.220 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:09:52.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.220 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167255 7.c 7:372d90c4:::benchmark_data_smithi052_71865_object46154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:09:52.221 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.812976+0000", 2024-09-13T13:09:52.221 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 303.40463238400002, 2024-09-13T13:09:52.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.970962349, 2024-09-13T13:09:52.221 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.221 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.221 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.221 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167255 2024-09-13T13:09:52.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.221 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812976+0000", 2024-09-13T13:09:52.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812976+0000", 2024-09-13T13:09:52.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812978+0000", 2024-09-13T13:09:52.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.547e-06 2024-09-13T13:09:52.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812985+0000", 2024-09-13T13:09:52.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1389999999999998e-06 2024-09-13T13:09:52.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812994+0000", 2024-09-13T13:09:52.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0580000000000004e-06 2024-09-13T13:09:52.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812999+0000", 2024-09-13T13:09:52.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1719999999999998e-06 2024-09-13T13:09:52.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.813060+0000", 2024-09-13T13:09:52.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0238000000000002e-05 2024-09-13T13:09:52.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.813067+0000", 2024-09-13T13:09:52.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1019999999999999e-06 2024-09-13T13:09:52.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.778497+0000", 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96543056400000005 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.778584+0000", 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6466999999999997e-05 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.778664+0000", 2024-09-13T13:09:52.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0111000000000006e-05 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783699+0000", 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050350940000000004 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783753+0000", 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4301000000000001e-05 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783880+0000", 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012635700000000001 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783912+0000", 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2736e-05 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783939+0000", 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6463000000000001e-05 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.227 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.228 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167302 7.c 7:3bad7f06:::benchmark_data_smithi052_71865_object46201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:09:52.228 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.859143+0000", 2024-09-13T13:09:52.228 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 303.35846563699999, 2024-09-13T13:09:52.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92766838399999996, 2024-09-13T13:09:52.228 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.228 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.228 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.228 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167302 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859143+0000", 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859143+0000", 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859145+0000", 2024-09-13T13:09:52.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-09-13T13:09:52.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859153+0000", 2024-09-13T13:09:52.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4540000000000007e-06 2024-09-13T13:09:52.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859165+0000", 2024-09-13T13:09:52.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1246e-05 2024-09-13T13:09:52.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859170+0000", 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859231+0000", 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1163999999999999e-05 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859244+0000", 2024-09-13T13:09:52.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2539e-05 2024-09-13T13:09:52.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.780456+0000", 2024-09-13T13:09:52.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92121202899999999 2024-09-13T13:09:52.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.780530+0000", 2024-09-13T13:09:52.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4692999999999998e-05 2024-09-13T13:09:52.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:52.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.780605+0000", 2024-09-13T13:09:52.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4281000000000003e-05 2024-09-13T13:09:52.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783805+0000", 2024-09-13T13:09:52.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031999709999999998 2024-09-13T13:09:52.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783844+0000", 2024-09-13T13:09:52.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9496000000000003e-05 2024-09-13T13:09:52.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786656+0000", 2024-09-13T13:09:52.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028116249999999999 2024-09-13T13:09:52.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786765+0000", 2024-09-13T13:09:52.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109205 2024-09-13T13:09:52.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786812+0000", 2024-09-13T13:09:52.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6440999999999998e-05 2024-09-13T13:09:52.235 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.235 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.235 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.235 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167310 7.c 7:3e43ee27:::benchmark_data_smithi052_71865_object46209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:09:52.236 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.867904+0000", 2024-09-13T13:09:52.236 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 303.34970501100003, 2024-09-13T13:09:52.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91916685399999998, 2024-09-13T13:09:52.236 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.236 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.236 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.236 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167310 2024-09-13T13:09:52.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.236 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867904+0000", 2024-09-13T13:09:52.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867904+0000", 2024-09-13T13:09:52.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867906+0000", 2024-09-13T13:09:52.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.827e-06 2024-09-13T13:09:52.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867911+0000", 2024-09-13T13:09:52.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.733e-06 2024-09-13T13:09:52.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867925+0000", 2024-09-13T13:09:52.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3701e-05 2024-09-13T13:09:52.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867932+0000", 2024-09-13T13:09:52.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8689999999999998e-06 2024-09-13T13:09:52.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867995+0000", 2024-09-13T13:09:52.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3208999999999997e-05 2024-09-13T13:09:52.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.868005+0000", 2024-09-13T13:09:52.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0379e-05 2024-09-13T13:09:52.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.782379+0000", 2024-09-13T13:09:52.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91437379500000004 2024-09-13T13:09:52.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.782458+0000", 2024-09-13T13:09:52.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8485999999999994e-05 2024-09-13T13:09:52.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.782522+0000", 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4598999999999996e-05 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786890+0000", 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0043677120000000002 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786952+0000", 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1827000000000003e-05 2024-09-13T13:09:52.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786989+0000", 2024-09-13T13:09:52.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6674999999999997e-05 2024-09-13T13:09:52.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787034+0000", 2024-09-13T13:09:52.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5003000000000003e-05 2024-09-13T13:09:52.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787071+0000", 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7039000000000002e-05 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167324 7.c 7:3d57ad43:::benchmark_data_smithi052_71865_object46223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.880402+0000", 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 303.33720720899998, 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91706507599999998, 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.243 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167324 2024-09-13T13:09:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880402+0000", 2024-09-13T13:09:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880402+0000", 2024-09-13T13:09:52.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880404+0000", 2024-09-13T13:09:52.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3329999999999999e-06 2024-09-13T13:09:52.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880410+0000", 2024-09-13T13:09:52.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3199999999999996e-06 2024-09-13T13:09:52.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880424+0000", 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3396e-05 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880431+0000", 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8010000000000004e-06 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880504+0000", 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2482000000000002e-05 2024-09-13T13:09:52.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880515+0000", 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0745e-05 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783970+0000", 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.90345520499999998 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.784030+0000", 2024-09-13T13:09:52.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0668000000000003e-05 2024-09-13T13:09:52.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:52.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.784102+0000", 2024-09-13T13:09:52.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1852000000000003e-05 2024-09-13T13:09:52.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.788798+0000", 2024-09-13T13:09:52.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004696034 2024-09-13T13:09:52.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.790728+0000", 2024-09-13T13:09:52.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001929828 2024-09-13T13:09:52.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797366+0000", 2024-09-13T13:09:52.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066377340000000002 2024-09-13T13:09:52.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797415+0000", 2024-09-13T13:09:52.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9502000000000001e-05 2024-09-13T13:09:52.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797467+0000", 2024-09-13T13:09:52.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1175999999999999e-05 2024-09-13T13:09:52.250 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.250 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.250 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.250 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167331 7.c 7:314bd5a0:::benchmark_data_smithi052_71865_object46230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:09:52.250 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.885575+0000", 2024-09-13T13:09:52.250 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 303.33203379700001, 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91216846600000001, 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167331 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885575+0000", 2024-09-13T13:09:52.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885575+0000", 2024-09-13T13:09:52.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885577+0000", 2024-09-13T13:09:52.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.446e-06 2024-09-13T13:09:52.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885583+0000", 2024-09-13T13:09:52.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0759999999999999e-06 2024-09-13T13:09:52.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885597+0000", 2024-09-13T13:09:52.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3909e-05 2024-09-13T13:09:52.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885610+0000", 2024-09-13T13:09:52.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2288e-05 2024-09-13T13:09:52.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885692+0000", 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2060999999999999e-05 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885703+0000", 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1331e-05 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.784949+0000", 2024-09-13T13:09:52.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.89924553699999998 2024-09-13T13:09:52.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.785047+0000", 2024-09-13T13:09:52.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8639000000000002e-05 2024-09-13T13:09:52.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:52.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.785146+0000", 2024-09-13T13:09:52.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8556999999999998e-05 2024-09-13T13:09:52.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.788803+0000", 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003657637 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.790769+0000", 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019660250000000002 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797662+0000", 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068928160000000004 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797708+0000", 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6038999999999997e-05 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797743+0000", 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5104999999999997e-05 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.257 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167336 7.c 7:3d629c66:::benchmark_data_smithi052_71865_object46235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:09:52.258 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.887959+0000", 2024-09-13T13:09:52.258 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 303.32964999500001, 2024-09-13T13:09:52.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.90987360799999994, 2024-09-13T13:09:52.258 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.258 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.258 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.258 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167336 2024-09-13T13:09:52.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.258 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887959+0000", 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887959+0000", 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887960+0000", 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.728e-06 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887965+0000", 2024-09-13T13:09:52.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7319999999999998e-06 2024-09-13T13:09:52.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887978+0000", 2024-09-13T13:09:52.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2702e-05 2024-09-13T13:09:52.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887985+0000", 2024-09-13T13:09:52.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.097e-06 2024-09-13T13:09:52.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.888051+0000", 2024-09-13T13:09:52.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6277000000000003e-05 2024-09-13T13:09:52.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.888062+0000", 2024-09-13T13:09:52.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049e-05 2024-09-13T13:09:52.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787109+0000", 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.89904651199999996 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787218+0000", 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010893200000000001 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787319+0000", 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101605 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.792171+0000", 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048516380000000001 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.794176+0000", 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020051280000000001 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797764+0000", 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003588277 2024-09-13T13:09:52.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797807+0000", 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3000999999999997e-05 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797832+0000", 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.493e-05 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189427 7.d 7:b2417336:::benchmark_data_smithi052_71865_object68326:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.325139+0000", 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.89246959500002, 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0242147180000001, 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189427 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325139+0000", 2024-09-13T13:09:52.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325139+0000", 2024-09-13T13:09:52.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325142+0000", 2024-09-13T13:09:52.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.526e-06 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325151+0000", 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4080000000000004e-06 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325164+0000", 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2676e-05 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325173+0000", 2024-09-13T13:09:52.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0070000000000003e-06 2024-09-13T13:09:52.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.327118+0000", 2024-09-13T13:09:52.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001944723 2024-09-13T13:09:52.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-09-13T13:09:52.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.327125+0000", 2024-09-13T13:09:52.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7729999999999999e-06 2024-09-13T13:09:52.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.333636+0000", 2024-09-13T13:09:52.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065108279999999998 2024-09-13T13:09:52.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.333645+0000", 2024-09-13T13:09:52.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5639999999999997e-06 2024-09-13T13:09:52.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349274+0000", 2024-09-13T13:09:52.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0156289759999999 2024-09-13T13:09:52.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349354+0000", 2024-09-13T13:09:52.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0236999999999998e-05 2024-09-13T13:09:52.272 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.272 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.272 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.272 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189429 7.d 7:b1726e7a:::benchmark_data_smithi052_71865_object68328:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.272 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.325193+0000", 2024-09-13T13:09:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.89241592500002, 2024-09-13T13:09:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0246123009999999, 2024-09-13T13:09:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:09:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189429 2024-09-13T13:09:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325193+0000", 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325193+0000", 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325194+0000", 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325197+0000", 2024-09-13T13:09:52.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2200000000000001e-06 2024-09-13T13:09:52.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325204+0000", 2024-09-13T13:09:52.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6989999999999997e-06 2024-09-13T13:09:52.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325209+0000", 2024-09-13T13:09:52.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.118e-06 2024-09-13T13:09:52.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.327226+0000", 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020175979999999998 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.327232+0000", 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9789999999999998e-06 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.333665+0000", 2024-09-13T13:09:52.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064329620000000004 2024-09-13T13:09:52.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.333671+0000", 2024-09-13T13:09:52.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0599999999999996e-06 2024-09-13T13:09:52.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349742+0000", 2024-09-13T13:09:52.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.016070799 2024-09-13T13:09:52.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349805+0000", 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3083000000000006e-05 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189477 7.d 7:bf75cb66:::benchmark_data_smithi052_71865_object68376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.372477+0000", 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.845132127, 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98440433000000005, 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.278 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189477 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372477+0000", 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372477+0000", 2024-09-13T13:09:52.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.280 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:52.281 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:52 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:09:52.281 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:52 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:09:52.281 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:52 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:52.281 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:52 smithi052 ceph-mon[36682]: from='client.35591 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:09:52.281 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:52 smithi052 ceph-mon[36682]: osdmap e235: 8 total, 8 up, 8 in 2024-09-13T13:09:52.282 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:52 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:09:52.282 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:52 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:09:52.282 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:52 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:52.282 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:52 smithi052 ceph-mon[29471]: from='client.35591 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:09:52.282 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:52 smithi052 ceph-mon[29471]: osdmap e235: 8 total, 8 up, 8 in 2024-09-13T13:09:52.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372479+0000", 2024-09-13T13:09:52.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8950000000000001e-06 2024-09-13T13:09:52.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372485+0000", 2024-09-13T13:09:52.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9900000000000002e-06 2024-09-13T13:09:52.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372494+0000", 2024-09-13T13:09:52.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5689999999999994e-06 2024-09-13T13:09:52.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372499+0000", 2024-09-13T13:09:52.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3310000000000003e-06 2024-09-13T13:09:52.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372566+0000", 2024-09-13T13:09:52.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6716000000000003e-05 2024-09-13T13:09:52.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372586+0000", 2024-09-13T13:09:52.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9987e-05 2024-09-13T13:09:52.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349836+0000", 2024-09-13T13:09:52.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97724946000000001 2024-09-13T13:09:52.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349909+0000", 2024-09-13T13:09:52.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3281999999999994e-05 2024-09-13T13:09:52.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:09:52.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349980+0000", 2024-09-13T13:09:52.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0748999999999996e-05 2024-09-13T13:09:52.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356657+0000", 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066769580000000002 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356719+0000", 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2872000000000004e-05 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356776+0000", 2024-09-13T13:09:52.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6314000000000003e-05 2024-09-13T13:09:52.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356840+0000", 2024-09-13T13:09:52.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4484000000000001e-05 2024-09-13T13:09:52.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356881+0000", 2024-09-13T13:09:52.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0723000000000003e-05 2024-09-13T13:09:52.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.287 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189510 7.d 7:b635578a:::benchmark_data_smithi052_71865_object68409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.399587+0000", 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.81802168199999, 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96507304999999999, 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189510 2024-09-13T13:09:52.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399587+0000", 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399587+0000", 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399589+0000", 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2759999999999999e-06 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399599+0000", 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4730000000000006e-06 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399613+0000", 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.403e-05 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399621+0000", 2024-09-13T13:09:52.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2609999999999993e-06 2024-09-13T13:09:52.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399690+0000", 2024-09-13T13:09:52.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8448000000000006e-05 2024-09-13T13:09:52.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399700+0000", 2024-09-13T13:09:52.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9490000000000008e-06 2024-09-13T13:09:52.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.353851+0000", 2024-09-13T13:09:52.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95415173900000005 2024-09-13T13:09:52.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.353952+0000", 2024-09-13T13:09:52.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101093 2024-09-13T13:09:52.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:09:52.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.354065+0000", 2024-09-13T13:09:52.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112314 2024-09-13T13:09:52.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.362390+0000", 2024-09-13T13:09:52.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0083252619999999999 2024-09-13T13:09:52.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364495+0000", 2024-09-13T13:09:52.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021055179999999998 2024-09-13T13:09:52.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364585+0000", 2024-09-13T13:09:52.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9598000000000005e-05 2024-09-13T13:09:52.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364629+0000", 2024-09-13T13:09:52.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3711000000000002e-05 2024-09-13T13:09:52.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364660+0000", 2024-09-13T13:09:52.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1377999999999997e-05 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189512 7.d 7:b7cfc9e5:::benchmark_data_smithi052_71865_object68411:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.400157+0000", 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.81745128400001, 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96465484199999996, 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.295 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.296 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.296 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189512 2024-09-13T13:09:52.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.296 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400157+0000", 2024-09-13T13:09:52.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400157+0000", 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400158+0000", 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.16e-07 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400162+0000", 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.304e-06 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400170+0000", 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400182+0000", 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2254e-05 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400234+0000", 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.198e-05 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400244+0000", 2024-09-13T13:09:52.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0036e-05 2024-09-13T13:09:52.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356925+0000", 2024-09-13T13:09:52.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95668097299999999 2024-09-13T13:09:52.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.357056+0000", 2024-09-13T13:09:52.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013119499999999999 2024-09-13T13:09:52.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:09:52.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.357181+0000", 2024-09-13T13:09:52.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012478000000000001 2024-09-13T13:09:52.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364436+0000", 2024-09-13T13:09:52.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072547779999999999 2024-09-13T13:09:52.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364542+0000", 2024-09-13T13:09:52.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010571 2024-09-13T13:09:52.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364710+0000", 2024-09-13T13:09:52.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168486 2024-09-13T13:09:52.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364754+0000", 2024-09-13T13:09:52.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4141000000000003e-05 2024-09-13T13:09:52.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364812+0000", 2024-09-13T13:09:52.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8180000000000002e-05 2024-09-13T13:09:52.302 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.302 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.302 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.302 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189519 7.d 7:b156376f:::benchmark_data_smithi052_71865_object68418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.302 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.408717+0000", 2024-09-13T13:09:52.302 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.80889209399999, 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97666052999999997, 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189519 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408717+0000", 2024-09-13T13:09:52.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408717+0000", 2024-09-13T13:09:52.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408718+0000", 2024-09-13T13:09:52.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3939999999999999e-06 2024-09-13T13:09:52.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408722+0000", 2024-09-13T13:09:52.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2740000000000001e-06 2024-09-13T13:09:52.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408732+0000", 2024-09-13T13:09:52.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9129999999999998e-06 2024-09-13T13:09:52.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408739+0000", 2024-09-13T13:09:52.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7789999999999998e-06 2024-09-13T13:09:52.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408803+0000", 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3750000000000005e-05 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408811+0000", 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4079999999999998e-06 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.359633+0000", 2024-09-13T13:09:52.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95082174900000005 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.359882+0000", 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024927 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.359990+0000", 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010738099999999999 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364551+0000", 2024-09-13T13:09:52.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045615020000000003 2024-09-13T13:09:52.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371210+0000", 2024-09-13T13:09:52.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066585400000000001 2024-09-13T13:09:52.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.385259+0000", 2024-09-13T13:09:52.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014048932 2024-09-13T13:09:52.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.385324+0000", 2024-09-13T13:09:52.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5908999999999995e-05 2024-09-13T13:09:52.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.385377+0000", 2024-09-13T13:09:52.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2729000000000003e-05 2024-09-13T13:09:52.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.309 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189523 7.d 7:ba812c0f:::benchmark_data_smithi052_71865_object68422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.410273+0000", 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.807336007, 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98121212400000002, 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189523 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.311 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:52 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:09:52.311 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:52 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:09:52.311 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:52 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:52.311 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:52 smithi167 ceph-mon[33178]: from='client.35591 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:09:52.311 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:52 smithi167 ceph-mon[33178]: osdmap e235: 8 total, 8 up, 8 in 2024-09-13T13:09:52.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410273+0000", 2024-09-13T13:09:52.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410273+0000", 2024-09-13T13:09:52.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410274+0000", 2024-09-13T13:09:52.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3179999999999999e-06 2024-09-13T13:09:52.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410281+0000", 2024-09-13T13:09:52.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8700000000000003e-06 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410293+0000", 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.225e-05 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410300+0000", 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6529999999999997e-06 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410356+0000", 2024-09-13T13:09:52.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6366000000000002e-05 2024-09-13T13:09:52.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410366+0000", 2024-09-13T13:09:52.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0166999999999999e-05 2024-09-13T13:09:52.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.362418+0000", 2024-09-13T13:09:52.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95205126100000004 2024-09-13T13:09:52.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.362525+0000", 2024-09-13T13:09:52.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106965 2024-09-13T13:09:52.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:09:52.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.362594+0000", 2024-09-13T13:09:52.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8967999999999994e-05 2024-09-13T13:09:52.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.368931+0000", 2024-09-13T13:09:52.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063376480000000004 2024-09-13T13:09:52.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.387384+0000", 2024-09-13T13:09:52.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018452995999999999 2024-09-13T13:09:52.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391424+0000", 2024-09-13T13:09:52.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040394380000000002 2024-09-13T13:09:52.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391455+0000", 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1152999999999998e-05 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391485+0000", 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0071000000000001e-05 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189527 7.d 7:b61412a6:::benchmark_data_smithi052_71865_object68426:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.318 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.414261+0000", 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.803347718, 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97733418299999997, 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189527 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414261+0000", 2024-09-13T13:09:52.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414261+0000", 2024-09-13T13:09:52.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414262+0000", 2024-09-13T13:09:52.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6599999999999995e-07 2024-09-13T13:09:52.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414264+0000", 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4830000000000002e-06 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414273+0000", 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5669999999999995e-06 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414277+0000", 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9530000000000001e-06 2024-09-13T13:09:52.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414327+0000", 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9759999999999998e-05 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414336+0000", 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3330000000000003e-06 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364845+0000", 2024-09-13T13:09:52.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95050922199999999 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364910+0000", 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5036999999999994e-05 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364972+0000", 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1741000000000004e-05 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371169+0000", 2024-09-13T13:09:52.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061969069999999998 2024-09-13T13:09:52.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.387420+0000", 2024-09-13T13:09:52.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016250882000000001 2024-09-13T13:09:52.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391522+0000", 2024-09-13T13:09:52.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041024420000000004 2024-09-13T13:09:52.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391561+0000", 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9026999999999998e-05 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391595+0000", 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4063000000000002e-05 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.325 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189540 7.d 7:bd5b765e:::benchmark_data_smithi052_71865_object68439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.326 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.427873+0000", 2024-09-13T13:09:52.326 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.78973623299999, 2024-09-13T13:09:52.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96380134799999995, 2024-09-13T13:09:52.326 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.326 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.326 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.326 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.326 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.326 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.326 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189540 2024-09-13T13:09:52.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.326 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427873+0000", 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427873+0000", 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427874+0000", 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.765e-06 2024-09-13T13:09:52.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427880+0000", 2024-09-13T13:09:52.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2299999999999999e-06 2024-09-13T13:09:52.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427892+0000", 2024-09-13T13:09:52.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2578999999999999e-05 2024-09-13T13:09:52.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427901+0000", 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8480000000000007e-06 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427976+0000", 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4636999999999997e-05 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427982+0000", 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.366695+0000", 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93871229899999997 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.366759+0000", 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4239000000000006e-05 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.366861+0000", 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102506 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.372748+0000", 2024-09-13T13:09:52.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058863960000000003 2024-09-13T13:09:52.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.387463+0000", 2024-09-13T13:09:52.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014715045 2024-09-13T13:09:52.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391626+0000", 2024-09-13T13:09:52.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041627900000000004 2024-09-13T13:09:52.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391651+0000", 2024-09-13T13:09:52.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5678e-05 2024-09-13T13:09:52.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391674+0000", 2024-09-13T13:09:52.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2586e-05 2024-09-13T13:09:52.332 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189552 7.d 7:b0ca6a67:::benchmark_data_smithi052_71865_object68451:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.438326+0000", 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.77928256000001, 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96655187600000003, 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.333 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.334 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189552 2024-09-13T13:09:52.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.334 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438326+0000", 2024-09-13T13:09:52.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438326+0000", 2024-09-13T13:09:52.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438328+0000", 2024-09-13T13:09:52.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.587e-06 2024-09-13T13:09:52.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438337+0000", 2024-09-13T13:09:52.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7280000000000001e-06 2024-09-13T13:09:52.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438349+0000", 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2944e-05 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438356+0000", 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3369999999999996e-06 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438420+0000", 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4082000000000001e-05 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438430+0000", 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0356e-05 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.368958+0000", 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93052786899999995 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.369054+0000", 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5421999999999995e-05 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:09:52.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.369151+0000", 2024-09-13T13:09:52.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7264999999999998e-05 2024-09-13T13:09:52.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.375369+0000", 2024-09-13T13:09:52.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0062185900000000004 2024-09-13T13:09:52.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391714+0000", 2024-09-13T13:09:52.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016344250000000001 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404746+0000", 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013032871999999999 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404838+0000", 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1419e-05 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404878+0000", 2024-09-13T13:09:52.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0154999999999998e-05 2024-09-13T13:09:52.340 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.340 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.340 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.340 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189574 7.d 7:bbe21a3a:::benchmark_data_smithi052_71865_object68473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.340 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.459595+0000", 2024-09-13T13:09:52.340 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.758013597, 2024-09-13T13:09:52.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94540613600000001, 2024-09-13T13:09:52.340 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.340 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.340 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189574 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459595+0000", 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459595+0000", 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459597+0000", 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9920000000000002e-06 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459603+0000", 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7629999999999997e-06 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459615+0000", 2024-09-13T13:09:52.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1909999999999999e-05 2024-09-13T13:09:52.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459621+0000", 2024-09-13T13:09:52.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8139999999999997e-06 2024-09-13T13:09:52.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459683+0000", 2024-09-13T13:09:52.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2174000000000005e-05 2024-09-13T13:09:52.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459694+0000", 2024-09-13T13:09:52.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1297e-05 2024-09-13T13:09:52.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371239+0000", 2024-09-13T13:09:52.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.911544507 2024-09-13T13:09:52.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371305+0000", 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6738999999999999e-05 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371370+0000", 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4368000000000004e-05 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.380489+0000", 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091196799999999998 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391735+0000", 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011245965 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404921+0000", 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013185357 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404969+0000", 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8319999999999998e-05 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405001+0000", 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2249999999999998e-05 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189587 7.d 7:bb16c9f3:::benchmark_data_smithi052_71865_object68486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.470029+0000", 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.74757988599998, 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93511254399999999, 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.347 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189587 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470029+0000", 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470029+0000", 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470030+0000", 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.42e-06 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470034+0000", 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1919999999999998e-06 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470045+0000", 2024-09-13T13:09:52.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0239e-05 2024-09-13T13:09:52.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470050+0000", 2024-09-13T13:09:52.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2569999999999998e-06 2024-09-13T13:09:52.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470113+0000", 2024-09-13T13:09:52.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3417999999999995e-05 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470123+0000", 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3379999999999994e-06 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.372770+0000", 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.90264689300000001 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.372854+0000", 2024-09-13T13:09:52.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4221000000000006e-05 2024-09-13T13:09:52.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:09:52.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.372914+0000", 2024-09-13T13:09:52.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0451000000000001e-05 2024-09-13T13:09:52.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.380494+0000", 2024-09-13T13:09:52.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075800609999999999 2024-09-13T13:09:52.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.403571+0000", 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02307698 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405053+0000", 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014813879999999999 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405106+0000", 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2904e-05 2024-09-13T13:09:52.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405141+0000", 2024-09-13T13:09:52.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5781999999999997e-05 2024-09-13T13:09:52.354 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.354 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.354 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.354 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189600 7.d 7:b65df3b7:::benchmark_data_smithi052_71865_object68499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.354 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.492940+0000", 2024-09-13T13:09:52.354 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.724668748, 2024-09-13T13:09:52.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91826516899999999, 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189600 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492940+0000", 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492940+0000", 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492942+0000", 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3539999999999998e-06 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492952+0000", 2024-09-13T13:09:52.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.448e-06 2024-09-13T13:09:52.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492967+0000", 2024-09-13T13:09:52.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.554e-05 2024-09-13T13:09:52.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492975+0000", 2024-09-13T13:09:52.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:09:52.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.493051+0000", 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6112999999999994e-05 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.493064+0000", 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2235000000000001e-05 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.374296+0000", 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.88123260599999997 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.374360+0000", 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4077000000000003e-05 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.374430+0000", 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9789999999999994e-05 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.380501+0000", 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060705790000000004 2024-09-13T13:09:52.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405185+0000", 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024683738 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411130+0000", 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059459279999999996 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411163+0000", 2024-09-13T13:09:52.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2898000000000002e-05 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411205+0000", 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1841000000000001e-05 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189604 7.d 7:b85ca2e8:::benchmark_data_smithi052_71865_object68503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.496890+0000", 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.72071856299999, 2024-09-13T13:09:52.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91447940100000003, 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189604 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496890+0000", 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496890+0000", 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496892+0000", 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5149999999999999e-06 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496896+0000", 2024-09-13T13:09:52.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8569999999999997e-06 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496905+0000", 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4280000000000002e-06 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496910+0000", 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3499999999999996e-06 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496968+0000", 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7939000000000002e-05 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496978+0000", 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0209e-05 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.375402+0000", 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87842360799999997 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.375497+0000", 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4949000000000002e-05 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.375606+0000", 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108652 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.380505+0000", 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048993880000000002 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405223+0000", 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024717682000000001 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411282+0000", 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060587799999999997 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411332+0000", 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0365000000000003e-05 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:09:52.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:09:52.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:09:52.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411370+0000", 2024-09-13T13:09:52.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7679000000000003e-05 2024-09-13T13:09:52.368 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.368 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.368 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.368 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:09:52.368 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:09:52.368 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:09:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:52.370 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:52.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:52.468 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:09:52.468 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:09:52.472 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:52.504 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:09:52.504 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:09:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:52.515 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:52.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:52.536 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:09:52.536 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:09:52.564 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:09:52.564 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:09:52.598 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:09:52.599 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:09:52.616 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:52.632 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:09:52.632 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:09:52.647 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:52.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:52.664 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:09:52.665 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:09:52.699 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:09:52.700 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:09:52.749 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:52.781 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:52.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:52.884 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:52.924 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:52.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:52.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:53.026 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:53.074 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:53.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:53.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:53.176 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:53.219 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:53.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:53.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:53 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:09:53.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:53 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:09:53.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:53 smithi052 ceph-mon[36682]: pgmap v416: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 381 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 898 op/s 2024-09-13T13:09:53.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:53 smithi052 ceph-mon[36682]: osdmap e236: 8 total, 8 up, 8 in 2024-09-13T13:09:53.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:53 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3649380709' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:53.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:53 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:09:53.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:53 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:09:53.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:53 smithi052 ceph-mon[29471]: pgmap v416: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 381 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 898 op/s 2024-09-13T13:09:53.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:53 smithi052 ceph-mon[29471]: osdmap e236: 8 total, 8 up, 8 in 2024-09-13T13:09:53.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3649380709' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:53.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:53 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:09:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:53 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:09:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:53 smithi167 ceph-mon[33178]: pgmap v416: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 381 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 898 op/s 2024-09-13T13:09:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:53 smithi167 ceph-mon[33178]: osdmap e236: 8 total, 8 up, 8 in 2024-09-13T13:09:53.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:53 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3649380709' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:53.321 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:53.357 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:53.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:53.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:53.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T13:09:53.460 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:53.490 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:53.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:53.592 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:53.644 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:53.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:53.715 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:09:53.746 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:53.829 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:53.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:53.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:53.927 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:53.933 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:54.001 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:54.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:54.106 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:54.123 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Removed 127926 objects 2024-09-13T13:09:54.123 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Clean up completed and total clean up time :112.332 2024-09-13T13:09:54.138 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:54.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:54.240 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:54.262 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:54.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:54.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:09:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:09:54.364 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:54.386 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:54.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:54.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:54.470 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:09:54.471 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:09:54.488 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:54.511 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:54.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:54.613 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:54.635 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:54.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:54.737 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:54.759 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:54.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:54.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:54.861 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:54.885 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:54.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:54.987 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:55.008 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:55.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:55.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:55.110 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:55.135 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:55.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:55.237 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:55.261 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:55.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:55.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:55 smithi167 ceph-mon[33178]: pgmap v418: 148 pgs: 148 active+clean; 131 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-13T13:09:55.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:55.363 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:55.387 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:55.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:55 smithi052 ceph-mon[29471]: pgmap v418: 148 pgs: 148 active+clean; 131 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-13T13:09:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:55 smithi052 ceph-mon[36682]: pgmap v418: 148 pgs: 148 active+clean; 131 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-13T13:09:55.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:09:55.489 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:55.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:55.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:55.523 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:09:55.543 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T13:09:55.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:55.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:55.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:55.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:55.745 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:55.770 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:55.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:55.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:55.871 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:55.895 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:55.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:55.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:55.988 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:09:56.072 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:09:56.073 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:09:56.073 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:56.074 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:56.091 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:56.104 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:56.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:56.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:56 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:09:56.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:56 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:09:56.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:56 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:09:56.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:56 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:09:56.207 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:56.235 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:56.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:56.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:56 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:09:56.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:56 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:09:56.336 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:56.359 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:56.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:56.461 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:56.482 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:09:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:09:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:09:56.487 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:56.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:56.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:56.589 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:56.613 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:56.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:56.714 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:56.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:56.738 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:56.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:56.839 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:09:56.861 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:56.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:09:56.935 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:09:56.935 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:09:56.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:56.988 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:56.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:56.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:57.054 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:09:57.091 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:57.113 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:57.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:57.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:57.144 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:57 smithi052 ceph-mon[36682]: pgmap v419: 148 pgs: 148 active+clean; 64 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 752 B/s rd, 0 B/s wr, 846 op/s 2024-09-13T13:09:57.145 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:57 smithi052 ceph-mon[36682]: osdmap e237: 8 total, 8 up, 8 in 2024-09-13T13:09:57.145 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:57 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3926857944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:09:57.145 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:57 smithi052 ceph-mon[36682]: from='client.35597 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:09:57.145 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:57 smithi052 ceph-mon[29471]: pgmap v419: 148 pgs: 148 active+clean; 64 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 752 B/s rd, 0 B/s wr, 846 op/s 2024-09-13T13:09:57.145 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:57 smithi052 ceph-mon[29471]: osdmap e237: 8 total, 8 up, 8 in 2024-09-13T13:09:57.145 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:57 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3926857944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:09:57.145 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:57 smithi052 ceph-mon[29471]: from='client.35597 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:09:57.215 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:57.238 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:57.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:57.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:57 smithi167 ceph-mon[33178]: pgmap v419: 148 pgs: 148 active+clean; 64 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 752 B/s rd, 0 B/s wr, 846 op/s 2024-09-13T13:09:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:57 smithi167 ceph-mon[33178]: osdmap e237: 8 total, 8 up, 8 in 2024-09-13T13:09:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:57 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3926857944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:09:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:57 smithi167 ceph-mon[33178]: from='client.35597 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:09:57.339 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:57.365 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:57.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:57.467 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:57.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:57.491 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:57.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:57.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:09:57.618 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:57.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:57.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:09:57.720 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:57.744 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:57.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:57.845 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:57.870 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:57.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:57.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:57.970 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:09:57.970 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:09:57.971 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:09:57.987 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:09:57.987 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:09:57.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:57.999 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:57.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:09:58.003 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:09:58.003 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:09:58.019 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:09:58.019 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:09:58.037 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:09:58.037 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:09:58.058 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:09:58.058 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:09:58.077 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:09:58.077 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:09:58.093 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:09:58.094 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:09:58.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:58.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:58.125 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:58.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:58.226 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:58 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:58 smithi052 ceph-mon[36682]: from='client.35597 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:09:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:58 smithi052 ceph-mon[36682]: osdmap e238: 8 total, 8 up, 8 in 2024-09-13T13:09:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:58 smithi052 ceph-mon[36682]: 4.1 scrub starts 2024-09-13T13:09:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:58 smithi052 ceph-mon[36682]: 4.1 scrub ok 2024-09-13T13:09:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:58 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3283257106' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:58.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:58 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:58.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:58 smithi052 ceph-mon[29471]: from='client.35597 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:09:58.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:58 smithi052 ceph-mon[29471]: osdmap e238: 8 total, 8 up, 8 in 2024-09-13T13:09:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:58 smithi052 ceph-mon[29471]: 4.1 scrub starts 2024-09-13T13:09:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:58 smithi052 ceph-mon[29471]: 4.1 scrub ok 2024-09-13T13:09:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:58 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3283257106' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:58.249 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:58.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:58.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:58.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:58 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:09:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:58 smithi167 ceph-mon[33178]: from='client.35597 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:09:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:58 smithi167 ceph-mon[33178]: osdmap e238: 8 total, 8 up, 8 in 2024-09-13T13:09:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:58 smithi167 ceph-mon[33178]: 4.1 scrub starts 2024-09-13T13:09:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:58 smithi167 ceph-mon[33178]: 4.1 scrub ok 2024-09-13T13:09:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:58 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3283257106' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:09:58.350 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:09:58.374 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:58.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:09:58.385 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T13:09:58.476 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:58.500 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:58.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:58.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:09:58.626 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:58.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:09:58.728 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:58.753 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:58.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:58.768 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:58.803 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:09:58.855 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:58.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:58.878 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:58.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:58.979 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:59.005 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:59.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:59.107 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:09:59.130 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:59.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:59.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:09:59.159 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-13T13:09:59.160 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-13T13:09:59.193 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:09:59 smithi052 ceph-mon[36682]: pgmap v422: 148 pgs: 148 active+clean; 37 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 918 op/s 2024-09-13T13:09:59.193 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:09:59 smithi052 ceph-mon[29471]: pgmap v422: 148 pgs: 148 active+clean; 37 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 918 op/s 2024-09-13T13:09:59.232 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:09:59.255 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:09:59.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:59.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:09:59.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:09:59 smithi167 ceph-mon[33178]: pgmap v422: 148 pgs: 148 active+clean; 37 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 918 op/s 2024-09-13T13:09:59.357 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:59.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:59.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:59.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:59.506 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:09:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:09:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:09:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:09:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83976 7.7 7:ec00590c:::benchmark_data_smithi052_71865_object83975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:09:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.558362+0000", 2024-09-13T13:09:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 404.94646006099998, 2024-09-13T13:09:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99679333000000003, 2024-09-13T13:09:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83976 2024-09-13T13:09:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558362+0000", 2024-09-13T13:09:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558362+0000", 2024-09-13T13:09:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558364+0000", 2024-09-13T13:09:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0870000000000001e-06 2024-09-13T13:09:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558389+0000", 2024-09-13T13:09:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5051000000000001e-05 2024-09-13T13:09:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558401+0000", 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1943000000000001e-05 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558409+0000", 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1580000000000005e-06 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558524+0000", 2024-09-13T13:09:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114991 2024-09-13T13:09:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:09:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558532+0000", 2024-09-13T13:09:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0450000000000001e-06 2024-09-13T13:09:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.542003+0000", 2024-09-13T13:09:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98347105700000004 2024-09-13T13:09:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.544289+0000", 2024-09-13T13:09:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002285493 2024-09-13T13:09:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:09:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.544471+0000", 2024-09-13T13:09:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018247200000000001 2024-09-13T13:09:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555006+0000", 2024-09-13T13:09:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010534559000000001 2024-09-13T13:09:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555018+0000", 2024-09-13T13:09:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1756999999999999e-05 2024-09-13T13:09:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555068+0000", 2024-09-13T13:09:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9982999999999999e-05 2024-09-13T13:09:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555155+0000", 2024-09-13T13:09:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7733999999999997e-05 2024-09-13T13:09:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83980 7.7 7:e454f641:::benchmark_data_smithi052_71865_object83979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:09:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.562536+0000", 2024-09-13T13:09:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 404.94228568099999, 2024-09-13T13:09:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99324992999999995, 2024-09-13T13:09:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83980 2024-09-13T13:09:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562536+0000", 2024-09-13T13:09:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562536+0000", 2024-09-13T13:09:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562538+0000", 2024-09-13T13:09:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9460000000000001e-06 2024-09-13T13:09:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562564+0000", 2024-09-13T13:09:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5812e-05 2024-09-13T13:09:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562579+0000", 2024-09-13T13:09:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4515000000000001e-05 2024-09-13T13:09:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562587+0000", 2024-09-13T13:09:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1529999999999997e-06 2024-09-13T13:09:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562654+0000", 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7094000000000005e-05 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562662+0000", 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2870000000000004e-06 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548143+0000", 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98548045299999998 2024-09-13T13:09:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548261+0000", 2024-09-13T13:09:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118776 2024-09-13T13:09:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:09:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548419+0000", 2024-09-13T13:09:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015719799999999999 2024-09-13T13:09:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555169+0000", 2024-09-13T13:09:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0067500839999999999 2024-09-13T13:09:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555200+0000", 2024-09-13T13:09:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1285999999999998e-05 2024-09-13T13:09:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555285+0000", 2024-09-13T13:09:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4899000000000001e-05 2024-09-13T13:09:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555786+0000", 2024-09-13T13:09:59.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00050142700000000004 2024-09-13T13:09:59.521 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.521 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.521 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:59.523 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83984 7.7 7:e1036ce8:::benchmark_data_smithi052_71865_object83983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:09:59.523 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.567703+0000", 2024-09-13T13:09:59.523 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 404.93711856499999, 2024-09-13T13:09:59.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99284181800000004, 2024-09-13T13:09:59.523 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.523 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.523 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.523 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.523 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:59.523 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:59.523 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83984 2024-09-13T13:09:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567703+0000", 2024-09-13T13:09:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567703+0000", 2024-09-13T13:09:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567706+0000", 2024-09-13T13:09:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3889999999999999e-06 2024-09-13T13:09:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567733+0000", 2024-09-13T13:09:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.69e-05 2024-09-13T13:09:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567754+0000", 2024-09-13T13:09:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1772000000000001e-05 2024-09-13T13:09:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567765+0000", 2024-09-13T13:09:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0553999999999999e-05 2024-09-13T13:09:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567856+0000", 2024-09-13T13:09:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0926000000000003e-05 2024-09-13T13:09:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:09:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567874+0000", 2024-09-13T13:09:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8481000000000001e-05 2024-09-13T13:09:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552014+0000", 2024-09-13T13:09:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98413990600000001 2024-09-13T13:09:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552117+0000", 2024-09-13T13:09:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010265200000000001 2024-09-13T13:09:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:09:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552243+0000", 2024-09-13T13:09:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012594900000000001 2024-09-13T13:09:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560036+0000", 2024-09-13T13:09:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0077931570000000002 2024-09-13T13:09:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560388+0000", 2024-09-13T13:09:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035232500000000001 2024-09-13T13:09:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560444+0000", 2024-09-13T13:09:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5460000000000001e-05 2024-09-13T13:09:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560545+0000", 2024-09-13T13:09:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101347 2024-09-13T13:09:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.14493.0:1217 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94064716851456 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:09:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:15.061486+0000", 2024-09-13T13:09:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 404.44333641200001, 2024-09-13T13:09:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.48025142399999998, 2024-09-13T13:09:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:09:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.14493", 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 1217 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061486+0000", 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061486+0000", 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061489+0000", 2024-09-13T13:09:59.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3249999999999999e-06 2024-09-13T13:09:59.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061499+0000", 2024-09-13T13:09:59.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0463e-05 2024-09-13T13:09:59.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061513+0000", 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3269e-05 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061521+0000", 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.909e-06 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061582+0000", 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1568000000000004e-05 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:09:59.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061589+0000", 2024-09-13T13:09:59.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2060000000000001e-06 2024-09-13T13:09:59.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.537301+0000", 2024-09-13T13:09:59.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.47571140000000001 2024-09-13T13:09:59.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.537446+0000", 2024-09-13T13:09:59.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000145339 2024-09-13T13:09:59.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.541737+0000", 2024-09-13T13:09:59.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.004290945 2024-09-13T13:09:59.535 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.535 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.535 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.535 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:118713 7.d e115/103 7:b0c4bc72:::benchmark_data_smithi052_71865_object118712:head v 115'7521, mlcod=115'7517)", 2024-09-13T13:09:59.535 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:49.563909+0000", 2024-09-13T13:09:59.535 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 369.94091256799999, 2024-09-13T13:09:59.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.118718926, 2024-09-13T13:09:59.536 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.536 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.536 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.536 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.563909+0000", 2024-09-13T13:09:59.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.563909+0000", 2024-09-13T13:09:59.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.563910+0000", 2024-09-13T13:09:59.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.02e-07 2024-09-13T13:09:59.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.563942+0000", 2024-09-13T13:09:59.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1276999999999999e-05 2024-09-13T13:09:59.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.563949+0000", 2024-09-13T13:09:59.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8709999999999997e-06 2024-09-13T13:09:59.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.563955+0000", 2024-09-13T13:09:59.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7919999999999998e-06 2024-09-13T13:09:59.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.671251+0000", 2024-09-13T13:09:59.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.107295289 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.671270+0000", 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9649000000000001e-05 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.682606+0000", 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011335567 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.539 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.682628+0000", 2024-09-13T13:09:59.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2578999999999998e-05 2024-09-13T13:09:59.540 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.540 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.540 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.540 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120039 7.6 7:6fe353e4:::benchmark_data_smithi052_71865_object120038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:09:59.540 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.024096+0000", 2024-09-13T13:09:59.540 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 368.48072575999998, 2024-09-13T13:09:59.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.121330047, 2024-09-13T13:09:59.540 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.540 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120039 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024096+0000", 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024096+0000", 2024-09-13T13:09:59.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.542 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024098+0000", 2024-09-13T13:09:59.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3459999999999999e-06 2024-09-13T13:09:59.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.542 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024118+0000", 2024-09-13T13:09:59.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0363000000000001e-05 2024-09-13T13:09:59.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024129+0000", 2024-09-13T13:09:59.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1396000000000001e-05 2024-09-13T13:09:59.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.024136+0000", 2024-09-13T13:09:59.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8369999999999999e-06 2024-09-13T13:09:59.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032888+0000", 2024-09-13T13:09:59.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0087514399999999992 2024-09-13T13:09:59.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032997+0000", 2024-09-13T13:09:59.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109399 2024-09-13T13:09:59.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:59.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033123+0000", 2024-09-13T13:09:59.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012575100000000001 2024-09-13T13:09:59.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138649+0000", 2024-09-13T13:09:59.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10552584600000001 2024-09-13T13:09:59.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:59.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.140107+0000", 2024-09-13T13:09:59.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0014585620000000001 2024-09-13T13:09:59.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145128+0000", 2024-09-13T13:09:59.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050206909999999999 2024-09-13T13:09:59.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145376+0000", 2024-09-13T13:09:59.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024774999999999998 2024-09-13T13:09:59.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145426+0000", 2024-09-13T13:09:59.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0665999999999999e-05 2024-09-13T13:09:59.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.546 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.547 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120046 7.6 7:64955858:::benchmark_data_smithi052_71865_object120045:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:09:59.547 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.028119+0000", 2024-09-13T13:09:59.547 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 368.47670284700001, 2024-09-13T13:09:59.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12583912, 2024-09-13T13:09:59.547 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.547 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.547 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.547 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.547 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:59.547 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:59.547 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120046 2024-09-13T13:09:59.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.548 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028119+0000", 2024-09-13T13:09:59.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028119+0000", 2024-09-13T13:09:59.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028120+0000", 2024-09-13T13:09:59.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2999999999999999e-07 2024-09-13T13:09:59.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028143+0000", 2024-09-13T13:09:59.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3104000000000001e-05 2024-09-13T13:09:59.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028151+0000", 2024-09-13T13:09:59.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9850000000000006e-06 2024-09-13T13:09:59.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028158+0000", 2024-09-13T13:09:59.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5969999999999997e-06 2024-09-13T13:09:59.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036251+0000", 2024-09-13T13:09:59.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0080934660000000005 2024-09-13T13:09:59.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036367+0000", 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000115832 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036497+0000", 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012981399999999999 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145466+0000", 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10896932099999999 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145645+0000", 2024-09-13T13:09:59.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017834400000000001 2024-09-13T13:09:59.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:59.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153238+0000", 2024-09-13T13:09:59.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0075929919999999998 2024-09-13T13:09:59.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153469+0000", 2024-09-13T13:09:59.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00023148999999999999 2024-09-13T13:09:59.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153958+0000", 2024-09-13T13:09:59.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00048924500000000002 2024-09-13T13:09:59.553 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.553 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.553 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.553 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120049 7.6 7:6e13283c:::benchmark_data_smithi052_71865_object120048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:09:59.553 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.031872+0000", 2024-09-13T13:09:59.553 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 368.47295036700001, 2024-09-13T13:09:59.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.125660096, 2024-09-13T13:09:59.554 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.554 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.554 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.554 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.554 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:59.554 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:59.554 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120049 2024-09-13T13:09:59.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.554 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031872+0000", 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031872+0000", 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031873+0000", 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0130000000000001e-06 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031889+0000", 2024-09-13T13:09:59.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6218999999999998e-05 2024-09-13T13:09:59.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031895+0000", 2024-09-13T13:09:59.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3300000000000004e-06 2024-09-13T13:09:59.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031900+0000", 2024-09-13T13:09:59.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4129999999999999e-06 2024-09-13T13:09:59.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145674+0000", 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.113774598 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145885+0000", 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000210929 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.146035+0000", 2024-09-13T13:09:59.557 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015038000000000001 2024-09-13T13:09:59.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.149376+0000", 2024-09-13T13:09:59.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033409690000000001 2024-09-13T13:09:59.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153999+0000", 2024-09-13T13:09:59.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0046225670000000002 2024-09-13T13:09:59.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157357+0000", 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033581800000000001 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157485+0000", 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012788599999999999 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157532+0000", 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6612e-05 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.559 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.560 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.560 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:120050 7.e e115/84 7:72296fb6:::benchmark_data_smithi052_71865_object120049:head v 115'7271, mlcod=115'7268)", 2024-09-13T13:09:59.560 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.036753+0000", 2024-09-13T13:09:59.560 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 368.46806907400003, 2024-09-13T13:09:59.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.120595173, 2024-09-13T13:09:59.560 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.560 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.560 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.560 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036753+0000", 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036753+0000", 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036754+0000", 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5099999999999998e-07 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036779+0000", 2024-09-13T13:09:59.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5122000000000001e-05 2024-09-13T13:09:59.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036786+0000", 2024-09-13T13:09:59.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9879999999999998e-06 2024-09-13T13:09:59.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036790+0000", 2024-09-13T13:09:59.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6e-06 2024-09-13T13:09:59.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138925+0000", 2024-09-13T13:09:59.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.102134962 2024-09-13T13:09:59.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138968+0000", 2024-09-13T13:09:59.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2760999999999999e-05 2024-09-13T13:09:59.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157292+0000", 2024-09-13T13:09:59.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.018324258 2024-09-13T13:09:59.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.564 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157348+0000", 2024-09-13T13:09:59.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5631000000000003e-05 2024-09-13T13:09:59.564 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.564 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.564 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.564 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.34781.0:120054 7.e e115/84 7:7dad863d:::benchmark_data_smithi052_71865_object120053:head v 115'7272, mlcod=115'7268)", 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.038814+0000", 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 368.46600762200001, 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.118752855, 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038814+0000", 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038814+0000", 2024-09-13T13:09:59.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038815+0000", 2024-09-13T13:09:59.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0640000000000001e-06 2024-09-13T13:09:59.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038839+0000", 2024-09-13T13:09:59.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3828999999999998e-05 2024-09-13T13:09:59.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038847+0000", 2024-09-13T13:09:59.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9990000000000008e-06 2024-09-13T13:09:59.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038853+0000", 2024-09-13T13:09:59.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8329999999999999e-06 2024-09-13T13:09:59.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145132+0000", 2024-09-13T13:09:59.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.106279179 2024-09-13T13:09:59.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145152+0000", 2024-09-13T13:09:59.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9264999999999999e-05 2024-09-13T13:09:59.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157535+0000", 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.012383939 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157567+0000", 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1746999999999999e-05 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.570 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120055 7.6 7:6a228e92:::benchmark_data_smithi052_71865_object120054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:09:59.570 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.039067+0000", 2024-09-13T13:09:59.570 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 368.46575535599999, 2024-09-13T13:09:59.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.124460581, 2024-09-13T13:09:59.570 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.570 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.570 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.570 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.570 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:59.570 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:59.570 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120055 2024-09-13T13:09:59.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.570 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039067+0000", 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039067+0000", 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039068+0000", 2024-09-13T13:09:59.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7859999999999999e-06 2024-09-13T13:09:59.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039092+0000", 2024-09-13T13:09:59.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3323999999999999e-05 2024-09-13T13:09:59.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039103+0000", 2024-09-13T13:09:59.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1408e-05 2024-09-13T13:09:59.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039110+0000", 2024-09-13T13:09:59.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0700000000000001e-06 2024-09-13T13:09:59.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.149407+0000", 2024-09-13T13:09:59.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.110296832 2024-09-13T13:09:59.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.149533+0000", 2024-09-13T13:09:59.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012555699999999999 2024-09-13T13:09:59.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:09:59.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.149681+0000", 2024-09-13T13:09:59.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014797000000000001 2024-09-13T13:09:59.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153150+0000", 2024-09-13T13:09:59.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0034697669999999999 2024-09-13T13:09:59.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:59.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157574+0000", 2024-09-13T13:09:59.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.004423382 2024-09-13T13:09:59.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.163426+0000", 2024-09-13T13:09:59.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058521060000000001 2024-09-13T13:09:59.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.163482+0000", 2024-09-13T13:09:59.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6328e-05 2024-09-13T13:09:59.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.163527+0000", 2024-09-13T13:09:59.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5050999999999999e-05 2024-09-13T13:09:59.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.576 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.576 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:199048 7.7 7:ef6de7c0:::benchmark_data_smithi052_71865_object77947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T13:09:59.577 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:05:25.185200+0000", 2024-09-13T13:09:59.577 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 274.31962156999998, 2024-09-13T13:09:59.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.127661889, 2024-09-13T13:09:59.577 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.577 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.577 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.577 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.577 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:09:59.577 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:09:59.577 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 199048 2024-09-13T13:09:59.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.577 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185200+0000", 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185200+0000", 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185201+0000", 2024-09-13T13:09:59.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3900000000000004e-07 2024-09-13T13:09:59.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185205+0000", 2024-09-13T13:09:59.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.455e-06 2024-09-13T13:09:59.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185358+0000", 2024-09-13T13:09:59.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000152919 2024-09-13T13:09:59.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185368+0000", 2024-09-13T13:09:59.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0465000000000001e-05 2024-09-13T13:09:59.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.201869+0000", 2024-09-13T13:09:59.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.016500959999999999 2024-09-13T13:09:59.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.202064+0000", 2024-09-13T13:09:59.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019464999999999999 2024-09-13T13:09:59.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:09:59.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.202323+0000", 2024-09-13T13:09:59.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00025961499999999999 2024-09-13T13:09:59.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.210108+0000", 2024-09-13T13:09:59.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:59.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:59.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.007784549 2024-09-13T13:09:59.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:59.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.219023+0000", 2024-09-13T13:09:59.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0089156979999999997 2024-09-13T13:09:59.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.312149+0000", 2024-09-13T13:09:59.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.093125873999999997 2024-09-13T13:09:59.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.312287+0000", 2024-09-13T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013777700000000001 2024-09-13T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.312862+0000", 2024-09-13T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00057518800000000002 2024-09-13T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237925 8.6 8:64810c19:::benchmark_data_smithi052_101302_object109996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.068404+0000", 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 21.436418073999999, 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.14961675099999999, 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237925 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068404+0000", 2024-09-13T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068404+0000", 2024-09-13T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068406+0000", 2024-09-13T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.728e-06 2024-09-13T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068411+0000", 2024-09-13T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6019999999999999e-06 2024-09-13T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068422+0000", 2024-09-13T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0769e-05 2024-09-13T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068429+0000", 2024-09-13T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7889999999999997e-06 2024-09-13T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068490+0000", 2024-09-13T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0983000000000002e-05 2024-09-13T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068617+0000", 2024-09-13T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000127267 2024-09-13T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068733+0000", 2024-09-13T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116445 2024-09-13T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074303+0000", 2024-09-13T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055691739999999997 2024-09-13T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217789+0000", 2024-09-13T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.143486802 2024-09-13T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217899+0000", 2024-09-13T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110011 2024-09-13T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217970+0000", 2024-09-13T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0506999999999994e-05 2024-09-13T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218021+0000", 2024-09-13T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0674000000000003e-05 2024-09-13T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237927 8.6 8:66a2a178:::benchmark_data_smithi052_101302_object109998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.068600+0000", 2024-09-13T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 21.436221603, 2024-09-13T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.149587739, 2024-09-13T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237927 2024-09-13T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068600+0000", 2024-09-13T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068600+0000", 2024-09-13T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068602+0000", 2024-09-13T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.176e-06 2024-09-13T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068606+0000", 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3039999999999998e-06 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068615+0000", 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9369999999999993e-06 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068624+0000", 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8400000000000001e-06 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071818+0000", 2024-09-13T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003194625 2024-09-13T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071929+0000", 2024-09-13T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110269 2024-09-13T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.072044+0000", 2024-09-13T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000115396 2024-09-13T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077466+0000", 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.005422568 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217827+0000", 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.14036069200000001 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218099+0000", 2024-09-13T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000271511 2024-09-13T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218152+0000", 2024-09-13T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2831000000000003e-05 2024-09-13T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218188+0000", 2024-09-13T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.659e-05 2024-09-13T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237928 8.6 8:692bc360:::benchmark_data_smithi052_101302_object109999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.068748+0000", 2024-09-13T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 21.436074256000001, 2024-09-13T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15122808900000001, 2024-09-13T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237928 2024-09-13T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068748+0000", 2024-09-13T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068748+0000", 2024-09-13T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.600 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.600 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068749+0000", 2024-09-13T13:09:59.600 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8599999999999996e-07 2024-09-13T13:09:59.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.600 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.600 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068753+0000", 2024-09-13T13:09:59.600 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9589999999999997e-06 2024-09-13T13:09:59.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.600 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.600 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068760+0000", 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6860000000000006e-06 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068766+0000", 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8050000000000003e-06 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074343+0000", 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055769210000000003 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.601 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.602 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074450+0000", 2024-09-13T13:09:59.602 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107321 2024-09-13T13:09:59.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.602 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:09:59.602 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074571+0000", 2024-09-13T13:09:59.602 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012104699999999999 2024-09-13T13:09:59.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.602 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:59.602 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214148+0000", 2024-09-13T13:09:59.602 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.139576799 2024-09-13T13:09:59.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.603 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.603 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217865+0000", 2024-09-13T13:09:59.603 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037168190000000001 2024-09-13T13:09:59.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.603 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.603 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218583+0000", 2024-09-13T13:09:59.603 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00071750199999999996 2024-09-13T13:09:59.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.603 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.603 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.219928+0000", 2024-09-13T13:09:59.604 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0013456729999999999 2024-09-13T13:09:59.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.604 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.604 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.604 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.219976+0000", 2024-09-13T13:09:59.604 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7571000000000002e-05 2024-09-13T13:09:59.604 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.604 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.604 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.604 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.604 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237931 8.e 8:77956f02:::benchmark_data_smithi052_101302_object110002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:09:59.604 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.071964+0000", 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 21.432858233000001, 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.14610564600000001, 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237931 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.605 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071964+0000", 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071964+0000", 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071966+0000", 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8560000000000001e-06 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.606 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.607 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071971+0000", 2024-09-13T13:09:59.607 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9660000000000004e-06 2024-09-13T13:09:59.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.607 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.607 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071982+0000", 2024-09-13T13:09:59.607 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1214e-05 2024-09-13T13:09:59.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.607 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.607 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071989+0000", 2024-09-13T13:09:59.607 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.644e-06 2024-09-13T13:09:59.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074334+0000", 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00234475 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074450+0000", 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011572499999999999 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074571+0000", 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121546 2024-09-13T13:09:59.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.609 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.609 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.609 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214179+0000", 2024-09-13T13:09:59.609 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.139607703 2024-09-13T13:09:59.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.609 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.609 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:59.609 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217833+0000", 2024-09-13T13:09:59.609 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036534179999999999 2024-09-13T13:09:59.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.609 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.609 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.609 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217883+0000", 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0846000000000001e-05 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217960+0000", 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6120999999999998e-05 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218069+0000", 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109857 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.25222.0:237924 8.9 e232/169 8:9137082c:::benchmark_data_smithi052_101302_object109995:head v 232'14652, mlcod=232'14649)", 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.076510+0000", 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 21.428311517000001, 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13780474600000001, 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076510+0000", 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.611 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.612 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.612 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076510+0000", 2024-09-13T13:09:59.612 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.612 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.612 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.612 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.612 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076512+0000", 2024-09-13T13:09:59.612 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.486e-06 2024-09-13T13:09:59.612 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.612 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.612 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.612 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076518+0000", 2024-09-13T13:09:59.612 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7239999999999997e-06 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076528+0000", 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8400000000000007e-06 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076533+0000", 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3090000000000003e-06 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.613 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076572+0000", 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.879e-05 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076585+0000", 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2955e-05 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214214+0000", 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13762922 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.614 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214315+0000", 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010142200000000001 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.25222.0:237931 8.e e232/169 ondisk, result = 0)", 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.077192+0000", 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 21.427630412999999, 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13702303699999999, 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.615 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.616 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.616 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077192+0000", 2024-09-13T13:09:59.616 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.616 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.616 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.616 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.616 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077192+0000", 2024-09-13T13:09:59.616 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.616 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.616 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.616 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.616 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077193+0000", 2024-09-13T13:09:59.616 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.184e-06 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077198+0000", 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0470000000000001e-06 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077207+0000", 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3619999999999995e-06 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077214+0000", 2024-09-13T13:09:59.617 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7020000000000004e-06 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214138+0000", 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.136924569 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214162+0000", 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3496999999999999e-05 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214215+0000", 2024-09-13T13:09:59.618 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2676000000000002e-05 2024-09-13T13:09:59.619 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.619 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.619 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.619 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.619 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.619 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.25222.0:237935 8.1 e232/169 8:84af4e89:::benchmark_data_smithi052_101302_object110006:head v 232'14986, mlcod=232'14984)", 2024-09-13T13:09:59.619 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.080810+0000", 2024-09-13T13:09:59.619 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 21.424012298000001, 2024-09-13T13:09:59.619 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.149619258, 2024-09-13T13:09:59.619 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.619 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.619 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.619 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.620 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.620 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.620 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.080810+0000", 2024-09-13T13:09:59.620 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.620 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.620 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.620 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.620 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.080810+0000", 2024-09-13T13:09:59.620 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.620 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.620 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.620 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.620 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.080812+0000", 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.857e-06 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.080821+0000", 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4930000000000004e-06 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214079+0000", 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13325792 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.621 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.622 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214106+0000", 2024-09-13T13:09:59.622 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7107e-05 2024-09-13T13:09:59.622 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.622 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.622 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.622 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214348+0000", 2024-09-13T13:09:59.622 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024191600000000001 2024-09-13T13:09:59.622 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.622 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.622 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.622 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214369+0000", 2024-09-13T13:09:59.622 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1149e-05 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.230395+0000", 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.016025604999999998 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.230429+0000", 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4211000000000002e-05 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237940 8.e 8:7ee1d0e2:::benchmark_data_smithi052_101302_object110011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.085058+0000", 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 21.419764392000001, 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13638705000000001, 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237940 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.624 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:09:59.625 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.625 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:09:59.625 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.085058+0000", 2024-09-13T13:09:59.625 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.625 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.625 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:09:59.625 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.085058+0000", 2024-09-13T13:09:59.625 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:09:59.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.625 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.625 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:09:59.625 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.085059+0000", 2024-09-13T13:09:59.626 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.666e-06 2024-09-13T13:09:59.626 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.626 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.626 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:09:59.626 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.085065+0000", 2024-09-13T13:09:59.626 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5770000000000001e-06 2024-09-13T13:09:59.626 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.626 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.626 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:09:59.626 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214099+0000", 2024-09-13T13:09:59.626 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.129034225 2024-09-13T13:09:59.626 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.626 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.627 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:09:59.627 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214130+0000", 2024-09-13T13:09:59.627 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1145000000000001e-05 2024-09-13T13:09:59.627 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.627 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.627 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:09:59.627 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214387+0000", 2024-09-13T13:09:59.627 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000257302 2024-09-13T13:09:59.627 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.627 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.627 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:09:59.627 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214491+0000", 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010342 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214593+0000", 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010197299999999999 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218079+0000", 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003486266 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.629 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218228+0000", 2024-09-13T13:09:59.629 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014901399999999999 2024-09-13T13:09:59.629 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.629 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.629 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:09:59.629 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.221356+0000", 2024-09-13T13:09:59.629 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031274430000000001 2024-09-13T13:09:59.629 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.629 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.629 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:09:59.629 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.221405+0000", 2024-09-13T13:09:59.629 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9570000000000001e-05 2024-09-13T13:09:59.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:09:59.630 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:09:59.630 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:09:59.630 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.221445+0000", 2024-09-13T13:09:59.630 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9449000000000001e-05 2024-09-13T13:09:59.630 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.630 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.630 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.630 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:09:59.630 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:09:59.630 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:09:59.683 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:59.706 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:59.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:59.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:59.808 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:59.831 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:59.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:59.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:09:59.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:09:59.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:09:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:09:59.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:00.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:00.081 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:00.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:00.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:00.205 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:00.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:00.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:00.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:00 smithi052 ceph-mon[36682]: 6.2 scrub starts 2024-09-13T13:10:00.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:00 smithi052 ceph-mon[36682]: 6.2 scrub ok 2024-09-13T13:10:00.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:00 smithi052 ceph-mon[36682]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-13T13:10:00.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:00 smithi052 ceph-mon[36682]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-13T13:10:00.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:00 smithi052 ceph-mon[29471]: 6.2 scrub starts 2024-09-13T13:10:00.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:00 smithi052 ceph-mon[29471]: 6.2 scrub ok 2024-09-13T13:10:00.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:00 smithi052 ceph-mon[29471]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-13T13:10:00.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:00 smithi052 ceph-mon[29471]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-13T13:10:00.307 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:00.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:00 smithi167 ceph-mon[33178]: 6.2 scrub starts 2024-09-13T13:10:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:00 smithi167 ceph-mon[33178]: 6.2 scrub ok 2024-09-13T13:10:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:00 smithi167 ceph-mon[33178]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-13T13:10:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:00 smithi167 ceph-mon[33178]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-13T13:10:00.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:00.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:00.415 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T13:10:00.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:00.452 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:00.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:00.526 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:10:00.554 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:00.591 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:00.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:00.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:00.693 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:00.716 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:00.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:00.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:00.818 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:00.841 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:00.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:00.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:00.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:00.966 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:00.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:01.067 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:01.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:01.107 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:01.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:01.121 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:01.208 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:01.239 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:01.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:01 smithi052 ceph-mon[36682]: pgmap v423: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 787 op/s 2024-09-13T13:10:01.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:01 smithi052 ceph-mon[36682]: 5.1d scrub starts 2024-09-13T13:10:01.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:01 smithi052 ceph-mon[36682]: 5.1d scrub ok 2024-09-13T13:10:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:01 smithi052 ceph-mon[29471]: pgmap v423: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 787 op/s 2024-09-13T13:10:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:01 smithi052 ceph-mon[29471]: 5.1d scrub starts 2024-09-13T13:10:01.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:01 smithi052 ceph-mon[29471]: 5.1d scrub ok 2024-09-13T13:10:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:01.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:01.277 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:10:01.284 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:10:01.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:01 smithi167 ceph-mon[33178]: pgmap v423: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 787 op/s 2024-09-13T13:10:01.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:01 smithi167 ceph-mon[33178]: 5.1d scrub starts 2024-09-13T13:10:01.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:01 smithi167 ceph-mon[33178]: 5.1d scrub ok 2024-09-13T13:10:01.342 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:01.365 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:01.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:01.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:01.467 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:01.489 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:01.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:01.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:01.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:01.614 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:01.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:01.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:01.716 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:01.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:01.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:01.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:01.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:01.861 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:01.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:01.963 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:01.986 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:01.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:02.088 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:02.150 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:02.151 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-09-13T13:09:59.787318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1376,"num_read_kb":1233,"num_write":256403,"num_write_kb":8188938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86114,"ondisk_log_size":86114,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2165552,"kb_used_data":7812,"kb_used_omap":15,"kb_used_meta":2157552,"kb_avail":747795664,"statfs":{"total":767960285184,"available":765742759936,"internally_reserved":0,"allocated":7999488,"data_stored":4907687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":2209334041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-399310872,"num_objects":-6094,"num_object_clones":0,"num_object_copies":-18282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6094,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6094,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.754236"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":386,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.053989+0000","last_change":"2024-09-13T13:06:44.304525+0000","last_active":"2024-09-13T13:09:57.053989+0000","last_peered":"2024-09-13T13:09:57.053989+0000","last_clean":"2024-09-13T13:09:57.053989+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:09:57.053989+0000","last_undegraded":"2024-09-13T13:09:57.053989+0000","last_fullsized":"2024-09-13T13:09:57.053989+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:44.304444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.304444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.250196+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":410,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.056882+0000","last_change":"2024-09-13T13:09:06.567774+0000","last_active":"2024-09-13T13:09:57.056882+0000","last_peered":"2024-09-13T13:09:57.056882+0000","last_clean":"2024-09-13T13:09:57.056882+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:09:57.056882+0000","last_undegraded":"2024-09-13T13:09:57.056882+0000","last_fullsized":"2024-09-13T13:09:57.056882+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:06.567719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:06.567719+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:30.867489+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":379,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429725+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:09:57.429725+0000","last_peered":"2024-09-13T13:09:57.429725+0000","last_clean":"2024-09-13T13:09:57.429725+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:09:57.429725+0000","last_undegraded":"2024-09-13T13:09:57.429725+0000","last_fullsized":"2024-09-13T13:09:57.429725+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:08.843909+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":375,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057112+0000","last_change":"2024-09-13T13:09:42.788562+0000","last_active":"2024-09-13T13:09:57.057112+0000","last_peered":"2024-09-13T13:09:57.057112+0000","last_clean":"2024-09-13T13:09:57.057112+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:09:57.057112+0000","last_undegraded":"2024-09-13T13:09:57.057112+0000","last_fullsized":"2024-09-13T13:09:57.057112+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:42.787492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:42.787492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:47.723359+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":393,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.051973+0000","last_change":"2024-09-13T13:08:55.504675+0000","last_active":"2024-09-13T13:09:57.051973+0000","last_peered":"2024-09-13T13:09:57.051973+0000","last_clean":"2024-09-13T13:09:57.051973+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:09:57.051973+0000","last_undegraded":"2024-09-13T13:09:57.051973+0000","last_fullsized":"2024-09-13T13:09:57.051973+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.504475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.504475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.958611+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":398,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.054865+0000","last_change":"2024-09-13T13:09:31.491481+0000","last_active":"2024-09-13T13:09:57.054865+0000","last_peered":"2024-09-13T13:09:57.054865+0000","last_clean":"2024-09-13T13:09:57.054865+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:09:57.054865+0000","last_undegraded":"2024-09-13T13:09:57.054865+0000","last_fullsized":"2024-09-13T13:09:57.054865+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:31.491404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:31.491404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.774257+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":423,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052319+0000","last_change":"2024-09-13T13:08:53.510436+0000","last_active":"2024-09-13T13:09:57.052319+0000","last_peered":"2024-09-13T13:09:57.052319+0000","last_clean":"2024-09-13T13:09:57.052319+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:09:57.052319+0000","last_undegraded":"2024-09-13T13:09:57.052319+0000","last_fullsized":"2024-09-13T13:09:57.052319+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:08:53.510335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.510335+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.247321+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":398,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.056864+0000","last_change":"2024-09-13T13:08:55.513716+0000","last_active":"2024-09-13T13:09:57.056864+0000","last_peered":"2024-09-13T13:09:57.056864+0000","last_clean":"2024-09-13T13:09:57.056864+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:09:57.056864+0000","last_undegraded":"2024-09-13T13:09:57.056864+0000","last_fullsized":"2024-09-13T13:09:57.056864+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.513667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.513667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.654393+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":419,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430901+0000","last_change":"2024-09-13T13:08:59.654555+0000","last_active":"2024-09-13T13:09:57.430901+0000","last_peered":"2024-09-13T13:09:57.430901+0000","last_clean":"2024-09-13T13:09:57.430901+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:09:57.430901+0000","last_undegraded":"2024-09-13T13:09:57.430901+0000","last_fullsized":"2024-09-13T13:09:57.430901+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:08:59.654388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.654388+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.560420+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":415,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057452+0000","last_change":"2024-09-13T13:09:00.426054+0000","last_active":"2024-09-13T13:09:57.057452+0000","last_peered":"2024-09-13T13:09:57.057452+0000","last_clean":"2024-09-13T13:09:57.057452+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:09:57.057452+0000","last_undegraded":"2024-09-13T13:09:57.057452+0000","last_fullsized":"2024-09-13T13:09:57.057452+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:00.425965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:00.425965+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:24.968404+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":386,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430101+0000","last_change":"2024-09-13T13:06:50.665256+0000","last_active":"2024-09-13T13:09:57.430101+0000","last_peered":"2024-09-13T13:09:57.430101+0000","last_clean":"2024-09-13T13:09:57.430101+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:09:57.430101+0000","last_undegraded":"2024-09-13T13:09:57.430101+0000","last_fullsized":"2024-09-13T13:09:57.430101+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:50.665191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.665191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.666378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":381,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052206+0000","last_change":"2024-09-13T13:08:49.280146+0000","last_active":"2024-09-13T13:09:57.052206+0000","last_peered":"2024-09-13T13:09:57.052206+0000","last_clean":"2024-09-13T13:09:57.052206+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:09:57.052206+0000","last_undegraded":"2024-09-13T13:09:57.052206+0000","last_fullsized":"2024-09-13T13:09:57.052206+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:49.280073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:49.280073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.252935+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":402,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429716+0000","last_change":"2024-09-13T13:07:57.425510+0000","last_active":"2024-09-13T13:09:57.429716+0000","last_peered":"2024-09-13T13:09:57.429716+0000","last_clean":"2024-09-13T13:09:57.429716+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:09:57.429716+0000","last_undegraded":"2024-09-13T13:09:57.429716+0000","last_fullsized":"2024-09-13T13:09:57.429716+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.425436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.425436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:24.739469+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":393,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057041+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:09:57.057041+0000","last_peered":"2024-09-13T13:09:57.057041+0000","last_clean":"2024-09-13T13:09:57.057041+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:09:57.057041+0000","last_undegraded":"2024-09-13T13:09:57.057041+0000","last_fullsized":"2024-09-13T13:09:57.057041+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:46.784902+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":386,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429587+0000","last_change":"2024-09-13T13:07:50.530956+0000","last_active":"2024-09-13T13:09:57.429587+0000","last_peered":"2024-09-13T13:09:57.429587+0000","last_clean":"2024-09-13T13:09:57.429587+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:09:57.429587+0000","last_undegraded":"2024-09-13T13:09:57.429587+0000","last_fullsized":"2024-09-13T13:09:57.429587+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:50.530904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:50.530904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.091714+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":366,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428479+0000","last_change":"2024-09-13T13:07:19.641119+0000","last_active":"2024-09-13T13:09:57.428479+0000","last_peered":"2024-09-13T13:09:57.428479+0000","last_clean":"2024-09-13T13:09:57.428479+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:09:57.428479+0000","last_undegraded":"2024-09-13T13:09:57.428479+0000","last_fullsized":"2024-09-13T13:09:57.428479+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:19.640697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:19.640697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:50.864276+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":396,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057849+0000","last_change":"2024-09-13T13:07:46.772159+0000","last_active":"2024-09-13T13:09:57.057849+0000","last_peered":"2024-09-13T13:09:57.057849+0000","last_clean":"2024-09-13T13:09:57.057849+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:09:57.057849+0000","last_undegraded":"2024-09-13T13:09:57.057849+0000","last_fullsized":"2024-09-13T13:09:57.057849+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:07:46.772111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:46.772111+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:22.644441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":393,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.053857+0000","last_change":"2024-09-13T13:08:19.181017+0000","last_active":"2024-09-13T13:09:57.053857+0000","last_peered":"2024-09-13T13:09:57.053857+0000","last_clean":"2024-09-13T13:09:57.053857+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:09:57.053857+0000","last_undegraded":"2024-09-13T13:09:57.053857+0000","last_fullsized":"2024-09-13T13:09:57.053857+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:19.180948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.180948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:54.211909+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":424,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429280+0000","last_change":"2024-09-13T13:06:44.532305+0000","last_active":"2024-09-13T13:09:57.429280+0000","last_peered":"2024-09-13T13:09:57.429280+0000","last_clean":"2024-09-13T13:09:57.429280+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:09:57.429280+0000","last_undegraded":"2024-09-13T13:09:57.429280+0000","last_fullsized":"2024-09-13T13:09:57.429280+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:06:44.532186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.532186+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.375109+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":387,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429667+0000","last_change":"2024-09-13T13:08:18.639588+0000","last_active":"2024-09-13T13:09:57.429667+0000","last_peered":"2024-09-13T13:09:57.429667+0000","last_clean":"2024-09-13T13:09:57.429667+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:09:57.429667+0000","last_undegraded":"2024-09-13T13:09:57.429667+0000","last_fullsized":"2024-09-13T13:09:57.429667+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:18.639517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:18.639517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":394,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.054951+0000","last_change":"2024-09-13T13:09:29.517189+0000","last_active":"2024-09-13T13:09:57.054951+0000","last_peered":"2024-09-13T13:09:57.054951+0000","last_clean":"2024-09-13T13:09:57.054951+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:09:57.054951+0000","last_undegraded":"2024-09-13T13:09:57.054951+0000","last_fullsized":"2024-09-13T13:09:57.054951+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:29.517034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:29.517034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:34.784832+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":395,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057572+0000","last_change":"2024-09-13T13:09:28.614216+0000","last_active":"2024-09-13T13:09:57.057572+0000","last_peered":"2024-09-13T13:09:57.057572+0000","last_clean":"2024-09-13T13:09:57.057572+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:09:57.057572+0000","last_undegraded":"2024-09-13T13:09:57.057572+0000","last_fullsized":"2024-09-13T13:09:57.057572+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:28.614027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.614027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:31.496891+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":405,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.054654+0000","last_change":"2024-09-13T13:09:42.310274+0000","last_active":"2024-09-13T13:09:57.054654+0000","last_peered":"2024-09-13T13:09:57.054654+0000","last_clean":"2024-09-13T13:09:57.054654+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:09:57.054654+0000","last_undegraded":"2024-09-13T13:09:57.054654+0000","last_fullsized":"2024-09-13T13:09:57.054654+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:09:42.310185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:42.310185+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:09.881670+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":387,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057403+0000","last_change":"2024-09-13T13:07:12.879813+0000","last_active":"2024-09-13T13:09:57.057403+0000","last_peered":"2024-09-13T13:09:57.057403+0000","last_clean":"2024-09-13T13:09:57.057403+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:09:57.057403+0000","last_undegraded":"2024-09-13T13:09:57.057403+0000","last_fullsized":"2024-09-13T13:09:57.057403+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.879727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.879727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.866101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":391,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430741+0000","last_change":"2024-09-13T13:08:59.165835+0000","last_active":"2024-09-13T13:09:57.430741+0000","last_peered":"2024-09-13T13:09:57.430741+0000","last_clean":"2024-09-13T13:09:57.430741+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:09:57.430741+0000","last_undegraded":"2024-09-13T13:09:57.430741+0000","last_fullsized":"2024-09-13T13:09:57.430741+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:08:59.165754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.165754+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.048878+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":388,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429955+0000","last_change":"2024-09-13T13:07:54.442276+0000","last_active":"2024-09-13T13:09:57.429955+0000","last_peered":"2024-09-13T13:09:57.429955+0000","last_clean":"2024-09-13T13:09:57.429955+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:09:57.429955+0000","last_undegraded":"2024-09-13T13:09:57.429955+0000","last_fullsized":"2024-09-13T13:09:57.429955+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:07:54.439269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:54.439269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.738173+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":413,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.053908+0000","last_change":"2024-09-13T13:09:28.434007+0000","last_active":"2024-09-13T13:09:57.053908+0000","last_peered":"2024-09-13T13:09:57.053908+0000","last_clean":"2024-09-13T13:09:57.053908+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:09:57.053908+0000","last_undegraded":"2024-09-13T13:09:57.053908+0000","last_fullsized":"2024-09-13T13:09:57.053908+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:28.433899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.433899+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.061308+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":447,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T13:09:52.031436+0000","last_change":"2024-09-13T13:09:39.831623+0000","last_active":"2024-09-13T13:09:52.031436+0000","last_peered":"2024-09-13T13:09:52.031436+0000","last_clean":"2024-09-13T13:09:52.031436+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:09:52.031436+0000","last_undegraded":"2024-09-13T13:09:52.031436+0000","last_fullsized":"2024-09-13T13:09:52.031436+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:09:39.831547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:39.831547+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.629532+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"236'16138","reported_seq":16311,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429511+0000","last_change":"2024-09-13T13:09:34.404950+0000","last_active":"2024-09-13T13:09:57.429511+0000","last_peered":"2024-09-13T13:09:57.429511+0000","last_clean":"2024-09-13T13:09:57.429511+0000","last_became_active":"2024-09-13T13:06:21.814002+0000","last_became_peered":"2024-09-13T13:06:21.814002+0000","last_unstale":"2024-09-13T13:09:57.429511+0000","last_undegraded":"2024-09-13T13:09:57.429511+0000","last_fullsized":"2024-09-13T13:09:57.429511+0000","mapping_epoch":169,"log_start":"217'11200","ondisk_log_start":"217'11200","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'14788","last_scrub_stamp":"2024-09-13T13:09:34.404844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:09:34.404844+0000","objects_scrubbed":1367,"log_size":4938,"log_dups_size":0,"ondisk_log_size":4938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:36.235023+0000","scrub_duration":561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16138,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":416,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429950+0000","last_change":"2024-09-13T13:07:15.243958+0000","last_active":"2024-09-13T13:09:57.429950+0000","last_peered":"2024-09-13T13:09:57.429950+0000","last_clean":"2024-09-13T13:09:57.429950+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:09:57.429950+0000","last_undegraded":"2024-09-13T13:09:57.429950+0000","last_fullsized":"2024-09-13T13:09:57.429950+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:07:15.243715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:15.243715+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:48.866586+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":403,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052152+0000","last_change":"2024-09-13T13:08:19.560359+0000","last_active":"2024-09-13T13:09:57.052152+0000","last_peered":"2024-09-13T13:09:57.052152+0000","last_clean":"2024-09-13T13:09:57.052152+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:09:57.052152+0000","last_undegraded":"2024-09-13T13:09:57.052152+0000","last_fullsized":"2024-09-13T13:09:57.052152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:19.560157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.560157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":503,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057913+0000","last_change":"2024-09-13T13:08:53.508763+0000","last_active":"2024-09-13T13:09:57.057913+0000","last_peered":"2024-09-13T13:09:57.057913+0000","last_clean":"2024-09-13T13:09:57.057913+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:09:57.057913+0000","last_undegraded":"2024-09-13T13:09:57.057913+0000","last_fullsized":"2024-09-13T13:09:57.057913+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:08:53.508690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.508690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:06.126931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":382,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430392+0000","last_change":"2024-09-13T13:08:05.356744+0000","last_active":"2024-09-13T13:09:57.430392+0000","last_peered":"2024-09-13T13:09:57.430392+0000","last_clean":"2024-09-13T13:09:57.430392+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:09:57.430392+0000","last_undegraded":"2024-09-13T13:09:57.430392+0000","last_fullsized":"2024-09-13T13:09:57.430392+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:05.356688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:05.356688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"236'15808","reported_seq":15957,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052958+0000","last_change":"2024-09-13T13:06:22.000536+0000","last_active":"2024-09-13T13:09:57.052958+0000","last_peered":"2024-09-13T13:09:57.052958+0000","last_clean":"2024-09-13T13:09:57.052958+0000","last_became_active":"2024-09-13T13:06:22.000244+0000","last_became_peered":"2024-09-13T13:06:22.000244+0000","last_unstale":"2024-09-13T13:09:57.052958+0000","last_undegraded":"2024-09-13T13:09:57.052958+0000","last_fullsized":"2024-09-13T13:09:57.052958+0000","mapping_epoch":169,"log_start":"213'10000","ondisk_log_start":"213'10000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5808,"log_dups_size":0,"ondisk_log_size":5808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:50.274083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15808,"num_write_kb":505856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":407,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052016+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:09:57.052016+0000","last_peered":"2024-09-13T13:09:57.052016+0000","last_clean":"2024-09-13T13:09:57.052016+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:09:57.052016+0000","last_undegraded":"2024-09-13T13:09:57.052016+0000","last_fullsized":"2024-09-13T13:09:57.052016+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.546922+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":393,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.055154+0000","last_change":"2024-09-13T13:06:44.785235+0000","last_active":"2024-09-13T13:09:57.055154+0000","last_peered":"2024-09-13T13:09:57.055154+0000","last_clean":"2024-09-13T13:09:57.055154+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:09:57.055154+0000","last_undegraded":"2024-09-13T13:09:57.055154+0000","last_fullsized":"2024-09-13T13:09:57.055154+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:06:44.785017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.785017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.624176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":380,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.056882+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:09:57.056882+0000","last_peered":"2024-09-13T13:09:57.056882+0000","last_clean":"2024-09-13T13:09:57.056882+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:09:57.056882+0000","last_undegraded":"2024-09-13T13:09:57.056882+0000","last_fullsized":"2024-09-13T13:09:57.056882+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.668051+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":384,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057366+0000","last_change":"2024-09-13T13:08:22.683148+0000","last_active":"2024-09-13T13:09:57.057366+0000","last_peered":"2024-09-13T13:09:57.057366+0000","last_clean":"2024-09-13T13:09:57.057366+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:09:57.057366+0000","last_undegraded":"2024-09-13T13:09:57.057366+0000","last_fullsized":"2024-09-13T13:09:57.057366+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:22.683019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:22.683019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"236'15934","reported_seq":16085,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430285+0000","last_change":"2024-09-13T13:06:22.000928+0000","last_active":"2024-09-13T13:09:57.430285+0000","last_peered":"2024-09-13T13:09:57.430285+0000","last_clean":"2024-09-13T13:09:57.430285+0000","last_became_active":"2024-09-13T13:06:22.000610+0000","last_became_peered":"2024-09-13T13:06:22.000610+0000","last_unstale":"2024-09-13T13:09:57.430285+0000","last_undegraded":"2024-09-13T13:09:57.430285+0000","last_fullsized":"2024-09-13T13:09:57.430285+0000","mapping_epoch":169,"log_start":"215'10500","ondisk_log_start":"215'10500","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5434,"log_dups_size":0,"ondisk_log_size":5434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:18.686758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15934,"num_write_kb":509825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":432,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.768541+0000","last_change":"2024-09-13T13:09:57.768541+0000","last_active":"2024-09-13T13:09:57.768541+0000","last_peered":"2024-09-13T13:09:57.768541+0000","last_clean":"2024-09-13T13:09:57.768541+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:09:57.768541+0000","last_undegraded":"2024-09-13T13:09:57.768541+0000","last_fullsized":"2024-09-13T13:09:57.768541+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:09:57.768472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:57.768472+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:07.273441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":379,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.054355+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:09:57.054355+0000","last_peered":"2024-09-13T13:09:57.054355+0000","last_clean":"2024-09-13T13:09:57.054355+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:09:57.054355+0000","last_undegraded":"2024-09-13T13:09:57.054355+0000","last_fullsized":"2024-09-13T13:09:57.054355+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.235292+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":495,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.051959+0000","last_change":"2024-09-13T13:09:28.477414+0000","last_active":"2024-09-13T13:09:57.051959+0000","last_peered":"2024-09-13T13:09:57.051959+0000","last_clean":"2024-09-13T13:09:57.051959+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:09:57.051959+0000","last_undegraded":"2024-09-13T13:09:57.051959+0000","last_fullsized":"2024-09-13T13:09:57.051959+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:09:28.477338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.477338+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:42.900990+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":386,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429910+0000","last_change":"2024-09-13T13:06:09.230872+0000","last_active":"2024-09-13T13:09:57.429910+0000","last_peered":"2024-09-13T13:09:57.429910+0000","last_clean":"2024-09-13T13:09:57.429910+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:09:57.429910+0000","last_undegraded":"2024-09-13T13:09:57.429910+0000","last_fullsized":"2024-09-13T13:09:57.429910+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:09.230824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.230824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:44.042037+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"236'15880","reported_seq":16029,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.054369+0000","last_change":"2024-09-13T13:06:21.812511+0000","last_active":"2024-09-13T13:09:57.054369+0000","last_peered":"2024-09-13T13:09:57.054369+0000","last_clean":"2024-09-13T13:09:57.054369+0000","last_became_active":"2024-09-13T13:06:21.812226+0000","last_became_peered":"2024-09-13T13:06:21.812226+0000","last_unstale":"2024-09-13T13:09:57.054369+0000","last_undegraded":"2024-09-13T13:09:57.054369+0000","last_fullsized":"2024-09-13T13:09:57.054369+0000","mapping_epoch":169,"log_start":"214'10100","ondisk_log_start":"214'10100","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.258097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":508160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":407,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429627+0000","last_change":"2024-09-13T13:09:27.854186+0000","last_active":"2024-09-13T13:09:57.429627+0000","last_peered":"2024-09-13T13:09:57.429627+0000","last_clean":"2024-09-13T13:09:57.429627+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:09:57.429627+0000","last_undegraded":"2024-09-13T13:09:57.429627+0000","last_fullsized":"2024-09-13T13:09:57.429627+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:27.854114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:27.854114+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:39.809612+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":395,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430999+0000","last_change":"2024-09-13T13:08:17.298581+0000","last_active":"2024-09-13T13:09:57.430999+0000","last_peered":"2024-09-13T13:09:57.430999+0000","last_clean":"2024-09-13T13:09:57.430999+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:09:57.430999+0000","last_undegraded":"2024-09-13T13:09:57.430999+0000","last_fullsized":"2024-09-13T13:09:57.430999+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:17.298430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:17.298430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":500,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.053424+0000","last_change":"2024-09-13T13:07:57.028959+0000","last_active":"2024-09-13T13:09:57.053424+0000","last_peered":"2024-09-13T13:09:57.053424+0000","last_clean":"2024-09-13T13:09:57.053424+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:09:57.053424+0000","last_undegraded":"2024-09-13T13:09:57.053424+0000","last_fullsized":"2024-09-13T13:09:57.053424+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:07:57.028875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.028875+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.174262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":390,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.051668+0000","last_change":"2024-09-13T13:08:52.207903+0000","last_active":"2024-09-13T13:09:57.051668+0000","last_peered":"2024-09-13T13:09:57.051668+0000","last_clean":"2024-09-13T13:09:57.051668+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:09:57.051668+0000","last_undegraded":"2024-09-13T13:09:57.051668+0000","last_fullsized":"2024-09-13T13:09:57.051668+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:52.207841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:52.207841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.868681+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"236'15970","reported_seq":16172,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.055081+0000","last_change":"2024-09-13T13:08:51.116458+0000","last_active":"2024-09-13T13:09:57.055081+0000","last_peered":"2024-09-13T13:09:57.055081+0000","last_clean":"2024-09-13T13:09:57.055081+0000","last_became_active":"2024-09-13T13:06:21.812208+0000","last_became_peered":"2024-09-13T13:06:21.812208+0000","last_unstale":"2024-09-13T13:09:57.055081+0000","last_undegraded":"2024-09-13T13:09:57.055081+0000","last_fullsized":"2024-09-13T13:09:57.055081+0000","mapping_epoch":169,"log_start":"217'11100","ondisk_log_start":"217'11100","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'11587","last_scrub_stamp":"2024-09-13T13:08:51.116305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:08:51.116305+0000","objects_scrubbed":4435,"log_size":4870,"log_dups_size":0,"ondisk_log_size":4870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.050196+0000","scrub_duration":1647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15970,"num_write_kb":511040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":434,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.053525+0000","last_change":"2024-09-13T13:09:05.142515+0000","last_active":"2024-09-13T13:09:57.053525+0000","last_peered":"2024-09-13T13:09:57.053525+0000","last_clean":"2024-09-13T13:09:57.053525+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:09:57.053525+0000","last_undegraded":"2024-09-13T13:09:57.053525+0000","last_fullsized":"2024-09-13T13:09:57.053525+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:09:05.142427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.142427+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:06.191632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":380,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429326+0000","last_change":"2024-09-13T13:09:38.867734+0000","last_active":"2024-09-13T13:09:57.429326+0000","last_peered":"2024-09-13T13:09:57.429326+0000","last_clean":"2024-09-13T13:09:57.429326+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:09:57.429326+0000","last_undegraded":"2024-09-13T13:09:57.429326+0000","last_fullsized":"2024-09-13T13:09:57.429326+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.867688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.867688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:54.044585+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":344,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T13:09:52.030696+0000","last_change":"2024-09-13T13:09:38.854654+0000","last_active":"2024-09-13T13:09:52.030696+0000","last_peered":"2024-09-13T13:09:52.030696+0000","last_clean":"2024-09-13T13:09:52.030696+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:09:52.030696+0000","last_undegraded":"2024-09-13T13:09:52.030696+0000","last_fullsized":"2024-09-13T13:09:52.030696+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.854581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.854581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:50.922952+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":395,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429669+0000","last_change":"2024-09-13T13:06:43.969251+0000","last_active":"2024-09-13T13:09:57.429669+0000","last_peered":"2024-09-13T13:09:57.429669+0000","last_clean":"2024-09-13T13:09:57.429669+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:09:57.429669+0000","last_undegraded":"2024-09-13T13:09:57.429669+0000","last_fullsized":"2024-09-13T13:09:57.429669+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.969176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.969176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.710958+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"236'15778","reported_seq":15928,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428216+0000","last_change":"2024-09-13T13:06:22.000658+0000","last_active":"2024-09-13T13:09:57.428216+0000","last_peered":"2024-09-13T13:09:57.428216+0000","last_clean":"2024-09-13T13:09:57.428216+0000","last_became_active":"2024-09-13T13:06:22.000261+0000","last_became_peered":"2024-09-13T13:06:22.000261+0000","last_unstale":"2024-09-13T13:09:57.428216+0000","last_undegraded":"2024-09-13T13:09:57.428216+0000","last_fullsized":"2024-09-13T13:09:57.428216+0000","mapping_epoch":169,"log_start":"215'10400","ondisk_log_start":"215'10400","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5378,"log_dups_size":0,"ondisk_log_size":5378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.720130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15778,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":548,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429191+0000","last_change":"2024-09-13T13:07:25.438610+0000","last_active":"2024-09-13T13:09:57.429191+0000","last_peered":"2024-09-13T13:09:57.429191+0000","last_clean":"2024-09-13T13:09:57.429191+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:09:57.429191+0000","last_undegraded":"2024-09-13T13:09:57.429191+0000","last_fullsized":"2024-09-13T13:09:57.429191+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:07:25.438528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:25.438528+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:50.888420+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":407,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057612+0000","last_change":"2024-09-13T13:07:18.888103+0000","last_active":"2024-09-13T13:09:57.057612+0000","last_peered":"2024-09-13T13:09:57.057612+0000","last_clean":"2024-09-13T13:09:57.057612+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:09:57.057612+0000","last_undegraded":"2024-09-13T13:09:57.057612+0000","last_fullsized":"2024-09-13T13:09:57.057612+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:18.888060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:18.888060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:51.825948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":398,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428905+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:09:57.428905+0000","last_peered":"2024-09-13T13:09:57.428905+0000","last_clean":"2024-09-13T13:09:57.428905+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:09:57.428905+0000","last_undegraded":"2024-09-13T13:09:57.428905+0000","last_fullsized":"2024-09-13T13:09:57.428905+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.818550+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":386,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.053312+0000","last_change":"2024-09-13T13:09:40.277472+0000","last_active":"2024-09-13T13:09:57.053312+0000","last_peered":"2024-09-13T13:09:57.053312+0000","last_clean":"2024-09-13T13:09:57.053312+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:09:57.053312+0000","last_undegraded":"2024-09-13T13:09:57.053312+0000","last_fullsized":"2024-09-13T13:09:57.053312+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:40.277312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:40.277312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:56.382493+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":383,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.054700+0000","last_change":"2024-09-13T13:09:00.570589+0000","last_active":"2024-09-13T13:09:57.054700+0000","last_peered":"2024-09-13T13:09:57.054700+0000","last_clean":"2024-09-13T13:09:57.054700+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:09:57.054700+0000","last_undegraded":"2024-09-13T13:09:57.054700+0000","last_fullsized":"2024-09-13T13:09:57.054700+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:00.570518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:00.570518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.670665+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"236'15886","reported_seq":16089,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.051842+0000","last_change":"2024-09-13T13:08:52.266749+0000","last_active":"2024-09-13T13:09:57.051842+0000","last_peered":"2024-09-13T13:09:57.051842+0000","last_clean":"2024-09-13T13:09:57.051842+0000","last_became_active":"2024-09-13T13:06:21.812869+0000","last_became_peered":"2024-09-13T13:06:21.812869+0000","last_unstale":"2024-09-13T13:09:57.051842+0000","last_undegraded":"2024-09-13T13:09:57.051842+0000","last_fullsized":"2024-09-13T13:09:57.051842+0000","mapping_epoch":169,"log_start":"217'11000","ondisk_log_start":"217'11000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'11526","last_scrub_stamp":"2024-09-13T13:08:52.266659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:08:52.266659+0000","objects_scrubbed":4408,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.678811+0000","scrub_duration":1778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15886,"num_write_kb":508352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":411,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429990+0000","last_change":"2024-09-13T13:08:04.680025+0000","last_active":"2024-09-13T13:09:57.429990+0000","last_peered":"2024-09-13T13:09:57.429990+0000","last_clean":"2024-09-13T13:09:57.429990+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:09:57.429990+0000","last_undegraded":"2024-09-13T13:09:57.429990+0000","last_fullsized":"2024-09-13T13:09:57.429990+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:08:04.679766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:04.679766+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:22.837118+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":415,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429385+0000","last_change":"2024-09-13T13:08:56.771534+0000","last_active":"2024-09-13T13:09:57.429385+0000","last_peered":"2024-09-13T13:09:57.429385+0000","last_clean":"2024-09-13T13:09:57.429385+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:09:57.429385+0000","last_undegraded":"2024-09-13T13:09:57.429385+0000","last_fullsized":"2024-09-13T13:09:57.429385+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:56.771432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:56.771432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.588929+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":411,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428759+0000","last_change":"2024-09-13T13:08:19.038471+0000","last_active":"2024-09-13T13:09:57.428759+0000","last_peered":"2024-09-13T13:09:57.428759+0000","last_clean":"2024-09-13T13:09:57.428759+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:09:57.428759+0000","last_undegraded":"2024-09-13T13:09:57.428759+0000","last_fullsized":"2024-09-13T13:09:57.428759+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:08:19.038393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.038393+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:25.295684+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":372,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429252+0000","last_change":"2024-09-13T13:03:14.282866+0000","last_active":"2024-09-13T13:09:57.429252+0000","last_peered":"2024-09-13T13:09:57.429252+0000","last_clean":"2024-09-13T13:09:57.429252+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:09:57.429252+0000","last_undegraded":"2024-09-13T13:09:57.429252+0000","last_fullsized":"2024-09-13T13:09:57.429252+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:44.192025+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":394,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430651+0000","last_change":"2024-09-13T13:08:54.232191+0000","last_active":"2024-09-13T13:09:57.430651+0000","last_peered":"2024-09-13T13:09:57.430651+0000","last_clean":"2024-09-13T13:09:57.430651+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:09:57.430651+0000","last_undegraded":"2024-09-13T13:09:57.430651+0000","last_fullsized":"2024-09-13T13:09:57.430651+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:54.232130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:54.232130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.097713+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"236'15842","reported_seq":16072,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429246+0000","last_change":"2024-09-13T13:07:49.115654+0000","last_active":"2024-09-13T13:09:57.429246+0000","last_peered":"2024-09-13T13:09:57.429246+0000","last_clean":"2024-09-13T13:09:57.429246+0000","last_became_active":"2024-09-13T13:06:21.814772+0000","last_became_peered":"2024-09-13T13:06:21.814772+0000","last_unstale":"2024-09-13T13:09:57.429246+0000","last_undegraded":"2024-09-13T13:09:57.429246+0000","last_fullsized":"2024-09-13T13:09:57.429246+0000","mapping_epoch":169,"log_start":"217'10900","ondisk_log_start":"217'10900","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'7112","last_scrub_stamp":"2024-09-13T13:07:49.115526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:07:49.115526+0000","objects_scrubbed":7029,"log_size":4942,"log_dups_size":0,"ondisk_log_size":4942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.794726+0000","scrub_duration":2765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15842,"num_write_kb":506944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":408,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.054052+0000","last_change":"2024-09-13T13:08:21.219398+0000","last_active":"2024-09-13T13:09:57.054052+0000","last_peered":"2024-09-13T13:09:57.054052+0000","last_clean":"2024-09-13T13:09:57.054052+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:09:57.054052+0000","last_undegraded":"2024-09-13T13:09:57.054052+0000","last_fullsized":"2024-09-13T13:09:57.054052+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:08:21.219300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:21.219300+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:52.209890+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":394,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.053055+0000","last_change":"2024-09-13T13:08:02.152950+0000","last_active":"2024-09-13T13:09:57.053055+0000","last_peered":"2024-09-13T13:09:57.053055+0000","last_clean":"2024-09-13T13:09:57.053055+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:09:57.053055+0000","last_undegraded":"2024-09-13T13:09:57.053055+0000","last_fullsized":"2024-09-13T13:09:57.053055+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:02.152859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:02.152859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:25.356503+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":933,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428844+0000","last_change":"2024-09-13T13:08:49.264831+0000","last_active":"2024-09-13T13:09:57.428844+0000","last_peered":"2024-09-13T13:09:57.428844+0000","last_clean":"2024-09-13T13:09:57.428844+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:09:57.428844+0000","last_undegraded":"2024-09-13T13:09:57.428844+0000","last_fullsized":"2024-09-13T13:09:57.428844+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:08:49.264775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:49.264775+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.716838+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":529,"num_read_kb":524,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":371,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430020+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:09:57.430020+0000","last_peered":"2024-09-13T13:09:57.430020+0000","last_clean":"2024-09-13T13:09:57.430020+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:09:57.430020+0000","last_undegraded":"2024-09-13T13:09:57.430020+0000","last_fullsized":"2024-09-13T13:09:57.430020+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:47.867215+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":392,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428862+0000","last_change":"2024-09-13T13:08:55.738138+0000","last_active":"2024-09-13T13:09:57.428862+0000","last_peered":"2024-09-13T13:09:57.428862+0000","last_clean":"2024-09-13T13:09:57.428862+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:09:57.428862+0000","last_undegraded":"2024-09-13T13:09:57.428862+0000","last_fullsized":"2024-09-13T13:09:57.428862+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.738069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.738069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:25.032322+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"236'15812","reported_seq":15979,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T13:09:54.122972+0000","last_change":"2024-09-13T13:09:38.386284+0000","last_active":"2024-09-13T13:09:54.122972+0000","last_peered":"2024-09-13T13:09:54.122972+0000","last_clean":"2024-09-13T13:09:54.122972+0000","last_became_active":"2024-09-13T13:06:21.814144+0000","last_became_peered":"2024-09-13T13:06:21.814144+0000","last_unstale":"2024-09-13T13:09:54.122972+0000","last_undegraded":"2024-09-13T13:09:54.122972+0000","last_fullsized":"2024-09-13T13:09:54.122972+0000","mapping_epoch":169,"log_start":"211'9500","ondisk_log_start":"211'9500","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'14664","last_scrub_stamp":"2024-09-13T13:09:38.386024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.386024+0000","objects_scrubbed":1158,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:01.089959+0000","scrub_duration":546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15812,"num_write_kb":505984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":375,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T13:09:52.031188+0000","last_change":"2024-09-13T13:06:51.193749+0000","last_active":"2024-09-13T13:09:52.031188+0000","last_peered":"2024-09-13T13:09:52.031188+0000","last_clean":"2024-09-13T13:09:52.031188+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:09:52.031188+0000","last_undegraded":"2024-09-13T13:09:52.031188+0000","last_fullsized":"2024-09-13T13:09:52.031188+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'18","last_scrub_stamp":"2024-09-13T13:06:51.193678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.193678+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.989653+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":398,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052570+0000","last_change":"2024-09-13T13:06:10.715371+0000","last_active":"2024-09-13T13:09:57.052570+0000","last_peered":"2024-09-13T13:09:57.052570+0000","last_clean":"2024-09-13T13:09:57.052570+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:09:57.052570+0000","last_undegraded":"2024-09-13T13:09:57.052570+0000","last_fullsized":"2024-09-13T13:09:57.052570+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:10.715313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.715313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.220475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":443,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429076+0000","last_change":"2024-09-13T13:06:39.564599+0000","last_active":"2024-09-13T13:09:57.429076+0000","last_peered":"2024-09-13T13:09:57.429076+0000","last_clean":"2024-09-13T13:09:57.429076+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:09:57.429076+0000","last_undegraded":"2024-09-13T13:09:57.429076+0000","last_fullsized":"2024-09-13T13:09:57.429076+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:06:39.564549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:06:39.564549+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:15.102832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":395,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430880+0000","last_change":"2024-09-13T13:08:25.500731+0000","last_active":"2024-09-13T13:09:57.430880+0000","last_peered":"2024-09-13T13:09:57.430880+0000","last_clean":"2024-09-13T13:09:57.430880+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:09:57.430880+0000","last_undegraded":"2024-09-13T13:09:57.430880+0000","last_fullsized":"2024-09-13T13:09:57.430880+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:25.500660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:25.500660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.959857+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":391,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429830+0000","last_change":"2024-09-13T13:08:20.287277+0000","last_active":"2024-09-13T13:09:57.429830+0000","last_peered":"2024-09-13T13:09:57.429830+0000","last_clean":"2024-09-13T13:09:57.429830+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:09:57.429830+0000","last_undegraded":"2024-09-13T13:09:57.429830+0000","last_fullsized":"2024-09-13T13:09:57.429830+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:20.287215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:20.287215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"236'16020","reported_seq":16190,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.056916+0000","last_change":"2024-09-13T13:07:26.381412+0000","last_active":"2024-09-13T13:09:57.056916+0000","last_peered":"2024-09-13T13:09:57.056916+0000","last_clean":"2024-09-13T13:09:57.056916+0000","last_became_active":"2024-09-13T13:06:21.812061+0000","last_became_peered":"2024-09-13T13:06:21.812061+0000","last_unstale":"2024-09-13T13:09:57.056916+0000","last_undegraded":"2024-09-13T13:09:57.056916+0000","last_fullsized":"2024-09-13T13:09:57.056916+0000","mapping_epoch":169,"log_start":"217'11200","ondisk_log_start":"217'11200","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":1649,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.683639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16020,"num_write_kb":512640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":413,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.058013+0000","last_change":"2024-09-13T13:08:01.790332+0000","last_active":"2024-09-13T13:09:57.058013+0000","last_peered":"2024-09-13T13:09:57.058013+0000","last_clean":"2024-09-13T13:09:57.058013+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:09:57.058013+0000","last_undegraded":"2024-09-13T13:09:57.058013+0000","last_fullsized":"2024-09-13T13:09:57.058013+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:08:01.790280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:01.790280+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:24.650459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":396,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057298+0000","last_change":"2024-09-13T13:08:51.546064+0000","last_active":"2024-09-13T13:09:57.057298+0000","last_peered":"2024-09-13T13:09:57.057298+0000","last_clean":"2024-09-13T13:09:57.057298+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:09:57.057298+0000","last_undegraded":"2024-09-13T13:09:57.057298+0000","last_fullsized":"2024-09-13T13:09:57.057298+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:51.546003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:51.546003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:09:59.798244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":397,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429645+0000","last_change":"2024-09-13T13:07:23.602111+0000","last_active":"2024-09-13T13:09:57.429645+0000","last_peered":"2024-09-13T13:09:57.429645+0000","last_clean":"2024-09-13T13:09:57.429645+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:09:57.429645+0000","last_undegraded":"2024-09-13T13:09:57.429645+0000","last_fullsized":"2024-09-13T13:09:57.429645+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:23.602052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:23.602052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":385,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052758+0000","last_change":"2024-09-13T13:08:27.451600+0000","last_active":"2024-09-13T13:09:57.052758+0000","last_peered":"2024-09-13T13:09:57.052758+0000","last_clean":"2024-09-13T13:09:57.052758+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:09:57.052758+0000","last_undegraded":"2024-09-13T13:09:57.052758+0000","last_fullsized":"2024-09-13T13:09:57.052758+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:27.448636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:27.448636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"236'16126","reported_seq":16357,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429532+0000","last_change":"2024-09-13T13:07:52.135686+0000","last_active":"2024-09-13T13:09:57.429532+0000","last_peered":"2024-09-13T13:09:57.429532+0000","last_clean":"2024-09-13T13:09:57.429532+0000","last_became_active":"2024-09-13T13:06:21.811526+0000","last_became_peered":"2024-09-13T13:06:21.811526+0000","last_unstale":"2024-09-13T13:09:57.429532+0000","last_undegraded":"2024-09-13T13:09:57.429532+0000","last_fullsized":"2024-09-13T13:09:57.429532+0000","mapping_epoch":169,"log_start":"217'11200","ondisk_log_start":"217'11200","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'7389","last_scrub_stamp":"2024-09-13T13:07:52.135527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:07:52.135527+0000","objects_scrubbed":7297,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.778806+0000","scrub_duration":2761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16126,"num_write_kb":516032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":411,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057931+0000","last_change":"2024-09-13T13:05:34.166464+0000","last_active":"2024-09-13T13:09:57.057931+0000","last_peered":"2024-09-13T13:09:57.057931+0000","last_clean":"2024-09-13T13:09:57.057931+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:09:57.057931+0000","last_undegraded":"2024-09-13T13:09:57.057931+0000","last_fullsized":"2024-09-13T13:09:57.057931+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.686209+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":368,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T13:09:52.030868+0000","last_change":"2024-09-13T13:09:05.049507+0000","last_active":"2024-09-13T13:09:52.030868+0000","last_peered":"2024-09-13T13:09:52.030868+0000","last_clean":"2024-09-13T13:09:52.030868+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:09:52.030868+0000","last_undegraded":"2024-09-13T13:09:52.030868+0000","last_fullsized":"2024-09-13T13:09:52.030868+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:05.049446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.049446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.951824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":379,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428979+0000","last_change":"2024-09-13T13:08:53.747370+0000","last_active":"2024-09-13T13:09:57.428979+0000","last_peered":"2024-09-13T13:09:57.428979+0000","last_clean":"2024-09-13T13:09:57.428979+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:09:57.428979+0000","last_undegraded":"2024-09-13T13:09:57.428979+0000","last_fullsized":"2024-09-13T13:09:57.428979+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.747302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.747302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.330071+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":383,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429115+0000","last_change":"2024-09-13T13:08:03.015302+0000","last_active":"2024-09-13T13:09:57.429115+0000","last_peered":"2024-09-13T13:09:57.429115+0000","last_clean":"2024-09-13T13:09:57.429115+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:09:57.429115+0000","last_undegraded":"2024-09-13T13:09:57.429115+0000","last_fullsized":"2024-09-13T13:09:57.429115+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:03.015233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.015233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:23.253063+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"236'15822","reported_seq":16055,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.051751+0000","last_change":"2024-09-13T13:07:54.357736+0000","last_active":"2024-09-13T13:09:57.051751+0000","last_peered":"2024-09-13T13:09:57.051751+0000","last_clean":"2024-09-13T13:09:57.051751+0000","last_became_active":"2024-09-13T13:06:21.812966+0000","last_became_peered":"2024-09-13T13:06:21.812966+0000","last_unstale":"2024-09-13T13:09:57.051751+0000","last_undegraded":"2024-09-13T13:09:57.051751+0000","last_fullsized":"2024-09-13T13:09:57.051751+0000","mapping_epoch":169,"log_start":"217'10900","ondisk_log_start":"217'10900","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'7426","last_scrub_stamp":"2024-09-13T13:07:54.357373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:07:54.357373+0000","objects_scrubbed":7324,"log_size":4922,"log_dups_size":0,"ondisk_log_size":4922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:51.223721+0000","scrub_duration":2702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15822,"num_write_kb":506304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":425,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057488+0000","last_change":"2024-09-13T13:09:05.619225+0000","last_active":"2024-09-13T13:09:57.057488+0000","last_peered":"2024-09-13T13:09:57.057488+0000","last_clean":"2024-09-13T13:09:57.057488+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:09:57.057488+0000","last_undegraded":"2024-09-13T13:09:57.057488+0000","last_fullsized":"2024-09-13T13:09:57.057488+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:05.619171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.619171+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:07.343034+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":386,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052859+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:09:57.052859+0000","last_peered":"2024-09-13T13:09:57.052859+0000","last_clean":"2024-09-13T13:09:57.052859+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:09:57.052859+0000","last_undegraded":"2024-09-13T13:09:57.052859+0000","last_fullsized":"2024-09-13T13:09:57.052859+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:15.617560+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":495,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429336+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:09:57.429336+0000","last_peered":"2024-09-13T13:09:57.429336+0000","last_clean":"2024-09-13T13:09:57.429336+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:09:57.429336+0000","last_undegraded":"2024-09-13T13:09:57.429336+0000","last_fullsized":"2024-09-13T13:09:57.429336+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.422585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":375,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057652+0000","last_change":"2024-09-13T13:09:37.812544+0000","last_active":"2024-09-13T13:09:57.057652+0000","last_peered":"2024-09-13T13:09:57.057652+0000","last_clean":"2024-09-13T13:09:57.057652+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:09:57.057652+0000","last_undegraded":"2024-09-13T13:09:57.057652+0000","last_fullsized":"2024-09-13T13:09:57.057652+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:37.812451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:37.812451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:00.130077+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"236'16286","reported_seq":16440,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429679+0000","last_change":"2024-09-13T13:09:42.955518+0000","last_active":"2024-09-13T13:09:57.429679+0000","last_peered":"2024-09-13T13:09:57.429679+0000","last_clean":"2024-09-13T13:09:57.429679+0000","last_became_active":"2024-09-13T13:06:21.814433+0000","last_became_peered":"2024-09-13T13:06:21.814433+0000","last_unstale":"2024-09-13T13:09:57.429679+0000","last_undegraded":"2024-09-13T13:09:57.429679+0000","last_fullsized":"2024-09-13T13:09:57.429679+0000","mapping_epoch":169,"log_start":"217'11300","ondisk_log_start":"217'11300","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":99,"log_size":4986,"log_dups_size":0,"ondisk_log_size":4986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.924104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16286,"num_write_kb":521152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":389,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.053513+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:09:57.053513+0000","last_peered":"2024-09-13T13:09:57.053513+0000","last_clean":"2024-09-13T13:09:57.053513+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:09:57.053513+0000","last_undegraded":"2024-09-13T13:09:57.053513+0000","last_fullsized":"2024-09-13T13:09:57.053513+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:13.254641+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":413,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429905+0000","last_change":"2024-09-13T13:09:06.668526+0000","last_active":"2024-09-13T13:09:57.429905+0000","last_peered":"2024-09-13T13:09:57.429905+0000","last_clean":"2024-09-13T13:09:57.429905+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:09:57.429905+0000","last_undegraded":"2024-09-13T13:09:57.429905+0000","last_fullsized":"2024-09-13T13:09:57.429905+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:06.668478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:06.668478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.164862+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":396,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428960+0000","last_change":"2024-09-13T13:07:11.400066+0000","last_active":"2024-09-13T13:09:57.428960+0000","last_peered":"2024-09-13T13:09:57.428960+0000","last_clean":"2024-09-13T13:09:57.428960+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:09:57.428960+0000","last_undegraded":"2024-09-13T13:09:57.428960+0000","last_fullsized":"2024-09-13T13:09:57.428960+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.399840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.399840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:45.151594+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":374,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.051689+0000","last_change":"2024-09-13T13:09:29.473480+0000","last_active":"2024-09-13T13:09:57.051689+0000","last_peered":"2024-09-13T13:09:57.051689+0000","last_clean":"2024-09-13T13:09:57.051689+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:09:57.051689+0000","last_undegraded":"2024-09-13T13:09:57.051689+0000","last_fullsized":"2024-09-13T13:09:57.051689+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:29.473418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:29.473418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:55.955954+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"236'16236","reported_seq":16385,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.053621+0000","last_change":"2024-09-13T13:06:22.000224+0000","last_active":"2024-09-13T13:09:57.053621+0000","last_peered":"2024-09-13T13:09:57.053621+0000","last_clean":"2024-09-13T13:09:57.053621+0000","last_became_active":"2024-09-13T13:06:21.999889+0000","last_became_peered":"2024-09-13T13:06:21.999889+0000","last_unstale":"2024-09-13T13:09:57.053621+0000","last_undegraded":"2024-09-13T13:09:57.053621+0000","last_fullsized":"2024-09-13T13:09:57.053621+0000","mapping_epoch":169,"log_start":"214'10400","ondisk_log_start":"214'10400","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":0,"log_size":5836,"log_dups_size":0,"ondisk_log_size":5836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:46.296208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16236,"num_write_kb":519552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":379,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T13:09:52.030777+0000","last_change":"2024-09-13T13:09:32.849360+0000","last_active":"2024-09-13T13:09:52.030777+0000","last_peered":"2024-09-13T13:09:52.030777+0000","last_clean":"2024-09-13T13:09:52.030777+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:09:52.030777+0000","last_undegraded":"2024-09-13T13:09:52.030777+0000","last_fullsized":"2024-09-13T13:09:52.030777+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:09:32.849277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:32.849277+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:00.662384+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":405,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.431054+0000","last_change":"2024-09-13T13:07:48.579717+0000","last_active":"2024-09-13T13:09:57.431054+0000","last_peered":"2024-09-13T13:09:57.431054+0000","last_clean":"2024-09-13T13:09:57.431054+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:09:57.431054+0000","last_undegraded":"2024-09-13T13:09:57.431054+0000","last_fullsized":"2024-09-13T13:09:57.431054+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:48.579658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:48.579658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.184890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":395,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428780+0000","last_change":"2024-09-13T13:08:26.033729+0000","last_active":"2024-09-13T13:09:57.428780+0000","last_peered":"2024-09-13T13:09:57.428780+0000","last_clean":"2024-09-13T13:09:57.428780+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:09:57.428780+0000","last_undegraded":"2024-09-13T13:09:57.428780+0000","last_fullsized":"2024-09-13T13:09:57.428780+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:26.033663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:26.033663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:46.198638+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":383,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428970+0000","last_change":"2024-09-13T13:07:49.584576+0000","last_active":"2024-09-13T13:09:57.428970+0000","last_peered":"2024-09-13T13:09:57.428970+0000","last_clean":"2024-09-13T13:09:57.428970+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:09:57.428970+0000","last_undegraded":"2024-09-13T13:09:57.428970+0000","last_fullsized":"2024-09-13T13:09:57.428970+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:49.584505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:49.584505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.019080+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"236'16304","reported_seq":16460,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T13:09:54.114916+0000","last_change":"2024-09-13T13:09:42.953580+0000","last_active":"2024-09-13T13:09:54.114916+0000","last_peered":"2024-09-13T13:09:54.114916+0000","last_clean":"2024-09-13T13:09:54.114916+0000","last_became_active":"2024-09-13T13:06:22.001404+0000","last_became_peered":"2024-09-13T13:06:22.001404+0000","last_unstale":"2024-09-13T13:09:54.114916+0000","last_undegraded":"2024-09-13T13:09:54.114916+0000","last_fullsized":"2024-09-13T13:09:54.114916+0000","mapping_epoch":169,"log_start":"212'10000","ondisk_log_start":"212'10000","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:06:20.791662+0000","objects_scrubbed":699,"log_size":6304,"log_dups_size":0,"ondisk_log_size":6304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.928166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16304,"num_write_kb":521728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":407,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057147+0000","last_change":"2024-09-13T13:09:30.614842+0000","last_active":"2024-09-13T13:09:57.057147+0000","last_peered":"2024-09-13T13:09:57.057147+0000","last_clean":"2024-09-13T13:09:57.057147+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:09:57.057147+0000","last_undegraded":"2024-09-13T13:09:57.057147+0000","last_fullsized":"2024-09-13T13:09:57.057147+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:09:30.614748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.614748+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:58.916392+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":394,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.431392+0000","last_change":"2024-09-13T13:05:29.345863+0000","last_active":"2024-09-13T13:09:57.431392+0000","last_peered":"2024-09-13T13:09:57.431392+0000","last_clean":"2024-09-13T13:09:57.431392+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:09:57.431392+0000","last_undegraded":"2024-09-13T13:09:57.431392+0000","last_fullsized":"2024-09-13T13:09:57.431392+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.345796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.345796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:14.146169+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":380,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428567+0000","last_change":"2024-09-13T13:06:42.450669+0000","last_active":"2024-09-13T13:09:57.428567+0000","last_peered":"2024-09-13T13:09:57.428567+0000","last_clean":"2024-09-13T13:09:57.428567+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:09:57.428567+0000","last_undegraded":"2024-09-13T13:09:57.428567+0000","last_fullsized":"2024-09-13T13:09:57.428567+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:42.450598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:42.450598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:14.400638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":366,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052741+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:09:57.052741+0000","last_peered":"2024-09-13T13:09:57.052741+0000","last_clean":"2024-09-13T13:09:57.052741+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:09:57.052741+0000","last_undegraded":"2024-09-13T13:09:57.052741+0000","last_fullsized":"2024-09-13T13:09:57.052741+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:10.602171+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"236'16012","reported_seq":16250,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429390+0000","last_change":"2024-09-13T13:08:03.659578+0000","last_active":"2024-09-13T13:09:57.429390+0000","last_peered":"2024-09-13T13:09:57.429390+0000","last_clean":"2024-09-13T13:09:57.429390+0000","last_became_active":"2024-09-13T13:06:21.816096+0000","last_became_peered":"2024-09-13T13:06:21.816096+0000","last_unstale":"2024-09-13T13:09:57.429390+0000","last_undegraded":"2024-09-13T13:09:57.429390+0000","last_fullsized":"2024-09-13T13:09:57.429390+0000","mapping_epoch":169,"log_start":"215'10600","ondisk_log_start":"215'10600","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'8119","last_scrub_stamp":"2024-09-13T13:08:03.659418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:06:20.791662+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.659418+0000","objects_scrubbed":7960,"log_size":5412,"log_dups_size":0,"ondisk_log_size":5412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2349,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16012,"num_write_kb":512384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":427,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428352+0000","last_change":"2024-09-13T13:09:33.843654+0000","last_active":"2024-09-13T13:09:57.428352+0000","last_peered":"2024-09-13T13:09:57.428352+0000","last_clean":"2024-09-13T13:09:57.428352+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:09:57.428352+0000","last_undegraded":"2024-09-13T13:09:57.428352+0000","last_fullsized":"2024-09-13T13:09:57.428352+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:09:33.843579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:33.843579+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:40.459268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":406,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430820+0000","last_change":"2024-09-13T13:06:51.627764+0000","last_active":"2024-09-13T13:09:57.430820+0000","last_peered":"2024-09-13T13:09:57.430820+0000","last_clean":"2024-09-13T13:09:57.430820+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:09:57.430820+0000","last_undegraded":"2024-09-13T13:09:57.430820+0000","last_fullsized":"2024-09-13T13:09:57.430820+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:06:51.627691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.627691+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.064623+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":512,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429545+0000","last_change":"2024-09-13T13:09:41.018928+0000","last_active":"2024-09-13T13:09:57.429545+0000","last_peered":"2024-09-13T13:09:57.429545+0000","last_clean":"2024-09-13T13:09:57.429545+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:09:57.429545+0000","last_undegraded":"2024-09-13T13:09:57.429545+0000","last_fullsized":"2024-09-13T13:09:57.429545+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:09:41.018848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:41.018848+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:56.940353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":386,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428692+0000","last_change":"2024-09-13T13:09:30.836048+0000","last_active":"2024-09-13T13:09:57.428692+0000","last_peered":"2024-09-13T13:09:57.428692+0000","last_clean":"2024-09-13T13:09:57.428692+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:09:57.428692+0000","last_undegraded":"2024-09-13T13:09:57.428692+0000","last_fullsized":"2024-09-13T13:09:57.428692+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.836006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.836006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:38.200874+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":360,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T13:09:52.030620+0000","last_change":"2024-09-13T13:07:10.058616+0000","last_active":"2024-09-13T13:09:52.030620+0000","last_peered":"2024-09-13T13:09:52.030620+0000","last_clean":"2024-09-13T13:09:52.030620+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:09:52.030620+0000","last_undegraded":"2024-09-13T13:09:52.030620+0000","last_fullsized":"2024-09-13T13:09:52.030620+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:10.058555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.058555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:47.933640+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":391,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.055070+0000","last_change":"2024-09-13T13:09:37.358795+0000","last_active":"2024-09-13T13:09:57.055070+0000","last_peered":"2024-09-13T13:09:57.055070+0000","last_clean":"2024-09-13T13:09:57.055070+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:09:57.055070+0000","last_undegraded":"2024-09-13T13:09:57.055070+0000","last_fullsized":"2024-09-13T13:09:57.055070+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:09:37.358710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:37.358710+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:38.147999+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":394,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430001+0000","last_change":"2024-09-13T13:08:21.310761+0000","last_active":"2024-09-13T13:09:57.430001+0000","last_peered":"2024-09-13T13:09:57.430001+0000","last_clean":"2024-09-13T13:09:57.430001+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:09:57.430001+0000","last_undegraded":"2024-09-13T13:09:57.430001+0000","last_fullsized":"2024-09-13T13:09:57.430001+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:21.310681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:21.310681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":401,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430556+0000","last_change":"2024-09-13T13:09:30.959919+0000","last_active":"2024-09-13T13:09:57.430556+0000","last_peered":"2024-09-13T13:09:57.430556+0000","last_clean":"2024-09-13T13:09:57.430556+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:09:57.430556+0000","last_undegraded":"2024-09-13T13:09:57.430556+0000","last_fullsized":"2024-09-13T13:09:57.430556+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.959870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.959870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:45.945701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":397,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.054213+0000","last_change":"2024-09-13T13:09:34.426362+0000","last_active":"2024-09-13T13:09:57.054213+0000","last_peered":"2024-09-13T13:09:57.054213+0000","last_clean":"2024-09-13T13:09:57.054213+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:09:57.054213+0000","last_undegraded":"2024-09-13T13:09:57.054213+0000","last_fullsized":"2024-09-13T13:09:57.054213+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:34.426272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:34.426272+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.871047+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":389,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430155+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:09:57.430155+0000","last_peered":"2024-09-13T13:09:57.430155+0000","last_clean":"2024-09-13T13:09:57.430155+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:09:57.430155+0000","last_undegraded":"2024-09-13T13:09:57.430155+0000","last_fullsized":"2024-09-13T13:09:57.430155+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:43.057636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":378,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.054972+0000","last_change":"2024-09-13T13:09:40.333896+0000","last_active":"2024-09-13T13:09:57.054972+0000","last_peered":"2024-09-13T13:09:57.054972+0000","last_clean":"2024-09-13T13:09:57.054972+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:09:57.054972+0000","last_undegraded":"2024-09-13T13:09:57.054972+0000","last_fullsized":"2024-09-13T13:09:57.054972+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:40.333836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:40.333836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:03.786971+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":391,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428706+0000","last_change":"2024-09-13T13:09:07.378142+0000","last_active":"2024-09-13T13:09:57.428706+0000","last_peered":"2024-09-13T13:09:57.428706+0000","last_clean":"2024-09-13T13:09:57.428706+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:09:57.428706+0000","last_undegraded":"2024-09-13T13:09:57.428706+0000","last_fullsized":"2024-09-13T13:09:57.428706+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:07.378066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:07.378066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:33.759039+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":393,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430803+0000","last_change":"2024-09-13T13:09:48.990167+0000","last_active":"2024-09-13T13:09:57.430803+0000","last_peered":"2024-09-13T13:09:57.430803+0000","last_clean":"2024-09-13T13:09:57.430803+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:09:57.430803+0000","last_undegraded":"2024-09-13T13:09:57.430803+0000","last_fullsized":"2024-09-13T13:09:57.430803+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:12.930034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":385,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.053238+0000","last_change":"2024-09-13T13:06:41.308449+0000","last_active":"2024-09-13T13:09:57.053238+0000","last_peered":"2024-09-13T13:09:57.053238+0000","last_clean":"2024-09-13T13:09:57.053238+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:09:57.053238+0000","last_undegraded":"2024-09-13T13:09:57.053238+0000","last_fullsized":"2024-09-13T13:09:57.053238+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:41.308367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:41.308367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.223962+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":486,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.053035+0000","last_change":"2024-09-13T13:08:02.779173+0000","last_active":"2024-09-13T13:09:57.053035+0000","last_peered":"2024-09-13T13:09:57.053035+0000","last_clean":"2024-09-13T13:09:57.053035+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:09:57.053035+0000","last_undegraded":"2024-09-13T13:09:57.053035+0000","last_fullsized":"2024-09-13T13:09:57.053035+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:08:02.778891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:02.778891+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":383,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057861+0000","last_change":"2024-09-13T13:07:21.958114+0000","last_active":"2024-09-13T13:09:57.057861+0000","last_peered":"2024-09-13T13:09:57.057861+0000","last_clean":"2024-09-13T13:09:57.057861+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:09:57.057861+0000","last_undegraded":"2024-09-13T13:09:57.057861+0000","last_fullsized":"2024-09-13T13:09:57.057861+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:21.958030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:21.958030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:52.786745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":405,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.053999+0000","last_change":"2024-09-13T13:09:04.635198+0000","last_active":"2024-09-13T13:09:57.053999+0000","last_peered":"2024-09-13T13:09:57.053999+0000","last_clean":"2024-09-13T13:09:57.053999+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:09:57.053999+0000","last_undegraded":"2024-09-13T13:09:57.053999+0000","last_fullsized":"2024-09-13T13:09:57.053999+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:04.599040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:04.599040+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:30.955987+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":378,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430207+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:09:57.430207+0000","last_peered":"2024-09-13T13:09:57.430207+0000","last_clean":"2024-09-13T13:09:57.430207+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:09:57.430207+0000","last_undegraded":"2024-09-13T13:09:57.430207+0000","last_fullsized":"2024-09-13T13:09:57.430207+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:11.161132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":510,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430555+0000","last_change":"2024-09-13T13:07:08.781005+0000","last_active":"2024-09-13T13:09:57.430555+0000","last_peered":"2024-09-13T13:09:57.430555+0000","last_clean":"2024-09-13T13:09:57.430555+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:09:57.430555+0000","last_undegraded":"2024-09-13T13:09:57.430555+0000","last_fullsized":"2024-09-13T13:09:57.430555+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:07:08.780952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.780952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:54.891948+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":381,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.051731+0000","last_change":"2024-09-13T13:09:36.414394+0000","last_active":"2024-09-13T13:09:57.051731+0000","last_peered":"2024-09-13T13:09:57.051731+0000","last_clean":"2024-09-13T13:09:57.051731+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:09:57.051731+0000","last_undegraded":"2024-09-13T13:09:57.051731+0000","last_fullsized":"2024-09-13T13:09:57.051731+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:36.414322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:36.414322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:39.835131+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":413,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057246+0000","last_change":"2024-09-13T13:05:26.186899+0000","last_active":"2024-09-13T13:09:57.057246+0000","last_peered":"2024-09-13T13:09:57.057246+0000","last_clean":"2024-09-13T13:09:57.057246+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:09:57.057246+0000","last_undegraded":"2024-09-13T13:09:57.057246+0000","last_fullsized":"2024-09-13T13:09:57.057246+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:05:26.186830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.186830+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:14.738553+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":398,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.057626+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:09:57.057626+0000","last_peered":"2024-09-13T13:09:57.057626+0000","last_clean":"2024-09-13T13:09:57.057626+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:09:57.057626+0000","last_undegraded":"2024-09-13T13:09:57.057626+0000","last_fullsized":"2024-09-13T13:09:57.057626+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.696749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":387,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.429536+0000","last_change":"2024-09-13T13:09:05.882862+0000","last_active":"2024-09-13T13:09:57.429536+0000","last_peered":"2024-09-13T13:09:57.429536+0000","last_clean":"2024-09-13T13:09:57.429536+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:09:57.429536+0000","last_undegraded":"2024-09-13T13:09:57.429536+0000","last_fullsized":"2024-09-13T13:09:57.429536+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:05.882769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.882769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:09.443670+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":382,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052161+0000","last_change":"2024-09-13T13:08:59.585721+0000","last_active":"2024-09-13T13:09:57.052161+0000","last_peered":"2024-09-13T13:09:57.052161+0000","last_clean":"2024-09-13T13:09:57.052161+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:09:57.052161+0000","last_undegraded":"2024-09-13T13:09:57.052161+0000","last_fullsized":"2024-09-13T13:09:57.052161+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:59.585646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.585646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.171797+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":342,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428454+0000","last_change":"2024-09-13T13:05:35.506093+0000","last_active":"2024-09-13T13:09:57.428454+0000","last_peered":"2024-09-13T13:09:57.428454+0000","last_clean":"2024-09-13T13:09:57.428454+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:09:57.428454+0000","last_undegraded":"2024-09-13T13:09:57.428454+0000","last_fullsized":"2024-09-13T13:09:57.428454+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:05:35.506037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:35.506037+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:10.670571+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":394,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430963+0000","last_change":"2024-09-13T13:09:07.118545+0000","last_active":"2024-09-13T13:09:57.430963+0000","last_peered":"2024-09-13T13:09:57.430963+0000","last_clean":"2024-09-13T13:09:57.430963+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:09:57.430963+0000","last_undegraded":"2024-09-13T13:09:57.430963+0000","last_fullsized":"2024-09-13T13:09:57.430963+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:07.118467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:07.118467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.947652+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":378,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052694+0000","last_change":"2024-09-13T13:09:32.467362+0000","last_active":"2024-09-13T13:09:57.052694+0000","last_peered":"2024-09-13T13:09:57.052694+0000","last_clean":"2024-09-13T13:09:57.052694+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:09:57.052694+0000","last_undegraded":"2024-09-13T13:09:57.052694+0000","last_fullsized":"2024-09-13T13:09:57.052694+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:32.467151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:32.467151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:55.111614+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":381,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.051987+0000","last_change":"2024-09-13T13:08:56.227109+0000","last_active":"2024-09-13T13:09:57.051987+0000","last_peered":"2024-09-13T13:09:57.051987+0000","last_clean":"2024-09-13T13:09:57.051987+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:09:57.051987+0000","last_undegraded":"2024-09-13T13:09:57.051987+0000","last_fullsized":"2024-09-13T13:09:57.051987+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:56.227027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:56.227027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:14.111530+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":404,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052268+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:09:57.052268+0000","last_peered":"2024-09-13T13:09:57.052268+0000","last_clean":"2024-09-13T13:09:57.052268+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:09:57.052268+0000","last_undegraded":"2024-09-13T13:09:57.052268+0000","last_fullsized":"2024-09-13T13:09:57.052268+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:11.582994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":394,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.430108+0000","last_change":"2024-09-13T13:08:50.517587+0000","last_active":"2024-09-13T13:09:57.430108+0000","last_peered":"2024-09-13T13:09:57.430108+0000","last_clean":"2024-09-13T13:09:57.430108+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:09:57.430108+0000","last_undegraded":"2024-09-13T13:09:57.430108+0000","last_fullsized":"2024-09-13T13:09:57.430108+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:50.517526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:50.517526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:03.602123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":397,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428466+0000","last_change":"2024-09-13T13:08:53.215325+0000","last_active":"2024-09-13T13:09:57.428466+0000","last_peered":"2024-09-13T13:09:57.428466+0000","last_clean":"2024-09-13T13:09:57.428466+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:09:57.428466+0000","last_undegraded":"2024-09-13T13:09:57.428466+0000","last_fullsized":"2024-09-13T13:09:57.428466+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.215263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.215263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:10.458784+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":377,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052850+0000","last_change":"2024-09-13T13:07:57.596272+0000","last_active":"2024-09-13T13:09:57.052850+0000","last_peered":"2024-09-13T13:09:57.052850+0000","last_clean":"2024-09-13T13:09:57.052850+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:09:57.052850+0000","last_undegraded":"2024-09-13T13:09:57.052850+0000","last_fullsized":"2024-09-13T13:09:57.052850+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.596097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:53.148768+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":412,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.051945+0000","last_change":"2024-09-13T13:09:39.368581+0000","last_active":"2024-09-13T13:09:57.051945+0000","last_peered":"2024-09-13T13:09:57.051945+0000","last_clean":"2024-09-13T13:09:57.051945+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:09:57.051945+0000","last_undegraded":"2024-09-13T13:09:57.051945+0000","last_fullsized":"2024-09-13T13:09:57.051945+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2024-09-13T13:09:39.368381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:39.368381+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.235364+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":401,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.054167+0000","last_change":"2024-09-13T13:09:30.487486+0000","last_active":"2024-09-13T13:09:57.054167+0000","last_peered":"2024-09-13T13:09:57.054167+0000","last_clean":"2024-09-13T13:09:57.054167+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:09:57.054167+0000","last_undegraded":"2024-09-13T13:09:57.054167+0000","last_fullsized":"2024-09-13T13:09:57.054167+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.487413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.487413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:53.159481+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":284,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T13:09:52.030682+0000","last_change":"2024-09-13T13:09:27.844085+0000","last_active":"2024-09-13T13:09:52.030682+0000","last_peered":"2024-09-13T13:09:52.030682+0000","last_clean":"2024-09-13T13:09:52.030682+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:09:52.030682+0000","last_undegraded":"2024-09-13T13:09:52.030682+0000","last_fullsized":"2024-09-13T13:09:52.030682+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:27.844003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:27.844003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:31.056528+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":390,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428959+0000","last_change":"2024-09-13T13:09:31.943644+0000","last_active":"2024-09-13T13:09:57.428959+0000","last_peered":"2024-09-13T13:09:57.428959+0000","last_clean":"2024-09-13T13:09:57.428959+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:09:57.428959+0000","last_undegraded":"2024-09-13T13:09:57.428959+0000","last_fullsized":"2024-09-13T13:09:57.428959+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:31.943568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:31.943568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.734605+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":383,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.428859+0000","last_change":"2024-09-13T13:08:27.641188+0000","last_active":"2024-09-13T13:09:57.428859+0000","last_peered":"2024-09-13T13:09:57.428859+0000","last_clean":"2024-09-13T13:09:57.428859+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:09:57.428859+0000","last_undegraded":"2024-09-13T13:09:57.428859+0000","last_fullsized":"2024-09-13T13:09:57.428859+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:27.641105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:27.641105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":386,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.052492+0000","last_change":"2024-09-13T13:09:35.355561+0000","last_active":"2024-09-13T13:09:57.052492+0000","last_peered":"2024-09-13T13:09:57.052492+0000","last_clean":"2024-09-13T13:09:57.052492+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:09:57.052492+0000","last_undegraded":"2024-09-13T13:09:57.052492+0000","last_fullsized":"2024-09-13T13:09:57.052492+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:35.355369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:35.355369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:52.921712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":378,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T13:09:52.031258+0000","last_change":"2024-09-13T13:06:09.928585+0000","last_active":"2024-09-13T13:09:52.031258+0000","last_peered":"2024-09-13T13:09:52.031258+0000","last_clean":"2024-09-13T13:09:52.031258+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:09:52.031258+0000","last_undegraded":"2024-09-13T13:09:52.031258+0000","last_fullsized":"2024-09-13T13:09:52.031258+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:06:09.928504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.928504+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:43.812961+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":394,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T13:09:57.053777+0000","last_change":"2024-09-13T13:09:38.337230+0000","last_active":"2024-09-13T13:09:57.053777+0000","last_peered":"2024-09-13T13:09:57.053777+0000","last_clean":"2024-09-13T13:09:57.053777+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:09:57.053777+0000","last_undegraded":"2024-09-13T13:09:57.053777+0000","last_fullsized":"2024-09-13T13:09:57.053777+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.337156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.337156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:40.248177+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":255854,"num_write_kb":8187265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85554,"ondisk_log_size":85554,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":529,"num_read_kb":524,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201390,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":194960,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93550192,"statfs":{"total":95995035648,"available":95795396608,"internally_reserved":0,"allocated":1372160,"data_stored":984990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":198244810},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141000,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":259492,"kb_used_data":1300,"kb_used_omap":4,"kb_used_meta":258171,"kb_avail":93485660,"statfs":{"total":95995035648,"available":95729315840,"internally_reserved":0,"allocated":1331200,"data_stored":969599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":264367789},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691962,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":274888,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93470264,"statfs":{"total":95995035648,"available":95713550336,"internally_reserved":0,"allocated":770048,"data_stored":393462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920896,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":194396,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93550756,"statfs":{"total":95995035648,"available":95795974144,"internally_reserved":0,"allocated":790528,"data_stored":393598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":198244806},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117127,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":360992,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93384160,"statfs":{"total":95995035648,"available":95625379840,"internally_reserved":0,"allocated":794624,"data_stored":394428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":368835015},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411466,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":274868,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93470284,"statfs":{"total":95995035648,"available":95713570816,"internally_reserved":0,"allocated":753664,"data_stored":392802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574993,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298544,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93446608,"statfs":{"total":95995035648,"available":95689326592,"internally_reserved":0,"allocated":811008,"data_stored":394109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738520,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307412,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93437740,"statfs":{"total":95995035648,"available":95680245760,"internally_reserved":0,"allocated":1376256,"data_stored":984699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:10:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:02.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:02.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[36682]: 5.f scrub starts 2024-09-13T13:10:02.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[36682]: 5.f scrub ok 2024-09-13T13:10:02.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[36682]: from='client.35609 v1:172.21.15.52:0/518887388' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:10:02.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[36682]: 3.9 scrub starts 2024-09-13T13:10:02.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[36682]: 3.9 scrub ok 2024-09-13T13:10:02.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3691987789' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:10:02.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[36682]: from='client.35615 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:10:02.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[29471]: 5.f scrub starts 2024-09-13T13:10:02.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[29471]: 5.f scrub ok 2024-09-13T13:10:02.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[29471]: from='client.35609 v1:172.21.15.52:0/518887388' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:10:02.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[29471]: 3.9 scrub starts 2024-09-13T13:10:02.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[29471]: 3.9 scrub ok 2024-09-13T13:10:02.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3691987789' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:10:02.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:02 smithi052 ceph-mon[29471]: from='client.35615 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:10:02.260 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:02.281 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:02.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:02.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:02.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:02 smithi167 ceph-mon[33178]: 5.f scrub starts 2024-09-13T13:10:02.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:02 smithi167 ceph-mon[33178]: 5.f scrub ok 2024-09-13T13:10:02.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:02 smithi167 ceph-mon[33178]: from='client.35609 v1:172.21.15.52:0/518887388' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:10:02.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:02 smithi167 ceph-mon[33178]: 3.9 scrub starts 2024-09-13T13:10:02.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:02 smithi167 ceph-mon[33178]: 3.9 scrub ok 2024-09-13T13:10:02.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:02 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3691987789' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:10:02.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:02 smithi167 ceph-mon[33178]: from='client.35615 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:10:02.385 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:02.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:02.408 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:02.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:02.510 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:02.523 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'unique_pool_1' removed 2024-09-13T13:10:02.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:02.532 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:02.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:02.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:02.655 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:02.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:02.757 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:02.783 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:02.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:02.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:02.885 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:10:02.885 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:02.910 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:02.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:02.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:03.012 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:03.034 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:03.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:03.136 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:03.162 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:03.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:03.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:03.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:03 smithi052 ceph-mon[29471]: pgmap v424: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 262 op/s 2024-09-13T13:10:03.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:03 smithi052 ceph-mon[29471]: from='client.35615 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:10:03.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:03 smithi052 ceph-mon[29471]: osdmap e239: 8 total, 8 up, 8 in 2024-09-13T13:10:03.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:03 smithi052 ceph-mon[36682]: pgmap v424: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 262 op/s 2024-09-13T13:10:03.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:03 smithi052 ceph-mon[36682]: from='client.35615 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:10:03.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:03 smithi052 ceph-mon[36682]: osdmap e239: 8 total, 8 up, 8 in 2024-09-13T13:10:03.264 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:03.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:03.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:03.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:03 smithi167 ceph-mon[33178]: pgmap v424: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 262 op/s 2024-09-13T13:10:03.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:03 smithi167 ceph-mon[33178]: from='client.35615 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:10:03.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:03 smithi167 ceph-mon[33178]: osdmap e239: 8 total, 8 up, 8 in 2024-09-13T13:10:03.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:03.409 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:03.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:03.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:03.492 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T13:10:03.493 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T13:10:03.493 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-13T13:10:03.494 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-13T13:10:03.510 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:03.562 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:03.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:03.582 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:03.584 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:03.627 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:03.663 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:03.685 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:03.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:03.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:03.809 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:03.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:03.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:03.911 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:03.934 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:03.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:03.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:04.036 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:04.060 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:04.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:04.162 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:04.184 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:04.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:04.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:04.286 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:04.303 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:10:04.303 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:10:04.304 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 0 2024-09-13T13:10:04.304 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:10:04.304 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:04 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:10:04.304 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:04 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:10:04.307 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:04.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:04.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:04 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:10:04.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:10:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:10:04.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:04.432 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:04.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:04.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:04.555 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:04.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:04.657 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:04.680 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:04.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:04.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:04.782 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:04.804 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:04.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:04.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:04.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:04.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:04.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:05.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:05.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:05.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:05.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:05.072 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:10:05.152 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:05.173 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:05.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:05.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:05 smithi052 ceph-mon[36682]: pgmap v426: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:10:05.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:05 smithi052 ceph-mon[36682]: 3.5 scrub starts 2024-09-13T13:10:05.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:05 smithi052 ceph-mon[36682]: 3.5 scrub ok 2024-09-13T13:10:05.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:05.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:05 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1052501403' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:05.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:05 smithi052 ceph-mon[36682]: from='client.35621 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:05 smithi052 ceph-mon[29471]: pgmap v426: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:10:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:05 smithi052 ceph-mon[29471]: 3.5 scrub starts 2024-09-13T13:10:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:05 smithi052 ceph-mon[29471]: 3.5 scrub ok 2024-09-13T13:10:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:05 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1052501403' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:05.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:05 smithi052 ceph-mon[29471]: from='client.35621 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:05.276 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:05.297 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:05.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:05.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:05.420 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:05.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:05.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:05.522 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:05.543 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:05.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:05.551 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:10:05.551 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:10:05.562 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:10:05.562 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:10:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:05 smithi167 ceph-mon[33178]: pgmap v426: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:10:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:05 smithi167 ceph-mon[33178]: 3.5 scrub starts 2024-09-13T13:10:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:05 smithi167 ceph-mon[33178]: 3.5 scrub ok 2024-09-13T13:10:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:05 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1052501403' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:05 smithi167 ceph-mon[33178]: from='client.35621 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:05.579 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:10:05.579 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:10:05.595 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:10:05.596 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:10:05.613 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:10:05.614 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:10:05.629 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:10:05.629 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:10:05.645 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:05.646 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:10:05.646 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:10:05.663 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:10:05.663 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:10:05.666 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:05.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:05.768 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:05.791 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:05.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:05.892 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:05.913 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:05.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:05.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:06.015 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:06.037 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:06.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:06.139 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:06.161 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:06.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:06.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:06.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:06.286 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:06.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:06.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:06.291 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:06.383 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:10:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:10:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:10:06.384 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:06 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:10:06.384 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:06 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:10:06.384 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:06 smithi052 ceph-mon[36682]: from='client.35621 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:10:06.384 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:06 smithi052 ceph-mon[36682]: osdmap e240: 8 total, 8 up, 8 in 2024-09-13T13:10:06.384 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:06 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1580477856' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:06.385 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:06 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:10:06.385 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:06 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:10:06.385 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:06 smithi052 ceph-mon[29471]: from='client.35621 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:10:06.385 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:06 smithi052 ceph-mon[29471]: osdmap e240: 8 total, 8 up, 8 in 2024-09-13T13:10:06.385 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:06 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1580477856' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:06.388 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:06.410 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:06.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:06.512 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:06.534 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:06.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:06.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:06 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:10:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:06 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:10:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:06 smithi167 ceph-mon[33178]: from='client.35621 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:10:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:06 smithi167 ceph-mon[33178]: osdmap e240: 8 total, 8 up, 8 in 2024-09-13T13:10:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:06 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1580477856' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:06.636 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:06.662 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:06.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:06.670 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T13:10:06.764 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:06.787 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:06.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:06.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:06.889 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:06.910 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:06.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:06.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:07.012 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:07.033 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:07.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:07.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:07.135 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:07.156 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:07.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:07.159 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:10:07.160 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:10:07.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:07.279 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:07.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:07.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:07.295 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:07 smithi052 ceph-mon[36682]: pgmap v428: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:10:07.295 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:07 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:10:07.296 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:07 smithi052 ceph-mon[29471]: pgmap v428: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:10:07.296 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:07 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:10:07.381 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:07.405 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:07.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:07.506 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:07.528 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:07.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:07.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:07 smithi167 ceph-mon[33178]: pgmap v428: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:10:07.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:07 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:10:07.630 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:07.638 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:07.654 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:07.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:07.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:07.755 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:07.777 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:07.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:07.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:07.900 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:07.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:08.001 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:08.022 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:08.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:08.026 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:10:08.125 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:08.135 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:10:08.135 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:10:08.148 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:08.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:08.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[29471]: 5.14 scrub starts 2024-09-13T13:10:08.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[29471]: 5.14 scrub ok 2024-09-13T13:10:08.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:10:08.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:10:08.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:10:08.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[29471]: 5.e scrub starts 2024-09-13T13:10:08.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[29471]: 5.e scrub ok 2024-09-13T13:10:08.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4012028501' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T13:10:08.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[36682]: 5.14 scrub starts 2024-09-13T13:10:08.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[36682]: 5.14 scrub ok 2024-09-13T13:10:08.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:10:08.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:10:08.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:10:08.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[36682]: 5.e scrub starts 2024-09-13T13:10:08.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[36682]: 5.e scrub ok 2024-09-13T13:10:08.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4012028501' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T13:10:08.250 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:08.257 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'unique_pool_2' created 2024-09-13T13:10:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:08.280 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:08.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:08.381 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:08.404 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:08.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:08.505 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:08.527 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:08.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:08.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:08.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:08 smithi167 ceph-mon[33178]: 5.14 scrub starts 2024-09-13T13:10:08.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:08 smithi167 ceph-mon[33178]: 5.14 scrub ok 2024-09-13T13:10:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:08 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:10:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:08 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:10:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:08 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:10:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:08 smithi167 ceph-mon[33178]: 5.e scrub starts 2024-09-13T13:10:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:08 smithi167 ceph-mon[33178]: 5.e scrub ok 2024-09-13T13:10:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4012028501' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T13:10:08.629 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:08.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:08.653 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:08.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:08.754 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:08.777 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:08.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:08.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:08.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:08.901 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:08.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:08.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:09.003 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-13T13:10:09.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:09.025 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:09.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:09.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:09.127 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:09.130 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:10:09.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:09.150 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:09.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:09.251 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:09.281 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:09.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:09.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:09.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:09.405 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:09.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:09.506 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:09.519 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:09 smithi052 ceph-mon[36682]: pgmap v429: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-13T13:10:09.520 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:09 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4012028501' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T13:10:09.520 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:09 smithi052 ceph-mon[36682]: osdmap e241: 8 total, 8 up, 8 in 2024-09-13T13:10:09.520 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:09 smithi052 ceph-mon[29471]: pgmap v429: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-13T13:10:09.520 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:09 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4012028501' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T13:10:09.520 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:09 smithi052 ceph-mon[29471]: osdmap e241: 8 total, 8 up, 8 in 2024-09-13T13:10:09.531 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:09.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:09.556 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T13:10:09.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:09 smithi167 ceph-mon[33178]: pgmap v429: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-13T13:10:09.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:09 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4012028501' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T13:10:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:09 smithi167 ceph-mon[33178]: osdmap e241: 8 total, 8 up, 8 in 2024-09-13T13:10:09.602 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:09.603 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:09.612 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:09.633 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:09.658 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:09.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:09.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:09.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:09.782 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:09.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:09.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:09.884 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:09.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:09.906 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:09.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:10.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:10.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:10.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:10.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:10.130 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:10.151 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:10.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:10.253 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:10.281 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:10.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:10.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:10.361 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:10 smithi052 ceph-mon[36682]: 6.d scrub starts 2024-09-13T13:10:10.361 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:10 smithi052 ceph-mon[36682]: 6.d scrub ok 2024-09-13T13:10:10.361 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:10 smithi052 ceph-mon[36682]: osdmap e242: 8 total, 8 up, 8 in 2024-09-13T13:10:10.361 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:10 smithi052 ceph-mon[36682]: 4.c scrub starts 2024-09-13T13:10:10.361 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:10 smithi052 ceph-mon[36682]: 4.c scrub ok 2024-09-13T13:10:10.361 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:10 smithi052 ceph-mon[29471]: 6.d scrub starts 2024-09-13T13:10:10.361 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:10 smithi052 ceph-mon[29471]: 6.d scrub ok 2024-09-13T13:10:10.361 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:10 smithi052 ceph-mon[29471]: osdmap e242: 8 total, 8 up, 8 in 2024-09-13T13:10:10.361 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:10 smithi052 ceph-mon[29471]: 4.c scrub starts 2024-09-13T13:10:10.362 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:10 smithi052 ceph-mon[29471]: 4.c scrub ok 2024-09-13T13:10:10.368 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:10:10.369 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:10:10.383 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:10.386 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:10:10.386 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:10:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:10:10.403 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:10:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:10.406 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:10.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:10.419 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:10:10.419 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:10:10.436 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:10:10.437 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:10:10.453 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:10:10.454 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:10:10.470 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:10:10.470 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:10:10.487 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:10:10.487 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:10:10.507 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:10.529 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:10.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:10.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:10 smithi167 ceph-mon[33178]: 6.d scrub starts 2024-09-13T13:10:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:10 smithi167 ceph-mon[33178]: 6.d scrub ok 2024-09-13T13:10:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:10 smithi167 ceph-mon[33178]: osdmap e242: 8 total, 8 up, 8 in 2024-09-13T13:10:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:10 smithi167 ceph-mon[33178]: 4.c scrub starts 2024-09-13T13:10:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:10 smithi167 ceph-mon[33178]: 4.c scrub ok 2024-09-13T13:10:10.631 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:10.689 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:10.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:10.791 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:10.815 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:10.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:10.916 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:10.943 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:10.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:10.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:11.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:11.066 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:11.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:11.169 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:11.192 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:11.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:11.277 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:10:11.294 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:11.316 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:11.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:11.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:11.418 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:11.440 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:11.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:11.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[29471]: 6.1c scrub starts 2024-09-13T13:10:11.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[29471]: 6.1c scrub ok 2024-09-13T13:10:11.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[29471]: pgmap v432: 148 pgs: 9 creating+peering, 7 unknown, 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:10:11.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[29471]: 6.7 scrub starts 2024-09-13T13:10:11.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[29471]: 6.7 scrub ok 2024-09-13T13:10:11.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[29471]: 5.3 scrub starts 2024-09-13T13:10:11.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[29471]: 5.3 scrub ok 2024-09-13T13:10:11.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[29471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:10:11.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[29471]: osdmap e243: 8 total, 8 up, 8 in 2024-09-13T13:10:11.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4238451965' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:11.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2407759729' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:10:11.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[36682]: 6.1c scrub starts 2024-09-13T13:10:11.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[36682]: 6.1c scrub ok 2024-09-13T13:10:11.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[36682]: pgmap v432: 148 pgs: 9 creating+peering, 7 unknown, 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:10:11.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[36682]: 6.7 scrub starts 2024-09-13T13:10:11.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[36682]: 6.7 scrub ok 2024-09-13T13:10:11.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[36682]: 5.3 scrub starts 2024-09-13T13:10:11.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[36682]: 5.3 scrub ok 2024-09-13T13:10:11.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[36682]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:10:11.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[36682]: osdmap e243: 8 total, 8 up, 8 in 2024-09-13T13:10:11.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4238451965' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:11.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2407759729' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:10:11.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:11.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:11.565 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:11.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:11 smithi167 ceph-mon[33178]: 6.1c scrub starts 2024-09-13T13:10:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:11 smithi167 ceph-mon[33178]: 6.1c scrub ok 2024-09-13T13:10:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:11 smithi167 ceph-mon[33178]: pgmap v432: 148 pgs: 9 creating+peering, 7 unknown, 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:10:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:11 smithi167 ceph-mon[33178]: 6.7 scrub starts 2024-09-13T13:10:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:11 smithi167 ceph-mon[33178]: 6.7 scrub ok 2024-09-13T13:10:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:11 smithi167 ceph-mon[33178]: 5.3 scrub starts 2024-09-13T13:10:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:11 smithi167 ceph-mon[33178]: 5.3 scrub ok 2024-09-13T13:10:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:11 smithi167 ceph-mon[33178]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:10:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:11 smithi167 ceph-mon[33178]: osdmap e243: 8 total, 8 up, 8 in 2024-09-13T13:10:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4238451965' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2407759729' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:10:11.666 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:11.689 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:11.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:11.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:11.791 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:11.814 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:11.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:11.915 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:11.938 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:11.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:12.039 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:12.062 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:12.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:12.164 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:12.186 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:12.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:12.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:12.287 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:12.310 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:12.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:12.382 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[36682]: 3.a scrub starts 2024-09-13T13:10:12.383 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[36682]: 3.a scrub ok 2024-09-13T13:10:12.383 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[36682]: 4.12 scrub starts 2024-09-13T13:10:12.383 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[36682]: 4.12 scrub ok 2024-09-13T13:10:12.383 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:12.383 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2407759729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:10:12.383 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[36682]: osdmap e244: 8 total, 8 up, 8 in 2024-09-13T13:10:12.383 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[29471]: 3.a scrub starts 2024-09-13T13:10:12.383 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[29471]: 3.a scrub ok 2024-09-13T13:10:12.383 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[29471]: 4.12 scrub starts 2024-09-13T13:10:12.383 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[29471]: 4.12 scrub ok 2024-09-13T13:10:12.384 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:12.384 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2407759729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:10:12.384 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:12 smithi052 ceph-mon[29471]: osdmap e244: 8 total, 8 up, 8 in 2024-09-13T13:10:12.388 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:12.412 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:12.435 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:12.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:12.523 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:12 smithi167 ceph-mon[33178]: 3.a scrub starts 2024-09-13T13:10:12.523 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:12 smithi167 ceph-mon[33178]: 3.a scrub ok 2024-09-13T13:10:12.523 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:12 smithi167 ceph-mon[33178]: 4.12 scrub starts 2024-09-13T13:10:12.523 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:12 smithi167 ceph-mon[33178]: 4.12 scrub ok 2024-09-13T13:10:12.524 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:12 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:12.524 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:12 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2407759729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:10:12.524 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:12 smithi167 ceph-mon[33178]: osdmap e244: 8 total, 8 up, 8 in 2024-09-13T13:10:12.536 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:12.557 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:12.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:12.659 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:12.681 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:12.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:12.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:12.803 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:12.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:12.884 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:10:12.884 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:10:12.884 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:10:12.884 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:10:12.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.884 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83958 7.b 7:d1e60c84:::benchmark_data_smithi052_71865_object83957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:10:12.884 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.524868+0000", 2024-09-13T13:10:12.884 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 418.355646337, 2024-09-13T13:10:12.884 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.019764474, 2024-09-13T13:10:12.885 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.885 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.885 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:10:12.885 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.885 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.886 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.886 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83958 2024-09-13T13:10:12.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.886 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.886 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.886 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524868+0000", 2024-09-13T13:10:12.886 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524868+0000", 2024-09-13T13:10:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524870+0000", 2024-09-13T13:10:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8929999999999999e-06 2024-09-13T13:10:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524893+0000", 2024-09-13T13:10:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2925000000000002e-05 2024-09-13T13:10:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524908+0000", 2024-09-13T13:10:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4839e-05 2024-09-13T13:10:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524916+0000", 2024-09-13T13:10:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0539999999999995e-06 2024-09-13T13:10:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525086+0000", 2024-09-13T13:10:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016999100000000001 2024-09-13T13:10:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525228+0000", 2024-09-13T13:10:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014245600000000001 2024-09-13T13:10:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:10:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525369+0000", 2024-09-13T13:10:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014030100000000001 2024-09-13T13:10:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.532247+0000", 2024-09-13T13:10:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0068779770000000004 2024-09-13T13:10:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.532262+0000", 2024-09-13T13:10:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5274e-05 2024-09-13T13:10:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548202+0000", 2024-09-13T13:10:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015939813000000001 2024-09-13T13:10:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548214+0000", 2024-09-13T13:10:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.24e-05 2024-09-13T13:10:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.544509+0000", 2024-09-13T13:10:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99629462499999999 2024-09-13T13:10:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.544633+0000", 2024-09-13T13:10:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012392600000000001 2024-09-13T13:10:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83959 7.b 7:de17d8b6:::benchmark_data_smithi052_71865_object83958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:10:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.525528+0000", 2024-09-13T13:10:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 418.35498672099999, 2024-09-13T13:10:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0337471460000001, 2024-09-13T13:10:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83959 2024-09-13T13:10:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525528+0000", 2024-09-13T13:10:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525528+0000", 2024-09-13T13:10:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525529+0000", 2024-09-13T13:10:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.248e-06 2024-09-13T13:10:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525551+0000", 2024-09-13T13:10:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1900999999999999e-05 2024-09-13T13:10:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525560+0000", 2024-09-13T13:10:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1810000000000007e-06 2024-09-13T13:10:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525568+0000", 2024-09-13T13:10:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7929999999999998e-06 2024-09-13T13:10:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548247+0000", 2024-09-13T13:10:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.022678957 2024-09-13T13:10:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548253+0000", 2024-09-13T13:10:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.84e-06 2024-09-13T13:10:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545053+0000", 2024-09-13T13:10:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.996799993 2024-09-13T13:10:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545157+0000", 2024-09-13T13:10:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104693 2024-09-13T13:10:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5", 2024-09-13T13:10:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545303+0000", 2024-09-13T13:10:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000145434 2024-09-13T13:10:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552563+0000", 2024-09-13T13:10:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0072600859999999998 2024-09-13T13:10:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559142+0000", 2024-09-13T13:10:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0065787509999999999 2024-09-13T13:10:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559190+0000", 2024-09-13T13:10:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7794999999999999e-05 2024-09-13T13:10:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559275+0000", 2024-09-13T13:10:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5474000000000002e-05 2024-09-13T13:10:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83968 7.b 7:d987446b:::benchmark_data_smithi052_71865_object83967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:10:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.545178+0000", 2024-09-13T13:10:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 418.33533628700002, 2024-09-13T13:10:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.014227582, 2024-09-13T13:10:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83968 2024-09-13T13:10:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545178+0000", 2024-09-13T13:10:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545178+0000", 2024-09-13T13:10:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545180+0000", 2024-09-13T13:10:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.728e-06 2024-09-13T13:10:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545203+0000", 2024-09-13T13:10:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3269999999999999e-05 2024-09-13T13:10:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545212+0000", 2024-09-13T13:10:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5620000000000004e-06 2024-09-13T13:10:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545218+0000", 2024-09-13T13:10:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7599999999999999e-06 2024-09-13T13:10:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548273+0000", 2024-09-13T13:10:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0030558460000000001 2024-09-13T13:10:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548279+0000", 2024-09-13T13:10:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7989999999999999e-06 2024-09-13T13:10:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.549411+0000", 2024-09-13T13:10:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0011321479999999 2024-09-13T13:10:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.549546+0000", 2024-09-13T13:10:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013478399999999999 2024-09-13T13:10:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5", 2024-09-13T13:10:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.549734+0000", 2024-09-13T13:10:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000187999 2024-09-13T13:10:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555255+0000", 2024-09-13T13:10:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055203860000000004 2024-09-13T13:10:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559284+0000", 2024-09-13T13:10:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0040296300000000002 2024-09-13T13:10:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559328+0000", 2024-09-13T13:10:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3339e-05 2024-09-13T13:10:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559406+0000", 2024-09-13T13:10:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8331000000000006e-05 2024-09-13T13:10:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83950 7.d 7:b84df720:::benchmark_data_smithi052_71865_object83949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:10:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:10:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 418.332590219, 2024-09-13T13:10:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99935902799999998, 2024-09-13T13:10:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83950 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:10:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547926+0000", 2024-09-13T13:10:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6199999999999999e-06 2024-09-13T13:10:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547961+0000", 2024-09-13T13:10:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4916999999999998e-05 2024-09-13T13:10:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547971+0000", 2024-09-13T13:10:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8789999999999998e-06 2024-09-13T13:10:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547978+0000", 2024-09-13T13:10:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6120000000000001e-06 2024-09-13T13:10:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548150+0000", 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017159 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548159+0000", 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7979999999999995e-06 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547139+0000", 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99898018700000002 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547283+0000", 2024-09-13T13:10:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000144425 2024-09-13T13:10:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83951 7.d 7:bd55e938:::benchmark_data_smithi052_71865_object83950:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:10:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.547994+0000", 2024-09-13T13:10:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 418.33252013100002, 2024-09-13T13:10:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0078699250000001, 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83951 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547994+0000", 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547994+0000", 2024-09-13T13:10:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547995+0000", 2024-09-13T13:10:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6899999999999996e-07 2024-09-13T13:10:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.918 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:12.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548019+0000", 2024-09-13T13:10:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3451999999999998e-05 2024-09-13T13:10:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548026+0000", 2024-09-13T13:10:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1600000000000001e-06 2024-09-13T13:10:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548032+0000", 2024-09-13T13:10:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8359999999999997e-06 2024-09-13T13:10:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548182+0000", 2024-09-13T13:10:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015005300000000001 2024-09-13T13:10:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548188+0000", 2024-09-13T13:10:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7869999999999998e-06 2024-09-13T13:10:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547487+0000", 2024-09-13T13:10:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99929913400000003 2024-09-13T13:10:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for missing object", 2024-09-13T13:10:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547629+0000", 2024-09-13T13:10:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000142106 2024-09-13T13:10:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.923 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.923 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555757+0000", 2024-09-13T13:10:12.923 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0081283139999999993 2024-09-13T13:10:12.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.923 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.923 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555864+0000", 2024-09-13T13:10:12.923 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010721399999999999 2024-09-13T13:10:12.923 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.923 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.923 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.924 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83978 7.d 7:b4a32006:::benchmark_data_smithi052_71865_object83977:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:10:12.924 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.562795+0000", 2024-09-13T13:10:12.924 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 418.31772000900003, 2024-09-13T13:10:12.924 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99338139199999997, 2024-09-13T13:10:12.924 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.924 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.924 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:12.924 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.925 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.925 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.925 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83978 2024-09-13T13:10:12.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.925 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.925 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.925 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562795+0000", 2024-09-13T13:10:12.925 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.926 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.926 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562795+0000", 2024-09-13T13:10:12.926 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.926 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.926 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.926 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562797+0000", 2024-09-13T13:10:12.926 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2309999999999999e-06 2024-09-13T13:10:12.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.926 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.926 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.926 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562824+0000", 2024-09-13T13:10:12.927 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6838999999999998e-05 2024-09-13T13:10:12.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.927 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.927 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562836+0000", 2024-09-13T13:10:12.927 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2812e-05 2024-09-13T13:10:12.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.927 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.927 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562845+0000", 2024-09-13T13:10:12.928 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2320000000000001e-06 2024-09-13T13:10:12.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.928 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.928 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.928 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.563048+0000", 2024-09-13T13:10:12.928 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020349699999999999 2024-09-13T13:10:12.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.928 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.928 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.928 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.563058+0000", 2024-09-13T13:10:12.928 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5820000000000008e-06 2024-09-13T13:10:12.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.929 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.929 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.929 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547871+0000", 2024-09-13T13:10:12.929 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98481351699999997 2024-09-13T13:10:12.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.929 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.929 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:12.929 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548017+0000", 2024-09-13T13:10:12.929 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014600999999999999 2024-09-13T13:10:12.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.929 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.930 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0", 2024-09-13T13:10:12.930 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548480+0000", 2024-09-13T13:10:12.930 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00046313600000000001 2024-09-13T13:10:12.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.930 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:12.930 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555942+0000", 2024-09-13T13:10:12.930 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0074612120000000001 2024-09-13T13:10:12.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.931 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.931 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556034+0000", 2024-09-13T13:10:12.931 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2584000000000007e-05 2024-09-13T13:10:12.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.931 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.931 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:12.931 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556087+0000", 2024-09-13T13:10:12.931 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3118999999999997e-05 2024-09-13T13:10:12.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.931 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.931 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.932 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556176+0000", 2024-09-13T13:10:12.932 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8621000000000004e-05 2024-09-13T13:10:12.932 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.932 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.932 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.932 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.932 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83979 7.b 7:d79e8e3c:::benchmark_data_smithi052_71865_object83978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:10:12.932 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.562864+0000", 2024-09-13T13:10:12.932 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 418.31765024600003, 2024-09-13T13:10:12.932 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99667409100000004, 2024-09-13T13:10:12.932 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.933 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.933 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:12.933 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.933 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.933 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.933 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83979 2024-09-13T13:10:12.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.933 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.933 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.933 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.933 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562864+0000", 2024-09-13T13:10:12.934 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.934 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.934 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562864+0000", 2024-09-13T13:10:12.934 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.934 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.934 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562865+0000", 2024-09-13T13:10:12.934 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0900000000000001e-07 2024-09-13T13:10:12.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.935 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.935 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562887+0000", 2024-09-13T13:10:12.935 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1977999999999999e-05 2024-09-13T13:10:12.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.935 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.935 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562897+0000", 2024-09-13T13:10:12.935 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8740000000000007e-06 2024-09-13T13:10:12.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.936 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.936 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562902+0000", 2024-09-13T13:10:12.936 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2780000000000001e-06 2024-09-13T13:10:12.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.936 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.936 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562985+0000", 2024-09-13T13:10:12.936 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3119000000000002e-05 2024-09-13T13:10:12.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.936 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.937 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562994+0000", 2024-09-13T13:10:12.937 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1190000000000003e-06 2024-09-13T13:10:12.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.937 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.937 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.937 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552593+0000", 2024-09-13T13:10:12.937 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98959876899999999 2024-09-13T13:10:12.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.937 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.937 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:12.937 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552700+0000", 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010694199999999999 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5", 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552844+0000", 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014343099999999999 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555298+0000", 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024543220000000001 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.938 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559413+0000", 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041153700000000001 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559458+0000", 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5105999999999998e-05 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559538+0000", 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0074000000000006e-05 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.939 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83988 7.b 7:d1d56cdb:::benchmark_data_smithi052_71865_object83987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.571369+0000", 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 418.30914557400001, 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98902711700000001, 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.940 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83988 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571369+0000", 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571369+0000", 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571371+0000", 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0760000000000001e-06 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571391+0000", 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9936999999999999e-05 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571404+0000", 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3241000000000001e-05 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.942 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.943 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.943 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571410+0000", 2024-09-13T13:10:12.943 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7799999999999997e-06 2024-09-13T13:10:12.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.943 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.943 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.943 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571466+0000", 2024-09-13T13:10:12.943 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6030999999999999e-05 2024-09-13T13:10:12.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.943 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.943 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.943 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571473+0000", 2024-09-13T13:10:12.943 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8539999999999999e-06 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556198+0000", 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98472465899999995 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556486+0000", 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00028855500000000002 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5", 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556637+0000", 2024-09-13T13:10:12.944 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000151218 2024-09-13T13:10:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559546+0000", 2024-09-13T13:10:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029090740000000002 2024-09-13T13:10:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560209+0000", 2024-09-13T13:10:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00066297199999999995 2024-09-13T13:10:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560301+0000", 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1712000000000006e-05 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560396+0000", 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5007999999999996e-05 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:166990 7.b 7:db1a2623:::benchmark_data_smithi052_71865_object45889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T13:10:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.766249+0000", 2024-09-13T13:10:12.947 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 325.11426510799998, 2024-09-13T13:10:12.947 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0195013719999999, 2024-09-13T13:10:12.947 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.947 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.947 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:10:12.947 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.947 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.947 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.947 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 166990 2024-09-13T13:10:12.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.947 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.948 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.948 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.948 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766249+0000", 2024-09-13T13:10:12.948 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.948 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.948 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.948 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766249+0000", 2024-09-13T13:10:12.948 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.948 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.948 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.949 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766252+0000", 2024-09-13T13:10:12.949 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.48e-06 2024-09-13T13:10:12.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.949 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.949 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.949 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766259+0000", 2024-09-13T13:10:12.949 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0849999999999999e-06 2024-09-13T13:10:12.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.949 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.949 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.950 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766272+0000", 2024-09-13T13:10:12.950 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.305e-05 2024-09-13T13:10:12.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.950 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.950 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.950 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766281+0000", 2024-09-13T13:10:12.950 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5750000000000001e-06 2024-09-13T13:10:12.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.950 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.950 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.950 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.767413+0000", 2024-09-13T13:10:12.950 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011321529999999999 2024-09-13T13:10:12.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.951 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.951 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:12.951 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.767559+0000", 2024-09-13T13:10:12.951 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000146217 2024-09-13T13:10:12.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.951 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.951 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:10:12.951 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.767706+0000", 2024-09-13T13:10:12.951 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014731799999999999 2024-09-13T13:10:12.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.952 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.952 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.952 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.776972+0000", 2024-09-13T13:10:12.952 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0092654350000000007 2024-09-13T13:10:12.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.952 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.952 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.952 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.776984+0000", 2024-09-13T13:10:12.952 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2018000000000001e-05 2024-09-13T13:10:12.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.952 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.953 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.953 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.785665+0000", 2024-09-13T13:10:12.953 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0086813429999999 2024-09-13T13:10:12.953 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.953 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.953 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.953 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.785751+0000", 2024-09-13T13:10:12.953 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5698000000000005e-05 2024-09-13T13:10:12.953 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.953 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.953 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.954 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.954 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167004 7.b 7:d0e4af5f:::benchmark_data_smithi052_71865_object45903:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:10:12.954 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.789600+0000", 2024-09-13T13:10:12.954 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 325.09091499800002, 2024-09-13T13:10:12.954 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0016245429999999, 2024-09-13T13:10:12.954 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.954 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.954 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:12.954 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.954 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167004 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789600+0000", 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789600+0000", 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789602+0000", 2024-09-13T13:10:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2230000000000001e-06 2024-09-13T13:10:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789608+0000", 2024-09-13T13:10:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.37e-06 2024-09-13T13:10:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.956 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789624+0000", 2024-09-13T13:10:12.957 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6167e-05 2024-09-13T13:10:12.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.957 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.957 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.957 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789632+0000", 2024-09-13T13:10:12.957 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7270000000000007e-06 2024-09-13T13:10:12.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.957 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.957 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.957 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789705+0000", 2024-09-13T13:10:12.957 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2799999999999994e-05 2024-09-13T13:10:12.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789718+0000", 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2653999999999999e-05 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.786269+0000", 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99655117400000004 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.786357+0000", 2024-09-13T13:10:12.958 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7865e-05 2024-09-13T13:10:12.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.959 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.959 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:10:12.959 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.786482+0000", 2024-09-13T13:10:12.959 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000125451 2024-09-13T13:10:12.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.959 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.959 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.959 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791065+0000", 2024-09-13T13:10:12.959 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045828450000000003 2024-09-13T13:10:12.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.959 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.959 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:12.960 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791120+0000", 2024-09-13T13:10:12.960 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5562000000000001e-05 2024-09-13T13:10:12.960 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.960 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.960 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.960 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791155+0000", 2024-09-13T13:10:12.960 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4326999999999998e-05 2024-09-13T13:10:12.960 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.960 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.960 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:12.960 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791188+0000", 2024-09-13T13:10:12.960 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2929000000000003e-05 2024-09-13T13:10:12.961 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.961 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.961 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.961 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791224+0000", 2024-09-13T13:10:12.961 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6449000000000003e-05 2024-09-13T13:10:12.961 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.961 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.961 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.961 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.961 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.961 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167017 7.b 7:d2ad2293:::benchmark_data_smithi052_71865_object45916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:10:12.961 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.800317+0000", 2024-09-13T13:10:12.961 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 325.080197166, 2024-09-13T13:10:12.962 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99742454800000002, 2024-09-13T13:10:12.962 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.962 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.962 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:12.962 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.962 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.962 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.962 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167017 2024-09-13T13:10:12.962 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.962 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.962 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.962 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.963 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800317+0000", 2024-09-13T13:10:12.963 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.963 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.963 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.963 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800317+0000", 2024-09-13T13:10:12.963 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.963 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.963 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.963 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800320+0000", 2024-09-13T13:10:12.964 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.148e-06 2024-09-13T13:10:12.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.964 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.964 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.964 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800326+0000", 2024-09-13T13:10:12.964 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2960000000000004e-06 2024-09-13T13:10:12.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.964 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.964 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.964 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800343+0000", 2024-09-13T13:10:12.964 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7079e-05 2024-09-13T13:10:12.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.965 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.965 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.965 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800351+0000", 2024-09-13T13:10:12.965 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1340000000000004e-06 2024-09-13T13:10:12.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.965 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.965 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.965 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800433+0000", 2024-09-13T13:10:12.965 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1509999999999997e-05 2024-09-13T13:10:12.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.965 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.966 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.966 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800445+0000", 2024-09-13T13:10:12.966 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2414e-05 2024-09-13T13:10:12.966 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.966 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.966 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.966 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.788619+0000", 2024-09-13T13:10:12.966 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98817433300000002 2024-09-13T13:10:12.966 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.966 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.966 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:12.967 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.788727+0000", 2024-09-13T13:10:12.967 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108059 2024-09-13T13:10:12.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.967 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.967 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:10:12.967 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.788817+0000", 2024-09-13T13:10:12.967 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9117999999999996e-05 2024-09-13T13:10:12.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.967 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.967 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.967 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791266+0000", 2024-09-13T13:10:12.968 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024498599999999999 2024-09-13T13:10:12.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.968 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.968 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.968 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791306+0000", 2024-09-13T13:10:12.968 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9691000000000003e-05 2024-09-13T13:10:12.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.968 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.968 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:12.968 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797479+0000", 2024-09-13T13:10:12.968 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0061724299999999996 2024-09-13T13:10:12.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.969 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:12.969 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797695+0000", 2024-09-13T13:10:12.969 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021655399999999999 2024-09-13T13:10:12.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.969 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.969 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797742+0000", 2024-09-13T13:10:12.969 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6922000000000002e-05 2024-09-13T13:10:12.969 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.969 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.970 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.970 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167030 7.b 7:d39cbe03:::benchmark_data_smithi052_71865_object45929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:10:12.970 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.815695+0000", 2024-09-13T13:10:12.970 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 325.06481999900001, 2024-09-13T13:10:12.970 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98232124600000004, 2024-09-13T13:10:12.970 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.970 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.970 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:12.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.970 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.971 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167030 2024-09-13T13:10:12.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.971 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.971 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.971 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815695+0000", 2024-09-13T13:10:12.971 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.971 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.971 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815695+0000", 2024-09-13T13:10:12.971 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.972 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.972 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815697+0000", 2024-09-13T13:10:12.972 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8380000000000002e-06 2024-09-13T13:10:12.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.972 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.972 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815705+0000", 2024-09-13T13:10:12.972 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1430000000000001e-06 2024-09-13T13:10:12.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.972 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.972 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815723+0000", 2024-09-13T13:10:12.973 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.838e-05 2024-09-13T13:10:12.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.973 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.973 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815732+0000", 2024-09-13T13:10:12.973 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2010000000000005e-06 2024-09-13T13:10:12.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.973 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.973 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815818+0000", 2024-09-13T13:10:12.973 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5518000000000004e-05 2024-09-13T13:10:12.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.974 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.974 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815827+0000", 2024-09-13T13:10:12.974 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6940000000000007e-06 2024-09-13T13:10:12.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.974 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.974 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.974 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791337+0000", 2024-09-13T13:10:12.974 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97550924100000003 2024-09-13T13:10:12.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.975 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:12.975 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791455+0000", 2024-09-13T13:10:12.975 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118081 2024-09-13T13:10:12.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.975 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:10:12.975 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791545+0000", 2024-09-13T13:10:12.975 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0480000000000001e-05 2024-09-13T13:10:12.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797789+0000", 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0062433799999999998 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797800+0000", 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1153e-05 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797829+0000", 2024-09-13T13:10:12.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9527999999999999e-05 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797977+0000", 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000148252 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798016+0000", 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8356999999999999e-05 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167044 7.b 7:d626642e:::benchmark_data_smithi052_71865_object45943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.832092+0000", 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 325.04842287000002, 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98043175900000001, 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167044 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.978 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832092+0000", 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832092+0000", 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832094+0000", 2024-09-13T13:10:12.979 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1299999999999999e-06 2024-09-13T13:10:12.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.980 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.980 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.980 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832100+0000", 2024-09-13T13:10:12.980 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9349999999999999e-06 2024-09-13T13:10:12.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.980 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.980 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.980 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832115+0000", 2024-09-13T13:10:12.980 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5104e-05 2024-09-13T13:10:12.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.981 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.981 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832122+0000", 2024-09-13T13:10:12.981 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5769999999999996e-06 2024-09-13T13:10:12.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.981 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.981 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832207+0000", 2024-09-13T13:10:12.981 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4742999999999998e-05 2024-09-13T13:10:12.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.981 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.982 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832217+0000", 2024-09-13T13:10:12.982 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6069999999999997e-06 2024-09-13T13:10:12.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.982 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.982 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.794154+0000", 2024-09-13T13:10:12.982 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96193714900000005 2024-09-13T13:10:12.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.982 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:12.982 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.794275+0000", 2024-09-13T13:10:12.983 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012122500000000001 2024-09-13T13:10:12.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.983 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:10:12.983 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.794399+0000", 2024-09-13T13:10:12.983 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012333799999999999 2024-09-13T13:10:12.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.983 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.983 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798054+0000", 2024-09-13T13:10:12.983 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036553319999999999 2024-09-13T13:10:12.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.984 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.984 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798090+0000", 2024-09-13T13:10:12.984 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6616999999999999e-05 2024-09-13T13:10:12.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.984 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:12.984 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812409+0000", 2024-09-13T13:10:12.984 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014318587000000001 2024-09-13T13:10:12.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.985 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:12.985 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812475+0000", 2024-09-13T13:10:12.985 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5811999999999999e-05 2024-09-13T13:10:12.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.985 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.985 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.985 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812523+0000", 2024-09-13T13:10:12.985 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8603000000000002e-05 2024-09-13T13:10:12.985 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.985 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.985 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.986 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.986 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167053 7.b 7:d291c1c0:::benchmark_data_smithi052_71865_object45952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:10:12.986 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.840143+0000", 2024-09-13T13:10:12.986 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 325.04037156700002, 2024-09-13T13:10:12.986 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97266568200000003, 2024-09-13T13:10:12.986 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.986 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.986 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:12.987 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.987 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.987 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.987 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167053 2024-09-13T13:10:12.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.987 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.987 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.987 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.987 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840143+0000", 2024-09-13T13:10:12.987 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.988 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.988 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.988 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840143+0000", 2024-09-13T13:10:12.988 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.988 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.988 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.988 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840145+0000", 2024-09-13T13:10:12.988 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:10:12.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.989 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.989 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.989 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840153+0000", 2024-09-13T13:10:12.989 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4080000000000001e-06 2024-09-13T13:10:12.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.989 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.989 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.989 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840184+0000", 2024-09-13T13:10:12.989 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0988000000000002e-05 2024-09-13T13:10:12.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.990 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.990 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840408+0000", 2024-09-13T13:10:12.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00022425 2024-09-13T13:10:12.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.990 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.990 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840568+0000", 2024-09-13T13:10:12.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015967999999999999 2024-09-13T13:10:12.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.991 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.991 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840580+0000", 2024-09-13T13:10:12.991 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.199e-05 2024-09-13T13:10:12.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.991 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.991 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798117+0000", 2024-09-13T13:10:12.991 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95753740700000001 2024-09-13T13:10:12.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.992 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.992 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:12.992 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798208+0000", 2024-09-13T13:10:12.992 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1225999999999997e-05 2024-09-13T13:10:12.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.992 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.992 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:10:12.992 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798311+0000", 2024-09-13T13:10:12.993 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103112 2024-09-13T13:10:12.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.993 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.993 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804034+0000", 2024-09-13T13:10:12.993 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057222940000000002 2024-09-13T13:10:12.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.993 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:12.993 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804070+0000", 2024-09-13T13:10:12.993 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6854999999999999e-05 2024-09-13T13:10:12.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.994 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:12.994 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812721+0000", 2024-09-13T13:10:12.994 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0086508160000000004 2024-09-13T13:10:12.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.994 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:12.994 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812768+0000", 2024-09-13T13:10:12.994 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6251000000000002e-05 2024-09-13T13:10:12.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.995 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.995 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:12.995 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812809+0000", 2024-09-13T13:10:12.995 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1153999999999999e-05 2024-09-13T13:10:12.995 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.995 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:12.995 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:12.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.995 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.995 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167055 7.b 7:d9aeaf19:::benchmark_data_smithi052_71865_object45954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:10:12.995 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.840429+0000", 2024-09-13T13:10:12.996 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 325.04008546699998, 2024-09-13T13:10:12.996 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97246315999999999, 2024-09-13T13:10:12.996 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:12.996 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:12.996 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:12.996 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:12.996 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:12.996 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:12.996 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167055 2024-09-13T13:10:12.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.996 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:12.997 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.997 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:12.997 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840429+0000", 2024-09-13T13:10:12.997 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.997 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.997 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:12.997 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840429+0000", 2024-09-13T13:10:12.997 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:12.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.998 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.998 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:12.998 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840430+0000", 2024-09-13T13:10:12.998 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0499999999999999e-06 2024-09-13T13:10:12.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.998 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.998 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:12.998 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840435+0000", 2024-09-13T13:10:12.998 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6979999999999997e-06 2024-09-13T13:10:12.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.998 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.998 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:12.998 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840443+0000", 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5709999999999998e-06 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840450+0000", 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3990000000000001e-06 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840607+0000", 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015671000000000001 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:12.999 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840614+0000", 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2459999999999997e-06 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.801126+0000", 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96051206300000003 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.801236+0000", 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110356 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.000 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:10:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.801360+0000", 2024-09-13T13:10:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012337399999999999 2024-09-13T13:10:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804101+0000", 2024-09-13T13:10:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027418619999999999 2024-09-13T13:10:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806730+0000", 2024-09-13T13:10:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002629022 2024-09-13T13:10:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812818+0000", 2024-09-13T13:10:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0060879970000000004 2024-09-13T13:10:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812858+0000", 2024-09-13T13:10:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9833000000000003e-05 2024-09-13T13:10:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812892+0000", 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3979000000000001e-05 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167059 7.b 7:d9156452:::benchmark_data_smithi052_71865_object45958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.843970+0000", 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 325.036544521, 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96901141300000004, 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:13.003 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:13.004 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:13.004 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:13.004 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167059 2024-09-13T13:10:13.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.004 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:13.004 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.004 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:13.004 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843970+0000", 2024-09-13T13:10:13.004 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:13.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.004 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843970+0000", 2024-09-13T13:10:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843972+0000", 2024-09-13T13:10:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1660000000000001e-06 2024-09-13T13:10:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843979+0000", 2024-09-13T13:10:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6490000000000002e-06 2024-09-13T13:10:13.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.006 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:13.006 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843997+0000", 2024-09-13T13:10:13.006 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8493999999999999e-05 2024-09-13T13:10:13.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.006 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:13.006 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.844006+0000", 2024-09-13T13:10:13.006 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3459999999999994e-06 2024-09-13T13:10:13.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.007 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:13.007 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.844089+0000", 2024-09-13T13:10:13.007 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3424999999999995e-05 2024-09-13T13:10:13.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.007 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.007 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:13.007 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.844101+0000", 2024-09-13T13:10:13.007 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1984000000000001e-05 2024-09-13T13:10:13.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.007 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.007 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:13.008 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804127+0000", 2024-09-13T13:10:13.008 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96002581099999995 2024-09-13T13:10:13.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.008 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:13.008 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804221+0000", 2024-09-13T13:10:13.008 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4267999999999999e-05 2024-09-13T13:10:13.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.008 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:10:13.009 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804324+0000", 2024-09-13T13:10:13.009 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103071 2024-09-13T13:10:13.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.009 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.009 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:13.009 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806768+0000", 2024-09-13T13:10:13.009 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024435590000000001 2024-09-13T13:10:13.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.009 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.010 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:13.010 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806802+0000", 2024-09-13T13:10:13.010 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4023999999999998e-05 2024-09-13T13:10:13.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.010 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.010 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:13.010 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812898+0000", 2024-09-13T13:10:13.010 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0060959630000000002 2024-09-13T13:10:13.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.010 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.011 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:13.011 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812946+0000", 2024-09-13T13:10:13.011 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8430000000000002e-05 2024-09-13T13:10:13.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.011 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.011 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:13.011 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812981+0000", 2024-09-13T13:10:13.011 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5222999999999998e-05 2024-09-13T13:10:13.011 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:13.011 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:13.011 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:13.012 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.012 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.012 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167062 7.b 7:daa2d453:::benchmark_data_smithi052_71865_object45961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:10:13.012 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.847548+0000", 2024-09-13T13:10:13.012 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 325.032966208, 2024-09-13T13:10:13.012 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96551024600000002, 2024-09-13T13:10:13.012 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:13.012 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:13.013 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:13.013 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:13.013 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:13.013 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:13.013 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167062 2024-09-13T13:10:13.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.013 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:13.013 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.013 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:13.013 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847548+0000", 2024-09-13T13:10:13.013 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:13.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.014 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.014 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:13.014 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847548+0000", 2024-09-13T13:10:13.014 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:13.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.014 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.014 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:13.014 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847551+0000", 2024-09-13T13:10:13.014 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.745e-06 2024-09-13T13:10:13.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.014 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.015 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:13.015 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847558+0000", 2024-09-13T13:10:13.015 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4359999999999999e-06 2024-09-13T13:10:13.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.015 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.015 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:13.015 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847575+0000", 2024-09-13T13:10:13.015 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7405e-05 2024-09-13T13:10:13.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.015 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.015 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:13.016 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847584+0000", 2024-09-13T13:10:13.016 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2869999999999993e-06 2024-09-13T13:10:13.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.016 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:13.016 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847663+0000", 2024-09-13T13:10:13.016 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8288000000000007e-05 2024-09-13T13:10:13.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.016 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:13.017 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847676+0000", 2024-09-13T13:10:13.017 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.367e-05 2024-09-13T13:10:13.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.018 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.018 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:13.018 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806829+0000", 2024-09-13T13:10:13.018 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95915302099999999 2024-09-13T13:10:13.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.018 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.018 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:13.018 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806951+0000", 2024-09-13T13:10:13.018 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012156599999999999 2024-09-13T13:10:13.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:10:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.807058+0000", 2024-09-13T13:10:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010687399999999999 2024-09-13T13:10:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.809517+0000", 2024-09-13T13:10:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024595379999999998 2024-09-13T13:10:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:13.019 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812643+0000", 2024-09-13T13:10:13.020 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031254239999999999 2024-09-13T13:10:13.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.020 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:13.020 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812987+0000", 2024-09-13T13:10:13.020 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00034455199999999998 2024-09-13T13:10:13.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.020 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:13.020 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813026+0000", 2024-09-13T13:10:13.020 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8333e-05 2024-09-13T13:10:13.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.021 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.021 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:13.021 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813059+0000", 2024-09-13T13:10:13.021 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3107e-05 2024-09-13T13:10:13.021 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:13.021 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:13.021 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:13.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.021 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.021 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167076 7.b 7:d73b9ec1:::benchmark_data_smithi052_71865_object45975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:10:13.021 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.861557+0000", 2024-09-13T13:10:13.021 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 325.01895756699997, 2024-09-13T13:10:13.022 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95706732900000002, 2024-09-13T13:10:13.022 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:13.022 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:13.022 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:13.022 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:13.022 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:13.022 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:13.022 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167076 2024-09-13T13:10:13.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.022 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:13.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.023 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:13.023 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861557+0000", 2024-09-13T13:10:13.023 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:13.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.023 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:13.023 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861557+0000", 2024-09-13T13:10:13.023 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:13.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.023 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:13.023 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861559+0000", 2024-09-13T13:10:13.024 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3750000000000001e-06 2024-09-13T13:10:13.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.024 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:13.024 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861566+0000", 2024-09-13T13:10:13.024 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0990000000000004e-06 2024-09-13T13:10:13.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.024 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:13.025 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861583+0000", 2024-09-13T13:10:13.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7405e-05 2024-09-13T13:10:13.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:13.025 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861591+0000", 2024-09-13T13:10:13.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1459999999999996e-06 2024-09-13T13:10:13.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:13.026 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861653+0000", 2024-09-13T13:10:13.026 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1723000000000005e-05 2024-09-13T13:10:13.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.026 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:13.026 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861662+0000", 2024-09-13T13:10:13.026 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9879999999999993e-06 2024-09-13T13:10:13.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.026 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:13.026 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.809546+0000", 2024-09-13T13:10:13.026 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.94788374500000006 2024-09-13T13:10:13.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.027 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:13.027 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.809649+0000", 2024-09-13T13:10:13.027 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103309 2024-09-13T13:10:13.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.027 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:10:13.027 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.809753+0000", 2024-09-13T13:10:13.027 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104096 2024-09-13T13:10:13.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.028 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:13.028 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812707+0000", 2024-09-13T13:10:13.028 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029539829999999999 2024-09-13T13:10:13.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.028 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:13.028 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813064+0000", 2024-09-13T13:10:13.028 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035704999999999998 2024-09-13T13:10:13.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.028 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:13.028 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818518+0000", 2024-09-13T13:10:13.029 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0054542200000000001 2024-09-13T13:10:13.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.029 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.029 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:13.029 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818577+0000", 2024-09-13T13:10:13.029 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8894000000000002e-05 2024-09-13T13:10:13.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.029 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.029 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:13.029 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818624+0000", 2024-09-13T13:10:13.029 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7296000000000003e-05 2024-09-13T13:10:13.029 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:13.029 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167129 7.b 7:d5d8d4da:::benchmark_data_smithi052_71865_object46028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.929446+0000", 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 324.95106882699997, 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.88935671599999999, 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:13.030 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167129 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929446+0000", 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929446+0000", 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.031 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929448+0000", 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.593e-06 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929456+0000", 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7800000000000001e-06 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929474+0000", 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7521999999999999e-05 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.033 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:13.033 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929481+0000", 2024-09-13T13:10:13.033 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3189999999999997e-06 2024-09-13T13:10:13.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.033 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:13.033 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929545+0000", 2024-09-13T13:10:13.033 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4216999999999999e-05 2024-09-13T13:10:13.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.033 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:13.033 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929555+0000", 2024-09-13T13:10:13.033 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8609999999999993e-06 2024-09-13T13:10:13.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.034 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:13.034 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813093+0000", 2024-09-13T13:10:13.034 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.88353817400000001 2024-09-13T13:10:13.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.034 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:13.034 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813205+0000", 2024-09-13T13:10:13.034 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000111549 2024-09-13T13:10:13.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.034 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:10:13.035 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813319+0000", 2024-09-13T13:10:13.035 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114021 2024-09-13T13:10:13.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.035 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:13.035 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818667+0000", 2024-09-13T13:10:13.035 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053481539999999999 2024-09-13T13:10:13.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.035 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:13.035 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818682+0000", 2024-09-13T13:10:13.035 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5438e-05 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818723+0000", 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0284000000000003e-05 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818766+0000", 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3798000000000003e-05 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:13.036 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818802+0000", 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6006000000000001e-05 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167134 7.b 7:d3cd8c35:::benchmark_data_smithi052_71865_object46033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.936758+0000", 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 324.94375643400002, 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.89043407500000005, 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:13.037 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167134 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936758+0000", 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936758+0000", 2024-09-13T13:10:13.038 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936760+0000", 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2199999999999999e-06 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936768+0000", 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.605e-06 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:13.039 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936784+0000", 2024-09-13T13:10:13.040 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5608999999999999e-05 2024-09-13T13:10:13.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.040 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:13.040 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936791+0000", 2024-09-13T13:10:13.040 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:10:13.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.040 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:13.040 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936875+0000", 2024-09-13T13:10:13.040 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3662e-05 2024-09-13T13:10:13.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.041 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.041 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:13.041 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936885+0000", 2024-09-13T13:10:13.041 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0105999999999999e-05 2024-09-13T13:10:13.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.041 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.041 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:13.041 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.815979+0000", 2024-09-13T13:10:13.041 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.87909423799999997 2024-09-13T13:10:13.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.041 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.041 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:13.042 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.816085+0000", 2024-09-13T13:10:13.042 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000106469 2024-09-13T13:10:13.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.042 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:10:13.042 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.816202+0000", 2024-09-13T13:10:13.042 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116344 2024-09-13T13:10:13.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.042 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:13.042 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818689+0000", 2024-09-13T13:10:13.042 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024874559999999999 2024-09-13T13:10:13.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:13.043 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818848+0000", 2024-09-13T13:10:13.043 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015898999999999999 2024-09-13T13:10:13.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:13.043 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.827048+0000", 2024-09-13T13:10:13.043 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0081994749999999995 2024-09-13T13:10:13.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:13.043 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.827147+0000", 2024-09-13T13:10:13.044 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9359999999999997e-05 2024-09-13T13:10:13.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:13.044 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:13.044 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:13.044 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.827192+0000", 2024-09-13T13:10:13.044 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5194999999999997e-05 2024-09-13T13:10:13.044 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:13.044 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:13.044 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:13.044 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:13.044 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:13.045 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:10:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:13.046 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:13.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:13.148 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:13.169 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:13.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:13.271 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:13.285 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-13T13:10:13.294 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:13.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:13.396 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:13.419 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:13.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:13.520 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:13.543 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:13.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:13.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:13.548 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:10:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:13 smithi167 ceph-mon[33178]: pgmap v435: 148 pgs: 9 creating+peering, 7 unknown, 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:10:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4098285805' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:10:13.606 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:13 smithi052 ceph-mon[36682]: pgmap v435: 148 pgs: 9 creating+peering, 7 unknown, 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:10:13.606 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4098285805' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:10:13.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:13 smithi052 ceph-mon[29471]: pgmap v435: 148 pgs: 9 creating+peering, 7 unknown, 132 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:10:13.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4098285805' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:10:13.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:13.667 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:13.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:13.769 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:13.791 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:13.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:13.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:13.915 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:13.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:14.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:14.042 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:14.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:14.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:14.058 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T13:10:14.144 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:14.151 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:10:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:14.168 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:14.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:14.269 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:14.290 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:14 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4098285805' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:10:14.291 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:14 smithi052 ceph-mon[29471]: osdmap e245: 8 total, 8 up, 8 in 2024-09-13T13:10:14.297 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:14.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:14.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:14.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:14 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4098285805' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:10:14.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:14 smithi167 ceph-mon[33178]: osdmap e245: 8 total, 8 up, 8 in 2024-09-13T13:10:14.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:10:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:10:14.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:14.421 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:14.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:14.523 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:14.548 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:14.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:14.647 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:14 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4098285805' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:10:14.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:14 smithi052 ceph-mon[36682]: osdmap e245: 8 total, 8 up, 8 in 2024-09-13T13:10:14.650 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:14.673 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:14.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:14.775 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:14.782 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:14.802 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:14.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:14.813 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:14.820 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:14.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:14.925 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:14.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:15.027 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:15.049 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:15.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:15.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:15.151 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:15.172 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:15.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:15.274 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:15.296 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:15.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:15.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:15.398 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:15.419 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:15.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:15.510 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:15 smithi052 ceph-mon[36682]: pgmap v437: 148 pgs: 3 creating+peering, 145 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:10:15.511 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:15.511 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:15 smithi052 ceph-mon[36682]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:10:15.511 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:15 smithi052 ceph-mon[29471]: pgmap v437: 148 pgs: 3 creating+peering, 145 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:10:15.511 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:15.511 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:15 smithi052 ceph-mon[29471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:10:15.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:15.539 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:10:15.539 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:10:15.546 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:15.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:15.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:15.556 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:10:15.557 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:10:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:15 smithi167 ceph-mon[33178]: pgmap v437: 148 pgs: 3 creating+peering, 145 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:10:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:15 smithi167 ceph-mon[33178]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:10:15.581 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:10:15.581 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:10:15.598 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:10:15.599 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:10:15.616 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:10:15.616 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:10:15.633 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:10:15.633 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:10:15.648 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:15.649 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:10:15.650 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:10:15.668 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:10:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:10:15.674 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:15.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:15.679 DEBUG:teuthology.orchestra.run.smithi052:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-13T13:10:15.679 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_2 2024-09-13T13:10:15.679 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_2 by 10 min 8 2024-09-13T13:10:15.679 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:10:15.719 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T13:10:15.719 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T13:10:15.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:hints = 1 2024-09-13T13:10:15.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T13:10:15.772 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Object prefix: benchmark_data_smithi052_128427 2024-09-13T13:10:15.773 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:10:15.773 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 0 0 0 0 0 0 - 0 2024-09-13T13:10:15.776 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:15.806 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:15.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:15.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:15.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:15.930 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:15.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:16.032 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:16.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:16.079 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:16.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:16.180 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:16.222 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:16.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:16.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:16.323 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:16.355 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:16.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:16.436 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:10:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:10:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:10:16.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:16 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2781372510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:16.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:16 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2781372510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:16.457 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:16.499 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:16.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:16.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:16.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:16 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2781372510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:16.601 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:16.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:16.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:16.730 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:16.773 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:16.773 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 1 16 2696 2680 167.433 167.5 0.0127409 0.00594547 2024-09-13T13:10:16.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:16.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:16.875 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:16.917 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:16.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:16.971 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:10:16.971 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:10:17.018 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:17.056 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:17.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:17.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:17.187 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:17.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:17.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:17.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:17.306 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:10:17.321 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:17.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:17.423 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:17.466 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:17.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:17.555 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:17 smithi052 ceph-mon[29471]: pgmap v438: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:10:17.555 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:17 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1534760429' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:17.555 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:17 smithi052 ceph-mon[36682]: pgmap v438: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:10:17.556 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:17 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1534760429' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:17.567 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:17 smithi167 ceph-mon[33178]: pgmap v438: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:10:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:17 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1534760429' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:17.607 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:17.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:17.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:17.709 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:17.748 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:17.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:17.773 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 2 15 5938 5923 185.049 202.688 0.00250775 0.00539426 2024-09-13T13:10:17.851 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:17.885 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:17.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:17.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:17.987 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:18.028 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:18.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:18.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:18.129 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:18.167 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:18.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:18.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:18.269 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:18.302 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:18.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:18.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:18.395 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T13:10:18.404 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:18.479 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:18.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:18.581 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:18.609 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:18.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:18.711 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:18 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:18 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1534760429' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:10:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:18 smithi052 ceph-mon[36682]: osdmap e246: 8 total, 8 up, 8 in 2024-09-13T13:10:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:18 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:10:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:18 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:18 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1534760429' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:10:18.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:18 smithi052 ceph-mon[29471]: osdmap e246: 8 total, 8 up, 8 in 2024-09-13T13:10:18.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:18 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:10:18.759 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:18.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:18.774 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 3 16 9117 9101 189.554 198.625 0.00486706 0.00526699 2024-09-13T13:10:18.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:18 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:18.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:18 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1534760429' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:10:18.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:18 smithi167 ceph-mon[33178]: osdmap e246: 8 total, 8 up, 8 in 2024-09-13T13:10:18.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:18 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:10:18.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:18.896 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:18.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:18.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:19.031 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:19.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:19.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:19.133 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:19.174 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:19.181 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:19.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:19.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:19.285 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:19.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:19.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:19.430 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:19.472 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:19.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:19.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:19.499 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:19.574 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:19.607 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:19.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:19.628 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:19 smithi052 ceph-mon[36682]: pgmap v440: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 785 B/s rd, 0 op/s 2024-09-13T13:10:19.628 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:19 smithi052 ceph-mon[29471]: pgmap v440: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 785 B/s rd, 0 op/s 2024-09-13T13:10:19.709 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:19.754 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:19.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:19.774 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 4 16 12018 12002 187.49 181.312 0.0104747 0.00532468 2024-09-13T13:10:19.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:19 smithi167 ceph-mon[33178]: pgmap v440: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 785 B/s rd, 0 op/s 2024-09-13T13:10:19.857 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:19.886 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:19.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:19.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:19.989 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:20.027 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:20.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:20.129 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:20.156 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:20.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:20.212 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:10:20.214 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:10:20.257 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:20.286 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:20.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:20.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:20.387 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:10:20.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:20.418 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:20.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:20.463 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:10:20.463 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:10:20.520 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:20.564 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:20.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:20.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:20.697 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:20.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:20.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:20.774 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 5 16 14583 14567 182.051 160.312 0.00407358 0.00548677 2024-09-13T13:10:20.799 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:20.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:20.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:20.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:20.958 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:20.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:21.059 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:21.098 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:21.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:21.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:21.147 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:10:21.200 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:21.235 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:21.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:21.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:21.336 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:21.359 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:21.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:21.460 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:21.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:21 smithi052 ceph-mon[36682]: pgmap v441: 148 pgs: 148 active+clean; 504 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-13T13:10:21.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:21 smithi052 ceph-mon[36682]: from='client.25666 v1:172.21.15.52:0/140847924' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:10:21.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:21 smithi052 ceph-mon[29471]: pgmap v441: 148 pgs: 148 active+clean; 504 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-13T13:10:21.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:21 smithi052 ceph-mon[29471]: from='client.25666 v1:172.21.15.52:0/140847924' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:10:21.495 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:21.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:21.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:21.597 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:21.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:21.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:21.731 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:21.772 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:21.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:21.774 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 6 16 16948 16932 176.343 147.812 0.00379086 0.00566463 2024-09-13T13:10:21.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:21 smithi167 ceph-mon[33178]: pgmap v441: 148 pgs: 148 active+clean; 504 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-13T13:10:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:21 smithi167 ceph-mon[33178]: from='client.25666 v1:172.21.15.52:0/140847924' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:10:21.874 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:21.916 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:21.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:21.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:22.018 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:22.055 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:22.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:22.158 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:22.199 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:22.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:22.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:22.301 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:22.334 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:22.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:22.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:22.366 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2024-09-13T13:10:19.792318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":528947739,"num_objects":8261,"num_object_clones":0,"num_object_copies":24783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8261,"num_whiteouts":0,"num_read":1395,"num_read_kb":1252,"num_write":8611,"num_write_kb":517641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8622,"ondisk_log_size":8622,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3446860,"kb_used_data":1201648,"kb_used_omap":15,"kb_used_meta":2245040,"kb_avail":746514356,"statfs":{"total":767960285184,"available":764430700544,"internally_reserved":0,"allocated":1230487552,"data_stored":1227015313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":2298921753},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":528351232,"num_objects":8062,"num_object_clones":0,"num_object_copies":24186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8062,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8062,"num_write_kb":515968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002363"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":402,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.318194+0000","last_change":"2024-09-13T13:06:44.304525+0000","last_active":"2024-09-13T13:10:17.318194+0000","last_peered":"2024-09-13T13:10:17.318194+0000","last_clean":"2024-09-13T13:10:17.318194+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:10:17.318194+0000","last_undegraded":"2024-09-13T13:10:17.318194+0000","last_fullsized":"2024-09-13T13:10:17.318194+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:44.304444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.304444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.250196+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":426,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.307943+0000","last_change":"2024-09-13T13:09:06.567774+0000","last_active":"2024-09-13T13:10:17.307943+0000","last_peered":"2024-09-13T13:10:17.307943+0000","last_clean":"2024-09-13T13:10:17.307943+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:10:17.307943+0000","last_undegraded":"2024-09-13T13:10:17.307943+0000","last_fullsized":"2024-09-13T13:10:17.307943+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:06.567719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:06.567719+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:30.867489+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":395,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.316979+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:10:17.316979+0000","last_peered":"2024-09-13T13:10:17.316979+0000","last_clean":"2024-09-13T13:10:17.316979+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:10:17.316979+0000","last_undegraded":"2024-09-13T13:10:17.316979+0000","last_fullsized":"2024-09-13T13:10:17.316979+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:14.841596+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":391,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308130+0000","last_change":"2024-09-13T13:09:42.788562+0000","last_active":"2024-09-13T13:10:17.308130+0000","last_peered":"2024-09-13T13:10:17.308130+0000","last_clean":"2024-09-13T13:10:17.308130+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:10:17.308130+0000","last_undegraded":"2024-09-13T13:10:17.308130+0000","last_fullsized":"2024-09-13T13:10:17.308130+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:42.787492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:42.787492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:47.723359+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":409,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.309168+0000","last_change":"2024-09-13T13:08:55.504675+0000","last_active":"2024-09-13T13:10:17.309168+0000","last_peered":"2024-09-13T13:10:17.309168+0000","last_clean":"2024-09-13T13:10:17.309168+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:10:17.309168+0000","last_undegraded":"2024-09-13T13:10:17.309168+0000","last_fullsized":"2024-09-13T13:10:17.309168+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.504475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.504475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.958611+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":414,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.310134+0000","last_change":"2024-09-13T13:09:31.491481+0000","last_active":"2024-09-13T13:10:17.310134+0000","last_peered":"2024-09-13T13:10:17.310134+0000","last_clean":"2024-09-13T13:10:17.310134+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:10:17.310134+0000","last_undegraded":"2024-09-13T13:10:17.310134+0000","last_fullsized":"2024-09-13T13:10:17.310134+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:31.491404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:31.491404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.774257+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":439,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313629+0000","last_change":"2024-09-13T13:08:53.510436+0000","last_active":"2024-09-13T13:10:17.313629+0000","last_peered":"2024-09-13T13:10:17.313629+0000","last_clean":"2024-09-13T13:10:17.313629+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:10:17.313629+0000","last_undegraded":"2024-09-13T13:10:17.313629+0000","last_fullsized":"2024-09-13T13:10:17.313629+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:08:53.510335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.510335+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":414,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.307841+0000","last_change":"2024-09-13T13:08:55.513716+0000","last_active":"2024-09-13T13:10:17.307841+0000","last_peered":"2024-09-13T13:10:17.307841+0000","last_clean":"2024-09-13T13:10:17.307841+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:10:17.307841+0000","last_undegraded":"2024-09-13T13:10:17.307841+0000","last_fullsized":"2024-09-13T13:10:17.307841+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.513667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.513667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:12.754122+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":435,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317372+0000","last_change":"2024-09-13T13:08:59.654555+0000","last_active":"2024-09-13T13:10:17.317372+0000","last_peered":"2024-09-13T13:10:17.317372+0000","last_clean":"2024-09-13T13:10:17.317372+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:10:17.317372+0000","last_undegraded":"2024-09-13T13:10:17.317372+0000","last_fullsized":"2024-09-13T13:10:17.317372+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:08:59.654388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.654388+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":431,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308415+0000","last_change":"2024-09-13T13:09:00.426054+0000","last_active":"2024-09-13T13:10:17.308415+0000","last_peered":"2024-09-13T13:10:17.308415+0000","last_clean":"2024-09-13T13:10:17.308415+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:10:17.308415+0000","last_undegraded":"2024-09-13T13:10:17.308415+0000","last_fullsized":"2024-09-13T13:10:17.308415+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:00.425965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:00.425965+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:24.968404+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":402,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.315834+0000","last_change":"2024-09-13T13:06:50.665256+0000","last_active":"2024-09-13T13:10:17.315834+0000","last_peered":"2024-09-13T13:10:17.315834+0000","last_clean":"2024-09-13T13:10:17.315834+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:10:17.315834+0000","last_undegraded":"2024-09-13T13:10:17.315834+0000","last_fullsized":"2024-09-13T13:10:17.315834+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:50.665191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.665191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.666378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":397,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.315412+0000","last_change":"2024-09-13T13:08:49.280146+0000","last_active":"2024-09-13T13:10:17.315412+0000","last_peered":"2024-09-13T13:10:17.315412+0000","last_clean":"2024-09-13T13:10:17.315412+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:10:17.315412+0000","last_undegraded":"2024-09-13T13:10:17.315412+0000","last_fullsized":"2024-09-13T13:10:17.315412+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:49.280073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:49.280073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":418,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.315950+0000","last_change":"2024-09-13T13:07:57.425510+0000","last_active":"2024-09-13T13:10:17.315950+0000","last_peered":"2024-09-13T13:10:17.315950+0000","last_clean":"2024-09-13T13:10:17.315950+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:10:17.315950+0000","last_undegraded":"2024-09-13T13:10:17.315950+0000","last_fullsized":"2024-09-13T13:10:17.315950+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.425436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.425436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:24.739469+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":409,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308300+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:10:17.308300+0000","last_peered":"2024-09-13T13:10:17.308300+0000","last_clean":"2024-09-13T13:10:17.308300+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:10:17.308300+0000","last_undegraded":"2024-09-13T13:10:17.308300+0000","last_fullsized":"2024-09-13T13:10:17.308300+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:46.784902+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":402,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313526+0000","last_change":"2024-09-13T13:07:50.530956+0000","last_active":"2024-09-13T13:10:17.313526+0000","last_peered":"2024-09-13T13:10:17.313526+0000","last_clean":"2024-09-13T13:10:17.313526+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:10:17.313526+0000","last_undegraded":"2024-09-13T13:10:17.313526+0000","last_fullsized":"2024-09-13T13:10:17.313526+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:50.530904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:50.530904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":382,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.315466+0000","last_change":"2024-09-13T13:07:19.641119+0000","last_active":"2024-09-13T13:10:17.315466+0000","last_peered":"2024-09-13T13:10:17.315466+0000","last_clean":"2024-09-13T13:10:17.315466+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:10:17.315466+0000","last_undegraded":"2024-09-13T13:10:17.315466+0000","last_fullsized":"2024-09-13T13:10:17.315466+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:19.640697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:19.640697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:50.864276+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":412,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.309080+0000","last_change":"2024-09-13T13:07:46.772159+0000","last_active":"2024-09-13T13:10:17.309080+0000","last_peered":"2024-09-13T13:10:17.309080+0000","last_clean":"2024-09-13T13:10:17.309080+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:10:17.309080+0000","last_undegraded":"2024-09-13T13:10:17.309080+0000","last_fullsized":"2024-09-13T13:10:17.309080+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:07:46.772111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:46.772111+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:22.644441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":409,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.323221+0000","last_change":"2024-09-13T13:08:19.181017+0000","last_active":"2024-09-13T13:10:17.323221+0000","last_peered":"2024-09-13T13:10:17.323221+0000","last_clean":"2024-09-13T13:10:17.323221+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:10:17.323221+0000","last_undegraded":"2024-09-13T13:10:17.323221+0000","last_fullsized":"2024-09-13T13:10:17.323221+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:19.180948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.180948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:54.211909+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":440,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313564+0000","last_change":"2024-09-13T13:06:44.532305+0000","last_active":"2024-09-13T13:10:17.313564+0000","last_peered":"2024-09-13T13:10:17.313564+0000","last_clean":"2024-09-13T13:10:17.313564+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:10:17.313564+0000","last_undegraded":"2024-09-13T13:10:17.313564+0000","last_fullsized":"2024-09-13T13:10:17.313564+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:06:44.532186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.532186+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":411,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.318117+0000","last_change":"2024-09-13T13:09:59.936868+0000","last_active":"2024-09-13T13:10:17.318117+0000","last_peered":"2024-09-13T13:10:17.318117+0000","last_clean":"2024-09-13T13:10:17.318117+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:10:17.318117+0000","last_undegraded":"2024-09-13T13:10:17.318117+0000","last_fullsized":"2024-09-13T13:10:17.318117+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:59.936811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:59.936811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:09.372858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":410,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.312558+0000","last_change":"2024-09-13T13:09:29.517189+0000","last_active":"2024-09-13T13:10:17.312558+0000","last_peered":"2024-09-13T13:10:17.312558+0000","last_clean":"2024-09-13T13:10:17.312558+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:10:17.312558+0000","last_undegraded":"2024-09-13T13:10:17.312558+0000","last_fullsized":"2024-09-13T13:10:17.312558+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:29.517034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:29.517034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:34.784832+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":411,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308549+0000","last_change":"2024-09-13T13:09:28.614216+0000","last_active":"2024-09-13T13:10:17.308549+0000","last_peered":"2024-09-13T13:10:17.308549+0000","last_clean":"2024-09-13T13:10:17.308549+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:10:17.308549+0000","last_undegraded":"2024-09-13T13:10:17.308549+0000","last_fullsized":"2024-09-13T13:10:17.308549+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:28.614027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.614027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:31.496891+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":421,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.312581+0000","last_change":"2024-09-13T13:09:42.310274+0000","last_active":"2024-09-13T13:10:17.312581+0000","last_peered":"2024-09-13T13:10:17.312581+0000","last_clean":"2024-09-13T13:10:17.312581+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:10:17.312581+0000","last_undegraded":"2024-09-13T13:10:17.312581+0000","last_fullsized":"2024-09-13T13:10:17.312581+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:09:42.310185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:42.310185+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:09.881670+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":403,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308181+0000","last_change":"2024-09-13T13:07:12.879813+0000","last_active":"2024-09-13T13:10:17.308181+0000","last_peered":"2024-09-13T13:10:17.308181+0000","last_clean":"2024-09-13T13:10:17.308181+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:10:17.308181+0000","last_undegraded":"2024-09-13T13:10:17.308181+0000","last_fullsized":"2024-09-13T13:10:17.308181+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.879727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.879727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.866101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":415,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313758+0000","last_change":"2024-09-13T13:10:09.765203+0000","last_active":"2024-09-13T13:10:17.313758+0000","last_peered":"2024-09-13T13:10:17.313758+0000","last_clean":"2024-09-13T13:10:17.313758+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:10:17.313758+0000","last_undegraded":"2024-09-13T13:10:17.313758+0000","last_fullsized":"2024-09-13T13:10:17.313758+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:09.765153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.765153+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:26.707398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":404,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317133+0000","last_change":"2024-09-13T13:07:54.442276+0000","last_active":"2024-09-13T13:10:17.317133+0000","last_peered":"2024-09-13T13:10:17.317133+0000","last_clean":"2024-09-13T13:10:17.317133+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:10:17.317133+0000","last_undegraded":"2024-09-13T13:10:17.317133+0000","last_fullsized":"2024-09-13T13:10:17.317133+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:07:54.439269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:54.439269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":429,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317238+0000","last_change":"2024-09-13T13:09:28.434007+0000","last_active":"2024-09-13T13:10:17.317238+0000","last_peered":"2024-09-13T13:10:17.317238+0000","last_clean":"2024-09-13T13:10:17.317238+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:10:17.317238+0000","last_undegraded":"2024-09-13T13:10:17.317238+0000","last_fullsized":"2024-09-13T13:10:17.317238+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:28.433899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.433899+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.061308+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":469,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T13:10:14.530722+0000","last_change":"2024-09-13T13:09:39.831623+0000","last_active":"2024-09-13T13:10:14.530722+0000","last_peered":"2024-09-13T13:10:14.530722+0000","last_clean":"2024-09-13T13:10:14.530722+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:10:14.530722+0000","last_undegraded":"2024-09-13T13:10:14.530722+0000","last_fullsized":"2024-09-13T13:10:14.530722+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:09:39.831547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:39.831547+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.629532+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"246'510","reported_seq":531,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.544190+0000","last_change":"2024-09-13T13:10:09.272561+0000","last_active":"2024-09-13T13:10:18.544190+0000","last_peered":"2024-09-13T13:10:18.544190+0000","last_clean":"2024-09-13T13:10:18.544190+0000","last_became_active":"2024-09-13T13:10:09.272264+0000","last_became_peered":"2024-09-13T13:10:09.272264+0000","last_unstale":"2024-09-13T13:10:18.544190+0000","last_undegraded":"2024-09-13T13:10:18.544190+0000","last_fullsized":"2024-09-13T13:10:18.544190+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:35.365520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33423360,"num_objects":510,"num_object_clones":0,"num_object_copies":1530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":510,"num_write_kb":32640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":433,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.318428+0000","last_change":"2024-09-13T13:07:15.243958+0000","last_active":"2024-09-13T13:10:17.318428+0000","last_peered":"2024-09-13T13:10:17.318428+0000","last_clean":"2024-09-13T13:10:17.318428+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:10:17.318428+0000","last_undegraded":"2024-09-13T13:10:17.318428+0000","last_fullsized":"2024-09-13T13:10:17.318428+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:07:15.243715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:15.243715+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:48.866586+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":419,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.309464+0000","last_change":"2024-09-13T13:08:19.560359+0000","last_active":"2024-09-13T13:10:17.309464+0000","last_peered":"2024-09-13T13:10:17.309464+0000","last_clean":"2024-09-13T13:10:17.309464+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:10:17.309464+0000","last_undegraded":"2024-09-13T13:10:17.309464+0000","last_fullsized":"2024-09-13T13:10:17.309464+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:19.560157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.560157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":532,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.309212+0000","last_change":"2024-09-13T13:10:07.726810+0000","last_active":"2024-09-13T13:10:17.309212+0000","last_peered":"2024-09-13T13:10:17.309212+0000","last_clean":"2024-09-13T13:10:17.309212+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:10:17.309212+0000","last_undegraded":"2024-09-13T13:10:17.309212+0000","last_fullsized":"2024-09-13T13:10:17.309212+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:10:07.726751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:07.726751+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:32.746229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":407,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.316327+0000","last_change":"2024-09-13T13:10:08.973968+0000","last_active":"2024-09-13T13:10:17.316327+0000","last_peered":"2024-09-13T13:10:17.316327+0000","last_clean":"2024-09-13T13:10:17.316327+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:10:17.316327+0000","last_undegraded":"2024-09-13T13:10:17.316327+0000","last_fullsized":"2024-09-13T13:10:17.316327+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.973917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.973917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:12.329655+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"246'416","reported_seq":437,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.033701+0000","last_change":"2024-09-13T13:10:09.273532+0000","last_active":"2024-09-13T13:10:18.033701+0000","last_peered":"2024-09-13T13:10:18.033701+0000","last_clean":"2024-09-13T13:10:18.033701+0000","last_became_active":"2024-09-13T13:10:09.273246+0000","last_became_peered":"2024-09-13T13:10:09.273246+0000","last_unstale":"2024-09-13T13:10:18.033701+0000","last_undegraded":"2024-09-13T13:10:18.033701+0000","last_fullsized":"2024-09-13T13:10:18.033701+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.627323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":423,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.309090+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:10:17.309090+0000","last_peered":"2024-09-13T13:10:17.309090+0000","last_clean":"2024-09-13T13:10:17.309090+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:10:17.309090+0000","last_undegraded":"2024-09-13T13:10:17.309090+0000","last_fullsized":"2024-09-13T13:10:17.309090+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":409,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313016+0000","last_change":"2024-09-13T13:06:44.785235+0000","last_active":"2024-09-13T13:10:17.313016+0000","last_peered":"2024-09-13T13:10:17.313016+0000","last_clean":"2024-09-13T13:10:17.313016+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:10:17.313016+0000","last_undegraded":"2024-09-13T13:10:17.313016+0000","last_fullsized":"2024-09-13T13:10:17.313016+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:06:44.785017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.785017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.624176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":396,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.307873+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:10:17.307873+0000","last_peered":"2024-09-13T13:10:17.307873+0000","last_clean":"2024-09-13T13:10:17.307873+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:10:17.307873+0000","last_undegraded":"2024-09-13T13:10:17.307873+0000","last_fullsized":"2024-09-13T13:10:17.307873+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.668051+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":408,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308321+0000","last_change":"2024-09-13T13:09:59.756770+0000","last_active":"2024-09-13T13:10:17.308321+0000","last_peered":"2024-09-13T13:10:17.308321+0000","last_clean":"2024-09-13T13:10:17.308321+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:10:17.308321+0000","last_undegraded":"2024-09-13T13:10:17.308321+0000","last_fullsized":"2024-09-13T13:10:17.308321+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:59.756715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:59.756715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:04.491806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"246'533","reported_seq":554,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.564716+0000","last_change":"2024-09-13T13:10:09.270421+0000","last_active":"2024-09-13T13:10:18.564716+0000","last_peered":"2024-09-13T13:10:18.564716+0000","last_clean":"2024-09-13T13:10:18.564716+0000","last_became_active":"2024-09-13T13:10:09.270129+0000","last_became_peered":"2024-09-13T13:10:09.270129+0000","last_unstale":"2024-09-13T13:10:18.564716+0000","last_undegraded":"2024-09-13T13:10:18.564716+0000","last_fullsized":"2024-09-13T13:10:18.564716+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:37.119106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34930688,"num_objects":533,"num_object_clones":0,"num_object_copies":1599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":533,"num_write_kb":34112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":448,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.307922+0000","last_change":"2024-09-13T13:09:57.768541+0000","last_active":"2024-09-13T13:10:17.307922+0000","last_peered":"2024-09-13T13:10:17.307922+0000","last_clean":"2024-09-13T13:10:17.307922+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:10:17.307922+0000","last_undegraded":"2024-09-13T13:10:17.307922+0000","last_fullsized":"2024-09-13T13:10:17.307922+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:09:57.768472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:57.768472+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:07.273441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":395,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.318113+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:10:17.318113+0000","last_peered":"2024-09-13T13:10:17.318113+0000","last_clean":"2024-09-13T13:10:17.318113+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:10:17.318113+0000","last_undegraded":"2024-09-13T13:10:17.318113+0000","last_fullsized":"2024-09-13T13:10:17.318113+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":515,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308913+0000","last_change":"2024-09-13T13:09:28.477414+0000","last_active":"2024-09-13T13:10:17.308913+0000","last_peered":"2024-09-13T13:10:17.308913+0000","last_clean":"2024-09-13T13:10:17.308913+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:10:17.308913+0000","last_undegraded":"2024-09-13T13:10:17.308913+0000","last_fullsized":"2024-09-13T13:10:17.308913+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:09:28.477338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.477338+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:42.900990+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":402,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313969+0000","last_change":"2024-09-13T13:06:09.230872+0000","last_active":"2024-09-13T13:10:17.313969+0000","last_peered":"2024-09-13T13:10:17.313969+0000","last_clean":"2024-09-13T13:10:17.313969+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:10:17.313969+0000","last_undegraded":"2024-09-13T13:10:17.313969+0000","last_fullsized":"2024-09-13T13:10:17.313969+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:09.230824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.230824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:44.042037+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"246'465","reported_seq":485,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.111202+0000","last_change":"2024-09-13T13:10:09.273698+0000","last_active":"2024-09-13T13:10:18.111202+0000","last_peered":"2024-09-13T13:10:18.111202+0000","last_clean":"2024-09-13T13:10:18.111202+0000","last_became_active":"2024-09-13T13:10:09.273311+0000","last_became_peered":"2024-09-13T13:10:09.273311+0000","last_unstale":"2024-09-13T13:10:18.111202+0000","last_undegraded":"2024-09-13T13:10:18.111202+0000","last_fullsized":"2024-09-13T13:10:18.111202+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:20.896856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30474240,"num_objects":465,"num_object_clones":0,"num_object_copies":1395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":465,"num_write_kb":29760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":423,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.319052+0000","last_change":"2024-09-13T13:09:27.854186+0000","last_active":"2024-09-13T13:10:17.319052+0000","last_peered":"2024-09-13T13:10:17.319052+0000","last_clean":"2024-09-13T13:10:17.319052+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:10:17.319052+0000","last_undegraded":"2024-09-13T13:10:17.319052+0000","last_fullsized":"2024-09-13T13:10:17.319052+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:27.854114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:27.854114+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:39.809612+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":419,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317461+0000","last_change":"2024-09-13T13:10:03.895450+0000","last_active":"2024-09-13T13:10:17.317461+0000","last_peered":"2024-09-13T13:10:17.317461+0000","last_clean":"2024-09-13T13:10:17.317461+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:10:17.317461+0000","last_undegraded":"2024-09-13T13:10:17.317461+0000","last_fullsized":"2024-09-13T13:10:17.317461+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:03.895396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:03.895396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:33.888709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":528,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.321622+0000","last_change":"2024-09-13T13:10:10.134973+0000","last_active":"2024-09-13T13:10:17.321622+0000","last_peered":"2024-09-13T13:10:17.321622+0000","last_clean":"2024-09-13T13:10:17.321622+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:10:17.321622+0000","last_undegraded":"2024-09-13T13:10:17.321622+0000","last_fullsized":"2024-09-13T13:10:17.321622+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:10:10.134873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.134873+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:12.943527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":406,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.315613+0000","last_change":"2024-09-13T13:08:52.207903+0000","last_active":"2024-09-13T13:10:17.315613+0000","last_peered":"2024-09-13T13:10:17.315613+0000","last_clean":"2024-09-13T13:10:17.315613+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:10:17.315613+0000","last_undegraded":"2024-09-13T13:10:17.315613+0000","last_fullsized":"2024-09-13T13:10:17.315613+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:52.207841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:52.207841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"246'446","reported_seq":467,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.032916+0000","last_change":"2024-09-13T13:10:09.270982+0000","last_active":"2024-09-13T13:10:18.032916+0000","last_peered":"2024-09-13T13:10:18.032916+0000","last_clean":"2024-09-13T13:10:18.032916+0000","last_became_active":"2024-09-13T13:10:09.270486+0000","last_became_peered":"2024-09-13T13:10:09.270486+0000","last_unstale":"2024-09-13T13:10:18.032916+0000","last_undegraded":"2024-09-13T13:10:18.032916+0000","last_fullsized":"2024-09-13T13:10:18.032916+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:31.811638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229056,"num_objects":446,"num_object_clones":0,"num_object_copies":1338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":446,"num_write_kb":28544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":450,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.321860+0000","last_change":"2024-09-13T13:09:05.142515+0000","last_active":"2024-09-13T13:10:17.321860+0000","last_peered":"2024-09-13T13:10:17.321860+0000","last_clean":"2024-09-13T13:10:17.321860+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:10:17.321860+0000","last_undegraded":"2024-09-13T13:10:17.321860+0000","last_fullsized":"2024-09-13T13:10:17.321860+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:09:05.142427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.142427+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:12.209335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":396,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.320099+0000","last_change":"2024-09-13T13:09:38.867734+0000","last_active":"2024-09-13T13:10:17.320099+0000","last_peered":"2024-09-13T13:10:17.320099+0000","last_clean":"2024-09-13T13:10:17.320099+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:10:17.320099+0000","last_undegraded":"2024-09-13T13:10:17.320099+0000","last_fullsized":"2024-09-13T13:10:17.320099+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.867688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.867688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:54.044585+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":362,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T13:10:14.530630+0000","last_change":"2024-09-13T13:09:38.854654+0000","last_active":"2024-09-13T13:10:14.530630+0000","last_peered":"2024-09-13T13:10:14.530630+0000","last_clean":"2024-09-13T13:10:14.530630+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:10:14.530630+0000","last_undegraded":"2024-09-13T13:10:14.530630+0000","last_fullsized":"2024-09-13T13:10:14.530630+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.854581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.854581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:50.922952+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":411,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.316875+0000","last_change":"2024-09-13T13:06:43.969251+0000","last_active":"2024-09-13T13:10:17.316875+0000","last_peered":"2024-09-13T13:10:17.316875+0000","last_clean":"2024-09-13T13:10:17.316875+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:10:17.316875+0000","last_undegraded":"2024-09-13T13:10:17.316875+0000","last_fullsized":"2024-09-13T13:10:17.316875+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.969176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.969176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:16.852941+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"246'617","reported_seq":637,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.955294+0000","last_change":"2024-09-13T13:10:09.277094+0000","last_active":"2024-09-13T13:10:18.955294+0000","last_peered":"2024-09-13T13:10:18.955294+0000","last_clean":"2024-09-13T13:10:18.955294+0000","last_became_active":"2024-09-13T13:10:09.275394+0000","last_became_peered":"2024-09-13T13:10:09.275394+0000","last_unstale":"2024-09-13T13:10:18.955294+0000","last_undegraded":"2024-09-13T13:10:18.955294+0000","last_fullsized":"2024-09-13T13:10:18.955294+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:11.852222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40435712,"num_objects":617,"num_object_clones":0,"num_object_copies":1851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":617,"num_write_kb":39488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":568,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.319953+0000","last_change":"2024-09-13T13:07:25.438610+0000","last_active":"2024-09-13T13:10:17.319953+0000","last_peered":"2024-09-13T13:10:17.319953+0000","last_clean":"2024-09-13T13:10:17.319953+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:10:17.319953+0000","last_undegraded":"2024-09-13T13:10:17.319953+0000","last_fullsized":"2024-09-13T13:10:17.319953+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:07:25.438528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:25.438528+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:50.888420+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":423,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308568+0000","last_change":"2024-09-13T13:07:18.888103+0000","last_active":"2024-09-13T13:10:17.308568+0000","last_peered":"2024-09-13T13:10:17.308568+0000","last_clean":"2024-09-13T13:10:17.308568+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:10:17.308568+0000","last_undegraded":"2024-09-13T13:10:17.308568+0000","last_fullsized":"2024-09-13T13:10:17.308568+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:18.888060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:18.888060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:51.825948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":414,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.315817+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:10:17.315817+0000","last_peered":"2024-09-13T13:10:17.315817+0000","last_clean":"2024-09-13T13:10:17.315817+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:10:17.315817+0000","last_undegraded":"2024-09-13T13:10:17.315817+0000","last_fullsized":"2024-09-13T13:10:17.315817+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.818550+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":402,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.318671+0000","last_change":"2024-09-13T13:09:40.277472+0000","last_active":"2024-09-13T13:10:17.318671+0000","last_peered":"2024-09-13T13:10:17.318671+0000","last_clean":"2024-09-13T13:10:17.318671+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:10:17.318671+0000","last_undegraded":"2024-09-13T13:10:17.318671+0000","last_fullsized":"2024-09-13T13:10:17.318671+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:40.277312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:40.277312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:56.382493+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":399,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.309835+0000","last_change":"2024-09-13T13:09:00.570589+0000","last_active":"2024-09-13T13:10:17.309835+0000","last_peered":"2024-09-13T13:10:17.309835+0000","last_clean":"2024-09-13T13:10:17.309835+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:10:17.309835+0000","last_undegraded":"2024-09-13T13:10:17.309835+0000","last_fullsized":"2024-09-13T13:10:17.309835+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:00.570518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:00.570518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.670665+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"246'437","reported_seq":458,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.103996+0000","last_change":"2024-09-13T13:10:09.270343+0000","last_active":"2024-09-13T13:10:18.103996+0000","last_peered":"2024-09-13T13:10:18.103996+0000","last_clean":"2024-09-13T13:10:18.103996+0000","last_became_active":"2024-09-13T13:10:09.270073+0000","last_became_peered":"2024-09-13T13:10:09.270073+0000","last_unstale":"2024-09-13T13:10:18.103996+0000","last_undegraded":"2024-09-13T13:10:18.103996+0000","last_fullsized":"2024-09-13T13:10:18.103996+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:35.580323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28639232,"num_objects":437,"num_object_clones":0,"num_object_copies":1311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":437,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":427,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.318522+0000","last_change":"2024-09-13T13:08:04.680025+0000","last_active":"2024-09-13T13:10:17.318522+0000","last_peered":"2024-09-13T13:10:17.318522+0000","last_clean":"2024-09-13T13:10:17.318522+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:10:17.318522+0000","last_undegraded":"2024-09-13T13:10:17.318522+0000","last_fullsized":"2024-09-13T13:10:17.318522+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:08:04.679766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:04.679766+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:22.837118+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":431,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.318739+0000","last_change":"2024-09-13T13:08:56.771534+0000","last_active":"2024-09-13T13:10:17.318739+0000","last_peered":"2024-09-13T13:10:17.318739+0000","last_clean":"2024-09-13T13:10:17.318739+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:10:17.318739+0000","last_undegraded":"2024-09-13T13:10:17.318739+0000","last_fullsized":"2024-09-13T13:10:17.318739+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:56.771432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:56.771432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.588929+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":427,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.314175+0000","last_change":"2024-09-13T13:08:19.038471+0000","last_active":"2024-09-13T13:10:17.314175+0000","last_peered":"2024-09-13T13:10:17.314175+0000","last_clean":"2024-09-13T13:10:17.314175+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:10:17.314175+0000","last_undegraded":"2024-09-13T13:10:17.314175+0000","last_fullsized":"2024-09-13T13:10:17.314175+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:08:19.038393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.038393+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:25.295684+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":388,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313562+0000","last_change":"2024-09-13T13:03:14.282866+0000","last_active":"2024-09-13T13:10:17.313562+0000","last_peered":"2024-09-13T13:10:17.313562+0000","last_clean":"2024-09-13T13:10:17.313562+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:10:17.313562+0000","last_undegraded":"2024-09-13T13:10:17.313562+0000","last_fullsized":"2024-09-13T13:10:17.313562+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:44.192025+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":410,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313894+0000","last_change":"2024-09-13T13:08:54.232191+0000","last_active":"2024-09-13T13:10:17.313894+0000","last_peered":"2024-09-13T13:10:17.313894+0000","last_clean":"2024-09-13T13:10:17.313894+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:10:17.313894+0000","last_undegraded":"2024-09-13T13:10:17.313894+0000","last_fullsized":"2024-09-13T13:10:17.313894+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:54.232130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:54.232130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.097713+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"246'597","reported_seq":618,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.952988+0000","last_change":"2024-09-13T13:10:09.737716+0000","last_active":"2024-09-13T13:10:18.952988+0000","last_peered":"2024-09-13T13:10:18.952988+0000","last_clean":"2024-09-13T13:10:18.952988+0000","last_became_active":"2024-09-13T13:10:09.737415+0000","last_became_peered":"2024-09-13T13:10:09.737415+0000","last_unstale":"2024-09-13T13:10:18.952988+0000","last_undegraded":"2024-09-13T13:10:18.952988+0000","last_fullsized":"2024-09-13T13:10:18.952988+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.782551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39124992,"num_objects":597,"num_object_clones":0,"num_object_copies":1791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":597,"num_write_kb":38208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":424,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317900+0000","last_change":"2024-09-13T13:08:21.219398+0000","last_active":"2024-09-13T13:10:17.317900+0000","last_peered":"2024-09-13T13:10:17.317900+0000","last_clean":"2024-09-13T13:10:17.317900+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:10:17.317900+0000","last_undegraded":"2024-09-13T13:10:17.317900+0000","last_fullsized":"2024-09-13T13:10:17.317900+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:08:21.219300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:21.219300+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:52.209890+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":410,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.316568+0000","last_change":"2024-09-13T13:08:02.152950+0000","last_active":"2024-09-13T13:10:17.316568+0000","last_peered":"2024-09-13T13:10:17.316568+0000","last_clean":"2024-09-13T13:10:17.316568+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:10:17.316568+0000","last_undegraded":"2024-09-13T13:10:17.316568+0000","last_fullsized":"2024-09-13T13:10:17.316568+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:02.152859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:02.152859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:25.356503+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":969,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313413+0000","last_change":"2024-09-13T13:08:49.264831+0000","last_active":"2024-09-13T13:10:17.313413+0000","last_peered":"2024-09-13T13:10:17.313413+0000","last_clean":"2024-09-13T13:10:17.313413+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:10:17.313413+0000","last_undegraded":"2024-09-13T13:10:17.313413+0000","last_fullsized":"2024-09-13T13:10:17.313413+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:08:49.264775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:49.264775+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":549,"num_read_kb":544,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":387,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317485+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:10:17.317485+0000","last_peered":"2024-09-13T13:10:17.317485+0000","last_clean":"2024-09-13T13:10:17.317485+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:10:17.317485+0000","last_undegraded":"2024-09-13T13:10:17.317485+0000","last_fullsized":"2024-09-13T13:10:17.317485+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:47.867215+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":408,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317522+0000","last_change":"2024-09-13T13:08:55.738138+0000","last_active":"2024-09-13T13:10:17.317522+0000","last_peered":"2024-09-13T13:10:17.317522+0000","last_clean":"2024-09-13T13:10:17.317522+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:10:17.317522+0000","last_undegraded":"2024-09-13T13:10:17.317522+0000","last_fullsized":"2024-09-13T13:10:17.317522+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.738069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.738069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:25.032322+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"246'545","reported_seq":566,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.567973+0000","last_change":"2024-09-13T13:10:09.276701+0000","last_active":"2024-09-13T13:10:18.567973+0000","last_peered":"2024-09-13T13:10:18.567973+0000","last_clean":"2024-09-13T13:10:18.567973+0000","last_became_active":"2024-09-13T13:10:09.276418+0000","last_became_peered":"2024-09-13T13:10:09.276418+0000","last_unstale":"2024-09-13T13:10:18.567973+0000","last_undegraded":"2024-09-13T13:10:18.567973+0000","last_fullsized":"2024-09-13T13:10:18.567973+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":545,"log_dups_size":0,"ondisk_log_size":545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:26.865438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35717120,"num_objects":545,"num_object_clones":0,"num_object_copies":1635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":545,"num_write_kb":34880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":397,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T13:10:14.530480+0000","last_change":"2024-09-13T13:06:51.193749+0000","last_active":"2024-09-13T13:10:14.530480+0000","last_peered":"2024-09-13T13:10:14.530480+0000","last_clean":"2024-09-13T13:10:14.530480+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:10:14.530480+0000","last_undegraded":"2024-09-13T13:10:14.530480+0000","last_fullsized":"2024-09-13T13:10:14.530480+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'18","last_scrub_stamp":"2024-09-13T13:06:51.193678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.193678+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:16.989653+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":414,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.309547+0000","last_change":"2024-09-13T13:06:10.715371+0000","last_active":"2024-09-13T13:10:17.309547+0000","last_peered":"2024-09-13T13:10:17.309547+0000","last_clean":"2024-09-13T13:10:17.309547+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:10:17.309547+0000","last_undegraded":"2024-09-13T13:10:17.309547+0000","last_fullsized":"2024-09-13T13:10:17.309547+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:10.715313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.715313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.220475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":459,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313537+0000","last_change":"2024-09-13T13:06:39.564599+0000","last_active":"2024-09-13T13:10:17.313537+0000","last_peered":"2024-09-13T13:10:17.313537+0000","last_clean":"2024-09-13T13:10:17.313537+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:10:17.313537+0000","last_undegraded":"2024-09-13T13:10:17.313537+0000","last_fullsized":"2024-09-13T13:10:17.313537+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:06:39.564549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:06:39.564549+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":411,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313858+0000","last_change":"2024-09-13T13:08:25.500731+0000","last_active":"2024-09-13T13:10:17.313858+0000","last_peered":"2024-09-13T13:10:17.313858+0000","last_clean":"2024-09-13T13:10:17.313858+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:10:17.313858+0000","last_undegraded":"2024-09-13T13:10:17.313858+0000","last_fullsized":"2024-09-13T13:10:17.313858+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:25.500660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:25.500660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.959857+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":415,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317386+0000","last_change":"2024-09-13T13:10:09.926369+0000","last_active":"2024-09-13T13:10:17.317386+0000","last_peered":"2024-09-13T13:10:17.317386+0000","last_clean":"2024-09-13T13:10:17.317386+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:10:17.317386+0000","last_undegraded":"2024-09-13T13:10:17.317386+0000","last_fullsized":"2024-09-13T13:10:17.317386+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:09.926319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.926319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:39.338826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"246'408","reported_seq":429,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.027749+0000","last_change":"2024-09-13T13:10:09.273573+0000","last_active":"2024-09-13T13:10:18.027749+0000","last_peered":"2024-09-13T13:10:18.027749+0000","last_clean":"2024-09-13T13:10:18.027749+0000","last_became_active":"2024-09-13T13:10:09.273226+0000","last_became_peered":"2024-09-13T13:10:09.273226+0000","last_unstale":"2024-09-13T13:10:18.027749+0000","last_undegraded":"2024-09-13T13:10:18.027749+0000","last_fullsized":"2024-09-13T13:10:18.027749+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:13.577725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26738688,"num_objects":408,"num_object_clones":0,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":408,"num_write_kb":26112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":430,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.309262+0000","last_change":"2024-09-13T13:08:01.790332+0000","last_active":"2024-09-13T13:10:17.309262+0000","last_peered":"2024-09-13T13:10:17.309262+0000","last_clean":"2024-09-13T13:10:17.309262+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:10:17.309262+0000","last_undegraded":"2024-09-13T13:10:17.309262+0000","last_fullsized":"2024-09-13T13:10:17.309262+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:08:01.790280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:01.790280+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:24.650459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":420,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308326+0000","last_change":"2024-09-13T13:10:01.834881+0000","last_active":"2024-09-13T13:10:17.308326+0000","last_peered":"2024-09-13T13:10:17.308326+0000","last_clean":"2024-09-13T13:10:17.308326+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:10:17.308326+0000","last_undegraded":"2024-09-13T13:10:17.308326+0000","last_fullsized":"2024-09-13T13:10:17.308326+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:01.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:01.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.226771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":422,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317152+0000","last_change":"2024-09-13T13:10:00.879081+0000","last_active":"2024-09-13T13:10:17.317152+0000","last_peered":"2024-09-13T13:10:17.317152+0000","last_clean":"2024-09-13T13:10:17.317152+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:10:17.317152+0000","last_undegraded":"2024-09-13T13:10:17.317152+0000","last_fullsized":"2024-09-13T13:10:17.317152+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:00.879024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:00.879024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:11.616602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":402,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.309919+0000","last_change":"2024-09-13T13:08:27.451600+0000","last_active":"2024-09-13T13:10:17.309919+0000","last_peered":"2024-09-13T13:10:17.309919+0000","last_clean":"2024-09-13T13:10:17.309919+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:10:17.309919+0000","last_undegraded":"2024-09-13T13:10:17.309919+0000","last_fullsized":"2024-09-13T13:10:17.309919+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:27.448636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:27.448636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"246'532","reported_seq":553,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.516107+0000","last_change":"2024-09-13T13:10:09.737339+0000","last_active":"2024-09-13T13:10:18.516107+0000","last_peered":"2024-09-13T13:10:18.516107+0000","last_clean":"2024-09-13T13:10:18.516107+0000","last_became_active":"2024-09-13T13:10:09.736988+0000","last_became_peered":"2024-09-13T13:10:09.736988+0000","last_unstale":"2024-09-13T13:10:18.516107+0000","last_undegraded":"2024-09-13T13:10:18.516107+0000","last_fullsized":"2024-09-13T13:10:18.516107+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:19.898448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34865152,"num_objects":532,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":532,"num_write_kb":34048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":428,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308944+0000","last_change":"2024-09-13T13:05:34.166464+0000","last_active":"2024-09-13T13:10:17.308944+0000","last_peered":"2024-09-13T13:10:17.308944+0000","last_clean":"2024-09-13T13:10:17.308944+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:10:17.308944+0000","last_undegraded":"2024-09-13T13:10:17.308944+0000","last_fullsized":"2024-09-13T13:10:17.308944+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:17.686209+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":393,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T13:10:14.530931+0000","last_change":"2024-09-13T13:10:10.787443+0000","last_active":"2024-09-13T13:10:14.530931+0000","last_peered":"2024-09-13T13:10:14.530931+0000","last_clean":"2024-09-13T13:10:14.530931+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:10:14.530931+0000","last_undegraded":"2024-09-13T13:10:14.530931+0000","last_fullsized":"2024-09-13T13:10:14.530931+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:10.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:39.817372+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":396,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.319585+0000","last_change":"2024-09-13T13:08:53.747370+0000","last_active":"2024-09-13T13:10:17.319585+0000","last_peered":"2024-09-13T13:10:17.319585+0000","last_clean":"2024-09-13T13:10:17.319585+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:10:17.319585+0000","last_undegraded":"2024-09-13T13:10:17.319585+0000","last_fullsized":"2024-09-13T13:10:17.319585+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.747302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.747302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.330071+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":400,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313097+0000","last_change":"2024-09-13T13:08:03.015302+0000","last_active":"2024-09-13T13:10:17.313097+0000","last_peered":"2024-09-13T13:10:17.313097+0000","last_clean":"2024-09-13T13:10:17.313097+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:10:17.313097+0000","last_undegraded":"2024-09-13T13:10:17.313097+0000","last_fullsized":"2024-09-13T13:10:17.313097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:03.015233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.015233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:23.253063+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"246'511","reported_seq":532,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.552626+0000","last_change":"2024-09-13T13:10:09.737741+0000","last_active":"2024-09-13T13:10:18.552626+0000","last_peered":"2024-09-13T13:10:18.552626+0000","last_clean":"2024-09-13T13:10:18.552626+0000","last_became_active":"2024-09-13T13:10:09.737415+0000","last_became_peered":"2024-09-13T13:10:09.737415+0000","last_unstale":"2024-09-13T13:10:18.552626+0000","last_undegraded":"2024-09-13T13:10:18.552626+0000","last_fullsized":"2024-09-13T13:10:18.552626+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:35.194931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":511,"num_object_clones":0,"num_object_copies":1533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":511,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":450,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308236+0000","last_change":"2024-09-13T13:10:09.739258+0000","last_active":"2024-09-13T13:10:17.308236+0000","last_peered":"2024-09-13T13:10:17.308236+0000","last_clean":"2024-09-13T13:10:17.308236+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:10:17.308236+0000","last_undegraded":"2024-09-13T13:10:17.308236+0000","last_fullsized":"2024-09-13T13:10:17.308236+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:09.739184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.739184+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.869202+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":403,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.311864+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:10:17.311864+0000","last_peered":"2024-09-13T13:10:17.311864+0000","last_clean":"2024-09-13T13:10:17.311864+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:10:17.311864+0000","last_undegraded":"2024-09-13T13:10:17.311864+0000","last_fullsized":"2024-09-13T13:10:17.311864+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":516,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313833+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:10:17.313833+0000","last_peered":"2024-09-13T13:10:17.313833+0000","last_clean":"2024-09-13T13:10:17.313833+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:10:17.313833+0000","last_undegraded":"2024-09-13T13:10:17.313833+0000","last_fullsized":"2024-09-13T13:10:17.313833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:14.921175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":392,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308967+0000","last_change":"2024-09-13T13:09:37.812544+0000","last_active":"2024-09-13T13:10:17.308967+0000","last_peered":"2024-09-13T13:10:17.308967+0000","last_clean":"2024-09-13T13:10:17.308967+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:10:17.308967+0000","last_undegraded":"2024-09-13T13:10:17.308967+0000","last_fullsized":"2024-09-13T13:10:17.308967+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:37.812451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:37.812451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:00.130077+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"246'451","reported_seq":472,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.094289+0000","last_change":"2024-09-13T13:10:09.274865+0000","last_active":"2024-09-13T13:10:18.094289+0000","last_peered":"2024-09-13T13:10:18.094289+0000","last_clean":"2024-09-13T13:10:18.094289+0000","last_became_active":"2024-09-13T13:10:09.274589+0000","last_became_peered":"2024-09-13T13:10:09.274589+0000","last_unstale":"2024-09-13T13:10:18.094289+0000","last_undegraded":"2024-09-13T13:10:18.094289+0000","last_fullsized":"2024-09-13T13:10:18.094289+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":451,"log_dups_size":0,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:08.647431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29556736,"num_objects":451,"num_object_clones":0,"num_object_copies":1353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":451,"num_write_kb":28864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":406,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.321771+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:10:17.321771+0000","last_peered":"2024-09-13T13:10:17.321771+0000","last_clean":"2024-09-13T13:10:17.321771+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:10:17.321771+0000","last_undegraded":"2024-09-13T13:10:17.321771+0000","last_fullsized":"2024-09-13T13:10:17.321771+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":430,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.316149+0000","last_change":"2024-09-13T13:09:06.668526+0000","last_active":"2024-09-13T13:10:17.316149+0000","last_peered":"2024-09-13T13:10:17.316149+0000","last_clean":"2024-09-13T13:10:17.316149+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:10:17.316149+0000","last_undegraded":"2024-09-13T13:10:17.316149+0000","last_fullsized":"2024-09-13T13:10:17.316149+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:06.668478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:06.668478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.164862+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":412,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313319+0000","last_change":"2024-09-13T13:07:11.400066+0000","last_active":"2024-09-13T13:10:17.313319+0000","last_peered":"2024-09-13T13:10:17.313319+0000","last_clean":"2024-09-13T13:10:17.313319+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:10:17.313319+0000","last_undegraded":"2024-09-13T13:10:17.313319+0000","last_fullsized":"2024-09-13T13:10:17.313319+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.399840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.399840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:45.151594+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":390,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.316181+0000","last_change":"2024-09-13T13:09:29.473480+0000","last_active":"2024-09-13T13:10:17.316181+0000","last_peered":"2024-09-13T13:10:17.316181+0000","last_clean":"2024-09-13T13:10:17.316181+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:10:17.316181+0000","last_undegraded":"2024-09-13T13:10:17.316181+0000","last_fullsized":"2024-09-13T13:10:17.316181+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:29.473418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:29.473418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:55.955954+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"246'449","reported_seq":470,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.111161+0000","last_change":"2024-09-13T13:10:09.271210+0000","last_active":"2024-09-13T13:10:18.111161+0000","last_peered":"2024-09-13T13:10:18.111161+0000","last_clean":"2024-09-13T13:10:18.111161+0000","last_became_active":"2024-09-13T13:10:09.270909+0000","last_became_peered":"2024-09-13T13:10:09.270909+0000","last_unstale":"2024-09-13T13:10:18.111161+0000","last_undegraded":"2024-09-13T13:10:18.111161+0000","last_fullsized":"2024-09-13T13:10:18.111161+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:15.899099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29425664,"num_objects":449,"num_object_clones":0,"num_object_copies":1347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":449,"num_write_kb":28736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":396,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T13:10:14.530683+0000","last_change":"2024-09-13T13:09:32.849360+0000","last_active":"2024-09-13T13:10:14.530683+0000","last_peered":"2024-09-13T13:10:14.530683+0000","last_clean":"2024-09-13T13:10:14.530683+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:10:14.530683+0000","last_undegraded":"2024-09-13T13:10:14.530683+0000","last_fullsized":"2024-09-13T13:10:14.530683+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:09:32.849277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:32.849277+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:00.662384+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":422,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.314341+0000","last_change":"2024-09-13T13:07:48.579717+0000","last_active":"2024-09-13T13:10:17.314341+0000","last_peered":"2024-09-13T13:10:17.314341+0000","last_clean":"2024-09-13T13:10:17.314341+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:10:17.314341+0000","last_undegraded":"2024-09-13T13:10:17.314341+0000","last_fullsized":"2024-09-13T13:10:17.314341+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:48.579658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:48.579658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":412,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.312957+0000","last_change":"2024-09-13T13:08:26.033729+0000","last_active":"2024-09-13T13:10:17.312957+0000","last_peered":"2024-09-13T13:10:17.312957+0000","last_clean":"2024-09-13T13:10:17.312957+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:10:17.312957+0000","last_undegraded":"2024-09-13T13:10:17.312957+0000","last_fullsized":"2024-09-13T13:10:17.312957+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:26.033663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:26.033663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:46.198638+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":399,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313437+0000","last_change":"2024-09-13T13:07:49.584576+0000","last_active":"2024-09-13T13:10:17.313437+0000","last_peered":"2024-09-13T13:10:17.313437+0000","last_clean":"2024-09-13T13:10:17.313437+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:10:17.313437+0000","last_undegraded":"2024-09-13T13:10:17.313437+0000","last_fullsized":"2024-09-13T13:10:17.313437+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:49.584505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:49.584505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:19.019080+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"246'544","reported_seq":565,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.505091+0000","last_change":"2024-09-13T13:10:09.270525+0000","last_active":"2024-09-13T13:10:18.505091+0000","last_peered":"2024-09-13T13:10:18.505091+0000","last_clean":"2024-09-13T13:10:18.505091+0000","last_became_active":"2024-09-13T13:10:09.270190+0000","last_became_peered":"2024-09-13T13:10:09.270190+0000","last_unstale":"2024-09-13T13:10:18.505091+0000","last_undegraded":"2024-09-13T13:10:18.505091+0000","last_fullsized":"2024-09-13T13:10:18.505091+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:08.505734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35651584,"num_objects":544,"num_object_clones":0,"num_object_copies":1632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":544,"num_write_kb":34816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":423,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308151+0000","last_change":"2024-09-13T13:09:30.614842+0000","last_active":"2024-09-13T13:10:17.308151+0000","last_peered":"2024-09-13T13:10:17.308151+0000","last_clean":"2024-09-13T13:10:17.308151+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:10:17.308151+0000","last_undegraded":"2024-09-13T13:10:17.308151+0000","last_fullsized":"2024-09-13T13:10:17.308151+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:09:30.614748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.614748+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:58.916392+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":411,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.314393+0000","last_change":"2024-09-13T13:05:29.345863+0000","last_active":"2024-09-13T13:10:17.314393+0000","last_peered":"2024-09-13T13:10:17.314393+0000","last_clean":"2024-09-13T13:10:17.314393+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:10:17.314393+0000","last_undegraded":"2024-09-13T13:10:17.314393+0000","last_fullsized":"2024-09-13T13:10:17.314393+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.345796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.345796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:16.704307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":396,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.312890+0000","last_change":"2024-09-13T13:06:42.450669+0000","last_active":"2024-09-13T13:10:17.312890+0000","last_peered":"2024-09-13T13:10:17.312890+0000","last_clean":"2024-09-13T13:10:17.312890+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:10:17.312890+0000","last_undegraded":"2024-09-13T13:10:17.312890+0000","last_fullsized":"2024-09-13T13:10:17.312890+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:42.450598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:42.450598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":383,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.309274+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:10:17.309274+0000","last_peered":"2024-09-13T13:10:17.309274+0000","last_clean":"2024-09-13T13:10:17.309274+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:10:17.309274+0000","last_undegraded":"2024-09-13T13:10:17.309274+0000","last_fullsized":"2024-09-13T13:10:17.309274+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:16.374703+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"246'601","reported_seq":622,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:18.938612+0000","last_change":"2024-09-13T13:10:09.275649+0000","last_active":"2024-09-13T13:10:18.938612+0000","last_peered":"2024-09-13T13:10:18.938612+0000","last_clean":"2024-09-13T13:10:18.938612+0000","last_became_active":"2024-09-13T13:10:09.275320+0000","last_became_peered":"2024-09-13T13:10:09.275320+0000","last_unstale":"2024-09-13T13:10:18.938612+0000","last_undegraded":"2024-09-13T13:10:18.938612+0000","last_fullsized":"2024-09-13T13:10:18.938612+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":601,"log_dups_size":0,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:10.488044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39387136,"num_objects":601,"num_object_clones":0,"num_object_copies":1803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":601,"num_write_kb":38464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":443,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.315435+0000","last_change":"2024-09-13T13:09:33.843654+0000","last_active":"2024-09-13T13:10:17.315435+0000","last_peered":"2024-09-13T13:10:17.315435+0000","last_clean":"2024-09-13T13:10:17.315435+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:10:17.315435+0000","last_undegraded":"2024-09-13T13:10:17.315435+0000","last_fullsized":"2024-09-13T13:10:17.315435+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:09:33.843579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:33.843579+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:40.459268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":423,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.314229+0000","last_change":"2024-09-13T13:06:51.627764+0000","last_active":"2024-09-13T13:10:17.314229+0000","last_peered":"2024-09-13T13:10:17.314229+0000","last_clean":"2024-09-13T13:10:17.314229+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:10:17.314229+0000","last_undegraded":"2024-09-13T13:10:17.314229+0000","last_fullsized":"2024-09-13T13:10:17.314229+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:06:51.627691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.627691+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":532,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313551+0000","last_change":"2024-09-13T13:09:41.018928+0000","last_active":"2024-09-13T13:10:17.313551+0000","last_peered":"2024-09-13T13:10:17.313551+0000","last_clean":"2024-09-13T13:10:17.313551+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:10:17.313551+0000","last_undegraded":"2024-09-13T13:10:17.313551+0000","last_fullsized":"2024-09-13T13:10:17.313551+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:09:41.018848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:41.018848+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:56.940353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":402,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.315687+0000","last_change":"2024-09-13T13:09:30.836048+0000","last_active":"2024-09-13T13:10:17.315687+0000","last_peered":"2024-09-13T13:10:17.315687+0000","last_clean":"2024-09-13T13:10:17.315687+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:10:17.315687+0000","last_undegraded":"2024-09-13T13:10:17.315687+0000","last_fullsized":"2024-09-13T13:10:17.315687+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.836006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.836006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:38.200874+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":377,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T13:10:14.531456+0000","last_change":"2024-09-13T13:07:10.058616+0000","last_active":"2024-09-13T13:10:14.531456+0000","last_peered":"2024-09-13T13:10:14.531456+0000","last_clean":"2024-09-13T13:10:14.531456+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:10:14.531456+0000","last_undegraded":"2024-09-13T13:10:14.531456+0000","last_fullsized":"2024-09-13T13:10:14.531456+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:10.058555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.058555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:47.933640+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":407,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.312863+0000","last_change":"2024-09-13T13:09:37.358795+0000","last_active":"2024-09-13T13:10:17.312863+0000","last_peered":"2024-09-13T13:10:17.312863+0000","last_clean":"2024-09-13T13:10:17.312863+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:10:17.312863+0000","last_undegraded":"2024-09-13T13:10:17.312863+0000","last_fullsized":"2024-09-13T13:10:17.312863+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:09:37.358710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:37.358710+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:38.147999+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":410,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.316621+0000","last_change":"2024-09-13T13:08:21.310761+0000","last_active":"2024-09-13T13:10:17.316621+0000","last_peered":"2024-09-13T13:10:17.316621+0000","last_clean":"2024-09-13T13:10:17.316621+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:10:17.316621+0000","last_undegraded":"2024-09-13T13:10:17.316621+0000","last_fullsized":"2024-09-13T13:10:17.316621+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:21.310681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:21.310681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:11.923454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":417,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313890+0000","last_change":"2024-09-13T13:09:30.959919+0000","last_active":"2024-09-13T13:10:17.313890+0000","last_peered":"2024-09-13T13:10:17.313890+0000","last_clean":"2024-09-13T13:10:17.313890+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:10:17.313890+0000","last_undegraded":"2024-09-13T13:10:17.313890+0000","last_fullsized":"2024-09-13T13:10:17.313890+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.959870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.959870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:45.945701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":413,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.318016+0000","last_change":"2024-09-13T13:09:34.426362+0000","last_active":"2024-09-13T13:10:17.318016+0000","last_peered":"2024-09-13T13:10:17.318016+0000","last_clean":"2024-09-13T13:10:17.318016+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:10:17.318016+0000","last_undegraded":"2024-09-13T13:10:17.318016+0000","last_fullsized":"2024-09-13T13:10:17.318016+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:34.426272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:34.426272+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.871047+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":405,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313725+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:10:17.313725+0000","last_peered":"2024-09-13T13:10:17.313725+0000","last_clean":"2024-09-13T13:10:17.313725+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:10:17.313725+0000","last_undegraded":"2024-09-13T13:10:17.313725+0000","last_fullsized":"2024-09-13T13:10:17.313725+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:43.057636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":394,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.312655+0000","last_change":"2024-09-13T13:09:40.333896+0000","last_active":"2024-09-13T13:10:17.312655+0000","last_peered":"2024-09-13T13:10:17.312655+0000","last_clean":"2024-09-13T13:10:17.312655+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:10:17.312655+0000","last_undegraded":"2024-09-13T13:10:17.312655+0000","last_fullsized":"2024-09-13T13:10:17.312655+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:40.333836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:40.333836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:03.786971+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":407,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313133+0000","last_change":"2024-09-13T13:09:07.378142+0000","last_active":"2024-09-13T13:10:17.313133+0000","last_peered":"2024-09-13T13:10:17.313133+0000","last_clean":"2024-09-13T13:10:17.313133+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:10:17.313133+0000","last_undegraded":"2024-09-13T13:10:17.313133+0000","last_fullsized":"2024-09-13T13:10:17.313133+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:07.378066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:07.378066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:33.759039+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":409,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317275+0000","last_change":"2024-09-13T13:09:48.990167+0000","last_active":"2024-09-13T13:10:17.317275+0000","last_peered":"2024-09-13T13:10:17.317275+0000","last_clean":"2024-09-13T13:10:17.317275+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:10:17.317275+0000","last_undegraded":"2024-09-13T13:10:17.317275+0000","last_fullsized":"2024-09-13T13:10:17.317275+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:16.980190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":401,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.316651+0000","last_change":"2024-09-13T13:06:41.308449+0000","last_active":"2024-09-13T13:10:17.316651+0000","last_peered":"2024-09-13T13:10:17.316651+0000","last_clean":"2024-09-13T13:10:17.316651+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:10:17.316651+0000","last_undegraded":"2024-09-13T13:10:17.316651+0000","last_fullsized":"2024-09-13T13:10:17.316651+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:41.308367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:41.308367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":514,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.310113+0000","last_change":"2024-09-13T13:10:07.230375+0000","last_active":"2024-09-13T13:10:17.310113+0000","last_peered":"2024-09-13T13:10:17.310113+0000","last_clean":"2024-09-13T13:10:17.310113+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:10:17.310113+0000","last_undegraded":"2024-09-13T13:10:17.310113+0000","last_fullsized":"2024-09-13T13:10:17.310113+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:10:07.230276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:07.230276+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:15.266427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":399,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308645+0000","last_change":"2024-09-13T13:07:21.958114+0000","last_active":"2024-09-13T13:10:17.308645+0000","last_peered":"2024-09-13T13:10:17.308645+0000","last_clean":"2024-09-13T13:10:17.308645+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:10:17.308645+0000","last_undegraded":"2024-09-13T13:10:17.308645+0000","last_fullsized":"2024-09-13T13:10:17.308645+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:21.958030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:21.958030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:52.786745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":421,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.311955+0000","last_change":"2024-09-13T13:09:04.635198+0000","last_active":"2024-09-13T13:10:17.311955+0000","last_peered":"2024-09-13T13:10:17.311955+0000","last_clean":"2024-09-13T13:10:17.311955+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:10:17.311955+0000","last_undegraded":"2024-09-13T13:10:17.311955+0000","last_fullsized":"2024-09-13T13:10:17.311955+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:04.599040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:04.599040+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:30.955987+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":394,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.314149+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:10:17.314149+0000","last_peered":"2024-09-13T13:10:17.314149+0000","last_clean":"2024-09-13T13:10:17.314149+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:10:17.314149+0000","last_undegraded":"2024-09-13T13:10:17.314149+0000","last_fullsized":"2024-09-13T13:10:17.314149+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:13.751570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":530,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317002+0000","last_change":"2024-09-13T13:07:08.781005+0000","last_active":"2024-09-13T13:10:17.317002+0000","last_peered":"2024-09-13T13:10:17.317002+0000","last_clean":"2024-09-13T13:10:17.317002+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:10:17.317002+0000","last_undegraded":"2024-09-13T13:10:17.317002+0000","last_fullsized":"2024-09-13T13:10:17.317002+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:07:08.780952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.780952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:54.891948+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":397,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317002+0000","last_change":"2024-09-13T13:09:36.414394+0000","last_active":"2024-09-13T13:10:17.317002+0000","last_peered":"2024-09-13T13:10:17.317002+0000","last_clean":"2024-09-13T13:10:17.317002+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:10:17.317002+0000","last_undegraded":"2024-09-13T13:10:17.317002+0000","last_fullsized":"2024-09-13T13:10:17.317002+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:36.414322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:36.414322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:39.835131+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":429,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308428+0000","last_change":"2024-09-13T13:05:26.186899+0000","last_active":"2024-09-13T13:10:17.308428+0000","last_peered":"2024-09-13T13:10:17.308428+0000","last_clean":"2024-09-13T13:10:17.308428+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:10:17.308428+0000","last_undegraded":"2024-09-13T13:10:17.308428+0000","last_fullsized":"2024-09-13T13:10:17.308428+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:05:26.186830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.186830+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":414,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308424+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:10:17.308424+0000","last_peered":"2024-09-13T13:10:17.308424+0000","last_clean":"2024-09-13T13:10:17.308424+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:10:17.308424+0000","last_undegraded":"2024-09-13T13:10:17.308424+0000","last_fullsized":"2024-09-13T13:10:17.308424+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":403,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.318188+0000","last_change":"2024-09-13T13:09:05.882862+0000","last_active":"2024-09-13T13:10:17.318188+0000","last_peered":"2024-09-13T13:10:17.318188+0000","last_clean":"2024-09-13T13:10:17.318188+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:10:17.318188+0000","last_undegraded":"2024-09-13T13:10:17.318188+0000","last_fullsized":"2024-09-13T13:10:17.318188+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:05.882769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.882769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":398,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313180+0000","last_change":"2024-09-13T13:08:59.585721+0000","last_active":"2024-09-13T13:10:17.313180+0000","last_peered":"2024-09-13T13:10:17.313180+0000","last_clean":"2024-09-13T13:10:17.313180+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:10:17.313180+0000","last_undegraded":"2024-09-13T13:10:17.313180+0000","last_fullsized":"2024-09-13T13:10:17.313180+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:59.585646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.585646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:21.171797+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":366,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.315599+0000","last_change":"2024-09-13T13:10:10.932948+0000","last_active":"2024-09-13T13:10:17.315599+0000","last_peered":"2024-09-13T13:10:17.315599+0000","last_clean":"2024-09-13T13:10:17.315599+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:10:17.315599+0000","last_undegraded":"2024-09-13T13:10:17.315599+0000","last_fullsized":"2024-09-13T13:10:17.315599+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:10.932889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.932889+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:33.657024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":410,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.314278+0000","last_change":"2024-09-13T13:09:07.118545+0000","last_active":"2024-09-13T13:10:17.314278+0000","last_peered":"2024-09-13T13:10:17.314278+0000","last_clean":"2024-09-13T13:10:17.314278+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:10:17.314278+0000","last_undegraded":"2024-09-13T13:10:17.314278+0000","last_fullsized":"2024-09-13T13:10:17.314278+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:07.118467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:07.118467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:20.947652+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":394,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.311745+0000","last_change":"2024-09-13T13:09:32.467362+0000","last_active":"2024-09-13T13:10:17.311745+0000","last_peered":"2024-09-13T13:10:17.311745+0000","last_clean":"2024-09-13T13:10:17.311745+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:10:17.311745+0000","last_undegraded":"2024-09-13T13:10:17.311745+0000","last_fullsized":"2024-09-13T13:10:17.311745+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:32.467151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:32.467151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:55.111614+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":397,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.315248+0000","last_change":"2024-09-13T13:08:56.227109+0000","last_active":"2024-09-13T13:10:17.315248+0000","last_peered":"2024-09-13T13:10:17.315248+0000","last_clean":"2024-09-13T13:10:17.315248+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:10:17.315248+0000","last_undegraded":"2024-09-13T13:10:17.315248+0000","last_fullsized":"2024-09-13T13:10:17.315248+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:56.227027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:56.227027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":420,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313435+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:10:17.313435+0000","last_peered":"2024-09-13T13:10:17.313435+0000","last_clean":"2024-09-13T13:10:17.313435+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:10:17.313435+0000","last_undegraded":"2024-09-13T13:10:17.313435+0000","last_fullsized":"2024-09-13T13:10:17.313435+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":410,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.316650+0000","last_change":"2024-09-13T13:08:50.517587+0000","last_active":"2024-09-13T13:10:17.316650+0000","last_peered":"2024-09-13T13:10:17.316650+0000","last_clean":"2024-09-13T13:10:17.316650+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:10:17.316650+0000","last_undegraded":"2024-09-13T13:10:17.316650+0000","last_fullsized":"2024-09-13T13:10:17.316650+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:50.517526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:50.517526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":413,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313203+0000","last_change":"2024-09-13T13:08:53.215325+0000","last_active":"2024-09-13T13:10:17.313203+0000","last_peered":"2024-09-13T13:10:17.313203+0000","last_clean":"2024-09-13T13:10:17.313203+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:10:17.313203+0000","last_undegraded":"2024-09-13T13:10:17.313203+0000","last_fullsized":"2024-09-13T13:10:17.313203+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.215263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.215263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":393,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.311958+0000","last_change":"2024-09-13T13:07:57.596272+0000","last_active":"2024-09-13T13:10:17.311958+0000","last_peered":"2024-09-13T13:10:17.311958+0000","last_clean":"2024-09-13T13:10:17.311958+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:10:17.311958+0000","last_undegraded":"2024-09-13T13:10:17.311958+0000","last_fullsized":"2024-09-13T13:10:17.311958+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.596097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:53.148768+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":428,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.308986+0000","last_change":"2024-09-13T13:09:39.368581+0000","last_active":"2024-09-13T13:10:17.308986+0000","last_peered":"2024-09-13T13:10:17.308986+0000","last_clean":"2024-09-13T13:10:17.308986+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:10:17.308986+0000","last_undegraded":"2024-09-13T13:10:17.308986+0000","last_fullsized":"2024-09-13T13:10:17.308986+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2024-09-13T13:09:39.368381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:39.368381+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:41.235364+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":417,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317794+0000","last_change":"2024-09-13T13:09:30.487486+0000","last_active":"2024-09-13T13:10:17.317794+0000","last_peered":"2024-09-13T13:10:17.317794+0000","last_clean":"2024-09-13T13:10:17.317794+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:10:17.317794+0000","last_undegraded":"2024-09-13T13:10:17.317794+0000","last_fullsized":"2024-09-13T13:10:17.317794+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.487413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.487413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:53.159481+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":301,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T13:10:14.530371+0000","last_change":"2024-09-13T13:09:27.844085+0000","last_active":"2024-09-13T13:10:14.530371+0000","last_peered":"2024-09-13T13:10:14.530371+0000","last_clean":"2024-09-13T13:10:14.530371+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:10:14.530371+0000","last_undegraded":"2024-09-13T13:10:14.530371+0000","last_fullsized":"2024-09-13T13:10:14.530371+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:27.844003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:27.844003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:31.056528+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":406,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.313520+0000","last_change":"2024-09-13T13:09:31.943644+0000","last_active":"2024-09-13T13:10:17.313520+0000","last_peered":"2024-09-13T13:10:17.313520+0000","last_clean":"2024-09-13T13:10:17.313520+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:10:17.313520+0000","last_undegraded":"2024-09-13T13:10:17.313520+0000","last_fullsized":"2024-09-13T13:10:17.313520+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:31.943568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:31.943568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:49.734605+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":399,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317778+0000","last_change":"2024-09-13T13:08:27.641188+0000","last_active":"2024-09-13T13:10:17.317778+0000","last_peered":"2024-09-13T13:10:17.317778+0000","last_clean":"2024-09-13T13:10:17.317778+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:10:17.317778+0000","last_undegraded":"2024-09-13T13:10:17.317778+0000","last_fullsized":"2024-09-13T13:10:17.317778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:27.641105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:27.641105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":402,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.309375+0000","last_change":"2024-09-13T13:09:35.355561+0000","last_active":"2024-09-13T13:10:17.309375+0000","last_peered":"2024-09-13T13:10:17.309375+0000","last_clean":"2024-09-13T13:10:17.309375+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:10:17.309375+0000","last_undegraded":"2024-09-13T13:10:17.309375+0000","last_fullsized":"2024-09-13T13:10:17.309375+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:35.355369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:35.355369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:52.921712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":396,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T13:10:14.530576+0000","last_change":"2024-09-13T13:06:09.928585+0000","last_active":"2024-09-13T13:10:14.530576+0000","last_peered":"2024-09-13T13:10:14.530576+0000","last_clean":"2024-09-13T13:10:14.530576+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:10:14.530576+0000","last_undegraded":"2024-09-13T13:10:14.530576+0000","last_fullsized":"2024-09-13T13:10:14.530576+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:06:09.928504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.928504+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:43.812961+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":410,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T13:10:17.317156+0000","last_change":"2024-09-13T13:09:38.337230+0000","last_active":"2024-09-13T13:10:17.317156+0000","last_peered":"2024-09-13T13:10:17.317156+0000","last_clean":"2024-09-13T13:10:17.317156+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:10:17.317156+0000","last_undegraded":"2024-09-13T13:10:17.317156+0000","last_fullsized":"2024-09-13T13:10:17.317156+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.337156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.337156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:10:40.248177+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":528351232,"num_objects":8062,"num_object_clones":0,"num_object_copies":24186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8062,"num_write_kb":515968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1452277760,"data_stored":1452277760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8062,"ondisk_log_size":8062,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":549,"num_read_kb":544,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201394,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":469556,"kb_used_data":207904,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93275596,"statfs":{"total":95995035648,"available":95514210304,"internally_reserved":0,"allocated":212893696,"data_stored":212459644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":267909578},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141004,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":259608,"kb_used_data":1412,"kb_used_omap":4,"kb_used_meta":258171,"kb_avail":93485544,"statfs":{"total":95995035648,"available":95729197056,"internally_reserved":0,"allocated":1445888,"data_stored":1032023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":264367789},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691966,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":534312,"kb_used_data":246292,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93210840,"statfs":{"total":95995035648,"available":95447900160,"internally_reserved":0,"allocated":252203008,"data_stored":251779540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920900,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":274684,"kb_used_data":76904,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93470468,"statfs":{"total":95995035648,"available":95713759232,"internally_reserved":0,"allocated":78749696,"data_stored":78305884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":202504646},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117131,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":608448,"kb_used_data":252396,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93136704,"statfs":{"total":95995035648,"available":95371984896,"internally_reserved":0,"allocated":258453504,"data_stored":258006426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364575175},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411470,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":337364,"kb_used_data":75716,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93407788,"statfs":{"total":95995035648,"available":95649574912,"internally_reserved":0,"allocated":77533184,"data_stored":77125440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":267909574},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574997,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546832,"kb_used_data":222716,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93198320,"statfs":{"total":95995035648,"available":95435079680,"internally_reserved":0,"allocated":228061184,"data_stored":227597403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738524,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":416056,"kb_used_data":118308,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93329096,"statfs":{"total":95995035648,"available":95568994304,"internally_reserved":0,"allocated":121147392,"data_stored":120708953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160038912,"data_stored":160038912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":310444032,"data_stored":310444032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95092736,"data_stored":95092736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":279379968,"data_stored":279379968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":85524480,"data_stored":85524480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":263585792,"data_stored":263585792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6881280,"data_stored":6881280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":251330560,"data_stored":251330560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:10:22.430 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-13T13:10:22.435 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:22.475 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:22.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:22.577 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:22.607 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:22.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:22.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:22.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:22.741 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:22.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:22.774 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 7 16 19710 19694 175.81 172.625 0.00346947 0.00568118 2024-09-13T13:10:22.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:22.876 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:22.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:22.977 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:22.999 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:22.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:23.100 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:23.141 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:23.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:23.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:23.243 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:23.281 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:23.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:23.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:23.417 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:23.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:23.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:23.519 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:23.561 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:23.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:23.662 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:23.694 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:23.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:23.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:23 smithi052 ceph-mon[36682]: pgmap v442: 148 pgs: 148 active+clean; 504 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 601 B/s rd, 59 MiB/s wr, 948 op/s 2024-09-13T13:10:23.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:23 smithi052 ceph-mon[29471]: pgmap v442: 148 pgs: 148 active+clean; 504 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 601 B/s rd, 59 MiB/s wr, 948 op/s 2024-09-13T13:10:23.774 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 8 16 22085 22069 172.385 148.438 0.00157509 0.00578266 2024-09-13T13:10:23.796 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:23 smithi167 ceph-mon[33178]: pgmap v442: 148 pgs: 148 active+clean; 504 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 601 B/s rd, 59 MiB/s wr, 948 op/s 2024-09-13T13:10:23.822 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:23.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:23.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:23.925 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:23.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:23.963 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:23.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:24.064 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:24.086 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:24.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:24.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:24.188 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:24.222 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:24.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:24.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:24.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:10:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:10:24.324 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:24.361 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:24.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:24.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:24.463 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:24.495 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:24.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:24.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:24.597 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:24.636 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:24.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:24.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:24.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:24.775 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 9 16 24360 24344 169.028 142.188 0.00572612 0.00590873 2024-09-13T13:10:24.776 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:24.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:24.870 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:24.878 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:24.899 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T13:10:24.910 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:24.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:24.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:25.012 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:25.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:25.071 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:25.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:25.172 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:25.198 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:25.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:25.300 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:25.333 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:25.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:25.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:25.435 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:25.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:25 smithi052 ceph-mon[29471]: pgmap v443: 148 pgs: 148 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 118 MiB/s wr, 1.88k op/s 2024-09-13T13:10:25.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:25.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:25 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:10:25.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:25 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:10:25.436 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:25 smithi052 ceph-mon[36682]: pgmap v443: 148 pgs: 148 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 118 MiB/s wr, 1.88k op/s 2024-09-13T13:10:25.436 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:25.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:25 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:10:25.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:25 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:10:25.464 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:25.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:25.512 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:25.526 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:10:25.526 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:10:25.538 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:25.565 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:10:25.565 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:10:25.566 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:25.596 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:25.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:25.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:25.602 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:10:25.602 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:10:25.640 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:10:25.640 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:10:25.676 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:10:25.676 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:10:25.698 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:25.711 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:10:25.711 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:10:25.727 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:25.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:25.745 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:10:25.745 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:10:25.775 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 10 16 26391 26375 164.819 126.938 0.00905743 0.00606099 2024-09-13T13:10:25.780 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:10:25.780 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:10:25.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:25 smithi167 ceph-mon[33178]: pgmap v443: 148 pgs: 148 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 118 MiB/s wr, 1.88k op/s 2024-09-13T13:10:25.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:25 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:10:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:25 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:10:25.829 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:25.858 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:25.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:25.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:25.993 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:25.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:26.095 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:26.125 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:26.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:26.227 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:26.268 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:26.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:26.366 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:10:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:10:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:10:26.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:26.401 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:26.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:26.502 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:26.542 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:26.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:26.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:26.628 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:26 smithi052 ceph-mon[36682]: osdmap e247: 8 total, 8 up, 8 in 2024-09-13T13:10:26.634 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:26 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2630154069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:26.634 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:26 smithi052 ceph-mon[29471]: osdmap e247: 8 total, 8 up, 8 in 2024-09-13T13:10:26.634 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:26 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2630154069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:26.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:26.684 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:26.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:26.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:26.774 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 11 16 28439 28423 161.471 128 0.00610181 0.0061878 2024-09-13T13:10:26.794 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:26.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:26 smithi167 ceph-mon[33178]: osdmap e247: 8 total, 8 up, 8 in 2024-09-13T13:10:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:26 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2630154069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:26.832 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:26.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:26.933 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:26.968 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:10:26.968 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:10:26.968 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:10:26.968 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:10:26.968 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.968 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:95502 7.e 7:73e7d3a1:::benchmark_data_smithi052_71865_object95501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:10:26.968 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.170950+0000", 2024-09-13T13:10:26.969 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 420.793904107, 2024-09-13T13:10:26.969 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10520948400000001, 2024-09-13T13:10:26.969 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:26.969 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:26.969 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:26.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:26.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:26.969 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:26.969 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95502 2024-09-13T13:10:26.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.969 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:26.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170950+0000", 2024-09-13T13:10:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170950+0000", 2024-09-13T13:10:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:26.970 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170952+0000", 2024-09-13T13:10:26.971 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.872e-06 2024-09-13T13:10:26.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.971 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:26.971 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170974+0000", 2024-09-13T13:10:26.971 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:10:26.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.971 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:26.971 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170982+0000", 2024-09-13T13:10:26.971 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6349999999999995e-06 2024-09-13T13:10:26.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.972 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:26.972 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170989+0000", 2024-09-13T13:10:26.972 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1619999999999996e-06 2024-09-13T13:10:26.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.972 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:26.972 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.171116+0000", 2024-09-13T13:10:26.972 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000127293 2024-09-13T13:10:26.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.973 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:26.973 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.171296+0000", 2024-09-13T13:10:26.973 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000180278 2024-09-13T13:10:26.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.973 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:26.973 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.171419+0000", 2024-09-13T13:10:26.973 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012266299999999999 2024-09-13T13:10:26.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.973 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:26.973 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.175039+0000", 2024-09-13T13:10:26.974 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036198469999999998 2024-09-13T13:10:26.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.974 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.974 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:26.974 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.175665+0000", 2024-09-13T13:10:26.974 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00062585400000000002 2024-09-13T13:10:26.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.974 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.974 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:26.975 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276069+0000", 2024-09-13T13:10:26.975 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10040423499999999 2024-09-13T13:10:26.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.975 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:26.975 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276119+0000", 2024-09-13T13:10:26.975 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0290999999999997e-05 2024-09-13T13:10:26.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.975 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:26.975 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276160+0000", 2024-09-13T13:10:26.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0516999999999998e-05 2024-09-13T13:10:26.976 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:26.976 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:26.976 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:26.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.976 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:95513 7.e 7:760c2885:::benchmark_data_smithi052_71865_object95512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:10:26.976 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.177025+0000", 2024-09-13T13:10:26.976 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 420.78782897600001, 2024-09-13T13:10:26.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.099458682000000007, 2024-09-13T13:10:26.976 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:26.976 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:26.977 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:26.977 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:26.977 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:26.977 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:26.977 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95513 2024-09-13T13:10:26.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.977 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:26.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.977 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:26.977 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177025+0000", 2024-09-13T13:10:26.977 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:26.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.978 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:26.978 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177025+0000", 2024-09-13T13:10:26.978 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:26.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.978 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:26.978 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177027+0000", 2024-09-13T13:10:26.978 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8109999999999999e-06 2024-09-13T13:10:26.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.978 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:26.978 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177059+0000", 2024-09-13T13:10:26.979 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2125000000000002e-05 2024-09-13T13:10:26.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.979 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.979 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:26.979 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177078+0000", 2024-09-13T13:10:26.979 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9239999999999999e-05 2024-09-13T13:10:26.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.979 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.980 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:26.980 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177085+0000", 2024-09-13T13:10:26.980 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9460000000000001e-06 2024-09-13T13:10:26.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.980 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.980 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:26.980 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177194+0000", 2024-09-13T13:10:26.980 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010910500000000001 2024-09-13T13:10:26.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.981 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:26.981 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177399+0000", 2024-09-13T13:10:26.981 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000205019 2024-09-13T13:10:26.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.981 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:26.981 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177525+0000", 2024-09-13T13:10:26.981 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000125969 2024-09-13T13:10:26.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.981 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276198+0000", 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098672383000000002 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276236+0000", 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7988000000000003e-05 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276248+0000", 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1858000000000001e-05 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276284+0000", 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6350000000000003e-05 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276484+0000", 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019988800000000001 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.983 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95494 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:10:26.984 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.177847+0000", 2024-09-13T13:10:26.984 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 420.78700684199998, 2024-09-13T13:10:26.984 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098173131999999996, 2024-09-13T13:10:26.984 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:26.984 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:26.984 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:10:26.984 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:26.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.984 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:26.984 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177847+0000", 2024-09-13T13:10:26.985 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:26.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.985 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.985 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:26.985 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177847+0000", 2024-09-13T13:10:26.985 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:26.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.985 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.985 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177849+0000", 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3179999999999999e-06 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177853+0000", 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1799999999999998e-06 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177862+0000", 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1390000000000001e-06 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.986 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177868+0000", 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4119999999999998e-06 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180221+0000", 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023525130000000001 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180227+0000", 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4930000000000003e-06 2024-09-13T13:10:26.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276020+0000", 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095793077000000004 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95502 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.180124+0000", 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 420.78473015399999, 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.09604087, 2024-09-13T13:10:26.988 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:26.989 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:26.989 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:10:26.989 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:26.989 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.989 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:26.989 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180124+0000", 2024-09-13T13:10:26.989 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:26.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.989 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.989 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:26.989 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180124+0000", 2024-09-13T13:10:26.989 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:26.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180125+0000", 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.311e-06 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180130+0000", 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7500000000000003e-06 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180140+0000", 2024-09-13T13:10:26.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9310000000000003e-06 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180147+0000", 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0600000000000002e-06 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276055+0000", 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095907611000000004 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:26.991 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276063+0000", 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9719999999999992e-06 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276165+0000", 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102235 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95513 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.180192+0000", 2024-09-13T13:10:26.992 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 420.78466216499999, 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.096011869999999999, 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180192+0000", 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180192+0000", 2024-09-13T13:10:26.993 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180193+0000", 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.032e-06 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180197+0000", 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7079999999999999e-06 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:26.994 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180204+0000", 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2819999999999999e-06 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180209+0000", 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7330000000000003e-06 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276187+0000", 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095978049999999995 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.995 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276193+0000", 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1480000000000003e-06 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276204+0000", 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0917000000000001e-05 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95513 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:10:26.996 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.180247+0000", 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 420.78460688500002, 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095994805000000002, 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180247+0000", 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:26.997 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180247+0000", 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180248+0000", 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.004e-06 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180252+0000", 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7890000000000001e-06 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:26.998 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180261+0000", 2024-09-13T13:10:26.999 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5089999999999993e-06 2024-09-13T13:10:26.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.999 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.999 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:26.999 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180267+0000", 2024-09-13T13:10:26.999 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0079999999999999e-06 2024-09-13T13:10:26.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:26.999 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:26.999 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:27.000 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:27.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:27.001 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276226+0000", 2024-09-13T13:10:27.001 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095959424000000001 2024-09-13T13:10:27.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.001 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.001 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.001 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276232+0000", 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6189999999999999e-06 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276242+0000", 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0451999999999999e-05 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:95658 7.e 7:79f5f817:::benchmark_data_smithi052_71865_object95657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.394482+0000", 2024-09-13T13:10:27.002 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 420.570372491, 2024-09-13T13:10:27.003 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.081968365000000001, 2024-09-13T13:10:27.003 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.003 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.003 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.003 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.003 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.003 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.003 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95658 2024-09-13T13:10:27.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.003 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.003 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.003 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394482+0000", 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394482+0000", 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394484+0000", 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4660000000000002e-06 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.004 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394512+0000", 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7447999999999999e-05 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394525+0000", 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3064000000000001e-05 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.394533+0000", 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5930000000000006e-06 2024-09-13T13:10:27.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.399009+0000", 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0044759159999999999 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.399121+0000", 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011148400000000001 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.399244+0000", 2024-09-13T13:10:27.006 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000123192 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.402074+0000", 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0028305230000000002 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.403012+0000", 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00093742399999999996 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.007 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.475889+0000", 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.072877493000000002 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.476237+0000", 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00034722900000000001 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.476450+0000", 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021353299999999999 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.008 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.009 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.009 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:96740 7.e 7:7b47086e:::benchmark_data_smithi052_71865_object96739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:10:27.009 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:27.478436+0000", 2024-09-13T13:10:27.009 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.48641779600001, 2024-09-13T13:10:27.009 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.082652384999999995, 2024-09-13T13:10:27.009 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.009 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.009 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.009 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.009 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.009 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.009 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 96740 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478436+0000", 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478436+0000", 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.010 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.011 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478439+0000", 2024-09-13T13:10:27.011 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0990000000000001e-06 2024-09-13T13:10:27.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.011 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.011 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.011 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478466+0000", 2024-09-13T13:10:27.011 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7987000000000002e-05 2024-09-13T13:10:27.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.011 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.011 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.011 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478477+0000", 2024-09-13T13:10:27.011 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0587999999999999e-05 2024-09-13T13:10:27.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.478484+0000", 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2799999999999998e-06 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.479227+0000", 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00074284600000000002 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.479355+0000", 2024-09-13T13:10:27.012 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012828000000000001 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.479490+0000", 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000134859 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.483095+0000", 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036047169999999999 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.483243+0000", 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014764499999999999 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.560910+0000", 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.077667016000000005 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.561046+0000", 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013630099999999999 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:27.561089+0000", 2024-09-13T13:10:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2766999999999999e-05 2024-09-13T13:10:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:118717 7.e 7:7eac196e:::benchmark_data_smithi052_71865_object118716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:10:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:10:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 397.40463552699998, 2024-09-13T13:10:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.120995147, 2024-09-13T13:10:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 118717 2024-09-13T13:10:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:10:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560220+0000", 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4200000000000004e-07 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560244+0000", 2024-09-13T13:10:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4354000000000001e-05 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560252+0000", 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5189999999999992e-06 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560259+0000", 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.302e-06 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560301+0000", 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2318999999999999e-05 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560415+0000", 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114269 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560572+0000", 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000156217 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.564231+0000", 2024-09-13T13:10:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036597489999999999 2024-09-13T13:10:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.565180+0000", 2024-09-13T13:10:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00094882300000000001 2024-09-13T13:10:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.680844+0000", 2024-09-13T13:10:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.115664164 2024-09-13T13:10:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.681157+0000", 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00031297600000000002 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.681214+0000", 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6512999999999999e-05 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120031 7.e 7:704b0327:::benchmark_data_smithi052_71865_object120030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 395.94404827699998, 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11824775799999999, 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120031 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020808+0000", 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0439999999999998e-06 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020838+0000", 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9791e-05 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020856+0000", 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8664999999999998e-05 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020866+0000", 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3670000000000003e-06 2024-09-13T13:10:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021203+0000", 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033735200000000002 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021578+0000", 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00037465399999999998 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022015+0000", 2024-09-13T13:10:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000437176 2024-09-13T13:10:27.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.026 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.026 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.026523+0000", 2024-09-13T13:10:27.026 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045084249999999999 2024-09-13T13:10:27.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.026 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.026 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.027199+0000", 2024-09-13T13:10:27.026 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00067598400000000002 2024-09-13T13:10:27.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.026 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138915+0000", 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11171563499999999 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139008+0000", 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3329e-05 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139054+0000", 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5336000000000001e-05 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.027 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.028 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.028 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120050 7.e 7:72296fb6:::benchmark_data_smithi052_71865_object120049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:10:27.028 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:10:27.028 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 395.93200023200001, 2024-09-13T13:10:27.028 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12504636999999999, 2024-09-13T13:10:27.028 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.028 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.028 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.028 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.029 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.029 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.029 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120050 2024-09-13T13:10:27.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.029 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.029 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.029 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032857+0000", 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1140000000000002e-06 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032888+0000", 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1115999999999998e-05 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032905+0000", 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6452000000000001e-05 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032915+0000", 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8889999999999997e-06 2024-09-13T13:10:27.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033665+0000", 2024-09-13T13:10:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00075015300000000004 2024-09-13T13:10:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033782+0000", 2024-09-13T13:10:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117642 2024-09-13T13:10:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033916+0000", 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013400799999999999 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.037705+0000", 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037884289999999998 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.037719+0000", 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4034e-05 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157684+0000", 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.119964979 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157805+0000", 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120715 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157900+0000", 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5839000000000002e-05 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.034 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.035 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120054 7.e 7:7dad863d:::benchmark_data_smithi052_71865_object120053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:10:27.035 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:10:27.035 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 395.92648495600002, 2024-09-13T13:10:27.035 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11988120300000001, 2024-09-13T13:10:27.035 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.035 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.035 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.035 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.035 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.035 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.035 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120054 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038370+0000", 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.051e-06 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038395+0000", 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4224999999999999e-05 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038403+0000", 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3399999999999998e-06 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038409+0000", 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3160000000000002e-06 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038462+0000", 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3012e-05 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038579+0000", 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116483 2024-09-13T13:10:27.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038720+0000", 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014139099999999999 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.042075+0000", 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033552830000000001 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.042095+0000", 2024-09-13T13:10:27.039 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9279999999999998e-05 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158031+0000", 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11593616299999999 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158169+0000", 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013835799999999999 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.040 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158250+0000", 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1301e-05 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:135717 7.e 7:74ed6606:::benchmark_data_smithi052_71865_object14616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:09.110240+0000", 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 377.85461389199997, 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.076080909000000002, 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.041 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 135717 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110240+0000", 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110240+0000", 2024-09-13T13:10:27.042 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.043 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110241+0000", 2024-09-13T13:10:27.043 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:10:27.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.043 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110243+0000", 2024-09-13T13:10:27.043 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4090000000000001e-06 2024-09-13T13:10:27.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.044 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110248+0000", 2024-09-13T13:10:27.044 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1250000000000001e-06 2024-09-13T13:10:27.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.044 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.044 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.044 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.110253+0000", 2024-09-13T13:10:27.044 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8350000000000003e-06 2024-09-13T13:10:27.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.044 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.044 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.044 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.113974+0000", 2024-09-13T13:10:27.044 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003720737 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.114066+0000", 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1833e-05 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.114179+0000", 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112695 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.045 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.116490+0000", 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023109210000000001 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.116845+0000", 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035593400000000002 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.186206+0000", 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.069360755999999996 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.046 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.186246+0000", 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9888000000000002e-05 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:09.186321+0000", 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5136999999999996e-05 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151582 7.e 7:7a3dbf39:::benchmark_data_smithi052_71865_object30481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:10:27.047 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.170441+0000", 2024-09-13T13:10:27.048 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 357.79441317200002, 2024-09-13T13:10:27.048 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.091679105999999996, 2024-09-13T13:10:27.048 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.048 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.048 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.048 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.048 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.048 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.048 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151582 2024-09-13T13:10:27.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.048 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.048 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.049 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170441+0000", 2024-09-13T13:10:27.049 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.049 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.049 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170441+0000", 2024-09-13T13:10:27.049 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.049 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.049 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170442+0000", 2024-09-13T13:10:27.049 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5000000000000002e-07 2024-09-13T13:10:27.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.050 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.050 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170445+0000", 2024-09-13T13:10:27.050 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.88e-06 2024-09-13T13:10:27.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.050 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.050 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170450+0000", 2024-09-13T13:10:27.050 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3419999999999998e-06 2024-09-13T13:10:27.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.052 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.052 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170455+0000", 2024-09-13T13:10:27.052 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.096e-06 2024-09-13T13:10:27.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.052 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.052 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174840+0000", 2024-09-13T13:10:27.052 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0043852609999999997 2024-09-13T13:10:27.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.052 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.052 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174948+0000", 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010804 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.175063+0000", 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0001145 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.178533+0000", 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003470065 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.178591+0000", 2024-09-13T13:10:27.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8149000000000002e-05 2024-09-13T13:10:27.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.054 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261725+0000", 2024-09-13T13:10:27.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.083133548000000002 2024-09-13T13:10:27.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.054 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261955+0000", 2024-09-13T13:10:27.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00023013299999999999 2024-09-13T13:10:27.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.262120+0000", 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016544199999999999 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151596 7.e 7:74fc7c8c:::benchmark_data_smithi052_71865_object30495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.187262+0000", 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 357.77759198699999, 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.084009381999999994, 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.055 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151596 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187262+0000", 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.057 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187262+0000", 2024-09-13T13:10:27.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.057 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.057 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187264+0000", 2024-09-13T13:10:27.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2199999999999999e-06 2024-09-13T13:10:27.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.057 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.057 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187273+0000", 2024-09-13T13:10:27.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4500000000000004e-06 2024-09-13T13:10:27.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.058 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187285+0000", 2024-09-13T13:10:27.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2446000000000001e-05 2024-09-13T13:10:27.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.058 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187293+0000", 2024-09-13T13:10:27.059 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4000000000000003e-06 2024-09-13T13:10:27.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.059 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.059 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187360+0000", 2024-09-13T13:10:27.059 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6908000000000004e-05 2024-09-13T13:10:27.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.059 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.059 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187487+0000", 2024-09-13T13:10:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000126859 2024-09-13T13:10:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.187631+0000", 2024-09-13T13:10:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014448999999999999 2024-09-13T13:10:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193046+0000", 2024-09-13T13:10:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.005415352 2024-09-13T13:10:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193098+0000", 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1375999999999997e-05 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.271006+0000", 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.077907893000000006 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.271225+0000", 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021893800000000001 2024-09-13T13:10:27.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.271272+0000", 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7049999999999998e-05 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151600 7.e 7:7fe6ca51:::benchmark_data_smithi052_71865_object30499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.189256+0000", 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 357.77559854399999, 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.086529871999999994, 2024-09-13T13:10:27.062 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151600 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189256+0000", 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189256+0000", 2024-09-13T13:10:27.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189257+0000", 2024-09-13T13:10:27.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7209999999999999e-06 2024-09-13T13:10:27.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189263+0000", 2024-09-13T13:10:27.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3580000000000002e-06 2024-09-13T13:10:27.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189275+0000", 2024-09-13T13:10:27.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1973e-05 2024-09-13T13:10:27.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189283+0000", 2024-09-13T13:10:27.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9039999999999992e-06 2024-09-13T13:10:27.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.190358+0000", 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010753099999999999 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.190465+0000", 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010736800000000001 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.190557+0000", 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193101+0000", 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025442569999999999 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193130+0000", 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8728999999999999e-05 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.275647+0000", 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.082517141000000002 2024-09-13T13:10:27.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.068 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.275742+0000", 2024-09-13T13:10:27.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5156999999999997e-05 2024-09-13T13:10:27.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.068 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.275786+0000", 2024-09-13T13:10:27.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3047000000000003e-05 2024-09-13T13:10:27.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.068 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.069 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:153862 7.e 7:797e5c7a:::benchmark_data_smithi052_71865_object32761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:10:27.069 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:10:27.069 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 354.70004748500003, 2024-09-13T13:10:27.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.106314172, 2024-09-13T13:10:27.069 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.069 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.069 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.069 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.069 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.069 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.069 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 153862 2024-09-13T13:10:27.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264809+0000", 2024-09-13T13:10:27.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.148e-06 2024-09-13T13:10:27.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264815+0000", 2024-09-13T13:10:27.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7610000000000004e-06 2024-09-13T13:10:27.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264826+0000", 2024-09-13T13:10:27.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1215e-05 2024-09-13T13:10:27.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264833+0000", 2024-09-13T13:10:27.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0090000000000001e-06 2024-09-13T13:10:27.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264885+0000", 2024-09-13T13:10:27.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1845000000000003e-05 2024-09-13T13:10:27.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.265001+0000", 2024-09-13T13:10:27.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116795 2024-09-13T13:10:27.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:27.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.265104+0000", 2024-09-13T13:10:27.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102979 2024-09-13T13:10:27.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.268877+0000", 2024-09-13T13:10:27.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037720779999999999 2024-09-13T13:10:27.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.269086+0000", 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000209829 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.370873+0000", 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10178622700000001 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.371063+0000", 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000190349 2024-09-13T13:10:27.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.371121+0000", 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7936999999999998e-05 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:153884 7.8 7:1006e86e:::benchmark_data_smithi052_71865_object32783:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:32.288977+0000", 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 354.67587671899997, 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.086944270000000004, 2024-09-13T13:10:27.075 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 153884 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288977+0000", 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288977+0000", 2024-09-13T13:10:27.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288979+0000", 2024-09-13T13:10:27.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9e-06 2024-09-13T13:10:27.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288985+0000", 2024-09-13T13:10:27.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4580000000000001e-06 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288997+0000", 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2004999999999999e-05 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.289005+0000", 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0069999999999997e-06 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.289044+0000", 2024-09-13T13:10:27.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9205000000000002e-05 2024-09-13T13:10:27.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.289111+0000", 2024-09-13T13:10:27.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6952000000000005e-05 2024-09-13T13:10:27.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:10:27.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.359719+0000", 2024-09-13T13:10:27.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.070607721999999998 2024-09-13T13:10:27.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.363436+0000", 2024-09-13T13:10:27.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037171370000000001 2024-09-13T13:10:27.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375313+0000", 2024-09-13T13:10:27.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011876639 2024-09-13T13:10:27.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375551+0000", 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000238939 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375761+0000", 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021001599999999999 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375922+0000", 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016029 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.081 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:153886 7.e 7:718ea587:::benchmark_data_smithi052_71865_object32785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:32.293022+0000", 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 354.67183235599998, 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.081446456, 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.082 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 153886 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.293022+0000", 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.293022+0000", 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.293025+0000", 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7580000000000001e-06 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.293032+0000", 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7619999999999995e-06 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.357646+0000", 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.064613241000000002 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.357665+0000", 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9284999999999999e-05 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.358627+0000", 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00096251400000000003 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.358770+0000", 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014308699999999999 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.358905+0000", 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000134785 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.362681+0000", 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037752800000000002 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.364635+0000", 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019547779999999999 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.374309+0000", 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0096736170000000007 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.374416+0000", 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107389 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.374468+0000", 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1959999999999997e-05 2024-09-13T13:10:27.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:199092 7.e 7:7925136c:::benchmark_data_smithi052_71865_object77991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:05:25.233281+0000", 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 301.73157339199997, 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.076015174000000005, 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:27.088 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 199092 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233281+0000", 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233281+0000", 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233282+0000", 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2929999999999999e-06 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233287+0000", 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0039999999999999e-06 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233297+0000", 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8509999999999994e-06 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233304+0000", 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6680000000000004e-06 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233364+0000", 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0167999999999997e-05 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233462+0000", 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7813999999999996e-05 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:27.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.233561+0000", 2024-09-13T13:10:27.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9557999999999998e-05 2024-09-13T13:10:27.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.236795+0000", 2024-09-13T13:10:27.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0032333890000000001 2024-09-13T13:10:27.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:27.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.236836+0000", 2024-09-13T13:10:27.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1826000000000002e-05 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.309149+0000", 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.072312716999999999 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.309250+0000", 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010125599999999999 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.309296+0000", 2024-09-13T13:10:27.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5630000000000002e-05 2024-09-13T13:10:27.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.094 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:27.094 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:27.094 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:10:27.101 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:27.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:27.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:27.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:27.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:27.257 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:27.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:27.358 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:27.385 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:10:27.397 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:27.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:27.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:27.431 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:10:27.431 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T13:10:27.499 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:27.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:27.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:27.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:27.632 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:27.662 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:27.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:27.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:27 smithi052 ceph-mon[36682]: pgmap v445: 148 pgs: 148 active+clean; 1.3 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 156 MiB/s wr, 2.50k op/s 2024-09-13T13:10:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:27 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3809332582' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:10:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:27 smithi052 ceph-mon[36682]: from='client.25678 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:10:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:27 smithi052 ceph-mon[29471]: pgmap v445: 148 pgs: 148 active+clean; 1.3 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 156 MiB/s wr, 2.50k op/s 2024-09-13T13:10:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:27 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3809332582' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:10:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:27 smithi052 ceph-mon[29471]: from='client.25678 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:10:27.763 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:27.776 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 12 16 30367 30351 158.036 120.5 0.0102755 0.00632106 2024-09-13T13:10:27.817 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:27.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:27.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:27.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:27 smithi167 ceph-mon[33178]: pgmap v445: 148 pgs: 148 active+clean; 1.3 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 156 MiB/s wr, 2.50k op/s 2024-09-13T13:10:27.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:27 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3809332582' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:10:27.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:27 smithi167 ceph-mon[33178]: from='client.25678 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:10:27.919 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:27.953 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:27.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:27.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:28.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:28.087 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:28.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:28.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:28.189 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:28.235 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:28.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:28.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:28.337 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:28.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:28.373 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:28.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:28.474 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:28.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:28 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:28.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:28 smithi052 ceph-mon[36682]: from='client.25678 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:10:28.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:28 smithi052 ceph-mon[36682]: osdmap e248: 8 total, 8 up, 8 in 2024-09-13T13:10:28.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:28 smithi052 ceph-mon[36682]: 3.d scrub starts 2024-09-13T13:10:28.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:28 smithi052 ceph-mon[36682]: 3.d scrub ok 2024-09-13T13:10:28.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:28 smithi052 ceph-mon[36682]: osdmap e249: 8 total, 8 up, 8 in 2024-09-13T13:10:28.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:28 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:28.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:28 smithi052 ceph-mon[29471]: from='client.25678 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:10:28.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:28 smithi052 ceph-mon[29471]: osdmap e248: 8 total, 8 up, 8 in 2024-09-13T13:10:28.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:28 smithi052 ceph-mon[29471]: 3.d scrub starts 2024-09-13T13:10:28.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:28 smithi052 ceph-mon[29471]: 3.d scrub ok 2024-09-13T13:10:28.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:28 smithi052 ceph-mon[29471]: osdmap e249: 8 total, 8 up, 8 in 2024-09-13T13:10:28.505 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:28.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:28.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:28.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:28.633 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:28.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:28.735 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:28.775 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:28.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:28.776 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 13 15 32440 32425 155.85 129.625 0.00466284 0.00641151 2024-09-13T13:10:28.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:28 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:28.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:28 smithi167 ceph-mon[33178]: from='client.25678 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:10:28.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:28 smithi167 ceph-mon[33178]: osdmap e248: 8 total, 8 up, 8 in 2024-09-13T13:10:28.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:28 smithi167 ceph-mon[33178]: 3.d scrub starts 2024-09-13T13:10:28.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:28 smithi167 ceph-mon[33178]: 3.d scrub ok 2024-09-13T13:10:28.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:28 smithi167 ceph-mon[33178]: osdmap e249: 8 total, 8 up, 8 in 2024-09-13T13:10:28.876 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:28.907 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:28.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:28.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:28.979 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T13:10:29.009 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:29.042 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:29.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:29.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:29.144 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:29.174 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:29.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:29.276 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:29.319 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:29.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:29.421 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:29.454 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:29.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:29.556 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:29.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:29.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:29.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:29.591 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:29.685 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:29.728 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:29.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:29.732 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:29 smithi052 ceph-mon[36682]: pgmap v447: 148 pgs: 148 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-13T13:10:29.732 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:29 smithi052 ceph-mon[36682]: 6.1e scrub starts 2024-09-13T13:10:29.732 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:29 smithi052 ceph-mon[36682]: 6.1e scrub ok 2024-09-13T13:10:29.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:29 smithi052 ceph-mon[29471]: pgmap v447: 148 pgs: 148 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-13T13:10:29.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:29 smithi052 ceph-mon[29471]: 6.1e scrub starts 2024-09-13T13:10:29.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:29 smithi052 ceph-mon[29471]: 6.1e scrub ok 2024-09-13T13:10:29.778 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 14 16 34471 34455 153.772 126.875 0.010036 0.00649755 2024-09-13T13:10:29.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:29 smithi167 ceph-mon[33178]: pgmap v447: 148 pgs: 148 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-13T13:10:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:29 smithi167 ceph-mon[33178]: 6.1e scrub starts 2024-09-13T13:10:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:29 smithi167 ceph-mon[33178]: 6.1e scrub ok 2024-09-13T13:10:29.831 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:29.854 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:10:29.875 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:29.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:29.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:29.978 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:30.012 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:30.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:30.114 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:30.159 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:30.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:30.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:30.261 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:30.305 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:30.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:30.407 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:30.438 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:30.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:30.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:30.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:30.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:30.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:30.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:30.706 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:30.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:30.777 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 15 16 36431 36415 151.687 122.5 0.00744784 0.00658732 2024-09-13T13:10:30.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:30.840 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:30.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:30.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:30.942 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:30.967 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:30.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:31.069 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:31.105 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:31.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:31.206 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:31.227 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:31.236 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:31.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:31.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:31.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:31.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:31.464 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[36682]: 4.7 scrub starts 2024-09-13T13:10:31.464 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[36682]: pgmap v449: 148 pgs: 148 active+clean; 1.9 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 135 MiB/s wr, 2.15k op/s 2024-09-13T13:10:31.464 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[36682]: 4.7 scrub ok 2024-09-13T13:10:31.465 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[36682]: 3.8 scrub starts 2024-09-13T13:10:31.465 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[36682]: 3.8 scrub ok 2024-09-13T13:10:31.465 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[36682]: 4.e scrub starts 2024-09-13T13:10:31.465 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[36682]: 4.e scrub ok 2024-09-13T13:10:31.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[29471]: 4.7 scrub starts 2024-09-13T13:10:31.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[29471]: pgmap v449: 148 pgs: 148 active+clean; 1.9 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 135 MiB/s wr, 2.15k op/s 2024-09-13T13:10:31.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[29471]: 4.7 scrub ok 2024-09-13T13:10:31.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[29471]: 3.8 scrub starts 2024-09-13T13:10:31.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[29471]: 3.8 scrub ok 2024-09-13T13:10:31.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[29471]: 4.e scrub starts 2024-09-13T13:10:31.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:31 smithi052 ceph-mon[29471]: 4.e scrub ok 2024-09-13T13:10:31.471 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:31.500 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:31.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:31.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:31.589 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:10:31.602 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:31.631 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:31.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:31.732 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:31.779 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 16 15 38147 38132 148.897 107.312 0.0111579 0.00670891 2024-09-13T13:10:31.805 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:31.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:31.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:31 smithi167 ceph-mon[33178]: 4.7 scrub starts 2024-09-13T13:10:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:31 smithi167 ceph-mon[33178]: pgmap v449: 148 pgs: 148 active+clean; 1.9 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 135 MiB/s wr, 2.15k op/s 2024-09-13T13:10:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:31 smithi167 ceph-mon[33178]: 4.7 scrub ok 2024-09-13T13:10:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:31 smithi167 ceph-mon[33178]: 3.8 scrub starts 2024-09-13T13:10:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:31 smithi167 ceph-mon[33178]: 3.8 scrub ok 2024-09-13T13:10:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:31 smithi167 ceph-mon[33178]: 4.e scrub starts 2024-09-13T13:10:31.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:31 smithi167 ceph-mon[33178]: 4.e scrub ok 2024-09-13T13:10:31.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:31.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:31.943 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:31.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:32.044 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:32.093 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:32.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:32.195 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:32.238 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:32.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:32.286 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:10:32.286 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:10:32.286 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.286 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:38764 9.2 9:4d245939:::benchmark_data_smithi052_128427_object38763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-13T13:10:32.286 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:10:32.162987+0000", 2024-09-13T13:10:32.286 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0097979899999999995, 2024-09-13T13:10:32.286 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0098306520000000005, 2024-09-13T13:10:32.286 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:32.286 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:32.286 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:10:32.286 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:32.287 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:10:32.287 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:10:32.287 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 38764 2024-09-13T13:10:32.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.287 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:32.287 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.287 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:32.287 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.162987+0000", 2024-09-13T13:10:32.287 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.287 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.287 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:32.287 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.162987+0000", 2024-09-13T13:10:32.288 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.288 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.288 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:32.288 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.162987+0000", 2024-09-13T13:10:32.288 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8299999999999996e-07 2024-09-13T13:10:32.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.288 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.288 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:32.288 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.163011+0000", 2024-09-13T13:10:32.288 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3439999999999999e-05 2024-09-13T13:10:32.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.288 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.289 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:32.289 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.163017+0000", 2024-09-13T13:10:32.289 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3409999999999999e-06 2024-09-13T13:10:32.289 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.289 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.289 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:32.289 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.163022+0000", 2024-09-13T13:10:32.289 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1449999999999999e-06 2024-09-13T13:10:32.289 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.289 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.289 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:32.289 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.163122+0000", 2024-09-13T13:10:32.290 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9734000000000004e-05 2024-09-13T13:10:32.290 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.290 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.290 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:32.290 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.163252+0000", 2024-09-13T13:10:32.290 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012975500000000001 2024-09-13T13:10:32.290 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.290 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.290 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:10:32.290 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.163407+0000", 2024-09-13T13:10:32.290 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015542099999999999 2024-09-13T13:10:32.290 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.290 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.291 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:32.291 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.166510+0000", 2024-09-13T13:10:32.291 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031029249999999999 2024-09-13T13:10:32.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.291 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.291 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:32.291 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168391+0000", 2024-09-13T13:10:32.291 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001881169 2024-09-13T13:10:32.291 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.291 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:32.291 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.291 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.292 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:38766 9.2 9:4af8f8c5:::benchmark_data_smithi052_128427_object38765:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-13T13:10:32.292 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:10:32.163082+0000", 2024-09-13T13:10:32.292 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0097024209999999993, 2024-09-13T13:10:32.292 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0098264519999999994, 2024-09-13T13:10:32.292 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:32.292 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:32.292 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:10:32.292 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:32.292 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:10:32.292 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:10:32.292 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 38766 2024-09-13T13:10:32.292 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.292 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:32.293 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.293 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:32.293 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.163082+0000", 2024-09-13T13:10:32.293 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.293 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.293 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.293 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:32.293 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.163082+0000", 2024-09-13T13:10:32.293 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.293 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.293 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.293 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.163083+0000", 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0299999999999999e-07 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.163155+0000", 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1859000000000004e-05 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.163162+0000", 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9290000000000001e-06 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.294 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.295 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:32.295 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.163168+0000", 2024-09-13T13:10:32.295 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1469999999999998e-06 2024-09-13T13:10:32.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.295 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.295 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:32.295 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.164833+0000", 2024-09-13T13:10:32.295 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016650110000000001 2024-09-13T13:10:32.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.295 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.295 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:32.295 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.164945+0000", 2024-09-13T13:10:32.295 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011258499999999999 2024-09-13T13:10:32.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.296 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.296 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:10:32.296 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.165074+0000", 2024-09-13T13:10:32.296 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012872599999999999 2024-09-13T13:10:32.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.296 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.296 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:32.296 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.166704+0000", 2024-09-13T13:10:32.296 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016300080000000001 2024-09-13T13:10:32.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.296 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.296 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:32.297 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171733+0000", 2024-09-13T13:10:32.297 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050287760000000004 2024-09-13T13:10:32.297 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.297 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:32.297 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.297 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.297 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:38775 9.8 9:1c889645:::benchmark_data_smithi052_128427_object38774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-13T13:10:32.297 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:10:32.168542+0000", 2024-09-13T13:10:32.297 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0042431079999999998, 2024-09-13T13:10:32.297 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.004461072, 2024-09-13T13:10:32.297 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:32.297 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:32.298 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:32.298 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:32.298 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:10:32.298 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:10:32.298 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 38775 2024-09-13T13:10:32.298 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.298 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:32.298 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.298 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:32.299 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168542+0000", 2024-09-13T13:10:32.299 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.299 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:32.299 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168542+0000", 2024-09-13T13:10:32.299 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.299 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:32.299 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168544+0000", 2024-09-13T13:10:32.299 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1110000000000002e-06 2024-09-13T13:10:32.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.300 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:32.300 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168569+0000", 2024-09-13T13:10:32.300 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5235000000000001e-05 2024-09-13T13:10:32.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.300 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:32.300 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168579+0000", 2024-09-13T13:10:32.300 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0411000000000001e-05 2024-09-13T13:10:32.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.301 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.301 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:32.301 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168586+0000", 2024-09-13T13:10:32.301 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0820000000000001e-06 2024-09-13T13:10:32.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.301 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.301 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:32.301 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168715+0000", 2024-09-13T13:10:32.301 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012882700000000001 2024-09-13T13:10:32.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.301 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.301 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:32.302 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168845+0000", 2024-09-13T13:10:32.302 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013016199999999999 2024-09-13T13:10:32.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.302 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.302 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:10:32.302 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.169017+0000", 2024-09-13T13:10:32.302 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017198300000000001 2024-09-13T13:10:32.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.302 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.302 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:32.302 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172375+0000", 2024-09-13T13:10:32.302 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033575290000000002 2024-09-13T13:10:32.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172462+0000", 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6737000000000007e-05 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172481+0000", 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9638000000000001e-05 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172532+0000", 2024-09-13T13:10:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0356999999999999e-05 2024-09-13T13:10:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:38776 9.c 9:3bfb20ce:::benchmark_data_smithi052_128427_object38775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-13T13:10:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:10:32.168611+0000", 2024-09-13T13:10:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0041741879999999997, 2024-09-13T13:10:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0044845450000000004, 2024-09-13T13:10:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:10:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:10:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:10:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 38776 2024-09-13T13:10:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168611+0000", 2024-09-13T13:10:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168611+0000", 2024-09-13T13:10:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168611+0000", 2024-09-13T13:10:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:10:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168633+0000", 2024-09-13T13:10:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1991e-05 2024-09-13T13:10:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168641+0000", 2024-09-13T13:10:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2810000000000003e-06 2024-09-13T13:10:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.168646+0000", 2024-09-13T13:10:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3399999999999997e-06 2024-09-13T13:10:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.169117+0000", 2024-09-13T13:10:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00047120799999999998 2024-09-13T13:10:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.169275+0000", 2024-09-13T13:10:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015768800000000001 2024-09-13T13:10:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:10:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.169419+0000", 2024-09-13T13:10:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000144022 2024-09-13T13:10:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:38778 9.f e249/241 9:fa306a27:::benchmark_data_smithi052_128427_object38777:head v 249'2420, mlcod=249'2418)", 2024-09-13T13:10:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:10:32.171389+0000", 2024-09-13T13:10:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0013961900000000001, 2024-09-13T13:10:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001774842, 2024-09-13T13:10:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:10:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171389+0000", 2024-09-13T13:10:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171389+0000", 2024-09-13T13:10:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171390+0000", 2024-09-13T13:10:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0279999999999999e-06 2024-09-13T13:10:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171427+0000", 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7159000000000001e-05 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171439+0000", 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2031000000000001e-05 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171445+0000", 2024-09-13T13:10:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5490000000000003e-06 2024-09-13T13:10:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171666+0000", 2024-09-13T13:10:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000220468 2024-09-13T13:10:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171683+0000", 2024-09-13T13:10:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7371999999999999e-05 2024-09-13T13:10:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:38780 9.c 9:3e3ae3cd:::benchmark_data_smithi052_128427_object38779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-13T13:10:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:10:32.171648+0000", 2024-09-13T13:10:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.001136471, 2024-09-13T13:10:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001572036, 2024-09-13T13:10:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 38780 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171648+0000", 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:32.315 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171648+0000", 2024-09-13T13:10:32.315 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171650+0000", 2024-09-13T13:10:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.55e-06 2024-09-13T13:10:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171671+0000", 2024-09-13T13:10:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1619000000000001e-05 2024-09-13T13:10:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171679+0000", 2024-09-13T13:10:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5360000000000004e-06 2024-09-13T13:10:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.171685+0000", 2024-09-13T13:10:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.913e-06 2024-09-13T13:10:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172025+0000", 2024-09-13T13:10:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033993100000000002 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172130+0000", 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000105115 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172374+0000", 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024379299999999999 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.15789.0:38776 9.c e249/241)", 2024-09-13T13:10:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:10:32.172235+0000", 2024-09-13T13:10:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.00054940999999999996, 2024-09-13T13:10:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001046209, 2024-09-13T13:10:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:10:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172235+0000", 2024-09-13T13:10:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172235+0000", 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172236+0000", 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0279999999999999e-06 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172241+0000", 2024-09-13T13:10:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3120000000000004e-06 2024-09-13T13:10:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172250+0000", 2024-09-13T13:10:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5270000000000004e-06 2024-09-13T13:10:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172256+0000", 2024-09-13T13:10:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0399999999999998e-06 2024-09-13T13:10:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:38783 9.2 9:4459f0cf:::benchmark_data_smithi052_128427_object38782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-13T13:10:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:10:32.172364+0000", 2024-09-13T13:10:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.000421218, 2024-09-13T13:10:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00096642400000000002, 2024-09-13T13:10:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:10:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:10:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 38783 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172364+0000", 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172364+0000", 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172365+0000", 2024-09-13T13:10:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3230000000000001e-06 2024-09-13T13:10:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172400+0000", 2024-09-13T13:10:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5039000000000001e-05 2024-09-13T13:10:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172410+0000", 2024-09-13T13:10:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0183999999999999e-05 2024-09-13T13:10:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172417+0000", 2024-09-13T13:10:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0990000000000001e-06 2024-09-13T13:10:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172509+0000", 2024-09-13T13:10:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1827e-05 2024-09-13T13:10:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172652+0000", 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014267199999999999 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172776+0000", 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000124778 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.15789.0:38764 9.2 e249/241)", 2024-09-13T13:10:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:10:32.172982+0000", 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 4294967295.9998026, 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00043385500000000003, 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172982+0000", 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172982+0000", 2024-09-13T13:10:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172983+0000", 2024-09-13T13:10:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0720000000000001e-06 2024-09-13T13:10:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.172988+0000", 2024-09-13T13:10:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.301e-06 2024-09-13T13:10:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.173002+0000", 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3787e-05 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:10:32.173008+0000", 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1020000000000002e-06 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:10:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 9 2024-09-13T13:10:32.330 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:10:32.339 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:32.374 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:32.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:32.418 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T13:10:32.476 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:32.514 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:32.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:32.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:32.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:32 smithi167 ceph-mon[33178]: 2.1 scrub starts 2024-09-13T13:10:32.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:32 smithi167 ceph-mon[33178]: 2.1 scrub ok 2024-09-13T13:10:32.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:32 smithi167 ceph-mon[33178]: 3.1c scrub starts 2024-09-13T13:10:32.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:32 smithi167 ceph-mon[33178]: 3.1c scrub ok 2024-09-13T13:10:32.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:32 smithi167 ceph-mon[33178]: 4.1a scrub starts 2024-09-13T13:10:32.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:32 smithi167 ceph-mon[33178]: 4.1a scrub ok 2024-09-13T13:10:32.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:32 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1948197857' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:10:32.616 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:32.619 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[29471]: 2.1 scrub starts 2024-09-13T13:10:32.619 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[29471]: 2.1 scrub ok 2024-09-13T13:10:32.619 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[29471]: 3.1c scrub starts 2024-09-13T13:10:32.619 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[29471]: 3.1c scrub ok 2024-09-13T13:10:32.619 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[29471]: 4.1a scrub starts 2024-09-13T13:10:32.619 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[29471]: 4.1a scrub ok 2024-09-13T13:10:32.619 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1948197857' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:10:32.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[36682]: 2.1 scrub starts 2024-09-13T13:10:32.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[36682]: 2.1 scrub ok 2024-09-13T13:10:32.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[36682]: 3.1c scrub starts 2024-09-13T13:10:32.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[36682]: 3.1c scrub ok 2024-09-13T13:10:32.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[36682]: 4.1a scrub starts 2024-09-13T13:10:32.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[36682]: 4.1a scrub ok 2024-09-13T13:10:32.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:32 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1948197857' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:10:32.654 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:32.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:32.756 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:32.779 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 17 16 39895 39879 146.56 109.188 0.00963427 0.00681771 2024-09-13T13:10:32.797 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:32.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:32.858 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:32.900 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:32.934 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:32.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:32.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:33.036 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:33.078 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:33.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:33.180 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:33.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:33.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:33.318 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T13:10:33.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:33.351 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:33.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:33.360 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:33.453 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:33.487 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:33.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:33.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:33.589 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:33.627 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[36682]: pgmap v450: 148 pgs: 148 active+clean; 1.9 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 796 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-09-13T13:10:33.627 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[36682]: 3.1f scrub starts 2024-09-13T13:10:33.627 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[36682]: 3.1f scrub ok 2024-09-13T13:10:33.627 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1948197857' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:10:33.627 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[36682]: osdmap e250: 8 total, 8 up, 8 in 2024-09-13T13:10:33.627 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[36682]: 4.18 scrub starts 2024-09-13T13:10:33.627 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[36682]: 4.18 scrub ok 2024-09-13T13:10:33.627 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[36682]: 5.1a scrub starts 2024-09-13T13:10:33.627 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[36682]: 5.1a scrub ok 2024-09-13T13:10:33.627 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[36682]: osdmap e251: 8 total, 8 up, 8 in 2024-09-13T13:10:33.628 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:10:33.630 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[29471]: pgmap v450: 148 pgs: 148 active+clean; 1.9 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 796 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-09-13T13:10:33.630 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[29471]: 3.1f scrub starts 2024-09-13T13:10:33.630 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[29471]: 3.1f scrub ok 2024-09-13T13:10:33.630 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1948197857' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:10:33.630 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[29471]: osdmap e250: 8 total, 8 up, 8 in 2024-09-13T13:10:33.631 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[29471]: 4.18 scrub starts 2024-09-13T13:10:33.631 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[29471]: 4.18 scrub ok 2024-09-13T13:10:33.631 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[29471]: 5.1a scrub starts 2024-09-13T13:10:33.631 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[29471]: 5.1a scrub ok 2024-09-13T13:10:33.631 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[29471]: osdmap e251: 8 total, 8 up, 8 in 2024-09-13T13:10:33.631 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:33 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:10:33.639 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:33.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:33.741 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:33.779 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 18 16 41599 41583 144.334 106.5 0.00669125 0.00692312 2024-09-13T13:10:33.791 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:33.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:33.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:33 smithi167 ceph-mon[33178]: pgmap v450: 148 pgs: 148 active+clean; 1.9 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 796 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-09-13T13:10:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:33 smithi167 ceph-mon[33178]: 3.1f scrub starts 2024-09-13T13:10:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:33 smithi167 ceph-mon[33178]: 3.1f scrub ok 2024-09-13T13:10:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:33 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1948197857' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:10:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:33 smithi167 ceph-mon[33178]: osdmap e250: 8 total, 8 up, 8 in 2024-09-13T13:10:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:33 smithi167 ceph-mon[33178]: 4.18 scrub starts 2024-09-13T13:10:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:33 smithi167 ceph-mon[33178]: 4.18 scrub ok 2024-09-13T13:10:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:33 smithi167 ceph-mon[33178]: 5.1a scrub starts 2024-09-13T13:10:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:33 smithi167 ceph-mon[33178]: 5.1a scrub ok 2024-09-13T13:10:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:33 smithi167 ceph-mon[33178]: osdmap e251: 8 total, 8 up, 8 in 2024-09-13T13:10:33.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:33 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:10:33.893 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:33.933 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:33.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:34.035 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:34.070 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:34.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:34.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:34.172 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:34.212 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:34.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:34.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:34.314 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:34.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:10:34 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:10:34.354 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:34.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:34.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:34.456 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:34.500 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:34.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:34.602 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:34.642 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:34 smithi052 ceph-mon[36682]: 6.4 scrub starts 2024-09-13T13:10:34.643 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:34 smithi052 ceph-mon[36682]: 6.4 scrub ok 2024-09-13T13:10:34.643 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:34 smithi052 ceph-mon[29471]: 6.4 scrub starts 2024-09-13T13:10:34.643 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:34 smithi052 ceph-mon[29471]: 6.4 scrub ok 2024-09-13T13:10:34.644 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:34.644 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T13:10:34.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:34.746 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:34.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:34.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:34.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:34.779 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 19 16 43258 43242 142.194 103.688 0.0150677 0.00702534 2024-09-13T13:10:34.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:34 smithi167 ceph-mon[33178]: 6.4 scrub starts 2024-09-13T13:10:34.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:34 smithi167 ceph-mon[33178]: 6.4 scrub ok 2024-09-13T13:10:34.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:34.899 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:34.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:34.951 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:34.962 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:10:34.963 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:10:35.000 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:10:35.000 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:10:35.001 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:35.030 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:10:35.031 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:10:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:35.032 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:35.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:35.065 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:10:35.065 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:10:35.101 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:10:35.101 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:10:35.132 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:35.137 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:10:35.137 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:10:35.170 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:10:35.171 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:10:35.175 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:35.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:35.206 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:10:35.206 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:10:35.277 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:35.306 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:35.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:35.407 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:35.447 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:35.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:35.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:35.459 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:10:35.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[29471]: pgmap v453: 148 pgs: 148 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 188 MiB/s wr, 3.01k op/s 2024-09-13T13:10:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[29471]: 6.c scrub starts 2024-09-13T13:10:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[29471]: 6.c scrub ok 2024-09-13T13:10:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:10:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:10:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[29471]: 3.1a scrub starts 2024-09-13T13:10:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[29471]: 3.1a scrub ok 2024-09-13T13:10:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2115781541' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1946484512' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[29471]: from='client.25687 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:35.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[36682]: pgmap v453: 148 pgs: 148 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 188 MiB/s wr, 3.01k op/s 2024-09-13T13:10:35.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:35.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[36682]: 6.c scrub starts 2024-09-13T13:10:35.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[36682]: 6.c scrub ok 2024-09-13T13:10:35.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:10:35.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:10:35.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[36682]: 3.1a scrub starts 2024-09-13T13:10:35.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[36682]: 3.1a scrub ok 2024-09-13T13:10:35.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2115781541' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:35.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1946484512' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:35.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:35 smithi052 ceph-mon[36682]: from='client.25687 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:35.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:35.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:35.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:35.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:35.684 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:35.724 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:35.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:35.779 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:10:35.779640+0000 min lat: 0.00134564 max lat: 0.100892 avg lat: 0.0071364 2024-09-13T13:10:35.780 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:10:35.780 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 20 16 44839 44823 140.025 98.8125 0.00426977 0.0071364 2024-09-13T13:10:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:35 smithi167 ceph-mon[33178]: pgmap v453: 148 pgs: 148 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 188 MiB/s wr, 3.01k op/s 2024-09-13T13:10:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:35 smithi167 ceph-mon[33178]: 6.c scrub starts 2024-09-13T13:10:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:35 smithi167 ceph-mon[33178]: 6.c scrub ok 2024-09-13T13:10:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:35 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:10:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:35 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:10:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:35 smithi167 ceph-mon[33178]: 3.1a scrub starts 2024-09-13T13:10:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:35 smithi167 ceph-mon[33178]: 3.1a scrub ok 2024-09-13T13:10:35.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:35 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2115781541' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:35.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:35 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1946484512' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:35.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:35 smithi167 ceph-mon[33178]: from='client.25687 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:35.826 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:35.870 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:35.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:35.972 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:36.009 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:36.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:36.110 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:36.146 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:36.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:36.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:36.281 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:36.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:36.382 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:36.422 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:36.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:36.442 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:10:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:10:36.523 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:36.543 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:36.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:36.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:36.682 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:36.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:36.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[36682]: 6.14 scrub starts 2024-09-13T13:10:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[36682]: 6.14 scrub ok 2024-09-13T13:10:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[36682]: 3.1d scrub starts 2024-09-13T13:10:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[36682]: 3.1d scrub ok 2024-09-13T13:10:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:10:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:10:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[36682]: from='client.25687 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:10:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[36682]: osdmap e252: 8 total, 8 up, 8 in 2024-09-13T13:10:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[29471]: 6.14 scrub starts 2024-09-13T13:10:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[29471]: 6.14 scrub ok 2024-09-13T13:10:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[29471]: 3.1d scrub starts 2024-09-13T13:10:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[29471]: 3.1d scrub ok 2024-09-13T13:10:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:10:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:10:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[29471]: from='client.25687 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:10:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:36 smithi052 ceph-mon[29471]: osdmap e252: 8 total, 8 up, 8 in 2024-09-13T13:10:36.779 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 21 16 46564 46548 138.491 107.812 0.0042523 0.00721454 2024-09-13T13:10:36.784 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:36.815 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:36.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:36.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:36 smithi167 ceph-mon[33178]: 6.14 scrub starts 2024-09-13T13:10:36.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:36 smithi167 ceph-mon[33178]: 6.14 scrub ok 2024-09-13T13:10:36.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:36 smithi167 ceph-mon[33178]: 3.1d scrub starts 2024-09-13T13:10:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:36 smithi167 ceph-mon[33178]: 3.1d scrub ok 2024-09-13T13:10:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:36 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:10:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:36 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:10:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:36 smithi167 ceph-mon[33178]: from='client.25687 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:10:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:36 smithi167 ceph-mon[33178]: osdmap e252: 8 total, 8 up, 8 in 2024-09-13T13:10:36.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:36.951 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:36.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:37.053 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:37.095 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:37.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:37.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:37.197 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:37.240 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:37.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:37.342 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:37.376 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:37.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:37.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:37.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:37.514 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:37.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:37.616 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:37.662 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:37.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:37.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:37 smithi052 ceph-mon[29471]: pgmap v455: 148 pgs: 148 active+clean; 2.5 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T13:10:37.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:37 smithi052 ceph-mon[29471]: 5.11 scrub starts 2024-09-13T13:10:37.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:37 smithi052 ceph-mon[29471]: 5.11 scrub ok 2024-09-13T13:10:37.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:37 smithi052 ceph-mon[36682]: pgmap v455: 148 pgs: 148 active+clean; 2.5 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T13:10:37.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:37 smithi052 ceph-mon[36682]: 5.11 scrub starts 2024-09-13T13:10:37.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:37 smithi052 ceph-mon[36682]: 5.11 scrub ok 2024-09-13T13:10:37.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:37.779 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 22 16 48382 48366 137.361 113.625 0.00394639 0.0072681 2024-09-13T13:10:37.785 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:37.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:37.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:37.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:37 smithi167 ceph-mon[33178]: pgmap v455: 148 pgs: 148 active+clean; 2.5 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T13:10:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:37 smithi167 ceph-mon[33178]: 5.11 scrub starts 2024-09-13T13:10:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:37 smithi167 ceph-mon[33178]: 5.11 scrub ok 2024-09-13T13:10:37.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:37.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:37.915 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:37.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:38.016 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:38.058 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:38.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:38.132 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:10:38.159 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:38.191 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:38.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:38.293 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:38.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:38.334 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:38.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:38.435 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:38.479 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:38.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:38.582 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:38.614 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:38.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:38.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:38.716 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:38.722 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:38.776 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:38.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:38.779 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 23 16 49911 49895 135.543 95.5625 0.0161818 0.00737241 2024-09-13T13:10:38.878 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:38.909 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:38.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:38.911 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:38 smithi052 ceph-mon[36682]: 3.1 scrub starts 2024-09-13T13:10:38.911 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:38 smithi052 ceph-mon[36682]: 3.1 scrub ok 2024-09-13T13:10:38.911 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:38 smithi052 ceph-mon[36682]: pgmap v456: 148 pgs: 148 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T13:10:38.911 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:38 smithi052 ceph-mon[36682]: osdmap e253: 8 total, 8 up, 8 in 2024-09-13T13:10:38.911 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:38 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/70444631' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:10:38.911 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:38 smithi052 ceph-mon[36682]: from='client.35696 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:10:38.911 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T13:10:38.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:38 smithi052 ceph-mon[29471]: 3.1 scrub starts 2024-09-13T13:10:38.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:38 smithi052 ceph-mon[29471]: 3.1 scrub ok 2024-09-13T13:10:38.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:38 smithi052 ceph-mon[29471]: pgmap v456: 148 pgs: 148 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T13:10:38.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:38 smithi052 ceph-mon[29471]: osdmap e253: 8 total, 8 up, 8 in 2024-09-13T13:10:38.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/70444631' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:10:38.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:38 smithi052 ceph-mon[29471]: from='client.35696 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:10:39.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:39.046 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:39.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:39.147 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:39.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:39.179 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:39.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:39.280 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:39.317 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:39.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:38 smithi167 ceph-mon[33178]: 3.1 scrub starts 2024-09-13T13:10:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:38 smithi167 ceph-mon[33178]: 3.1 scrub ok 2024-09-13T13:10:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:38 smithi167 ceph-mon[33178]: pgmap v456: 148 pgs: 148 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T13:10:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:38 smithi167 ceph-mon[33178]: osdmap e253: 8 total, 8 up, 8 in 2024-09-13T13:10:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:38 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/70444631' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:10:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:38 smithi167 ceph-mon[33178]: from='client.35696 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:10:39.394 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:39.419 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:39.446 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:39.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:39.452 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:10:39.547 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:39.577 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:39.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:39.679 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:39.708 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:39.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:39.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:39.781 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 24 16 51541 51525 134.134 101.875 0.00806785 0.00744948 2024-09-13T13:10:39.810 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:39.851 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:39.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:39.953 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:39.981 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:39.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:40.005 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T13:10:40.082 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:40.143 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:40.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:40.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:40.234 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:39 smithi052 ceph-mon[36682]: from='client.35696 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:10:40.234 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:39 smithi052 ceph-mon[36682]: osdmap e254: 8 total, 8 up, 8 in 2024-09-13T13:10:40.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:39 smithi052 ceph-mon[29471]: from='client.35696 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:10:40.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:39 smithi052 ceph-mon[29471]: osdmap e254: 8 total, 8 up, 8 in 2024-09-13T13:10:40.245 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:40.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:40.287 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:40.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:40.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:39 smithi167 ceph-mon[33178]: from='client.35696 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:10:40.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:39 smithi167 ceph-mon[33178]: osdmap e254: 8 total, 8 up, 8 in 2024-09-13T13:10:40.388 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:40.420 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:40.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:40.484 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:10:40.484 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:10:40.521 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:40.563 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:40.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:40.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:40.665 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:40.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:40.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:40.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:40.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 25 16 53088 53072 132.626 96.6875 0.0117539 0.0075343 2024-09-13T13:10:40.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:40.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:40.836 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:40.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:40.937 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:40.980 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:40.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:40.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:41.081 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:41.122 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:41.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:41.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:41.165 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:40 smithi052 ceph-mon[36682]: 4.19 scrub starts 2024-09-13T13:10:41.165 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:40 smithi052 ceph-mon[36682]: 4.19 scrub ok 2024-09-13T13:10:41.165 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:40 smithi052 ceph-mon[36682]: 6.10 scrub starts 2024-09-13T13:10:41.165 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:40 smithi052 ceph-mon[36682]: 6.10 scrub ok 2024-09-13T13:10:41.166 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:40 smithi052 ceph-mon[36682]: pgmap v459: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-09-13T13:10:41.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:40 smithi052 ceph-mon[29471]: 4.19 scrub starts 2024-09-13T13:10:41.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:40 smithi052 ceph-mon[29471]: 4.19 scrub ok 2024-09-13T13:10:41.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:40 smithi052 ceph-mon[29471]: 6.10 scrub starts 2024-09-13T13:10:41.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:40 smithi052 ceph-mon[29471]: 6.10 scrub ok 2024-09-13T13:10:41.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:40 smithi052 ceph-mon[29471]: pgmap v459: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-09-13T13:10:41.219 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T13:10:41.223 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:41.254 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:41.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:41.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:40 smithi167 ceph-mon[33178]: 4.19 scrub starts 2024-09-13T13:10:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:40 smithi167 ceph-mon[33178]: 4.19 scrub ok 2024-09-13T13:10:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:40 smithi167 ceph-mon[33178]: 6.10 scrub starts 2024-09-13T13:10:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:40 smithi167 ceph-mon[33178]: 6.10 scrub ok 2024-09-13T13:10:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:40 smithi167 ceph-mon[33178]: pgmap v459: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-09-13T13:10:41.356 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:41.396 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:41.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:41.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:41.532 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:41.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:41.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:41.634 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:41.671 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:41.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:41.773 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:41.783 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 26 16 54648 54632 131.272 97.5 0.0168267 0.00761297 2024-09-13T13:10:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:41.821 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:41.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:41.922 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:41.961 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:41.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:41.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:41.972 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:10:41.972 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:10:41.973 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:10:41.973 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:10:41.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[29471]: 4.2 scrub starts 2024-09-13T13:10:41.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[29471]: 4.2 scrub ok 2024-09-13T13:10:41.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[29471]: 6.8 scrub starts 2024-09-13T13:10:41.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[29471]: 6.8 scrub ok 2024-09-13T13:10:41.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[29471]: 3.c scrub starts 2024-09-13T13:10:41.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[29471]: 3.c scrub ok 2024-09-13T13:10:41.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3397517893' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:41.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[36682]: 4.2 scrub starts 2024-09-13T13:10:41.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[36682]: 4.2 scrub ok 2024-09-13T13:10:41.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[36682]: 6.8 scrub starts 2024-09-13T13:10:41.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[36682]: 6.8 scrub ok 2024-09-13T13:10:41.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[36682]: 3.c scrub starts 2024-09-13T13:10:41.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[36682]: 3.c scrub ok 2024-09-13T13:10:41.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:41 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3397517893' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:42.004 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:10:42.005 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:10:42.040 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:10:42.041 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:10:42.063 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:42.065 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:42.075 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:10:42.075 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:10:42.084 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:42.092 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:42.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:42.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:42.095 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:42.113 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:10:42.114 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:10:42.148 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:10:42.149 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:10:42.183 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:10:42.183 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:10:42.194 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:42.230 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:42.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:42.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:42.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:41 smithi167 ceph-mon[33178]: 4.2 scrub starts 2024-09-13T13:10:42.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:41 smithi167 ceph-mon[33178]: 4.2 scrub ok 2024-09-13T13:10:42.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:41 smithi167 ceph-mon[33178]: 6.8 scrub starts 2024-09-13T13:10:42.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:41 smithi167 ceph-mon[33178]: 6.8 scrub ok 2024-09-13T13:10:42.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:41 smithi167 ceph-mon[33178]: 3.c scrub starts 2024-09-13T13:10:42.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:41 smithi167 ceph-mon[33178]: 3.c scrub ok 2024-09-13T13:10:42.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:41 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3397517893' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:42.332 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:42.378 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:42.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:42.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:42.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:42.506 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:42.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:42.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:42.608 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:42.641 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:42.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:42.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:42.743 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:42.774 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:42.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:42.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:42.784 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 27 16 56060 56044 129.679 88.25 0.0068537 0.00770599 2024-09-13T13:10:42.876 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:42.926 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:42.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:43.028 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:43.064 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:43.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:43.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:43.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:42 smithi167 ceph-mon[33178]: pgmap v460: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 806 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-13T13:10:43.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:42 smithi167 ceph-mon[33178]: 5.1f scrub starts 2024-09-13T13:10:43.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:42 smithi167 ceph-mon[33178]: 5.1f scrub ok 2024-09-13T13:10:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:42 smithi167 ceph-mon[33178]: 6.a scrub starts 2024-09-13T13:10:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:42 smithi167 ceph-mon[33178]: 6.a scrub ok 2024-09-13T13:10:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:42 smithi167 ceph-mon[33178]: osdmap e255: 8 total, 8 up, 8 in 2024-09-13T13:10:43.165 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:43.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:43.206 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:43.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:42 smithi052 ceph-mon[36682]: pgmap v460: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 806 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-13T13:10:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:42 smithi052 ceph-mon[36682]: 5.1f scrub starts 2024-09-13T13:10:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:42 smithi052 ceph-mon[36682]: 5.1f scrub ok 2024-09-13T13:10:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:42 smithi052 ceph-mon[36682]: 6.a scrub starts 2024-09-13T13:10:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:42 smithi052 ceph-mon[36682]: 6.a scrub ok 2024-09-13T13:10:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:42 smithi052 ceph-mon[36682]: osdmap e255: 8 total, 8 up, 8 in 2024-09-13T13:10:43.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:42 smithi052 ceph-mon[29471]: pgmap v460: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 806 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-13T13:10:43.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:42 smithi052 ceph-mon[29471]: 5.1f scrub starts 2024-09-13T13:10:43.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:42 smithi052 ceph-mon[29471]: 5.1f scrub ok 2024-09-13T13:10:43.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:42 smithi052 ceph-mon[29471]: 6.a scrub starts 2024-09-13T13:10:43.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:42 smithi052 ceph-mon[29471]: 6.a scrub ok 2024-09-13T13:10:43.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:42 smithi052 ceph-mon[29471]: osdmap e255: 8 total, 8 up, 8 in 2024-09-13T13:10:43.307 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:43.342 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:43.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:43.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:43.443 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:43.480 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:43.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:43.581 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:43.620 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:43.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:43.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:43.722 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:43.750 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:43.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:43.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:43.784 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 28 15 57671 57656 128.646 100.75 0.00624558 0.00776828 2024-09-13T13:10:43.852 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:43.886 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:43.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:43.939 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:43 smithi052 ceph-mon[29471]: 6.0 scrub starts 2024-09-13T13:10:43.940 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:43 smithi052 ceph-mon[29471]: 6.0 scrub ok 2024-09-13T13:10:43.940 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:43 smithi052 ceph-mon[29471]: 6.6 scrub starts 2024-09-13T13:10:43.940 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:43 smithi052 ceph-mon[29471]: 6.6 scrub ok 2024-09-13T13:10:43.986 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:44.030 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:44.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:44.132 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:44.160 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:44.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:44.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:43 smithi052 ceph-mon[36682]: 6.0 scrub starts 2024-09-13T13:10:44.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:43 smithi052 ceph-mon[36682]: 6.0 scrub ok 2024-09-13T13:10:44.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:43 smithi052 ceph-mon[36682]: 6.6 scrub starts 2024-09-13T13:10:44.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:43 smithi052 ceph-mon[36682]: 6.6 scrub ok 2024-09-13T13:10:44.263 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:44.311 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:44.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:44.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:44.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:43 smithi167 ceph-mon[33178]: 6.0 scrub starts 2024-09-13T13:10:44.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:43 smithi167 ceph-mon[33178]: 6.0 scrub ok 2024-09-13T13:10:44.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:43 smithi167 ceph-mon[33178]: 6.6 scrub starts 2024-09-13T13:10:44.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:43 smithi167 ceph-mon[33178]: 6.6 scrub ok 2024-09-13T13:10:44.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:10:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:10:44.413 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:44.451 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:44.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:44.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:44.458 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:10:44.458 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:10:44.458 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:10:44.458 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:10:44.458 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.458 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83963 7.c 7:30d6a006:::benchmark_data_smithi052_71865_object83962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:10:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:10:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 449.904811967, 2024-09-13T13:10:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.013163161, 2024-09-13T13:10:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83963 2024-09-13T13:10:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547884+0000", 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.779e-06 2024-09-13T13:10:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547907+0000", 2024-09-13T13:10:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2779999999999999e-05 2024-09-13T13:10:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547917+0000", 2024-09-13T13:10:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8439999999999993e-06 2024-09-13T13:10:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5229999999999998e-06 2024-09-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554417+0000", 2024-09-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064933880000000001 2024-09-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554427+0000", 2024-09-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4280000000000002e-06 2024-09-13T13:10:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.543883+0000", 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98945604700000001 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545504+0000", 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016211660000000001 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545747+0000", 2024-09-13T13:10:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024302199999999999 2024-09-13T13:10:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552352+0000", 2024-09-13T13:10:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.006605133 2024-09-13T13:10:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560880+0000", 2024-09-13T13:10:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0085281650000000007 2024-09-13T13:10:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560953+0000", 2024-09-13T13:10:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2096999999999998e-05 2024-09-13T13:10:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561045+0000", 2024-09-13T13:10:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2788999999999995e-05 2024-09-13T13:10:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83964 7.c 7:37783165:::benchmark_data_smithi052_71865_object83963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:10:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:10:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 449.90467580000001, 2024-09-13T13:10:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0132684729999999, 2024-09-13T13:10:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83964 2024-09-13T13:10:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548019+0000", 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0279999999999999e-06 2024-09-13T13:10:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548044+0000", 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4581999999999999e-05 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548052+0000", 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3799999999999994e-06 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548059+0000", 2024-09-13T13:10:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8329999999999996e-06 2024-09-13T13:10:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554452+0000", 2024-09-13T13:10:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0063932119999999997 2024-09-13T13:10:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554459+0000", 2024-09-13T13:10:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3319999999999997e-06 2024-09-13T13:10:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552381+0000", 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99792229399999999 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552485+0000", 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010354800000000001 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552626+0000", 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014122000000000001 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560863+0000", 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0082373700000000008 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561139+0000", 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027546899999999997 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561192+0000", 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2986999999999999e-05 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.472 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.472 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561287+0000", 2024-09-13T13:10:44.472 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5217999999999995e-05 2024-09-13T13:10:44.472 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.472 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.472 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.472 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83985 7.c 7:386b3d8b:::benchmark_data_smithi052_71865_object83984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:10:44.472 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:10:44.472 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 449.88475444599999, 2024-09-13T13:10:44.472 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99349958199999999, 2024-09-13T13:10:44.472 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83985 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.473 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567942+0000", 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.888e-06 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567971+0000", 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9380999999999999e-05 2024-09-13T13:10:44.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568031+0000", 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9731000000000001e-05 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568039+0000", 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8109999999999992e-06 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568102+0000", 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2271000000000001e-05 2024-09-13T13:10:44.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.476 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:44.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568109+0000", 2024-09-13T13:10:44.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2570000000000001e-06 2024-09-13T13:10:44.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.476 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.557963+0000", 2024-09-13T13:10:44.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.989854491 2024-09-13T13:10:44.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.476 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.558066+0000", 2024-09-13T13:10:44.477 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102929 2024-09-13T13:10:44.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.477 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:10:44.477 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.558206+0000", 2024-09-13T13:10:44.477 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013928000000000001 2024-09-13T13:10:44.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.477 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.477 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561121+0000", 2024-09-13T13:10:44.477 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029151849999999998 2024-09-13T13:10:44.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561301+0000", 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018019299999999999 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561349+0000", 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7639999999999998e-05 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561439+0000", 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0525000000000005e-05 2024-09-13T13:10:44.478 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83989 7.c 7:3cf9e082:::benchmark_data_smithi052_71865_object83988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 449.87811224299998, 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99086929999999995, 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:10:44.479 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83989 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.480 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574584+0000", 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0480000000000001e-06 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574603+0000", 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8913e-05 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574615+0000", 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1678000000000001e-05 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.482 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.482 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574622+0000", 2024-09-13T13:10:44.482 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:10:44.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.482 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.482 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574674+0000", 2024-09-13T13:10:44.482 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1053000000000001e-05 2024-09-13T13:10:44.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.482 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:10:44.482 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574681+0000", 2024-09-13T13:10:44.482 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0110000000000003e-06 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561482+0000", 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98680099200000004 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561622+0000", 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014003799999999999 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561773+0000", 2024-09-13T13:10:44.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015180599999999999 2024-09-13T13:10:44.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.564308+0000", 2024-09-13T13:10:44.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025348390000000001 2024-09-13T13:10:44.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565272+0000", 2024-09-13T13:10:44.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00096360599999999999 2024-09-13T13:10:44.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565362+0000", 2024-09-13T13:10:44.485 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0253000000000005e-05 2024-09-13T13:10:44.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.485 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565451+0000", 2024-09-13T13:10:44.485 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9090000000000003e-05 2024-09-13T13:10:44.485 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.485 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.485 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.486 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36744 8.f 8:f94e36b9:::benchmark_data_smithi052_101302_object36743:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:10:44.486 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.446006+0000", 2024-09-13T13:10:44.486 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 237.00668834000001, 2024-09-13T13:10:44.486 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.128763765, 2024-09-13T13:10:44.486 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.486 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.486 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36744 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446006+0000", 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446006+0000", 2024-09-13T13:10:44.487 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.488 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446008+0000", 2024-09-13T13:10:44.488 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0949999999999998e-06 2024-09-13T13:10:44.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.488 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446034+0000", 2024-09-13T13:10:44.488 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6186999999999998e-05 2024-09-13T13:10:44.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446051+0000", 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6549e-05 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.446058+0000", 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6480000000000003e-06 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.448035+0000", 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019772129999999998 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.448146+0000", 2024-09-13T13:10:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011084199999999999 2024-09-13T13:10:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:10:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.448268+0000", 2024-09-13T13:10:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121266 2024-09-13T13:10:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451458+0000", 2024-09-13T13:10:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031904899999999998 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.452999+0000", 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001540823 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.574662+0000", 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.121663207 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.574726+0000", 2024-09-13T13:10:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4369000000000007e-05 2024-09-13T13:10:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.574770+0000", 2024-09-13T13:10:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3075999999999999e-05 2024-09-13T13:10:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36751 8.0 8:0c704d38:::benchmark_data_smithi052_101302_object36750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:10:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.450874+0000", 2024-09-13T13:10:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 237.001819665, 2024-09-13T13:10:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12726658099999999, 2024-09-13T13:10:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36751 2024-09-13T13:10:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450874+0000", 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450874+0000", 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450877+0000", 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.159e-06 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450910+0000", 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.345e-05 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450922+0000", 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1912e-05 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450929+0000", 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3180000000000001e-06 2024-09-13T13:10:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.496 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.496 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451081+0000", 2024-09-13T13:10:44.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015215699999999999 2024-09-13T13:10:44.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.496 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.496 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451225+0000", 2024-09-13T13:10:44.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014390400000000001 2024-09-13T13:10:44.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.496 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:10:44.496 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451361+0000", 2024-09-13T13:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013605400000000001 2024-09-13T13:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454511+0000", 2024-09-13T13:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031497119999999998 2024-09-13T13:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454545+0000", 2024-09-13T13:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3543000000000001e-05 2024-09-13T13:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578050+0000", 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.123505225 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578103+0000", 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2679999999999997e-05 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578141+0000", 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8467000000000003e-05 2024-09-13T13:10:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36752 8.0 8:0ff7cdfb:::benchmark_data_smithi052_101302_object36751:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.450957+0000", 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 237.001736792, 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12729238900000001, 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36752 2024-09-13T13:10:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450957+0000", 2024-09-13T13:10:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450957+0000", 2024-09-13T13:10:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450958+0000", 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1699999999999997e-07 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450980+0000", 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1466000000000001e-05 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450986+0000", 2024-09-13T13:10:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.516e-06 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.450991+0000", 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3090000000000003e-06 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.452820+0000", 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00182898 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.452929+0000", 2024-09-13T13:10:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108448 2024-09-13T13:10:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:10:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.453078+0000", 2024-09-13T13:10:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000148663 2024-09-13T13:10:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455024+0000", 2024-09-13T13:10:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019467499999999999 2024-09-13T13:10:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455035+0000", 2024-09-13T13:10:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:10:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578177+0000", 2024-09-13T13:10:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.123141319 2024-09-13T13:10:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578219+0000", 2024-09-13T13:10:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2348999999999997e-05 2024-09-13T13:10:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578250+0000", 2024-09-13T13:10:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0783e-05 2024-09-13T13:10:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36753 8.f 8:f05f76d5:::benchmark_data_smithi052_101302_object36752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:10:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.451293+0000", 2024-09-13T13:10:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 237.00140075900001, 2024-09-13T13:10:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.123777199, 2024-09-13T13:10:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36753 2024-09-13T13:10:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451293+0000", 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451293+0000", 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451294+0000", 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0699999999999999e-06 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451317+0000", 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2759e-05 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451325+0000", 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1419999999999993e-06 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451332+0000", 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.63e-06 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451491+0000", 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015913900000000001 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451597+0000", 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010614400000000001 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.451725+0000", 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012755899999999999 2024-09-13T13:10:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454052+0000", 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002327487 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458004+0000", 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039520670000000001 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.574824+0000", 2024-09-13T13:10:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11682004999999999 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.575030+0000", 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000205658 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.575071+0000", 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0494000000000002e-05 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36762 8.0 8:04e03c57:::benchmark_data_smithi052_101302_object36761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:10:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.454968+0000", 2024-09-13T13:10:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 236.99772593, 2024-09-13T13:10:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12340187399999999, 2024-09-13T13:10:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36762 2024-09-13T13:10:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454968+0000", 2024-09-13T13:10:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454968+0000", 2024-09-13T13:10:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454970+0000", 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.854e-06 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.454996+0000", 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5721e-05 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455007+0000", 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1294000000000001e-05 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455014+0000", 2024-09-13T13:10:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7970000000000003e-06 2024-09-13T13:10:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455064+0000", 2024-09-13T13:10:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0141e-05 2024-09-13T13:10:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455227+0000", 2024-09-13T13:10:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016291299999999999 2024-09-13T13:10:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:10:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455350+0000", 2024-09-13T13:10:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000122884 2024-09-13T13:10:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458138+0000", 2024-09-13T13:10:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027878180000000001 2024-09-13T13:10:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458328+0000", 2024-09-13T13:10:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000190352 2024-09-13T13:10:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578285+0000", 2024-09-13T13:10:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11995714 2024-09-13T13:10:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578335+0000", 2024-09-13T13:10:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9969999999999998e-05 2024-09-13T13:10:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.578370+0000", 2024-09-13T13:10:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4990000000000002e-05 2024-09-13T13:10:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36765 8.f 8:fc16e397:::benchmark_data_smithi052_101302_object36764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:10:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.455809+0000", 2024-09-13T13:10:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 236.996884962, 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12131874400000001, 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36765 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455809+0000", 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455809+0000", 2024-09-13T13:10:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455811+0000", 2024-09-13T13:10:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.452e-06 2024-09-13T13:10:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455840+0000", 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9050000000000001e-05 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455850+0000", 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.013e-05 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455856+0000", 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6630000000000004e-06 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.455915+0000", 2024-09-13T13:10:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8628999999999997e-05 2024-09-13T13:10:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.456126+0000", 2024-09-13T13:10:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021076100000000001 2024-09-13T13:10:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:10:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.456250+0000", 2024-09-13T13:10:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012409000000000001 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458014+0000", 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0017640080000000001 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458626+0000", 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00061228300000000003 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.577037+0000", 2024-09-13T13:10:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.118410709 2024-09-13T13:10:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.577084+0000", 2024-09-13T13:10:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7490000000000001e-05 2024-09-13T13:10:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.577128+0000", 2024-09-13T13:10:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3479000000000002e-05 2024-09-13T13:10:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.524 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36768 8.b 8:dd28fad3:::benchmark_data_smithi052_101302_object36767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:10:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.458820+0000", 2024-09-13T13:10:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 236.99387440800001, 2024-09-13T13:10:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.127385796, 2024-09-13T13:10:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.525 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36768 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458820+0000", 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458820+0000", 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458821+0000", 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5770000000000001e-06 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458842+0000", 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1013999999999999e-05 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458853+0000", 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0876000000000001e-05 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458871+0000", 2024-09-13T13:10:44.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7371999999999999e-05 2024-09-13T13:10:44.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.458943+0000", 2024-09-13T13:10:44.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2386999999999997e-05 2024-09-13T13:10:44.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459025+0000", 2024-09-13T13:10:44.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1864000000000001e-05 2024-09-13T13:10:44.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:10:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459148+0000", 2024-09-13T13:10:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012354900000000001 2024-09-13T13:10:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.460382+0000", 2024-09-13T13:10:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012340999999999999 2024-09-13T13:10:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.460748+0000", 2024-09-13T13:10:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00036591599999999998 2024-09-13T13:10:44.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.586104+0000", 2024-09-13T13:10:44.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.125355614 2024-09-13T13:10:44.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.586163+0000", 2024-09-13T13:10:44.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8563999999999997e-05 2024-09-13T13:10:44.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.586206+0000", 2024-09-13T13:10:44.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2963000000000002e-05 2024-09-13T13:10:44.532 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.532 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.532 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.532 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:36769 8.0 8:0a2fab5a:::benchmark_data_smithi052_101302_object36768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:10:44.532 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:06:47.459028+0000", 2024-09-13T13:10:44.532 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 236.99366636799999, 2024-09-13T13:10:44.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12155476799999999, 2024-09-13T13:10:44.532 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.532 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.533 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.533 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.533 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.533 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.533 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 36769 2024-09-13T13:10:44.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.533 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459028+0000", 2024-09-13T13:10:44.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459028+0000", 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459029+0000", 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.283e-06 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459052+0000", 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3315e-05 2024-09-13T13:10:44.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459062+0000", 2024-09-13T13:10:44.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2969999999999992e-06 2024-09-13T13:10:44.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459069+0000", 2024-09-13T13:10:44.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1690000000000003e-06 2024-09-13T13:10:44.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459120+0000", 2024-09-13T13:10:44.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1684999999999997e-05 2024-09-13T13:10:44.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459235+0000", 2024-09-13T13:10:44.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114246 2024-09-13T13:10:44.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:10:44.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.459394+0000", 2024-09-13T13:10:44.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015965599999999999 2024-09-13T13:10:44.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.460679+0000", 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012848759999999999 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.461249+0000", 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00057000599999999996 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.580496+0000", 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.119247045 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.580546+0000", 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9638000000000001e-05 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:06:47.580583+0000", 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6551999999999998e-05 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.538 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:135978 8.0 8:077a9a64:::benchmark_data_smithi052_101302_object8049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:08.813985+0000", 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 155.63870870100001, 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19821777300000001, 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 135978 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.813985+0000", 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.813985+0000", 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.813987+0000", 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7489999999999999e-06 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.813992+0000", 2024-09-13T13:10:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0679999999999996e-06 2024-09-13T13:10:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814245+0000", 2024-09-13T13:10:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00025265800000000001 2024-09-13T13:10:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814253+0000", 2024-09-13T13:10:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7039999999999994e-06 2024-09-13T13:10:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814370+0000", 2024-09-13T13:10:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117674 2024-09-13T13:10:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814476+0000", 2024-09-13T13:10:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000105457 2024-09-13T13:10:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:10:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814578+0000", 2024-09-13T13:10:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102204 2024-09-13T13:10:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.817977+0000", 2024-09-13T13:10:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033988999999999998 2024-09-13T13:10:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819611+0000", 2024-09-13T13:10:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016344199999999999 2024-09-13T13:10:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.012080+0000", 2024-09-13T13:10:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.192468369 2024-09-13T13:10:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.012156+0000", 2024-09-13T13:10:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6459999999999996e-05 2024-09-13T13:10:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.012203+0000", 2024-09-13T13:10:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7110000000000001e-05 2024-09-13T13:10:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:135984 8.0 8:0135d23d:::benchmark_data_smithi052_101302_object8055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:10:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:08.819395+0000", 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 155.63329869699999, 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.197854312, 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 135984 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819395+0000", 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819395+0000", 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819397+0000", 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.945e-06 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819403+0000", 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0739999999999998e-06 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819419+0000", 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5481999999999998e-05 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819427+0000", 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7300000000000005e-06 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819641+0000", 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000214606 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819763+0000", 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121311 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819881+0000", 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118528 2024-09-13T13:10:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.822831+0000", 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002949587 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.822973+0000", 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014218999999999999 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.016953+0000", 2024-09-13T13:10:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19398000800000001 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.017192+0000", 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000238879 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.017250+0000", 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7972e-05 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:135989 8.0 8:0d783973:::benchmark_data_smithi052_101302_object8060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:10:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:08.824790+0000", 2024-09-13T13:10:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 155.62790459999999, 2024-09-13T13:10:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.204002616, 2024-09-13T13:10:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 135989 2024-09-13T13:10:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824790+0000", 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824790+0000", 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824792+0000", 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.04e-06 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824797+0000", 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6840000000000001e-06 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824813+0000", 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5529999999999999e-05 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824821+0000", 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.259e-06 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824872+0000", 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1159999999999998e-05 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824997+0000", 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012481499999999999 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.825118+0000", 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120907 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.828631+0000", 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0035128339999999998 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.828649+0000", 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8048000000000001e-05 2024-09-13T13:10:44.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.557 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.027817+0000", 2024-09-13T13:10:44.557 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19916857399999999 2024-09-13T13:10:44.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.557 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.028750+0000", 2024-09-13T13:10:44.557 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00093269499999999999 2024-09-13T13:10:44.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.557 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.028792+0000", 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2070000000000002e-05 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:228709 8.f 8:f5930bd7:::benchmark_data_smithi052_101302_object100780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:29.777978+0000", 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 74.674716020000005, 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.111901846, 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.558 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 228709 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.777978+0000", 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.777978+0000", 2024-09-13T13:10:44.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.777979+0000", 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4399999999999998e-07 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.777983+0000", 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9090000000000002e-06 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.777994+0000", 2024-09-13T13:10:44.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0774999999999999e-05 2024-09-13T13:10:44.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.778000+0000", 2024-09-13T13:10:44.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.054e-06 2024-09-13T13:10:44.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.787619+0000", 2024-09-13T13:10:44.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0096195010000000008 2024-09-13T13:10:44.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.787743+0000", 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000123931 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.787866+0000", 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012251100000000001 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.792690+0000", 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048240829999999998 2024-09-13T13:10:44.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.792731+0000", 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1078000000000002e-05 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.889701+0000", 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.096970361000000005 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.889833+0000", 2024-09-13T13:10:44.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013170900000000001 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.889880+0000", 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6990000000000002e-05 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:228717 8.f 8:f37b4b8e:::benchmark_data_smithi052_101302_object100788:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:29.784011+0000", 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 74.668682803999999, 2024-09-13T13:10:44.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12196863600000001, 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 228717 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.784011+0000", 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.784011+0000", 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.784013+0000", 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.265e-06 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.784017+0000", 2024-09-13T13:10:44.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7029999999999997e-06 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.784027+0000", 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.38e-06 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.784033+0000", 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7410000000000003e-06 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.790257+0000", 2024-09-13T13:10:44.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.006223302 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.790385+0000", 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012805999999999999 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.790504+0000", 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000119361 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.792735+0000", 2024-09-13T13:10:44.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0022308459999999999 2024-09-13T13:10:44.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.793168+0000", 2024-09-13T13:10:44.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000432576 2024-09-13T13:10:44.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.905876+0000", 2024-09-13T13:10:44.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.112708219 2024-09-13T13:10:44.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.905936+0000", 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0032999999999999e-05 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:29.905980+0000", 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4150000000000003e-05 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237582 8.b 8:d863b546:::benchmark_data_smithi052_101302_object109653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:10:44.570 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:37.691188+0000", 2024-09-13T13:10:44.571 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 66.761506427, 2024-09-13T13:10:44.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.124043958, 2024-09-13T13:10:44.571 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.571 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.571 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:44.572 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.572 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.572 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.572 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.572 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237582 2024-09-13T13:10:44.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.573 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.691188+0000", 2024-09-13T13:10:44.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.691188+0000", 2024-09-13T13:10:44.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.691188+0000", 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8500000000000001e-07 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.691192+0000", 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6799999999999999e-06 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.691199+0000", 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5250000000000002e-06 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.691204+0000", 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6230000000000002e-06 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.691320+0000", 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000115869 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.691441+0000", 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121136 2024-09-13T13:10:44.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:10:44.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.691622+0000", 2024-09-13T13:10:44.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018044099999999999 2024-09-13T13:10:44.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.694536+0000", 2024-09-13T13:10:44.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029143239999999998 2024-09-13T13:10:44.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.695496+0000", 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00096022900000000005 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.815129+0000", 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11963267 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.815185+0000", 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6589999999999999e-05 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.815232+0000", 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6186000000000001e-05 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237593 8.b 8:d03a3173:::benchmark_data_smithi052_101302_object109664:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:37.697565+0000", 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 66.755129526000005, 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12435009599999999, 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.578 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237593 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.697565+0000", 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.697565+0000", 2024-09-13T13:10:44.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.697567+0000", 2024-09-13T13:10:44.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2689999999999998e-06 2024-09-13T13:10:44.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.697573+0000", 2024-09-13T13:10:44.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9540000000000001e-06 2024-09-13T13:10:44.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.697586+0000", 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3366e-05 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.697595+0000", 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6610000000000006e-06 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.697652+0000", 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7420999999999998e-05 2024-09-13T13:10:44.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.697765+0000", 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112879 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.697873+0000", 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108046 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.702510+0000", 2024-09-13T13:10:44.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0046364570000000001 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.702922+0000", 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00041236799999999998 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.821802+0000", 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11888041000000001 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.821861+0000", 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8616000000000003e-05 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.821915+0000", 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3649000000000001e-05 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237600 8.0 8:03089901:::benchmark_data_smithi052_101302_object109671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:37.704285+0000", 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 66.748408690000005, 2024-09-13T13:10:44.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.118060099, 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237600 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.704285+0000", 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.704285+0000", 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.704287+0000", 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0530000000000001e-06 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.704294+0000", 2024-09-13T13:10:44.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0599999999999996e-06 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.704305+0000", 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1568e-05 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.704313+0000", 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4070000000000004e-06 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:10:44.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.705486+0000", 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011737169999999999 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.705604+0000", 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011796 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.705725+0000", 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012087 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.708391+0000", 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026661520000000002 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.708887+0000", 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000495778 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.822220+0000", 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.113333076 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:10:44.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.822304+0000", 2024-09-13T13:10:44.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3436999999999994e-05 2024-09-13T13:10:44.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:10:44.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:10:44.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:10:44.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:37.822346+0000", 2024-09-13T13:10:44.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2021000000000003e-05 2024-09-13T13:10:44.590 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.590 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.590 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.590 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:10:44.590 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:10:44.590 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:10:44.627 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:44.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:44.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:44.766 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:44.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:44.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:44.784 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 29 16 59030 59014 127.137 84.875 0.00848568 0.0078607 2024-09-13T13:10:44.867 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:44.910 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:44.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:44.942 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:10:44.957 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T13:10:45.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:45.043 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:45.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:45.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:45.145 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:45.184 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:45.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:45.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[36682]: 6.18 scrub starts 2024-09-13T13:10:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[36682]: 6.18 scrub ok 2024-09-13T13:10:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[36682]: pgmap v462: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-13T13:10:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4121172371' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:10:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2145632469' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:10:45.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[36682]: from='client.35705 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:10:45.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[29471]: 6.18 scrub starts 2024-09-13T13:10:45.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[29471]: 6.18 scrub ok 2024-09-13T13:10:45.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[29471]: pgmap v462: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-13T13:10:45.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:45.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4121172371' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:10:45.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2145632469' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:10:45.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:44 smithi052 ceph-mon[29471]: from='client.35705 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:10:45.286 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:45.320 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:44 smithi167 ceph-mon[33178]: 6.18 scrub starts 2024-09-13T13:10:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:44 smithi167 ceph-mon[33178]: 6.18 scrub ok 2024-09-13T13:10:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:44 smithi167 ceph-mon[33178]: pgmap v462: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-13T13:10:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:44 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4121172371' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:10:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2145632469' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:10:45.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:44 smithi167 ceph-mon[33178]: from='client.35705 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:10:45.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:45.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:45.423 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:45.458 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:45.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:45.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:45.560 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:45.599 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:45.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:45.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:45.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:45.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:45.745 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:10:45.778 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:10:45.784 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 30 16 60607 60591 126.184 98.5625 0.0290005 0.00792032 2024-09-13T13:10:45.837 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:45.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:45.880 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:45.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:45.981 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:45.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:45 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:45 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4121172371' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:10:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:45 smithi052 ceph-mon[36682]: from='client.35705 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:10:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:45 smithi052 ceph-mon[36682]: osdmap e256: 8 total, 8 up, 8 in 2024-09-13T13:10:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:45 smithi052 ceph-mon[36682]: 5.6 scrub starts 2024-09-13T13:10:45.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:45 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:45.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:45 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4121172371' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:10:45.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:45 smithi052 ceph-mon[29471]: from='client.35705 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:10:45.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:45 smithi052 ceph-mon[29471]: osdmap e256: 8 total, 8 up, 8 in 2024-09-13T13:10:45.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:45 smithi052 ceph-mon[29471]: 5.6 scrub starts 2024-09-13T13:10:46.016 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:46.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:45 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:45 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4121172371' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:10:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:45 smithi167 ceph-mon[33178]: from='client.35705 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:10:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:45 smithi167 ceph-mon[33178]: osdmap e256: 8 total, 8 up, 8 in 2024-09-13T13:10:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:45 smithi167 ceph-mon[33178]: 5.6 scrub starts 2024-09-13T13:10:46.118 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:46.160 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:10:46.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:46.166 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:46.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:46.267 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:46.313 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:46.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:46.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:46.415 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:46.453 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:46.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:46.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:10:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:10:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:10:46.553 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:46.591 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:46.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:46.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:46.693 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:46.729 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:46.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:46.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:46.784 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 31 16 61982 61966 124.886 85.9375 0.0129218 0.00800175 2024-09-13T13:10:46.808 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:10:46.808 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:10:46.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.808 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.15789.0:62009 9.c e256/241 9:3233f1a0:::benchmark_data_smithi052_128427_object62008:head v 256'3812, mlcod=256'3809)", 2024-09-13T13:10:46.808 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:46.804146+0000", 2024-09-13T13:10:46.809 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0016819319999999999, 2024-09-13T13:10:46.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001709201, 2024-09-13T13:10:46.809 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:46.809 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:46.809 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:10:46.809 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:46.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:46.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.804146+0000", 2024-09-13T13:10:46.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:46.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.804146+0000", 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.804147+0000", 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.362e-06 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.804196+0000", 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9243999999999998e-05 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.804210+0000", 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4e-05 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.804217+0000", 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.754e-06 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.805584+0000", 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001366698 2024-09-13T13:10:46.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:46.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.805679+0000", 2024-09-13T13:10:46.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5618999999999994e-05 2024-09-13T13:10:46.812 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:46.812 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:46.812 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:46.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.812 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.15789.0:62002 9.c e256/241 9:35586591:::benchmark_data_smithi052_128427_object62001:head v 256'3811, mlcod=256'3809)", 2024-09-13T13:10:46.812 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:46.798378+0000", 2024-09-13T13:10:46.812 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0074496839999999998, 2024-09-13T13:10:46.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075319829999999999, 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.798378+0000", 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.798378+0000", 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:46.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.798379+0000", 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2449999999999999e-06 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.798420+0000", 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0633000000000002e-05 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.798430+0000", 2024-09-13T13:10:46.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0309000000000001e-05 2024-09-13T13:10:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.798436+0000", 2024-09-13T13:10:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8769999999999998e-06 2024-09-13T13:10:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.800582+0000", 2024-09-13T13:10:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002146387 2024-09-13T13:10:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.800627+0000", 2024-09-13T13:10:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4128999999999997e-05 2024-09-13T13:10:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.15789.0:62001 9.4 e256/241 9:21badcaa:::benchmark_data_smithi052_128427_object62000:head v 256'3856, mlcod=256'3854)", 2024-09-13T13:10:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:46.800464+0000", 2024-09-13T13:10:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0053639539999999998, 2024-09-13T13:10:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0054923660000000003, 2024-09-13T13:10:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.800464+0000", 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.800464+0000", 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.800474+0000", 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8560000000000002e-06 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.800506+0000", 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2382999999999998e-05 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.800517+0000", 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0811e-05 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.800523+0000", 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2770000000000002e-06 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.801546+0000", 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010224640000000001 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.801587+0000", 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1303000000000001e-05 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.15789.0:62004 9.4 e256/241 9:2c457c6a:::benchmark_data_smithi052_128427_object62003:head v 256'3857, mlcod=256'3854)", 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:46.803307+0000", 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.002520179, 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026932900000000001, 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.803307+0000", 2024-09-13T13:10:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.803307+0000", 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.803309+0000", 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.412e-06 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.803351+0000", 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2344e-05 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.803363+0000", 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2023e-05 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.803370+0000", 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9959999999999996e-06 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.805550+0000", 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021798030000000001 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:46.805651+0000", 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010061200000000001 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 4 2024-09-13T13:10:46.823 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:10:46.831 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:46.861 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:46.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:46.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:46.963 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:46.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:46 smithi052 ceph-mon[29471]: pgmap v464: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 742 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T13:10:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:46 smithi052 ceph-mon[36682]: pgmap v464: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 742 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T13:10:46.990 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:46.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:46.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:47.092 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:47.134 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:47.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:47.236 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:47.277 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:47.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:47.302 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:46 smithi167 ceph-mon[33178]: pgmap v464: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 742 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T13:10:47.303 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:46 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:10:47.303 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:46 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:10:47.380 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:47.422 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:47.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:47.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:47 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:10:47.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:47 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:10:47.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:47 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:10:47.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:47 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:10:47.523 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:47.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:47.569 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:47.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:47.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:47.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:47.708 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:47.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:47.766 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T13:10:47.784 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 32 16 63263 63247 123.485 80.0625 0.00762372 0.00808596 2024-09-13T13:10:47.809 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:47.844 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:47.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:47.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:47.946 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:47.976 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:47.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:48.081 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:48.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:48.114 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:48.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:48.171 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:10:48.215 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:48.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:48.243 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:48.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:48.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:48 smithi167 ceph-mon[33178]: osdmap e257: 8 total, 8 up, 8 in 2024-09-13T13:10:48.343 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:48.377 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:48.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:48.479 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:48.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:48 smithi052 ceph-mon[36682]: osdmap e257: 8 total, 8 up, 8 in 2024-09-13T13:10:48.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:48 smithi052 ceph-mon[29471]: osdmap e257: 8 total, 8 up, 8 in 2024-09-13T13:10:48.506 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:48.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:48.608 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:48.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:48.645 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:48.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:48.747 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:48.784 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 33 15 64539 64524 122.162 79.8125 0.00876687 0.00818055 2024-09-13T13:10:48.785 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:48.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:48.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:48.795 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:10:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:10:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:10:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:10:48.796 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:10:48.886 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:48.925 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:48.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:49.027 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:49.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:49.064 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:49.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:49.165 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:49.202 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:49.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:49.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:49 smithi052 ceph-mon[36682]: pgmap v466: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-13T13:10:49.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:49 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:10:49.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:49 smithi052 ceph-mon[29471]: pgmap v466: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-13T13:10:49.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:49 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:10:49.303 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:49.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:49 smithi167 ceph-mon[33178]: pgmap v466: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-13T13:10:49.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:49 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:10:49.346 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:49.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:49.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:49.448 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:49.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:49.489 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:49.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:49.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T13:10:49.590 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:49.627 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:49.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:49.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:49.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:49.756 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:49.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:49.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:49.786 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 34 16 66061 66045 121.358 95.0625 0.0102695 0.00823457 2024-09-13T13:10:49.796 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T13:10:49.858 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:49.864 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:49.922 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:49.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:49.947 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:10:50.025 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:50.063 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:50.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:50.165 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:50.196 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:50.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:50.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:50.322 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:50.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:50.350 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:50 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:10:50.350 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:50 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:10:50.350 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:50 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:10:50.350 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:50 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:10:50.423 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:50.462 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:50.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:50.522 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:50.550 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:10:50.550 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:10:50.564 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:50.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:50 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:10:50.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:50 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:10:50.576 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:10:50.576 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:10:50.587 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:50.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:50.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:50.596 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:10:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:10:50.629 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:10:50.629 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:10:50.665 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:10:50.665 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:10:50.690 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:50.701 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:10:50.701 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:10:50.722 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:50.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:50.736 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:10:50.737 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:10:50.771 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:10:50.771 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:10:50.786 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 35 16 67188 67172 119.903 70.4375 0.0123631 0.00833514 2024-09-13T13:10:50.825 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:50.872 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:50.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:50.974 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:51.017 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:51.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:51.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:51.119 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:51.155 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:51.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:51.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:51.257 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:51.294 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:51.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:51.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:51.396 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:51.433 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:51.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:51.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:51.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:51 smithi052 ceph-mon[29471]: pgmap v467: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 133 MiB/s wr, 2.13k op/s 2024-09-13T13:10:51.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:51 smithi052 ceph-mon[29471]: osdmap e258: 8 total, 8 up, 8 in 2024-09-13T13:10:51.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:51 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2642303386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:51.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:51 smithi052 ceph-mon[36682]: pgmap v467: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 133 MiB/s wr, 2.13k op/s 2024-09-13T13:10:51.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:51 smithi052 ceph-mon[36682]: osdmap e258: 8 total, 8 up, 8 in 2024-09-13T13:10:51.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:51 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2642303386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:51.535 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:51.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:51.570 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:51.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:51.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:51 smithi167 ceph-mon[33178]: pgmap v467: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 133 MiB/s wr, 2.13k op/s 2024-09-13T13:10:51.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:51 smithi167 ceph-mon[33178]: osdmap e258: 8 total, 8 up, 8 in 2024-09-13T13:10:51.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:51 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2642303386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:51.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:51.707 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:51.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:51.745 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:10:51.788 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 36 16 68593 68577 119.007 87.8125 0.0174395 0.0083974 2024-09-13T13:10:51.807 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:51.859 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:51.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:51.961 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:51.992 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:51.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:51.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:52.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:52.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:52.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:52.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:52.226 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:52.238 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:52.261 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:52.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:52.363 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:52.411 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:52.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:52.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:52.513 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:52.551 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:52.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:52.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:52.653 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:52.694 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:52.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:52.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:52.788 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 37 16 69849 69833 117.913 78.5 0.0116019 0.00847577 2024-09-13T13:10:52.795 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:52.829 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:52.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:52.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:52.834 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:10:52.834 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:10:52.880 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T13:10:52.932 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:52.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:52.971 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:52.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:53.072 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:53.101 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:53 smithi052 ceph-mon[36682]: pgmap v469: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 746 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-13T13:10:53.101 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:53 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/60503413' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:10:53.101 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:53 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/60503413' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:10:53.101 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:53 smithi052 ceph-mon[36682]: osdmap e259: 8 total, 8 up, 8 in 2024-09-13T13:10:53.101 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:53 smithi052 ceph-mon[29471]: pgmap v469: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 746 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-13T13:10:53.101 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/60503413' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:10:53.101 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/60503413' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:10:53.101 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:53 smithi052 ceph-mon[29471]: osdmap e259: 8 total, 8 up, 8 in 2024-09-13T13:10:53.111 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:53.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:53.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:53.213 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:53.252 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:53.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:53.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:53.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:53.386 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:53.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:53.488 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:53.515 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:53.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:53.554 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:53 smithi167 ceph-mon[33178]: pgmap v469: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 746 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-13T13:10:53.554 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:53 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/60503413' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:10:53.554 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:53 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/60503413' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:10:53.554 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:53 smithi167 ceph-mon[33178]: osdmap e259: 8 total, 8 up, 8 in 2024-09-13T13:10:53.608 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T13:10:53.617 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:53.654 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:53.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:53.756 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:53.788 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 38 16 71130 71114 116.917 80.0625 0.0135571 0.00854726 2024-09-13T13:10:53.795 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:53.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:53.812 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:10:53.897 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:53.912 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T13:10:53.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:53.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:54.045 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:54.056 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:10:54.099 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:54.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:54.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:54.105 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:54 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1745089532' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:54.105 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:54 smithi052 ceph-mon[29471]: from='client.25723 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:54.109 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:10:54.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:54.232 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:54.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:54.334 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:54.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:10:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:10:54.353 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:54 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1745089532' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:54.353 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:54 smithi167 ceph-mon[33178]: from='client.25723 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:54.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:54 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1745089532' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:54.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:54 smithi052 ceph-mon[36682]: from='client.25723 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:10:54.390 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:54.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:54.492 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:54.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:54.531 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:54.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:54.632 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:54.670 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:54.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:54.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:54.676 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:10:54.676 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:10:54.773 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:54.790 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 39 16 72392 72376 115.937 78.875 0.00785258 0.00862038 2024-09-13T13:10:54.812 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:54.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:54.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:54.958 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:54.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:55.060 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:55.097 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:55.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:55.199 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:55.208 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:55.221 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:55.230 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:55.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:55.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:55 smithi167 ceph-mon[33178]: pgmap v471: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 115 MiB/s wr, 1.83k op/s 2024-09-13T13:10:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:55 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:55 smithi167 ceph-mon[33178]: from='client.25723 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:10:55.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:55 smithi167 ceph-mon[33178]: osdmap e260: 8 total, 8 up, 8 in 2024-09-13T13:10:55.332 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:55.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:55 smithi052 ceph-mon[29471]: pgmap v471: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 115 MiB/s wr, 1.83k op/s 2024-09-13T13:10:55.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:55 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:55.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:55.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:55 smithi052 ceph-mon[29471]: from='client.25723 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:10:55.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:55 smithi052 ceph-mon[29471]: osdmap e260: 8 total, 8 up, 8 in 2024-09-13T13:10:55.358 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:55 smithi052 ceph-mon[36682]: pgmap v471: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 115 MiB/s wr, 1.83k op/s 2024-09-13T13:10:55.358 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:55 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:10:55.358 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:10:55.358 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:55 smithi052 ceph-mon[36682]: from='client.25723 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:10:55.358 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:55 smithi052 ceph-mon[36682]: osdmap e260: 8 total, 8 up, 8 in 2024-09-13T13:10:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:55.370 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:55.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:55.375 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T13:10:55.471 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:55.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:55.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:55.652 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:55.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:55.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:55.788 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:55.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:55.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:55.790 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:10:55.790080+0000 min lat: 0.00134564 max lat: 0.125715 avg lat: 0.00868538 2024-09-13T13:10:55.790 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:10:55.790 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 40 16 73682 73666 115.054 80.625 0.00757934 0.00868538 2024-09-13T13:10:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:10:55.890 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:55.937 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:55.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:56.039 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:56.071 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:56.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:56.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:56 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:10:56.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:56 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:10:56.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:56 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:10:56.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:56 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:10:56.173 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:56.220 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:56.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:56.303 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:56 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:10:56.303 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:56 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:10:56.322 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:56.325 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:10:56.325 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:10:56.365 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:10:56.366 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:10:56.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:56.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:56.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:56.410 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:10:56.411 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:10:56.411 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:10:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:10:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:10:56.447 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:10:56.447 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:10:56.460 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:10:56.460 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:10:56.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:56.484 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:10:56.485 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:10:56.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:56.506 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:56.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:56.520 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:10:56.520 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:10:56.552 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:10:56.552 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:10:56.590 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:10:56.590 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:10:56.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:56.639 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:56.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:56.740 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:56.777 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:56.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:56.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:56.790 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 41 16 74912 74896 114.123 76.875 0.00994463 0.00875712 2024-09-13T13:10:56.880 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:56.922 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:56.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:56.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:57.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:57.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:57.059 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:57.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:57.127 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T13:10:57.153 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:10:57.160 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:57.198 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:57.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:57.202 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T13:10:57.300 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:57.351 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:57.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:57.423 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:57 smithi052 ceph-mon[36682]: pgmap v473: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-13T13:10:57.423 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:57 smithi052 ceph-mon[36682]: osdmap e261: 8 total, 8 up, 8 in 2024-09-13T13:10:57.423 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:57 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2010949858' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:57.423 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:57 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3522283636' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:10:57.423 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:57 smithi052 ceph-mon[36682]: from='client.25729 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:10:57.425 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:57 smithi052 ceph-mon[29471]: pgmap v473: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-13T13:10:57.425 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:57 smithi052 ceph-mon[29471]: osdmap e261: 8 total, 8 up, 8 in 2024-09-13T13:10:57.425 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:57 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2010949858' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:57.425 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:57 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3522283636' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:10:57.426 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:57 smithi052 ceph-mon[29471]: from='client.25729 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:10:57.453 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:57.508 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:57.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:57 smithi167 ceph-mon[33178]: pgmap v473: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-13T13:10:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:57 smithi167 ceph-mon[33178]: osdmap e261: 8 total, 8 up, 8 in 2024-09-13T13:10:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:57 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2010949858' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:10:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:57 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3522283636' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:10:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:57 smithi167 ceph-mon[33178]: from='client.25729 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:10:57.611 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:57.659 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:10:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:57.690 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:57.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:57.790 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 42 16 76208 76192 113.334 81 0.00852084 0.00881898 2024-09-13T13:10:57.792 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:57.832 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:57.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:57.934 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:57.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:57.972 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:57.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:58.073 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:58.111 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:58.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:58.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:58.213 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:10:58.248 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:58.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:58.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:10:58.272 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:10:58.272 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:10:58.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:10:58.272 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:10:58.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.273 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:79331 8.8 8:13289dff:::benchmark_data_smithi052_101302_object79330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T13:10:58.273 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:17.879266+0000", 2024-09-13T13:10:58.273 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 220.391092773, 2024-09-13T13:10:58.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084187697000000006, 2024-09-13T13:10:58.273 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.273 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.273 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:10:58.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:10:58.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:58.273 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 79331 2024-09-13T13:10:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879266+0000", 2024-09-13T13:10:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879266+0000", 2024-09-13T13:10:58.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879268+0000", 2024-09-13T13:10:58.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.454e-06 2024-09-13T13:10:58.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879330+0000", 2024-09-13T13:10:58.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2682999999999996e-05 2024-09-13T13:10:58.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879339+0000", 2024-09-13T13:10:58.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6389999999999998e-06 2024-09-13T13:10:58.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.879344+0000", 2024-09-13T13:10:58.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.977e-06 2024-09-13T13:10:58.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.882487+0000", 2024-09-13T13:10:58.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003142677 2024-09-13T13:10:58.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.882606+0000", 2024-09-13T13:10:58.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011966 2024-09-13T13:10:58.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:10:58.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.882740+0000", 2024-09-13T13:10:58.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013381299999999999 2024-09-13T13:10:58.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:10:58.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.885073+0000", 2024-09-13T13:10:58.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023330849999999999 2024-09-13T13:10:58.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.885588+0000", 2024-09-13T13:10:58.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051493199999999995 2024-09-13T13:10:58.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.963389+0000", 2024-09-13T13:10:58.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077800367999999995 2024-09-13T13:10:58.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.963428+0000", 2024-09-13T13:10:58.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9514999999999997e-05 2024-09-13T13:10:58.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:17.963454+0000", 2024-09-13T13:10:58.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5894e-05 2024-09-13T13:10:58.280 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.280 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.280 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.280 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135962 8.3 e207/169 8:c288bb32:::benchmark_data_smithi052_101302_object8033:head v 207'8600, mlcod=207'8598)", 2024-09-13T13:10:58.281 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.798736+0000", 2024-09-13T13:10:58.281 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.47162313800001, 2024-09-13T13:10:58.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.214192363, 2024-09-13T13:10:58.281 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.281 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.281 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.281 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798736+0000", 2024-09-13T13:10:58.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798736+0000", 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798738+0000", 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.542e-06 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798746+0000", 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7299999999999994e-06 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798754+0000", 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7770000000000002e-06 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798760+0000", 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7769999999999999e-06 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798809+0000", 2024-09-13T13:10:58.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8714000000000001e-05 2024-09-13T13:10:58.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798822+0000", 2024-09-13T13:10:58.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3737000000000001e-05 2024-09-13T13:10:58.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012889+0000", 2024-09-13T13:10:58.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.21406626500000001 2024-09-13T13:10:58.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012928+0000", 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9820999999999997e-05 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:135964 8.8 8:1e05dc47:::benchmark_data_smithi052_101302_object8035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.800142+0000", 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.47021704299999, 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20392829400000001, 2024-09-13T13:10:58.285 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135964 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800142+0000", 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800142+0000", 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800144+0000", 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5719999999999999e-06 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800149+0000", 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2059999999999998e-06 2024-09-13T13:10:58.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800159+0000", 2024-09-13T13:10:58.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9110000000000005e-06 2024-09-13T13:10:58.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800165+0000", 2024-09-13T13:10:58.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6640000000000001e-06 2024-09-13T13:10:58.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800242+0000", 2024-09-13T13:10:58.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6415999999999995e-05 2024-09-13T13:10:58.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800369+0000", 2024-09-13T13:10:58.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000127238 2024-09-13T13:10:58.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:10:58.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800490+0000", 2024-09-13T13:10:58.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120827 2024-09-13T13:10:58.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:10:58.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803447+0000", 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029568620000000002 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803750+0000", 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030363800000000001 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003963+0000", 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200212532 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004013+0000", 2024-09-13T13:10:58.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9765999999999998e-05 2024-09-13T13:10:58.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004070+0000", 2024-09-13T13:10:58.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7661999999999998e-05 2024-09-13T13:10:58.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.292 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:135967 8.4 8:2cd9f03c:::benchmark_data_smithi052_101302_object8038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.800185+0000", 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.47017370099999, 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20402237600000001, 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135967 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800185+0000", 2024-09-13T13:10:58.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800185+0000", 2024-09-13T13:10:58.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800186+0000", 2024-09-13T13:10:58.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8999999999999996e-07 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800189+0000", 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.689e-06 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800194+0000", 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1309999999999997e-06 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800199+0000", 2024-09-13T13:10:58.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6959999999999996e-06 2024-09-13T13:10:58.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800250+0000", 2024-09-13T13:10:58.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1402e-05 2024-09-13T13:10:58.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800389+0000", 2024-09-13T13:10:58.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000138596 2024-09-13T13:10:58.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800490+0000", 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010134899999999999 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803436+0000", 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029464529999999999 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004082+0000", 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200645886 2024-09-13T13:10:58.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004122+0000", 2024-09-13T13:10:58.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9864000000000003e-05 2024-09-13T13:10:58.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004169+0000", 2024-09-13T13:10:58.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7052000000000003e-05 2024-09-13T13:10:58.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004208+0000", 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8668000000000003e-05 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135966 8.3 e207/169 8:cd1f5880:::benchmark_data_smithi052_101302_object8037:head v 207'8601, mlcod=207'8598)", 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.802377+0000", 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.46798244600001, 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.21058527299999999, 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.299 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802377+0000", 2024-09-13T13:10:58.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802377+0000", 2024-09-13T13:10:58.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802379+0000", 2024-09-13T13:10:58.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.827e-06 2024-09-13T13:10:58.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802390+0000", 2024-09-13T13:10:58.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1623999999999999e-05 2024-09-13T13:10:58.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802402+0000", 2024-09-13T13:10:58.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1564e-05 2024-09-13T13:10:58.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802410+0000", 2024-09-13T13:10:58.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2719999999999997e-06 2024-09-13T13:10:58.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802483+0000", 2024-09-13T13:10:58.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2840000000000001e-05 2024-09-13T13:10:58.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802503+0000", 2024-09-13T13:10:58.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0452e-05 2024-09-13T13:10:58.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012933+0000", 2024-09-13T13:10:58.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.210429965 2024-09-13T13:10:58.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012962+0000", 2024-09-13T13:10:58.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8728999999999999e-05 2024-09-13T13:10:58.304 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.304 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.304 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.304 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135968 8.2 e207/169 8:4f16015f:::benchmark_data_smithi052_101302_object8039:head v 207'8502, mlcod=207'8500)", 2024-09-13T13:10:58.304 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.803228+0000", 2024-09-13T13:10:58.304 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.46713142900001, 2024-09-13T13:10:58.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20068414800000001, 2024-09-13T13:10:58.305 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.305 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.305 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.305 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803228+0000", 2024-09-13T13:10:58.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803228+0000", 2024-09-13T13:10:58.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803229+0000", 2024-09-13T13:10:58.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2399999999999996e-07 2024-09-13T13:10:58.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803234+0000", 2024-09-13T13:10:58.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6400000000000002e-06 2024-09-13T13:10:58.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803244+0000", 2024-09-13T13:10:58.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6770000000000007e-06 2024-09-13T13:10:58.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803251+0000", 2024-09-13T13:10:58.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0149999999999998e-06 2024-09-13T13:10:58.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803344+0000", 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2656000000000002e-05 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803370+0000", 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6156000000000002e-05 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003858+0000", 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20048846400000001 2024-09-13T13:10:58.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003912+0000", 2024-09-13T13:10:58.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3616000000000003e-05 2024-09-13T13:10:58.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.309 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.309 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.25222.0:135964 8.8 e207/169 ondisk, result = 0)", 2024-09-13T13:10:58.309 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.803563+0000", 2024-09-13T13:10:58.309 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.46679607900001, 2024-09-13T13:10:58.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200512417, 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803563+0000", 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803563+0000", 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.804856+0000", 2024-09-13T13:10:58.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001293055 2024-09-13T13:10:58.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.804861+0000", 2024-09-13T13:10:58.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5800000000000002e-06 2024-09-13T13:10:58.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003842+0000", 2024-09-13T13:10:58.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19898099699999999 2024-09-13T13:10:58.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003857+0000", 2024-09-13T13:10:58.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5021999999999999e-05 2024-09-13T13:10:58.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003949+0000", 2024-09-13T13:10:58.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1824999999999996e-05 2024-09-13T13:10:58.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003956+0000", 2024-09-13T13:10:58.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9070000000000007e-06 2024-09-13T13:10:58.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004075+0000", 2024-09-13T13:10:58.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011903100000000001 2024-09-13T13:10:58.317 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.317 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.317 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.317 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.25222.0:135967 8.4 e207/169 ondisk, result = 0)", 2024-09-13T13:10:58.318 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.803807+0000", 2024-09-13T13:10:58.318 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.46655169900001, 2024-09-13T13:10:58.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200283038, 2024-09-13T13:10:58.318 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.318 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.318 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:10:58.318 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803807+0000", 2024-09-13T13:10:58.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803807+0000", 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803808+0000", 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.032e-06 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803812+0000", 2024-09-13T13:10:58.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0069999999999999e-06 2024-09-13T13:10:58.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803820+0000", 2024-09-13T13:10:58.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5599999999999996e-06 2024-09-13T13:10:58.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003847+0000", 2024-09-13T13:10:58.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20002690300000001 2024-09-13T13:10:58.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004067+0000", 2024-09-13T13:10:58.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021978000000000001 2024-09-13T13:10:58.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004075+0000", 2024-09-13T13:10:58.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6430000000000001e-06 2024-09-13T13:10:58.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004090+0000", 2024-09-13T13:10:58.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5113000000000001e-05 2024-09-13T13:10:58.322 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.322 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.322 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.322 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135970 8.3 e207/169 8:ce0cc5da:::benchmark_data_smithi052_101302_object8041:head v 207'8602, mlcod=207'8598)", 2024-09-13T13:10:58.322 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.806702+0000", 2024-09-13T13:10:58.322 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.46365666599999, 2024-09-13T13:10:58.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20628901199999999, 2024-09-13T13:10:58.323 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.323 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.323 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.323 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806702+0000", 2024-09-13T13:10:58.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806702+0000", 2024-09-13T13:10:58.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806703+0000", 2024-09-13T13:10:58.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0109999999999999e-06 2024-09-13T13:10:58.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806711+0000", 2024-09-13T13:10:58.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.52e-06 2024-09-13T13:10:58.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003876+0000", 2024-09-13T13:10:58.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19716455899999999 2024-09-13T13:10:58.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003884+0000", 2024-09-13T13:10:58.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2269999999999993e-06 2024-09-13T13:10:58.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003930+0000", 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6368000000000001e-05 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003955+0000", 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4374000000000001e-05 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012964+0000", 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090092720000000005 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012991+0000", 2024-09-13T13:10:58.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7681000000000001e-05 2024-09-13T13:10:58.327 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.327 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.327 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.327 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:29372 9.f 9:f38803ea:::benchmark_data_smithi052_128427_object29371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-13T13:10:58.327 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:27.216958+0000", 2024-09-13T13:10:58.327 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.053400977999999, 2024-09-13T13:10:58.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082973796000000002, 2024-09-13T13:10:58.327 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 29372 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.216958+0000", 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.216958+0000", 2024-09-13T13:10:58.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.216959+0000", 2024-09-13T13:10:58.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7000000000000003e-07 2024-09-13T13:10:58.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.217035+0000", 2024-09-13T13:10:58.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6367999999999998e-05 2024-09-13T13:10:58.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.267229+0000", 2024-09-13T13:10:58.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050193854000000003 2024-09-13T13:10:58.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.267247+0000", 2024-09-13T13:10:58.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7884999999999999e-05 2024-09-13T13:10:58.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.268796+0000", 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015484450000000001 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.268919+0000", 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123664 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.269035+0000", 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115641 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:10:58.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.271234+0000", 2024-09-13T13:10:58.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021986800000000002 2024-09-13T13:10:58.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.273174+0000", 2024-09-13T13:10:58.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019405850000000001 2024-09-13T13:10:58.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.299795+0000", 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026620923000000001 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.299889+0000", 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4207000000000001e-05 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:27.299932+0000", 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2574000000000003e-05 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.333 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:66603 9.f 9:f57ab56d:::benchmark_data_smithi052_128427_object66602:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:50.216721+0000", 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.0536376529999991, 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.110807181, 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 66603 2024-09-13T13:10:58.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.216721+0000", 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.216721+0000", 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.216723+0000", 2024-09-13T13:10:58.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5510000000000001e-06 2024-09-13T13:10:58.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.216815+0000", 2024-09-13T13:10:58.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2025000000000001e-05 2024-09-13T13:10:58.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.216827+0000", 2024-09-13T13:10:58.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1922e-05 2024-09-13T13:10:58.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.216835+0000", 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8509999999999999e-06 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.216893+0000", 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8514000000000003e-05 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.217007+0000", 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011394400000000001 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:10:58.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.217127+0000", 2024-09-13T13:10:58.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119306 2024-09-13T13:10:58.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:10:58.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.219454+0000", 2024-09-13T13:10:58.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023274120000000001 2024-09-13T13:10:58.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.224341+0000", 2024-09-13T13:10:58.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048871909999999999 2024-09-13T13:10:58.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239277+0000", 2024-09-13T13:10:58.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014935887 2024-09-13T13:10:58.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.327471+0000", 2024-09-13T13:10:58.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088193650999999998 2024-09-13T13:10:58.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.327529+0000", 2024-09-13T13:10:58.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7927000000000003e-05 2024-09-13T13:10:58.340 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.340 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.340 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.340 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:66607 9.f 9:fb21c6f2:::benchmark_data_smithi052_128427_object66606:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-13T13:10:58.340 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:50.221233+0000", 2024-09-13T13:10:58.340 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.0491257770000004, 2024-09-13T13:10:58.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10811032900000001, 2024-09-13T13:10:58.340 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.341 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.341 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.341 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:10:58.341 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:10:58.341 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:10:58.341 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 66607 2024-09-13T13:10:58.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.342 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.221233+0000", 2024-09-13T13:10:58.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.221233+0000", 2024-09-13T13:10:58.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.221235+0000", 2024-09-13T13:10:58.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.018e-06 2024-09-13T13:10:58.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.221274+0000", 2024-09-13T13:10:58.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8322999999999999e-05 2024-09-13T13:10:58.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.221285+0000", 2024-09-13T13:10:58.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1297999999999999e-05 2024-09-13T13:10:58.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.221291+0000", 2024-09-13T13:10:58.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7880000000000003e-06 2024-09-13T13:10:58.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.221338+0000", 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7723999999999999e-05 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.221437+0000", 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8737e-05 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.221521+0000", 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3599999999999999e-05 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:10:58.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.223340+0000", 2024-09-13T13:10:58.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001819504 2024-09-13T13:10:58.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329022+0000", 2024-09-13T13:10:58.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10568166 2024-09-13T13:10:58.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329092+0000", 2024-09-13T13:10:58.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9754999999999999e-05 2024-09-13T13:10:58.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329296+0000", 2024-09-13T13:10:58.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020390100000000001 2024-09-13T13:10:58.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329344+0000", 2024-09-13T13:10:58.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8021e-05 2024-09-13T13:10:58.347 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.347 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.347 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:66612 9.a 9:524f10c0:::benchmark_data_smithi052_128427_object66611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-13T13:10:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:50.224860+0000", 2024-09-13T13:10:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.0454986220000002, 2024-09-13T13:10:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.110938143, 2024-09-13T13:10:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:10:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:10:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:10:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 66612 2024-09-13T13:10:58.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.224860+0000", 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.224860+0000", 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.224862+0000", 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4130000000000001e-06 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.224968+0000", 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106507 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.224976+0000", 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4170000000000003e-06 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.224983+0000", 2024-09-13T13:10:58.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9800000000000001e-06 2024-09-13T13:10:58.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225032+0000", 2024-09-13T13:10:58.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9067999999999998e-05 2024-09-13T13:10:58.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225137+0000", 2024-09-13T13:10:58.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104864 2024-09-13T13:10:58.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,7", 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225238+0000", 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101121 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.228502+0000", 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032637260000000002 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.326791+0000", 2024-09-13T13:10:58.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098289073000000005 2024-09-13T13:10:58.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.353 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:58.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.335683+0000", 2024-09-13T13:10:58.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088924750000000004 2024-09-13T13:10:58.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.335742+0000", 2024-09-13T13:10:58.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8768999999999999e-05 2024-09-13T13:10:58.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.335799+0000", 2024-09-13T13:10:58.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6730000000000001e-05 2024-09-13T13:10:58.355 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.355 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.356 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.356 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:66614 9.f 9:ffd5a164:::benchmark_data_smithi052_128427_object66613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-13T13:10:58.356 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:50.224997+0000", 2024-09-13T13:10:58.356 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.045362184, 2024-09-13T13:10:58.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.104484643, 2024-09-13T13:10:58.356 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.356 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.356 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.356 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:10:58.356 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:10:58.356 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:10:58.357 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 66614 2024-09-13T13:10:58.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.357 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.224997+0000", 2024-09-13T13:10:58.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.224997+0000", 2024-09-13T13:10:58.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.224998+0000", 2024-09-13T13:10:58.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5300000000000003e-07 2024-09-13T13:10:58.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225031+0000", 2024-09-13T13:10:58.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3485999999999998e-05 2024-09-13T13:10:58.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225038+0000", 2024-09-13T13:10:58.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4370000000000004e-06 2024-09-13T13:10:58.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225042+0000", 2024-09-13T13:10:58.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9809999999999997e-06 2024-09-13T13:10:58.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225086+0000", 2024-09-13T13:10:58.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4271999999999999e-05 2024-09-13T13:10:58.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225192+0000", 2024-09-13T13:10:58.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105914 2024-09-13T13:10:58.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:10:58.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225269+0000", 2024-09-13T13:10:58.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6773000000000005e-05 2024-09-13T13:10:58.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:10:58.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.227012+0000", 2024-09-13T13:10:58.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017437399999999999 2024-09-13T13:10:58.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329059+0000", 2024-09-13T13:10:58.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10204643100000001 2024-09-13T13:10:58.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329388+0000", 2024-09-13T13:10:58.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032888200000000001 2024-09-13T13:10:58.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329448+0000", 2024-09-13T13:10:58.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.016e-05 2024-09-13T13:10:58.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329482+0000", 2024-09-13T13:10:58.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3714000000000003e-05 2024-09-13T13:10:58.362 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.362 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.362 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.362 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:66615 9.a 9:5862c55c:::benchmark_data_smithi052_128427_object66614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-13T13:10:58.363 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:50.225052+0000", 2024-09-13T13:10:58.363 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.0453071349999998, 2024-09-13T13:10:58.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.110873413, 2024-09-13T13:10:58.363 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.363 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.363 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:10:58.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:10:58.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:10:58.363 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 66615 2024-09-13T13:10:58.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.363 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225052+0000", 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225052+0000", 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225053+0000", 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8599999999999997e-07 2024-09-13T13:10:58.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225104+0000", 2024-09-13T13:10:58.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1411e-05 2024-09-13T13:10:58.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225110+0000", 2024-09-13T13:10:58.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.049e-06 2024-09-13T13:10:58.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.225114+0000", 2024-09-13T13:10:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8510000000000001e-06 2024-09-13T13:10:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.226941+0000", 2024-09-13T13:10:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018264329999999999 2024-09-13T13:10:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.227052+0000", 2024-09-13T13:10:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011122 2024-09-13T13:10:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,7", 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.227170+0000", 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118738 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.228623+0000", 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014526859999999999 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.328978+0000", 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10035503799999999 2024-09-13T13:10:58.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.335845+0000", 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068667329999999999 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.335891+0000", 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5655999999999998e-05 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.335925+0000", 2024-09-13T13:10:58.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4811999999999998e-05 2024-09-13T13:10:58.369 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.369 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.369 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.369 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.15789.0:66603 9.f e258/241 ondisk, result = 0)", 2024-09-13T13:10:58.369 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:50.239079+0000", 2024-09-13T13:10:58.369 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.0312805869999995, 2024-09-13T13:10:58.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088457279999999999, 2024-09-13T13:10:58.369 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.369 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.369 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:10:58.369 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239079+0000", 2024-09-13T13:10:58.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239079+0000", 2024-09-13T13:10:58.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239081+0000", 2024-09-13T13:10:58.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-09-13T13:10:58.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239091+0000", 2024-09-13T13:10:58.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6099999999999995e-06 2024-09-13T13:10:58.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239115+0000", 2024-09-13T13:10:58.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4465999999999999e-05 2024-09-13T13:10:58.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239140+0000", 2024-09-13T13:10:58.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4508999999999999e-05 2024-09-13T13:10:58.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239263+0000", 2024-09-13T13:10:58.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012338799999999999 2024-09-13T13:10:58.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239271+0000", 2024-09-13T13:10:58.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2449999999999998e-06 2024-09-13T13:10:58.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.327536+0000", 2024-09-13T13:10:58.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088264322000000006 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.15789.0:66618 9.9 e258/241 9:9336ebb3:::benchmark_data_smithi052_128427_object66617:head v 258'4306, mlcod=258'4304)", 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:50.239134+0000", 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.0312251579999998, 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090682262999999999, 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239134+0000", 2024-09-13T13:10:58.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239134+0000", 2024-09-13T13:10:58.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239137+0000", 2024-09-13T13:10:58.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0029999999999999e-06 2024-09-13T13:10:58.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239326+0000", 2024-09-13T13:10:58.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018859599999999999 2024-09-13T13:10:58.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239333+0000", 2024-09-13T13:10:58.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5730000000000002e-06 2024-09-13T13:10:58.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239338+0000", 2024-09-13T13:10:58.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9710000000000003e-06 2024-09-13T13:10:58.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.326836+0000", 2024-09-13T13:10:58.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.087497733999999994 2024-09-13T13:10:58.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.326857+0000", 2024-09-13T13:10:58.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1282000000000001e-05 2024-09-13T13:10:58.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:10:58.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329771+0000", 2024-09-13T13:10:58.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029140400000000001 2024-09-13T13:10:58.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329816+0000", 2024-09-13T13:10:58.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5064000000000001e-05 2024-09-13T13:10:58.378 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.378 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.378 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.379 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.15789.0:66607 9.f e258/241 ondisk, result = 0)", 2024-09-13T13:10:58.379 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:50.239184+0000", 2024-09-13T13:10:58.379 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.0311751650000005, 2024-09-13T13:10:58.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089845916999999997, 2024-09-13T13:10:58.379 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.379 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.379 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:10:58.379 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239184+0000", 2024-09-13T13:10:58.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239184+0000", 2024-09-13T13:10:58.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239185+0000", 2024-09-13T13:10:58.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5430000000000001e-06 2024-09-13T13:10:58.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239193+0000", 2024-09-13T13:10:58.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5100000000000001e-06 2024-09-13T13:10:58.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239210+0000", 2024-09-13T13:10:58.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7462999999999999e-05 2024-09-13T13:10:58.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239225+0000", 2024-09-13T13:10:58.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4323e-05 2024-09-13T13:10:58.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329010+0000", 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089785208000000005 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329017+0000", 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5810000000000002e-06 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329030+0000", 2024-09-13T13:10:58.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3288999999999999e-05 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.15789.0:66614 9.f e258/241 ondisk, result = 0)", 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:50.239251+0000", 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.0311079759999995, 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089814030000000003, 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239251+0000", 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239251+0000", 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239252+0000", 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.51e-07 2024-09-13T13:10:58.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239254+0000", 2024-09-13T13:10:58.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5560000000000002e-06 2024-09-13T13:10:58.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239260+0000", 2024-09-13T13:10:58.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8200000000000002e-06 2024-09-13T13:10:58.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.239264+0000", 2024-09-13T13:10:58.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3599999999999998e-06 2024-09-13T13:10:58.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329049+0000", 2024-09-13T13:10:58.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089784638999999999 2024-09-13T13:10:58.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329054+0000", 2024-09-13T13:10:58.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1259999999999997e-06 2024-09-13T13:10:58.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.329065+0000", 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1078e-05 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.15789.0:66612 9.a e258/241 ondisk, result = 0)", 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:10:50.243130+0000", 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.0272287250000005, 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083669001000000007, 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:10:58.387 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:10:58.388 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:10:58.388 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:10:58.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:10:58.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.243130+0000", 2024-09-13T13:10:58.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:10:58.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.243130+0000", 2024-09-13T13:10:58.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:10:58.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.243132+0000", 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.35e-06 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.243137+0000", 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4779999999999999e-06 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.326666+0000", 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083528757999999995 2024-09-13T13:10:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.326688+0000", 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2130999999999999e-05 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.326768+0000", 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0222999999999994e-05 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.326778+0000", 2024-09-13T13:10:58.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9830000000000008e-06 2024-09-13T13:10:58.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:10:58.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:10:58.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:10:58.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:10:50.326799+0000", 2024-09-13T13:10:58.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1078e-05 2024-09-13T13:10:58.391 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.391 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.391 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.391 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:10:58.391 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:10:58.391 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:10:58.394 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:58.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:58.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:58 smithi052 ceph-mon[36682]: from='client.25729 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:10:58.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:58 smithi052 ceph-mon[36682]: osdmap e262: 8 total, 8 up, 8 in 2024-09-13T13:10:58.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:58 smithi052 ceph-mon[36682]: osdmap e263: 8 total, 8 up, 8 in 2024-09-13T13:10:58.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:58 smithi052 ceph-mon[29471]: from='client.25729 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:10:58.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:58 smithi052 ceph-mon[29471]: osdmap e262: 8 total, 8 up, 8 in 2024-09-13T13:10:58.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:58 smithi052 ceph-mon[29471]: osdmap e263: 8 total, 8 up, 8 in 2024-09-13T13:10:58.496 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:58.535 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:58.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:58.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:58 smithi167 ceph-mon[33178]: from='client.25729 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:10:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:58 smithi167 ceph-mon[33178]: osdmap e262: 8 total, 8 up, 8 in 2024-09-13T13:10:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:58 smithi167 ceph-mon[33178]: osdmap e263: 8 total, 8 up, 8 in 2024-09-13T13:10:58.637 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:58.675 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:58.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:58.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:58.777 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:10:58.790 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 43 16 77412 77396 112.449 75.25 0.00818654 0.00888765 2024-09-13T13:10:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:58.815 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:58.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:10:58.916 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:10:58.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:58.943 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:58.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:10:59.044 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:10:59.065 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:59.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:59.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:10:59.167 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:59.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:59.209 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:59.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:59.247 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T13:10:59.310 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:59.333 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:59.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:59.435 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:10:59.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:59.472 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:59.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:10:59.485 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:10:59.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:10:59 smithi052 ceph-mon[36682]: pgmap v476: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-13T13:10:59.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:10:59 smithi052 ceph-mon[29471]: pgmap v476: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-13T13:10:59.573 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:10:59.574 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:10:59 smithi167 ceph-mon[33178]: pgmap v476: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-13T13:10:59.618 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:10:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:59.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:10:59.639 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:10:59.719 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:10:59.748 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:59.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:10:59.790 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 44 16 78501 78485 111.44 68.0625 0.00823895 0.00896844 2024-09-13T13:10:59.854 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:10:59.878 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T13:10:59.885 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:10:59.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:10:59.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:10:59.909 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:10:59.986 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:00.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:00.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:00.125 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:00.176 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:00.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:00.278 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:00.320 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:11:00.320 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:11:00.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.320 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:78957 9.3 9:c91327ce:::benchmark_data_smithi052_128427_object78956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-13T13:11:00.320 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:11:00.214487+0000", 2024-09-13T13:11:00.320 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.051491083999999999, 2024-09-13T13:11:00.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051519082000000001, 2024-09-13T13:11:00.320 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:00.321 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:00.321 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:11:00.321 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:00.321 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:11:00.321 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:11:00.321 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78957 2024-09-13T13:11:00.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.321 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:00.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:00.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214487+0000", 2024-09-13T13:11:00.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:00.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214487+0000", 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214489+0000", 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.429e-06 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214540+0000", 2024-09-13T13:11:00.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1116999999999999e-05 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214551+0000", 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0749e-05 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214556+0000", 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:00.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214601+0000", 2024-09-13T13:11:00.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4736e-05 2024-09-13T13:11:00.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:00.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214698+0000", 2024-09-13T13:11:00.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6831999999999997e-05 2024-09-13T13:11:00.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:11:00.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214804+0000", 2024-09-13T13:11:00.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106358 2024-09-13T13:11:00.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:00.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.220020+0000", 2024-09-13T13:11:00.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005216287 2024-09-13T13:11:00.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:00.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.223681+0000", 2024-09-13T13:11:00.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003660309 2024-09-13T13:11:00.325 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:00.325 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:00.325 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:00.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.326 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:78959 9.3 9:c1b6c049:::benchmark_data_smithi052_128427_object78958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-13T13:11:00.326 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:11:00.214597+0000", 2024-09-13T13:11:00.326 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.051381727000000002, 2024-09-13T13:11:00.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051474399999999997, 2024-09-13T13:11:00.326 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:00.326 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:00.326 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:11:00.326 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:00.326 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:11:00.326 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:11:00.327 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78959 2024-09-13T13:11:00.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.327 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:00.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:00.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214597+0000", 2024-09-13T13:11:00.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:00.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:00.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214597+0000", 2024-09-13T13:11:00.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214598+0000", 2024-09-13T13:11:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3600000000000002e-07 2024-09-13T13:11:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214668+0000", 2024-09-13T13:11:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0702000000000002e-05 2024-09-13T13:11:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214675+0000", 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8310000000000003e-06 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.214679+0000", 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6399999999999999e-06 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.217190+0000", 2024-09-13T13:11:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025110089999999998 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.217325+0000", 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134757 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.217469+0000", 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014494700000000001 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.220469+0000", 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029995830000000001 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.226535+0000", 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060662759999999998 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:78968 9.3 9:cd54190a:::benchmark_data_smithi052_128427_object78967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:11:00.220999+0000", 2024-09-13T13:11:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.044979388000000002, 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045110895999999998, 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78968 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.220999+0000", 2024-09-13T13:11:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.220999+0000", 2024-09-13T13:11:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.221001+0000", 2024-09-13T13:11:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6300000000000001e-06 2024-09-13T13:11:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.221046+0000", 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5290000000000002e-05 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.221058+0000", 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-05 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.221066+0000", 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9529999999999999e-06 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.221114+0000", 2024-09-13T13:11:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8600000000000002e-05 2024-09-13T13:11:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.221260+0000", 2024-09-13T13:11:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014517499999999999 2024-09-13T13:11:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:11:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.221432+0000", 2024-09-13T13:11:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017246500000000001 2024-09-13T13:11:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.226559+0000", 2024-09-13T13:11:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005126884 2024-09-13T13:11:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.226584+0000", 2024-09-13T13:11:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4522999999999999e-05 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:78970 9.3 9:cfc957e4:::benchmark_data_smithi052_128427_object78969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:11:00.223585+0000", 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.042393666000000003, 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042563075999999998, 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:11:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:11:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78970 2024-09-13T13:11:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.223585+0000", 2024-09-13T13:11:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.223585+0000", 2024-09-13T13:11:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.223586+0000", 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.761e-06 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.223646+0000", 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9936999999999999e-05 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.223656+0000", 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4970000000000007e-06 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.223662+0000", 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.494e-06 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.223715+0000", 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2197999999999997e-05 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.224229+0000", 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000514859 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.224410+0000", 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000180607 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.226577+0000", 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00216723 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:00.226587+0000", 2024-09-13T13:11:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6970000000000005e-06 2024-09-13T13:11:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:11:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 4 2024-09-13T13:11:00.342 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:11:00.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:00.344 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:00.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:00.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:00.479 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:00.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:00.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:00.580 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:00.618 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:00.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:00.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:00.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:00.750 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:00.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:00.790 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 45 16 79549 79533 110.419 65.5 0.0085586 0.00904139 2024-09-13T13:11:00.853 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:00.891 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:00.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:00.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:01.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:01.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:01.130 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:01.166 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:01.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:01.203 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T13:11:01.270 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:01.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:01.305 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:01.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:01.406 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:01.440 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:01.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:01.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:01.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:01 smithi052 ceph-mon[36682]: pgmap v478: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 897 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-09-13T13:11:01.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:01 smithi052 ceph-mon[29471]: pgmap v478: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 897 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-09-13T13:11:01.542 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:01.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:01 smithi167 ceph-mon[33178]: pgmap v478: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 897 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-09-13T13:11:01.578 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:01.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:01.666 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:11:01.679 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:01.713 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:01.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:01.790 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 46 16 80737 80721 109.633 74.25 0.0100115 0.00911571 2024-09-13T13:11:01.815 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:01.855 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:01.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:01.957 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:01.991 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:01.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:02.093 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:02.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:02.139 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:02.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:02.240 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:02.277 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:02.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:02.301 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:11:02.301 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:11:02.301 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:11:02.301 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:11:02.301 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:11:02.379 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:02.414 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:02.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:02.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:02.457 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:02 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:11:02.457 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:02 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:11:02.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:02 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:11:02.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:02 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:11:02.498 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:02.517 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:02.558 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:02.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:02.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:02.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:02 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:11:02.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:02 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:11:02.660 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:02.691 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:02.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:02.792 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 47 16 81901 81885 108.845 72.75 0.0104303 0.00918208 2024-09-13T13:11:02.793 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:02.829 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:02.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:02.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:02.931 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:02.966 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:02.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:02.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:03.047 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T13:11:03.069 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:03.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:03.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:03.233 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:03.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:03.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:03.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:03.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:03 smithi167 ceph-mon[33178]: pgmap v479: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-13T13:11:03.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:03 smithi167 ceph-mon[33178]: osdmap e264: 8 total, 8 up, 8 in 2024-09-13T13:11:03.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:03 smithi167 ceph-mon[33178]: osdmap e265: 8 total, 8 up, 8 in 2024-09-13T13:11:03.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:03 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2592553641' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:11:03.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:03 smithi167 ceph-mon[33178]: from='client.35741 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:11:03.325 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:03 smithi052 ceph-mon[36682]: pgmap v479: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-13T13:11:03.325 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:03 smithi052 ceph-mon[36682]: osdmap e264: 8 total, 8 up, 8 in 2024-09-13T13:11:03.325 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:03 smithi052 ceph-mon[36682]: osdmap e265: 8 total, 8 up, 8 in 2024-09-13T13:11:03.325 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:03 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2592553641' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:11:03.325 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:03 smithi052 ceph-mon[36682]: from='client.35741 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:11:03.326 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:03 smithi052 ceph-mon[29471]: pgmap v479: 148 pgs: 148 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-13T13:11:03.326 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:03 smithi052 ceph-mon[29471]: osdmap e264: 8 total, 8 up, 8 in 2024-09-13T13:11:03.326 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:03 smithi052 ceph-mon[29471]: osdmap e265: 8 total, 8 up, 8 in 2024-09-13T13:11:03.326 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:03 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2592553641' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:11:03.326 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:03 smithi052 ceph-mon[29471]: from='client.35741 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:11:03.342 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:03.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:03.398 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:11:03.398 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:11:03.412 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:03.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:03.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:03.425 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:11:03.426 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:11:03.457 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:11:03.457 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:11:03.469 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:11:03.482 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:11:03.483 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:11:03.514 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:03.516 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:11:03.516 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:11:03.563 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:03.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:03.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:03.565 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:11:03.565 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:11:03.596 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:11:03.596 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:11:03.633 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:11:03.634 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:11:03.665 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:03.712 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:03.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:03.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:03.792 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 48 16 82994 82978 108.001 68.3125 0.00887747 0.00925372 2024-09-13T13:11:03.813 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:03.877 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:03.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:03.898 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:11:03.979 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:04.017 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:04.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:04.106 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:11:04.106 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:11:04.118 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:04.158 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:04.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:04.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:11:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:11:04.237 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:04 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:11:04.237 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:04 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4098160689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:04.237 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:04 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:04.237 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:04 smithi167 ceph-mon[33178]: from='client.35741 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:11:04.237 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:04 smithi167 ceph-mon[33178]: osdmap e266: 8 total, 8 up, 8 in 2024-09-13T13:11:04.260 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:04.297 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:04.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:04.399 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:04.433 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:04.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:04 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:11:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:04 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4098160689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:04 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:04 smithi052 ceph-mon[36682]: from='client.35741 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:11:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:04 smithi052 ceph-mon[36682]: osdmap e266: 8 total, 8 up, 8 in 2024-09-13T13:11:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:04 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:11:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:04 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4098160689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:04 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:04 smithi052 ceph-mon[29471]: from='client.35741 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:11:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:04 smithi052 ceph-mon[29471]: osdmap e266: 8 total, 8 up, 8 in 2024-09-13T13:11:04.535 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:04.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:04.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:04.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:04.707 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:04.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:04.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:04.792 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 49 16 84225 84209 107.367 76.9375 0.0142016 0.009309 2024-09-13T13:11:04.808 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:04.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T13:11:04.847 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:04.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:04.949 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:04.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:04.987 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:04.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:05.088 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:05.125 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:05.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:05.227 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:11:05.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:05.276 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:05.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:05.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:05.378 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:05.415 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:05.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:05.444 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:05 smithi052 ceph-mon[29471]: pgmap v482: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-09-13T13:11:05.444 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:05 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:11:05.444 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:05 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:11:05.444 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:05.444 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:05 smithi052 ceph-mon[29471]: 3.17 scrub starts 2024-09-13T13:11:05.444 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:05 smithi052 ceph-mon[29471]: 3.17 scrub ok 2024-09-13T13:11:05.444 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:05 smithi052 ceph-mon[36682]: pgmap v482: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-09-13T13:11:05.444 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:05 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:11:05.445 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:05 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:11:05.445 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:05.445 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:05 smithi052 ceph-mon[36682]: 3.17 scrub starts 2024-09-13T13:11:05.445 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:05 smithi052 ceph-mon[36682]: 3.17 scrub ok 2024-09-13T13:11:05.517 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:05.533 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:05 smithi167 ceph-mon[33178]: pgmap v482: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-09-13T13:11:05.533 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:05 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:11:05.534 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:05 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:11:05.534 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:05.534 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:05 smithi167 ceph-mon[33178]: 3.17 scrub starts 2024-09-13T13:11:05.534 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:05 smithi167 ceph-mon[33178]: 3.17 scrub ok 2024-09-13T13:11:05.555 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:05.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:05.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:05.693 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:05.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:05.792 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 50 16 85454 85438 106.756 76.8125 0.00992901 0.00936233 2024-09-13T13:11:05.795 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:05.829 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:05.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:05.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:05.876 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:11:05.876 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:11:05.931 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:05.967 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:05.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:06.069 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:06.107 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:06.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:06.184 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T13:11:06.208 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:06.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:06.240 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:06.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:06.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:06.373 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:06.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:06.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:06.475 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:06 smithi052 ceph-mon[29471]: 6.e scrub starts 2024-09-13T13:11:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:06 smithi052 ceph-mon[29471]: 6.e scrub ok 2024-09-13T13:11:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:06 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2419297154' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:11:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:06 smithi052 ceph-mon[29471]: from='client.35753 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:11:06.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:11:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:11:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:11:06.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:06 smithi052 ceph-mon[36682]: 6.e scrub starts 2024-09-13T13:11:06.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:06 smithi052 ceph-mon[36682]: 6.e scrub ok 2024-09-13T13:11:06.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:06 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2419297154' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:11:06.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:06 smithi052 ceph-mon[36682]: from='client.35753 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:11:06.525 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:06.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:06.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:06 smithi167 ceph-mon[33178]: 6.e scrub starts 2024-09-13T13:11:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:06 smithi167 ceph-mon[33178]: 6.e scrub ok 2024-09-13T13:11:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:06 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2419297154' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:11:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:06 smithi167 ceph-mon[33178]: from='client.35753 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:11:06.626 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:06.663 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:06.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:06.704 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T13:11:06.747 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:11:06.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:06.793 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 51 16 86772 86756 106.278 82.375 0.0137817 0.00940426 2024-09-13T13:11:06.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:06.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:06.937 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:06.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:06.983 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:06.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:07.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:07.134 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:07.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:07.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:07.153 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:11:07.165 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T13:11:07.236 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:07.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:07.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:07.288 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:07.371 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:07.410 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:07.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:07.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:07.429 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:07 smithi052 ceph-mon[36682]: 3.1e scrub starts 2024-09-13T13:11:07.429 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:07 smithi052 ceph-mon[36682]: 3.1e scrub ok 2024-09-13T13:11:07.430 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:07 smithi052 ceph-mon[36682]: pgmap v484: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 932 op/s 2024-09-13T13:11:07.430 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:07 smithi052 ceph-mon[36682]: from='client.35753 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:11:07.430 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:07 smithi052 ceph-mon[36682]: osdmap e267: 8 total, 8 up, 8 in 2024-09-13T13:11:07.430 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:07 smithi052 ceph-mon[29471]: 3.1e scrub starts 2024-09-13T13:11:07.430 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:07 smithi052 ceph-mon[29471]: 3.1e scrub ok 2024-09-13T13:11:07.430 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:07 smithi052 ceph-mon[29471]: pgmap v484: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 932 op/s 2024-09-13T13:11:07.430 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:07 smithi052 ceph-mon[29471]: from='client.35753 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:11:07.430 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:07 smithi052 ceph-mon[29471]: osdmap e267: 8 total, 8 up, 8 in 2024-09-13T13:11:07.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:11:07.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:07.540 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:07.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:07.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:07.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:07 smithi167 ceph-mon[33178]: 3.1e scrub starts 2024-09-13T13:11:07.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:07 smithi167 ceph-mon[33178]: 3.1e scrub ok 2024-09-13T13:11:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:07 smithi167 ceph-mon[33178]: pgmap v484: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 932 op/s 2024-09-13T13:11:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:07 smithi167 ceph-mon[33178]: from='client.35753 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:11:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:07 smithi167 ceph-mon[33178]: osdmap e267: 8 total, 8 up, 8 in 2024-09-13T13:11:07.642 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:07.668 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:07.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:07.769 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:07.795 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 52 16 87800 87784 105.465 64.25 0.0105015 0.00947748 2024-09-13T13:11:07.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:07.824 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:07.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:07.925 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:07.991 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:07.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:08.014 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:08.025 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:11:08.025 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:11:08.076 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:11:08.077 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:11:08.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:08.122 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:08.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:08.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:08.126 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:11:08.126 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:11:08.147 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:11:08.147 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:11:08.179 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:11:08.180 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:11:08.211 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:11:08.211 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:11:08.224 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:08.247 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:11:08.248 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:11:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:08.266 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:08.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:08.281 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:11:08.281 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:11:08.339 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:08 smithi052 ceph-mon[36682]: 5.0 scrub starts 2024-09-13T13:11:08.339 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:08 smithi052 ceph-mon[36682]: 5.0 scrub ok 2024-09-13T13:11:08.339 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:08 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:11:08.339 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4093485808' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:08.340 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:08 smithi052 ceph-mon[29471]: 5.0 scrub starts 2024-09-13T13:11:08.340 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:08 smithi052 ceph-mon[29471]: 5.0 scrub ok 2024-09-13T13:11:08.340 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:08 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:11:08.340 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4093485808' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:08.368 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:08.408 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:08.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:08.510 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:08.543 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:08.553 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:08.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:08.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:08 smithi167 ceph-mon[33178]: 5.0 scrub starts 2024-09-13T13:11:08.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:08 smithi167 ceph-mon[33178]: 5.0 scrub ok 2024-09-13T13:11:08.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:08 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:11:08.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4093485808' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:08.654 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:08.715 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:08.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:08.796 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 53 16 88824 88808 104.68 64 0.00935139 0.0095469 2024-09-13T13:11:08.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:08.846 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:08.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:08.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:08.948 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:08.984 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:08.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:08.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:09.086 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:09.120 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:09.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:09.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:09.222 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:09.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:09.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:09.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:09.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:09.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:09.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:09 smithi052 ceph-mon[36682]: 3.18 scrub starts 2024-09-13T13:11:09.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:09 smithi052 ceph-mon[36682]: 3.18 scrub ok 2024-09-13T13:11:09.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:09 smithi052 ceph-mon[36682]: pgmap v486: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 905 B/s rd, 62 MiB/s wr, 990 op/s 2024-09-13T13:11:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:09 smithi052 ceph-mon[29471]: 3.18 scrub starts 2024-09-13T13:11:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:09 smithi052 ceph-mon[29471]: 3.18 scrub ok 2024-09-13T13:11:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:09 smithi052 ceph-mon[29471]: pgmap v486: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 905 B/s rd, 62 MiB/s wr, 990 op/s 2024-09-13T13:11:09.534 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:09 smithi167 ceph-mon[33178]: 3.18 scrub starts 2024-09-13T13:11:09.534 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:09 smithi167 ceph-mon[33178]: 3.18 scrub ok 2024-09-13T13:11:09.534 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:09 smithi167 ceph-mon[33178]: pgmap v486: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 905 B/s rd, 62 MiB/s wr, 990 op/s 2024-09-13T13:11:09.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:09.574 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:11:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:11:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:11:09.574 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:11:09.575 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:11:09.584 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:09.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:09.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:09.686 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:09.720 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:09.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:09.796 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 54 16 89859 89843 103.94 64.6875 0.0080327 0.0096152 2024-09-13T13:11:09.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:09.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:09.862 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:09.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:09.963 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:10.010 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:10.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:10.112 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:10.142 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:10.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:10.234 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T13:11:10.244 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:10.286 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:10.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:10.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:10 smithi167 ceph-mon[33178]: 5.13 scrub starts 2024-09-13T13:11:10.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:10 smithi167 ceph-mon[33178]: 5.13 scrub ok 2024-09-13T13:11:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:10 smithi167 ceph-mon[33178]: 6.12 scrub starts 2024-09-13T13:11:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:10 smithi167 ceph-mon[33178]: 6.12 scrub ok 2024-09-13T13:11:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:10 smithi167 ceph-mon[33178]: 9.6 scrub starts 2024-09-13T13:11:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:10 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3241117401' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:11:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:10 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:11:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:10 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:11:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:10 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:11:10.377 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T13:11:10.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:10.473 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:10.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:10.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[36682]: 5.13 scrub starts 2024-09-13T13:11:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[36682]: 5.13 scrub ok 2024-09-13T13:11:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[36682]: 6.12 scrub starts 2024-09-13T13:11:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[36682]: 6.12 scrub ok 2024-09-13T13:11:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[36682]: 9.6 scrub starts 2024-09-13T13:11:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3241117401' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:11:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:11:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:11:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:11:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[29471]: 5.13 scrub starts 2024-09-13T13:11:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[29471]: 5.13 scrub ok 2024-09-13T13:11:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[29471]: 6.12 scrub starts 2024-09-13T13:11:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[29471]: 6.12 scrub ok 2024-09-13T13:11:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[29471]: 9.6 scrub starts 2024-09-13T13:11:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3241117401' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:11:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:11:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:11:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:10 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:11:10.575 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:10.614 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:10.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:10.716 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:10.757 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:10.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:10.796 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 55 16 90904 90888 103.238 65.3125 0.00459299 0.00968053 2024-09-13T13:11:10.807 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:11:10.859 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:10.895 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:10.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:10.997 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:11.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:11.035 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:11.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:11.101 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T13:11:11.136 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:11.169 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:11.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:11.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:11.257 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:11:11.271 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:11.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:11.345 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:11.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:11.376 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:11:11.427 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[29471]: pgmap v487: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 103 MiB/s wr, 1.66k op/s 2024-09-13T13:11:11.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[29471]: 6.2 scrub starts 2024-09-13T13:11:11.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[29471]: 6.2 scrub ok 2024-09-13T13:11:11.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3241117401' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:11:11.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[29471]: osdmap e268: 8 total, 8 up, 8 in 2024-09-13T13:11:11.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/670474510' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:11.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[29471]: from='client.25759 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:11.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[29471]: 6.9 scrub starts 2024-09-13T13:11:11.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[29471]: 6.9 scrub ok 2024-09-13T13:11:11.428 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[36682]: pgmap v487: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 103 MiB/s wr, 1.66k op/s 2024-09-13T13:11:11.428 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[36682]: 6.2 scrub starts 2024-09-13T13:11:11.428 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[36682]: 6.2 scrub ok 2024-09-13T13:11:11.429 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3241117401' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:11:11.429 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[36682]: osdmap e268: 8 total, 8 up, 8 in 2024-09-13T13:11:11.429 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/670474510' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:11.429 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[36682]: from='client.25759 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:11.429 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[36682]: 6.9 scrub starts 2024-09-13T13:11:11.429 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:11 smithi052 ceph-mon[36682]: 6.9 scrub ok 2024-09-13T13:11:11.446 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:11.475 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:11.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:11 smithi167 ceph-mon[33178]: pgmap v487: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 103 MiB/s wr, 1.66k op/s 2024-09-13T13:11:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:11 smithi167 ceph-mon[33178]: 6.2 scrub starts 2024-09-13T13:11:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:11 smithi167 ceph-mon[33178]: 6.2 scrub ok 2024-09-13T13:11:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3241117401' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:11:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:11 smithi167 ceph-mon[33178]: osdmap e268: 8 total, 8 up, 8 in 2024-09-13T13:11:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/670474510' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:11 smithi167 ceph-mon[33178]: from='client.25759 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:11 smithi167 ceph-mon[33178]: 6.9 scrub starts 2024-09-13T13:11:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:11 smithi167 ceph-mon[33178]: 6.9 scrub ok 2024-09-13T13:11:11.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:11.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:11.618 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:11.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:11.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:11.749 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:11:11.749 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:11:11.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:11:11.749 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167255 7.c 7:372d90c4:::benchmark_data_smithi052_71865_object46154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.812976+0000", 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 382.93286313700003, 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.970962349, 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167255 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.750 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812976+0000", 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812976+0000", 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812978+0000", 2024-09-13T13:11:11.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.547e-06 2024-09-13T13:11:11.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812985+0000", 2024-09-13T13:11:11.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1389999999999998e-06 2024-09-13T13:11:11.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812994+0000", 2024-09-13T13:11:11.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0580000000000004e-06 2024-09-13T13:11:11.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.812999+0000", 2024-09-13T13:11:11.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1719999999999998e-06 2024-09-13T13:11:11.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.813060+0000", 2024-09-13T13:11:11.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0238000000000002e-05 2024-09-13T13:11:11.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.813067+0000", 2024-09-13T13:11:11.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1019999999999999e-06 2024-09-13T13:11:11.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.778497+0000", 2024-09-13T13:11:11.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96543056400000005 2024-09-13T13:11:11.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.778584+0000", 2024-09-13T13:11:11.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6466999999999997e-05 2024-09-13T13:11:11.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:11:11.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.778664+0000", 2024-09-13T13:11:11.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0111000000000006e-05 2024-09-13T13:11:11.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783699+0000", 2024-09-13T13:11:11.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050350940000000004 2024-09-13T13:11:11.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783753+0000", 2024-09-13T13:11:11.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4301000000000001e-05 2024-09-13T13:11:11.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783880+0000", 2024-09-13T13:11:11.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012635700000000001 2024-09-13T13:11:11.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783912+0000", 2024-09-13T13:11:11.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2736e-05 2024-09-13T13:11:11.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783939+0000", 2024-09-13T13:11:11.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6463000000000001e-05 2024-09-13T13:11:11.757 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.757 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.757 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167302 7.c 7:3bad7f06:::benchmark_data_smithi052_71865_object46201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.859143+0000", 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 382.88669639, 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92766838399999996, 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167302 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.758 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859143+0000", 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859143+0000", 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859145+0000", 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-09-13T13:11:11.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859153+0000", 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4540000000000007e-06 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859165+0000", 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1246e-05 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859170+0000", 2024-09-13T13:11:11.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859231+0000", 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1163999999999999e-05 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.859244+0000", 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2539e-05 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.780456+0000", 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92121202899999999 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.780530+0000", 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4692999999999998e-05 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.780605+0000", 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4281000000000003e-05 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783805+0000", 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031999709999999998 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783844+0000", 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9496000000000003e-05 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786656+0000", 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028116249999999999 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786765+0000", 2024-09-13T13:11:11.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109205 2024-09-13T13:11:11.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786812+0000", 2024-09-13T13:11:11.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6440999999999998e-05 2024-09-13T13:11:11.765 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.765 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.765 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.765 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167310 7.c 7:3e43ee27:::benchmark_data_smithi052_71865_object46209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:11:11.765 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.867904+0000", 2024-09-13T13:11:11.766 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 382.87793576399997, 2024-09-13T13:11:11.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91916685399999998, 2024-09-13T13:11:11.766 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.766 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.766 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.766 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.766 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.766 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.766 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167310 2024-09-13T13:11:11.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.766 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867904+0000", 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867904+0000", 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867906+0000", 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.827e-06 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867911+0000", 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.733e-06 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867925+0000", 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3701e-05 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867932+0000", 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8689999999999998e-06 2024-09-13T13:11:11.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.867995+0000", 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3208999999999997e-05 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.868005+0000", 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0379e-05 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.782379+0000", 2024-09-13T13:11:11.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91437379500000004 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.782458+0000", 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8485999999999994e-05 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.782522+0000", 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4598999999999996e-05 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786890+0000", 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0043677120000000002 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786952+0000", 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1827000000000003e-05 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.786989+0000", 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6674999999999997e-05 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787034+0000", 2024-09-13T13:11:11.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5003000000000003e-05 2024-09-13T13:11:11.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787071+0000", 2024-09-13T13:11:11.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7039000000000002e-05 2024-09-13T13:11:11.772 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.772 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.772 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.772 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167324 7.c 7:3d57ad43:::benchmark_data_smithi052_71865_object46223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.880402+0000", 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 382.86543796199999, 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91706507599999998, 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167324 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880402+0000", 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880402+0000", 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880404+0000", 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3329999999999999e-06 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880410+0000", 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3199999999999996e-06 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880424+0000", 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3396e-05 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880431+0000", 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8010000000000004e-06 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880504+0000", 2024-09-13T13:11:11.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2482000000000002e-05 2024-09-13T13:11:11.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.880515+0000", 2024-09-13T13:11:11.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0745e-05 2024-09-13T13:11:11.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.783970+0000", 2024-09-13T13:11:11.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.90345520499999998 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.784030+0000", 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0668000000000003e-05 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.784102+0000", 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1852000000000003e-05 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.788798+0000", 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004696034 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.790728+0000", 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001929828 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797366+0000", 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066377340000000002 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797415+0000", 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9502000000000001e-05 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797467+0000", 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1175999999999999e-05 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167331 7.c 7:314bd5a0:::benchmark_data_smithi052_71865_object46230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:11:11.779 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.885575+0000", 2024-09-13T13:11:11.780 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 382.86026455000001, 2024-09-13T13:11:11.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91216846600000001, 2024-09-13T13:11:11.780 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.780 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.780 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.780 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167331 2024-09-13T13:11:11.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.780 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885575+0000", 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885575+0000", 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885577+0000", 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.446e-06 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885583+0000", 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0759999999999999e-06 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885597+0000", 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3909e-05 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885610+0000", 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2288e-05 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885692+0000", 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2060999999999999e-05 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.885703+0000", 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1331e-05 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.784949+0000", 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.89924553699999998 2024-09-13T13:11:11.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.785047+0000", 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8639000000000002e-05 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.785146+0000", 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8556999999999998e-05 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.788803+0000", 2024-09-13T13:11:11.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003657637 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.790769+0000", 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019660250000000002 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797662+0000", 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068928160000000004 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797708+0000", 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6038999999999997e-05 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797743+0000", 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5104999999999997e-05 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:167336 7.c 7:3d629c66:::benchmark_data_smithi052_71865_object46235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:48.887959+0000", 2024-09-13T13:11:11.786 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 382.85788074800001, 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.90987360799999994, 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 167336 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887959+0000", 2024-09-13T13:11:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887959+0000", 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887960+0000", 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.728e-06 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887965+0000", 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7319999999999998e-06 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887978+0000", 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2702e-05 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.887985+0000", 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.097e-06 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.888051+0000", 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6277000000000003e-05 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:48.888062+0000", 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049e-05 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787109+0000", 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.89904651199999996 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787218+0000", 2024-09-13T13:11:11.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010893200000000001 2024-09-13T13:11:11.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:11:11.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.787319+0000", 2024-09-13T13:11:11.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101605 2024-09-13T13:11:11.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.792171+0000", 2024-09-13T13:11:11.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048516380000000001 2024-09-13T13:11:11.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.794176+0000", 2024-09-13T13:11:11.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020051280000000001 2024-09-13T13:11:11.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797764+0000", 2024-09-13T13:11:11.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003588277 2024-09-13T13:11:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:11.793 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:11.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:11.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797807+0000", 2024-09-13T13:11:11.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3000999999999997e-05 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:49.797832+0000", 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.493e-05 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189427 7.d 7:b2417336:::benchmark_data_smithi052_71865_object68326:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.325139+0000", 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.42070034800003, 2024-09-13T13:11:11.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0242147180000001, 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189427 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325139+0000", 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325139+0000", 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325142+0000", 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.526e-06 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325151+0000", 2024-09-13T13:11:11.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4080000000000004e-06 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325164+0000", 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2676e-05 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325173+0000", 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0070000000000003e-06 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.327118+0000", 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001944723 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.327125+0000", 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7729999999999999e-06 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.333636+0000", 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065108279999999998 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.333645+0000", 2024-09-13T13:11:11.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5639999999999997e-06 2024-09-13T13:11:11.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349274+0000", 2024-09-13T13:11:11.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0156289759999999 2024-09-13T13:11:11.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349354+0000", 2024-09-13T13:11:11.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0236999999999998e-05 2024-09-13T13:11:11.800 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189429 7.d 7:b1726e7a:::benchmark_data_smithi052_71865_object68328:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.325193+0000", 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.42064667800003, 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0246123009999999, 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.801 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189429 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325193+0000", 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325193+0000", 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325194+0000", 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325197+0000", 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2200000000000001e-06 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325204+0000", 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6989999999999997e-06 2024-09-13T13:11:11.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.325209+0000", 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.118e-06 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.327226+0000", 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020175979999999998 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.327232+0000", 2024-09-13T13:11:11.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9789999999999998e-06 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.333665+0000", 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064329620000000004 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.333671+0000", 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0599999999999996e-06 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349742+0000", 2024-09-13T13:11:11.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.016070799 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349805+0000", 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3083000000000006e-05 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189477 7.d 7:bf75cb66:::benchmark_data_smithi052_71865_object68376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.372477+0000", 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.37336288, 2024-09-13T13:11:11.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98440433000000005, 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189477 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372477+0000", 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372477+0000", 2024-09-13T13:11:11.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372479+0000", 2024-09-13T13:11:11.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8950000000000001e-06 2024-09-13T13:11:11.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372485+0000", 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9900000000000002e-06 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372494+0000", 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5689999999999994e-06 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372499+0000", 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3310000000000003e-06 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372566+0000", 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6716000000000003e-05 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.372586+0000", 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9987e-05 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349836+0000", 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97724946000000001 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349909+0000", 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3281999999999994e-05 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.349980+0000", 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0748999999999996e-05 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356657+0000", 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066769580000000002 2024-09-13T13:11:11.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356719+0000", 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2872000000000004e-05 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356776+0000", 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6314000000000003e-05 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356840+0000", 2024-09-13T13:11:11.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4484000000000001e-05 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356881+0000", 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0723000000000003e-05 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189510 7.d 7:b635578a:::benchmark_data_smithi052_71865_object68409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.399587+0000", 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.346252435, 2024-09-13T13:11:11.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96507304999999999, 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189510 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399587+0000", 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399587+0000", 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399589+0000", 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2759999999999999e-06 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399599+0000", 2024-09-13T13:11:11.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4730000000000006e-06 2024-09-13T13:11:11.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399613+0000", 2024-09-13T13:11:11.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.403e-05 2024-09-13T13:11:11.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399621+0000", 2024-09-13T13:11:11.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2609999999999993e-06 2024-09-13T13:11:11.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399690+0000", 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8448000000000006e-05 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.399700+0000", 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9490000000000008e-06 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.353851+0000", 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95415173900000005 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.353952+0000", 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101093 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.354065+0000", 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112314 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.362390+0000", 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0083252619999999999 2024-09-13T13:11:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364495+0000", 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021055179999999998 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364585+0000", 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9598000000000005e-05 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364629+0000", 2024-09-13T13:11:11.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3711000000000002e-05 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364660+0000", 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1377999999999997e-05 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189512 7.d 7:b7cfc9e5:::benchmark_data_smithi052_71865_object68411:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.400157+0000", 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.34568203700002, 2024-09-13T13:11:11.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96465484199999996, 2024-09-13T13:11:11.821 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.821 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.821 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.821 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.821 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.821 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.821 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189512 2024-09-13T13:11:11.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.821 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400157+0000", 2024-09-13T13:11:11.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400157+0000", 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400158+0000", 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.16e-07 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400162+0000", 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.304e-06 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400170+0000", 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400182+0000", 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2254e-05 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400234+0000", 2024-09-13T13:11:11.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.198e-05 2024-09-13T13:11:11.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.400244+0000", 2024-09-13T13:11:11.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0036e-05 2024-09-13T13:11:11.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.356925+0000", 2024-09-13T13:11:11.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95668097299999999 2024-09-13T13:11:11.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.357056+0000", 2024-09-13T13:11:11.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013119499999999999 2024-09-13T13:11:11.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:11:11.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.357181+0000", 2024-09-13T13:11:11.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012478000000000001 2024-09-13T13:11:11.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364436+0000", 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072547779999999999 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364542+0000", 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010571 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364710+0000", 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168486 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364754+0000", 2024-09-13T13:11:11.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4141000000000003e-05 2024-09-13T13:11:11.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364812+0000", 2024-09-13T13:11:11.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8180000000000002e-05 2024-09-13T13:11:11.827 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.827 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.827 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.827 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189519 7.d 7:b156376f:::benchmark_data_smithi052_71865_object68418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.828 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.408717+0000", 2024-09-13T13:11:11.828 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.33712284699999, 2024-09-13T13:11:11.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97666052999999997, 2024-09-13T13:11:11.828 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.828 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.828 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.828 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.828 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.828 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.828 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189519 2024-09-13T13:11:11.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.828 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408717+0000", 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408717+0000", 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408718+0000", 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3939999999999999e-06 2024-09-13T13:11:11.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408722+0000", 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2740000000000001e-06 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408732+0000", 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9129999999999998e-06 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408739+0000", 2024-09-13T13:11:11.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7789999999999998e-06 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408803+0000", 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3750000000000005e-05 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.408811+0000", 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4079999999999998e-06 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.359633+0000", 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95082174900000005 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.359882+0000", 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024927 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.359990+0000", 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010738099999999999 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364551+0000", 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045615020000000003 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371210+0000", 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066585400000000001 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.385259+0000", 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014048932 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.385324+0000", 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5908999999999995e-05 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.385377+0000", 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2729000000000003e-05 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.834 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189523 7.d 7:ba812c0f:::benchmark_data_smithi052_71865_object68422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.410273+0000", 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.33556676000001, 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98121212400000002, 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189523 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.836 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 56 16 92061 92045 102.686 72.3125 0.00714445 0.0097341 2024-09-13T13:11:11.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410273+0000", 2024-09-13T13:11:11.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410273+0000", 2024-09-13T13:11:11.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410274+0000", 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3179999999999999e-06 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410281+0000", 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8700000000000003e-06 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410293+0000", 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.225e-05 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410300+0000", 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6529999999999997e-06 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410356+0000", 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6366000000000002e-05 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.410366+0000", 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0166999999999999e-05 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.362418+0000", 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95205126100000004 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.362525+0000", 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106965 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.362594+0000", 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8967999999999994e-05 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.368931+0000", 2024-09-13T13:11:11.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063376480000000004 2024-09-13T13:11:11.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.387384+0000", 2024-09-13T13:11:11.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018452995999999999 2024-09-13T13:11:11.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391424+0000", 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040394380000000002 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391455+0000", 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1152999999999998e-05 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391485+0000", 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0071000000000001e-05 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189527 7.d 7:b61412a6:::benchmark_data_smithi052_71865_object68426:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.414261+0000", 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.331578471, 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97733418299999997, 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189527 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414261+0000", 2024-09-13T13:11:11.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414261+0000", 2024-09-13T13:11:11.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414262+0000", 2024-09-13T13:11:11.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6599999999999995e-07 2024-09-13T13:11:11.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414264+0000", 2024-09-13T13:11:11.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4830000000000002e-06 2024-09-13T13:11:11.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414273+0000", 2024-09-13T13:11:11.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5669999999999995e-06 2024-09-13T13:11:11.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414277+0000", 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9530000000000001e-06 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414327+0000", 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9759999999999998e-05 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.414336+0000", 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3330000000000003e-06 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364845+0000", 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95050922199999999 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364910+0000", 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5036999999999994e-05 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.364972+0000", 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1741000000000004e-05 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371169+0000", 2024-09-13T13:11:11.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061969069999999998 2024-09-13T13:11:11.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.387420+0000", 2024-09-13T13:11:11.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016250882000000001 2024-09-13T13:11:11.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391522+0000", 2024-09-13T13:11:11.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041024420000000004 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391561+0000", 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9026999999999998e-05 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391595+0000", 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4063000000000002e-05 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.850 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189540 7.d 7:bd5b765e:::benchmark_data_smithi052_71865_object68439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.850 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.427873+0000", 2024-09-13T13:11:11.850 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.31796698599999, 2024-09-13T13:11:11.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96380134799999995, 2024-09-13T13:11:11.850 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.850 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.850 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.850 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.850 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.850 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.850 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189540 2024-09-13T13:11:11.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427873+0000", 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427873+0000", 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427874+0000", 2024-09-13T13:11:11.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.765e-06 2024-09-13T13:11:11.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427880+0000", 2024-09-13T13:11:11.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2299999999999999e-06 2024-09-13T13:11:11.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427892+0000", 2024-09-13T13:11:11.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2578999999999999e-05 2024-09-13T13:11:11.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427901+0000", 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8480000000000007e-06 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427976+0000", 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4636999999999997e-05 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.427982+0000", 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.366695+0000", 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93871229899999997 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.366759+0000", 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4239000000000006e-05 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.366861+0000", 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102506 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.372748+0000", 2024-09-13T13:11:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058863960000000003 2024-09-13T13:11:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.387463+0000", 2024-09-13T13:11:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014715045 2024-09-13T13:11:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391626+0000", 2024-09-13T13:11:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041627900000000004 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391651+0000", 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5678e-05 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391674+0000", 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2586e-05 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189552 7.d 7:b0ca6a67:::benchmark_data_smithi052_71865_object68451:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.438326+0000", 2024-09-13T13:11:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.30751331300002, 2024-09-13T13:11:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96655187600000003, 2024-09-13T13:11:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189552 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438326+0000", 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438326+0000", 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438328+0000", 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.587e-06 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438337+0000", 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7280000000000001e-06 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438349+0000", 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2944e-05 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438356+0000", 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3369999999999996e-06 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438420+0000", 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4082000000000001e-05 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.438430+0000", 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0356e-05 2024-09-13T13:11:11.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.368958+0000", 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93052786899999995 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.369054+0000", 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5421999999999995e-05 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.369151+0000", 2024-09-13T13:11:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7264999999999998e-05 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.375369+0000", 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0062185900000000004 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391714+0000", 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016344250000000001 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404746+0000", 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013032871999999999 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404838+0000", 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1419e-05 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404878+0000", 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0154999999999998e-05 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.864 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189574 7.d 7:bbe21a3a:::benchmark_data_smithi052_71865_object68473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.864 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.459595+0000", 2024-09-13T13:11:11.864 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.28624435, 2024-09-13T13:11:11.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94540613600000001, 2024-09-13T13:11:11.864 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.864 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.864 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.864 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.864 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.864 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.864 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189574 2024-09-13T13:11:11.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459595+0000", 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459595+0000", 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459597+0000", 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9920000000000002e-06 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459603+0000", 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7629999999999997e-06 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459615+0000", 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1909999999999999e-05 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459621+0000", 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8139999999999997e-06 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459683+0000", 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2174000000000005e-05 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.459694+0000", 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1297e-05 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371239+0000", 2024-09-13T13:11:11.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.911544507 2024-09-13T13:11:11.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371305+0000", 2024-09-13T13:11:11.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6738999999999999e-05 2024-09-13T13:11:11.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:11:11.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.371370+0000", 2024-09-13T13:11:11.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4368000000000004e-05 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.380489+0000", 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091196799999999998 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.391735+0000", 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011245965 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404921+0000", 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013185357 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.404969+0000", 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8319999999999998e-05 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405001+0000", 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2249999999999998e-05 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.870 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189587 7.d 7:bb16c9f3:::benchmark_data_smithi052_71865_object68486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.470029+0000", 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.27581063899999, 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93511254399999999, 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189587 2024-09-13T13:11:11.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.872 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470029+0000", 2024-09-13T13:11:11.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470029+0000", 2024-09-13T13:11:11.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470030+0000", 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.42e-06 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470034+0000", 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1919999999999998e-06 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470045+0000", 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0239e-05 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470050+0000", 2024-09-13T13:11:11.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2569999999999998e-06 2024-09-13T13:11:11.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470113+0000", 2024-09-13T13:11:11.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3417999999999995e-05 2024-09-13T13:11:11.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.470123+0000", 2024-09-13T13:11:11.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3379999999999994e-06 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.372770+0000", 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.90264689300000001 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.372854+0000", 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4221000000000006e-05 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.372914+0000", 2024-09-13T13:11:11.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0451000000000001e-05 2024-09-13T13:11:11.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.380494+0000", 2024-09-13T13:11:11.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075800609999999999 2024-09-13T13:11:11.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.403571+0000", 2024-09-13T13:11:11.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02307698 2024-09-13T13:11:11.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405053+0000", 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014813879999999999 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405106+0000", 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2904e-05 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405141+0000", 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5781999999999997e-05 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.877 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.878 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.878 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189600 7.d 7:b65df3b7:::benchmark_data_smithi052_71865_object68499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.878 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.492940+0000", 2024-09-13T13:11:11.878 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.252899501, 2024-09-13T13:11:11.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91826516899999999, 2024-09-13T13:11:11.878 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.878 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.878 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.878 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.878 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.878 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189600 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492940+0000", 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492940+0000", 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492942+0000", 2024-09-13T13:11:11.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3539999999999998e-06 2024-09-13T13:11:11.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492952+0000", 2024-09-13T13:11:11.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.448e-06 2024-09-13T13:11:11.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492967+0000", 2024-09-13T13:11:11.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.554e-05 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.492975+0000", 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.493051+0000", 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6112999999999994e-05 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.493064+0000", 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2235000000000001e-05 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.374296+0000", 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.88123260599999997 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.374360+0000", 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4077000000000003e-05 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:11:11.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.374430+0000", 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9789999999999994e-05 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.380501+0000", 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060705790000000004 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405185+0000", 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024683738 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411130+0000", 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059459279999999996 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411163+0000", 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2898000000000002e-05 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411205+0000", 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1841000000000001e-05 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.884 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.885 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:189604 7.d 7:b85ca2e8:::benchmark_data_smithi052_71865_object68503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:11:11.885 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:05:14.496890+0000", 2024-09-13T13:11:11.885 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 357.24894931599999, 2024-09-13T13:11:11.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91447940100000003, 2024-09-13T13:11:11.885 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:11:11.885 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:11:11.885 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:11.885 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:11:11.885 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:11:11.885 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 189604 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496890+0000", 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496890+0000", 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496892+0000", 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5149999999999999e-06 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496896+0000", 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8569999999999997e-06 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496905+0000", 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4280000000000002e-06 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:11:11.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496910+0000", 2024-09-13T13:11:11.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3499999999999996e-06 2024-09-13T13:11:11.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496968+0000", 2024-09-13T13:11:11.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7939000000000002e-05 2024-09-13T13:11:11.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:11:11.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:14.496978+0000", 2024-09-13T13:11:11.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0209e-05 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.375402+0000", 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87842360799999997 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.375497+0000", 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4949000000000002e-05 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.375606+0000", 2024-09-13T13:11:11.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108652 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.380505+0000", 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048993880000000002 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.405223+0000", 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024717682000000001 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:11.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411282+0000", 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060587799999999997 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411332+0000", 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0365000000000003e-05 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:05:15.411370+0000", 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7679000000000003e-05 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.891 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.892 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:11:11.892 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:11:11.892 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:11:11.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:11.943 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:11.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:12.045 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:12.093 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:12.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:12.123 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:12.177 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:12.195 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:12.235 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:12.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:12.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:12.305 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[36682]: 9.6 scrub ok 2024-09-13T13:11:12.306 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:11:12.306 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:11:12.306 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[36682]: from='client.25759 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:11:12.306 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[36682]: osdmap e269: 8 total, 8 up, 8 in 2024-09-13T13:11:12.306 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:11:12.306 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:11:12.306 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[36682]: 5.5 scrub starts 2024-09-13T13:11:12.306 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[36682]: 5.5 scrub ok 2024-09-13T13:11:12.306 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[29471]: 9.6 scrub ok 2024-09-13T13:11:12.306 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:11:12.306 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:11:12.307 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[29471]: from='client.25759 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:11:12.307 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[29471]: osdmap e269: 8 total, 8 up, 8 in 2024-09-13T13:11:12.307 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:11:12.307 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:11:12.307 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[29471]: 5.5 scrub starts 2024-09-13T13:11:12.307 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:12 smithi052 ceph-mon[29471]: 5.5 scrub ok 2024-09-13T13:11:12.336 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:12.376 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:12.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:12.477 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:12.523 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:12.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:12 smithi167 ceph-mon[33178]: 9.6 scrub ok 2024-09-13T13:11:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:12 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:11:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:12 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:11:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:12 smithi167 ceph-mon[33178]: from='client.25759 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:11:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:12 smithi167 ceph-mon[33178]: osdmap e269: 8 total, 8 up, 8 in 2024-09-13T13:11:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:12 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:11:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:12 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:11:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:12 smithi167 ceph-mon[33178]: 5.5 scrub starts 2024-09-13T13:11:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:12 smithi167 ceph-mon[33178]: 5.5 scrub ok 2024-09-13T13:11:12.624 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:12.660 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:12.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:12.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:12.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:12.792 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:12.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:12.796 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 57 16 93243 93227 102.18 73.875 0.00655433 0.00978205 2024-09-13T13:11:12.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:12.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:12.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:12.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:12.948 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T13:11:13.014 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:11:13.014 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:11:13.030 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:13.055 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:11:13.055 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:11:13.083 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:13.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:13.099 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:11:13.099 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:11:13.132 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:11:13.132 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:11:13.163 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:11:13.163 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:11:13.185 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:13.196 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:11:13.196 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:11:13.243 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:13.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:13.245 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:11:13.245 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:11:13.292 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:11:13.293 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:11:13.345 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:13.404 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:13.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[36682]: 5.17 scrub starts 2024-09-13T13:11:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[36682]: 5.17 scrub ok 2024-09-13T13:11:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[36682]: 5.9 scrub starts 2024-09-13T13:11:13.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[36682]: 5.9 scrub ok 2024-09-13T13:11:13.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[36682]: pgmap v490: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 7.3 KiB/s rd, 61 MiB/s wr, 989 op/s 2024-09-13T13:11:13.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[36682]: 5.16 scrub starts 2024-09-13T13:11:13.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[36682]: 5.16 scrub ok 2024-09-13T13:11:13.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[36682]: osdmap e270: 8 total, 8 up, 8 in 2024-09-13T13:11:13.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[36682]: osdmap e271: 8 total, 8 up, 8 in 2024-09-13T13:11:13.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/586234451' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:13.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[29471]: 5.17 scrub starts 2024-09-13T13:11:13.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[29471]: 5.17 scrub ok 2024-09-13T13:11:13.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[29471]: 5.9 scrub starts 2024-09-13T13:11:13.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[29471]: 5.9 scrub ok 2024-09-13T13:11:13.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[29471]: pgmap v490: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 7.3 KiB/s rd, 61 MiB/s wr, 989 op/s 2024-09-13T13:11:13.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[29471]: 5.16 scrub starts 2024-09-13T13:11:13.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[29471]: 5.16 scrub ok 2024-09-13T13:11:13.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[29471]: osdmap e270: 8 total, 8 up, 8 in 2024-09-13T13:11:13.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[29471]: osdmap e271: 8 total, 8 up, 8 in 2024-09-13T13:11:13.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/586234451' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:13.505 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:13.549 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:13.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:13.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:13 smithi167 ceph-mon[33178]: 5.17 scrub starts 2024-09-13T13:11:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:13 smithi167 ceph-mon[33178]: 5.17 scrub ok 2024-09-13T13:11:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:13 smithi167 ceph-mon[33178]: 5.9 scrub starts 2024-09-13T13:11:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:13 smithi167 ceph-mon[33178]: 5.9 scrub ok 2024-09-13T13:11:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:13 smithi167 ceph-mon[33178]: pgmap v490: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 7.3 KiB/s rd, 61 MiB/s wr, 989 op/s 2024-09-13T13:11:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:13 smithi167 ceph-mon[33178]: 5.16 scrub starts 2024-09-13T13:11:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:13 smithi167 ceph-mon[33178]: 5.16 scrub ok 2024-09-13T13:11:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:13 smithi167 ceph-mon[33178]: osdmap e270: 8 total, 8 up, 8 in 2024-09-13T13:11:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:13 smithi167 ceph-mon[33178]: osdmap e271: 8 total, 8 up, 8 in 2024-09-13T13:11:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/586234451' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:13.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:13.682 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:13.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:13.784 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:13.796 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 58 16 94318 94302 101.577 67.1875 0.0108734 0.00983954 2024-09-13T13:11:13.824 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:13.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:13.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:13.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:13.965 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:13.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:13.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:14.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:14.104 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:11:14.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:14.105 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:14.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:14.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:14.267 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T13:11:14.273 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:14.294 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:14.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:14.395 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:14.438 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:14.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:14 smithi052 ceph-mon[36682]: 5.f scrub starts 2024-09-13T13:11:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:14 smithi052 ceph-mon[36682]: 5.f scrub ok 2024-09-13T13:11:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:14 smithi052 ceph-mon[36682]: 5.18 scrub starts 2024-09-13T13:11:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:14 smithi052 ceph-mon[36682]: 5.18 scrub ok 2024-09-13T13:11:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:14 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3765729911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:11:14.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:14 smithi052 ceph-mon[36682]: from='client.25771 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:11:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:14 smithi052 ceph-mon[29471]: 5.f scrub starts 2024-09-13T13:11:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:14 smithi052 ceph-mon[29471]: 5.f scrub ok 2024-09-13T13:11:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:14 smithi052 ceph-mon[29471]: 5.18 scrub starts 2024-09-13T13:11:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:14 smithi052 ceph-mon[29471]: 5.18 scrub ok 2024-09-13T13:11:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:14 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3765729911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:11:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:14 smithi052 ceph-mon[29471]: from='client.25771 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:11:14.540 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:14.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:11:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:11:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:14 smithi167 ceph-mon[33178]: 5.f scrub starts 2024-09-13T13:11:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:14 smithi167 ceph-mon[33178]: 5.f scrub ok 2024-09-13T13:11:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:14 smithi167 ceph-mon[33178]: 5.18 scrub starts 2024-09-13T13:11:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:14 smithi167 ceph-mon[33178]: 5.18 scrub ok 2024-09-13T13:11:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:14 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3765729911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:11:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:14 smithi167 ceph-mon[33178]: from='client.25771 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:11:14.594 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:14.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:14.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:14.695 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:14.734 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:14.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:14.797 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 59 16 95441 95425 101.045 70.1875 0.00847838 0.00989116 2024-09-13T13:11:14.836 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:14.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:14.874 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:14.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:14.975 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:14.993 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T13:11:14.993 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:11:15.003 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:15.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:15.109 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:15.144 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:15.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:15.246 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:15.267 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:15.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:15.369 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:15.398 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:15.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:15.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[36682]: 3.13 scrub starts 2024-09-13T13:11:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[36682]: 3.13 scrub ok 2024-09-13T13:11:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[36682]: pgmap v493: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 88 KiB/s rd, 69 MiB/s wr, 1.24k op/s 2024-09-13T13:11:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[36682]: 9.4 scrub starts 2024-09-13T13:11:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[36682]: from='client.25771 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:11:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[36682]: osdmap e272: 8 total, 8 up, 8 in 2024-09-13T13:11:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[29471]: 3.13 scrub starts 2024-09-13T13:11:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[29471]: 3.13 scrub ok 2024-09-13T13:11:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[29471]: pgmap v493: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 88 KiB/s rd, 69 MiB/s wr, 1.24k op/s 2024-09-13T13:11:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[29471]: 9.4 scrub starts 2024-09-13T13:11:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[29471]: from='client.25771 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:11:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:15 smithi052 ceph-mon[29471]: osdmap e272: 8 total, 8 up, 8 in 2024-09-13T13:11:15.500 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:15.532 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:15.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:15 smithi167 ceph-mon[33178]: 3.13 scrub starts 2024-09-13T13:11:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:15 smithi167 ceph-mon[33178]: 3.13 scrub ok 2024-09-13T13:11:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:15 smithi167 ceph-mon[33178]: pgmap v493: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 88 KiB/s rd, 69 MiB/s wr, 1.24k op/s 2024-09-13T13:11:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:15 smithi167 ceph-mon[33178]: 9.4 scrub starts 2024-09-13T13:11:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:15 smithi167 ceph-mon[33178]: from='client.25771 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:11:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:15 smithi167 ceph-mon[33178]: osdmap e272: 8 total, 8 up, 8 in 2024-09-13T13:11:15.635 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:15.661 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:15.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:15.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:15.763 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:15.791 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:11:15.791 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:11:15.796 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:11:15.796880+0000 min lat: 0.00134564 max lat: 0.125715 avg lat: 0.00995452 2024-09-13T13:11:15.797 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:11:15.797 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 60 16 96444 96428 100.406 62.6875 0.0173075 0.00995452 2024-09-13T13:11:15.798 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:15.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:15.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:15.900 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:15.937 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:15.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:16.039 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:16.076 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:16.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:16.170 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:11:16.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:16.212 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:16.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:16.314 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:16.354 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:16.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:16.456 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:16.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:11:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:11:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:11:16.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:16 smithi052 ceph-mon[29471]: 4.1d scrub starts 2024-09-13T13:11:16.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:16 smithi052 ceph-mon[29471]: 4.1d scrub ok 2024-09-13T13:11:16.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:16 smithi052 ceph-mon[29471]: 9.3 scrub starts 2024-09-13T13:11:16.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:16 smithi052 ceph-mon[36682]: 4.1d scrub starts 2024-09-13T13:11:16.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:16 smithi052 ceph-mon[36682]: 4.1d scrub ok 2024-09-13T13:11:16.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:16 smithi052 ceph-mon[36682]: 9.3 scrub starts 2024-09-13T13:11:16.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:16.498 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:16.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:16.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:16 smithi167 ceph-mon[33178]: 4.1d scrub starts 2024-09-13T13:11:16.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:16 smithi167 ceph-mon[33178]: 4.1d scrub ok 2024-09-13T13:11:16.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:16 smithi167 ceph-mon[33178]: 9.3 scrub starts 2024-09-13T13:11:16.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:16.612 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T13:11:16.626 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:16.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:16.683 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:16.725 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:16.728 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:16.739 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:16.797 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 61 16 97536 97520 99.8788 68.25 0.0139392 0.0100068 2024-09-13T13:11:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:16.827 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:16.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:16.928 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:16.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:16.980 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:16.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:17.081 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:17.108 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:17.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:17.210 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:17.247 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:17.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:17.349 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:17.379 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:17.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:17.440 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:17 smithi052 ceph-mon[36682]: pgmap v495: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 83 KiB/s rd, 75 MiB/s wr, 1.33k op/s 2024-09-13T13:11:17.440 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:17 smithi052 ceph-mon[36682]: 9.4 scrub ok 2024-09-13T13:11:17.441 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:17 smithi052 ceph-mon[29471]: pgmap v495: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 83 KiB/s rd, 75 MiB/s wr, 1.33k op/s 2024-09-13T13:11:17.441 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:17 smithi052 ceph-mon[29471]: 9.4 scrub ok 2024-09-13T13:11:17.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:17.515 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:17.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:17.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:17 smithi167 ceph-mon[33178]: pgmap v495: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 83 KiB/s rd, 75 MiB/s wr, 1.33k op/s 2024-09-13T13:11:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:17 smithi167 ceph-mon[33178]: 9.4 scrub ok 2024-09-13T13:11:17.617 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:17.642 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:17.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:17.744 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:17.785 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:17.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:17.796 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 62 16 98645 98629 99.3858 69.3125 0.00996277 0.0100565 2024-09-13T13:11:17.887 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:17.918 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:17.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:17.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:18.019 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:18.048 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:18.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:18.151 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:18.183 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:18.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:18.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:18.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:18.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:18.416 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:18.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:18.451 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:18.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:18.552 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:18 smithi167 ceph-mon[33178]: 9.3 scrub ok 2024-09-13T13:11:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:18 smithi167 ceph-mon[33178]: 4.1 scrub starts 2024-09-13T13:11:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:18 smithi167 ceph-mon[33178]: 4.1 scrub ok 2024-09-13T13:11:18.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:18 smithi167 ceph-mon[33178]: osdmap e273: 8 total, 8 up, 8 in 2024-09-13T13:11:18.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:18 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2876638089' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:11:18.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:18 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:11:18.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:18 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:11:18.587 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:18.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:18.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:18.690 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:18.728 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:18.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:18.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[36682]: 9.3 scrub ok 2024-09-13T13:11:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[36682]: 4.1 scrub starts 2024-09-13T13:11:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[36682]: 4.1 scrub ok 2024-09-13T13:11:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[36682]: osdmap e273: 8 total, 8 up, 8 in 2024-09-13T13:11:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2876638089' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:11:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:11:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:11:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[29471]: 9.3 scrub ok 2024-09-13T13:11:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[29471]: 4.1 scrub starts 2024-09-13T13:11:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[29471]: 4.1 scrub ok 2024-09-13T13:11:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[29471]: osdmap e273: 8 total, 8 up, 8 in 2024-09-13T13:11:18.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2876638089' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:11:18.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:11:18.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:18 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:11:18.797 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 63 16 99737 99721 98.8917 68.25 0.00764168 0.0101071 2024-09-13T13:11:18.830 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:18.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:18.871 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:18.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:18.907 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:11:18.972 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:19.006 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:19.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:19.108 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:19.146 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:19.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:19.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:19.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:19.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:19.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:19.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:19.386 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:19.420 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:19.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:19.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:19.522 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:19.562 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:19.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:19.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:19.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:19 smithi167 ceph-mon[33178]: 6.b scrub starts 2024-09-13T13:11:19.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:19 smithi167 ceph-mon[33178]: 6.b scrub ok 2024-09-13T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:19 smithi167 ceph-mon[33178]: pgmap v496: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 66 KiB/s rd, 57 MiB/s wr, 1.02k op/s 2024-09-13T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:19 smithi167 ceph-mon[33178]: 4.13 scrub starts 2024-09-13T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:19 smithi167 ceph-mon[33178]: 4.13 scrub ok 2024-09-13T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:19 smithi167 ceph-mon[33178]: from='client.15921 v1:172.21.15.52:0/582306813' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:19 smithi167 ceph-mon[33178]: 4.3 scrub starts 2024-09-13T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:19 smithi167 ceph-mon[33178]: 4.3 scrub ok 2024-09-13T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:19 smithi167 ceph-mon[33178]: 5.6 scrub starts 2024-09-13T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:19 smithi167 ceph-mon[33178]: 5.6 scrub ok 2024-09-13T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:19 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:19 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2876638089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:11:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:19 smithi167 ceph-mon[33178]: osdmap e274: 8 total, 8 up, 8 in 2024-09-13T13:11:19.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[36682]: 6.b scrub starts 2024-09-13T13:11:19.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[36682]: 6.b scrub ok 2024-09-13T13:11:19.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[36682]: pgmap v496: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 66 KiB/s rd, 57 MiB/s wr, 1.02k op/s 2024-09-13T13:11:19.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[36682]: 4.13 scrub starts 2024-09-13T13:11:19.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[36682]: 4.13 scrub ok 2024-09-13T13:11:19.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[36682]: from='client.15921 v1:172.21.15.52:0/582306813' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:11:19.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[36682]: 4.3 scrub starts 2024-09-13T13:11:19.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[36682]: 4.3 scrub ok 2024-09-13T13:11:19.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[36682]: 5.6 scrub starts 2024-09-13T13:11:19.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[36682]: 5.6 scrub ok 2024-09-13T13:11:19.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:19.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2876638089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:11:19.603 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[36682]: osdmap e274: 8 total, 8 up, 8 in 2024-09-13T13:11:19.603 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[29471]: 6.b scrub starts 2024-09-13T13:11:19.603 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[29471]: 6.b scrub ok 2024-09-13T13:11:19.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[29471]: pgmap v496: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 66 KiB/s rd, 57 MiB/s wr, 1.02k op/s 2024-09-13T13:11:19.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[29471]: 4.13 scrub starts 2024-09-13T13:11:19.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[29471]: 4.13 scrub ok 2024-09-13T13:11:19.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[29471]: from='client.15921 v1:172.21.15.52:0/582306813' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:11:19.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[29471]: 4.3 scrub starts 2024-09-13T13:11:19.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[29471]: 4.3 scrub ok 2024-09-13T13:11:19.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[29471]: 5.6 scrub starts 2024-09-13T13:11:19.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[29471]: 5.6 scrub ok 2024-09-13T13:11:19.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:19.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2876638089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:11:19.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:19 smithi052 ceph-mon[29471]: osdmap e274: 8 total, 8 up, 8 in 2024-09-13T13:11:19.665 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:19.698 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:19.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:19.797 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 64 16 101038 101022 98.617 81.3125 0.0103407 0.0101358 2024-09-13T13:11:19.800 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:19.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:19.852 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:19.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:19.953 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:19.987 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:19.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:19.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:20.089 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:20.131 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:20.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:20.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:20.194 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:11:20.198 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:11:20.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:20.266 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:20.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:20.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:20.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:20.409 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:20.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:20.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:20 smithi052 ceph-mon[36682]: 3.e scrub starts 2024-09-13T13:11:20.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:20 smithi052 ceph-mon[36682]: 3.e scrub ok 2024-09-13T13:11:20.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:20 smithi052 ceph-mon[29471]: 3.e scrub starts 2024-09-13T13:11:20.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:20 smithi052 ceph-mon[29471]: 3.e scrub ok 2024-09-13T13:11:20.511 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:20.545 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:20.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:20.647 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:20.687 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:20.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:20.789 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:20.797 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 65 16 102310 102294 98.3229 79.5 0.00640048 0.0101653 2024-09-13T13:11:20.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:20 smithi167 ceph-mon[33178]: 3.e scrub starts 2024-09-13T13:11:20.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:20 smithi167 ceph-mon[33178]: 3.e scrub ok 2024-09-13T13:11:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:20.822 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:20.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:20.923 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:20.947 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:11:20.948 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:11:20.962 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:20.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:20.987 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:11:20.987 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:11:21.010 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:11:21.010 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:11:21.065 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:21.087 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:21.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:21.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:21.122 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:11:21.122 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:11:21.158 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:11:21.158 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:11:21.190 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:21.193 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:11:21.193 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:11:21.228 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:11:21.229 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:11:21.233 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:21.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:21.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:21.261 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:11:21.261 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:11:21.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:21.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:21.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:21.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:21.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:21.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:21 smithi052 ceph-mon[36682]: pgmap v499: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 14 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-09-13T13:11:21.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:21 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/713170212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:21.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:21 smithi052 ceph-mon[29471]: pgmap v499: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 14 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-09-13T13:11:21.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:21 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/713170212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:21.503 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:21.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:21.605 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:21.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:21.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:21.731 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:21.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:21.766 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:21.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:21.797 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 66 16 103368 103352 97.8351 66.125 0.0134612 0.0102166 2024-09-13T13:11:21.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:21 smithi167 ceph-mon[33178]: pgmap v499: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 14 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-09-13T13:11:21.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:21 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/713170212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:21.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:21.904 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:21.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:22.006 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:22.041 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:22.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:22.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:22.143 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:22.175 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:22.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:22.277 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:22.318 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:22.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:22.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:22.420 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:22.442 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:22.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:22.544 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:22.586 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:22.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:22.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:22.688 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:22.723 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:22.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:22.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:22.797 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 67 16 104481 104465 97.4131 69.5625 0.00659132 0.0102606 2024-09-13T13:11:22.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:22.861 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:22.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:22.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:22.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:23.003 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:23.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:23.104 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:23.141 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:23.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:23.245 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:23.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:23.277 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:23.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:23.378 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:23.418 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:23.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:23.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:23 smithi052 ceph-mon[36682]: pgmap v500: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 2.6 KiB/s rd, 45 MiB/s wr, 721 op/s 2024-09-13T13:11:23.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:23 smithi052 ceph-mon[36682]: osdmap e275: 8 total, 8 up, 8 in 2024-09-13T13:11:23.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:23 smithi052 ceph-mon[29471]: pgmap v500: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 2.6 KiB/s rd, 45 MiB/s wr, 721 op/s 2024-09-13T13:11:23.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:23 smithi052 ceph-mon[29471]: osdmap e275: 8 total, 8 up, 8 in 2024-09-13T13:11:23.520 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:23.555 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:23.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:23.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:23.657 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:23.698 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2024-09-13T13:11:17.896670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6158031387,"num_objects":94154,"num_object_clones":0,"num_object_copies":282462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94154,"num_whiteouts":0,"num_read":1879,"num_read_kb":1736,"num_write":94790,"num_write_kb":6014793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83401,"ondisk_log_size":83401,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20618204,"kb_used_data":17975916,"kb_used_omap":15,"kb_used_meta":2642096,"kb_avail":729343012,"statfs":{"total":767960285184,"available":746847244288,"internally_reserved":0,"allocated":18407337984,"data_stored":18402799247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":2705507097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,15,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":358547456,"num_objects":5471,"num_object_clones":0,"num_object_copies":16413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5471,"num_whiteouts":0,"num_read":395,"num_read_kb":395,"num_write":5731,"num_write_kb":350144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.635852"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":460,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.907171+0000","last_change":"2024-09-13T13:10:32.368663+0000","last_active":"2024-09-13T13:11:12.907171+0000","last_peered":"2024-09-13T13:11:12.907171+0000","last_clean":"2024-09-13T13:11:12.907171+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:11:12.907171+0000","last_undegraded":"2024-09-13T13:11:12.907171+0000","last_fullsized":"2024-09-13T13:11:12.907171+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.368588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.368588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:00.402096+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":494,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.888542+0000","last_change":"2024-09-13T13:10:32.777554+0000","last_active":"2024-09-13T13:11:12.888542+0000","last_peered":"2024-09-13T13:11:12.888542+0000","last_clean":"2024-09-13T13:11:12.888542+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:11:12.888542+0000","last_undegraded":"2024-09-13T13:11:12.888542+0000","last_fullsized":"2024-09-13T13:11:12.888542+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:32.777451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.777451+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:00.660196+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.911878+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:11:12.911878+0000","last_peered":"2024-09-13T13:11:12.911878+0000","last_clean":"2024-09-13T13:11:12.911878+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:11:12.911878+0000","last_undegraded":"2024-09-13T13:11:12.911878+0000","last_fullsized":"2024-09-13T13:11:12.911878+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:14.841596+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":442,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.889299+0000","last_change":"2024-09-13T13:09:42.788562+0000","last_active":"2024-09-13T13:11:12.889299+0000","last_peered":"2024-09-13T13:11:12.889299+0000","last_clean":"2024-09-13T13:11:12.889299+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:11:12.889299+0000","last_undegraded":"2024-09-13T13:11:12.889299+0000","last_fullsized":"2024-09-13T13:11:12.889299+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:42.787492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:42.787492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:02.026348+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":460,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.904349+0000","last_change":"2024-09-13T13:08:55.504675+0000","last_active":"2024-09-13T13:11:12.904349+0000","last_peered":"2024-09-13T13:11:12.904349+0000","last_clean":"2024-09-13T13:11:12.904349+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:11:12.904349+0000","last_undegraded":"2024-09-13T13:11:12.904349+0000","last_fullsized":"2024-09-13T13:11:12.904349+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.504475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.504475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:45.483728+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":473,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902647+0000","last_change":"2024-09-13T13:11:05.501305+0000","last_active":"2024-09-13T13:11:12.902647+0000","last_peered":"2024-09-13T13:11:12.902647+0000","last_clean":"2024-09-13T13:11:12.902647+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:11:12.902647+0000","last_undegraded":"2024-09-13T13:11:12.902647+0000","last_fullsized":"2024-09-13T13:11:12.902647+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:05.501234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:05.501234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.182488+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":516,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.904166+0000","last_change":"2024-09-13T13:10:39.460577+0000","last_active":"2024-09-13T13:11:12.904166+0000","last_peered":"2024-09-13T13:11:12.904166+0000","last_clean":"2024-09-13T13:11:12.904166+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:11:12.904166+0000","last_undegraded":"2024-09-13T13:11:12.904166+0000","last_fullsized":"2024-09-13T13:11:12.904166+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:10:39.460500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.460500+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:00.679480+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":465,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.888489+0000","last_change":"2024-09-13T13:08:55.513716+0000","last_active":"2024-09-13T13:11:12.888489+0000","last_peered":"2024-09-13T13:11:12.888489+0000","last_clean":"2024-09-13T13:11:12.888489+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:11:12.888489+0000","last_undegraded":"2024-09-13T13:11:12.888489+0000","last_fullsized":"2024-09-13T13:11:12.888489+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.513667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.513667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":494,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.904499+0000","last_change":"2024-09-13T13:10:35.137237+0000","last_active":"2024-09-13T13:11:12.904499+0000","last_peered":"2024-09-13T13:11:12.904499+0000","last_clean":"2024-09-13T13:11:12.904499+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:11:12.904499+0000","last_undegraded":"2024-09-13T13:11:12.904499+0000","last_fullsized":"2024-09-13T13:11:12.904499+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:10:35.137160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:35.137160+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.584514+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":499,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.889573+0000","last_change":"2024-09-13T13:10:31.738782+0000","last_active":"2024-09-13T13:11:12.889573+0000","last_peered":"2024-09-13T13:11:12.889573+0000","last_clean":"2024-09-13T13:11:12.889573+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:11:12.889573+0000","last_undegraded":"2024-09-13T13:11:12.889573+0000","last_fullsized":"2024-09-13T13:11:12.889573+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:31.738685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.738685+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:43.720157+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":453,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902259+0000","last_change":"2024-09-13T13:06:50.665256+0000","last_active":"2024-09-13T13:11:12.902259+0000","last_peered":"2024-09-13T13:11:12.902259+0000","last_clean":"2024-09-13T13:11:12.902259+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:11:12.902259+0000","last_undegraded":"2024-09-13T13:11:12.902259+0000","last_fullsized":"2024-09-13T13:11:12.902259+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:50.665191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.665191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:21.020152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":456,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905681+0000","last_change":"2024-09-13T13:10:43.363157+0000","last_active":"2024-09-13T13:11:12.905681+0000","last_peered":"2024-09-13T13:11:12.905681+0000","last_clean":"2024-09-13T13:11:12.905681+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:11:12.905681+0000","last_undegraded":"2024-09-13T13:11:12.905681+0000","last_fullsized":"2024-09-13T13:11:12.905681+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:43.362967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:43.362967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:11.841904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":477,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.901349+0000","last_change":"2024-09-13T13:10:31.103576+0000","last_active":"2024-09-13T13:11:12.901349+0000","last_peered":"2024-09-13T13:11:12.901349+0000","last_clean":"2024-09-13T13:11:12.901349+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:11:12.901349+0000","last_undegraded":"2024-09-13T13:11:12.901349+0000","last_fullsized":"2024-09-13T13:11:12.901349+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:31.103509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.103509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:52.926445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":463,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.888623+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:11:12.888623+0000","last_peered":"2024-09-13T13:11:12.888623+0000","last_clean":"2024-09-13T13:11:12.888623+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:11:12.888623+0000","last_undegraded":"2024-09-13T13:11:12.888623+0000","last_fullsized":"2024-09-13T13:11:12.888623+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.107577+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":461,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905495+0000","last_change":"2024-09-13T13:10:32.803006+0000","last_active":"2024-09-13T13:11:12.905495+0000","last_peered":"2024-09-13T13:11:12.905495+0000","last_clean":"2024-09-13T13:11:12.905495+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:11:12.905495+0000","last_undegraded":"2024-09-13T13:11:12.905495+0000","last_fullsized":"2024-09-13T13:11:12.905495+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.802931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.802931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:58.651202+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":433,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902374+0000","last_change":"2024-09-13T13:07:19.641119+0000","last_active":"2024-09-13T13:11:12.902374+0000","last_peered":"2024-09-13T13:11:12.902374+0000","last_clean":"2024-09-13T13:11:12.902374+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:11:12.902374+0000","last_undegraded":"2024-09-13T13:11:12.902374+0000","last_fullsized":"2024-09-13T13:11:12.902374+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:19.640697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:19.640697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:57.084952+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":471,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890725+0000","last_change":"2024-09-13T13:10:28.805013+0000","last_active":"2024-09-13T13:11:12.890725+0000","last_peered":"2024-09-13T13:11:12.890725+0000","last_clean":"2024-09-13T13:11:12.890725+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:11:12.890725+0000","last_undegraded":"2024-09-13T13:11:12.890725+0000","last_fullsized":"2024-09-13T13:11:12.890725+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:28.804735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:28.804735+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.220025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":460,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906725+0000","last_change":"2024-09-13T13:08:19.181017+0000","last_active":"2024-09-13T13:11:12.906725+0000","last_peered":"2024-09-13T13:11:12.906725+0000","last_clean":"2024-09-13T13:11:12.906725+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:11:12.906725+0000","last_undegraded":"2024-09-13T13:11:12.906725+0000","last_fullsized":"2024-09-13T13:11:12.906725+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:19.180948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.180948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:57.228082+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":509,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890618+0000","last_change":"2024-09-13T13:06:44.532305+0000","last_active":"2024-09-13T13:11:12.890618+0000","last_peered":"2024-09-13T13:11:12.890618+0000","last_clean":"2024-09-13T13:11:12.890618+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:11:12.890618+0000","last_undegraded":"2024-09-13T13:11:12.890618+0000","last_fullsized":"2024-09-13T13:11:12.890618+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:06:44.532186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.532186+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:17.921942+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":462,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.912430+0000","last_change":"2024-09-13T13:09:59.936868+0000","last_active":"2024-09-13T13:11:12.912430+0000","last_peered":"2024-09-13T13:11:12.912430+0000","last_clean":"2024-09-13T13:11:12.912430+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:11:12.912430+0000","last_undegraded":"2024-09-13T13:11:12.912430+0000","last_fullsized":"2024-09-13T13:11:12.912430+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:59.936811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:59.936811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":460,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906951+0000","last_change":"2024-09-13T13:09:29.517189+0000","last_active":"2024-09-13T13:11:12.906951+0000","last_peered":"2024-09-13T13:11:12.906951+0000","last_clean":"2024-09-13T13:11:12.906951+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:11:12.906951+0000","last_undegraded":"2024-09-13T13:11:12.906951+0000","last_fullsized":"2024-09-13T13:11:12.906951+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:29.517034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:29.517034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:54.544083+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":470,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.889704+0000","last_change":"2024-09-13T13:10:34.788637+0000","last_active":"2024-09-13T13:11:12.889704+0000","last_peered":"2024-09-13T13:11:12.889704+0000","last_clean":"2024-09-13T13:11:12.889704+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:11:12.889704+0000","last_undegraded":"2024-09-13T13:11:12.889704+0000","last_fullsized":"2024-09-13T13:11:12.889704+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:34.788562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:34.788562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:03.723901+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":481,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.903348+0000","last_change":"2024-09-13T13:09:42.310274+0000","last_active":"2024-09-13T13:11:12.903348+0000","last_peered":"2024-09-13T13:11:12.903348+0000","last_clean":"2024-09-13T13:11:12.903348+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:11:12.903348+0000","last_undegraded":"2024-09-13T13:11:12.903348+0000","last_fullsized":"2024-09-13T13:11:12.903348+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:09:42.310185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:42.310185+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":454,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890065+0000","last_change":"2024-09-13T13:07:12.879813+0000","last_active":"2024-09-13T13:11:12.890065+0000","last_peered":"2024-09-13T13:11:12.890065+0000","last_clean":"2024-09-13T13:11:12.890065+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:11:12.890065+0000","last_undegraded":"2024-09-13T13:11:12.890065+0000","last_fullsized":"2024-09-13T13:11:12.890065+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.879727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.879727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:52.066816+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":466,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905150+0000","last_change":"2024-09-13T13:10:09.765203+0000","last_active":"2024-09-13T13:11:12.905150+0000","last_peered":"2024-09-13T13:11:12.905150+0000","last_clean":"2024-09-13T13:11:12.905150+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:11:12.905150+0000","last_undegraded":"2024-09-13T13:11:12.905150+0000","last_fullsized":"2024-09-13T13:11:12.905150+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:09.765153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.765153+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:26.707398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":455,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.912025+0000","last_change":"2024-09-13T13:07:54.442276+0000","last_active":"2024-09-13T13:11:12.912025+0000","last_peered":"2024-09-13T13:11:12.912025+0000","last_clean":"2024-09-13T13:11:12.912025+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:11:12.912025+0000","last_undegraded":"2024-09-13T13:11:12.912025+0000","last_fullsized":"2024-09-13T13:11:12.912025+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:07:54.439269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:54.439269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:18.846115+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":495,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906171+0000","last_change":"2024-09-13T13:09:28.434007+0000","last_active":"2024-09-13T13:11:12.906171+0000","last_peered":"2024-09-13T13:11:12.906171+0000","last_clean":"2024-09-13T13:11:12.906171+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:11:12.906171+0000","last_undegraded":"2024-09-13T13:11:12.906171+0000","last_fullsized":"2024-09-13T13:11:12.906171+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:28.433899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.433899+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:03.236560+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":543,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T13:11:14.277472+0000","last_change":"2024-09-13T13:10:42.010918+0000","last_active":"2024-09-13T13:11:14.277472+0000","last_peered":"2024-09-13T13:11:14.277472+0000","last_clean":"2024-09-13T13:11:14.277472+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:11:14.277472+0000","last_undegraded":"2024-09-13T13:11:14.277472+0000","last_fullsized":"2024-09-13T13:11:14.277472+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:10:42.010847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.010847+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.459829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"271'5939","reported_seq":6010,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.667846+0000","last_change":"2024-09-13T13:10:09.272561+0000","last_active":"2024-09-13T13:11:13.667846+0000","last_peered":"2024-09-13T13:11:13.667846+0000","last_clean":"2024-09-13T13:11:13.667846+0000","last_became_active":"2024-09-13T13:10:09.272264+0000","last_became_peered":"2024-09-13T13:10:09.272264+0000","last_unstale":"2024-09-13T13:11:13.667846+0000","last_undegraded":"2024-09-13T13:11:13.667846+0000","last_fullsized":"2024-09-13T13:11:13.667846+0000","mapping_epoch":241,"log_start":"246'1200","ondisk_log_start":"246'1200","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:35.365520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389218304,"num_objects":5939,"num_object_clones":0,"num_object_copies":17817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5939,"num_write_kb":380096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":501,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.915302+0000","last_change":"2024-09-13T13:07:15.243958+0000","last_active":"2024-09-13T13:11:12.915302+0000","last_peered":"2024-09-13T13:11:12.915302+0000","last_clean":"2024-09-13T13:11:12.915302+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:11:12.915302+0000","last_undegraded":"2024-09-13T13:11:12.915302+0000","last_fullsized":"2024-09-13T13:11:12.915302+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:07:15.243715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:15.243715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.652407+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":477,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902841+0000","last_change":"2024-09-13T13:10:30.552177+0000","last_active":"2024-09-13T13:11:12.902841+0000","last_peered":"2024-09-13T13:11:12.902841+0000","last_clean":"2024-09-13T13:11:12.902841+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:11:12.902841+0000","last_undegraded":"2024-09-13T13:11:12.902841+0000","last_fullsized":"2024-09-13T13:11:12.902841+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:30.552113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.552113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:48.898081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":593,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890716+0000","last_change":"2024-09-13T13:10:07.726810+0000","last_active":"2024-09-13T13:11:12.890716+0000","last_peered":"2024-09-13T13:11:12.890716+0000","last_clean":"2024-09-13T13:11:12.890716+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:11:12.890716+0000","last_undegraded":"2024-09-13T13:11:12.890716+0000","last_fullsized":"2024-09-13T13:11:12.890716+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:10:07.726751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:07.726751+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:32.746229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":457,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.904202+0000","last_change":"2024-09-13T13:10:08.973968+0000","last_active":"2024-09-13T13:11:12.904202+0000","last_peered":"2024-09-13T13:11:12.904202+0000","last_clean":"2024-09-13T13:11:12.904202+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:11:12.904202+0000","last_undegraded":"2024-09-13T13:11:12.904202+0000","last_fullsized":"2024-09-13T13:11:12.904202+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.973917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.973917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"271'5869","reported_seq":5940,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.054565+0000","last_change":"2024-09-13T13:10:09.273532+0000","last_active":"2024-09-13T13:11:13.054565+0000","last_peered":"2024-09-13T13:11:13.054565+0000","last_clean":"2024-09-13T13:11:13.054565+0000","last_became_active":"2024-09-13T13:10:09.273246+0000","last_became_peered":"2024-09-13T13:10:09.273246+0000","last_unstale":"2024-09-13T13:11:13.054565+0000","last_undegraded":"2024-09-13T13:11:13.054565+0000","last_fullsized":"2024-09-13T13:11:13.054565+0000","mapping_epoch":241,"log_start":"246'1000","ondisk_log_start":"246'1000","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4869,"log_dups_size":0,"ondisk_log_size":4869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.627323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384630784,"num_objects":5869,"num_object_clones":0,"num_object_copies":17607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5869,"num_write_kb":375616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":492,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902372+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:11:12.902372+0000","last_peered":"2024-09-13T13:11:12.902372+0000","last_clean":"2024-09-13T13:11:12.902372+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:11:12.902372+0000","last_undegraded":"2024-09-13T13:11:12.902372+0000","last_fullsized":"2024-09-13T13:11:12.902372+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:20.400526+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":459,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.907414+0000","last_change":"2024-09-13T13:06:44.785235+0000","last_active":"2024-09-13T13:11:12.907414+0000","last_peered":"2024-09-13T13:11:12.907414+0000","last_clean":"2024-09-13T13:11:12.907414+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:11:12.907414+0000","last_undegraded":"2024-09-13T13:11:12.907414+0000","last_fullsized":"2024-09-13T13:11:12.907414+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:06:44.785017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.785017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:22.444611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.888703+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:11:12.888703+0000","last_peered":"2024-09-13T13:11:12.888703+0000","last_clean":"2024-09-13T13:11:12.888703+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:11:12.888703+0000","last_undegraded":"2024-09-13T13:11:12.888703+0000","last_fullsized":"2024-09-13T13:11:12.888703+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:21.838553+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":466,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.889485+0000","last_change":"2024-09-13T13:11:10.104997+0000","last_active":"2024-09-13T13:11:12.889485+0000","last_peered":"2024-09-13T13:11:12.889485+0000","last_clean":"2024-09-13T13:11:12.889485+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:11:12.889485+0000","last_undegraded":"2024-09-13T13:11:12.889485+0000","last_fullsized":"2024-09-13T13:11:12.889485+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:10.104926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:10.104926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:33.972960+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"271'5785","reported_seq":5856,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.643606+0000","last_change":"2024-09-13T13:10:09.270421+0000","last_active":"2024-09-13T13:11:13.643606+0000","last_peered":"2024-09-13T13:11:13.643606+0000","last_clean":"2024-09-13T13:11:13.643606+0000","last_became_active":"2024-09-13T13:10:09.270129+0000","last_became_peered":"2024-09-13T13:10:09.270129+0000","last_unstale":"2024-09-13T13:11:13.643606+0000","last_undegraded":"2024-09-13T13:11:13.643606+0000","last_fullsized":"2024-09-13T13:11:13.643606+0000","mapping_epoch":241,"log_start":"246'1000","ondisk_log_start":"246'1000","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4785,"log_dups_size":0,"ondisk_log_size":4785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:37.119106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379125760,"num_objects":5785,"num_object_clones":0,"num_object_copies":17355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5785,"num_write_kb":370240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":519,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.888792+0000","last_change":"2024-09-13T13:09:57.768541+0000","last_active":"2024-09-13T13:11:12.888792+0000","last_peered":"2024-09-13T13:11:12.888792+0000","last_clean":"2024-09-13T13:11:12.888792+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:11:12.888792+0000","last_undegraded":"2024-09-13T13:11:12.888792+0000","last_fullsized":"2024-09-13T13:11:12.888792+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'14","last_scrub_stamp":"2024-09-13T13:09:57.768472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:57.768472+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":445,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906396+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:11:12.906396+0000","last_peered":"2024-09-13T13:11:12.906396+0000","last_clean":"2024-09-13T13:11:12.906396+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:11:12.906396+0000","last_undegraded":"2024-09-13T13:11:12.906396+0000","last_fullsized":"2024-09-13T13:11:12.906396+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.301338+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":585,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902221+0000","last_change":"2024-09-13T13:11:06.526723+0000","last_active":"2024-09-13T13:11:12.902221+0000","last_peered":"2024-09-13T13:11:12.902221+0000","last_clean":"2024-09-13T13:11:12.902221+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:11:12.902221+0000","last_undegraded":"2024-09-13T13:11:12.902221+0000","last_fullsized":"2024-09-13T13:11:12.902221+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:11:06.526646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:06.526646+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:18.710007+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":452,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905014+0000","last_change":"2024-09-13T13:06:09.230872+0000","last_active":"2024-09-13T13:11:12.905014+0000","last_peered":"2024-09-13T13:11:12.905014+0000","last_clean":"2024-09-13T13:11:12.905014+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:11:12.905014+0000","last_undegraded":"2024-09-13T13:11:12.905014+0000","last_fullsized":"2024-09-13T13:11:12.905014+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:09.230824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.230824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:45.998027+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"271'5820","reported_seq":5891,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.170307+0000","last_change":"2024-09-13T13:10:09.273698+0000","last_active":"2024-09-13T13:11:13.170307+0000","last_peered":"2024-09-13T13:11:13.170307+0000","last_clean":"2024-09-13T13:11:13.170307+0000","last_became_active":"2024-09-13T13:10:09.273311+0000","last_became_peered":"2024-09-13T13:10:09.273311+0000","last_unstale":"2024-09-13T13:11:13.170307+0000","last_undegraded":"2024-09-13T13:11:13.170307+0000","last_fullsized":"2024-09-13T13:11:13.170307+0000","mapping_epoch":241,"log_start":"246'500","ondisk_log_start":"246'500","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":5320,"log_dups_size":0,"ondisk_log_size":5320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:20.896856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381419520,"num_objects":5820,"num_object_clones":0,"num_object_copies":17460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5820,"num_write_kb":372480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":496,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.917795+0000","last_change":"2024-09-13T13:10:40.651563+0000","last_active":"2024-09-13T13:11:12.917795+0000","last_peered":"2024-09-13T13:11:12.917795+0000","last_clean":"2024-09-13T13:11:12.917795+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:11:12.917795+0000","last_undegraded":"2024-09-13T13:11:12.917795+0000","last_fullsized":"2024-09-13T13:11:12.917795+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:02.404183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":469,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.904586+0000","last_change":"2024-09-13T13:10:03.895450+0000","last_active":"2024-09-13T13:11:12.904586+0000","last_peered":"2024-09-13T13:11:12.904586+0000","last_clean":"2024-09-13T13:11:12.904586+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:11:12.904586+0000","last_undegraded":"2024-09-13T13:11:12.904586+0000","last_fullsized":"2024-09-13T13:11:12.904586+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:03.895396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:03.895396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:33.888709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":589,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906166+0000","last_change":"2024-09-13T13:10:10.134973+0000","last_active":"2024-09-13T13:11:12.906166+0000","last_peered":"2024-09-13T13:11:12.906166+0000","last_clean":"2024-09-13T13:11:12.906166+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:11:12.906166+0000","last_undegraded":"2024-09-13T13:11:12.906166+0000","last_fullsized":"2024-09-13T13:11:12.906166+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:10:10.134873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.134873+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:12.943527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":465,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905911+0000","last_change":"2024-09-13T13:10:42.390672+0000","last_active":"2024-09-13T13:11:12.905911+0000","last_peered":"2024-09-13T13:11:12.905911+0000","last_clean":"2024-09-13T13:11:12.905911+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:11:12.905911+0000","last_undegraded":"2024-09-13T13:11:12.905911+0000","last_fullsized":"2024-09-13T13:11:12.905911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:42.390473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.390473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:43.033158+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"271'5796","reported_seq":5867,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.016903+0000","last_change":"2024-09-13T13:10:09.270982+0000","last_active":"2024-09-13T13:11:13.016903+0000","last_peered":"2024-09-13T13:11:13.016903+0000","last_clean":"2024-09-13T13:11:13.016903+0000","last_became_active":"2024-09-13T13:10:09.270486+0000","last_became_peered":"2024-09-13T13:10:09.270486+0000","last_unstale":"2024-09-13T13:11:13.016903+0000","last_undegraded":"2024-09-13T13:11:13.016903+0000","last_fullsized":"2024-09-13T13:11:13.016903+0000","mapping_epoch":241,"log_start":"246'900","ondisk_log_start":"246'900","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4896,"log_dups_size":0,"ondisk_log_size":4896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:31.811638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379846656,"num_objects":5796,"num_object_clones":0,"num_object_copies":17388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5796,"num_write_kb":370944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":515,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906466+0000","last_change":"2024-09-13T13:09:05.142515+0000","last_active":"2024-09-13T13:11:12.906466+0000","last_peered":"2024-09-13T13:11:12.906466+0000","last_clean":"2024-09-13T13:11:12.906466+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:11:12.906466+0000","last_undegraded":"2024-09-13T13:11:12.906466+0000","last_fullsized":"2024-09-13T13:11:12.906466+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:09:05.142427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.142427+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.912995+0000","last_change":"2024-09-13T13:09:38.867734+0000","last_active":"2024-09-13T13:11:12.912995+0000","last_peered":"2024-09-13T13:11:12.912995+0000","last_clean":"2024-09-13T13:11:12.912995+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:11:12.912995+0000","last_undegraded":"2024-09-13T13:11:12.912995+0000","last_fullsized":"2024-09-13T13:11:12.912995+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.867688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.867688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:57.738458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":416,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T13:11:14.276037+0000","last_change":"2024-09-13T13:09:38.854654+0000","last_active":"2024-09-13T13:11:14.276037+0000","last_peered":"2024-09-13T13:11:14.276037+0000","last_clean":"2024-09-13T13:11:14.276037+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:11:14.276037+0000","last_undegraded":"2024-09-13T13:11:14.276037+0000","last_fullsized":"2024-09-13T13:11:14.276037+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.854581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.854581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:54.127592+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":461,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.911764+0000","last_change":"2024-09-13T13:06:43.969251+0000","last_active":"2024-09-13T13:11:12.911764+0000","last_peered":"2024-09-13T13:11:12.911764+0000","last_clean":"2024-09-13T13:11:12.911764+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:11:12.911764+0000","last_undegraded":"2024-09-13T13:11:12.911764+0000","last_fullsized":"2024-09-13T13:11:12.911764+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.969176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.969176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:16.852941+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"271'6083","reported_seq":6154,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.979559+0000","last_change":"2024-09-13T13:10:09.277094+0000","last_active":"2024-09-13T13:11:13.979559+0000","last_peered":"2024-09-13T13:11:13.979559+0000","last_clean":"2024-09-13T13:11:13.979559+0000","last_became_active":"2024-09-13T13:10:09.275394+0000","last_became_peered":"2024-09-13T13:10:09.275394+0000","last_unstale":"2024-09-13T13:11:13.979559+0000","last_undegraded":"2024-09-13T13:11:13.979559+0000","last_fullsized":"2024-09-13T13:11:13.979559+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":6083,"log_dups_size":0,"ondisk_log_size":6083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398655488,"num_objects":6083,"num_object_clones":0,"num_object_copies":18249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":389312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"268'40","reported_seq":647,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.912854+0000","last_change":"2024-09-13T13:07:25.438610+0000","last_active":"2024-09-13T13:11:12.912854+0000","last_peered":"2024-09-13T13:11:12.912854+0000","last_clean":"2024-09-13T13:11:12.912854+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:11:12.912854+0000","last_undegraded":"2024-09-13T13:11:12.912854+0000","last_fullsized":"2024-09-13T13:11:12.912854+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:07:25.438528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:25.438528+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:54.678903+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":473,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.888863+0000","last_change":"2024-09-13T13:07:18.888103+0000","last_active":"2024-09-13T13:11:12.888863+0000","last_peered":"2024-09-13T13:11:12.888863+0000","last_clean":"2024-09-13T13:11:12.888863+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:11:12.888863+0000","last_undegraded":"2024-09-13T13:11:12.888863+0000","last_fullsized":"2024-09-13T13:11:12.888863+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:18.888060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:18.888060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:56.046611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":464,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902658+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:11:12.902658+0000","last_peered":"2024-09-13T13:11:12.902658+0000","last_clean":"2024-09-13T13:11:12.902658+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:11:12.902658+0000","last_undegraded":"2024-09-13T13:11:12.902658+0000","last_fullsized":"2024-09-13T13:11:12.902658+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:54.110286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":460,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.907366+0000","last_change":"2024-09-13T13:11:11.413254+0000","last_active":"2024-09-13T13:11:12.907366+0000","last_peered":"2024-09-13T13:11:12.907366+0000","last_clean":"2024-09-13T13:11:12.907366+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:11:12.907366+0000","last_undegraded":"2024-09-13T13:11:12.907366+0000","last_fullsized":"2024-09-13T13:11:12.907366+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:11.413050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.413050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:12.384938+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":457,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902372+0000","last_change":"2024-09-13T13:10:42.489307+0000","last_active":"2024-09-13T13:11:12.902372+0000","last_peered":"2024-09-13T13:11:12.902372+0000","last_clean":"2024-09-13T13:11:12.902372+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:11:12.902372+0000","last_undegraded":"2024-09-13T13:11:12.902372+0000","last_fullsized":"2024-09-13T13:11:12.902372+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:42.489264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.489264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:02.930639+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"271'5844","reported_seq":5915,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.159813+0000","last_change":"2024-09-13T13:10:09.270343+0000","last_active":"2024-09-13T13:11:13.159813+0000","last_peered":"2024-09-13T13:11:13.159813+0000","last_clean":"2024-09-13T13:11:13.159813+0000","last_became_active":"2024-09-13T13:10:09.270073+0000","last_became_peered":"2024-09-13T13:10:09.270073+0000","last_unstale":"2024-09-13T13:11:13.159813+0000","last_undegraded":"2024-09-13T13:11:13.159813+0000","last_fullsized":"2024-09-13T13:11:13.159813+0000","mapping_epoch":241,"log_start":"246'500","ondisk_log_start":"246'500","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":5344,"log_dups_size":0,"ondisk_log_size":5344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:35.580323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382992384,"num_objects":5844,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5844,"num_write_kb":374016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":491,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.915396+0000","last_change":"2024-09-13T13:10:29.799315+0000","last_active":"2024-09-13T13:11:12.915396+0000","last_peered":"2024-09-13T13:11:12.915396+0000","last_clean":"2024-09-13T13:11:12.915396+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:11:12.915396+0000","last_undegraded":"2024-09-13T13:11:12.915396+0000","last_fullsized":"2024-09-13T13:11:12.915396+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:10:29.799185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:29.799185+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:36.850709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":482,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.916599+0000","last_change":"2024-09-13T13:08:56.771534+0000","last_active":"2024-09-13T13:11:12.916599+0000","last_peered":"2024-09-13T13:11:12.916599+0000","last_clean":"2024-09-13T13:11:12.916599+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:11:12.916599+0000","last_undegraded":"2024-09-13T13:11:12.916599+0000","last_fullsized":"2024-09-13T13:11:12.916599+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:56.771432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:56.771432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:21.866535+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":485,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.891232+0000","last_change":"2024-09-13T13:10:30.919483+0000","last_active":"2024-09-13T13:11:12.891232+0000","last_peered":"2024-09-13T13:11:12.891232+0000","last_clean":"2024-09-13T13:11:12.891232+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:11:12.891232+0000","last_undegraded":"2024-09-13T13:11:12.891232+0000","last_fullsized":"2024-09-13T13:11:12.891232+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:10:30.919398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.919398+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:56.559722+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":441,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.891522+0000","last_change":"2024-09-13T13:10:47.053458+0000","last_active":"2024-09-13T13:11:12.891522+0000","last_peered":"2024-09-13T13:11:12.891522+0000","last_clean":"2024-09-13T13:11:12.891522+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:11:12.891522+0000","last_undegraded":"2024-09-13T13:11:12.891522+0000","last_fullsized":"2024-09-13T13:11:12.891522+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:14.282792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:03:14.282792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:14.964426+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":460,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905094+0000","last_change":"2024-09-13T13:08:54.232191+0000","last_active":"2024-09-13T13:11:12.905094+0000","last_peered":"2024-09-13T13:11:12.905094+0000","last_clean":"2024-09-13T13:11:12.905094+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:11:12.905094+0000","last_undegraded":"2024-09-13T13:11:12.905094+0000","last_fullsized":"2024-09-13T13:11:12.905094+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:54.232130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:54.232130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:58.884996+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"271'5960","reported_seq":6031,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.988324+0000","last_change":"2024-09-13T13:10:09.737716+0000","last_active":"2024-09-13T13:11:13.988324+0000","last_peered":"2024-09-13T13:11:13.988324+0000","last_clean":"2024-09-13T13:11:13.988324+0000","last_became_active":"2024-09-13T13:10:09.737415+0000","last_became_peered":"2024-09-13T13:10:09.737415+0000","last_unstale":"2024-09-13T13:11:13.988324+0000","last_undegraded":"2024-09-13T13:11:13.988324+0000","last_fullsized":"2024-09-13T13:11:13.988324+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":5960,"log_dups_size":0,"ondisk_log_size":5960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.782551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390594560,"num_objects":5960,"num_object_clones":0,"num_object_copies":17880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5960,"num_write_kb":381440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":492,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906225+0000","last_change":"2024-09-13T13:08:21.219398+0000","last_active":"2024-09-13T13:11:12.906225+0000","last_peered":"2024-09-13T13:11:12.906225+0000","last_clean":"2024-09-13T13:11:12.906225+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:11:12.906225+0000","last_undegraded":"2024-09-13T13:11:12.906225+0000","last_fullsized":"2024-09-13T13:11:12.906225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:08:21.219300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:21.219300+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:55.256946+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":468,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.907089+0000","last_change":"2024-09-13T13:10:37.465330+0000","last_active":"2024-09-13T13:11:12.907089+0000","last_peered":"2024-09-13T13:11:12.907089+0000","last_clean":"2024-09-13T13:11:12.907089+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:11:12.907089+0000","last_undegraded":"2024-09-13T13:11:12.907089+0000","last_fullsized":"2024-09-13T13:11:12.907089+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:37.465261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:37.465261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:56.060539+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1075,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.904532+0000","last_change":"2024-09-13T13:08:49.264831+0000","last_active":"2024-09-13T13:11:12.904532+0000","last_peered":"2024-09-13T13:11:12.904532+0000","last_clean":"2024-09-13T13:11:12.904532+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:11:12.904532+0000","last_undegraded":"2024-09-13T13:11:12.904532+0000","last_fullsized":"2024-09-13T13:11:12.904532+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:08:49.264775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:49.264775+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:56.918802+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":604,"num_read_kb":599,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":437,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.903011+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:11:12.903011+0000","last_peered":"2024-09-13T13:11:12.903011+0000","last_clean":"2024-09-13T13:11:12.903011+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:11:12.903011+0000","last_undegraded":"2024-09-13T13:11:12.903011+0000","last_fullsized":"2024-09-13T13:11:12.903011+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:48.121624+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":466,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.912484+0000","last_change":"2024-09-13T13:10:32.720742+0000","last_active":"2024-09-13T13:11:12.912484+0000","last_peered":"2024-09-13T13:11:12.912484+0000","last_clean":"2024-09-13T13:11:12.912484+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:11:12.912484+0000","last_undegraded":"2024-09-13T13:11:12.912484+0000","last_fullsized":"2024-09-13T13:11:12.912484+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:41.718820+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"271'5883","reported_seq":5954,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.621427+0000","last_change":"2024-09-13T13:10:09.276701+0000","last_active":"2024-09-13T13:11:13.621427+0000","last_peered":"2024-09-13T13:11:13.621427+0000","last_clean":"2024-09-13T13:11:13.621427+0000","last_became_active":"2024-09-13T13:10:09.276418+0000","last_became_peered":"2024-09-13T13:10:09.276418+0000","last_unstale":"2024-09-13T13:11:13.621427+0000","last_undegraded":"2024-09-13T13:11:13.621427+0000","last_fullsized":"2024-09-13T13:11:13.621427+0000","mapping_epoch":241,"log_start":"246'1100","ondisk_log_start":"246'1100","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:26.865438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385548288,"num_objects":5883,"num_object_clones":0,"num_object_copies":17649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5883,"num_write_kb":376512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"268'30","reported_seq":481,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T13:11:14.276849+0000","last_change":"2024-09-13T13:06:51.193749+0000","last_active":"2024-09-13T13:11:14.276849+0000","last_peered":"2024-09-13T13:11:14.276849+0000","last_clean":"2024-09-13T13:11:14.276849+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:11:14.276849+0000","last_undegraded":"2024-09-13T13:11:14.276849+0000","last_fullsized":"2024-09-13T13:11:14.276849+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'18","last_scrub_stamp":"2024-09-13T13:06:51.193678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.193678+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:17.798203+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":464,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.903058+0000","last_change":"2024-09-13T13:06:10.715371+0000","last_active":"2024-09-13T13:11:12.903058+0000","last_peered":"2024-09-13T13:11:12.903058+0000","last_clean":"2024-09-13T13:11:12.903058+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:11:12.903058+0000","last_undegraded":"2024-09-13T13:11:12.903058+0000","last_fullsized":"2024-09-13T13:11:12.903058+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:10.715313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.715313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.480796+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":510,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.904637+0000","last_change":"2024-09-13T13:06:39.564599+0000","last_active":"2024-09-13T13:11:12.904637+0000","last_peered":"2024-09-13T13:11:12.904637+0000","last_clean":"2024-09-13T13:11:12.904637+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:11:12.904637+0000","last_undegraded":"2024-09-13T13:11:12.904637+0000","last_fullsized":"2024-09-13T13:11:12.904637+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:06:39.564549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:06:39.564549+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:19.739076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":461,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905951+0000","last_change":"2024-09-13T13:08:25.500731+0000","last_active":"2024-09-13T13:11:12.905951+0000","last_peered":"2024-09-13T13:11:12.905951+0000","last_clean":"2024-09-13T13:11:12.905951+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:11:12.905951+0000","last_undegraded":"2024-09-13T13:11:12.905951+0000","last_fullsized":"2024-09-13T13:11:12.905951+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:25.500660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:25.500660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.962133+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":465,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902870+0000","last_change":"2024-09-13T13:10:09.926369+0000","last_active":"2024-09-13T13:11:12.902870+0000","last_peered":"2024-09-13T13:11:12.902870+0000","last_clean":"2024-09-13T13:11:12.902870+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:11:12.902870+0000","last_undegraded":"2024-09-13T13:11:12.902870+0000","last_fullsized":"2024-09-13T13:11:12.902870+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:09.926319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.926319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:39.338826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"271'5890","reported_seq":5961,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.043036+0000","last_change":"2024-09-13T13:10:09.273573+0000","last_active":"2024-09-13T13:11:13.043036+0000","last_peered":"2024-09-13T13:11:13.043036+0000","last_clean":"2024-09-13T13:11:13.043036+0000","last_became_active":"2024-09-13T13:10:09.273226+0000","last_became_peered":"2024-09-13T13:10:09.273226+0000","last_unstale":"2024-09-13T13:11:13.043036+0000","last_undegraded":"2024-09-13T13:11:13.043036+0000","last_fullsized":"2024-09-13T13:11:13.043036+0000","mapping_epoch":241,"log_start":"246'1000","ondisk_log_start":"246'1000","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4890,"log_dups_size":0,"ondisk_log_size":4890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:13.577725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386007040,"num_objects":5890,"num_object_clones":0,"num_object_copies":17670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5890,"num_write_kb":376960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":500,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890775+0000","last_change":"2024-09-13T13:10:30.736738+0000","last_active":"2024-09-13T13:11:12.890775+0000","last_peered":"2024-09-13T13:11:12.890775+0000","last_clean":"2024-09-13T13:11:12.890775+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:11:12.890775+0000","last_undegraded":"2024-09-13T13:11:12.890775+0000","last_fullsized":"2024-09-13T13:11:12.890775+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:10:30.736543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.736543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.681138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":470,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.889101+0000","last_change":"2024-09-13T13:10:01.834881+0000","last_active":"2024-09-13T13:11:12.889101+0000","last_peered":"2024-09-13T13:11:12.889101+0000","last_clean":"2024-09-13T13:11:12.889101+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:11:12.889101+0000","last_undegraded":"2024-09-13T13:11:12.889101+0000","last_fullsized":"2024-09-13T13:11:12.889101+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:01.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:01.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.226771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":480,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.010250+0000","last_change":"2024-09-13T13:11:13.010250+0000","last_active":"2024-09-13T13:11:13.010250+0000","last_peered":"2024-09-13T13:11:13.010250+0000","last_clean":"2024-09-13T13:11:13.010250+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:11:13.010250+0000","last_undegraded":"2024-09-13T13:11:13.010250+0000","last_fullsized":"2024-09-13T13:11:13.010250+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:13.010123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:13.010123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:33.715146+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":460,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902842+0000","last_change":"2024-09-13T13:10:34.526736+0000","last_active":"2024-09-13T13:11:12.902842+0000","last_peered":"2024-09-13T13:11:12.902842+0000","last_clean":"2024-09-13T13:11:12.902842+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:11:12.902842+0000","last_undegraded":"2024-09-13T13:11:12.902842+0000","last_fullsized":"2024-09-13T13:11:12.902842+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:34.526666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:34.526666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.167713+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"271'5889","reported_seq":5960,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.537009+0000","last_change":"2024-09-13T13:10:09.737339+0000","last_active":"2024-09-13T13:11:13.537009+0000","last_peered":"2024-09-13T13:11:13.537009+0000","last_clean":"2024-09-13T13:11:13.537009+0000","last_became_active":"2024-09-13T13:10:09.736988+0000","last_became_peered":"2024-09-13T13:10:09.736988+0000","last_unstale":"2024-09-13T13:11:13.537009+0000","last_undegraded":"2024-09-13T13:11:13.537009+0000","last_fullsized":"2024-09-13T13:11:13.537009+0000","mapping_epoch":241,"log_start":"246'1100","ondisk_log_start":"246'1100","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4789,"log_dups_size":0,"ondisk_log_size":4789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:19.898448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385941504,"num_objects":5889,"num_object_clones":0,"num_object_copies":17667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5889,"num_write_kb":376896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":499,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890884+0000","last_change":"2024-09-13T13:05:34.166464+0000","last_active":"2024-09-13T13:11:12.890884+0000","last_peered":"2024-09-13T13:11:12.890884+0000","last_clean":"2024-09-13T13:11:12.890884+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:11:12.890884+0000","last_undegraded":"2024-09-13T13:11:12.890884+0000","last_fullsized":"2024-09-13T13:11:12.890884+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:20.859961+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":448,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T13:11:14.276367+0000","last_change":"2024-09-13T13:10:10.787443+0000","last_active":"2024-09-13T13:11:14.276367+0000","last_peered":"2024-09-13T13:11:14.276367+0000","last_clean":"2024-09-13T13:11:14.276367+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:11:14.276367+0000","last_undegraded":"2024-09-13T13:11:14.276367+0000","last_fullsized":"2024-09-13T13:11:14.276367+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:10.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:39.817372+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.912567+0000","last_change":"2024-09-13T13:08:53.747370+0000","last_active":"2024-09-13T13:11:12.912567+0000","last_peered":"2024-09-13T13:11:12.912567+0000","last_clean":"2024-09-13T13:11:12.912567+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:11:12.912567+0000","last_undegraded":"2024-09-13T13:11:12.912567+0000","last_fullsized":"2024-09-13T13:11:12.912567+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.747302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.747302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:22.821654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":450,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.891079+0000","last_change":"2024-09-13T13:08:03.015302+0000","last_active":"2024-09-13T13:11:12.891079+0000","last_peered":"2024-09-13T13:11:12.891079+0000","last_clean":"2024-09-13T13:11:12.891079+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:11:12.891079+0000","last_undegraded":"2024-09-13T13:11:12.891079+0000","last_fullsized":"2024-09-13T13:11:12.891079+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:03.015233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.015233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:26.850341+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"271'5695","reported_seq":5766,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.637570+0000","last_change":"2024-09-13T13:10:09.737741+0000","last_active":"2024-09-13T13:11:13.637570+0000","last_peered":"2024-09-13T13:11:13.637570+0000","last_clean":"2024-09-13T13:11:13.637570+0000","last_became_active":"2024-09-13T13:10:09.737415+0000","last_became_peered":"2024-09-13T13:10:09.737415+0000","last_unstale":"2024-09-13T13:11:13.637570+0000","last_undegraded":"2024-09-13T13:11:13.637570+0000","last_fullsized":"2024-09-13T13:11:13.637570+0000","mapping_epoch":241,"log_start":"246'1000","ondisk_log_start":"246'1000","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4695,"log_dups_size":0,"ondisk_log_size":4695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:35.194931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373227520,"num_objects":5695,"num_object_clones":0,"num_object_copies":17085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5695,"num_write_kb":364480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":515,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890154+0000","last_change":"2024-09-13T13:10:09.739258+0000","last_active":"2024-09-13T13:11:12.890154+0000","last_peered":"2024-09-13T13:11:12.890154+0000","last_clean":"2024-09-13T13:11:12.890154+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:11:12.890154+0000","last_undegraded":"2024-09-13T13:11:12.890154+0000","last_fullsized":"2024-09-13T13:11:12.890154+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:09.739184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.739184+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.869202+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":453,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.907640+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:11:12.907640+0000","last_peered":"2024-09-13T13:11:12.907640+0000","last_clean":"2024-09-13T13:11:12.907640+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:11:12.907640+0000","last_undegraded":"2024-09-13T13:11:12.907640+0000","last_fullsized":"2024-09-13T13:11:12.907640+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:21.445014+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":577,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890767+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:11:12.890767+0000","last_peered":"2024-09-13T13:11:12.890767+0000","last_clean":"2024-09-13T13:11:12.890767+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:11:12.890767+0000","last_undegraded":"2024-09-13T13:11:12.890767+0000","last_fullsized":"2024-09-13T13:11:12.890767+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:14.921175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":450,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890594+0000","last_change":"2024-09-13T13:11:05.084909+0000","last_active":"2024-09-13T13:11:12.890594+0000","last_peered":"2024-09-13T13:11:12.890594+0000","last_clean":"2024-09-13T13:11:12.890594+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:11:12.890594+0000","last_undegraded":"2024-09-13T13:11:12.890594+0000","last_fullsized":"2024-09-13T13:11:12.890594+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:05.084772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:05.084772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.857054+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"271'5857","reported_seq":5992,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.151764+0000","last_change":"2024-09-13T13:11:11.072082+0000","last_active":"2024-09-13T13:11:13.151764+0000","last_peered":"2024-09-13T13:11:13.151764+0000","last_clean":"2024-09-13T13:11:13.151764+0000","last_became_active":"2024-09-13T13:10:09.274589+0000","last_became_peered":"2024-09-13T13:10:09.274589+0000","last_unstale":"2024-09-13T13:11:13.151764+0000","last_undegraded":"2024-09-13T13:11:13.151764+0000","last_fullsized":"2024-09-13T13:11:13.151764+0000","mapping_epoch":241,"log_start":"246'500","ondisk_log_start":"246'500","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'5724","last_scrub_stamp":"2024-09-13T13:11:11.071957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.071957+0000","objects_scrubbed":5659,"log_size":5357,"log_dups_size":0,"ondisk_log_size":5357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:14.421303+0000","scrub_duration":1921,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383844352,"num_objects":5857,"num_object_clones":0,"num_object_copies":17571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5857,"num_write_kb":374848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":465,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906371+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:11:12.906371+0000","last_peered":"2024-09-13T13:11:12.906371+0000","last_clean":"2024-09-13T13:11:12.906371+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:11:12.906371+0000","last_undegraded":"2024-09-13T13:11:12.906371+0000","last_fullsized":"2024-09-13T13:11:12.906371+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:20.208272+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":488,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.901477+0000","last_change":"2024-09-13T13:10:41.028120+0000","last_active":"2024-09-13T13:11:12.901477+0000","last_peered":"2024-09-13T13:11:12.901477+0000","last_clean":"2024-09-13T13:11:12.901477+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:11:12.901477+0000","last_undegraded":"2024-09-13T13:11:12.901477+0000","last_fullsized":"2024-09-13T13:11:12.901477+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:41.028047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:41.028047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:06.786834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":462,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890150+0000","last_change":"2024-09-13T13:07:11.400066+0000","last_active":"2024-09-13T13:11:12.890150+0000","last_peered":"2024-09-13T13:11:12.890150+0000","last_clean":"2024-09-13T13:11:12.890150+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:11:12.890150+0000","last_undegraded":"2024-09-13T13:11:12.890150+0000","last_fullsized":"2024-09-13T13:11:12.890150+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.399840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.399840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:45.908383+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":448,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906996+0000","last_change":"2024-09-13T13:11:10.432159+0000","last_active":"2024-09-13T13:11:12.906996+0000","last_peered":"2024-09-13T13:11:12.906996+0000","last_clean":"2024-09-13T13:11:12.906996+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:11:12.906996+0000","last_undegraded":"2024-09-13T13:11:12.906996+0000","last_fullsized":"2024-09-13T13:11:12.906996+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:10.432086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:10.432086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:13.775919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"271'5838","reported_seq":5909,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.160877+0000","last_change":"2024-09-13T13:10:09.271210+0000","last_active":"2024-09-13T13:11:13.160877+0000","last_peered":"2024-09-13T13:11:13.160877+0000","last_clean":"2024-09-13T13:11:13.160877+0000","last_became_active":"2024-09-13T13:10:09.270909+0000","last_became_peered":"2024-09-13T13:10:09.270909+0000","last_unstale":"2024-09-13T13:11:13.160877+0000","last_undegraded":"2024-09-13T13:11:13.160877+0000","last_fullsized":"2024-09-13T13:11:13.160877+0000","mapping_epoch":241,"log_start":"246'500","ondisk_log_start":"246'500","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":5338,"log_dups_size":0,"ondisk_log_size":5338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:15.899099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382599168,"num_objects":5838,"num_object_clones":0,"num_object_copies":17514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5838,"num_write_kb":373632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":475,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T13:11:14.276135+0000","last_change":"2024-09-13T13:09:32.849360+0000","last_active":"2024-09-13T13:11:14.276135+0000","last_peered":"2024-09-13T13:11:14.276135+0000","last_clean":"2024-09-13T13:11:14.276135+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:11:14.276135+0000","last_undegraded":"2024-09-13T13:11:14.276135+0000","last_fullsized":"2024-09-13T13:11:14.276135+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:09:32.849277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:32.849277+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:01.179140+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":480,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906112+0000","last_change":"2024-09-13T13:10:27.802685+0000","last_active":"2024-09-13T13:11:12.906112+0000","last_peered":"2024-09-13T13:11:12.906112+0000","last_clean":"2024-09-13T13:11:12.906112+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:11:12.906112+0000","last_undegraded":"2024-09-13T13:11:12.906112+0000","last_fullsized":"2024-09-13T13:11:12.906112+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:27.802621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:27.802621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:40.450332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":462,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890143+0000","last_change":"2024-09-13T13:08:26.033729+0000","last_active":"2024-09-13T13:11:12.890143+0000","last_peered":"2024-09-13T13:11:12.890143+0000","last_clean":"2024-09-13T13:11:12.890143+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:11:12.890143+0000","last_undegraded":"2024-09-13T13:11:12.890143+0000","last_fullsized":"2024-09-13T13:11:12.890143+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:26.033663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:26.033663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:55.961768+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":457,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.904553+0000","last_change":"2024-09-13T13:10:40.961803+0000","last_active":"2024-09-13T13:11:12.904553+0000","last_peered":"2024-09-13T13:11:12.904553+0000","last_clean":"2024-09-13T13:11:12.904553+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:11:12.904553+0000","last_undegraded":"2024-09-13T13:11:12.904553+0000","last_fullsized":"2024-09-13T13:11:12.904553+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:40.961744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.961744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:59.893697+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"271'5892","reported_seq":5961,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.579366+0000","last_change":"2024-09-13T13:10:09.270525+0000","last_active":"2024-09-13T13:11:13.579366+0000","last_peered":"2024-09-13T13:11:13.579366+0000","last_clean":"2024-09-13T13:11:13.579366+0000","last_became_active":"2024-09-13T13:10:09.270190+0000","last_became_peered":"2024-09-13T13:10:09.270190+0000","last_unstale":"2024-09-13T13:11:13.579366+0000","last_undegraded":"2024-09-13T13:11:13.579366+0000","last_fullsized":"2024-09-13T13:11:13.579366+0000","mapping_epoch":241,"log_start":"246'1100","ondisk_log_start":"246'1100","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386138112,"num_objects":5892,"num_object_clones":0,"num_object_copies":17676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5892,"num_write_kb":377088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":482,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.888962+0000","last_change":"2024-09-13T13:09:30.614842+0000","last_active":"2024-09-13T13:11:12.888962+0000","last_peered":"2024-09-13T13:11:12.888962+0000","last_clean":"2024-09-13T13:11:12.888962+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:11:12.888962+0000","last_undegraded":"2024-09-13T13:11:12.888962+0000","last_fullsized":"2024-09-13T13:11:12.888962+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:09:30.614748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.614748+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:03.028024+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":461,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905554+0000","last_change":"2024-09-13T13:05:29.345863+0000","last_active":"2024-09-13T13:11:12.905554+0000","last_peered":"2024-09-13T13:11:12.905554+0000","last_clean":"2024-09-13T13:11:12.905554+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:11:12.905554+0000","last_undegraded":"2024-09-13T13:11:12.905554+0000","last_fullsized":"2024-09-13T13:11:12.905554+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.345796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.345796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:16.704307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":446,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.896111+0000","last_change":"2024-09-13T13:06:42.450669+0000","last_active":"2024-09-13T13:11:12.896111+0000","last_peered":"2024-09-13T13:11:12.896111+0000","last_clean":"2024-09-13T13:11:12.896111+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:11:12.896111+0000","last_undegraded":"2024-09-13T13:11:12.896111+0000","last_fullsized":"2024-09-13T13:11:12.896111+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:42.450598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:42.450598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:20.962607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":433,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.904448+0000","last_change":"2024-09-13T13:03:20.523572+0000","last_active":"2024-09-13T13:11:12.904448+0000","last_peered":"2024-09-13T13:11:12.904448+0000","last_clean":"2024-09-13T13:11:12.904448+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:11:12.904448+0000","last_undegraded":"2024-09-13T13:11:12.904448+0000","last_fullsized":"2024-09-13T13:11:12.904448+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:20.523300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:03:20.523300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:16.374703+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"271'5915","reported_seq":5986,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:13.984678+0000","last_change":"2024-09-13T13:10:09.275649+0000","last_active":"2024-09-13T13:11:13.984678+0000","last_peered":"2024-09-13T13:11:13.984678+0000","last_clean":"2024-09-13T13:11:13.984678+0000","last_became_active":"2024-09-13T13:10:09.275320+0000","last_became_peered":"2024-09-13T13:10:09.275320+0000","last_unstale":"2024-09-13T13:11:13.984678+0000","last_undegraded":"2024-09-13T13:11:13.984678+0000","last_fullsized":"2024-09-13T13:11:13.984678+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":5915,"log_dups_size":0,"ondisk_log_size":5915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387645440,"num_objects":5915,"num_object_clones":0,"num_object_copies":17745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":378560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":511,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902106+0000","last_change":"2024-09-13T13:09:33.843654+0000","last_active":"2024-09-13T13:11:12.902106+0000","last_peered":"2024-09-13T13:11:12.902106+0000","last_clean":"2024-09-13T13:11:12.902106+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:11:12.902106+0000","last_undegraded":"2024-09-13T13:11:12.902106+0000","last_fullsized":"2024-09-13T13:11:12.902106+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:09:33.843579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:33.843579+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:46.077623+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":473,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905878+0000","last_change":"2024-09-13T13:06:51.627764+0000","last_active":"2024-09-13T13:11:12.905878+0000","last_peered":"2024-09-13T13:11:12.905878+0000","last_clean":"2024-09-13T13:11:12.905878+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:11:12.905878+0000","last_undegraded":"2024-09-13T13:11:12.905878+0000","last_fullsized":"2024-09-13T13:11:12.905878+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:06:51.627691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.627691+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:21.738706+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":601,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.904604+0000","last_change":"2024-09-13T13:11:11.751140+0000","last_active":"2024-09-13T13:11:12.904604+0000","last_peered":"2024-09-13T13:11:12.904604+0000","last_clean":"2024-09-13T13:11:12.904604+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:11:12.904604+0000","last_undegraded":"2024-09-13T13:11:12.904604+0000","last_fullsized":"2024-09-13T13:11:12.904604+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:11:11.751078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.751078+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.052505+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":460,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902523+0000","last_change":"2024-09-13T13:10:42.034653+0000","last_active":"2024-09-13T13:11:12.902523+0000","last_peered":"2024-09-13T13:11:12.902523+0000","last_clean":"2024-09-13T13:11:12.902523+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:11:12.902523+0000","last_undegraded":"2024-09-13T13:11:12.902523+0000","last_fullsized":"2024-09-13T13:11:12.902523+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:42.034601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.034601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.440553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":432,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T13:11:14.276154+0000","last_change":"2024-09-13T13:07:10.058616+0000","last_active":"2024-09-13T13:11:14.276154+0000","last_peered":"2024-09-13T13:11:14.276154+0000","last_clean":"2024-09-13T13:11:14.276154+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:11:14.276154+0000","last_undegraded":"2024-09-13T13:11:14.276154+0000","last_fullsized":"2024-09-13T13:11:14.276154+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:10.058555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.058555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:50.118617+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":467,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.907262+0000","last_change":"2024-09-13T13:09:37.358795+0000","last_active":"2024-09-13T13:11:12.907262+0000","last_peered":"2024-09-13T13:11:12.907262+0000","last_clean":"2024-09-13T13:11:12.907262+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:11:12.907262+0000","last_undegraded":"2024-09-13T13:11:12.907262+0000","last_fullsized":"2024-09-13T13:11:12.907262+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:09:37.358710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:37.358710+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:55.589722+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":469,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902837+0000","last_change":"2024-09-13T13:11:12.054173+0000","last_active":"2024-09-13T13:11:12.902837+0000","last_peered":"2024-09-13T13:11:12.902837+0000","last_clean":"2024-09-13T13:11:12.902837+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:11:12.902837+0000","last_undegraded":"2024-09-13T13:11:12.902837+0000","last_fullsized":"2024-09-13T13:11:12.902837+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:12.054125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:12.054125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.049869+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":468,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905007+0000","last_change":"2024-09-13T13:09:30.959919+0000","last_active":"2024-09-13T13:11:12.905007+0000","last_peered":"2024-09-13T13:11:12.905007+0000","last_clean":"2024-09-13T13:11:12.905007+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:11:12.905007+0000","last_undegraded":"2024-09-13T13:11:12.905007+0000","last_fullsized":"2024-09-13T13:11:12.905007+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.959870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.959870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:01.787830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":473,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906309+0000","last_change":"2024-09-13T13:09:34.426362+0000","last_active":"2024-09-13T13:11:12.906309+0000","last_peered":"2024-09-13T13:11:12.906309+0000","last_clean":"2024-09-13T13:11:12.906309+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:11:12.906309+0000","last_undegraded":"2024-09-13T13:11:12.906309+0000","last_fullsized":"2024-09-13T13:11:12.906309+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:34.426272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:34.426272+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:59.241784+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":456,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906191+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:11:12.906191+0000","last_peered":"2024-09-13T13:11:12.906191+0000","last_clean":"2024-09-13T13:11:12.906191+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:11:12.906191+0000","last_undegraded":"2024-09-13T13:11:12.906191+0000","last_fullsized":"2024-09-13T13:11:12.906191+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.986182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":453,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.907058+0000","last_change":"2024-09-13T13:11:11.553996+0000","last_active":"2024-09-13T13:11:12.907058+0000","last_peered":"2024-09-13T13:11:12.907058+0000","last_clean":"2024-09-13T13:11:12.907058+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:11:12.907058+0000","last_undegraded":"2024-09-13T13:11:12.907058+0000","last_fullsized":"2024-09-13T13:11:12.907058+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:11.553908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.553908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:41.237817+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":466,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890294+0000","last_change":"2024-09-13T13:10:34.954684+0000","last_active":"2024-09-13T13:11:12.890294+0000","last_peered":"2024-09-13T13:11:12.890294+0000","last_clean":"2024-09-13T13:11:12.890294+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:11:12.890294+0000","last_undegraded":"2024-09-13T13:11:12.890294+0000","last_fullsized":"2024-09-13T13:11:12.890294+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:34.954585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:34.954585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.029347+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":469,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.904459+0000","last_change":"2024-09-13T13:09:48.990167+0000","last_active":"2024-09-13T13:11:12.904459+0000","last_peered":"2024-09-13T13:11:12.904459+0000","last_clean":"2024-09-13T13:11:12.904459+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:11:12.904459+0000","last_undegraded":"2024-09-13T13:11:12.904459+0000","last_fullsized":"2024-09-13T13:11:12.904459+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:16.980190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":452,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906026+0000","last_change":"2024-09-13T13:06:41.308449+0000","last_active":"2024-09-13T13:11:12.906026+0000","last_peered":"2024-09-13T13:11:12.906026+0000","last_clean":"2024-09-13T13:11:12.906026+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:11:12.906026+0000","last_undegraded":"2024-09-13T13:11:12.906026+0000","last_fullsized":"2024-09-13T13:11:12.906026+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:41.308367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:41.308367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:27.297312+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":576,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902993+0000","last_change":"2024-09-13T13:10:07.230375+0000","last_active":"2024-09-13T13:11:12.902993+0000","last_peered":"2024-09-13T13:11:12.902993+0000","last_clean":"2024-09-13T13:11:12.902993+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:11:12.902993+0000","last_undegraded":"2024-09-13T13:11:12.902993+0000","last_fullsized":"2024-09-13T13:11:12.902993+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:10:07.230276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:07.230276+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:15.266427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":450,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890733+0000","last_change":"2024-09-13T13:07:21.958114+0000","last_active":"2024-09-13T13:11:12.890733+0000","last_peered":"2024-09-13T13:11:12.890733+0000","last_clean":"2024-09-13T13:11:12.890733+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:11:12.890733+0000","last_undegraded":"2024-09-13T13:11:12.890733+0000","last_fullsized":"2024-09-13T13:11:12.890733+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:21.958030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:21.958030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:59.015030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":487,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.903327+0000","last_change":"2024-09-13T13:09:04.635198+0000","last_active":"2024-09-13T13:11:12.903327+0000","last_peered":"2024-09-13T13:11:12.903327+0000","last_clean":"2024-09-13T13:11:12.903327+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:11:12.903327+0000","last_undegraded":"2024-09-13T13:11:12.903327+0000","last_fullsized":"2024-09-13T13:11:12.903327+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:04.599040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:04.599040+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:52.519601+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":445,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905412+0000","last_change":"2024-09-13T13:04:50.432966+0000","last_active":"2024-09-13T13:11:12.905412+0000","last_peered":"2024-09-13T13:11:12.905412+0000","last_clean":"2024-09-13T13:11:12.905412+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:11:12.905412+0000","last_undegraded":"2024-09-13T13:11:12.905412+0000","last_fullsized":"2024-09-13T13:11:12.905412+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:50.432923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:50.432923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:13.751570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":592,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.904299+0000","last_change":"2024-09-13T13:07:08.781005+0000","last_active":"2024-09-13T13:11:12.904299+0000","last_peered":"2024-09-13T13:11:12.904299+0000","last_clean":"2024-09-13T13:11:12.904299+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:11:12.904299+0000","last_undegraded":"2024-09-13T13:11:12.904299+0000","last_fullsized":"2024-09-13T13:11:12.904299+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:07:08.780952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.780952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:00.142438+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":448,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905876+0000","last_change":"2024-09-13T13:09:36.414394+0000","last_active":"2024-09-13T13:11:12.905876+0000","last_peered":"2024-09-13T13:11:12.905876+0000","last_clean":"2024-09-13T13:11:12.905876+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:11:12.905876+0000","last_undegraded":"2024-09-13T13:11:12.905876+0000","last_fullsized":"2024-09-13T13:11:12.905876+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:36.414322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:36.414322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:48.346115+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":492,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.888708+0000","last_change":"2024-09-13T13:05:26.186899+0000","last_active":"2024-09-13T13:11:12.888708+0000","last_peered":"2024-09-13T13:11:12.888708+0000","last_clean":"2024-09-13T13:11:12.888708+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:11:12.888708+0000","last_undegraded":"2024-09-13T13:11:12.888708+0000","last_fullsized":"2024-09-13T13:11:12.888708+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:05:26.186830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:26.186830+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:17.825948+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":465,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.890382+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:11:12.890382+0000","last_peered":"2024-09-13T13:11:12.890382+0000","last_clean":"2024-09-13T13:11:12.890382+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:11:12.890382+0000","last_undegraded":"2024-09-13T13:11:12.890382+0000","last_fullsized":"2024-09-13T13:11:12.890382+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:18.854424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":454,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.917627+0000","last_change":"2024-09-13T13:09:05.882862+0000","last_active":"2024-09-13T13:11:12.917627+0000","last_peered":"2024-09-13T13:11:12.917627+0000","last_clean":"2024-09-13T13:11:12.917627+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:11:12.917627+0000","last_undegraded":"2024-09-13T13:11:12.917627+0000","last_fullsized":"2024-09-13T13:11:12.917627+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:05.882769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.882769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:20.891630+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":449,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.903613+0000","last_change":"2024-09-13T13:08:59.585721+0000","last_active":"2024-09-13T13:11:12.903613+0000","last_peered":"2024-09-13T13:11:12.903613+0000","last_clean":"2024-09-13T13:11:12.903613+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:11:12.903613+0000","last_undegraded":"2024-09-13T13:11:12.903613+0000","last_fullsized":"2024-09-13T13:11:12.903613+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:59.585646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.585646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:51.544074+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":425,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902376+0000","last_change":"2024-09-13T13:10:10.932948+0000","last_active":"2024-09-13T13:11:12.902376+0000","last_peered":"2024-09-13T13:11:12.902376+0000","last_clean":"2024-09-13T13:11:12.902376+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:11:12.902376+0000","last_undegraded":"2024-09-13T13:11:12.902376+0000","last_fullsized":"2024-09-13T13:11:12.902376+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:10.932889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.932889+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:33.657024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":461,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905971+0000","last_change":"2024-09-13T13:09:07.118545+0000","last_active":"2024-09-13T13:11:12.905971+0000","last_peered":"2024-09-13T13:11:12.905971+0000","last_clean":"2024-09-13T13:11:12.905971+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:11:12.905971+0000","last_undegraded":"2024-09-13T13:11:12.905971+0000","last_fullsized":"2024-09-13T13:11:12.905971+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:07.118467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:07.118467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:59.839803+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":453,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.907404+0000","last_change":"2024-09-13T13:11:08.551743+0000","last_active":"2024-09-13T13:11:12.907404+0000","last_peered":"2024-09-13T13:11:12.907404+0000","last_clean":"2024-09-13T13:11:12.907404+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:11:12.907404+0000","last_undegraded":"2024-09-13T13:11:12.907404+0000","last_fullsized":"2024-09-13T13:11:12.907404+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:08.551449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:08.551449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:26.007369+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":456,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905533+0000","last_change":"2024-09-13T13:10:39.477882+0000","last_active":"2024-09-13T13:11:12.905533+0000","last_peered":"2024-09-13T13:11:12.905533+0000","last_clean":"2024-09-13T13:11:12.905533+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:11:12.905533+0000","last_undegraded":"2024-09-13T13:11:12.905533+0000","last_fullsized":"2024-09-13T13:11:12.905533+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:39.477683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.477683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:04.956182+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":483,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.903997+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:11:12.903997+0000","last_peered":"2024-09-13T13:11:12.903997+0000","last_clean":"2024-09-13T13:11:12.903997+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:11:12.903997+0000","last_undegraded":"2024-09-13T13:11:12.903997+0000","last_fullsized":"2024-09-13T13:11:12.903997+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:18.352140+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":461,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902829+0000","last_change":"2024-09-13T13:08:50.517587+0000","last_active":"2024-09-13T13:11:12.902829+0000","last_peered":"2024-09-13T13:11:12.902829+0000","last_clean":"2024-09-13T13:11:12.902829+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:11:12.902829+0000","last_undegraded":"2024-09-13T13:11:12.902829+0000","last_fullsized":"2024-09-13T13:11:12.902829+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:50.517526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:50.517526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:20.011573+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":464,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.904297+0000","last_change":"2024-09-13T13:08:53.215325+0000","last_active":"2024-09-13T13:11:12.904297+0000","last_peered":"2024-09-13T13:11:12.904297+0000","last_clean":"2024-09-13T13:11:12.904297+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:11:12.904297+0000","last_undegraded":"2024-09-13T13:11:12.904297+0000","last_fullsized":"2024-09-13T13:11:12.904297+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.215263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.215263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:50.963832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":444,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.907557+0000","last_change":"2024-09-13T13:07:57.596272+0000","last_active":"2024-09-13T13:11:12.907557+0000","last_peered":"2024-09-13T13:11:12.907557+0000","last_clean":"2024-09-13T13:11:12.907557+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:11:12.907557+0000","last_undegraded":"2024-09-13T13:11:12.907557+0000","last_fullsized":"2024-09-13T13:11:12.907557+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.596097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:53.509678+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":485,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902279+0000","last_change":"2024-09-13T13:09:39.368581+0000","last_active":"2024-09-13T13:11:12.902279+0000","last_peered":"2024-09-13T13:11:12.902279+0000","last_clean":"2024-09-13T13:11:12.902279+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:11:12.902279+0000","last_undegraded":"2024-09-13T13:11:12.902279+0000","last_fullsized":"2024-09-13T13:11:12.902279+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2024-09-13T13:09:39.368381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:39.368381+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:57.544832+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":476,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.907245+0000","last_change":"2024-09-13T13:11:04.275826+0000","last_active":"2024-09-13T13:11:12.907245+0000","last_peered":"2024-09-13T13:11:12.907245+0000","last_clean":"2024-09-13T13:11:12.907245+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:11:12.907245+0000","last_undegraded":"2024-09-13T13:11:12.907245+0000","last_fullsized":"2024-09-13T13:11:12.907245+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:04.275756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:04.275756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:29.145451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":364,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T13:11:14.277973+0000","last_change":"2024-09-13T13:10:36.918019+0000","last_active":"2024-09-13T13:11:14.277973+0000","last_peered":"2024-09-13T13:11:14.277973+0000","last_clean":"2024-09-13T13:11:14.277973+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:11:14.277973+0000","last_undegraded":"2024-09-13T13:11:14.277973+0000","last_fullsized":"2024-09-13T13:11:14.277973+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:36.917945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:36.917945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.082612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":465,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.905428+0000","last_change":"2024-09-13T13:11:08.704496+0000","last_active":"2024-09-13T13:11:12.905428+0000","last_peered":"2024-09-13T13:11:12.905428+0000","last_clean":"2024-09-13T13:11:12.905428+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:11:12.905428+0000","last_undegraded":"2024-09-13T13:11:12.905428+0000","last_fullsized":"2024-09-13T13:11:12.905428+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:08.704450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:08.704450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.469645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":450,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.912231+0000","last_change":"2024-09-13T13:08:27.641188+0000","last_active":"2024-09-13T13:11:12.912231+0000","last_peered":"2024-09-13T13:11:12.912231+0000","last_clean":"2024-09-13T13:11:12.912231+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:11:12.912231+0000","last_undegraded":"2024-09-13T13:11:12.912231+0000","last_fullsized":"2024-09-13T13:11:12.912231+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:27.641105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:27.641105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:19.856337+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":461,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.902762+0000","last_change":"2024-09-13T13:11:07.564094+0000","last_active":"2024-09-13T13:11:12.902762+0000","last_peered":"2024-09-13T13:11:12.902762+0000","last_clean":"2024-09-13T13:11:12.902762+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:11:12.902762+0000","last_undegraded":"2024-09-13T13:11:12.902762+0000","last_fullsized":"2024-09-13T13:11:12.902762+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:07.564025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:07.564025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:18.616035+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":462,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T13:11:14.277333+0000","last_change":"2024-09-13T13:06:09.928585+0000","last_active":"2024-09-13T13:11:14.277333+0000","last_peered":"2024-09-13T13:11:14.277333+0000","last_clean":"2024-09-13T13:11:14.277333+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:11:14.277333+0000","last_undegraded":"2024-09-13T13:11:14.277333+0000","last_fullsized":"2024-09-13T13:11:14.277333+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:06:09.928504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.928504+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.088075+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":461,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T13:11:12.906082+0000","last_change":"2024-09-13T13:09:38.337230+0000","last_active":"2024-09-13T13:11:12.906082+0000","last_peered":"2024-09-13T13:11:12.906082+0000","last_clean":"2024-09-13T13:11:12.906082+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:11:12.906082+0000","last_undegraded":"2024-09-13T13:11:12.906082+0000","last_fullsized":"2024-09-13T13:11:12.906082+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.337156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.337156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:50.333108+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6157434880,"num_objects":93955,"num_object_clones":0,"num_object_copies":281865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93955,"num_write_kb":6013120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18492424192,"data_stored":18492424192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82555,"ondisk_log_size":82555,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":681,"ondisk_log_size":681,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":604,"num_read_kb":599,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201405,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3678992,"kb_used_data":3350720,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":90066160,"statfs":{"total":95995035648,"available":92227747840,"internally_reserved":0,"allocated":3431137280,"data_stored":3430573316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":336132554},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141016,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1430300,"kb_used_data":1144328,"kb_used_omap":4,"kb_used_meta":285947,"kb_avail":92314852,"statfs":{"total":95995035648,"available":94530408448,"internally_reserved":0,"allocated":1171791872,"data_stored":1171231807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":292810413},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691977,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3712796,"kb_used_data":3378944,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":90032356,"statfs":{"total":95995035648,"available":92193132544,"internally_reserved":0,"allocated":3460038656,"data_stored":3459479844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,8,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920911,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1371024,"kb_used_data":1101052,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92374128,"statfs":{"total":95995035648,"available":94591107072,"internally_reserved":0,"allocated":1127477248,"data_stored":1126908906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":276429254},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117142,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3420596,"kb_used_data":2997912,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":90324556,"statfs":{"total":95995035648,"available":92492345344,"internally_reserved":0,"allocated":3069861888,"data_stored":3069279466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":432798151},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411481,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1419768,"kb_used_data":1122020,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92325384,"statfs":{"total":95995035648,"available":94541193216,"internally_reserved":0,"allocated":1148948480,"data_stored":1148410824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":304871878},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575008,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3733492,"kb_used_data":3363548,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":90011660,"statfs":{"total":95995035648,"available":92171939840,"internally_reserved":0,"allocated":3444273152,"data_stored":3443679459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738535,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1851236,"kb_used_data":1517392,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91893916,"statfs":{"total":95995035648,"available":94099369984,"internally_reserved":0,"allocated":1553809408,"data_stored":1553235625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1563885568,"data_stored":1563885568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3471900672,"data_stored":3471900672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1157234688,"data_stored":1157234688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3071737856,"data_stored":3071737856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1140064256,"data_stored":1140064256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3465543680,"data_stored":3465543680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1181220864,"data_stored":1181220864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3440836608,"data_stored":3440836608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:11:23.705 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:11:23.706 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:23.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:23.797 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 68 16 105772 105756 97.1671 80.6875 0.00982509 0.0102867 2024-09-13T13:11:23.812 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:23 smithi167 ceph-mon[33178]: pgmap v500: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 2.6 KiB/s rd, 45 MiB/s wr, 721 op/s 2024-09-13T13:11:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:23 smithi167 ceph-mon[33178]: osdmap e275: 8 total, 8 up, 8 in 2024-09-13T13:11:23.853 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:23.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:23.955 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:24.013 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:24.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:24.115 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:24.158 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:24.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:24.260 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:24.293 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:24.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:24.395 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:24.433 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:24.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:24.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:24.535 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:24.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:11:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:11:24.574 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:24.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:24.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:24.645 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:24.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:24.707 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:24.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:24.798 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 69 16 107015 106999 96.884 77.6875 0.0145997 0.0103169 2024-09-13T13:11:24.809 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:24.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:24.856 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:24.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:24.957 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:24.987 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:24.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:25.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:25.122 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:25.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:25.223 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:25.234 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:25.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:25.269 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:25.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:25.289 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:11:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:11:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:11:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:11:25.290 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:11:25.370 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:25.410 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:25.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:25.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:25.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:25 smithi052 ceph-mon[36682]: pgmap v502: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-13T13:11:25.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:25.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:25 smithi052 ceph-mon[29471]: pgmap v502: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-13T13:11:25.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:25.513 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:25.554 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:25.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:25.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:25.656 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:25.693 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:25.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:25.794 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:25.796 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:11:25.798 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 70 16 108248 108232 96.6008 77.0625 0.00867082 0.0103471 2024-09-13T13:11:25.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:25 smithi167 ceph-mon[33178]: pgmap v502: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-13T13:11:25.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:25.831 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:25.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:25.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:25.934 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:25.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:25.990 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:25.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:26.014 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T13:11:26.091 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:26.122 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:26.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:26.224 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:26.241 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:11:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:11:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:11:26.275 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:26.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:26.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:26.377 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:26.434 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:26.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:26.536 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:26.579 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:26.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:26.674 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:11:26.682 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:26.760 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:26.773 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:26.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:26.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:26.775 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:26.802 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 71 16 109301 109285 96.1617 65.8125 0.0794885 0.0103881 2024-09-13T13:11:26.875 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:26.935 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:26.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:26.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:27.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:27.067 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:27.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:27.091 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:11:27.093 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:11:27.170 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:27.206 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:27.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:27.309 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:27.353 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:27.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:27.454 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:27.475 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:27.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:27.577 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:27.617 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:27.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:27.718 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:27.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:27 smithi052 ceph-mon[36682]: pgmap v503: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 90 MiB/s wr, 1.45k op/s 2024-09-13T13:11:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:27 smithi052 ceph-mon[29471]: pgmap v503: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 90 MiB/s wr, 1.45k op/s 2024-09-13T13:11:27.752 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:27.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:27.802 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 72 16 110326 110310 95.7159 64.0625 0.0054499 0.0104382 2024-09-13T13:11:27.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:27 smithi167 ceph-mon[33178]: pgmap v503: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 90 MiB/s wr, 1.45k op/s 2024-09-13T13:11:27.855 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:27.889 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:27.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:27.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:27.991 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:28.028 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:28.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:28.130 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:28.165 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:28.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:28.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:28.267 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:28.305 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:28.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:28.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:28.407 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:28.475 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:28.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:28.576 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:28.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:28.610 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:28.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:28.710 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:28.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:28.734 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:28.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:28.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:28 smithi052 ceph-mon[36682]: from='client.15933 v1:172.21.15.52:0/3938641706' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:11:28.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:28 smithi052 ceph-mon[29471]: from='client.15933 v1:172.21.15.52:0/3938641706' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:11:28.802 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 73 16 111482 111466 95.3945 72.25 0.00750508 0.0104785 2024-09-13T13:11:28.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:28 smithi167 ceph-mon[33178]: from='client.15933 v1:172.21.15.52:0/3938641706' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:11:28.835 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:28.875 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:28.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:28.977 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:29.009 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:29.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:29.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:29.111 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:29.144 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:29.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:29.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:29.246 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:29.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:29.287 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:29.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:29.388 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:29.418 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:29.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:29.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:29.562 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:29.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:29.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:29.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:29.693 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:29.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:29.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:29 smithi052 ceph-mon[36682]: pgmap v504: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 574 B/s rd, 49 MiB/s wr, 790 op/s 2024-09-13T13:11:29.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:29 smithi052 ceph-mon[29471]: pgmap v504: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 574 B/s rd, 49 MiB/s wr, 790 op/s 2024-09-13T13:11:29.795 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:29.802 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 74 16 112502 112486 94.9669 63.75 0.0176288 0.0105254 2024-09-13T13:11:29.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:29 smithi167 ceph-mon[33178]: pgmap v504: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 574 B/s rd, 49 MiB/s wr, 790 op/s 2024-09-13T13:11:29.831 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:29.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:29.933 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:29.959 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:29.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:30.062 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:30.103 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:30.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:30.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:30.206 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:30.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:30.236 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:30.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:30.337 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:30.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:30.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:30.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:30.469 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:30.505 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:30.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:30.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:30.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:30.641 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:30.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:30.743 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:30.764 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-09-13T13:11:25.821123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6908746267,"num_objects":105609,"num_object_clones":0,"num_object_copies":316827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105609,"num_whiteouts":0,"num_read":1889,"num_read_kb":1746,"num_write":106245,"num_write_kb":6747913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83456,"ondisk_log_size":83456,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22820456,"kb_used_data":20097696,"kb_used_omap":15,"kb_used_meta":2722608,"kb_avail":727140760,"statfs":{"total":767960285184,"available":744592138240,"internally_reserved":0,"allocated":20580040704,"data_stored":20575333681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":2787951385},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,8,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":750714880,"num_objects":11455,"num_object_clones":0,"num_object_copies":34365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11455,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11455,"num_write_kb":733120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.924452"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":468,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.900301+0000","last_change":"2024-09-13T13:10:32.368663+0000","last_active":"2024-09-13T13:11:22.900301+0000","last_peered":"2024-09-13T13:11:22.900301+0000","last_clean":"2024-09-13T13:11:22.900301+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:11:22.900301+0000","last_undegraded":"2024-09-13T13:11:22.900301+0000","last_fullsized":"2024-09-13T13:11:22.900301+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.368588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.368588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:00.402096+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":502,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895347+0000","last_change":"2024-09-13T13:10:32.777554+0000","last_active":"2024-09-13T13:11:22.895347+0000","last_peered":"2024-09-13T13:11:22.895347+0000","last_clean":"2024-09-13T13:11:22.895347+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:11:22.895347+0000","last_undegraded":"2024-09-13T13:11:22.895347+0000","last_fullsized":"2024-09-13T13:11:22.895347+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:32.777451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.777451+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:00.660196+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903687+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:11:22.903687+0000","last_peered":"2024-09-13T13:11:22.903687+0000","last_clean":"2024-09-13T13:11:22.903687+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:11:22.903687+0000","last_undegraded":"2024-09-13T13:11:22.903687+0000","last_fullsized":"2024-09-13T13:11:22.903687+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.772899+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":450,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895548+0000","last_change":"2024-09-13T13:09:42.788562+0000","last_active":"2024-09-13T13:11:22.895548+0000","last_peered":"2024-09-13T13:11:22.895548+0000","last_clean":"2024-09-13T13:11:22.895548+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:11:22.895548+0000","last_undegraded":"2024-09-13T13:11:22.895548+0000","last_fullsized":"2024-09-13T13:11:22.895548+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:42.787492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:42.787492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:02.026348+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":468,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.908838+0000","last_change":"2024-09-13T13:08:55.504675+0000","last_active":"2024-09-13T13:11:22.908838+0000","last_peered":"2024-09-13T13:11:22.908838+0000","last_clean":"2024-09-13T13:11:22.908838+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:11:22.908838+0000","last_undegraded":"2024-09-13T13:11:22.908838+0000","last_fullsized":"2024-09-13T13:11:22.908838+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.504475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.504475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:45.483728+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":481,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.909196+0000","last_change":"2024-09-13T13:11:05.501305+0000","last_active":"2024-09-13T13:11:22.909196+0000","last_peered":"2024-09-13T13:11:22.909196+0000","last_clean":"2024-09-13T13:11:22.909196+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:11:22.909196+0000","last_undegraded":"2024-09-13T13:11:22.909196+0000","last_fullsized":"2024-09-13T13:11:22.909196+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:05.501234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:05.501234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.182488+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":524,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.909148+0000","last_change":"2024-09-13T13:10:39.460577+0000","last_active":"2024-09-13T13:11:22.909148+0000","last_peered":"2024-09-13T13:11:22.909148+0000","last_clean":"2024-09-13T13:11:22.909148+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:11:22.909148+0000","last_undegraded":"2024-09-13T13:11:22.909148+0000","last_fullsized":"2024-09-13T13:11:22.909148+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:10:39.460500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.460500+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:00.679480+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":481,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895267+0000","last_change":"2024-09-13T13:11:13.201530+0000","last_active":"2024-09-13T13:11:22.895267+0000","last_peered":"2024-09-13T13:11:22.895267+0000","last_clean":"2024-09-13T13:11:22.895267+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:11:22.895267+0000","last_undegraded":"2024-09-13T13:11:22.895267+0000","last_fullsized":"2024-09-13T13:11:22.895267+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:13.201484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:13.201484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.299188+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":502,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905952+0000","last_change":"2024-09-13T13:10:35.137237+0000","last_active":"2024-09-13T13:11:22.905952+0000","last_peered":"2024-09-13T13:11:22.905952+0000","last_clean":"2024-09-13T13:11:22.905952+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:11:22.905952+0000","last_undegraded":"2024-09-13T13:11:22.905952+0000","last_fullsized":"2024-09-13T13:11:22.905952+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:10:35.137160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:35.137160+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.584514+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":507,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895783+0000","last_change":"2024-09-13T13:10:31.738782+0000","last_active":"2024-09-13T13:11:22.895783+0000","last_peered":"2024-09-13T13:11:22.895783+0000","last_clean":"2024-09-13T13:11:22.895783+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:11:22.895783+0000","last_undegraded":"2024-09-13T13:11:22.895783+0000","last_fullsized":"2024-09-13T13:11:22.895783+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:31.738685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.738685+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:43.720157+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":461,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905325+0000","last_change":"2024-09-13T13:06:50.665256+0000","last_active":"2024-09-13T13:11:22.905325+0000","last_peered":"2024-09-13T13:11:22.905325+0000","last_clean":"2024-09-13T13:11:22.905325+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:11:22.905325+0000","last_undegraded":"2024-09-13T13:11:22.905325+0000","last_fullsized":"2024-09-13T13:11:22.905325+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:50.665191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.665191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:21.077433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":464,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898775+0000","last_change":"2024-09-13T13:10:43.363157+0000","last_active":"2024-09-13T13:11:22.898775+0000","last_peered":"2024-09-13T13:11:22.898775+0000","last_clean":"2024-09-13T13:11:22.898775+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:11:22.898775+0000","last_undegraded":"2024-09-13T13:11:22.898775+0000","last_fullsized":"2024-09-13T13:11:22.898775+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:43.362967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:43.362967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:11.841904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":485,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902158+0000","last_change":"2024-09-13T13:10:31.103576+0000","last_active":"2024-09-13T13:11:22.902158+0000","last_peered":"2024-09-13T13:11:22.902158+0000","last_clean":"2024-09-13T13:11:22.902158+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:11:22.902158+0000","last_undegraded":"2024-09-13T13:11:22.902158+0000","last_fullsized":"2024-09-13T13:11:22.902158+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:31.103509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.103509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:52.926445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":471,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895230+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:11:22.895230+0000","last_peered":"2024-09-13T13:11:22.895230+0000","last_clean":"2024-09-13T13:11:22.895230+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:11:22.895230+0000","last_undegraded":"2024-09-13T13:11:22.895230+0000","last_fullsized":"2024-09-13T13:11:22.895230+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.107577+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":469,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903809+0000","last_change":"2024-09-13T13:10:32.803006+0000","last_active":"2024-09-13T13:11:22.903809+0000","last_peered":"2024-09-13T13:11:22.903809+0000","last_clean":"2024-09-13T13:11:22.903809+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:11:22.903809+0000","last_undegraded":"2024-09-13T13:11:22.903809+0000","last_fullsized":"2024-09-13T13:11:22.903809+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.802931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.802931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:58.651202+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":441,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.901908+0000","last_change":"2024-09-13T13:07:19.641119+0000","last_active":"2024-09-13T13:11:22.901908+0000","last_peered":"2024-09-13T13:11:22.901908+0000","last_clean":"2024-09-13T13:11:22.901908+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:11:22.901908+0000","last_undegraded":"2024-09-13T13:11:22.901908+0000","last_fullsized":"2024-09-13T13:11:22.901908+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:19.640697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:19.640697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:57.084952+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":479,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898537+0000","last_change":"2024-09-13T13:10:28.805013+0000","last_active":"2024-09-13T13:11:22.898537+0000","last_peered":"2024-09-13T13:11:22.898537+0000","last_clean":"2024-09-13T13:11:22.898537+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:11:22.898537+0000","last_undegraded":"2024-09-13T13:11:22.898537+0000","last_fullsized":"2024-09-13T13:11:22.898537+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:28.804735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:28.804735+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.220025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":468,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.906754+0000","last_change":"2024-09-13T13:08:19.181017+0000","last_active":"2024-09-13T13:11:22.906754+0000","last_peered":"2024-09-13T13:11:22.906754+0000","last_clean":"2024-09-13T13:11:22.906754+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:11:22.906754+0000","last_undegraded":"2024-09-13T13:11:22.906754+0000","last_fullsized":"2024-09-13T13:11:22.906754+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:19.180948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.180948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:57.228082+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":517,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.923571+0000","last_change":"2024-09-13T13:06:44.532305+0000","last_active":"2024-09-13T13:11:22.923571+0000","last_peered":"2024-09-13T13:11:22.923571+0000","last_clean":"2024-09-13T13:11:22.923571+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:11:22.923571+0000","last_undegraded":"2024-09-13T13:11:22.923571+0000","last_fullsized":"2024-09-13T13:11:22.923571+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:06:44.532186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.532186+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":470,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.907103+0000","last_change":"2024-09-13T13:09:59.936868+0000","last_active":"2024-09-13T13:11:22.907103+0000","last_peered":"2024-09-13T13:11:22.907103+0000","last_clean":"2024-09-13T13:11:22.907103+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:11:22.907103+0000","last_undegraded":"2024-09-13T13:11:22.907103+0000","last_fullsized":"2024-09-13T13:11:22.907103+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:59.936811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:59.936811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":468,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.910436+0000","last_change":"2024-09-13T13:09:29.517189+0000","last_active":"2024-09-13T13:11:22.910436+0000","last_peered":"2024-09-13T13:11:22.910436+0000","last_clean":"2024-09-13T13:11:22.910436+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:11:22.910436+0000","last_undegraded":"2024-09-13T13:11:22.910436+0000","last_fullsized":"2024-09-13T13:11:22.910436+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:29.517034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:29.517034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:54.544083+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":478,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895839+0000","last_change":"2024-09-13T13:10:34.788637+0000","last_active":"2024-09-13T13:11:22.895839+0000","last_peered":"2024-09-13T13:11:22.895839+0000","last_clean":"2024-09-13T13:11:22.895839+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:11:22.895839+0000","last_undegraded":"2024-09-13T13:11:22.895839+0000","last_fullsized":"2024-09-13T13:11:22.895839+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:34.788562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:34.788562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:03.723901+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":497,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.909147+0000","last_change":"2024-09-13T13:11:14.593530+0000","last_active":"2024-09-13T13:11:22.909147+0000","last_peered":"2024-09-13T13:11:22.909147+0000","last_clean":"2024-09-13T13:11:22.909147+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:11:22.909147+0000","last_undegraded":"2024-09-13T13:11:22.909147+0000","last_fullsized":"2024-09-13T13:11:22.909147+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:11:14.593438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:14.593438+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:43.465667+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":462,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898090+0000","last_change":"2024-09-13T13:07:12.879813+0000","last_active":"2024-09-13T13:11:22.898090+0000","last_peered":"2024-09-13T13:11:22.898090+0000","last_clean":"2024-09-13T13:11:22.898090+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:11:22.898090+0000","last_undegraded":"2024-09-13T13:11:22.898090+0000","last_fullsized":"2024-09-13T13:11:22.898090+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.879727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.879727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:52.066816+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":474,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904636+0000","last_change":"2024-09-13T13:10:09.765203+0000","last_active":"2024-09-13T13:11:22.904636+0000","last_peered":"2024-09-13T13:11:22.904636+0000","last_clean":"2024-09-13T13:11:22.904636+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:11:22.904636+0000","last_undegraded":"2024-09-13T13:11:22.904636+0000","last_fullsized":"2024-09-13T13:11:22.904636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:09.765153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.765153+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:26.707398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":463,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903825+0000","last_change":"2024-09-13T13:07:54.442276+0000","last_active":"2024-09-13T13:11:22.903825+0000","last_peered":"2024-09-13T13:11:22.903825+0000","last_clean":"2024-09-13T13:11:22.903825+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:11:22.903825+0000","last_undegraded":"2024-09-13T13:11:22.903825+0000","last_fullsized":"2024-09-13T13:11:22.903825+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:07:54.439269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:54.439269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":503,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.899662+0000","last_change":"2024-09-13T13:09:28.434007+0000","last_active":"2024-09-13T13:11:22.899662+0000","last_peered":"2024-09-13T13:11:22.899662+0000","last_clean":"2024-09-13T13:11:22.899662+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:11:22.899662+0000","last_undegraded":"2024-09-13T13:11:22.899662+0000","last_fullsized":"2024-09-13T13:11:22.899662+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:28.433899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.433899+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:03.236560+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":548,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-13T13:11:18.924634+0000","last_change":"2024-09-13T13:10:42.010918+0000","last_active":"2024-09-13T13:11:18.924634+0000","last_peered":"2024-09-13T13:11:18.924634+0000","last_clean":"2024-09-13T13:11:18.924634+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:11:18.924634+0000","last_undegraded":"2024-09-13T13:11:18.924634+0000","last_fullsized":"2024-09-13T13:11:18.924634+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:10:42.010847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.010847+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.459829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"275'6634","reported_seq":6713,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.689911+0000","last_change":"2024-09-13T13:10:09.272561+0000","last_active":"2024-09-13T13:11:23.689911+0000","last_peered":"2024-09-13T13:11:23.689911+0000","last_clean":"2024-09-13T13:11:23.689911+0000","last_became_active":"2024-09-13T13:10:09.272264+0000","last_became_peered":"2024-09-13T13:10:09.272264+0000","last_unstale":"2024-09-13T13:11:23.689911+0000","last_undegraded":"2024-09-13T13:11:23.689911+0000","last_fullsized":"2024-09-13T13:11:23.689911+0000","mapping_epoch":241,"log_start":"249'1900","ondisk_log_start":"249'1900","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:35.365520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434765824,"num_objects":6634,"num_object_clones":0,"num_object_copies":19902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6634,"num_write_kb":424576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":509,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904261+0000","last_change":"2024-09-13T13:07:15.243958+0000","last_active":"2024-09-13T13:11:22.904261+0000","last_peered":"2024-09-13T13:11:22.904261+0000","last_clean":"2024-09-13T13:11:22.904261+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:11:22.904261+0000","last_undegraded":"2024-09-13T13:11:22.904261+0000","last_fullsized":"2024-09-13T13:11:22.904261+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:07:15.243715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:15.243715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.652407+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":485,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.911823+0000","last_change":"2024-09-13T13:10:30.552177+0000","last_active":"2024-09-13T13:11:22.911823+0000","last_peered":"2024-09-13T13:11:22.911823+0000","last_clean":"2024-09-13T13:11:22.911823+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:11:22.911823+0000","last_undegraded":"2024-09-13T13:11:22.911823+0000","last_fullsized":"2024-09-13T13:11:22.911823+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:30.552113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.552113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:48.898081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":603,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898631+0000","last_change":"2024-09-13T13:10:07.726810+0000","last_active":"2024-09-13T13:11:22.898631+0000","last_peered":"2024-09-13T13:11:22.898631+0000","last_clean":"2024-09-13T13:11:22.898631+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:11:22.898631+0000","last_undegraded":"2024-09-13T13:11:22.898631+0000","last_fullsized":"2024-09-13T13:11:22.898631+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:10:07.726751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:07.726751+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:32.746229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":465,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905066+0000","last_change":"2024-09-13T13:10:08.973968+0000","last_active":"2024-09-13T13:11:22.905066+0000","last_peered":"2024-09-13T13:11:22.905066+0000","last_clean":"2024-09-13T13:11:22.905066+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:11:22.905066+0000","last_undegraded":"2024-09-13T13:11:22.905066+0000","last_fullsized":"2024-09-13T13:11:22.905066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.973917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.973917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"275'6586","reported_seq":6665,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.084546+0000","last_change":"2024-09-13T13:10:09.273532+0000","last_active":"2024-09-13T13:11:23.084546+0000","last_peered":"2024-09-13T13:11:23.084546+0000","last_clean":"2024-09-13T13:11:23.084546+0000","last_became_active":"2024-09-13T13:10:09.273246+0000","last_became_peered":"2024-09-13T13:10:09.273246+0000","last_unstale":"2024-09-13T13:11:23.084546+0000","last_undegraded":"2024-09-13T13:11:23.084546+0000","last_fullsized":"2024-09-13T13:11:23.084546+0000","mapping_epoch":241,"log_start":"247'1700","ondisk_log_start":"247'1700","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.627323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431620096,"num_objects":6586,"num_object_clones":0,"num_object_copies":19758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6586,"num_write_kb":421504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":500,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.911543+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:11:22.911543+0000","last_peered":"2024-09-13T13:11:22.911543+0000","last_clean":"2024-09-13T13:11:22.911543+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:11:22.911543+0000","last_undegraded":"2024-09-13T13:11:22.911543+0000","last_fullsized":"2024-09-13T13:11:22.911543+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":467,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.910824+0000","last_change":"2024-09-13T13:06:44.785235+0000","last_active":"2024-09-13T13:11:22.910824+0000","last_peered":"2024-09-13T13:11:22.910824+0000","last_clean":"2024-09-13T13:11:22.910824+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:11:22.910824+0000","last_undegraded":"2024-09-13T13:11:22.910824+0000","last_fullsized":"2024-09-13T13:11:22.910824+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:06:44.785017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.785017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895864+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:11:22.895864+0000","last_peered":"2024-09-13T13:11:22.895864+0000","last_clean":"2024-09-13T13:11:22.895864+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:11:22.895864+0000","last_undegraded":"2024-09-13T13:11:22.895864+0000","last_fullsized":"2024-09-13T13:11:22.895864+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":474,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895710+0000","last_change":"2024-09-13T13:11:10.104997+0000","last_active":"2024-09-13T13:11:22.895710+0000","last_peered":"2024-09-13T13:11:22.895710+0000","last_clean":"2024-09-13T13:11:22.895710+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:11:22.895710+0000","last_undegraded":"2024-09-13T13:11:22.895710+0000","last_fullsized":"2024-09-13T13:11:22.895710+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:10.104926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:10.104926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:33.972960+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"275'6478","reported_seq":6557,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.666820+0000","last_change":"2024-09-13T13:10:09.270421+0000","last_active":"2024-09-13T13:11:23.666820+0000","last_peered":"2024-09-13T13:11:23.666820+0000","last_clean":"2024-09-13T13:11:23.666820+0000","last_became_active":"2024-09-13T13:10:09.270129+0000","last_became_peered":"2024-09-13T13:10:09.270129+0000","last_unstale":"2024-09-13T13:11:23.666820+0000","last_undegraded":"2024-09-13T13:11:23.666820+0000","last_fullsized":"2024-09-13T13:11:23.666820+0000","mapping_epoch":241,"log_start":"247'1700","ondisk_log_start":"247'1700","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:37.119106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424542208,"num_objects":6478,"num_object_clones":0,"num_object_copies":19434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6478,"num_write_kb":414592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":535,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895876+0000","last_change":"2024-09-13T13:11:17.167120+0000","last_active":"2024-09-13T13:11:22.895876+0000","last_peered":"2024-09-13T13:11:22.895876+0000","last_clean":"2024-09-13T13:11:22.895876+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:11:22.895876+0000","last_undegraded":"2024-09-13T13:11:22.895876+0000","last_fullsized":"2024-09-13T13:11:22.895876+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:11:17.167044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:17.167044+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.955826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":453,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.899902+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:11:22.899902+0000","last_peered":"2024-09-13T13:11:22.899902+0000","last_clean":"2024-09-13T13:11:22.899902+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:11:22.899902+0000","last_undegraded":"2024-09-13T13:11:22.899902+0000","last_fullsized":"2024-09-13T13:11:22.899902+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.301338+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":595,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.908637+0000","last_change":"2024-09-13T13:11:06.526723+0000","last_active":"2024-09-13T13:11:22.908637+0000","last_peered":"2024-09-13T13:11:22.908637+0000","last_clean":"2024-09-13T13:11:22.908637+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:11:22.908637+0000","last_undegraded":"2024-09-13T13:11:22.908637+0000","last_fullsized":"2024-09-13T13:11:22.908637+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:11:06.526646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:06.526646+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:18.710007+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":460,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904190+0000","last_change":"2024-09-13T13:06:09.230872+0000","last_active":"2024-09-13T13:11:22.904190+0000","last_peered":"2024-09-13T13:11:22.904190+0000","last_clean":"2024-09-13T13:11:22.904190+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:11:22.904190+0000","last_undegraded":"2024-09-13T13:11:22.904190+0000","last_fullsized":"2024-09-13T13:11:22.904190+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:09.230824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.230824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:45.998027+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"275'6534","reported_seq":6613,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.181722+0000","last_change":"2024-09-13T13:10:09.273698+0000","last_active":"2024-09-13T13:11:23.181722+0000","last_peered":"2024-09-13T13:11:23.181722+0000","last_clean":"2024-09-13T13:11:23.181722+0000","last_became_active":"2024-09-13T13:10:09.273311+0000","last_became_peered":"2024-09-13T13:10:09.273311+0000","last_unstale":"2024-09-13T13:11:23.181722+0000","last_undegraded":"2024-09-13T13:11:23.181722+0000","last_fullsized":"2024-09-13T13:11:23.181722+0000","mapping_epoch":241,"log_start":"246'1200","ondisk_log_start":"246'1200","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":5334,"log_dups_size":0,"ondisk_log_size":5334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428212224,"num_objects":6534,"num_object_clones":0,"num_object_copies":19602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6534,"num_write_kb":418176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":504,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.906428+0000","last_change":"2024-09-13T13:10:40.651563+0000","last_active":"2024-09-13T13:11:22.906428+0000","last_peered":"2024-09-13T13:11:22.906428+0000","last_clean":"2024-09-13T13:11:22.906428+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:11:22.906428+0000","last_undegraded":"2024-09-13T13:11:22.906428+0000","last_fullsized":"2024-09-13T13:11:22.906428+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:02.404183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":477,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.908865+0000","last_change":"2024-09-13T13:10:03.895450+0000","last_active":"2024-09-13T13:11:22.908865+0000","last_peered":"2024-09-13T13:11:22.908865+0000","last_clean":"2024-09-13T13:11:22.908865+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:11:22.908865+0000","last_undegraded":"2024-09-13T13:11:22.908865+0000","last_fullsized":"2024-09-13T13:11:22.908865+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:03.895396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:03.895396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:33.888709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":599,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.900283+0000","last_change":"2024-09-13T13:10:10.134973+0000","last_active":"2024-09-13T13:11:22.900283+0000","last_peered":"2024-09-13T13:11:22.900283+0000","last_clean":"2024-09-13T13:11:22.900283+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:11:22.900283+0000","last_undegraded":"2024-09-13T13:11:22.900283+0000","last_fullsized":"2024-09-13T13:11:22.900283+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:10:10.134873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.134873+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":473,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898991+0000","last_change":"2024-09-13T13:10:42.390672+0000","last_active":"2024-09-13T13:11:22.898991+0000","last_peered":"2024-09-13T13:11:22.898991+0000","last_clean":"2024-09-13T13:11:22.898991+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:11:22.898991+0000","last_undegraded":"2024-09-13T13:11:22.898991+0000","last_fullsized":"2024-09-13T13:11:22.898991+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:42.390473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.390473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:43.033158+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"275'6487","reported_seq":6566,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.081482+0000","last_change":"2024-09-13T13:10:09.270982+0000","last_active":"2024-09-13T13:11:23.081482+0000","last_peered":"2024-09-13T13:11:23.081482+0000","last_clean":"2024-09-13T13:11:23.081482+0000","last_became_active":"2024-09-13T13:10:09.270486+0000","last_became_peered":"2024-09-13T13:10:09.270486+0000","last_unstale":"2024-09-13T13:11:23.081482+0000","last_undegraded":"2024-09-13T13:11:23.081482+0000","last_fullsized":"2024-09-13T13:11:23.081482+0000","mapping_epoch":241,"log_start":"247'1600","ondisk_log_start":"247'1600","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:31.811638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425132032,"num_objects":6487,"num_object_clones":0,"num_object_copies":19461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":415168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":531,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.906234+0000","last_change":"2024-09-13T13:11:18.533559+0000","last_active":"2024-09-13T13:11:22.906234+0000","last_peered":"2024-09-13T13:11:22.906234+0000","last_clean":"2024-09-13T13:11:22.906234+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:11:22.906234+0000","last_undegraded":"2024-09-13T13:11:22.906234+0000","last_fullsized":"2024-09-13T13:11:22.906234+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:11:18.533452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:18.533452+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:47.552700+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905718+0000","last_change":"2024-09-13T13:09:38.867734+0000","last_active":"2024-09-13T13:11:22.905718+0000","last_peered":"2024-09-13T13:11:22.905718+0000","last_clean":"2024-09-13T13:11:22.905718+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:11:22.905718+0000","last_undegraded":"2024-09-13T13:11:22.905718+0000","last_fullsized":"2024-09-13T13:11:22.905718+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.867688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.867688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:57.738458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":420,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-13T13:11:18.923883+0000","last_change":"2024-09-13T13:09:38.854654+0000","last_active":"2024-09-13T13:11:18.923883+0000","last_peered":"2024-09-13T13:11:18.923883+0000","last_clean":"2024-09-13T13:11:18.923883+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:11:18.923883+0000","last_undegraded":"2024-09-13T13:11:18.923883+0000","last_fullsized":"2024-09-13T13:11:18.923883+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.854581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.854581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:54.127592+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":469,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903597+0000","last_change":"2024-09-13T13:06:43.969251+0000","last_active":"2024-09-13T13:11:22.903597+0000","last_peered":"2024-09-13T13:11:22.903597+0000","last_clean":"2024-09-13T13:11:22.903597+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:11:22.903597+0000","last_undegraded":"2024-09-13T13:11:22.903597+0000","last_fullsized":"2024-09-13T13:11:22.903597+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.969176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.969176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"275'6776","reported_seq":6855,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:24.004992+0000","last_change":"2024-09-13T13:10:09.277094+0000","last_active":"2024-09-13T13:11:24.004992+0000","last_peered":"2024-09-13T13:11:24.004992+0000","last_clean":"2024-09-13T13:11:24.004992+0000","last_became_active":"2024-09-13T13:10:09.275394+0000","last_became_peered":"2024-09-13T13:10:09.275394+0000","last_unstale":"2024-09-13T13:11:24.004992+0000","last_undegraded":"2024-09-13T13:11:24.004992+0000","last_fullsized":"2024-09-13T13:11:24.004992+0000","mapping_epoch":241,"log_start":"246'700","ondisk_log_start":"246'700","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444071936,"num_objects":6776,"num_object_clones":0,"num_object_copies":20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6776,"num_write_kb":433664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"268'40","reported_seq":657,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905572+0000","last_change":"2024-09-13T13:07:25.438610+0000","last_active":"2024-09-13T13:11:22.905572+0000","last_peered":"2024-09-13T13:11:22.905572+0000","last_clean":"2024-09-13T13:11:22.905572+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:11:22.905572+0000","last_undegraded":"2024-09-13T13:11:22.905572+0000","last_fullsized":"2024-09-13T13:11:22.905572+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:07:25.438528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:25.438528+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:54.678903+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":481,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.896228+0000","last_change":"2024-09-13T13:07:18.888103+0000","last_active":"2024-09-13T13:11:22.896228+0000","last_peered":"2024-09-13T13:11:22.896228+0000","last_clean":"2024-09-13T13:11:22.896228+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:11:22.896228+0000","last_undegraded":"2024-09-13T13:11:22.896228+0000","last_fullsized":"2024-09-13T13:11:22.896228+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:18.888060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:18.888060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:56.046611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":472,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902253+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:11:22.902253+0000","last_peered":"2024-09-13T13:11:22.902253+0000","last_clean":"2024-09-13T13:11:22.902253+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:11:22.902253+0000","last_undegraded":"2024-09-13T13:11:22.902253+0000","last_fullsized":"2024-09-13T13:11:22.902253+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:54.110286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":468,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902455+0000","last_change":"2024-09-13T13:11:11.413254+0000","last_active":"2024-09-13T13:11:22.902455+0000","last_peered":"2024-09-13T13:11:22.902455+0000","last_clean":"2024-09-13T13:11:22.902455+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:11:22.902455+0000","last_undegraded":"2024-09-13T13:11:22.902455+0000","last_fullsized":"2024-09-13T13:11:22.902455+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:11.413050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.413050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:12.384938+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":465,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.908964+0000","last_change":"2024-09-13T13:10:42.489307+0000","last_active":"2024-09-13T13:11:22.908964+0000","last_peered":"2024-09-13T13:11:22.908964+0000","last_clean":"2024-09-13T13:11:22.908964+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:11:22.908964+0000","last_undegraded":"2024-09-13T13:11:22.908964+0000","last_fullsized":"2024-09-13T13:11:22.908964+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:42.489264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.489264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:02.930639+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"275'6571","reported_seq":6650,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.173934+0000","last_change":"2024-09-13T13:10:09.270343+0000","last_active":"2024-09-13T13:11:23.173934+0000","last_peered":"2024-09-13T13:11:23.173934+0000","last_clean":"2024-09-13T13:11:23.173934+0000","last_became_active":"2024-09-13T13:10:09.270073+0000","last_became_peered":"2024-09-13T13:10:09.270073+0000","last_unstale":"2024-09-13T13:11:23.173934+0000","last_undegraded":"2024-09-13T13:11:23.173934+0000","last_fullsized":"2024-09-13T13:11:23.173934+0000","mapping_epoch":241,"log_start":"246'1300","ondisk_log_start":"246'1300","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:35.580323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430637056,"num_objects":6571,"num_object_clones":0,"num_object_copies":19713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6571,"num_write_kb":420544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":499,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904356+0000","last_change":"2024-09-13T13:10:29.799315+0000","last_active":"2024-09-13T13:11:22.904356+0000","last_peered":"2024-09-13T13:11:22.904356+0000","last_clean":"2024-09-13T13:11:22.904356+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:11:22.904356+0000","last_undegraded":"2024-09-13T13:11:22.904356+0000","last_fullsized":"2024-09-13T13:11:22.904356+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:10:29.799185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:29.799185+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:36.850709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":490,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903959+0000","last_change":"2024-09-13T13:08:56.771534+0000","last_active":"2024-09-13T13:11:22.903959+0000","last_peered":"2024-09-13T13:11:22.903959+0000","last_clean":"2024-09-13T13:11:22.903959+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:11:22.903959+0000","last_undegraded":"2024-09-13T13:11:22.903959+0000","last_fullsized":"2024-09-13T13:11:22.903959+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:56.771432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:56.771432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":493,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.922615+0000","last_change":"2024-09-13T13:10:30.919483+0000","last_active":"2024-09-13T13:11:22.922615+0000","last_peered":"2024-09-13T13:11:22.922615+0000","last_clean":"2024-09-13T13:11:22.922615+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:11:22.922615+0000","last_undegraded":"2024-09-13T13:11:22.922615+0000","last_fullsized":"2024-09-13T13:11:22.922615+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:10:30.919398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.919398+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:56.559722+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":457,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.925049+0000","last_change":"2024-09-13T13:11:18.610942+0000","last_active":"2024-09-13T13:11:22.925049+0000","last_peered":"2024-09-13T13:11:22.925049+0000","last_clean":"2024-09-13T13:11:22.925049+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:11:22.925049+0000","last_undegraded":"2024-09-13T13:11:22.925049+0000","last_fullsized":"2024-09-13T13:11:22.925049+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:18.610714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:18.610714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:40.534665+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":468,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904884+0000","last_change":"2024-09-13T13:08:54.232191+0000","last_active":"2024-09-13T13:11:22.904884+0000","last_peered":"2024-09-13T13:11:22.904884+0000","last_clean":"2024-09-13T13:11:22.904884+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:11:22.904884+0000","last_undegraded":"2024-09-13T13:11:22.904884+0000","last_fullsized":"2024-09-13T13:11:22.904884+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:54.232130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:54.232130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:58.884996+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"275'6670","reported_seq":6749,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.990813+0000","last_change":"2024-09-13T13:10:09.737716+0000","last_active":"2024-09-13T13:11:23.990813+0000","last_peered":"2024-09-13T13:11:23.990813+0000","last_clean":"2024-09-13T13:11:23.990813+0000","last_became_active":"2024-09-13T13:10:09.737415+0000","last_became_peered":"2024-09-13T13:10:09.737415+0000","last_unstale":"2024-09-13T13:11:23.990813+0000","last_undegraded":"2024-09-13T13:11:23.990813+0000","last_fullsized":"2024-09-13T13:11:23.990813+0000","mapping_epoch":241,"log_start":"246'600","ondisk_log_start":"246'600","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":6070,"log_dups_size":0,"ondisk_log_size":6070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437125120,"num_objects":6670,"num_object_clones":0,"num_object_copies":20010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6670,"num_write_kb":426880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":500,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.899744+0000","last_change":"2024-09-13T13:08:21.219398+0000","last_active":"2024-09-13T13:11:22.899744+0000","last_peered":"2024-09-13T13:11:22.899744+0000","last_clean":"2024-09-13T13:11:22.899744+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:11:22.899744+0000","last_undegraded":"2024-09-13T13:11:22.899744+0000","last_fullsized":"2024-09-13T13:11:22.899744+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:08:21.219300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:21.219300+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:55.256946+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":476,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.899103+0000","last_change":"2024-09-13T13:10:37.465330+0000","last_active":"2024-09-13T13:11:22.899103+0000","last_peered":"2024-09-13T13:11:22.899103+0000","last_clean":"2024-09-13T13:11:22.899103+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:11:22.899103+0000","last_undegraded":"2024-09-13T13:11:22.899103+0000","last_fullsized":"2024-09-13T13:11:22.899103+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:37.465261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:37.465261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:56.060539+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1093,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903424+0000","last_change":"2024-09-13T13:08:49.264831+0000","last_active":"2024-09-13T13:11:22.903424+0000","last_peered":"2024-09-13T13:11:22.903424+0000","last_clean":"2024-09-13T13:11:22.903424+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:11:22.903424+0000","last_undegraded":"2024-09-13T13:11:22.903424+0000","last_fullsized":"2024-09-13T13:11:22.903424+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:08:49.264775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:49.264775+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:56.918802+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":614,"num_read_kb":609,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":445,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903345+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:11:22.903345+0000","last_peered":"2024-09-13T13:11:22.903345+0000","last_clean":"2024-09-13T13:11:22.903345+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:11:22.903345+0000","last_undegraded":"2024-09-13T13:11:22.903345+0000","last_fullsized":"2024-09-13T13:11:22.903345+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:48.121624+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":474,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904958+0000","last_change":"2024-09-13T13:10:32.720742+0000","last_active":"2024-09-13T13:11:22.904958+0000","last_peered":"2024-09-13T13:11:22.904958+0000","last_clean":"2024-09-13T13:11:22.904958+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:11:22.904958+0000","last_undegraded":"2024-09-13T13:11:22.904958+0000","last_fullsized":"2024-09-13T13:11:22.904958+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:41.718820+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"275'6619","reported_seq":6698,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.654241+0000","last_change":"2024-09-13T13:10:09.276701+0000","last_active":"2024-09-13T13:11:23.654241+0000","last_peered":"2024-09-13T13:11:23.654241+0000","last_clean":"2024-09-13T13:11:23.654241+0000","last_became_active":"2024-09-13T13:10:09.276418+0000","last_became_peered":"2024-09-13T13:10:09.276418+0000","last_unstale":"2024-09-13T13:11:23.654241+0000","last_undegraded":"2024-09-13T13:11:23.654241+0000","last_fullsized":"2024-09-13T13:11:23.654241+0000","mapping_epoch":241,"log_start":"248'1900","ondisk_log_start":"248'1900","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:26.865438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433782784,"num_objects":6619,"num_object_clones":0,"num_object_copies":19857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6619,"num_write_kb":423616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"268'30","reported_seq":486,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-13T13:11:18.923983+0000","last_change":"2024-09-13T13:06:51.193749+0000","last_active":"2024-09-13T13:11:18.923983+0000","last_peered":"2024-09-13T13:11:18.923983+0000","last_clean":"2024-09-13T13:11:18.923983+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:11:18.923983+0000","last_undegraded":"2024-09-13T13:11:18.923983+0000","last_fullsized":"2024-09-13T13:11:18.923983+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'18","last_scrub_stamp":"2024-09-13T13:06:51.193678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.193678+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":472,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.909691+0000","last_change":"2024-09-13T13:06:10.715371+0000","last_active":"2024-09-13T13:11:22.909691+0000","last_peered":"2024-09-13T13:11:22.909691+0000","last_clean":"2024-09-13T13:11:22.909691+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:11:22.909691+0000","last_undegraded":"2024-09-13T13:11:22.909691+0000","last_fullsized":"2024-09-13T13:11:22.909691+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:10.715313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.715313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.480796+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":518,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903607+0000","last_change":"2024-09-13T13:06:39.564599+0000","last_active":"2024-09-13T13:11:22.903607+0000","last_peered":"2024-09-13T13:11:22.903607+0000","last_clean":"2024-09-13T13:11:22.903607+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:11:22.903607+0000","last_undegraded":"2024-09-13T13:11:22.903607+0000","last_fullsized":"2024-09-13T13:11:22.903607+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:06:39.564549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:06:39.564549+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":469,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904789+0000","last_change":"2024-09-13T13:08:25.500731+0000","last_active":"2024-09-13T13:11:22.904789+0000","last_peered":"2024-09-13T13:11:22.904789+0000","last_clean":"2024-09-13T13:11:22.904789+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:11:22.904789+0000","last_undegraded":"2024-09-13T13:11:22.904789+0000","last_fullsized":"2024-09-13T13:11:22.904789+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:25.500660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:25.500660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.962133+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":473,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903218+0000","last_change":"2024-09-13T13:10:09.926369+0000","last_active":"2024-09-13T13:11:22.903218+0000","last_peered":"2024-09-13T13:11:22.903218+0000","last_clean":"2024-09-13T13:11:22.903218+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:11:22.903218+0000","last_undegraded":"2024-09-13T13:11:22.903218+0000","last_fullsized":"2024-09-13T13:11:22.903218+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:09.926319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.926319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:39.338826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"275'6612","reported_seq":6761,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.077083+0000","last_change":"2024-09-13T13:11:16.781686+0000","last_active":"2024-09-13T13:11:23.077083+0000","last_peered":"2024-09-13T13:11:23.077083+0000","last_clean":"2024-09-13T13:11:23.077083+0000","last_became_active":"2024-09-13T13:10:09.273226+0000","last_became_peered":"2024-09-13T13:10:09.273226+0000","last_unstale":"2024-09-13T13:11:23.077083+0000","last_undegraded":"2024-09-13T13:11:23.077083+0000","last_fullsized":"2024-09-13T13:11:23.077083+0000","mapping_epoch":241,"log_start":"247'1700","ondisk_log_start":"247'1700","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'6147","last_scrub_stamp":"2024-09-13T13:11:16.781526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:11:16.781526+0000","objects_scrubbed":6090,"log_size":4912,"log_dups_size":0,"ondisk_log_size":4912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:20.503549+0000","scrub_duration":1943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433324032,"num_objects":6612,"num_object_clones":0,"num_object_copies":19836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6612,"num_write_kb":423168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":508,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898671+0000","last_change":"2024-09-13T13:10:30.736738+0000","last_active":"2024-09-13T13:11:22.898671+0000","last_peered":"2024-09-13T13:11:22.898671+0000","last_clean":"2024-09-13T13:11:22.898671+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:11:22.898671+0000","last_undegraded":"2024-09-13T13:11:22.898671+0000","last_fullsized":"2024-09-13T13:11:22.898671+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:10:30.736543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.736543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.681138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":478,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.896091+0000","last_change":"2024-09-13T13:10:01.834881+0000","last_active":"2024-09-13T13:11:22.896091+0000","last_peered":"2024-09-13T13:11:22.896091+0000","last_clean":"2024-09-13T13:11:22.896091+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:11:22.896091+0000","last_undegraded":"2024-09-13T13:11:22.896091+0000","last_fullsized":"2024-09-13T13:11:22.896091+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:01.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:01.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.226771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":488,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902993+0000","last_change":"2024-09-13T13:11:13.010250+0000","last_active":"2024-09-13T13:11:22.902993+0000","last_peered":"2024-09-13T13:11:22.902993+0000","last_clean":"2024-09-13T13:11:22.902993+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:11:22.902993+0000","last_undegraded":"2024-09-13T13:11:22.902993+0000","last_fullsized":"2024-09-13T13:11:22.902993+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:13.010123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:13.010123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:33.715146+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":468,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.909894+0000","last_change":"2024-09-13T13:10:34.526736+0000","last_active":"2024-09-13T13:11:22.909894+0000","last_peered":"2024-09-13T13:11:22.909894+0000","last_clean":"2024-09-13T13:11:22.909894+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:11:22.909894+0000","last_undegraded":"2024-09-13T13:11:22.909894+0000","last_fullsized":"2024-09-13T13:11:22.909894+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:34.526666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:34.526666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.167713+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"275'6602","reported_seq":6681,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.607140+0000","last_change":"2024-09-13T13:10:09.737339+0000","last_active":"2024-09-13T13:11:23.607140+0000","last_peered":"2024-09-13T13:11:23.607140+0000","last_clean":"2024-09-13T13:11:23.607140+0000","last_became_active":"2024-09-13T13:10:09.736988+0000","last_became_peered":"2024-09-13T13:10:09.736988+0000","last_unstale":"2024-09-13T13:11:23.607140+0000","last_undegraded":"2024-09-13T13:11:23.607140+0000","last_fullsized":"2024-09-13T13:11:23.607140+0000","mapping_epoch":241,"log_start":"247'1800","ondisk_log_start":"247'1800","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4802,"log_dups_size":0,"ondisk_log_size":4802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432668672,"num_objects":6602,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6602,"num_write_kb":422528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":507,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898323+0000","last_change":"2024-09-13T13:05:34.166464+0000","last_active":"2024-09-13T13:11:22.898323+0000","last_peered":"2024-09-13T13:11:22.898323+0000","last_clean":"2024-09-13T13:11:22.898323+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:11:22.898323+0000","last_undegraded":"2024-09-13T13:11:22.898323+0000","last_fullsized":"2024-09-13T13:11:22.898323+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:22.122358+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":452,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-13T13:11:18.924143+0000","last_change":"2024-09-13T13:10:10.787443+0000","last_active":"2024-09-13T13:11:18.924143+0000","last_peered":"2024-09-13T13:11:18.924143+0000","last_clean":"2024-09-13T13:11:18.924143+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:11:18.924143+0000","last_undegraded":"2024-09-13T13:11:18.924143+0000","last_fullsized":"2024-09-13T13:11:18.924143+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:10.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:39.817372+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905053+0000","last_change":"2024-09-13T13:08:53.747370+0000","last_active":"2024-09-13T13:11:22.905053+0000","last_peered":"2024-09-13T13:11:22.905053+0000","last_clean":"2024-09-13T13:11:22.905053+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:11:22.905053+0000","last_undegraded":"2024-09-13T13:11:22.905053+0000","last_fullsized":"2024-09-13T13:11:22.905053+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.747302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.747302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":458,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.923825+0000","last_change":"2024-09-13T13:08:03.015302+0000","last_active":"2024-09-13T13:11:22.923825+0000","last_peered":"2024-09-13T13:11:22.923825+0000","last_clean":"2024-09-13T13:11:22.923825+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:11:22.923825+0000","last_undegraded":"2024-09-13T13:11:22.923825+0000","last_fullsized":"2024-09-13T13:11:22.923825+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:03.015233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.015233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:26.850341+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"275'6413","reported_seq":6492,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.678054+0000","last_change":"2024-09-13T13:10:09.737741+0000","last_active":"2024-09-13T13:11:23.678054+0000","last_peered":"2024-09-13T13:11:23.678054+0000","last_clean":"2024-09-13T13:11:23.678054+0000","last_became_active":"2024-09-13T13:10:09.737415+0000","last_became_peered":"2024-09-13T13:10:09.737415+0000","last_unstale":"2024-09-13T13:11:23.678054+0000","last_undegraded":"2024-09-13T13:11:23.678054+0000","last_fullsized":"2024-09-13T13:11:23.678054+0000","mapping_epoch":241,"log_start":"247'1700","ondisk_log_start":"247'1700","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4713,"log_dups_size":0,"ondisk_log_size":4713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:35.194931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420282368,"num_objects":6413,"num_object_clones":0,"num_object_copies":19239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6413,"num_write_kb":410432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":523,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898171+0000","last_change":"2024-09-13T13:10:09.739258+0000","last_active":"2024-09-13T13:11:22.898171+0000","last_peered":"2024-09-13T13:11:22.898171+0000","last_clean":"2024-09-13T13:11:22.898171+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:11:22.898171+0000","last_undegraded":"2024-09-13T13:11:22.898171+0000","last_fullsized":"2024-09-13T13:11:22.898171+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:09.739184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.739184+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.869202+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":461,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.909918+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:11:22.909918+0000","last_peered":"2024-09-13T13:11:22.909918+0000","last_clean":"2024-09-13T13:11:22.909918+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:11:22.909918+0000","last_undegraded":"2024-09-13T13:11:22.909918+0000","last_fullsized":"2024-09-13T13:11:22.909918+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":587,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.923731+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:11:22.923731+0000","last_peered":"2024-09-13T13:11:22.923731+0000","last_clean":"2024-09-13T13:11:22.923731+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:11:22.923731+0000","last_undegraded":"2024-09-13T13:11:22.923731+0000","last_fullsized":"2024-09-13T13:11:22.923731+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":458,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898564+0000","last_change":"2024-09-13T13:11:05.084909+0000","last_active":"2024-09-13T13:11:22.898564+0000","last_peered":"2024-09-13T13:11:22.898564+0000","last_clean":"2024-09-13T13:11:22.898564+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:11:22.898564+0000","last_undegraded":"2024-09-13T13:11:22.898564+0000","last_fullsized":"2024-09-13T13:11:22.898564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:05.084772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:05.084772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.857054+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"275'6605","reported_seq":6749,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.177744+0000","last_change":"2024-09-13T13:11:11.072082+0000","last_active":"2024-09-13T13:11:23.177744+0000","last_peered":"2024-09-13T13:11:23.177744+0000","last_clean":"2024-09-13T13:11:23.177744+0000","last_became_active":"2024-09-13T13:10:09.274589+0000","last_became_peered":"2024-09-13T13:10:09.274589+0000","last_unstale":"2024-09-13T13:11:23.177744+0000","last_undegraded":"2024-09-13T13:11:23.177744+0000","last_fullsized":"2024-09-13T13:11:23.177744+0000","mapping_epoch":241,"log_start":"246'1300","ondisk_log_start":"246'1300","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'5724","last_scrub_stamp":"2024-09-13T13:11:11.071957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.071957+0000","objects_scrubbed":5659,"log_size":5305,"log_dups_size":0,"ondisk_log_size":5305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:14.421303+0000","scrub_duration":1921,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432865280,"num_objects":6605,"num_object_clones":0,"num_object_copies":19815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6605,"num_write_kb":422720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":473,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.906137+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:11:22.906137+0000","last_peered":"2024-09-13T13:11:22.906137+0000","last_clean":"2024-09-13T13:11:22.906137+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:11:22.906137+0000","last_undegraded":"2024-09-13T13:11:22.906137+0000","last_fullsized":"2024-09-13T13:11:22.906137+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:20.576847+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":496,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902294+0000","last_change":"2024-09-13T13:10:41.028120+0000","last_active":"2024-09-13T13:11:22.902294+0000","last_peered":"2024-09-13T13:11:22.902294+0000","last_clean":"2024-09-13T13:11:22.902294+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:11:22.902294+0000","last_undegraded":"2024-09-13T13:11:22.902294+0000","last_fullsized":"2024-09-13T13:11:22.902294+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:41.028047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:41.028047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:06.786834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":470,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.922848+0000","last_change":"2024-09-13T13:07:11.400066+0000","last_active":"2024-09-13T13:11:22.922848+0000","last_peered":"2024-09-13T13:11:22.922848+0000","last_clean":"2024-09-13T13:11:22.922848+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:11:22.922848+0000","last_undegraded":"2024-09-13T13:11:22.922848+0000","last_fullsized":"2024-09-13T13:11:22.922848+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.399840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.399840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:45.908383+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":456,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.899006+0000","last_change":"2024-09-13T13:11:10.432159+0000","last_active":"2024-09-13T13:11:22.899006+0000","last_peered":"2024-09-13T13:11:22.899006+0000","last_clean":"2024-09-13T13:11:22.899006+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:11:22.899006+0000","last_undegraded":"2024-09-13T13:11:22.899006+0000","last_fullsized":"2024-09-13T13:11:22.899006+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:10.432086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:10.432086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:13.775919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"275'6590","reported_seq":6669,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.182628+0000","last_change":"2024-09-13T13:10:09.271210+0000","last_active":"2024-09-13T13:11:23.182628+0000","last_peered":"2024-09-13T13:11:23.182628+0000","last_clean":"2024-09-13T13:11:23.182628+0000","last_became_active":"2024-09-13T13:10:09.270909+0000","last_became_peered":"2024-09-13T13:10:09.270909+0000","last_unstale":"2024-09-13T13:11:23.182628+0000","last_undegraded":"2024-09-13T13:11:23.182628+0000","last_fullsized":"2024-09-13T13:11:23.182628+0000","mapping_epoch":241,"log_start":"246'1300","ondisk_log_start":"246'1300","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":5290,"log_dups_size":0,"ondisk_log_size":5290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431882240,"num_objects":6590,"num_object_clones":0,"num_object_copies":19770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6590,"num_write_kb":421760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":479,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-13T13:11:18.923968+0000","last_change":"2024-09-13T13:09:32.849360+0000","last_active":"2024-09-13T13:11:18.923968+0000","last_peered":"2024-09-13T13:11:18.923968+0000","last_clean":"2024-09-13T13:11:18.923968+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:11:18.923968+0000","last_undegraded":"2024-09-13T13:11:18.923968+0000","last_fullsized":"2024-09-13T13:11:18.923968+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:09:32.849277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:32.849277+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:01.179140+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":488,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905694+0000","last_change":"2024-09-13T13:10:27.802685+0000","last_active":"2024-09-13T13:11:22.905694+0000","last_peered":"2024-09-13T13:11:22.905694+0000","last_clean":"2024-09-13T13:11:22.905694+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:11:22.905694+0000","last_undegraded":"2024-09-13T13:11:22.905694+0000","last_fullsized":"2024-09-13T13:11:22.905694+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:27.802621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:27.802621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:40.450332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":470,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.923104+0000","last_change":"2024-09-13T13:08:26.033729+0000","last_active":"2024-09-13T13:11:22.923104+0000","last_peered":"2024-09-13T13:11:22.923104+0000","last_clean":"2024-09-13T13:11:22.923104+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:11:22.923104+0000","last_undegraded":"2024-09-13T13:11:22.923104+0000","last_fullsized":"2024-09-13T13:11:22.923104+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:26.033663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:26.033663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:55.961768+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":465,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903520+0000","last_change":"2024-09-13T13:10:40.961803+0000","last_active":"2024-09-13T13:11:22.903520+0000","last_peered":"2024-09-13T13:11:22.903520+0000","last_clean":"2024-09-13T13:11:22.903520+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:11:22.903520+0000","last_undegraded":"2024-09-13T13:11:22.903520+0000","last_fullsized":"2024-09-13T13:11:22.903520+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:40.961744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.961744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:59.893697+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"275'6609","reported_seq":6755,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.567154+0000","last_change":"2024-09-13T13:11:15.826994+0000","last_active":"2024-09-13T13:11:23.567154+0000","last_peered":"2024-09-13T13:11:23.567154+0000","last_clean":"2024-09-13T13:11:23.567154+0000","last_became_active":"2024-09-13T13:10:09.270190+0000","last_became_peered":"2024-09-13T13:10:09.270190+0000","last_unstale":"2024-09-13T13:11:23.567154+0000","last_undegraded":"2024-09-13T13:11:23.567154+0000","last_fullsized":"2024-09-13T13:11:23.567154+0000","mapping_epoch":241,"log_start":"247'1800","ondisk_log_start":"247'1800","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'6035","last_scrub_stamp":"2024-09-13T13:11:15.826744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:11:15.826744+0000","objects_scrubbed":5970,"log_size":4809,"log_dups_size":0,"ondisk_log_size":4809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:31.364522+0000","scrub_duration":1936,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433127424,"num_objects":6609,"num_object_clones":0,"num_object_copies":19827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6609,"num_write_kb":422976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":490,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895996+0000","last_change":"2024-09-13T13:09:30.614842+0000","last_active":"2024-09-13T13:11:22.895996+0000","last_peered":"2024-09-13T13:11:22.895996+0000","last_clean":"2024-09-13T13:11:22.895996+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:11:22.895996+0000","last_undegraded":"2024-09-13T13:11:22.895996+0000","last_fullsized":"2024-09-13T13:11:22.895996+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:09:30.614748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.614748+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:03.028024+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":477,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905490+0000","last_change":"2024-09-13T13:11:18.821936+0000","last_active":"2024-09-13T13:11:22.905490+0000","last_peered":"2024-09-13T13:11:22.905490+0000","last_clean":"2024-09-13T13:11:22.905490+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:11:22.905490+0000","last_undegraded":"2024-09-13T13:11:22.905490+0000","last_fullsized":"2024-09-13T13:11:22.905490+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:18.821868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:18.821868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:40.874513+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.923042+0000","last_change":"2024-09-13T13:06:42.450669+0000","last_active":"2024-09-13T13:11:22.923042+0000","last_peered":"2024-09-13T13:11:22.923042+0000","last_clean":"2024-09-13T13:11:22.923042+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:11:22.923042+0000","last_undegraded":"2024-09-13T13:11:22.923042+0000","last_fullsized":"2024-09-13T13:11:22.923042+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:42.450598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:42.450598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":449,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.909071+0000","last_change":"2024-09-13T13:11:17.667223+0000","last_active":"2024-09-13T13:11:22.909071+0000","last_peered":"2024-09-13T13:11:22.909071+0000","last_clean":"2024-09-13T13:11:22.909071+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:11:22.909071+0000","last_undegraded":"2024-09-13T13:11:22.909071+0000","last_fullsized":"2024-09-13T13:11:22.909071+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:17.667151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:17.667151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:46.207860+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"275'6624","reported_seq":6703,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:23.960614+0000","last_change":"2024-09-13T13:10:09.275649+0000","last_active":"2024-09-13T13:11:23.960614+0000","last_peered":"2024-09-13T13:11:23.960614+0000","last_clean":"2024-09-13T13:11:23.960614+0000","last_became_active":"2024-09-13T13:10:09.275320+0000","last_became_peered":"2024-09-13T13:10:09.275320+0000","last_unstale":"2024-09-13T13:11:23.960614+0000","last_undegraded":"2024-09-13T13:11:23.960614+0000","last_fullsized":"2024-09-13T13:11:23.960614+0000","mapping_epoch":241,"log_start":"246'600","ondisk_log_start":"246'600","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":6024,"log_dups_size":0,"ondisk_log_size":6024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:21.621303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434110464,"num_objects":6624,"num_object_clones":0,"num_object_copies":19872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6624,"num_write_kb":423936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":519,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902945+0000","last_change":"2024-09-13T13:09:33.843654+0000","last_active":"2024-09-13T13:11:22.902945+0000","last_peered":"2024-09-13T13:11:22.902945+0000","last_clean":"2024-09-13T13:11:22.902945+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:11:22.902945+0000","last_undegraded":"2024-09-13T13:11:22.902945+0000","last_fullsized":"2024-09-13T13:11:22.902945+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:09:33.843579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:33.843579+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:46.077623+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":481,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905331+0000","last_change":"2024-09-13T13:06:51.627764+0000","last_active":"2024-09-13T13:11:22.905331+0000","last_peered":"2024-09-13T13:11:22.905331+0000","last_clean":"2024-09-13T13:11:22.905331+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:11:22.905331+0000","last_undegraded":"2024-09-13T13:11:22.905331+0000","last_fullsized":"2024-09-13T13:11:22.905331+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:06:51.627691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.627691+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":611,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903886+0000","last_change":"2024-09-13T13:11:11.751140+0000","last_active":"2024-09-13T13:11:22.903886+0000","last_peered":"2024-09-13T13:11:22.903886+0000","last_clean":"2024-09-13T13:11:22.903886+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:11:22.903886+0000","last_undegraded":"2024-09-13T13:11:22.903886+0000","last_fullsized":"2024-09-13T13:11:22.903886+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:11:11.751078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.751078+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.052505+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":468,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902024+0000","last_change":"2024-09-13T13:10:42.034653+0000","last_active":"2024-09-13T13:11:22.902024+0000","last_peered":"2024-09-13T13:11:22.902024+0000","last_clean":"2024-09-13T13:11:22.902024+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:11:22.902024+0000","last_undegraded":"2024-09-13T13:11:22.902024+0000","last_fullsized":"2024-09-13T13:11:22.902024+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:42.034601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.034601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.440553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":436,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-13T13:11:18.923948+0000","last_change":"2024-09-13T13:07:10.058616+0000","last_active":"2024-09-13T13:11:18.923948+0000","last_peered":"2024-09-13T13:11:18.923948+0000","last_clean":"2024-09-13T13:11:18.923948+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:11:18.923948+0000","last_undegraded":"2024-09-13T13:11:18.923948+0000","last_fullsized":"2024-09-13T13:11:18.923948+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:10.058555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.058555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:50.118617+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":475,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.910661+0000","last_change":"2024-09-13T13:09:37.358795+0000","last_active":"2024-09-13T13:11:22.910661+0000","last_peered":"2024-09-13T13:11:22.910661+0000","last_clean":"2024-09-13T13:11:22.910661+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:11:22.910661+0000","last_undegraded":"2024-09-13T13:11:22.910661+0000","last_fullsized":"2024-09-13T13:11:22.910661+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:09:37.358710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:37.358710+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:55.589722+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":477,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902712+0000","last_change":"2024-09-13T13:11:12.054173+0000","last_active":"2024-09-13T13:11:22.902712+0000","last_peered":"2024-09-13T13:11:22.902712+0000","last_clean":"2024-09-13T13:11:22.902712+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:11:22.902712+0000","last_undegraded":"2024-09-13T13:11:22.902712+0000","last_fullsized":"2024-09-13T13:11:22.902712+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:12.054125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:12.054125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.049869+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":476,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904782+0000","last_change":"2024-09-13T13:09:30.959919+0000","last_active":"2024-09-13T13:11:22.904782+0000","last_peered":"2024-09-13T13:11:22.904782+0000","last_clean":"2024-09-13T13:11:22.904782+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:11:22.904782+0000","last_undegraded":"2024-09-13T13:11:22.904782+0000","last_fullsized":"2024-09-13T13:11:22.904782+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.959870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.959870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:01.787830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":481,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.899823+0000","last_change":"2024-09-13T13:09:34.426362+0000","last_active":"2024-09-13T13:11:22.899823+0000","last_peered":"2024-09-13T13:11:22.899823+0000","last_clean":"2024-09-13T13:11:22.899823+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:11:22.899823+0000","last_undegraded":"2024-09-13T13:11:22.899823+0000","last_fullsized":"2024-09-13T13:11:22.899823+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:34.426272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:34.426272+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:59.241784+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":464,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904180+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:11:22.904180+0000","last_peered":"2024-09-13T13:11:22.904180+0000","last_clean":"2024-09-13T13:11:22.904180+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:11:22.904180+0000","last_undegraded":"2024-09-13T13:11:22.904180+0000","last_fullsized":"2024-09-13T13:11:22.904180+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.986182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":461,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.910443+0000","last_change":"2024-09-13T13:11:11.553996+0000","last_active":"2024-09-13T13:11:22.910443+0000","last_peered":"2024-09-13T13:11:22.910443+0000","last_clean":"2024-09-13T13:11:22.910443+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:11:22.910443+0000","last_undegraded":"2024-09-13T13:11:22.910443+0000","last_fullsized":"2024-09-13T13:11:22.910443+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:11.553908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.553908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:41.237817+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":474,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.923246+0000","last_change":"2024-09-13T13:10:34.954684+0000","last_active":"2024-09-13T13:11:22.923246+0000","last_peered":"2024-09-13T13:11:22.923246+0000","last_clean":"2024-09-13T13:11:22.923246+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:11:22.923246+0000","last_undegraded":"2024-09-13T13:11:22.923246+0000","last_fullsized":"2024-09-13T13:11:22.923246+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:34.954585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:34.954585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.029347+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":477,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905860+0000","last_change":"2024-09-13T13:09:48.990167+0000","last_active":"2024-09-13T13:11:22.905860+0000","last_peered":"2024-09-13T13:11:22.905860+0000","last_clean":"2024-09-13T13:11:22.905860+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:11:22.905860+0000","last_undegraded":"2024-09-13T13:11:22.905860+0000","last_fullsized":"2024-09-13T13:11:22.905860+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.057014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":460,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.900512+0000","last_change":"2024-09-13T13:06:41.308449+0000","last_active":"2024-09-13T13:11:22.900512+0000","last_peered":"2024-09-13T13:11:22.900512+0000","last_clean":"2024-09-13T13:11:22.900512+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:11:22.900512+0000","last_undegraded":"2024-09-13T13:11:22.900512+0000","last_fullsized":"2024-09-13T13:11:22.900512+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:41.308367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:41.308367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:27.297312+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":586,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.910060+0000","last_change":"2024-09-13T13:10:07.230375+0000","last_active":"2024-09-13T13:11:22.910060+0000","last_peered":"2024-09-13T13:11:22.910060+0000","last_clean":"2024-09-13T13:11:22.910060+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:11:22.910060+0000","last_undegraded":"2024-09-13T13:11:22.910060+0000","last_fullsized":"2024-09-13T13:11:22.910060+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:10:07.230276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:07.230276+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":458,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.897717+0000","last_change":"2024-09-13T13:07:21.958114+0000","last_active":"2024-09-13T13:11:22.897717+0000","last_peered":"2024-09-13T13:11:22.897717+0000","last_clean":"2024-09-13T13:11:22.897717+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:11:22.897717+0000","last_undegraded":"2024-09-13T13:11:22.897717+0000","last_fullsized":"2024-09-13T13:11:22.897717+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:21.958030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:21.958030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:59.015030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":495,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.910388+0000","last_change":"2024-09-13T13:09:04.635198+0000","last_active":"2024-09-13T13:11:22.910388+0000","last_peered":"2024-09-13T13:11:22.910388+0000","last_clean":"2024-09-13T13:11:22.910388+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:11:22.910388+0000","last_undegraded":"2024-09-13T13:11:22.910388+0000","last_fullsized":"2024-09-13T13:11:22.910388+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:04.599040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:04.599040+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:52.519601+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":461,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904436+0000","last_change":"2024-09-13T13:11:13.811691+0000","last_active":"2024-09-13T13:11:22.904436+0000","last_peered":"2024-09-13T13:11:22.904436+0000","last_clean":"2024-09-13T13:11:22.904436+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:11:22.904436+0000","last_undegraded":"2024-09-13T13:11:22.904436+0000","last_fullsized":"2024-09-13T13:11:22.904436+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:13.811631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:13.811631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.142292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":602,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905646+0000","last_change":"2024-09-13T13:07:08.781005+0000","last_active":"2024-09-13T13:11:22.905646+0000","last_peered":"2024-09-13T13:11:22.905646+0000","last_clean":"2024-09-13T13:11:22.905646+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:11:22.905646+0000","last_undegraded":"2024-09-13T13:11:22.905646+0000","last_fullsized":"2024-09-13T13:11:22.905646+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:07:08.780952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.780952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:00.142438+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":456,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.899440+0000","last_change":"2024-09-13T13:09:36.414394+0000","last_active":"2024-09-13T13:11:22.899440+0000","last_peered":"2024-09-13T13:11:22.899440+0000","last_clean":"2024-09-13T13:11:22.899440+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:11:22.899440+0000","last_undegraded":"2024-09-13T13:11:22.899440+0000","last_fullsized":"2024-09-13T13:11:22.899440+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:36.414322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:36.414322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:48.346115+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":508,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.896012+0000","last_change":"2024-09-13T13:11:18.158667+0000","last_active":"2024-09-13T13:11:22.896012+0000","last_peered":"2024-09-13T13:11:22.896012+0000","last_clean":"2024-09-13T13:11:22.896012+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:11:22.896012+0000","last_undegraded":"2024-09-13T13:11:22.896012+0000","last_fullsized":"2024-09-13T13:11:22.896012+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:18.158584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:18.158584+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:40.118653+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":473,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898385+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:11:22.898385+0000","last_peered":"2024-09-13T13:11:22.898385+0000","last_clean":"2024-09-13T13:11:22.898385+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:11:22.898385+0000","last_undegraded":"2024-09-13T13:11:22.898385+0000","last_fullsized":"2024-09-13T13:11:22.898385+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:21.138701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":462,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.906294+0000","last_change":"2024-09-13T13:09:05.882862+0000","last_active":"2024-09-13T13:11:22.906294+0000","last_peered":"2024-09-13T13:11:22.906294+0000","last_clean":"2024-09-13T13:11:22.906294+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:11:22.906294+0000","last_undegraded":"2024-09-13T13:11:22.906294+0000","last_fullsized":"2024-09-13T13:11:22.906294+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:05.882769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.882769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":457,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.908712+0000","last_change":"2024-09-13T13:08:59.585721+0000","last_active":"2024-09-13T13:11:22.908712+0000","last_peered":"2024-09-13T13:11:22.908712+0000","last_clean":"2024-09-13T13:11:22.908712+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:11:22.908712+0000","last_undegraded":"2024-09-13T13:11:22.908712+0000","last_fullsized":"2024-09-13T13:11:22.908712+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:59.585646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.585646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:51.544074+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":433,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.901883+0000","last_change":"2024-09-13T13:10:10.932948+0000","last_active":"2024-09-13T13:11:22.901883+0000","last_peered":"2024-09-13T13:11:22.901883+0000","last_clean":"2024-09-13T13:11:22.901883+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:11:22.901883+0000","last_undegraded":"2024-09-13T13:11:22.901883+0000","last_fullsized":"2024-09-13T13:11:22.901883+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:10.932889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.932889+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:33.657024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":469,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905548+0000","last_change":"2024-09-13T13:09:07.118545+0000","last_active":"2024-09-13T13:11:22.905548+0000","last_peered":"2024-09-13T13:11:22.905548+0000","last_clean":"2024-09-13T13:11:22.905548+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:11:22.905548+0000","last_undegraded":"2024-09-13T13:11:22.905548+0000","last_fullsized":"2024-09-13T13:11:22.905548+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:07.118467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:07.118467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:59.839803+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":461,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.909170+0000","last_change":"2024-09-13T13:11:08.551743+0000","last_active":"2024-09-13T13:11:22.909170+0000","last_peered":"2024-09-13T13:11:22.909170+0000","last_clean":"2024-09-13T13:11:22.909170+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:11:22.909170+0000","last_undegraded":"2024-09-13T13:11:22.909170+0000","last_fullsized":"2024-09-13T13:11:22.909170+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:08.551449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:08.551449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:26.007369+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":464,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898612+0000","last_change":"2024-09-13T13:10:39.477882+0000","last_active":"2024-09-13T13:11:22.898612+0000","last_peered":"2024-09-13T13:11:22.898612+0000","last_clean":"2024-09-13T13:11:22.898612+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:11:22.898612+0000","last_undegraded":"2024-09-13T13:11:22.898612+0000","last_fullsized":"2024-09-13T13:11:22.898612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:39.477683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.477683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:04.956182+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":491,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.908852+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:11:22.908852+0000","last_peered":"2024-09-13T13:11:22.908852+0000","last_clean":"2024-09-13T13:11:22.908852+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:11:22.908852+0000","last_undegraded":"2024-09-13T13:11:22.908852+0000","last_fullsized":"2024-09-13T13:11:22.908852+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":469,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902799+0000","last_change":"2024-09-13T13:08:50.517587+0000","last_active":"2024-09-13T13:11:22.902799+0000","last_peered":"2024-09-13T13:11:22.902799+0000","last_clean":"2024-09-13T13:11:22.902799+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:11:22.902799+0000","last_undegraded":"2024-09-13T13:11:22.902799+0000","last_fullsized":"2024-09-13T13:11:22.902799+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:50.517526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:50.517526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:20.085000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":472,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903110+0000","last_change":"2024-09-13T13:08:53.215325+0000","last_active":"2024-09-13T13:11:22.903110+0000","last_peered":"2024-09-13T13:11:22.903110+0000","last_clean":"2024-09-13T13:11:22.903110+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:11:22.903110+0000","last_undegraded":"2024-09-13T13:11:22.903110+0000","last_fullsized":"2024-09-13T13:11:22.903110+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.215263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.215263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:50.963832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":452,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.909871+0000","last_change":"2024-09-13T13:07:57.596272+0000","last_active":"2024-09-13T13:11:22.909871+0000","last_peered":"2024-09-13T13:11:22.909871+0000","last_clean":"2024-09-13T13:11:22.909871+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:11:22.909871+0000","last_undegraded":"2024-09-13T13:11:22.909871+0000","last_fullsized":"2024-09-13T13:11:22.909871+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.596097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:53.509678+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":493,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.911454+0000","last_change":"2024-09-13T13:09:39.368581+0000","last_active":"2024-09-13T13:11:22.911454+0000","last_peered":"2024-09-13T13:11:22.911454+0000","last_clean":"2024-09-13T13:11:22.911454+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:11:22.911454+0000","last_undegraded":"2024-09-13T13:11:22.911454+0000","last_fullsized":"2024-09-13T13:11:22.911454+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2024-09-13T13:09:39.368381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:39.368381+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:57.544832+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":484,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.900379+0000","last_change":"2024-09-13T13:11:04.275826+0000","last_active":"2024-09-13T13:11:22.900379+0000","last_peered":"2024-09-13T13:11:22.900379+0000","last_clean":"2024-09-13T13:11:22.900379+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:11:22.900379+0000","last_undegraded":"2024-09-13T13:11:22.900379+0000","last_fullsized":"2024-09-13T13:11:22.900379+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:04.275756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:04.275756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:29.145451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":368,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-13T13:11:18.925673+0000","last_change":"2024-09-13T13:10:36.918019+0000","last_active":"2024-09-13T13:11:18.925673+0000","last_peered":"2024-09-13T13:11:18.925673+0000","last_clean":"2024-09-13T13:11:18.925673+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:11:18.925673+0000","last_undegraded":"2024-09-13T13:11:18.925673+0000","last_fullsized":"2024-09-13T13:11:18.925673+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:36.917945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:36.917945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.082612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":473,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903165+0000","last_change":"2024-09-13T13:11:08.704496+0000","last_active":"2024-09-13T13:11:22.903165+0000","last_peered":"2024-09-13T13:11:22.903165+0000","last_clean":"2024-09-13T13:11:22.903165+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:11:22.903165+0000","last_undegraded":"2024-09-13T13:11:22.903165+0000","last_fullsized":"2024-09-13T13:11:22.903165+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:08.704450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:08.704450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.469645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":458,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.906823+0000","last_change":"2024-09-13T13:08:27.641188+0000","last_active":"2024-09-13T13:11:22.906823+0000","last_peered":"2024-09-13T13:11:22.906823+0000","last_clean":"2024-09-13T13:11:22.906823+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:11:22.906823+0000","last_undegraded":"2024-09-13T13:11:22.906823+0000","last_fullsized":"2024-09-13T13:11:22.906823+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:27.641105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:27.641105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":469,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.911754+0000","last_change":"2024-09-13T13:11:07.564094+0000","last_active":"2024-09-13T13:11:22.911754+0000","last_peered":"2024-09-13T13:11:22.911754+0000","last_clean":"2024-09-13T13:11:22.911754+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:11:22.911754+0000","last_undegraded":"2024-09-13T13:11:22.911754+0000","last_fullsized":"2024-09-13T13:11:22.911754+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:07.564025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:07.564025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:18.616035+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":466,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-13T13:11:18.924489+0000","last_change":"2024-09-13T13:06:09.928585+0000","last_active":"2024-09-13T13:11:18.924489+0000","last_peered":"2024-09-13T13:11:18.924489+0000","last_clean":"2024-09-13T13:11:18.924489+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:11:18.924489+0000","last_undegraded":"2024-09-13T13:11:18.924489+0000","last_fullsized":"2024-09-13T13:11:18.924489+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:06:09.928504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.928504+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.088075+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":469,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.899572+0000","last_change":"2024-09-13T13:09:38.337230+0000","last_active":"2024-09-13T13:11:22.899572+0000","last_peered":"2024-09-13T13:11:22.899572+0000","last_clean":"2024-09-13T13:11:22.899572+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:11:22.899572+0000","last_undegraded":"2024-09-13T13:11:22.899572+0000","last_fullsized":"2024-09-13T13:11:22.899572+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.337156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.337156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:50.333108+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6908149760,"num_objects":105410,"num_object_clones":0,"num_object_copies":316230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105410,"num_write_kb":6746240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20670644224,"data_stored":20670644224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82610,"ondisk_log_size":82610,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":681,"ondisk_log_size":681,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":614,"num_read_kb":609,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201407,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4078592,"kb_used_data":3758640,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":89666560,"statfs":{"total":95995035648,"available":91818557440,"internally_reserved":0,"allocated":3848847360,"data_stored":3848261676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":327612874},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141017,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1502128,"kb_used_data":1212000,"kb_used_omap":4,"kb_used_meta":290107,"kb_avail":92243024,"statfs":{"total":95995035648,"available":94456856576,"internally_reserved":0,"allocated":1241088000,"data_stored":1240517071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":297070253},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691979,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4154692,"kb_used_data":3794480,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":89590460,"statfs":{"total":95995035648,"available":91740631040,"internally_reserved":0,"allocated":3885547520,"data_stored":3884966988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920913,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1496008,"kb_used_data":1234360,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":92249144,"statfs":{"total":95995035648,"available":94463123456,"internally_reserved":0,"allocated":1263984640,"data_stored":1263389196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":267909574},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117144,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3833496,"kb_used_data":3369160,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":89911656,"statfs":{"total":95995035648,"available":92069535744,"internally_reserved":0,"allocated":3450019840,"data_stored":3449415698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":475462087},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411483,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1562916,"kb_used_data":1256852,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92182236,"statfs":{"total":95995035648,"available":94394609664,"internally_reserved":0,"allocated":1287016448,"data_stored":1286457072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":313391558},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4142492,"kb_used_data":3768396,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":89602660,"statfs":{"total":95995035648,"available":91753123840,"internally_reserved":0,"allocated":3858837504,"data_stored":3858222091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738537,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2050132,"kb_used_data":1703808,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91695020,"statfs":{"total":95995035648,"available":93895700480,"internally_reserved":0,"allocated":1744699392,"data_stored":1744103889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1751056384,"data_stored":1751056384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3897491456,"data_stored":3897491456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1297612800,"data_stored":1297612800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3449356288,"data_stored":3449356288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1276444672,"data_stored":1276444672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3889168384,"data_stored":3889168384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1252524032,"data_stored":1252524032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3856990208,"data_stored":3856990208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:11:30.770 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T13:11:30.770 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:11:30.771 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-13T13:11:30.771 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:11:30.786 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:30.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:30.802 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 75 16 113742 113726 94.7338 77.5 0.0117031 0.0105511 2024-09-13T13:11:30.888 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:30.950 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:30.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:30.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:31.052 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:31.059 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:11:31.060 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:11:31.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:11:31.060 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:11:31.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.060 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83976 7.7 7:ec00590c:::benchmark_data_smithi052_71865_object83975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:11:31.060 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.558362+0000", 2024-09-13T13:11:31.060 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 496.49633062200002, 2024-09-13T13:11:31.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99679333000000003, 2024-09-13T13:11:31.060 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.060 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.060 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.060 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.061 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:31.061 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:31.061 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83976 2024-09-13T13:11:31.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.061 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558362+0000", 2024-09-13T13:11:31.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558362+0000", 2024-09-13T13:11:31.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558364+0000", 2024-09-13T13:11:31.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0870000000000001e-06 2024-09-13T13:11:31.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558389+0000", 2024-09-13T13:11:31.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5051000000000001e-05 2024-09-13T13:11:31.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558401+0000", 2024-09-13T13:11:31.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1943000000000001e-05 2024-09-13T13:11:31.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558409+0000", 2024-09-13T13:11:31.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1580000000000005e-06 2024-09-13T13:11:31.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558524+0000", 2024-09-13T13:11:31.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114991 2024-09-13T13:11:31.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:31.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.558532+0000", 2024-09-13T13:11:31.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0450000000000001e-06 2024-09-13T13:11:31.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.542003+0000", 2024-09-13T13:11:31.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98347105700000004 2024-09-13T13:11:31.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.544289+0000", 2024-09-13T13:11:31.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002285493 2024-09-13T13:11:31.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:11:31.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.544471+0000", 2024-09-13T13:11:31.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018247200000000001 2024-09-13T13:11:31.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555006+0000", 2024-09-13T13:11:31.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010534559000000001 2024-09-13T13:11:31.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555018+0000", 2024-09-13T13:11:31.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1756999999999999e-05 2024-09-13T13:11:31.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555068+0000", 2024-09-13T13:11:31.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9982999999999999e-05 2024-09-13T13:11:31.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555155+0000", 2024-09-13T13:11:31.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7733999999999997e-05 2024-09-13T13:11:31.067 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.067 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.067 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.067 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83980 7.7 7:e454f641:::benchmark_data_smithi052_71865_object83979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:11:31.067 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.562536+0000", 2024-09-13T13:11:31.067 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 496.49215624200002, 2024-09-13T13:11:31.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99324992999999995, 2024-09-13T13:11:31.068 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.068 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.068 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.068 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.068 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:31.068 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:31.068 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83980 2024-09-13T13:11:31.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.069 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.069 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.069 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562536+0000", 2024-09-13T13:11:31.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.069 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.069 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562536+0000", 2024-09-13T13:11:31.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562538+0000", 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9460000000000001e-06 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562564+0000", 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5812e-05 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562579+0000", 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4515000000000001e-05 2024-09-13T13:11:31.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562587+0000", 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1529999999999997e-06 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562654+0000", 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7094000000000005e-05 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562662+0000", 2024-09-13T13:11:31.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2870000000000004e-06 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548143+0000", 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98548045299999998 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548261+0000", 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118776 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:11:31.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548419+0000", 2024-09-13T13:11:31.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015719799999999999 2024-09-13T13:11:31.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555169+0000", 2024-09-13T13:11:31.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0067500839999999999 2024-09-13T13:11:31.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555200+0000", 2024-09-13T13:11:31.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1285999999999998e-05 2024-09-13T13:11:31.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555285+0000", 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4899000000000001e-05 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555786+0000", 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00050142700000000004 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.074 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83984 7.7 7:e1036ce8:::benchmark_data_smithi052_71865_object83983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:11:31.075 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.567703+0000", 2024-09-13T13:11:31.075 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 496.48698912600003, 2024-09-13T13:11:31.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99284181800000004, 2024-09-13T13:11:31.075 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.075 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.075 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.075 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.075 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:31.075 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:31.075 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83984 2024-09-13T13:11:31.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.075 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567703+0000", 2024-09-13T13:11:31.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567703+0000", 2024-09-13T13:11:31.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567706+0000", 2024-09-13T13:11:31.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3889999999999999e-06 2024-09-13T13:11:31.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567733+0000", 2024-09-13T13:11:31.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.69e-05 2024-09-13T13:11:31.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567754+0000", 2024-09-13T13:11:31.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1772000000000001e-05 2024-09-13T13:11:31.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567765+0000", 2024-09-13T13:11:31.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0553999999999999e-05 2024-09-13T13:11:31.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567856+0000", 2024-09-13T13:11:31.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0926000000000003e-05 2024-09-13T13:11:31.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567874+0000", 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8481000000000001e-05 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552014+0000", 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98413990600000001 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552117+0000", 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010265200000000001 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3", 2024-09-13T13:11:31.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552243+0000", 2024-09-13T13:11:31.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012594900000000001 2024-09-13T13:11:31.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560036+0000", 2024-09-13T13:11:31.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0077931570000000002 2024-09-13T13:11:31.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560388+0000", 2024-09-13T13:11:31.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035232500000000001 2024-09-13T13:11:31.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560444+0000", 2024-09-13T13:11:31.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5460000000000001e-05 2024-09-13T13:11:31.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560545+0000", 2024-09-13T13:11:31.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101347 2024-09-13T13:11:31.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.081 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.14493.0:1217 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94064716851456 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:15.061486+0000", 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 495.99320697299999, 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.48025142399999998, 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.14493", 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 1217 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.082 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061486+0000", 2024-09-13T13:11:31.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061486+0000", 2024-09-13T13:11:31.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061489+0000", 2024-09-13T13:11:31.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3249999999999999e-06 2024-09-13T13:11:31.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061499+0000", 2024-09-13T13:11:31.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0463e-05 2024-09-13T13:11:31.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061513+0000", 2024-09-13T13:11:31.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3269e-05 2024-09-13T13:11:31.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061521+0000", 2024-09-13T13:11:31.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.909e-06 2024-09-13T13:11:31.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061582+0000", 2024-09-13T13:11:31.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1568000000000004e-05 2024-09-13T13:11:31.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.061589+0000", 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2060000000000001e-06 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.537301+0000", 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.47571140000000001 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.537446+0000", 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000145339 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.541737+0000", 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.004290945 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120046 7.6 7:64955858:::benchmark_data_smithi052_71865_object120045:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.028119+0000", 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 460.02657340799999, 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12583912, 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.087 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120046 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028119+0000", 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028119+0000", 2024-09-13T13:11:31.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028120+0000", 2024-09-13T13:11:31.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2999999999999999e-07 2024-09-13T13:11:31.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028143+0000", 2024-09-13T13:11:31.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3104000000000001e-05 2024-09-13T13:11:31.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028151+0000", 2024-09-13T13:11:31.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9850000000000006e-06 2024-09-13T13:11:31.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.028158+0000", 2024-09-13T13:11:31.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5969999999999997e-06 2024-09-13T13:11:31.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036251+0000", 2024-09-13T13:11:31.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0080934660000000005 2024-09-13T13:11:31.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036367+0000", 2024-09-13T13:11:31.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000115832 2024-09-13T13:11:31.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.036497+0000", 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012981399999999999 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145466+0000", 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10896932099999999 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145645+0000", 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017834400000000001 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153238+0000", 2024-09-13T13:11:31.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0075929919999999998 2024-09-13T13:11:31.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153469+0000", 2024-09-13T13:11:31.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00023148999999999999 2024-09-13T13:11:31.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153958+0000", 2024-09-13T13:11:31.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00048924500000000002 2024-09-13T13:11:31.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.096 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.096 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120049 7.6 7:6e13283c:::benchmark_data_smithi052_71865_object120048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:11:31.096 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.031872+0000", 2024-09-13T13:11:31.096 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 460.02282092799999, 2024-09-13T13:11:31.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.125660096, 2024-09-13T13:11:31.096 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.096 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.096 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.096 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120049 2024-09-13T13:11:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031872+0000", 2024-09-13T13:11:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031872+0000", 2024-09-13T13:11:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031873+0000", 2024-09-13T13:11:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0130000000000001e-06 2024-09-13T13:11:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031889+0000", 2024-09-13T13:11:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6218999999999998e-05 2024-09-13T13:11:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031895+0000", 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3300000000000004e-06 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.031900+0000", 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4129999999999999e-06 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145674+0000", 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.113774598 2024-09-13T13:11:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.145885+0000", 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000210929 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.146035+0000", 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015038000000000001 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.149376+0000", 2024-09-13T13:11:31.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033409690000000001 2024-09-13T13:11:31.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153999+0000", 2024-09-13T13:11:31.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0046225670000000002 2024-09-13T13:11:31.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157357+0000", 2024-09-13T13:11:31.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033581800000000001 2024-09-13T13:11:31.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157485+0000", 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012788599999999999 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157532+0000", 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6612e-05 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.102 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120055 7.6 7:6a228e92:::benchmark_data_smithi052_71865_object120054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.039067+0000", 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 460.01562591700002, 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.124460581, 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120055 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039067+0000", 2024-09-13T13:11:31.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039067+0000", 2024-09-13T13:11:31.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039068+0000", 2024-09-13T13:11:31.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7859999999999999e-06 2024-09-13T13:11:31.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039092+0000", 2024-09-13T13:11:31.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3323999999999999e-05 2024-09-13T13:11:31.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039103+0000", 2024-09-13T13:11:31.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1408e-05 2024-09-13T13:11:31.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.039110+0000", 2024-09-13T13:11:31.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0700000000000001e-06 2024-09-13T13:11:31.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.149407+0000", 2024-09-13T13:11:31.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.110296832 2024-09-13T13:11:31.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.149533+0000", 2024-09-13T13:11:31.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012555699999999999 2024-09-13T13:11:31.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.149681+0000", 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014797000000000001 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.153150+0000", 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0034697669999999999 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157574+0000", 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.004423382 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.163426+0000", 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058521060000000001 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.163482+0000", 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6328e-05 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.163527+0000", 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5050999999999999e-05 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.108 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.109 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.109 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:199048 7.7 7:ef6de7c0:::benchmark_data_smithi052_71865_object77947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-13T13:11:31.109 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:05:25.185200+0000", 2024-09-13T13:11:31.109 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 365.86949213100002, 2024-09-13T13:11:31.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.127661889, 2024-09-13T13:11:31.109 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.109 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.109 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.109 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.109 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 199048 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185200+0000", 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185200+0000", 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185201+0000", 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3900000000000004e-07 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185205+0000", 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.455e-06 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185358+0000", 2024-09-13T13:11:31.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000152919 2024-09-13T13:11:31.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.185368+0000", 2024-09-13T13:11:31.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0465000000000001e-05 2024-09-13T13:11:31.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.201869+0000", 2024-09-13T13:11:31.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.016500959999999999 2024-09-13T13:11:31.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.202064+0000", 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019464999999999999 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.202323+0000", 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00025961499999999999 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.210108+0000", 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.007784549 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.219023+0000", 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0089156979999999997 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.312149+0000", 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.093125873999999997 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.312287+0000", 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013777700000000001 2024-09-13T13:11:31.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:05:25.312862+0000", 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00057518800000000002 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237925 8.6 8:64810c19:::benchmark_data_smithi052_101302_object109996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.068404+0000", 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 112.98628863499999, 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.14961675099999999, 2024-09-13T13:11:31.115 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.116 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.116 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.116 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.116 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:11:31.116 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:11:31.116 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237925 2024-09-13T13:11:31.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.116 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068404+0000", 2024-09-13T13:11:31.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068404+0000", 2024-09-13T13:11:31.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068406+0000", 2024-09-13T13:11:31.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.728e-06 2024-09-13T13:11:31.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068411+0000", 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6019999999999999e-06 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068422+0000", 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0769e-05 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068429+0000", 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7889999999999997e-06 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.119 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068490+0000", 2024-09-13T13:11:31.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0983000000000002e-05 2024-09-13T13:11:31.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.119 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.119 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068617+0000", 2024-09-13T13:11:31.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000127267 2024-09-13T13:11:31.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.119 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:11:31.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:31.120 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:31.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:31.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068733+0000", 2024-09-13T13:11:31.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116445 2024-09-13T13:11:31.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074303+0000", 2024-09-13T13:11:31.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055691739999999997 2024-09-13T13:11:31.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217789+0000", 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.143486802 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217899+0000", 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110011 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217970+0000", 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0506999999999994e-05 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218021+0000", 2024-09-13T13:11:31.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0674000000000003e-05 2024-09-13T13:11:31.123 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.123 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.123 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.123 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237927 8.6 8:66a2a178:::benchmark_data_smithi052_101302_object109998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:11:31.123 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.068600+0000", 2024-09-13T13:11:31.123 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 112.986092164, 2024-09-13T13:11:31.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.149587739, 2024-09-13T13:11:31.123 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.123 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.124 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.124 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.124 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:11:31.124 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:11:31.124 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237927 2024-09-13T13:11:31.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.124 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068600+0000", 2024-09-13T13:11:31.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068600+0000", 2024-09-13T13:11:31.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068602+0000", 2024-09-13T13:11:31.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.176e-06 2024-09-13T13:11:31.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068606+0000", 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3039999999999998e-06 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068615+0000", 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9369999999999993e-06 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068624+0000", 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8400000000000001e-06 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071818+0000", 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003194625 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071929+0000", 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110269 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.072044+0000", 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000115396 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077466+0000", 2024-09-13T13:11:31.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.005422568 2024-09-13T13:11:31.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217827+0000", 2024-09-13T13:11:31.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.14036069200000001 2024-09-13T13:11:31.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218099+0000", 2024-09-13T13:11:31.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000271511 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218152+0000", 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2831000000000003e-05 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218188+0000", 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.659e-05 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237928 8.6 8:692bc360:::benchmark_data_smithi052_101302_object109999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.068748+0000", 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 112.985944817, 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15122808900000001, 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237928 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068748+0000", 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068748+0000", 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068749+0000", 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8599999999999996e-07 2024-09-13T13:11:31.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068753+0000", 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9589999999999997e-06 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068760+0000", 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6860000000000006e-06 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068766+0000", 2024-09-13T13:11:31.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8050000000000003e-06 2024-09-13T13:11:31.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074343+0000", 2024-09-13T13:11:31.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055769210000000003 2024-09-13T13:11:31.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074450+0000", 2024-09-13T13:11:31.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107321 2024-09-13T13:11:31.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074571+0000", 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012104699999999999 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214148+0000", 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.139576799 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217865+0000", 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037168190000000001 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218583+0000", 2024-09-13T13:11:31.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00071750199999999996 2024-09-13T13:11:31.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.219928+0000", 2024-09-13T13:11:31.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0013456729999999999 2024-09-13T13:11:31.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.219976+0000", 2024-09-13T13:11:31.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7571000000000002e-05 2024-09-13T13:11:31.135 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.136 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.136 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.136 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237931 8.e 8:77956f02:::benchmark_data_smithi052_101302_object110002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:11:31.136 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.071964+0000", 2024-09-13T13:11:31.136 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 112.982728794, 2024-09-13T13:11:31.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.14610564600000001, 2024-09-13T13:11:31.136 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.136 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.136 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.136 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.136 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:11:31.137 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:11:31.137 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237931 2024-09-13T13:11:31.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.137 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071964+0000", 2024-09-13T13:11:31.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071964+0000", 2024-09-13T13:11:31.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071966+0000", 2024-09-13T13:11:31.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8560000000000001e-06 2024-09-13T13:11:31.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071971+0000", 2024-09-13T13:11:31.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9660000000000004e-06 2024-09-13T13:11:31.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071982+0000", 2024-09-13T13:11:31.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1214e-05 2024-09-13T13:11:31.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.071989+0000", 2024-09-13T13:11:31.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.644e-06 2024-09-13T13:11:31.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074334+0000", 2024-09-13T13:11:31.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00234475 2024-09-13T13:11:31.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074450+0000", 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011572499999999999 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074571+0000", 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121546 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214179+0000", 2024-09-13T13:11:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.139607703 2024-09-13T13:11:31.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217833+0000", 2024-09-13T13:11:31.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036534179999999999 2024-09-13T13:11:31.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217883+0000", 2024-09-13T13:11:31.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0846000000000001e-05 2024-09-13T13:11:31.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217960+0000", 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6120999999999998e-05 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218069+0000", 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109857 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.142 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.25222.0:237924 8.9 e232/169 8:9137082c:::benchmark_data_smithi052_101302_object109995:head v 232'14652, mlcod=232'14649)", 2024-09-13T13:11:31.143 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.076510+0000", 2024-09-13T13:11:31.143 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 112.978182078, 2024-09-13T13:11:31.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13780474600000001, 2024-09-13T13:11:31.143 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.143 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.143 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.143 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076510+0000", 2024-09-13T13:11:31.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076510+0000", 2024-09-13T13:11:31.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076512+0000", 2024-09-13T13:11:31.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.486e-06 2024-09-13T13:11:31.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076518+0000", 2024-09-13T13:11:31.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7239999999999997e-06 2024-09-13T13:11:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076528+0000", 2024-09-13T13:11:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8400000000000007e-06 2024-09-13T13:11:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076533+0000", 2024-09-13T13:11:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3090000000000003e-06 2024-09-13T13:11:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076572+0000", 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.879e-05 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.076585+0000", 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2955e-05 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214214+0000", 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13762922 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214315+0000", 2024-09-13T13:11:31.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010142200000000001 2024-09-13T13:11:31.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.147 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.147 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.25222.0:237931 8.e e232/169 ondisk, result = 0)", 2024-09-13T13:11:31.147 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.077192+0000", 2024-09-13T13:11:31.147 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 112.97750097399999, 2024-09-13T13:11:31.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13702303699999999, 2024-09-13T13:11:31.147 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077192+0000", 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077192+0000", 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077193+0000", 2024-09-13T13:11:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.184e-06 2024-09-13T13:11:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077198+0000", 2024-09-13T13:11:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0470000000000001e-06 2024-09-13T13:11:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077207+0000", 2024-09-13T13:11:31.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3619999999999995e-06 2024-09-13T13:11:31.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.077214+0000", 2024-09-13T13:11:31.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7020000000000004e-06 2024-09-13T13:11:31.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214138+0000", 2024-09-13T13:11:31.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.136924569 2024-09-13T13:11:31.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214162+0000", 2024-09-13T13:11:31.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3496999999999999e-05 2024-09-13T13:11:31.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.152 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214215+0000", 2024-09-13T13:11:31.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2676000000000002e-05 2024-09-13T13:11:31.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.152 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.152 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.25222.0:237935 8.1 e232/169 8:84af4e89:::benchmark_data_smithi052_101302_object110006:head v 232'14986, mlcod=232'14984)", 2024-09-13T13:11:31.152 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.080810+0000", 2024-09-13T13:11:31.152 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 112.973882859, 2024-09-13T13:11:31.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.149619258, 2024-09-13T13:11:31.152 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.152 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.153 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.153 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.080810+0000", 2024-09-13T13:11:31.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.080810+0000", 2024-09-13T13:11:31.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.080812+0000", 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.857e-06 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.080821+0000", 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4930000000000004e-06 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214079+0000", 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13325792 2024-09-13T13:11:31.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214106+0000", 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7107e-05 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214348+0000", 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024191600000000001 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214369+0000", 2024-09-13T13:11:31.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1149e-05 2024-09-13T13:11:31.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.230395+0000", 2024-09-13T13:11:31.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.016025604999999998 2024-09-13T13:11:31.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.230429+0000", 2024-09-13T13:11:31.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4211000000000002e-05 2024-09-13T13:11:31.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.156 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.157 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237940 8.e 8:7ee1d0e2:::benchmark_data_smithi052_101302_object110011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:11:31.157 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.085058+0000", 2024-09-13T13:11:31.157 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 112.969634953, 2024-09-13T13:11:31.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13638705000000001, 2024-09-13T13:11:31.157 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.157 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.157 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.157 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.157 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:11:31.157 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:11:31.157 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237940 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.085058+0000", 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.085058+0000", 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.085059+0000", 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.666e-06 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.085065+0000", 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5770000000000001e-06 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214099+0000", 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.129034225 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214130+0000", 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1145000000000001e-05 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214387+0000", 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000257302 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214491+0000", 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010342 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:11:31.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214593+0000", 2024-09-13T13:11:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010197299999999999 2024-09-13T13:11:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218079+0000", 2024-09-13T13:11:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003486266 2024-09-13T13:11:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218228+0000", 2024-09-13T13:11:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014901399999999999 2024-09-13T13:11:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.221356+0000", 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031274430000000001 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.221405+0000", 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9570000000000001e-05 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.221445+0000", 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9449000000000001e-05 2024-09-13T13:11:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:110001 9.b 9:df88dd83:::benchmark_data_smithi052_128427_object110000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:27.411752+0000", 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 3.6429407559999998, 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.124718806, 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:11:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:11:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 110001 2024-09-13T13:11:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.411752+0000", 2024-09-13T13:11:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.411752+0000", 2024-09-13T13:11:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.411753+0000", 2024-09-13T13:11:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6509999999999999e-06 2024-09-13T13:11:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.411777+0000", 2024-09-13T13:11:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3686999999999999e-05 2024-09-13T13:11:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.411788+0000", 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1371e-05 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.411796+0000", 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.114e-06 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.411855+0000", 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9042999999999997e-05 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.411978+0000", 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012306599999999999 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.412107+0000", 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000129598 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.415094+0000", 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002986397 2024-09-13T13:11:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.415896+0000", 2024-09-13T13:11:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00080200400000000004 2024-09-13T13:11:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.536407+0000", 2024-09-13T13:11:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.120511511 2024-09-13T13:11:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.536442+0000", 2024-09-13T13:11:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4873000000000003e-05 2024-09-13T13:11:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.536471+0000", 2024-09-13T13:11:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8490999999999999e-05 2024-09-13T13:11:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:110009 9.9 9:9a2cd811:::benchmark_data_smithi052_128427_object110008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-09-13T13:11:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:27.417259+0000", 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 3.6374340109999999, 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12580545600000001, 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 110009 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.417259+0000", 2024-09-13T13:11:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.417259+0000", 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.417260+0000", 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.42e-06 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.417283+0000", 2024-09-13T13:11:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2753999999999999e-05 2024-09-13T13:11:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.417293+0000", 2024-09-13T13:11:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.859e-06 2024-09-13T13:11:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.417372+0000", 2024-09-13T13:11:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9089000000000001e-05 2024-09-13T13:11:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.417884+0000", 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00051269399999999995 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.418015+0000", 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013025499999999999 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.418158+0000", 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014376999999999999 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.542834+0000", 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12467552899999999 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.542912+0000", 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8023999999999997e-05 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.542965+0000", 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2788999999999999e-05 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.543025+0000", 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0111000000000001e-05 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.543064+0000", 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9162000000000002e-05 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:110010 9.9 9:9b6038c1:::benchmark_data_smithi052_128427_object110009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:27.417395+0000", 2024-09-13T13:11:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 3.6372970680000001, 2024-09-13T13:11:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12584240399999999, 2024-09-13T13:11:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:11:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:11:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 110010 2024-09-13T13:11:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.417395+0000", 2024-09-13T13:11:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.417395+0000", 2024-09-13T13:11:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.417396+0000", 2024-09-13T13:11:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.017e-06 2024-09-13T13:11:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.417421+0000", 2024-09-13T13:11:31.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4192000000000001e-05 2024-09-13T13:11:31.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.417430+0000", 2024-09-13T13:11:31.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6080000000000002e-06 2024-09-13T13:11:31.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.417437+0000", 2024-09-13T13:11:31.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.562e-06 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.420748+0000", 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033113389999999999 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.536010+0000", 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.115261882 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:11:31.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.536171+0000", 2024-09-13T13:11:31.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000161111 2024-09-13T13:11:31.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.542923+0000", 2024-09-13T13:11:31.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0067516420000000004 2024-09-13T13:11:31.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.543101+0000", 2024-09-13T13:11:31.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000177951 2024-09-13T13:11:31.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.543165+0000", 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4437000000000007e-05 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.543207+0000", 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1897999999999998e-05 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.543238+0000", 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0765000000000001e-05 2024-09-13T13:11:31.181 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.182 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.182 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.182 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:110013 9.9 9:963f100e:::benchmark_data_smithi052_128427_object110012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-09-13T13:11:31.182 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:27.418179+0000", 2024-09-13T13:11:31.182 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 3.6365137729999999, 2024-09-13T13:11:31.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13458378800000001, 2024-09-13T13:11:31.182 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:31.182 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:31.182 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:31.182 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:31.182 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:11:31.183 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:11:31.183 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 110013 2024-09-13T13:11:31.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.183 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:31.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:31.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.418179+0000", 2024-09-13T13:11:31.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:31.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.418179+0000", 2024-09-13T13:11:31.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:31.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:31.184 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.418180+0000", 2024-09-13T13:11:31.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4199999999999995e-07 2024-09-13T13:11:31.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:31.184 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.418201+0000", 2024-09-13T13:11:31.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1639999999999999e-05 2024-09-13T13:11:31.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:31.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.418208+0000", 2024-09-13T13:11:31.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2880000000000004e-06 2024-09-13T13:11:31.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:31.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.418214+0000", 2024-09-13T13:11:31.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9070000000000004e-06 2024-09-13T13:11:31.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:31.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.539026+0000", 2024-09-13T13:11:31.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.120811483 2024-09-13T13:11:31.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:31.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.539172+0000", 2024-09-13T13:11:31.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014610400000000001 2024-09-13T13:11:31.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:11:31.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.539314+0000", 2024-09-13T13:11:31.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014167300000000001 2024-09-13T13:11:31.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.543133+0000", 2024-09-13T13:11:31.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003819537 2024-09-13T13:11:31.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:31.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.543276+0000", 2024-09-13T13:11:31.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000143113 2024-09-13T13:11:31.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:31.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.552665+0000", 2024-09-13T13:11:31.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0093892009999999998 2024-09-13T13:11:31.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.552718+0000", 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3004999999999998e-05 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:27.552763+0000", 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4094999999999997e-05 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:31.188 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:11:31.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:31.258 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:31.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:31.360 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:31.395 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:31.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:31.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:31.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:31.527 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:31.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:31.623 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:31 smithi167 ceph-mon[33178]: pgmap v505: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-13T13:11:31.630 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:31.672 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:31.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:31 smithi052 ceph-mon[29471]: pgmap v505: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-13T13:11:31.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:31 smithi052 ceph-mon[36682]: pgmap v505: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-13T13:11:31.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:31.804 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 76 16 114939 114923 94.4706 74.8125 0.0092782 0.0105808 2024-09-13T13:11:31.812 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:31.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:31.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:31.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:31.948 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:31.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:32.050 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:32.095 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:32.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:32.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:32.197 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:32.272 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:32.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:32.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:32.374 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:32.414 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:32.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:32.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:32.516 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:32.556 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:32.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:32.658 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:32.686 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:32.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:32.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:32.803 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 77 16 116068 116052 94.1602 70.5625 0.0161965 0.0106158 2024-09-13T13:11:32.826 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:32.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:32.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:32.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:32.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:33.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:33.091 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:33.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:33.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:33.193 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:33.215 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:33.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:33.317 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:33.352 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:33.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:33.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:33.378 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:11:33.455 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:33.492 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:33.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:33.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:33.594 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:33.634 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:33.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:33.735 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:33 smithi167 ceph-mon[33178]: pgmap v506: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-13T13:11:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:33 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1648156747' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:33 smithi167 ceph-mon[33178]: from='client.25795 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:33 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:11:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:33 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:11:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:33 smithi052 ceph-mon[29471]: pgmap v506: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-13T13:11:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:33 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1648156747' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:33 smithi052 ceph-mon[29471]: from='client.25795 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:33 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:11:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:33 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:11:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:33 smithi052 ceph-mon[36682]: pgmap v506: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-13T13:11:33.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:33 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1648156747' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:33.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:33 smithi052 ceph-mon[36682]: from='client.25795 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:33.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:33 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:11:33.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:33 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:11:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:33.774 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:33.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:33.804 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 78 16 117033 117017 93.7263 60.3125 0.0084322 0.0106651 2024-09-13T13:11:33.820 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T13:11:33.875 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:33.911 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:33.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:34.013 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:34.049 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:34.052 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:34.059 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:34.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:34.160 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:34.205 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:34.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:34.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:34.307 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:34.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:34.350 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:34.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:34.451 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:34.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:34 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:34.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:34 smithi052 ceph-mon[36682]: from='client.25795 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:11:34.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:34 smithi052 ceph-mon[36682]: osdmap e276: 8 total, 8 up, 8 in 2024-09-13T13:11:34.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:34 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:34.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:34 smithi052 ceph-mon[29471]: from='client.25795 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:11:34.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:34 smithi052 ceph-mon[29471]: osdmap e276: 8 total, 8 up, 8 in 2024-09-13T13:11:34.491 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:34.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:34.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:34.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:34 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:34.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:34 smithi167 ceph-mon[33178]: from='client.25795 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:11:34.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:34 smithi167 ceph-mon[33178]: osdmap e276: 8 total, 8 up, 8 in 2024-09-13T13:11:34.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:11:34 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:11:34.593 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:34.641 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:34.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:34.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:34.743 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:34.789 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:34.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:34.804 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 79 16 118252 118236 93.5043 76.1875 0.0111155 0.0106901 2024-09-13T13:11:34.891 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:34.922 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:34.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:34.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:35.023 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:35.060 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:35.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:35.162 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:35.205 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:35.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:35.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:35.306 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:35.340 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:35.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:35.442 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:35.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:35.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:35.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:35 smithi052 ceph-mon[36682]: pgmap v508: 148 pgs: 148 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T13:11:35.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:35.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:35 smithi052 ceph-mon[29471]: pgmap v508: 148 pgs: 148 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T13:11:35.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:35.584 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:35.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:35.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:35.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:35.718 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:35.730 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:35.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:35.778 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:35.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:35.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:11:35.804233+0000 min lat: 0.00134564 max lat: 0.270998 avg lat: 0.0107176 2024-09-13T13:11:35.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:11:35.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 80 16 119438 119422 93.262 74.125 0.0275729 0.0107176 2024-09-13T13:11:35.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:35 smithi167 ceph-mon[33178]: pgmap v508: 148 pgs: 148 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T13:11:35.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:35.822 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:11:35.822 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:11:35.871 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:11:35.872 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:11:35.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:35.906 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:35.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:35.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:35.907 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:11:35.908 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:11:35.932 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:11:35.934 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:11:35.948 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:11:35.948 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:11:35.991 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:11:35.992 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:11:36.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:36.030 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:11:36.030 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:11:36.033 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:36.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:36.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:36.066 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:11:36.066 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:11:36.105 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:11:36.105 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:11:36.135 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:36.166 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:36.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:36.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:36.268 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:36.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:36.306 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:36.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:36.366 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:11:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:11:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:11:36.407 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:36.438 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:36.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:36.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:36.540 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:36.576 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:36.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:36.636 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:11:36.677 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:36.678 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:36 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2955785882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:36.679 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:36 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2955785882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:36.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:36.718 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:36.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:36.804 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 81 16 120537 120521 92.9586 68.6875 0.00347293 0.0107426 2024-09-13T13:11:36.819 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:36.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:36 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2955785882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:36.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:36.962 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:37.003 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:37.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:37.106 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:37.146 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:37.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:37.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:37.281 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:37.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:37.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:37.395 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-13T13:11:35.823997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7649761819,"num_objects":116916,"num_object_clones":0,"num_object_copies":350748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116916,"num_whiteouts":0,"num_read":1899,"num_read_kb":1756,"num_write":117552,"num_write_kb":7471561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83363,"ondisk_log_size":83363,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":25114604,"kb_used_data":22268148,"kb_used_omap":15,"kb_used_meta":2846128,"kb_avail":724846612,"statfs":{"total":767960285184,"available":742242930688,"internally_reserved":0,"allocated":22802583552,"data_stored":22797925573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":2914435865},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,9,9,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":741015552,"num_objects":11307,"num_object_clones":0,"num_object_copies":33921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11307,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11307,"num_write_kb":723648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002874"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":471,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.375959+0000","last_change":"2024-09-13T13:10:32.368663+0000","last_active":"2024-09-13T13:11:33.375959+0000","last_peered":"2024-09-13T13:11:33.375959+0000","last_clean":"2024-09-13T13:11:33.375959+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:11:33.375959+0000","last_undegraded":"2024-09-13T13:11:33.375959+0000","last_fullsized":"2024-09-13T13:11:33.375959+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.368588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.368588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:00.402096+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":502,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895347+0000","last_change":"2024-09-13T13:10:32.777554+0000","last_active":"2024-09-13T13:11:22.895347+0000","last_peered":"2024-09-13T13:11:22.895347+0000","last_clean":"2024-09-13T13:11:22.895347+0000","last_became_active":"2024-09-13T13:04:02.763500+0000","last_became_peered":"2024-09-13T13:04:02.763500+0000","last_unstale":"2024-09-13T13:11:22.895347+0000","last_undegraded":"2024-09-13T13:11:22.895347+0000","last_fullsized":"2024-09-13T13:11:22.895347+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:32.777451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.777451+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:00.660196+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":456,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.386410+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:11:33.386410+0000","last_peered":"2024-09-13T13:11:33.386410+0000","last_clean":"2024-09-13T13:11:33.386410+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:11:33.386410+0000","last_undegraded":"2024-09-13T13:11:33.386410+0000","last_fullsized":"2024-09-13T13:11:33.386410+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.772899+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":450,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895548+0000","last_change":"2024-09-13T13:09:42.788562+0000","last_active":"2024-09-13T13:11:22.895548+0000","last_peered":"2024-09-13T13:11:22.895548+0000","last_clean":"2024-09-13T13:11:22.895548+0000","last_became_active":"2024-09-13T13:01:08.213798+0000","last_became_peered":"2024-09-13T13:01:08.213798+0000","last_unstale":"2024-09-13T13:11:22.895548+0000","last_undegraded":"2024-09-13T13:11:22.895548+0000","last_fullsized":"2024-09-13T13:11:22.895548+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:42.787492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:42.787492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:02.026348+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":470,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.382408+0000","last_change":"2024-09-13T13:08:55.504675+0000","last_active":"2024-09-13T13:11:33.382408+0000","last_peered":"2024-09-13T13:11:33.382408+0000","last_clean":"2024-09-13T13:11:33.382408+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:11:33.382408+0000","last_undegraded":"2024-09-13T13:11:33.382408+0000","last_fullsized":"2024-09-13T13:11:33.382408+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:55.504475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:55.504475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:45.483728+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":483,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.383347+0000","last_change":"2024-09-13T13:11:05.501305+0000","last_active":"2024-09-13T13:11:33.383347+0000","last_peered":"2024-09-13T13:11:33.383347+0000","last_clean":"2024-09-13T13:11:33.383347+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:11:33.383347+0000","last_undegraded":"2024-09-13T13:11:33.383347+0000","last_fullsized":"2024-09-13T13:11:33.383347+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:05.501234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:05.501234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.182488+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":526,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.384268+0000","last_change":"2024-09-13T13:10:39.460577+0000","last_active":"2024-09-13T13:11:33.384268+0000","last_peered":"2024-09-13T13:11:33.384268+0000","last_clean":"2024-09-13T13:11:33.384268+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:11:33.384268+0000","last_undegraded":"2024-09-13T13:11:33.384268+0000","last_fullsized":"2024-09-13T13:11:33.384268+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:10:39.460500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.460500+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:00.679480+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":481,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895267+0000","last_change":"2024-09-13T13:11:13.201530+0000","last_active":"2024-09-13T13:11:22.895267+0000","last_peered":"2024-09-13T13:11:22.895267+0000","last_clean":"2024-09-13T13:11:22.895267+0000","last_became_active":"2024-09-13T13:04:02.765030+0000","last_became_peered":"2024-09-13T13:04:02.765030+0000","last_unstale":"2024-09-13T13:11:22.895267+0000","last_undegraded":"2024-09-13T13:11:22.895267+0000","last_fullsized":"2024-09-13T13:11:22.895267+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:13.201484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:13.201484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.299188+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":502,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905952+0000","last_change":"2024-09-13T13:10:35.137237+0000","last_active":"2024-09-13T13:11:22.905952+0000","last_peered":"2024-09-13T13:11:22.905952+0000","last_clean":"2024-09-13T13:11:22.905952+0000","last_became_active":"2024-09-13T13:04:02.754554+0000","last_became_peered":"2024-09-13T13:04:02.754554+0000","last_unstale":"2024-09-13T13:11:22.905952+0000","last_undegraded":"2024-09-13T13:11:22.905952+0000","last_fullsized":"2024-09-13T13:11:22.905952+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:10:35.137160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:35.137160+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.584514+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":507,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895783+0000","last_change":"2024-09-13T13:10:31.738782+0000","last_active":"2024-09-13T13:11:22.895783+0000","last_peered":"2024-09-13T13:11:22.895783+0000","last_clean":"2024-09-13T13:11:22.895783+0000","last_became_active":"2024-09-13T13:01:04.189136+0000","last_became_peered":"2024-09-13T13:01:04.189136+0000","last_unstale":"2024-09-13T13:11:22.895783+0000","last_undegraded":"2024-09-13T13:11:22.895783+0000","last_fullsized":"2024-09-13T13:11:22.895783+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:31.738685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.738685+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:43.720157+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":461,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905325+0000","last_change":"2024-09-13T13:06:50.665256+0000","last_active":"2024-09-13T13:11:22.905325+0000","last_peered":"2024-09-13T13:11:22.905325+0000","last_clean":"2024-09-13T13:11:22.905325+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:11:22.905325+0000","last_undegraded":"2024-09-13T13:11:22.905325+0000","last_fullsized":"2024-09-13T13:11:22.905325+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:50.665191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.665191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:21.077433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":466,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.375293+0000","last_change":"2024-09-13T13:10:43.363157+0000","last_active":"2024-09-13T13:11:33.375293+0000","last_peered":"2024-09-13T13:11:33.375293+0000","last_clean":"2024-09-13T13:11:33.375293+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:11:33.375293+0000","last_undegraded":"2024-09-13T13:11:33.375293+0000","last_fullsized":"2024-09-13T13:11:33.375293+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:43.362967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:43.362967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:11.841904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":485,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902158+0000","last_change":"2024-09-13T13:10:31.103576+0000","last_active":"2024-09-13T13:11:22.902158+0000","last_peered":"2024-09-13T13:11:22.902158+0000","last_clean":"2024-09-13T13:11:22.902158+0000","last_became_active":"2024-09-13T13:01:02.184277+0000","last_became_peered":"2024-09-13T13:01:02.184277+0000","last_unstale":"2024-09-13T13:11:22.902158+0000","last_undegraded":"2024-09-13T13:11:22.902158+0000","last_fullsized":"2024-09-13T13:11:22.902158+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:31.103509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.103509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:52.926445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":471,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895230+0000","last_change":"2024-09-13T13:05:12.093696+0000","last_active":"2024-09-13T13:11:22.895230+0000","last_peered":"2024-09-13T13:11:22.895230+0000","last_clean":"2024-09-13T13:11:22.895230+0000","last_became_active":"2024-09-13T13:01:04.195541+0000","last_became_peered":"2024-09-13T13:01:04.195541+0000","last_unstale":"2024-09-13T13:11:22.895230+0000","last_undegraded":"2024-09-13T13:11:22.895230+0000","last_fullsized":"2024-09-13T13:11:22.895230+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-13T13:05:12.093388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:12.093388+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.107577+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":469,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903809+0000","last_change":"2024-09-13T13:10:32.803006+0000","last_active":"2024-09-13T13:11:22.903809+0000","last_peered":"2024-09-13T13:11:22.903809+0000","last_clean":"2024-09-13T13:11:22.903809+0000","last_became_active":"2024-09-13T13:01:06.210596+0000","last_became_peered":"2024-09-13T13:01:06.210596+0000","last_unstale":"2024-09-13T13:11:22.903809+0000","last_undegraded":"2024-09-13T13:11:22.903809+0000","last_fullsized":"2024-09-13T13:11:22.903809+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.802931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.802931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:58.651202+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":441,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.901908+0000","last_change":"2024-09-13T13:07:19.641119+0000","last_active":"2024-09-13T13:11:22.901908+0000","last_peered":"2024-09-13T13:11:22.901908+0000","last_clean":"2024-09-13T13:11:22.901908+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:11:22.901908+0000","last_undegraded":"2024-09-13T13:11:22.901908+0000","last_fullsized":"2024-09-13T13:11:22.901908+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:19.640697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:19.640697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:57.084952+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":479,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898537+0000","last_change":"2024-09-13T13:10:28.805013+0000","last_active":"2024-09-13T13:11:22.898537+0000","last_peered":"2024-09-13T13:11:22.898537+0000","last_clean":"2024-09-13T13:11:22.898537+0000","last_became_active":"2024-09-13T13:04:02.763772+0000","last_became_peered":"2024-09-13T13:04:02.763772+0000","last_unstale":"2024-09-13T13:11:22.898537+0000","last_undegraded":"2024-09-13T13:11:22.898537+0000","last_fullsized":"2024-09-13T13:11:22.898537+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:28.804735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:28.804735+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.220025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":470,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.379981+0000","last_change":"2024-09-13T13:08:19.181017+0000","last_active":"2024-09-13T13:11:33.379981+0000","last_peered":"2024-09-13T13:11:33.379981+0000","last_clean":"2024-09-13T13:11:33.379981+0000","last_became_active":"2024-09-13T13:01:02.183593+0000","last_became_peered":"2024-09-13T13:01:02.183593+0000","last_unstale":"2024-09-13T13:11:33.379981+0000","last_undegraded":"2024-09-13T13:11:33.379981+0000","last_fullsized":"2024-09-13T13:11:33.379981+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:19.180948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.180948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:57.228082+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":517,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.923571+0000","last_change":"2024-09-13T13:06:44.532305+0000","last_active":"2024-09-13T13:11:22.923571+0000","last_peered":"2024-09-13T13:11:22.923571+0000","last_clean":"2024-09-13T13:11:22.923571+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:11:22.923571+0000","last_undegraded":"2024-09-13T13:11:22.923571+0000","last_fullsized":"2024-09-13T13:11:22.923571+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:06:44.532186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.532186+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":472,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.390781+0000","last_change":"2024-09-13T13:09:59.936868+0000","last_active":"2024-09-13T13:11:33.390781+0000","last_peered":"2024-09-13T13:11:33.390781+0000","last_clean":"2024-09-13T13:11:33.390781+0000","last_became_active":"2024-09-13T13:01:06.209036+0000","last_became_peered":"2024-09-13T13:01:06.209036+0000","last_unstale":"2024-09-13T13:11:33.390781+0000","last_undegraded":"2024-09-13T13:11:33.390781+0000","last_fullsized":"2024-09-13T13:11:33.390781+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:59.936811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:59.936811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:32.765574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":471,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.383575+0000","last_change":"2024-09-13T13:09:29.517189+0000","last_active":"2024-09-13T13:11:33.383575+0000","last_peered":"2024-09-13T13:11:33.383575+0000","last_clean":"2024-09-13T13:11:33.383575+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:11:33.383575+0000","last_undegraded":"2024-09-13T13:11:33.383575+0000","last_fullsized":"2024-09-13T13:11:33.383575+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:29.517034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:29.517034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:54.544083+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":478,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895839+0000","last_change":"2024-09-13T13:10:34.788637+0000","last_active":"2024-09-13T13:11:22.895839+0000","last_peered":"2024-09-13T13:11:22.895839+0000","last_clean":"2024-09-13T13:11:22.895839+0000","last_became_active":"2024-09-13T13:01:02.173191+0000","last_became_peered":"2024-09-13T13:01:02.173191+0000","last_unstale":"2024-09-13T13:11:22.895839+0000","last_undegraded":"2024-09-13T13:11:22.895839+0000","last_fullsized":"2024-09-13T13:11:22.895839+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:34.788562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:34.788562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:03.723901+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":499,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.382181+0000","last_change":"2024-09-13T13:11:14.593530+0000","last_active":"2024-09-13T13:11:33.382181+0000","last_peered":"2024-09-13T13:11:33.382181+0000","last_clean":"2024-09-13T13:11:33.382181+0000","last_became_active":"2024-09-13T13:01:04.194790+0000","last_became_peered":"2024-09-13T13:01:04.194790+0000","last_unstale":"2024-09-13T13:11:33.382181+0000","last_undegraded":"2024-09-13T13:11:33.382181+0000","last_fullsized":"2024-09-13T13:11:33.382181+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:11:14.593438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:14.593438+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:43.465667+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":462,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898090+0000","last_change":"2024-09-13T13:07:12.879813+0000","last_active":"2024-09-13T13:11:22.898090+0000","last_peered":"2024-09-13T13:11:22.898090+0000","last_clean":"2024-09-13T13:11:22.898090+0000","last_became_active":"2024-09-13T13:01:06.206907+0000","last_became_peered":"2024-09-13T13:01:06.206907+0000","last_unstale":"2024-09-13T13:11:22.898090+0000","last_undegraded":"2024-09-13T13:11:22.898090+0000","last_fullsized":"2024-09-13T13:11:22.898090+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.879727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.879727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:52.066816+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":474,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904636+0000","last_change":"2024-09-13T13:10:09.765203+0000","last_active":"2024-09-13T13:11:22.904636+0000","last_peered":"2024-09-13T13:11:22.904636+0000","last_clean":"2024-09-13T13:11:22.904636+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:11:22.904636+0000","last_undegraded":"2024-09-13T13:11:22.904636+0000","last_fullsized":"2024-09-13T13:11:22.904636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:09.765153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.765153+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:26.707398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":465,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.386544+0000","last_change":"2024-09-13T13:07:54.442276+0000","last_active":"2024-09-13T13:11:33.386544+0000","last_peered":"2024-09-13T13:11:33.386544+0000","last_clean":"2024-09-13T13:11:33.386544+0000","last_became_active":"2024-09-13T13:01:02.179905+0000","last_became_peered":"2024-09-13T13:01:02.179905+0000","last_unstale":"2024-09-13T13:11:33.386544+0000","last_undegraded":"2024-09-13T13:11:33.386544+0000","last_fullsized":"2024-09-13T13:11:33.386544+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:07:54.439269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:54.439269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:24.760172+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":505,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.375389+0000","last_change":"2024-09-13T13:09:28.434007+0000","last_active":"2024-09-13T13:11:33.375389+0000","last_peered":"2024-09-13T13:11:33.375389+0000","last_clean":"2024-09-13T13:11:33.375389+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:11:33.375389+0000","last_undegraded":"2024-09-13T13:11:33.375389+0000","last_fullsized":"2024-09-13T13:11:33.375389+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:28.433899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.433899+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:03.236560+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":552,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904023+0000","last_change":"2024-09-13T13:10:42.010918+0000","last_active":"2024-09-13T13:11:22.904023+0000","last_peered":"2024-09-13T13:11:22.904023+0000","last_clean":"2024-09-13T13:11:22.904023+0000","last_became_active":"2024-09-13T13:04:02.762940+0000","last_became_peered":"2024-09-13T13:04:02.762940+0000","last_unstale":"2024-09-13T13:11:22.904023+0000","last_undegraded":"2024-09-13T13:11:22.904023+0000","last_fullsized":"2024-09-13T13:11:22.904023+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:10:42.010847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.010847+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.459829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"276'7309","reported_seq":7390,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.745681+0000","last_change":"2024-09-13T13:10:09.272561+0000","last_active":"2024-09-13T13:11:33.745681+0000","last_peered":"2024-09-13T13:11:33.745681+0000","last_clean":"2024-09-13T13:11:33.745681+0000","last_became_active":"2024-09-13T13:10:09.272264+0000","last_became_peered":"2024-09-13T13:10:09.272264+0000","last_unstale":"2024-09-13T13:11:33.745681+0000","last_undegraded":"2024-09-13T13:11:33.745681+0000","last_fullsized":"2024-09-13T13:11:33.745681+0000","mapping_epoch":241,"log_start":"251'2600","ondisk_log_start":"251'2600","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:35.365520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479002624,"num_objects":7309,"num_object_clones":0,"num_object_copies":21927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7309,"num_write_kb":467776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":511,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.391437+0000","last_change":"2024-09-13T13:07:15.243958+0000","last_active":"2024-09-13T13:11:33.391437+0000","last_peered":"2024-09-13T13:11:33.391437+0000","last_clean":"2024-09-13T13:11:33.391437+0000","last_became_active":"2024-09-13T13:01:04.198613+0000","last_became_peered":"2024-09-13T13:01:04.198613+0000","last_unstale":"2024-09-13T13:11:33.391437+0000","last_undegraded":"2024-09-13T13:11:33.391437+0000","last_fullsized":"2024-09-13T13:11:33.391437+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:07:15.243715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:15.243715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.652407+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":487,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.385006+0000","last_change":"2024-09-13T13:10:30.552177+0000","last_active":"2024-09-13T13:11:33.385006+0000","last_peered":"2024-09-13T13:11:33.385006+0000","last_clean":"2024-09-13T13:11:33.385006+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:11:33.385006+0000","last_undegraded":"2024-09-13T13:11:33.385006+0000","last_fullsized":"2024-09-13T13:11:33.385006+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:30.552113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.552113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:48.898081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":605,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:27.361665+0000","last_change":"2024-09-13T13:10:07.726810+0000","last_active":"2024-09-13T13:11:27.361665+0000","last_peered":"2024-09-13T13:11:27.361665+0000","last_clean":"2024-09-13T13:11:27.361665+0000","last_became_active":"2024-09-13T13:01:06.205815+0000","last_became_peered":"2024-09-13T13:01:06.205815+0000","last_unstale":"2024-09-13T13:11:27.361665+0000","last_undegraded":"2024-09-13T13:11:27.361665+0000","last_fullsized":"2024-09-13T13:11:27.361665+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:10:07.726751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:07.726751+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:32.746229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":465,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905066+0000","last_change":"2024-09-13T13:10:08.973968+0000","last_active":"2024-09-13T13:11:22.905066+0000","last_peered":"2024-09-13T13:11:22.905066+0000","last_clean":"2024-09-13T13:11:22.905066+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:11:22.905066+0000","last_undegraded":"2024-09-13T13:11:22.905066+0000","last_fullsized":"2024-09-13T13:11:22.905066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.973917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.973917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"275'7300","reported_seq":7379,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.051177+0000","last_change":"2024-09-13T13:10:09.273532+0000","last_active":"2024-09-13T13:11:33.051177+0000","last_peered":"2024-09-13T13:11:33.051177+0000","last_clean":"2024-09-13T13:11:33.051177+0000","last_became_active":"2024-09-13T13:10:09.273246+0000","last_became_peered":"2024-09-13T13:10:09.273246+0000","last_unstale":"2024-09-13T13:11:33.051177+0000","last_undegraded":"2024-09-13T13:11:33.051177+0000","last_fullsized":"2024-09-13T13:11:33.051177+0000","mapping_epoch":241,"log_start":"249'2400","ondisk_log_start":"249'2400","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4900,"log_dups_size":0,"ondisk_log_size":4900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:29.955904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478412800,"num_objects":7300,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7300,"num_write_kb":467200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":502,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.384629+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:11:33.384629+0000","last_peered":"2024-09-13T13:11:33.384629+0000","last_clean":"2024-09-13T13:11:33.384629+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:11:33.384629+0000","last_undegraded":"2024-09-13T13:11:33.384629+0000","last_fullsized":"2024-09-13T13:11:33.384629+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:26.731371+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":469,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.384455+0000","last_change":"2024-09-13T13:06:44.785235+0000","last_active":"2024-09-13T13:11:33.384455+0000","last_peered":"2024-09-13T13:11:33.384455+0000","last_clean":"2024-09-13T13:11:33.384455+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:11:33.384455+0000","last_undegraded":"2024-09-13T13:11:33.384455+0000","last_fullsized":"2024-09-13T13:11:33.384455+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:06:44.785017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.785017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.777133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":455,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895901+0000","last_change":"2024-09-13T13:05:29.213998+0000","last_active":"2024-09-13T13:11:22.895901+0000","last_peered":"2024-09-13T13:11:22.895901+0000","last_clean":"2024-09-13T13:11:22.895901+0000","last_became_active":"2024-09-13T13:01:06.208768+0000","last_became_peered":"2024-09-13T13:01:06.208768+0000","last_unstale":"2024-09-13T13:11:22.895901+0000","last_undegraded":"2024-09-13T13:11:22.895901+0000","last_fullsized":"2024-09-13T13:11:22.895901+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:26.143018+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":474,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895710+0000","last_change":"2024-09-13T13:11:10.104997+0000","last_active":"2024-09-13T13:11:22.895710+0000","last_peered":"2024-09-13T13:11:22.895710+0000","last_clean":"2024-09-13T13:11:22.895710+0000","last_became_active":"2024-09-13T13:01:08.204496+0000","last_became_peered":"2024-09-13T13:01:08.204496+0000","last_unstale":"2024-09-13T13:11:22.895710+0000","last_undegraded":"2024-09-13T13:11:22.895710+0000","last_fullsized":"2024-09-13T13:11:22.895710+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:10.104926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:10.104926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:33.972960+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"276'7122","reported_seq":7203,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.761205+0000","last_change":"2024-09-13T13:10:09.270421+0000","last_active":"2024-09-13T13:11:33.761205+0000","last_peered":"2024-09-13T13:11:33.761205+0000","last_clean":"2024-09-13T13:11:33.761205+0000","last_became_active":"2024-09-13T13:10:09.270129+0000","last_became_peered":"2024-09-13T13:10:09.270129+0000","last_unstale":"2024-09-13T13:11:33.761205+0000","last_undegraded":"2024-09-13T13:11:33.761205+0000","last_fullsized":"2024-09-13T13:11:33.761205+0000","mapping_epoch":241,"log_start":"249'2400","ondisk_log_start":"249'2400","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4722,"log_dups_size":0,"ondisk_log_size":4722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:37.119106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466747392,"num_objects":7122,"num_object_clones":0,"num_object_copies":21366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7122,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":536,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895925+0000","last_change":"2024-09-13T13:11:17.167120+0000","last_active":"2024-09-13T13:11:22.895925+0000","last_peered":"2024-09-13T13:11:22.895925+0000","last_clean":"2024-09-13T13:11:22.895925+0000","last_became_active":"2024-09-13T13:04:02.772405+0000","last_became_peered":"2024-09-13T13:04:02.772405+0000","last_unstale":"2024-09-13T13:11:22.895925+0000","last_undegraded":"2024-09-13T13:11:22.895925+0000","last_fullsized":"2024-09-13T13:11:22.895925+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:11:17.167044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:17.167044+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.955826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":455,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.375645+0000","last_change":"2024-09-13T13:03:22.498759+0000","last_active":"2024-09-13T13:11:33.375645+0000","last_peered":"2024-09-13T13:11:33.375645+0000","last_clean":"2024-09-13T13:11:33.375645+0000","last_became_active":"2024-09-13T13:01:02.187495+0000","last_became_peered":"2024-09-13T13:01:02.187495+0000","last_unstale":"2024-09-13T13:11:33.375645+0000","last_undegraded":"2024-09-13T13:11:33.375645+0000","last_fullsized":"2024-09-13T13:11:33.375645+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:25.548394+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":599,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.384440+0000","last_change":"2024-09-13T13:11:06.526723+0000","last_active":"2024-09-13T13:11:33.384440+0000","last_peered":"2024-09-13T13:11:33.384440+0000","last_clean":"2024-09-13T13:11:33.384440+0000","last_became_active":"2024-09-13T13:01:06.210886+0000","last_became_peered":"2024-09-13T13:01:06.210886+0000","last_unstale":"2024-09-13T13:11:33.384440+0000","last_undegraded":"2024-09-13T13:11:33.384440+0000","last_fullsized":"2024-09-13T13:11:33.384440+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:11:06.526646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:06.526646+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:18.710007+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":460,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904190+0000","last_change":"2024-09-13T13:06:09.230872+0000","last_active":"2024-09-13T13:11:22.904190+0000","last_peered":"2024-09-13T13:11:22.904190+0000","last_clean":"2024-09-13T13:11:22.904190+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:11:22.904190+0000","last_undegraded":"2024-09-13T13:11:22.904190+0000","last_fullsized":"2024-09-13T13:11:22.904190+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:09.230824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.230824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:45.998027+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"275'7279","reported_seq":7358,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.110040+0000","last_change":"2024-09-13T13:10:09.273698+0000","last_active":"2024-09-13T13:11:33.110040+0000","last_peered":"2024-09-13T13:11:33.110040+0000","last_clean":"2024-09-13T13:11:33.110040+0000","last_became_active":"2024-09-13T13:10:09.273311+0000","last_became_peered":"2024-09-13T13:10:09.273311+0000","last_unstale":"2024-09-13T13:11:33.110040+0000","last_undegraded":"2024-09-13T13:11:33.110040+0000","last_fullsized":"2024-09-13T13:11:33.110040+0000","mapping_epoch":241,"log_start":"249'2000","ondisk_log_start":"249'2000","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":5279,"log_dups_size":0,"ondisk_log_size":5279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:29.006601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477036544,"num_objects":7279,"num_object_clones":0,"num_object_copies":21837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7279,"num_write_kb":465856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":507,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.391576+0000","last_change":"2024-09-13T13:10:40.651563+0000","last_active":"2024-09-13T13:11:33.391576+0000","last_peered":"2024-09-13T13:11:33.391576+0000","last_clean":"2024-09-13T13:11:33.391576+0000","last_became_active":"2024-09-13T13:01:04.198376+0000","last_became_peered":"2024-09-13T13:01:04.198376+0000","last_unstale":"2024-09-13T13:11:33.391576+0000","last_undegraded":"2024-09-13T13:11:33.391576+0000","last_fullsized":"2024-09-13T13:11:33.391576+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:02.404183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":478,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.908951+0000","last_change":"2024-09-13T13:10:03.895450+0000","last_active":"2024-09-13T13:11:22.908951+0000","last_peered":"2024-09-13T13:11:22.908951+0000","last_clean":"2024-09-13T13:11:22.908951+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:11:22.908951+0000","last_undegraded":"2024-09-13T13:11:22.908951+0000","last_fullsized":"2024-09-13T13:11:22.908951+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:03.895396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:03.895396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:33.888709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":604,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.379414+0000","last_change":"2024-09-13T13:10:10.134973+0000","last_active":"2024-09-13T13:11:33.379414+0000","last_peered":"2024-09-13T13:11:33.379414+0000","last_clean":"2024-09-13T13:11:33.379414+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:11:33.379414+0000","last_undegraded":"2024-09-13T13:11:33.379414+0000","last_fullsized":"2024-09-13T13:11:33.379414+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:10:10.134873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.134873+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:32.576287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":475,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.375506+0000","last_change":"2024-09-13T13:10:42.390672+0000","last_active":"2024-09-13T13:11:33.375506+0000","last_peered":"2024-09-13T13:11:33.375506+0000","last_clean":"2024-09-13T13:11:33.375506+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:11:33.375506+0000","last_undegraded":"2024-09-13T13:11:33.375506+0000","last_fullsized":"2024-09-13T13:11:33.375506+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:42.390473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.390473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:43.033158+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"275'7183","reported_seq":7262,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.057601+0000","last_change":"2024-09-13T13:10:09.270982+0000","last_active":"2024-09-13T13:11:33.057601+0000","last_peered":"2024-09-13T13:11:33.057601+0000","last_clean":"2024-09-13T13:11:33.057601+0000","last_became_active":"2024-09-13T13:10:09.270486+0000","last_became_peered":"2024-09-13T13:10:09.270486+0000","last_unstale":"2024-09-13T13:11:33.057601+0000","last_undegraded":"2024-09-13T13:11:33.057601+0000","last_fullsized":"2024-09-13T13:11:33.057601+0000","mapping_epoch":241,"log_start":"249'2300","ondisk_log_start":"249'2300","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4883,"log_dups_size":0,"ondisk_log_size":4883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470745088,"num_objects":7183,"num_object_clones":0,"num_object_copies":21549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7183,"num_write_kb":459712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":534,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.379764+0000","last_change":"2024-09-13T13:11:18.533559+0000","last_active":"2024-09-13T13:11:33.379764+0000","last_peered":"2024-09-13T13:11:33.379764+0000","last_clean":"2024-09-13T13:11:33.379764+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:11:33.379764+0000","last_undegraded":"2024-09-13T13:11:33.379764+0000","last_fullsized":"2024-09-13T13:11:33.379764+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:11:18.533452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:18.533452+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:47.552700+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":457,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.391574+0000","last_change":"2024-09-13T13:09:38.867734+0000","last_active":"2024-09-13T13:11:33.391574+0000","last_peered":"2024-09-13T13:11:33.391574+0000","last_clean":"2024-09-13T13:11:33.391574+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:11:33.391574+0000","last_undegraded":"2024-09-13T13:11:33.391574+0000","last_fullsized":"2024-09-13T13:11:33.391574+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.867688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.867688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:57.738458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":422,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902634+0000","last_change":"2024-09-13T13:09:38.854654+0000","last_active":"2024-09-13T13:11:22.902634+0000","last_peered":"2024-09-13T13:11:22.902634+0000","last_clean":"2024-09-13T13:11:22.902634+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:11:22.902634+0000","last_undegraded":"2024-09-13T13:11:22.902634+0000","last_fullsized":"2024-09-13T13:11:22.902634+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.854581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.854581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:54.127592+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":472,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.386362+0000","last_change":"2024-09-13T13:06:43.969251+0000","last_active":"2024-09-13T13:11:33.386362+0000","last_peered":"2024-09-13T13:11:33.386362+0000","last_clean":"2024-09-13T13:11:33.386362+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:11:33.386362+0000","last_undegraded":"2024-09-13T13:11:33.386362+0000","last_fullsized":"2024-09-13T13:11:33.386362+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.969176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.969176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:23.798758+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"276'7500","reported_seq":7580,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:34.007563+0000","last_change":"2024-09-13T13:10:09.277094+0000","last_active":"2024-09-13T13:11:34.007563+0000","last_peered":"2024-09-13T13:11:34.007563+0000","last_clean":"2024-09-13T13:11:34.007563+0000","last_became_active":"2024-09-13T13:10:09.275394+0000","last_became_peered":"2024-09-13T13:10:09.275394+0000","last_unstale":"2024-09-13T13:11:34.007563+0000","last_undegraded":"2024-09-13T13:11:34.007563+0000","last_fullsized":"2024-09-13T13:11:34.007563+0000","mapping_epoch":241,"log_start":"246'1400","ondisk_log_start":"246'1400","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":6100,"log_dups_size":0,"ondisk_log_size":6100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:30.543594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491520000,"num_objects":7500,"num_object_clones":0,"num_object_copies":22500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7500,"num_write_kb":480000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"268'40","reported_seq":662,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.391476+0000","last_change":"2024-09-13T13:07:25.438610+0000","last_active":"2024-09-13T13:11:33.391476+0000","last_peered":"2024-09-13T13:11:33.391476+0000","last_clean":"2024-09-13T13:11:33.391476+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:11:33.391476+0000","last_undegraded":"2024-09-13T13:11:33.391476+0000","last_fullsized":"2024-09-13T13:11:33.391476+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'28","last_scrub_stamp":"2024-09-13T13:07:25.438528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:07:25.438528+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:54.678903+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":482,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.896258+0000","last_change":"2024-09-13T13:07:18.888103+0000","last_active":"2024-09-13T13:11:22.896258+0000","last_peered":"2024-09-13T13:11:22.896258+0000","last_clean":"2024-09-13T13:11:22.896258+0000","last_became_active":"2024-09-13T13:04:02.764675+0000","last_became_peered":"2024-09-13T13:04:02.764675+0000","last_unstale":"2024-09-13T13:11:22.896258+0000","last_undegraded":"2024-09-13T13:11:22.896258+0000","last_fullsized":"2024-09-13T13:11:22.896258+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:18.888060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:18.888060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:56.046611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":473,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902291+0000","last_change":"2024-09-13T13:04:59.739126+0000","last_active":"2024-09-13T13:11:22.902291+0000","last_peered":"2024-09-13T13:11:22.902291+0000","last_clean":"2024-09-13T13:11:22.902291+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:11:22.902291+0000","last_undegraded":"2024-09-13T13:11:22.902291+0000","last_fullsized":"2024-09-13T13:11:22.902291+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:04:59.739079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:04:59.739079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:54.110286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":470,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.377741+0000","last_change":"2024-09-13T13:11:11.413254+0000","last_active":"2024-09-13T13:11:33.377741+0000","last_peered":"2024-09-13T13:11:33.377741+0000","last_clean":"2024-09-13T13:11:33.377741+0000","last_became_active":"2024-09-13T13:01:06.215839+0000","last_became_peered":"2024-09-13T13:01:06.215839+0000","last_unstale":"2024-09-13T13:11:33.377741+0000","last_undegraded":"2024-09-13T13:11:33.377741+0000","last_fullsized":"2024-09-13T13:11:33.377741+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:11.413050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.413050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:12.384938+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":467,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.382836+0000","last_change":"2024-09-13T13:10:42.489307+0000","last_active":"2024-09-13T13:11:33.382836+0000","last_peered":"2024-09-13T13:11:33.382836+0000","last_clean":"2024-09-13T13:11:33.382836+0000","last_became_active":"2024-09-13T13:01:08.203140+0000","last_became_peered":"2024-09-13T13:01:08.203140+0000","last_unstale":"2024-09-13T13:11:33.382836+0000","last_undegraded":"2024-09-13T13:11:33.382836+0000","last_fullsized":"2024-09-13T13:11:33.382836+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:42.489264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.489264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:02.930639+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"275'7300","reported_seq":7379,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.076398+0000","last_change":"2024-09-13T13:10:09.270343+0000","last_active":"2024-09-13T13:11:33.076398+0000","last_peered":"2024-09-13T13:11:33.076398+0000","last_clean":"2024-09-13T13:11:33.076398+0000","last_became_active":"2024-09-13T13:10:09.270073+0000","last_became_peered":"2024-09-13T13:10:09.270073+0000","last_unstale":"2024-09-13T13:11:33.076398+0000","last_undegraded":"2024-09-13T13:11:33.076398+0000","last_fullsized":"2024-09-13T13:11:33.076398+0000","mapping_epoch":241,"log_start":"249'2000","ondisk_log_start":"249'2000","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":5300,"log_dups_size":0,"ondisk_log_size":5300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:35.580323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478412800,"num_objects":7300,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7300,"num_write_kb":467200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":501,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.391742+0000","last_change":"2024-09-13T13:10:29.799315+0000","last_active":"2024-09-13T13:11:33.391742+0000","last_peered":"2024-09-13T13:11:33.391742+0000","last_clean":"2024-09-13T13:11:33.391742+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:11:33.391742+0000","last_undegraded":"2024-09-13T13:11:33.391742+0000","last_fullsized":"2024-09-13T13:11:33.391742+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:10:29.799185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:29.799185+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:36.850709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":492,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.389150+0000","last_change":"2024-09-13T13:08:56.771534+0000","last_active":"2024-09-13T13:11:33.389150+0000","last_peered":"2024-09-13T13:11:33.389150+0000","last_clean":"2024-09-13T13:11:33.389150+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:11:33.389150+0000","last_undegraded":"2024-09-13T13:11:33.389150+0000","last_fullsized":"2024-09-13T13:11:33.389150+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:56.771432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:56.771432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:30.760708+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":494,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.922665+0000","last_change":"2024-09-13T13:10:30.919483+0000","last_active":"2024-09-13T13:11:22.922665+0000","last_peered":"2024-09-13T13:11:22.922665+0000","last_clean":"2024-09-13T13:11:22.922665+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:11:22.922665+0000","last_undegraded":"2024-09-13T13:11:22.922665+0000","last_fullsized":"2024-09-13T13:11:22.922665+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:10:30.919398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.919398+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:56.559722+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":457,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.925049+0000","last_change":"2024-09-13T13:11:18.610942+0000","last_active":"2024-09-13T13:11:22.925049+0000","last_peered":"2024-09-13T13:11:22.925049+0000","last_clean":"2024-09-13T13:11:22.925049+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:11:22.925049+0000","last_undegraded":"2024-09-13T13:11:22.925049+0000","last_fullsized":"2024-09-13T13:11:22.925049+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:18.610714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:18.610714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:40.534665+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":468,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904884+0000","last_change":"2024-09-13T13:08:54.232191+0000","last_active":"2024-09-13T13:11:22.904884+0000","last_peered":"2024-09-13T13:11:22.904884+0000","last_clean":"2024-09-13T13:11:22.904884+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:11:22.904884+0000","last_undegraded":"2024-09-13T13:11:22.904884+0000","last_fullsized":"2024-09-13T13:11:22.904884+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:54.232130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:54.232130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:58.884996+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"276'7406","reported_seq":7487,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:34.011380+0000","last_change":"2024-09-13T13:10:09.737716+0000","last_active":"2024-09-13T13:11:34.011380+0000","last_peered":"2024-09-13T13:11:34.011380+0000","last_clean":"2024-09-13T13:11:34.011380+0000","last_became_active":"2024-09-13T13:10:09.737415+0000","last_became_peered":"2024-09-13T13:10:09.737415+0000","last_unstale":"2024-09-13T13:11:34.011380+0000","last_undegraded":"2024-09-13T13:11:34.011380+0000","last_fullsized":"2024-09-13T13:11:34.011380+0000","mapping_epoch":241,"log_start":"246'1400","ondisk_log_start":"246'1400","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485359616,"num_objects":7406,"num_object_clones":0,"num_object_copies":22218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7406,"num_write_kb":473984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":502,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.375471+0000","last_change":"2024-09-13T13:08:21.219398+0000","last_active":"2024-09-13T13:11:33.375471+0000","last_peered":"2024-09-13T13:11:33.375471+0000","last_clean":"2024-09-13T13:11:33.375471+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:11:33.375471+0000","last_undegraded":"2024-09-13T13:11:33.375471+0000","last_fullsized":"2024-09-13T13:11:33.375471+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:08:21.219300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:21.219300+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:55.256946+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":479,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.378782+0000","last_change":"2024-09-13T13:10:37.465330+0000","last_active":"2024-09-13T13:11:33.378782+0000","last_peered":"2024-09-13T13:11:33.378782+0000","last_clean":"2024-09-13T13:11:33.378782+0000","last_became_active":"2024-09-13T13:01:02.186034+0000","last_became_peered":"2024-09-13T13:01:02.186034+0000","last_unstale":"2024-09-13T13:11:33.378782+0000","last_undegraded":"2024-09-13T13:11:33.378782+0000","last_fullsized":"2024-09-13T13:11:33.378782+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:37.465261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:37.465261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:56.060539+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1103,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:31.307122+0000","last_change":"2024-09-13T13:08:49.264831+0000","last_active":"2024-09-13T13:11:31.307122+0000","last_peered":"2024-09-13T13:11:31.307122+0000","last_clean":"2024-09-13T13:11:31.307122+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:11:31.307122+0000","last_undegraded":"2024-09-13T13:11:31.307122+0000","last_fullsized":"2024-09-13T13:11:31.307122+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:08:49.264775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:49.264775+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:56.918802+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":624,"num_read_kb":619,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":445,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903345+0000","last_change":"2024-09-13T13:04:54.703851+0000","last_active":"2024-09-13T13:11:22.903345+0000","last_peered":"2024-09-13T13:11:22.903345+0000","last_clean":"2024-09-13T13:11:22.903345+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:11:22.903345+0000","last_undegraded":"2024-09-13T13:11:22.903345+0000","last_fullsized":"2024-09-13T13:11:22.903345+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:04:54.703771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:04:54.703771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:48.121624+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":476,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.391076+0000","last_change":"2024-09-13T13:10:32.720742+0000","last_active":"2024-09-13T13:11:33.391076+0000","last_peered":"2024-09-13T13:11:33.391076+0000","last_clean":"2024-09-13T13:11:33.391076+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:11:33.391076+0000","last_undegraded":"2024-09-13T13:11:33.391076+0000","last_fullsized":"2024-09-13T13:11:33.391076+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:41.718820+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"276'7352","reported_seq":7433,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.736146+0000","last_change":"2024-09-13T13:10:09.276701+0000","last_active":"2024-09-13T13:11:33.736146+0000","last_peered":"2024-09-13T13:11:33.736146+0000","last_clean":"2024-09-13T13:11:33.736146+0000","last_became_active":"2024-09-13T13:10:09.276418+0000","last_became_peered":"2024-09-13T13:10:09.276418+0000","last_unstale":"2024-09-13T13:11:33.736146+0000","last_undegraded":"2024-09-13T13:11:33.736146+0000","last_fullsized":"2024-09-13T13:11:33.736146+0000","mapping_epoch":241,"log_start":"251'2600","ondisk_log_start":"251'2600","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4752,"log_dups_size":0,"ondisk_log_size":4752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481820672,"num_objects":7352,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7352,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"268'30","reported_seq":490,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903265+0000","last_change":"2024-09-13T13:06:51.193749+0000","last_active":"2024-09-13T13:11:22.903265+0000","last_peered":"2024-09-13T13:11:22.903265+0000","last_clean":"2024-09-13T13:11:22.903265+0000","last_became_active":"2024-09-13T13:04:02.784741+0000","last_became_peered":"2024-09-13T13:04:02.784741+0000","last_unstale":"2024-09-13T13:11:22.903265+0000","last_undegraded":"2024-09-13T13:11:22.903265+0000","last_fullsized":"2024-09-13T13:11:22.903265+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'18","last_scrub_stamp":"2024-09-13T13:06:51.193678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.193678+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:20.170952+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":475,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.384641+0000","last_change":"2024-09-13T13:06:10.715371+0000","last_active":"2024-09-13T13:11:33.384641+0000","last_peered":"2024-09-13T13:11:33.384641+0000","last_clean":"2024-09-13T13:11:33.384641+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:11:33.384641+0000","last_undegraded":"2024-09-13T13:11:33.384641+0000","last_fullsized":"2024-09-13T13:11:33.384641+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:10.715313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:10.715313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.480796+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":518,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903607+0000","last_change":"2024-09-13T13:06:39.564599+0000","last_active":"2024-09-13T13:11:22.903607+0000","last_peered":"2024-09-13T13:11:22.903607+0000","last_clean":"2024-09-13T13:11:22.903607+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:11:22.903607+0000","last_undegraded":"2024-09-13T13:11:22.903607+0000","last_fullsized":"2024-09-13T13:11:22.903607+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:06:39.564549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:06:39.564549+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":470,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904825+0000","last_change":"2024-09-13T13:08:25.500731+0000","last_active":"2024-09-13T13:11:22.904825+0000","last_peered":"2024-09-13T13:11:22.904825+0000","last_clean":"2024-09-13T13:11:22.904825+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:11:22.904825+0000","last_undegraded":"2024-09-13T13:11:22.904825+0000","last_fullsized":"2024-09-13T13:11:22.904825+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:25.500660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:25.500660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.962133+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":473,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903218+0000","last_change":"2024-09-13T13:10:09.926369+0000","last_active":"2024-09-13T13:11:22.903218+0000","last_peered":"2024-09-13T13:11:22.903218+0000","last_clean":"2024-09-13T13:11:22.903218+0000","last_became_active":"2024-09-13T13:01:08.209050+0000","last_became_peered":"2024-09-13T13:01:08.209050+0000","last_unstale":"2024-09-13T13:11:22.903218+0000","last_undegraded":"2024-09-13T13:11:22.903218+0000","last_fullsized":"2024-09-13T13:11:22.903218+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:09.926319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.926319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:39.338826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"275'7358","reported_seq":7506,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.084005+0000","last_change":"2024-09-13T13:11:16.781686+0000","last_active":"2024-09-13T13:11:33.084005+0000","last_peered":"2024-09-13T13:11:33.084005+0000","last_clean":"2024-09-13T13:11:33.084005+0000","last_became_active":"2024-09-13T13:10:09.273226+0000","last_became_peered":"2024-09-13T13:10:09.273226+0000","last_unstale":"2024-09-13T13:11:33.084005+0000","last_undegraded":"2024-09-13T13:11:33.084005+0000","last_fullsized":"2024-09-13T13:11:33.084005+0000","mapping_epoch":241,"log_start":"251'2500","ondisk_log_start":"251'2500","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'6147","last_scrub_stamp":"2024-09-13T13:11:16.781526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:11:16.781526+0000","objects_scrubbed":6090,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:20.503549+0000","scrub_duration":1943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482213888,"num_objects":7358,"num_object_clones":0,"num_object_copies":22074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7358,"num_write_kb":470912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":508,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898671+0000","last_change":"2024-09-13T13:10:30.736738+0000","last_active":"2024-09-13T13:11:22.898671+0000","last_peered":"2024-09-13T13:11:22.898671+0000","last_clean":"2024-09-13T13:11:22.898671+0000","last_became_active":"2024-09-13T13:04:02.763327+0000","last_became_peered":"2024-09-13T13:04:02.763327+0000","last_unstale":"2024-09-13T13:11:22.898671+0000","last_undegraded":"2024-09-13T13:11:22.898671+0000","last_fullsized":"2024-09-13T13:11:22.898671+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:10:30.736543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.736543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.681138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":478,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.896091+0000","last_change":"2024-09-13T13:10:01.834881+0000","last_active":"2024-09-13T13:11:22.896091+0000","last_peered":"2024-09-13T13:11:22.896091+0000","last_clean":"2024-09-13T13:11:22.896091+0000","last_became_active":"2024-09-13T13:01:02.180042+0000","last_became_peered":"2024-09-13T13:01:02.180042+0000","last_unstale":"2024-09-13T13:11:22.896091+0000","last_undegraded":"2024-09-13T13:11:22.896091+0000","last_fullsized":"2024-09-13T13:11:22.896091+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:01.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:01.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.226771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":488,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902993+0000","last_change":"2024-09-13T13:11:13.010250+0000","last_active":"2024-09-13T13:11:22.902993+0000","last_peered":"2024-09-13T13:11:22.902993+0000","last_clean":"2024-09-13T13:11:22.902993+0000","last_became_active":"2024-09-13T13:04:02.753900+0000","last_became_peered":"2024-09-13T13:04:02.753900+0000","last_unstale":"2024-09-13T13:11:22.902993+0000","last_undegraded":"2024-09-13T13:11:22.902993+0000","last_fullsized":"2024-09-13T13:11:22.902993+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:13.010123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:13.010123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:33.715146+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":470,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.385155+0000","last_change":"2024-09-13T13:10:34.526736+0000","last_active":"2024-09-13T13:11:33.385155+0000","last_peered":"2024-09-13T13:11:33.385155+0000","last_clean":"2024-09-13T13:11:33.385155+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:11:33.385155+0000","last_undegraded":"2024-09-13T13:11:33.385155+0000","last_fullsized":"2024-09-13T13:11:33.385155+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:34.526666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:34.526666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.167713+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"276'7263","reported_seq":7344,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.568072+0000","last_change":"2024-09-13T13:10:09.737339+0000","last_active":"2024-09-13T13:11:33.568072+0000","last_peered":"2024-09-13T13:11:33.568072+0000","last_clean":"2024-09-13T13:11:33.568072+0000","last_became_active":"2024-09-13T13:10:09.736988+0000","last_became_peered":"2024-09-13T13:10:09.736988+0000","last_unstale":"2024-09-13T13:11:33.568072+0000","last_undegraded":"2024-09-13T13:11:33.568072+0000","last_fullsized":"2024-09-13T13:11:33.568072+0000","mapping_epoch":241,"log_start":"250'2500","ondisk_log_start":"250'2500","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475987968,"num_objects":7263,"num_object_clones":0,"num_object_copies":21789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7263,"num_write_kb":464832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":507,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898323+0000","last_change":"2024-09-13T13:05:34.166464+0000","last_active":"2024-09-13T13:11:22.898323+0000","last_peered":"2024-09-13T13:11:22.898323+0000","last_clean":"2024-09-13T13:11:22.898323+0000","last_became_active":"2024-09-13T13:01:04.194709+0000","last_became_peered":"2024-09-13T13:01:04.194709+0000","last_unstale":"2024-09-13T13:11:22.898323+0000","last_undegraded":"2024-09-13T13:11:22.898323+0000","last_fullsized":"2024-09-13T13:11:22.898323+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:22.122358+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902945+0000","last_change":"2024-09-13T13:10:10.787443+0000","last_active":"2024-09-13T13:11:22.902945+0000","last_peered":"2024-09-13T13:11:22.902945+0000","last_clean":"2024-09-13T13:11:22.902945+0000","last_became_active":"2024-09-13T13:04:02.764323+0000","last_became_peered":"2024-09-13T13:04:02.764323+0000","last_unstale":"2024-09-13T13:11:22.902945+0000","last_undegraded":"2024-09-13T13:11:22.902945+0000","last_fullsized":"2024-09-13T13:11:22.902945+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:10.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:39.817372+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":456,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.391135+0000","last_change":"2024-09-13T13:08:53.747370+0000","last_active":"2024-09-13T13:11:33.391135+0000","last_peered":"2024-09-13T13:11:33.391135+0000","last_clean":"2024-09-13T13:11:33.391135+0000","last_became_active":"2024-09-13T13:01:06.208253+0000","last_became_peered":"2024-09-13T13:01:06.208253+0000","last_unstale":"2024-09-13T13:11:33.391135+0000","last_undegraded":"2024-09-13T13:11:33.391135+0000","last_fullsized":"2024-09-13T13:11:33.391135+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.747302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.747302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:31.723272+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":458,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.923825+0000","last_change":"2024-09-13T13:08:03.015302+0000","last_active":"2024-09-13T13:11:22.923825+0000","last_peered":"2024-09-13T13:11:22.923825+0000","last_clean":"2024-09-13T13:11:22.923825+0000","last_became_active":"2024-09-13T13:01:08.203700+0000","last_became_peered":"2024-09-13T13:01:08.203700+0000","last_unstale":"2024-09-13T13:11:22.923825+0000","last_undegraded":"2024-09-13T13:11:22.923825+0000","last_fullsized":"2024-09-13T13:11:22.923825+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:03.015233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.015233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:26.850341+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"276'7143","reported_seq":7224,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.752557+0000","last_change":"2024-09-13T13:10:09.737741+0000","last_active":"2024-09-13T13:11:33.752557+0000","last_peered":"2024-09-13T13:11:33.752557+0000","last_clean":"2024-09-13T13:11:33.752557+0000","last_became_active":"2024-09-13T13:10:09.737415+0000","last_became_peered":"2024-09-13T13:10:09.737415+0000","last_unstale":"2024-09-13T13:11:33.752557+0000","last_undegraded":"2024-09-13T13:11:33.752557+0000","last_fullsized":"2024-09-13T13:11:33.752557+0000","mapping_epoch":241,"log_start":"251'2400","ondisk_log_start":"251'2400","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4743,"log_dups_size":0,"ondisk_log_size":4743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:35.194931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468123648,"num_objects":7143,"num_object_clones":0,"num_object_copies":21429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7143,"num_write_kb":457152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":523,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898171+0000","last_change":"2024-09-13T13:10:09.739258+0000","last_active":"2024-09-13T13:11:22.898171+0000","last_peered":"2024-09-13T13:11:22.898171+0000","last_clean":"2024-09-13T13:11:22.898171+0000","last_became_active":"2024-09-13T13:04:02.766357+0000","last_became_peered":"2024-09-13T13:04:02.766357+0000","last_unstale":"2024-09-13T13:11:22.898171+0000","last_undegraded":"2024-09-13T13:11:22.898171+0000","last_fullsized":"2024-09-13T13:11:22.898171+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:09.739184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.739184+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:23.869202+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":463,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.382809+0000","last_change":"2024-09-13T13:03:38.515673+0000","last_active":"2024-09-13T13:11:33.382809+0000","last_peered":"2024-09-13T13:11:33.382809+0000","last_clean":"2024-09-13T13:11:33.382809+0000","last_became_active":"2024-09-13T13:01:02.187742+0000","last_became_peered":"2024-09-13T13:01:02.187742+0000","last_unstale":"2024-09-13T13:11:33.382809+0000","last_undegraded":"2024-09-13T13:11:33.382809+0000","last_fullsized":"2024-09-13T13:11:33.382809+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:03:38.515455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:38.515455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:27.749993+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":589,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:27.361595+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:11:27.361595+0000","last_peered":"2024-09-13T13:11:27.361595+0000","last_clean":"2024-09-13T13:11:27.361595+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:11:27.361595+0000","last_undegraded":"2024-09-13T13:11:27.361595+0000","last_fullsized":"2024-09-13T13:11:27.361595+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:26.540591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":458,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898564+0000","last_change":"2024-09-13T13:11:05.084909+0000","last_active":"2024-09-13T13:11:22.898564+0000","last_peered":"2024-09-13T13:11:22.898564+0000","last_clean":"2024-09-13T13:11:22.898564+0000","last_became_active":"2024-09-13T13:01:08.212717+0000","last_became_peered":"2024-09-13T13:01:08.212717+0000","last_unstale":"2024-09-13T13:11:22.898564+0000","last_undegraded":"2024-09-13T13:11:22.898564+0000","last_fullsized":"2024-09-13T13:11:22.898564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:05.084772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:05.084772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.857054+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"275'7274","reported_seq":7418,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:32.917289+0000","last_change":"2024-09-13T13:11:11.072082+0000","last_active":"2024-09-13T13:11:32.917289+0000","last_peered":"2024-09-13T13:11:32.917289+0000","last_clean":"2024-09-13T13:11:32.917289+0000","last_became_active":"2024-09-13T13:10:09.274589+0000","last_became_peered":"2024-09-13T13:10:09.274589+0000","last_unstale":"2024-09-13T13:11:32.917289+0000","last_undegraded":"2024-09-13T13:11:32.917289+0000","last_fullsized":"2024-09-13T13:11:32.917289+0000","mapping_epoch":241,"log_start":"249'2000","ondisk_log_start":"249'2000","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'5724","last_scrub_stamp":"2024-09-13T13:11:11.071957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.071957+0000","objects_scrubbed":5659,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:14.421303+0000","scrub_duration":1921,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476708864,"num_objects":7274,"num_object_clones":0,"num_object_copies":21822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7274,"num_write_kb":465536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":475,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.379639+0000","last_change":"2024-09-13T13:03:16.475874+0000","last_active":"2024-09-13T13:11:33.379639+0000","last_peered":"2024-09-13T13:11:33.379639+0000","last_clean":"2024-09-13T13:11:33.379639+0000","last_became_active":"2024-09-13T13:01:04.192576+0000","last_became_peered":"2024-09-13T13:01:04.192576+0000","last_unstale":"2024-09-13T13:11:33.379639+0000","last_undegraded":"2024-09-13T13:11:33.379639+0000","last_fullsized":"2024-09-13T13:11:33.379639+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:20.576847+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":496,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902294+0000","last_change":"2024-09-13T13:10:41.028120+0000","last_active":"2024-09-13T13:11:22.902294+0000","last_peered":"2024-09-13T13:11:22.902294+0000","last_clean":"2024-09-13T13:11:22.902294+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:11:22.902294+0000","last_undegraded":"2024-09-13T13:11:22.902294+0000","last_fullsized":"2024-09-13T13:11:22.902294+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:41.028047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:41.028047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:06.786834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":470,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.922848+0000","last_change":"2024-09-13T13:07:11.400066+0000","last_active":"2024-09-13T13:11:22.922848+0000","last_peered":"2024-09-13T13:11:22.922848+0000","last_clean":"2024-09-13T13:11:22.922848+0000","last_became_active":"2024-09-13T13:01:06.207070+0000","last_became_peered":"2024-09-13T13:01:06.207070+0000","last_unstale":"2024-09-13T13:11:22.922848+0000","last_undegraded":"2024-09-13T13:11:22.922848+0000","last_fullsized":"2024-09-13T13:11:22.922848+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:11.399840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:11.399840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:45.908383+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":458,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.378696+0000","last_change":"2024-09-13T13:11:10.432159+0000","last_active":"2024-09-13T13:11:33.378696+0000","last_peered":"2024-09-13T13:11:33.378696+0000","last_clean":"2024-09-13T13:11:33.378696+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:11:33.378696+0000","last_undegraded":"2024-09-13T13:11:33.378696+0000","last_fullsized":"2024-09-13T13:11:33.378696+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:10.432086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:10.432086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:13.775919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"275'7328","reported_seq":7407,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.100134+0000","last_change":"2024-09-13T13:10:09.271210+0000","last_active":"2024-09-13T13:11:33.100134+0000","last_peered":"2024-09-13T13:11:33.100134+0000","last_clean":"2024-09-13T13:11:33.100134+0000","last_became_active":"2024-09-13T13:10:09.270909+0000","last_became_peered":"2024-09-13T13:10:09.270909+0000","last_unstale":"2024-09-13T13:11:33.100134+0000","last_undegraded":"2024-09-13T13:11:33.100134+0000","last_fullsized":"2024-09-13T13:11:33.100134+0000","mapping_epoch":241,"log_start":"249'2000","ondisk_log_start":"249'2000","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":5328,"log_dups_size":0,"ondisk_log_size":5328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:27.029894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480247808,"num_objects":7328,"num_object_clones":0,"num_object_copies":21984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7328,"num_write_kb":468992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":481,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902736+0000","last_change":"2024-09-13T13:09:32.849360+0000","last_active":"2024-09-13T13:11:22.902736+0000","last_peered":"2024-09-13T13:11:22.902736+0000","last_clean":"2024-09-13T13:11:22.902736+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:11:22.902736+0000","last_undegraded":"2024-09-13T13:11:22.902736+0000","last_fullsized":"2024-09-13T13:11:22.902736+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:09:32.849277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:32.849277+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:01.179140+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":488,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905694+0000","last_change":"2024-09-13T13:10:27.802685+0000","last_active":"2024-09-13T13:11:22.905694+0000","last_peered":"2024-09-13T13:11:22.905694+0000","last_clean":"2024-09-13T13:11:22.905694+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:11:22.905694+0000","last_undegraded":"2024-09-13T13:11:22.905694+0000","last_fullsized":"2024-09-13T13:11:22.905694+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:27.802621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:27.802621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:40.450332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":470,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.923104+0000","last_change":"2024-09-13T13:08:26.033729+0000","last_active":"2024-09-13T13:11:22.923104+0000","last_peered":"2024-09-13T13:11:22.923104+0000","last_clean":"2024-09-13T13:11:22.923104+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:11:22.923104+0000","last_undegraded":"2024-09-13T13:11:22.923104+0000","last_fullsized":"2024-09-13T13:11:22.923104+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:26.033663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:26.033663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:55.961768+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":465,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903520+0000","last_change":"2024-09-13T13:10:40.961803+0000","last_active":"2024-09-13T13:11:22.903520+0000","last_peered":"2024-09-13T13:11:22.903520+0000","last_clean":"2024-09-13T13:11:22.903520+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:11:22.903520+0000","last_undegraded":"2024-09-13T13:11:22.903520+0000","last_fullsized":"2024-09-13T13:11:22.903520+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:40.961744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.961744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:59.893697+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"276'7315","reported_seq":7463,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.618879+0000","last_change":"2024-09-13T13:11:15.826994+0000","last_active":"2024-09-13T13:11:33.618879+0000","last_peered":"2024-09-13T13:11:33.618879+0000","last_clean":"2024-09-13T13:11:33.618879+0000","last_became_active":"2024-09-13T13:10:09.270190+0000","last_became_peered":"2024-09-13T13:10:09.270190+0000","last_unstale":"2024-09-13T13:11:33.618879+0000","last_undegraded":"2024-09-13T13:11:33.618879+0000","last_fullsized":"2024-09-13T13:11:33.618879+0000","mapping_epoch":241,"log_start":"250'2500","ondisk_log_start":"250'2500","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'6035","last_scrub_stamp":"2024-09-13T13:11:15.826744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:11:15.826744+0000","objects_scrubbed":5970,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:31.364522+0000","scrub_duration":1936,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479395840,"num_objects":7315,"num_object_clones":0,"num_object_copies":21945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7315,"num_write_kb":468160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":490,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.895996+0000","last_change":"2024-09-13T13:09:30.614842+0000","last_active":"2024-09-13T13:11:22.895996+0000","last_peered":"2024-09-13T13:11:22.895996+0000","last_clean":"2024-09-13T13:11:22.895996+0000","last_became_active":"2024-09-13T13:01:04.195665+0000","last_became_peered":"2024-09-13T13:01:04.195665+0000","last_unstale":"2024-09-13T13:11:22.895996+0000","last_undegraded":"2024-09-13T13:11:22.895996+0000","last_fullsized":"2024-09-13T13:11:22.895996+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:09:30.614748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.614748+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:03.028024+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":477,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905490+0000","last_change":"2024-09-13T13:11:18.821936+0000","last_active":"2024-09-13T13:11:22.905490+0000","last_peered":"2024-09-13T13:11:22.905490+0000","last_clean":"2024-09-13T13:11:22.905490+0000","last_became_active":"2024-09-13T13:01:02.186934+0000","last_became_peered":"2024-09-13T13:01:02.186934+0000","last_unstale":"2024-09-13T13:11:22.905490+0000","last_undegraded":"2024-09-13T13:11:22.905490+0000","last_fullsized":"2024-09-13T13:11:22.905490+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:18.821868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:18.821868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:40.874513+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.923042+0000","last_change":"2024-09-13T13:06:42.450669+0000","last_active":"2024-09-13T13:11:22.923042+0000","last_peered":"2024-09-13T13:11:22.923042+0000","last_clean":"2024-09-13T13:11:22.923042+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:11:22.923042+0000","last_undegraded":"2024-09-13T13:11:22.923042+0000","last_fullsized":"2024-09-13T13:11:22.923042+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:42.450598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:42.450598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":451,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.382498+0000","last_change":"2024-09-13T13:11:17.667223+0000","last_active":"2024-09-13T13:11:33.382498+0000","last_peered":"2024-09-13T13:11:33.382498+0000","last_clean":"2024-09-13T13:11:33.382498+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:11:33.382498+0000","last_undegraded":"2024-09-13T13:11:33.382498+0000","last_fullsized":"2024-09-13T13:11:33.382498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:17.667151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:17.667151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:46.207860+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"276'7285","reported_seq":7366,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:34.006905+0000","last_change":"2024-09-13T13:10:09.275649+0000","last_active":"2024-09-13T13:11:34.006905+0000","last_peered":"2024-09-13T13:11:34.006905+0000","last_clean":"2024-09-13T13:11:34.006905+0000","last_became_active":"2024-09-13T13:10:09.275320+0000","last_became_peered":"2024-09-13T13:10:09.275320+0000","last_unstale":"2024-09-13T13:11:34.006905+0000","last_undegraded":"2024-09-13T13:11:34.006905+0000","last_fullsized":"2024-09-13T13:11:34.006905+0000","mapping_epoch":241,"log_start":"246'1200","ondisk_log_start":"246'1200","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":6085,"log_dups_size":0,"ondisk_log_size":6085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:21.621303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477429760,"num_objects":7285,"num_object_clones":0,"num_object_copies":21855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7285,"num_write_kb":466240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":519,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902945+0000","last_change":"2024-09-13T13:09:33.843654+0000","last_active":"2024-09-13T13:11:22.902945+0000","last_peered":"2024-09-13T13:11:22.902945+0000","last_clean":"2024-09-13T13:11:22.902945+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:11:22.902945+0000","last_undegraded":"2024-09-13T13:11:22.902945+0000","last_fullsized":"2024-09-13T13:11:22.902945+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:09:33.843579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:33.843579+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:46.077623+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":481,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905331+0000","last_change":"2024-09-13T13:06:51.627764+0000","last_active":"2024-09-13T13:11:22.905331+0000","last_peered":"2024-09-13T13:11:22.905331+0000","last_clean":"2024-09-13T13:11:22.905331+0000","last_became_active":"2024-09-13T13:01:02.183027+0000","last_became_peered":"2024-09-13T13:01:02.183027+0000","last_unstale":"2024-09-13T13:11:22.905331+0000","last_undegraded":"2024-09-13T13:11:22.905331+0000","last_fullsized":"2024-09-13T13:11:22.905331+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:06:51.627691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.627691+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":613,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:27.361475+0000","last_change":"2024-09-13T13:11:11.751140+0000","last_active":"2024-09-13T13:11:27.361475+0000","last_peered":"2024-09-13T13:11:27.361475+0000","last_clean":"2024-09-13T13:11:27.361475+0000","last_became_active":"2024-09-13T13:04:02.759214+0000","last_became_peered":"2024-09-13T13:04:02.759214+0000","last_unstale":"2024-09-13T13:11:27.361475+0000","last_undegraded":"2024-09-13T13:11:27.361475+0000","last_fullsized":"2024-09-13T13:11:27.361475+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:11:11.751078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.751078+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.052505+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":468,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902024+0000","last_change":"2024-09-13T13:10:42.034653+0000","last_active":"2024-09-13T13:11:22.902024+0000","last_peered":"2024-09-13T13:11:22.902024+0000","last_clean":"2024-09-13T13:11:22.902024+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:11:22.902024+0000","last_undegraded":"2024-09-13T13:11:22.902024+0000","last_fullsized":"2024-09-13T13:11:22.902024+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:42.034601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.034601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.440553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":438,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902836+0000","last_change":"2024-09-13T13:07:10.058616+0000","last_active":"2024-09-13T13:11:22.902836+0000","last_peered":"2024-09-13T13:11:22.902836+0000","last_clean":"2024-09-13T13:11:22.902836+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:11:22.902836+0000","last_undegraded":"2024-09-13T13:11:22.902836+0000","last_fullsized":"2024-09-13T13:11:22.902836+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:10.058555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:10.058555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:50.118617+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":477,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.383927+0000","last_change":"2024-09-13T13:09:37.358795+0000","last_active":"2024-09-13T13:11:33.383927+0000","last_peered":"2024-09-13T13:11:33.383927+0000","last_clean":"2024-09-13T13:11:33.383927+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:11:33.383927+0000","last_undegraded":"2024-09-13T13:11:33.383927+0000","last_fullsized":"2024-09-13T13:11:33.383927+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:09:37.358710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:37.358710+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:55.589722+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":477,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902712+0000","last_change":"2024-09-13T13:11:12.054173+0000","last_active":"2024-09-13T13:11:22.902712+0000","last_peered":"2024-09-13T13:11:22.902712+0000","last_clean":"2024-09-13T13:11:22.902712+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:11:22.902712+0000","last_undegraded":"2024-09-13T13:11:22.902712+0000","last_fullsized":"2024-09-13T13:11:22.902712+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:12.054125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:12.054125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.049869+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":476,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904782+0000","last_change":"2024-09-13T13:09:30.959919+0000","last_active":"2024-09-13T13:11:22.904782+0000","last_peered":"2024-09-13T13:11:22.904782+0000","last_clean":"2024-09-13T13:11:22.904782+0000","last_became_active":"2024-09-13T13:04:02.751938+0000","last_became_peered":"2024-09-13T13:04:02.751938+0000","last_unstale":"2024-09-13T13:11:22.904782+0000","last_undegraded":"2024-09-13T13:11:22.904782+0000","last_fullsized":"2024-09-13T13:11:22.904782+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.959870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.959870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:01.787830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":483,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.375555+0000","last_change":"2024-09-13T13:09:34.426362+0000","last_active":"2024-09-13T13:11:33.375555+0000","last_peered":"2024-09-13T13:11:33.375555+0000","last_clean":"2024-09-13T13:11:33.375555+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:11:33.375555+0000","last_undegraded":"2024-09-13T13:11:33.375555+0000","last_fullsized":"2024-09-13T13:11:33.375555+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:34.426272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:34.426272+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:59.241784+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":464,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904180+0000","last_change":"2024-09-13T13:05:01.425215+0000","last_active":"2024-09-13T13:11:22.904180+0000","last_peered":"2024-09-13T13:11:22.904180+0000","last_clean":"2024-09-13T13:11:22.904180+0000","last_became_active":"2024-09-13T13:04:02.759083+0000","last_became_peered":"2024-09-13T13:04:02.759083+0000","last_unstale":"2024-09-13T13:11:22.904180+0000","last_undegraded":"2024-09-13T13:11:22.904180+0000","last_fullsized":"2024-09-13T13:11:22.904180+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:01.425165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:01.425165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.986182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":463,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.383471+0000","last_change":"2024-09-13T13:11:11.553996+0000","last_active":"2024-09-13T13:11:33.383471+0000","last_peered":"2024-09-13T13:11:33.383471+0000","last_clean":"2024-09-13T13:11:33.383471+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:11:33.383471+0000","last_undegraded":"2024-09-13T13:11:33.383471+0000","last_fullsized":"2024-09-13T13:11:33.383471+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:11.553908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.553908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:41.237817+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":474,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.923246+0000","last_change":"2024-09-13T13:10:34.954684+0000","last_active":"2024-09-13T13:11:22.923246+0000","last_peered":"2024-09-13T13:11:22.923246+0000","last_clean":"2024-09-13T13:11:22.923246+0000","last_became_active":"2024-09-13T13:01:08.205358+0000","last_became_peered":"2024-09-13T13:01:08.205358+0000","last_unstale":"2024-09-13T13:11:22.923246+0000","last_undegraded":"2024-09-13T13:11:22.923246+0000","last_fullsized":"2024-09-13T13:11:22.923246+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:34.954585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:34.954585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:47.029347+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":477,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905860+0000","last_change":"2024-09-13T13:09:48.990167+0000","last_active":"2024-09-13T13:11:22.905860+0000","last_peered":"2024-09-13T13:11:22.905860+0000","last_clean":"2024-09-13T13:11:22.905860+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:11:22.905860+0000","last_undegraded":"2024-09-13T13:11:22.905860+0000","last_fullsized":"2024-09-13T13:11:22.905860+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.057014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":462,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.376230+0000","last_change":"2024-09-13T13:06:41.308449+0000","last_active":"2024-09-13T13:11:33.376230+0000","last_peered":"2024-09-13T13:11:33.376230+0000","last_clean":"2024-09-13T13:11:33.376230+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:11:33.376230+0000","last_undegraded":"2024-09-13T13:11:33.376230+0000","last_fullsized":"2024-09-13T13:11:33.376230+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:41.308367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:41.308367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:29.533283+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":590,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.385566+0000","last_change":"2024-09-13T13:10:07.230375+0000","last_active":"2024-09-13T13:11:33.385566+0000","last_peered":"2024-09-13T13:11:33.385566+0000","last_clean":"2024-09-13T13:11:33.385566+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:11:33.385566+0000","last_undegraded":"2024-09-13T13:11:33.385566+0000","last_fullsized":"2024-09-13T13:11:33.385566+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:10:07.230276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:07.230276+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:32.849812+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":458,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.897717+0000","last_change":"2024-09-13T13:07:21.958114+0000","last_active":"2024-09-13T13:11:22.897717+0000","last_peered":"2024-09-13T13:11:22.897717+0000","last_clean":"2024-09-13T13:11:22.897717+0000","last_became_active":"2024-09-13T13:01:08.205836+0000","last_became_peered":"2024-09-13T13:01:08.205836+0000","last_unstale":"2024-09-13T13:11:22.897717+0000","last_undegraded":"2024-09-13T13:11:22.897717+0000","last_fullsized":"2024-09-13T13:11:22.897717+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:21.958030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:21.958030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:59.015030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":497,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.385917+0000","last_change":"2024-09-13T13:09:04.635198+0000","last_active":"2024-09-13T13:11:33.385917+0000","last_peered":"2024-09-13T13:11:33.385917+0000","last_clean":"2024-09-13T13:11:33.385917+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:11:33.385917+0000","last_undegraded":"2024-09-13T13:11:33.385917+0000","last_fullsized":"2024-09-13T13:11:33.385917+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:04.599040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:04.599040+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:52.519601+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":461,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.904436+0000","last_change":"2024-09-13T13:11:13.811691+0000","last_active":"2024-09-13T13:11:22.904436+0000","last_peered":"2024-09-13T13:11:22.904436+0000","last_clean":"2024-09-13T13:11:22.904436+0000","last_became_active":"2024-09-13T13:01:02.180412+0000","last_became_peered":"2024-09-13T13:01:02.180412+0000","last_unstale":"2024-09-13T13:11:22.904436+0000","last_undegraded":"2024-09-13T13:11:22.904436+0000","last_fullsized":"2024-09-13T13:11:22.904436+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:13.811631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:13.811631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.142292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":604,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:27.362106+0000","last_change":"2024-09-13T13:07:08.781005+0000","last_active":"2024-09-13T13:11:27.362106+0000","last_peered":"2024-09-13T13:11:27.362106+0000","last_clean":"2024-09-13T13:11:27.362106+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:11:27.362106+0000","last_undegraded":"2024-09-13T13:11:27.362106+0000","last_fullsized":"2024-09-13T13:11:27.362106+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:07:08.780952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.780952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:00.142438+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":458,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.375152+0000","last_change":"2024-09-13T13:09:36.414394+0000","last_active":"2024-09-13T13:11:33.375152+0000","last_peered":"2024-09-13T13:11:33.375152+0000","last_clean":"2024-09-13T13:11:33.375152+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:11:33.375152+0000","last_undegraded":"2024-09-13T13:11:33.375152+0000","last_fullsized":"2024-09-13T13:11:33.375152+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:36.414322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:36.414322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:48.346115+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":508,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.896012+0000","last_change":"2024-09-13T13:11:18.158667+0000","last_active":"2024-09-13T13:11:22.896012+0000","last_peered":"2024-09-13T13:11:22.896012+0000","last_clean":"2024-09-13T13:11:22.896012+0000","last_became_active":"2024-09-13T13:04:02.763561+0000","last_became_peered":"2024-09-13T13:04:02.763561+0000","last_unstale":"2024-09-13T13:11:22.896012+0000","last_undegraded":"2024-09-13T13:11:22.896012+0000","last_fullsized":"2024-09-13T13:11:22.896012+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:18.158584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:18.158584+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:40.118653+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":473,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.898385+0000","last_change":"2024-09-13T13:05:10.066345+0000","last_active":"2024-09-13T13:11:22.898385+0000","last_peered":"2024-09-13T13:11:22.898385+0000","last_clean":"2024-09-13T13:11:22.898385+0000","last_became_active":"2024-09-13T13:04:02.764021+0000","last_became_peered":"2024-09-13T13:04:02.764021+0000","last_unstale":"2024-09-13T13:11:22.898385+0000","last_undegraded":"2024-09-13T13:11:22.898385+0000","last_fullsized":"2024-09-13T13:11:22.898385+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:21.138701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":464,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.391353+0000","last_change":"2024-09-13T13:09:05.882862+0000","last_active":"2024-09-13T13:11:33.391353+0000","last_peered":"2024-09-13T13:11:33.391353+0000","last_clean":"2024-09-13T13:11:33.391353+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:11:33.391353+0000","last_undegraded":"2024-09-13T13:11:33.391353+0000","last_fullsized":"2024-09-13T13:11:33.391353+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:05.882769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.882769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:26.806407+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":459,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.383327+0000","last_change":"2024-09-13T13:08:59.585721+0000","last_active":"2024-09-13T13:11:33.383327+0000","last_peered":"2024-09-13T13:11:33.383327+0000","last_clean":"2024-09-13T13:11:33.383327+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:11:33.383327+0000","last_undegraded":"2024-09-13T13:11:33.383327+0000","last_fullsized":"2024-09-13T13:11:33.383327+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:59.585646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.585646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:51.544074+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":433,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.901883+0000","last_change":"2024-09-13T13:10:10.932948+0000","last_active":"2024-09-13T13:11:22.901883+0000","last_peered":"2024-09-13T13:11:22.901883+0000","last_clean":"2024-09-13T13:11:22.901883+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:11:22.901883+0000","last_undegraded":"2024-09-13T13:11:22.901883+0000","last_fullsized":"2024-09-13T13:11:22.901883+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:10.932889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.932889+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:33.657024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":469,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.905548+0000","last_change":"2024-09-13T13:09:07.118545+0000","last_active":"2024-09-13T13:11:22.905548+0000","last_peered":"2024-09-13T13:11:22.905548+0000","last_clean":"2024-09-13T13:11:22.905548+0000","last_became_active":"2024-09-13T13:01:02.181843+0000","last_became_peered":"2024-09-13T13:01:02.181843+0000","last_unstale":"2024-09-13T13:11:22.905548+0000","last_undegraded":"2024-09-13T13:11:22.905548+0000","last_fullsized":"2024-09-13T13:11:22.905548+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:07.118467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:07.118467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:59.839803+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":463,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.382574+0000","last_change":"2024-09-13T13:11:08.551743+0000","last_active":"2024-09-13T13:11:33.382574+0000","last_peered":"2024-09-13T13:11:33.382574+0000","last_clean":"2024-09-13T13:11:33.382574+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:11:33.382574+0000","last_undegraded":"2024-09-13T13:11:33.382574+0000","last_fullsized":"2024-09-13T13:11:33.382574+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:08.551449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:08.551449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:26.007369+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":466,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.375134+0000","last_change":"2024-09-13T13:10:39.477882+0000","last_active":"2024-09-13T13:11:33.375134+0000","last_peered":"2024-09-13T13:11:33.375134+0000","last_clean":"2024-09-13T13:11:33.375134+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:11:33.375134+0000","last_undegraded":"2024-09-13T13:11:33.375134+0000","last_fullsized":"2024-09-13T13:11:33.375134+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:39.477683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.477683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:04.956182+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":493,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.383683+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:11:33.383683+0000","last_peered":"2024-09-13T13:11:33.383683+0000","last_clean":"2024-09-13T13:11:33.383683+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:11:33.383683+0000","last_undegraded":"2024-09-13T13:11:33.383683+0000","last_fullsized":"2024-09-13T13:11:33.383683+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:23.701189+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":469,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902799+0000","last_change":"2024-09-13T13:08:50.517587+0000","last_active":"2024-09-13T13:11:22.902799+0000","last_peered":"2024-09-13T13:11:22.902799+0000","last_clean":"2024-09-13T13:11:22.902799+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:11:22.902799+0000","last_undegraded":"2024-09-13T13:11:22.902799+0000","last_fullsized":"2024-09-13T13:11:22.902799+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:50.517526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:50.517526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:20.085000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":472,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903110+0000","last_change":"2024-09-13T13:08:53.215325+0000","last_active":"2024-09-13T13:11:22.903110+0000","last_peered":"2024-09-13T13:11:22.903110+0000","last_clean":"2024-09-13T13:11:22.903110+0000","last_became_active":"2024-09-13T13:04:02.752155+0000","last_became_peered":"2024-09-13T13:04:02.752155+0000","last_unstale":"2024-09-13T13:11:22.903110+0000","last_undegraded":"2024-09-13T13:11:22.903110+0000","last_fullsized":"2024-09-13T13:11:22.903110+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.215263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.215263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:50.963832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":454,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.382728+0000","last_change":"2024-09-13T13:07:57.596272+0000","last_active":"2024-09-13T13:11:33.382728+0000","last_peered":"2024-09-13T13:11:33.382728+0000","last_clean":"2024-09-13T13:11:33.382728+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:11:33.382728+0000","last_undegraded":"2024-09-13T13:11:33.382728+0000","last_fullsized":"2024-09-13T13:11:33.382728+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:57.596097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:57.596097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:53.509678+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":495,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.384538+0000","last_change":"2024-09-13T13:09:39.368581+0000","last_active":"2024-09-13T13:11:33.384538+0000","last_peered":"2024-09-13T13:11:33.384538+0000","last_clean":"2024-09-13T13:11:33.384538+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:11:33.384538+0000","last_undegraded":"2024-09-13T13:11:33.384538+0000","last_fullsized":"2024-09-13T13:11:33.384538+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2024-09-13T13:09:39.368381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:39.368381+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:57.544832+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":486,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.376036+0000","last_change":"2024-09-13T13:11:04.275826+0000","last_active":"2024-09-13T13:11:33.376036+0000","last_peered":"2024-09-13T13:11:33.376036+0000","last_clean":"2024-09-13T13:11:33.376036+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:11:33.376036+0000","last_undegraded":"2024-09-13T13:11:33.376036+0000","last_fullsized":"2024-09-13T13:11:33.376036+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:04.275756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:04.275756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:29.145451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":370,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.902626+0000","last_change":"2024-09-13T13:10:36.918019+0000","last_active":"2024-09-13T13:11:22.902626+0000","last_peered":"2024-09-13T13:11:22.902626+0000","last_clean":"2024-09-13T13:11:22.902626+0000","last_became_active":"2024-09-13T13:04:02.764734+0000","last_became_peered":"2024-09-13T13:04:02.764734+0000","last_unstale":"2024-09-13T13:11:22.902626+0000","last_undegraded":"2024-09-13T13:11:22.902626+0000","last_fullsized":"2024-09-13T13:11:22.902626+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:36.917945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:36.917945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.082612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":473,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903165+0000","last_change":"2024-09-13T13:11:08.704496+0000","last_active":"2024-09-13T13:11:22.903165+0000","last_peered":"2024-09-13T13:11:22.903165+0000","last_clean":"2024-09-13T13:11:22.903165+0000","last_became_active":"2024-09-13T13:01:08.209935+0000","last_became_peered":"2024-09-13T13:01:08.209935+0000","last_unstale":"2024-09-13T13:11:22.903165+0000","last_undegraded":"2024-09-13T13:11:22.903165+0000","last_fullsized":"2024-09-13T13:11:22.903165+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:08.704450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:08.704450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.469645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":460,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.390544+0000","last_change":"2024-09-13T13:08:27.641188+0000","last_active":"2024-09-13T13:11:33.390544+0000","last_peered":"2024-09-13T13:11:33.390544+0000","last_clean":"2024-09-13T13:11:33.390544+0000","last_became_active":"2024-09-13T13:01:08.214479+0000","last_became_peered":"2024-09-13T13:01:08.214479+0000","last_unstale":"2024-09-13T13:11:33.390544+0000","last_undegraded":"2024-09-13T13:11:33.390544+0000","last_fullsized":"2024-09-13T13:11:33.390544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:27.641105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:27.641105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:25.778705+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":471,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.384891+0000","last_change":"2024-09-13T13:11:07.564094+0000","last_active":"2024-09-13T13:11:33.384891+0000","last_peered":"2024-09-13T13:11:33.384891+0000","last_clean":"2024-09-13T13:11:33.384891+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:11:33.384891+0000","last_undegraded":"2024-09-13T13:11:33.384891+0000","last_fullsized":"2024-09-13T13:11:33.384891+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:07.564025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:07.564025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:18.616035+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":468,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T13:11:22.903836+0000","last_change":"2024-09-13T13:06:09.928585+0000","last_active":"2024-09-13T13:11:22.903836+0000","last_peered":"2024-09-13T13:11:22.903836+0000","last_clean":"2024-09-13T13:11:22.903836+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:11:22.903836+0000","last_undegraded":"2024-09-13T13:11:22.903836+0000","last_fullsized":"2024-09-13T13:11:22.903836+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:06:09.928504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:09.928504+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:49.088075+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":471,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T13:11:33.375309+0000","last_change":"2024-09-13T13:09:38.337230+0000","last_active":"2024-09-13T13:11:33.375309+0000","last_peered":"2024-09-13T13:11:33.375309+0000","last_clean":"2024-09-13T13:11:33.375309+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:11:33.375309+0000","last_undegraded":"2024-09-13T13:11:33.375309+0000","last_fullsized":"2024-09-13T13:11:33.375309+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.337156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.337156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:50.333108+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7649165312,"num_objects":116717,"num_object_clones":0,"num_object_copies":350151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116717,"num_write_kb":7469888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22895001600,"data_stored":22895001600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82517,"ondisk_log_size":82517,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":681,"ondisk_log_size":681,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":624,"num_read_kb":619,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201409,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4522252,"kb_used_data":4180092,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":89222900,"statfs":{"total":95995035648,"available":91364249600,"internally_reserved":0,"allocated":4280414208,"data_stored":4279823092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":350353866},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141019,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1646780,"kb_used_data":1348332,"kb_used_omap":4,"kb_used_meta":298427,"kb_avail":92098372,"statfs":{"total":95995035648,"available":94308732928,"internally_reserved":0,"allocated":1380691968,"data_stored":1380115607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":305589933},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691981,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4577984,"kb_used_data":4209456,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":89167168,"statfs":{"total":95995035648,"available":91307180032,"internally_reserved":0,"allocated":4310482944,"data_stored":4309902412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920915,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1656148,"kb_used_data":1368132,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92089004,"statfs":{"total":95995035648,"available":94299140096,"internally_reserved":0,"allocated":1400967168,"data_stored":1400366292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":294910406},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117146,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4190104,"kb_used_data":3707720,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":89555048,"statfs":{"total":95995035648,"available":91704369152,"internally_reserved":0,"allocated":3796705280,"data_stored":3796101138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":493943239},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411485,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1720240,"kb_used_data":1387808,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":92024912,"statfs":{"total":95995035648,"available":94233509888,"internally_reserved":0,"allocated":1421115392,"data_stored":1420550584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":340392390},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4577448,"kb_used_data":4181144,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":89167704,"statfs":{"total":95995035648,"available":91307728896,"internally_reserved":0,"allocated":4281491456,"data_stored":4280870611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738539,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2223648,"kb_used_data":1885464,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":91521504,"statfs":{"total":95995035648,"available":93718020096,"internally_reserved":0,"allocated":1930715136,"data_stored":1930195837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1937178624,"data_stored":1937178624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4309516288,"data_stored":4309516288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1429536768,"data_stored":1429536768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3823763456,"data_stored":3823763456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4311089152,"data_stored":4311089152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1391329280,"data_stored":1391329280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4280745984,"data_stored":4280745984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:11:37.402 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-09-13T13:11:37.418 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:37.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:37.520 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:37.560 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:37.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:37.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:37.662 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:37.693 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:37.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:37.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:37.716 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:11:37.716 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:11:37.717 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:37 smithi052 ceph-mon[36682]: pgmap v509: 148 pgs: 148 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-13T13:11:37.717 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:37 smithi052 ceph-mon[36682]: from='client.35813 v1:172.21.15.52:0/556382654' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:11:37.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:37 smithi052 ceph-mon[29471]: pgmap v509: 148 pgs: 148 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-13T13:11:37.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:37 smithi052 ceph-mon[29471]: from='client.35813 v1:172.21.15.52:0/556382654' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:11:37.795 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:37.804 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 82 16 121749 121733 92.7487 75.75 0.00805197 0.0107771 2024-09-13T13:11:37.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:37 smithi167 ceph-mon[33178]: pgmap v509: 148 pgs: 148 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-13T13:11:37.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:37 smithi167 ceph-mon[33178]: from='client.35813 v1:172.21.15.52:0/556382654' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:11:37.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:37.840 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:37.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:37.942 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:37.974 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:37.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:38.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:38.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:38.110 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:38.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:38.211 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:38.246 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:38.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:38.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:38.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:38.384 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:38.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:38.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:38.486 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:38.518 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:38.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:38.590 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T13:11:38.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:38.680 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:38.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:38.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:38.804 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 83 16 123025 123009 92.5919 79.75 0.0193889 0.0107957 2024-09-13T13:11:38.814 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:38.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:38.916 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:38.949 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:38.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:38.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:39.051 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:39.082 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:39.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:39.152 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:39.168 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:39.182 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:39.210 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:39.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:39.243 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:39.311 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:39.349 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:39.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:39.451 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:39.492 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:39.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:39.576 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:39 smithi052 ceph-mon[29471]: pgmap v510: 148 pgs: 148 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-13T13:11:39.576 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:39 smithi052 ceph-mon[36682]: pgmap v510: 148 pgs: 148 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-13T13:11:39.594 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:39.638 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:39.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:39.740 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:39.779 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:39.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:39.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 84 16 124233 124217 92.3884 75.5 0.00687933 0.0108193 2024-09-13T13:11:39.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:39 smithi167 ceph-mon[33178]: pgmap v510: 148 pgs: 148 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-13T13:11:39.881 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:39.914 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:39.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:39.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:40.016 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:40.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:40.046 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:40.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:40.147 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:40.173 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:40.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:40.275 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:40.310 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:40.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:40.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:40.403 INFO:teuthology.orchestra.run.smithi052.stderr:set pool 1 pgp_num to 1 2024-09-13T13:11:40.412 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:40.436 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:40.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:40.538 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:40.560 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:40.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:40.662 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:40.697 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:40.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:40.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:40.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:40 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/575941239' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T13:11:40.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:40 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/575941239' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T13:11:40.801 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:40.804 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 85 16 125195 125179 92.0089 60.125 0.0153521 0.0108639 2024-09-13T13:11:40.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:40 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/575941239' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T13:11:40.823 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:40.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:40.924 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:40.968 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:40.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:40.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:41.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:41.101 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:41.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:41.136 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:11:41.203 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:41.244 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:41.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:41.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:41.346 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:41.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:41.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:41.475 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:41 smithi052 ceph-mon[29471]: pgmap v511: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-13T13:11:41.475 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:41 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/575941239' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-13T13:11:41.475 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:41 smithi052 ceph-mon[29471]: osdmap e277: 8 total, 8 up, 8 in 2024-09-13T13:11:41.475 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:41 smithi052 ceph-mon[36682]: pgmap v511: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-13T13:11:41.475 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:41 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/575941239' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-13T13:11:41.475 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:41 smithi052 ceph-mon[36682]: osdmap e277: 8 total, 8 up, 8 in 2024-09-13T13:11:41.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:41.516 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:41.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:41.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:41.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:41.647 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:41.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:41.748 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:41.785 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:41.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:41.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 86 16 126013 125997 91.5335 51.125 0.00989418 0.0109184 2024-09-13T13:11:41.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:41 smithi167 ceph-mon[33178]: pgmap v511: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-13T13:11:41.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:41 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/575941239' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-13T13:11:41.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:41 smithi167 ceph-mon[33178]: osdmap e277: 8 total, 8 up, 8 in 2024-09-13T13:11:41.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:41.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:41.912 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:41.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:42.013 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:42.049 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:42.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:42.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:42.151 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:42.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:42.199 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:42.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:42.300 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:42.342 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:11:42.342 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:11:42.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:42.346 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:42.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:42.447 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:42.475 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:42.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:42.577 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:42.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:42.601 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:42.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:42.702 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:42.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:42.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:42.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:42.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 87 15 126958 126943 91.161 59.125 0.0113036 0.0109653 2024-09-13T13:11:42.839 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:42.872 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:42.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:42.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:42.974 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:43.008 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:43.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:43.110 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:43.156 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:43.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:43.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:43.292 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:43.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:43.394 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:43.416 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:11:43.443 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:43.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:43.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:43.545 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:43.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:43.581 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:43.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:43.682 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:43.718 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:43.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:43.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:43 smithi052 ceph-mon[36682]: pgmap v513: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 605 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T13:11:43.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/286121770' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:11:43.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:43 smithi052 ceph-mon[29471]: pgmap v513: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 605 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T13:11:43.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/286121770' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:11:43.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 88 16 128237 128221 91.0327 79.875 0.0113058 0.0109802 2024-09-13T13:11:43.821 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:43 smithi167 ceph-mon[33178]: pgmap v513: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 605 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-13T13:11:43.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/286121770' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:11:43.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:43.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:43.958 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:43.996 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:43.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:43.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:44.098 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:44.133 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:44.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:44.235 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:44.267 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:44.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:44.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:44.369 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:44.403 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:44.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:44.458 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:11:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:11:44.506 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:44.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:44.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:44.634 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:44.675 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:44.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:44.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:44.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:44 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:44.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/286121770' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:11:44.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:44 smithi052 ceph-mon[36682]: osdmap e278: 8 total, 8 up, 8 in 2024-09-13T13:11:44.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:44 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:44.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/286121770' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:11:44.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:44 smithi052 ceph-mon[29471]: osdmap e278: 8 total, 8 up, 8 in 2024-09-13T13:11:44.777 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:44.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 89 16 129548 129532 90.9305 81.9375 0.0132607 0.0109932 2024-09-13T13:11:44.815 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:44.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:44.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:44.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:44 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:44.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/286121770' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:11:44.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:44 smithi167 ceph-mon[33178]: osdmap e278: 8 total, 8 up, 8 in 2024-09-13T13:11:44.917 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:44.945 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:44.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:44.956 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T13:11:45.002 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:45.047 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:45.078 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:45.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:45.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:45.180 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:45.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:45.243 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:45.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:45.344 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:45.398 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:45.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:45.500 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:45.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:45.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:45.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:45.632 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:45.687 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:45.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:45.694 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:45.705 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:11:45.705 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:11:45.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:45 smithi052 ceph-mon[29471]: pgmap v515: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-09-13T13:11:45.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:45 smithi052 ceph-mon[29471]: 3.5 scrub starts 2024-09-13T13:11:45.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:45 smithi052 ceph-mon[29471]: 3.5 scrub ok 2024-09-13T13:11:45.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:45 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:45.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:45 smithi052 ceph-mon[29471]: 6.0 scrub starts 2024-09-13T13:11:45.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:45 smithi052 ceph-mon[29471]: 6.0 scrub ok 2024-09-13T13:11:45.742 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:45 smithi052 ceph-mon[36682]: pgmap v515: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-09-13T13:11:45.742 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:45 smithi052 ceph-mon[36682]: 3.5 scrub starts 2024-09-13T13:11:45.742 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:45 smithi052 ceph-mon[36682]: 3.5 scrub ok 2024-09-13T13:11:45.742 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:45 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:45.743 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:45 smithi052 ceph-mon[36682]: 6.0 scrub starts 2024-09-13T13:11:45.743 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:45 smithi052 ceph-mon[36682]: 6.0 scrub ok 2024-09-13T13:11:45.745 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:11:45.745 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:11:45.777 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:11:45.778 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:11:45.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:45.794 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:11:45.794 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:11:45.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Total time run: 90.0085 2024-09-13T13:11:45.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Total writes made: 130666 2024-09-13T13:11:45.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Write size: 65536 2024-09-13T13:11:45.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Object size: 65536 2024-09-13T13:11:45.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Bandwidth (MB/sec): 90.7317 2024-09-13T13:11:45.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev Bandwidth: 32.1529 2024-09-13T13:11:45.805 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max bandwidth (MB/sec): 202.688 2024-09-13T13:11:45.806 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min bandwidth (MB/sec): 51.125 2024-09-13T13:11:45.806 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Average IOPS: 1451 2024-09-13T13:11:45.806 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev IOPS: 514.447 2024-09-13T13:11:45.806 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max IOPS: 3243 2024-09-13T13:11:45.806 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min IOPS: 818 2024-09-13T13:11:45.806 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Average Latency(s): 0.0110175 2024-09-13T13:11:45.806 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev Latency(s): 0.00874329 2024-09-13T13:11:45.806 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max latency(s): 0.270998 2024-09-13T13:11:45.806 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min latency(s): 0.00134564 2024-09-13T13:11:45.809 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:45.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:45.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:45.810 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T13:11:45.811 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:11:45.811 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:11:45.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:45 smithi167 ceph-mon[33178]: pgmap v515: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-09-13T13:11:45.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:45 smithi167 ceph-mon[33178]: 3.5 scrub starts 2024-09-13T13:11:45.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:45 smithi167 ceph-mon[33178]: 3.5 scrub ok 2024-09-13T13:11:45.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:45 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:45 smithi167 ceph-mon[33178]: 6.0 scrub starts 2024-09-13T13:11:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:45 smithi167 ceph-mon[33178]: 6.0 scrub ok 2024-09-13T13:11:45.848 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:11:45.849 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:11:45.882 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:11:45.883 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:11:45.911 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:45.918 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:11:45.919 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:11:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:45.948 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:45.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:46.049 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:46.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:46.090 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:46.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:46.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:46.230 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:46.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:46.321 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:11:46.331 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:46.368 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:46.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:46.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:46.472 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:11:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:11:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:11:46.503 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:46.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:46.605 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:46.636 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:46.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:46.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:46 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/934017622' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:46.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:46 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/934017622' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:46.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:46.784 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:46.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:46.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:46.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:46 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/934017622' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:46.886 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:46.917 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:11:46.918 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:11:46.918 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:11:46.918 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:11:46.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.918 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83958 7.b 7:d1e60c84:::benchmark_data_smithi052_71865_object83957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:11:46.918 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.524868+0000", 2024-09-13T13:11:46.918 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 512.38732295399996, 2024-09-13T13:11:46.918 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.019764474, 2024-09-13T13:11:46.918 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:46.918 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:46.918 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:11:46.919 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:46.919 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:46.919 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:46.919 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83958 2024-09-13T13:11:46.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.919 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:46.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:46.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524868+0000", 2024-09-13T13:11:46.919 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:46.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524868+0000", 2024-09-13T13:11:46.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:46.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524870+0000", 2024-09-13T13:11:46.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8929999999999999e-06 2024-09-13T13:11:46.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:46.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524893+0000", 2024-09-13T13:11:46.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2925000000000002e-05 2024-09-13T13:11:46.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:46.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524908+0000", 2024-09-13T13:11:46.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4839e-05 2024-09-13T13:11:46.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:46.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.524916+0000", 2024-09-13T13:11:46.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0539999999999995e-06 2024-09-13T13:11:46.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.922 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.922 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525086+0000", 2024-09-13T13:11:46.922 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016999100000000001 2024-09-13T13:11:46.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.922 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.922 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:46.922 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525228+0000", 2024-09-13T13:11:46.922 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014245600000000001 2024-09-13T13:11:46.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.922 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.922 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:46.922 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525369+0000", 2024-09-13T13:11:46.922 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014030100000000001 2024-09-13T13:11:46.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.923 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:46.923 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.532247+0000", 2024-09-13T13:11:46.923 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0068779770000000004 2024-09-13T13:11:46.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.923 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:46.923 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.532262+0000", 2024-09-13T13:11:46.923 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5274e-05 2024-09-13T13:11:46.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.924 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.924 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548202+0000", 2024-09-13T13:11:46.924 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015939813000000001 2024-09-13T13:11:46.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.924 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:46.924 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548214+0000", 2024-09-13T13:11:46.924 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.24e-05 2024-09-13T13:11:46.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.924 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.924 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.544509+0000", 2024-09-13T13:11:46.924 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99629462499999999 2024-09-13T13:11:46.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.925 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:46.925 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.544633+0000", 2024-09-13T13:11:46.925 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012392600000000001 2024-09-13T13:11:46.925 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.925 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:46.925 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.925 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83959 7.b 7:de17d8b6:::benchmark_data_smithi052_71865_object83958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:11:46.925 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.525528+0000", 2024-09-13T13:11:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 512.38666333799995, 2024-09-13T13:11:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0337471460000001, 2024-09-13T13:11:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83959 2024-09-13T13:11:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.926 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525528+0000", 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525528+0000", 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525529+0000", 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.248e-06 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.928 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:46.928 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525551+0000", 2024-09-13T13:11:46.928 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1900999999999999e-05 2024-09-13T13:11:46.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.928 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.928 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:46.928 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525560+0000", 2024-09-13T13:11:46.928 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1810000000000007e-06 2024-09-13T13:11:46.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.928 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.928 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:46.929 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.525568+0000", 2024-09-13T13:11:46.929 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7929999999999998e-06 2024-09-13T13:11:46.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.929 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.929 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.929 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548247+0000", 2024-09-13T13:11:46.929 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.022678957 2024-09-13T13:11:46.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.929 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.929 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:46.929 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548253+0000", 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.84e-06 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545053+0000", 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.996799993 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545157+0000", 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104693 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.930 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5", 2024-09-13T13:11:46.931 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545303+0000", 2024-09-13T13:11:46.931 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000145434 2024-09-13T13:11:46.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.931 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.931 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:46.931 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552563+0000", 2024-09-13T13:11:46.931 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0072600859999999998 2024-09-13T13:11:46.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.931 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.931 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:46.931 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559142+0000", 2024-09-13T13:11:46.931 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0065787509999999999 2024-09-13T13:11:46.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.932 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.932 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:46.932 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559190+0000", 2024-09-13T13:11:46.932 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7794999999999999e-05 2024-09-13T13:11:46.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.932 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.932 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:46.932 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559275+0000", 2024-09-13T13:11:46.932 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5474000000000002e-05 2024-09-13T13:11:46.932 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.932 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:46.932 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83968 7.b 7:d987446b:::benchmark_data_smithi052_71865_object83967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-13T13:11:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.545178+0000", 2024-09-13T13:11:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 512.36701290400003, 2024-09-13T13:11:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.014227582, 2024-09-13T13:11:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83968 2024-09-13T13:11:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545178+0000", 2024-09-13T13:11:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545178+0000", 2024-09-13T13:11:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545180+0000", 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.728e-06 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545203+0000", 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3269999999999999e-05 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545212+0000", 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5620000000000004e-06 2024-09-13T13:11:46.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.936 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:46.936 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.545218+0000", 2024-09-13T13:11:46.936 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7599999999999999e-06 2024-09-13T13:11:46.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.936 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.936 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548273+0000", 2024-09-13T13:11:46.936 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0030558460000000001 2024-09-13T13:11:46.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.936 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:46.936 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548279+0000", 2024-09-13T13:11:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7989999999999999e-06 2024-09-13T13:11:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.549411+0000", 2024-09-13T13:11:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0011321479999999 2024-09-13T13:11:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.549546+0000", 2024-09-13T13:11:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013478399999999999 2024-09-13T13:11:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.937 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5", 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.549734+0000", 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000187999 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555255+0000", 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055203860000000004 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559284+0000", 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0040296300000000002 2024-09-13T13:11:46.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559328+0000", 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3339e-05 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559406+0000", 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8331000000000006e-05 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.939 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.940 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83950 7.d 7:b84df720:::benchmark_data_smithi052_71865_object83949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:11:46.940 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:11:46.940 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 512.36426683599996, 2024-09-13T13:11:46.940 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99935902799999998, 2024-09-13T13:11:46.940 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:46.940 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:46.940 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:11:46.940 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:46.940 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:46.940 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:46.940 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83950 2024-09-13T13:11:46.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:46.941 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547926+0000", 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6199999999999999e-06 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547961+0000", 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4916999999999998e-05 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547971+0000", 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8789999999999998e-06 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.942 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:46.943 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547978+0000", 2024-09-13T13:11:46.943 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6120000000000001e-06 2024-09-13T13:11:46.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.943 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.943 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.943 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548150+0000", 2024-09-13T13:11:46.943 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017159 2024-09-13T13:11:46.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.943 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.943 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:46.943 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548159+0000", 2024-09-13T13:11:46.943 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7979999999999995e-06 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547139+0000", 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99898018700000002 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547283+0000", 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000144425 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.945 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.945 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83951 7.d 7:bd55e938:::benchmark_data_smithi052_71865_object83950:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:11:46.945 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.547994+0000", 2024-09-13T13:11:46.945 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 512.36419674800004, 2024-09-13T13:11:46.945 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0078699250000001, 2024-09-13T13:11:46.945 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:46.945 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:46.945 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:11:46.945 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:46.945 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:46.945 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:46.945 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83951 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547994+0000", 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547994+0000", 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.946 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:46.947 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547995+0000", 2024-09-13T13:11:46.947 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6899999999999996e-07 2024-09-13T13:11:46.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.947 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.947 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:46.947 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548019+0000", 2024-09-13T13:11:46.947 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3451999999999998e-05 2024-09-13T13:11:46.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.947 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.948 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:46.948 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548026+0000", 2024-09-13T13:11:46.948 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1600000000000001e-06 2024-09-13T13:11:46.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.948 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.948 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:46.948 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548032+0000", 2024-09-13T13:11:46.948 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8359999999999997e-06 2024-09-13T13:11:46.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.948 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.948 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.948 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548182+0000", 2024-09-13T13:11:46.949 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015005300000000001 2024-09-13T13:11:46.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.949 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.949 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:46.949 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548188+0000", 2024-09-13T13:11:46.949 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7869999999999998e-06 2024-09-13T13:11:46.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.949 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.949 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.949 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547487+0000", 2024-09-13T13:11:46.949 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99929913400000003 2024-09-13T13:11:46.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.949 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.950 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for missing object", 2024-09-13T13:11:46.950 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547629+0000", 2024-09-13T13:11:46.950 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000142106 2024-09-13T13:11:46.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.950 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.950 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.950 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555757+0000", 2024-09-13T13:11:46.950 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0081283139999999993 2024-09-13T13:11:46.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.950 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.950 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:46.950 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555864+0000", 2024-09-13T13:11:46.951 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010721399999999999 2024-09-13T13:11:46.951 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.951 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:46.951 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.951 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.951 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83978 7.d 7:b4a32006:::benchmark_data_smithi052_71865_object83977:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:11:46.951 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.562795+0000", 2024-09-13T13:11:46.951 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 512.34939662600004, 2024-09-13T13:11:46.951 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99338139199999997, 2024-09-13T13:11:46.951 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:46.951 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:46.951 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83978 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562795+0000", 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:46.952 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562795+0000", 2024-09-13T13:11:46.953 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.953 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.953 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.953 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:46.953 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562797+0000", 2024-09-13T13:11:46.953 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2309999999999999e-06 2024-09-13T13:11:46.953 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.953 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.953 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:46.953 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562824+0000", 2024-09-13T13:11:46.953 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6838999999999998e-05 2024-09-13T13:11:46.953 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.953 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562836+0000", 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2812e-05 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562845+0000", 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2320000000000001e-06 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.563048+0000", 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020349699999999999 2024-09-13T13:11:46.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.955 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.955 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:46.955 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.563058+0000", 2024-09-13T13:11:46.955 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5820000000000008e-06 2024-09-13T13:11:46.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.955 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.955 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.955 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.547871+0000", 2024-09-13T13:11:46.955 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98481351699999997 2024-09-13T13:11:46.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.955 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.955 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:46.955 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548017+0000", 2024-09-13T13:11:46.956 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014600999999999999 2024-09-13T13:11:46.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.956 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.956 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0", 2024-09-13T13:11:46.956 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.548480+0000", 2024-09-13T13:11:46.956 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00046313600000000001 2024-09-13T13:11:46.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.956 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.956 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:46.956 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555942+0000", 2024-09-13T13:11:46.956 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0074612120000000001 2024-09-13T13:11:46.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.956 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556034+0000", 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2584000000000007e-05 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556087+0000", 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3118999999999997e-05 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556176+0000", 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8621000000000004e-05 2024-09-13T13:11:46.957 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83979 7.b 7:d79e8e3c:::benchmark_data_smithi052_71865_object83978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.562864+0000", 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 512.34932686299999, 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99667409100000004, 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:46.958 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83979 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562864+0000", 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562864+0000", 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.959 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.960 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:46.960 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562865+0000", 2024-09-13T13:11:46.960 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0900000000000001e-07 2024-09-13T13:11:46.960 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.960 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.960 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:46.960 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562887+0000", 2024-09-13T13:11:46.960 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1977999999999999e-05 2024-09-13T13:11:46.960 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.960 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.960 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:46.960 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562897+0000", 2024-09-13T13:11:46.960 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8740000000000007e-06 2024-09-13T13:11:46.961 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.961 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.961 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:46.961 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562902+0000", 2024-09-13T13:11:46.961 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2780000000000001e-06 2024-09-13T13:11:46.961 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.961 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.961 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.961 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562985+0000", 2024-09-13T13:11:46.961 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3119000000000002e-05 2024-09-13T13:11:46.962 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.962 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.962 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:46.962 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.562994+0000", 2024-09-13T13:11:46.962 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1190000000000003e-06 2024-09-13T13:11:46.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.963 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.963 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.963 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552593+0000", 2024-09-13T13:11:46.963 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98959876899999999 2024-09-13T13:11:46.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.963 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.963 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:46.963 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552700+0000", 2024-09-13T13:11:46.963 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010694199999999999 2024-09-13T13:11:46.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.963 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.963 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5", 2024-09-13T13:11:46.964 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552844+0000", 2024-09-13T13:11:46.964 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014343099999999999 2024-09-13T13:11:46.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.964 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.964 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:46.964 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.555298+0000", 2024-09-13T13:11:46.964 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024543220000000001 2024-09-13T13:11:46.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.964 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.964 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:46.964 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559413+0000", 2024-09-13T13:11:46.964 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041153700000000001 2024-09-13T13:11:46.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.965 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.965 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:46.965 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559458+0000", 2024-09-13T13:11:46.965 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5105999999999998e-05 2024-09-13T13:11:46.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.965 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.965 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:46.965 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559538+0000", 2024-09-13T13:11:46.965 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0074000000000006e-05 2024-09-13T13:11:46.965 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.965 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:46.965 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.966 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.966 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83988 7.b 7:d1d56cdb:::benchmark_data_smithi052_71865_object83987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:11:46.966 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.571369+0000", 2024-09-13T13:11:46.966 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 512.34082219100003, 2024-09-13T13:11:46.966 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98902711700000001, 2024-09-13T13:11:46.966 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:46.966 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:46.966 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:46.966 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:46.966 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:46.966 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:46.966 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83988 2024-09-13T13:11:46.966 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.967 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:46.967 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.967 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:46.967 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571369+0000", 2024-09-13T13:11:46.967 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.967 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.967 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:46.967 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571369+0000", 2024-09-13T13:11:46.967 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.967 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.967 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:46.968 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571371+0000", 2024-09-13T13:11:46.968 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0760000000000001e-06 2024-09-13T13:11:46.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.968 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.968 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:46.968 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571391+0000", 2024-09-13T13:11:46.968 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9936999999999999e-05 2024-09-13T13:11:46.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.968 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.968 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:46.968 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571404+0000", 2024-09-13T13:11:46.968 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3241000000000001e-05 2024-09-13T13:11:46.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.969 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:46.969 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571410+0000", 2024-09-13T13:11:46.969 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7799999999999997e-06 2024-09-13T13:11:46.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.969 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.969 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571466+0000", 2024-09-13T13:11:46.969 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6030999999999999e-05 2024-09-13T13:11:46.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.969 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:46.969 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.571473+0000", 2024-09-13T13:11:46.970 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8539999999999999e-06 2024-09-13T13:11:46.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.970 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.970 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556198+0000", 2024-09-13T13:11:46.970 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98472465899999995 2024-09-13T13:11:46.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.970 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:46.970 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556486+0000", 2024-09-13T13:11:46.970 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00028855500000000002 2024-09-13T13:11:46.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.971 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5", 2024-09-13T13:11:46.971 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.556637+0000", 2024-09-13T13:11:46.971 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000151218 2024-09-13T13:11:46.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.971 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:46.971 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.559546+0000", 2024-09-13T13:11:46.971 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029090740000000002 2024-09-13T13:11:46.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.971 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:46.971 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560209+0000", 2024-09-13T13:11:46.971 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00066297199999999995 2024-09-13T13:11:46.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.972 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:46.972 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560301+0000", 2024-09-13T13:11:46.972 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1712000000000006e-05 2024-09-13T13:11:46.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.972 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:46.972 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560396+0000", 2024-09-13T13:11:46.972 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5007999999999996e-05 2024-09-13T13:11:46.972 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.972 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:46.972 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.973 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:166990 7.b 7:db1a2623:::benchmark_data_smithi052_71865_object45889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T13:11:46.973 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.766249+0000", 2024-09-13T13:11:46.973 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.145941725, 2024-09-13T13:11:46.973 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0195013719999999, 2024-09-13T13:11:46.973 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:46.973 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:46.973 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:11:46.973 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:46.973 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:46.973 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:46.974 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 166990 2024-09-13T13:11:46.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.974 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:46.974 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.974 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:46.974 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766249+0000", 2024-09-13T13:11:46.974 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.974 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.974 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:46.974 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766249+0000", 2024-09-13T13:11:46.974 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.975 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:46.975 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766252+0000", 2024-09-13T13:11:46.975 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.48e-06 2024-09-13T13:11:46.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.975 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:46.975 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766259+0000", 2024-09-13T13:11:46.975 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0849999999999999e-06 2024-09-13T13:11:46.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.975 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:46.975 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766272+0000", 2024-09-13T13:11:46.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.305e-05 2024-09-13T13:11:46.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.976 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:46.976 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766281+0000", 2024-09-13T13:11:46.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5750000000000001e-06 2024-09-13T13:11:46.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.976 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.976 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.767413+0000", 2024-09-13T13:11:46.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011321529999999999 2024-09-13T13:11:46.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.977 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:46.977 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.767559+0000", 2024-09-13T13:11:46.977 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000146217 2024-09-13T13:11:46.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.977 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:46.977 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.767706+0000", 2024-09-13T13:11:46.977 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014731799999999999 2024-09-13T13:11:46.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.977 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.977 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.776972+0000", 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0092654350000000007 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.776984+0000", 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2018000000000001e-05 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.785665+0000", 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0086813429999999 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:46.979 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.785751+0000", 2024-09-13T13:11:46.979 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5698000000000005e-05 2024-09-13T13:11:46.979 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.979 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:46.979 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.979 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.979 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167004 7.b 7:d0e4af5f:::benchmark_data_smithi052_71865_object45903:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:11:46.979 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.789600+0000", 2024-09-13T13:11:46.979 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.12259161499998, 2024-09-13T13:11:46.979 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0016245429999999, 2024-09-13T13:11:46.979 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:46.979 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:46.980 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:46.980 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:46.980 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:46.980 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:46.980 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167004 2024-09-13T13:11:46.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.980 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:46.980 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.980 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:46.980 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789600+0000", 2024-09-13T13:11:46.980 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789600+0000", 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789602+0000", 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2230000000000001e-06 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789608+0000", 2024-09-13T13:11:46.981 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.37e-06 2024-09-13T13:11:46.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.982 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:46.982 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789624+0000", 2024-09-13T13:11:46.982 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6167e-05 2024-09-13T13:11:46.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.982 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:46.982 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789632+0000", 2024-09-13T13:11:46.982 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7270000000000007e-06 2024-09-13T13:11:46.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.982 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.983 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789705+0000", 2024-09-13T13:11:46.983 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2799999999999994e-05 2024-09-13T13:11:46.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.983 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:46.983 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789718+0000", 2024-09-13T13:11:46.983 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2653999999999999e-05 2024-09-13T13:11:46.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.983 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.983 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.786269+0000", 2024-09-13T13:11:46.983 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99655117400000004 2024-09-13T13:11:46.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.984 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:46.984 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.786357+0000", 2024-09-13T13:11:46.984 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7865e-05 2024-09-13T13:11:46.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.984 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:46.984 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.786482+0000", 2024-09-13T13:11:46.984 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000125451 2024-09-13T13:11:46.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.984 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:46.985 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791065+0000", 2024-09-13T13:11:46.985 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045828450000000003 2024-09-13T13:11:46.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.985 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.985 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:46.985 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791120+0000", 2024-09-13T13:11:46.985 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5562000000000001e-05 2024-09-13T13:11:46.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.985 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.985 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:46.985 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791155+0000", 2024-09-13T13:11:46.985 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4326999999999998e-05 2024-09-13T13:11:46.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.986 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.986 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:46.986 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791188+0000", 2024-09-13T13:11:46.986 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2929000000000003e-05 2024-09-13T13:11:46.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.986 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.986 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:46.986 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791224+0000", 2024-09-13T13:11:46.986 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6449000000000003e-05 2024-09-13T13:11:46.986 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.986 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:46.986 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167017 7.b 7:d2ad2293:::benchmark_data_smithi052_71865_object45916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.800317+0000", 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.11187378300002, 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99742454800000002, 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167017 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.987 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:46.988 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.988 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:46.988 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800317+0000", 2024-09-13T13:11:46.988 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.988 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.988 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:46.988 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800317+0000", 2024-09-13T13:11:46.988 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.988 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.988 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:46.989 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800320+0000", 2024-09-13T13:11:46.989 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.148e-06 2024-09-13T13:11:46.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.989 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.989 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:46.989 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800326+0000", 2024-09-13T13:11:46.989 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2960000000000004e-06 2024-09-13T13:11:46.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.989 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.989 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:46.989 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800343+0000", 2024-09-13T13:11:46.989 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7079e-05 2024-09-13T13:11:46.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800351+0000", 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1340000000000004e-06 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800433+0000", 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1509999999999997e-05 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800445+0000", 2024-09-13T13:11:46.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2414e-05 2024-09-13T13:11:46.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.991 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:46.991 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.788619+0000", 2024-09-13T13:11:46.991 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98817433300000002 2024-09-13T13:11:46.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.991 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:46.991 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.788727+0000", 2024-09-13T13:11:46.991 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108059 2024-09-13T13:11:46.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:46.993 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:46.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:46.994 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:46.994 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.788817+0000", 2024-09-13T13:11:46.994 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9117999999999996e-05 2024-09-13T13:11:46.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.994 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:46.994 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791266+0000", 2024-09-13T13:11:46.994 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024498599999999999 2024-09-13T13:11:46.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.994 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:46.994 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791306+0000", 2024-09-13T13:11:46.995 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9691000000000003e-05 2024-09-13T13:11:46.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.995 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.995 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:46.995 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797479+0000", 2024-09-13T13:11:46.995 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0061724299999999996 2024-09-13T13:11:46.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.995 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.995 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:46.995 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797695+0000", 2024-09-13T13:11:46.995 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021655399999999999 2024-09-13T13:11:46.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.996 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.996 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:46.996 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797742+0000", 2024-09-13T13:11:46.996 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6922000000000002e-05 2024-09-13T13:11:46.996 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.996 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:46.996 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:46.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.996 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.996 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167030 7.b 7:d39cbe03:::benchmark_data_smithi052_71865_object45929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:11:46.996 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.815695+0000", 2024-09-13T13:11:46.996 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.09649661600002, 2024-09-13T13:11:46.996 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98232124600000004, 2024-09-13T13:11:46.997 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:46.997 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:46.997 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:46.997 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:46.997 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:46.997 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:46.997 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167030 2024-09-13T13:11:46.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.997 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:46.997 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.997 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:46.997 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815695+0000", 2024-09-13T13:11:46.997 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815695+0000", 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815697+0000", 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8380000000000002e-06 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:46.998 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815705+0000", 2024-09-13T13:11:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1430000000000001e-06 2024-09-13T13:11:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815723+0000", 2024-09-13T13:11:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.838e-05 2024-09-13T13:11:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815732+0000", 2024-09-13T13:11:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2010000000000005e-06 2024-09-13T13:11:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:46.999 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815818+0000", 2024-09-13T13:11:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5518000000000004e-05 2024-09-13T13:11:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.815827+0000", 2024-09-13T13:11:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6940000000000007e-06 2024-09-13T13:11:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791337+0000", 2024-09-13T13:11:47.000 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97550924100000003 2024-09-13T13:11:47.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.001 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.001 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:47.001 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791455+0000", 2024-09-13T13:11:47.001 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118081 2024-09-13T13:11:47.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.001 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.001 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:47.001 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791545+0000", 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0480000000000001e-05 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797789+0000", 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0062433799999999998 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797800+0000", 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1153e-05 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.002 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.003 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797829+0000", 2024-09-13T13:11:47.003 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9527999999999999e-05 2024-09-13T13:11:47.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.003 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.003 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:47.003 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797977+0000", 2024-09-13T13:11:47.003 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000148252 2024-09-13T13:11:47.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.003 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.003 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:47.003 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798016+0000", 2024-09-13T13:11:47.004 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8356999999999999e-05 2024-09-13T13:11:47.004 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.004 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:47.004 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.004 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.004 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167044 7.b 7:d626642e:::benchmark_data_smithi052_71865_object45943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:11:47.004 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.832092+0000", 2024-09-13T13:11:47.004 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.08009948699998, 2024-09-13T13:11:47.004 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98043175900000001, 2024-09-13T13:11:47.004 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:47.004 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:47.004 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:47.005 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:47.005 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:47.005 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:47.005 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167044 2024-09-13T13:11:47.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.005 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:47.005 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.005 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:47.005 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832092+0000", 2024-09-13T13:11:47.005 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.005 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.005 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832092+0000", 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832094+0000", 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1299999999999999e-06 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832100+0000", 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9349999999999999e-06 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.007 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:47.007 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832115+0000", 2024-09-13T13:11:47.007 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5104e-05 2024-09-13T13:11:47.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.007 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.007 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:47.007 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832122+0000", 2024-09-13T13:11:47.007 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5769999999999996e-06 2024-09-13T13:11:47.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.007 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.007 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.007 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832207+0000", 2024-09-13T13:11:47.007 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4742999999999998e-05 2024-09-13T13:11:47.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.008 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:47.008 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.832217+0000", 2024-09-13T13:11:47.008 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6069999999999997e-06 2024-09-13T13:11:47.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.008 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.008 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.794154+0000", 2024-09-13T13:11:47.008 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96193714900000005 2024-09-13T13:11:47.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.008 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:47.009 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.794275+0000", 2024-09-13T13:11:47.009 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012122500000000001 2024-09-13T13:11:47.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.009 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.009 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:47.009 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.794399+0000", 2024-09-13T13:11:47.009 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012333799999999999 2024-09-13T13:11:47.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.009 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.009 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.009 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798054+0000", 2024-09-13T13:11:47.009 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036553319999999999 2024-09-13T13:11:47.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.010 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.010 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.010 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798090+0000", 2024-09-13T13:11:47.010 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6616999999999999e-05 2024-09-13T13:11:47.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.010 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.010 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:47.010 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812409+0000", 2024-09-13T13:11:47.010 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014318587000000001 2024-09-13T13:11:47.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.010 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.010 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:47.010 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812475+0000", 2024-09-13T13:11:47.011 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5811999999999999e-05 2024-09-13T13:11:47.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.011 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.011 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:47.011 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812523+0000", 2024-09-13T13:11:47.011 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8603000000000002e-05 2024-09-13T13:11:47.011 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.011 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:47.011 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.011 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.011 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167053 7.b 7:d291c1c0:::benchmark_data_smithi052_71865_object45952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:11:47.011 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.840143+0000", 2024-09-13T13:11:47.012 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.07204818399998, 2024-09-13T13:11:47.012 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97266568200000003, 2024-09-13T13:11:47.012 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:47.012 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:47.012 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:47.012 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:47.012 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:47.012 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:47.012 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167053 2024-09-13T13:11:47.012 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.012 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:47.012 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.013 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:47.013 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840143+0000", 2024-09-13T13:11:47.013 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.013 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.013 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:47.013 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840143+0000", 2024-09-13T13:11:47.013 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.013 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.013 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:47.013 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840145+0000", 2024-09-13T13:11:47.014 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:11:47.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.014 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.014 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:47.014 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840153+0000", 2024-09-13T13:11:47.014 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4080000000000001e-06 2024-09-13T13:11:47.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.014 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.014 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:47.014 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840184+0000", 2024-09-13T13:11:47.014 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0988000000000002e-05 2024-09-13T13:11:47.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.014 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840408+0000", 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00022425 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840568+0000", 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015967999999999999 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840580+0000", 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.199e-05 2024-09-13T13:11:47.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.016 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.016 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798117+0000", 2024-09-13T13:11:47.016 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95753740700000001 2024-09-13T13:11:47.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.016 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:47.016 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798208+0000", 2024-09-13T13:11:47.016 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1225999999999997e-05 2024-09-13T13:11:47.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.016 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:47.016 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.798311+0000", 2024-09-13T13:11:47.017 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103112 2024-09-13T13:11:47.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.017 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.017 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.017 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804034+0000", 2024-09-13T13:11:47.017 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057222940000000002 2024-09-13T13:11:47.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.017 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.017 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.017 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804070+0000", 2024-09-13T13:11:47.017 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6854999999999999e-05 2024-09-13T13:11:47.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.017 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812721+0000", 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0086508160000000004 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812768+0000", 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6251000000000002e-05 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812809+0000", 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1153999999999999e-05 2024-09-13T13:11:47.018 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.019 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:47.019 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.019 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.019 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167055 7.b 7:d9aeaf19:::benchmark_data_smithi052_71865_object45954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:11:47.019 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.840429+0000", 2024-09-13T13:11:47.019 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.071762084, 2024-09-13T13:11:47.019 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97246315999999999, 2024-09-13T13:11:47.019 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:47.019 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:47.019 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:47.019 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:47.019 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:47.020 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:47.020 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167055 2024-09-13T13:11:47.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.020 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:47.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.020 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:47.020 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840429+0000", 2024-09-13T13:11:47.020 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.020 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:47.020 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840429+0000", 2024-09-13T13:11:47.020 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840430+0000", 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0499999999999999e-06 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840435+0000", 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6979999999999997e-06 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:47.021 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840443+0000", 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5709999999999998e-06 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840450+0000", 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3990000000000001e-06 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840607+0000", 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015671000000000001 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.022 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.840614+0000", 2024-09-13T13:11:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2459999999999997e-06 2024-09-13T13:11:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.801126+0000", 2024-09-13T13:11:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96051206300000003 2024-09-13T13:11:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.801236+0000", 2024-09-13T13:11:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110356 2024-09-13T13:11:47.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.024 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:47.024 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.801360+0000", 2024-09-13T13:11:47.024 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012337399999999999 2024-09-13T13:11:47.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.024 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.024 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804101+0000", 2024-09-13T13:11:47.024 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027418619999999999 2024-09-13T13:11:47.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.024 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.024 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806730+0000", 2024-09-13T13:11:47.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002629022 2024-09-13T13:11:47.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:47.025 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812818+0000", 2024-09-13T13:11:47.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0060879970000000004 2024-09-13T13:11:47.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:47.025 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812858+0000", 2024-09-13T13:11:47.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9833000000000003e-05 2024-09-13T13:11:47.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.026 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:47.026 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812892+0000", 2024-09-13T13:11:47.026 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3979000000000001e-05 2024-09-13T13:11:47.026 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.026 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:47.026 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.026 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167059 7.b 7:d9156452:::benchmark_data_smithi052_71865_object45958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:11:47.026 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.843970+0000", 2024-09-13T13:11:47.026 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.06822113800001, 2024-09-13T13:11:47.026 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96901141300000004, 2024-09-13T13:11:47.026 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167059 2024-09-13T13:11:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843970+0000", 2024-09-13T13:11:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.028 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:47.028 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843970+0000", 2024-09-13T13:11:47.028 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.028 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:47.028 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843972+0000", 2024-09-13T13:11:47.028 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1660000000000001e-06 2024-09-13T13:11:47.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.028 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:47.028 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843979+0000", 2024-09-13T13:11:47.029 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6490000000000002e-06 2024-09-13T13:11:47.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.029 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.029 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:47.029 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.843997+0000", 2024-09-13T13:11:47.029 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8493999999999999e-05 2024-09-13T13:11:47.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.029 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.029 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:47.029 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.844006+0000", 2024-09-13T13:11:47.029 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3459999999999994e-06 2024-09-13T13:11:47.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.030 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.030 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.844089+0000", 2024-09-13T13:11:47.030 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3424999999999995e-05 2024-09-13T13:11:47.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.030 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:47.030 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.844101+0000", 2024-09-13T13:11:47.030 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1984000000000001e-05 2024-09-13T13:11:47.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.030 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.030 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804127+0000", 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96002581099999995 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804221+0000", 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4267999999999999e-05 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.804324+0000", 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103071 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.031 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.032 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806768+0000", 2024-09-13T13:11:47.032 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024435590000000001 2024-09-13T13:11:47.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.032 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.032 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806802+0000", 2024-09-13T13:11:47.032 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4023999999999998e-05 2024-09-13T13:11:47.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.032 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:47.032 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812898+0000", 2024-09-13T13:11:47.032 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0060959630000000002 2024-09-13T13:11:47.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.033 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:47.033 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812946+0000", 2024-09-13T13:11:47.033 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8430000000000002e-05 2024-09-13T13:11:47.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.033 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:47.033 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812981+0000", 2024-09-13T13:11:47.033 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5222999999999998e-05 2024-09-13T13:11:47.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.033 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:47.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.034 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167062 7.b 7:daa2d453:::benchmark_data_smithi052_71865_object45961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:11:47.034 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.847548+0000", 2024-09-13T13:11:47.034 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.06464282500002, 2024-09-13T13:11:47.034 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96551024600000002, 2024-09-13T13:11:47.034 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:47.034 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:47.034 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:47.034 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:47.034 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:47.034 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:47.034 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167062 2024-09-13T13:11:47.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.035 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:47.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.035 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:47.035 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847548+0000", 2024-09-13T13:11:47.035 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.035 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:47.035 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847548+0000", 2024-09-13T13:11:47.035 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.035 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:47.036 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847551+0000", 2024-09-13T13:11:47.036 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.745e-06 2024-09-13T13:11:47.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:47.036 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847558+0000", 2024-09-13T13:11:47.036 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4359999999999999e-06 2024-09-13T13:11:47.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:47.036 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847575+0000", 2024-09-13T13:11:47.036 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7405e-05 2024-09-13T13:11:47.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.037 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:47.037 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847584+0000", 2024-09-13T13:11:47.037 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2869999999999993e-06 2024-09-13T13:11:47.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.037 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.037 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847663+0000", 2024-09-13T13:11:47.037 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8288000000000007e-05 2024-09-13T13:11:47.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.037 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:47.038 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.847676+0000", 2024-09-13T13:11:47.038 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.367e-05 2024-09-13T13:11:47.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.038 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.039 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806829+0000", 2024-09-13T13:11:47.039 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95915302099999999 2024-09-13T13:11:47.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.039 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:47.039 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.806951+0000", 2024-09-13T13:11:47.039 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012156599999999999 2024-09-13T13:11:47.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.039 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:47.039 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.807058+0000", 2024-09-13T13:11:47.039 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010687399999999999 2024-09-13T13:11:47.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.040 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.040 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.809517+0000", 2024-09-13T13:11:47.040 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024595379999999998 2024-09-13T13:11:47.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.040 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.040 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812643+0000", 2024-09-13T13:11:47.040 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031254239999999999 2024-09-13T13:11:47.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.040 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:47.040 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812987+0000", 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00034455199999999998 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813026+0000", 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8333e-05 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813059+0000", 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3107e-05 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:47.041 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.042 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167076 7.b 7:d73b9ec1:::benchmark_data_smithi052_71865_object45975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:11:47.042 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.861557+0000", 2024-09-13T13:11:47.042 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.05063418399999, 2024-09-13T13:11:47.042 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95706732900000002, 2024-09-13T13:11:47.042 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:47.042 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:47.042 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:47.042 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:47.042 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:47.042 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:47.042 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167076 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861557+0000", 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861557+0000", 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:47.044 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861559+0000", 2024-09-13T13:11:47.044 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3750000000000001e-06 2024-09-13T13:11:47.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.044 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.044 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:47.044 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861566+0000", 2024-09-13T13:11:47.044 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0990000000000004e-06 2024-09-13T13:11:47.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.044 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.044 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:47.044 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861583+0000", 2024-09-13T13:11:47.044 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7405e-05 2024-09-13T13:11:47.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.045 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:47.045 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861591+0000", 2024-09-13T13:11:47.045 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1459999999999996e-06 2024-09-13T13:11:47.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.045 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.045 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861653+0000", 2024-09-13T13:11:47.045 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1723000000000005e-05 2024-09-13T13:11:47.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.045 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:47.046 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.861662+0000", 2024-09-13T13:11:47.046 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9879999999999993e-06 2024-09-13T13:11:47.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.046 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.046 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.809546+0000", 2024-09-13T13:11:47.046 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.94788374500000006 2024-09-13T13:11:47.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.046 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:47.046 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.809649+0000", 2024-09-13T13:11:47.046 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103309 2024-09-13T13:11:47.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.047 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.047 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:47.047 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.809753+0000", 2024-09-13T13:11:47.047 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104096 2024-09-13T13:11:47.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.047 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.047 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.047 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.812707+0000", 2024-09-13T13:11:47.047 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029539829999999999 2024-09-13T13:11:47.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.047 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.047 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:47.047 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813064+0000", 2024-09-13T13:11:47.048 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035704999999999998 2024-09-13T13:11:47.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.048 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.048 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818518+0000", 2024-09-13T13:11:47.048 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0054542200000000001 2024-09-13T13:11:47.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.048 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:47.048 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818577+0000", 2024-09-13T13:11:47.048 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8894000000000002e-05 2024-09-13T13:11:47.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818624+0000", 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7296000000000003e-05 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167129 7.b 7:d5d8d4da:::benchmark_data_smithi052_71865_object46028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.929446+0000", 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 418.98274544399999, 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.88935671599999999, 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:47.049 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:47.050 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:47.050 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:47.050 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:47.050 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:47.050 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167129 2024-09-13T13:11:47.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.050 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:47.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.050 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:47.050 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929446+0000", 2024-09-13T13:11:47.050 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929446+0000", 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929448+0000", 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.593e-06 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929456+0000", 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7800000000000001e-06 2024-09-13T13:11:47.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.052 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:47.052 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929474+0000", 2024-09-13T13:11:47.052 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7521999999999999e-05 2024-09-13T13:11:47.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.052 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:47.052 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929481+0000", 2024-09-13T13:11:47.052 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3189999999999997e-06 2024-09-13T13:11:47.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.052 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.052 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929545+0000", 2024-09-13T13:11:47.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4216999999999999e-05 2024-09-13T13:11:47.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.053 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:47.053 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.929555+0000", 2024-09-13T13:11:47.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8609999999999993e-06 2024-09-13T13:11:47.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.053 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.053 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813093+0000", 2024-09-13T13:11:47.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.88353817400000001 2024-09-13T13:11:47.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.054 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:47.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813205+0000", 2024-09-13T13:11:47.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000111549 2024-09-13T13:11:47.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.054 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:47.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.813319+0000", 2024-09-13T13:11:47.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114021 2024-09-13T13:11:47.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.054 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818667+0000", 2024-09-13T13:11:47.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053481539999999999 2024-09-13T13:11:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818682+0000", 2024-09-13T13:11:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5438e-05 2024-09-13T13:11:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818723+0000", 2024-09-13T13:11:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0284000000000003e-05 2024-09-13T13:11:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.055 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:47.056 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818766+0000", 2024-09-13T13:11:47.056 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3798000000000003e-05 2024-09-13T13:11:47.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:47.056 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818802+0000", 2024-09-13T13:11:47.056 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6006000000000001e-05 2024-09-13T13:11:47.056 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.056 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:47.056 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.056 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167134 7.b 7:d3cd8c35:::benchmark_data_smithi052_71865_object46033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:11:47.057 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.936758+0000", 2024-09-13T13:11:47.057 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 418.97543305099998, 2024-09-13T13:11:47.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.89043407500000005, 2024-09-13T13:11:47.057 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:11:47.057 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:11:47.057 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:11:47.057 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:11:47.057 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:11:47.057 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:11:47.057 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167134 2024-09-13T13:11:47.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.057 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:11:47.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.058 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:11:47.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936758+0000", 2024-09-13T13:11:47.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.058 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:11:47.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936758+0000", 2024-09-13T13:11:47.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:11:47.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.058 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:11:47.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936760+0000", 2024-09-13T13:11:47.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2199999999999999e-06 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936768+0000", 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.605e-06 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936784+0000", 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5608999999999999e-05 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:11:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936791+0000", 2024-09-13T13:11:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:11:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936875+0000", 2024-09-13T13:11:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3662e-05 2024-09-13T13:11:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:11:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.936885+0000", 2024-09-13T13:11:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0105999999999999e-05 2024-09-13T13:11:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:11:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.815979+0000", 2024-09-13T13:11:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.87909423799999997 2024-09-13T13:11:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:11:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.816085+0000", 2024-09-13T13:11:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000106469 2024-09-13T13:11:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:11:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.816202+0000", 2024-09-13T13:11:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116344 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818689+0000", 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024874559999999999 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.818848+0000", 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015898999999999999 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:11:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.827048+0000", 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0081994749999999995 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.827147+0000", 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9359999999999997e-05 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.827192+0000", 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5194999999999997e-05 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.064 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:11:47.064 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:11:47.064 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:11:47.064 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:47.095 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:47.129 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:47.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:47.231 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:47.269 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:47.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:47.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:47.371 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:47.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:47.414 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:47.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:47.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:47.548 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:47.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:47.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:47.650 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:47.678 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:47.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:47.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:47 smithi052 ceph-mon[29471]: pgmap v516: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-13T13:11:47.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:47 smithi052 ceph-mon[29471]: 9.a scrub starts 2024-09-13T13:11:47.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:47 smithi052 ceph-mon[36682]: pgmap v516: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-13T13:11:47.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:47 smithi052 ceph-mon[36682]: 9.a scrub starts 2024-09-13T13:11:47.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:47.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:47 smithi167 ceph-mon[33178]: pgmap v516: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-13T13:11:47.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:47 smithi167 ceph-mon[33178]: 9.a scrub starts 2024-09-13T13:11:47.823 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:47.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:47.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:47.925 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:47.963 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:47.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:48.065 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:48.102 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:48.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:48.203 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:48.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:48.239 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:48.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:48.340 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:48.374 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:48.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:48.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:48.476 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:48.514 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:48.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:48.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:48.616 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:48.653 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:11:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:48.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[36682]: 5.a scrub starts 2024-09-13T13:11:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[36682]: 5.a scrub ok 2024-09-13T13:11:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:11:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:11:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[36682]: 6.1b scrub starts 2024-09-13T13:11:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[36682]: 6.1b scrub ok 2024-09-13T13:11:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[36682]: 3.11 scrub starts 2024-09-13T13:11:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[36682]: 3.11 scrub ok 2024-09-13T13:11:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[36682]: 3.a scrub starts 2024-09-13T13:11:48.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[36682]: 3.a scrub ok 2024-09-13T13:11:48.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:11:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[29471]: 5.a scrub starts 2024-09-13T13:11:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[29471]: 5.a scrub ok 2024-09-13T13:11:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:11:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:11:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[29471]: 6.1b scrub starts 2024-09-13T13:11:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[29471]: 6.1b scrub ok 2024-09-13T13:11:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[29471]: 3.11 scrub starts 2024-09-13T13:11:48.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[29471]: 3.11 scrub ok 2024-09-13T13:11:48.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[29471]: 3.a scrub starts 2024-09-13T13:11:48.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[29471]: 3.a scrub ok 2024-09-13T13:11:48.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:48 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:11:48.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:48.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:48.794 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:48.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:48 smithi167 ceph-mon[33178]: 5.a scrub starts 2024-09-13T13:11:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:48 smithi167 ceph-mon[33178]: 5.a scrub ok 2024-09-13T13:11:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:48 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:11:48.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:48 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:11:48.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:48 smithi167 ceph-mon[33178]: 6.1b scrub starts 2024-09-13T13:11:48.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:48 smithi167 ceph-mon[33178]: 6.1b scrub ok 2024-09-13T13:11:48.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:48 smithi167 ceph-mon[33178]: 3.11 scrub starts 2024-09-13T13:11:48.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:48 smithi167 ceph-mon[33178]: 3.11 scrub ok 2024-09-13T13:11:48.823 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:48 smithi167 ceph-mon[33178]: 3.a scrub starts 2024-09-13T13:11:48.823 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:48 smithi167 ceph-mon[33178]: 3.a scrub ok 2024-09-13T13:11:48.823 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:48 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:11:48.851 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:11:48.851 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:11:48.851 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T13:11:48.852 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd out 4 2024-09-13T13:11:48.895 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:48.922 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:48.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:48.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:49.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:49.048 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:49.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:49.150 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:49.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:49.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:49.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:49.322 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:49.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:49.423 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:49.462 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:49.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:49.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:49.565 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:49.594 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:49.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:49.695 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:49.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[29471]: pgmap v517: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 651 op/s 2024-09-13T13:11:49.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[29471]: 9.a scrub ok 2024-09-13T13:11:49.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[29471]: osdmap e279: 8 total, 8 up, 8 in 2024-09-13T13:11:49.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[29471]: 6.14 scrub starts 2024-09-13T13:11:49.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[29471]: 6.14 scrub ok 2024-09-13T13:11:49.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[29471]: 4.1f scrub starts 2024-09-13T13:11:49.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[29471]: 4.1f scrub ok 2024-09-13T13:11:49.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[36682]: pgmap v517: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 651 op/s 2024-09-13T13:11:49.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[36682]: 9.a scrub ok 2024-09-13T13:11:49.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[36682]: osdmap e279: 8 total, 8 up, 8 in 2024-09-13T13:11:49.742 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[36682]: 6.14 scrub starts 2024-09-13T13:11:49.742 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[36682]: 6.14 scrub ok 2024-09-13T13:11:49.742 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[36682]: 4.1f scrub starts 2024-09-13T13:11:49.742 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:49 smithi052 ceph-mon[36682]: 4.1f scrub ok 2024-09-13T13:11:49.742 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:49.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:49.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:49 smithi167 ceph-mon[33178]: pgmap v517: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 651 op/s 2024-09-13T13:11:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:49 smithi167 ceph-mon[33178]: 9.a scrub ok 2024-09-13T13:11:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:49 smithi167 ceph-mon[33178]: osdmap e279: 8 total, 8 up, 8 in 2024-09-13T13:11:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:49 smithi167 ceph-mon[33178]: 6.14 scrub starts 2024-09-13T13:11:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:49 smithi167 ceph-mon[33178]: 6.14 scrub ok 2024-09-13T13:11:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:49 smithi167 ceph-mon[33178]: 4.1f scrub starts 2024-09-13T13:11:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:49 smithi167 ceph-mon[33178]: 4.1f scrub ok 2024-09-13T13:11:49.844 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:49.877 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:49.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:49.978 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:50.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:50.001 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:50.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:50.102 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:50.137 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:50.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:50.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:50.242 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:50.281 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:50.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:50.342 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T13:11:50.384 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:50.415 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:50.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:50.516 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:50.559 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:50.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:50.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:50.569 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:11:50.569 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:11:50.569 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[36682]: osdmap e280: 8 total, 8 up, 8 in 2024-09-13T13:11:50.570 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[36682]: 6.16 scrub starts 2024-09-13T13:11:50.570 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[36682]: 6.16 scrub ok 2024-09-13T13:11:50.570 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[36682]: 6.3 scrub starts 2024-09-13T13:11:50.570 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[36682]: 6.3 scrub ok 2024-09-13T13:11:50.570 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:11:50.570 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:11:50.570 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[29471]: osdmap e280: 8 total, 8 up, 8 in 2024-09-13T13:11:50.570 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[29471]: 6.16 scrub starts 2024-09-13T13:11:50.570 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[29471]: 6.16 scrub ok 2024-09-13T13:11:50.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[29471]: 6.3 scrub starts 2024-09-13T13:11:50.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:50 smithi052 ceph-mon[29471]: 6.3 scrub ok 2024-09-13T13:11:50.661 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:50.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:50.701 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:50.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:50.802 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:50.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:50 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:11:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:50 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:11:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:50 smithi167 ceph-mon[33178]: osdmap e280: 8 total, 8 up, 8 in 2024-09-13T13:11:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:50 smithi167 ceph-mon[33178]: 6.16 scrub starts 2024-09-13T13:11:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:50 smithi167 ceph-mon[33178]: 6.16 scrub ok 2024-09-13T13:11:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:50 smithi167 ceph-mon[33178]: 6.3 scrub starts 2024-09-13T13:11:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:50 smithi167 ceph-mon[33178]: 6.3 scrub ok 2024-09-13T13:11:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:50.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:50.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:50.936 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:51.002 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:51.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:51.066 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:51.071 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:51.086 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:51.104 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:51.130 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:51.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:51.232 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:51.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:51.277 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:51.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:51.378 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:51.407 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:51.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:51.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:51.509 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:51.535 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:11:51.542 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:51.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:51.644 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:51.685 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:51.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:51.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[29471]: pgmap v520: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 958 B/s rd, 56 MiB/s wr, 1.42k op/s 2024-09-13T13:11:51.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[29471]: 5.7 scrub starts 2024-09-13T13:11:51.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[29471]: 5.7 scrub ok 2024-09-13T13:11:51.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[29471]: 5.4 scrub starts 2024-09-13T13:11:51.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[29471]: 5.4 scrub ok 2024-09-13T13:11:51.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4202385245' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:51.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[29471]: from='client.25816 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:51.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[36682]: pgmap v520: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 958 B/s rd, 56 MiB/s wr, 1.42k op/s 2024-09-13T13:11:51.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[36682]: 5.7 scrub starts 2024-09-13T13:11:51.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[36682]: 5.7 scrub ok 2024-09-13T13:11:51.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[36682]: 5.4 scrub starts 2024-09-13T13:11:51.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[36682]: 5.4 scrub ok 2024-09-13T13:11:51.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4202385245' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:51.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:51 smithi052 ceph-mon[36682]: from='client.25816 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:51.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:51.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:51 smithi167 ceph-mon[33178]: pgmap v520: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 958 B/s rd, 56 MiB/s wr, 1.42k op/s 2024-09-13T13:11:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:51 smithi167 ceph-mon[33178]: 5.7 scrub starts 2024-09-13T13:11:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:51 smithi167 ceph-mon[33178]: 5.7 scrub ok 2024-09-13T13:11:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:51 smithi167 ceph-mon[33178]: 5.4 scrub starts 2024-09-13T13:11:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:51 smithi167 ceph-mon[33178]: 5.4 scrub ok 2024-09-13T13:11:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:51 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4202385245' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:51 smithi167 ceph-mon[33178]: from='client.25816 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:11:51.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:51.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:51.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:51.962 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:51.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:51.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:52.063 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:52.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:52.199 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:52.232 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:52.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:52.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:52.334 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:52.375 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:52.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:52.477 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:52.511 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:52.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:52.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:52.613 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:52.657 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:52.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:11:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:11:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[36682]: from='client.25816 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:11:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[36682]: osdmap e281: 8 total, 8 up, 8 in 2024-09-13T13:11:52.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[36682]: 5.1e scrub starts 2024-09-13T13:11:52.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[36682]: 5.1e scrub ok 2024-09-13T13:11:52.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[36682]: 3.10 scrub starts 2024-09-13T13:11:52.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[36682]: 3.10 scrub ok 2024-09-13T13:11:52.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:11:52.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:11:52.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[29471]: from='client.25816 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:11:52.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[29471]: osdmap e281: 8 total, 8 up, 8 in 2024-09-13T13:11:52.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[29471]: 5.1e scrub starts 2024-09-13T13:11:52.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[29471]: 5.1e scrub ok 2024-09-13T13:11:52.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[29471]: 3.10 scrub starts 2024-09-13T13:11:52.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:52 smithi052 ceph-mon[29471]: 3.10 scrub ok 2024-09-13T13:11:52.753 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:11:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:11:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 0 2024-09-13T13:11:52.754 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:11:52.758 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:52.789 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:52.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:52.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:52.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:52 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:11:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:52 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:11:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:52 smithi167 ceph-mon[33178]: from='client.25816 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:11:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:52 smithi167 ceph-mon[33178]: osdmap e281: 8 total, 8 up, 8 in 2024-09-13T13:11:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:52 smithi167 ceph-mon[33178]: 5.1e scrub starts 2024-09-13T13:11:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:52 smithi167 ceph-mon[33178]: 5.1e scrub ok 2024-09-13T13:11:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:52 smithi167 ceph-mon[33178]: 3.10 scrub starts 2024-09-13T13:11:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:52 smithi167 ceph-mon[33178]: 3.10 scrub ok 2024-09-13T13:11:52.892 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:52.946 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:52.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:52.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:52.988 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:11:52.988 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:11:53.027 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:11:53.028 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:11:53.048 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:53.062 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:11:53.062 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:11:53.082 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:53.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:53.100 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:11:53.100 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:11:53.136 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:11:53.136 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:11:53.171 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:11:53.171 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:11:53.184 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:53.204 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:11:53.204 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:11:53.217 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:53.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:53.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:53.239 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:11:53.239 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:11:53.319 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:53.351 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:53.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:53.453 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:53.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:53.493 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:53.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:53.594 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:53.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:53.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:53.731 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:53.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:53 smithi052 ceph-mon[36682]: pgmap v522: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 29 MiB/s wr, 1.03k op/s 2024-09-13T13:11:53.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:53 smithi052 ceph-mon[36682]: 4.8 scrub starts 2024-09-13T13:11:53.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:53 smithi052 ceph-mon[36682]: 4.8 scrub ok 2024-09-13T13:11:53.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:53 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3347477360' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:53.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:53 smithi052 ceph-mon[36682]: 5.11 scrub starts 2024-09-13T13:11:53.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:53 smithi052 ceph-mon[36682]: 5.11 scrub ok 2024-09-13T13:11:53.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:53 smithi052 ceph-mon[29471]: pgmap v522: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 29 MiB/s wr, 1.03k op/s 2024-09-13T13:11:53.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:53 smithi052 ceph-mon[29471]: 4.8 scrub starts 2024-09-13T13:11:53.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:53 smithi052 ceph-mon[29471]: 4.8 scrub ok 2024-09-13T13:11:53.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3347477360' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:53.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:53 smithi052 ceph-mon[29471]: 5.11 scrub starts 2024-09-13T13:11:53.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:53 smithi052 ceph-mon[29471]: 5.11 scrub ok 2024-09-13T13:11:53.767 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:53.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:53.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:53 smithi167 ceph-mon[33178]: pgmap v522: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 29 MiB/s wr, 1.03k op/s 2024-09-13T13:11:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:53 smithi167 ceph-mon[33178]: 4.8 scrub starts 2024-09-13T13:11:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:53 smithi167 ceph-mon[33178]: 4.8 scrub ok 2024-09-13T13:11:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:53 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3347477360' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:11:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:53 smithi167 ceph-mon[33178]: 5.11 scrub starts 2024-09-13T13:11:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:53 smithi167 ceph-mon[33178]: 5.11 scrub ok 2024-09-13T13:11:53.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:53.901 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:53.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:53.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:54.003 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:54.042 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:54.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:54.144 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:54.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:54.185 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:54.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:54.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:53 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a[29467]: 2024-09-13T13:11:53.790+0000 7effa8af8640 -1 mon.a@0(leader).osd e281 definitely_dead 0 2024-09-13T13:11:54.271 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:11:54.286 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:54.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:54.321 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:54.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:54.422 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:54.446 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:54.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:54.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:54.526 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:11:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:11:54.548 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:54.572 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:54.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:54.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:54.695 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:54.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[29471]: 4.f scrub starts 2024-09-13T13:11:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[29471]: 4.f scrub ok 2024-09-13T13:11:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[29471]: 4.1b scrub starts 2024-09-13T13:11:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[29471]: 4.1b scrub ok 2024-09-13T13:11:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1887116725' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T13:11:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[29471]: from='client.35852 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T13:11:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[29471]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T13:11:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[29471]: 3.2 scrub starts 2024-09-13T13:11:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[29471]: 3.2 scrub ok 2024-09-13T13:11:54.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[29471]: 5.2 scrub starts 2024-09-13T13:11:54.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[29471]: 5.2 scrub ok 2024-09-13T13:11:54.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[36682]: 4.f scrub starts 2024-09-13T13:11:54.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[36682]: 4.f scrub ok 2024-09-13T13:11:54.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[36682]: 4.1b scrub starts 2024-09-13T13:11:54.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[36682]: 4.1b scrub ok 2024-09-13T13:11:54.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1887116725' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T13:11:54.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[36682]: from='client.35852 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T13:11:54.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[36682]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T13:11:54.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[36682]: 3.2 scrub starts 2024-09-13T13:11:54.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[36682]: 3.2 scrub ok 2024-09-13T13:11:54.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[36682]: 5.2 scrub starts 2024-09-13T13:11:54.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:54 smithi052 ceph-mon[36682]: 5.2 scrub ok 2024-09-13T13:11:54.796 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:54.819 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:54.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:54 smithi167 ceph-mon[33178]: 4.f scrub starts 2024-09-13T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:54 smithi167 ceph-mon[33178]: 4.f scrub ok 2024-09-13T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:54 smithi167 ceph-mon[33178]: 4.1b scrub starts 2024-09-13T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:54 smithi167 ceph-mon[33178]: 4.1b scrub ok 2024-09-13T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:54 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1887116725' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:54 smithi167 ceph-mon[33178]: from='client.35852 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:54 smithi167 ceph-mon[33178]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:54 smithi167 ceph-mon[33178]: 3.2 scrub starts 2024-09-13T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:54 smithi167 ceph-mon[33178]: 3.2 scrub ok 2024-09-13T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:54 smithi167 ceph-mon[33178]: 5.2 scrub starts 2024-09-13T13:11:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:54 smithi167 ceph-mon[33178]: 5.2 scrub ok 2024-09-13T13:11:54.921 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:54.945 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:54.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:55.046 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:55.069 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:55.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:55.120 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T13:11:55.170 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:55.194 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:55.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:55.208 INFO:teuthology.orchestra.run.smithi052.stderr:marked out osd.4. 2024-09-13T13:11:55.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:55.319 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:55.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:55.421 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:11:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:55.445 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:55.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:11:55.547 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:55.583 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:55.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:55.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:55.707 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:55.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:55.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:55.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:55 smithi052 ceph-mon[29471]: pgmap v523: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.8 KiB/s rd, 29 MiB/s wr, 1.85k op/s 2024-09-13T13:11:55.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:55.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:55 smithi052 ceph-mon[29471]: from='client.35852 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T13:11:55.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:55 smithi052 ceph-mon[29471]: osdmap e282: 8 total, 8 up, 7 in 2024-09-13T13:11:55.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:55 smithi052 ceph-mon[29471]: 6.13 scrub starts 2024-09-13T13:11:55.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:55 smithi052 ceph-mon[29471]: 6.13 scrub ok 2024-09-13T13:11:55.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:55 smithi052 ceph-mon[36682]: pgmap v523: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.8 KiB/s rd, 29 MiB/s wr, 1.85k op/s 2024-09-13T13:11:55.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:55.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:55 smithi052 ceph-mon[36682]: from='client.35852 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T13:11:55.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:55 smithi052 ceph-mon[36682]: osdmap e282: 8 total, 8 up, 7 in 2024-09-13T13:11:55.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:55 smithi052 ceph-mon[36682]: 6.13 scrub starts 2024-09-13T13:11:55.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:55 smithi052 ceph-mon[36682]: 6.13 scrub ok 2024-09-13T13:11:55.809 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:55.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:55 smithi167 ceph-mon[33178]: pgmap v523: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.8 KiB/s rd, 29 MiB/s wr, 1.85k op/s 2024-09-13T13:11:55.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:11:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:55 smithi167 ceph-mon[33178]: from='client.35852 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T13:11:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:55 smithi167 ceph-mon[33178]: osdmap e282: 8 total, 8 up, 7 in 2024-09-13T13:11:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:55 smithi167 ceph-mon[33178]: 6.13 scrub starts 2024-09-13T13:11:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:55 smithi167 ceph-mon[33178]: 6.13 scrub ok 2024-09-13T13:11:55.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:55.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:55.934 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:55.958 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:55.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:56.012 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:56.059 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:56.081 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:56.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:56.182 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:56.205 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:56.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:56.306 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:56.329 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:56.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:56.430 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:56.453 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:56.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:56.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:11:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:11:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:11:56.555 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:56.578 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:56.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:56.679 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:11:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:56.702 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:56.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:11:56.803 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:56.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:56 smithi167 ceph-mon[33178]: 4.4 scrub starts 2024-09-13T13:11:56.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:56 smithi167 ceph-mon[33178]: 4.4 scrub ok 2024-09-13T13:11:56.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:56 smithi167 ceph-mon[33178]: osdmap e283: 8 total, 8 up, 7 in 2024-09-13T13:11:56.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:56 smithi167 ceph-mon[33178]: 3.b scrub starts 2024-09-13T13:11:56.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:56 smithi167 ceph-mon[33178]: 3.b scrub ok 2024-09-13T13:11:56.833 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:56.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:56.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:56.936 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:56.970 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:56.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:56 smithi052 ceph-mon[36682]: 4.4 scrub starts 2024-09-13T13:11:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:56 smithi052 ceph-mon[36682]: 4.4 scrub ok 2024-09-13T13:11:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:56 smithi052 ceph-mon[36682]: osdmap e283: 8 total, 8 up, 7 in 2024-09-13T13:11:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:56 smithi052 ceph-mon[36682]: 3.b scrub starts 2024-09-13T13:11:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:56 smithi052 ceph-mon[36682]: 3.b scrub ok 2024-09-13T13:11:56.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:56 smithi052 ceph-mon[29471]: 4.4 scrub starts 2024-09-13T13:11:56.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:56 smithi052 ceph-mon[29471]: 4.4 scrub ok 2024-09-13T13:11:56.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:56 smithi052 ceph-mon[29471]: osdmap e283: 8 total, 8 up, 7 in 2024-09-13T13:11:56.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:56 smithi052 ceph-mon[29471]: 3.b scrub starts 2024-09-13T13:11:56.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:56 smithi052 ceph-mon[29471]: 3.b scrub ok 2024-09-13T13:11:57.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:57.099 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:57.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:57.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:57.201 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:57.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:57.265 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:57.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:57.366 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:57.394 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:57.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:57.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:57.496 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:57.532 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:57.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:57.550 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:57.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:57.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:57.670 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[29471]: pgmap v526: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-13T13:11:57.670 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[29471]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 281'8598 2024-09-13T13:11:57.670 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[29471]: osdmap e284: 8 total, 8 up, 7 in 2024-09-13T13:11:57.670 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[29471]: 9.4 starting backfill to osd.0 from (0'0,0'0] MAX to 281'8867 2024-09-13T13:11:57.670 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[29471]: 9.e starting backfill to osd.0 from (0'0,0'0] MAX to 281'8753 2024-09-13T13:11:57.670 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[29471]: 2.2 scrub starts 2024-09-13T13:11:57.670 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[29471]: 2.2 scrub ok 2024-09-13T13:11:57.670 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[36682]: pgmap v526: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-13T13:11:57.671 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[36682]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 281'8598 2024-09-13T13:11:57.671 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[36682]: osdmap e284: 8 total, 8 up, 7 in 2024-09-13T13:11:57.671 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[36682]: 9.4 starting backfill to osd.0 from (0'0,0'0] MAX to 281'8867 2024-09-13T13:11:57.671 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[36682]: 9.e starting backfill to osd.0 from (0'0,0'0] MAX to 281'8753 2024-09-13T13:11:57.671 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[36682]: 2.2 scrub starts 2024-09-13T13:11:57.671 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:57 smithi052 ceph-mon[36682]: 2.2 scrub ok 2024-09-13T13:11:57.678 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:57.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:57.780 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:57.808 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:57.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:57.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:57 smithi167 ceph-mon[33178]: pgmap v526: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-13T13:11:57.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:57 smithi167 ceph-mon[33178]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 281'8598 2024-09-13T13:11:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:57 smithi167 ceph-mon[33178]: osdmap e284: 8 total, 8 up, 7 in 2024-09-13T13:11:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:57 smithi167 ceph-mon[33178]: 9.4 starting backfill to osd.0 from (0'0,0'0] MAX to 281'8867 2024-09-13T13:11:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:57 smithi167 ceph-mon[33178]: 9.e starting backfill to osd.0 from (0'0,0'0] MAX to 281'8753 2024-09-13T13:11:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:57 smithi167 ceph-mon[33178]: 2.2 scrub starts 2024-09-13T13:11:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:57 smithi167 ceph-mon[33178]: 2.2 scrub ok 2024-09-13T13:11:57.912 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:57.955 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:57.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:57.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:58.056 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:11:58.097 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:58.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:11:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:11:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:11:58.199 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:11:58.236 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:58.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:11:58.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:58.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:58.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:58.470 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:58.509 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:58.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:58.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:58.562 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:11:58.597 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:11:58.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:11:58.638 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:58.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:11:58.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:58 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1377762138' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:11:58.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:58 smithi052 ceph-mon[36682]: from='client.25828 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:11:58.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:58 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1377762138' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:11:58.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:58 smithi052 ceph-mon[29471]: from='client.25828 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:11:58.739 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:11:58.768 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:11:58.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:58.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:11:58.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:58 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1377762138' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:11:58.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:58 smithi167 ceph-mon[33178]: from='client.25828 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:11:58.870 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:58.917 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:58.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:59.020 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:11:59.090 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T13:11:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:59.116 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:59.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:11:59.216 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:59.280 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:11:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:11:59.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:11:59.382 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:11:59.946 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:59 smithi052 ceph-mon[36682]: pgmap v528: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 1.01k op/s; 82 B/s, 3 objects/s recovering 2024-09-13T13:11:59.946 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:59 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:59.946 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:59 smithi052 ceph-mon[36682]: from='client.25828 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:11:59.946 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:11:59 smithi052 ceph-mon[36682]: osdmap e285: 8 total, 8 up, 7 in 2024-09-13T13:11:59.947 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:59 smithi052 ceph-mon[29471]: pgmap v528: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 1.01k op/s; 82 B/s, 3 objects/s recovering 2024-09-13T13:11:59.947 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:59 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:59.947 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:59 smithi052 ceph-mon[29471]: from='client.25828 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:11:59.947 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:11:59 smithi052 ceph-mon[29471]: osdmap e285: 8 total, 8 up, 7 in 2024-09-13T13:11:59.948 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:11:59.948 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:59 smithi167 ceph-mon[33178]: pgmap v528: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 1.01k op/s; 82 B/s, 3 objects/s recovering 2024-09-13T13:11:59.948 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:59 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:11:59.949 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:59 smithi167 ceph-mon[33178]: from='client.25828 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:11:59.949 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:11:59 smithi167 ceph-mon[33178]: osdmap e285: 8 total, 8 up, 7 in 2024-09-13T13:12:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:00.012 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:00.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:00.021 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:00.113 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:00.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:00.149 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:00.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:00.250 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:00.322 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:00.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:00.422 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:00.463 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:00.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:00.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:00.565 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:00.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:00.604 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:00.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:00.705 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:00.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:00.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:00.795 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:12:00.795 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:12:00.795 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-13T13:12:00.796 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-13T13:12:00.812 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:12:00.812 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:12:00.838 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:00.847 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:12:00.847 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:12:00.880 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:12:00.881 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:12:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:00.901 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:00.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:00.917 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:12:00.917 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:12:00.961 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:12:00.961 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:12:01.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:01.006 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:12:01.006 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:12:01.046 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:12:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:12:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:12:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:12:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:95502 7.e 7:73e7d3a1:::benchmark_data_smithi052_71865_object95501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:12:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.170950+0000", 2024-09-13T13:12:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 514.86811522200003, 2024-09-13T13:12:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10520948400000001, 2024-09-13T13:12:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95502 2024-09-13T13:12:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170950+0000", 2024-09-13T13:12:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170950+0000", 2024-09-13T13:12:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170952+0000", 2024-09-13T13:12:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.872e-06 2024-09-13T13:12:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170974+0000", 2024-09-13T13:12:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:12:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.049 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170982+0000", 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6349999999999995e-06 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.170989+0000", 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1619999999999996e-06 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.171116+0000", 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000127293 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.050 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.171296+0000", 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000180278 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.171419+0000", 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012266299999999999 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.175039+0000", 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036198469999999998 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.175665+0000", 2024-09-13T13:12:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00062585400000000002 2024-09-13T13:12:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276069+0000", 2024-09-13T13:12:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10040423499999999 2024-09-13T13:12:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276119+0000", 2024-09-13T13:12:01.052 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0290999999999997e-05 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276160+0000", 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0516999999999998e-05 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:95513 7.e 7:760c2885:::benchmark_data_smithi052_71865_object95512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.177025+0000", 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 514.86204009100004, 2024-09-13T13:12:01.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.099458682000000007, 2024-09-13T13:12:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95513 2024-09-13T13:12:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177025+0000", 2024-09-13T13:12:01.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177025+0000", 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177027+0000", 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8109999999999999e-06 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.055 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177059+0000", 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2125000000000002e-05 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177078+0000", 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9239999999999999e-05 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177085+0000", 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9460000000000001e-06 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177194+0000", 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010910500000000001 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177399+0000", 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000205019 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177525+0000", 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000125969 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276198+0000", 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098672383000000002 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276236+0000", 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7988000000000003e-05 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276248+0000", 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1858000000000001e-05 2024-09-13T13:12:01.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276284+0000", 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6350000000000003e-05 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276484+0000", 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019988800000000001 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95494 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:12:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.177847+0000", 2024-09-13T13:12:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 514.86121795700001, 2024-09-13T13:12:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098173131999999996, 2024-09-13T13:12:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:12:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177847+0000", 2024-09-13T13:12:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177847+0000", 2024-09-13T13:12:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177849+0000", 2024-09-13T13:12:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3179999999999999e-06 2024-09-13T13:12:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177853+0000", 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1799999999999998e-06 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177862+0000", 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1390000000000001e-06 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.177868+0000", 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4119999999999998e-06 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180221+0000", 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023525130000000001 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180227+0000", 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4930000000000003e-06 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276020+0000", 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095793077000000004 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.063 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95502 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:12:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.180124+0000", 2024-09-13T13:12:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 514.85894126899996, 2024-09-13T13:12:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.09604087, 2024-09-13T13:12:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:12:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180124+0000", 2024-09-13T13:12:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180124+0000", 2024-09-13T13:12:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180125+0000", 2024-09-13T13:12:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.311e-06 2024-09-13T13:12:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180130+0000", 2024-09-13T13:12:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7500000000000003e-06 2024-09-13T13:12:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180140+0000", 2024-09-13T13:12:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9310000000000003e-06 2024-09-13T13:12:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180147+0000", 2024-09-13T13:12:01.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0600000000000002e-06 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276055+0000", 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095907611000000004 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276063+0000", 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9719999999999992e-06 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276165+0000", 2024-09-13T13:12:01.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102235 2024-09-13T13:12:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95513 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:12:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.180192+0000", 2024-09-13T13:12:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 514.85887328000001, 2024-09-13T13:12:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.096011869999999999, 2024-09-13T13:12:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.068 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:12:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180192+0000", 2024-09-13T13:12:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180192+0000", 2024-09-13T13:12:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.069 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180193+0000", 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.032e-06 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180197+0000", 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7079999999999999e-06 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180204+0000", 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2819999999999999e-06 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180209+0000", 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7330000000000003e-06 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276187+0000", 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095978049999999995 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276193+0000", 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1480000000000003e-06 2024-09-13T13:12:01.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276204+0000", 2024-09-13T13:12:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0917000000000001e-05 2024-09-13T13:12:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.34781.0:95513 7.e e108/84 ondisk, result = 0)", 2024-09-13T13:12:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:26.180247+0000", 2024-09-13T13:12:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 514.85881800000004, 2024-09-13T13:12:01.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095994805000000002, 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180247+0000", 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180247+0000", 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180248+0000", 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.004e-06 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180252+0000", 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7890000000000001e-06 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180261+0000", 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5089999999999993e-06 2024-09-13T13:12:01.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.180267+0000", 2024-09-13T13:12:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0079999999999999e-06 2024-09-13T13:12:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.075 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.075 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:12:01.076 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:12:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:01.077 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:01.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276226+0000", 2024-09-13T13:12:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095959424000000001 2024-09-13T13:12:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276232+0000", 2024-09-13T13:12:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6189999999999999e-06 2024-09-13T13:12:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:26.276242+0000", 2024-09-13T13:12:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0451999999999999e-05 2024-09-13T13:12:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:118717 7.e 7:7eac196e:::benchmark_data_smithi052_71865_object118716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:12:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:12:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 491.47884664200001, 2024-09-13T13:12:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.120995147, 2024-09-13T13:12:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.079 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 118717 2024-09-13T13:12:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:12:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:12:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560220+0000", 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4200000000000004e-07 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560244+0000", 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4354000000000001e-05 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560252+0000", 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5189999999999992e-06 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560259+0000", 2024-09-13T13:12:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.302e-06 2024-09-13T13:12:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560301+0000", 2024-09-13T13:12:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2318999999999999e-05 2024-09-13T13:12:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560415+0000", 2024-09-13T13:12:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114269 2024-09-13T13:12:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560572+0000", 2024-09-13T13:12:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000156217 2024-09-13T13:12:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.564231+0000", 2024-09-13T13:12:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036597489999999999 2024-09-13T13:12:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.565180+0000", 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00094882300000000001 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.680844+0000", 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.115664164 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.681157+0000", 2024-09-13T13:12:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00031297600000000002 2024-09-13T13:12:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.681214+0000", 2024-09-13T13:12:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6512999999999999e-05 2024-09-13T13:12:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120031 7.e 7:704b0327:::benchmark_data_smithi052_71865_object120030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:12:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:12:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 490.018259392, 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11824775799999999, 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120031 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:12:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:12:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020808+0000", 2024-09-13T13:12:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0439999999999998e-06 2024-09-13T13:12:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020838+0000", 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9791e-05 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020856+0000", 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8664999999999998e-05 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020866+0000", 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3670000000000003e-06 2024-09-13T13:12:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021203+0000", 2024-09-13T13:12:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033735200000000002 2024-09-13T13:12:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021578+0000", 2024-09-13T13:12:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00037465399999999998 2024-09-13T13:12:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022015+0000", 2024-09-13T13:12:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000437176 2024-09-13T13:12:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.026523+0000", 2024-09-13T13:12:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045084249999999999 2024-09-13T13:12:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.027199+0000", 2024-09-13T13:12:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00067598400000000002 2024-09-13T13:12:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138915+0000", 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11171563499999999 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139008+0000", 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3329e-05 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139054+0000", 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5336000000000001e-05 2024-09-13T13:12:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120050 7.e 7:72296fb6:::benchmark_data_smithi052_71865_object120049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:12:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:12:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 490.00621134699998, 2024-09-13T13:12:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12504636999999999, 2024-09-13T13:12:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120050 2024-09-13T13:12:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:12:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032857+0000", 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1140000000000002e-06 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032888+0000", 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1115999999999998e-05 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032905+0000", 2024-09-13T13:12:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6452000000000001e-05 2024-09-13T13:12:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032915+0000", 2024-09-13T13:12:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8889999999999997e-06 2024-09-13T13:12:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033665+0000", 2024-09-13T13:12:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00075015300000000004 2024-09-13T13:12:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033782+0000", 2024-09-13T13:12:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117642 2024-09-13T13:12:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033916+0000", 2024-09-13T13:12:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013400799999999999 2024-09-13T13:12:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.037705+0000", 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037884289999999998 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.037719+0000", 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4034e-05 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157684+0000", 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.119964979 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157805+0000", 2024-09-13T13:12:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120715 2024-09-13T13:12:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157900+0000", 2024-09-13T13:12:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5839000000000002e-05 2024-09-13T13:12:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120054 7.e 7:7dad863d:::benchmark_data_smithi052_71865_object120053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 490.00069607099999, 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11988120300000001, 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120054 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:12:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:12:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038370+0000", 2024-09-13T13:12:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.051e-06 2024-09-13T13:12:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038395+0000", 2024-09-13T13:12:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4224999999999999e-05 2024-09-13T13:12:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038403+0000", 2024-09-13T13:12:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3399999999999998e-06 2024-09-13T13:12:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038409+0000", 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3160000000000002e-06 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038462+0000", 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3012e-05 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038579+0000", 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116483 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038720+0000", 2024-09-13T13:12:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014139099999999999 2024-09-13T13:12:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.042075+0000", 2024-09-13T13:12:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033552830000000001 2024-09-13T13:12:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.042095+0000", 2024-09-13T13:12:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9279999999999998e-05 2024-09-13T13:12:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158031+0000", 2024-09-13T13:12:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11593616299999999 2024-09-13T13:12:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158169+0000", 2024-09-13T13:12:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013835799999999999 2024-09-13T13:12:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158250+0000", 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1301e-05 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151582 7.e 7:7a3dbf39:::benchmark_data_smithi052_71865_object30481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.170441+0000", 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 451.86862428699999, 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.091679105999999996, 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151582 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170441+0000", 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170441+0000", 2024-09-13T13:12:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170442+0000", 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5000000000000002e-07 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170445+0000", 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.88e-06 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170450+0000", 2024-09-13T13:12:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3419999999999998e-06 2024-09-13T13:12:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.170455+0000", 2024-09-13T13:12:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.096e-06 2024-09-13T13:12:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174840+0000", 2024-09-13T13:12:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0043852609999999997 2024-09-13T13:12:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.174948+0000", 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010804 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.175063+0000", 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0001145 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.178533+0000", 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003470065 2024-09-13T13:12:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.178591+0000", 2024-09-13T13:12:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8149000000000002e-05 2024-09-13T13:12:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261725+0000", 2024-09-13T13:12:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.083133548000000002 2024-09-13T13:12:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.261955+0000", 2024-09-13T13:12:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00023013299999999999 2024-09-13T13:12:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.262120+0000", 2024-09-13T13:12:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016544199999999999 2024-09-13T13:12:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:151600 7.e 7:7fe6ca51:::benchmark_data_smithi052_71865_object30499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:12:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:29.189256+0000", 2024-09-13T13:12:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 451.84980965900002, 2024-09-13T13:12:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.086529871999999994, 2024-09-13T13:12:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151600 2024-09-13T13:12:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189256+0000", 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189256+0000", 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189257+0000", 2024-09-13T13:12:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7209999999999999e-06 2024-09-13T13:12:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189263+0000", 2024-09-13T13:12:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3580000000000002e-06 2024-09-13T13:12:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189275+0000", 2024-09-13T13:12:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1973e-05 2024-09-13T13:12:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.189283+0000", 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9039999999999992e-06 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.190358+0000", 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010753099999999999 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.190465+0000", 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010736800000000001 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.190557+0000", 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193101+0000", 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025442569999999999 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.193130+0000", 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8728999999999999e-05 2024-09-13T13:12:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.275647+0000", 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.082517141000000002 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.275742+0000", 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5156999999999997e-05 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:29.275786+0000", 2024-09-13T13:12:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3047000000000003e-05 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:153862 7.e 7:797e5c7a:::benchmark_data_smithi052_71865_object32761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 448.7742586, 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.106314172, 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 153862 2024-09-13T13:12:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:12:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:12:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264809+0000", 2024-09-13T13:12:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.148e-06 2024-09-13T13:12:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264815+0000", 2024-09-13T13:12:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7610000000000004e-06 2024-09-13T13:12:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264826+0000", 2024-09-13T13:12:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1215e-05 2024-09-13T13:12:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264833+0000", 2024-09-13T13:12:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0090000000000001e-06 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264885+0000", 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1845000000000003e-05 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.265001+0000", 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116795 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.265104+0000", 2024-09-13T13:12:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102979 2024-09-13T13:12:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.268877+0000", 2024-09-13T13:12:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037720779999999999 2024-09-13T13:12:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.269086+0000", 2024-09-13T13:12:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000209829 2024-09-13T13:12:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.370873+0000", 2024-09-13T13:12:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10178622700000001 2024-09-13T13:12:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.371063+0000", 2024-09-13T13:12:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000190349 2024-09-13T13:12:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.371121+0000", 2024-09-13T13:12:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7936999999999998e-05 2024-09-13T13:12:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:153884 7.8 7:1006e86e:::benchmark_data_smithi052_71865_object32783:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:12:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:32.288977+0000", 2024-09-13T13:12:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 448.750087834, 2024-09-13T13:12:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.086944270000000004, 2024-09-13T13:12:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 153884 2024-09-13T13:12:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288977+0000", 2024-09-13T13:12:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288977+0000", 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288979+0000", 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9e-06 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288985+0000", 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4580000000000001e-06 2024-09-13T13:12:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.288997+0000", 2024-09-13T13:12:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2004999999999999e-05 2024-09-13T13:12:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.289005+0000", 2024-09-13T13:12:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0069999999999997e-06 2024-09-13T13:12:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.289044+0000", 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9205000000000002e-05 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.289111+0000", 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6952000000000005e-05 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.359719+0000", 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.070607721999999998 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.363436+0000", 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037171370000000001 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375313+0000", 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011876639 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375551+0000", 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000238939 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375761+0000", 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021001599999999999 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.375922+0000", 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016029 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112327 9.2 e275/241 9:464b334b:::benchmark_data_smithi052_128427_object112326:head v 275'7019, mlcod=275'7017)", 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.572648+0000", 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 31.466417371999999, 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.103891961, 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572648+0000", 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572648+0000", 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572650+0000", 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.717e-06 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572678+0000", 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7985999999999999e-05 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572690+0000", 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2262999999999999e-05 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572699+0000", 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9600000000000006e-06 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572782+0000", 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3338999999999996e-05 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572804+0000", 2024-09-13T13:12:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1314999999999998e-05 2024-09-13T13:12:01.136 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:12:01.136 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:12:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676479+0000", 2024-09-13T13:12:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.103675184 2024-09-13T13:12:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676540+0000", 2024-09-13T13:12:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1197000000000004e-05 2024-09-13T13:12:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112328 9.2 e275/241 9:45698ec9:::benchmark_data_smithi052_128427_object112327:head v 275'7020, mlcod=275'7017)", 2024-09-13T13:12:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.576519+0000", 2024-09-13T13:12:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 31.462546246999999, 2024-09-13T13:12:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10005291099999999, 2024-09-13T13:12:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576519+0000", 2024-09-13T13:12:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576519+0000", 2024-09-13T13:12:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576521+0000", 2024-09-13T13:12:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6080000000000001e-06 2024-09-13T13:12:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576544+0000", 2024-09-13T13:12:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3791999999999999e-05 2024-09-13T13:12:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576556+0000", 2024-09-13T13:12:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1671000000000001e-05 2024-09-13T13:12:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576564+0000", 2024-09-13T13:12:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8290000000000008e-06 2024-09-13T13:12:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.578118+0000", 2024-09-13T13:12:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0015537820000000001 2024-09-13T13:12:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.578138+0000", 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0152999999999998e-05 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676545+0000", 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098407005000000006 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676572+0000", 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7070999999999999e-05 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112330 9.2 e275/241 9:4591f68b:::benchmark_data_smithi052_128427_object112329:head v 275'7021, mlcod=275'7017)", 2024-09-13T13:12:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.579400+0000", 2024-09-13T13:12:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 31.459665026, 2024-09-13T13:12:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10414786299999999, 2024-09-13T13:12:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579400+0000", 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579400+0000", 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579402+0000", 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7e-06 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579446+0000", 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.439e-05 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579458+0000", 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1477999999999999e-05 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579465+0000", 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.114e-06 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676607+0000", 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.097141839999999993 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676630+0000", 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3332e-05 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.683496+0000", 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0068658679999999998 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.683548+0000", 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2141000000000001e-05 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112336 9.0 e275/241 9:041ef402:::benchmark_data_smithi052_128427_object112335:head v 275'7019, mlcod=275'7017)", 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.580484+0000", 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 31.458581311, 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10093202900000001, 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580484+0000", 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580484+0000", 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580486+0000", 2024-09-13T13:12:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.53e-06 2024-09-13T13:12:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580530+0000", 2024-09-13T13:12:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4589999999999998e-05 2024-09-13T13:12:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580541+0000", 2024-09-13T13:12:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0779e-05 2024-09-13T13:12:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580548+0000", 2024-09-13T13:12:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3019999999999997e-06 2024-09-13T13:12:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580604+0000", 2024-09-13T13:12:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6271999999999999e-05 2024-09-13T13:12:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676458+0000", 2024-09-13T13:12:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095853820000000006 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.681299+0000", 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048409280000000004 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.681416+0000", 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116808 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112346 9.6 e275/241 9:67a240c7:::benchmark_data_smithi052_128427_object112345:head v 275'7035, mlcod=275'7032)", 2024-09-13T13:12:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.587147+0000", 2024-09-13T13:12:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 31.451918760000002, 2024-09-13T13:12:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.097407753, 2024-09-13T13:12:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587147+0000", 2024-09-13T13:12:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587147+0000", 2024-09-13T13:12:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587149+0000", 2024-09-13T13:12:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.075e-06 2024-09-13T13:12:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587225+0000", 2024-09-13T13:12:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5964000000000007e-05 2024-09-13T13:12:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.677080+0000", 2024-09-13T13:12:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.089855655000000006 2024-09-13T13:12:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.677087+0000", 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0199999999999997e-06 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.677138+0000", 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0773000000000003e-05 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.677157+0000", 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8910999999999999e-05 2024-09-13T13:12:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.684462+0000", 2024-09-13T13:12:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0073047269999999996 2024-09-13T13:12:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.684554+0000", 2024-09-13T13:12:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2627999999999995e-05 2024-09-13T13:12:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112348 9.2 e275/241 9:414aa897:::benchmark_data_smithi052_128427_object112347:head v 275'7022, mlcod=275'7017)", 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.587656+0000", 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 31.451409608999999, 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098073530000000006, 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587656+0000", 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587656+0000", 2024-09-13T13:12:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587657+0000", 2024-09-13T13:12:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.658e-06 2024-09-13T13:12:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587683+0000", 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5253999999999999e-05 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676615+0000", 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.088932251000000004 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676624+0000", 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4150000000000005e-06 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.679045+0000", 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024209600000000002 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.679070+0000", 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4967e-05 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.685690+0000", 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0066195150000000003 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.685729+0000", 2024-09-13T13:12:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9509999999999999e-05 2024-09-13T13:12:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:01.161 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:12:01.178 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:01.222 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:01.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:01.324 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:01.358 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:01.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:01.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:01.460 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:01.502 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:01.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:01.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:01.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:01.637 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:01.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:01.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:01 smithi052 ceph-mon[36682]: pgmap v530: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 7.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 615 op/s; 21305/354441 objects misplaced (6.011%); 6.2 MiB/s, 115 objects/s recovering 2024-09-13T13:12:01.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:01 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/362929395' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:01.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:01 smithi052 ceph-mon[29471]: pgmap v530: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 7.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 615 op/s; 21305/354441 objects misplaced (6.011%); 6.2 MiB/s, 115 objects/s recovering 2024-09-13T13:12:01.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:01 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/362929395' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:01.740 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:01.780 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:01.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:01.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:01 smithi167 ceph-mon[33178]: pgmap v530: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 7.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 615 op/s; 21305/354441 objects misplaced (6.011%); 6.2 MiB/s, 115 objects/s recovering 2024-09-13T13:12:01.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:01 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/362929395' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:01.882 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:01.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:01.913 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:01.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:01.949 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:12:02.014 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:02.062 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:02.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:02.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:02.164 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:02.217 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:02.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:02.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:02.319 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:02.370 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:02.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:02.472 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:02.501 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:02.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:02.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:02.632 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:02.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:02.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:02.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:02.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:02.769 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:02.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:02.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:02 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:12:02.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:02 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:12:02.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:02 smithi052 ceph-mon[29471]: osdmap e286: 8 total, 8 up, 7 in 2024-09-13T13:12:02.833 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:02 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:12:02.833 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:02 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:12:02.833 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:02 smithi052 ceph-mon[36682]: osdmap e286: 8 total, 8 up, 7 in 2024-09-13T13:12:02.870 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:02.872 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T13:12:02.892 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:02.903 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:02.916 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:02.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:03.019 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:03.067 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:03.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:02 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:12:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:02 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:12:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:02 smithi167 ceph-mon[33178]: osdmap e286: 8 total, 8 up, 7 in 2024-09-13T13:12:03.169 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:03.195 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:03.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:03.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:03.297 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:03.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:03.344 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:03.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:03.445 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:03.489 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:03.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:03.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:03.591 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:03.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:03.632 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:03.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:03.733 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:03.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:03 smithi052 ceph-mon[36682]: pgmap v532: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 7.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 544 op/s; 21305/354441 objects misplaced (6.011%); 5.5 MiB/s, 101 objects/s recovering 2024-09-13T13:12:03.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:03 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:12:03.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:03 smithi052 ceph-mon[29471]: pgmap v532: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 7.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 544 op/s; 21305/354441 objects misplaced (6.011%); 5.5 MiB/s, 101 objects/s recovering 2024-09-13T13:12:03.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:03 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:12:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:03.763 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:03.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:03.864 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:03.893 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:03.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:03.994 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:04.018 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:04.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:04.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:03 smithi167 ceph-mon[33178]: pgmap v532: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 7.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 544 op/s; 21305/354441 objects misplaced (6.011%); 5.5 MiB/s, 101 objects/s recovering 2024-09-13T13:12:04.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:03 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:12:04.072 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:04.121 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:04.165 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:04.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:04.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:04.269 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:04.308 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:04.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:04.349 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:12:04.349 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:12:04.349 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 286, 2024-09-13T13:12:04.349 INFO:teuthology.orchestra.run.smithi052.stdout: "fsid": "2b415b38-71cf-11ef-bceb-c7b262605968", 2024-09-13T13:12:04.349 INFO:teuthology.orchestra.run.smithi052.stdout: "created": "2024-09-13T12:55:27.893882+0000", 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: "modified": "2024-09-13T13:12:01.556224+0000", 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: "last_up_change": "2024-09-13T13:04:02.281376+0000", 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: "last_in_change": "2024-09-13T13:11:54.520634+0000", 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_num": 5801984, 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_set": [ 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: "noscrub", 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: "pglog_hardlimit", 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: "purged_snapdirs", 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: "recovery_deletes", 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: "sortbitwise" 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_version": 19, 2024-09-13T13:12:04.350 INFO:teuthology.orchestra.run.smithi052.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T13:12:04.351 INFO:teuthology.orchestra.run.smithi052.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T13:12:04.351 INFO:teuthology.orchestra.run.smithi052.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T13:12:04.351 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_snapshot": "", 2024-09-13T13:12:04.351 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_max": 9, 2024-09-13T13:12:04.351 INFO:teuthology.orchestra.run.smithi052.stdout: "max_osd": 8, 2024-09-13T13:12:04.351 INFO:teuthology.orchestra.run.smithi052.stdout: "require_min_compat_client": "luminous", 2024-09-13T13:12:04.351 INFO:teuthology.orchestra.run.smithi052.stdout: "min_compat_client": "luminous", 2024-09-13T13:12:04.351 INFO:teuthology.orchestra.run.smithi052.stdout: "require_osd_release": "squid", 2024-09-13T13:12:04.351 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_crimson": false, 2024-09-13T13:12:04.351 INFO:teuthology.orchestra.run.smithi052.stdout: "pools": [ 2024-09-13T13:12:04.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.351 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 1, 2024-09-13T13:12:04.351 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": ".mgr", 2024-09-13T13:12:04.352 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T12:58:31.271893+0000", 2024-09-13T13:12:04.352 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:12:04.352 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:12:04.352 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:12:04.352 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:12:04.352 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:12:04.352 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:12:04.352 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:12:04.352 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:12:04.352 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:12:04.352 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:12:04.352 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:12:04.353 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:12:04.353 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:12:04.353 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 1, 2024-09-13T13:12:04.353 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 1, 2024-09-13T13:12:04.353 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 1, 2024-09-13T13:12:04.354 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 1, 2024-09-13T13:12:04.354 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 1, 2024-09-13T13:12:04.354 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:12:04.354 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:12:04.354 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:12:04.354 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:12:04.354 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:12:04.354 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:12:04.354 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:12:04.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.354 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "277", 2024-09-13T13:12:04.354 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:12:04.354 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:12:04.355 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:12:04.355 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:12:04.355 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:12:04.355 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:12:04.355 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:12:04.355 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:12:04.355 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:12:04.355 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:12:04.355 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:12:04.355 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:12:04.355 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:12:04.355 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:12:04.355 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:12:04.356 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:12:04.356 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:12:04.356 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:12:04.356 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:12:04.356 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:12:04.356 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:12:04.356 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:12:04.356 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:12:04.356 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:12:04.356 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:12:04.356 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:12:04.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.357 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:12:04.357 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:12:04.357 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:12:04.357 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:12:04.357 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:12:04.357 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:12:04.357 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:12:04.357 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:12:04.358 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:12:04.358 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:12:04.358 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:12:04.358 INFO:teuthology.orchestra.run.smithi052.stdout: "options": { 2024-09-13T13:12:04.358 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_max": 32, 2024-09-13T13:12:04.358 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_min": 1 2024-09-13T13:12:04.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.358 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:12:04.358 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr": {} 2024-09-13T13:12:04.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.358 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:12:04.358 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:12:04.359 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 7.8899998664855957, 2024-09-13T13:12:04.359 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 7.8899998664855957, 2024-09-13T13:12:04.359 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 0.37999999523162842, 2024-09-13T13:12:04.359 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 3, 2024-09-13T13:12:04.359 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 3, 2024-09-13T13:12:04.359 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 1, 2024-09-13T13:12:04.359 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 1, 2024-09-13T13:12:04.359 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 1 2024-09-13T13:12:04.359 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.359 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.359 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 2, 2024-09-13T13:12:04.359 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "datapool", 2024-09-13T13:12:04.360 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:00.306747+0000", 2024-09-13T13:12:04.360 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 8193, 2024-09-13T13:12:04.360 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T13:12:04.360 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:12:04.360 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:12:04.360 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:12:04.360 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:12:04.360 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:12:04.360 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:12:04.360 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:12:04.360 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:12:04.360 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:12:04.360 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:12:04.361 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:12:04.361 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 3, 2024-09-13T13:12:04.361 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 3, 2024-09-13T13:12:04.361 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 3, 2024-09-13T13:12:04.361 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 3, 2024-09-13T13:12:04.361 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 3, 2024-09-13T13:12:04.361 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:12:04.361 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:12:04.361 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:12:04.361 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:12:04.361 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:12:04.361 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:12:04.361 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:12:04.362 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.362 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "63", 2024-09-13T13:12:04.362 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:12:04.362 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:12:04.362 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:12:04.362 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:12:04.362 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:12:04.362 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 2, 2024-09-13T13:12:04.362 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 63, 2024-09-13T13:12:04.362 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:12:04.362 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:12:04.362 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:12:04.362 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:12:04.363 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:12:04.363 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:12:04.363 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:12:04.363 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:12:04.363 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:12:04.363 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:12:04.363 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:12:04.363 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:12:04.363 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:12:04.363 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:12:04.363 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:12:04.363 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:12:04.363 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:12:04.364 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:12:04.364 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:12:04.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.364 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:12:04.364 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:12:04.364 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:12:04.364 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:12:04.364 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:12:04.364 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:12:04.364 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:12:04.364 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:12:04.364 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd": {} 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 2.6500000953674316, 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 2.6500000953674316, 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-13T13:12:04.365 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 1, 2024-09-13T13:12:04.366 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 1, 2024-09-13T13:12:04.366 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 1 2024-09-13T13:12:04.366 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.366 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.366 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 3, 2024-09-13T13:12:04.366 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": ".rgw.root", 2024-09-13T13:12:04.366 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:01.103523+0000", 2024-09-13T13:12:04.366 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:12:04.366 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:12:04.366 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:12:04.366 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:12:04.367 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:12:04.367 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:12:04.367 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:12:04.367 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:12:04.367 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:12:04.367 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:12:04.367 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:12:04.367 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:12:04.367 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:12:04.367 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:12:04.367 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:12:04.367 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:12:04.368 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:12:04.368 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:12:04.368 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:12:04.368 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:12:04.368 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:12:04.368 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:12:04.368 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:12:04.368 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:12:04.368 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:12:04.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.368 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "61", 2024-09-13T13:12:04.368 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:12:04.368 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:12:04.369 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:12:04.369 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:12:04.369 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:12:04.369 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:12:04.369 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:12:04.369 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:12:04.369 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:12:04.369 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:12:04.369 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:12:04.369 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:12:04.369 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:12:04.369 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:12:04.370 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:12:04.370 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:12:04.370 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:12:04.370 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:12:04.370 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:12:04.370 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:12:04.370 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:12:04.370 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:12:04.370 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:12:04.370 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:12:04.370 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:12:04.370 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:12:04.370 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.371 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:12:04.371 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:12:04.371 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:12:04.371 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:12:04.371 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:12:04.371 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:12:04.371 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:12:04.371 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:12:04.371 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:12:04.371 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:12:04.371 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:12:04.371 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:12:04.371 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:12:04.372 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:12:04.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.372 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:12:04.372 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:12:04.372 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 1.7400000095367432, 2024-09-13T13:12:04.372 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 1.7400000095367432, 2024-09-13T13:12:04.372 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T13:12:04.372 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-13T13:12:04.372 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-13T13:12:04.372 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 1, 2024-09-13T13:12:04.372 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 1, 2024-09-13T13:12:04.372 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 1 2024-09-13T13:12:04.373 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.373 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.373 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 4, 2024-09-13T13:12:04.373 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "default.rgw.log", 2024-09-13T13:12:04.373 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:02.249986+0000", 2024-09-13T13:12:04.373 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:12:04.373 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:12:04.373 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:12:04.373 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:12:04.373 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:12:04.373 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:12:04.373 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:12:04.374 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:12:04.374 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:12:04.374 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:12:04.374 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:12:04.374 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:12:04.374 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:12:04.374 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:12:04.374 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:12:04.374 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:12:04.374 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:12:04.374 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:12:04.374 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:12:04.374 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:12:04.375 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:12:04.375 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:12:04.375 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:12:04.375 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:12:04.375 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:12:04.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.375 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "63", 2024-09-13T13:12:04.375 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:12:04.375 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:12:04.375 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:12:04.375 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:12:04.375 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:12:04.375 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:12:04.376 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:12:04.376 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:12:04.376 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:12:04.376 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:12:04.376 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:12:04.376 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:12:04.376 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:12:04.376 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:12:04.376 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:12:04.376 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:12:04.376 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:12:04.376 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:12:04.377 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:12:04.377 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:12:04.377 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:12:04.377 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:12:04.377 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:12:04.377 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:12:04.377 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:12:04.377 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:12:04.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.377 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:12:04.377 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:12:04.377 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:12:04.377 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:12:04.378 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:12:04.378 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:12:04.378 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:12:04.378 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:12:04.378 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:12:04.378 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:12:04.378 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:12:04.378 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:12:04.378 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:12:04.378 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:12:04.378 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.378 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:12:04.378 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:12:04.379 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 2.4900000095367432, 2024-09-13T13:12:04.379 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 2.4900000095367432, 2024-09-13T13:12:04.379 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T13:12:04.379 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 2.190000057220459, 2024-09-13T13:12:04.379 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 2.190000057220459, 2024-09-13T13:12:04.379 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 1, 2024-09-13T13:12:04.379 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 1, 2024-09-13T13:12:04.379 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 1 2024-09-13T13:12:04.379 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.379 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.379 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 5, 2024-09-13T13:12:04.379 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "default.rgw.control", 2024-09-13T13:12:04.380 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:04.203764+0000", 2024-09-13T13:12:04.380 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:12:04.380 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:12:04.380 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:12:04.380 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:12:04.380 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:12:04.380 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:12:04.380 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:12:04.380 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:12:04.380 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:12:04.380 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:12:04.380 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:12:04.380 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:12:04.381 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:12:04.381 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:12:04.381 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:12:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:12:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:12:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:12:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:12:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:12:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:12:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:12:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:12:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:12:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:12:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "65", 2024-09-13T13:12:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:12:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:12:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:12:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:12:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:12:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:12:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:12:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:12:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:12:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:12:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:12:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:12:04.387 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:12:04.387 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:12:04.387 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:12:04.387 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:12:04.387 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:12:04.387 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:12:04.387 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:12:04.387 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:12:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:12:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:12:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:12:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:12:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:12:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:12:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:12:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:12:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:12:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:12:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:12:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 1.4900000095367432, 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 1.4900000095367432, 2024-09-13T13:12:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T13:12:04.390 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 1.309999942779541, 2024-09-13T13:12:04.390 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:12:04.391 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 1.309999942779541, 2024-09-13T13:12:04.391 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 1, 2024-09-13T13:12:04.391 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 1, 2024-09-13T13:12:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 1 2024-09-13T13:12:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 6, 2024-09-13T13:12:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "default.rgw.meta", 2024-09-13T13:12:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:06.293676+0000", 2024-09-13T13:12:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:12:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:12:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:12:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:12:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:12:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:12:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:12:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:12:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:12:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:12:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:12:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:12:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:12:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:12:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:12:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:12:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:12:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:12:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:12:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:12:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:12:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:12:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:12:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:12:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:12:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "67", 2024-09-13T13:12:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:12:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:12:04.396 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:12:04.396 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:12:04.396 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:12:04.396 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:12:04.396 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:12:04.396 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:12:04.396 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:12:04.396 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:12:04.396 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:12:04.396 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:12:04.396 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:12:04.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:12:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:12:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:12:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:12:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:12:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:12:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:12:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:12:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:12:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:12:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:12:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: "options": { 2024-09-13T13:12:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_bias": 4 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 1.9900000095367432, 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 1.9900000095367432, 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 1.75, 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 1.75, 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 1, 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 1, 2024-09-13T13:12:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 1 2024-09-13T13:12:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 9, 2024-09-13T13:12:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "unique_pool_2", 2024-09-13T13:12:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:10:07.820868+0000", 2024-09-13T13:12:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:12:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:12:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:12:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:12:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:12:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:12:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:12:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:12:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:12:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:12:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:12:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:12:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:12:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 16, 2024-09-13T13:12:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 16, 2024-09-13T13:12:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 16, 2024-09-13T13:12:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 16, 2024-09-13T13:12:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 16, 2024-09-13T13:12:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:12:04.401 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:12:04.401 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:12:04.401 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:12:04.401 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:12:04.401 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:12:04.401 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:12:04.401 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "245", 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:12:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:12:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:12:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:12:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:12:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:12:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:12:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:12:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:12:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:12:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:12:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:12:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:12:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:12:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:12:04.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.404 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:12:04.404 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:12:04.404 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:12:04.404 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:12:04.404 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:12:04.404 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:12:04.404 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:12:04.404 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:12:04.404 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:12:04.404 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:12:04.404 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:12:04.404 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:12:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:12:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: "rados": {} 2024-09-13T13:12:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:12:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:12:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 1.9900000095367432, 2024-09-13T13:12:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 1.9900000095367432, 2024-09-13T13:12:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T13:12:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 1.75, 2024-09-13T13:12:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 1.75, 2024-09-13T13:12:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 1, 2024-09-13T13:12:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 1, 2024-09-13T13:12:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 1 2024-09-13T13:12:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:12:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: "osds": [ 2024-09-13T13:12:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 0, 2024-09-13T13:12:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "d52dca1a-94c1-4e9e-93f5-687c038cd7c7", 2024-09-13T13:12:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:12:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:12:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:12:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:12:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:12:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:12:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 8, 2024-09-13T13:12:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 282, 2024-09-13T13:12:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:12:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:12:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:12:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6801", 2024-09-13T13:12:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:12:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:12:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6802", 2024-09-13T13:12:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:12:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:12:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6804", 2024-09-13T13:12:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:12:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:12:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6803", 2024-09-13T13:12:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:12:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6801/542298937", 2024-09-13T13:12:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6802/542298937", 2024-09-13T13:12:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6804/542298937", 2024-09-13T13:12:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6803/542298937", 2024-09-13T13:12:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:12:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:12:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:12:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 1, 2024-09-13T13:12:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "b40a3574-3c6f-4810-a651-8d64aee6801b", 2024-09-13T13:12:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:12:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:12:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:12:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:12:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:12:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:12:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 13, 2024-09-13T13:12:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 283, 2024-09-13T13:12:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:12:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:12:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:12:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6805", 2024-09-13T13:12:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:12:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:12:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6806", 2024-09-13T13:12:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:12:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:12:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6808", 2024-09-13T13:12:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:12:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:12:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6807", 2024-09-13T13:12:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:12:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6805/26289431", 2024-09-13T13:12:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6806/26289431", 2024-09-13T13:12:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6808/26289431", 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6807/26289431", 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 2, 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "1e8e49a0-ccf8-4b7f-839b-104b64601408", 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:12:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:12:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:12:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 18, 2024-09-13T13:12:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 282, 2024-09-13T13:12:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:12:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:12:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:12:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6809", 2024-09-13T13:12:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:12:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:12:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6810", 2024-09-13T13:12:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:12:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:12:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6812", 2024-09-13T13:12:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:12:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:12:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6811", 2024-09-13T13:12:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:12:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6809/3960240393", 2024-09-13T13:12:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6810/3960240393", 2024-09-13T13:12:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6812/3960240393", 2024-09-13T13:12:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6811/3960240393", 2024-09-13T13:12:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:12:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:12:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:12:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 3, 2024-09-13T13:12:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "9f6f1d22-be68-4d86-9d4b-86fd9e87d141", 2024-09-13T13:12:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:12:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:12:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:12:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:12:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:12:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:12:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 27, 2024-09-13T13:12:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 283, 2024-09-13T13:12:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:12:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:12:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:12:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6813", 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6814", 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:12:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:12:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6816", 2024-09-13T13:12:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:12:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:12:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6815", 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6813/2025522885", 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6814/2025522885", 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6816/2025522885", 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6815/2025522885", 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:12:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:12:04.424 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.424 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.424 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 4, 2024-09-13T13:12:04.424 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "1326d850-caf4-484e-90a1-052f54f9f306", 2024-09-13T13:12:04.424 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:12:04.424 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 0, 2024-09-13T13:12:04.424 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 0, 2024-09-13T13:12:04.424 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:12:04.424 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:12:04.424 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:12:04.424 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 33, 2024-09-13T13:12:04.424 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 169, 2024-09-13T13:12:04.425 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:12:04.425 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:12:04.425 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:12:04.425 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.425 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.425 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.425 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6800", 2024-09-13T13:12:04.425 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:12:04.425 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.425 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.425 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:12:04.425 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.426 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.426 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.426 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6801", 2024-09-13T13:12:04.426 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:12:04.426 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.426 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.426 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:12:04.426 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.426 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.426 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.426 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6803", 2024-09-13T13:12:04.426 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:12:04.427 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.427 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.427 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:12:04.427 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.427 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.427 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.427 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6802", 2024-09-13T13:12:04.427 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:12:04.427 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.427 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.427 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6800/2137138858", 2024-09-13T13:12:04.428 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6801/2137138858", 2024-09-13T13:12:04.428 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6803/2137138858", 2024-09-13T13:12:04.428 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6802/2137138858", 2024-09-13T13:12:04.428 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:12:04.428 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:12:04.428 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:12:04.428 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.428 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.428 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 5, 2024-09-13T13:12:04.428 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2", 2024-09-13T13:12:04.428 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:12:04.428 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:12:04.429 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:12:04.429 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:12:04.429 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:12:04.429 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:12:04.429 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 40, 2024-09-13T13:12:04.429 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 282, 2024-09-13T13:12:04.429 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:12:04.429 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:12:04.429 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:12:04.429 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.429 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.429 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.429 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6804", 2024-09-13T13:12:04.430 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:12:04.430 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.430 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.430 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:12:04.430 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.430 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.430 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.431 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:04.432 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6805", 2024-09-13T13:12:04.432 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:12:04.432 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.432 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.432 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:12:04.432 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.432 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.432 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.432 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6807", 2024-09-13T13:12:04.433 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:12:04.433 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.433 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.433 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:12:04.433 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.433 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.433 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.433 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6806", 2024-09-13T13:12:04.433 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:12:04.433 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.433 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.434 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6804/2906884337", 2024-09-13T13:12:04.434 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6805/2906884337", 2024-09-13T13:12:04.434 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6807/2906884337", 2024-09-13T13:12:04.434 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6806/2906884337", 2024-09-13T13:12:04.434 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:12:04.434 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:12:04.434 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:12:04.434 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.434 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.434 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 6, 2024-09-13T13:12:04.434 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "2471907d-fcbd-44e0-9d29-54151ada9df0", 2024-09-13T13:12:04.434 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:12:04.435 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:12:04.435 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:12:04.435 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:12:04.435 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 48, 2024-09-13T13:12:04.435 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 102, 2024-09-13T13:12:04.435 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 118, 2024-09-13T13:12:04.435 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 282, 2024-09-13T13:12:04.435 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 103, 2024-09-13T13:12:04.435 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:12:04.435 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:12:04.435 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.435 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.435 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.436 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6808", 2024-09-13T13:12:04.436 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:12:04.436 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.436 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.436 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:12:04.436 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.436 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.436 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.436 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6809", 2024-09-13T13:12:04.436 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:12:04.436 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.436 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.437 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.437 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:12:04.437 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.437 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.437 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.437 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6811", 2024-09-13T13:12:04.437 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:12:04.437 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.437 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.437 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.437 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:12:04.437 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.437 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6810", 2024-09-13T13:12:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:12:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6808/143093129", 2024-09-13T13:12:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6809/143093129", 2024-09-13T13:12:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6811/143093129", 2024-09-13T13:12:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6810/143093129", 2024-09-13T13:12:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:12:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:12:04.438 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:12:04.439 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.439 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.439 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 7, 2024-09-13T13:12:04.439 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "97d2c6d7-7989-4c11-91e0-32ee87e76c6d", 2024-09-13T13:12:04.439 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:12:04.439 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:12:04.439 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:12:04.439 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:12:04.439 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:12:04.439 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:12:04.439 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 55, 2024-09-13T13:12:04.439 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 283, 2024-09-13T13:12:04.440 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:12:04.440 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:12:04.440 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:12:04.440 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.440 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.440 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.440 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6812", 2024-09-13T13:12:04.440 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:12:04.441 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.441 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.441 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.441 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:12:04.441 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.441 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.441 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.441 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6813", 2024-09-13T13:12:04.441 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:12:04.441 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.441 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.441 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.442 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:12:04.442 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.442 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.442 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.442 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6815", 2024-09-13T13:12:04.442 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:12:04.442 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.442 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.442 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:12:04.442 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:12:04.442 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.443 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:12:04.443 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6814", 2024-09-13T13:12:04.443 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:12:04.443 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.443 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.443 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.443 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6812/403155155", 2024-09-13T13:12:04.443 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6813/403155155", 2024-09-13T13:12:04.443 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6815/403155155", 2024-09-13T13:12:04.443 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6814/403155155", 2024-09-13T13:12:04.443 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:12:04.443 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:12:04.443 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:12:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:12:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_xinfo": [ 2024-09-13T13:12:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 0, 2024-09-13T13:12:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:12:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:12:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:12:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:12:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:12:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:11:01.544487+0000", 2024-09-13T13:12:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:12:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 1, 2024-09-13T13:12:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:12:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:12:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:12:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:12:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:12:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:10:49.504398+0000", 2024-09-13T13:12:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:12:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 2, 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:11:49.115434+0000", 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 3, 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:12:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:12:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:12:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:12:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:11:11.299080+0000", 2024-09-13T13:12:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:12:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 4, 2024-09-13T13:12:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:12:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:12:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:12:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:12:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 65536, 2024-09-13T13:12:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:12:01.459078+0000", 2024-09-13T13:12:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:12:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 5, 2024-09-13T13:12:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:12:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:12:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:12:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:12:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:12:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:11:03.831231+0000", 2024-09-13T13:12:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:12:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 6, 2024-09-13T13:12:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:12:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:12:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:12:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:12:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:12:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:11:47.650259+0000", 2024-09-13T13:12:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 102 2024-09-13T13:12:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 7, 2024-09-13T13:12:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:12:04.450 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:12:04.450 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:12:04.450 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:12:04.450 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:12:04.450 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:10:46.191988+0000", 2024-09-13T13:12:04.450 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:12:04.450 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.450 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:12:04.450 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_upmap": [], 2024-09-13T13:12:04.450 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_upmap_items": [ 2024-09-13T13:12:04.450 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.450 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.10", 2024-09-13T13:12:04.450 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 1, 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 2 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.12", 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 1, 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 5 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.451 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.452 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.452 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.1f", 2024-09-13T13:12:04.452 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:12:04.452 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.452 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 1, 2024-09-13T13:12:04.452 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 2 2024-09-13T13:12:04.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.452 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.452 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:12:04.452 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_upmap_primaries": [], 2024-09-13T13:12:04.452 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_temp": [ 2024-09-13T13:12:04.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.453 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "9.4", 2024-09-13T13:12:04.453 INFO:teuthology.orchestra.run.smithi052.stdout: "osds": [ 2024-09-13T13:12:04.453 INFO:teuthology.orchestra.run.smithi052.stdout: 1, 2024-09-13T13:12:04.453 INFO:teuthology.orchestra.run.smithi052.stdout: 3, 2024-09-13T13:12:04.453 INFO:teuthology.orchestra.run.smithi052.stdout: 4 2024-09-13T13:12:04.453 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.453 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.454 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "9.c", 2024-09-13T13:12:04.454 INFO:teuthology.orchestra.run.smithi052.stdout: "osds": [ 2024-09-13T13:12:04.454 INFO:teuthology.orchestra.run.smithi052.stdout: 3, 2024-09-13T13:12:04.454 INFO:teuthology.orchestra.run.smithi052.stdout: 7, 2024-09-13T13:12:04.454 INFO:teuthology.orchestra.run.smithi052.stdout: 4 2024-09-13T13:12:04.454 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.454 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:04.454 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "9.e", 2024-09-13T13:12:04.454 INFO:teuthology.orchestra.run.smithi052.stdout: "osds": [ 2024-09-13T13:12:04.454 INFO:teuthology.orchestra.run.smithi052.stdout: 7, 2024-09-13T13:12:04.454 INFO:teuthology.orchestra.run.smithi052.stdout: 1, 2024-09-13T13:12:04.455 INFO:teuthology.orchestra.run.smithi052.stdout: 4 2024-09-13T13:12:04.455 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:04.455 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.455 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:12:04.455 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_temp": [], 2024-09-13T13:12:04.455 INFO:teuthology.orchestra.run.smithi052.stdout: "blocklist": { 2024-09-13T13:12:04.455 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/2104786675": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:12:04.455 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:6800/910649698": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:12:04.455 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1471436716": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:12:04.455 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/4151877300": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:12:04.455 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/3080924817": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:12:04.455 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/4248579079": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:12:04.455 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/455730280": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1887735845": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:6800/1437783219": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:6800/326637643": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/991228238": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/3415403690": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1933060029": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1009267179": "2024-09-14T13:01:31.849627+0000" 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: "range_blocklist": {}, 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profiles": { 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: "default": { 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: "crush-failure-domain": "osd", 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: "k": "2", 2024-09-13T13:12:04.456 INFO:teuthology.orchestra.run.smithi052.stdout: "m": "1", 2024-09-13T13:12:04.457 INFO:teuthology.orchestra.run.smithi052.stdout: "plugin": "jerasure", 2024-09-13T13:12:04.457 INFO:teuthology.orchestra.run.smithi052.stdout: "technique": "reed_sol_van" 2024-09-13T13:12:04.457 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:04.457 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps_queue": [], 2024-09-13T13:12:04.457 INFO:teuthology.orchestra.run.smithi052.stdout: "new_removed_snaps": [], 2024-09-13T13:12:04.457 INFO:teuthology.orchestra.run.smithi052.stdout: "new_purged_snaps": [], 2024-09-13T13:12:04.457 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_node_flags": {}, 2024-09-13T13:12:04.457 INFO:teuthology.orchestra.run.smithi052.stdout: "device_class_flags": {}, 2024-09-13T13:12:04.457 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_mode": { 2024-09-13T13:12:04.457 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_mode_enabled": false, 2024-09-13T13:12:04.457 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_bucket_count": 0, 2024-09-13T13:12:04.457 INFO:teuthology.orchestra.run.smithi052.stdout: "degraded_stretch_mode": 0, 2024-09-13T13:12:04.458 INFO:teuthology.orchestra.run.smithi052.stdout: "recovering_stretch_mode": 0, 2024-09-13T13:12:04.458 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_mode_bucket": 0 2024-09-13T13:12:04.458 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:04.458 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:12:04.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:04.503 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:04.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:04.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:12:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:12:04.602 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:12:04.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:04.631 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:04.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:04.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:04.733 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:04.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:04 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2351932452' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:04.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:04 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3734468696' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-13T13:12:04.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:04 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2351932452' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:04.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:04 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3734468696' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-13T13:12:04.759 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:04.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:04.862 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:04.900 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:04.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:04.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:05.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:05.047 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:05.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:05.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:04 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2351932452' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:05.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:04 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3734468696' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-13T13:12:05.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:05.178 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:05.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:05.281 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:05.315 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:05.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:05.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:05.447 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:05.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:05.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:05.549 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:05.586 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:05.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:05.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:05.688 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:05.754 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:05.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:05.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:05.856 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:05.902 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:05.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:05.968 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:05 smithi052 ceph-mon[36682]: pgmap v533: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 7.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 561 B/s rd, 0 B/s wr, 908 op/s; 19514/344364 objects misplaced (5.667%); 14 MiB/s, 234 objects/s recovering 2024-09-13T13:12:05.968 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:05.968 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:05 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:05.968 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:05 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2351932452' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:12:05.968 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:05 smithi052 ceph-mon[36682]: osdmap e287: 8 total, 8 up, 7 in 2024-09-13T13:12:05.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:05 smithi052 ceph-mon[29471]: pgmap v533: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 7.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 561 B/s rd, 0 B/s wr, 908 op/s; 19514/344364 objects misplaced (5.667%); 14 MiB/s, 234 objects/s recovering 2024-09-13T13:12:05.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:05.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:05 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:05.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:05 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2351932452' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:12:05.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:05 smithi052 ceph-mon[29471]: osdmap e287: 8 total, 8 up, 7 in 2024-09-13T13:12:06.005 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:06.033 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:06.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:06.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:05 smithi167 ceph-mon[33178]: pgmap v533: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 7.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 561 B/s rd, 0 B/s wr, 908 op/s; 19514/344364 objects misplaced (5.667%); 14 MiB/s, 234 objects/s recovering 2024-09-13T13:12:06.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:06.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:05 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:06.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:05 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2351932452' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:12:06.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:05 smithi167 ceph-mon[33178]: osdmap e287: 8 total, 8 up, 7 in 2024-09-13T13:12:06.134 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:06.145 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:12:06.145 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:12:06.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:152240 9.c 9:32a171f5:::benchmark_data_smithi052_128427_object21571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:06.130869+0000", 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.013038645999999999, 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.013072606000000001, 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 152240 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.130869+0000", 2024-09-13T13:12:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.130869+0000", 2024-09-13T13:12:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.130871+0000", 2024-09-13T13:12:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7859999999999999e-06 2024-09-13T13:12:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.130876+0000", 2024-09-13T13:12:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5099999999999998e-06 2024-09-13T13:12:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.130887+0000", 2024-09-13T13:12:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.101e-05 2024-09-13T13:12:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.130894+0000", 2024-09-13T13:12:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2089999999999999e-06 2024-09-13T13:12:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.135434+0000", 2024-09-13T13:12:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045393810000000003 2024-09-13T13:12:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.135555+0000", 2024-09-13T13:12:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121463 2024-09-13T13:12:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4,7", 2024-09-13T13:12:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.135687+0000", 2024-09-13T13:12:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000132297 2024-09-13T13:12:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.142234+0000", 2024-09-13T13:12:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0065464520000000003 2024-09-13T13:12:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143132+0000", 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00089790300000000003 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143608+0000", 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000476109 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:152249 9.7 e287/241 9:e6bdeb7f:::benchmark_data_smithi052_128427_object21580:head v 287'9703, mlcod=287'9701)", 2024-09-13T13:12:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:06.135221+0000", 2024-09-13T13:12:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0086858879999999992, 2024-09-13T13:12:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0088030609999999992, 2024-09-13T13:12:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:12:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.135221+0000", 2024-09-13T13:12:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.135221+0000", 2024-09-13T13:12:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.135223+0000", 2024-09-13T13:12:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1480000000000001e-06 2024-09-13T13:12:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.135229+0000", 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1299999999999998e-06 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.135238+0000", 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8789999999999992e-06 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.135244+0000", 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1589999999999998e-06 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.135469+0000", 2024-09-13T13:12:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000225341 2024-09-13T13:12:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.135487+0000", 2024-09-13T13:12:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7909000000000001e-05 2024-09-13T13:12:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:152251 9.c 9:3ace6dc7:::benchmark_data_smithi052_128427_object21582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-13T13:12:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:06.136365+0000", 2024-09-13T13:12:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0075419049999999998, 2024-09-13T13:12:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0077099789999999996, 2024-09-13T13:12:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:12:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 152251 2024-09-13T13:12:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.136365+0000", 2024-09-13T13:12:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.136365+0000", 2024-09-13T13:12:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.136367+0000", 2024-09-13T13:12:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6729999999999999e-06 2024-09-13T13:12:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.136372+0000", 2024-09-13T13:12:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.091e-06 2024-09-13T13:12:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.136383+0000", 2024-09-13T13:12:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0294e-05 2024-09-13T13:12:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.136390+0000", 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1389999999999998e-06 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.138278+0000", 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018887559999999999 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.138507+0000", 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000229015 2024-09-13T13:12:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4,7", 2024-09-13T13:12:06.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.138634+0000", 2024-09-13T13:12:06.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012670499999999999 2024-09-13T13:12:06.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:06.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143137+0000", 2024-09-13T13:12:06.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045029240000000002 2024-09-13T13:12:06.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.161 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:06.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:152254 9.7 e287/241 9:e93c215d:::benchmark_data_smithi052_128427_object21585:head v 287'9704, mlcod=287'9701)", 2024-09-13T13:12:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:06.141399+0000", 2024-09-13T13:12:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0025087500000000001, 2024-09-13T13:12:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027451699999999999, 2024-09-13T13:12:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:12:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.141399+0000", 2024-09-13T13:12:06.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:06.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.141399+0000", 2024-09-13T13:12:06.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:06.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.141400+0000", 2024-09-13T13:12:06.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3799999999999999e-06 2024-09-13T13:12:06.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:06.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.141408+0000", 2024-09-13T13:12:06.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5399999999999998e-06 2024-09-13T13:12:06.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:06.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.141417+0000", 2024-09-13T13:12:06.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8239999999999995e-06 2024-09-13T13:12:06.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.141424+0000", 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9029999999999998e-06 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.142306+0000", 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00088218699999999999 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.142326+0000", 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9106e-05 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.165 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.15789.0:152251 9.c e287/283)", 2024-09-13T13:12:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:06.143550+0000", 2024-09-13T13:12:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.00035773100000000002, 2024-09-13T13:12:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00064041300000000005, 2024-09-13T13:12:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:12:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:06.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143550+0000", 2024-09-13T13:12:06.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:06.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143550+0000", 2024-09-13T13:12:06.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:06.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143551+0000", 2024-09-13T13:12:06.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.406e-06 2024-09-13T13:12:06.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:06.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143556+0000", 2024-09-13T13:12:06.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8600000000000001e-06 2024-09-13T13:12:06.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:06.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143564+0000", 2024-09-13T13:12:06.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2050000000000002e-06 2024-09-13T13:12:06.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143571+0000", 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3969999999999999e-06 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.15789.0:152239 9.c e287/283)", 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:06.143705+0000", 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.000202785, 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00052071799999999998, 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:06.169 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:12:06.170 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:06.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:06.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143705+0000", 2024-09-13T13:12:06.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:06.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143705+0000", 2024-09-13T13:12:06.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:06.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143705+0000", 2024-09-13T13:12:06.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:12:06.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:06.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143709+0000", 2024-09-13T13:12:06.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9580000000000001e-06 2024-09-13T13:12:06.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:06.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143717+0000", 2024-09-13T13:12:06.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5399999999999998e-06 2024-09-13T13:12:06.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:06.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143722+0000", 2024-09-13T13:12:06.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1830000000000002e-06 2024-09-13T13:12:06.172 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.172 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:06.172 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.172 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.15789.0:152240 9.c e287/283)", 2024-09-13T13:12:06.172 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:06.143731+0000", 2024-09-13T13:12:06.172 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.000176682, 2024-09-13T13:12:06.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00052973099999999997, 2024-09-13T13:12:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:12:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143731+0000", 2024-09-13T13:12:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143731+0000", 2024-09-13T13:12:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143731+0000", 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.27e-07 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143733+0000", 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.227e-06 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143737+0000", 2024-09-13T13:12:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1860000000000002e-06 2024-09-13T13:12:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143741+0000", 2024-09-13T13:12:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0550000000000001e-06 2024-09-13T13:12:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:152239 9.c 9:3e0db1e2:::benchmark_data_smithi052_128427_object21570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-13T13:12:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:06.125800+0000", 2024-09-13T13:12:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.018107109, 2024-09-13T13:12:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.018503167000000001, 2024-09-13T13:12:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:12:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 152239 2024-09-13T13:12:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.125800+0000", 2024-09-13T13:12:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.125800+0000", 2024-09-13T13:12:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.125801+0000", 2024-09-13T13:12:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2799999999999995e-07 2024-09-13T13:12:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.125804+0000", 2024-09-13T13:12:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7250000000000002e-06 2024-09-13T13:12:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.125810+0000", 2024-09-13T13:12:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9089999999999997e-06 2024-09-13T13:12:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.125815+0000", 2024-09-13T13:12:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2490000000000001e-06 2024-09-13T13:12:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.130403+0000", 2024-09-13T13:12:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045885600000000002 2024-09-13T13:12:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.130524+0000", 2024-09-13T13:12:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012043499999999999 2024-09-13T13:12:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4,7", 2024-09-13T13:12:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.130652+0000", 2024-09-13T13:12:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012836499999999999 2024-09-13T13:12:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.135404+0000", 2024-09-13T13:12:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047522459999999999 2024-09-13T13:12:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.138243+0000", 2024-09-13T13:12:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002838794 2024-09-13T13:12:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:06.143126+0000", 2024-09-13T13:12:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048824910000000001 2024-09-13T13:12:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:12:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 8 2024-09-13T13:12:06.182 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:12:06.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:06.183 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:06.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:06.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:12:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:12:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:12:06.283 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:06.327 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:06.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:06.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:06.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:06.456 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:06.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:06.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:06.589 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:06.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:06.690 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:06.731 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:06.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:06.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:06.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:06.864 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:06.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:06.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:06.870 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:06 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:12:06.870 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:06 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:12:06.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:06 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:12:06.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:06 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:12:06.924 INFO:tasks.thrashosds.thrasher:j is {'epoch': 286, 'fsid': '2b415b38-71cf-11ef-bceb-c7b262605968', 'created': '2024-09-13T12:55:27.893882+0000', 'modified': '2024-09-13T13:12:01.556224+0000', 'last_up_change': '2024-09-13T13:04:02.281376+0000', 'last_in_change': '2024-09-13T13:11:54.520634+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T12:58:31.271893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '277', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-13T13:01:00.306747+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-13T13:01:01.103523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-13T13:01:02.249986+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-13T13:01:04.203764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-13T13:01:06.293676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-13T13:10:07.820868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '245', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'd52dca1a-94c1-4e9e-93f5-687c038cd7c7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 282, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6801', 'nonce': 542298937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6802', 'nonce': 542298937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6804', 'nonce': 542298937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6803', 'nonce': 542298937}]}, 'public_addr': '172.21.15.52:6801/542298937', 'cluster_addr': '172.21.15.52:6802/542298937', 'heartbeat_back_addr': '172.21.15.52:6804/542298937', 'heartbeat_front_addr': '172.21.15.52:6803/542298937', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b40a3574-3c6f-4810-a651-8d64aee6801b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 283, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6805', 'nonce': 26289431}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6806', 'nonce': 26289431}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6808', 'nonce': 26289431}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6807', 'nonce': 26289431}]}, 'public_addr': '172.21.15.52:6805/26289431', 'cluster_addr': '172.21.15.52:6806/26289431', 'heartbeat_back_addr': '172.21.15.52:6808/26289431', 'heartbeat_front_addr': '172.21.15.52:6807/26289431', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1e8e49a0-ccf8-4b7f-839b-104b64601408', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 282, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6809', 'nonce': 3960240393}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6810', 'nonce': 3960240393}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6812', 'nonce': 3960240393}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6811', 'nonce': 3960240393}]}, 'public_addr': '172.21.15.52:6809/3960240393', 'cluster_addr': '172.21.15.52:6810/3960240393', 'heartbeat_back_addr': '172.21.15.52:6812/3960240393', 'heartbeat_front_addr': '172.21.15.52:6811/3960240393', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9f6f1d22-be68-4d86-9d4b-86fd9e87d141', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 283, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6813', 'nonce': 2025522885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6814', 'nonce': 2025522885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6816', 'nonce': 2025522885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6815', 'nonce': 2025522885}]}, 'public_addr': '172.21.15.52:6813/2025522885', 'cluster_addr': '172.21.15.52:6814/2025522885', 'heartbeat_back_addr': '172.21.15.52:6816/2025522885', 'heartbeat_front_addr': '172.21.15.52:6815/2025522885', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1326d850-caf4-484e-90a1-052f54f9f306', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 169, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6800', 'nonce': 2137138858}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 2137138858}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 2137138858}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6802', 'nonce': 2137138858}]}, 'public_addr': '172.21.15.167:6800/2137138858', 'cluster_addr': '172.21.15.167:6801/2137138858', 'heartbeat_back_addr': '172.21.15.167:6803/2137138858', 'heartbeat_front_addr': '172.21.15.167:6802/2137138858', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 282, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6804', 'nonce': 2906884337}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 2906884337}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 2906884337}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6806', 'nonce': 2906884337}]}, 'public_addr': '172.21.15.167:6804/2906884337', 'cluster_addr': '172.21.15.167:6805/2906884337', 'heartbeat_back_addr': '172.21.15.167:6807/2906884337', 'heartbeat_front_addr': '172.21.15.167:6806/2906884337', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2471907d-fcbd-44e0-9d29-54151ada9df0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 48, 'last_clean_end': 102, 'up_from': 118, 'up_thru': 282, 'down_at': 103, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6808', 'nonce': 143093129}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 143093129}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 143093129}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6810', 'nonce': 143093129}]}, 'public_addr': '172.21.15.167:6808/143093129', 'cluster_addr': '172.21.15.167:6809/143093129', 'heartbeat_back_addr': '172.21.15.167:6811/143093129', 'heartbeat_front_addr': '172.21.15.167:6810/143093129', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '97d2c6d7-7989-4c11-91e0-32ee87e76c6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 283, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6812', 'nonce': 403155155}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 403155155}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 403155155}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6814', 'nonce': 403155155}]}, 'public_addr': '172.21.15.167:6812/403155155', 'cluster_addr': '172.21.15.167:6813/403155155', 'heartbeat_back_addr': '172.21.15.167:6815/403155155', 'heartbeat_front_addr': '172.21.15.167:6814/403155155', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:11:01.544487+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:10:49.504398+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:11:49.115434+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:11:11.299080+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-13T13:12:01.459078+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:11:03.831231+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:11:47.650259+0000', 'dead_epoch': 102}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:10:46.191988+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.4', 'osds': [1, 3, 4]}, {'pgid': '9.c', 'osds': [3, 7, 4]}, {'pgid': '9.e', 'osds': [7, 1, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.52:0/2104786675': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:6800/910649698': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:0/1471436716': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:0/4151877300': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:0/3080924817': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:0/4248579079': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:0/455730280': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/1887735845': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:6800/1437783219': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:6800/326637643': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/991228238': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/3415403690': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/1933060029': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:0/1009267179': '2024-09-14T13:01:31.849627+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T13:12:06.925 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:12:06.953 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:06.966 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:06.991 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:06.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:06.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:07.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:06 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:12:07.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:06 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:12:07.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:07.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:07.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:07.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:07.225 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:07.254 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:07.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:07.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:07.356 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:07.396 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:07.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:07.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:07.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:07.525 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:07.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:07.627 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:07.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:07.666 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:07.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:07.728 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:12:07.752 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T13:12:07.767 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:07.795 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:07.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:07.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:07.868 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:07 smithi052 ceph-mon[36682]: pgmap v535: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 140 B/s rd, 0 B/s wr, 1.06k op/s; 18696/336813 objects misplaced (5.551%); 17 MiB/s, 271 objects/s recovering 2024-09-13T13:12:07.868 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:07 smithi052 ceph-mon[36682]: osdmap e288: 8 total, 8 up, 7 in 2024-09-13T13:12:07.869 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:07 smithi052 ceph-mon[29471]: pgmap v535: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 140 B/s rd, 0 B/s wr, 1.06k op/s; 18696/336813 objects misplaced (5.551%); 17 MiB/s, 271 objects/s recovering 2024-09-13T13:12:07.869 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:07 smithi052 ceph-mon[29471]: osdmap e288: 8 total, 8 up, 7 in 2024-09-13T13:12:07.897 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:07.974 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:07.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:07.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:08.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:07 smithi167 ceph-mon[33178]: pgmap v535: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 140 B/s rd, 0 B/s wr, 1.06k op/s; 18696/336813 objects misplaced (5.551%); 17 MiB/s, 271 objects/s recovering 2024-09-13T13:12:08.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:07 smithi167 ceph-mon[33178]: osdmap e288: 8 total, 8 up, 7 in 2024-09-13T13:12:08.076 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:08.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:08.125 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:08.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:08.226 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:08.273 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:08.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:08.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:08.298 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:08.375 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:08.379 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:12:08.379 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:12:08.400 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:08.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:08.422 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:12:08.422 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:12:08.445 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:12:08.445 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:12:08.501 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:08.524 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:08.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:08.570 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:12:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=sqlstore.transactions t=2024-09-13T13:12:08.120624644Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T13:12:08.570 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:12:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=cleanup t=2024-09-13T13:12:08.125901393Z level=info msg="Completed cleanup jobs" duration=16.092453ms 2024-09-13T13:12:08.571 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:12:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=plugins.update.checker t=2024-09-13T13:12:08.263585626Z level=info msg="Update check succeeded" duration=63.207461ms 2024-09-13T13:12:08.571 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:12:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=grafana.update.checker t=2024-09-13T13:12:08.264850326Z level=info msg="Update check succeeded" duration=68.657499ms 2024-09-13T13:12:08.579 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:12:08.579 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:12:08.615 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:12:08.615 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:12:08.626 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:08.653 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:12:08.654 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:12:08.671 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:08.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:08.689 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:12:08.689 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:12:08.724 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:12:08.724 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:12:08.773 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:08.801 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:08.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:08.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:08.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:08.935 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:08.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:08 smithi052 ceph-mon[36682]: pgmap v537: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 815 B/s rd, 0 B/s wr, 937 op/s; 18696/336813 objects misplaced (5.551%); 16 MiB/s, 254 objects/s recovering 2024-09-13T13:12:08.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:08 smithi052 ceph-mon[29471]: pgmap v537: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 815 B/s rd, 0 B/s wr, 937 op/s; 18696/336813 objects misplaced (5.551%); 16 MiB/s, 254 objects/s recovering 2024-09-13T13:12:09.037 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:09.077 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:09.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:09.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:09.179 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:09.221 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:09.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:09.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:09.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:08 smithi167 ceph-mon[33178]: pgmap v537: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 815 B/s rd, 0 B/s wr, 937 op/s; 18696/336813 objects misplaced (5.551%); 16 MiB/s, 254 objects/s recovering 2024-09-13T13:12:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3224675507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:09.323 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:09.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:09 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3224675507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:09.324 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3224675507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:09.370 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:09.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:09.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:09.472 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:09.502 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:09.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:09.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:09.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:09.633 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:09.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:09.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:09.735 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:09.764 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:09.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:09.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:09.788 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:09.866 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:09.898 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:09.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:10.001 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:10.006 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:12:10.015 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:12:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:10.031 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:10.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:10.132 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:10.163 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:10.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:10.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:10.299 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:10.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:10.401 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:10.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:10.445 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:10.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:10.546 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:10.572 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:10.601 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:10.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:10.703 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:10.735 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:10.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:10.837 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:10.864 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:10.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:10.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:10.965 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:10.995 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:10.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:11.053 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2024-09-13T13:12:09.837961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6972447283,"num_objects":106582,"num_object_clones":0,"num_object_copies":319746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16167,"num_objects_unfound":0,"num_objects_dirty":106582,"num_whiteouts":0,"num_read":1926,"num_read_kb":1783,"num_write":155786,"num_write_kb":8364298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3866,"num_bytes_recovered":247859913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73469,"ondisk_log_size":73469,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":22366448,"kb_used_data":19539952,"kb_used_omap":13,"kb_used_meta":2826098,"kb_avail":633849616,"statfs":{"total":671965249536,"available":649062006784,"internally_reserved":0,"allocated":20008910848,"data_stored":20006276275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13997,"internal_metadata":2893924691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,7,11,11,7,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-757923840,"num_objects":-11565,"num_object_clones":0,"num_object_copies":-34695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-5138,"num_objects_unfound":0,"num_objects_dirty":-11565,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11565,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3246,"num_bytes_recovered":212729856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001989"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":495,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.614911+0000","last_change":"2024-09-13T13:10:32.368663+0000","last_active":"2024-09-13T13:12:06.614911+0000","last_peered":"2024-09-13T13:12:06.614911+0000","last_clean":"2024-09-13T13:12:06.614911+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:12:06.614911+0000","last_undegraded":"2024-09-13T13:12:06.614911+0000","last_fullsized":"2024-09-13T13:12:06.614911+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.368588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.368588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":515,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-13T13:12:04.618967+0000","last_change":"2024-09-13T13:11:55.594727+0000","last_active":"2024-09-13T13:12:04.618967+0000","last_peered":"2024-09-13T13:12:04.618967+0000","last_clean":"2024-09-13T13:12:04.618967+0000","last_became_active":"2024-09-13T13:11:55.555979+0000","last_became_peered":"2024-09-13T13:11:55.555979+0000","last_unstale":"2024-09-13T13:12:04.618967+0000","last_undegraded":"2024-09-13T13:12:04.618967+0000","last_fullsized":"2024-09-13T13:12:04.618967+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:32.777451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.777451+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":480,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.610821+0000","last_change":"2024-09-13T13:05:33.773491+0000","last_active":"2024-09-13T13:12:06.610821+0000","last_peered":"2024-09-13T13:12:06.610821+0000","last_clean":"2024-09-13T13:12:06.610821+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:12:06.610821+0000","last_undegraded":"2024-09-13T13:12:06.610821+0000","last_fullsized":"2024-09-13T13:12:06.610821+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:33.773423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:33.773423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.772899+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":23,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620671+0000","last_change":"2024-09-13T13:11:55.563956+0000","last_active":"2024-09-13T13:12:06.620671+0000","last_peered":"2024-09-13T13:12:06.620671+0000","last_clean":"2024-09-13T13:12:06.620671+0000","last_became_active":"2024-09-13T13:11:55.563496+0000","last_became_peered":"2024-09-13T13:11:55.563496+0000","last_unstale":"2024-09-13T13:12:06.620671+0000","last_undegraded":"2024-09-13T13:12:06.620671+0000","last_fullsized":"2024-09-13T13:12:06.620671+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:42.787492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:42.787492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":502,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619801+0000","last_change":"2024-09-13T13:11:47.852762+0000","last_active":"2024-09-13T13:12:06.619801+0000","last_peered":"2024-09-13T13:12:06.619801+0000","last_clean":"2024-09-13T13:12:06.619801+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:12:06.619801+0000","last_undegraded":"2024-09-13T13:12:06.619801+0000","last_fullsized":"2024-09-13T13:12:06.619801+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:13.219176+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":507,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620658+0000","last_change":"2024-09-13T13:11:05.501305+0000","last_active":"2024-09-13T13:12:06.620658+0000","last_peered":"2024-09-13T13:12:06.620658+0000","last_clean":"2024-09-13T13:12:06.620658+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:12:06.620658+0000","last_undegraded":"2024-09-13T13:12:06.620658+0000","last_fullsized":"2024-09-13T13:12:06.620658+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:05.501234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:05.501234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.182488+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":550,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.622172+0000","last_change":"2024-09-13T13:10:39.460577+0000","last_active":"2024-09-13T13:12:06.622172+0000","last_peered":"2024-09-13T13:12:06.622172+0000","last_clean":"2024-09-13T13:12:06.622172+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:12:06.622172+0000","last_undegraded":"2024-09-13T13:12:06.622172+0000","last_fullsized":"2024-09-13T13:12:06.622172+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:10:39.460500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.460500+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":153,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-13T13:12:04.619088+0000","last_change":"2024-09-13T13:11:55.557019+0000","last_active":"2024-09-13T13:12:04.619088+0000","last_peered":"2024-09-13T13:12:04.619088+0000","last_clean":"2024-09-13T13:12:04.619088+0000","last_became_active":"2024-09-13T13:11:55.556548+0000","last_became_peered":"2024-09-13T13:11:55.556548+0000","last_unstale":"2024-09-13T13:12:04.619088+0000","last_undegraded":"2024-09-13T13:12:04.619088+0000","last_fullsized":"2024-09-13T13:12:04.619088+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:13.201484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:13.201484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:16.627254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":539,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620600+0000","last_change":"2024-09-13T13:11:55.564397+0000","last_active":"2024-09-13T13:12:06.620600+0000","last_peered":"2024-09-13T13:12:06.620600+0000","last_clean":"2024-09-13T13:12:06.620600+0000","last_became_active":"2024-09-13T13:11:55.555432+0000","last_became_peered":"2024-09-13T13:11:55.555432+0000","last_unstale":"2024-09-13T13:12:06.620600+0000","last_undegraded":"2024-09-13T13:12:06.620600+0000","last_fullsized":"2024-09-13T13:12:06.620600+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:10:35.137160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:35.137160+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":524,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620850+0000","last_change":"2024-09-13T13:11:55.568139+0000","last_active":"2024-09-13T13:12:06.620850+0000","last_peered":"2024-09-13T13:12:06.620850+0000","last_clean":"2024-09-13T13:12:06.620850+0000","last_became_active":"2024-09-13T13:11:55.547420+0000","last_became_peered":"2024-09-13T13:11:55.547420+0000","last_unstale":"2024-09-13T13:12:06.620850+0000","last_undegraded":"2024-09-13T13:12:06.620850+0000","last_fullsized":"2024-09-13T13:12:06.620850+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:31.738685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.738685+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":487,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620073+0000","last_change":"2024-09-13T13:06:50.665256+0000","last_active":"2024-09-13T13:12:06.620073+0000","last_peered":"2024-09-13T13:12:06.620073+0000","last_clean":"2024-09-13T13:12:06.620073+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:12:06.620073+0000","last_undegraded":"2024-09-13T13:12:06.620073+0000","last_fullsized":"2024-09-13T13:12:06.620073+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:50.665191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:50.665191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:21.077433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":490,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.613235+0000","last_change":"2024-09-13T13:10:43.363157+0000","last_active":"2024-09-13T13:12:06.613235+0000","last_peered":"2024-09-13T13:12:06.613235+0000","last_clean":"2024-09-13T13:12:06.613235+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:12:06.613235+0000","last_undegraded":"2024-09-13T13:12:06.613235+0000","last_fullsized":"2024-09-13T13:12:06.613235+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:43.362967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:43.362967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:11.841904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":518,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620391+0000","last_change":"2024-09-13T13:11:55.559060+0000","last_active":"2024-09-13T13:12:06.620391+0000","last_peered":"2024-09-13T13:12:06.620391+0000","last_clean":"2024-09-13T13:12:06.620391+0000","last_became_active":"2024-09-13T13:11:55.558732+0000","last_became_peered":"2024-09-13T13:11:55.558732+0000","last_unstale":"2024-09-13T13:12:06.620391+0000","last_undegraded":"2024-09-13T13:12:06.620391+0000","last_fullsized":"2024-09-13T13:12:06.620391+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:31.103509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.103509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":484,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620204+0000","last_change":"2024-09-13T13:11:55.573811+0000","last_active":"2024-09-13T13:12:06.620204+0000","last_peered":"2024-09-13T13:12:06.620204+0000","last_clean":"2024-09-13T13:12:06.620204+0000","last_became_active":"2024-09-13T13:11:55.556591+0000","last_became_peered":"2024-09-13T13:11:55.556591+0000","last_unstale":"2024-09-13T13:12:06.620204+0000","last_undegraded":"2024-09-13T13:12:06.620204+0000","last_fullsized":"2024-09-13T13:12:06.620204+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:11:53.120451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:53.120451+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:07.862466+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":502,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620170+0000","last_change":"2024-09-13T13:11:55.563232+0000","last_active":"2024-09-13T13:12:06.620170+0000","last_peered":"2024-09-13T13:12:06.620170+0000","last_clean":"2024-09-13T13:12:06.620170+0000","last_became_active":"2024-09-13T13:11:55.562883+0000","last_became_peered":"2024-09-13T13:11:55.562883+0000","last_unstale":"2024-09-13T13:12:06.620170+0000","last_undegraded":"2024-09-13T13:12:06.620170+0000","last_fullsized":"2024-09-13T13:12:06.620170+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.802931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.802931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":467,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619620+0000","last_change":"2024-09-13T13:07:19.641119+0000","last_active":"2024-09-13T13:12:06.619620+0000","last_peered":"2024-09-13T13:12:06.619620+0000","last_clean":"2024-09-13T13:12:06.619620+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:12:06.619620+0000","last_undegraded":"2024-09-13T13:12:06.619620+0000","last_fullsized":"2024-09-13T13:12:06.619620+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:19.640697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:19.640697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:59.949351+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":37,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-13T13:12:04.619632+0000","last_change":"2024-09-13T13:11:55.558822+0000","last_active":"2024-09-13T13:12:04.619632+0000","last_peered":"2024-09-13T13:12:04.619632+0000","last_clean":"2024-09-13T13:12:04.619632+0000","last_became_active":"2024-09-13T13:11:55.551872+0000","last_became_peered":"2024-09-13T13:11:55.551872+0000","last_unstale":"2024-09-13T13:12:04.619632+0000","last_undegraded":"2024-09-13T13:12:04.619632+0000","last_fullsized":"2024-09-13T13:12:04.619632+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:28.804735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:28.804735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":501,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.613849+0000","last_change":"2024-09-13T13:11:55.553455+0000","last_active":"2024-09-13T13:12:06.613849+0000","last_peered":"2024-09-13T13:12:06.613849+0000","last_clean":"2024-09-13T13:12:06.613849+0000","last_became_active":"2024-09-13T13:11:55.552618+0000","last_became_peered":"2024-09-13T13:11:55.552618+0000","last_unstale":"2024-09-13T13:12:06.613849+0000","last_undegraded":"2024-09-13T13:12:06.613849+0000","last_fullsized":"2024-09-13T13:12:06.613849+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:19.180948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:19.180948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":543,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.626924+0000","last_change":"2024-09-13T13:06:44.532305+0000","last_active":"2024-09-13T13:12:06.626924+0000","last_peered":"2024-09-13T13:12:06.626924+0000","last_clean":"2024-09-13T13:12:06.626924+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:12:06.626924+0000","last_undegraded":"2024-09-13T13:12:06.626924+0000","last_fullsized":"2024-09-13T13:12:06.626924+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:06:44.532186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.532186+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.519028+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":503,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.614505+0000","last_change":"2024-09-13T13:11:55.562079+0000","last_active":"2024-09-13T13:12:06.614505+0000","last_peered":"2024-09-13T13:12:06.614505+0000","last_clean":"2024-09-13T13:12:06.614505+0000","last_became_active":"2024-09-13T13:11:55.561742+0000","last_became_peered":"2024-09-13T13:11:55.561742+0000","last_unstale":"2024-09-13T13:12:06.614505+0000","last_undegraded":"2024-09-13T13:12:06.614505+0000","last_fullsized":"2024-09-13T13:12:06.614505+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:59.936811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:59.936811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":495,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620807+0000","last_change":"2024-09-13T13:09:29.517189+0000","last_active":"2024-09-13T13:12:06.620807+0000","last_peered":"2024-09-13T13:12:06.620807+0000","last_clean":"2024-09-13T13:12:06.620807+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:12:06.620807+0000","last_undegraded":"2024-09-13T13:12:06.620807+0000","last_fullsized":"2024-09-13T13:12:06.620807+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:29.517034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:29.517034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.617164+0000","last_change":"2024-09-13T13:11:55.562962+0000","last_active":"2024-09-13T13:12:06.617164+0000","last_peered":"2024-09-13T13:12:06.617164+0000","last_clean":"2024-09-13T13:12:06.617164+0000","last_became_active":"2024-09-13T13:11:55.556448+0000","last_became_peered":"2024-09-13T13:11:55.556448+0000","last_unstale":"2024-09-13T13:12:06.617164+0000","last_undegraded":"2024-09-13T13:12:06.617164+0000","last_fullsized":"2024-09-13T13:12:06.617164+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:34.788562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:34.788562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":542,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619744+0000","last_change":"2024-09-13T13:11:55.585405+0000","last_active":"2024-09-13T13:12:06.619744+0000","last_peered":"2024-09-13T13:12:06.619744+0000","last_clean":"2024-09-13T13:12:06.619744+0000","last_became_active":"2024-09-13T13:11:55.557005+0000","last_became_peered":"2024-09-13T13:11:55.557005+0000","last_unstale":"2024-09-13T13:12:06.619744+0000","last_undegraded":"2024-09-13T13:12:06.619744+0000","last_fullsized":"2024-09-13T13:12:06.619744+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:11:14.593438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:14.593438+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:24.720108+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":23,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.621355+0000","last_change":"2024-09-13T13:11:55.541329+0000","last_active":"2024-09-13T13:12:06.621355+0000","last_peered":"2024-09-13T13:12:06.621355+0000","last_clean":"2024-09-13T13:12:06.621355+0000","last_became_active":"2024-09-13T13:11:55.540475+0000","last_became_peered":"2024-09-13T13:11:55.540475+0000","last_unstale":"2024-09-13T13:12:06.621355+0000","last_undegraded":"2024-09-13T13:12:06.621355+0000","last_fullsized":"2024-09-13T13:12:06.621355+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:12.879727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:12.879727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":500,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619133+0000","last_change":"2024-09-13T13:10:09.765203+0000","last_active":"2024-09-13T13:12:06.619133+0000","last_peered":"2024-09-13T13:12:06.619133+0000","last_clean":"2024-09-13T13:12:06.619133+0000","last_became_active":"2024-09-13T13:01:08.210040+0000","last_became_peered":"2024-09-13T13:01:08.210040+0000","last_unstale":"2024-09-13T13:12:06.619133+0000","last_undegraded":"2024-09-13T13:12:06.619133+0000","last_fullsized":"2024-09-13T13:12:06.619133+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:09.765153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.765153+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:34.922825+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":500,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.610988+0000","last_change":"2024-09-13T13:11:55.569188+0000","last_active":"2024-09-13T13:12:06.610988+0000","last_peered":"2024-09-13T13:12:06.610988+0000","last_clean":"2024-09-13T13:12:06.610988+0000","last_became_active":"2024-09-13T13:11:55.559301+0000","last_became_peered":"2024-09-13T13:11:55.559301+0000","last_unstale":"2024-09-13T13:12:06.610988+0000","last_undegraded":"2024-09-13T13:12:06.610988+0000","last_fullsized":"2024-09-13T13:12:06.610988+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:07:54.439269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:54.439269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":529,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.615759+0000","last_change":"2024-09-13T13:09:28.434007+0000","last_active":"2024-09-13T13:12:06.615759+0000","last_peered":"2024-09-13T13:12:06.615759+0000","last_clean":"2024-09-13T13:12:06.615759+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:12:06.615759+0000","last_undegraded":"2024-09-13T13:12:06.615759+0000","last_fullsized":"2024-09-13T13:12:06.615759+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:28.433899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:28.433899+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"284'11","reported_seq":594,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-13T13:12:04.619692+0000","last_change":"2024-09-13T13:11:55.558710+0000","last_active":"2024-09-13T13:12:04.619692+0000","last_peered":"2024-09-13T13:12:04.619692+0000","last_clean":"2024-09-13T13:12:04.619692+0000","last_became_active":"2024-09-13T13:11:55.544769+0000","last_became_peered":"2024-09-13T13:11:55.544769+0000","last_unstale":"2024-09-13T13:12:04.619692+0000","last_undegraded":"2024-09-13T13:12:04.619692+0000","last_fullsized":"2024-09-13T13:12:04.619692+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:10:42.010847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.010847+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"288'9620","reported_seq":9725,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:08.819527+0000","last_change":"2024-09-13T13:10:09.272561+0000","last_active":"2024-09-13T13:12:08.819527+0000","last_peered":"2024-09-13T13:12:08.819527+0000","last_clean":"2024-09-13T13:12:08.819527+0000","last_became_active":"2024-09-13T13:10:09.272264+0000","last_became_peered":"2024-09-13T13:10:09.272264+0000","last_unstale":"2024-09-13T13:12:08.819527+0000","last_undegraded":"2024-09-13T13:12:08.819527+0000","last_fullsized":"2024-09-13T13:12:08.819527+0000","mapping_epoch":241,"log_start":"266'5308","ondisk_log_start":"266'5308","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4312,"log_dups_size":0,"ondisk_log_size":4312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:38.733071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439091200,"num_objects":6700,"num_object_clones":0,"num_object_copies":20100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9620,"num_write_kb":522240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":566,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.618254+0000","last_change":"2024-09-13T13:11:55.575321+0000","last_active":"2024-09-13T13:12:06.618254+0000","last_peered":"2024-09-13T13:12:06.618254+0000","last_clean":"2024-09-13T13:12:06.618254+0000","last_became_active":"2024-09-13T13:11:55.558987+0000","last_became_peered":"2024-09-13T13:11:55.558987+0000","last_unstale":"2024-09-13T13:12:06.618254+0000","last_undegraded":"2024-09-13T13:12:06.618254+0000","last_fullsized":"2024-09-13T13:12:06.618254+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:11:53.007997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:53.007997+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:58.163112+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":512,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.626747+0000","last_change":"2024-09-13T13:10:30.552177+0000","last_active":"2024-09-13T13:12:06.626747+0000","last_peered":"2024-09-13T13:12:06.626747+0000","last_clean":"2024-09-13T13:12:06.626747+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:12:06.626747+0000","last_undegraded":"2024-09-13T13:12:06.626747+0000","last_fullsized":"2024-09-13T13:12:06.626747+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:30.552113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.552113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"284'9","reported_seq":56,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619998+0000","last_change":"2024-09-13T13:11:55.565027+0000","last_active":"2024-09-13T13:12:06.619998+0000","last_peered":"2024-09-13T13:12:06.619998+0000","last_clean":"2024-09-13T13:12:06.619998+0000","last_became_active":"2024-09-13T13:11:55.555340+0000","last_became_peered":"2024-09-13T13:11:55.555340+0000","last_unstale":"2024-09-13T13:12:06.619998+0000","last_undegraded":"2024-09-13T13:12:06.619998+0000","last_fullsized":"2024-09-13T13:12:06.619998+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:10:07.726751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:07.726751+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":491,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620106+0000","last_change":"2024-09-13T13:10:08.973968+0000","last_active":"2024-09-13T13:12:06.620106+0000","last_peered":"2024-09-13T13:12:06.620106+0000","last_clean":"2024-09-13T13:12:06.620106+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:12:06.620106+0000","last_undegraded":"2024-09-13T13:12:06.620106+0000","last_fullsized":"2024-09-13T13:12:06.620106+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.973917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.973917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:23.093183+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"288'9661","reported_seq":9767,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:08.116305+0000","last_change":"2024-09-13T13:10:09.273532+0000","last_active":"2024-09-13T13:12:08.116305+0000","last_peered":"2024-09-13T13:12:08.116305+0000","last_clean":"2024-09-13T13:12:08.116305+0000","last_became_active":"2024-09-13T13:10:09.273246+0000","last_became_peered":"2024-09-13T13:10:09.273246+0000","last_unstale":"2024-09-13T13:12:08.116305+0000","last_undegraded":"2024-09-13T13:12:08.116305+0000","last_fullsized":"2024-09-13T13:12:08.116305+0000","mapping_epoch":241,"log_start":"264'5098","ondisk_log_start":"264'5098","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4563,"log_dups_size":0,"ondisk_log_size":4563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:29.955904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441581592,"num_objects":6739,"num_object_clones":0,"num_object_copies":20217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6739,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9661,"num_write_kb":524737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":526,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.626414+0000","last_change":"2024-09-13T13:03:22.436912+0000","last_active":"2024-09-13T13:12:06.626414+0000","last_peered":"2024-09-13T13:12:06.626414+0000","last_clean":"2024-09-13T13:12:06.626414+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:12:06.626414+0000","last_undegraded":"2024-09-13T13:12:06.626414+0000","last_fullsized":"2024-09-13T13:12:06.626414+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'18","last_scrub_stamp":"2024-09-13T13:03:22.436685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.436685+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:26.731371+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":494,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.621793+0000","last_change":"2024-09-13T13:06:44.785235+0000","last_active":"2024-09-13T13:12:06.621793+0000","last_peered":"2024-09-13T13:12:06.621793+0000","last_clean":"2024-09-13T13:12:06.621793+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:12:06.621793+0000","last_undegraded":"2024-09-13T13:12:06.621793+0000","last_fullsized":"2024-09-13T13:12:06.621793+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:06:44.785017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:44.785017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.777133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":23,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620583+0000","last_change":"2024-09-13T13:11:55.553540+0000","last_active":"2024-09-13T13:12:06.620583+0000","last_peered":"2024-09-13T13:12:06.620583+0000","last_clean":"2024-09-13T13:12:06.620583+0000","last_became_active":"2024-09-13T13:11:55.553046+0000","last_became_peered":"2024-09-13T13:11:55.553046+0000","last_unstale":"2024-09-13T13:12:06.620583+0000","last_undegraded":"2024-09-13T13:12:06.620583+0000","last_fullsized":"2024-09-13T13:12:06.620583+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":23,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620164+0000","last_change":"2024-09-13T13:11:55.549044+0000","last_active":"2024-09-13T13:12:06.620164+0000","last_peered":"2024-09-13T13:12:06.620164+0000","last_clean":"2024-09-13T13:12:06.620164+0000","last_became_active":"2024-09-13T13:11:55.548156+0000","last_became_peered":"2024-09-13T13:11:55.548156+0000","last_unstale":"2024-09-13T13:12:06.620164+0000","last_undegraded":"2024-09-13T13:12:06.620164+0000","last_fullsized":"2024-09-13T13:12:06.620164+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:10.104926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:10.104926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:38.782309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"288'9457","reported_seq":11808,"reported_epoch":288,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T13:12:08.849669+0000","last_change":"2024-09-13T13:11:56.751379+0000","last_active":"2024-09-13T13:12:08.849669+0000","last_peered":"2024-09-13T13:12:08.849669+0000","last_clean":"2024-09-13T13:11:54.415418+0000","last_became_active":"2024-09-13T13:11:56.666909+0000","last_became_peered":"2024-09-13T13:11:56.666909+0000","last_unstale":"2024-09-13T13:12:08.849669+0000","last_undegraded":"2024-09-13T13:12:08.849669+0000","last_fullsized":"2024-09-13T13:12:08.849669+0000","mapping_epoch":283,"log_start":"266'5113","ondisk_log_start":"266'5113","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4344,"log_dups_size":0,"ondisk_log_size":4344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423165952,"num_objects":6457,"num_object_clones":0,"num_object_copies":19371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5417,"num_objects_unfound":0,"num_objects_dirty":6457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9457,"num_write_kb":509248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1120,"num_bytes_recovered":73400320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,4],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":6457}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":550,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619722+0000","last_change":"2024-09-13T13:11:55.579068+0000","last_active":"2024-09-13T13:12:06.619722+0000","last_peered":"2024-09-13T13:12:06.619722+0000","last_clean":"2024-09-13T13:12:06.619722+0000","last_became_active":"2024-09-13T13:11:55.556835+0000","last_became_peered":"2024-09-13T13:11:55.556835+0000","last_unstale":"2024-09-13T13:12:06.619722+0000","last_undegraded":"2024-09-13T13:12:06.619722+0000","last_fullsized":"2024-09-13T13:12:06.619722+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:11:17.167044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:17.167044+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:40.852281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":491,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.615885+0000","last_change":"2024-09-13T13:11:55.568683+0000","last_active":"2024-09-13T13:12:06.615885+0000","last_peered":"2024-09-13T13:12:06.615885+0000","last_clean":"2024-09-13T13:12:06.615885+0000","last_became_active":"2024-09-13T13:11:55.552534+0000","last_became_peered":"2024-09-13T13:11:55.552534+0000","last_unstale":"2024-09-13T13:12:06.615885+0000","last_undegraded":"2024-09-13T13:12:06.615885+0000","last_fullsized":"2024-09-13T13:12:06.615885+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:03:22.498673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:03:22.498673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"284'9","reported_seq":641,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.623253+0000","last_change":"2024-09-13T13:11:55.589830+0000","last_active":"2024-09-13T13:12:06.623253+0000","last_peered":"2024-09-13T13:12:06.623253+0000","last_clean":"2024-09-13T13:12:06.623253+0000","last_became_active":"2024-09-13T13:11:55.558541+0000","last_became_peered":"2024-09-13T13:11:55.558541+0000","last_unstale":"2024-09-13T13:12:06.623253+0000","last_undegraded":"2024-09-13T13:12:06.623253+0000","last_fullsized":"2024-09-13T13:12:06.623253+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:11:06.526646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:06.526646+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:36.177175+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":495,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619397+0000","last_change":"2024-09-13T13:11:49.982172+0000","last_active":"2024-09-13T13:12:06.619397+0000","last_peered":"2024-09-13T13:12:06.619397+0000","last_clean":"2024-09-13T13:12:06.619397+0000","last_became_active":"2024-09-13T13:04:02.763587+0000","last_became_peered":"2024-09-13T13:04:02.763587+0000","last_unstale":"2024-09-13T13:12:06.619397+0000","last_undegraded":"2024-09-13T13:12:06.619397+0000","last_fullsized":"2024-09-13T13:12:06.619397+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:49.982122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.982122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:50.066525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"288'9653","reported_seq":9758,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:08.249943+0000","last_change":"2024-09-13T13:10:09.273698+0000","last_active":"2024-09-13T13:12:08.249943+0000","last_peered":"2024-09-13T13:12:08.249943+0000","last_clean":"2024-09-13T13:12:08.249943+0000","last_became_active":"2024-09-13T13:10:09.273311+0000","last_became_peered":"2024-09-13T13:10:09.273311+0000","last_unstale":"2024-09-13T13:12:08.249943+0000","last_undegraded":"2024-09-13T13:12:08.249943+0000","last_fullsized":"2024-09-13T13:12:08.249943+0000","mapping_epoch":241,"log_start":"263'4764","ondisk_log_start":"263'4764","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4889,"log_dups_size":0,"ondisk_log_size":4889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:29.006601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432865280,"num_objects":6605,"num_object_clones":0,"num_object_copies":19815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9653,"num_write_kb":520256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":550,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.617058+0000","last_change":"2024-09-13T13:11:55.584738+0000","last_active":"2024-09-13T13:12:06.617058+0000","last_peered":"2024-09-13T13:12:06.617058+0000","last_clean":"2024-09-13T13:12:06.617058+0000","last_became_active":"2024-09-13T13:11:55.556607+0000","last_became_peered":"2024-09-13T13:11:55.556607+0000","last_unstale":"2024-09-13T13:12:06.617058+0000","last_undegraded":"2024-09-13T13:12:06.617058+0000","last_fullsized":"2024-09-13T13:12:06.617058+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":512,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620688+0000","last_change":"2024-09-13T13:11:44.046330+0000","last_active":"2024-09-13T13:12:06.620688+0000","last_peered":"2024-09-13T13:12:06.620688+0000","last_clean":"2024-09-13T13:12:06.620688+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:12:06.620688+0000","last_undegraded":"2024-09-13T13:12:06.620688+0000","last_fullsized":"2024-09-13T13:12:06.620688+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:44.046261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:44.046261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:03.579263+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":635,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.612980+0000","last_change":"2024-09-13T13:10:10.134973+0000","last_active":"2024-09-13T13:12:06.612980+0000","last_peered":"2024-09-13T13:12:06.612980+0000","last_clean":"2024-09-13T13:12:06.612980+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:12:06.612980+0000","last_undegraded":"2024-09-13T13:12:06.612980+0000","last_fullsized":"2024-09-13T13:12:06.612980+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:10:10.134873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.134873+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:32.576287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":507,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.613433+0000","last_change":"2024-09-13T13:11:44.756719+0000","last_active":"2024-09-13T13:12:06.613433+0000","last_peered":"2024-09-13T13:12:06.613433+0000","last_clean":"2024-09-13T13:12:06.613433+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:12:06.613433+0000","last_undegraded":"2024-09-13T13:12:06.613433+0000","last_fullsized":"2024-09-13T13:12:06.613433+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:44.756518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:44.756518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:55.883051+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"288'9567","reported_seq":12798,"reported_epoch":288,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T13:12:08.130085+0000","last_change":"2024-09-13T13:11:56.749317+0000","last_active":"2024-09-13T13:12:08.130085+0000","last_peered":"2024-09-13T13:12:08.130085+0000","last_clean":"2024-09-13T13:11:54.368025+0000","last_became_active":"2024-09-13T13:11:56.669031+0000","last_became_peered":"2024-09-13T13:11:56.669031+0000","last_unstale":"2024-09-13T13:12:08.130085+0000","last_undegraded":"2024-09-13T13:12:08.130085+0000","last_fullsized":"2024-09-13T13:12:08.130085+0000","mapping_epoch":283,"log_start":"263'5008","ondisk_log_start":"263'5008","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434438144,"num_objects":6629,"num_object_clones":0,"num_object_copies":19887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5156,"num_objects_unfound":0,"num_objects_dirty":6629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9567,"num_write_kb":518272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1561,"num_bytes_recovered":102301696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":6629}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":558,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.613720+0000","last_change":"2024-09-13T13:11:18.533559+0000","last_active":"2024-09-13T13:12:06.613720+0000","last_peered":"2024-09-13T13:12:06.613720+0000","last_clean":"2024-09-13T13:12:06.613720+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:12:06.613720+0000","last_undegraded":"2024-09-13T13:12:06.613720+0000","last_fullsized":"2024-09-13T13:12:06.613720+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:11:18.533452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:18.533452+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:47.552700+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":481,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620002+0000","last_change":"2024-09-13T13:09:38.867734+0000","last_active":"2024-09-13T13:12:06.620002+0000","last_peered":"2024-09-13T13:12:06.620002+0000","last_clean":"2024-09-13T13:12:06.620002+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:12:06.620002+0000","last_undegraded":"2024-09-13T13:12:06.620002+0000","last_fullsized":"2024-09-13T13:12:06.620002+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:38.867688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:38.867688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":454,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-13T13:12:04.619242+0000","last_change":"2024-09-13T13:11:54.199900+0000","last_active":"2024-09-13T13:12:04.619242+0000","last_peered":"2024-09-13T13:12:04.619242+0000","last_clean":"2024-09-13T13:12:04.619242+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:12:04.619242+0000","last_undegraded":"2024-09-13T13:12:04.619242+0000","last_fullsized":"2024-09-13T13:12:04.619242+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:54.199831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:54.199831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:11.064459+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":496,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.610744+0000","last_change":"2024-09-13T13:06:43.969251+0000","last_active":"2024-09-13T13:12:06.610744+0000","last_peered":"2024-09-13T13:12:06.610744+0000","last_clean":"2024-09-13T13:12:06.610744+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:12:06.610744+0000","last_undegraded":"2024-09-13T13:12:06.610744+0000","last_fullsized":"2024-09-13T13:12:06.610744+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:43.969176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:06:43.969176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:23.798758+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"288'9987","reported_seq":10090,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:09.086752+0000","last_change":"2024-09-13T13:10:09.277094+0000","last_active":"2024-09-13T13:12:09.086752+0000","last_peered":"2024-09-13T13:12:09.086752+0000","last_clean":"2024-09-13T13:12:09.086752+0000","last_became_active":"2024-09-13T13:10:09.275394+0000","last_became_peered":"2024-09-13T13:10:09.275394+0000","last_unstale":"2024-09-13T13:12:09.086752+0000","last_undegraded":"2024-09-13T13:12:09.086752+0000","last_fullsized":"2024-09-13T13:12:09.086752+0000","mapping_epoch":241,"log_start":"267'5552","ondisk_log_start":"267'5552","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:30.543594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445710336,"num_objects":6801,"num_object_clones":0,"num_object_copies":20403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9987,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"268'40","reported_seq":701,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619611+0000","last_change":"2024-09-13T13:11:54.902633+0000","last_active":"2024-09-13T13:12:06.619611+0000","last_peered":"2024-09-13T13:12:06.619611+0000","last_clean":"2024-09-13T13:12:06.619611+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:12:06.619611+0000","last_undegraded":"2024-09-13T13:12:06.619611+0000","last_fullsized":"2024-09-13T13:12:06.619611+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'40","last_scrub_stamp":"2024-09-13T13:11:54.902575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:54.902575+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:14.462492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":155,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.613948+0000","last_change":"2024-09-13T13:11:55.553777+0000","last_active":"2024-09-13T13:12:06.613948+0000","last_peered":"2024-09-13T13:12:06.613948+0000","last_clean":"2024-09-13T13:12:06.613948+0000","last_became_active":"2024-09-13T13:11:55.553072+0000","last_became_peered":"2024-09-13T13:11:55.553072+0000","last_unstale":"2024-09-13T13:12:06.613948+0000","last_undegraded":"2024-09-13T13:12:06.613948+0000","last_fullsized":"2024-09-13T13:12:06.613948+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:18.888060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:07:18.888060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":507,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620891+0000","last_change":"2024-09-13T13:11:56.946203+0000","last_active":"2024-09-13T13:12:06.620891+0000","last_peered":"2024-09-13T13:12:06.620891+0000","last_clean":"2024-09-13T13:12:06.620891+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:12:06.620891+0000","last_undegraded":"2024-09-13T13:12:06.620891+0000","last_fullsized":"2024-09-13T13:12:06.620891+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:11:56.946121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:56.946121+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:13.534347+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":502,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.613300+0000","last_change":"2024-09-13T13:11:55.553329+0000","last_active":"2024-09-13T13:12:06.613300+0000","last_peered":"2024-09-13T13:12:06.613300+0000","last_clean":"2024-09-13T13:12:06.613300+0000","last_became_active":"2024-09-13T13:11:55.552893+0000","last_became_peered":"2024-09-13T13:11:55.552893+0000","last_unstale":"2024-09-13T13:12:06.613300+0000","last_undegraded":"2024-09-13T13:12:06.613300+0000","last_fullsized":"2024-09-13T13:12:06.613300+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:11.413050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.413050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:40.678557+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":499,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620151+0000","last_change":"2024-09-13T13:11:55.549318+0000","last_active":"2024-09-13T13:12:06.620151+0000","last_peered":"2024-09-13T13:12:06.620151+0000","last_clean":"2024-09-13T13:12:06.620151+0000","last_became_active":"2024-09-13T13:11:55.548984+0000","last_became_peered":"2024-09-13T13:11:55.548984+0000","last_unstale":"2024-09-13T13:12:06.620151+0000","last_undegraded":"2024-09-13T13:12:06.620151+0000","last_fullsized":"2024-09-13T13:12:06.620151+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:42.489264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.489264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"288'9768","reported_seq":9964,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:08.200089+0000","last_change":"2024-09-13T13:11:48.441719+0000","last_active":"2024-09-13T13:12:08.200089+0000","last_peered":"2024-09-13T13:12:08.200089+0000","last_clean":"2024-09-13T13:12:08.200089+0000","last_became_active":"2024-09-13T13:10:09.270073+0000","last_became_peered":"2024-09-13T13:10:09.270073+0000","last_unstale":"2024-09-13T13:12:08.200089+0000","last_undegraded":"2024-09-13T13:12:08.200089+0000","last_fullsized":"2024-09-13T13:12:08.200089+0000","mapping_epoch":241,"log_start":"263'4857","ondisk_log_start":"263'4857","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'8433","last_scrub_stamp":"2024-09-13T13:11:48.441462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:11:48.441462+0000","objects_scrubbed":8103,"log_size":4911,"log_dups_size":0,"ondisk_log_size":4911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:54.080211+0000","scrub_duration":2466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437518336,"num_objects":6676,"num_object_clones":0,"num_object_copies":20028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9768,"num_write_kb":526208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":526,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.618342+0000","last_change":"2024-09-13T13:10:29.799315+0000","last_active":"2024-09-13T13:12:06.618342+0000","last_peered":"2024-09-13T13:12:06.618342+0000","last_clean":"2024-09-13T13:12:06.618342+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:12:06.618342+0000","last_undegraded":"2024-09-13T13:12:06.618342+0000","last_fullsized":"2024-09-13T13:12:06.618342+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:10:29.799185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:29.799185+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:39.768765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":516,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.610512+0000","last_change":"2024-09-13T13:08:56.771534+0000","last_active":"2024-09-13T13:12:06.610512+0000","last_peered":"2024-09-13T13:12:06.610512+0000","last_clean":"2024-09-13T13:12:06.610512+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:12:06.610512+0000","last_undegraded":"2024-09-13T13:12:06.610512+0000","last_fullsized":"2024-09-13T13:12:06.610512+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:56.771432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:56.771432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:30.760708+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":520,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.627990+0000","last_change":"2024-09-13T13:10:30.919483+0000","last_active":"2024-09-13T13:12:06.627990+0000","last_peered":"2024-09-13T13:12:06.627990+0000","last_clean":"2024-09-13T13:12:06.627990+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:12:06.627990+0000","last_undegraded":"2024-09-13T13:12:06.627990+0000","last_fullsized":"2024-09-13T13:12:06.627990+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:10:30.919398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.919398+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":484,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.626681+0000","last_change":"2024-09-13T13:11:18.610942+0000","last_active":"2024-09-13T13:12:06.626681+0000","last_peered":"2024-09-13T13:12:06.626681+0000","last_clean":"2024-09-13T13:12:06.626681+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:12:06.626681+0000","last_undegraded":"2024-09-13T13:12:06.626681+0000","last_fullsized":"2024-09-13T13:12:06.626681+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:18.610714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:18.610714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:40.534665+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":495,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619375+0000","last_change":"2024-09-13T13:08:54.232191+0000","last_active":"2024-09-13T13:12:06.619375+0000","last_peered":"2024-09-13T13:12:06.619375+0000","last_clean":"2024-09-13T13:12:06.619375+0000","last_became_active":"2024-09-13T13:04:02.751216+0000","last_became_peered":"2024-09-13T13:04:02.751216+0000","last_unstale":"2024-09-13T13:12:06.619375+0000","last_undegraded":"2024-09-13T13:12:06.619375+0000","last_fullsized":"2024-09-13T13:12:06.619375+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:54.232130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:54.232130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"288'9811","reported_seq":9916,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:09.043213+0000","last_change":"2024-09-13T13:10:09.737716+0000","last_active":"2024-09-13T13:12:09.043213+0000","last_peered":"2024-09-13T13:12:09.043213+0000","last_clean":"2024-09-13T13:12:09.043213+0000","last_became_active":"2024-09-13T13:10:09.737415+0000","last_became_peered":"2024-09-13T13:10:09.737415+0000","last_unstale":"2024-09-13T13:12:09.043213+0000","last_undegraded":"2024-09-13T13:12:09.043213+0000","last_fullsized":"2024-09-13T13:12:09.043213+0000","mapping_epoch":241,"log_start":"266'5306","ondisk_log_start":"266'5306","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:37.668471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439681024,"num_objects":6709,"num_object_clones":0,"num_object_copies":20127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":528640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":527,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.615791+0000","last_change":"2024-09-13T13:08:21.219398+0000","last_active":"2024-09-13T13:12:06.615791+0000","last_peered":"2024-09-13T13:12:06.615791+0000","last_clean":"2024-09-13T13:12:06.615791+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:12:06.615791+0000","last_undegraded":"2024-09-13T13:12:06.615791+0000","last_fullsized":"2024-09-13T13:12:06.615791+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:08:21.219300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:08:21.219300+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":510,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.614179+0000","last_change":"2024-09-13T13:11:55.548203+0000","last_active":"2024-09-13T13:12:06.614179+0000","last_peered":"2024-09-13T13:12:06.614179+0000","last_clean":"2024-09-13T13:12:06.614179+0000","last_became_active":"2024-09-13T13:11:55.547881+0000","last_became_peered":"2024-09-13T13:11:55.547881+0000","last_unstale":"2024-09-13T13:12:06.614179+0000","last_undegraded":"2024-09-13T13:12:06.614179+0000","last_fullsized":"2024-09-13T13:12:06.614179+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:37.465261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:37.465261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1164,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.618850+0000","last_change":"2024-09-13T13:08:49.264831+0000","last_active":"2024-09-13T13:12:06.618850+0000","last_peered":"2024-09-13T13:12:06.618850+0000","last_clean":"2024-09-13T13:12:06.618850+0000","last_became_active":"2024-09-13T13:01:02.177378+0000","last_became_peered":"2024-09-13T13:01:02.177378+0000","last_unstale":"2024-09-13T13:12:06.618850+0000","last_undegraded":"2024-09-13T13:12:06.618850+0000","last_fullsized":"2024-09-13T13:12:06.618850+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:08:49.264775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:49.264775+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":659,"num_read_kb":654,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":480,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620471+0000","last_change":"2024-09-13T13:11:49.911602+0000","last_active":"2024-09-13T13:12:06.620471+0000","last_peered":"2024-09-13T13:12:06.620471+0000","last_clean":"2024-09-13T13:12:06.620471+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:12:06.620471+0000","last_undegraded":"2024-09-13T13:12:06.620471+0000","last_fullsized":"2024-09-13T13:12:06.620471+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:49.911558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.911558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:54.115361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":501,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619170+0000","last_change":"2024-09-13T13:10:32.720742+0000","last_active":"2024-09-13T13:12:06.619170+0000","last_peered":"2024-09-13T13:12:06.619170+0000","last_clean":"2024-09-13T13:12:06.619170+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:12:06.619170+0000","last_undegraded":"2024-09-13T13:12:06.619170+0000","last_fullsized":"2024-09-13T13:12:06.619170+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:41.757119+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"288'9752","reported_seq":9857,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:08.821459+0000","last_change":"2024-09-13T13:10:09.276701+0000","last_active":"2024-09-13T13:12:08.821459+0000","last_peered":"2024-09-13T13:12:08.821459+0000","last_clean":"2024-09-13T13:12:08.821459+0000","last_became_active":"2024-09-13T13:10:09.276418+0000","last_became_peered":"2024-09-13T13:10:09.276418+0000","last_unstale":"2024-09-13T13:12:08.821459+0000","last_undegraded":"2024-09-13T13:12:08.821459+0000","last_fullsized":"2024-09-13T13:12:08.821459+0000","mapping_epoch":241,"log_start":"267'5372","ondisk_log_start":"267'5372","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:35.800142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433979392,"num_objects":6622,"num_object_clones":0,"num_object_copies":19866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9752,"num_write_kb":523968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"284'31","reported_seq":546,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-13T13:12:04.619566+0000","last_change":"2024-09-13T13:11:55.574964+0000","last_active":"2024-09-13T13:12:04.619566+0000","last_peered":"2024-09-13T13:12:04.619566+0000","last_clean":"2024-09-13T13:12:04.619566+0000","last_became_active":"2024-09-13T13:11:55.550342+0000","last_became_peered":"2024-09-13T13:11:55.550342+0000","last_unstale":"2024-09-13T13:12:04.619566+0000","last_undegraded":"2024-09-13T13:12:04.619566+0000","last_fullsized":"2024-09-13T13:12:04.619566+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'18","last_scrub_stamp":"2024-09-13T13:06:51.193678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.193678+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:03.218034+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":507,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.621017+0000","last_change":"2024-09-13T13:11:54.028616+0000","last_active":"2024-09-13T13:12:06.621017+0000","last_peered":"2024-09-13T13:12:06.621017+0000","last_clean":"2024-09-13T13:12:06.621017+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:12:06.621017+0000","last_undegraded":"2024-09-13T13:12:06.621017+0000","last_fullsized":"2024-09-13T13:12:06.621017+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:54.028552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:54.028552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:06.409077+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":545,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.618919+0000","last_change":"2024-09-13T13:06:39.564599+0000","last_active":"2024-09-13T13:12:06.618919+0000","last_peered":"2024-09-13T13:12:06.618919+0000","last_clean":"2024-09-13T13:12:06.618919+0000","last_became_active":"2024-09-13T13:04:02.752942+0000","last_became_peered":"2024-09-13T13:04:02.752942+0000","last_unstale":"2024-09-13T13:12:06.618919+0000","last_undegraded":"2024-09-13T13:12:06.618919+0000","last_fullsized":"2024-09-13T13:12:06.618919+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:06:39.564549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:06:39.564549+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:40.953558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":504,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619234+0000","last_change":"2024-09-13T13:11:50.986822+0000","last_active":"2024-09-13T13:12:06.619234+0000","last_peered":"2024-09-13T13:12:06.619234+0000","last_clean":"2024-09-13T13:12:06.619234+0000","last_became_active":"2024-09-13T13:01:06.206750+0000","last_became_peered":"2024-09-13T13:01:06.206750+0000","last_unstale":"2024-09-13T13:12:06.619234+0000","last_undegraded":"2024-09-13T13:12:06.619234+0000","last_fullsized":"2024-09-13T13:12:06.619234+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:50.986772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:50.986772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:18.743697+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":507,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620307+0000","last_change":"2024-09-13T13:11:55.551866+0000","last_active":"2024-09-13T13:12:06.620307+0000","last_peered":"2024-09-13T13:12:06.620307+0000","last_clean":"2024-09-13T13:12:06.620307+0000","last_became_active":"2024-09-13T13:11:55.551448+0000","last_became_peered":"2024-09-13T13:11:55.551448+0000","last_unstale":"2024-09-13T13:12:06.620307+0000","last_undegraded":"2024-09-13T13:12:06.620307+0000","last_fullsized":"2024-09-13T13:12:06.620307+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:09.926319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.926319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"288'9671","reported_seq":9846,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:08.124013+0000","last_change":"2024-09-13T13:11:16.781686+0000","last_active":"2024-09-13T13:12:08.124013+0000","last_peered":"2024-09-13T13:12:08.124013+0000","last_clean":"2024-09-13T13:12:08.124013+0000","last_became_active":"2024-09-13T13:10:09.273226+0000","last_became_peered":"2024-09-13T13:10:09.273226+0000","last_unstale":"2024-09-13T13:12:08.124013+0000","last_undegraded":"2024-09-13T13:12:08.124013+0000","last_fullsized":"2024-09-13T13:12:08.124013+0000","mapping_epoch":241,"log_start":"263'5030","ondisk_log_start":"263'5030","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'6147","last_scrub_stamp":"2024-09-13T13:11:16.781526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:11:16.781526+0000","objects_scrubbed":6090,"log_size":4641,"log_dups_size":0,"ondisk_log_size":4641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:20.503549+0000","scrub_duration":1943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443613184,"num_objects":6769,"num_object_clones":0,"num_object_copies":20307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9671,"num_write_kb":526080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":525,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-13T13:12:04.619527+0000","last_change":"2024-09-13T13:11:55.585935+0000","last_active":"2024-09-13T13:12:04.619527+0000","last_peered":"2024-09-13T13:12:04.619527+0000","last_clean":"2024-09-13T13:12:04.619527+0000","last_became_active":"2024-09-13T13:11:55.556430+0000","last_became_peered":"2024-09-13T13:11:55.556430+0000","last_unstale":"2024-09-13T13:12:04.619527+0000","last_undegraded":"2024-09-13T13:12:04.619527+0000","last_fullsized":"2024-09-13T13:12:04.619527+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:10:30.736543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.736543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.627855+0000","last_change":"2024-09-13T13:11:55.551962+0000","last_active":"2024-09-13T13:12:06.627855+0000","last_peered":"2024-09-13T13:12:06.627855+0000","last_clean":"2024-09-13T13:12:06.627855+0000","last_became_active":"2024-09-13T13:11:55.551612+0000","last_became_peered":"2024-09-13T13:11:55.551612+0000","last_unstale":"2024-09-13T13:12:06.627855+0000","last_undegraded":"2024-09-13T13:12:06.627855+0000","last_fullsized":"2024-09-13T13:12:06.627855+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:01.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:01.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":521,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620060+0000","last_change":"2024-09-13T13:11:55.546875+0000","last_active":"2024-09-13T13:12:06.620060+0000","last_peered":"2024-09-13T13:12:06.620060+0000","last_clean":"2024-09-13T13:12:06.620060+0000","last_became_active":"2024-09-13T13:11:55.546138+0000","last_became_peered":"2024-09-13T13:11:55.546138+0000","last_unstale":"2024-09-13T13:12:06.620060+0000","last_undegraded":"2024-09-13T13:12:06.620060+0000","last_fullsized":"2024-09-13T13:12:06.620060+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:13.010123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:13.010123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:33.551178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":494,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.621138+0000","last_change":"2024-09-13T13:10:34.526736+0000","last_active":"2024-09-13T13:12:06.621138+0000","last_peered":"2024-09-13T13:12:06.621138+0000","last_clean":"2024-09-13T13:12:06.621138+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:12:06.621138+0000","last_undegraded":"2024-09-13T13:12:06.621138+0000","last_fullsized":"2024-09-13T13:12:06.621138+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:34.526666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:34.526666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"288'9679","reported_seq":9785,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:08.743438+0000","last_change":"2024-09-13T13:10:09.737339+0000","last_active":"2024-09-13T13:12:08.743438+0000","last_peered":"2024-09-13T13:12:08.743438+0000","last_clean":"2024-09-13T13:12:08.743438+0000","last_became_active":"2024-09-13T13:10:09.736988+0000","last_became_peered":"2024-09-13T13:10:09.736988+0000","last_unstale":"2024-09-13T13:12:08.743438+0000","last_undegraded":"2024-09-13T13:12:08.743438+0000","last_fullsized":"2024-09-13T13:12:08.743438+0000","mapping_epoch":241,"log_start":"267'5421","ondisk_log_start":"267'5421","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4258,"log_dups_size":0,"ondisk_log_size":4258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:34.795205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430112768,"num_objects":6563,"num_object_clones":0,"num_object_copies":19689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9679,"num_write_kb":519744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":532,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.627256+0000","last_change":"2024-09-13T13:11:55.577009+0000","last_active":"2024-09-13T13:12:06.627256+0000","last_peered":"2024-09-13T13:12:06.627256+0000","last_clean":"2024-09-13T13:12:06.627256+0000","last_became_active":"2024-09-13T13:11:55.552178+0000","last_became_peered":"2024-09-13T13:11:55.552178+0000","last_unstale":"2024-09-13T13:12:06.627256+0000","last_undegraded":"2024-09-13T13:12:06.627256+0000","last_fullsized":"2024-09-13T13:12:06.627256+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'17","last_scrub_stamp":"2024-09-13T13:05:34.166378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:05:34.166378+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:59.515418+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":493,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-13T13:12:04.619390+0000","last_change":"2024-09-13T13:11:55.564657+0000","last_active":"2024-09-13T13:12:04.619390+0000","last_peered":"2024-09-13T13:12:04.619390+0000","last_clean":"2024-09-13T13:12:04.619390+0000","last_became_active":"2024-09-13T13:11:55.564155+0000","last_became_peered":"2024-09-13T13:11:55.564155+0000","last_unstale":"2024-09-13T13:12:04.619390+0000","last_undegraded":"2024-09-13T13:12:04.619390+0000","last_fullsized":"2024-09-13T13:12:04.619390+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:48.254929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:48.254929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:14.250286+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":487,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619268+0000","last_change":"2024-09-13T13:11:55.562178+0000","last_active":"2024-09-13T13:12:06.619268+0000","last_peered":"2024-09-13T13:12:06.619268+0000","last_clean":"2024-09-13T13:12:06.619268+0000","last_became_active":"2024-09-13T13:11:55.559489+0000","last_became_peered":"2024-09-13T13:11:55.559489+0000","last_unstale":"2024-09-13T13:12:06.619268+0000","last_undegraded":"2024-09-13T13:12:06.619268+0000","last_fullsized":"2024-09-13T13:12:06.619268+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.747302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.747302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":491,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.627622+0000","last_change":"2024-09-13T13:11:55.541212+0000","last_active":"2024-09-13T13:12:06.627622+0000","last_peered":"2024-09-13T13:12:06.627622+0000","last_clean":"2024-09-13T13:12:06.627622+0000","last_became_active":"2024-09-13T13:11:55.539746+0000","last_became_peered":"2024-09-13T13:11:55.539746+0000","last_unstale":"2024-09-13T13:12:06.627622+0000","last_undegraded":"2024-09-13T13:12:06.627622+0000","last_fullsized":"2024-09-13T13:12:06.627622+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:03.015233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:03.015233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"288'9454","reported_seq":9557,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:08.833519+0000","last_change":"2024-09-13T13:10:09.737741+0000","last_active":"2024-09-13T13:12:08.833519+0000","last_peered":"2024-09-13T13:12:08.833519+0000","last_clean":"2024-09-13T13:12:08.833519+0000","last_became_active":"2024-09-13T13:10:09.737415+0000","last_became_peered":"2024-09-13T13:10:09.737415+0000","last_unstale":"2024-09-13T13:12:08.833519+0000","last_undegraded":"2024-09-13T13:12:08.833519+0000","last_fullsized":"2024-09-13T13:12:08.833519+0000","mapping_epoch":241,"log_start":"266'5125","ondisk_log_start":"266'5125","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:36.751388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425852928,"num_objects":6498,"num_object_clones":0,"num_object_copies":19494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9454,"num_write_kb":510464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":542,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.621624+0000","last_change":"2024-09-13T13:11:55.597192+0000","last_active":"2024-09-13T13:12:06.621624+0000","last_peered":"2024-09-13T13:12:06.621624+0000","last_clean":"2024-09-13T13:12:06.621624+0000","last_became_active":"2024-09-13T13:11:55.559435+0000","last_became_peered":"2024-09-13T13:11:55.559435+0000","last_unstale":"2024-09-13T13:12:06.621624+0000","last_undegraded":"2024-09-13T13:12:06.621624+0000","last_fullsized":"2024-09-13T13:12:06.621624+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:09.739184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.739184+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":502,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620843+0000","last_change":"2024-09-13T13:11:55.996403+0000","last_active":"2024-09-13T13:12:06.620843+0000","last_peered":"2024-09-13T13:12:06.620843+0000","last_clean":"2024-09-13T13:12:06.620843+0000","last_became_active":"2024-09-13T13:11:55.557309+0000","last_became_peered":"2024-09-13T13:11:55.557309+0000","last_unstale":"2024-09-13T13:12:06.620843+0000","last_undegraded":"2024-09-13T13:12:06.620843+0000","last_fullsized":"2024-09-13T13:12:06.620843+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.996346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.996346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:20.097787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":622,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.627209+0000","last_change":"2024-09-13T13:05:32.256537+0000","last_active":"2024-09-13T13:12:06.627209+0000","last_peered":"2024-09-13T13:12:06.627209+0000","last_clean":"2024-09-13T13:12:06.627209+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:12:06.627209+0000","last_undegraded":"2024-09-13T13:12:06.627209+0000","last_fullsized":"2024-09-13T13:12:06.627209+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:05:32.256448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:32.256448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:26.540591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":23,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620169+0000","last_change":"2024-09-13T13:11:55.555873+0000","last_active":"2024-09-13T13:12:06.620169+0000","last_peered":"2024-09-13T13:12:06.620169+0000","last_clean":"2024-09-13T13:12:06.620169+0000","last_became_active":"2024-09-13T13:11:55.555400+0000","last_became_peered":"2024-09-13T13:11:55.555400+0000","last_unstale":"2024-09-13T13:12:06.620169+0000","last_undegraded":"2024-09-13T13:12:06.620169+0000","last_fullsized":"2024-09-13T13:12:06.620169+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:05.084772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:05.084772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:15.626243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"288'9663","reported_seq":9833,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:08.229374+0000","last_change":"2024-09-13T13:11:11.072082+0000","last_active":"2024-09-13T13:12:08.229374+0000","last_peered":"2024-09-13T13:12:08.229374+0000","last_clean":"2024-09-13T13:12:08.229374+0000","last_became_active":"2024-09-13T13:10:09.274589+0000","last_became_peered":"2024-09-13T13:10:09.274589+0000","last_unstale":"2024-09-13T13:12:08.229374+0000","last_undegraded":"2024-09-13T13:12:08.229374+0000","last_fullsized":"2024-09-13T13:12:08.229374+0000","mapping_epoch":241,"log_start":"262'4763","ondisk_log_start":"262'4763","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'5724","last_scrub_stamp":"2024-09-13T13:11:11.071957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.071957+0000","objects_scrubbed":5659,"log_size":4900,"log_dups_size":0,"ondisk_log_size":4900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:14.421303+0000","scrub_duration":1921,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434569216,"num_objects":6631,"num_object_clones":0,"num_object_copies":19893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9663,"num_write_kb":521408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":516,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.613684+0000","last_change":"2024-09-13T13:11:55.575538+0000","last_active":"2024-09-13T13:12:06.613684+0000","last_peered":"2024-09-13T13:12:06.613684+0000","last_clean":"2024-09-13T13:12:06.613684+0000","last_became_active":"2024-09-13T13:11:55.553805+0000","last_became_peered":"2024-09-13T13:11:55.553805+0000","last_unstale":"2024-09-13T13:12:06.613684+0000","last_undegraded":"2024-09-13T13:12:06.613684+0000","last_fullsized":"2024-09-13T13:12:06.613684+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:03:16.475781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:16.475781+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":522,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620490+0000","last_change":"2024-09-13T13:10:41.028120+0000","last_active":"2024-09-13T13:12:06.620490+0000","last_peered":"2024-09-13T13:12:06.620490+0000","last_clean":"2024-09-13T13:12:06.620490+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:12:06.620490+0000","last_undegraded":"2024-09-13T13:12:06.620490+0000","last_fullsized":"2024-09-13T13:12:06.620490+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:41.028047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:41.028047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:06.786834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":512,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.626228+0000","last_change":"2024-09-13T13:11:55.537205+0000","last_active":"2024-09-13T13:12:06.626228+0000","last_peered":"2024-09-13T13:12:06.626228+0000","last_clean":"2024-09-13T13:12:06.626228+0000","last_became_active":"2024-09-13T13:11:55.536582+0000","last_became_peered":"2024-09-13T13:11:55.536582+0000","last_unstale":"2024-09-13T13:12:06.626228+0000","last_undegraded":"2024-09-13T13:12:06.626228+0000","last_fullsized":"2024-09-13T13:12:06.626228+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.527187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.527187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:57.145054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":483,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.614082+0000","last_change":"2024-09-13T13:11:10.432159+0000","last_active":"2024-09-13T13:12:06.614082+0000","last_peered":"2024-09-13T13:12:06.614082+0000","last_clean":"2024-09-13T13:12:06.614082+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:12:06.614082+0000","last_undegraded":"2024-09-13T13:12:06.614082+0000","last_fullsized":"2024-09-13T13:12:06.614082+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:10.432086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:10.432086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:13.775919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"288'9862","reported_seq":9967,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:08.240169+0000","last_change":"2024-09-13T13:10:09.271210+0000","last_active":"2024-09-13T13:12:08.240169+0000","last_peered":"2024-09-13T13:12:08.240169+0000","last_clean":"2024-09-13T13:12:08.240169+0000","last_became_active":"2024-09-13T13:10:09.270909+0000","last_became_peered":"2024-09-13T13:10:09.270909+0000","last_unstale":"2024-09-13T13:12:08.240169+0000","last_undegraded":"2024-09-13T13:12:08.240169+0000","last_fullsized":"2024-09-13T13:12:08.240169+0000","mapping_epoch":241,"log_start":"263'4971","ondisk_log_start":"263'4971","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4891,"log_dups_size":0,"ondisk_log_size":4891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:27.029894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443023360,"num_objects":6760,"num_object_clones":0,"num_object_copies":20280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9862,"num_write_kb":531904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":505,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-13T13:12:04.619266+0000","last_change":"2024-09-13T13:09:32.849360+0000","last_active":"2024-09-13T13:12:04.619266+0000","last_peered":"2024-09-13T13:12:04.619266+0000","last_clean":"2024-09-13T13:12:04.619266+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:12:04.619266+0000","last_undegraded":"2024-09-13T13:12:04.619266+0000","last_fullsized":"2024-09-13T13:12:04.619266+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'19","last_scrub_stamp":"2024-09-13T13:09:32.849277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:32.849277+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":514,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620271+0000","last_change":"2024-09-13T13:10:27.802685+0000","last_active":"2024-09-13T13:12:06.620271+0000","last_peered":"2024-09-13T13:12:06.620271+0000","last_clean":"2024-09-13T13:12:06.620271+0000","last_became_active":"2024-09-13T13:04:02.759113+0000","last_became_peered":"2024-09-13T13:04:02.759113+0000","last_unstale":"2024-09-13T13:12:06.620271+0000","last_undegraded":"2024-09-13T13:12:06.620271+0000","last_fullsized":"2024-09-13T13:12:06.620271+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:27.802621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:27.802621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:41.947273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":496,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.626367+0000","last_change":"2024-09-13T13:08:26.033729+0000","last_active":"2024-09-13T13:12:06.626367+0000","last_peered":"2024-09-13T13:12:06.626367+0000","last_clean":"2024-09-13T13:12:06.626367+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:12:06.626367+0000","last_undegraded":"2024-09-13T13:12:06.626367+0000","last_fullsized":"2024-09-13T13:12:06.626367+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:26.033663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:26.033663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":492,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.618850+0000","last_change":"2024-09-13T13:10:40.961803+0000","last_active":"2024-09-13T13:12:06.618850+0000","last_peered":"2024-09-13T13:12:06.618850+0000","last_clean":"2024-09-13T13:12:06.618850+0000","last_became_active":"2024-09-13T13:01:08.211835+0000","last_became_peered":"2024-09-13T13:01:08.211835+0000","last_unstale":"2024-09-13T13:12:06.618850+0000","last_undegraded":"2024-09-13T13:12:06.618850+0000","last_fullsized":"2024-09-13T13:12:06.618850+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:40.961744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.961744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"288'9730","reported_seq":12095,"reported_epoch":288,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T13:12:08.744715+0000","last_change":"2024-09-13T13:11:56.786537+0000","last_active":"2024-09-13T13:12:08.744715+0000","last_peered":"2024-09-13T13:12:08.744715+0000","last_clean":"2024-09-13T13:11:54.424909+0000","last_became_active":"2024-09-13T13:11:56.666906+0000","last_became_peered":"2024-09-13T13:11:56.666906+0000","last_unstale":"2024-09-13T13:12:08.744715+0000","last_undegraded":"2024-09-13T13:12:08.744715+0000","last_fullsized":"2024-09-13T13:12:08.744715+0000","mapping_epoch":283,"log_start":"267'5501","ondisk_log_start":"267'5501","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'6035","last_scrub_stamp":"2024-09-13T13:11:15.826744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:11:15.826744+0000","objects_scrubbed":5970,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1936,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434110464,"num_objects":6624,"num_object_clones":0,"num_object_copies":19872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5594,"num_objects_unfound":0,"num_objects_dirty":6624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9730,"num_write_kb":523328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1093,"num_bytes_recovered":71565312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,4],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"1,3,4","objects":6624}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":507,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.610896+0000","last_change":"2024-09-13T13:11:55.590099+0000","last_active":"2024-09-13T13:12:06.610896+0000","last_peered":"2024-09-13T13:12:06.610896+0000","last_clean":"2024-09-13T13:12:06.610896+0000","last_became_active":"2024-09-13T13:11:55.558767+0000","last_became_peered":"2024-09-13T13:11:55.558767+0000","last_unstale":"2024-09-13T13:12:06.610896+0000","last_undegraded":"2024-09-13T13:12:06.610896+0000","last_fullsized":"2024-09-13T13:12:06.610896+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2024-09-13T13:09:30.614748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.614748+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":510,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620154+0000","last_change":"2024-09-13T13:11:55.556814+0000","last_active":"2024-09-13T13:12:06.620154+0000","last_peered":"2024-09-13T13:12:06.620154+0000","last_clean":"2024-09-13T13:12:06.620154+0000","last_became_active":"2024-09-13T13:11:55.556060+0000","last_became_peered":"2024-09-13T13:11:55.556060+0000","last_unstale":"2024-09-13T13:12:06.620154+0000","last_undegraded":"2024-09-13T13:12:06.620154+0000","last_fullsized":"2024-09-13T13:12:06.620154+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:18.821868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:18.821868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:44.420419+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":481,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.628170+0000","last_change":"2024-09-13T13:06:42.450669+0000","last_active":"2024-09-13T13:12:06.628170+0000","last_peered":"2024-09-13T13:12:06.628170+0000","last_clean":"2024-09-13T13:12:06.628170+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:12:06.628170+0000","last_undegraded":"2024-09-13T13:12:06.628170+0000","last_fullsized":"2024-09-13T13:12:06.628170+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:42.450598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:06:42.450598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:35.443401+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":475,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619898+0000","last_change":"2024-09-13T13:11:17.667223+0000","last_active":"2024-09-13T13:12:06.619898+0000","last_peered":"2024-09-13T13:12:06.619898+0000","last_clean":"2024-09-13T13:12:06.619898+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:12:06.619898+0000","last_undegraded":"2024-09-13T13:12:06.619898+0000","last_fullsized":"2024-09-13T13:12:06.619898+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:17.667151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:17.667151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:46.207860+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"288'9616","reported_seq":9720,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:09.090149+0000","last_change":"2024-09-13T13:10:09.275649+0000","last_active":"2024-09-13T13:12:09.090149+0000","last_peered":"2024-09-13T13:12:09.090149+0000","last_clean":"2024-09-13T13:12:09.090149+0000","last_became_active":"2024-09-13T13:10:09.275320+0000","last_became_peered":"2024-09-13T13:10:09.275320+0000","last_unstale":"2024-09-13T13:12:09.090149+0000","last_undegraded":"2024-09-13T13:12:09.090149+0000","last_fullsized":"2024-09-13T13:12:09.090149+0000","mapping_epoch":241,"log_start":"265'5144","ondisk_log_start":"265'5144","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:10:08.250457+0000","last_clean_scrub_stamp":"2024-09-13T13:10:08.250457+0000","objects_scrubbed":0,"log_size":4472,"log_dups_size":0,"ondisk_log_size":4472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:21.621303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432537600,"num_objects":6600,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9616,"num_write_kb":518912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":554,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.618877+0000","last_change":"2024-09-13T13:11:51.970014+0000","last_active":"2024-09-13T13:12:06.618877+0000","last_peered":"2024-09-13T13:12:06.618877+0000","last_clean":"2024-09-13T13:12:06.618877+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:12:06.618877+0000","last_undegraded":"2024-09-13T13:12:06.618877+0000","last_fullsized":"2024-09-13T13:12:06.618877+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:11:51.969805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:51.969805+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:12.216327+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":520,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619624+0000","last_change":"2024-09-13T13:11:55.590960+0000","last_active":"2024-09-13T13:12:06.619624+0000","last_peered":"2024-09-13T13:12:06.619624+0000","last_clean":"2024-09-13T13:12:06.619624+0000","last_became_active":"2024-09-13T13:11:55.556327+0000","last_became_peered":"2024-09-13T13:11:55.556327+0000","last_unstale":"2024-09-13T13:12:06.619624+0000","last_undegraded":"2024-09-13T13:12:06.619624+0000","last_fullsized":"2024-09-13T13:12:06.619624+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:06:51.627691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:51.627691+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"284'11","reported_seq":658,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.618942+0000","last_change":"2024-09-13T13:11:55.565463+0000","last_active":"2024-09-13T13:12:06.618942+0000","last_peered":"2024-09-13T13:12:06.618942+0000","last_clean":"2024-09-13T13:12:06.618942+0000","last_became_active":"2024-09-13T13:11:55.550120+0000","last_became_peered":"2024-09-13T13:11:55.550120+0000","last_unstale":"2024-09-13T13:12:06.618942+0000","last_undegraded":"2024-09-13T13:12:06.618942+0000","last_fullsized":"2024-09-13T13:12:06.618942+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:11:11.751078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.751078+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:40.136066+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":495,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620722+0000","last_change":"2024-09-13T13:10:42.034653+0000","last_active":"2024-09-13T13:12:06.620722+0000","last_peered":"2024-09-13T13:12:06.620722+0000","last_clean":"2024-09-13T13:12:06.620722+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:12:06.620722+0000","last_undegraded":"2024-09-13T13:12:06.620722+0000","last_fullsized":"2024-09-13T13:12:06.620722+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:42.034601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.034601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":470,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-13T13:12:04.619184+0000","last_change":"2024-09-13T13:11:52.186613+0000","last_active":"2024-09-13T13:12:04.619184+0000","last_peered":"2024-09-13T13:12:04.619184+0000","last_clean":"2024-09-13T13:12:04.619184+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:12:04.619184+0000","last_undegraded":"2024-09-13T13:12:04.619184+0000","last_fullsized":"2024-09-13T13:12:04.619184+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:52.186542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:52.186542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:52.263627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":501,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.621120+0000","last_change":"2024-09-13T13:09:37.358795+0000","last_active":"2024-09-13T13:12:06.621120+0000","last_peered":"2024-09-13T13:12:06.621120+0000","last_clean":"2024-09-13T13:12:06.621120+0000","last_became_active":"2024-09-13T13:01:04.207895+0000","last_became_peered":"2024-09-13T13:01:04.207895+0000","last_unstale":"2024-09-13T13:12:06.621120+0000","last_undegraded":"2024-09-13T13:12:06.621120+0000","last_fullsized":"2024-09-13T13:12:06.621120+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'6","last_scrub_stamp":"2024-09-13T13:09:37.358710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:37.358710+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":503,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620105+0000","last_change":"2024-09-13T13:11:12.054173+0000","last_active":"2024-09-13T13:12:06.620105+0000","last_peered":"2024-09-13T13:12:06.620105+0000","last_clean":"2024-09-13T13:12:06.620105+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:12:06.620105+0000","last_undegraded":"2024-09-13T13:12:06.620105+0000","last_fullsized":"2024-09-13T13:12:06.620105+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:12.054125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:12.054125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:28.049869+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":509,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619232+0000","last_change":"2024-09-13T13:11:55.552839+0000","last_active":"2024-09-13T13:12:06.619232+0000","last_peered":"2024-09-13T13:12:06.619232+0000","last_clean":"2024-09-13T13:12:06.619232+0000","last_became_active":"2024-09-13T13:11:55.552440+0000","last_became_peered":"2024-09-13T13:11:55.552440+0000","last_unstale":"2024-09-13T13:12:06.619232+0000","last_undegraded":"2024-09-13T13:12:06.619232+0000","last_fullsized":"2024-09-13T13:12:06.619232+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:30.959870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:09:30.959870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":507,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.615862+0000","last_change":"2024-09-13T13:09:34.426362+0000","last_active":"2024-09-13T13:12:06.615862+0000","last_peered":"2024-09-13T13:12:06.615862+0000","last_clean":"2024-09-13T13:12:06.615862+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:12:06.615862+0000","last_undegraded":"2024-09-13T13:12:06.615862+0000","last_fullsized":"2024-09-13T13:12:06.615862+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:09:34.426272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:34.426272+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":505,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619696+0000","last_change":"2024-09-13T13:11:55.559711+0000","last_active":"2024-09-13T13:12:06.619696+0000","last_peered":"2024-09-13T13:12:06.619696+0000","last_clean":"2024-09-13T13:12:06.619696+0000","last_became_active":"2024-09-13T13:11:55.559059+0000","last_became_peered":"2024-09-13T13:11:55.559059+0000","last_unstale":"2024-09-13T13:12:06.619696+0000","last_undegraded":"2024-09-13T13:12:06.619696+0000","last_fullsized":"2024-09-13T13:12:06.619696+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.967181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.967181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:01.131109+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":487,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620905+0000","last_change":"2024-09-13T13:11:11.553996+0000","last_active":"2024-09-13T13:12:06.620905+0000","last_peered":"2024-09-13T13:12:06.620905+0000","last_clean":"2024-09-13T13:12:06.620905+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:12:06.620905+0000","last_undegraded":"2024-09-13T13:12:06.620905+0000","last_fullsized":"2024-09-13T13:12:06.620905+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:11.553908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:11.553908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:41.237817+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":515,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.626565+0000","last_change":"2024-09-13T13:11:55.540011+0000","last_active":"2024-09-13T13:12:06.626565+0000","last_peered":"2024-09-13T13:12:06.626565+0000","last_clean":"2024-09-13T13:12:06.626565+0000","last_became_active":"2024-09-13T13:11:55.539664+0000","last_became_peered":"2024-09-13T13:11:55.539664+0000","last_unstale":"2024-09-13T13:12:06.626565+0000","last_undegraded":"2024-09-13T13:12:06.626565+0000","last_fullsized":"2024-09-13T13:12:06.626565+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:48.506365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:48.506365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:49.570360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":503,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620538+0000","last_change":"2024-09-13T13:09:48.990167+0000","last_active":"2024-09-13T13:12:06.620538+0000","last_peered":"2024-09-13T13:12:06.620538+0000","last_clean":"2024-09-13T13:12:06.620538+0000","last_became_active":"2024-09-13T13:01:04.208012+0000","last_became_peered":"2024-09-13T13:01:04.208012+0000","last_unstale":"2024-09-13T13:12:06.620538+0000","last_undegraded":"2024-09-13T13:12:06.620538+0000","last_fullsized":"2024-09-13T13:12:06.620538+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:04:53.687665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:04:53.687665+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:19.057014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":486,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.612562+0000","last_change":"2024-09-13T13:06:41.308449+0000","last_active":"2024-09-13T13:12:06.612562+0000","last_peered":"2024-09-13T13:12:06.612562+0000","last_clean":"2024-09-13T13:12:06.612562+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:12:06.612562+0000","last_undegraded":"2024-09-13T13:12:06.612562+0000","last_fullsized":"2024-09-13T13:12:06.612562+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:06:41.308367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:06:41.308367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:29.533283+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":621,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.621492+0000","last_change":"2024-09-13T13:10:07.230375+0000","last_active":"2024-09-13T13:12:06.621492+0000","last_peered":"2024-09-13T13:12:06.621492+0000","last_clean":"2024-09-13T13:12:06.621492+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:12:06.621492+0000","last_undegraded":"2024-09-13T13:12:06.621492+0000","last_fullsized":"2024-09-13T13:12:06.621492+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:10:07.230276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:07.230276+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:32.849812+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":23,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.627544+0000","last_change":"2024-09-13T13:11:55.549096+0000","last_active":"2024-09-13T13:12:06.627544+0000","last_peered":"2024-09-13T13:12:06.627544+0000","last_clean":"2024-09-13T13:12:06.627544+0000","last_became_active":"2024-09-13T13:11:55.548697+0000","last_became_peered":"2024-09-13T13:11:55.548697+0000","last_unstale":"2024-09-13T13:12:06.627544+0000","last_undegraded":"2024-09-13T13:12:06.627544+0000","last_fullsized":"2024-09-13T13:12:06.627544+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:07:21.958030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:07:21.958030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:01.461818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":521,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.621691+0000","last_change":"2024-09-13T13:09:04.635198+0000","last_active":"2024-09-13T13:12:06.621691+0000","last_peered":"2024-09-13T13:12:06.621691+0000","last_clean":"2024-09-13T13:12:06.621691+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:12:06.621691+0000","last_undegraded":"2024-09-13T13:12:06.621691+0000","last_fullsized":"2024-09-13T13:12:06.621691+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:09:04.599040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:04.599040+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":494,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619702+0000","last_change":"2024-09-13T13:11:55.544681+0000","last_active":"2024-09-13T13:12:06.619702+0000","last_peered":"2024-09-13T13:12:06.619702+0000","last_clean":"2024-09-13T13:12:06.619702+0000","last_became_active":"2024-09-13T13:11:55.543835+0000","last_became_peered":"2024-09-13T13:11:55.543835+0000","last_unstale":"2024-09-13T13:12:06.619702+0000","last_undegraded":"2024-09-13T13:12:06.619702+0000","last_fullsized":"2024-09-13T13:12:06.619702+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:13.811631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:13.811631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:35.037233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":637,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620311+0000","last_change":"2024-09-13T13:07:08.781005+0000","last_active":"2024-09-13T13:12:06.620311+0000","last_peered":"2024-09-13T13:12:06.620311+0000","last_clean":"2024-09-13T13:12:06.620311+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:12:06.620311+0000","last_undegraded":"2024-09-13T13:12:06.620311+0000","last_fullsized":"2024-09-13T13:12:06.620311+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:07:08.780952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:07:08.780952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:02.884190+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":490,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.614325+0000","last_change":"2024-09-13T13:11:49.729147+0000","last_active":"2024-09-13T13:12:06.614325+0000","last_peered":"2024-09-13T13:12:06.614325+0000","last_clean":"2024-09-13T13:12:06.614325+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:12:06.614325+0000","last_undegraded":"2024-09-13T13:12:06.614325+0000","last_fullsized":"2024-09-13T13:12:06.614325+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:49.729073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.729073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:52.700574+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":517,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-13T13:12:04.619282+0000","last_change":"2024-09-13T13:11:55.584157+0000","last_active":"2024-09-13T13:12:04.619282+0000","last_peered":"2024-09-13T13:12:04.619282+0000","last_clean":"2024-09-13T13:12:04.619282+0000","last_became_active":"2024-09-13T13:11:55.554728+0000","last_became_peered":"2024-09-13T13:11:55.554728+0000","last_unstale":"2024-09-13T13:12:04.619282+0000","last_undegraded":"2024-09-13T13:12:04.619282+0000","last_fullsized":"2024-09-13T13:12:04.619282+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:18.158584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:18.158584+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:45.536567+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":163,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.619378+0000","last_change":"2024-09-13T13:11:55.546193+0000","last_active":"2024-09-13T13:12:06.619378+0000","last_peered":"2024-09-13T13:12:06.619378+0000","last_clean":"2024-09-13T13:12:06.619378+0000","last_became_active":"2024-09-13T13:11:55.545890+0000","last_became_peered":"2024-09-13T13:11:55.545890+0000","last_unstale":"2024-09-13T13:12:06.619378+0000","last_undegraded":"2024-09-13T13:12:06.619378+0000","last_fullsized":"2024-09-13T13:12:06.619378+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:10.066281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:05:10.066281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":488,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.616917+0000","last_change":"2024-09-13T13:09:05.882862+0000","last_active":"2024-09-13T13:12:06.616917+0000","last_peered":"2024-09-13T13:12:06.616917+0000","last_clean":"2024-09-13T13:12:06.616917+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:12:06.616917+0000","last_undegraded":"2024-09-13T13:12:06.616917+0000","last_fullsized":"2024-09-13T13:12:06.616917+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:05.882769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:09:05.882769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:26.806407+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":483,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620475+0000","last_change":"2024-09-13T13:08:59.585721+0000","last_active":"2024-09-13T13:12:06.620475+0000","last_peered":"2024-09-13T13:12:06.620475+0000","last_clean":"2024-09-13T13:12:06.620475+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:12:06.620475+0000","last_undegraded":"2024-09-13T13:12:06.620475+0000","last_fullsized":"2024-09-13T13:12:06.620475+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:59.585646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:59.585646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":460,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620471+0000","last_change":"2024-09-13T13:10:10.932948+0000","last_active":"2024-09-13T13:12:06.620471+0000","last_peered":"2024-09-13T13:12:06.620471+0000","last_clean":"2024-09-13T13:12:06.620471+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:12:06.620471+0000","last_undegraded":"2024-09-13T13:12:06.620471+0000","last_fullsized":"2024-09-13T13:12:06.620471+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:10.932889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.932889+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:39.052523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":502,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620130+0000","last_change":"2024-09-13T13:11:55.561373+0000","last_active":"2024-09-13T13:12:06.620130+0000","last_peered":"2024-09-13T13:12:06.620130+0000","last_clean":"2024-09-13T13:12:06.620130+0000","last_became_active":"2024-09-13T13:11:55.560735+0000","last_became_peered":"2024-09-13T13:11:55.560735+0000","last_unstale":"2024-09-13T13:12:06.620130+0000","last_undegraded":"2024-09-13T13:12:06.620130+0000","last_fullsized":"2024-09-13T13:12:06.620130+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:09:07.118467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:09:07.118467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":487,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620110+0000","last_change":"2024-09-13T13:11:08.551743+0000","last_active":"2024-09-13T13:12:06.620110+0000","last_peered":"2024-09-13T13:12:06.620110+0000","last_clean":"2024-09-13T13:12:06.620110+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:12:06.620110+0000","last_undegraded":"2024-09-13T13:12:06.620110+0000","last_fullsized":"2024-09-13T13:12:06.620110+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:08.551449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:08.551449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:26.007369+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":490,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.613102+0000","last_change":"2024-09-13T13:10:39.477882+0000","last_active":"2024-09-13T13:12:06.613102+0000","last_peered":"2024-09-13T13:12:06.613102+0000","last_clean":"2024-09-13T13:12:06.613102+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:12:06.613102+0000","last_undegraded":"2024-09-13T13:12:06.613102+0000","last_fullsized":"2024-09-13T13:12:06.613102+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:39.477683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.477683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":517,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620808+0000","last_change":"2024-09-13T13:04:02.755696+0000","last_active":"2024-09-13T13:12:06.620808+0000","last_peered":"2024-09-13T13:12:06.620808+0000","last_clean":"2024-09-13T13:12:06.620808+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:12:06.620808+0000","last_undegraded":"2024-09-13T13:12:06.620808+0000","last_fullsized":"2024-09-13T13:12:06.620808+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'11","last_scrub_stamp":"2024-09-13T13:03:12.467503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:03:12.467503+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:23.701189+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":495,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620271+0000","last_change":"2024-09-13T13:08:50.517587+0000","last_active":"2024-09-13T13:12:06.620271+0000","last_peered":"2024-09-13T13:12:06.620271+0000","last_clean":"2024-09-13T13:12:06.620271+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:12:06.620271+0000","last_undegraded":"2024-09-13T13:12:06.620271+0000","last_fullsized":"2024-09-13T13:12:06.620271+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:50.517526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:08:50.517526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:20.085000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":505,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.618519+0000","last_change":"2024-09-13T13:11:55.555160+0000","last_active":"2024-09-13T13:12:06.618519+0000","last_peered":"2024-09-13T13:12:06.618519+0000","last_clean":"2024-09-13T13:12:06.618519+0000","last_became_active":"2024-09-13T13:11:55.554818+0000","last_became_peered":"2024-09-13T13:11:55.554818+0000","last_unstale":"2024-09-13T13:12:06.618519+0000","last_undegraded":"2024-09-13T13:12:06.618519+0000","last_fullsized":"2024-09-13T13:12:06.618519+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:53.215263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:08:53.215263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":486,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620300+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:12:06.620300+0000","last_peered":"2024-09-13T13:12:06.620300+0000","last_clean":"2024-09-13T13:12:06.620300+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:12:06.620300+0000","last_undegraded":"2024-09-13T13:12:06.620300+0000","last_fullsized":"2024-09-13T13:12:06.620300+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:09.752773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":519,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.623361+0000","last_change":"2024-09-13T13:09:39.368581+0000","last_active":"2024-09-13T13:12:06.623361+0000","last_peered":"2024-09-13T13:12:06.623361+0000","last_clean":"2024-09-13T13:12:06.623361+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:12:06.623361+0000","last_undegraded":"2024-09-13T13:12:06.623361+0000","last_fullsized":"2024-09-13T13:12:06.623361+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2024-09-13T13:09:39.368381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:09:39.368381+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":510,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.615007+0000","last_change":"2024-09-13T13:11:04.275826+0000","last_active":"2024-09-13T13:12:06.615007+0000","last_peered":"2024-09-13T13:12:06.615007+0000","last_clean":"2024-09-13T13:12:06.615007+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:12:06.615007+0000","last_undegraded":"2024-09-13T13:12:06.615007+0000","last_fullsized":"2024-09-13T13:12:06.615007+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:04.275756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:04.275756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:29.145451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":409,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-13T13:12:04.620521+0000","last_change":"2024-09-13T13:11:55.548460+0000","last_active":"2024-09-13T13:12:04.620521+0000","last_peered":"2024-09-13T13:12:04.620521+0000","last_clean":"2024-09-13T13:12:04.620521+0000","last_became_active":"2024-09-13T13:11:55.547762+0000","last_became_peered":"2024-09-13T13:11:55.547762+0000","last_unstale":"2024-09-13T13:12:04.620521+0000","last_undegraded":"2024-09-13T13:12:04.620521+0000","last_fullsized":"2024-09-13T13:12:04.620521+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:53.201129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:53.201129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:54.263434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":506,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.620221+0000","last_change":"2024-09-13T13:11:55.546694+0000","last_active":"2024-09-13T13:12:06.620221+0000","last_peered":"2024-09-13T13:12:06.620221+0000","last_clean":"2024-09-13T13:12:06.620221+0000","last_became_active":"2024-09-13T13:11:55.545749+0000","last_became_peered":"2024-09-13T13:11:55.545749+0000","last_unstale":"2024-09-13T13:12:06.620221+0000","last_undegraded":"2024-09-13T13:12:06.620221+0000","last_fullsized":"2024-09-13T13:12:06.620221+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:08.704450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:08.704450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:37.024704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":491,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.610204+0000","last_change":"2024-09-13T13:11:55.555753+0000","last_active":"2024-09-13T13:12:06.610204+0000","last_peered":"2024-09-13T13:12:06.610204+0000","last_clean":"2024-09-13T13:12:06.610204+0000","last_became_active":"2024-09-13T13:11:55.554881+0000","last_became_peered":"2024-09-13T13:11:55.554881+0000","last_unstale":"2024-09-13T13:12:06.610204+0000","last_undegraded":"2024-09-13T13:12:06.610204+0000","last_fullsized":"2024-09-13T13:12:06.610204+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:08:27.641105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:08:27.641105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":495,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.626662+0000","last_change":"2024-09-13T13:11:07.564094+0000","last_active":"2024-09-13T13:12:06.626662+0000","last_peered":"2024-09-13T13:12:06.626662+0000","last_clean":"2024-09-13T13:12:06.626662+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:12:06.626662+0000","last_undegraded":"2024-09-13T13:12:06.626662+0000","last_fullsized":"2024-09-13T13:12:06.626662+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:07.564025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:07.564025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:12:18.616035+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":500,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-13T13:12:04.619583+0000","last_change":"2024-09-13T13:11:49.264206+0000","last_active":"2024-09-13T13:12:04.619583+0000","last_peered":"2024-09-13T13:12:04.619583+0000","last_clean":"2024-09-13T13:12:04.619583+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:12:04.619583+0000","last_undegraded":"2024-09-13T13:12:04.619583+0000","last_fullsized":"2024-09-13T13:12:04.619583+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:49.264128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.264128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:12.991320+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":503,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-09-13T13:12:06.615318+0000","last_change":"2024-09-13T13:11:51.690505+0000","last_active":"2024-09-13T13:12:06.615318+0000","last_peered":"2024-09-13T13:12:06.615318+0000","last_clean":"2024-09-13T13:12:06.615318+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:12:06.615318+0000","last_undegraded":"2024-09-13T13:12:06.615318+0000","last_fullsized":"2024-09-13T13:12:06.615318+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:51.690425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:11:51.690425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:13:01.938580+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6971850776,"num_objects":106383,"num_object_clones":0,"num_object_copies":319149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16167,"num_objects_unfound":0,"num_objects_dirty":106383,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154951,"num_write_kb":8362625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3774,"num_bytes_recovered":247267328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21212180480,"data_stored":21212168264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72618,"ondisk_log_size":72618,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1116,"num_read_kb":873,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":1584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":682,"ondisk_log_size":682,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":5459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":659,"num_read_kb":654,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201416,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4235820,"kb_used_data":3857564,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":89509332,"statfs":{"total":95995035648,"available":91657555968,"internally_reserved":0,"allocated":3950145536,"data_stored":3949768224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":387316170},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141026,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1664740,"kb_used_data":1316172,"kb_used_omap":4,"kb_used_meta":348411,"kb_avail":92080412,"statfs":{"total":95995035648,"available":94290341888,"internally_reserved":0,"allocated":1347760128,"data_stored":1347390982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":356773549},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691988,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4306272,"kb_used_data":3869708,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":89438880,"statfs":{"total":95995035648,"available":91585413120,"internally_reserved":0,"allocated":3962580992,"data_stored":3962218314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":141733920922,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117153,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3913088,"kb_used_data":3412588,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":89832064,"statfs":{"total":95995035648,"available":91988033536,"internally_reserved":0,"allocated":3494490112,"data_stored":3494095841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":512424391},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411492,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1617676,"kb_used_data":1289404,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92127476,"statfs":{"total":95995035648,"available":94338535424,"internally_reserved":0,"allocated":1320349696,"data_stored":1320002828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":336132550},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575019,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4304460,"kb_used_data":3844284,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":89440692,"statfs":{"total":95995035648,"available":91587268608,"internally_reserved":0,"allocated":3936546816,"data_stored":3936136051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,6,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738546,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2324392,"kb_used_data":1950232,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91420760,"statfs":{"total":95995035648,"available":93614858240,"internally_reserved":0,"allocated":1997037568,"data_stored":1996664035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1995440128,"data_stored":1995440128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3918270464,"data_stored":3918266392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1305935872,"data_stored":1305935872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3469148160,"data_stored":3469148160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1297612800,"data_stored":1297612800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3951497216,"data_stored":3951493144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1332674560,"data_stored":1332674560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3941601280,"data_stored":3941597208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:12:11.060 INFO:tasks.thrashosds.thrasher:Setting 5.1c to [7, 1, 5] 2024-09-13T13:12:11.061 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.1c', '7', '1', '5'] 2024-09-13T13:12:11.061 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 5.1c 7 1 5 2024-09-13T13:12:11.097 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:11.161 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:11.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:11.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:11.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[29471]: pgmap v538: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 16167/319746 objects misplaced (5.056%); 25 MiB/s, 405 objects/s recovering 2024-09-13T13:12:11.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[29471]: from='client.16011 v1:172.21.15.52:0/3083509336' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:12:11.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:12:11.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:11.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[36682]: pgmap v538: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 16167/319746 objects misplaced (5.056%); 25 MiB/s, 405 objects/s recovering 2024-09-13T13:12:11.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[36682]: from='client.16011 v1:172.21.15.52:0/3083509336' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:12:11.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:12:11.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:10 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:11.264 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:11.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:10 smithi167 ceph-mon[33178]: pgmap v538: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 16167/319746 objects misplaced (5.056%); 25 MiB/s, 405 objects/s recovering 2024-09-13T13:12:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:10 smithi167 ceph-mon[33178]: from='client.16011 v1:172.21.15.52:0/3083509336' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:12:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:10 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:12:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:10 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:10 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:10 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:10 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:10 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:10 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:10 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:10 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:10 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:11.376 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:11.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:11.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:11.493 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:11.543 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:11.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:11.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:11.703 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:12:11.703 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:12:11.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:11.704 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:11.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:11.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:11.842 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:11.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:11.944 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:11.988 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:11.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:11.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:12.090 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:12.128 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:12.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:12.187 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: Adjusting osd_memory_target on smithi167 to 3248M 2024-09-13T13:12:12.188 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.188 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.188 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.188 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.188 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.188 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.188 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.188 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.188 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: Adjusting osd_memory_target on smithi052 to 3249M 2024-09-13T13:12:12.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:12.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:12:12.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:12:12.189 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:12.189 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: Adjusting osd_memory_target on smithi167 to 3248M 2024-09-13T13:12:12.189 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.189 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: Adjusting osd_memory_target on smithi052 to 3249M 2024-09-13T13:12:12.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:12.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:12:12.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:12:12.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:11 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:12.230 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:12.269 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:12.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: Adjusting osd_memory_target on smithi167 to 3248M 2024-09-13T13:12:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-13T13:12:12.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: Adjusting osd_memory_target on smithi052 to 3249M 2024-09-13T13:12:12.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:12.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:12:12.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:12:12.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:11 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:12.370 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:12.380 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:12:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:12.413 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:12.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:12.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:12.543 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:12.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:12.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:12.703 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:12.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:12.750 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T13:12:12.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:12.858 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:12.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:12.949 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:12.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:12.985 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:12.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:12.996 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:13.087 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:13.116 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:13.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:13.218 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:13.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:13.266 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:13.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:13.367 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:13.414 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:13.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:13.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:13.416 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:12:13.517 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:13.548 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:13.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:13.650 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:13.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:13.690 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:13.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:13.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:13 smithi052 ceph-mon[36682]: pgmap v539: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 16167/319746 objects misplaced (5.056%); 17 MiB/s, 265 objects/s recovering 2024-09-13T13:12:13.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2585372703' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:12:13.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:13 smithi052 ceph-mon[36682]: from='client.25858 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:12:13.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:13 smithi052 ceph-mon[29471]: pgmap v539: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 16167/319746 objects misplaced (5.056%); 17 MiB/s, 265 objects/s recovering 2024-09-13T13:12:13.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2585372703' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:12:13.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:13 smithi052 ceph-mon[29471]: from='client.25858 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:12:13.792 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:13.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:13 smithi167 ceph-mon[33178]: pgmap v539: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 16167/319746 objects misplaced (5.056%); 17 MiB/s, 265 objects/s recovering 2024-09-13T13:12:13.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2585372703' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:12:13.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:13 smithi167 ceph-mon[33178]: from='client.25858 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:12:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:13.826 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:13.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:13.927 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:13.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:13.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:14.054 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:14.085 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:14.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:14.186 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:14.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:14.232 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:14.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:14.334 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:14.364 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:14.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:14.468 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:14.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:14.493 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:14.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:14.553 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:14 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:14.553 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:14 smithi052 ceph-mon[29471]: from='client.25858 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:12:14.553 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:14 smithi052 ceph-mon[29471]: osdmap e289: 8 total, 8 up, 7 in 2024-09-13T13:12:14.553 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:14 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:14.553 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:14 smithi052 ceph-mon[36682]: from='client.25858 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:12:14.553 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:14 smithi052 ceph-mon[36682]: osdmap e289: 8 total, 8 up, 7 in 2024-09-13T13:12:14.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:14 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:14 smithi167 ceph-mon[33178]: from='client.25858 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:12:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:14 smithi167 ceph-mon[33178]: osdmap e289: 8 total, 8 up, 7 in 2024-09-13T13:12:14.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:12:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:12:14.593 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:14.625 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:14.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:14.727 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:14.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:14.769 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:14.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:14.870 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:14.892 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:14.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:14.985 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:12:14.985 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:12:14.985 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:12:14.985 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:12:14.985 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:12:14.985 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:12:14.985 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:12:14.985 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:12:14.986 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:12:14.986 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:12:14.986 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:12:14.986 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:12:14.986 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:12:14.986 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:12:14.986 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:12:14.986 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:12:14.994 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:15.031 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:15.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:15.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:15.161 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:15.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:15.263 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:15.304 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:15.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:15.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:15.406 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:15.429 INFO:teuthology.orchestra.run.smithi052.stderr:set 5.1c pg_upmap mapping to [7,1,5] 2024-09-13T13:12:15.444 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:15.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:15.546 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:15.585 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:15.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:15.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:15.716 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:15.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:15.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:15.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[36682]: pgmap v541: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 14576/310512 objects misplaced (4.694%); 21 MiB/s, 340 objects/s recovering 2024-09-13T13:12:15.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:15.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[36682]: 4.18 scrub starts 2024-09-13T13:12:15.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[36682]: 4.18 scrub ok 2024-09-13T13:12:15.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/75223666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:15.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3500407873' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 1, 5]} : dispatch 2024-09-13T13:12:15.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[36682]: from='client.25864 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 1, 5]} : dispatch 2024-09-13T13:12:15.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[29471]: pgmap v541: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 14576/310512 objects misplaced (4.694%); 21 MiB/s, 340 objects/s recovering 2024-09-13T13:12:15.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:15.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[29471]: 4.18 scrub starts 2024-09-13T13:12:15.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[29471]: 4.18 scrub ok 2024-09-13T13:12:15.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/75223666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:15.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3500407873' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 1, 5]} : dispatch 2024-09-13T13:12:15.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:15 smithi052 ceph-mon[29471]: from='client.25864 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 1, 5]} : dispatch 2024-09-13T13:12:15.818 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:15.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:15 smithi167 ceph-mon[33178]: pgmap v541: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 14576/310512 objects misplaced (4.694%); 21 MiB/s, 340 objects/s recovering 2024-09-13T13:12:15.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:15.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:15 smithi167 ceph-mon[33178]: 4.18 scrub starts 2024-09-13T13:12:15.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:15 smithi167 ceph-mon[33178]: 4.18 scrub ok 2024-09-13T13:12:15.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:15 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/75223666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:15.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:15 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3500407873' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 1, 5]} : dispatch 2024-09-13T13:12:15.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:15 smithi167 ceph-mon[33178]: from='client.25864 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 1, 5]} : dispatch 2024-09-13T13:12:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:15.853 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:15.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:15.954 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:15.996 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:15.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:16.098 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:16.137 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:16.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:16.239 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:16.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:16.277 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:16.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:16.378 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:16.408 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:16.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:16.439 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:12:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:12:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:12:16.510 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:16.531 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:16.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:16.633 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:16.675 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:16.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:16.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:16 smithi052 ceph-mon[36682]: 6.7 scrub starts 2024-09-13T13:12:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:16 smithi052 ceph-mon[36682]: 6.7 scrub ok 2024-09-13T13:12:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:16 smithi052 ceph-mon[36682]: from='client.25864 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 1, 5]}]': finished 2024-09-13T13:12:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:16 smithi052 ceph-mon[36682]: osdmap e290: 8 total, 8 up, 7 in 2024-09-13T13:12:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:16 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:12:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:16 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:12:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:16 smithi052 ceph-mon[29471]: 6.7 scrub starts 2024-09-13T13:12:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:16 smithi052 ceph-mon[29471]: 6.7 scrub ok 2024-09-13T13:12:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:16 smithi052 ceph-mon[29471]: from='client.25864 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 1, 5]}]': finished 2024-09-13T13:12:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:16 smithi052 ceph-mon[29471]: osdmap e290: 8 total, 8 up, 7 in 2024-09-13T13:12:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:16 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:12:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:16 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:12:16.777 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:16.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:16.802 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:16.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:16.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:16 smithi167 ceph-mon[33178]: 6.7 scrub starts 2024-09-13T13:12:16.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:16 smithi167 ceph-mon[33178]: 6.7 scrub ok 2024-09-13T13:12:16.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:16 smithi167 ceph-mon[33178]: from='client.25864 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 1, 5]}]': finished 2024-09-13T13:12:16.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:16 smithi167 ceph-mon[33178]: osdmap e290: 8 total, 8 up, 7 in 2024-09-13T13:12:16.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:16 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:12:16.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:16 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:12:16.903 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:16.944 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:16.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:17.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:17.075 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:17.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:17.177 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:17.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:17.224 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:17.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:17.326 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:17.360 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:17.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:17.462 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:17.484 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:17.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:17.586 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:17.628 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:17.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:17.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:17.730 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:17.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:17 smithi052 ceph-mon[36682]: pgmap v543: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 13867/303390 objects misplaced (4.571%); 25 MiB/s, 395 objects/s recovering 2024-09-13T13:12:17.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:17 smithi052 ceph-mon[36682]: osdmap e291: 8 total, 8 up, 7 in 2024-09-13T13:12:17.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:17 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:12:17.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:17 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:12:17.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:17 smithi052 ceph-mon[36682]: 5.18 scrub starts 2024-09-13T13:12:17.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:17 smithi052 ceph-mon[29471]: pgmap v543: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 13867/303390 objects misplaced (4.571%); 25 MiB/s, 395 objects/s recovering 2024-09-13T13:12:17.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:17 smithi052 ceph-mon[29471]: osdmap e291: 8 total, 8 up, 7 in 2024-09-13T13:12:17.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:17 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:12:17.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:17 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:12:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:17 smithi052 ceph-mon[29471]: 5.18 scrub starts 2024-09-13T13:12:17.758 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:17.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:17.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:17.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:17 smithi167 ceph-mon[33178]: pgmap v543: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 13867/303390 objects misplaced (4.571%); 25 MiB/s, 395 objects/s recovering 2024-09-13T13:12:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:17 smithi167 ceph-mon[33178]: osdmap e291: 8 total, 8 up, 7 in 2024-09-13T13:12:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:17 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:12:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:17 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:12:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:17 smithi167 ceph-mon[33178]: 5.18 scrub starts 2024-09-13T13:12:17.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:17.893 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:17.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:17.994 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:18.023 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:18.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:18.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:18.125 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:18.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:18.149 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:18.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:18.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:18.281 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:18.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:18.383 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:18.427 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:18.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:18.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:18.529 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:18.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:18.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:18.656 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:18.692 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:18.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:18 smithi052 ceph-mon[36682]: 3.1c scrub starts 2024-09-13T13:12:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:18 smithi052 ceph-mon[36682]: 3.1c scrub ok 2024-09-13T13:12:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:18 smithi052 ceph-mon[36682]: 5.18 scrub ok 2024-09-13T13:12:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:18 smithi052 ceph-mon[36682]: osdmap e292: 8 total, 8 up, 7 in 2024-09-13T13:12:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:18 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:12:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:18 smithi052 ceph-mon[29471]: 3.1c scrub starts 2024-09-13T13:12:18.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:18 smithi052 ceph-mon[29471]: 3.1c scrub ok 2024-09-13T13:12:18.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:18 smithi052 ceph-mon[29471]: 5.18 scrub ok 2024-09-13T13:12:18.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:18 smithi052 ceph-mon[29471]: osdmap e292: 8 total, 8 up, 7 in 2024-09-13T13:12:18.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:18 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:12:18.793 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:18.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:18.818 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:18.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:18.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:18 smithi167 ceph-mon[33178]: 3.1c scrub starts 2024-09-13T13:12:18.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:18 smithi167 ceph-mon[33178]: 3.1c scrub ok 2024-09-13T13:12:18.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:18 smithi167 ceph-mon[33178]: 5.18 scrub ok 2024-09-13T13:12:18.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:18 smithi167 ceph-mon[33178]: osdmap e292: 8 total, 8 up, 7 in 2024-09-13T13:12:18.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:18 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:12:18.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:18.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:18.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:19.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:19.090 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:19.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:19.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:19.192 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:19.213 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:19.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:19.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:19.315 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:19.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:19.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:19.446 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:19.483 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:19.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:19.585 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:19.621 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:19.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:19.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:19.723 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:19.766 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:19.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:19.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:19 smithi167 ceph-mon[33178]: pgmap v546: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 13867/303390 objects misplaced (4.571%); 12 MiB/s, 192 objects/s recovering 2024-09-13T13:12:19.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:19 smithi167 ceph-mon[33178]: 3.1d scrub starts 2024-09-13T13:12:19.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:19 smithi167 ceph-mon[33178]: 3.1d scrub ok 2024-09-13T13:12:19.867 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:19.890 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:19.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:19.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:19.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:19 smithi052 ceph-mon[29471]: pgmap v546: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 13867/303390 objects misplaced (4.571%); 12 MiB/s, 192 objects/s recovering 2024-09-13T13:12:19.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:19 smithi052 ceph-mon[29471]: 3.1d scrub starts 2024-09-13T13:12:19.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:19 smithi052 ceph-mon[29471]: 3.1d scrub ok 2024-09-13T13:12:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:19 smithi052 ceph-mon[36682]: pgmap v546: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 6.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 13867/303390 objects misplaced (4.571%); 12 MiB/s, 192 objects/s recovering 2024-09-13T13:12:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:19 smithi052 ceph-mon[36682]: 3.1d scrub starts 2024-09-13T13:12:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:19 smithi052 ceph-mon[36682]: 3.1d scrub ok 2024-09-13T13:12:19.993 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:20.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:20.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:20.130 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:20.160 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:20.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:20.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:20.262 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:20.293 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:20.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:20.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:20.395 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:20.431 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:20.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:20.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:20.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:20.565 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:20.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:20.668 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:20.692 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:20.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:20.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:20.794 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:20.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:20 smithi167 ceph-mon[33178]: 4.15 scrub starts 2024-09-13T13:12:20.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:20 smithi167 ceph-mon[33178]: 4.15 scrub ok 2024-09-13T13:12:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:20.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:20.930 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:20.959 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:20.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:20.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:20 smithi052 ceph-mon[36682]: 4.15 scrub starts 2024-09-13T13:12:20.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:20 smithi052 ceph-mon[36682]: 4.15 scrub ok 2024-09-13T13:12:20.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:20 smithi052 ceph-mon[29471]: 4.15 scrub starts 2024-09-13T13:12:20.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:20 smithi052 ceph-mon[29471]: 4.15 scrub ok 2024-09-13T13:12:21.061 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:21.093 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:21.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:21.195 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:21.226 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:21.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:21.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:21.328 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:21.360 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:21.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:21.461 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:21.497 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:21.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:21.598 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:21.635 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:21.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:21.737 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:21.768 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:21.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:21.869 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:21.913 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:21.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:21.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:21.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:21 smithi052 ceph-mon[36682]: pgmap v547: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.25k op/s; 11522/287931 objects misplaced (4.002%); 22 MiB/s, 350 objects/s recovering 2024-09-13T13:12:21.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:21 smithi052 ceph-mon[29471]: pgmap v547: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.25k op/s; 11522/287931 objects misplaced (4.002%); 22 MiB/s, 350 objects/s recovering 2024-09-13T13:12:22.014 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:22.051 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:22.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:22.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:21 smithi167 ceph-mon[33178]: pgmap v547: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.25k op/s; 11522/287931 objects misplaced (4.002%); 22 MiB/s, 350 objects/s recovering 2024-09-13T13:12:22.153 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:22.175 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:22.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:22.277 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:22.315 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:22.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:22.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:22.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:22.444 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:22.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:22.545 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:22.570 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:22.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:22.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:22.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:22.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:22.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:22.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:22.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:22.836 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:22.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:22.937 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:22.976 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:22.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:23.078 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:23.117 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:23.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:23.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:23.218 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:23.263 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:23.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:23.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:23.365 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:23.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:23.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:23.498 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:23.522 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:23.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:23.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:23.624 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:23.660 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:23.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:23.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:23.793 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:23.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:23.895 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:23.918 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:23.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:23.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:23 smithi052 ceph-mon[36682]: pgmap v548: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 797 B/s rd, 0 B/s wr, 803 op/s; 11522/287931 objects misplaced (4.002%); 16 MiB/s, 259 objects/s recovering 2024-09-13T13:12:23.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:23 smithi052 ceph-mon[29471]: pgmap v548: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 797 B/s rd, 0 B/s wr, 803 op/s; 11522/287931 objects misplaced (4.002%); 16 MiB/s, 259 objects/s recovering 2024-09-13T13:12:24.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:24.053 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:24.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:24.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:23 smithi167 ceph-mon[33178]: pgmap v548: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 797 B/s rd, 0 B/s wr, 803 op/s; 11522/287931 objects misplaced (4.002%); 16 MiB/s, 259 objects/s recovering 2024-09-13T13:12:24.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:24.183 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:24.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:24.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:24.310 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:24.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:24.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:24.413 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:24.442 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:24.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:24.544 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:24.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:12:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:12:24.575 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:24.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:24.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:24.678 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:24.721 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:24.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:24.822 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:24.844 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:24.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:24.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:24.946 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:24.976 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:24.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:24.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:25.078 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:25.121 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:25.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:25.223 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:25.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:25.362 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:25.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:25.385 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:25.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:25.486 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:25.516 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:25.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:25.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:25.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:25.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:25.652 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:25.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:25.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:25.790 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:25.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:25.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:25.892 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:25.934 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:25.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:25 smithi052 ceph-mon[36682]: pgmap v549: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 10154/279063 objects misplaced (3.639%); 21 MiB/s, 332 objects/s recovering 2024-09-13T13:12:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:25 smithi052 ceph-mon[36682]: 5.1b scrub starts 2024-09-13T13:12:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:25 smithi052 ceph-mon[36682]: 5.1b scrub ok 2024-09-13T13:12:25.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:25 smithi052 ceph-mon[29471]: pgmap v549: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 10154/279063 objects misplaced (3.639%); 21 MiB/s, 332 objects/s recovering 2024-09-13T13:12:25.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:25.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:25 smithi052 ceph-mon[29471]: 5.1b scrub starts 2024-09-13T13:12:25.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:25 smithi052 ceph-mon[29471]: 5.1b scrub ok 2024-09-13T13:12:26.035 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:26.067 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:26.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:26.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:25 smithi167 ceph-mon[33178]: pgmap v549: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 10154/279063 objects misplaced (3.639%); 21 MiB/s, 332 objects/s recovering 2024-09-13T13:12:26.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:26.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:25 smithi167 ceph-mon[33178]: 5.1b scrub starts 2024-09-13T13:12:26.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:25 smithi167 ceph-mon[33178]: 5.1b scrub ok 2024-09-13T13:12:26.169 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:26.198 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:26.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:26.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:26.300 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:26.330 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:26.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:26.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:26.432 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:26.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:26.476 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:26.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:26.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:12:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:12:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:12:26.577 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:26.607 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:26.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:26.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:26.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:26.742 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:26.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:26.766 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:26.843 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:26.898 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:26.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:26.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:26.909 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:26 smithi052 ceph-mon[36682]: 3.16 scrub starts 2024-09-13T13:12:26.909 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:26 smithi052 ceph-mon[36682]: 3.16 scrub ok 2024-09-13T13:12:26.909 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:26 smithi052 ceph-mon[29471]: 3.16 scrub starts 2024-09-13T13:12:26.909 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:26 smithi052 ceph-mon[29471]: 3.16 scrub ok 2024-09-13T13:12:27.000 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:27.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:27.032 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:27.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:27.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:26 smithi167 ceph-mon[33178]: 3.16 scrub starts 2024-09-13T13:12:27.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:26 smithi167 ceph-mon[33178]: 3.16 scrub ok 2024-09-13T13:12:27.134 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:27.163 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:27.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:27.265 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:27.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:27.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:27.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:27.430 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:27.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:27.452 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:12:27.452 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:12:27.452 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:12:27.453 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:12:27.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.453 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83963 7.c 7:30d6a006:::benchmark_data_smithi052_71865_object83962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:12:27.453 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:12:27.453 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 552.90064303300005, 2024-09-13T13:12:27.453 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.013163161, 2024-09-13T13:12:27.453 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.453 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.453 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.453 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.453 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:27.454 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:27.454 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83963 2024-09-13T13:12:27.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.454 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.454 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.454 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.454 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:12:27.454 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.454 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.454 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.454 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547882+0000", 2024-09-13T13:12:27.454 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.455 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.455 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.455 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547884+0000", 2024-09-13T13:12:27.455 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.779e-06 2024-09-13T13:12:27.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.455 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.455 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.455 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547907+0000", 2024-09-13T13:12:27.455 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2779999999999999e-05 2024-09-13T13:12:27.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.455 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.455 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.456 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547917+0000", 2024-09-13T13:12:27.456 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8439999999999993e-06 2024-09-13T13:12:27.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.456 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.456 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.547924+0000", 2024-09-13T13:12:27.456 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5229999999999998e-06 2024-09-13T13:12:27.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.456 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.456 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554417+0000", 2024-09-13T13:12:27.456 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064933880000000001 2024-09-13T13:12:27.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.457 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:12:27.457 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554427+0000", 2024-09-13T13:12:27.457 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4280000000000002e-06 2024-09-13T13:12:27.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.457 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.457 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.543883+0000", 2024-09-13T13:12:27.457 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98945604700000001 2024-09-13T13:12:27.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.457 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545504+0000", 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016211660000000001 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.545747+0000", 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024302199999999999 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552352+0000", 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.006605133 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.458 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560880+0000", 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0085281650000000007 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560953+0000", 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2096999999999998e-05 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561045+0000", 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2788999999999995e-05 2024-09-13T13:12:27.459 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.460 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.460 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.460 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.460 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83964 7.c 7:37783165:::benchmark_data_smithi052_71865_object83963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:12:27.460 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:12:27.460 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 552.900506866, 2024-09-13T13:12:27.460 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0132684729999999, 2024-09-13T13:12:27.460 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.460 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.460 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.460 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.460 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:27.461 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:27.461 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83964 2024-09-13T13:12:27.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.461 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.461 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.461 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:12:27.461 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.461 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.461 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548018+0000", 2024-09-13T13:12:27.461 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548019+0000", 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0279999999999999e-06 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548044+0000", 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4581999999999999e-05 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548052+0000", 2024-09-13T13:12:27.463 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3799999999999994e-06 2024-09-13T13:12:27.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.463 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.463 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.548059+0000", 2024-09-13T13:12:27.463 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8329999999999996e-06 2024-09-13T13:12:27.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.463 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.463 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554452+0000", 2024-09-13T13:12:27.463 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0063932119999999997 2024-09-13T13:12:27.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.464 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:12:27.464 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.554459+0000", 2024-09-13T13:12:27.464 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3319999999999997e-06 2024-09-13T13:12:27.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.464 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.464 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552381+0000", 2024-09-13T13:12:27.464 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99792229399999999 2024-09-13T13:12:27.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.464 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.464 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552485+0000", 2024-09-13T13:12:27.464 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010354800000000001 2024-09-13T13:12:27.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.465 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:12:27.465 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.552626+0000", 2024-09-13T13:12:27.465 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014122000000000001 2024-09-13T13:12:27.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.465 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.465 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.560863+0000", 2024-09-13T13:12:27.465 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0082373700000000008 2024-09-13T13:12:27.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.466 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.466 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561139+0000", 2024-09-13T13:12:27.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027546899999999997 2024-09-13T13:12:27.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.466 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.466 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.466 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561192+0000", 2024-09-13T13:12:27.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2986999999999999e-05 2024-09-13T13:12:27.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.466 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.466 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.466 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561287+0000", 2024-09-13T13:12:27.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5217999999999995e-05 2024-09-13T13:12:27.467 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.467 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.467 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.467 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.467 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83985 7.c 7:386b3d8b:::benchmark_data_smithi052_71865_object83984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:12:27.467 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:12:27.467 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 552.88058551200004, 2024-09-13T13:12:27.467 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99349958199999999, 2024-09-13T13:12:27.467 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.467 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.467 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.467 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.468 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:27.468 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:27.468 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83985 2024-09-13T13:12:27.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.468 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.468 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.468 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:12:27.468 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.468 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.468 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567940+0000", 2024-09-13T13:12:27.469 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.469 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.469 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567942+0000", 2024-09-13T13:12:27.469 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.888e-06 2024-09-13T13:12:27.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.469 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.469 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.567971+0000", 2024-09-13T13:12:27.469 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9380999999999999e-05 2024-09-13T13:12:27.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.470 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.470 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568031+0000", 2024-09-13T13:12:27.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9731000000000001e-05 2024-09-13T13:12:27.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.470 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.470 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.470 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568039+0000", 2024-09-13T13:12:27.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8109999999999992e-06 2024-09-13T13:12:27.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.470 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.470 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.470 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568102+0000", 2024-09-13T13:12:27.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2271000000000001e-05 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.568109+0000", 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2570000000000001e-06 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.557963+0000", 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.989854491 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.471 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.558066+0000", 2024-09-13T13:12:27.472 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102929 2024-09-13T13:12:27.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.472 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:12:27.472 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.558206+0000", 2024-09-13T13:12:27.472 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013928000000000001 2024-09-13T13:12:27.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.472 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.472 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561121+0000", 2024-09-13T13:12:27.472 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029151849999999998 2024-09-13T13:12:27.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.473 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.473 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561301+0000", 2024-09-13T13:12:27.473 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018019299999999999 2024-09-13T13:12:27.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.473 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.473 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.473 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561349+0000", 2024-09-13T13:12:27.473 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7639999999999998e-05 2024-09-13T13:12:27.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.473 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.473 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.473 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561439+0000", 2024-09-13T13:12:27.473 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0525000000000005e-05 2024-09-13T13:12:27.474 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.474 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.474 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.474 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.474 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:83989 7.c 7:3cf9e082:::benchmark_data_smithi052_71865_object83988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:12:27.474 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:12:27.474 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 552.87394330899997, 2024-09-13T13:12:27.474 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99086929999999995, 2024-09-13T13:12:27.474 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.474 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.474 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.475 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.475 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:12:27.475 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:12:27.475 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 83989 2024-09-13T13:12:27.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.475 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.475 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.475 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:12:27.475 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.475 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574582+0000", 2024-09-13T13:12:27.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.476 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574584+0000", 2024-09-13T13:12:27.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0480000000000001e-06 2024-09-13T13:12:27.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.476 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574603+0000", 2024-09-13T13:12:27.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8913e-05 2024-09-13T13:12:27.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.477 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.477 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574615+0000", 2024-09-13T13:12:27.477 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1678000000000001e-05 2024-09-13T13:12:27.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.477 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.477 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574622+0000", 2024-09-13T13:12:27.477 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:12:27.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.477 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.477 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574674+0000", 2024-09-13T13:12:27.478 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1053000000000001e-05 2024-09-13T13:12:27.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.478 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:12:27.478 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:14.574681+0000", 2024-09-13T13:12:27.478 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0110000000000003e-06 2024-09-13T13:12:27.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.478 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.478 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561482+0000", 2024-09-13T13:12:27.478 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98680099200000004 2024-09-13T13:12:27.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561622+0000", 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014003799999999999 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 7", 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.561773+0000", 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015180599999999999 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.564308+0000", 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025348390000000001 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.480 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.480 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565272+0000", 2024-09-13T13:12:27.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00096360599999999999 2024-09-13T13:12:27.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.480 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.480 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.480 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565362+0000", 2024-09-13T13:12:27.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0253000000000005e-05 2024-09-13T13:12:27.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.480 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.480 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.480 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:15.565451+0000", 2024-09-13T13:12:27.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9090000000000003e-05 2024-09-13T13:12:27.481 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.481 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.481 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:135978 8.0 8:077a9a64:::benchmark_data_smithi052_101302_object8049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:12:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:08.813985+0000", 2024-09-13T13:12:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 258.63453976699998, 2024-09-13T13:12:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19821777300000001, 2024-09-13T13:12:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:12:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:12:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 135978 2024-09-13T13:12:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.813985+0000", 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.813985+0000", 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.813987+0000", 2024-09-13T13:12:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7489999999999999e-06 2024-09-13T13:12:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.813992+0000", 2024-09-13T13:12:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0679999999999996e-06 2024-09-13T13:12:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814245+0000", 2024-09-13T13:12:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00025265800000000001 2024-09-13T13:12:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814253+0000", 2024-09-13T13:12:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7039999999999994e-06 2024-09-13T13:12:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814370+0000", 2024-09-13T13:12:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117674 2024-09-13T13:12:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814476+0000", 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000105457 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.814578+0000", 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102204 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.817977+0000", 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033988999999999998 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819611+0000", 2024-09-13T13:12:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016344199999999999 2024-09-13T13:12:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.012080+0000", 2024-09-13T13:12:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.192468369 2024-09-13T13:12:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.012156+0000", 2024-09-13T13:12:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6459999999999996e-05 2024-09-13T13:12:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.012203+0000", 2024-09-13T13:12:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7110000000000001e-05 2024-09-13T13:12:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:135984 8.0 8:0135d23d:::benchmark_data_smithi052_101302_object8055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:12:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:08.819395+0000", 2024-09-13T13:12:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 258.62912976299998, 2024-09-13T13:12:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.197854312, 2024-09-13T13:12:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:12:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:12:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 135984 2024-09-13T13:12:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819395+0000", 2024-09-13T13:12:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819395+0000", 2024-09-13T13:12:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819397+0000", 2024-09-13T13:12:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.945e-06 2024-09-13T13:12:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819403+0000", 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0739999999999998e-06 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819419+0000", 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5481999999999998e-05 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819427+0000", 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7300000000000005e-06 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819641+0000", 2024-09-13T13:12:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000214606 2024-09-13T13:12:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819763+0000", 2024-09-13T13:12:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121311 2024-09-13T13:12:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:12:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.819881+0000", 2024-09-13T13:12:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118528 2024-09-13T13:12:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.822831+0000", 2024-09-13T13:12:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002949587 2024-09-13T13:12:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.822973+0000", 2024-09-13T13:12:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014218999999999999 2024-09-13T13:12:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.016953+0000", 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19398000800000001 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.017192+0000", 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000238879 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.017250+0000", 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7972e-05 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:135989 8.0 8:0d783973:::benchmark_data_smithi052_101302_object8060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:12:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:08:08.824790+0000", 2024-09-13T13:12:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 258.62373566600002, 2024-09-13T13:12:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.204002616, 2024-09-13T13:12:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:12:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:12:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 135989 2024-09-13T13:12:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824790+0000", 2024-09-13T13:12:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824790+0000", 2024-09-13T13:12:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824792+0000", 2024-09-13T13:12:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.04e-06 2024-09-13T13:12:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824797+0000", 2024-09-13T13:12:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6840000000000001e-06 2024-09-13T13:12:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824813+0000", 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5529999999999999e-05 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824821+0000", 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.259e-06 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824872+0000", 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1159999999999998e-05 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.824997+0000", 2024-09-13T13:12:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012481499999999999 2024-09-13T13:12:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:12:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.825118+0000", 2024-09-13T13:12:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120907 2024-09-13T13:12:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.828631+0000", 2024-09-13T13:12:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0035128339999999998 2024-09-13T13:12:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:08.828649+0000", 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8048000000000001e-05 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.027817+0000", 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19916857399999999 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.028750+0000", 2024-09-13T13:12:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00093269499999999999 2024-09-13T13:12:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:08:09.028792+0000", 2024-09-13T13:12:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2070000000000002e-05 2024-09-13T13:12:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:116255 9.8 9:16d2e4bc:::benchmark_data_smithi052_128427_object116254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-09-13T13:12:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:33.062098+0000", 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 54.386427562999998, 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16702249599999999, 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 116255 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062098+0000", 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062098+0000", 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062100+0000", 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9180000000000001e-06 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062126+0000", 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6418999999999999e-05 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062134+0000", 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4419999999999998e-06 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062140+0000", 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6679999999999998e-06 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062189+0000", 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8779999999999997e-05 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062313+0000", 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000124189 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062454+0000", 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014138099999999999 2024-09-13T13:12:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.070948+0000", 2024-09-13T13:12:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0084940050000000007 2024-09-13T13:12:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.070996+0000", 2024-09-13T13:12:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7095999999999998e-05 2024-09-13T13:12:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.229030+0000", 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.158034759 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.229086+0000", 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5992999999999997e-05 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.229120+0000", 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3846000000000001e-05 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:116256 9.8 9:113e5c3c:::benchmark_data_smithi052_128427_object116255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-09-13T13:12:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:33.062156+0000", 2024-09-13T13:12:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 54.386368957000002, 2024-09-13T13:12:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16707438299999999, 2024-09-13T13:12:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 116256 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062156+0000", 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062156+0000", 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062157+0000", 2024-09-13T13:12:27.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5899999999999995e-07 2024-09-13T13:12:27.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062180+0000", 2024-09-13T13:12:27.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3045999999999999e-05 2024-09-13T13:12:27.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062188+0000", 2024-09-13T13:12:27.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4159999999999998e-06 2024-09-13T13:12:27.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062194+0000", 2024-09-13T13:12:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2899999999999999e-06 2024-09-13T13:12:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.065053+0000", 2024-09-13T13:12:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0028593519999999999 2024-09-13T13:12:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.065186+0000", 2024-09-13T13:12:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000133137 2024-09-13T13:12:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.065318+0000", 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013146699999999999 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.070985+0000", 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0056676570000000004 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.071000+0000", 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.411e-05 2024-09-13T13:12:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.229147+0000", 2024-09-13T13:12:27.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15814703699999999 2024-09-13T13:12:27.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.229194+0000", 2024-09-13T13:12:27.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.706e-05 2024-09-13T13:12:27.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.229231+0000", 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6952000000000001e-05 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:116274 9.8 9:1889f5f5:::benchmark_data_smithi052_128427_object116273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:33.062393+0000", 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 54.386132660000001, 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16696314400000001, 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 116274 2024-09-13T13:12:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062393+0000", 2024-09-13T13:12:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062393+0000", 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062393+0000", 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9100000000000003e-07 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062417+0000", 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3388e-05 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062424+0000", 2024-09-13T13:12:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8329999999999996e-06 2024-09-13T13:12:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062431+0000", 2024-09-13T13:12:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4880000000000001e-06 2024-09-13T13:12:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.067850+0000", 2024-09-13T13:12:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0054193740000000002 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.067968+0000", 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011770899999999999 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.068096+0000", 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012752600000000001 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.071119+0000", 2024-09-13T13:12:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003023838 2024-09-13T13:12:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.071316+0000", 2024-09-13T13:12:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019686700000000001 2024-09-13T13:12:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.229269+0000", 2024-09-13T13:12:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15795253000000001 2024-09-13T13:12:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.229319+0000", 2024-09-13T13:12:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9916000000000001e-05 2024-09-13T13:12:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.229356+0000", 2024-09-13T13:12:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6884000000000001e-05 2024-09-13T13:12:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:161595 9.2 9:4df2328c:::benchmark_data_smithi052_128427_object30926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T13:12:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.716484+0000", 2024-09-13T13:12:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 12.732041626000001, 2024-09-13T13:12:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.235957154, 2024-09-13T13:12:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 161595 2024-09-13T13:12:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716484+0000", 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716484+0000", 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716485+0000", 2024-09-13T13:12:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2500000000000001e-06 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716489+0000", 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4510000000000002e-06 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716497+0000", 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9889999999999992e-06 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716504+0000", 2024-09-13T13:12:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0779999999999998e-06 2024-09-13T13:12:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716824+0000", 2024-09-13T13:12:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000320064 2024-09-13T13:12:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716961+0000", 2024-09-13T13:12:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013622100000000001 2024-09-13T13:12:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:12:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.717058+0000", 2024-09-13T13:12:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7355999999999994e-05 2024-09-13T13:12:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.722850+0000", 2024-09-13T13:12:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057922770000000002 2024-09-13T13:12:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.725892+0000", 2024-09-13T13:12:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0030412809999999998 2024-09-13T13:12:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952206+0000", 2024-09-13T13:12:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.226314716 2024-09-13T13:12:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952289+0000", 2024-09-13T13:12:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2841000000000002e-05 2024-09-13T13:12:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952441+0000", 2024-09-13T13:12:27.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015163000000000001 2024-09-13T13:12:27.528 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.528 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.528 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.528 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:161597 9.2 9:4b6690a8:::benchmark_data_smithi052_128427_object30928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T13:12:27.528 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.722218+0000", 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 12.726307093999999, 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.23041990900000001, 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 161597 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.722218+0000", 2024-09-13T13:12:27.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.722218+0000", 2024-09-13T13:12:27.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.722220+0000", 2024-09-13T13:12:27.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.499e-06 2024-09-13T13:12:27.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.722224+0000", 2024-09-13T13:12:27.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8829999999999997e-06 2024-09-13T13:12:27.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.722236+0000", 2024-09-13T13:12:27.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1219e-05 2024-09-13T13:12:27.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.722243+0000", 2024-09-13T13:12:27.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2919999999999998e-06 2024-09-13T13:12:27.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.722889+0000", 2024-09-13T13:12:27.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00064559399999999999 2024-09-13T13:12:27.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.722995+0000", 2024-09-13T13:12:27.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000106918 2024-09-13T13:12:27.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:12:27.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.723101+0000", 2024-09-13T13:12:27.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000105301 2024-09-13T13:12:27.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.725898+0000", 2024-09-13T13:12:27.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027969650000000002 2024-09-13T13:12:27.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.727688+0000", 2024-09-13T13:12:27.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001789793 2024-09-13T13:12:27.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952549+0000", 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.22486166799999999 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952600+0000", 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0315999999999997e-05 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952638+0000", 2024-09-13T13:12:27.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8461000000000003e-05 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:161607 9.2 9:44ba942b:::benchmark_data_smithi052_128427_object30938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.732041+0000", 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 12.716484487000001, 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.22075366799999999, 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.535 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:27.536 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:27.536 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 161607 2024-09-13T13:12:27.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.536 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.732041+0000", 2024-09-13T13:12:27.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.732041+0000", 2024-09-13T13:12:27.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.732043+0000", 2024-09-13T13:12:27.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1040000000000001e-06 2024-09-13T13:12:27.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.732050+0000", 2024-09-13T13:12:27.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1099999999999997e-06 2024-09-13T13:12:27.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.732063+0000", 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3515e-05 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.732072+0000", 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2679999999999994e-06 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.732144+0000", 2024-09-13T13:12:27.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2015999999999996e-05 2024-09-13T13:12:27.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.539 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.732275+0000", 2024-09-13T13:12:27.539 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013123 2024-09-13T13:12:27.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:12:27.539 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.732387+0000", 2024-09-13T13:12:27.539 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112188 2024-09-13T13:12:27.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.736546+0000", 2024-09-13T13:12:27.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041590949999999998 2024-09-13T13:12:27.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.740545+0000", 2024-09-13T13:12:27.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039990260000000001 2024-09-13T13:12:27.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952697+0000", 2024-09-13T13:12:27.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.212152118 2024-09-13T13:12:27.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952751+0000", 2024-09-13T13:12:27.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4126000000000003e-05 2024-09-13T13:12:27.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952794+0000", 2024-09-13T13:12:27.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2871999999999999e-05 2024-09-13T13:12:27.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.541 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.542 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:161642 9.2 9:4f40cf9a:::benchmark_data_smithi052_128427_object30973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T13:12:27.542 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.764851+0000", 2024-09-13T13:12:27.542 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 12.683674671, 2024-09-13T13:12:27.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.188088281, 2024-09-13T13:12:27.542 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.542 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.542 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.542 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.542 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:27.542 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:27.542 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 161642 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.764851+0000", 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.764851+0000", 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.764852+0000", 2024-09-13T13:12:27.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.858e-06 2024-09-13T13:12:27.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.764858+0000", 2024-09-13T13:12:27.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4859999999999997e-06 2024-09-13T13:12:27.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.764895+0000", 2024-09-13T13:12:27.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7126000000000003e-05 2024-09-13T13:12:27.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.764904+0000", 2024-09-13T13:12:27.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1279999999999997e-06 2024-09-13T13:12:27.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.764973+0000", 2024-09-13T13:12:27.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8523999999999996e-05 2024-09-13T13:12:27.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.765086+0000", 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011386400000000001 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.765246+0000", 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000159448 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.767987+0000", 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027408039999999999 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.770070+0000", 2024-09-13T13:12:27.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020829210000000002 2024-09-13T13:12:27.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952850+0000", 2024-09-13T13:12:27.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.18278081700000001 2024-09-13T13:12:27.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952898+0000", 2024-09-13T13:12:27.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7034999999999999e-05 2024-09-13T13:12:27.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952939+0000", 2024-09-13T13:12:27.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1270000000000003e-05 2024-09-13T13:12:27.548 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.548 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.548 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.548 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:161648 9.2 9:4f4e0c5d:::benchmark_data_smithi052_128427_object30979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T13:12:27.548 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.772858+0000", 2024-09-13T13:12:27.549 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 12.675667287, 2024-09-13T13:12:27.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.18027220699999999, 2024-09-13T13:12:27.549 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.549 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.549 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.549 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.549 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:27.549 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:27.549 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 161648 2024-09-13T13:12:27.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.549 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.772858+0000", 2024-09-13T13:12:27.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.772858+0000", 2024-09-13T13:12:27.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.772860+0000", 2024-09-13T13:12:27.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9379999999999999e-06 2024-09-13T13:12:27.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.772865+0000", 2024-09-13T13:12:27.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4779999999999999e-06 2024-09-13T13:12:27.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.772874+0000", 2024-09-13T13:12:27.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4430000000000003e-06 2024-09-13T13:12:27.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.772880+0000", 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2790000000000004e-06 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.772952+0000", 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1831999999999999e-05 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.773077+0000", 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012463799999999999 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:12:27.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.773222+0000", 2024-09-13T13:12:27.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014515599999999999 2024-09-13T13:12:27.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.778307+0000", 2024-09-13T13:12:27.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050851250000000002 2024-09-13T13:12:27.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.778346+0000", 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9439e-05 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.953035+0000", 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.17468882799999999 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.953090+0000", 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4571999999999999e-05 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.555 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:27.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.953130+0000", 2024-09-13T13:12:27.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0479000000000003e-05 2024-09-13T13:12:27.556 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.556 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.556 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:161662 9.2 9:47e53c48:::benchmark_data_smithi052_128427_object30993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T13:12:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.787369+0000", 2024-09-13T13:12:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 12.661155947999999, 2024-09-13T13:12:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.17177567099999999, 2024-09-13T13:12:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 161662 2024-09-13T13:12:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.787369+0000", 2024-09-13T13:12:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.787369+0000", 2024-09-13T13:12:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.787371+0000", 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0190000000000001e-06 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.787378+0000", 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5899999999999996e-06 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.787391+0000", 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3008e-05 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.787400+0000", 2024-09-13T13:12:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6510000000000007e-06 2024-09-13T13:12:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.787472+0000", 2024-09-13T13:12:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2602999999999996e-05 2024-09-13T13:12:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.787612+0000", 2024-09-13T13:12:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000139942 2024-09-13T13:12:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:12:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.787749+0000", 2024-09-13T13:12:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000137293 2024-09-13T13:12:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.790802+0000", 2024-09-13T13:12:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0030526389999999998 2024-09-13T13:12:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.791573+0000", 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00077101000000000003 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.959045+0000", 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.167471756 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.959103+0000", 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8006e-05 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.959145+0000", 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2154000000000003e-05 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:163395 9.8 9:1aff1fb1:::benchmark_data_smithi052_128427_object32726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:16.425312+0000", 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 11.023213135000001, 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16665848799999999, 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 163395 2024-09-13T13:12:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.425312+0000", 2024-09-13T13:12:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.425312+0000", 2024-09-13T13:12:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.425313+0000", 2024-09-13T13:12:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.155e-06 2024-09-13T13:12:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.425318+0000", 2024-09-13T13:12:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.989e-06 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.425328+0000", 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0185e-05 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.425335+0000", 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.427323+0000", 2024-09-13T13:12:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001987754 2024-09-13T13:12:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.427436+0000", 2024-09-13T13:12:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112784 2024-09-13T13:12:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.427548+0000", 2024-09-13T13:12:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112033 2024-09-13T13:12:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.444697+0000", 2024-09-13T13:12:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.017148991999999998 2024-09-13T13:12:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.444899+0000", 2024-09-13T13:12:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020186299999999999 2024-09-13T13:12:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.591866+0000", 2024-09-13T13:12:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.14696756499999999 2024-09-13T13:12:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.591916+0000", 2024-09-13T13:12:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9653e-05 2024-09-13T13:12:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.591971+0000", 2024-09-13T13:12:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4706000000000001e-05 2024-09-13T13:12:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.570 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.570 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:163402 9.8 9:11f6e92a:::benchmark_data_smithi052_128427_object32733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-13T13:12:27.570 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:16.434277+0000", 2024-09-13T13:12:27.570 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 11.014248565000001, 2024-09-13T13:12:27.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.158062753, 2024-09-13T13:12:27.570 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.570 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.570 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.570 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.570 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:27.570 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:27.571 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 163402 2024-09-13T13:12:27.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.571 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.434277+0000", 2024-09-13T13:12:27.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.434277+0000", 2024-09-13T13:12:27.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.434278+0000", 2024-09-13T13:12:27.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.176e-06 2024-09-13T13:12:27.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.434282+0000", 2024-09-13T13:12:27.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5600000000000004e-06 2024-09-13T13:12:27.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.434293+0000", 2024-09-13T13:12:27.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1e-05 2024-09-13T13:12:27.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.434301+0000", 2024-09-13T13:12:27.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2860000000000002e-06 2024-09-13T13:12:27.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.444968+0000", 2024-09-13T13:12:27.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010667326 2024-09-13T13:12:27.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.445065+0000", 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6744999999999996e-05 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.445133+0000", 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8612e-05 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.588835+0000", 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.14370170700000001 2024-09-13T13:12:27.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.589721+0000", 2024-09-13T13:12:27.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00088626099999999995 2024-09-13T13:12:27.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.591982+0000", 2024-09-13T13:12:27.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0022603459999999999 2024-09-13T13:12:27.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.592304+0000", 2024-09-13T13:12:27.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00032281200000000001 2024-09-13T13:12:27.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:16.592339+0000", 2024-09-13T13:12:27.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4922000000000002e-05 2024-09-13T13:12:27.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.576 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.576 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:169650 9.c 9:365b7fc3:::benchmark_data_smithi052_128427_object38981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:12:27.576 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.425224+0000", 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 5.0233011660000004, 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16245214299999999, 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 169650 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425224+0000", 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425224+0000", 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425226+0000", 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.75e-06 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425231+0000", 2024-09-13T13:12:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0359999999999997e-06 2024-09-13T13:12:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425241+0000", 2024-09-13T13:12:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.048e-05 2024-09-13T13:12:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425248+0000", 2024-09-13T13:12:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0500000000000003e-06 2024-09-13T13:12:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.426749+0000", 2024-09-13T13:12:27.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001500777 2024-09-13T13:12:27.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.426875+0000", 2024-09-13T13:12:27.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012603899999999999 2024-09-13T13:12:27.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4,7", 2024-09-13T13:12:27.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.427014+0000", 2024-09-13T13:12:27.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000138706 2024-09-13T13:12:27.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.559854+0000", 2024-09-13T13:12:27.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13284027800000001 2024-09-13T13:12:27.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.559907+0000", 2024-09-13T13:12:27.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2732999999999998e-05 2024-09-13T13:12:27.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.559974+0000", 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6845000000000001e-05 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.587473+0000", 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.027499425000000001 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.587539+0000", 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6184000000000002e-05 2024-09-13T13:12:27.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.587676+0000", 2024-09-13T13:12:27.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013684 2024-09-13T13:12:27.583 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.583 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.583 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.584 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:169654 9.8 9:1a381ad4:::benchmark_data_smithi052_128427_object38985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:12:27.584 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.428910+0000", 2024-09-13T13:12:27.584 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 5.0196149600000002, 2024-09-13T13:12:27.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.158464619, 2024-09-13T13:12:27.585 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:12:27.585 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:12:27.585 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:27.585 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:12:27.585 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:12:27.585 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:27.585 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 169654 2024-09-13T13:12:27.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.585 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:12:27.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:12:27.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.428910+0000", 2024-09-13T13:12:27.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:12:27.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.428910+0000", 2024-09-13T13:12:27.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:12:27.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:12:27.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.428912+0000", 2024-09-13T13:12:27.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9819999999999998e-06 2024-09-13T13:12:27.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:12:27.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.428918+0000", 2024-09-13T13:12:27.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.981e-06 2024-09-13T13:12:27.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:12:27.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.428930+0000", 2024-09-13T13:12:27.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1644999999999999e-05 2024-09-13T13:12:27.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:12:27.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.428949+0000", 2024-09-13T13:12:27.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8875000000000001e-05 2024-09-13T13:12:27.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:12:27.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.429007+0000", 2024-09-13T13:12:27.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8103999999999998e-05 2024-09-13T13:12:27.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:12:27.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.559639+0000", 2024-09-13T13:12:27.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.130632628 2024-09-13T13:12:27.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:12:27.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.559800+0000", 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016010500000000001 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.562422+0000", 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026220200000000001 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.562490+0000", 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8016999999999996e-05 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.587254+0000", 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.024764123999999998 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.587314+0000", 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9973999999999999e-05 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.587375+0000", 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1163999999999999e-05 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.592 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:12:27.592 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:12:27.592 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:12:27.635 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:27.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:27.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:27.736 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:27.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:27 smithi052 ceph-mon[36682]: pgmap v550: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s; 9534/271884 objects misplaced (3.507%); 20 MiB/s, 322 objects/s recovering 2024-09-13T13:12:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:27 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:12:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:27 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:12:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:27 smithi052 ceph-mon[36682]: osdmap e293: 8 total, 8 up, 7 in 2024-09-13T13:12:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:27 smithi052 ceph-mon[36682]: 6.d scrub starts 2024-09-13T13:12:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:27 smithi052 ceph-mon[36682]: 6.d scrub ok 2024-09-13T13:12:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:27 smithi052 ceph-mon[29471]: pgmap v550: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s; 9534/271884 objects misplaced (3.507%); 20 MiB/s, 322 objects/s recovering 2024-09-13T13:12:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:27 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:12:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:27 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:12:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:27 smithi052 ceph-mon[29471]: osdmap e293: 8 total, 8 up, 7 in 2024-09-13T13:12:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:27 smithi052 ceph-mon[29471]: 6.d scrub starts 2024-09-13T13:12:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:27 smithi052 ceph-mon[29471]: 6.d scrub ok 2024-09-13T13:12:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:27.769 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:27.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:27.870 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:27.910 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:27.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:28.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:28.046 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:28.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:28.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:27 smithi167 ceph-mon[33178]: pgmap v550: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s; 9534/271884 objects misplaced (3.507%); 20 MiB/s, 322 objects/s recovering 2024-09-13T13:12:28.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:27 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:12:28.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:27 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:12:28.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:27 smithi167 ceph-mon[33178]: osdmap e293: 8 total, 8 up, 7 in 2024-09-13T13:12:28.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:27 smithi167 ceph-mon[33178]: 6.d scrub starts 2024-09-13T13:12:28.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:27 smithi167 ceph-mon[33178]: 6.d scrub ok 2024-09-13T13:12:28.146 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:28.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:28.187 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:28.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:28.276 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:12:28.288 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:28.314 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:28.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:28.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:28.416 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:28.442 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:28.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:28.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:28.544 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:28.586 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:28.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:28.687 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:28.730 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:28.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:28.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:28.814 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:12:28.832 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:28.918 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:28.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:28.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:28.934 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:12:29.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:29.057 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:29.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:29.080 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:29.158 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:29.217 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:29.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:29.318 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:29.335 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:12:29.362 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:29.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:29.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:29.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:29.498 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:29.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:29.600 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:29.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:29.643 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:29.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:29.744 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:29.763 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:29.791 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:29.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:29.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:29.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:29 smithi167 ceph-mon[33178]: pgmap v552: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 9534/271884 objects misplaced (3.507%); 19 MiB/s, 303 objects/s recovering 2024-09-13T13:12:29.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:29 smithi167 ceph-mon[33178]: 9.7 scrub starts 2024-09-13T13:12:29.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:29.898 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:29 smithi052 ceph-mon[36682]: pgmap v552: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 9534/271884 objects misplaced (3.507%); 19 MiB/s, 303 objects/s recovering 2024-09-13T13:12:29.898 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:29 smithi052 ceph-mon[36682]: 9.7 scrub starts 2024-09-13T13:12:29.899 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:29 smithi052 ceph-mon[29471]: pgmap v552: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 9534/271884 objects misplaced (3.507%); 19 MiB/s, 303 objects/s recovering 2024-09-13T13:12:29.899 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:29 smithi052 ceph-mon[29471]: 9.7 scrub starts 2024-09-13T13:12:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:29.924 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:29.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:30.025 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:30.058 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:30.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:30.159 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:30.193 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:30.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:30.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:30.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:30.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:30.431 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:30.462 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:30.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:30.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:30.564 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:30.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:30.610 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:30.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:30.615 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:12:30.662 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:30 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3811473361' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:30.663 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:30 smithi167 ceph-mon[33178]: from='client.25873 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:30.663 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:30 smithi167 ceph-mon[33178]: 9.7 scrub ok 2024-09-13T13:12:30.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:30.732 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:12:30.732 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:12:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:30.751 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:30.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:30.766 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:12:30.766 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:12:30.802 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:12:30.802 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:12:30.803 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:30 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3811473361' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:30.803 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:30 smithi052 ceph-mon[36682]: from='client.25873 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:30.803 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:30 smithi052 ceph-mon[36682]: 9.7 scrub ok 2024-09-13T13:12:30.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:30 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3811473361' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:30.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:30 smithi052 ceph-mon[29471]: from='client.25873 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:30.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:30 smithi052 ceph-mon[29471]: 9.7 scrub ok 2024-09-13T13:12:30.822 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:12:30.822 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:12:30.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:30.822 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.15789.0:178849 9.4 e294/283 9:2cebafe0:::benchmark_data_smithi052_128427_object48180:head v 294'11226, mlcod=294'11223)", 2024-09-13T13:12:30.822 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:30.817322+0000", 2024-09-13T13:12:30.822 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00254025, 2024-09-13T13:12:30.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025657729999999999, 2024-09-13T13:12:30.822 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:30.822 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:30.822 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:12:30.822 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:30.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:30.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:30.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:30.817322+0000", 2024-09-13T13:12:30.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:30.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:30.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:30.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:30.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:30.817322+0000", 2024-09-13T13:12:30.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:30.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:30.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:30.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:30.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:30.817324+0000", 2024-09-13T13:12:30.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.674e-06 2024-09-13T13:12:30.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:30.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:30.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:30.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:30.817332+0000", 2024-09-13T13:12:30.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0069999999999997e-06 2024-09-13T13:12:30.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:30.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:30.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:30.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:30.817349+0000", 2024-09-13T13:12:30.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6867999999999998e-05 2024-09-13T13:12:30.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:30.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:30.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:30.817358+0000", 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7760000000000003e-06 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:30.817495+0000", 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000137272 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:30.817544+0000", 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8641999999999999e-05 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:30.825 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:30.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:30.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:30.826 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:12:30.826 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-09-13T13:12:30.826 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:12:30.837 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:12:30.837 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:12:30.851 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:30.871 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:12:30.871 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:12:30.889 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:30.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:30.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:30.905 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:12:30.906 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:12:30.941 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:12:30.942 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:12:30.975 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:12:30.975 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:12:30.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:31.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:31.026 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:31.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:31.127 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:31.165 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:31.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:31.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:31.267 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:31.302 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:31.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:31.403 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:31.457 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:31.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:31.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:31.593 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:31.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:31.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T13:12:31.640 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:31 smithi167 ceph-mon[33178]: pgmap v553: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s; 7530/254766 objects misplaced (2.956%); 18 MiB/s, 282 objects/s recovering 2024-09-13T13:12:31.640 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:31 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:12:31.640 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:31 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:12:31.640 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:31 smithi167 ceph-mon[33178]: from='client.25873 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:12:31.641 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:31 smithi167 ceph-mon[33178]: osdmap e294: 8 total, 8 up, 7 in 2024-09-13T13:12:31.694 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:31.730 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:31.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:31.832 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:31.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:31.870 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:31.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:31.971 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[36682]: pgmap v553: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s; 7530/254766 objects misplaced (2.956%); 18 MiB/s, 282 objects/s recovering 2024-09-13T13:12:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:12:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:12:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[36682]: from='client.25873 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:12:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[36682]: osdmap e294: 8 total, 8 up, 7 in 2024-09-13T13:12:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2666947468' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[36682]: 9.f scrub starts 2024-09-13T13:12:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:12:31.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:12:31.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[29471]: pgmap v553: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s; 7530/254766 objects misplaced (2.956%); 18 MiB/s, 282 objects/s recovering 2024-09-13T13:12:31.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:12:31.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:12:31.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[29471]: from='client.25873 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:12:31.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[29471]: osdmap e294: 8 total, 8 up, 7 in 2024-09-13T13:12:31.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2666947468' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:31.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[29471]: 9.f scrub starts 2024-09-13T13:12:31.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:12:31.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:31 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:12:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:32.021 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:32.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:31 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2666947468' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:31 smithi167 ceph-mon[33178]: 9.f scrub starts 2024-09-13T13:12:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:31 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:12:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:31 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:12:32.082 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:12:32.122 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:32.162 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:32.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:32.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:32.264 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:32.302 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:32.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:32.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:32.455 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:32.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:32.556 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:32.599 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:32.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:32.701 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:32.727 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:12:32.728 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:12:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:32.739 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:32.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:32 smithi167 ceph-mon[33178]: osdmap e295: 8 total, 8 up, 7 in 2024-09-13T13:12:32.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:32.887 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:32.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:32.910 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:12:32.910 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:12:32.910 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T13:12:32.910 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.315503 2024-09-13T13:12:32.910 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 0.3155027063472291 2024-09-13T13:12:32.951 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:32 smithi052 ceph-mon[36682]: osdmap e295: 8 total, 8 up, 7 in 2024-09-13T13:12:32.952 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:32 smithi052 ceph-mon[29471]: osdmap e295: 8 total, 8 up, 7 in 2024-09-13T13:12:32.988 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:33.041 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:33.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:33.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:33.145 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:33.203 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:33.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:33.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:33.333 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:33.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:33.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:33.364 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:33.435 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:33.455 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:12:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:33.467 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:33.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:33.475 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T13:12:33.569 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:33.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:33.619 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:33.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:33.720 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[36682]: pgmap v556: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 951 op/s; 7530/254766 objects misplaced (2.956%); 15 MiB/s, 241 objects/s recovering 2024-09-13T13:12:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-13T13:12:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-13T13:12:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 5]} : dispatch 2024-09-13T13:12:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 5]} : dispatch 2024-09-13T13:12:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]} : dispatch 2024-09-13T13:12:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]} : dispatch 2024-09-13T13:12:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[36682]: 9.f scrub ok 2024-09-13T13:12:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:33.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:12:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[29471]: pgmap v556: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 951 op/s; 7530/254766 objects misplaced (2.956%); 15 MiB/s, 241 objects/s recovering 2024-09-13T13:12:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-13T13:12:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-13T13:12:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 5]} : dispatch 2024-09-13T13:12:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 5]} : dispatch 2024-09-13T13:12:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]} : dispatch 2024-09-13T13:12:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]} : dispatch 2024-09-13T13:12:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[29471]: 9.f scrub ok 2024-09-13T13:12:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:33 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:12:33.742 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:33.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:33.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:33.840 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:12:33.844 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:33.868 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:33.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:33.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:33.962 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:33 smithi167 ceph-mon[33178]: pgmap v556: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 951 op/s; 7530/254766 objects misplaced (2.956%); 15 MiB/s, 241 objects/s recovering 2024-09-13T13:12:33.962 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:33 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-13T13:12:33.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:33 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-13T13:12:33.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:33 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 5]} : dispatch 2024-09-13T13:12:33.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:33 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 5]} : dispatch 2024-09-13T13:12:33.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:33 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]} : dispatch 2024-09-13T13:12:33.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:33 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]} : dispatch 2024-09-13T13:12:33.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:33 smithi167 ceph-mon[33178]: 9.f scrub ok 2024-09-13T13:12:33.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:33 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:33.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:33 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:12:33.970 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:33.991 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:33.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:34.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:34.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:12:34.119 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:34.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:34.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:34.221 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:34.256 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:34.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:34.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:34.358 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:34.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:34.409 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:34.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:34.458 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:12:34.458 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:12:34.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:12:34 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:12:34.510 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:34.548 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:34.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:34.650 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:34.682 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:34.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:34.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:34.784 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:34.794 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:34 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-13T13:12:34.795 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:34 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 5]}]': finished 2024-09-13T13:12:34.795 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:34 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]}]': finished 2024-09-13T13:12:34.795 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:34 smithi167 ceph-mon[33178]: osdmap e296: 8 total, 8 up, 7 in 2024-09-13T13:12:34.795 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:34 smithi167 ceph-mon[33178]: osdmap e297: 8 total, 8 up, 7 in 2024-09-13T13:12:34.819 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:34.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:34.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:34.921 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:34.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:34.961 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:34.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:34 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-13T13:12:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:34 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 5]}]': finished 2024-09-13T13:12:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:34 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]}]': finished 2024-09-13T13:12:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:34 smithi052 ceph-mon[36682]: osdmap e296: 8 total, 8 up, 7 in 2024-09-13T13:12:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:34 smithi052 ceph-mon[36682]: osdmap e297: 8 total, 8 up, 7 in 2024-09-13T13:12:34.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:34 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-13T13:12:34.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:34 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 5]}]': finished 2024-09-13T13:12:34.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:34 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]}]': finished 2024-09-13T13:12:34.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:34 smithi052 ceph-mon[29471]: osdmap e296: 8 total, 8 up, 7 in 2024-09-13T13:12:34.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:34 smithi052 ceph-mon[29471]: osdmap e297: 8 total, 8 up, 7 in 2024-09-13T13:12:35.062 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:35.102 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:35.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:35.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:35.144 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T13:12:35.204 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:35.239 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:35.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:35.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:35.341 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:35.380 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:35.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:35.481 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:35.526 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:35.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:35.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:35.552 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:12:35.628 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:35.667 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:35.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:35.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:35.769 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:35.810 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:35.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:35.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:35.946 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:35.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:35.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:35 smithi052 ceph-mon[36682]: pgmap v558: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.45k op/s; 5961/245724 objects misplaced (2.426%); 29 MiB/s, 461 objects/s recovering 2024-09-13T13:12:35.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:35.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:35 smithi052 ceph-mon[29471]: pgmap v558: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.45k op/s; 5961/245724 objects misplaced (2.426%); 29 MiB/s, 461 objects/s recovering 2024-09-13T13:12:35.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:36.008 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:35 smithi167 ceph-mon[33178]: pgmap v558: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.45k op/s; 5961/245724 objects misplaced (2.426%); 29 MiB/s, 461 objects/s recovering 2024-09-13T13:12:36.008 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:36.049 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:36.071 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:36.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:36.116 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:12:36.116 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:12:36.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:36.205 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:36.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:36.307 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:36.350 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:36.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:36.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:36.400 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:12:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:12:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:12:36.452 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:36.494 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:36.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:36.596 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:36.640 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:36.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:36.745 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:36.787 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:36.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:36.883 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T13:12:36.889 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:36.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:36.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:36.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:36.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:36 smithi052 ceph-mon[36682]: 4.17 scrub starts 2024-09-13T13:12:36.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:36 smithi052 ceph-mon[36682]: 4.17 scrub ok 2024-09-13T13:12:36.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:36 smithi052 ceph-mon[29471]: 4.17 scrub starts 2024-09-13T13:12:36.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:36 smithi052 ceph-mon[29471]: 4.17 scrub ok 2024-09-13T13:12:37.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:37.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:36 smithi167 ceph-mon[33178]: 4.17 scrub starts 2024-09-13T13:12:37.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:36 smithi167 ceph-mon[33178]: 4.17 scrub ok 2024-09-13T13:12:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:37.090 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:37.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:37.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:37.231 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:37.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:37.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:37.252 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:12:37.333 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:37.375 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:37.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:37.477 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:37.511 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:37.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:37.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:37.613 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:37.638 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:37.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:37.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:37.740 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:37.770 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:37.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:37.819 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:12:37.819 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:12:37.872 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:37.909 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:37.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:37.941 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:37 smithi052 ceph-mon[36682]: pgmap v560: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 978 B/s rd, 0 B/s wr, 1.01k op/s; 5035/238959 objects misplaced (2.107%); 25 MiB/s, 404 objects/s recovering 2024-09-13T13:12:37.942 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:37 smithi052 ceph-mon[36682]: 5.d scrub starts 2024-09-13T13:12:37.942 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:37 smithi052 ceph-mon[36682]: 5.d scrub ok 2024-09-13T13:12:37.942 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:37 smithi052 ceph-mon[29471]: pgmap v560: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 978 B/s rd, 0 B/s wr, 1.01k op/s; 5035/238959 objects misplaced (2.107%); 25 MiB/s, 404 objects/s recovering 2024-09-13T13:12:37.942 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:37 smithi052 ceph-mon[29471]: 5.d scrub starts 2024-09-13T13:12:37.942 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:37 smithi052 ceph-mon[29471]: 5.d scrub ok 2024-09-13T13:12:38.012 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:38.034 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:38.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:38.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:37 smithi167 ceph-mon[33178]: pgmap v560: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 978 B/s rd, 0 B/s wr, 1.01k op/s; 5035/238959 objects misplaced (2.107%); 25 MiB/s, 404 objects/s recovering 2024-09-13T13:12:38.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:37 smithi167 ceph-mon[33178]: 5.d scrub starts 2024-09-13T13:12:38.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:37 smithi167 ceph-mon[33178]: 5.d scrub ok 2024-09-13T13:12:38.136 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:38.178 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:38.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:38.280 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:38.323 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:38.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:38.424 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:38.455 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:38.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:38.557 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:38.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T13:12:38.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:38.599 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:38.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:38.700 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:38.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:38.730 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:38.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:38.831 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:38.881 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:38.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:38.953 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:12:38.983 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:39.017 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:39.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:39.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:39.118 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:39.139 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:39.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:39.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:39.234 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:38 smithi052 ceph-mon[36682]: pgmap v561: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 823 B/s rd, 0 B/s wr, 848 op/s; 5035/238959 objects misplaced (2.107%); 21 MiB/s, 340 objects/s recovering 2024-09-13T13:12:39.234 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:38 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3665128436' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.3155027063472291} : dispatch 2024-09-13T13:12:39.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:38 smithi052 ceph-mon[29471]: pgmap v561: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 823 B/s rd, 0 B/s wr, 848 op/s; 5035/238959 objects misplaced (2.107%); 21 MiB/s, 340 objects/s recovering 2024-09-13T13:12:39.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3665128436' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.3155027063472291} : dispatch 2024-09-13T13:12:39.234 INFO:teuthology.orchestra.run.smithi052.stderr:set osd.7 primary-affinity to 0.315503 (8206762) 2024-09-13T13:12:39.241 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:39.254 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:38 smithi167 ceph-mon[33178]: pgmap v561: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 823 B/s rd, 0 B/s wr, 848 op/s; 5035/238959 objects misplaced (2.107%); 21 MiB/s, 340 objects/s recovering 2024-09-13T13:12:39.254 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:38 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3665128436' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.3155027063472291} : dispatch 2024-09-13T13:12:39.269 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:39.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:39.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:39.393 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:39.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:39.489 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:12:39.489 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:12:39.496 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:39.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:39.522 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:39.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:39.623 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:39.652 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:39.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:39.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:39.755 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:39.777 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:39.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:39.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:39.879 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:39.903 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:39.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:39.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:40.005 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:40.029 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:40.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:40.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:40.131 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:40.166 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:40.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:39 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3665128436' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3155027063472291}]': finished 2024-09-13T13:12:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:39 smithi052 ceph-mon[36682]: osdmap e298: 8 total, 8 up, 7 in 2024-09-13T13:12:40.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:39 smithi052 ceph-mon[36682]: osdmap e299: 8 total, 8 up, 7 in 2024-09-13T13:12:40.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:39 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3665128436' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3155027063472291}]': finished 2024-09-13T13:12:40.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:39 smithi052 ceph-mon[29471]: osdmap e298: 8 total, 8 up, 7 in 2024-09-13T13:12:40.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:39 smithi052 ceph-mon[29471]: osdmap e299: 8 total, 8 up, 7 in 2024-09-13T13:12:40.268 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:40.300 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:40.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:40.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:39 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3665128436' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3155027063472291}]': finished 2024-09-13T13:12:40.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:39 smithi167 ceph-mon[33178]: osdmap e298: 8 total, 8 up, 7 in 2024-09-13T13:12:40.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:39 smithi167 ceph-mon[33178]: osdmap e299: 8 total, 8 up, 7 in 2024-09-13T13:12:40.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:40.426 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:40.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:40.456 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T13:12:40.528 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:40.554 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:40.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:40.656 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:40.680 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:40.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:40.741 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:12:40.783 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:40.811 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:40.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:40.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:40.935 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:40.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:40.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:41.037 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:41.070 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:41.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:41.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:41.172 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:41.198 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:41.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:40 smithi052 ceph-mon[36682]: pgmap v563: 148 pgs: 4 peering, 3 active+remapped+backfilling, 141 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.31k op/s; 3103/223314 objects misplaced (1.390%); 32 MiB/s, 514 objects/s recovering 2024-09-13T13:12:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:40 smithi052 ceph-mon[36682]: 4.12 scrub starts 2024-09-13T13:12:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:40 smithi052 ceph-mon[36682]: 4.12 scrub ok 2024-09-13T13:12:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:40 smithi052 ceph-mon[36682]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-13T13:12:41.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:40 smithi052 ceph-mon[36682]: osdmap e300: 8 total, 8 up, 7 in 2024-09-13T13:12:41.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:40 smithi052 ceph-mon[29471]: pgmap v563: 148 pgs: 4 peering, 3 active+remapped+backfilling, 141 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.31k op/s; 3103/223314 objects misplaced (1.390%); 32 MiB/s, 514 objects/s recovering 2024-09-13T13:12:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:40 smithi052 ceph-mon[29471]: 4.12 scrub starts 2024-09-13T13:12:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:40 smithi052 ceph-mon[29471]: 4.12 scrub ok 2024-09-13T13:12:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:40 smithi052 ceph-mon[29471]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-13T13:12:41.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:40 smithi052 ceph-mon[29471]: osdmap e300: 8 total, 8 up, 7 in 2024-09-13T13:12:41.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:41.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:40 smithi167 ceph-mon[33178]: pgmap v563: 148 pgs: 4 peering, 3 active+remapped+backfilling, 141 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.31k op/s; 3103/223314 objects misplaced (1.390%); 32 MiB/s, 514 objects/s recovering 2024-09-13T13:12:41.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:40 smithi167 ceph-mon[33178]: 4.12 scrub starts 2024-09-13T13:12:41.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:40 smithi167 ceph-mon[33178]: 4.12 scrub ok 2024-09-13T13:12:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:40 smithi167 ceph-mon[33178]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-13T13:12:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:40 smithi167 ceph-mon[33178]: osdmap e300: 8 total, 8 up, 7 in 2024-09-13T13:12:41.327 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:41.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:41.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:41.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:41.455 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:41.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:41.558 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:41.595 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:41.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:41.668 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:41.669 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:41.696 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:41.724 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:41.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:41.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:41.856 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:41.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:41.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:41.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:41.984 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:41.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:41.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:42.081 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:41 smithi052 ceph-mon[36682]: 4.1c scrub starts 2024-09-13T13:12:42.081 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:41 smithi052 ceph-mon[36682]: 4.1c scrub ok 2024-09-13T13:12:42.081 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:41 smithi052 ceph-mon[36682]: 3.14 scrub starts 2024-09-13T13:12:42.081 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:41 smithi052 ceph-mon[36682]: 3.14 scrub ok 2024-09-13T13:12:42.081 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:41 smithi052 ceph-mon[36682]: 9.e continuing backfill to osd.0 from (275'7008,297'11614] 9:7ec97926:::benchmark_data_smithi052_128427_object117487:head to 297'11614 2024-09-13T13:12:42.081 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:41 smithi052 ceph-mon[29471]: 4.1c scrub starts 2024-09-13T13:12:42.082 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:41 smithi052 ceph-mon[29471]: 4.1c scrub ok 2024-09-13T13:12:42.082 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:41 smithi052 ceph-mon[29471]: 3.14 scrub starts 2024-09-13T13:12:42.082 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:41 smithi052 ceph-mon[29471]: 3.14 scrub ok 2024-09-13T13:12:42.082 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:41 smithi052 ceph-mon[29471]: 9.e continuing backfill to osd.0 from (275'7008,297'11614] 9:7ec97926:::benchmark_data_smithi052_128427_object117487:head to 297'11614 2024-09-13T13:12:42.086 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:42.117 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:42.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:42.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:42.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:42.249 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:42.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:42.268 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:41 smithi167 ceph-mon[33178]: 4.1c scrub starts 2024-09-13T13:12:42.268 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:41 smithi167 ceph-mon[33178]: 4.1c scrub ok 2024-09-13T13:12:42.268 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:41 smithi167 ceph-mon[33178]: 3.14 scrub starts 2024-09-13T13:12:42.268 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:41 smithi167 ceph-mon[33178]: 3.14 scrub ok 2024-09-13T13:12:42.268 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:41 smithi167 ceph-mon[33178]: 9.e continuing backfill to osd.0 from (275'7008,297'11614] 9:7ec97926:::benchmark_data_smithi052_128427_object117487:head to 297'11614 2024-09-13T13:12:42.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:42.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:42.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:42.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:42.412 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:12:42.412 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:12:42.483 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:42.520 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:42.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:42.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:42.622 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:42.635 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:12:42.635 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:12:42.635 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:12:42.636 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:12:42.636 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:12:42.636 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:12:42.654 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:42.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:42.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:42.663 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:12:42.663 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:12:42.707 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:12:42.707 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:12:42.739 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:12:42.740 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:12:42.756 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:42.776 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:12:42.776 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:12:42.790 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:42.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:42.819 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:12:42.819 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:12:42.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:42.920 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:42.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:42.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:43.022 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:43.052 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:43.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:43.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:42 smithi167 ceph-mon[33178]: 6.5 scrub starts 2024-09-13T13:12:43.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:42 smithi167 ceph-mon[33178]: 6.5 scrub ok 2024-09-13T13:12:43.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:42 smithi167 ceph-mon[33178]: pgmap v566: 148 pgs: 4 peering, 3 active+remapped+backfilling, 141 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 868 op/s; 3103/223314 objects misplaced (1.390%); 17 MiB/s, 279 objects/s recovering 2024-09-13T13:12:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:42 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3260538095' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:42 smithi167 ceph-mon[33178]: osdmap e301: 8 total, 8 up, 7 in 2024-09-13T13:12:43.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T13:12:43.154 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:43.185 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:43.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:42 smithi052 ceph-mon[36682]: 6.5 scrub starts 2024-09-13T13:12:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:42 smithi052 ceph-mon[36682]: 6.5 scrub ok 2024-09-13T13:12:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:42 smithi052 ceph-mon[36682]: pgmap v566: 148 pgs: 4 peering, 3 active+remapped+backfilling, 141 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 868 op/s; 3103/223314 objects misplaced (1.390%); 17 MiB/s, 279 objects/s recovering 2024-09-13T13:12:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:42 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3260538095' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:42 smithi052 ceph-mon[36682]: osdmap e301: 8 total, 8 up, 7 in 2024-09-13T13:12:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:42 smithi052 ceph-mon[29471]: 6.5 scrub starts 2024-09-13T13:12:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:42 smithi052 ceph-mon[29471]: 6.5 scrub ok 2024-09-13T13:12:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:42 smithi052 ceph-mon[29471]: pgmap v566: 148 pgs: 4 peering, 3 active+remapped+backfilling, 141 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 868 op/s; 3103/223314 objects misplaced (1.390%); 17 MiB/s, 279 objects/s recovering 2024-09-13T13:12:43.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3260538095' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:43.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:42 smithi052 ceph-mon[29471]: osdmap e301: 8 total, 8 up, 7 in 2024-09-13T13:12:43.287 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:43.308 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:43.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:43.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:43.410 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:43.424 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:12:43.433 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:43.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:43.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:43.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:43.556 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:43.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:43.658 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:43.679 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:43.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:43.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:43.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:43.802 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:43.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:43.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:43.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:43.918 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:12:43.919 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:12:43.919 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:12:43.919 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:12:43.919 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:12:43.927 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:12:43.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:43.927 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:43.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:44.028 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:44.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:44.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:44.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:44.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:43 smithi167 ceph-mon[33178]: 5.10 scrub starts 2024-09-13T13:12:44.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:43 smithi167 ceph-mon[33178]: 5.10 scrub ok 2024-09-13T13:12:44.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:43 smithi167 ceph-mon[33178]: 6.1c scrub starts 2024-09-13T13:12:44.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:43 smithi167 ceph-mon[33178]: 6.1c scrub ok 2024-09-13T13:12:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2834826911' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:12:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:43 smithi167 ceph-mon[33178]: 5.19 scrub starts 2024-09-13T13:12:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:43 smithi167 ceph-mon[33178]: 5.19 scrub ok 2024-09-13T13:12:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:43 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2834826911' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:12:44.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:43 smithi167 ceph-mon[33178]: osdmap e302: 8 total, 8 up, 7 in 2024-09-13T13:12:44.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:44.183 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:44.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:44.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:44.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[36682]: 5.10 scrub starts 2024-09-13T13:12:44.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[36682]: 5.10 scrub ok 2024-09-13T13:12:44.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[36682]: 6.1c scrub starts 2024-09-13T13:12:44.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[36682]: 6.1c scrub ok 2024-09-13T13:12:44.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2834826911' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:12:44.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[36682]: 5.19 scrub starts 2024-09-13T13:12:44.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[36682]: 5.19 scrub ok 2024-09-13T13:12:44.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:44.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2834826911' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:12:44.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[36682]: osdmap e302: 8 total, 8 up, 7 in 2024-09-13T13:12:44.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[29471]: 5.10 scrub starts 2024-09-13T13:12:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[29471]: 5.10 scrub ok 2024-09-13T13:12:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[29471]: 6.1c scrub starts 2024-09-13T13:12:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[29471]: 6.1c scrub ok 2024-09-13T13:12:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2834826911' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:12:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[29471]: 5.19 scrub starts 2024-09-13T13:12:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[29471]: 5.19 scrub ok 2024-09-13T13:12:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2834826911' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:12:44.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:43 smithi052 ceph-mon[29471]: osdmap e302: 8 total, 8 up, 7 in 2024-09-13T13:12:44.285 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:44.307 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:44.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:44.408 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:44.431 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:44.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:44.533 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:44.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:12:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:12:44.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:44.556 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:44.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:44.568 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T13:12:44.657 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:44.679 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:44.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:44.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:44.803 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:44.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:44.816 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:12:44.905 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:44.927 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:44.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:44.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:44 smithi052 ceph-mon[36682]: 6.15 scrub starts 2024-09-13T13:12:44.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:44 smithi052 ceph-mon[36682]: 6.15 scrub ok 2024-09-13T13:12:44.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:44 smithi052 ceph-mon[36682]: pgmap v568: 148 pgs: 5 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 838 op/s; 1583/218232 objects misplaced (0.725%); 25 MiB/s, 394 objects/s recovering 2024-09-13T13:12:44.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:44 smithi052 ceph-mon[36682]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:12:44.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:44 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:44.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:44 smithi052 ceph-mon[36682]: osdmap e303: 8 total, 8 up, 7 in 2024-09-13T13:12:44.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:44 smithi052 ceph-mon[29471]: 6.15 scrub starts 2024-09-13T13:12:44.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:44 smithi052 ceph-mon[29471]: 6.15 scrub ok 2024-09-13T13:12:44.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:44 smithi052 ceph-mon[29471]: pgmap v568: 148 pgs: 5 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 838 op/s; 1583/218232 objects misplaced (0.725%); 25 MiB/s, 394 objects/s recovering 2024-09-13T13:12:44.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:44 smithi052 ceph-mon[29471]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:12:44.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:44 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:44.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:44 smithi052 ceph-mon[29471]: osdmap e303: 8 total, 8 up, 7 in 2024-09-13T13:12:45.028 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:45.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:45.065 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:45.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:45.166 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:45.203 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:45.204 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:44 smithi167 ceph-mon[33178]: 6.15 scrub starts 2024-09-13T13:12:45.204 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:44 smithi167 ceph-mon[33178]: 6.15 scrub ok 2024-09-13T13:12:45.204 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:44 smithi167 ceph-mon[33178]: pgmap v568: 148 pgs: 5 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 838 op/s; 1583/218232 objects misplaced (0.725%); 25 MiB/s, 394 objects/s recovering 2024-09-13T13:12:45.204 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:44 smithi167 ceph-mon[33178]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:12:45.204 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:44 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:45.205 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:44 smithi167 ceph-mon[33178]: osdmap e303: 8 total, 8 up, 7 in 2024-09-13T13:12:45.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:45.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:45.306 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:45.349 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:45.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:45.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:45.360 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:12:45.360 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:12:45.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:12:45.360 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:12:45.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.360 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14493.0:1220 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94064716861824 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-13T13:12:45.360 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:03:15.061504+0000", 2024-09-13T13:12:45.360 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 570.29324608100001, 2024-09-13T13:12:45.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.48614386900000001, 2024-09-13T13:12:45.361 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.361 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.361 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:12:45.361 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.361 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14493", 2024-09-13T13:12:45.361 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:12:45.361 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1220 2024-09-13T13:12:45.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.361 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061504+0000", 2024-09-13T13:12:45.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061504+0000", 2024-09-13T13:12:45.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061507+0000", 2024-09-13T13:12:45.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5599999999999998e-06 2024-09-13T13:12:45.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061517+0000", 2024-09-13T13:12:45.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3710000000000006e-06 2024-09-13T13:12:45.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061530+0000", 2024-09-13T13:12:45.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2943e-05 2024-09-13T13:12:45.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061539+0000", 2024-09-13T13:12:45.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1300000000000007e-06 2024-09-13T13:12:45.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061612+0000", 2024-09-13T13:12:45.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3455999999999996e-05 2024-09-13T13:12:45.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:12:45.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.061621+0000", 2024-09-13T13:12:45.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9379999999999998e-06 2024-09-13T13:12:45.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.540176+0000", 2024-09-13T13:12:45.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.47855515700000001 2024-09-13T13:12:45.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.540307+0000", 2024-09-13T13:12:45.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000130906 2024-09-13T13:12:45.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:03:15.547648+0000", 2024-09-13T13:12:45.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0073404079999999997 2024-09-13T13:12:45.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.366 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.366 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:135640 9.3 9:c8fab5fe:::benchmark_data_smithi052_128427_object4971:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T13:12:45.366 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:11:49.891399+0000", 2024-09-13T13:12:45.366 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.463350425999998, 2024-09-13T13:12:45.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12597461600000001, 2024-09-13T13:12:45.366 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.366 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.367 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.367 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135640 2024-09-13T13:12:45.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.367 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891399+0000", 2024-09-13T13:12:45.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891399+0000", 2024-09-13T13:12:45.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891401+0000", 2024-09-13T13:12:45.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.435e-06 2024-09-13T13:12:45.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891406+0000", 2024-09-13T13:12:45.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:12:45.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891419+0000", 2024-09-13T13:12:45.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3396e-05 2024-09-13T13:12:45.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891427+0000", 2024-09-13T13:12:45.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7079999999999997e-06 2024-09-13T13:12:45.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891504+0000", 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7261000000000004e-05 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891609+0000", 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104628 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891708+0000", 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8981999999999995e-05 2024-09-13T13:12:45.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.898451+0000", 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0067428699999999998 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.902316+0000", 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038657269999999998 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:50.017273+0000", 2024-09-13T13:12:45.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114956343 2024-09-13T13:12:45.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:50.017329+0000", 2024-09-13T13:12:45.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6198e-05 2024-09-13T13:12:45.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:50.017374+0000", 2024-09-13T13:12:45.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5274000000000001e-05 2024-09-13T13:12:45.372 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.372 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.372 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.373 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:135641 9.3 9:cf12db49:::benchmark_data_smithi052_128427_object4972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T13:12:45.373 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:11:49.891445+0000", 2024-09-13T13:12:45.373 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.463304833999999, 2024-09-13T13:12:45.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.126092432, 2024-09-13T13:12:45.373 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.373 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.373 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.373 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.373 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.373 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.373 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135641 2024-09-13T13:12:45.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.374 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891445+0000", 2024-09-13T13:12:45.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891445+0000", 2024-09-13T13:12:45.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891446+0000", 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3300000000000001e-07 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891449+0000", 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6739999999999999e-06 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891456+0000", 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4590000000000003e-06 2024-09-13T13:12:45.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891462+0000", 2024-09-13T13:12:45.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:12:45.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.894700+0000", 2024-09-13T13:12:45.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032381990000000002 2024-09-13T13:12:45.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.894802+0000", 2024-09-13T13:12:45.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010218899999999999 2024-09-13T13:12:45.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:12:45.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.894875+0000", 2024-09-13T13:12:45.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2380999999999997e-05 2024-09-13T13:12:45.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.901124+0000", 2024-09-13T13:12:45.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0062498019999999996 2024-09-13T13:12:45.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.902419+0000", 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012945159999999999 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:50.017440+0000", 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11502148099999999 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:50.017493+0000", 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3146999999999998e-05 2024-09-13T13:12:45.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:50.017538+0000", 2024-09-13T13:12:45.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4066000000000001e-05 2024-09-13T13:12:45.379 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.379 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.379 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.379 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:154786 9.3 9:cbb83ca5:::benchmark_data_smithi052_128427_object24117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T13:12:45.379 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:08.412441+0000", 2024-09-13T13:12:45.379 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.942308959999998, 2024-09-13T13:12:45.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13816605100000001, 2024-09-13T13:12:45.380 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.380 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.380 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.380 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.380 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.380 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.380 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154786 2024-09-13T13:12:45.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.380 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412441+0000", 2024-09-13T13:12:45.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412441+0000", 2024-09-13T13:12:45.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412442+0000", 2024-09-13T13:12:45.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2249999999999999e-06 2024-09-13T13:12:45.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412446+0000", 2024-09-13T13:12:45.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8650000000000003e-06 2024-09-13T13:12:45.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412455+0000", 2024-09-13T13:12:45.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2140000000000002e-06 2024-09-13T13:12:45.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412460+0000", 2024-09-13T13:12:45.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1909999999999999e-06 2024-09-13T13:12:45.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412498+0000", 2024-09-13T13:12:45.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7203000000000003e-05 2024-09-13T13:12:45.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412571+0000", 2024-09-13T13:12:45.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3002000000000003e-05 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412637+0000", 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6225999999999999e-05 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.417079+0000", 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044422200000000002 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.423099+0000", 2024-09-13T13:12:45.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00602003 2024-09-13T13:12:45.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.550526+0000", 2024-09-13T13:12:45.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.127427176 2024-09-13T13:12:45.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.550579+0000", 2024-09-13T13:12:45.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2822999999999999e-05 2024-09-13T13:12:45.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.550607+0000", 2024-09-13T13:12:45.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7875999999999999e-05 2024-09-13T13:12:45.386 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.386 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.386 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.386 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:154787 9.3 9:c6308295:::benchmark_data_smithi052_128427_object24118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T13:12:45.386 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:08.412476+0000", 2024-09-13T13:12:45.386 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.942273720000003, 2024-09-13T13:12:45.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14505712100000001, 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154787 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412476+0000", 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412476+0000", 2024-09-13T13:12:45.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412477+0000", 2024-09-13T13:12:45.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:12:45.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412479+0000", 2024-09-13T13:12:45.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1880000000000001e-06 2024-09-13T13:12:45.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412483+0000", 2024-09-13T13:12:45.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6299999999999997e-06 2024-09-13T13:12:45.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412487+0000", 2024-09-13T13:12:45.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7910000000000002e-06 2024-09-13T13:12:45.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.414686+0000", 2024-09-13T13:12:45.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021988009999999998 2024-09-13T13:12:45.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.414791+0000", 2024-09-13T13:12:45.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010519 2024-09-13T13:12:45.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:12:45.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.414904+0000", 2024-09-13T13:12:45.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112802 2024-09-13T13:12:45.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.419029+0000", 2024-09-13T13:12:45.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041248209999999999 2024-09-13T13:12:45.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.423145+0000", 2024-09-13T13:12:45.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041162229999999996 2024-09-13T13:12:45.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.557439+0000", 2024-09-13T13:12:45.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13429402300000001 2024-09-13T13:12:45.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.557490+0000", 2024-09-13T13:12:45.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0538000000000003e-05 2024-09-13T13:12:45.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.557533+0000", 2024-09-13T13:12:45.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3640000000000002e-05 2024-09-13T13:12:45.393 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.393 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.393 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.394 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:154791 9.3 9:c8f0a3cb:::benchmark_data_smithi052_128427_object24122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T13:12:45.394 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:08.412626+0000", 2024-09-13T13:12:45.394 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.942124272000001, 2024-09-13T13:12:45.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14510228, 2024-09-13T13:12:45.394 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.394 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.394 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.394 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.394 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.394 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.394 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154791 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412626+0000", 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412626+0000", 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412626+0000", 2024-09-13T13:12:45.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9999999999999999e-07 2024-09-13T13:12:45.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412627+0000", 2024-09-13T13:12:45.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.424e-06 2024-09-13T13:12:45.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412630+0000", 2024-09-13T13:12:45.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.807e-06 2024-09-13T13:12:45.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412633+0000", 2024-09-13T13:12:45.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7e-06 2024-09-13T13:12:45.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.417109+0000", 2024-09-13T13:12:45.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044762700000000001 2024-09-13T13:12:45.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.417222+0000", 2024-09-13T13:12:45.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113161 2024-09-13T13:12:45.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:12:45.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.417348+0000", 2024-09-13T13:12:45.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012564299999999999 2024-09-13T13:12:45.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.419142+0000", 2024-09-13T13:12:45.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001794365 2024-09-13T13:12:45.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.424527+0000", 2024-09-13T13:12:45.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0053843340000000002 2024-09-13T13:12:45.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.557643+0000", 2024-09-13T13:12:45.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13311679500000001 2024-09-13T13:12:45.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.557695+0000", 2024-09-13T13:12:45.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1272e-05 2024-09-13T13:12:45.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.557728+0000", 2024-09-13T13:12:45.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3209e-05 2024-09-13T13:12:45.400 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.400 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.400 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.400 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:154796 9.3 9:c3ab54a2:::benchmark_data_smithi052_128427_object24127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T13:12:45.400 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:08.420648+0000", 2024-09-13T13:12:45.400 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.934101896000001, 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16434322200000001, 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154796 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420648+0000", 2024-09-13T13:12:45.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420648+0000", 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420649+0000", 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2410000000000001e-06 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420654+0000", 2024-09-13T13:12:45.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3000000000000003e-06 2024-09-13T13:12:45.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420662+0000", 2024-09-13T13:12:45.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4740000000000005e-06 2024-09-13T13:12:45.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420667+0000", 2024-09-13T13:12:45.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.797e-06 2024-09-13T13:12:45.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420714+0000", 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6937999999999997e-05 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420838+0000", 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012402999999999999 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420949+0000", 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110874 2024-09-13T13:12:45.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.423161+0000", 2024-09-13T13:12:45.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022121889999999998 2024-09-13T13:12:45.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.424593+0000", 2024-09-13T13:12:45.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014320030000000001 2024-09-13T13:12:45.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.584895+0000", 2024-09-13T13:12:45.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16030223599999999 2024-09-13T13:12:45.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.584952+0000", 2024-09-13T13:12:45.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6428000000000003e-05 2024-09-13T13:12:45.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.584991+0000", 2024-09-13T13:12:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9712000000000002e-05 2024-09-13T13:12:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:163394 9.e 9:7c89c0d8:::benchmark_data_smithi052_128427_object32725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-13T13:12:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:16.424824+0000", 2024-09-13T13:12:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.929925535999999, 2024-09-13T13:12:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14081759499999999, 2024-09-13T13:12:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 163394 2024-09-13T13:12:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424824+0000", 2024-09-13T13:12:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424824+0000", 2024-09-13T13:12:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424825+0000", 2024-09-13T13:12:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.23e-07 2024-09-13T13:12:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424829+0000", 2024-09-13T13:12:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3330000000000001e-06 2024-09-13T13:12:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424837+0000", 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3750000000000003e-06 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424842+0000", 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3750000000000002e-06 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424889+0000", 2024-09-13T13:12:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6263000000000001e-05 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.425137+0000", 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024875199999999998 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.425257+0000", 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119881 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.428314+0000", 2024-09-13T13:12:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030571629999999999 2024-09-13T13:12:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.428325+0000", 2024-09-13T13:12:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.098e-05 2024-09-13T13:12:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.430320+0000", 2024-09-13T13:12:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019945649999999998 2024-09-13T13:12:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.565531+0000", 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13521107199999999 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.565574+0000", 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3133999999999997e-05 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.565642+0000", 2024-09-13T13:12:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7879000000000005e-05 2024-09-13T13:12:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:164430 9.3 9:c2c8b212:::benchmark_data_smithi052_128427_object33761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-13T13:12:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:17.424138+0000", 2024-09-13T13:12:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.930611835000001, 2024-09-13T13:12:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14083164200000001, 2024-09-13T13:12:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 164430 2024-09-13T13:12:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424138+0000", 2024-09-13T13:12:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424138+0000", 2024-09-13T13:12:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424140+0000", 2024-09-13T13:12:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8500000000000001e-06 2024-09-13T13:12:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424145+0000", 2024-09-13T13:12:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5500000000000002e-06 2024-09-13T13:12:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424157+0000", 2024-09-13T13:12:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1627999999999999e-05 2024-09-13T13:12:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424164+0000", 2024-09-13T13:12:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1890000000000001e-06 2024-09-13T13:12:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424228+0000", 2024-09-13T13:12:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3825000000000006e-05 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424327+0000", 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9273999999999999e-05 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424427+0000", 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9520999999999998e-05 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.426863+0000", 2024-09-13T13:12:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024363620000000001 2024-09-13T13:12:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.550466+0000", 2024-09-13T13:12:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123602541 2024-09-13T13:12:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.564891+0000", 2024-09-13T13:12:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014425097 2024-09-13T13:12:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.564926+0000", 2024-09-13T13:12:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5500999999999998e-05 2024-09-13T13:12:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.564970+0000", 2024-09-13T13:12:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3303999999999997e-05 2024-09-13T13:12:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:164434 9.3 9:c73c0368:::benchmark_data_smithi052_128427_object33765:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-13T13:12:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:17.428447+0000", 2024-09-13T13:12:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.926303399999998, 2024-09-13T13:12:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13666840799999999, 2024-09-13T13:12:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 164434 2024-09-13T13:12:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428447+0000", 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428447+0000", 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428448+0000", 2024-09-13T13:12:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9350000000000001e-06 2024-09-13T13:12:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428454+0000", 2024-09-13T13:12:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3140000000000003e-06 2024-09-13T13:12:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428466+0000", 2024-09-13T13:12:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2058999999999999e-05 2024-09-13T13:12:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428473+0000", 2024-09-13T13:12:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0029999999999997e-06 2024-09-13T13:12:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428526+0000", 2024-09-13T13:12:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2961999999999999e-05 2024-09-13T13:12:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428637+0000", 2024-09-13T13:12:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111221 2024-09-13T13:12:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428742+0000", 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105358 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.430849+0000", 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021066399999999999 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.555922+0000", 2024-09-13T13:12:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12507289899999999 2024-09-13T13:12:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.565017+0000", 2024-09-13T13:12:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090949110000000007 2024-09-13T13:12:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.565074+0000", 2024-09-13T13:12:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7535999999999999e-05 2024-09-13T13:12:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.565115+0000", 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.057e-05 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:169641 9.e 9:78334604:::benchmark_data_smithi052_128427_object38972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:22.418468+0000", 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.936281843, 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.169452977, 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169641 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418468+0000", 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418468+0000", 2024-09-13T13:12:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418469+0000", 2024-09-13T13:12:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3099999999999996e-07 2024-09-13T13:12:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418472+0000", 2024-09-13T13:12:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0730000000000001e-06 2024-09-13T13:12:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418478+0000", 2024-09-13T13:12:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6869999999999999e-06 2024-09-13T13:12:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418483+0000", 2024-09-13T13:12:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.899e-06 2024-09-13T13:12:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.421223+0000", 2024-09-13T13:12:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027399949999999998 2024-09-13T13:12:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.421339+0000", 2024-09-13T13:12:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116055 2024-09-13T13:12:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:12:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.421457+0000", 2024-09-13T13:12:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118276 2024-09-13T13:12:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.426437+0000", 2024-09-13T13:12:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049803579999999998 2024-09-13T13:12:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.429739+0000", 2024-09-13T13:12:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033021919999999998 2024-09-13T13:12:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.429814+0000", 2024-09-13T13:12:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4789999999999994e-05 2024-09-13T13:12:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587749+0000", 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.15793521199999999 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587804+0000", 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4314999999999998e-05 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587921+0000", 2024-09-13T13:12:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011729400000000001 2024-09-13T13:12:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:169651 9.e 9:73829da4:::benchmark_data_smithi052_128427_object38982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:12:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:22.425923+0000", 2024-09-13T13:12:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.928827282, 2024-09-13T13:12:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16214630499999999, 2024-09-13T13:12:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169651 2024-09-13T13:12:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425923+0000", 2024-09-13T13:12:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425923+0000", 2024-09-13T13:12:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425925+0000", 2024-09-13T13:12:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.153e-06 2024-09-13T13:12:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425930+0000", 2024-09-13T13:12:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1510000000000003e-06 2024-09-13T13:12:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425941+0000", 2024-09-13T13:12:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1219e-05 2024-09-13T13:12:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425948+0000", 2024-09-13T13:12:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4590000000000003e-06 2024-09-13T13:12:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.426533+0000", 2024-09-13T13:12:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058506300000000003 2024-09-13T13:12:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.426626+0000", 2024-09-13T13:12:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2878000000000001e-05 2024-09-13T13:12:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:12:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.426723+0000", 2024-09-13T13:12:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7369999999999998e-05 2024-09-13T13:12:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.429801+0000", 2024-09-13T13:12:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030778759999999998 2024-09-13T13:12:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.429820+0000", 2024-09-13T13:12:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9456000000000001e-05 2024-09-13T13:12:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.559932+0000", 2024-09-13T13:12:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13011128799999999 2024-09-13T13:12:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587959+0000", 2024-09-13T13:12:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028027237 2024-09-13T13:12:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587993+0000", 2024-09-13T13:12:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4285000000000001e-05 2024-09-13T13:12:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.588069+0000", 2024-09-13T13:12:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5870000000000004e-05 2024-09-13T13:12:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172101 9.e 9:7b187d27:::benchmark_data_smithi052_128427_object41432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:12:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.753747+0000", 2024-09-13T13:12:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.601002459, 2024-09-13T13:12:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14020244800000001, 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172101 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.753747+0000", 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.753747+0000", 2024-09-13T13:12:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.753749+0000", 2024-09-13T13:12:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.561e-06 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.753754+0000", 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8040000000000001e-06 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.753761+0000", 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2350000000000002e-06 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.753766+0000", 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6249999999999998e-06 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.754624+0000", 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00085874499999999995 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.754802+0000", 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178035 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.754954+0000", 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015183699999999999 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759833+0000", 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048790639999999998 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759871+0000", 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7533000000000001e-05 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759911+0000", 2024-09-13T13:12:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9796000000000003e-05 2024-09-13T13:12:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.893830+0000", 2024-09-13T13:12:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13391968200000001 2024-09-13T13:12:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.893870+0000", 2024-09-13T13:12:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9932999999999999e-05 2024-09-13T13:12:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.893950+0000", 2024-09-13T13:12:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9598000000000006e-05 2024-09-13T13:12:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172108 9.e 9:76b5c7a7:::benchmark_data_smithi052_128427_object41439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:12:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.758691+0000", 2024-09-13T13:12:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.596058895999999, 2024-09-13T13:12:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14327330599999999, 2024-09-13T13:12:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172108 2024-09-13T13:12:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.758691+0000", 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.758691+0000", 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.758693+0000", 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2850000000000001e-06 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.758699+0000", 2024-09-13T13:12:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3369999999999999e-06 2024-09-13T13:12:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.758708+0000", 2024-09-13T13:12:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2280000000000004e-06 2024-09-13T13:12:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.758715+0000", 2024-09-13T13:12:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2529999999999998e-06 2024-09-13T13:12:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759950+0000", 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001234635 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760063+0000", 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113378 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760608+0000", 2024-09-13T13:12:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00054462300000000005 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769006+0000", 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0083978569999999999 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769057+0000", 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1439e-05 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.772669+0000", 2024-09-13T13:12:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003612189 2024-09-13T13:12:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.901781+0000", 2024-09-13T13:12:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12911177200000001 2024-09-13T13:12:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.901867+0000", 2024-09-13T13:12:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5834000000000005e-05 2024-09-13T13:12:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.901964+0000", 2024-09-13T13:12:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7476e-05 2024-09-13T13:12:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172109 9.e 9:7a4e12b2:::benchmark_data_smithi052_128427_object41440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:12:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.759294+0000", 2024-09-13T13:12:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.595456311, 2024-09-13T13:12:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14287778800000001, 2024-09-13T13:12:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172109 2024-09-13T13:12:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759294+0000", 2024-09-13T13:12:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759294+0000", 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759295+0000", 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759303+0000", 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6380000000000004e-06 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759324+0000", 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.16e-05 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759332+0000", 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3599999999999998e-06 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.763556+0000", 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042248720000000002 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.763639+0000", 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2668000000000002e-05 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.763741+0000", 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010195 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769103+0000", 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0053623550000000001 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769151+0000", 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7472000000000002e-05 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.772674+0000", 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035229229999999999 2024-09-13T13:12:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902009+0000", 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.129334853 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902054+0000", 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4892000000000003e-05 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902171+0000", 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117885 2024-09-13T13:12:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172112 9.e 9:778ddad4:::benchmark_data_smithi052_128427_object41443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:12:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.760357+0000", 2024-09-13T13:12:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.594392682999999, 2024-09-13T13:12:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14215998599999999, 2024-09-13T13:12:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172112 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760357+0000", 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760357+0000", 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760359+0000", 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.494e-06 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760364+0000", 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9359999999999998e-06 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760374+0000", 2024-09-13T13:12:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0584999999999999e-05 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760381+0000", 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1389999999999998e-06 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766044+0000", 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056627689999999998 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766169+0000", 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012447900000000001 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766305+0000", 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013649800000000001 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.772633+0000", 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063277079999999996 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.772661+0000", 2024-09-13T13:12:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8588000000000001e-05 2024-09-13T13:12:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.772677+0000", 2024-09-13T13:12:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.535e-05 2024-09-13T13:12:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902234+0000", 2024-09-13T13:12:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12955698600000001 2024-09-13T13:12:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902412+0000", 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000177756 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902517+0000", 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105698 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172113 9.3 9:ca684d45:::benchmark_data_smithi052_128427_object41444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:12:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.760502+0000", 2024-09-13T13:12:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.594248159999999, 2024-09-13T13:12:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13345124, 2024-09-13T13:12:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172113 2024-09-13T13:12:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.471 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760502+0000", 2024-09-13T13:12:45.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760502+0000", 2024-09-13T13:12:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760503+0000", 2024-09-13T13:12:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.035e-06 2024-09-13T13:12:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760506+0000", 2024-09-13T13:12:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6200000000000001e-06 2024-09-13T13:12:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760513+0000", 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.601e-06 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760519+0000", 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.032e-06 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760579+0000", 2024-09-13T13:12:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9484999999999997e-05 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760677+0000", 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8138000000000001e-05 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760777+0000", 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100692 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.763309+0000", 2024-09-13T13:12:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025319129999999998 2024-09-13T13:12:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766405+0000", 2024-09-13T13:12:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030952670000000001 2024-09-13T13:12:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.893840+0000", 2024-09-13T13:12:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12743517200000001 2024-09-13T13:12:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.893904+0000", 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4003000000000004e-05 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.893953+0000", 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9282e-05 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172114 9.d 9:ba712465:::benchmark_data_smithi052_128427_object41445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:12:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.762227+0000", 2024-09-13T13:12:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.592523378999999, 2024-09-13T13:12:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13646831200000001, 2024-09-13T13:12:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172114 2024-09-13T13:12:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762227+0000", 2024-09-13T13:12:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762227+0000", 2024-09-13T13:12:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762228+0000", 2024-09-13T13:12:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5290000000000001e-06 2024-09-13T13:12:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762233+0000", 2024-09-13T13:12:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4680000000000003e-06 2024-09-13T13:12:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762242+0000", 2024-09-13T13:12:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6399999999999992e-06 2024-09-13T13:12:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762249+0000", 2024-09-13T13:12:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9990000000000002e-06 2024-09-13T13:12:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762294+0000", 2024-09-13T13:12:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5114999999999997e-05 2024-09-13T13:12:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762394+0000", 2024-09-13T13:12:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9232000000000001e-05 2024-09-13T13:12:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:12:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762493+0000", 2024-09-13T13:12:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9932000000000005e-05 2024-09-13T13:12:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.767265+0000", 2024-09-13T13:12:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047712140000000002 2024-09-13T13:12:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.771179+0000", 2024-09-13T13:12:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039145350000000002 2024-09-13T13:12:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.898613+0000", 2024-09-13T13:12:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12743336799999999 2024-09-13T13:12:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.898652+0000", 2024-09-13T13:12:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9481999999999999e-05 2024-09-13T13:12:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.898695+0000", 2024-09-13T13:12:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2797999999999999e-05 2024-09-13T13:12:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172120 9.e 9:700d9b1c:::benchmark_data_smithi052_128427_object41451:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:12:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.766538+0000", 2024-09-13T13:12:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.588211762, 2024-09-13T13:12:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.136210366, 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172120 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766538+0000", 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766538+0000", 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766540+0000", 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.686e-06 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766544+0000", 2024-09-13T13:12:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9400000000000004e-06 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766560+0000", 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6358000000000002e-05 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766568+0000", 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9170000000000006e-06 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769182+0000", 2024-09-13T13:12:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026134589999999998 2024-09-13T13:12:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769301+0000", 2024-09-13T13:12:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119703 2024-09-13T13:12:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:12:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769434+0000", 2024-09-13T13:12:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000133242 2024-09-13T13:12:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.772679+0000", 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032450090000000001 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.773396+0000", 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00071618400000000003 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.896041+0000", 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.122645191 2024-09-13T13:12:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902575+0000", 2024-09-13T13:12:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065343750000000003 2024-09-13T13:12:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902637+0000", 2024-09-13T13:12:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1747000000000004e-05 2024-09-13T13:12:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902749+0000", 2024-09-13T13:12:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111555 2024-09-13T13:12:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172122 9.d 9:bd4fcb71:::benchmark_data_smithi052_128427_object41453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:12:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.768424+0000", 2024-09-13T13:12:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.586326261, 2024-09-13T13:12:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13067949700000001, 2024-09-13T13:12:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:12:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:12:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:12:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:12:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:12:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:12:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172122 2024-09-13T13:12:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:12:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:12:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768424+0000", 2024-09-13T13:12:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768424+0000", 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768425+0000", 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3909999999999999e-06 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768430+0000", 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9540000000000003e-06 2024-09-13T13:12:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:12:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768440+0000", 2024-09-13T13:12:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6160000000000008e-06 2024-09-13T13:12:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:12:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768447+0000", 2024-09-13T13:12:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7109999999999995e-06 2024-09-13T13:12:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:12:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768540+0000", 2024-09-13T13:12:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2726999999999995e-05 2024-09-13T13:12:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:12:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768703+0000", 2024-09-13T13:12:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016333999999999999 2024-09-13T13:12:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:12:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768828+0000", 2024-09-13T13:12:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124356 2024-09-13T13:12:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:12:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.771519+0000", 2024-09-13T13:12:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002691681 2024-09-13T13:12:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.771667+0000", 2024-09-13T13:12:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014779300000000001 2024-09-13T13:12:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:12:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.899015+0000", 2024-09-13T13:12:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12734757899999999 2024-09-13T13:12:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:12:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.899066+0000", 2024-09-13T13:12:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0747e-05 2024-09-13T13:12:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:12:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:12:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:12:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.899103+0000", 2024-09-13T13:12:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7601999999999997e-05 2024-09-13T13:12:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:12:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:12:45.498 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:12:45.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:45.537 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:45.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:45.594 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:12:45.639 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:45.692 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:45.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:45.794 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:45.823 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:45.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:45.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:45.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:45.954 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:45.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:46.029 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:46.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:46.085 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:46.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:46.186 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:46.227 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:46.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:46.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:46 smithi167 ceph-mon[33178]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-13T13:12:46.329 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:46.377 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:46.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:46.387 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:46 smithi052 ceph-mon[36682]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-13T13:12:46.387 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:12:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:12:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:12:46.387 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:46 smithi052 ceph-mon[29471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-13T13:12:46.480 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:46.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:46.521 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:46.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:46.575 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:12:46.575 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:12:46.576 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-13T13:12:46.576 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd in 4 2024-09-13T13:12:46.623 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:46.668 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:46.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:46.757 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:12:46.758 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T13:12:46.770 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:46.826 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:46.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:46.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:46.928 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:46.983 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:46.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:46.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:47.085 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:47.086 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:12:47.108 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:47.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:47.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:47.210 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:47.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:47 smithi052 ceph-mon[29471]: pgmap v571: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 145 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 639 op/s; 308/213900 objects misplaced (0.144%); 35 MiB/s, 557 objects/s recovering 2024-09-13T13:12:47.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:47 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3456607714' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:47 smithi052 ceph-mon[36682]: pgmap v571: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 145 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 639 op/s; 308/213900 objects misplaced (0.144%); 35 MiB/s, 557 objects/s recovering 2024-09-13T13:12:47.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:47 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3456607714' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:47.252 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:47.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:47.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:47.309 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:47 smithi167 ceph-mon[33178]: pgmap v571: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 145 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 639 op/s; 308/213900 objects misplaced (0.144%); 35 MiB/s, 557 objects/s recovering 2024-09-13T13:12:47.312 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:47 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3456607714' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:47.354 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:47.377 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:47.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:47.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:47.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:47.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:47.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:47.610 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:47.641 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:47.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:47.743 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:47.777 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:47.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:47.878 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:47.893 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:12:47.894 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:12:47.894 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:12:47.894 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:12:47.894 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:12:47.894 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:12:47.894 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:12:47.894 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:12:47.895 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:12:47.895 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:12:47.895 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:12:47.895 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:12:47.895 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:12:47.896 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:12:47.896 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:12:47.896 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:12:47.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:47.906 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:47.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:48.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:48.023 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:12:48.023 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:12:48.028 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:48.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:48.130 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:48.166 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:48.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:48.268 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:48.304 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:48.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:48.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:48.406 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:48.434 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:48.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:48.535 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:48.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:48.576 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:48.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:48.677 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:48.680 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T13:12:48.700 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:48.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:48.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:48.802 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:48.841 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:48.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:48.942 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:48.983 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:48.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:49.037 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:12:49.085 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:49.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:49.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:49.217 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:49.257 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:49.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:49.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:49.359 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:49.395 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:49.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:49.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:49.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:49.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:49.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:49.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:49.635 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:49.656 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:12:49.657 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:12:49.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:49.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:49 smithi052 ceph-mon[36682]: pgmap v572: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 145 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 522 op/s; 308/213900 objects misplaced (0.144%); 29 MiB/s, 456 objects/s recovering 2024-09-13T13:12:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:49 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:49 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:12:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:49 smithi052 ceph-mon[29471]: pgmap v572: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 145 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 522 op/s; 308/213900 objects misplaced (0.144%); 29 MiB/s, 456 objects/s recovering 2024-09-13T13:12:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:49 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:49 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:12:49.767 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:49.799 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:49.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:49.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:49 smithi167 ceph-mon[33178]: pgmap v572: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 145 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 522 op/s; 308/213900 objects misplaced (0.144%); 29 MiB/s, 456 objects/s recovering 2024-09-13T13:12:49.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:49 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:12:49.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:49 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:12:49.901 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:49.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:49.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:49.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:50.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:50.055 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:50.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:50.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:50.148 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:50.152 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:50.157 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:50.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:50.213 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:50.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:50.314 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:50.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:50.348 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:50.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:50.449 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:50.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:50.478 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:50.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:50.579 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:50.616 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T13:12:50.617 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:50.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:50.720 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:50.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:50.765 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:50.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:50.866 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:50.903 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:50.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:50.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:50.950 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:12:51.005 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:51.032 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:51.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:51.134 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:51.187 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:51.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:51.288 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:51.317 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:51.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:51.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:51.419 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:51.427 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:51 smithi052 ceph-mon[29471]: pgmap v573: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s; 296/199974 objects misplaced (0.148%); 25 MiB/s, 395 objects/s recovering 2024-09-13T13:12:51.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:51 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/253629642' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:51.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:51 smithi052 ceph-mon[29471]: from='client.25906 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:51.442 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:12:51.459 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:51.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:51.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:51.561 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:51.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:51.585 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:51.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:51.686 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:51.718 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:51 smithi167 ceph-mon[33178]: pgmap v573: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s; 296/199974 objects misplaced (0.148%); 25 MiB/s, 395 objects/s recovering 2024-09-13T13:12:51.719 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:51 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/253629642' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:51.719 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:51 smithi167 ceph-mon[33178]: from='client.25906 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:51.725 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:51.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:51.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:51 smithi052 ceph-mon[36682]: pgmap v573: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s; 296/199974 objects misplaced (0.148%); 25 MiB/s, 395 objects/s recovering 2024-09-13T13:12:51.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:51 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/253629642' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:51.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:51 smithi052 ceph-mon[36682]: from='client.25906 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:12:51.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:51 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a[29467]: 2024-09-13T13:12:51.474+0000 7effa8af8640 -1 mon.a@0(leader).osd e304 definitely_dead 0 2024-09-13T13:12:51.827 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:51.858 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:51.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:51.960 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:51.998 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:51.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:51.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:52.040 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:12:52.100 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:52.154 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:52.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:52.256 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:52.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:52.294 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:52.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:52.395 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:52.442 INFO:teuthology.orchestra.run.smithi052.stderr:marked in osd.4. 2024-09-13T13:12:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:52.457 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:52.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:52.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:52.581 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:52.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:52.644 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:12:52.644 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:12:52.682 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:52.692 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:52.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:52.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:52.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:52 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:52 smithi052 ceph-mon[36682]: from='client.25906 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:12:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:52 smithi052 ceph-mon[36682]: osdmap e304: 8 total, 8 up, 7 in 2024-09-13T13:12:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:52 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2943684927' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T13:12:52.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:52 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:52.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:52 smithi052 ceph-mon[29471]: from='client.25906 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:12:52.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:52 smithi052 ceph-mon[29471]: osdmap e304: 8 total, 8 up, 7 in 2024-09-13T13:12:52.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:52 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2943684927' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T13:12:52.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:52.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:52 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:52 smithi167 ceph-mon[33178]: from='client.25906 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:12:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:52 smithi167 ceph-mon[33178]: osdmap e304: 8 total, 8 up, 7 in 2024-09-13T13:12:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:52 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2943684927' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T13:12:52.829 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:52.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:52.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:52.930 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:52.972 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:52.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:53.074 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:53.096 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:53.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:53.167 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-13T13:12:53.198 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:53.221 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:53.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:53.323 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:53.345 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:53.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:53.447 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:53.470 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:53.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:53.571 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:53.597 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:53.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:53.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:53.659 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:12:53.660 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:12:53.660 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:53 smithi052 ceph-mon[36682]: pgmap v575: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 645 B/s rd, 0 B/s wr, 768 op/s; 296/199974 objects misplaced (0.148%); 10 MiB/s, 159 objects/s recovering 2024-09-13T13:12:53.660 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:53 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2943684927' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T13:12:53.660 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:53 smithi052 ceph-mon[36682]: osdmap e305: 8 total, 8 up, 8 in 2024-09-13T13:12:53.660 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:53 smithi052 ceph-mon[36682]: osdmap e306: 8 total, 8 up, 8 in 2024-09-13T13:12:53.660 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:53 smithi052 ceph-mon[29471]: pgmap v575: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 645 B/s rd, 0 B/s wr, 768 op/s; 296/199974 objects misplaced (0.148%); 10 MiB/s, 159 objects/s recovering 2024-09-13T13:12:53.661 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2943684927' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T13:12:53.661 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:53 smithi052 ceph-mon[29471]: osdmap e305: 8 total, 8 up, 8 in 2024-09-13T13:12:53.661 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:53 smithi052 ceph-mon[29471]: osdmap e306: 8 total, 8 up, 8 in 2024-09-13T13:12:53.676 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:12:53.677 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:12:53.693 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:12:53.693 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:12:53.699 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:53.710 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:12:53.710 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:12:53.721 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:53.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:53.727 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:12:53.727 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:12:53.745 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:12:53.745 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:12:53.762 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:12:53.763 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:12:53.779 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:12:53.780 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:12:53.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:53 smithi167 ceph-mon[33178]: pgmap v575: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 645 B/s rd, 0 B/s wr, 768 op/s; 296/199974 objects misplaced (0.148%); 10 MiB/s, 159 objects/s recovering 2024-09-13T13:12:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:53 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2943684927' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T13:12:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:53 smithi167 ceph-mon[33178]: osdmap e305: 8 total, 8 up, 8 in 2024-09-13T13:12:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:53 smithi167 ceph-mon[33178]: osdmap e306: 8 total, 8 up, 8 in 2024-09-13T13:12:53.823 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:53.846 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:53.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:53.947 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:53.972 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:53.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:53.995 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T13:12:54.073 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:54.126 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:54.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:54.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:54.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:54.256 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:54.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:54.358 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:54.368 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:12:54.396 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:54.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:54.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:54.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:54.549 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:54.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:54.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:54 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3258725315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:54.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:54 smithi167 ceph-mon[33178]: osdmap e307: 8 total, 8 up, 8 in 2024-09-13T13:12:54.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:12:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:12:54.651 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:54.698 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:54.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:54.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:54 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3258725315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:54 smithi052 ceph-mon[29471]: osdmap e307: 8 total, 8 up, 8 in 2024-09-13T13:12:54.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:54 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3258725315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:12:54.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:54 smithi052 ceph-mon[36682]: osdmap e307: 8 total, 8 up, 8 in 2024-09-13T13:12:54.800 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:54.811 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:54.828 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:54.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:54.930 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:54.963 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:54.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:54.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:55.065 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:55.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:55.094 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:55.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:55.137 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:55.195 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:55.246 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:55.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:55.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:55.378 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:55.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:55.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:55.490 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:12:55.491 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:12:55.491 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 0 2024-09-13T13:12:55.491 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:12:55.509 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:55.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:55.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:55.563 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:55 smithi052 ceph-mon[36682]: pgmap v578: 148 pgs: 1 peering, 1 remapped+peering, 146 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.18k op/s; 150 B/s, 3 objects/s recovering 2024-09-13T13:12:55.563 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:55 smithi052 ceph-mon[36682]: 9.c starting backfill to osd.4 from (276'7158,301'11602] MIN to 304'12188 2024-09-13T13:12:55.563 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:55 smithi052 ceph-mon[36682]: 9.4 starting backfill to osd.4 from (276'7701,300'11891] MIN to 304'12448 2024-09-13T13:12:55.563 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:55.563 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:55 smithi052 ceph-mon[36682]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T13:12:55.563 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:55 smithi052 ceph-mon[29471]: pgmap v578: 148 pgs: 1 peering, 1 remapped+peering, 146 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.18k op/s; 150 B/s, 3 objects/s recovering 2024-09-13T13:12:55.564 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:55 smithi052 ceph-mon[29471]: 9.c starting backfill to osd.4 from (276'7158,301'11602] MIN to 304'12188 2024-09-13T13:12:55.564 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:55 smithi052 ceph-mon[29471]: 9.4 starting backfill to osd.4 from (276'7701,300'11891] MIN to 304'12448 2024-09-13T13:12:55.564 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:55.564 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:55 smithi052 ceph-mon[29471]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T13:12:55.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:55.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:55.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:55.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:55.745 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:55.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:55.780 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:55.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:55.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:55 smithi167 ceph-mon[33178]: pgmap v578: 148 pgs: 1 peering, 1 remapped+peering, 146 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.18k op/s; 150 B/s, 3 objects/s recovering 2024-09-13T13:12:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:55 smithi167 ceph-mon[33178]: 9.c starting backfill to osd.4 from (276'7158,301'11602] MIN to 304'12188 2024-09-13T13:12:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:55 smithi167 ceph-mon[33178]: 9.4 starting backfill to osd.4 from (276'7701,300'11891] MIN to 304'12448 2024-09-13T13:12:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:12:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:55 smithi167 ceph-mon[33178]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T13:12:55.881 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:55.923 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:55.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:56.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:56.046 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:56.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:56.148 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:12:56.174 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:56.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:56.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:12:56.276 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:56.302 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:56.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:56.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:56.379 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:12:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:12:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:12:56.405 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:56.421 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T13:12:56.428 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:56.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:56.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:56.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:56.552 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:56.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:56.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:56.648 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:12:56.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:56.676 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:56.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:56.778 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:56.835 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:56.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:56.867 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:56.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:56.960 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:56.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:56.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:57.062 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:57.091 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:57.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:57.193 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:57.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:57.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:57.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:57.322 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:57.346 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:57.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:57.444 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:12:57.444 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:12:57.444 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:12:57.445 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:12:57.445 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:12:57.448 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:57.473 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:57.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:57.493 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:12:57.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:57.598 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:57.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:57.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:57.647 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:57 smithi052 ceph-mon[36682]: pgmap v580: 148 pgs: 1 peering, 1 activating+remapped, 1 remapped+peering, 145 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 3726/186405 objects misplaced (1.999%); 290 B/s, 7 objects/s recovering 2024-09-13T13:12:57.647 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:57 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3576549596' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:12:57.647 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:57 smithi052 ceph-mon[36682]: from='client.35945 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:12:57.648 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:57 smithi052 ceph-mon[29471]: pgmap v580: 148 pgs: 1 peering, 1 activating+remapped, 1 remapped+peering, 145 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 3726/186405 objects misplaced (1.999%); 290 B/s, 7 objects/s recovering 2024-09-13T13:12:57.648 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:57 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3576549596' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:12:57.648 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:57 smithi052 ceph-mon[29471]: from='client.35945 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:12:57.699 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:12:57.733 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:57.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:57.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:12:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:57 smithi167 ceph-mon[33178]: pgmap v580: 148 pgs: 1 peering, 1 activating+remapped, 1 remapped+peering, 145 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 3726/186405 objects misplaced (1.999%); 290 B/s, 7 objects/s recovering 2024-09-13T13:12:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:57 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3576549596' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:12:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:57 smithi167 ceph-mon[33178]: from='client.35945 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:12:57.835 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:57.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:57.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:57.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:57.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:58.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:58.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:58.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:58.168 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:12:58.168 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:12:58.168 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-13T13:12:58.168 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-13T13:12:58.217 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:58.240 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:58.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:58.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:58.364 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:58.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:58.465 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:12:58.478 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:58.492 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:58.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:12:58.594 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:58.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:58.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:58.718 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:12:58.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:58 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:58.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:58 smithi052 ceph-mon[36682]: from='client.35945 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:12:58.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:58 smithi052 ceph-mon[36682]: osdmap e308: 8 total, 8 up, 8 in 2024-09-13T13:12:58.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:58 smithi052 ceph-mon[36682]: osdmap e309: 8 total, 8 up, 8 in 2024-09-13T13:12:58.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:58 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:12:58.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:58 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:12:58.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:58 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:58.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:58 smithi052 ceph-mon[29471]: from='client.35945 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:12:58.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:58 smithi052 ceph-mon[29471]: osdmap e308: 8 total, 8 up, 8 in 2024-09-13T13:12:58.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:58 smithi052 ceph-mon[29471]: osdmap e309: 8 total, 8 up, 8 in 2024-09-13T13:12:58.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:58 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:12:58.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:58 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:12:58.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:58.741 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:58.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:12:58.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:58 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:12:58.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:58 smithi167 ceph-mon[33178]: from='client.35945 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:12:58.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:58 smithi167 ceph-mon[33178]: osdmap e308: 8 total, 8 up, 8 in 2024-09-13T13:12:58.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:58 smithi167 ceph-mon[33178]: osdmap e309: 8 total, 8 up, 8 in 2024-09-13T13:12:58.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:58 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:12:58.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:58 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:12:58.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:58.863 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:58.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:58.883 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T13:12:58.920 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:12:58.965 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:58.986 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:58.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:58.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:59.088 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:59.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:59.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:59.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:59.247 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:59.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:59.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:59.349 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:12:59.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:59.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:12:59.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:59 smithi052 ceph-mon[29471]: pgmap v582: 148 pgs: 1 peering, 1 activating+remapped, 1 remapped+peering, 145 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 945 B/s rd, 0 B/s wr, 836 op/s; 3726/186405 objects misplaced (1.999%); 264 B/s, 7 objects/s recovering 2024-09-13T13:12:59.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:59 smithi052 ceph-mon[29471]: 3.15 scrub starts 2024-09-13T13:12:59.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:59 smithi052 ceph-mon[29471]: 3.15 scrub ok 2024-09-13T13:12:59.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:59 smithi052 ceph-mon[29471]: osdmap e310: 8 total, 8 up, 8 in 2024-09-13T13:12:59.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:59 smithi052 ceph-mon[29471]: 4.d scrub starts 2024-09-13T13:12:59.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:12:59 smithi052 ceph-mon[29471]: 4.d scrub ok 2024-09-13T13:12:59.483 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:12:59.523 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:12:59.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:59.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:12:59.625 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:59.654 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:59.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:59.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:59 smithi052 ceph-mon[36682]: pgmap v582: 148 pgs: 1 peering, 1 activating+remapped, 1 remapped+peering, 145 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 945 B/s rd, 0 B/s wr, 836 op/s; 3726/186405 objects misplaced (1.999%); 264 B/s, 7 objects/s recovering 2024-09-13T13:12:59.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:59 smithi052 ceph-mon[36682]: 3.15 scrub starts 2024-09-13T13:12:59.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:59 smithi052 ceph-mon[36682]: 3.15 scrub ok 2024-09-13T13:12:59.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:59 smithi052 ceph-mon[36682]: osdmap e310: 8 total, 8 up, 8 in 2024-09-13T13:12:59.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:59 smithi052 ceph-mon[36682]: 4.d scrub starts 2024-09-13T13:12:59.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:12:59 smithi052 ceph-mon[36682]: 4.d scrub ok 2024-09-13T13:12:59.756 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:12:59.782 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:59.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:59.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:12:59.826 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:59 smithi167 ceph-mon[33178]: pgmap v582: 148 pgs: 1 peering, 1 activating+remapped, 1 remapped+peering, 145 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 945 B/s rd, 0 B/s wr, 836 op/s; 3726/186405 objects misplaced (1.999%); 264 B/s, 7 objects/s recovering 2024-09-13T13:12:59.826 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:59 smithi167 ceph-mon[33178]: 3.15 scrub starts 2024-09-13T13:12:59.826 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:59 smithi167 ceph-mon[33178]: 3.15 scrub ok 2024-09-13T13:12:59.826 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:59 smithi167 ceph-mon[33178]: osdmap e310: 8 total, 8 up, 8 in 2024-09-13T13:12:59.826 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:59 smithi167 ceph-mon[33178]: 4.d scrub starts 2024-09-13T13:12:59.826 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:12:59 smithi167 ceph-mon[33178]: 4.d scrub ok 2024-09-13T13:12:59.884 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:12:59.913 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:12:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:12:59.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:12:59.930 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:12:59.930 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:12:59.931 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:12:59.931 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:12:59.931 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:12:59.931 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:12:59.931 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:12:59.931 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:12:59.943 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:12:59.944 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:12:59.982 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:12:59.982 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:13:00.011 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:00.014 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:00.029 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:13:00.029 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:13:00.060 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:13:00.060 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:13:00.067 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:00.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:00.169 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:00.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:00.213 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:00.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:00.251 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:13:00.252 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:13:00.252 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 310, 2024-09-13T13:13:00.252 INFO:teuthology.orchestra.run.smithi052.stdout: "fsid": "2b415b38-71cf-11ef-bceb-c7b262605968", 2024-09-13T13:13:00.252 INFO:teuthology.orchestra.run.smithi052.stdout: "created": "2024-09-13T12:55:27.893882+0000", 2024-09-13T13:13:00.252 INFO:teuthology.orchestra.run.smithi052.stdout: "modified": "2024-09-13T13:12:58.939156+0000", 2024-09-13T13:13:00.252 INFO:teuthology.orchestra.run.smithi052.stdout: "last_up_change": "2024-09-13T13:04:02.281376+0000", 2024-09-13T13:13:00.252 INFO:teuthology.orchestra.run.smithi052.stdout: "last_in_change": "2024-09-13T13:12:52.437622+0000", 2024-09-13T13:13:00.252 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T13:13:00.252 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_num": 5804032, 2024-09-13T13:13:00.252 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_set": [ 2024-09-13T13:13:00.252 INFO:teuthology.orchestra.run.smithi052.stdout: "nodeep-scrub", 2024-09-13T13:13:00.252 INFO:teuthology.orchestra.run.smithi052.stdout: "pglog_hardlimit", 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: "purged_snapdirs", 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: "recovery_deletes", 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: "sortbitwise" 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_version": 19, 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_snapshot": "", 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_max": 9, 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: "max_osd": 8, 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: "require_min_compat_client": "luminous", 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: "min_compat_client": "luminous", 2024-09-13T13:13:00.253 INFO:teuthology.orchestra.run.smithi052.stdout: "require_osd_release": "squid", 2024-09-13T13:13:00.254 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_crimson": false, 2024-09-13T13:13:00.254 INFO:teuthology.orchestra.run.smithi052.stdout: "pools": [ 2024-09-13T13:13:00.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.254 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 1, 2024-09-13T13:13:00.254 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": ".mgr", 2024-09-13T13:13:00.254 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T12:58:31.271893+0000", 2024-09-13T13:13:00.254 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:13:00.254 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:13:00.254 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:13:00.254 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:13:00.254 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:13:00.254 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:13:00.254 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:13:00.255 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:13:00.255 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:13:00.255 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:13:00.255 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:13:00.255 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:13:00.255 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:13:00.255 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 1, 2024-09-13T13:13:00.255 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 1, 2024-09-13T13:13:00.255 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 1, 2024-09-13T13:13:00.255 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 1, 2024-09-13T13:13:00.255 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 1, 2024-09-13T13:13:00.255 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:13:00.256 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:13:00.256 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:13:00.256 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:13:00.256 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:13:00.256 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:13:00.256 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:13:00.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.256 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "277", 2024-09-13T13:13:00.256 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:13:00.256 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:13:00.256 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:13:00.256 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:13:00.256 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:13:00.257 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:13:00.257 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:13:00.257 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:13:00.257 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:13:00.257 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:13:00.257 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:13:00.257 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:13:00.257 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:13:00.257 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:13:00.257 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:13:00.257 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:13:00.257 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:13:00.257 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:13:00.258 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:13:00.259 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:13:00.259 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:13:00.259 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:13:00.259 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:13:00.259 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:13:00.259 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:13:00.259 INFO:teuthology.orchestra.run.smithi052.stdout: "options": { 2024-09-13T13:13:00.259 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_max": 32, 2024-09-13T13:13:00.259 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_min": 1 2024-09-13T13:13:00.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.259 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:13:00.259 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr": {} 2024-09-13T13:13:00.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.260 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:13:00.260 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:13:00.260 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 7.8899998664855957, 2024-09-13T13:13:00.260 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 7.8899998664855957, 2024-09-13T13:13:00.260 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 0.37999999523162842, 2024-09-13T13:13:00.260 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 3, 2024-09-13T13:13:00.260 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 3, 2024-09-13T13:13:00.260 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:13:00.260 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:13:00.260 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:13:00.260 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.261 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 2, 2024-09-13T13:13:00.261 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "datapool", 2024-09-13T13:13:00.261 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:00.306747+0000", 2024-09-13T13:13:00.261 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 8193, 2024-09-13T13:13:00.261 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T13:13:00.261 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:13:00.261 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:13:00.261 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:13:00.261 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:13:00.261 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:13:00.261 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:13:00.261 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:13:00.261 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:13:00.262 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:13:00.262 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:13:00.262 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:13:00.262 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 3, 2024-09-13T13:13:00.262 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 3, 2024-09-13T13:13:00.262 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 3, 2024-09-13T13:13:00.262 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 3, 2024-09-13T13:13:00.262 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 3, 2024-09-13T13:13:00.262 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:13:00.262 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:13:00.262 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:13:00.262 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:13:00.262 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:13:00.263 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:13:00.263 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:13:00.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.263 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "63", 2024-09-13T13:13:00.263 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:13:00.263 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:13:00.263 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:13:00.263 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:13:00.263 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:13:00.263 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 2, 2024-09-13T13:13:00.263 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 63, 2024-09-13T13:13:00.263 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:13:00.263 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:13:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:13:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:13:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:13:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:13:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:13:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:13:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:13:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:13:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:13:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:13:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:13:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:13:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:13:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:13:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:13:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:13:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:13:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:13:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:13:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:13:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:13:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:13:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:13:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:13:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:13:00.266 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:13:00.266 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:13:00.266 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:13:00.266 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:13:00.266 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:13:00.266 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd": {} 2024-09-13T13:13:00.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.266 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:13:00.266 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:13:00.266 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 2.6500000953674316, 2024-09-13T13:13:00.266 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 2.6500000953674316, 2024-09-13T13:13:00.266 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T13:13:00.266 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-13T13:13:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-13T13:13:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:13:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:13:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:13:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 3, 2024-09-13T13:13:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": ".rgw.root", 2024-09-13T13:13:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:01.103523+0000", 2024-09-13T13:13:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:13:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:13:00.267 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:13:00.268 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:13:00.268 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:13:00.268 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:13:00.268 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:13:00.268 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:13:00.268 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:13:00.268 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:13:00.268 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:13:00.268 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:13:00.268 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:13:00.268 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:13:00.269 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:13:00.269 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:13:00.269 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:13:00.269 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:13:00.269 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:13:00.270 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:13:00.270 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:13:00.270 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:13:00.270 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:13:00.270 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:13:00.270 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:13:00.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.270 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "61", 2024-09-13T13:13:00.270 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:13:00.270 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:13:00.270 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:13:00.271 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:13:00.271 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:13:00.271 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:13:00.271 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:13:00.271 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:13:00.271 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:13:00.271 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:13:00.271 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:13:00.271 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:13:00.271 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:13:00.271 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:13:00.271 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:13:00.271 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:13:00.272 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:13:00.272 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:13:00.272 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:13:00.272 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:13:00.272 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:13:00.272 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:13:00.272 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:13:00.272 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:13:00.272 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:13:00.272 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:13:00.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.273 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:13:00.273 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:13:00.273 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:13:00.273 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:13:00.273 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:13:00.273 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:13:00.273 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:13:00.273 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:13:00.273 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:13:00.273 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:13:00.273 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:13:00.274 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:13:00.274 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:13:00.274 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:13:00.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.274 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:13:00.274 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:13:00.274 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 1.75, 2024-09-13T13:13:00.274 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 1.75, 2024-09-13T13:13:00.274 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:13:00.274 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 1.75, 2024-09-13T13:13:00.274 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 1.75, 2024-09-13T13:13:00.274 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:13:00.274 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:13:00.275 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:13:00.275 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.275 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 4, 2024-09-13T13:13:00.275 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "default.rgw.log", 2024-09-13T13:13:00.275 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:02.249986+0000", 2024-09-13T13:13:00.275 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:13:00.275 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:13:00.275 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:13:00.275 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:13:00.275 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:13:00.276 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:13:00.276 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:13:00.276 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:13:00.276 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:13:00.276 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:13:00.276 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:13:00.276 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:13:00.276 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:13:00.276 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:13:00.276 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:13:00.276 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:13:00.276 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:13:00.276 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:13:00.277 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:13:00.277 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:13:00.277 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:13:00.277 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:13:00.277 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:13:00.277 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:13:00.277 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:13:00.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.277 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "63", 2024-09-13T13:13:00.277 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:13:00.277 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:13:00.277 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:13:00.277 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:13:00.278 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:13:00.278 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:13:00.278 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:13:00.278 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:13:00.278 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:13:00.278 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:13:00.278 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:13:00.278 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:13:00.279 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:13:00.279 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:13:00.279 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:13:00.279 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:13:00.279 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:13:00.279 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:13:00.279 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:13:00.279 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:13:00.279 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:13:00.279 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:13:00.279 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:13:00.279 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:13:00.279 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:13:00.280 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:13:00.281 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:13:00.281 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:13:00.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.281 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:13:00.281 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:13:00.281 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 2.25, 2024-09-13T13:13:00.281 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 2.25, 2024-09-13T13:13:00.281 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:13:00.281 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 2.25, 2024-09-13T13:13:00.281 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 2.25, 2024-09-13T13:13:00.281 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:13:00.282 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:13:00.282 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:13:00.282 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.282 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.282 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 5, 2024-09-13T13:13:00.282 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "default.rgw.control", 2024-09-13T13:13:00.282 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:04.203764+0000", 2024-09-13T13:13:00.282 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:13:00.282 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:13:00.282 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:13:00.282 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:13:00.282 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:13:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:13:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:13:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:13:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:13:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:13:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:13:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:13:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:13:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:13:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:13:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:13:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:13:00.283 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:13:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:13:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:13:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:13:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:13:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:13:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:13:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:13:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "65", 2024-09-13T13:13:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:13:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:13:00.284 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:13:00.285 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:13:00.285 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:13:00.285 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:13:00.285 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:13:00.285 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:13:00.285 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:13:00.285 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:13:00.285 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:13:00.285 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:13:00.285 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:13:00.285 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:13:00.285 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:13:00.286 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:13:00.286 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:13:00.286 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:13:00.286 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:13:00.286 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:13:00.286 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:13:00.286 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:13:00.286 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:13:00.286 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:13:00.287 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:13:00.287 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:13:00.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.287 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:13:00.287 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:13:00.287 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:13:00.287 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:13:00.287 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:13:00.287 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:13:00.287 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:13:00.287 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:13:00.287 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:13:00.288 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:13:00.288 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:13:00.288 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:13:00.288 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:13:00.288 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:13:00.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.288 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:13:00.288 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:13:00.288 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 1.5, 2024-09-13T13:13:00.288 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 1.5, 2024-09-13T13:13:00.288 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:13:00.289 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 1.5, 2024-09-13T13:13:00.289 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 1.5, 2024-09-13T13:13:00.289 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:13:00.289 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:13:00.289 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:13:00.289 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.289 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.289 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.289 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 6, 2024-09-13T13:13:00.289 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "default.rgw.meta", 2024-09-13T13:13:00.289 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:06.293676+0000", 2024-09-13T13:13:00.290 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:13:00.290 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:13:00.290 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:13:00.290 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:13:00.290 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:13:00.290 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:13:00.290 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:13:00.290 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:13:00.290 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:13:00.290 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:13:00.290 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:13:00.290 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:13:00.290 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:13:00.291 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:13:00.291 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:13:00.291 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:13:00.291 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:13:00.291 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:13:00.291 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:13:00.291 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:13:00.291 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:13:00.291 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:13:00.291 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:13:00.291 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:13:00.291 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:13:00.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.292 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "67", 2024-09-13T13:13:00.292 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:13:00.292 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:13:00.292 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:13:00.292 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:13:00.292 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:13:00.292 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:13:00.292 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:13:00.292 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:13:00.292 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:13:00.293 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:13:00.293 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:13:00.293 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:13:00.293 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:13:00.293 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:13:00.293 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:13:00.293 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:13:00.293 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:13:00.293 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:13:00.293 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:13:00.293 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:13:00.293 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:13:00.294 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:13:00.294 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:13:00.294 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:13:00.294 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:13:00.294 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:13:00.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.294 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:13:00.294 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:13:00.294 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:13:00.295 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:13:00.295 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:13:00.295 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:13:00.295 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:13:00.295 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:13:00.295 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:13:00.295 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:13:00.295 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:13:00.295 INFO:teuthology.orchestra.run.smithi052.stdout: "options": { 2024-09-13T13:13:00.295 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_bias": 4 2024-09-13T13:13:00.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.295 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:13:00.295 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:13:00.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.296 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:13:00.296 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:13:00.296 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 1.75, 2024-09-13T13:13:00.296 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 1.75, 2024-09-13T13:13:00.296 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:13:00.296 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 1.75, 2024-09-13T13:13:00.296 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 1.75, 2024-09-13T13:13:00.296 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:13:00.296 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:13:00.296 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:13:00.296 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 9, 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "unique_pool_2", 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:10:07.820868+0000", 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:13:00.297 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 16, 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 16, 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 16, 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 16, 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 16, 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:13:00.298 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "245", 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:13:00.299 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:13:00.300 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:13:00.300 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:13:00.300 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:13:00.300 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:13:00.300 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:13:00.300 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:13:00.300 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:13:00.300 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:13:00.300 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:13:00.300 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:13:00.300 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:13:00.300 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:13:00.300 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:13:00.301 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:13:00.301 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:13:00.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.301 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:13:00.301 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:13:00.301 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:13:00.301 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:13:00.301 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:13:00.301 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:13:00.301 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:13:00.302 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:13:00.302 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:13:00.302 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:13:00.302 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:13:00.302 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:13:00.302 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:13:00.302 INFO:teuthology.orchestra.run.smithi052.stdout: "rados": {} 2024-09-13T13:13:00.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.302 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:13:00.302 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:13:00.302 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 2.5, 2024-09-13T13:13:00.302 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 2.5, 2024-09-13T13:13:00.302 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 2.5, 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 2.5, 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: "osds": [ 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 0, 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "d52dca1a-94c1-4e9e-93f5-687c038cd7c7", 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:13:00.303 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:13:00.304 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:13:00.304 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:13:00.304 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:13:00.304 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:13:00.304 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 8, 2024-09-13T13:13:00.304 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 305, 2024-09-13T13:13:00.304 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:13:00.304 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:13:00.304 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:13:00.304 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.304 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.304 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.304 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6801", 2024-09-13T13:13:00.305 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:13:00.305 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.305 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.305 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:13:00.305 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.305 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.305 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6802", 2024-09-13T13:13:00.305 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:13:00.305 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.305 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6804", 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.306 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6803", 2024-09-13T13:13:00.307 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:13:00.307 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.307 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.307 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6801/542298937", 2024-09-13T13:13:00.307 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6802/542298937", 2024-09-13T13:13:00.308 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6804/542298937", 2024-09-13T13:13:00.308 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6803/542298937", 2024-09-13T13:13:00.308 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:13:00.308 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:13:00.308 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:13:00.308 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.308 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 1, 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "b40a3574-3c6f-4810-a651-8d64aee6801b", 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 13, 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 309, 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.310 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.310 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6805", 2024-09-13T13:13:00.310 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:13:00.310 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.310 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.310 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:13:00.310 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.310 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.310 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.310 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6806", 2024-09-13T13:13:00.310 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:13:00.310 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.311 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.311 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:13:00.311 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.311 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.311 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6808", 2024-09-13T13:13:00.311 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:13:00.311 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.311 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.311 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:13:00.312 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.312 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.312 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.312 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6807", 2024-09-13T13:13:00.312 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:13:00.312 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.312 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.312 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6805/26289431", 2024-09-13T13:13:00.312 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6806/26289431", 2024-09-13T13:13:00.312 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6808/26289431", 2024-09-13T13:13:00.312 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6807/26289431", 2024-09-13T13:13:00.313 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:13:00.313 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:13:00.313 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:13:00.313 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.313 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.313 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.313 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 2, 2024-09-13T13:13:00.313 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "1e8e49a0-ccf8-4b7f-839b-104b64601408", 2024-09-13T13:13:00.313 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:13:00.313 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:13:00.313 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:13:00.313 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:13:00.314 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:13:00.314 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:13:00.314 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 18, 2024-09-13T13:13:00.314 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 305, 2024-09-13T13:13:00.314 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:13:00.314 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:13:00.314 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:13:00.314 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.314 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.314 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.314 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6809", 2024-09-13T13:13:00.314 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:13:00.314 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.315 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.315 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:13:00.315 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.315 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.315 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.315 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6810", 2024-09-13T13:13:00.315 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:13:00.315 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.315 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.315 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:13:00.316 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.316 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.316 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.316 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6812", 2024-09-13T13:13:00.316 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:13:00.316 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.316 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.316 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:13:00.316 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.316 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.316 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.316 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6811", 2024-09-13T13:13:00.317 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:13:00.317 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.318 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.318 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6809/3960240393", 2024-09-13T13:13:00.318 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6810/3960240393", 2024-09-13T13:13:00.318 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6812/3960240393", 2024-09-13T13:13:00.318 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6811/3960240393", 2024-09-13T13:13:00.318 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:13:00.318 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:13:00.318 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:13:00.318 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 3, 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "9f6f1d22-be68-4d86-9d4b-86fd9e87d141", 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 27, 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 309, 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:13:00.319 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:13:00.320 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.320 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.320 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6813", 2024-09-13T13:13:00.320 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:13:00.320 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.320 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.320 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:13:00.320 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.321 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.321 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6814", 2024-09-13T13:13:00.321 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:13:00.321 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.321 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.321 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:13:00.321 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.321 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.321 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.321 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6816", 2024-09-13T13:13:00.321 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:13:00.321 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.322 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.322 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:13:00.322 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.322 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.322 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.322 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6815", 2024-09-13T13:13:00.322 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:13:00.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.322 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.323 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6813/2025522885", 2024-09-13T13:13:00.323 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6814/2025522885", 2024-09-13T13:13:00.323 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6816/2025522885", 2024-09-13T13:13:00.323 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6815/2025522885", 2024-09-13T13:13:00.324 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:13:00.324 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:13:00.324 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:13:00.324 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.324 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 4, 2024-09-13T13:13:00.324 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "1326d850-caf4-484e-90a1-052f54f9f306", 2024-09-13T13:13:00.324 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:13:00.324 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:13:00.324 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:13:00.324 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:13:00.324 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:13:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:13:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 33, 2024-09-13T13:13:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 306, 2024-09-13T13:13:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:13:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:13:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:13:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6800", 2024-09-13T13:13:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:13:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6801", 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6803", 2024-09-13T13:13:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:13:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:13:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6802", 2024-09-13T13:13:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:13:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6800/2137138858", 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6801/2137138858", 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6803/2137138858", 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6802/2137138858", 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 5, 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2", 2024-09-13T13:13:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:13:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:13:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:13:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:13:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:13:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:13:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 40, 2024-09-13T13:13:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 305, 2024-09-13T13:13:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:13:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:13:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:13:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6804", 2024-09-13T13:13:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:13:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:13:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6805", 2024-09-13T13:13:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:13:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:13:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6807", 2024-09-13T13:13:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:13:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:13:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6806", 2024-09-13T13:13:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:13:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6804/2906884337", 2024-09-13T13:13:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6805/2906884337", 2024-09-13T13:13:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6807/2906884337", 2024-09-13T13:13:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6806/2906884337", 2024-09-13T13:13:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:13:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:13:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:13:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 6, 2024-09-13T13:13:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "2471907d-fcbd-44e0-9d29-54151ada9df0", 2024-09-13T13:13:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:13:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:13:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:13:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:13:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 48, 2024-09-13T13:13:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 102, 2024-09-13T13:13:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 118, 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 305, 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 103, 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6808", 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:13:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6809", 2024-09-13T13:13:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:13:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:13:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6811", 2024-09-13T13:13:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:13:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:13:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6810", 2024-09-13T13:13:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:13:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6808/143093129", 2024-09-13T13:13:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6809/143093129", 2024-09-13T13:13:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6811/143093129", 2024-09-13T13:13:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6810/143093129", 2024-09-13T13:13:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:13:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:13:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:13:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 7, 2024-09-13T13:13:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "97d2c6d7-7989-4c11-91e0-32ee87e76c6d", 2024-09-13T13:13:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:13:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 0.31549072265625, 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 55, 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 305, 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6812", 2024-09-13T13:13:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6813", 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:13:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6815", 2024-09-13T13:13:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:13:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:13:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:13:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:13:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6814", 2024-09-13T13:13:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:13:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6812/403155155", 2024-09-13T13:13:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6813/403155155", 2024-09-13T13:13:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6815/403155155", 2024-09-13T13:13:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6814/403155155", 2024-09-13T13:13:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:13:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:13:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:13:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:13:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_xinfo": [ 2024-09-13T13:13:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 0, 2024-09-13T13:13:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:13:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:13:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:13:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:13:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:13:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:12:16.176074+0000", 2024-09-13T13:13:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:13:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 1, 2024-09-13T13:13:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:13:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:13:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:13:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:13:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:13:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:12:05.682291+0000", 2024-09-13T13:13:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:13:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 2, 2024-09-13T13:13:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:13:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:11:49.115434+0000", 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 3, 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:13:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:12:26.543807+0000", 2024-09-13T13:13:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:13:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 4, 2024-09-13T13:13:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:13:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:13:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:13:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:13:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:13:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:12:01.459078+0000", 2024-09-13T13:13:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:13:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 5, 2024-09-13T13:13:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:13:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:13:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:13:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:13:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:13:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:12:31.235507+0000", 2024-09-13T13:13:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:13:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 6, 2024-09-13T13:13:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:12:57.940918+0000", 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 102 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 7, 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:13:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:13:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:12:16.795987+0000", 2024-09-13T13:13:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:13:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:13:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_upmap": [ 2024-09-13T13:13:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "5.1c", 2024-09-13T13:13:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: "osds": [ 2024-09-13T13:13:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: 7, 2024-09-13T13:13:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: 1, 2024-09-13T13:13:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: 5 2024-09-13T13:13:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:13:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_upmap_items": [ 2024-09-13T13:13:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.10", 2024-09-13T13:13:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:13:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 1, 2024-09-13T13:13:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 2 2024-09-13T13:13:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.12", 2024-09-13T13:13:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:13:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 1, 2024-09-13T13:13:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 5 2024-09-13T13:13:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.15", 2024-09-13T13:13:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:13:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 3, 2024-09-13T13:13:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 2 2024-09-13T13:13:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.17", 2024-09-13T13:13:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:13:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 3, 2024-09-13T13:13:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 5 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.1f", 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 1, 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 2 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:13:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_upmap_primaries": [], 2024-09-13T13:13:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_temp": [], 2024-09-13T13:13:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_temp": [], 2024-09-13T13:13:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "blocklist": { 2024-09-13T13:13:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/2104786675": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:13:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:6800/910649698": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:13:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1471436716": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:13:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/4151877300": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:13:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/3080924817": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:13:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/4248579079": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:13:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/455730280": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:13:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1887735845": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:13:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:6800/1437783219": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:13:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:6800/326637643": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/991228238": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/3415403690": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1933060029": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1009267179": "2024-09-14T13:01:31.849627+0000" 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "range_blocklist": {}, 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profiles": { 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "default": { 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "crush-failure-domain": "osd", 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "k": "2", 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "m": "1", 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "plugin": "jerasure", 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "technique": "reed_sol_van" 2024-09-13T13:13:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps_queue": [], 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: "new_removed_snaps": [], 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: "new_purged_snaps": [], 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_node_flags": {}, 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: "device_class_flags": {}, 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_mode": { 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_mode_enabled": false, 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_bucket_count": 0, 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: "degraded_stretch_mode": 0, 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: "recovering_stretch_mode": 0, 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_mode_bucket": 0 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:00.355 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:13:00.356 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:00.389 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:00.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:00.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:00.441 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:13:00.491 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:00.535 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:00.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:00.636 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:00.719 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:00.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:00.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:00.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:00 smithi052 ceph-mon[36682]: 3.c scrub starts 2024-09-13T13:13:00.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:00 smithi052 ceph-mon[36682]: 3.c scrub ok 2024-09-13T13:13:00.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:00 smithi052 ceph-mon[36682]: 6.11 scrub starts 2024-09-13T13:13:00.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:00 smithi052 ceph-mon[36682]: 6.11 scrub ok 2024-09-13T13:13:00.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:00 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/795860819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:00.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:00 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3971597019' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-13T13:13:00.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:00 smithi052 ceph-mon[29471]: 3.c scrub starts 2024-09-13T13:13:00.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:00 smithi052 ceph-mon[29471]: 3.c scrub ok 2024-09-13T13:13:00.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:00 smithi052 ceph-mon[29471]: 6.11 scrub starts 2024-09-13T13:13:00.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:00 smithi052 ceph-mon[29471]: 6.11 scrub ok 2024-09-13T13:13:00.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:00 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/795860819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:00.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:00 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3971597019' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-13T13:13:00.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:00 smithi167 ceph-mon[33178]: 3.c scrub starts 2024-09-13T13:13:00.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:00 smithi167 ceph-mon[33178]: 3.c scrub ok 2024-09-13T13:13:00.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:00 smithi167 ceph-mon[33178]: 6.11 scrub starts 2024-09-13T13:13:00.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:00 smithi167 ceph-mon[33178]: 6.11 scrub ok 2024-09-13T13:13:00.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:00 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/795860819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:00.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:00 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3971597019' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-13T13:13:00.821 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:00.856 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:00.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:00.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:00.959 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:00.996 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:00.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:00.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:01.098 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:01.134 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:01.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:01.235 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:01.255 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:01.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:01.383 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:01.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:01.423 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:01.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:01.524 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:01.558 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:01.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:01.659 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:01.694 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:01.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:01.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:01.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:01 smithi052 ceph-mon[36682]: pgmap v585: 148 pgs: 1 active+degraded, 1 peering, 146 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 864 B/s rd, 0 B/s wr, 902 op/s; 1/176739 objects degraded (0.001%); 4.5 MiB/s, 76 objects/s recovering 2024-09-13T13:13:01.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:01 smithi052 ceph-mon[36682]: 9.6 scrub starts 2024-09-13T13:13:01.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:01 smithi052 ceph-mon[36682]: Health check failed: Degraded data redundancy: 1/176739 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:13:01.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:01 smithi052 ceph-mon[36682]: 4.14 scrub starts 2024-09-13T13:13:01.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:01 smithi052 ceph-mon[36682]: 4.14 scrub ok 2024-09-13T13:13:01.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:01 smithi052 ceph-mon[29471]: pgmap v585: 148 pgs: 1 active+degraded, 1 peering, 146 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 864 B/s rd, 0 B/s wr, 902 op/s; 1/176739 objects degraded (0.001%); 4.5 MiB/s, 76 objects/s recovering 2024-09-13T13:13:01.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:01 smithi052 ceph-mon[29471]: 9.6 scrub starts 2024-09-13T13:13:01.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:01 smithi052 ceph-mon[29471]: Health check failed: Degraded data redundancy: 1/176739 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:13:01.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:01 smithi052 ceph-mon[29471]: 4.14 scrub starts 2024-09-13T13:13:01.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:01 smithi052 ceph-mon[29471]: 4.14 scrub ok 2024-09-13T13:13:01.795 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:01.812 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:01 smithi167 ceph-mon[33178]: pgmap v585: 148 pgs: 1 active+degraded, 1 peering, 146 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 864 B/s rd, 0 B/s wr, 902 op/s; 1/176739 objects degraded (0.001%); 4.5 MiB/s, 76 objects/s recovering 2024-09-13T13:13:01.812 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:01 smithi167 ceph-mon[33178]: 9.6 scrub starts 2024-09-13T13:13:01.812 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:01 smithi167 ceph-mon[33178]: Health check failed: Degraded data redundancy: 1/176739 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:13:01.812 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:01 smithi167 ceph-mon[33178]: 4.14 scrub starts 2024-09-13T13:13:01.812 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:01 smithi167 ceph-mon[33178]: 4.14 scrub ok 2024-09-13T13:13:01.836 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:01.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:01.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:01.938 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:01.973 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:01.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:02.004 INFO:tasks.thrashosds.thrasher:j is {'epoch': 310, 'fsid': '2b415b38-71cf-11ef-bceb-c7b262605968', 'created': '2024-09-13T12:55:27.893882+0000', 'modified': '2024-09-13T13:12:58.939156+0000', 'last_up_change': '2024-09-13T13:04:02.281376+0000', 'last_in_change': '2024-09-13T13:12:52.437622+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T12:58:31.271893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '277', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-13T13:01:00.306747+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-13T13:01:01.103523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-13T13:01:02.249986+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-13T13:01:04.203764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-13T13:01:06.293676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-13T13:10:07.820868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '245', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}], 'osds': [{'osd': 0, 'uuid': 'd52dca1a-94c1-4e9e-93f5-687c038cd7c7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 305, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6801', 'nonce': 542298937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6802', 'nonce': 542298937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6804', 'nonce': 542298937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6803', 'nonce': 542298937}]}, 'public_addr': '172.21.15.52:6801/542298937', 'cluster_addr': '172.21.15.52:6802/542298937', 'heartbeat_back_addr': '172.21.15.52:6804/542298937', 'heartbeat_front_addr': '172.21.15.52:6803/542298937', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b40a3574-3c6f-4810-a651-8d64aee6801b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 309, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6805', 'nonce': 26289431}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6806', 'nonce': 26289431}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6808', 'nonce': 26289431}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6807', 'nonce': 26289431}]}, 'public_addr': '172.21.15.52:6805/26289431', 'cluster_addr': '172.21.15.52:6806/26289431', 'heartbeat_back_addr': '172.21.15.52:6808/26289431', 'heartbeat_front_addr': '172.21.15.52:6807/26289431', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1e8e49a0-ccf8-4b7f-839b-104b64601408', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 305, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6809', 'nonce': 3960240393}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6810', 'nonce': 3960240393}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6812', 'nonce': 3960240393}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6811', 'nonce': 3960240393}]}, 'public_addr': '172.21.15.52:6809/3960240393', 'cluster_addr': '172.21.15.52:6810/3960240393', 'heartbeat_back_addr': '172.21.15.52:6812/3960240393', 'heartbeat_front_addr': '172.21.15.52:6811/3960240393', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9f6f1d22-be68-4d86-9d4b-86fd9e87d141', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 309, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6813', 'nonce': 2025522885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6814', 'nonce': 2025522885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6816', 'nonce': 2025522885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6815', 'nonce': 2025522885}]}, 'public_addr': '172.21.15.52:6813/2025522885', 'cluster_addr': '172.21.15.52:6814/2025522885', 'heartbeat_back_addr': '172.21.15.52:6816/2025522885', 'heartbeat_front_addr': '172.21.15.52:6815/2025522885', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1326d850-caf4-484e-90a1-052f54f9f306', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6800', 'nonce': 2137138858}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 2137138858}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 2137138858}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6802', 'nonce': 2137138858}]}, 'public_addr': '172.21.15.167:6800/2137138858', 'cluster_addr': '172.21.15.167:6801/2137138858', 'heartbeat_back_addr': '172.21.15.167:6803/2137138858', 'heartbeat_front_addr': '172.21.15.167:6802/2137138858', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 305, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6804', 'nonce': 2906884337}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 2906884337}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 2906884337}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6806', 'nonce': 2906884337}]}, 'public_addr': '172.21.15.167:6804/2906884337', 'cluster_addr': '172.21.15.167:6805/2906884337', 'heartbeat_back_addr': '172.21.15.167:6807/2906884337', 'heartbeat_front_addr': '172.21.15.167:6806/2906884337', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2471907d-fcbd-44e0-9d29-54151ada9df0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 48, 'last_clean_end': 102, 'up_from': 118, 'up_thru': 305, 'down_at': 103, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6808', 'nonce': 143093129}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 143093129}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 143093129}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6810', 'nonce': 143093129}]}, 'public_addr': '172.21.15.167:6808/143093129', 'cluster_addr': '172.21.15.167:6809/143093129', 'heartbeat_back_addr': '172.21.15.167:6811/143093129', 'heartbeat_front_addr': '172.21.15.167:6810/143093129', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '97d2c6d7-7989-4c11-91e0-32ee87e76c6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.31549072265625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 305, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6812', 'nonce': 403155155}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 403155155}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 403155155}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6814', 'nonce': 403155155}]}, 'public_addr': '172.21.15.167:6812/403155155', 'cluster_addr': '172.21.15.167:6813/403155155', 'heartbeat_back_addr': '172.21.15.167:6815/403155155', 'heartbeat_front_addr': '172.21.15.167:6814/403155155', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:12:16.176074+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:12:05.682291+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:11:49.115434+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:12:26.543807+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:12:01.459078+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:12:31.235507+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:12:57.940918+0000', 'dead_epoch': 102}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:12:16.795987+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.1c', 'osds': [7, 1, 5]}], 'pg_upmap_items': [{'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.15', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.52:0/2104786675': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:6800/910649698': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:0/1471436716': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:0/4151877300': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:0/3080924817': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:0/4248579079': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:0/455730280': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/1887735845': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:6800/1437783219': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:6800/326637643': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/991228238': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/3415403690': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/1933060029': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:0/1009267179': '2024-09-14T13:01:31.849627+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T13:13:02.004 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 5.1c 2024-09-13T13:13:02.004 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd rm-pg-upmap 5.1c 2024-09-13T13:13:02.074 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:02.104 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:02.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:02.206 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:02.235 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:02.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:02.285 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:13:02.285 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:13:02.337 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:02.378 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:02.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:02.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:02.480 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:02.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:02.517 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:02.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:02.618 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:02.652 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:02.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:02.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:02.754 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:02.787 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:02.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:02.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:02 smithi167 ceph-mon[33178]: 9.6 scrub ok 2024-09-13T13:13:02.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:02 smithi167 ceph-mon[33178]: 4.5 scrub starts 2024-09-13T13:13:02.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:02 smithi167 ceph-mon[33178]: 4.5 scrub ok 2024-09-13T13:13:02.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:02 smithi167 ceph-mon[33178]: 3.f scrub starts 2024-09-13T13:13:02.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:02 smithi167 ceph-mon[33178]: 3.f scrub ok 2024-09-13T13:13:02.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:02 smithi167 ceph-mon[33178]: 5.1f scrub starts 2024-09-13T13:13:02.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:02 smithi167 ceph-mon[33178]: 5.1f scrub ok 2024-09-13T13:13:02.885 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[29471]: 9.6 scrub ok 2024-09-13T13:13:02.885 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[29471]: 4.5 scrub starts 2024-09-13T13:13:02.886 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[29471]: 4.5 scrub ok 2024-09-13T13:13:02.886 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[29471]: 3.f scrub starts 2024-09-13T13:13:02.886 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[29471]: 3.f scrub ok 2024-09-13T13:13:02.886 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[29471]: 5.1f scrub starts 2024-09-13T13:13:02.886 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[29471]: 5.1f scrub ok 2024-09-13T13:13:02.886 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[36682]: 9.6 scrub ok 2024-09-13T13:13:02.886 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[36682]: 4.5 scrub starts 2024-09-13T13:13:02.886 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[36682]: 4.5 scrub ok 2024-09-13T13:13:02.886 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[36682]: 3.f scrub starts 2024-09-13T13:13:02.886 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[36682]: 3.f scrub ok 2024-09-13T13:13:02.886 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[36682]: 5.1f scrub starts 2024-09-13T13:13:02.887 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:02 smithi052 ceph-mon[36682]: 5.1f scrub ok 2024-09-13T13:13:02.889 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:02.930 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:02.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:03.031 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:03.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:03.064 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:03.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:03.165 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:03.202 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:03.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:03.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:03.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:03.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:03.340 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:03.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:03.441 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:03.481 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:03.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:03.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:03.583 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:03.597 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T13:13:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:03.635 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:03.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:03.675 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:03.736 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:03.741 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:03.801 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:03.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:03.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:03.817 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:03 smithi052 ceph-mon[29471]: pgmap v586: 148 pgs: 1 active+degraded, 1 peering, 146 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 537 op/s; 1/176739 objects degraded (0.001%); 4.5 MiB/s, 73 objects/s recovering 2024-09-13T13:13:03.817 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:03 smithi052 ceph-mon[29471]: 6.1f scrub starts 2024-09-13T13:13:03.817 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:03 smithi052 ceph-mon[29471]: 6.1f scrub ok 2024-09-13T13:13:03.817 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:03 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:13:03.817 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:03 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:13:03.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:03 smithi052 ceph-mon[36682]: pgmap v586: 148 pgs: 1 active+degraded, 1 peering, 146 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 537 op/s; 1/176739 objects degraded (0.001%); 4.5 MiB/s, 73 objects/s recovering 2024-09-13T13:13:03.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:03 smithi052 ceph-mon[36682]: 6.1f scrub starts 2024-09-13T13:13:03.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:03 smithi052 ceph-mon[36682]: 6.1f scrub ok 2024-09-13T13:13:03.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:03 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:13:03.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:03 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:13:03.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:03 smithi167 ceph-mon[33178]: pgmap v586: 148 pgs: 1 active+degraded, 1 peering, 146 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 537 op/s; 1/176739 objects degraded (0.001%); 4.5 MiB/s, 73 objects/s recovering 2024-09-13T13:13:03.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:03 smithi167 ceph-mon[33178]: 6.1f scrub starts 2024-09-13T13:13:03.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:03 smithi167 ceph-mon[33178]: 6.1f scrub ok 2024-09-13T13:13:03.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:03 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:13:03.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:03 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:13:03.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:03.935 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:03.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:03.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:04.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:04.066 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:04.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:04.168 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:04.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:04.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:04.301 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:04.339 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:04.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:04.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:04.441 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:04.470 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:04.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:04.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:04.515 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:13:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:13:04.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:04.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:04.607 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:04.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:04.646 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[36682]: 6.19 scrub starts 2024-09-13T13:13:04.647 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[36682]: 6.19 scrub ok 2024-09-13T13:13:04.647 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:13:04.647 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:13:04.647 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[36682]: 5.6 scrub starts 2024-09-13T13:13:04.647 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[36682]: 5.6 scrub ok 2024-09-13T13:13:04.647 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[36682]: 5.9 scrub starts 2024-09-13T13:13:04.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[36682]: 5.9 scrub ok 2024-09-13T13:13:04.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[36682]: osdmap e311: 8 total, 8 up, 8 in 2024-09-13T13:13:04.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[36682]: 3.3 scrub starts 2024-09-13T13:13:04.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[36682]: 3.3 scrub ok 2024-09-13T13:13:04.648 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[29471]: 6.19 scrub starts 2024-09-13T13:13:04.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[29471]: 6.19 scrub ok 2024-09-13T13:13:04.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:13:04.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:13:04.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[29471]: 5.6 scrub starts 2024-09-13T13:13:04.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[29471]: 5.6 scrub ok 2024-09-13T13:13:04.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[29471]: 5.9 scrub starts 2024-09-13T13:13:04.650 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[29471]: 5.9 scrub ok 2024-09-13T13:13:04.650 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[29471]: osdmap e311: 8 total, 8 up, 8 in 2024-09-13T13:13:04.650 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[29471]: 3.3 scrub starts 2024-09-13T13:13:04.650 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:04 smithi052 ceph-mon[29471]: 3.3 scrub ok 2024-09-13T13:13:04.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:04.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:04.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:04.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:04 smithi167 ceph-mon[33178]: 6.19 scrub starts 2024-09-13T13:13:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:04 smithi167 ceph-mon[33178]: 6.19 scrub ok 2024-09-13T13:13:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:04 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:13:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:04 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:13:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:04 smithi167 ceph-mon[33178]: 5.6 scrub starts 2024-09-13T13:13:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:04 smithi167 ceph-mon[33178]: 5.6 scrub ok 2024-09-13T13:13:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:04 smithi167 ceph-mon[33178]: 5.9 scrub starts 2024-09-13T13:13:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:04 smithi167 ceph-mon[33178]: 5.9 scrub ok 2024-09-13T13:13:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:04 smithi167 ceph-mon[33178]: osdmap e311: 8 total, 8 up, 8 in 2024-09-13T13:13:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:04 smithi167 ceph-mon[33178]: 3.3 scrub starts 2024-09-13T13:13:04.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:04 smithi167 ceph-mon[33178]: 3.3 scrub ok 2024-09-13T13:13:04.838 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:04.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:04.866 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:04.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:04.923 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:04.931 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:13:04.967 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:05.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:05.039 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:05.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:05.141 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:05.189 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:05.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:05.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:05.292 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:05.350 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:05.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:05.452 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:05.482 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:05.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:05.544 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:13:05.557 INFO:teuthology.orchestra.run.smithi052.stderr:clear 5.1c pg_upmap mapping 2024-09-13T13:13:05.580 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:13:05.580 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:13:05.583 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:05.607 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:05.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:05.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:05.708 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:05.749 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:05.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:05.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:05.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:05 smithi167 ceph-mon[33178]: pgmap v588: 148 pgs: 1 active+degraded, 147 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 997 op/s; 1/168462 objects degraded (0.001%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T13:13:05.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:05.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:05 smithi167 ceph-mon[33178]: 5.a scrub starts 2024-09-13T13:13:05.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:05 smithi167 ceph-mon[33178]: 5.a scrub ok 2024-09-13T13:13:05.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:05 smithi167 ceph-mon[33178]: 3.10 scrub starts 2024-09-13T13:13:05.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:05 smithi167 ceph-mon[33178]: 3.10 scrub ok 2024-09-13T13:13:05.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:05 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2761660033' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:05.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:05 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1871005073' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap", "pgid": "5.1c"} : dispatch 2024-09-13T13:13:05.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:05 smithi167 ceph-mon[33178]: from='client.35969 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap", "pgid": "5.1c"} : dispatch 2024-09-13T13:13:05.851 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:05.880 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:05.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:05.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:05.982 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:05.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[29471]: pgmap v588: 148 pgs: 1 active+degraded, 147 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 997 op/s; 1/168462 objects degraded (0.001%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T13:13:05.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:05.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[29471]: 5.a scrub starts 2024-09-13T13:13:05.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[29471]: 5.a scrub ok 2024-09-13T13:13:05.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[29471]: 3.10 scrub starts 2024-09-13T13:13:05.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[29471]: 3.10 scrub ok 2024-09-13T13:13:05.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2761660033' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:05.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1871005073' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap", "pgid": "5.1c"} : dispatch 2024-09-13T13:13:05.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[29471]: from='client.35969 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap", "pgid": "5.1c"} : dispatch 2024-09-13T13:13:05.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[36682]: pgmap v588: 148 pgs: 1 active+degraded, 147 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 997 op/s; 1/168462 objects degraded (0.001%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T13:13:05.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:05.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[36682]: 5.a scrub starts 2024-09-13T13:13:05.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[36682]: 5.a scrub ok 2024-09-13T13:13:05.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[36682]: 3.10 scrub starts 2024-09-13T13:13:05.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[36682]: 3.10 scrub ok 2024-09-13T13:13:05.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2761660033' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:05.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1871005073' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap", "pgid": "5.1c"} : dispatch 2024-09-13T13:13:05.992 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:05 smithi052 ceph-mon[36682]: from='client.35969 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap", "pgid": "5.1c"} : dispatch 2024-09-13T13:13:06.018 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:06.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:06.120 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:06.157 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:06.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:06.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:06.259 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:06.287 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:13:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:13:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:06.298 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:06.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:06.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:06.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:06.434 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:06.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:06.535 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:06.557 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:06.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:06.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:06.617 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[36682]: 5.15 scrub starts 2024-09-13T13:13:06.618 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[29471]: 5.15 scrub starts 2024-09-13T13:13:06.618 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[29471]: 5.15 scrub ok 2024-09-13T13:13:06.660 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:06.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:06.694 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:06.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:06.795 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:06.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:06 smithi167 ceph-mon[33178]: 5.15 scrub starts 2024-09-13T13:13:06.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:06 smithi167 ceph-mon[33178]: 5.15 scrub ok 2024-09-13T13:13:06.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:06 smithi167 ceph-mon[33178]: 6.14 scrub starts 2024-09-13T13:13:06.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:06 smithi167 ceph-mon[33178]: 6.14 scrub ok 2024-09-13T13:13:06.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:06 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:13:06.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:06 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2761660033' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:13:06.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:06 smithi167 ceph-mon[33178]: from='client.35969 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.1c"}]': finished 2024-09-13T13:13:06.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:06 smithi167 ceph-mon[33178]: osdmap e312: 8 total, 8 up, 8 in 2024-09-13T13:13:06.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:06 smithi167 ceph-mon[33178]: 3.12 scrub starts 2024-09-13T13:13:06.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:06 smithi167 ceph-mon[33178]: 3.12 scrub ok 2024-09-13T13:13:06.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:06.832 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:06.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:06.934 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:06.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:06.963 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:06.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:06.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[36682]: 5.15 scrub ok 2024-09-13T13:13:06.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[36682]: 6.14 scrub starts 2024-09-13T13:13:06.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[36682]: 6.14 scrub ok 2024-09-13T13:13:06.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:13:06.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2761660033' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:13:06.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[36682]: from='client.35969 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.1c"}]': finished 2024-09-13T13:13:06.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[36682]: osdmap e312: 8 total, 8 up, 8 in 2024-09-13T13:13:06.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[36682]: 3.12 scrub starts 2024-09-13T13:13:06.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[36682]: 3.12 scrub ok 2024-09-13T13:13:06.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[29471]: 6.14 scrub starts 2024-09-13T13:13:06.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[29471]: 6.14 scrub ok 2024-09-13T13:13:06.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:13:06.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2761660033' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:13:06.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[29471]: from='client.35969 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.1c"}]': finished 2024-09-13T13:13:06.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[29471]: osdmap e312: 8 total, 8 up, 8 in 2024-09-13T13:13:06.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[29471]: 3.12 scrub starts 2024-09-13T13:13:06.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:06 smithi052 ceph-mon[29471]: 3.12 scrub ok 2024-09-13T13:13:07.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:07.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:07.094 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:07.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:07.195 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:07.233 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:07.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:07.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:07.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:07.370 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:07.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:07.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:07.472 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:07.507 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:07.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:07.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:07.609 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:07.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:07.646 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:07.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:07.748 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:07.786 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:07.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:07.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:07.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:07 smithi167 ceph-mon[33178]: 5.f scrub starts 2024-09-13T13:13:07.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:07 smithi167 ceph-mon[33178]: 5.f scrub ok 2024-09-13T13:13:07.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:07 smithi167 ceph-mon[33178]: pgmap v590: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 1.08k op/s; 1.8 MiB/s, 28 objects/s recovering 2024-09-13T13:13:07.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:07 smithi167 ceph-mon[33178]: 4.a scrub starts 2024-09-13T13:13:07.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:07 smithi167 ceph-mon[33178]: 4.a scrub ok 2024-09-13T13:13:07.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:07 smithi167 ceph-mon[33178]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/168462 objects degraded (0.001%), 1 pg degraded) 2024-09-13T13:13:07.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:07 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:13:07.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:07 smithi167 ceph-mon[33178]: osdmap e313: 8 total, 8 up, 8 in 2024-09-13T13:13:07.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:07 smithi167 ceph-mon[33178]: 1.0 scrub starts 2024-09-13T13:13:07.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:07 smithi167 ceph-mon[33178]: 1.0 scrub ok 2024-09-13T13:13:07.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:07.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:07.920 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:07.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:07.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[36682]: 5.f scrub starts 2024-09-13T13:13:07.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[36682]: 5.f scrub ok 2024-09-13T13:13:07.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[36682]: pgmap v590: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 1.08k op/s; 1.8 MiB/s, 28 objects/s recovering 2024-09-13T13:13:07.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[36682]: 4.a scrub starts 2024-09-13T13:13:07.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[36682]: 4.a scrub ok 2024-09-13T13:13:07.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[36682]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/168462 objects degraded (0.001%), 1 pg degraded) 2024-09-13T13:13:07.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:13:07.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[36682]: osdmap e313: 8 total, 8 up, 8 in 2024-09-13T13:13:07.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[36682]: 1.0 scrub starts 2024-09-13T13:13:07.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[36682]: 1.0 scrub ok 2024-09-13T13:13:07.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[29471]: 5.f scrub starts 2024-09-13T13:13:07.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[29471]: 5.f scrub ok 2024-09-13T13:13:07.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[29471]: pgmap v590: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 1.08k op/s; 1.8 MiB/s, 28 objects/s recovering 2024-09-13T13:13:07.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[29471]: 4.a scrub starts 2024-09-13T13:13:07.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[29471]: 4.a scrub ok 2024-09-13T13:13:07.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[29471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/168462 objects degraded (0.001%), 1 pg degraded) 2024-09-13T13:13:07.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:13:07.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[29471]: osdmap e313: 8 total, 8 up, 8 in 2024-09-13T13:13:07.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[29471]: 1.0 scrub starts 2024-09-13T13:13:07.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:07 smithi052 ceph-mon[29471]: 1.0 scrub ok 2024-09-13T13:13:08.022 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:08.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:08.059 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:08.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:08.160 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:08.189 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:08.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:08.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:08.291 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:08.324 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:08.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:08.426 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:08.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:08.449 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:08.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:08.550 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:08.586 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:08.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:08.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:08.688 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:08.724 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:08.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:08.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:08.852 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:08.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:08.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:08.954 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:08 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:13:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:08 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:13:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:08 smithi052 ceph-mon[36682]: osdmap e314: 8 total, 8 up, 8 in 2024-09-13T13:13:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:08 smithi052 ceph-mon[36682]: osdmap e315: 8 total, 8 up, 8 in 2024-09-13T13:13:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:08 smithi052 ceph-mon[36682]: 5.1c scrub starts 2024-09-13T13:13:08.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:08 smithi052 ceph-mon[36682]: 5.1c scrub ok 2024-09-13T13:13:08.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:08 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:13:08.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:08 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:13:08.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:08 smithi052 ceph-mon[29471]: osdmap e314: 8 total, 8 up, 8 in 2024-09-13T13:13:08.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:08 smithi052 ceph-mon[29471]: osdmap e315: 8 total, 8 up, 8 in 2024-09-13T13:13:08.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:08 smithi052 ceph-mon[29471]: 5.1c scrub starts 2024-09-13T13:13:08.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:08 smithi052 ceph-mon[29471]: 5.1c scrub ok 2024-09-13T13:13:08.994 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:08.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:08.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:09.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:08 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:13:09.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:08 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:13:09.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:08 smithi167 ceph-mon[33178]: osdmap e314: 8 total, 8 up, 8 in 2024-09-13T13:13:09.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:08 smithi167 ceph-mon[33178]: osdmap e315: 8 total, 8 up, 8 in 2024-09-13T13:13:09.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:08 smithi167 ceph-mon[33178]: 5.1c scrub starts 2024-09-13T13:13:09.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:08 smithi167 ceph-mon[33178]: 5.1c scrub ok 2024-09-13T13:13:09.097 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:09.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:09.133 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:09.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:09.234 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:09.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:09.268 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:09.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:09.369 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:09.406 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:09.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:09.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:09.508 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:09.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:09.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:09.634 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:09.673 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:09.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:09.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:09.776 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:09.804 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:09.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:09.906 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:09.943 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:09.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:09.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:09 smithi052 ceph-mon[36682]: pgmap v593: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 969 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T13:13:09.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:09 smithi052 ceph-mon[36682]: 6.1d scrub starts 2024-09-13T13:13:09.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:09 smithi052 ceph-mon[36682]: 6.1d scrub ok 2024-09-13T13:13:09.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:09 smithi052 ceph-mon[36682]: 4.10 scrub starts 2024-09-13T13:13:09.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:09 smithi052 ceph-mon[36682]: 4.10 scrub ok 2024-09-13T13:13:09.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:09 smithi052 ceph-mon[29471]: pgmap v593: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 969 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T13:13:09.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:09 smithi052 ceph-mon[29471]: 6.1d scrub starts 2024-09-13T13:13:09.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:09 smithi052 ceph-mon[29471]: 6.1d scrub ok 2024-09-13T13:13:09.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:09 smithi052 ceph-mon[29471]: 4.10 scrub starts 2024-09-13T13:13:09.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:09 smithi052 ceph-mon[29471]: 4.10 scrub ok 2024-09-13T13:13:10.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:10.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:09 smithi167 ceph-mon[33178]: pgmap v593: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 969 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T13:13:10.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:09 smithi167 ceph-mon[33178]: 6.1d scrub starts 2024-09-13T13:13:10.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:09 smithi167 ceph-mon[33178]: 6.1d scrub ok 2024-09-13T13:13:10.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:09 smithi167 ceph-mon[33178]: 4.10 scrub starts 2024-09-13T13:13:10.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:09 smithi167 ceph-mon[33178]: 4.10 scrub ok 2024-09-13T13:13:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:10.076 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:10.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:10.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:10.215 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:10.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:10.316 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:10.352 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:10.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:10.451 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T13:13:10.454 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:10.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:10.493 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:10.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:10.594 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:10.624 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:10.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:10.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:10.726 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:10.767 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:10.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:10.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:10.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:10.890 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:10.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:10.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:10 smithi052 ceph-mon[36682]: 5.16 scrub starts 2024-09-13T13:13:10.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:10 smithi052 ceph-mon[36682]: 5.16 scrub ok 2024-09-13T13:13:10.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:10 smithi052 ceph-mon[36682]: 3.1b scrub starts 2024-09-13T13:13:10.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:10 smithi052 ceph-mon[36682]: 3.1b scrub ok 2024-09-13T13:13:10.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:10 smithi052 ceph-mon[36682]: 5.14 scrub starts 2024-09-13T13:13:10.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:10 smithi052 ceph-mon[36682]: 5.14 scrub ok 2024-09-13T13:13:10.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:10 smithi052 ceph-mon[29471]: 5.16 scrub starts 2024-09-13T13:13:10.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:10 smithi052 ceph-mon[29471]: 5.16 scrub ok 2024-09-13T13:13:10.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:10 smithi052 ceph-mon[29471]: 3.1b scrub starts 2024-09-13T13:13:10.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:10 smithi052 ceph-mon[29471]: 3.1b scrub ok 2024-09-13T13:13:10.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:10 smithi052 ceph-mon[29471]: 5.14 scrub starts 2024-09-13T13:13:10.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:10 smithi052 ceph-mon[29471]: 5.14 scrub ok 2024-09-13T13:13:10.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:11.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:11.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:11.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:11.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:10 smithi167 ceph-mon[33178]: 5.16 scrub starts 2024-09-13T13:13:11.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:10 smithi167 ceph-mon[33178]: 5.16 scrub ok 2024-09-13T13:13:11.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:10 smithi167 ceph-mon[33178]: 3.1b scrub starts 2024-09-13T13:13:11.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:10 smithi167 ceph-mon[33178]: 3.1b scrub ok 2024-09-13T13:13:11.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:10 smithi167 ceph-mon[33178]: 5.14 scrub starts 2024-09-13T13:13:11.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:10 smithi167 ceph-mon[33178]: 5.14 scrub ok 2024-09-13T13:13:11.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:11.155 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:11.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:11.257 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:11.293 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:11.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:11.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:11.395 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:11.417 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:11.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:11.518 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:11.557 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:11.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:11.658 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:11.684 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:11.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:11.786 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:11.824 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:11.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:11.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:11.926 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:11.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:11.949 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:11.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:11 smithi052 ceph-mon[36682]: pgmap v595: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-13T13:13:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:11 smithi052 ceph-mon[36682]: 9.a scrub starts 2024-09-13T13:13:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:11 smithi052 ceph-mon[36682]: 4.6 scrub starts 2024-09-13T13:13:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:11 smithi052 ceph-mon[36682]: 4.6 scrub ok 2024-09-13T13:13:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:11 smithi052 ceph-mon[29471]: pgmap v595: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-13T13:13:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:11 smithi052 ceph-mon[29471]: 9.a scrub starts 2024-09-13T13:13:11.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:11 smithi052 ceph-mon[29471]: 4.6 scrub starts 2024-09-13T13:13:11.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:11 smithi052 ceph-mon[29471]: 4.6 scrub ok 2024-09-13T13:13:12.050 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:12.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:11 smithi167 ceph-mon[33178]: pgmap v595: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-13T13:13:12.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:11 smithi167 ceph-mon[33178]: 9.a scrub starts 2024-09-13T13:13:12.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:11 smithi167 ceph-mon[33178]: 4.6 scrub starts 2024-09-13T13:13:12.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:11 smithi167 ceph-mon[33178]: 4.6 scrub ok 2024-09-13T13:13:12.086 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:12.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:12.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:12.188 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:12.226 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:12.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:12.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:12.328 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:12.372 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:12.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:12.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:12.474 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:12.512 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:12.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:12.613 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:12.649 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:12.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:12.751 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:12.793 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:12.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:12.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:12.895 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:12.933 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:12.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:12.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:12.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[36682]: 5.b scrub starts 2024-09-13T13:13:12.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[36682]: 5.b scrub ok 2024-09-13T13:13:12.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[36682]: 5.c scrub starts 2024-09-13T13:13:12.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[36682]: 5.c scrub ok 2024-09-13T13:13:12.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[36682]: 5.7 scrub starts 2024-09-13T13:13:12.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[36682]: 5.7 scrub ok 2024-09-13T13:13:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[36682]: 9.a scrub ok 2024-09-13T13:13:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[36682]: 6.17 scrub starts 2024-09-13T13:13:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[36682]: 6.17 scrub ok 2024-09-13T13:13:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[36682]: 9.c scrub starts 2024-09-13T13:13:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:13:12.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[29471]: 5.b scrub starts 2024-09-13T13:13:12.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[29471]: 5.b scrub ok 2024-09-13T13:13:12.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[29471]: 5.c scrub starts 2024-09-13T13:13:12.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[29471]: 5.c scrub ok 2024-09-13T13:13:12.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[29471]: 5.7 scrub starts 2024-09-13T13:13:12.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[29471]: 5.7 scrub ok 2024-09-13T13:13:12.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[29471]: 9.a scrub ok 2024-09-13T13:13:12.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[29471]: 6.17 scrub starts 2024-09-13T13:13:12.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[29471]: 6.17 scrub ok 2024-09-13T13:13:12.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[29471]: 9.c scrub starts 2024-09-13T13:13:12.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:12 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:13:13.035 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:13.063 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:13.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:13.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:12 smithi167 ceph-mon[33178]: 5.b scrub starts 2024-09-13T13:13:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:12 smithi167 ceph-mon[33178]: 5.b scrub ok 2024-09-13T13:13:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:12 smithi167 ceph-mon[33178]: 5.c scrub starts 2024-09-13T13:13:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:12 smithi167 ceph-mon[33178]: 5.c scrub ok 2024-09-13T13:13:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:12 smithi167 ceph-mon[33178]: 5.7 scrub starts 2024-09-13T13:13:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:12 smithi167 ceph-mon[33178]: 5.7 scrub ok 2024-09-13T13:13:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:12 smithi167 ceph-mon[33178]: 9.a scrub ok 2024-09-13T13:13:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:12 smithi167 ceph-mon[33178]: 6.17 scrub starts 2024-09-13T13:13:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:12 smithi167 ceph-mon[33178]: 6.17 scrub ok 2024-09-13T13:13:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:12 smithi167 ceph-mon[33178]: 9.c scrub starts 2024-09-13T13:13:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:12 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:13:13.165 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:13.201 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:13.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:13.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:13.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:13.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:13.325 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:13.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:13.426 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:13.461 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:13.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:13.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:13.563 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:13.599 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:13.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:13.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:13.700 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:13.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:13.738 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:13.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:13.840 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:13.886 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:13.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:13.988 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:13.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[36682]: pgmap v596: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 998 op/s 2024-09-13T13:13:13.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[36682]: 2.0 scrub starts 2024-09-13T13:13:13.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[36682]: 2.0 scrub ok 2024-09-13T13:13:13.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[36682]: 3.5 scrub starts 2024-09-13T13:13:13.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[36682]: 3.5 scrub ok 2024-09-13T13:13:13.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[36682]: osdmap e316: 8 total, 8 up, 8 in 2024-09-13T13:13:13.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[36682]: 4.16 scrub starts 2024-09-13T13:13:13.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[36682]: 4.16 scrub ok 2024-09-13T13:13:13.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[36682]: 9.c scrub ok 2024-09-13T13:13:13.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[29471]: pgmap v596: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 998 op/s 2024-09-13T13:13:13.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[29471]: 2.0 scrub starts 2024-09-13T13:13:13.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[29471]: 2.0 scrub ok 2024-09-13T13:13:13.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[29471]: 3.5 scrub starts 2024-09-13T13:13:13.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[29471]: 3.5 scrub ok 2024-09-13T13:13:13.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[29471]: osdmap e316: 8 total, 8 up, 8 in 2024-09-13T13:13:13.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[29471]: 4.16 scrub starts 2024-09-13T13:13:13.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[29471]: 4.16 scrub ok 2024-09-13T13:13:13.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:13 smithi052 ceph-mon[29471]: 9.c scrub ok 2024-09-13T13:13:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:14.016 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:14.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:14.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:13 smithi167 ceph-mon[33178]: pgmap v596: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 998 op/s 2024-09-13T13:13:14.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:13 smithi167 ceph-mon[33178]: 2.0 scrub starts 2024-09-13T13:13:14.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:13 smithi167 ceph-mon[33178]: 2.0 scrub ok 2024-09-13T13:13:14.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:13 smithi167 ceph-mon[33178]: 3.5 scrub starts 2024-09-13T13:13:14.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:13 smithi167 ceph-mon[33178]: 3.5 scrub ok 2024-09-13T13:13:14.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:13 smithi167 ceph-mon[33178]: osdmap e316: 8 total, 8 up, 8 in 2024-09-13T13:13:14.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:13 smithi167 ceph-mon[33178]: 4.16 scrub starts 2024-09-13T13:13:14.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:13 smithi167 ceph-mon[33178]: 4.16 scrub ok 2024-09-13T13:13:14.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:13 smithi167 ceph-mon[33178]: 9.c scrub ok 2024-09-13T13:13:14.117 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:14.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:14.151 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:14.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:14.251 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:14.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:14.278 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:14.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:14.379 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:14.409 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:14.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:14.510 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:14.553 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:14.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:14.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:14.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:13:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:13:14.655 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:14.693 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:14.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:14.795 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:14.827 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:14.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:14.929 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:14.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:14.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:14.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:15.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:15.078 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:15.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:15.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:15.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:15.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:15.218 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:15.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:15.319 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:15.342 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:15.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:15.444 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:15.473 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:15.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:15.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:15.575 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:15.602 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:15.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:15.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:15.704 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:15.749 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:15.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:15.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:15.851 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:15.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:15.895 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:15.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:15.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: pgmap v598: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 822 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-13T13:13:15.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:13:15.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:13:15.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:15.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: 5.11 scrub starts 2024-09-13T13:13:15.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: 5.11 scrub ok 2024-09-13T13:13:15.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: 2.1 scrub starts 2024-09-13T13:13:15.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: 2.1 scrub ok 2024-09-13T13:13:15.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: osdmap e317: 8 total, 8 up, 8 in 2024-09-13T13:13:15.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: 4.1e scrub starts 2024-09-13T13:13:15.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: 4.1e scrub ok 2024-09-13T13:13:15.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: 9.8 scrub starts 2024-09-13T13:13:15.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:13:15.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:13:15.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:13:15.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: 4.8 scrub starts 2024-09-13T13:13:15.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[29471]: 4.8 scrub ok 2024-09-13T13:13:15.959 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: pgmap v598: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 822 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-13T13:13:15.959 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:13:15.959 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:13:15.959 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:15.960 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: 5.11 scrub starts 2024-09-13T13:13:15.960 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: 5.11 scrub ok 2024-09-13T13:13:15.960 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: 2.1 scrub starts 2024-09-13T13:13:15.960 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: 2.1 scrub ok 2024-09-13T13:13:15.960 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: osdmap e317: 8 total, 8 up, 8 in 2024-09-13T13:13:15.960 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: 4.1e scrub starts 2024-09-13T13:13:15.960 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: 4.1e scrub ok 2024-09-13T13:13:15.960 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: 9.8 scrub starts 2024-09-13T13:13:15.960 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:13:15.961 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:13:15.961 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:13:15.961 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: 4.8 scrub starts 2024-09-13T13:13:15.961 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:15 smithi052 ceph-mon[36682]: 4.8 scrub ok 2024-09-13T13:13:15.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:16.022 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:16.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: pgmap v598: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 822 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-13T13:13:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:13:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:13:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: 5.11 scrub starts 2024-09-13T13:13:16.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: 5.11 scrub ok 2024-09-13T13:13:16.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: 2.1 scrub starts 2024-09-13T13:13:16.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: 2.1 scrub ok 2024-09-13T13:13:16.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: osdmap e317: 8 total, 8 up, 8 in 2024-09-13T13:13:16.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: 4.1e scrub starts 2024-09-13T13:13:16.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: 4.1e scrub ok 2024-09-13T13:13:16.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: 9.8 scrub starts 2024-09-13T13:13:16.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:13:16.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:13:16.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:13:16.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: 4.8 scrub starts 2024-09-13T13:13:16.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:15 smithi167 ceph-mon[33178]: 4.8 scrub ok 2024-09-13T13:13:16.124 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:16.160 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:16.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:16.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:16.208 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:13:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:13:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:13:16.262 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:16.293 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:16.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:16.394 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:16.430 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:16.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:16.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:16.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:16.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:16.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:16.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:16.669 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:16.697 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:16.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:16.798 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:16.835 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:16.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:16.900 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:16.937 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:16.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:16 smithi052 ceph-mon[29471]: 5.12 scrub starts 2024-09-13T13:13:16.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:16 smithi052 ceph-mon[29471]: 5.12 scrub ok 2024-09-13T13:13:16.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:16 smithi052 ceph-mon[29471]: 9.8 scrub ok 2024-09-13T13:13:16.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:16 smithi052 ceph-mon[36682]: 5.12 scrub starts 2024-09-13T13:13:16.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:16 smithi052 ceph-mon[36682]: 5.12 scrub ok 2024-09-13T13:13:16.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:16 smithi052 ceph-mon[36682]: 9.8 scrub ok 2024-09-13T13:13:17.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:17.005 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:17.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:17.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:16 smithi167 ceph-mon[33178]: 5.12 scrub starts 2024-09-13T13:13:17.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:16 smithi167 ceph-mon[33178]: 5.12 scrub ok 2024-09-13T13:13:17.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:16 smithi167 ceph-mon[33178]: 9.8 scrub ok 2024-09-13T13:13:17.106 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:17.146 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:17.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:17.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:17.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:17.279 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:17.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:17.380 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:17.437 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:17.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:17.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:17.522 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:17.539 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:17.596 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:17.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:17.600 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:13:17.600 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:13:17.636 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:13:17.637 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:13:17.670 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:13:17.670 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:13:17.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:17.708 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:13:17.708 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:13:17.723 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:17.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:17.743 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:13:17.743 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:13:17.774 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:13:17.775 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:13:17.805 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:13:17.805 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:13:17.826 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:17.841 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:13:17.842 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:13:17.866 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:17.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:17.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:17.968 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:17 smithi052 ceph-mon[36682]: pgmap v600: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-13T13:13:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:17 smithi052 ceph-mon[36682]: 3.a scrub starts 2024-09-13T13:13:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:17 smithi052 ceph-mon[36682]: 3.a scrub ok 2024-09-13T13:13:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:17 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3431650459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:17.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:17 smithi052 ceph-mon[29471]: pgmap v600: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-13T13:13:17.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:17 smithi052 ceph-mon[29471]: 3.a scrub starts 2024-09-13T13:13:17.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:17 smithi052 ceph-mon[29471]: 3.a scrub ok 2024-09-13T13:13:17.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:17 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3431650459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:18.002 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:18.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:18.011 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:13:18.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:17 smithi167 ceph-mon[33178]: pgmap v600: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-13T13:13:18.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:17 smithi167 ceph-mon[33178]: 3.a scrub starts 2024-09-13T13:13:18.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:17 smithi167 ceph-mon[33178]: 3.a scrub ok 2024-09-13T13:13:18.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:17 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3431650459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:18.104 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:18.158 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:18.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:18.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:18.260 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:18.302 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:18.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:18.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:18.404 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:18.450 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:18.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:18.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:18.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:18.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:18.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:18.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:18.690 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:18.721 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:18.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:18.822 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:18.862 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:18.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:18.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[36682]: 5.4 scrub starts 2024-09-13T13:13:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[36682]: 5.4 scrub ok 2024-09-13T13:13:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[36682]: 3.0 scrub starts 2024-09-13T13:13:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[36682]: 3.0 scrub ok 2024-09-13T13:13:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[36682]: pgmap v601: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 671 op/s 2024-09-13T13:13:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[36682]: osdmap e318: 8 total, 8 up, 8 in 2024-09-13T13:13:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[36682]: 6.1a scrub starts 2024-09-13T13:13:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[36682]: 6.1a scrub ok 2024-09-13T13:13:18.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:13:18.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:13:18.990 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:13:18.990 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:13:18.990 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:13:18.990 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:13:18.990 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:13:18.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[29471]: 5.4 scrub starts 2024-09-13T13:13:18.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[29471]: 5.4 scrub ok 2024-09-13T13:13:18.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[29471]: 3.0 scrub starts 2024-09-13T13:13:18.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[29471]: 3.0 scrub ok 2024-09-13T13:13:18.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[29471]: pgmap v601: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 671 op/s 2024-09-13T13:13:18.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[29471]: osdmap e318: 8 total, 8 up, 8 in 2024-09-13T13:13:18.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[29471]: 6.1a scrub starts 2024-09-13T13:13:18.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[29471]: 6.1a scrub ok 2024-09-13T13:13:18.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:13:18.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:18 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:13:18.997 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:18.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:19.099 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:19.133 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:19.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:19.234 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:19.286 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:19.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:19.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:18 smithi167 ceph-mon[33178]: 5.4 scrub starts 2024-09-13T13:13:19.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:18 smithi167 ceph-mon[33178]: 5.4 scrub ok 2024-09-13T13:13:19.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:18 smithi167 ceph-mon[33178]: 3.0 scrub starts 2024-09-13T13:13:19.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:18 smithi167 ceph-mon[33178]: 3.0 scrub ok 2024-09-13T13:13:19.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:18 smithi167 ceph-mon[33178]: pgmap v601: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 671 op/s 2024-09-13T13:13:19.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:18 smithi167 ceph-mon[33178]: osdmap e318: 8 total, 8 up, 8 in 2024-09-13T13:13:19.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:18 smithi167 ceph-mon[33178]: 6.1a scrub starts 2024-09-13T13:13:19.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:18 smithi167 ceph-mon[33178]: 6.1a scrub ok 2024-09-13T13:13:19.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:18 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:13:19.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:18 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:13:19.388 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:19.421 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:19.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:19.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:19.523 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:19.562 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:19.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:19.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:19.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:19.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:19.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:19.809 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:19.826 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T13:13:19.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:19.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:19.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:19.893 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:19.937 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:19.989 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:19.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:19.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:20.035 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:19 smithi052 ceph-mon[29471]: 3.4 scrub starts 2024-09-13T13:13:20.035 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:19 smithi052 ceph-mon[29471]: 3.4 scrub ok 2024-09-13T13:13:20.035 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:19 smithi052 ceph-mon[29471]: 9.1 scrub starts 2024-09-13T13:13:20.035 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:19 smithi052 ceph-mon[36682]: 3.4 scrub starts 2024-09-13T13:13:20.035 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:19 smithi052 ceph-mon[36682]: 3.4 scrub ok 2024-09-13T13:13:20.035 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:19 smithi052 ceph-mon[36682]: 9.1 scrub starts 2024-09-13T13:13:20.091 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:20.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:20.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:20.223 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:20.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:20.262 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:20.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:20.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:19 smithi167 ceph-mon[33178]: 3.4 scrub starts 2024-09-13T13:13:20.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:19 smithi167 ceph-mon[33178]: 3.4 scrub ok 2024-09-13T13:13:20.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:19 smithi167 ceph-mon[33178]: 9.1 scrub starts 2024-09-13T13:13:20.363 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:20.400 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:20.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:20.502 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:20.532 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:20.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:20.633 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:20.660 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:20.697 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:20.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:20.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:20.752 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:13:20.800 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:20.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:20.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:20.929 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:20.964 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:13:20.980 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:20.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:21.044 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[36682]: 9.1 scrub ok 2024-09-13T13:13:21.045 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[36682]: pgmap v603: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-09-13T13:13:21.045 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[36682]: 4.9 scrub starts 2024-09-13T13:13:21.045 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[36682]: 4.9 scrub ok 2024-09-13T13:13:21.045 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[36682]: 9.2 scrub starts 2024-09-13T13:13:21.045 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[36682]: 2.2 scrub starts 2024-09-13T13:13:21.045 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[36682]: 2.2 scrub ok 2024-09-13T13:13:21.045 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/720451681' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:13:21.045 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[36682]: from='client.35975 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:13:21.046 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[29471]: 9.1 scrub ok 2024-09-13T13:13:21.046 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[29471]: pgmap v603: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-09-13T13:13:21.046 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[29471]: 4.9 scrub starts 2024-09-13T13:13:21.046 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[29471]: 4.9 scrub ok 2024-09-13T13:13:21.046 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[29471]: 9.2 scrub starts 2024-09-13T13:13:21.046 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[29471]: 2.2 scrub starts 2024-09-13T13:13:21.046 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[29471]: 2.2 scrub ok 2024-09-13T13:13:21.046 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/720451681' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:13:21.046 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:20 smithi052 ceph-mon[29471]: from='client.35975 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:13:21.083 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:21.113 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:21.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:21.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:21.215 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:21.266 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:21.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:21.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:20 smithi167 ceph-mon[33178]: 9.1 scrub ok 2024-09-13T13:13:21.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:20 smithi167 ceph-mon[33178]: pgmap v603: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-09-13T13:13:21.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:20 smithi167 ceph-mon[33178]: 4.9 scrub starts 2024-09-13T13:13:21.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:20 smithi167 ceph-mon[33178]: 4.9 scrub ok 2024-09-13T13:13:21.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:20 smithi167 ceph-mon[33178]: 9.2 scrub starts 2024-09-13T13:13:21.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:20 smithi167 ceph-mon[33178]: 2.2 scrub starts 2024-09-13T13:13:21.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:20 smithi167 ceph-mon[33178]: 2.2 scrub ok 2024-09-13T13:13:21.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:20 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/720451681' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:13:21.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:20 smithi167 ceph-mon[33178]: from='client.35975 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:13:21.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:21.400 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:21.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:21.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:21.502 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:21.521 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:13:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:13:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:13:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:13:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:166990 7.b 7:db1a2623:::benchmark_data_smithi052_71865_object45889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T13:13:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.766249+0000", 2024-09-13T13:13:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 513.75186127500001, 2024-09-13T13:13:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0195013719999999, 2024-09-13T13:13:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:13:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:13:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:13:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 166990 2024-09-13T13:13:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766249+0000", 2024-09-13T13:13:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766249+0000", 2024-09-13T13:13:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766252+0000", 2024-09-13T13:13:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.48e-06 2024-09-13T13:13:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766259+0000", 2024-09-13T13:13:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0849999999999999e-06 2024-09-13T13:13:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766272+0000", 2024-09-13T13:13:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.305e-05 2024-09-13T13:13:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.766281+0000", 2024-09-13T13:13:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5750000000000001e-06 2024-09-13T13:13:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.767413+0000", 2024-09-13T13:13:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011321529999999999 2024-09-13T13:13:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.767559+0000", 2024-09-13T13:13:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000146217 2024-09-13T13:13:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:13:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.767706+0000", 2024-09-13T13:13:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014731799999999999 2024-09-13T13:13:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.527 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.527 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.776972+0000", 2024-09-13T13:13:21.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0092654350000000007 2024-09-13T13:13:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.776984+0000", 2024-09-13T13:13:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2018000000000001e-05 2024-09-13T13:13:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.785665+0000", 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0086813429999999 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.785751+0000", 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5698000000000005e-05 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167004 7.b 7:d0e4af5f:::benchmark_data_smithi052_71865_object45903:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:13:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.789600+0000", 2024-09-13T13:13:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 513.72851116499999, 2024-09-13T13:13:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0016245429999999, 2024-09-13T13:13:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:13:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:13:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167004 2024-09-13T13:13:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789600+0000", 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789600+0000", 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789602+0000", 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2230000000000001e-06 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789608+0000", 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.37e-06 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789624+0000", 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6167e-05 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789632+0000", 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7270000000000007e-06 2024-09-13T13:13:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789705+0000", 2024-09-13T13:13:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2799999999999994e-05 2024-09-13T13:13:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.789718+0000", 2024-09-13T13:13:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2653999999999999e-05 2024-09-13T13:13:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.786269+0000", 2024-09-13T13:13:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99655117400000004 2024-09-13T13:13:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.786357+0000", 2024-09-13T13:13:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7865e-05 2024-09-13T13:13:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.786482+0000", 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000125451 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791065+0000", 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045828450000000003 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791120+0000", 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5562000000000001e-05 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791155+0000", 2024-09-13T13:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4326999999999998e-05 2024-09-13T13:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791188+0000", 2024-09-13T13:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2929000000000003e-05 2024-09-13T13:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791224+0000", 2024-09-13T13:13:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6449000000000003e-05 2024-09-13T13:13:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:167017 7.b 7:d2ad2293:::benchmark_data_smithi052_71865_object45916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:13:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:47.800317+0000", 2024-09-13T13:13:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 513.71779333300003, 2024-09-13T13:13:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99742454800000002, 2024-09-13T13:13:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 167017 2024-09-13T13:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800317+0000", 2024-09-13T13:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800317+0000", 2024-09-13T13:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800320+0000", 2024-09-13T13:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.148e-06 2024-09-13T13:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800326+0000", 2024-09-13T13:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2960000000000004e-06 2024-09-13T13:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800343+0000", 2024-09-13T13:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7079e-05 2024-09-13T13:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800351+0000", 2024-09-13T13:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1340000000000004e-06 2024-09-13T13:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800433+0000", 2024-09-13T13:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1509999999999997e-05 2024-09-13T13:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:47.800445+0000", 2024-09-13T13:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2414e-05 2024-09-13T13:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.788619+0000", 2024-09-13T13:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98817433300000002 2024-09-13T13:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.788727+0000", 2024-09-13T13:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108059 2024-09-13T13:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.788817+0000", 2024-09-13T13:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9117999999999996e-05 2024-09-13T13:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791266+0000", 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024498599999999999 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.791306+0000", 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9691000000000003e-05 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797479+0000", 2024-09-13T13:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0061724299999999996 2024-09-13T13:13:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797695+0000", 2024-09-13T13:13:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021655399999999999 2024-09-13T13:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:48.797742+0000", 2024-09-13T13:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6922000000000002e-05 2024-09-13T13:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189885 9.4 9:25913ce2:::benchmark_data_smithi052_128427_object59216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-09-13T13:13:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:13:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 38.608502719999997, 2024-09-13T13:13:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.048349118, 2024-09-13T13:13:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189885 2024-09-13T13:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909609+0000", 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0860000000000001e-06 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909614+0000", 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5199999999999999e-06 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909627+0000", 2024-09-13T13:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3623e-05 2024-09-13T13:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909634+0000", 2024-09-13T13:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0210000000000002e-06 2024-09-13T13:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909697+0000", 2024-09-13T13:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2769999999999997e-05 2024-09-13T13:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909840+0000", 2024-09-13T13:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014305399999999999 2024-09-13T13:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3,4", 2024-09-13T13:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909967+0000", 2024-09-13T13:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012707300000000001 2024-09-13T13:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.913962+0000", 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039947009999999998 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.913977+0000", 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.506e-05 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.916641+0000", 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026644839999999999 2024-09-13T13:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.937520+0000", 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.020878821999999998 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.937533+0000", 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2896e-05 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.957873+0000", 2024-09-13T13:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0203400570000001 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.957957+0000", 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3950999999999996e-05 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189915 9.4 9:2fa1035b:::benchmark_data_smithi052_128427_object59246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 38.575168896000001, 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0238765249999999, 2024-09-13T13:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189915 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:13:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942944+0000", 2024-09-13T13:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.655e-06 2024-09-13T13:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942948+0000", 2024-09-13T13:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3220000000000003e-06 2024-09-13T13:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942957+0000", 2024-09-13T13:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2609999999999999e-06 2024-09-13T13:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942964+0000", 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6379999999999998e-06 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.944838+0000", 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018746419999999999 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.944845+0000", 2024-09-13T13:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958435+0000", 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0135895370000001 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958549+0000", 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114144 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958677+0000", 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012798 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966592+0000", 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0079149440000000001 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966687+0000", 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4741999999999995e-05 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966725+0000", 2024-09-13T13:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7768000000000002e-05 2024-09-13T13:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966777+0000", 2024-09-13T13:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2605999999999997e-05 2024-09-13T13:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966818+0000", 2024-09-13T13:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1279999999999998e-05 2024-09-13T13:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189933 9.4 9:2eec938f:::benchmark_data_smithi052_128427_object59264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 38.560040297999997, 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.008927224, 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189933 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958072+0000", 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7e-06 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958078+0000", 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4990000000000002e-06 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958089+0000", 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0865e-05 2024-09-13T13:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958095+0000", 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8279999999999997e-06 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958183+0000", 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7655e-05 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958191+0000", 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9529999999999999e-06 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961131+0000", 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0029402620000001 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961259+0000", 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012816999999999999 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961395+0000", 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013549800000000001 2024-09-13T13:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966691+0000", 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0052965979999999996 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966864+0000", 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017277899999999999 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966899+0000", 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4377e-05 2024-09-13T13:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966955+0000", 2024-09-13T13:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6495e-05 2024-09-13T13:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966998+0000", 2024-09-13T13:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2545e-05 2024-09-13T13:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189960 9.4 9:2cd09825:::benchmark_data_smithi052_128427_object59291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 38.535994823999999, 2024-09-13T13:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99033151900000005, 2024-09-13T13:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189960 2024-09-13T13:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982118+0000", 2024-09-13T13:13:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982123+0000", 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8389999999999997e-06 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982137+0000", 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4433000000000001e-05 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982145+0000", 2024-09-13T13:13:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8290000000000008e-06 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982197+0000", 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2431e-05 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982205+0000", 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8630000000000008e-06 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964055+0000", 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98184965300000004 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964156+0000", 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101055 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964261+0000", 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104961 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972104+0000", 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0078433300000000008 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972147+0000", 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3022000000000002e-05 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972189+0000", 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1727000000000002e-05 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972384+0000", 2024-09-13T13:13:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019545200000000001 2024-09-13T13:13:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972447+0000", 2024-09-13T13:13:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3144000000000004e-05 2024-09-13T13:13:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189972 9.4 9:28374290:::benchmark_data_smithi052_128427_object59303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:13:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 38.527422657999999, 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99084523700000005, 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189972 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:13:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990690+0000", 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990694+0000", 2024-09-13T13:13:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6029999999999998e-06 2024-09-13T13:13:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990704+0000", 2024-09-13T13:13:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5060000000000001e-06 2024-09-13T13:13:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990710+0000", 2024-09-13T13:13:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:13:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990764+0000", 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4252000000000002e-05 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990773+0000", 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2379999999999997e-06 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967034+0000", 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97626137000000002 2024-09-13T13:13:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967153+0000", 2024-09-13T13:13:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118855 2024-09-13T13:13:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:13:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967267+0000", 2024-09-13T13:13:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113855 2024-09-13T13:13:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972462+0000", 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051953219999999996 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.976022+0000", 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003559563 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981419+0000", 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053977260000000003 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981488+0000", 2024-09-13T13:13:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8950999999999998e-05 2024-09-13T13:13:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981533+0000", 2024-09-13T13:13:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4981999999999997e-05 2024-09-13T13:13:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:21.584 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:21.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202468 9.4 9:2e7e6063:::benchmark_data_smithi052_128427_object71799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:13:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:13:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 23.552202340000001, 2024-09-13T13:13:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.030148343, 2024-09-13T13:13:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202468 2024-09-13T13:13:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:13:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:13:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965910+0000", 2024-09-13T13:13:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2810000000000001e-06 2024-09-13T13:13:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965915+0000", 2024-09-13T13:13:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0749999999999997e-06 2024-09-13T13:13:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965926+0000", 2024-09-13T13:13:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1056999999999999e-05 2024-09-13T13:13:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965943+0000", 2024-09-13T13:13:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7178000000000001e-05 2024-09-13T13:13:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.966053+0000", 2024-09-13T13:13:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109768 2024-09-13T13:13:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.966062+0000", 2024-09-13T13:13:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4029999999999996e-06 2024-09-13T13:13:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.971764+0000", 2024-09-13T13:13:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0057021909999999 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.971894+0000", 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000129987 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.972041+0000", 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000146398 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976575+0000", 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045341729999999998 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976634+0000", 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8996999999999998e-05 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.995927+0000", 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.019293457999999999 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.996003+0000", 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5221000000000004e-05 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.996057+0000", 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4156000000000001e-05 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202476 9.4 9:205ce74f:::benchmark_data_smithi052_128427_object71807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:13:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:13:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 23.545229627000001, 2024-09-13T13:13:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0253464960000001, 2024-09-13T13:13:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202476 2024-09-13T13:13:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972883+0000", 2024-09-13T13:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.792e-06 2024-09-13T13:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972888+0000", 2024-09-13T13:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7380000000000002e-06 2024-09-13T13:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972896+0000", 2024-09-13T13:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9230000000000002e-06 2024-09-13T13:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972903+0000", 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5310000000000004e-06 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.973002+0000", 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8484000000000001e-05 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.973012+0000", 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0016e-05 2024-09-13T13:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974404+0000", 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0013926820000001 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974530+0000", 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012599 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974651+0000", 2024-09-13T13:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012022400000000001 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976695+0000", 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020448139999999998 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981646+0000", 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049504229999999998 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998145+0000", 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.016499092 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998183+0000", 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8457999999999997e-05 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998228+0000", 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4329000000000002e-05 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202503 9.4 9:257a0b3a:::benchmark_data_smithi052_128427_object71834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:13:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:13:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 23.526181292, 2024-09-13T13:13:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.009335557, 2024-09-13T13:13:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202503 2024-09-13T13:13:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:13:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:13:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991936+0000", 2024-09-13T13:13:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1149999999999999e-06 2024-09-13T13:13:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991941+0000", 2024-09-13T13:13:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.463e-06 2024-09-13T13:13:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991954+0000", 2024-09-13T13:13:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2533e-05 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991961+0000", 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1690000000000003e-06 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.992017+0000", 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5856999999999997e-05 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.992026+0000", 2024-09-13T13:13:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:13:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976789+0000", 2024-09-13T13:13:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98476255999999995 2024-09-13T13:13:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976898+0000", 2024-09-13T13:13:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109548 2024-09-13T13:13:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.977048+0000", 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015011899999999999 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981632+0000", 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045837020000000003 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981701+0000", 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8511999999999997e-05 2024-09-13T13:13:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001159+0000", 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.019457953 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001221+0000", 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2229000000000003e-05 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001265+0000", 2024-09-13T13:13:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4267999999999997e-05 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202507 9.4 9:20f0d2e1:::benchmark_data_smithi052_128427_object71838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 23.524096149000002, 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.007409886, 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202507 2024-09-13T13:13:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:13:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:13:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994016+0000", 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.001e-06 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994020+0000", 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2320000000000003e-06 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994031+0000", 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1259e-05 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994036+0000", 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3580000000000002e-06 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994089+0000", 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2120000000000002e-05 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994097+0000", 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8000000000000004e-06 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979607+0000", 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98551001400000005 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979708+0000", 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000100108 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979813+0000", 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000105238 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981689+0000", 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018764299999999999 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987043+0000", 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053537139999999999 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001312+0000", 2024-09-13T13:13:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014268729000000001 2024-09-13T13:13:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001378+0000", 2024-09-13T13:13:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5980000000000002e-05 2024-09-13T13:13:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001424+0000", 2024-09-13T13:13:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6903000000000001e-05 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202513 9.4 9:293cc3ac:::benchmark_data_smithi052_128427_object71844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 23.518640207000001, 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0035790550000001, 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202513 2024-09-13T13:13:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:13:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:13:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999472+0000", 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.536e-06 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999477+0000", 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7260000000000002e-06 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999492+0000", 2024-09-13T13:13:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5495999999999999e-05 2024-09-13T13:13:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999500+0000", 2024-09-13T13:13:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3100000000000003e-06 2024-09-13T13:13:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999557+0000", 2024-09-13T13:13:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7491999999999998e-05 2024-09-13T13:13:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999565+0000", 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9359999999999999e-06 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981724+0000", 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98215919600000001 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981838+0000", 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114092 2024-09-13T13:13:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981966+0000", 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012748300000000001 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987109+0000", 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.005142771 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987186+0000", 2024-09-13T13:13:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7822000000000001e-05 2024-09-13T13:13:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.002955+0000", 2024-09-13T13:13:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015768880999999998 2024-09-13T13:13:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003010+0000", 2024-09-13T13:13:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4959e-05 2024-09-13T13:13:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003050+0000", 2024-09-13T13:13:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9354999999999998e-05 2024-09-13T13:13:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:227739 9.4 9:2c4e28d0:::benchmark_data_smithi052_128427_object97070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-13T13:13:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:20.619255+0000", 2024-09-13T13:13:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.89885556899999997, 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.040545204000000001, 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 227739 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.619255+0000", 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.619255+0000", 2024-09-13T13:13:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.619256+0000", 2024-09-13T13:13:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0399999999999996e-07 2024-09-13T13:13:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.619258+0000", 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4260000000000002e-06 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.619263+0000", 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0030000000000002e-06 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.619268+0000", 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7369999999999997e-06 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.625111+0000", 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058427119999999999 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.625212+0000", 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101535 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.625316+0000", 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103879 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.634244+0000", 2024-09-13T13:13:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0089280639999999994 2024-09-13T13:13:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.634365+0000", 2024-09-13T13:13:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121409 2024-09-13T13:13:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.659582+0000", 2024-09-13T13:13:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.025216125999999998 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.659758+0000", 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017611800000000001 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.659800+0000", 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2691000000000002e-05 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:227751 9.4 9:2450e7e8:::benchmark_data_smithi052_128427_object97082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:20.635182+0000", 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.88292911299999999, 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.062909829, 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 227751 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.635182+0000", 2024-09-13T13:13:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.635182+0000", 2024-09-13T13:13:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.635183+0000", 2024-09-13T13:13:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7179999999999999e-06 2024-09-13T13:13:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.635189+0000", 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9059999999999999e-06 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.635202+0000", 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2575000000000001e-05 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.635210+0000", 2024-09-13T13:13:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8849999999999999e-06 2024-09-13T13:13:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.635267+0000", 2024-09-13T13:13:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7550000000000003e-05 2024-09-13T13:13:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.635394+0000", 2024-09-13T13:13:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000126515 2024-09-13T13:13:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:13:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.635536+0000", 2024-09-13T13:13:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000141989 2024-09-13T13:13:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.638784+0000", 2024-09-13T13:13:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003247996 2024-09-13T13:13:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.642764+0000", 2024-09-13T13:13:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039797169999999998 2024-09-13T13:13:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.696858+0000", 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.054094353999999997 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.698039+0000", 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00118074 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.698091+0000", 2024-09-13T13:13:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2883999999999997e-05 2024-09-13T13:13:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:227753 9.4 9:200d3879:::benchmark_data_smithi052_128427_object97084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-13T13:13:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:20.637427+0000", 2024-09-13T13:13:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.88068397899999995, 2024-09-13T13:13:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.060893287999999997, 2024-09-13T13:13:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 227753 2024-09-13T13:13:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.637427+0000", 2024-09-13T13:13:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.637427+0000", 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.637430+0000", 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5619999999999999e-06 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.637433+0000", 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8679999999999999e-06 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.637439+0000", 2024-09-13T13:13:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5500000000000002e-06 2024-09-13T13:13:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.637444+0000", 2024-09-13T13:13:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9300000000000002e-06 2024-09-13T13:13:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.638819+0000", 2024-09-13T13:13:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001374835 2024-09-13T13:13:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.638948+0000", 2024-09-13T13:13:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012909900000000001 2024-09-13T13:13:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:13:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.639130+0000", 2024-09-13T13:13:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018264299999999999 2024-09-13T13:13:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.642804+0000", 2024-09-13T13:13:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036738729999999998 2024-09-13T13:13:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.652135+0000", 2024-09-13T13:13:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0093303789999999998 2024-09-13T13:13:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.698214+0000", 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.046079506999999999 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.698269+0000", 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4534999999999999e-05 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.698320+0000", 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1507e-05 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:227754 9.4 9:232dff54:::benchmark_data_smithi052_128427_object97085:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:20.637450+0000", 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.88066076599999998, 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.061045602999999997, 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 227754 2024-09-13T13:13:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.637450+0000", 2024-09-13T13:13:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.637450+0000", 2024-09-13T13:13:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.637450+0000", 2024-09-13T13:13:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2100000000000002e-07 2024-09-13T13:13:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.637453+0000", 2024-09-13T13:13:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2989999999999999e-06 2024-09-13T13:13:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.637457+0000", 2024-09-13T13:13:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5979999999999999e-06 2024-09-13T13:13:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.637462+0000", 2024-09-13T13:13:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4240000000000003e-06 2024-09-13T13:13:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.642841+0000", 2024-09-13T13:13:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053790260000000003 2024-09-13T13:13:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.648823+0000", 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0059821869999999999 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.648941+0000", 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117816 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.652122+0000", 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003181054 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.652140+0000", 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8393000000000002e-05 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.698399+0000", 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.046259116000000003 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.698462+0000", 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2799999999999995e-05 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.698496+0000", 2024-09-13T13:13:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3469000000000001e-05 2024-09-13T13:13:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:227848 9.4 9:2ca98001:::benchmark_data_smithi052_128427_object97179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-13T13:13:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:20.739536+0000", 2024-09-13T13:13:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.77857459100000004, 2024-09-13T13:13:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.039024615999999998, 2024-09-13T13:13:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 227848 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.739536+0000", 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.739536+0000", 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.739537+0000", 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.048e-06 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.739542+0000", 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4290000000000003e-06 2024-09-13T13:13:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.739550+0000", 2024-09-13T13:13:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9140000000000008e-06 2024-09-13T13:13:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.739556+0000", 2024-09-13T13:13:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6760000000000001e-06 2024-09-13T13:13:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.747122+0000", 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.007565904 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.747272+0000", 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015019100000000001 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.747411+0000", 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013894600000000001 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.753380+0000", 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00596876 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.753398+0000", 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8454e-05 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.778133+0000", 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.024734717999999999 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.778405+0000", 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027160999999999998 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.778561+0000", 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000155966 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:228030 9.a e318/241 9:5069f930:::benchmark_data_smithi052_128427_object97361:head v 318'14299, mlcod=318'14297)", 2024-09-13T13:13:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:20.940336+0000", 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.57777474600000001, 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.038751238, 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.940336+0000", 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.940336+0000", 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.940337+0000", 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.08e-06 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.940344+0000", 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2030000000000003e-06 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.940353+0000", 2024-09-13T13:13:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9169999999999995e-06 2024-09-13T13:13:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.940359+0000", 2024-09-13T13:13:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8300000000000001e-06 2024-09-13T13:13:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.942350+0000", 2024-09-13T13:13:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019907739999999998 2024-09-13T13:13:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.942371+0000", 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1143000000000001e-05 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.978996+0000", 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.036625425000000003 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.979087+0000", 2024-09-13T13:13:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0865999999999994e-05 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:228031 9.a e318/241 9:5f56dc07:::benchmark_data_smithi052_128427_object97362:head v 318'14300, mlcod=318'14297)", 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:20.942042+0000", 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.57606875999999996, 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.037160453000000003, 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.942042+0000", 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.942042+0000", 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.942044+0000", 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.528e-06 2024-09-13T13:13:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.942051+0000", 2024-09-13T13:13:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4900000000000003e-06 2024-09-13T13:13:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.942062+0000", 2024-09-13T13:13:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1094000000000001e-05 2024-09-13T13:13:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.942069+0000", 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.945450+0000", 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003380639 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.945468+0000", 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7941999999999999e-05 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.979166+0000", 2024-09-13T13:13:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.033698288999999999 2024-09-13T13:13:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:20.979202+0000", 2024-09-13T13:13:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6309000000000001e-05 2024-09-13T13:13:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:21.663 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:13:21.685 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:21.722 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:21.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:21.823 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:21.861 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:21.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:21.946 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:13:21.946 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T13:13:21.964 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:22.048 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:22.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:22.150 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:22.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:22.190 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:22.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:22.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[36682]: 5.2 scrub starts 2024-09-13T13:13:22.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[36682]: 5.2 scrub ok 2024-09-13T13:13:22.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[36682]: 5.1d scrub starts 2024-09-13T13:13:22.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[36682]: 5.1d scrub ok 2024-09-13T13:13:22.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[36682]: 9.2 scrub ok 2024-09-13T13:13:22.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:13:22.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[36682]: from='client.35975 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:13:22.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[36682]: osdmap e319: 8 total, 8 up, 8 in 2024-09-13T13:13:22.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[29471]: 5.2 scrub starts 2024-09-13T13:13:22.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[29471]: 5.2 scrub ok 2024-09-13T13:13:22.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[29471]: 5.1d scrub starts 2024-09-13T13:13:22.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[29471]: 5.1d scrub ok 2024-09-13T13:13:22.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[29471]: 9.2 scrub ok 2024-09-13T13:13:22.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:13:22.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[29471]: from='client.35975 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:13:22.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:21 smithi052 ceph-mon[29471]: osdmap e319: 8 total, 8 up, 8 in 2024-09-13T13:13:22.292 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:21 smithi167 ceph-mon[33178]: 5.2 scrub starts 2024-09-13T13:13:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:21 smithi167 ceph-mon[33178]: 5.2 scrub ok 2024-09-13T13:13:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:21 smithi167 ceph-mon[33178]: 5.1d scrub starts 2024-09-13T13:13:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:21 smithi167 ceph-mon[33178]: 5.1d scrub ok 2024-09-13T13:13:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:21 smithi167 ceph-mon[33178]: 9.2 scrub ok 2024-09-13T13:13:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:21 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:13:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:21 smithi167 ceph-mon[33178]: from='client.35975 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:13:22.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:21 smithi167 ceph-mon[33178]: osdmap e319: 8 total, 8 up, 8 in 2024-09-13T13:13:22.324 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:22.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:22.427 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:22.457 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:22.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:22.559 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:22.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:22.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:22.690 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:22.731 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:22.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:22.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:22.833 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:22.870 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T13:13:22.898 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:22.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:23.001 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:23.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:23.033 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:23.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:23.134 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:23.179 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:23.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:23.243 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:22 smithi052 ceph-mon[29471]: pgmap v605: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 713 B/s rd, 0 B/s wr, 801 op/s 2024-09-13T13:13:23.243 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:22 smithi052 ceph-mon[29471]: osdmap e320: 8 total, 8 up, 8 in 2024-09-13T13:13:23.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:22 smithi052 ceph-mon[36682]: pgmap v605: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 713 B/s rd, 0 B/s wr, 801 op/s 2024-09-13T13:13:23.243 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:22 smithi052 ceph-mon[36682]: osdmap e320: 8 total, 8 up, 8 in 2024-09-13T13:13:23.280 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:23.308 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:23.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:23.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:22 smithi167 ceph-mon[33178]: pgmap v605: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 713 B/s rd, 0 B/s wr, 801 op/s 2024-09-13T13:13:23.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:22 smithi167 ceph-mon[33178]: osdmap e320: 8 total, 8 up, 8 in 2024-09-13T13:13:23.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:23.423 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:23.472 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:23.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:23.485 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:23.493 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:23.575 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:23.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:23.621 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:23.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:23.722 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:23.747 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:23.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:23.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:23.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:23.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:23.883 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:23.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:23.985 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:24.023 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:24.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:24.126 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:24.153 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:24.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:24.255 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:24.289 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:24.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:24.391 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:24.421 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:24.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:24.522 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:24.548 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:24.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:24.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:24.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:13:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:13:24.650 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:24.684 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:24.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:24.715 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:13:24.716 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:13:24.716 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 0 2024-09-13T13:13:24.716 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:13:24.785 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:24.823 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:24.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:24.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:24.925 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:24.964 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:24.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:24.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:24.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:24 smithi052 ceph-mon[29471]: pgmap v607: 148 pgs: 148 active+clean; 2.1 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-13T13:13:24.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:24 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:24.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:24 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/878414103' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:13:24.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:24 smithi052 ceph-mon[29471]: from='client.25957 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:13:25.006 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T13:13:25.066 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:25.101 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:25.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:25.203 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:25.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:25.242 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:25.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:25.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:24 smithi167 ceph-mon[33178]: pgmap v607: 148 pgs: 148 active+clean; 2.1 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-13T13:13:25.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:24 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:25.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:24 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/878414103' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:13:25.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:24 smithi167 ceph-mon[33178]: from='client.25957 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:13:25.343 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:25.378 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:13:25.379 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:25.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:25.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:25.414 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:24 smithi052 ceph-mon[36682]: pgmap v607: 148 pgs: 148 active+clean; 2.1 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-13T13:13:25.414 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:24 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:25.414 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:24 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/878414103' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:13:25.414 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:24 smithi052 ceph-mon[36682]: from='client.25957 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:13:25.481 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:25.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:25.542 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:25.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:25.593 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T13:13:25.643 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:25.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:25.699 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:25.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:25.800 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:25.847 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:25.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:25.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:25.949 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:25.976 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:25.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:25.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:26.078 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:26.100 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:26 smithi052 ceph-mon[36682]: from='client.25957 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:13:26.100 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:26 smithi052 ceph-mon[36682]: osdmap e321: 8 total, 8 up, 8 in 2024-09-13T13:13:26.100 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:26 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3431527167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:26.100 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:25 smithi052 ceph-mon[29471]: from='client.25957 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:13:26.100 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:26 smithi052 ceph-mon[29471]: osdmap e321: 8 total, 8 up, 8 in 2024-09-13T13:13:26.101 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:26 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3431527167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:26.110 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:26.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:26.211 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:26.248 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:26.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:26.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:26.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:25 smithi167 ceph-mon[33178]: from='client.25957 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:13:26.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:25 smithi167 ceph-mon[33178]: osdmap e321: 8 total, 8 up, 8 in 2024-09-13T13:13:26.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:25 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3431527167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:26.350 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:26.389 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:26.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:26.460 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:13:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:13:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:13:26.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:26.514 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T13:13:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:26.516 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:26.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:26.593 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:13:26.593 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:13:26.594 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:13:26.594 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:13:26.594 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:13:26.594 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:13:26.594 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:13:26.594 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:13:26.594 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:13:26.594 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:13:26.594 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:13:26.594 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:13:26.594 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:13:26.594 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:13:26.595 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:13:26.595 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:13:26.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:26.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:26.660 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:26.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:26.761 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:26.813 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:26.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:26.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:26.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:26.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:27.054 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:27.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:27.098 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:27.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:27.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:27.232 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:27.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:27.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:27.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:27 smithi052 ceph-mon[36682]: pgmap v609: 148 pgs: 148 active+clean; 1.8 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 988 op/s 2024-09-13T13:13:27.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:27 smithi052 ceph-mon[29471]: pgmap v609: 148 pgs: 148 active+clean; 1.8 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 988 op/s 2024-09-13T13:13:27.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:27 smithi167 ceph-mon[33178]: pgmap v609: 148 pgs: 148 active+clean; 1.8 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 988 op/s 2024-09-13T13:13:27.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:27.349 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:27.361 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:27.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:27.364 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:27.419 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:27.463 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:27.493 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:27.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:27.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:27.595 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:27.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:27.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:27.731 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:27.766 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:27.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:27.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:27.868 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:27.905 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:27.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:27.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:28.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:28.032 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:28.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:28.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:28.134 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:28.171 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:28.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:28.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:28.273 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:28.313 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:28.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:28.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:28.456 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:28.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:28.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:28.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:28.591 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:28.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:28.693 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:28.725 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:28.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:28.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:28.854 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:28.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:28.952 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:28 smithi052 ceph-mon[29471]: pgmap v610: 148 pgs: 148 active+clean; 1.8 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 858 op/s 2024-09-13T13:13:28.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:28.981 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:28.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:28.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:29.083 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:29.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:29.124 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:29.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:29.225 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:29.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:28 smithi052 ceph-mon[29471]: osdmap e322: 8 total, 8 up, 8 in 2024-09-13T13:13:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:28 smithi052 ceph-mon[36682]: pgmap v610: 148 pgs: 148 active+clean; 1.8 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 858 op/s 2024-09-13T13:13:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:28 smithi052 ceph-mon[36682]: osdmap e322: 8 total, 8 up, 8 in 2024-09-13T13:13:29.257 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:29.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:29.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:29.303 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:13:29.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:28 smithi167 ceph-mon[33178]: pgmap v610: 148 pgs: 148 active+clean; 1.8 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 858 op/s 2024-09-13T13:13:29.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:28 smithi167 ceph-mon[33178]: osdmap e322: 8 total, 8 up, 8 in 2024-09-13T13:13:29.351 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:13:29.352 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:13:29.359 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:29.403 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:29.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:29.504 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:29.554 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:29.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:29.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:29.656 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:29.710 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:29.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:29.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:29.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:29.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:29.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:29.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:29.966 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T13:13:29.974 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:13:29.982 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:29.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:30.084 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:30.119 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:30.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:30.221 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:30.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:29 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4061262802' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:30.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:29 smithi052 ceph-mon[29471]: from='client.35990 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:30.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:29 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2856008868' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:13:30.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:29 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4061262802' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:30.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:29 smithi052 ceph-mon[36682]: from='client.35990 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:30.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:29 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2856008868' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:13:30.259 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:30.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:30.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:30.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:29 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4061262802' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:30.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:29 smithi167 ceph-mon[33178]: from='client.35990 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:30.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:29 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2856008868' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:13:30.361 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:30.414 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T13:13:30.425 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:30.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:30.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:30.527 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:30.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:30.576 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:30.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:30.677 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:30.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:30.710 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:30.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:30.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:30.846 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:30.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:30.890 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T13:13:30.948 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:30.990 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:30.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:31.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:31.120 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:31.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:31.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:31.221 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:31.222 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:30 smithi167 ceph-mon[33178]: pgmap v612: 148 pgs: 148 active+clean; 1.4 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.74k op/s 2024-09-13T13:13:31.222 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:30 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:13:31.222 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:30 smithi167 ceph-mon[33178]: from='client.35990 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:13:31.222 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:30 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2856008868' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:13:31.222 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:30 smithi167 ceph-mon[33178]: osdmap e323: 8 total, 8 up, 8 in 2024-09-13T13:13:31.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:30 smithi052 ceph-mon[29471]: pgmap v612: 148 pgs: 148 active+clean; 1.4 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.74k op/s 2024-09-13T13:13:31.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:30 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:13:31.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:30 smithi052 ceph-mon[29471]: from='client.35990 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:13:31.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:30 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2856008868' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:13:31.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:30 smithi052 ceph-mon[29471]: osdmap e323: 8 total, 8 up, 8 in 2024-09-13T13:13:31.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:30 smithi052 ceph-mon[36682]: pgmap v612: 148 pgs: 148 active+clean; 1.4 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.74k op/s 2024-09-13T13:13:31.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:30 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:13:31.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:30 smithi052 ceph-mon[36682]: from='client.35990 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:13:31.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:30 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2856008868' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:13:31.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:30 smithi052 ceph-mon[36682]: osdmap e323: 8 total, 8 up, 8 in 2024-09-13T13:13:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:31.269 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:31.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:31.370 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:31.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:31.410 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:31.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:31.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:31.537 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:31.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:31.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:31.639 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:31.670 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:31.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:31.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:31.772 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:31.808 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:31.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:31.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:31.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:31.948 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:31.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:32.050 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:32.088 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:32.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:32.191 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:32.212 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:32.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:32.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:31 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:13:32.257 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:31 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:13:32.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:31 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:13:32.257 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:31 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:13:32.258 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:31 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:13:32.258 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:31 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:13:32.314 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:32.349 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:32.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:32.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:32.451 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:32.474 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:32.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:32.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:32.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:32.610 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:32.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:32.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:32.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:32.754 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:32.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:32.855 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:32.903 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:32.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:32.970 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:32.979 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:33.006 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:33.025 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:33.062 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:33.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:33.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:33.139 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:32 smithi052 ceph-mon[29471]: pgmap v614: 148 pgs: 148 active+clean; 1.4 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 891 op/s 2024-09-13T13:13:33.140 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:32 smithi052 ceph-mon[29471]: osdmap e324: 8 total, 8 up, 8 in 2024-09-13T13:13:33.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:32 smithi052 ceph-mon[36682]: pgmap v614: 148 pgs: 148 active+clean; 1.4 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 891 op/s 2024-09-13T13:13:33.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:32 smithi052 ceph-mon[36682]: osdmap e324: 8 total, 8 up, 8 in 2024-09-13T13:13:33.164 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:33.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:33.200 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:33.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:33.301 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:33.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:32 smithi167 ceph-mon[33178]: pgmap v614: 148 pgs: 148 active+clean; 1.4 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 891 op/s 2024-09-13T13:13:33.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:32 smithi167 ceph-mon[33178]: osdmap e324: 8 total, 8 up, 8 in 2024-09-13T13:13:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:33.341 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:33.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:33.443 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:33.471 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:33.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:33.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:33.606 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:33.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:33.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:33.708 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:33.733 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:33.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:33.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:33.835 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:33.869 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:33.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:33.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:33.948 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:13:33.948 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:13:33.971 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:33.993 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:13:33.993 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:13:34.016 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:34.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:34.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:34.026 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:13:34.027 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:13:34.061 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:13:34.061 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:13:34.100 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:13:34.101 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:13:34.118 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:34.132 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:13:34.132 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:13:34.159 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:34.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:34.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:34.168 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:13:34.168 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:13:34.185 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:13:34.186 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:13:34.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:13:34.186 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:13:34.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.186 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:118717 7.e 7:7eac196e:::benchmark_data_smithi052_71865_object118716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:13:34.186 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:13:34.186 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 584.62286559400002, 2024-09-13T13:13:34.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.120995147, 2024-09-13T13:13:34.186 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.186 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.187 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.187 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:34.187 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:13:34.187 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:13:34.187 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 118717 2024-09-13T13:13:34.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.187 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:13:34.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560219+0000", 2024-09-13T13:13:34.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560220+0000", 2024-09-13T13:13:34.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4200000000000004e-07 2024-09-13T13:13:34.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560244+0000", 2024-09-13T13:13:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4354000000000001e-05 2024-09-13T13:13:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560252+0000", 2024-09-13T13:13:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5189999999999992e-06 2024-09-13T13:13:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560259+0000", 2024-09-13T13:13:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.302e-06 2024-09-13T13:13:34.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560301+0000", 2024-09-13T13:13:34.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2318999999999999e-05 2024-09-13T13:13:34.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560415+0000", 2024-09-13T13:13:34.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114269 2024-09-13T13:13:34.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.560572+0000", 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000156217 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.564231+0000", 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036597489999999999 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.565180+0000", 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00094882300000000001 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.680844+0000", 2024-09-13T13:13:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.115664164 2024-09-13T13:13:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.681157+0000", 2024-09-13T13:13:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00031297600000000002 2024-09-13T13:13:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:49.681214+0000", 2024-09-13T13:13:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6512999999999999e-05 2024-09-13T13:13:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120031 7.e 7:704b0327:::benchmark_data_smithi052_71865_object120030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:13:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:13:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 583.16227834400001, 2024-09-13T13:13:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11824775799999999, 2024-09-13T13:13:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:13:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:13:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120031 2024-09-13T13:13:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:13:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020806+0000", 2024-09-13T13:13:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020808+0000", 2024-09-13T13:13:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0439999999999998e-06 2024-09-13T13:13:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020838+0000", 2024-09-13T13:13:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9791e-05 2024-09-13T13:13:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020856+0000", 2024-09-13T13:13:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8664999999999998e-05 2024-09-13T13:13:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.020866+0000", 2024-09-13T13:13:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3670000000000003e-06 2024-09-13T13:13:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021203+0000", 2024-09-13T13:13:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033735200000000002 2024-09-13T13:13:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.021578+0000", 2024-09-13T13:13:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00037465399999999998 2024-09-13T13:13:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.022015+0000", 2024-09-13T13:13:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000437176 2024-09-13T13:13:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.026523+0000", 2024-09-13T13:13:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045084249999999999 2024-09-13T13:13:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.027199+0000", 2024-09-13T13:13:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00067598400000000002 2024-09-13T13:13:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.138915+0000", 2024-09-13T13:13:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11171563499999999 2024-09-13T13:13:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139008+0000", 2024-09-13T13:13:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3329e-05 2024-09-13T13:13:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.139054+0000", 2024-09-13T13:13:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5336000000000001e-05 2024-09-13T13:13:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120050 7.e 7:72296fb6:::benchmark_data_smithi052_71865_object120049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:13:34.201 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:13:34.201 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 583.15023029899999, 2024-09-13T13:13:34.201 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12504636999999999, 2024-09-13T13:13:34.201 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.201 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.201 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:13:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:13:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120050 2024-09-13T13:13:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:13:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032854+0000", 2024-09-13T13:13:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032857+0000", 2024-09-13T13:13:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1140000000000002e-06 2024-09-13T13:13:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032888+0000", 2024-09-13T13:13:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1115999999999998e-05 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032905+0000", 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6452000000000001e-05 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.032915+0000", 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8889999999999997e-06 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033665+0000", 2024-09-13T13:13:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00075015300000000004 2024-09-13T13:13:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033782+0000", 2024-09-13T13:13:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117642 2024-09-13T13:13:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.033916+0000", 2024-09-13T13:13:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013400799999999999 2024-09-13T13:13:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.037705+0000", 2024-09-13T13:13:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037884289999999998 2024-09-13T13:13:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.037719+0000", 2024-09-13T13:13:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4034e-05 2024-09-13T13:13:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157684+0000", 2024-09-13T13:13:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.119964979 2024-09-13T13:13:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157805+0000", 2024-09-13T13:13:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120715 2024-09-13T13:13:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.157900+0000", 2024-09-13T13:13:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5839000000000002e-05 2024-09-13T13:13:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:120054 7.e 7:7dad863d:::benchmark_data_smithi052_71865_object120053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-13T13:13:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:13:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 583.144715023, 2024-09-13T13:13:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11988120300000001, 2024-09-13T13:13:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.209 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.209 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:34.209 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:13:34.209 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:13:34.209 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 120054 2024-09-13T13:13:34.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.209 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.209 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:13:34.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038369+0000", 2024-09-13T13:13:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038370+0000", 2024-09-13T13:13:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.051e-06 2024-09-13T13:13:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038395+0000", 2024-09-13T13:13:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4224999999999999e-05 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038403+0000", 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3399999999999998e-06 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038409+0000", 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3160000000000002e-06 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038462+0000", 2024-09-13T13:13:34.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3012e-05 2024-09-13T13:13:34.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038579+0000", 2024-09-13T13:13:34.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116483 2024-09-13T13:13:34.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:34.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.038720+0000", 2024-09-13T13:13:34.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014139099999999999 2024-09-13T13:13:34.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.042075+0000", 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033552830000000001 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.042095+0000", 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9279999999999998e-05 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158031+0000", 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11593616299999999 2024-09-13T13:13:34.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.214 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.214 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158169+0000", 2024-09-13T13:13:34.214 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013835799999999999 2024-09-13T13:13:34.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.214 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.214 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:03:51.158250+0000", 2024-09-13T13:13:34.214 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1301e-05 2024-09-13T13:13:34.214 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.214 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.214 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.215 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.34781.0:153862 7.e 7:797e5c7a:::benchmark_data_smithi052_71865_object32761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T13:13:34.215 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:13:34.215 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 541.91827755199995, 2024-09-13T13:13:34.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.106314172, 2024-09-13T13:13:34.215 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.215 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.215 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.215 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:34.215 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.34781", 2024-09-13T13:13:34.215 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:13:34.215 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 153862 2024-09-13T13:13:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:13:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264807+0000", 2024-09-13T13:13:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264809+0000", 2024-09-13T13:13:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.148e-06 2024-09-13T13:13:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264815+0000", 2024-09-13T13:13:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7610000000000004e-06 2024-09-13T13:13:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264826+0000", 2024-09-13T13:13:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1215e-05 2024-09-13T13:13:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264833+0000", 2024-09-13T13:13:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0090000000000001e-06 2024-09-13T13:13:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.264885+0000", 2024-09-13T13:13:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1845000000000003e-05 2024-09-13T13:13:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.265001+0000", 2024-09-13T13:13:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116795 2024-09-13T13:13:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.265104+0000", 2024-09-13T13:13:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102979 2024-09-13T13:13:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.268877+0000", 2024-09-13T13:13:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037720779999999999 2024-09-13T13:13:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.269086+0000", 2024-09-13T13:13:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000209829 2024-09-13T13:13:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.370873+0000", 2024-09-13T13:13:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10178622700000001 2024-09-13T13:13:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.371063+0000", 2024-09-13T13:13:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000190349 2024-09-13T13:13:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:04:32.371121+0000", 2024-09-13T13:13:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7936999999999998e-05 2024-09-13T13:13:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112327 9.2 e275/241 9:464b334b:::benchmark_data_smithi052_128427_object112326:head v 275'7019, mlcod=275'7017)", 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.572648+0000", 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 124.61043632400001, 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.103891961, 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572648+0000", 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572648+0000", 2024-09-13T13:13:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572650+0000", 2024-09-13T13:13:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.717e-06 2024-09-13T13:13:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572678+0000", 2024-09-13T13:13:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7985999999999999e-05 2024-09-13T13:13:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572690+0000", 2024-09-13T13:13:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2262999999999999e-05 2024-09-13T13:13:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572699+0000", 2024-09-13T13:13:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9600000000000006e-06 2024-09-13T13:13:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572782+0000", 2024-09-13T13:13:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3338999999999996e-05 2024-09-13T13:13:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572804+0000", 2024-09-13T13:13:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1314999999999998e-05 2024-09-13T13:13:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676479+0000", 2024-09-13T13:13:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.103675184 2024-09-13T13:13:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676540+0000", 2024-09-13T13:13:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1197000000000004e-05 2024-09-13T13:13:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112328 9.2 e275/241 9:45698ec9:::benchmark_data_smithi052_128427_object112327:head v 275'7020, mlcod=275'7017)", 2024-09-13T13:13:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.576519+0000", 2024-09-13T13:13:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 124.606565199, 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10005291099999999, 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576519+0000", 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576519+0000", 2024-09-13T13:13:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576521+0000", 2024-09-13T13:13:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6080000000000001e-06 2024-09-13T13:13:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576544+0000", 2024-09-13T13:13:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3791999999999999e-05 2024-09-13T13:13:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576556+0000", 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1671000000000001e-05 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576564+0000", 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8290000000000008e-06 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.578118+0000", 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0015537820000000001 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.578138+0000", 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0152999999999998e-05 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676545+0000", 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098407005000000006 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676572+0000", 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7070999999999999e-05 2024-09-13T13:13:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.231 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.231 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.231 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.231 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112330 9.2 e275/241 9:4591f68b:::benchmark_data_smithi052_128427_object112329:head v 275'7021, mlcod=275'7017)", 2024-09-13T13:13:34.231 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.579400+0000", 2024-09-13T13:13:34.231 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 124.60368397800001, 2024-09-13T13:13:34.231 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10414786299999999, 2024-09-13T13:13:34.231 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.231 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.232 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.232 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.232 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.232 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579400+0000", 2024-09-13T13:13:34.232 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.232 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579400+0000", 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579402+0000", 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7e-06 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579446+0000", 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.439e-05 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.233 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.234 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.234 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579458+0000", 2024-09-13T13:13:34.234 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1477999999999999e-05 2024-09-13T13:13:34.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.234 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.234 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579465+0000", 2024-09-13T13:13:34.234 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.114e-06 2024-09-13T13:13:34.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.234 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.234 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676607+0000", 2024-09-13T13:13:34.234 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.097141839999999993 2024-09-13T13:13:34.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.235 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.235 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676630+0000", 2024-09-13T13:13:34.235 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3332e-05 2024-09-13T13:13:34.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.235 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.235 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.683496+0000", 2024-09-13T13:13:34.235 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0068658679999999998 2024-09-13T13:13:34.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.235 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.236 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.683548+0000", 2024-09-13T13:13:34.236 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2141000000000001e-05 2024-09-13T13:13:34.236 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.236 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.236 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.236 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.236 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112336 9.0 e275/241 9:041ef402:::benchmark_data_smithi052_128427_object112335:head v 275'7019, mlcod=275'7017)", 2024-09-13T13:13:34.237 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.580484+0000", 2024-09-13T13:13:34.237 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 124.602600263, 2024-09-13T13:13:34.237 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10093202900000001, 2024-09-13T13:13:34.237 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.237 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.237 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.237 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.237 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.237 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.237 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580484+0000", 2024-09-13T13:13:34.237 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580484+0000", 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580486+0000", 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.53e-06 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.238 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580530+0000", 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4589999999999998e-05 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580541+0000", 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0779e-05 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580548+0000", 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3019999999999997e-06 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.239 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.240 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580604+0000", 2024-09-13T13:13:34.240 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6271999999999999e-05 2024-09-13T13:13:34.240 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.240 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.240 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.240 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676458+0000", 2024-09-13T13:13:34.240 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095853820000000006 2024-09-13T13:13:34.240 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.240 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.240 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.240 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.681299+0000", 2024-09-13T13:13:34.241 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048409280000000004 2024-09-13T13:13:34.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.241 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.241 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.241 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.681416+0000", 2024-09-13T13:13:34.241 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116808 2024-09-13T13:13:34.241 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.241 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.241 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.241 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.241 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112346 9.6 e275/241 9:67a240c7:::benchmark_data_smithi052_128427_object112345:head v 275'7035, mlcod=275'7032)", 2024-09-13T13:13:34.241 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.587147+0000", 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 124.59593771199999, 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.097407753, 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587147+0000", 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.242 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587147+0000", 2024-09-13T13:13:34.243 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.243 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.243 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587149+0000", 2024-09-13T13:13:34.243 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.075e-06 2024-09-13T13:13:34.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.243 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.243 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587225+0000", 2024-09-13T13:13:34.243 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5964000000000007e-05 2024-09-13T13:13:34.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.244 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.244 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.677080+0000", 2024-09-13T13:13:34.244 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.089855655000000006 2024-09-13T13:13:34.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.244 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.244 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.244 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.677087+0000", 2024-09-13T13:13:34.244 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0199999999999997e-06 2024-09-13T13:13:34.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.244 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.245 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.245 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.677138+0000", 2024-09-13T13:13:34.245 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0773000000000003e-05 2024-09-13T13:13:34.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.245 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.245 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.245 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.677157+0000", 2024-09-13T13:13:34.245 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8910999999999999e-05 2024-09-13T13:13:34.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.245 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.245 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.245 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.684462+0000", 2024-09-13T13:13:34.245 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0073047269999999996 2024-09-13T13:13:34.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.246 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.684554+0000", 2024-09-13T13:13:34.246 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2627999999999995e-05 2024-09-13T13:13:34.246 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.246 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.246 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.246 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112348 9.2 e275/241 9:414aa897:::benchmark_data_smithi052_128427_object112347:head v 275'7022, mlcod=275'7017)", 2024-09-13T13:13:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.587656+0000", 2024-09-13T13:13:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 124.59542856100001, 2024-09-13T13:13:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098073530000000006, 2024-09-13T13:13:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587656+0000", 2024-09-13T13:13:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.248 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.248 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:13:34.248 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:13:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587656+0000", 2024-09-13T13:13:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587657+0000", 2024-09-13T13:13:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.658e-06 2024-09-13T13:13:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587683+0000", 2024-09-13T13:13:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5253999999999999e-05 2024-09-13T13:13:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676615+0000", 2024-09-13T13:13:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.088932251000000004 2024-09-13T13:13:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676624+0000", 2024-09-13T13:13:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4150000000000005e-06 2024-09-13T13:13:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.679045+0000", 2024-09-13T13:13:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024209600000000002 2024-09-13T13:13:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.679070+0000", 2024-09-13T13:13:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4967e-05 2024-09-13T13:13:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.685690+0000", 2024-09-13T13:13:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0066195150000000003 2024-09-13T13:13:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.685729+0000", 2024-09-13T13:13:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9509999999999999e-05 2024-09-13T13:13:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:161588 9.6 e289/241 9:6d59046d:::benchmark_data_smithi052_128427_object30919:head v 289'10101, mlcod=289'10099)", 2024-09-13T13:13:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.716022+0000", 2024-09-13T13:13:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 79.467061818999994, 2024-09-13T13:13:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.234028868, 2024-09-13T13:13:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716022+0000", 2024-09-13T13:13:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716022+0000", 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716024+0000", 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7880000000000001e-06 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716034+0000", 2024-09-13T13:13:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5219999999999996e-06 2024-09-13T13:13:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716214+0000", 2024-09-13T13:13:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018008299999999999 2024-09-13T13:13:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716459+0000", 2024-09-13T13:13:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024492300000000003 2024-09-13T13:13:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716537+0000", 2024-09-13T13:13:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8596000000000004e-05 2024-09-13T13:13:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942849+0000", 2024-09-13T13:13:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.22631210600000001 2024-09-13T13:13:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.949963+0000", 2024-09-13T13:13:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0071133680000000001 2024-09-13T13:13:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.950051+0000", 2024-09-13T13:13:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8481999999999998e-05 2024-09-13T13:13:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:161595 9.2 e289/241 9:4df2328c:::benchmark_data_smithi052_128427_object30926:head v 289'10015, mlcod=289'10013)", 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.717193+0000", 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 79.465891479000007, 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.23484664699999999, 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.717193+0000", 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.717193+0000", 2024-09-13T13:13:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.717194+0000", 2024-09-13T13:13:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.579e-06 2024-09-13T13:13:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.717201+0000", 2024-09-13T13:13:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942820+0000", 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.225618547 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942836+0000", 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6680999999999998e-05 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942951+0000", 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011408000000000001 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942972+0000", 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1250000000000002e-05 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.951998+0000", 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0090262640000000009 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952039+0000", 2024-09-13T13:13:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1378999999999998e-05 2024-09-13T13:13:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:161591 9.6 e289/241 9:60592e87:::benchmark_data_smithi052_128427_object30922:head v 289'10102, mlcod=289'10099)", 2024-09-13T13:13:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.718993+0000", 2024-09-13T13:13:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 79.464091308999997, 2024-09-13T13:13:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.23666279500000001, 2024-09-13T13:13:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.718993+0000", 2024-09-13T13:13:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.718993+0000", 2024-09-13T13:13:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.718995+0000", 2024-09-13T13:13:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7990000000000001e-06 2024-09-13T13:13:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.719004+0000", 2024-09-13T13:13:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3829999999999998e-06 2024-09-13T13:13:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942844+0000", 2024-09-13T13:13:34.265 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.22383993799999999 2024-09-13T13:13:34.265 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.265 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.265 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.265 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942855+0000", 2024-09-13T13:13:34.265 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0499000000000001e-05 2024-09-13T13:13:34.265 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.265 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.266 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.266 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.945174+0000", 2024-09-13T13:13:34.266 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023197339999999999 2024-09-13T13:13:34.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.266 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.270 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.945191+0000", 2024-09-13T13:13:34.270 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7088e-05 2024-09-13T13:13:34.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.270 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.270 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.955462+0000", 2024-09-13T13:13:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010270646 2024-09-13T13:13:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.955656+0000", 2024-09-13T13:13:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019370800000000001 2024-09-13T13:13:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:161608 9.0 e289/241 9:0a1fbe34:::benchmark_data_smithi052_128427_object30939:head v 289'10085, mlcod=289'10083)", 2024-09-13T13:13:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.734825+0000", 2024-09-13T13:13:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 79.448259495000002, 2024-09-13T13:13:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.21376037000000001, 2024-09-13T13:13:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.734825+0000", 2024-09-13T13:13:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.734825+0000", 2024-09-13T13:13:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.734827+0000", 2024-09-13T13:13:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1610000000000001e-06 2024-09-13T13:13:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.734836+0000", 2024-09-13T13:13:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2879999999999998e-06 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.943089+0000", 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.208252828 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.943096+0000", 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5429999999999999e-06 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:34.274 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.943141+0000", 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5708999999999999e-05 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.943164+0000", 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3146000000000002e-05 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.948488+0000", 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053236239999999999 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:34.275 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:34.276 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.948585+0000", 2024-09-13T13:13:34.276 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7071000000000006e-05 2024-09-13T13:13:34.276 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.276 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.276 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.276 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:34.276 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:34.276 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:13:34.276 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:34.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:34.312 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:34.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:34.413 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:34.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:34.449 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:34.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:34.550 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:34.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:34 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:13:34.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:34 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:13:34.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:34 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1696006873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:34.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:13:34 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:13:34.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:34.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:34.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:34.687 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:34.725 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:34.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:34.827 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:34.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:34.859 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:34.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:34.949 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:13:34.960 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:34 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:13:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:34 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:13:34.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:34 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1696006873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:34.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:34 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:13:34.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:34 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:13:34.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:34 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1696006873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:35.030 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:35.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:35.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:35.160 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:35.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:35.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:35.261 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:35.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:35.310 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:35.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:35.411 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:35.448 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:35.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:35.550 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:35.583 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:35.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:35.686 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:35.725 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:35.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:35 smithi052 ceph-mon[36682]: pgmap v616: 148 pgs: 148 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-13T13:13:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:35.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:35 smithi052 ceph-mon[29471]: pgmap v616: 148 pgs: 148 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-13T13:13:35.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:35.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:35 smithi167 ceph-mon[33178]: pgmap v616: 148 pgs: 148 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-13T13:13:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:35.826 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:35.865 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:35.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:35.967 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:36.003 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:36.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:36.059 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T13:13:36.104 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:36.159 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:36.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:36.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:36.261 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:36.287 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:36.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:36.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:36.388 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:36.436 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:36.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:36.475 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:13:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:13:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:13:36.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:36.566 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:36.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:36.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:36.614 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:36.642 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:36.668 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:36.695 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:36.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:36.797 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:36.848 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:36.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:36.950 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:36.979 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:36.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:36.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:37.081 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:37.121 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:37.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:37.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:37.223 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:37.256 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:37.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:37.358 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:37.386 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:37.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:37.488 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:13:37.489 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:37.500 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T13:13:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:37.527 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:37.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:37.628 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:37.654 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:37.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:37.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:37.757 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:37.787 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:37.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:37.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:37.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:37 smithi167 ceph-mon[33178]: pgmap v617: 148 pgs: 148 active+clean; 1.1 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-13T13:13:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:37 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/943503821' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:13:37.890 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:37.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:37.936 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:37.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:37.956 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:37 smithi052 ceph-mon[29471]: pgmap v617: 148 pgs: 148 active+clean; 1.1 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-13T13:13:37.956 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:37 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/943503821' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:13:37.956 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:37 smithi052 ceph-mon[36682]: pgmap v617: 148 pgs: 148 active+clean; 1.1 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-13T13:13:37.956 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:37 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/943503821' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:13:38.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:38.067 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:38.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:38.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:38.169 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:38.209 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:38.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:38.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:38.311 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:38.348 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:38.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:38.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:38.450 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:38.487 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:38.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:38.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:38.559 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/943503821' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:13:38.559 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/943503821' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:13:38.559 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[29471]: osdmap e325: 8 total, 8 up, 8 in 2024-09-13T13:13:38.590 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:38.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:38.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:38.732 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:38.764 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:38.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:38.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:38.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:38 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/943503821' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:13:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:38 smithi167 ceph-mon[33178]: osdmap e325: 8 total, 8 up, 8 in 2024-09-13T13:13:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:38 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3205148147' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:13:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:38 smithi167 ceph-mon[33178]: from='client.25993 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:13:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:38 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:13:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:38 smithi167 ceph-mon[33178]: from='client.25993 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:13:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:38 smithi167 ceph-mon[33178]: osdmap e326: 8 total, 8 up, 8 in 2024-09-13T13:13:38.866 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:38.903 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:38.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:38.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:38.910 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:13:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:13:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:248858 9.3 9:cc679d28:::benchmark_data_smithi052_128427_object118189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-13T13:13:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:38.260582+0000", 2024-09-13T13:13:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.014517756999999999, 2024-09-13T13:13:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014545971, 2024-09-13T13:13:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:13:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:38.911 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 248858 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260582+0000", 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260582+0000", 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:38.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260582+0000", 2024-09-13T13:13:38.913 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.09e-07 2024-09-13T13:13:38.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:38.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260586+0000", 2024-09-13T13:13:38.913 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5999999999999998e-06 2024-09-13T13:13:38.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:38.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260593+0000", 2024-09-13T13:13:38.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5480000000000004e-06 2024-09-13T13:13:38.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.914 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:38.914 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260599+0000", 2024-09-13T13:13:38.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.964e-06 2024-09-13T13:13:38.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.914 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:38.914 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.268077+0000", 2024-09-13T13:13:38.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0074774869999999997 2024-09-13T13:13:38.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.915 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:38.915 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.268175+0000", 2024-09-13T13:13:38.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7584000000000006e-05 2024-09-13T13:13:38.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.915 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,7", 2024-09-13T13:13:38.915 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.268277+0000", 2024-09-13T13:13:38.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010295500000000001 2024-09-13T13:13:38.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.916 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:38.916 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.272529+0000", 2024-09-13T13:13:38.916 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0042515720000000003 2024-09-13T13:13:38.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.916 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.916 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:38.916 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.275120+0000", 2024-09-13T13:13:38.916 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025906319999999998 2024-09-13T13:13:38.916 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.916 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:38.916 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.917 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:248859 9.2 9:44781f17:::benchmark_data_smithi052_128427_object118190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-13T13:13:38.917 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:38.260608+0000", 2024-09-13T13:13:38.917 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.014491418000000001, 2024-09-13T13:13:38.917 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014599881, 2024-09-13T13:13:38.917 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:38.917 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:38.917 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:13:38.917 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:38.917 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:38.917 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:38.917 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 248859 2024-09-13T13:13:38.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.918 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:38.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.918 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:38.918 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260608+0000", 2024-09-13T13:13:38.918 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.918 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:38.918 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260608+0000", 2024-09-13T13:13:38.918 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260608+0000", 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.44e-07 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260611+0000", 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.514e-06 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260616+0000", 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.169e-06 2024-09-13T13:13:38.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:38.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260621+0000", 2024-09-13T13:13:38.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8760000000000004e-06 2024-09-13T13:13:38.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:38.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.261851+0000", 2024-09-13T13:13:38.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012300600000000001 2024-09-13T13:13:38.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:38.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.261978+0000", 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012705400000000001 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.262841+0000", 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00086262999999999997 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.270409+0000", 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0075678550000000001 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:38.922 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.272669+0000", 2024-09-13T13:13:38.922 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002260027 2024-09-13T13:13:38.922 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.922 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:38.922 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.922 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.922 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:248860 9.2 9:4d3821e6:::benchmark_data_smithi052_128427_object118191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-13T13:13:38.922 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:38.261014+0000", 2024-09-13T13:13:38.922 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.014085024, 2024-09-13T13:13:38.922 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014265313, 2024-09-13T13:13:38.922 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:38.922 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 248860 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.261014+0000", 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.261014+0000", 2024-09-13T13:13:38.923 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.261015+0000", 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.203e-06 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.261025+0000", 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6979999999999993e-06 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:38.924 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.261058+0000", 2024-09-13T13:13:38.925 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2393e-05 2024-09-13T13:13:38.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.925 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:38.925 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.261064+0000", 2024-09-13T13:13:38.925 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8870000000000003e-06 2024-09-13T13:13:38.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.925 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:38.926 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.267485+0000", 2024-09-13T13:13:38.926 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064208620000000003 2024-09-13T13:13:38.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.926 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.926 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:38.926 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.267593+0000", 2024-09-13T13:13:38.926 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107934 2024-09-13T13:13:38.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.926 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.926 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:13:38.926 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.267814+0000", 2024-09-13T13:13:38.926 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00022108800000000001 2024-09-13T13:13:38.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.927 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:38.927 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.272674+0000", 2024-09-13T13:13:38.927 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048594689999999999 2024-09-13T13:13:38.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.927 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:38.927 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273697+0000", 2024-09-13T13:13:38.927 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010229550000000001 2024-09-13T13:13:38.927 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.927 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:38.927 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.928 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.928 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.15789.0:248848 9.3 e326/298 ondisk, result = 0)", 2024-09-13T13:13:38.928 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:38.272742+0000", 2024-09-13T13:13:38.928 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0023572509999999999, 2024-09-13T13:13:38.928 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026056790000000001, 2024-09-13T13:13:38.928 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:38.928 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.272742+0000", 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.272742+0000", 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.272743+0000", 2024-09-13T13:13:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:13:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.272745+0000", 2024-09-13T13:13:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9639999999999999e-06 2024-09-13T13:13:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.272752+0000", 2024-09-13T13:13:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2179999999999996e-06 2024-09-13T13:13:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.272757+0000", 2024-09-13T13:13:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2820000000000004e-06 2024-09-13T13:13:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.275263+0000", 2024-09-13T13:13:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025058120000000001 2024-09-13T13:13:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.275270+0000", 2024-09-13T13:13:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9e-06 2024-09-13T13:13:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:248872 9.2 9:4257750e:::benchmark_data_smithi052_128427_object118203:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-13T13:13:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:38.273524+0000", 2024-09-13T13:13:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.001575171, 2024-09-13T13:13:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001878631, 2024-09-13T13:13:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:13:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 248872 2024-09-13T13:13:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273524+0000", 2024-09-13T13:13:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273524+0000", 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273526+0000", 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.95e-06 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273531+0000", 2024-09-13T13:13:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9579999999999998e-06 2024-09-13T13:13:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273545+0000", 2024-09-13T13:13:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3839000000000001e-05 2024-09-13T13:13:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273552+0000", 2024-09-13T13:13:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9709999999999998e-06 2024-09-13T13:13:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273724+0000", 2024-09-13T13:13:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000172151 2024-09-13T13:13:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273824+0000", 2024-09-13T13:13:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9765000000000005e-05 2024-09-13T13:13:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:13:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273926+0000", 2024-09-13T13:13:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010253499999999999 2024-09-13T13:13:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:248874 9.2 9.117c5202 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-13T13:13:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:38.273569+0000", 2024-09-13T13:13:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0015303579999999999, 2024-09-13T13:13:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018959339999999999, 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 248874 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273569+0000", 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273569+0000", 2024-09-13T13:13:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273570+0000", 2024-09-13T13:13:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3600000000000003e-07 2024-09-13T13:13:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273573+0000", 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273579+0000", 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7490000000000004e-06 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.273583+0000", 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.651e-06 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:248853 9.2 9:45af4f1f:::benchmark_data_smithi052_128427_object118184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-13T13:13:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:38.256112+0000", 2024-09-13T13:13:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.018987522, 2024-09-13T13:13:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.019406626999999999, 2024-09-13T13:13:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:13:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 248853 2024-09-13T13:13:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.256112+0000", 2024-09-13T13:13:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.256112+0000", 2024-09-13T13:13:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.943 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.943 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:38.943 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.256113+0000", 2024-09-13T13:13:38.943 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7499999999999999e-07 2024-09-13T13:13:38.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.943 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.943 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:38.943 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.256115+0000", 2024-09-13T13:13:38.943 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9299999999999999e-06 2024-09-13T13:13:38.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.943 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.943 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:38.943 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.256121+0000", 2024-09-13T13:13:38.944 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4079999999999997e-06 2024-09-13T13:13:38.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.944 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.944 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:38.944 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.256126+0000", 2024-09-13T13:13:38.944 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6929999999999998e-06 2024-09-13T13:13:38.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.944 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.944 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:38.944 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.257194+0000", 2024-09-13T13:13:38.944 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001068479 2024-09-13T13:13:38.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.945 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.945 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:38.945 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.257308+0000", 2024-09-13T13:13:38.945 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113613 2024-09-13T13:13:38.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.945 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.945 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:13:38.945 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.257427+0000", 2024-09-13T13:13:38.946 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000119139 2024-09-13T13:13:38.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.946 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.946 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:38.946 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.270370+0000", 2024-09-13T13:13:38.946 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.012942896000000001 2024-09-13T13:13:38.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.946 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.946 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:38.946 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.270424+0000", 2024-09-13T13:13:38.947 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4528999999999999e-05 2024-09-13T13:13:38.947 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.947 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:38.947 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.947 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.947 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:248856 9.3 9:c9a84e98:::benchmark_data_smithi052_128427_object118187:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-13T13:13:38.947 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:38.260462+0000", 2024-09-13T13:13:38.947 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.01463706, 2024-09-13T13:13:38.947 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0151369, 2024-09-13T13:13:38.947 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:38.947 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:38.948 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:13:38.948 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:38.948 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:38.948 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:38.948 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 248856 2024-09-13T13:13:38.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.948 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:38.948 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.948 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:38.948 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260462+0000", 2024-09-13T13:13:38.948 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.949 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.949 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:38.949 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260462+0000", 2024-09-13T13:13:38.949 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:38.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.949 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.949 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:38.949 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260464+0000", 2024-09-13T13:13:38.949 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6249999999999999e-06 2024-09-13T13:13:38.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.949 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.949 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:38.950 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260469+0000", 2024-09-13T13:13:38.950 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6689999999999997e-06 2024-09-13T13:13:38.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.950 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.950 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:38.950 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260478+0000", 2024-09-13T13:13:38.950 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.55e-06 2024-09-13T13:13:38.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.950 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.950 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:38.950 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.260485+0000", 2024-09-13T13:13:38.950 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9909999999999996e-06 2024-09-13T13:13:38.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.951 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.951 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:38.951 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.264075+0000", 2024-09-13T13:13:38.951 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003590298 2024-09-13T13:13:38.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.951 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.951 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:38.951 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.264185+0000", 2024-09-13T13:13:38.951 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010945 2024-09-13T13:13:38.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.951 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.951 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,7", 2024-09-13T13:13:38.951 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.264295+0000", 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109787 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.272524+0000", 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.008228941 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:38.275084+0000", 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025605530000000001 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:38.952 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.953 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:38.953 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:13:38.953 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 8 2024-09-13T13:13:38.953 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:13:38.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3205148147' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:13:38.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[29471]: from='client.25993 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:13:38.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:13:38.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[29471]: from='client.25993 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:13:38.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[29471]: osdmap e326: 8 total, 8 up, 8 in 2024-09-13T13:13:38.954 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[36682]: osdmap e325: 8 total, 8 up, 8 in 2024-09-13T13:13:38.954 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3205148147' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:13:38.954 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[36682]: from='client.25993 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:13:38.954 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:13:38.954 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[36682]: from='client.25993 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:13:38.954 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:38 smithi052 ceph-mon[36682]: osdmap e326: 8 total, 8 up, 8 in 2024-09-13T13:13:39.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:39.041 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:39.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:39.142 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:39.165 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:39.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:39.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:39.296 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:39.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:39.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:39.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:39.429 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:39.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:39.531 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:39.566 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:39.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:39.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:39.668 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:39.703 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:39.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:39.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:39.805 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:39.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:39 smithi167 ceph-mon[33178]: pgmap v619: 148 pgs: 148 active+clean; 1.1 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 752 op/s 2024-09-13T13:13:39.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:39 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:13:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:39 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:13:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:39 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:13:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:39 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:13:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:39 smithi167 ceph-mon[33178]: 9.4 scrub starts 2024-09-13T13:13:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:39 smithi167 ceph-mon[33178]: 4.e scrub starts 2024-09-13T13:13:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:39 smithi167 ceph-mon[33178]: 4.e scrub ok 2024-09-13T13:13:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:39 smithi167 ceph-mon[33178]: osdmap e327: 8 total, 8 up, 8 in 2024-09-13T13:13:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:39 smithi167 ceph-mon[33178]: 9.4 scrub ok 2024-09-13T13:13:39.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:39.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:39.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:39.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:39.967 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:39.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:39.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[29471]: pgmap v619: 148 pgs: 148 active+clean; 1.1 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 752 op/s 2024-09-13T13:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[29471]: 9.4 scrub starts 2024-09-13T13:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[29471]: 4.e scrub starts 2024-09-13T13:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[29471]: 4.e scrub ok 2024-09-13T13:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[29471]: osdmap e327: 8 total, 8 up, 8 in 2024-09-13T13:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[29471]: 9.4 scrub ok 2024-09-13T13:13:39.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[36682]: pgmap v619: 148 pgs: 148 active+clean; 1.1 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 752 op/s 2024-09-13T13:13:39.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:13:39.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:13:39.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:13:39.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:13:39.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[36682]: 9.4 scrub starts 2024-09-13T13:13:39.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[36682]: 4.e scrub starts 2024-09-13T13:13:39.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[36682]: 4.e scrub ok 2024-09-13T13:13:39.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[36682]: osdmap e327: 8 total, 8 up, 8 in 2024-09-13T13:13:39.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:39 smithi052 ceph-mon[36682]: 9.4 scrub ok 2024-09-13T13:13:40.069 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:40.109 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:40.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:40.211 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:40.244 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:40.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:40.345 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:40.376 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:40.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:40.478 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:40.528 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:40.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:40.630 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:40.672 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:40.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:40.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:40 smithi052 ceph-mon[36682]: 3.1a scrub starts 2024-09-13T13:13:40.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:40 smithi052 ceph-mon[36682]: 3.1a scrub ok 2024-09-13T13:13:40.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:40 smithi052 ceph-mon[29471]: 3.1a scrub starts 2024-09-13T13:13:40.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:40 smithi052 ceph-mon[29471]: 3.1a scrub ok 2024-09-13T13:13:40.775 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:40.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:40.812 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:40.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:40.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:40.949 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:40.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:40.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:41.051 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:41.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:40 smithi167 ceph-mon[33178]: 3.1a scrub starts 2024-09-13T13:13:41.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:40 smithi167 ceph-mon[33178]: 3.1a scrub ok 2024-09-13T13:13:41.090 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:41.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:41.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:41.192 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:41.231 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T13:13:41.246 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:41.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:41.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:41.348 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:41.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:41.397 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:41.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:41.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:41.540 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:41.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:41.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:41.642 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:41.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:41.677 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:41.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:41.778 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:41.804 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:13:41.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:41.814 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:41.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:41.915 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:41.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:41.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:41.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:41.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:41 smithi052 ceph-mon[36682]: pgmap v622: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 736 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.68k op/s 2024-09-13T13:13:41.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:41 smithi052 ceph-mon[36682]: 6.18 scrub starts 2024-09-13T13:13:41.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:41 smithi052 ceph-mon[36682]: 6.18 scrub ok 2024-09-13T13:13:41.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:41 smithi052 ceph-mon[29471]: pgmap v622: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 736 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.68k op/s 2024-09-13T13:13:41.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:41 smithi052 ceph-mon[29471]: 6.18 scrub starts 2024-09-13T13:13:41.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:41 smithi052 ceph-mon[29471]: 6.18 scrub ok 2024-09-13T13:13:42.056 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:42.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:41 smithi167 ceph-mon[33178]: pgmap v622: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 736 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.68k op/s 2024-09-13T13:13:42.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:41 smithi167 ceph-mon[33178]: 6.18 scrub starts 2024-09-13T13:13:42.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:41 smithi167 ceph-mon[33178]: 6.18 scrub ok 2024-09-13T13:13:42.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:42.082 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:42.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:42.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:42.216 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:42.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:42.318 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:42.359 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:42.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:42.461 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:42.494 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:42.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:42.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:42.596 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:42.637 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:42.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:42.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:42.739 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:42.775 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:42.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:42.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:42.877 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:42.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:42.912 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:42.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:42.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[36682]: 4.1d scrub starts 2024-09-13T13:13:42.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[36682]: 4.1d scrub ok 2024-09-13T13:13:42.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[36682]: 3.1d scrub starts 2024-09-13T13:13:42.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[36682]: 3.1d scrub ok 2024-09-13T13:13:42.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[36682]: 6.9 scrub starts 2024-09-13T13:13:42.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[36682]: 6.9 scrub ok 2024-09-13T13:13:42.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[36682]: 6.e scrub starts 2024-09-13T13:13:42.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[36682]: 6.e scrub ok 2024-09-13T13:13:42.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[29471]: 4.1d scrub starts 2024-09-13T13:13:42.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[29471]: 4.1d scrub ok 2024-09-13T13:13:42.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[29471]: 3.1d scrub starts 2024-09-13T13:13:42.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[29471]: 3.1d scrub ok 2024-09-13T13:13:42.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[29471]: 6.9 scrub starts 2024-09-13T13:13:42.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[29471]: 6.9 scrub ok 2024-09-13T13:13:42.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[29471]: 6.e scrub starts 2024-09-13T13:13:42.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:42 smithi052 ceph-mon[29471]: 6.e scrub ok 2024-09-13T13:13:43.013 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:43.050 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:43.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:43.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:43.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:42 smithi167 ceph-mon[33178]: 4.1d scrub starts 2024-09-13T13:13:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:42 smithi167 ceph-mon[33178]: 4.1d scrub ok 2024-09-13T13:13:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:42 smithi167 ceph-mon[33178]: 3.1d scrub starts 2024-09-13T13:13:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:42 smithi167 ceph-mon[33178]: 3.1d scrub ok 2024-09-13T13:13:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:42 smithi167 ceph-mon[33178]: 6.9 scrub starts 2024-09-13T13:13:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:42 smithi167 ceph-mon[33178]: 6.9 scrub ok 2024-09-13T13:13:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:42 smithi167 ceph-mon[33178]: 6.e scrub starts 2024-09-13T13:13:43.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:42 smithi167 ceph-mon[33178]: 6.e scrub ok 2024-09-13T13:13:43.153 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:43.174 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:43.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:43.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:43.276 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:43.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:43.306 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:43.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:43.407 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:43.441 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:43.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:43.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:43.564 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:43.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:43.665 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:43.702 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:43.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:43.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:43.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:43 smithi167 ceph-mon[33178]: pgmap v623: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 736 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-13T13:13:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:43 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:13:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:43 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:13:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:43 smithi167 ceph-mon[33178]: osdmap e328: 8 total, 8 up, 8 in 2024-09-13T13:13:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:43 smithi167 ceph-mon[33178]: 3.e scrub starts 2024-09-13T13:13:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:43 smithi167 ceph-mon[33178]: 3.e scrub ok 2024-09-13T13:13:43.840 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:43.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:43.943 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:43.972 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:43.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:43.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:43 smithi052 ceph-mon[36682]: pgmap v623: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 736 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-13T13:13:43.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:43 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:13:43.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:43 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:13:43.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:43 smithi052 ceph-mon[36682]: osdmap e328: 8 total, 8 up, 8 in 2024-09-13T13:13:43.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:43 smithi052 ceph-mon[36682]: 3.e scrub starts 2024-09-13T13:13:43.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:43 smithi052 ceph-mon[36682]: 3.e scrub ok 2024-09-13T13:13:43.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:43 smithi052 ceph-mon[29471]: pgmap v623: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 736 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-13T13:13:43.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:43 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:13:43.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:43 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:13:43.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:43 smithi052 ceph-mon[29471]: osdmap e328: 8 total, 8 up, 8 in 2024-09-13T13:13:43.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:43 smithi052 ceph-mon[29471]: 3.e scrub starts 2024-09-13T13:13:43.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:43 smithi052 ceph-mon[29471]: 3.e scrub ok 2024-09-13T13:13:44.075 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:44.106 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:44.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:44.208 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:44.238 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:44.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:44.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:44.340 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:44.375 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:44.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:44.477 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:44.513 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:44.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:44.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:44.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:13:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:13:44.615 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:44.667 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:44.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:44.769 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:44.800 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:44.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:44.902 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:44.939 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:44.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:44.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:44 smithi052 ceph-mon[29471]: 5.1b scrub starts 2024-09-13T13:13:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:44 smithi052 ceph-mon[29471]: 5.1b scrub ok 2024-09-13T13:13:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:44 smithi052 ceph-mon[29471]: 5.5 scrub starts 2024-09-13T13:13:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:44 smithi052 ceph-mon[29471]: 5.5 scrub ok 2024-09-13T13:13:44.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:44 smithi052 ceph-mon[36682]: 5.1b scrub starts 2024-09-13T13:13:44.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:44 smithi052 ceph-mon[36682]: 5.1b scrub ok 2024-09-13T13:13:44.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:44 smithi052 ceph-mon[36682]: 5.5 scrub starts 2024-09-13T13:13:44.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:44 smithi052 ceph-mon[36682]: 5.5 scrub ok 2024-09-13T13:13:45.041 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:45.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:44 smithi167 ceph-mon[33178]: 5.1b scrub starts 2024-09-13T13:13:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:44 smithi167 ceph-mon[33178]: 5.1b scrub ok 2024-09-13T13:13:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:44 smithi167 ceph-mon[33178]: 5.5 scrub starts 2024-09-13T13:13:45.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:44 smithi167 ceph-mon[33178]: 5.5 scrub ok 2024-09-13T13:13:45.081 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:45.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:45.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:45.150 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T13:13:45.183 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:45.257 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:45.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:45.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:45.266 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:45.274 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:45.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:45.388 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:45.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:45.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:45.490 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:45.526 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:45.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:45.628 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:45.655 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:45.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:45.672 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[36682]: pgmap v625: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 611 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-13T13:13:45.672 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[36682]: 5.13 scrub starts 2024-09-13T13:13:45.672 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[36682]: 5.13 scrub ok 2024-09-13T13:13:45.672 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:45.672 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[36682]: 3.16 scrub starts 2024-09-13T13:13:45.673 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[36682]: 3.16 scrub ok 2024-09-13T13:13:45.673 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[36682]: 3.17 scrub starts 2024-09-13T13:13:45.673 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[36682]: 6.2 scrub starts 2024-09-13T13:13:45.673 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[36682]: 3.17 scrub ok 2024-09-13T13:13:45.673 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[36682]: 6.2 scrub ok 2024-09-13T13:13:45.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[29471]: pgmap v625: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 611 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-13T13:13:45.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[29471]: 5.13 scrub starts 2024-09-13T13:13:45.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[29471]: 5.13 scrub ok 2024-09-13T13:13:45.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:45.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[29471]: 3.16 scrub starts 2024-09-13T13:13:45.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[29471]: 3.16 scrub ok 2024-09-13T13:13:45.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[29471]: 3.17 scrub starts 2024-09-13T13:13:45.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[29471]: 6.2 scrub starts 2024-09-13T13:13:45.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[29471]: 3.17 scrub ok 2024-09-13T13:13:45.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:45 smithi052 ceph-mon[29471]: 6.2 scrub ok 2024-09-13T13:13:45.757 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:45.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:45.791 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:45.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:45.892 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:45.935 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:45.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:45.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:46.000 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:13:46.001 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:13:46.037 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:46.047 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:13:46.047 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:13:46.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:45 smithi167 ceph-mon[33178]: pgmap v625: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 611 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-13T13:13:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:45 smithi167 ceph-mon[33178]: 5.13 scrub starts 2024-09-13T13:13:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:45 smithi167 ceph-mon[33178]: 5.13 scrub ok 2024-09-13T13:13:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:45 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:45 smithi167 ceph-mon[33178]: 3.16 scrub starts 2024-09-13T13:13:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:45 smithi167 ceph-mon[33178]: 3.16 scrub ok 2024-09-13T13:13:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:45 smithi167 ceph-mon[33178]: 3.17 scrub starts 2024-09-13T13:13:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:45 smithi167 ceph-mon[33178]: 6.2 scrub starts 2024-09-13T13:13:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:45 smithi167 ceph-mon[33178]: 3.17 scrub ok 2024-09-13T13:13:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:45 smithi167 ceph-mon[33178]: 6.2 scrub ok 2024-09-13T13:13:46.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:46.078 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:46.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:46.081 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:13:46.081 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:13:46.119 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:13:46.119 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:13:46.154 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:13:46.155 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:13:46.176 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:13:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:13:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:13:46.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:46.189 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:13:46.190 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:13:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:46.215 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:46.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:46.230 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:13:46.231 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:13:46.261 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:13:46.261 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:13:46.316 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:46.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:46.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:46.447 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:46.476 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:46.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:46.578 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:46.620 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:46.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:46.654 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:13:46.721 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:46.775 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:46.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:46.877 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:46.907 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:46.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:46.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:46 smithi052 ceph-mon[36682]: 4.3 scrub starts 2024-09-13T13:13:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:46 smithi052 ceph-mon[36682]: 4.3 scrub ok 2024-09-13T13:13:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:46 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/977879345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:46.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:46 smithi052 ceph-mon[29471]: 4.3 scrub starts 2024-09-13T13:13:46.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:46 smithi052 ceph-mon[29471]: 4.3 scrub ok 2024-09-13T13:13:46.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:46 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/977879345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:47.009 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:47.050 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:47.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:47.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:46 smithi167 ceph-mon[33178]: 4.3 scrub starts 2024-09-13T13:13:47.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:46 smithi167 ceph-mon[33178]: 4.3 scrub ok 2024-09-13T13:13:47.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:46 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/977879345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:47.152 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:47.210 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:47.211 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:47.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:47.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:47.312 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:47.351 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:47.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:47.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:47.453 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:47.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:47.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:47.593 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:47.624 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:47.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:47.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:47.633 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T13:13:47.660 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:47 smithi052 ceph-mon[36682]: pgmap v626: 148 pgs: 148 active+clean; 345 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-13T13:13:47.660 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:47 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3094931758' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:13:47.661 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:47 smithi052 ceph-mon[29471]: pgmap v626: 148 pgs: 148 active+clean; 345 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-13T13:13:47.661 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:47 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3094931758' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:13:47.726 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:47.756 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:47.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:47.858 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:47.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:47.895 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:47.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:47.934 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:47.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:48.032 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:48.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:48.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:47 smithi167 ceph-mon[33178]: pgmap v626: 148 pgs: 148 active+clean; 345 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-13T13:13:48.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:47 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3094931758' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:13:48.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:48.172 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:48.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:48.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:48.273 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:48.308 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Removed 130666 objects 2024-09-13T13:13:48.308 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Clean up completed and total clean up time :122.497 2024-09-13T13:13:48.308 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:48.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:48.370 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:13:48.370 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:13:48.411 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:48.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:48.433 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:48.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:48.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:48.556 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:48.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:48.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:48.658 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:48.679 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:48.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:48.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:48.803 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:48.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:48.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:48.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:48.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:48.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:48.948 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3094931758' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:13:48.948 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[36682]: osdmap e329: 8 total, 8 up, 8 in 2024-09-13T13:13:48.948 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[36682]: 4.13 scrub starts 2024-09-13T13:13:48.949 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[36682]: 4.13 scrub ok 2024-09-13T13:13:48.949 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[36682]: osdmap e330: 8 total, 8 up, 8 in 2024-09-13T13:13:48.949 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[36682]: 9.7 scrub starts 2024-09-13T13:13:48.949 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[36682]: 9.7 scrub ok 2024-09-13T13:13:48.949 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:13:48.950 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3094931758' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:13:48.950 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[29471]: osdmap e329: 8 total, 8 up, 8 in 2024-09-13T13:13:48.950 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[29471]: 4.13 scrub starts 2024-09-13T13:13:48.950 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[29471]: 4.13 scrub ok 2024-09-13T13:13:48.951 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[29471]: osdmap e330: 8 total, 8 up, 8 in 2024-09-13T13:13:48.951 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[29471]: 9.7 scrub starts 2024-09-13T13:13:48.951 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[29471]: 9.7 scrub ok 2024-09-13T13:13:48.951 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:48 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:13:49.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:49.051 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:49.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:49.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:49.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:48 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3094931758' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:13:49.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:48 smithi167 ceph-mon[33178]: osdmap e329: 8 total, 8 up, 8 in 2024-09-13T13:13:49.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:48 smithi167 ceph-mon[33178]: 4.13 scrub starts 2024-09-13T13:13:49.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:48 smithi167 ceph-mon[33178]: 4.13 scrub ok 2024-09-13T13:13:49.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:48 smithi167 ceph-mon[33178]: osdmap e330: 8 total, 8 up, 8 in 2024-09-13T13:13:49.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:48 smithi167 ceph-mon[33178]: 9.7 scrub starts 2024-09-13T13:13:49.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:48 smithi167 ceph-mon[33178]: 9.7 scrub ok 2024-09-13T13:13:49.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:48 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:13:49.154 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:49.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:49.176 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:49.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:49.183 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:13:49.277 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:49.299 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:49.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:49.401 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:49.423 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:49.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:49.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:49.549 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:49.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:49.650 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:49.666 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:13:49.679 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:49.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:49.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:49.787 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-13T13:13:49.787 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-13T13:13:49.804 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:49.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:49.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:49.882 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T13:13:49.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:49.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:49.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:49.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:49.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:49 smithi052 ceph-mon[36682]: pgmap v628: 148 pgs: 148 active+clean; 345 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 782 op/s 2024-09-13T13:13:49.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:49 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2389630512' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:49.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:49 smithi052 ceph-mon[36682]: from='client.36029 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:49.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:49 smithi052 ceph-mon[36682]: 4.11 scrub starts 2024-09-13T13:13:49.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:49 smithi052 ceph-mon[36682]: 4.11 scrub ok 2024-09-13T13:13:49.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:49 smithi052 ceph-mon[29471]: pgmap v628: 148 pgs: 148 active+clean; 345 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 782 op/s 2024-09-13T13:13:49.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:49 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2389630512' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:49.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:49 smithi052 ceph-mon[29471]: from='client.36029 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:49.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:49 smithi052 ceph-mon[29471]: 4.11 scrub starts 2024-09-13T13:13:49.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:49 smithi052 ceph-mon[29471]: 4.11 scrub ok 2024-09-13T13:13:50.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:50.054 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:50.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:50.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:49 smithi167 ceph-mon[33178]: pgmap v628: 148 pgs: 148 active+clean; 345 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 782 op/s 2024-09-13T13:13:50.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:49 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2389630512' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:50.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:49 smithi167 ceph-mon[33178]: from='client.36029 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:13:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:49 smithi167 ceph-mon[33178]: 4.11 scrub starts 2024-09-13T13:13:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:49 smithi167 ceph-mon[33178]: 4.11 scrub ok 2024-09-13T13:13:50.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:50.177 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:50.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:50.279 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:50.302 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:50.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:50.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:50.403 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:50.426 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:50.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:50.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:50.485 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T13:13:50.528 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:50.557 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:50.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:50.658 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:50.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:50.685 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:50.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:50.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:13:50.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:13:50.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[29471]: from='client.36029 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:13:50.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[29471]: osdmap e331: 8 total, 8 up, 8 in 2024-09-13T13:13:50.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[29471]: 6.16 scrub starts 2024-09-13T13:13:50.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[29471]: 6.16 scrub ok 2024-09-13T13:13:50.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[29471]: 4.0 scrub starts 2024-09-13T13:13:50.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[29471]: 4.0 scrub ok 2024-09-13T13:13:50.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:13:50.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:13:50.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[36682]: from='client.36029 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:13:50.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[36682]: osdmap e331: 8 total, 8 up, 8 in 2024-09-13T13:13:50.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[36682]: 6.16 scrub starts 2024-09-13T13:13:50.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[36682]: 6.16 scrub ok 2024-09-13T13:13:50.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[36682]: 4.0 scrub starts 2024-09-13T13:13:50.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:50 smithi052 ceph-mon[36682]: 4.0 scrub ok 2024-09-13T13:13:50.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:50.812 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:50.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:50.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:50.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:50.938 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:50.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:50.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:51.006 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:51.025 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:51.025 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:51.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:51.063 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:51.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:51.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:51.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:50 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:13:51.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:50 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:13:51.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:50 smithi167 ceph-mon[33178]: from='client.36029 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:13:51.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:50 smithi167 ceph-mon[33178]: osdmap e331: 8 total, 8 up, 8 in 2024-09-13T13:13:51.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:50 smithi167 ceph-mon[33178]: 6.16 scrub starts 2024-09-13T13:13:51.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:50 smithi167 ceph-mon[33178]: 6.16 scrub ok 2024-09-13T13:13:51.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:50 smithi167 ceph-mon[33178]: 4.0 scrub starts 2024-09-13T13:13:51.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:50 smithi167 ceph-mon[33178]: 4.0 scrub ok 2024-09-13T13:13:51.166 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:51.188 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:51.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:51.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:51.291 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:51.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:51.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:51.418 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:51.443 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:51.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:51.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:51.545 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:51.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:51.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:51.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:51.673 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:51.696 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:51.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:51.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:51.749 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:13:51.749 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:13:51.760 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:13:51.761 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:13:51.777 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:13:51.778 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:13:51.793 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:13:51.794 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:13:51.798 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:51.811 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:13:51.811 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:13:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:51.825 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:51.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:51.828 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:13:51.829 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:13:51.845 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:13:51.846 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:13:51.863 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:13:51.864 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:13:51.891 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:51 smithi052 ceph-mon[36682]: pgmap v631: 148 pgs: 148 active+clean; 1.5 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-09-13T13:13:51.891 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:51 smithi052 ceph-mon[36682]: 4.1 scrub starts 2024-09-13T13:13:51.892 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:51 smithi052 ceph-mon[36682]: 4.1 scrub ok 2024-09-13T13:13:51.892 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:51 smithi052 ceph-mon[36682]: 6.1 scrub starts 2024-09-13T13:13:51.892 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:51 smithi052 ceph-mon[36682]: 6.1 scrub ok 2024-09-13T13:13:51.892 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:51 smithi052 ceph-mon[29471]: pgmap v631: 148 pgs: 148 active+clean; 1.5 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-09-13T13:13:51.892 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:51 smithi052 ceph-mon[29471]: 4.1 scrub starts 2024-09-13T13:13:51.892 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:51 smithi052 ceph-mon[29471]: 4.1 scrub ok 2024-09-13T13:13:51.892 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:51 smithi052 ceph-mon[29471]: 6.1 scrub starts 2024-09-13T13:13:51.892 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:51 smithi052 ceph-mon[29471]: 6.1 scrub ok 2024-09-13T13:13:51.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:51.951 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:51.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:51.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:52.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:52.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:51 smithi167 ceph-mon[33178]: pgmap v631: 148 pgs: 148 active+clean; 1.5 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-09-13T13:13:52.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:51 smithi167 ceph-mon[33178]: 4.1 scrub starts 2024-09-13T13:13:52.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:51 smithi167 ceph-mon[33178]: 4.1 scrub ok 2024-09-13T13:13:52.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:51 smithi167 ceph-mon[33178]: 6.1 scrub starts 2024-09-13T13:13:52.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:51 smithi167 ceph-mon[33178]: 6.1 scrub ok 2024-09-13T13:13:52.076 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:52.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:52.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:52.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:52.201 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:52.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:52.302 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:52.325 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:52.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:52.417 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:52.426 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:52.452 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:52.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:52.553 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:52.577 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:52.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:52.678 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:52.684 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'unique_pool_2' removed 2024-09-13T13:13:52.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:52.736 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:52.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:52.838 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:52.841 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[29471]: 5.8 scrub starts 2024-09-13T13:13:52.841 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[29471]: 5.8 scrub ok 2024-09-13T13:13:52.842 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/521043825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:52.842 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[29471]: 3.11 scrub starts 2024-09-13T13:13:52.842 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[29471]: 3.11 scrub ok 2024-09-13T13:13:52.842 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2504367473' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:13:52.842 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[29471]: from='client.36041 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:13:52.842 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[29471]: 3.1e scrub starts 2024-09-13T13:13:52.842 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[29471]: 3.1e scrub ok 2024-09-13T13:13:52.842 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[29471]: 3.d scrub starts 2024-09-13T13:13:52.842 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[29471]: 3.d scrub ok 2024-09-13T13:13:52.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[36682]: 5.8 scrub starts 2024-09-13T13:13:52.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[36682]: 5.8 scrub ok 2024-09-13T13:13:52.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/521043825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:52.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[36682]: 3.11 scrub starts 2024-09-13T13:13:52.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[36682]: 3.11 scrub ok 2024-09-13T13:13:52.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2504367473' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:13:52.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[36682]: from='client.36041 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:13:52.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[36682]: 3.1e scrub starts 2024-09-13T13:13:52.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[36682]: 3.1e scrub ok 2024-09-13T13:13:52.844 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[36682]: 3.d scrub starts 2024-09-13T13:13:52.844 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:52 smithi052 ceph-mon[36682]: 3.d scrub ok 2024-09-13T13:13:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:52.863 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:52.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:52.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:52.987 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:52.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:53.006 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:13:53.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:52 smithi167 ceph-mon[33178]: 5.8 scrub starts 2024-09-13T13:13:53.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:52 smithi167 ceph-mon[33178]: 5.8 scrub ok 2024-09-13T13:13:53.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:52 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/521043825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:13:53.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:52 smithi167 ceph-mon[33178]: 3.11 scrub starts 2024-09-13T13:13:53.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:52 smithi167 ceph-mon[33178]: 3.11 scrub ok 2024-09-13T13:13:53.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:52 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2504367473' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:13:53.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:52 smithi167 ceph-mon[33178]: from='client.36041 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:13:53.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:52 smithi167 ceph-mon[33178]: 3.1e scrub starts 2024-09-13T13:13:53.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:52 smithi167 ceph-mon[33178]: 3.1e scrub ok 2024-09-13T13:13:53.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:52 smithi167 ceph-mon[33178]: 3.d scrub starts 2024-09-13T13:13:53.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:52 smithi167 ceph-mon[33178]: 3.d scrub ok 2024-09-13T13:13:53.090 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:53.141 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:53.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:53.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:53.243 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:53.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:53.265 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:53.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:53.366 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:53.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:53.389 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:53.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:53.490 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:53.513 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:53.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:53.614 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:53.638 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:53.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:53.678 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T13:13:53.740 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:53.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:53.765 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:53.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:53.866 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:53.889 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:53.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:53.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[36682]: pgmap v632: 148 pgs: 148 active+clean; 1.5 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 915 op/s 2024-09-13T13:13:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[36682]: 6.3 scrub starts 2024-09-13T13:13:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[36682]: 6.3 scrub ok 2024-09-13T13:13:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[36682]: from='client.36041 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:13:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[36682]: osdmap e332: 8 total, 8 up, 8 in 2024-09-13T13:13:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[36682]: 6.0 scrub starts 2024-09-13T13:13:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[36682]: 6.0 scrub ok 2024-09-13T13:13:53.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4082455487' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:13:53.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[29471]: pgmap v632: 148 pgs: 148 active+clean; 1.5 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 915 op/s 2024-09-13T13:13:53.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[29471]: 6.3 scrub starts 2024-09-13T13:13:53.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[29471]: 6.3 scrub ok 2024-09-13T13:13:53.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[29471]: from='client.36041 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:13:53.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[29471]: osdmap e332: 8 total, 8 up, 8 in 2024-09-13T13:13:53.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[29471]: 6.0 scrub starts 2024-09-13T13:13:53.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[29471]: 6.0 scrub ok 2024-09-13T13:13:53.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4082455487' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:13:53.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:54.013 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:54.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:54.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:54.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:53 smithi167 ceph-mon[33178]: pgmap v632: 148 pgs: 148 active+clean; 1.5 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 915 op/s 2024-09-13T13:13:54.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:53 smithi167 ceph-mon[33178]: 6.3 scrub starts 2024-09-13T13:13:54.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:53 smithi167 ceph-mon[33178]: 6.3 scrub ok 2024-09-13T13:13:54.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:53 smithi167 ceph-mon[33178]: from='client.36041 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:13:54.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:53 smithi167 ceph-mon[33178]: osdmap e332: 8 total, 8 up, 8 in 2024-09-13T13:13:54.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:53 smithi167 ceph-mon[33178]: 6.0 scrub starts 2024-09-13T13:13:54.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:53 smithi167 ceph-mon[33178]: 6.0 scrub ok 2024-09-13T13:13:54.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:53 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4082455487' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:13:54.075 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:13:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:13:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:13:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:13:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190217 9.c 9:3da1d190:::benchmark_data_smithi052_128427_object59548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:13:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:43.982184+0000", 2024-09-13T13:13:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 69.306023952999993, 2024-09-13T13:13:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99036358000000002, 2024-09-13T13:13:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190217 2024-09-13T13:13:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982184+0000", 2024-09-13T13:13:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982184+0000", 2024-09-13T13:13:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982185+0000", 2024-09-13T13:13:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5859999999999999e-06 2024-09-13T13:13:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982190+0000", 2024-09-13T13:13:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9749999999999998e-06 2024-09-13T13:13:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982200+0000", 2024-09-13T13:13:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2310000000000002e-06 2024-09-13T13:13:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982209+0000", 2024-09-13T13:13:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9220000000000003e-06 2024-09-13T13:13:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982540+0000", 2024-09-13T13:13:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033174499999999999 2024-09-13T13:13:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982550+0000", 2024-09-13T13:13:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5629999999999998e-06 2024-09-13T13:13:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.966848+0000", 2024-09-13T13:13:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.9842978 2024-09-13T13:13:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.966978+0000", 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013044200000000001 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.967121+0000", 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000143096 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972248+0000", 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051265260000000002 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972262+0000", 2024-09-13T13:13:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4037e-05 2024-09-13T13:13:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972452+0000", 2024-09-13T13:13:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019067699999999999 2024-09-13T13:13:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972517+0000", 2024-09-13T13:13:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4935000000000001e-05 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972547+0000", 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0045000000000001e-05 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190219 9.c 9:30476813:::benchmark_data_smithi052_128427_object59550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:43.982580+0000", 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 69.305627810999994, 2024-09-13T13:13:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.990151738, 2024-09-13T13:13:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190219 2024-09-13T13:13:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982580+0000", 2024-09-13T13:13:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982580+0000", 2024-09-13T13:13:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982581+0000", 2024-09-13T13:13:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0249999999999999e-06 2024-09-13T13:13:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982585+0000", 2024-09-13T13:13:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1189999999999999e-06 2024-09-13T13:13:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982595+0000", 2024-09-13T13:13:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0074999999999999e-05 2024-09-13T13:13:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982601+0000", 2024-09-13T13:13:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2149999999999998e-06 2024-09-13T13:13:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982868+0000", 2024-09-13T13:13:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00026633000000000001 2024-09-13T13:13:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982876+0000", 2024-09-13T13:13:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8900000000000007e-06 2024-09-13T13:13:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.969658+0000", 2024-09-13T13:13:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98678257999999996 2024-09-13T13:13:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.969792+0000", 2024-09-13T13:13:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013425500000000001 2024-09-13T13:13:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:13:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.969926+0000", 2024-09-13T13:13:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013390200000000001 2024-09-13T13:13:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972556+0000", 2024-09-13T13:13:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026292170000000001 2024-09-13T13:13:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972600+0000", 2024-09-13T13:13:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4697999999999998e-05 2024-09-13T13:13:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972657+0000", 2024-09-13T13:13:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6453000000000003e-05 2024-09-13T13:13:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972711+0000", 2024-09-13T13:13:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.452e-05 2024-09-13T13:13:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972732+0000", 2024-09-13T13:13:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0458999999999998e-05 2024-09-13T13:13:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190252 9.c 9:3916637a:::benchmark_data_smithi052_128427_object59583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.013719+0000", 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 69.274489084999999, 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96423604299999999, 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190252 2024-09-13T13:13:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013719+0000", 2024-09-13T13:13:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013719+0000", 2024-09-13T13:13:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013720+0000", 2024-09-13T13:13:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7039999999999999e-06 2024-09-13T13:13:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013726+0000", 2024-09-13T13:13:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3299999999999998e-06 2024-09-13T13:13:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013745+0000", 2024-09-13T13:13:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9623000000000002e-05 2024-09-13T13:13:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013753+0000", 2024-09-13T13:13:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.498e-06 2024-09-13T13:13:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.014204+0000", 2024-09-13T13:13:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00045132700000000001 2024-09-13T13:13:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.014212+0000", 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1510000000000004e-06 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972762+0000", 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.958549706 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972843+0000", 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1415000000000006e-05 2024-09-13T13:13:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:13:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972915+0000", 2024-09-13T13:13:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:13:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976630+0000", 2024-09-13T13:13:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037151879999999999 2024-09-13T13:13:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976739+0000", 2024-09-13T13:13:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109011 2024-09-13T13:13:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.977890+0000", 2024-09-13T13:13:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011513350000000001 2024-09-13T13:13:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.977916+0000", 2024-09-13T13:13:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5948e-05 2024-09-13T13:13:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.977955+0000", 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8451999999999997e-05 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190263 9.c 9:307a9428:::benchmark_data_smithi052_128427_object59594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.022090+0000", 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 69.266118156000005, 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95977093599999996, 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190263 2024-09-13T13:13:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022090+0000", 2024-09-13T13:13:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022090+0000", 2024-09-13T13:13:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022092+0000", 2024-09-13T13:13:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8870000000000001e-06 2024-09-13T13:13:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022097+0000", 2024-09-13T13:13:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.31e-06 2024-09-13T13:13:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022108+0000", 2024-09-13T13:13:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0941000000000001e-05 2024-09-13T13:13:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022115+0000", 2024-09-13T13:13:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3050000000000004e-06 2024-09-13T13:13:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022210+0000", 2024-09-13T13:13:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5169999999999999e-05 2024-09-13T13:13:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022219+0000", 2024-09-13T13:13:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1500000000000005e-06 2024-09-13T13:13:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.974786+0000", 2024-09-13T13:13:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95256630799999997 2024-09-13T13:13:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.974884+0000", 2024-09-13T13:13:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8256000000000003e-05 2024-09-13T13:13:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:13:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.975003+0000", 2024-09-13T13:13:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011911100000000001 2024-09-13T13:13:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976679+0000", 2024-09-13T13:13:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016762019999999999 2024-09-13T13:13:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.978004+0000", 2024-09-13T13:13:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001324605 2024-09-13T13:13:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981764+0000", 2024-09-13T13:13:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003760556 2024-09-13T13:13:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981816+0000", 2024-09-13T13:13:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1412999999999997e-05 2024-09-13T13:13:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981861+0000", 2024-09-13T13:13:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4722000000000003e-05 2024-09-13T13:13:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190265 9.c 9:3c2318c0:::benchmark_data_smithi052_128427_object59596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:13:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.022643+0000", 2024-09-13T13:13:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 69.265565097000007, 2024-09-13T13:13:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95946595400000001, 2024-09-13T13:13:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190265 2024-09-13T13:13:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022643+0000", 2024-09-13T13:13:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022643+0000", 2024-09-13T13:13:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022644+0000", 2024-09-13T13:13:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.282e-06 2024-09-13T13:13:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022648+0000", 2024-09-13T13:13:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3189999999999996e-06 2024-09-13T13:13:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022657+0000", 2024-09-13T13:13:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1710000000000008e-06 2024-09-13T13:13:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022664+0000", 2024-09-13T13:13:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8600000000000004e-06 2024-09-13T13:13:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022843+0000", 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017918 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022851+0000", 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5970000000000003e-06 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976779+0000", 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95392760200000004 2024-09-13T13:13:54.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976846+0000", 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7539000000000005e-05 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976913+0000", 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6785999999999994e-05 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981913+0000", 2024-09-13T13:13:54.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050001179999999996 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981978+0000", 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4917000000000002e-05 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982009+0000", 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0499999999999999e-05 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982067+0000", 2024-09-13T13:13:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8573000000000003e-05 2024-09-13T13:13:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982109+0000", 2024-09-13T13:13:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1510000000000001e-05 2024-09-13T13:13:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190292 9.c 9:3187c4a7:::benchmark_data_smithi052_128427_object59623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:13:54.113 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.039937+0000", 2024-09-13T13:13:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 69.248271145000004, 2024-09-13T13:13:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95233577000000003, 2024-09-13T13:13:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190292 2024-09-13T13:13:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039937+0000", 2024-09-13T13:13:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039937+0000", 2024-09-13T13:13:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039938+0000", 2024-09-13T13:13:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.776e-06 2024-09-13T13:13:54.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039944+0000", 2024-09-13T13:13:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5949999999999998e-06 2024-09-13T13:13:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039959+0000", 2024-09-13T13:13:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4854e-05 2024-09-13T13:13:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039966+0000", 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2239999999999998e-06 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.040018+0000", 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1578e-05 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.040025+0000", 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6690000000000006e-06 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982498+0000", 2024-09-13T13:13:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.94247312100000002 2024-09-13T13:13:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982590+0000", 2024-09-13T13:13:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1283e-05 2024-09-13T13:13:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:13:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982692+0000", 2024-09-13T13:13:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102144 2024-09-13T13:13:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.987375+0000", 2024-09-13T13:13:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0046834329999999999 2024-09-13T13:13:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.987419+0000", 2024-09-13T13:13:54.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3282999999999999e-05 2024-09-13T13:13:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992172+0000", 2024-09-13T13:13:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047538839999999999 2024-09-13T13:13:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992231+0000", 2024-09-13T13:13:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8329999999999999e-05 2024-09-13T13:13:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992272+0000", 2024-09-13T13:13:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1596e-05 2024-09-13T13:13:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190295 9.c 9:3895a0c0:::benchmark_data_smithi052_128427_object59626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:13:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.041745+0000", 2024-09-13T13:13:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 69.246462647000001, 2024-09-13T13:13:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96494661000000004, 2024-09-13T13:13:54.121 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190295 2024-09-13T13:13:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041745+0000", 2024-09-13T13:13:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041745+0000", 2024-09-13T13:13:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041746+0000", 2024-09-13T13:13:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2619999999999999e-06 2024-09-13T13:13:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041751+0000", 2024-09-13T13:13:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.216e-06 2024-09-13T13:13:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041759+0000", 2024-09-13T13:13:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7129999999999994e-06 2024-09-13T13:13:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041805+0000", 2024-09-13T13:13:54.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5739999999999999e-05 2024-09-13T13:13:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041899+0000", 2024-09-13T13:13:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3598999999999996e-05 2024-09-13T13:13:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041907+0000", 2024-09-13T13:13:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4169999999999992e-06 2024-09-13T13:13:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.984980+0000", 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.94307334399999998 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.985078+0000", 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7633000000000005e-05 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.985188+0000", 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110245 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989611+0000", 2024-09-13T13:13:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0044224889999999999 2024-09-13T13:13:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.994760+0000", 2024-09-13T13:13:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051493700000000003 2024-09-13T13:13:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.006626+0000", 2024-09-13T13:13:54.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011865504000000001 2024-09-13T13:13:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.006651+0000", 2024-09-13T13:13:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5629000000000001e-05 2024-09-13T13:13:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.006692+0000", 2024-09-13T13:13:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0448999999999998e-05 2024-09-13T13:13:54.128 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190317 9.c 9:34b3085e:::benchmark_data_smithi052_128427_object59648:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:13:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.058087+0000", 2024-09-13T13:13:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 69.230121056000002, 2024-09-13T13:13:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95460188400000001, 2024-09-13T13:13:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.129 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190317 2024-09-13T13:13:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058087+0000", 2024-09-13T13:13:54.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058087+0000", 2024-09-13T13:13:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058088+0000", 2024-09-13T13:13:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.192e-06 2024-09-13T13:13:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058093+0000", 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7380000000000002e-06 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058102+0000", 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6590000000000002e-06 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058109+0000", 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9770000000000003e-06 2024-09-13T13:13:54.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058175+0000", 2024-09-13T13:13:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5343000000000001e-05 2024-09-13T13:13:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058182+0000", 2024-09-13T13:13:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8059999999999995e-06 2024-09-13T13:13:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989661+0000", 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.931478994 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989768+0000", 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010650900000000001 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989877+0000", 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109343 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.134 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.994834+0000", 2024-09-13T13:13:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049566610000000002 2024-09-13T13:13:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.994870+0000", 2024-09-13T13:13:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6399999999999997e-05 2024-09-13T13:13:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.012452+0000", 2024-09-13T13:13:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.017581513999999999 2024-09-13T13:13:54.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.012635+0000", 2024-09-13T13:13:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018336900000000001 2024-09-13T13:13:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.012689+0000", 2024-09-13T13:13:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3378999999999998e-05 2024-09-13T13:13:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.136 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190320 9.c 9:35ad6b23:::benchmark_data_smithi052_128427_object59651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:13:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.059653+0000", 2024-09-13T13:13:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 69.228554884999994, 2024-09-13T13:13:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95393949300000003, 2024-09-13T13:13:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.137 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190320 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059653+0000", 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059653+0000", 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059654+0000", 2024-09-13T13:13:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1370000000000001e-06 2024-09-13T13:13:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059658+0000", 2024-09-13T13:13:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.755e-06 2024-09-13T13:13:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059668+0000", 2024-09-13T13:13:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8709999999999992e-06 2024-09-13T13:13:54.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059675+0000", 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5870000000000004e-06 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059730+0000", 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5180999999999999e-05 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059739+0000", 2024-09-13T13:13:54.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0569999999999999e-06 2024-09-13T13:13:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992325+0000", 2024-09-13T13:13:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.93258522899999996 2024-09-13T13:13:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992426+0000", 2024-09-13T13:13:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101094 2024-09-13T13:13:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.141 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:13:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992530+0000", 2024-09-13T13:13:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104051 2024-09-13T13:13:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.999704+0000", 2024-09-13T13:13:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0071743629999999996 2024-09-13T13:13:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.004507+0000", 2024-09-13T13:13:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048030340000000003 2024-09-13T13:13:54.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.013507+0000", 2024-09-13T13:13:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0089995439999999999 2024-09-13T13:13:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.013555+0000", 2024-09-13T13:13:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8572000000000002e-05 2024-09-13T13:13:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.013592+0000", 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7017999999999997e-05 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202437 9.c 9:31d379dd:::benchmark_data_smithi052_128427_object71768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.936285+0000", 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 55.351922997999999, 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0668128299999999, 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:13:54.144 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202437 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936285+0000", 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936285+0000", 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936286+0000", 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5540000000000001e-06 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936291+0000", 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0100000000000003e-06 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936305+0000", 2024-09-13T13:13:54.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3562e-05 2024-09-13T13:13:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936312+0000", 2024-09-13T13:13:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3780000000000003e-06 2024-09-13T13:13:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936374+0000", 2024-09-13T13:13:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1909999999999995e-05 2024-09-13T13:13:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936528+0000", 2024-09-13T13:13:54.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015417 2024-09-13T13:13:54.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4,7", 2024-09-13T13:13:54.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936642+0000", 2024-09-13T13:13:54.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011353000000000001 2024-09-13T13:13:54.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.148 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.148 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.941575+0000", 2024-09-13T13:13:54.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049326969999999998 2024-09-13T13:13:54.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.947466+0000", 2024-09-13T13:13:54.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058910960000000002 2024-09-13T13:13:54.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.947475+0000", 2024-09-13T13:13:54.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6350000000000001e-06 2024-09-13T13:13:54.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.990524+0000", 2024-09-13T13:13:54.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0430484790000001 2024-09-13T13:13:54.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for missing object", 2024-09-13T13:13:54.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.990633+0000", 2024-09-13T13:13:54.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108872 2024-09-13T13:13:54.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003024+0000", 2024-09-13T13:13:54.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.012390988 2024-09-13T13:13:54.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003098+0000", 2024-09-13T13:13:54.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3949000000000006e-05 2024-09-13T13:13:54.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.151 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.151 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202453 9.c 9:3d8e06d1:::benchmark_data_smithi052_128427_object71784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:13:54.151 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.956183+0000", 2024-09-13T13:13:54.152 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 55.332024777000001, 2024-09-13T13:13:54.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.056296729, 2024-09-13T13:13:54.152 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.152 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.152 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.152 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.152 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.152 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.152 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202453 2024-09-13T13:13:54.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.153 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956183+0000", 2024-09-13T13:13:54.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956183+0000", 2024-09-13T13:13:54.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956185+0000", 2024-09-13T13:13:54.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.784e-06 2024-09-13T13:13:54.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956190+0000", 2024-09-13T13:13:54.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:13:54.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956200+0000", 2024-09-13T13:13:54.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2900000000000008e-06 2024-09-13T13:13:54.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956206+0000", 2024-09-13T13:13:54.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1489999999999999e-06 2024-09-13T13:13:54.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957496+0000", 2024-09-13T13:13:54.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012905519999999999 2024-09-13T13:13:54.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957505+0000", 2024-09-13T13:13:54.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9930000000000001e-06 2024-09-13T13:13:54.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003310+0000", 2024-09-13T13:13:54.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.045804368 2024-09-13T13:13:54.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003406+0000", 2024-09-13T13:13:54.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6335000000000005e-05 2024-09-13T13:13:54.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:13:54.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003506+0000", 2024-09-13T13:13:54.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000100295 2024-09-13T13:13:54.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012143+0000", 2024-09-13T13:13:54.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0086370389999999991 2024-09-13T13:13:54.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012233+0000", 2024-09-13T13:13:54.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9973000000000001e-05 2024-09-13T13:13:54.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012279+0000", 2024-09-13T13:13:54.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5966e-05 2024-09-13T13:13:54.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012376+0000", 2024-09-13T13:13:54.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6220999999999999e-05 2024-09-13T13:13:54.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012480+0000", 2024-09-13T13:13:54.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010412900000000001 2024-09-13T13:13:54.160 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.160 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.160 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.160 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202455 9.c 9:304f4878:::benchmark_data_smithi052_128427_object71786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:13:54.160 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:13:54.160 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 55.331989870000001, 2024-09-13T13:13:54.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.071265975, 2024-09-13T13:13:54.161 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.161 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.161 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.161 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.161 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.161 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.161 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202455 2024-09-13T13:13:54.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.161 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:13:54.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:13:54.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:13:54.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5000000000000003e-07 2024-09-13T13:13:54.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956221+0000", 2024-09-13T13:13:54.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.441e-06 2024-09-13T13:13:54.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956226+0000", 2024-09-13T13:13:54.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0089999999999998e-06 2024-09-13T13:13:54.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956230+0000", 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5410000000000002e-06 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957529+0000", 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012988469999999999 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957535+0000", 2024-09-13T13:13:54.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8109999999999999e-06 2024-09-13T13:13:54.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.005874+0000", 2024-09-13T13:13:54.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0483387770000001 2024-09-13T13:13:54.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.005991+0000", 2024-09-13T13:13:54.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116669 2024-09-13T13:13:54.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:13:54.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.006107+0000", 2024-09-13T13:13:54.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011618 2024-09-13T13:13:54.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012522+0000", 2024-09-13T13:13:54.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064157060000000002 2024-09-13T13:13:54.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012562+0000", 2024-09-13T13:13:54.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9756999999999999e-05 2024-09-13T13:13:54.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.026940+0000", 2024-09-13T13:13:54.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014377463 2024-09-13T13:13:54.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027300+0000", 2024-09-13T13:13:54.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00036073000000000002 2024-09-13T13:13:54.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027484+0000", 2024-09-13T13:13:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000183494 2024-09-13T13:13:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202464 9.c 9:3602984d:::benchmark_data_smithi052_128427_object71795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:13:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:13:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 55.326923970000003, 2024-09-13T13:13:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.066828723, 2024-09-13T13:13:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202464 2024-09-13T13:13:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:13:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:13:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:13:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4099999999999999e-07 2024-09-13T13:13:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961286+0000", 2024-09-13T13:13:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2010000000000002e-06 2024-09-13T13:13:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961291+0000", 2024-09-13T13:13:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5909999999999998e-06 2024-09-13T13:13:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961295+0000", 2024-09-13T13:13:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1110000000000001e-06 2024-09-13T13:13:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.962375+0000", 2024-09-13T13:13:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00107939 2024-09-13T13:13:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.962382+0000", 2024-09-13T13:13:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5900000000000002e-06 2024-09-13T13:13:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.008652+0000", 2024-09-13T13:13:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0462698880000001 2024-09-13T13:13:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.008749+0000", 2024-09-13T13:13:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7125000000000003e-05 2024-09-13T13:13:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:13:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.008858+0000", 2024-09-13T13:13:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010874699999999999 2024-09-13T13:13:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012711+0000", 2024-09-13T13:13:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0038533619999999999 2024-09-13T13:13:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012755+0000", 2024-09-13T13:13:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3773000000000003e-05 2024-09-13T13:13:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027802+0000", 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015047115 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027905+0000", 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103278 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028113+0000", 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000207111 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202480 9.c 9:3e2ac5d3:::benchmark_data_smithi052_128427_object71811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:13:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.976129+0000", 2024-09-13T13:13:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 55.312078970000002, 2024-09-13T13:13:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.052223253, 2024-09-13T13:13:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202480 2024-09-13T13:13:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976129+0000", 2024-09-13T13:13:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976129+0000", 2024-09-13T13:13:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976130+0000", 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3799999999999999e-06 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976135+0000", 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2809999999999999e-06 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976147+0000", 2024-09-13T13:13:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1182000000000001e-05 2024-09-13T13:13:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976154+0000", 2024-09-13T13:13:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4440000000000002e-06 2024-09-13T13:13:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976219+0000", 2024-09-13T13:13:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4414999999999999e-05 2024-09-13T13:13:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976227+0000", 2024-09-13T13:13:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:13:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012784+0000", 2024-09-13T13:13:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.036556815 2024-09-13T13:13:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012892+0000", 2024-09-13T13:13:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010768999999999999 2024-09-13T13:13:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:13:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.013010+0000", 2024-09-13T13:13:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118244 2024-09-13T13:13:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.018728+0000", 2024-09-13T13:13:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057179079999999998 2024-09-13T13:13:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.018768+0000", 2024-09-13T13:13:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.985e-05 2024-09-13T13:13:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028119+0000", 2024-09-13T13:13:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0093518460000000005 2024-09-13T13:13:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028244+0000", 2024-09-13T13:13:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012501799999999999 2024-09-13T13:13:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028352+0000", 2024-09-13T13:13:54.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107557 2024-09-13T13:13:54.183 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.183 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.183 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.183 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202488 9.c 9:3231b30e:::benchmark_data_smithi052_128427_object71819:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:13:54.184 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.980401+0000", 2024-09-13T13:13:54.184 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 55.307806497000001, 2024-09-13T13:13:54.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0481403709999999, 2024-09-13T13:13:54.184 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.184 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.184 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.184 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.184 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.184 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202488 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980401+0000", 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980401+0000", 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980403+0000", 2024-09-13T13:13:54.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.339e-06 2024-09-13T13:13:54.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980408+0000", 2024-09-13T13:13:54.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1909999999999999e-06 2024-09-13T13:13:54.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980421+0000", 2024-09-13T13:13:54.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3267e-05 2024-09-13T13:13:54.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980428+0000", 2024-09-13T13:13:54.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0310000000000001e-06 2024-09-13T13:13:54.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980485+0000", 2024-09-13T13:13:54.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6527999999999998e-05 2024-09-13T13:13:54.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980493+0000", 2024-09-13T13:13:54.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2460000000000003e-06 2024-09-13T13:13:54.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.015559+0000", 2024-09-13T13:13:54.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0350661029999999 2024-09-13T13:13:54.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.015857+0000", 2024-09-13T13:13:54.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000297726 2024-09-13T13:13:54.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:13:54.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.016037+0000", 2024-09-13T13:13:54.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018051499999999999 2024-09-13T13:13:54.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.021797+0000", 2024-09-13T13:13:54.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057596000000000001 2024-09-13T13:13:54.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.021836+0000", 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9180000000000001e-05 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028359+0000", 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0065231589999999997 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028449+0000", 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9874e-05 2024-09-13T13:13:54.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028542+0000", 2024-09-13T13:13:54.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2612e-05 2024-09-13T13:13:54.191 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.191 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.191 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.191 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202524 9.c 9:322e12cc:::benchmark_data_smithi052_128427_object71855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:13:54.191 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:58.008117+0000", 2024-09-13T13:13:54.191 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 55.280091224000003, 2024-09-13T13:13:54.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0206121399999999, 2024-09-13T13:13:54.192 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.192 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.192 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.192 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.192 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.192 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.192 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202524 2024-09-13T13:13:54.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.192 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.193 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008117+0000", 2024-09-13T13:13:54.193 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.193 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008117+0000", 2024-09-13T13:13:54.193 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.193 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008118+0000", 2024-09-13T13:13:54.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3200000000000001e-06 2024-09-13T13:13:54.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.194 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008123+0000", 2024-09-13T13:13:54.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8189999999999999e-06 2024-09-13T13:13:54.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.194 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008132+0000", 2024-09-13T13:13:54.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6979999999999993e-06 2024-09-13T13:13:54.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008141+0000", 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7620000000000001e-06 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008199+0000", 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7942999999999997e-05 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008208+0000", 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1069999999999994e-06 2024-09-13T13:13:54.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.018947+0000", 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0107389179999999 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.019046+0000", 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8896999999999999e-05 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.019147+0000", 2024-09-13T13:13:54.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101001 2024-09-13T13:13:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.021878+0000", 2024-09-13T13:13:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002730985 2024-09-13T13:13:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.026889+0000", 2024-09-13T13:13:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050107529999999997 2024-09-13T13:13:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028548+0000", 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016596950000000001 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028637+0000", 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8687e-05 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028729+0000", 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1555e-05 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.198 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.199 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.199 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:252968 9.1 9:8984707c:::benchmark_data_smithi052_128427_object122299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-13T13:13:54.199 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:41.446602+0000", 2024-09-13T13:13:54.199 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 11.841605861, 2024-09-13T13:13:54.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.077841553999999993, 2024-09-13T13:13:54.199 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.199 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.199 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.199 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.199 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.199 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 252968 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446602+0000", 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446602+0000", 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.201 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.201 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446616+0000", 2024-09-13T13:13:54.201 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4532000000000001e-05 2024-09-13T13:13:54.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.201 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.201 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446629+0000", 2024-09-13T13:13:54.201 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2934e-05 2024-09-13T13:13:54.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.201 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.201 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446646+0000", 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6416999999999999e-05 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446660+0000", 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4589999999999999e-05 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446783+0000", 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000122552 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446893+0000", 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110443 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.447025+0000", 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013190500000000001 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.515913+0000", 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.068887970000000007 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.516107+0000", 2024-09-13T13:13:54.204 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000194227 2024-09-13T13:13:54.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.204 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.204 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.524303+0000", 2024-09-13T13:13:54.204 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0081953330000000008 2024-09-13T13:13:54.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.204 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.204 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.524400+0000", 2024-09-13T13:13:54.204 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7317000000000004e-05 2024-09-13T13:13:54.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.205 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.205 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.524443+0000", 2024-09-13T13:13:54.205 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3334000000000002e-05 2024-09-13T13:13:54.205 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.205 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.205 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.205 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:252969 9.2 9:4692ee92:::benchmark_data_smithi052_128427_object122300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-13T13:13:54.205 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:41.446682+0000", 2024-09-13T13:13:54.205 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 11.841525912, 2024-09-13T13:13:54.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.076174663000000004, 2024-09-13T13:13:54.206 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.206 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.206 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.206 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.206 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.206 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.206 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 252969 2024-09-13T13:13:54.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.206 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446682+0000", 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446682+0000", 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446683+0000", 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0920000000000001e-06 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446687+0000", 2024-09-13T13:13:54.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6550000000000002e-06 2024-09-13T13:13:54.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446693+0000", 2024-09-13T13:13:54.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8759999999999999e-06 2024-09-13T13:13:54.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446699+0000", 2024-09-13T13:13:54.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8069999999999996e-06 2024-09-13T13:13:54.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446894+0000", 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000195119 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.447023+0000", 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000128885 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.447136+0000", 2024-09-13T13:13:54.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112732 2024-09-13T13:13:54.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.210 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.210 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.520247+0000", 2024-09-13T13:13:54.210 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.073111329000000003 2024-09-13T13:13:54.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.210 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.210 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.521136+0000", 2024-09-13T13:13:54.210 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00088822599999999999 2024-09-13T13:13:54.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.211 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.211 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.522752+0000", 2024-09-13T13:13:54.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001616417 2024-09-13T13:13:54.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.211 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.211 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.522811+0000", 2024-09-13T13:13:54.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8539999999999999e-05 2024-09-13T13:13:54.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.522857+0000", 2024-09-13T13:13:54.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5985000000000001e-05 2024-09-13T13:13:54.212 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.212 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.212 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.212 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:252972 9.1 9:8254345c:::benchmark_data_smithi052_128427_object122303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-13T13:13:54.212 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:41.446719+0000", 2024-09-13T13:13:54.212 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 11.841488761000001, 2024-09-13T13:13:54.213 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.077973789000000002, 2024-09-13T13:13:54.213 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.213 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.213 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.213 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.213 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.213 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.213 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 252972 2024-09-13T13:13:54.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.213 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.214 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.214 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446719+0000", 2024-09-13T13:13:54.214 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.214 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.214 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446719+0000", 2024-09-13T13:13:54.214 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.215 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446720+0000", 2024-09-13T13:13:54.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4e-07 2024-09-13T13:13:54.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.215 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446724+0000", 2024-09-13T13:13:54.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7979999999999999e-06 2024-09-13T13:13:54.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.215 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446730+0000", 2024-09-13T13:13:54.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7309999999999996e-06 2024-09-13T13:13:54.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446736+0000", 2024-09-13T13:13:54.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8479999999999997e-06 2024-09-13T13:13:54.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.512726+0000", 2024-09-13T13:13:54.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.065989347000000004 2024-09-13T13:13:54.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.217 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.217 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.217 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.512847+0000", 2024-09-13T13:13:54.217 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121556 2024-09-13T13:13:54.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.217 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.217 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:13:54.217 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.512984+0000", 2024-09-13T13:13:54.217 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013671700000000001 2024-09-13T13:13:54.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.217 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.218 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.516148+0000", 2024-09-13T13:13:54.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031641159999999998 2024-09-13T13:13:54.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.218 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.524454+0000", 2024-09-13T13:13:54.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0083055460000000005 2024-09-13T13:13:54.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.524612+0000", 2024-09-13T13:13:54.219 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015815600000000001 2024-09-13T13:13:54.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.219 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.524658+0000", 2024-09-13T13:13:54.219 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6555999999999999e-05 2024-09-13T13:13:54.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.219 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.524693+0000", 2024-09-13T13:13:54.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4578e-05 2024-09-13T13:13:54.220 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.220 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.220 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.220 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:252976 9.c 9:358b0c08:::benchmark_data_smithi052_128427_object122307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-13T13:13:54.220 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:41.447521+0000", 2024-09-13T13:13:54.220 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 11.840686452, 2024-09-13T13:13:54.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.076000634999999997, 2024-09-13T13:13:54.220 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:13:54.221 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:13:54.221 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:54.221 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:13:54.221 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:13:54.221 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:54.221 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 252976 2024-09-13T13:13:54.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.221 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:13:54.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.221 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:13:54.221 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.447521+0000", 2024-09-13T13:13:54.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.222 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:13:54.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.447521+0000", 2024-09-13T13:13:54.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:13:54.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.222 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:13:54.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.447522+0000", 2024-09-13T13:13:54.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.079e-06 2024-09-13T13:13:54.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.223 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:13:54.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.447527+0000", 2024-09-13T13:13:54.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1320000000000004e-06 2024-09-13T13:13:54.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.223 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:13:54.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.509607+0000", 2024-09-13T13:13:54.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.062080784 2024-09-13T13:13:54.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.224 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:13:54.224 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.509686+0000", 2024-09-13T13:13:54.224 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8917000000000004e-05 2024-09-13T13:13:54.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.224 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:13:54.224 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.509839+0000", 2024-09-13T13:13:54.224 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000152719 2024-09-13T13:13:54.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.225 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:13:54.225 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.510002+0000", 2024-09-13T13:13:54.225 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016316799999999999 2024-09-13T13:13:54.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.225 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:13:54.225 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.510123+0000", 2024-09-13T13:13:54.225 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012103399999999999 2024-09-13T13:13:54.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.226 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.226 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.523258+0000", 2024-09-13T13:13:54.226 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.013135262999999999 2024-09-13T13:13:54.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.226 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:54.226 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.523312+0000", 2024-09-13T13:13:54.226 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3931999999999998e-05 2024-09-13T13:13:54.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.227 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:13:54.227 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.523441+0000", 2024-09-13T13:13:54.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000129014 2024-09-13T13:13:54.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.227 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:13:54.227 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.523485+0000", 2024-09-13T13:13:54.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3609000000000002e-05 2024-09-13T13:13:54.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:13:54.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:13:54.227 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:13:54.228 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.523522+0000", 2024-09-13T13:13:54.228 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6983999999999997e-05 2024-09-13T13:13:54.228 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.228 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.228 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.228 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:13:54.228 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:13:54.228 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:13:54.228 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:54.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:54.251 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:54.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:54.352 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:54.377 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:54.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:54.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:54.499 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:54.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:54.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:54.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:13:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:13:54.602 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:54.628 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:54.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:54.730 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:54.755 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:54.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:54.857 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:54.879 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:54.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:54.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:54.979 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4082455487' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:13:54.979 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[36682]: osdmap e333: 8 total, 8 up, 8 in 2024-09-13T13:13:54.979 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[36682]: 3.19 scrub starts 2024-09-13T13:13:54.979 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[36682]: 3.19 scrub ok 2024-09-13T13:13:54.979 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[36682]: 5.1e scrub starts 2024-09-13T13:13:54.979 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[36682]: 5.1e scrub ok 2024-09-13T13:13:54.979 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[36682]: 5.1a scrub starts 2024-09-13T13:13:54.979 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[36682]: 5.1a scrub ok 2024-09-13T13:13:54.979 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[36682]: 3.7 scrub starts 2024-09-13T13:13:54.979 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[36682]: 3.7 scrub ok 2024-09-13T13:13:54.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4082455487' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:13:54.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[29471]: osdmap e333: 8 total, 8 up, 8 in 2024-09-13T13:13:54.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[29471]: 3.19 scrub starts 2024-09-13T13:13:54.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[29471]: 3.19 scrub ok 2024-09-13T13:13:54.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[29471]: 5.1e scrub starts 2024-09-13T13:13:54.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[29471]: 5.1e scrub ok 2024-09-13T13:13:54.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[29471]: 5.1a scrub starts 2024-09-13T13:13:54.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[29471]: 5.1a scrub ok 2024-09-13T13:13:54.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[29471]: 3.7 scrub starts 2024-09-13T13:13:54.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:54 smithi052 ceph-mon[29471]: 3.7 scrub ok 2024-09-13T13:13:54.981 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:55.002 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:55.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:54 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4082455487' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:13:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:54 smithi167 ceph-mon[33178]: osdmap e333: 8 total, 8 up, 8 in 2024-09-13T13:13:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:54 smithi167 ceph-mon[33178]: 3.19 scrub starts 2024-09-13T13:13:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:54 smithi167 ceph-mon[33178]: 3.19 scrub ok 2024-09-13T13:13:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:54 smithi167 ceph-mon[33178]: 5.1e scrub starts 2024-09-13T13:13:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:54 smithi167 ceph-mon[33178]: 5.1e scrub ok 2024-09-13T13:13:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:54 smithi167 ceph-mon[33178]: 5.1a scrub starts 2024-09-13T13:13:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:54 smithi167 ceph-mon[33178]: 5.1a scrub ok 2024-09-13T13:13:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:54 smithi167 ceph-mon[33178]: 3.7 scrub starts 2024-09-13T13:13:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:54 smithi167 ceph-mon[33178]: 3.7 scrub ok 2024-09-13T13:13:55.104 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:55.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:55.127 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:55.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:55.228 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:55.253 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:55.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:55.355 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:55.379 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:55.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:55.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:55.502 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:55.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:55.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:55.572 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:13:55.604 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:55.632 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:55.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:55.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:55.735 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:55.759 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:55.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:55.821 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:13:55.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:55.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[36682]: pgmap v635: 132 pgs: 132 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-09-13T13:13:55.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:55.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[36682]: 6.f scrub starts 2024-09-13T13:13:55.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[36682]: 6.f scrub ok 2024-09-13T13:13:55.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[36682]: 5.0 scrub starts 2024-09-13T13:13:55.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[36682]: 5.0 scrub ok 2024-09-13T13:13:55.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[36682]: 6.a scrub starts 2024-09-13T13:13:55.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[36682]: 6.a scrub ok 2024-09-13T13:13:55.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[29471]: pgmap v635: 132 pgs: 132 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-09-13T13:13:55.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:55.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[29471]: 6.f scrub starts 2024-09-13T13:13:55.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[29471]: 6.f scrub ok 2024-09-13T13:13:55.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[29471]: 5.0 scrub starts 2024-09-13T13:13:55.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[29471]: 5.0 scrub ok 2024-09-13T13:13:55.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[29471]: 6.a scrub starts 2024-09-13T13:13:55.870 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:55 smithi052 ceph-mon[29471]: 6.a scrub ok 2024-09-13T13:13:55.883 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:55.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:55.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:55.943 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:55 smithi167 ceph-mon[33178]: pgmap v635: 132 pgs: 132 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-09-13T13:13:55.943 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:13:55.943 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:55 smithi167 ceph-mon[33178]: 6.f scrub starts 2024-09-13T13:13:55.943 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:55 smithi167 ceph-mon[33178]: 6.f scrub ok 2024-09-13T13:13:55.943 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:55 smithi167 ceph-mon[33178]: 5.0 scrub starts 2024-09-13T13:13:55.943 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:55 smithi167 ceph-mon[33178]: 5.0 scrub ok 2024-09-13T13:13:55.943 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:55 smithi167 ceph-mon[33178]: 6.a scrub starts 2024-09-13T13:13:55.943 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:55 smithi167 ceph-mon[33178]: 6.a scrub ok 2024-09-13T13:13:55.985 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:13:56.007 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:56.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:13:56.109 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:56.131 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:56.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:56.188 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T13:13:56.189 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T13:13:56.189 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-13T13:13:56.190 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-13T13:13:56.233 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:56.257 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:56.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:56.358 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:56.373 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:13:56.373 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:13:56.373 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-09-13T13:13:56.373 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:13:56.380 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:56.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:56.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:56.482 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:56.488 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:13:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:13:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:13:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:56.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:56.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:56.606 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:56.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:56.634 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:56.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:56.686 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:56.735 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:56.757 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:56.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:56.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:56 smithi167 ceph-mon[33178]: 5.17 scrub starts 2024-09-13T13:13:56.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:56 smithi167 ceph-mon[33178]: 5.17 scrub ok 2024-09-13T13:13:56.859 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:56.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:56 smithi052 ceph-mon[36682]: 5.17 scrub starts 2024-09-13T13:13:56.869 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:56 smithi052 ceph-mon[36682]: 5.17 scrub ok 2024-09-13T13:13:56.869 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:56 smithi052 ceph-mon[29471]: 5.17 scrub starts 2024-09-13T13:13:56.869 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:56 smithi052 ceph-mon[29471]: 5.17 scrub ok 2024-09-13T13:13:56.883 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:56.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:56.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:56.985 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:57.008 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:57.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:57.110 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:57.120 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T13:13:57.134 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:57.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:57.159 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T13:13:57.236 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:57.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:57.360 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:57.384 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:57.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:57.415 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:13:57.486 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:57.508 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:57.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:57.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:57.610 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:13:57.627 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:57.631 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:57.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:57.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:13:57.662 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:13:57.733 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:57.755 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:57.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:57.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:57 smithi167 ceph-mon[33178]: pgmap v636: 132 pgs: 132 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-09-13T13:13:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:57 smithi167 ceph-mon[33178]: 4.b scrub starts 2024-09-13T13:13:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:57 smithi167 ceph-mon[33178]: 4.b scrub ok 2024-09-13T13:13:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:57 smithi167 ceph-mon[33178]: 3.13 scrub starts 2024-09-13T13:13:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:57 smithi167 ceph-mon[33178]: 3.13 scrub ok 2024-09-13T13:13:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:57 smithi167 ceph-mon[33178]: 6.d scrub starts 2024-09-13T13:13:57.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:57 smithi167 ceph-mon[33178]: 6.d scrub ok 2024-09-13T13:13:57.852 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[36682]: pgmap v636: 132 pgs: 132 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-09-13T13:13:57.852 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[36682]: 4.b scrub starts 2024-09-13T13:13:57.852 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[36682]: 4.b scrub ok 2024-09-13T13:13:57.852 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[36682]: 3.13 scrub starts 2024-09-13T13:13:57.852 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[36682]: 3.13 scrub ok 2024-09-13T13:13:57.853 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[36682]: 6.d scrub starts 2024-09-13T13:13:57.853 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[36682]: 6.d scrub ok 2024-09-13T13:13:57.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[29471]: pgmap v636: 132 pgs: 132 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-09-13T13:13:57.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[29471]: 4.b scrub starts 2024-09-13T13:13:57.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[29471]: 4.b scrub ok 2024-09-13T13:13:57.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[29471]: 3.13 scrub starts 2024-09-13T13:13:57.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[29471]: 3.13 scrub ok 2024-09-13T13:13:57.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[29471]: 6.d scrub starts 2024-09-13T13:13:57.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:57 smithi052 ceph-mon[29471]: 6.d scrub ok 2024-09-13T13:13:57.857 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:57.880 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:57.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:57.957 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'unique_pool_3' created 2024-09-13T13:13:57.981 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:57.992 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:13:57.992 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:13:57.992 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:13:57.993 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:13:57.993 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:13:58.005 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:58.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:58.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:58.128 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:58.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:58.230 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:13:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:58.253 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:58.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:13:58.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:13:58.376 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:13:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:58.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:13:58.478 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:58.502 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:58.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:58.603 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:58.626 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:13:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:58.628 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:58.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:58.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T13:13:58.729 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:58.752 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:58.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:58.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:58.854 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:58.872 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/102423899' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T13:13:58.872 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[36682]: from='client.36053 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T13:13:58.872 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[36682]: 3.6 scrub starts 2024-09-13T13:13:58.872 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[36682]: 3.6 scrub ok 2024-09-13T13:13:58.872 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[36682]: from='client.36053 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T13:13:58.872 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[36682]: osdmap e334: 8 total, 8 up, 8 in 2024-09-13T13:13:58.872 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[36682]: 4.17 scrub starts 2024-09-13T13:13:58.872 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[36682]: 4.17 scrub ok 2024-09-13T13:13:58.872 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/102423899' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T13:13:58.873 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[29471]: from='client.36053 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T13:13:58.873 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[29471]: 3.6 scrub starts 2024-09-13T13:13:58.873 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[29471]: 3.6 scrub ok 2024-09-13T13:13:58.873 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[29471]: from='client.36053 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T13:13:58.873 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[29471]: osdmap e334: 8 total, 8 up, 8 in 2024-09-13T13:13:58.873 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[29471]: 4.17 scrub starts 2024-09-13T13:13:58.873 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:58 smithi052 ceph-mon[29471]: 4.17 scrub ok 2024-09-13T13:13:58.877 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:58.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:58.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:58.921 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:13:58.953 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:13:58.979 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:59.004 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:59.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:59.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:59.025 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-13T13:13:59.041 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:58 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/102423899' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T13:13:59.042 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:58 smithi167 ceph-mon[33178]: from='client.36053 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T13:13:59.042 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:58 smithi167 ceph-mon[33178]: 3.6 scrub starts 2024-09-13T13:13:59.042 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:58 smithi167 ceph-mon[33178]: 3.6 scrub ok 2024-09-13T13:13:59.042 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:58 smithi167 ceph-mon[33178]: from='client.36053 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T13:13:59.042 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:58 smithi167 ceph-mon[33178]: osdmap e334: 8 total, 8 up, 8 in 2024-09-13T13:13:59.042 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:58 smithi167 ceph-mon[33178]: 4.17 scrub starts 2024-09-13T13:13:59.042 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:58 smithi167 ceph-mon[33178]: 4.17 scrub ok 2024-09-13T13:13:59.106 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:59.130 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:59.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:59.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:59.232 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:59.254 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:59.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:59.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:59.356 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:59.379 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:59.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:59.394 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:13:59.394 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:13:59.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:13:59.394 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:13:59.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.394 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.34781.0:155460 7.1 7:83ed900b:::benchmark_data_smithi052_71865_object34359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T13:13:59.394 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:04:34.105592+0000", 2024-09-13T13:13:59.394 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 565.28622084799997, 2024-09-13T13:13:59.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075149564000000002, 2024-09-13T13:13:59.394 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.394 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.395 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.395 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.395 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.34781", 2024-09-13T13:13:59.395 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3135988201", 2024-09-13T13:13:59.395 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 155460 2024-09-13T13:13:59.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.395 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105592+0000", 2024-09-13T13:13:59.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105592+0000", 2024-09-13T13:13:59.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105595+0000", 2024-09-13T13:13:59.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0759999999999999e-06 2024-09-13T13:13:59.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105608+0000", 2024-09-13T13:13:59.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2848e-05 2024-09-13T13:13:59.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105630+0000", 2024-09-13T13:13:59.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2146000000000001e-05 2024-09-13T13:13:59.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.105640+0000", 2024-09-13T13:13:59.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6860000000000001e-06 2024-09-13T13:13:59.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.106344+0000", 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070417500000000003 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.106500+0000", 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000155468 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.106631+0000", 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131137 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.109754+0000", 2024-09-13T13:13:59.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031233480000000002 2024-09-13T13:13:59.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.110324+0000", 2024-09-13T13:13:59.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000569843 2024-09-13T13:13:59.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.180592+0000", 2024-09-13T13:13:59.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070267674000000002 2024-09-13T13:13:59.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.180647+0000", 2024-09-13T13:13:59.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5067e-05 2024-09-13T13:13:59.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:04:34.180742+0000", 2024-09-13T13:13:59.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5095999999999999e-05 2024-09-13T13:13:59.400 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.400 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.400 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.401 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:34103 8.2 8:4dac86b7:::benchmark_data_smithi052_101302_object34102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T13:13:59.401 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:45.908967+0000", 2024-09-13T13:13:59.401 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 433.48284640200001, 2024-09-13T13:13:59.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073523294000000003, 2024-09-13T13:13:59.401 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.401 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.401 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.401 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.401 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:13:59.401 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:13:59.401 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 34103 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.908967+0000", 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.908967+0000", 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.908968+0000", 2024-09-13T13:13:59.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7010000000000001e-06 2024-09-13T13:13:59.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.909065+0000", 2024-09-13T13:13:59.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6872000000000004e-05 2024-09-13T13:13:59.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.909080+0000", 2024-09-13T13:13:59.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4277000000000001e-05 2024-09-13T13:13:59.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.909087+0000", 2024-09-13T13:13:59.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4039999999999998e-06 2024-09-13T13:13:59.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.910770+0000", 2024-09-13T13:13:59.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001683429 2024-09-13T13:13:59.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.910907+0000", 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000136191 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.911051+0000", 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014473399999999999 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.914088+0000", 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030367430000000002 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.914567+0000", 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00047879999999999998 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.982379+0000", 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067811689999999994 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.982442+0000", 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3378000000000002e-05 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.982490+0000", 2024-09-13T13:13:59.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8075000000000003e-05 2024-09-13T13:13:59.407 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.407 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.407 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.407 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:34104 8.2 8:47f11d8a:::benchmark_data_smithi052_101302_object34103:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T13:13:59.407 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:45.909098+0000", 2024-09-13T13:13:59.407 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 433.482715644, 2024-09-13T13:13:59.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079192135999999996, 2024-09-13T13:13:59.407 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.408 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.408 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.408 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.409 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:13:59.409 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:13:59.409 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 34104 2024-09-13T13:13:59.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.409 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.909098+0000", 2024-09-13T13:13:59.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.909098+0000", 2024-09-13T13:13:59.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.909098+0000", 2024-09-13T13:13:59.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.85e-07 2024-09-13T13:13:59.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.909133+0000", 2024-09-13T13:13:59.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4424000000000001e-05 2024-09-13T13:13:59.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.909139+0000", 2024-09-13T13:13:59.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8969999999999996e-06 2024-09-13T13:13:59.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.909142+0000", 2024-09-13T13:13:59.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7730000000000001e-06 2024-09-13T13:13:59.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.912489+0000", 2024-09-13T13:13:59.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033462349999999999 2024-09-13T13:13:59.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.912608+0000", 2024-09-13T13:13:59.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119283 2024-09-13T13:13:59.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:59.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.912738+0000", 2024-09-13T13:13:59.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012959700000000001 2024-09-13T13:13:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.914147+0000", 2024-09-13T13:13:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001409357 2024-09-13T13:13:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.915610+0000", 2024-09-13T13:13:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001463174 2024-09-13T13:13:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.988184+0000", 2024-09-13T13:13:59.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072574423999999998 2024-09-13T13:13:59.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.988238+0000", 2024-09-13T13:13:59.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3653000000000003e-05 2024-09-13T13:13:59.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:45.988290+0000", 2024-09-13T13:13:59.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1533999999999998e-05 2024-09-13T13:13:59.414 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.415 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.415 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.415 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:36965 8.2 8:4461e6ee:::benchmark_data_smithi052_101302_object36964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T13:13:59.415 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:47.666007+0000", 2024-09-13T13:13:59.415 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 431.72580608499999, 2024-09-13T13:13:59.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067357757000000004, 2024-09-13T13:13:59.415 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.415 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.415 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.415 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.415 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:13:59.416 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:13:59.416 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 36965 2024-09-13T13:13:59.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.416 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.666007+0000", 2024-09-13T13:13:59.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.666007+0000", 2024-09-13T13:13:59.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.666010+0000", 2024-09-13T13:13:59.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8959999999999999e-06 2024-09-13T13:13:59.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.666060+0000", 2024-09-13T13:13:59.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0148000000000002e-05 2024-09-13T13:13:59.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.666074+0000", 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3399e-05 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.666080+0000", 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5950000000000004e-06 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.666137+0000", 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6366999999999998e-05 2024-09-13T13:13:59.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.666272+0000", 2024-09-13T13:13:59.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013533499999999999 2024-09-13T13:13:59.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:59.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.666405+0000", 2024-09-13T13:13:59.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013348900000000001 2024-09-13T13:13:59.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.667622+0000", 2024-09-13T13:13:59.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012170989999999999 2024-09-13T13:13:59.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.668252+0000", 2024-09-13T13:13:59.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00062938200000000005 2024-09-13T13:13:59.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.733209+0000", 2024-09-13T13:13:59.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064956998000000002 2024-09-13T13:13:59.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.733320+0000", 2024-09-13T13:13:59.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111007 2024-09-13T13:13:59.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.733365+0000", 2024-09-13T13:13:59.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5042e-05 2024-09-13T13:13:59.421 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.421 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.421 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:37464 8.2 8:433b9a59:::benchmark_data_smithi052_101302_object37463:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:47.998024+0000", 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 431.39378913500002, 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088262342999999993, 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 37464 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.998024+0000", 2024-09-13T13:13:59.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.998024+0000", 2024-09-13T13:13:59.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.998026+0000", 2024-09-13T13:13:59.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2639999999999998e-06 2024-09-13T13:13:59.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.998073+0000", 2024-09-13T13:13:59.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6551000000000002e-05 2024-09-13T13:13:59.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.998085+0000", 2024-09-13T13:13:59.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2157999999999999e-05 2024-09-13T13:13:59.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.998091+0000", 2024-09-13T13:13:59.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6119999999999998e-06 2024-09-13T13:13:59.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.998144+0000", 2024-09-13T13:13:59.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3266999999999997e-05 2024-09-13T13:13:59.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.998230+0000", 2024-09-13T13:13:59.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5859000000000006e-05 2024-09-13T13:13:59.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.998321+0000", 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1508000000000005e-05 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.999478+0000", 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011564430000000001 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:47.999973+0000", 2024-09-13T13:13:59.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000495331 2024-09-13T13:13:59.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.086203+0000", 2024-09-13T13:13:59.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.086229700000000006 2024-09-13T13:13:59.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.086243+0000", 2024-09-13T13:13:59.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0398999999999997e-05 2024-09-13T13:13:59.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.086286+0000", 2024-09-13T13:13:59.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3251000000000003e-05 2024-09-13T13:13:59.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.428 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.428 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:37484 8.2 8:4087dd2d:::benchmark_data_smithi052_101302_object37483:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-13T13:13:59.428 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:06:48.009237+0000", 2024-09-13T13:13:59.428 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 431.38257654099999, 2024-09-13T13:13:59.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077239564999999996, 2024-09-13T13:13:59.428 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.428 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.429 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.429 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.429 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:13:59.429 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:13:59.429 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 37484 2024-09-13T13:13:59.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.429 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.009237+0000", 2024-09-13T13:13:59.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.009237+0000", 2024-09-13T13:13:59.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.009239+0000", 2024-09-13T13:13:59.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1890000000000001e-06 2024-09-13T13:13:59.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.009312+0000", 2024-09-13T13:13:59.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2955999999999997e-05 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.009324+0000", 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2275e-05 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.009331+0000", 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.81e-06 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.009377+0000", 2024-09-13T13:13:59.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6208000000000002e-05 2024-09-13T13:13:59.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.009464+0000", 2024-09-13T13:13:59.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7217000000000002e-05 2024-09-13T13:13:59.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:59.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.009557+0000", 2024-09-13T13:13:59.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2288999999999997e-05 2024-09-13T13:13:59.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.010640+0000", 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010837430000000001 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.010898+0000", 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025760900000000001 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.086386+0000", 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075487979999999996 2024-09-13T13:13:59.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.086440+0000", 2024-09-13T13:13:59.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3918000000000001e-05 2024-09-13T13:13:59.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:06:48.086476+0000", 2024-09-13T13:13:59.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6371000000000001e-05 2024-09-13T13:13:59.434 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.434 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.434 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:59322 8.2 8:489fe351:::benchmark_data_smithi052_101302_object59321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:01.966972+0000", 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 417.42484146800001, 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084009151000000004, 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 59322 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.435 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.966972+0000", 2024-09-13T13:13:59.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.966972+0000", 2024-09-13T13:13:59.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.966974+0000", 2024-09-13T13:13:59.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.458e-06 2024-09-13T13:13:59.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.967014+0000", 2024-09-13T13:13:59.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9876000000000002e-05 2024-09-13T13:13:59.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.967026+0000", 2024-09-13T13:13:59.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1474999999999999e-05 2024-09-13T13:13:59.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.967033+0000", 2024-09-13T13:13:59.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.746e-06 2024-09-13T13:13:59.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.967097+0000", 2024-09-13T13:13:59.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3639999999999994e-05 2024-09-13T13:13:59.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.967202+0000", 2024-09-13T13:13:59.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010464899999999999 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.967311+0000", 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010952599999999999 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.969139+0000", 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018281510000000001 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.969542+0000", 2024-09-13T13:13:59.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040241400000000003 2024-09-13T13:13:59.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.050856+0000", 2024-09-13T13:13:59.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.081314077999999998 2024-09-13T13:13:59.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.050932+0000", 2024-09-13T13:13:59.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6024000000000003e-05 2024-09-13T13:13:59.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.050981+0000", 2024-09-13T13:13:59.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9113999999999998e-05 2024-09-13T13:13:59.441 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.441 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.441 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.441 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:59334 8.2 8:45737f58:::benchmark_data_smithi052_101302_object59333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-13T13:13:59.441 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:01.983045+0000", 2024-09-13T13:13:59.441 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 417.408768076, 2024-09-13T13:13:59.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.068113566, 2024-09-13T13:13:59.441 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.441 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 59334 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.983045+0000", 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.983045+0000", 2024-09-13T13:13:59.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.983048+0000", 2024-09-13T13:13:59.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4849999999999999e-06 2024-09-13T13:13:59.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.983113+0000", 2024-09-13T13:13:59.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5845000000000004e-05 2024-09-13T13:13:59.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.983126+0000", 2024-09-13T13:13:59.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2729e-05 2024-09-13T13:13:59.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.983134+0000", 2024-09-13T13:13:59.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5530000000000004e-06 2024-09-13T13:13:59.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.983194+0000", 2024-09-13T13:13:59.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0390000000000003e-05 2024-09-13T13:13:59.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.983326+0000", 2024-09-13T13:13:59.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013207599999999999 2024-09-13T13:13:59.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:59.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.983475+0000", 2024-09-13T13:13:59.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014881000000000001 2024-09-13T13:13:59.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.985041+0000", 2024-09-13T13:13:59.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001566086 2024-09-13T13:13:59.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:01.985690+0000", 2024-09-13T13:13:59.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064879599999999999 2024-09-13T13:13:59.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.051057+0000", 2024-09-13T13:13:59.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065367357000000001 2024-09-13T13:13:59.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.051115+0000", 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7924999999999999e-05 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:02.051159+0000", 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3513999999999997e-05 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:61271 8.2 8:4fc57a4d:::benchmark_data_smithi052_101302_object61270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:07:03.545525+0000", 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 415.846287829, 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097038458999999994, 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 61271 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.545525+0000", 2024-09-13T13:13:59.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.545525+0000", 2024-09-13T13:13:59.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.545528+0000", 2024-09-13T13:13:59.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1600000000000001e-06 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.545562+0000", 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4232e-05 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.545574+0000", 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2299e-05 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.545582+0000", 2024-09-13T13:13:59.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9039999999999992e-06 2024-09-13T13:13:59.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.545639+0000", 2024-09-13T13:13:59.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6963999999999999e-05 2024-09-13T13:13:59.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.545763+0000", 2024-09-13T13:13:59.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012423299999999999 2024-09-13T13:13:59.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.545914+0000", 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015129499999999999 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.549156+0000", 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003241341 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.549654+0000", 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00049804700000000005 2024-09-13T13:13:59.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.642479+0000", 2024-09-13T13:13:59.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.092825198999999997 2024-09-13T13:13:59.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.642516+0000", 2024-09-13T13:13:59.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6981999999999999e-05 2024-09-13T13:13:59.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:07:03.642564+0000", 2024-09-13T13:13:59.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7803000000000003e-05 2024-09-13T13:13:59.454 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.454 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.454 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.454 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:135968 8.2 8:4f16015f:::benchmark_data_smithi052_101302_object8039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:13:59.454 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.800436+0000", 2024-09-13T13:13:59.454 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 350.59137730100002, 2024-09-13T13:13:59.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20382483900000001, 2024-09-13T13:13:59.454 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.454 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.454 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135968 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800436+0000", 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800436+0000", 2024-09-13T13:13:59.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800438+0000", 2024-09-13T13:13:59.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1880000000000001e-06 2024-09-13T13:13:59.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800444+0000", 2024-09-13T13:13:59.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0630000000000002e-06 2024-09-13T13:13:59.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800459+0000", 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4618e-05 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800466+0000", 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7209999999999994e-06 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800533+0000", 2024-09-13T13:13:59.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6579000000000001e-05 2024-09-13T13:13:59.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800676+0000", 2024-09-13T13:13:59.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000143182 2024-09-13T13:13:59.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:59.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800780+0000", 2024-09-13T13:13:59.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103553 2024-09-13T13:13:59.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803764+0000", 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002984273 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803775+0000", 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1088e-05 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004153+0000", 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200377894 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004207+0000", 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4042000000000002e-05 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004261+0000", 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3637999999999997e-05 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.460 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:188578 8.2 8:4872faa9:::benchmark_data_smithi052_101302_object60649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:55.001476+0000", 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 304.39033674699999, 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082732623000000005, 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 188578 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.001476+0000", 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.001476+0000", 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.001478+0000", 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8980000000000001e-06 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.001484+0000", 2024-09-13T13:13:59.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4249999999999997e-06 2024-09-13T13:13:59.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.001500+0000", 2024-09-13T13:13:59.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6518999999999999e-05 2024-09-13T13:13:59.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.001508+0000", 2024-09-13T13:13:59.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2180000000000002e-06 2024-09-13T13:13:59.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.001560+0000", 2024-09-13T13:13:59.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2506000000000002e-05 2024-09-13T13:13:59.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.001668+0000", 2024-09-13T13:13:59.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107963 2024-09-13T13:13:59.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:59.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.001793+0000", 2024-09-13T13:13:59.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000125232 2024-09-13T13:13:59.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.004142+0000", 2024-09-13T13:13:59.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023483660000000002 2024-09-13T13:13:59.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.004426+0000", 2024-09-13T13:13:59.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028453700000000002 2024-09-13T13:13:59.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.084124+0000", 2024-09-13T13:13:59.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079698242000000002 2024-09-13T13:13:59.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.084170+0000", 2024-09-13T13:13:59.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5791000000000003e-05 2024-09-13T13:13:59.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:55.084209+0000", 2024-09-13T13:13:59.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8926e-05 2024-09-13T13:13:59.468 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.468 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.468 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.468 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:237609 8.2 8:40f126b9:::benchmark_data_smithi052_101302_object109680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:13:59.468 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:09:37.710595+0000", 2024-09-13T13:13:59.468 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 261.68121831299999, 2024-09-13T13:13:59.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101345344, 2024-09-13T13:13:59.468 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.468 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.468 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.469 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.469 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:13:59.469 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:13:59.469 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 237609 2024-09-13T13:13:59.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.469 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.710595+0000", 2024-09-13T13:13:59.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.710595+0000", 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.710597+0000", 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1940000000000001e-06 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.710604+0000", 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1670000000000002e-06 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.710620+0000", 2024-09-13T13:13:59.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6249e-05 2024-09-13T13:13:59.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.710629+0000", 2024-09-13T13:13:59.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5720000000000003e-06 2024-09-13T13:13:59.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.710685+0000", 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6073999999999999e-05 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.710806+0000", 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012079900000000001 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.710916+0000", 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110042 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.713672+0000", 2024-09-13T13:13:59.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027557380000000002 2024-09-13T13:13:59.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.716947+0000", 2024-09-13T13:13:59.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032749749999999998 2024-09-13T13:13:59.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.811865+0000", 2024-09-13T13:13:59.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094918796999999999 2024-09-13T13:13:59.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.811906+0000", 2024-09-13T13:13:59.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0340999999999999e-05 2024-09-13T13:13:59.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.811940+0000", 2024-09-13T13:13:59.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4396000000000001e-05 2024-09-13T13:13:59.474 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.474 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.474 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.475 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:237610 8.2 8:4ab299f2:::benchmark_data_smithi052_101302_object109681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:13:59.475 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:09:37.713813+0000", 2024-09-13T13:13:59.475 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 261.67800050300002, 2024-09-13T13:13:59.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.104642312, 2024-09-13T13:13:59.475 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.475 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.475 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.475 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.475 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:13:59.475 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:13:59.475 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 237610 2024-09-13T13:13:59.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.476 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.713813+0000", 2024-09-13T13:13:59.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.713813+0000", 2024-09-13T13:13:59.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.713815+0000", 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2419999999999999e-06 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.713821+0000", 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9649999999999996e-06 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.713834+0000", 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2965999999999999e-05 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.713842+0000", 2024-09-13T13:13:59.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6539999999999999e-06 2024-09-13T13:13:59.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.713891+0000", 2024-09-13T13:13:59.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-09-13T13:13:59.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.713997+0000", 2024-09-13T13:13:59.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106044 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.714103+0000", 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106297 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.716743+0000", 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002639429 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.717053+0000", 2024-09-13T13:13:59.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030982200000000002 2024-09-13T13:13:59.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.818302+0000", 2024-09-13T13:13:59.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10124894 2024-09-13T13:13:59.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.818406+0000", 2024-09-13T13:13:59.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104892 2024-09-13T13:13:59.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:09:37.818455+0000", 2024-09-13T13:13:59.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8556000000000001e-05 2024-09-13T13:13:59.481 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.481 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.481 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.481 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:227306 9.e 9:7d4c2d62:::benchmark_data_smithi052_128427_object96637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-13T13:13:59.481 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:13:20.213703+0000", 2024-09-13T13:13:59.481 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 39.178110347000001, 2024-09-13T13:13:59.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038565387999999999, 2024-09-13T13:13:59.481 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.482 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.482 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.482 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.482 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:13:59.482 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:59.482 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 227306 2024-09-13T13:13:59.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.482 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.213703+0000", 2024-09-13T13:13:59.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.213703+0000", 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.213705+0000", 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.705e-06 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.213710+0000", 2024-09-13T13:13:59.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.271e-06 2024-09-13T13:13:59.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.213721+0000", 2024-09-13T13:13:59.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0855e-05 2024-09-13T13:13:59.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.213728+0000", 2024-09-13T13:13:59.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6660000000000008e-06 2024-09-13T13:13:59.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.213778+0000", 2024-09-13T13:13:59.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9209999999999998e-05 2024-09-13T13:13:59.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.213862+0000", 2024-09-13T13:13:59.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4859999999999997e-05 2024-09-13T13:13:59.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,7", 2024-09-13T13:13:59.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.213935+0000", 2024-09-13T13:13:59.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2180000000000003e-05 2024-09-13T13:13:59.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.216343+0000", 2024-09-13T13:13:59.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024088719999999998 2024-09-13T13:13:59.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.216827+0000", 2024-09-13T13:13:59.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00048341600000000002 2024-09-13T13:13:59.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.252123+0000", 2024-09-13T13:13:59.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035296098999999997 2024-09-13T13:13:59.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.252221+0000", 2024-09-13T13:13:59.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7936000000000006e-05 2024-09-13T13:13:59.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:20.252268+0000", 2024-09-13T13:13:59.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7318000000000003e-05 2024-09-13T13:13:59.487 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.488 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.488 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.488 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:248137 9.e 9:7e9898d7:::benchmark_data_smithi052_128427_object117468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-13T13:13:59.488 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:13:37.637152+0000", 2024-09-13T13:13:59.488 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.754661632000001, 2024-09-13T13:13:59.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037241934999999997, 2024-09-13T13:13:59.488 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.488 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.488 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.488 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.488 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 248137 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.637152+0000", 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.637152+0000", 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.637153+0000", 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.795e-06 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.637158+0000", 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1460000000000004e-06 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.637171+0000", 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2245e-05 2024-09-13T13:13:59.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.637177+0000", 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5289999999999997e-06 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.637228+0000", 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1044000000000001e-05 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.637344+0000", 2024-09-13T13:13:59.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115358 2024-09-13T13:13:59.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,7", 2024-09-13T13:13:59.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.637457+0000", 2024-09-13T13:13:59.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011328399999999999 2024-09-13T13:13:59.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.641958+0000", 2024-09-13T13:13:59.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045009539999999997 2024-09-13T13:13:59.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.641975+0000", 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6821999999999999e-05 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.674296+0000", 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032321170000000003 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.674350+0000", 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3990000000000003e-05 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:37.674393+0000", 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3597999999999998e-05 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:254525 9.e 9:74fe996b:::benchmark_data_smithi052_128427_object123856:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:13:42.708354+0000", 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 16.683458730000002, 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043133741000000003, 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.494 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.495 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.495 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.495 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:13:59.495 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:59.495 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 254525 2024-09-13T13:13:59.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.495 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.708354+0000", 2024-09-13T13:13:59.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.708354+0000", 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.708357+0000", 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3870000000000002e-06 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.708363+0000", 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5189999999999998e-06 2024-09-13T13:13:59.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.708382+0000", 2024-09-13T13:13:59.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8822e-05 2024-09-13T13:13:59.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.708390+0000", 2024-09-13T13:13:59.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0830000000000004e-06 2024-09-13T13:13:59.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.708449+0000", 2024-09-13T13:13:59.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8798000000000002e-05 2024-09-13T13:13:59.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.708588+0000", 2024-09-13T13:13:59.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013930400000000001 2024-09-13T13:13:59.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,7", 2024-09-13T13:13:59.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.708734+0000", 2024-09-13T13:13:59.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014585999999999999 2024-09-13T13:13:59.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.711935+0000", 2024-09-13T13:13:59.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032003600000000002 2024-09-13T13:13:59.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.714592+0000", 2024-09-13T13:13:59.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002657768 2024-09-13T13:13:59.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.751381+0000", 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.03678886 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.751435+0000", 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3699000000000002e-05 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:42.751488+0000", 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3281e-05 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.500 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.501 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:255494 9.e 9:7f1313fc:::benchmark_data_smithi052_128427_object124825:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-13T13:13:59.501 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:13:43.499661+0000", 2024-09-13T13:13:59.501 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.892152640999999, 2024-09-13T13:13:59.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.087666517999999999, 2024-09-13T13:13:59.501 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.501 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.501 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.501 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 255494 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.499661+0000", 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.499661+0000", 2024-09-13T13:13:59.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.499662+0000", 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.939e-06 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.499668+0000", 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3199999999999999e-06 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.499679+0000", 2024-09-13T13:13:59.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1517e-05 2024-09-13T13:13:59.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.499685+0000", 2024-09-13T13:13:59.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5709999999999997e-06 2024-09-13T13:13:59.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.499731+0000", 2024-09-13T13:13:59.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5994e-05 2024-09-13T13:13:59.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.499839+0000", 2024-09-13T13:13:59.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107904 2024-09-13T13:13:59.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,7", 2024-09-13T13:13:59.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.499908+0000", 2024-09-13T13:13:59.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9553000000000003e-05 2024-09-13T13:13:59.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.507139+0000", 2024-09-13T13:13:59.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072306469999999998 2024-09-13T13:13:59.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.507425+0000", 2024-09-13T13:13:59.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028566000000000003 2024-09-13T13:13:59.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.587197+0000", 2024-09-13T13:13:59.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079772755000000001 2024-09-13T13:13:59.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.587248+0000", 2024-09-13T13:13:59.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0572000000000003e-05 2024-09-13T13:13:59.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.587327+0000", 2024-09-13T13:13:59.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9085999999999995e-05 2024-09-13T13:13:59.507 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.507 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.507 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.507 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:255497 9.e 9:770787ba:::benchmark_data_smithi052_128427_object124828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:13:43.501829+0000", 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.889983898000001, 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.085657699000000004, 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 255497 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.501829+0000", 2024-09-13T13:13:59.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.501829+0000", 2024-09-13T13:13:59.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.501831+0000", 2024-09-13T13:13:59.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990000000000001e-06 2024-09-13T13:13:59.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.501839+0000", 2024-09-13T13:13:59.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5560000000000002e-06 2024-09-13T13:13:59.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.501848+0000", 2024-09-13T13:13:59.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0180000000000007e-06 2024-09-13T13:13:59.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.501855+0000", 2024-09-13T13:13:59.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8229999999999997e-06 2024-09-13T13:13:59.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.502705+0000", 2024-09-13T13:13:59.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00085039299999999996 2024-09-13T13:13:59.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.503934+0000", 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001228809 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,7", 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.504165+0000", 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023128999999999999 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.507165+0000", 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029998920000000001 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.507602+0000", 2024-09-13T13:13:59.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043682399999999999 2024-09-13T13:13:59.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:13:59.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.587389+0000", 2024-09-13T13:13:59.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079786989000000003 2024-09-13T13:13:59.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:13:59.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.587446+0000", 2024-09-13T13:13:59.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7129000000000001e-05 2024-09-13T13:13:59.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:43.587487+0000", 2024-09-13T13:13:59.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0877000000000002e-05 2024-09-13T13:13:59.517 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.517 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.517 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.517 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDScrubReserve(3.19,QREQUEST,e:333,from:1,reservation_nonce:3)", 2024-09-13T13:13:59.517 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:13:53.793386+0000", 2024-09-13T13:13:59.517 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.598427429, 2024-09-13T13:13:59.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001768618, 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:53.793386+0000", 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:53.793386+0000", 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:53.793389+0000", 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0469999999999998e-06 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:53.793400+0000", 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1506e-05 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:53.793427+0000", 2024-09-13T13:13:59.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6772999999999999e-05 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:53.793441+0000", 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.432e-05 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:53.793586+0000", 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014416799999999999 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-09-13T13:13:59.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:53.793614+0000", 2024-09-13T13:13:59.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8501999999999999e-05 2024-09-13T13:13:59.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:53.795092+0000", 2024-09-13T13:13:59.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014776699999999999 2024-09-13T13:13:59.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:53.795117+0000", 2024-09-13T13:13:59.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5211e-05 2024-09-13T13:13:59.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:53.795154+0000", 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7421e-05 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14493.0:2971 5.e 5:7759931f:::notify.3:head [watch ping cookie 94064716860672 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e331)", 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:13:57.394127+0000", 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.9976858929999999, 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00071848500000000004, 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:13:59.522 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14493", 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 2971 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:57.394127+0000", 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:13:59.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:57.394127+0000", 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:57.394132+0000", 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3769999999999998e-06 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:57.394145+0000", 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3782e-05 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:57.394175+0000", 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9207e-05 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:57.394187+0000", 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2700999999999999e-05 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:57.394644+0000", 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045636 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:57.394724+0000", 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0500999999999994e-05 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:57.394846+0000", 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121557 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:13:59.526 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:13:59.527 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:13:59.549 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:59.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:59.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:13:59.651 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:13:59.673 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:59.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:59.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:13:59.775 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:13:59.797 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:59.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:13:59.899 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:13:59.921 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:13:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:13:59.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:13:59.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:59 smithi052 ceph-mon[36682]: pgmap v637: 132 pgs: 132 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-13T13:13:59.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:59 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/673102296' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:13:59.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:59 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:13:59.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:59 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/673102296' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:13:59.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:13:59 smithi052 ceph-mon[36682]: osdmap e335: 8 total, 8 up, 8 in 2024-09-13T13:13:59.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:59 smithi052 ceph-mon[29471]: pgmap v637: 132 pgs: 132 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-13T13:13:59.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:59 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/673102296' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:13:59.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:59 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:13:59.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:59 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/673102296' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:13:59.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:13:59 smithi052 ceph-mon[29471]: osdmap e335: 8 total, 8 up, 8 in 2024-09-13T13:13:59.998 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:00.024 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:00.048 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:00.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:00.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:00.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:59 smithi167 ceph-mon[33178]: pgmap v637: 132 pgs: 132 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-13T13:14:00.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:59 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/673102296' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:14:00.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:59 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:00.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:59 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/673102296' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:14:00.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:13:59 smithi167 ceph-mon[33178]: osdmap e335: 8 total, 8 up, 8 in 2024-09-13T13:14:00.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:00.174 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:00.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:00.268 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T13:14:00.276 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:00.298 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:00.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:00.400 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:00.422 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:00.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:00.513 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:14:00.524 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:00.533 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:14:00.534 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:14:00.542 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:00.546 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:00.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:00.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:00.548 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:14:00.548 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:14:00.565 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:14:00.566 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:14:00.586 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:14:00.586 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:14:00.602 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:14:00.603 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:14:00.619 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:14:00.620 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:14:00.636 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:14:00.636 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:14:00.648 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:00.653 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:14:00.653 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:14:00.673 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:00.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:00.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:00.774 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:00.795 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:00.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:00.897 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:00.919 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:00.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:00.956 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T13:14:01.021 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:01.028 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:14:01.028 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:14:01.043 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:01.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:01.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:01.145 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:01.169 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:01.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:00 smithi052 ceph-mon[29471]: pgmap v640: 148 pgs: 8 creating+peering, 6 unknown, 134 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-13T13:14:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:00 smithi052 ceph-mon[29471]: osdmap e336: 8 total, 8 up, 8 in 2024-09-13T13:14:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:00 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1650115626' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:14:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:00 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2199729163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:00 smithi052 ceph-mon[29471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:14:01.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:00 smithi052 ceph-mon[36682]: pgmap v640: 148 pgs: 8 creating+peering, 6 unknown, 134 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-13T13:14:01.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:00 smithi052 ceph-mon[36682]: osdmap e336: 8 total, 8 up, 8 in 2024-09-13T13:14:01.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:00 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1650115626' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:14:01.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:00 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2199729163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:01.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:00 smithi052 ceph-mon[36682]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:14:01.270 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:01.292 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:01.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:01.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:00 smithi167 ceph-mon[33178]: pgmap v640: 148 pgs: 8 creating+peering, 6 unknown, 134 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-13T13:14:01.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:00 smithi167 ceph-mon[33178]: osdmap e336: 8 total, 8 up, 8 in 2024-09-13T13:14:01.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:00 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1650115626' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:14:01.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:00 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2199729163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:01.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:00 smithi167 ceph-mon[33178]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:14:01.394 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:01.417 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:01.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:01.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:01.541 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:01.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:01.642 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:01.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:01.666 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:01.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:01.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T13:14:01.740 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:14:01.768 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:01.806 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:01.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:01.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:01.908 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:01.932 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:01.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:01.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:01.942 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:14:01.969 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-13T13:14:02.034 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:02.044 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:01 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1650115626' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:14:02.044 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:01 smithi052 ceph-mon[36682]: osdmap e337: 8 total, 8 up, 8 in 2024-09-13T13:14:02.044 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:01 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/531746146' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:14:02.044 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:01 smithi052 ceph-mon[36682]: from='client.36077 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:14:02.045 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:01 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1650115626' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:14:02.045 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:01 smithi052 ceph-mon[29471]: osdmap e337: 8 total, 8 up, 8 in 2024-09-13T13:14:02.045 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:01 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/531746146' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:14:02.045 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:01 smithi052 ceph-mon[29471]: from='client.36077 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:14:02.059 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:02.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:02.161 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:02.182 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:02.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:02.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:02.284 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:02.302 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:01 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1650115626' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:14:02.303 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:01 smithi167 ceph-mon[33178]: osdmap e337: 8 total, 8 up, 8 in 2024-09-13T13:14:02.303 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:01 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/531746146' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:14:02.303 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:01 smithi167 ceph-mon[33178]: from='client.36077 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:14:02.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:02.307 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:02.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:02.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:02.413 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:14:02.413 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:14:02.433 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:02.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:02.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:02.516 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T13:14:02.535 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:02.582 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:02.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:02.684 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:02.706 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:02.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:02.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:02.809 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:02.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:02.832 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:02.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:02.934 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:02.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:02.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:02 smithi167 ceph-mon[33178]: pgmap v643: 148 pgs: 8 creating+peering, 140 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:14:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:02 smithi167 ceph-mon[33178]: from='client.36077 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:14:03.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:02 smithi167 ceph-mon[33178]: osdmap e338: 8 total, 8 up, 8 in 2024-09-13T13:14:03.081 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:03.094 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:02 smithi052 ceph-mon[36682]: pgmap v643: 148 pgs: 8 creating+peering, 140 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:14:03.094 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:02 smithi052 ceph-mon[36682]: from='client.36077 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:14:03.094 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:02 smithi052 ceph-mon[36682]: osdmap e338: 8 total, 8 up, 8 in 2024-09-13T13:14:03.094 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:02 smithi052 ceph-mon[29471]: pgmap v643: 148 pgs: 8 creating+peering, 140 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:14:03.094 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:02 smithi052 ceph-mon[29471]: from='client.36077 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:14:03.094 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:02 smithi052 ceph-mon[29471]: osdmap e338: 8 total, 8 up, 8 in 2024-09-13T13:14:03.105 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:03.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:03.117 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T13:14:03.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:03.215 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:03.231 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:03.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:03.235 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:03.332 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:03.355 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:03.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:03.406 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:14:03.457 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:03.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:03.479 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:03.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:03.580 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:03.603 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:03.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:03.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:03.705 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:03.728 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:03.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:03.829 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:03.851 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:03.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:03.938 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:14:03.938 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:14:03.954 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:03.976 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:03.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:04.075 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:14:04.077 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:04.100 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:04.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:04.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:04.122 DEBUG:teuthology.orchestra.run.smithi052:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-13T13:14:04.144 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T13:14:04.144 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T13:14:04.202 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:04.225 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:04.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:04.226 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:03 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:14:04.226 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:03 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:14:04.226 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:03 smithi052 ceph-mon[36682]: osdmap e339: 8 total, 8 up, 8 in 2024-09-13T13:14:04.226 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:03 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:14:04.227 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:03 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:14:04.227 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:03 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:14:04.227 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:03 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:14:04.227 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:03 smithi052 ceph-mon[29471]: osdmap e339: 8 total, 8 up, 8 in 2024-09-13T13:14:04.227 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:03 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:14:04.227 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:03 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:14:04.242 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:03 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:14:04.243 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:03 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:14:04.243 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:03 smithi167 ceph-mon[33178]: osdmap e339: 8 total, 8 up, 8 in 2024-09-13T13:14:04.243 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:03 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:14:04.243 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:03 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:14:04.294 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:hints = 1 2024-09-13T13:14:04.294 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T13:14:04.294 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Object prefix: benchmark_data_smithi052_153703 2024-09-13T13:14:04.294 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:14:04.295 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 0 0 0 0 0 0 - 0 2024-09-13T13:14:04.326 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:04.374 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:04.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:04.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:04.404 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:04.477 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:04.514 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:04.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:04.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:14:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:14:04.616 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:04.633 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T13:14:04.651 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:04.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:04.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:04.755 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:04.802 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:04.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:04.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:04.904 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:04.944 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:04.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:04.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:04.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:04 smithi167 ceph-mon[33178]: pgmap v646: 148 pgs: 1 creating+peering, 147 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-13T13:14:04.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:04 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2024111117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:14:04.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:04 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:04.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:04 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2643789032' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:04.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:04 smithi167 ceph-mon[33178]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:14:04.981 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:14:04.985 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T13:14:05.025 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:14:05.046 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:05.064 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:14:05.064 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:14:05.091 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:05.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:05.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:05.106 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:14:05.107 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:14:05.145 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:04 smithi052 ceph-mon[36682]: pgmap v646: 148 pgs: 1 creating+peering, 147 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-13T13:14:05.146 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:04 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2024111117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:14:05.146 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:04 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:05.146 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:04 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2643789032' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:05.146 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:04 smithi052 ceph-mon[36682]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:14:05.146 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:14:05.146 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:14:05.147 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:04 smithi052 ceph-mon[29471]: pgmap v646: 148 pgs: 1 creating+peering, 147 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-13T13:14:05.147 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:04 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2024111117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:14:05.147 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:04 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:05.147 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:04 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2643789032' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:05.147 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:04 smithi052 ceph-mon[29471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:14:05.178 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:14:05.179 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:14:05.193 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:05.214 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:14:05.215 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:14:05.232 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:05.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:05.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:05.250 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:14:05.250 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:14:05.290 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:14:05.290 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:14:05.294 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 1 16 3237 3221 201.296 201.312 0.00395976 0.00495567 2024-09-13T13:14:05.326 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:14:05.326 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:14:05.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:05.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:05.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:05.471 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:05.506 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:05.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:05.608 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:05.640 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:05.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:05.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:05.666 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:14:05.666 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:14:05.667 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-09-13T13:14:05.667 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:14:05.742 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:05.780 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:05.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:05.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:05.882 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:05.918 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:05.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:06.020 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:06.057 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:06.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:06.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:05 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:06.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:05 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2024111117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:14:06.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:05 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2643789032' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:14:06.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:05 smithi052 ceph-mon[36682]: osdmap e340: 8 total, 8 up, 8 in 2024-09-13T13:14:06.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:05 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3953622344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:06.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:05 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:06.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:05 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2024111117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:14:06.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:05 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2643789032' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:14:06.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:05 smithi052 ceph-mon[29471]: osdmap e340: 8 total, 8 up, 8 in 2024-09-13T13:14:06.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:05 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3953622344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:06.159 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:06.193 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:06.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:06.294 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 2 16 6685 6669 208.389 215.5 0.00661547 0.00478773 2024-09-13T13:14:06.294 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:06.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:05 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:05 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2024111117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:14:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:05 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2643789032' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:14:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:05 smithi167 ceph-mon[33178]: osdmap e340: 8 total, 8 up, 8 in 2024-09-13T13:14:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:05 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3953622344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:06.327 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:06.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:06.402 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:14:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:14:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:14:06.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:06.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T13:14:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:06.466 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:06.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:06.567 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:06.606 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:06.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:06.707 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:06.744 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:06.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:06.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:06.846 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:06.889 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:06.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:06.911 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:14:06.991 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:07.029 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:07.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:07.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:07.063 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:06 smithi052 ceph-mon[29471]: pgmap v648: 148 pgs: 148 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T13:14:07.064 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:06 smithi052 ceph-mon[36682]: pgmap v648: 148 pgs: 148 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T13:14:07.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T13:14:07.131 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:07.199 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:07.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:07.247 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:06 smithi167 ceph-mon[33178]: pgmap v648: 148 pgs: 148 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T13:14:07.294 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 3 16 9732 9716 202.399 190.438 0.002711 0.00493259 2024-09-13T13:14:07.300 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:07.341 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:07.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:07.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:07.443 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:07.481 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:07.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:07.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:07.545 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:14:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:14:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:14:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:14:07.545 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:14:07.583 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:07.619 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:07.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:07.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:07.720 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:07.767 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:07.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:07.826 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:07.861 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:14:07.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:07.903 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:07.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:07.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:08.005 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:08.038 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:08.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:08.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:08.140 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:08.170 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:08.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:08.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:08.272 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:08.294 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 4 15 12513 12498 195.264 173.875 0.00533578 0.00511327 2024-09-13T13:14:08.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T13:14:08.312 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:08.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:08.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:08.414 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:08.453 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:08.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:08.537 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:08.555 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:08.592 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:08.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:08.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:08.693 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:08.713 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:14:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:08.731 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:08.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:08.832 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:08.861 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:08.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:08.962 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:08.996 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:08.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:08.999 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T13:14:09.098 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:09.138 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:09.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:09.173 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:08 smithi167 ceph-mon[33178]: pgmap v649: 148 pgs: 148 active+clean; 135 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 22 MiB/s wr, 360 op/s 2024-09-13T13:14:09.173 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3452780670' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:14:09.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:08 smithi052 ceph-mon[36682]: pgmap v649: 148 pgs: 148 active+clean; 135 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 22 MiB/s wr, 360 op/s 2024-09-13T13:14:09.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3452780670' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:14:09.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:08 smithi052 ceph-mon[29471]: pgmap v649: 148 pgs: 148 active+clean; 135 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 22 MiB/s wr, 360 op/s 2024-09-13T13:14:09.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3452780670' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:14:09.240 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:09.278 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:09.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:09.294 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 5 16 15032 15016 187.684 157.375 0.00413237 0.00532062 2024-09-13T13:14:09.344 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:14:09.344 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:14:09.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:14:09.344 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:14:09.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.345 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:135640 9.3 9:c8fab5fe:::benchmark_data_smithi052_128427_object4971:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T13:14:09.345 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:11:49.891399+0000", 2024-09-13T13:14:09.345 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 139.44965384299999, 2024-09-13T13:14:09.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12597461600000001, 2024-09-13T13:14:09.345 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.345 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.345 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.345 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135640 2024-09-13T13:14:09.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.346 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891399+0000", 2024-09-13T13:14:09.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891399+0000", 2024-09-13T13:14:09.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891401+0000", 2024-09-13T13:14:09.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.435e-06 2024-09-13T13:14:09.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891406+0000", 2024-09-13T13:14:09.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:14:09.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891419+0000", 2024-09-13T13:14:09.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3396e-05 2024-09-13T13:14:09.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891427+0000", 2024-09-13T13:14:09.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7079999999999997e-06 2024-09-13T13:14:09.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891504+0000", 2024-09-13T13:14:09.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7261000000000004e-05 2024-09-13T13:14:09.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891609+0000", 2024-09-13T13:14:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104628 2024-09-13T13:14:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:14:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891708+0000", 2024-09-13T13:14:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8981999999999995e-05 2024-09-13T13:14:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.898451+0000", 2024-09-13T13:14:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0067428699999999998 2024-09-13T13:14:09.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.902316+0000", 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038657269999999998 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:50.017273+0000", 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114956343 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:50.017329+0000", 2024-09-13T13:14:09.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6198e-05 2024-09-13T13:14:09.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:50.017374+0000", 2024-09-13T13:14:09.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5274000000000001e-05 2024-09-13T13:14:09.351 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.351 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.351 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.351 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:135641 9.3 9:cf12db49:::benchmark_data_smithi052_128427_object4972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T13:14:09.351 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:11:49.891445+0000", 2024-09-13T13:14:09.351 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 139.449608251, 2024-09-13T13:14:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.126092432, 2024-09-13T13:14:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135641 2024-09-13T13:14:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891445+0000", 2024-09-13T13:14:09.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891445+0000", 2024-09-13T13:14:09.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891446+0000", 2024-09-13T13:14:09.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3300000000000001e-07 2024-09-13T13:14:09.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891449+0000", 2024-09-13T13:14:09.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6739999999999999e-06 2024-09-13T13:14:09.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891456+0000", 2024-09-13T13:14:09.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4590000000000003e-06 2024-09-13T13:14:09.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.891462+0000", 2024-09-13T13:14:09.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:14:09.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.894700+0000", 2024-09-13T13:14:09.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032381990000000002 2024-09-13T13:14:09.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.894802+0000", 2024-09-13T13:14:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010218899999999999 2024-09-13T13:14:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:14:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.894875+0000", 2024-09-13T13:14:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2380999999999997e-05 2024-09-13T13:14:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.901124+0000", 2024-09-13T13:14:09.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0062498019999999996 2024-09-13T13:14:09.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:49.902419+0000", 2024-09-13T13:14:09.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012945159999999999 2024-09-13T13:14:09.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:50.017440+0000", 2024-09-13T13:14:09.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11502148099999999 2024-09-13T13:14:09.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:50.017493+0000", 2024-09-13T13:14:09.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3146999999999998e-05 2024-09-13T13:14:09.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:11:50.017538+0000", 2024-09-13T13:14:09.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4066000000000001e-05 2024-09-13T13:14:09.359 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.359 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.359 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.359 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:154786 9.3 9:cbb83ca5:::benchmark_data_smithi052_128427_object24117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T13:14:09.359 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:08.412441+0000", 2024-09-13T13:14:09.359 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 120.92861237699999, 2024-09-13T13:14:09.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13816605100000001, 2024-09-13T13:14:09.359 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.359 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.360 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.361 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154786 2024-09-13T13:14:09.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.361 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412441+0000", 2024-09-13T13:14:09.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412441+0000", 2024-09-13T13:14:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412442+0000", 2024-09-13T13:14:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2249999999999999e-06 2024-09-13T13:14:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412446+0000", 2024-09-13T13:14:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8650000000000003e-06 2024-09-13T13:14:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412455+0000", 2024-09-13T13:14:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2140000000000002e-06 2024-09-13T13:14:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412460+0000", 2024-09-13T13:14:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1909999999999999e-06 2024-09-13T13:14:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412498+0000", 2024-09-13T13:14:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7203000000000003e-05 2024-09-13T13:14:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412571+0000", 2024-09-13T13:14:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3002000000000003e-05 2024-09-13T13:14:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:14:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412637+0000", 2024-09-13T13:14:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6225999999999999e-05 2024-09-13T13:14:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.417079+0000", 2024-09-13T13:14:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044422200000000002 2024-09-13T13:14:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.423099+0000", 2024-09-13T13:14:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00602003 2024-09-13T13:14:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.550526+0000", 2024-09-13T13:14:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.127427176 2024-09-13T13:14:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.550579+0000", 2024-09-13T13:14:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2822999999999999e-05 2024-09-13T13:14:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.550607+0000", 2024-09-13T13:14:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7875999999999999e-05 2024-09-13T13:14:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.367 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:154787 9.3 9:c6308295:::benchmark_data_smithi052_128427_object24118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T13:14:09.367 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:08.412476+0000", 2024-09-13T13:14:09.367 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 120.928577137, 2024-09-13T13:14:09.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14505712100000001, 2024-09-13T13:14:09.367 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.367 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.367 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.367 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154787 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412476+0000", 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412476+0000", 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412477+0000", 2024-09-13T13:14:09.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:14:09.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412479+0000", 2024-09-13T13:14:09.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1880000000000001e-06 2024-09-13T13:14:09.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412483+0000", 2024-09-13T13:14:09.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6299999999999997e-06 2024-09-13T13:14:09.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412487+0000", 2024-09-13T13:14:09.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7910000000000002e-06 2024-09-13T13:14:09.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.414686+0000", 2024-09-13T13:14:09.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021988009999999998 2024-09-13T13:14:09.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.414791+0000", 2024-09-13T13:14:09.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010519 2024-09-13T13:14:09.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:14:09.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.414904+0000", 2024-09-13T13:14:09.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112802 2024-09-13T13:14:09.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.419029+0000", 2024-09-13T13:14:09.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041248209999999999 2024-09-13T13:14:09.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.423145+0000", 2024-09-13T13:14:09.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041162229999999996 2024-09-13T13:14:09.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.557439+0000", 2024-09-13T13:14:09.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13429402300000001 2024-09-13T13:14:09.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.557490+0000", 2024-09-13T13:14:09.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0538000000000003e-05 2024-09-13T13:14:09.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.557533+0000", 2024-09-13T13:14:09.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3640000000000002e-05 2024-09-13T13:14:09.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.374 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:154791 9.3 9:c8f0a3cb:::benchmark_data_smithi052_128427_object24122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:08.412626+0000", 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 120.928427689, 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14510228, 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154791 2024-09-13T13:14:09.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.376 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412626+0000", 2024-09-13T13:14:09.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412626+0000", 2024-09-13T13:14:09.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412626+0000", 2024-09-13T13:14:09.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9999999999999999e-07 2024-09-13T13:14:09.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412627+0000", 2024-09-13T13:14:09.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.424e-06 2024-09-13T13:14:09.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412630+0000", 2024-09-13T13:14:09.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.807e-06 2024-09-13T13:14:09.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.412633+0000", 2024-09-13T13:14:09.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7e-06 2024-09-13T13:14:09.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.417109+0000", 2024-09-13T13:14:09.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044762700000000001 2024-09-13T13:14:09.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.417222+0000", 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113161 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.417348+0000", 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012564299999999999 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.419142+0000", 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001794365 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.424527+0000", 2024-09-13T13:14:09.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0053843340000000002 2024-09-13T13:14:09.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.557643+0000", 2024-09-13T13:14:09.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13311679500000001 2024-09-13T13:14:09.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.557695+0000", 2024-09-13T13:14:09.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1272e-05 2024-09-13T13:14:09.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.557728+0000", 2024-09-13T13:14:09.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3209e-05 2024-09-13T13:14:09.381 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.381 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.381 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.382 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:154796 9.3 9:c3ab54a2:::benchmark_data_smithi052_128427_object24127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T13:14:09.382 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:08.420648+0000", 2024-09-13T13:14:09.382 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 120.920405313, 2024-09-13T13:14:09.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16434322200000001, 2024-09-13T13:14:09.382 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.382 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.382 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.382 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.382 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.382 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.383 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154796 2024-09-13T13:14:09.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.383 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420648+0000", 2024-09-13T13:14:09.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420648+0000", 2024-09-13T13:14:09.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420649+0000", 2024-09-13T13:14:09.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2410000000000001e-06 2024-09-13T13:14:09.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420654+0000", 2024-09-13T13:14:09.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3000000000000003e-06 2024-09-13T13:14:09.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420662+0000", 2024-09-13T13:14:09.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4740000000000005e-06 2024-09-13T13:14:09.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420667+0000", 2024-09-13T13:14:09.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.797e-06 2024-09-13T13:14:09.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420714+0000", 2024-09-13T13:14:09.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6937999999999997e-05 2024-09-13T13:14:09.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420838+0000", 2024-09-13T13:14:09.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012402999999999999 2024-09-13T13:14:09.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:14:09.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420949+0000", 2024-09-13T13:14:09.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110874 2024-09-13T13:14:09.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.423161+0000", 2024-09-13T13:14:09.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022121889999999998 2024-09-13T13:14:09.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.424593+0000", 2024-09-13T13:14:09.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014320030000000001 2024-09-13T13:14:09.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.584895+0000", 2024-09-13T13:14:09.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16030223599999999 2024-09-13T13:14:09.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.584952+0000", 2024-09-13T13:14:09.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6428000000000003e-05 2024-09-13T13:14:09.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.584991+0000", 2024-09-13T13:14:09.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9712000000000002e-05 2024-09-13T13:14:09.388 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.389 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.389 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.389 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:163394 9.e 9:7c89c0d8:::benchmark_data_smithi052_128427_object32725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-13T13:14:09.389 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:16.424824+0000", 2024-09-13T13:14:09.389 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 112.916228953, 2024-09-13T13:14:09.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14081759499999999, 2024-09-13T13:14:09.389 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.389 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.389 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.389 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.390 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.390 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.390 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 163394 2024-09-13T13:14:09.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.390 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424824+0000", 2024-09-13T13:14:09.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424824+0000", 2024-09-13T13:14:09.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424825+0000", 2024-09-13T13:14:09.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.23e-07 2024-09-13T13:14:09.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424829+0000", 2024-09-13T13:14:09.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3330000000000001e-06 2024-09-13T13:14:09.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424837+0000", 2024-09-13T13:14:09.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3750000000000003e-06 2024-09-13T13:14:09.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424842+0000", 2024-09-13T13:14:09.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3750000000000002e-06 2024-09-13T13:14:09.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.424889+0000", 2024-09-13T13:14:09.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6263000000000001e-05 2024-09-13T13:14:09.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.425137+0000", 2024-09-13T13:14:09.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024875199999999998 2024-09-13T13:14:09.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:14:09.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.425257+0000", 2024-09-13T13:14:09.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119881 2024-09-13T13:14:09.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.428314+0000", 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030571629999999999 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.428325+0000", 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.098e-05 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.430320+0000", 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019945649999999998 2024-09-13T13:14:09.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.565531+0000", 2024-09-13T13:14:09.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13521107199999999 2024-09-13T13:14:09.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.565574+0000", 2024-09-13T13:14:09.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3133999999999997e-05 2024-09-13T13:14:09.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.565642+0000", 2024-09-13T13:14:09.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7879000000000005e-05 2024-09-13T13:14:09.396 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.396 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.396 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.396 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:164430 9.3 9:c2c8b212:::benchmark_data_smithi052_128427_object33761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-13T13:14:09.396 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:17.424138+0000", 2024-09-13T13:14:09.396 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 111.916915252, 2024-09-13T13:14:09.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14083164200000001, 2024-09-13T13:14:09.396 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.396 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 164430 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424138+0000", 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424138+0000", 2024-09-13T13:14:09.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424140+0000", 2024-09-13T13:14:09.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8500000000000001e-06 2024-09-13T13:14:09.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424145+0000", 2024-09-13T13:14:09.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5500000000000002e-06 2024-09-13T13:14:09.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424157+0000", 2024-09-13T13:14:09.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1627999999999999e-05 2024-09-13T13:14:09.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424164+0000", 2024-09-13T13:14:09.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1890000000000001e-06 2024-09-13T13:14:09.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424228+0000", 2024-09-13T13:14:09.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3825000000000006e-05 2024-09-13T13:14:09.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424327+0000", 2024-09-13T13:14:09.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9273999999999999e-05 2024-09-13T13:14:09.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:14:09.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.424427+0000", 2024-09-13T13:14:09.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9520999999999998e-05 2024-09-13T13:14:09.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.426863+0000", 2024-09-13T13:14:09.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024363620000000001 2024-09-13T13:14:09.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.550466+0000", 2024-09-13T13:14:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123602541 2024-09-13T13:14:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.564891+0000", 2024-09-13T13:14:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014425097 2024-09-13T13:14:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.564926+0000", 2024-09-13T13:14:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5500999999999998e-05 2024-09-13T13:14:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.564970+0000", 2024-09-13T13:14:09.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3303999999999997e-05 2024-09-13T13:14:09.403 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.403 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.403 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.403 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:164434 9.3 9:c73c0368:::benchmark_data_smithi052_128427_object33765:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-13T13:14:09.403 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:17.428447+0000", 2024-09-13T13:14:09.403 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 111.912606817, 2024-09-13T13:14:09.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13666840799999999, 2024-09-13T13:14:09.404 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.404 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.404 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.404 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.404 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.404 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.404 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 164434 2024-09-13T13:14:09.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.404 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428447+0000", 2024-09-13T13:14:09.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428447+0000", 2024-09-13T13:14:09.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428448+0000", 2024-09-13T13:14:09.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9350000000000001e-06 2024-09-13T13:14:09.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428454+0000", 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3140000000000003e-06 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428466+0000", 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2058999999999999e-05 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428473+0000", 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0029999999999997e-06 2024-09-13T13:14:09.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428526+0000", 2024-09-13T13:14:09.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2961999999999999e-05 2024-09-13T13:14:09.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428637+0000", 2024-09-13T13:14:09.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111221 2024-09-13T13:14:09.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:14:09.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.428742+0000", 2024-09-13T13:14:09.408 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:09.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105358 2024-09-13T13:14:09.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.430849+0000", 2024-09-13T13:14:09.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021066399999999999 2024-09-13T13:14:09.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.555922+0000", 2024-09-13T13:14:09.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12507289899999999 2024-09-13T13:14:09.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.565017+0000", 2024-09-13T13:14:09.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090949110000000007 2024-09-13T13:14:09.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.565074+0000", 2024-09-13T13:14:09.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7535999999999999e-05 2024-09-13T13:14:09.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:17.565115+0000", 2024-09-13T13:14:09.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.057e-05 2024-09-13T13:14:09.411 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.411 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.411 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:169641 9.e 9:78334604:::benchmark_data_smithi052_128427_object38972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:22.418468+0000", 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 106.92258526000001, 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.169452977, 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169641 2024-09-13T13:14:09.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418468+0000", 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418468+0000", 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418469+0000", 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3099999999999996e-07 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418472+0000", 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0730000000000001e-06 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418478+0000", 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6869999999999999e-06 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418483+0000", 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.899e-06 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.421223+0000", 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027399949999999998 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.421339+0000", 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116055 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:14:09.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.421457+0000", 2024-09-13T13:14:09.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118276 2024-09-13T13:14:09.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.426437+0000", 2024-09-13T13:14:09.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049803579999999998 2024-09-13T13:14:09.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.429739+0000", 2024-09-13T13:14:09.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033021919999999998 2024-09-13T13:14:09.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.429814+0000", 2024-09-13T13:14:09.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4789999999999994e-05 2024-09-13T13:14:09.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587749+0000", 2024-09-13T13:14:09.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.15793521199999999 2024-09-13T13:14:09.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587804+0000", 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4314999999999998e-05 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587921+0000", 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011729400000000001 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:169651 9.e 9:73829da4:::benchmark_data_smithi052_128427_object38982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:09.418 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:22.425923+0000", 2024-09-13T13:14:09.419 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 106.915130699, 2024-09-13T13:14:09.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16214630499999999, 2024-09-13T13:14:09.419 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.419 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.419 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.419 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.419 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.419 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.419 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169651 2024-09-13T13:14:09.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.419 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425923+0000", 2024-09-13T13:14:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425923+0000", 2024-09-13T13:14:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425925+0000", 2024-09-13T13:14:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.153e-06 2024-09-13T13:14:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425930+0000", 2024-09-13T13:14:09.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1510000000000003e-06 2024-09-13T13:14:09.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425941+0000", 2024-09-13T13:14:09.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1219e-05 2024-09-13T13:14:09.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425948+0000", 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4590000000000003e-06 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.426533+0000", 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058506300000000003 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.426626+0000", 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2878000000000001e-05 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:14:09.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.426723+0000", 2024-09-13T13:14:09.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7369999999999998e-05 2024-09-13T13:14:09.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.429801+0000", 2024-09-13T13:14:09.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030778759999999998 2024-09-13T13:14:09.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.429820+0000", 2024-09-13T13:14:09.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9456000000000001e-05 2024-09-13T13:14:09.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.559932+0000", 2024-09-13T13:14:09.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13011128799999999 2024-09-13T13:14:09.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587959+0000", 2024-09-13T13:14:09.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028027237 2024-09-13T13:14:09.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587993+0000", 2024-09-13T13:14:09.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4285000000000001e-05 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.588069+0000", 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5870000000000004e-05 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172101 9.e 9:7b187d27:::benchmark_data_smithi052_128427_object41432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.753747+0000", 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.587305876, 2024-09-13T13:14:09.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14020244800000001, 2024-09-13T13:14:09.426 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.426 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.426 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.426 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.426 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.426 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.426 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172101 2024-09-13T13:14:09.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.426 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.753747+0000", 2024-09-13T13:14:09.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.753747+0000", 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.753749+0000", 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.561e-06 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.753754+0000", 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8040000000000001e-06 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.753761+0000", 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2350000000000002e-06 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.753766+0000", 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6249999999999998e-06 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.754624+0000", 2024-09-13T13:14:09.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00085874499999999995 2024-09-13T13:14:09.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.754802+0000", 2024-09-13T13:14:09.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178035 2024-09-13T13:14:09.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:14:09.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.754954+0000", 2024-09-13T13:14:09.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015183699999999999 2024-09-13T13:14:09.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759833+0000", 2024-09-13T13:14:09.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048790639999999998 2024-09-13T13:14:09.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759871+0000", 2024-09-13T13:14:09.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7533000000000001e-05 2024-09-13T13:14:09.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759911+0000", 2024-09-13T13:14:09.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9796000000000003e-05 2024-09-13T13:14:09.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.893830+0000", 2024-09-13T13:14:09.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13391968200000001 2024-09-13T13:14:09.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.893870+0000", 2024-09-13T13:14:09.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9932999999999999e-05 2024-09-13T13:14:09.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.893950+0000", 2024-09-13T13:14:09.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9598000000000006e-05 2024-09-13T13:14:09.432 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.432 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.432 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.432 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172108 9.e 9:76b5c7a7:::benchmark_data_smithi052_128427_object41439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:09.433 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.758691+0000", 2024-09-13T13:14:09.433 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.582362313, 2024-09-13T13:14:09.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14327330599999999, 2024-09-13T13:14:09.433 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.433 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.433 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.433 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.433 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.433 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.433 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172108 2024-09-13T13:14:09.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.433 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.758691+0000", 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.758691+0000", 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.758693+0000", 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2850000000000001e-06 2024-09-13T13:14:09.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.758699+0000", 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3369999999999999e-06 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.758708+0000", 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2280000000000004e-06 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.758715+0000", 2024-09-13T13:14:09.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2529999999999998e-06 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759950+0000", 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001234635 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760063+0000", 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113378 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:14:09.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760608+0000", 2024-09-13T13:14:09.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00054462300000000005 2024-09-13T13:14:09.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769006+0000", 2024-09-13T13:14:09.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0083978569999999999 2024-09-13T13:14:09.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769057+0000", 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1439e-05 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.772669+0000", 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003612189 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.901781+0000", 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12911177200000001 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.901867+0000", 2024-09-13T13:14:09.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5834000000000005e-05 2024-09-13T13:14:09.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.901964+0000", 2024-09-13T13:14:09.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7476e-05 2024-09-13T13:14:09.439 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.439 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.439 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.439 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172109 9.e 9:7a4e12b2:::benchmark_data_smithi052_128427_object41440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:09.440 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.759294+0000", 2024-09-13T13:14:09.440 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.58175972799999, 2024-09-13T13:14:09.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14287778800000001, 2024-09-13T13:14:09.440 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.440 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.440 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.440 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.440 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.440 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.440 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172109 2024-09-13T13:14:09.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.440 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759294+0000", 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759294+0000", 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759295+0000", 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-09-13T13:14:09.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759303+0000", 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6380000000000004e-06 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759324+0000", 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.16e-05 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.759332+0000", 2024-09-13T13:14:09.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3599999999999998e-06 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.763556+0000", 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042248720000000002 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.763639+0000", 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2668000000000002e-05 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:14:09.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.763741+0000", 2024-09-13T13:14:09.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010195 2024-09-13T13:14:09.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769103+0000", 2024-09-13T13:14:09.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0053623550000000001 2024-09-13T13:14:09.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769151+0000", 2024-09-13T13:14:09.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7472000000000002e-05 2024-09-13T13:14:09.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.772674+0000", 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035229229999999999 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902009+0000", 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.129334853 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902054+0000", 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4892000000000003e-05 2024-09-13T13:14:09.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902171+0000", 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117885 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172112 9.e 9:778ddad4:::benchmark_data_smithi052_128427_object41443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.760357+0000", 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.5806961, 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14215998599999999, 2024-09-13T13:14:09.446 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.447 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.447 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.447 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.447 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.447 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.447 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172112 2024-09-13T13:14:09.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.447 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760357+0000", 2024-09-13T13:14:09.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760357+0000", 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760359+0000", 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.494e-06 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760364+0000", 2024-09-13T13:14:09.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9359999999999998e-06 2024-09-13T13:14:09.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760374+0000", 2024-09-13T13:14:09.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0584999999999999e-05 2024-09-13T13:14:09.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760381+0000", 2024-09-13T13:14:09.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1389999999999998e-06 2024-09-13T13:14:09.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766044+0000", 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056627689999999998 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766169+0000", 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012447900000000001 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766305+0000", 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013649800000000001 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.772633+0000", 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063277079999999996 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.772661+0000", 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8588000000000001e-05 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.772677+0000", 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.535e-05 2024-09-13T13:14:09.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902234+0000", 2024-09-13T13:14:09.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12955698600000001 2024-09-13T13:14:09.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902412+0000", 2024-09-13T13:14:09.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000177756 2024-09-13T13:14:09.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902517+0000", 2024-09-13T13:14:09.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105698 2024-09-13T13:14:09.453 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.454 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.454 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.454 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172113 9.3 9:ca684d45:::benchmark_data_smithi052_128427_object41444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:09.454 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.760502+0000", 2024-09-13T13:14:09.454 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.58055157699999, 2024-09-13T13:14:09.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13345124, 2024-09-13T13:14:09.454 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.454 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.454 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.454 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.454 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172113 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760502+0000", 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760502+0000", 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760503+0000", 2024-09-13T13:14:09.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.035e-06 2024-09-13T13:14:09.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760506+0000", 2024-09-13T13:14:09.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6200000000000001e-06 2024-09-13T13:14:09.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760513+0000", 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.601e-06 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760519+0000", 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.032e-06 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760579+0000", 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9484999999999997e-05 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760677+0000", 2024-09-13T13:14:09.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8138000000000001e-05 2024-09-13T13:14:09.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:14:09.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.760777+0000", 2024-09-13T13:14:09.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100692 2024-09-13T13:14:09.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.763309+0000", 2024-09-13T13:14:09.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025319129999999998 2024-09-13T13:14:09.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766405+0000", 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030952670000000001 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.893840+0000", 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12743517200000001 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.893904+0000", 2024-09-13T13:14:09.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4003000000000004e-05 2024-09-13T13:14:09.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.893953+0000", 2024-09-13T13:14:09.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9282e-05 2024-09-13T13:14:09.460 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.460 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.460 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.460 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172114 9.d 9:ba712465:::benchmark_data_smithi052_128427_object41445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:09.460 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.762227+0000", 2024-09-13T13:14:09.461 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.578826796, 2024-09-13T13:14:09.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13646831200000001, 2024-09-13T13:14:09.461 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.461 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.461 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.461 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.461 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.461 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.461 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172114 2024-09-13T13:14:09.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.461 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762227+0000", 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762227+0000", 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762228+0000", 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5290000000000001e-06 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762233+0000", 2024-09-13T13:14:09.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4680000000000003e-06 2024-09-13T13:14:09.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762242+0000", 2024-09-13T13:14:09.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6399999999999992e-06 2024-09-13T13:14:09.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762249+0000", 2024-09-13T13:14:09.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9990000000000002e-06 2024-09-13T13:14:09.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762294+0000", 2024-09-13T13:14:09.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5114999999999997e-05 2024-09-13T13:14:09.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762394+0000", 2024-09-13T13:14:09.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9232000000000001e-05 2024-09-13T13:14:09.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:14:09.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.762493+0000", 2024-09-13T13:14:09.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9932000000000005e-05 2024-09-13T13:14:09.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.767265+0000", 2024-09-13T13:14:09.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047712140000000002 2024-09-13T13:14:09.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.771179+0000", 2024-09-13T13:14:09.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039145350000000002 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.898613+0000", 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12743336799999999 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.898652+0000", 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9481999999999999e-05 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.898695+0000", 2024-09-13T13:14:09.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2797999999999999e-05 2024-09-13T13:14:09.467 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.467 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.467 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.467 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172120 9.e 9:700d9b1c:::benchmark_data_smithi052_128427_object41451:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:09.467 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.766538+0000", 2024-09-13T13:14:09.467 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.574515179, 2024-09-13T13:14:09.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.136210366, 2024-09-13T13:14:09.467 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.469 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.469 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.469 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.469 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.469 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.469 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172120 2024-09-13T13:14:09.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.469 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766538+0000", 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766538+0000", 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766540+0000", 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.686e-06 2024-09-13T13:14:09.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766544+0000", 2024-09-13T13:14:09.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9400000000000004e-06 2024-09-13T13:14:09.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766560+0000", 2024-09-13T13:14:09.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6358000000000002e-05 2024-09-13T13:14:09.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.766568+0000", 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9170000000000006e-06 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769182+0000", 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026134589999999998 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769301+0000", 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119703 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:14:09.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.769434+0000", 2024-09-13T13:14:09.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000133242 2024-09-13T13:14:09.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.772679+0000", 2024-09-13T13:14:09.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032450090000000001 2024-09-13T13:14:09.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.773396+0000", 2024-09-13T13:14:09.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00071618400000000003 2024-09-13T13:14:09.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.896041+0000", 2024-09-13T13:14:09.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.122645191 2024-09-13T13:14:09.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902575+0000", 2024-09-13T13:14:09.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065343750000000003 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902637+0000", 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1747000000000004e-05 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.902749+0000", 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111555 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.476 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:172122 9.d 9:bd4fcb71:::benchmark_data_smithi052_128427_object41453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:09.476 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:24.768424+0000", 2024-09-13T13:14:09.476 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.572629678, 2024-09-13T13:14:09.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13067949700000001, 2024-09-13T13:14:09.476 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.476 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.476 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.476 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:09.476 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:14:09.476 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:09.476 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 172122 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768424+0000", 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768424+0000", 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768425+0000", 2024-09-13T13:14:09.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3909999999999999e-06 2024-09-13T13:14:09.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768430+0000", 2024-09-13T13:14:09.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9540000000000003e-06 2024-09-13T13:14:09.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768440+0000", 2024-09-13T13:14:09.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6160000000000008e-06 2024-09-13T13:14:09.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768447+0000", 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7109999999999995e-06 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768540+0000", 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2726999999999995e-05 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768703+0000", 2024-09-13T13:14:09.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016333999999999999 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,5", 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.768828+0000", 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124356 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.771519+0000", 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002691681 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.771667+0000", 2024-09-13T13:14:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014779300000000001 2024-09-13T13:14:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.899015+0000", 2024-09-13T13:14:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12734757899999999 2024-09-13T13:14:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.899066+0000", 2024-09-13T13:14:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0747e-05 2024-09-13T13:14:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:24.899103+0000", 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7601999999999997e-05 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.15789.0:225197 9.5 e318/241 9:a03a3a8e:::benchmark_data_smithi052_128427_object94528:head v 318'14021, mlcod=318'14017)", 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:13:18.507551+0000", 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.833502181999997, 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.027205933000000002, 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:09.482 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:09.483 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:09.483 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:09.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:09.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:18.507551+0000", 2024-09-13T13:14:09.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:09.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:18.507551+0000", 2024-09-13T13:14:09.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:09.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:09.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:18.507553+0000", 2024-09-13T13:14:09.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5829999999999999e-06 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:18.507560+0000", 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4340000000000003e-06 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:18.507569+0000", 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0159999999999997e-06 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:18.507576+0000", 2024-09-13T13:14:09.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6370000000000002e-06 2024-09-13T13:14:09.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:09.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:18.507685+0000", 2024-09-13T13:14:09.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109456 2024-09-13T13:14:09.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:09.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:18.507708+0000", 2024-09-13T13:14:09.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2585000000000001e-05 2024-09-13T13:14:09.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:18.534730+0000", 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.027021829000000001 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:13:18.534757+0000", 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7393e-05 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:09.487 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:14:09.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:09.489 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:09.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:09.590 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:09.626 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:09.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:09.724 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:14:09.728 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:09.802 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:09.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:09.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:09.905 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:09.937 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:09.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:10.016 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:14:10.039 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:10.073 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:10.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:10.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:10.175 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:10.199 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:10.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:10.294 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 6 16 17297 17281 179.995 141.562 0.00489606 0.00554946 2024-09-13T13:14:10.296 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:10 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:14:10.296 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:10 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:14:10.296 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:10 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3452780670' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:14:10.296 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:10 smithi052 ceph-mon[29471]: osdmap e341: 8 total, 8 up, 8 in 2024-09-13T13:14:10.296 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:10 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3148410785' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:14:10.297 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:10 smithi052 ceph-mon[29471]: from='client.26083 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:14:10.297 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:10 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:14:10.297 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:10 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:14:10.297 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:10 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3452780670' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:14:10.297 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:10 smithi052 ceph-mon[36682]: osdmap e341: 8 total, 8 up, 8 in 2024-09-13T13:14:10.297 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:10 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3148410785' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:14:10.297 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:10 smithi052 ceph-mon[36682]: from='client.26083 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:14:10.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:10.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:10 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:14:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:10 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:14:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:10 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3452780670' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:14:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:10 smithi167 ceph-mon[33178]: osdmap e341: 8 total, 8 up, 8 in 2024-09-13T13:14:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:10 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3148410785' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:14:10.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:10 smithi167 ceph-mon[33178]: from='client.26083 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:14:10.333 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:10.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:10.355 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T13:14:10.356 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:14:10.435 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:10.469 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:10.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:10.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:10.572 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:10.605 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:10.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:10.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:10.743 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:10.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:10.845 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:10.886 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:10.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:10.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:10.988 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:11.028 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:11.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:11.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:11.090 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T13:14:11.130 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:11.175 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:11.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:11.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:11.218 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:11.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:11.283 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:11.294 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 7 16 19822 19806 176.823 157.812 0.00715932 0.0056485 2024-09-13T13:14:11.314 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:11.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:11.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:11 smithi167 ceph-mon[33178]: pgmap v651: 148 pgs: 148 active+clean; 649 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-13T13:14:11.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:11 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:11.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:11 smithi167 ceph-mon[33178]: from='client.26083 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:14:11.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:11 smithi167 ceph-mon[33178]: osdmap e342: 8 total, 8 up, 8 in 2024-09-13T13:14:11.358 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:11 smithi052 ceph-mon[29471]: pgmap v651: 148 pgs: 148 active+clean; 649 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-13T13:14:11.358 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:11 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:11.359 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:11 smithi052 ceph-mon[29471]: from='client.26083 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:14:11.359 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:11 smithi052 ceph-mon[29471]: osdmap e342: 8 total, 8 up, 8 in 2024-09-13T13:14:11.359 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:11 smithi052 ceph-mon[36682]: pgmap v651: 148 pgs: 148 active+clean; 649 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-13T13:14:11.359 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:11 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:11.359 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:11 smithi052 ceph-mon[36682]: from='client.26083 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:14:11.359 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:11 smithi052 ceph-mon[36682]: osdmap e342: 8 total, 8 up, 8 in 2024-09-13T13:14:11.416 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:11.454 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:11.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:11.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:11.505 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:14:11.557 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:11.600 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:11.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:11.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:11.738 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:11.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:11.839 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:11.867 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:11.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:11.968 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:12.003 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:12.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:12.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:12.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:12.142 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:12.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:12.150 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:14:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:14:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:21774 10.a 10:5cd350c5:::benchmark_data_smithi052_153703_object21773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-13T13:14:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:12.139591+0000", 2024-09-13T13:14:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0078114259999999998, 2024-09-13T13:14:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.007841743, 2024-09-13T13:14:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:14:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:14:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21774 2024-09-13T13:14:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139591+0000", 2024-09-13T13:14:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139591+0000", 2024-09-13T13:14:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139593+0000", 2024-09-13T13:14:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.697e-06 2024-09-13T13:14:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139653+0000", 2024-09-13T13:14:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9895000000000002e-05 2024-09-13T13:14:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139671+0000", 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7425e-05 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139685+0000", 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4549000000000001e-05 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139761+0000", 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6019000000000005e-05 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139909+0000", 2024-09-13T13:14:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014817699999999999 2024-09-13T13:14:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:14:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.140053+0000", 2024-09-13T13:14:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014389399999999999 2024-09-13T13:14:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.141168+0000", 2024-09-13T13:14:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001114624 2024-09-13T13:14:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.144496+0000", 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033277570000000002 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:21776 10.0 10:06b5bbd2:::benchmark_data_smithi052_153703_object21775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:12.139708+0000", 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0076952619999999996, 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.007805612, 2024-09-13T13:14:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:14:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:14:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21776 2024-09-13T13:14:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139708+0000", 2024-09-13T13:14:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139708+0000", 2024-09-13T13:14:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139708+0000", 2024-09-13T13:14:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8899999999999998e-07 2024-09-13T13:14:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139740+0000", 2024-09-13T13:14:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1168999999999999e-05 2024-09-13T13:14:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139746+0000", 2024-09-13T13:14:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8260000000000004e-06 2024-09-13T13:14:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139752+0000", 2024-09-13T13:14:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.293e-06 2024-09-13T13:14:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.139805+0000", 2024-09-13T13:14:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.363e-05 2024-09-13T13:14:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.140532+0000", 2024-09-13T13:14:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072673699999999998 2024-09-13T13:14:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:14:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.140667+0000", 2024-09-13T13:14:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013463900000000001 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.141795+0000", 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001127833 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.143859+0000", 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002064607 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:21779 10.a 10:588417d8:::benchmark_data_smithi052_153703_object21778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:12.141186+0000", 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0062165679999999996, 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00641013, 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21779 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.141186+0000", 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.141186+0000", 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.141188+0000", 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.886e-06 2024-09-13T13:14:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.141219+0000", 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1025000000000002e-05 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.141229+0000", 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4800000000000007e-06 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.141235+0000", 2024-09-13T13:14:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.302e-06 2024-09-13T13:14:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.141283+0000", 2024-09-13T13:14:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7895999999999997e-05 2024-09-13T13:14:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.141406+0000", 2024-09-13T13:14:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012313800000000001 2024-09-13T13:14:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.141533+0000", 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000127028 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.143904+0000", 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023713549999999999 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.144577+0000", 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00067278200000000002 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:21784 10.a 10:55a56b8e:::benchmark_data_smithi052_153703_object21783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:12.142696+0000", 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0047068609999999997, 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049905749999999997, 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21784 2024-09-13T13:14:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.142696+0000", 2024-09-13T13:14:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.142696+0000", 2024-09-13T13:14:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.142697+0000", 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.384e-06 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.142792+0000", 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4798999999999998e-05 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.142806+0000", 2024-09-13T13:14:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.382e-05 2024-09-13T13:14:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.142814+0000", 2024-09-13T13:14:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0479999999999999e-06 2024-09-13T13:14:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.142864+0000", 2024-09-13T13:14:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9838e-05 2024-09-13T13:14:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.142989+0000", 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012530899999999999 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.143123+0000", 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013400599999999999 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.144072+0000", 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00094914100000000003 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.144718+0000", 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064529699999999997 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:21787 10.b 10:d7a8d1ad:::benchmark_data_smithi052_153703_object21786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:12.144553+0000", 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0028502810000000001, 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032123030000000001, 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21787 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.144553+0000", 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.144553+0000", 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.144554+0000", 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5149999999999999e-06 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.144582+0000", 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7949e-05 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.144593+0000", 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0618000000000001e-05 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.144600+0000", 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8410000000000002e-06 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145771+0000", 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001171245 2024-09-13T13:14:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145908+0000", 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000136747 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.146040+0000", 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000132087 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:21786 10.e e342/334 10:70025a1d:::benchmark_data_smithi052_153703_object21785:head v 342'1347, mlcod=342'1345)", 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:12.145615+0000", 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0017881959999999999, 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022980050000000001, 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145615+0000", 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145615+0000", 2024-09-13T13:14:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145616+0000", 2024-09-13T13:14:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.316e-06 2024-09-13T13:14:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145640+0000", 2024-09-13T13:14:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3662000000000001e-05 2024-09-13T13:14:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145652+0000", 2024-09-13T13:14:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2303000000000001e-05 2024-09-13T13:14:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145659+0000", 2024-09-13T13:14:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5100000000000004e-06 2024-09-13T13:14:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145706+0000", 2024-09-13T13:14:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7898000000000001e-05 2024-09-13T13:14:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145725+0000", 2024-09-13T13:14:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8663000000000001e-05 2024-09-13T13:14:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:21789 10.0 10:0732a645:::benchmark_data_smithi052_153703_object21788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-13T13:14:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:12.145823+0000", 2024-09-13T13:14:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.001579446, 2024-09-13T13:14:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021496589999999999, 2024-09-13T13:14:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21789 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145823+0000", 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145823+0000", 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145825+0000", 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.525e-06 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145859+0000", 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4331e-05 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145873+0000", 2024-09-13T13:14:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4059999999999999e-05 2024-09-13T13:14:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.145880+0000", 2024-09-13T13:14:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0790000000000003e-06 2024-09-13T13:14:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.146177+0000", 2024-09-13T13:14:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029664900000000001 2024-09-13T13:14:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.146373+0000", 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019619400000000001 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.146515+0000", 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014123 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:21790 10.e e342/334 10:77579a17:::benchmark_data_smithi052_153703_object21789:head v 342'1348, mlcod=342'1345)", 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:12.147300+0000", 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.000103183, 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00074113399999999996, 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147300+0000", 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147300+0000", 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147301+0000", 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.159e-06 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147324+0000", 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2663e-05 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147336+0000", 2024-09-13T13:14:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2157000000000001e-05 2024-09-13T13:14:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147342+0000", 2024-09-13T13:14:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:14:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147457+0000", 2024-09-13T13:14:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011527 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147475+0000", 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8498000000000001e-05 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:21791 10.b 10:df8bc984:::benchmark_data_smithi052_153703_object21790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:12.147363+0000", 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.9804e-05, 2024-09-13T13:14:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00073596800000000004, 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21791 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147363+0000", 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147363+0000", 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147364+0000", 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.33e-06 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147393+0000", 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8271000000000001e-05 2024-09-13T13:14:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147402+0000", 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0340000000000002e-06 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147408+0000", 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4640000000000003e-06 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147496+0000", 2024-09-13T13:14:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7834e-05 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147618+0000", 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121507 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:12.147732+0000", 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114715 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:14:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:14:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 9 2024-09-13T13:14:12.194 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:14:12.244 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:12.282 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:12.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:12.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:12.295 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 8 16 22167 22151 173.039 146.562 0.00701797 0.00577307 2024-09-13T13:14:12.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:12.417 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:12.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:12.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:12.552 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:12.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:12.654 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:12.694 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:12.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:12.793 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:14:12.793 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:14:12.793 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:14:12.793 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:14:12.793 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:14:12.793 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:14:12.794 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:14:12.794 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:14:12.795 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:12.831 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:12.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:12.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:12.833 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:14:12.833 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:14:12.866 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:14:12.866 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:14:12.901 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:14:12.901 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:14:12.933 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:14:12.933 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:14:12.933 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:12.978 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:12.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:13.080 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:13.103 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:14:13.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:13.107 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:13.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:13.127 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:13 smithi167 ceph-mon[33178]: pgmap v653: 148 pgs: 148 active+clean; 963 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 742 B/s rd, 140 MiB/s wr, 2.23k op/s 2024-09-13T13:14:13.127 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:13 smithi167 ceph-mon[33178]: 4.10 scrub starts 2024-09-13T13:14:13.127 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:13 smithi167 ceph-mon[33178]: 4.10 scrub ok 2024-09-13T13:14:13.127 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/602686562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:13.208 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:13.249 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:13.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:13.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:13.295 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 9 15 24444 24429 169.63 142.375 0.0042833 0.00588876 2024-09-13T13:14:13.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:13.389 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T13:14:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:13.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:13.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:13 smithi052 ceph-mon[36682]: pgmap v653: 148 pgs: 148 active+clean; 963 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 742 B/s rd, 140 MiB/s wr, 2.23k op/s 2024-09-13T13:14:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:13 smithi052 ceph-mon[36682]: 4.10 scrub starts 2024-09-13T13:14:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:13 smithi052 ceph-mon[36682]: 4.10 scrub ok 2024-09-13T13:14:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/602686562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:13.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:13 smithi052 ceph-mon[29471]: pgmap v653: 148 pgs: 148 active+clean; 963 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 742 B/s rd, 140 MiB/s wr, 2.23k op/s 2024-09-13T13:14:13.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:13 smithi052 ceph-mon[29471]: 4.10 scrub starts 2024-09-13T13:14:13.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:13 smithi052 ceph-mon[29471]: 4.10 scrub ok 2024-09-13T13:14:13.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/602686562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:13.497 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:13.522 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:13.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:13.625 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:13.666 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:13.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:13.767 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:13.802 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:13.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:13.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:13.816 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:14:13.904 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:13.949 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:13.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:14.052 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:14.075 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:14.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:14.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:14.176 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:14.209 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:14.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:14.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:14.279 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:14 smithi167 ceph-mon[33178]: 5.19 scrub starts 2024-09-13T13:14:14.279 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:14 smithi167 ceph-mon[33178]: 5.19 scrub ok 2024-09-13T13:14:14.279 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:14 smithi167 ceph-mon[33178]: 3.15 scrub starts 2024-09-13T13:14:14.279 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:14 smithi167 ceph-mon[33178]: 3.15 scrub ok 2024-09-13T13:14:14.295 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 10 15 26639 26624 166.384 137.188 0.00559066 0.00600317 2024-09-13T13:14:14.311 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:14.342 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:14.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:14.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:14.460 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:14:14.460 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:14:14.477 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:14.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:14.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:14.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:14 smithi052 ceph-mon[36682]: 5.19 scrub starts 2024-09-13T13:14:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:14 smithi052 ceph-mon[36682]: 5.19 scrub ok 2024-09-13T13:14:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:14 smithi052 ceph-mon[36682]: 3.15 scrub starts 2024-09-13T13:14:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:14 smithi052 ceph-mon[36682]: 3.15 scrub ok 2024-09-13T13:14:14.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:14 smithi052 ceph-mon[29471]: 5.19 scrub starts 2024-09-13T13:14:14.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:14 smithi052 ceph-mon[29471]: 5.19 scrub ok 2024-09-13T13:14:14.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:14 smithi052 ceph-mon[29471]: 3.15 scrub starts 2024-09-13T13:14:14.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:14 smithi052 ceph-mon[29471]: 3.15 scrub ok 2024-09-13T13:14:14.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:14:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:14:14.580 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:14.614 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:14.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:14.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:14.716 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:14.748 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:14.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:14.850 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:14.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:14.878 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:14.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:14.907 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:14.979 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:15.014 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:15.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:15.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:15.098 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:15 smithi052 ceph-mon[29471]: 5.16 scrub starts 2024-09-13T13:14:15.098 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:15 smithi052 ceph-mon[29471]: 5.16 scrub ok 2024-09-13T13:14:15.098 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:15 smithi052 ceph-mon[29471]: pgmap v654: 148 pgs: 148 active+clean; 1.3 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 161 MiB/s wr, 2.58k op/s 2024-09-13T13:14:15.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:15 smithi052 ceph-mon[29471]: 6.15 scrub starts 2024-09-13T13:14:15.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:15 smithi052 ceph-mon[29471]: 6.15 scrub ok 2024-09-13T13:14:15.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:15.099 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:15 smithi052 ceph-mon[36682]: 5.16 scrub starts 2024-09-13T13:14:15.099 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:15 smithi052 ceph-mon[36682]: 5.16 scrub ok 2024-09-13T13:14:15.099 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:15 smithi052 ceph-mon[36682]: pgmap v654: 148 pgs: 148 active+clean; 1.3 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 161 MiB/s wr, 2.58k op/s 2024-09-13T13:14:15.099 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:15 smithi052 ceph-mon[36682]: 6.15 scrub starts 2024-09-13T13:14:15.099 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:15 smithi052 ceph-mon[36682]: 6.15 scrub ok 2024-09-13T13:14:15.099 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:15.116 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:15.162 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:15.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:15.176 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T13:14:15.222 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:14:15.225 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:14:15.264 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:15.295 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:15.295 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 11 16 28735 28719 163.161 130.938 0.00202542 0.00611725 2024-09-13T13:14:15.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:15.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:15.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:15 smithi167 ceph-mon[33178]: 5.16 scrub starts 2024-09-13T13:14:15.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:15 smithi167 ceph-mon[33178]: 5.16 scrub ok 2024-09-13T13:14:15.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:15 smithi167 ceph-mon[33178]: pgmap v654: 148 pgs: 148 active+clean; 1.3 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 161 MiB/s wr, 2.58k op/s 2024-09-13T13:14:15.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:15 smithi167 ceph-mon[33178]: 6.15 scrub starts 2024-09-13T13:14:15.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:15 smithi167 ceph-mon[33178]: 6.15 scrub ok 2024-09-13T13:14:15.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:15.396 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:15.420 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:15.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:15.452 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:14:15.521 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:15.543 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:15.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:15.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:15.685 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:15.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:15.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:15.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:15.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:15.822 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:15.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:15.923 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:15.946 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:15.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:16.048 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:16.069 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:16.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:16.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:16.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[29471]: 5.f scrub starts 2024-09-13T13:14:16.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[29471]: 5.f scrub ok 2024-09-13T13:14:16.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[29471]: 5.c scrub starts 2024-09-13T13:14:16.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[29471]: 5.c scrub ok 2024-09-13T13:14:16.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[29471]: 3.3 scrub starts 2024-09-13T13:14:16.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[29471]: 3.3 scrub ok 2024-09-13T13:14:16.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[29471]: from='client.36125 v1:172.21.15.52:0/586414888' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:14:16.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[36682]: 5.f scrub starts 2024-09-13T13:14:16.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[36682]: 5.f scrub ok 2024-09-13T13:14:16.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[36682]: 5.c scrub starts 2024-09-13T13:14:16.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[36682]: 5.c scrub ok 2024-09-13T13:14:16.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[36682]: 3.3 scrub starts 2024-09-13T13:14:16.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[36682]: 3.3 scrub ok 2024-09-13T13:14:16.143 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-mon[36682]: from='client.36125 v1:172.21.15.52:0/586414888' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:14:16.171 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:16.209 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:16.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:16.295 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 12 16 29720 29704 154.694 61.5625 0.00538572 0.00645899 2024-09-13T13:14:16.310 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:16.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:16 smithi167 ceph-mon[33178]: 5.f scrub starts 2024-09-13T13:14:16.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:16 smithi167 ceph-mon[33178]: 5.f scrub ok 2024-09-13T13:14:16.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:16 smithi167 ceph-mon[33178]: 5.c scrub starts 2024-09-13T13:14:16.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:16 smithi167 ceph-mon[33178]: 5.c scrub ok 2024-09-13T13:14:16.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:16 smithi167 ceph-mon[33178]: 3.3 scrub starts 2024-09-13T13:14:16.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:16 smithi167 ceph-mon[33178]: 3.3 scrub ok 2024-09-13T13:14:16.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:16 smithi167 ceph-mon[33178]: from='client.36125 v1:172.21.15.52:0/586414888' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:14:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:16.339 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:16.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:16.441 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:16.479 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:16.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:16.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:14:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:14:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:14:16.582 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:16.604 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:16.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:16.606 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:14:16.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:16.743 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:16.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:16.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:16.845 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:16.875 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:16.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:16.977 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:17.007 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:17.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:17.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:17.109 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:17.146 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:17.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:17.248 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:17.288 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:17.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:17.295 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 13 16 31218 31202 149.996 93.625 0.00965063 0.00666125 2024-09-13T13:14:17.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:17 smithi167 ceph-mon[33178]: 6.14 scrub starts 2024-09-13T13:14:17.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:17 smithi167 ceph-mon[33178]: 6.14 scrub ok 2024-09-13T13:14:17.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:17 smithi167 ceph-mon[33178]: pgmap v655: 148 pgs: 148 active+clean; 1.4 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-13T13:14:17.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:17 smithi167 ceph-mon[33178]: 5.1c scrub starts 2024-09-13T13:14:17.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:17 smithi167 ceph-mon[33178]: 5.1c scrub ok 2024-09-13T13:14:17.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:17 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:14:17.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:17 smithi167 ceph-mon[33178]: 4.14 scrub starts 2024-09-13T13:14:17.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:17 smithi167 ceph-mon[33178]: 4.14 scrub ok 2024-09-13T13:14:17.390 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:17.423 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:17.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[36682]: 6.14 scrub starts 2024-09-13T13:14:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[36682]: 6.14 scrub ok 2024-09-13T13:14:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[36682]: pgmap v655: 148 pgs: 148 active+clean; 1.4 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-13T13:14:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[36682]: 5.1c scrub starts 2024-09-13T13:14:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[36682]: 5.1c scrub ok 2024-09-13T13:14:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:14:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[36682]: 4.14 scrub starts 2024-09-13T13:14:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[36682]: 4.14 scrub ok 2024-09-13T13:14:17.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[29471]: 6.14 scrub starts 2024-09-13T13:14:17.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[29471]: 6.14 scrub ok 2024-09-13T13:14:17.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[29471]: pgmap v655: 148 pgs: 148 active+clean; 1.4 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-13T13:14:17.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[29471]: 5.1c scrub starts 2024-09-13T13:14:17.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[29471]: 5.1c scrub ok 2024-09-13T13:14:17.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:14:17.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[29471]: 4.14 scrub starts 2024-09-13T13:14:17.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:17 smithi052 ceph-mon[29471]: 4.14 scrub ok 2024-09-13T13:14:17.524 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:17.553 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:17.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:17.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:17.655 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:17.689 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:17.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:17.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:17.791 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:17.833 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:17.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:17.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:17.934 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:17.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:17.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:17.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:18.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:18.092 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:18.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:18.194 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:18.216 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:18.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:18.295 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 14 16 32316 32300 144.183 68.625 0.0142095 0.00692899 2024-09-13T13:14:18.319 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:18.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:18 smithi167 ceph-mon[33178]: 5.11 scrub starts 2024-09-13T13:14:18.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:18 smithi167 ceph-mon[33178]: 5.11 scrub ok 2024-09-13T13:14:18.363 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:18.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:18.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:18.465 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:18.487 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:18.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:18.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:18 smithi052 ceph-mon[36682]: 5.11 scrub starts 2024-09-13T13:14:18.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:18 smithi052 ceph-mon[36682]: 5.11 scrub ok 2024-09-13T13:14:18.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:18 smithi052 ceph-mon[29471]: 5.11 scrub starts 2024-09-13T13:14:18.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:18 smithi052 ceph-mon[29471]: 5.11 scrub ok 2024-09-13T13:14:18.588 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:18.624 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:18.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:18.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:18.726 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:18.748 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:18.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:18.850 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:18.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:18.872 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:18.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:18.973 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:19.013 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:19.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:19.114 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:19.137 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:19.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:19.238 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:19.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:19.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:19.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:19.295 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 15 16 33605 33589 139.941 80.5625 0.00409896 0.00714006 2024-09-13T13:14:19.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:19.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:19.406 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:19.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:19.507 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:19.544 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:19.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:19.646 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:19.668 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:19.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:19.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:19.769 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:19.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:19.792 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:19.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:19.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:19 smithi167 ceph-mon[33178]: pgmap v656: 148 pgs: 148 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 576 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-09-13T13:14:19.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:19 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:14:19.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:19 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:14:19.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:19 smithi167 ceph-mon[33178]: 3.1b scrub starts 2024-09-13T13:14:19.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:19 smithi167 ceph-mon[33178]: 3.1b scrub ok 2024-09-13T13:14:19.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:19 smithi167 ceph-mon[33178]: 5.a scrub starts 2024-09-13T13:14:19.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:19 smithi167 ceph-mon[33178]: 5.a scrub ok 2024-09-13T13:14:19.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:19 smithi167 ceph-mon[33178]: 6.1d scrub starts 2024-09-13T13:14:19.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:19 smithi167 ceph-mon[33178]: 6.1d scrub ok 2024-09-13T13:14:19.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:19.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:19.931 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:19.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[36682]: pgmap v656: 148 pgs: 148 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 576 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-09-13T13:14:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:14:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:14:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[36682]: 3.1b scrub starts 2024-09-13T13:14:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[36682]: 3.1b scrub ok 2024-09-13T13:14:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[36682]: 5.a scrub starts 2024-09-13T13:14:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[36682]: 5.a scrub ok 2024-09-13T13:14:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[36682]: 6.1d scrub starts 2024-09-13T13:14:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[36682]: 6.1d scrub ok 2024-09-13T13:14:19.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[29471]: pgmap v656: 148 pgs: 148 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 576 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-09-13T13:14:19.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:14:19.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:14:19.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[29471]: 3.1b scrub starts 2024-09-13T13:14:19.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[29471]: 3.1b scrub ok 2024-09-13T13:14:19.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[29471]: 5.a scrub starts 2024-09-13T13:14:19.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[29471]: 5.a scrub ok 2024-09-13T13:14:19.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[29471]: 6.1d scrub starts 2024-09-13T13:14:19.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:19 smithi052 ceph-mon[29471]: 6.1d scrub ok 2024-09-13T13:14:20.033 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:20.074 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:20.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:20.176 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:20.208 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:20.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:20.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:20.281 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":654,"stamp":"2024-09-13T13:14:13.884310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1355291163,"num_objects":20870,"num_object_clones":0,"num_object_copies":62610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20870,"num_whiteouts":0,"num_read":1336,"num_read_kb":1195,"num_write":21506,"num_write_kb":1324617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":594037,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21528,"ondisk_log_size":21528,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7152700,"kb_used_data":3617780,"kb_used_omap":15,"kb_used_meta":3534768,"kb_avail":742808516,"statfs":{"total":767960285184,"available":760635920384,"internally_reserved":0,"allocated":3704606720,"data_stored":3700396071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":3619603225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,6,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1354694656,"num_objects":20671,"num_object_clones":0,"num_object_copies":62013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20671,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":20671,"num_write_kb":1322944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002654"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":602,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.011165+0000","last_change":"2024-09-13T13:10:32.368663+0000","last_active":"2024-09-13T13:14:09.011165+0000","last_peered":"2024-09-13T13:14:09.011165+0000","last_clean":"2024-09-13T13:14:09.011165+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:14:09.011165+0000","last_undegraded":"2024-09-13T13:14:09.011165+0000","last_fullsized":"2024-09-13T13:14:09.011165+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.368588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.368588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:33.672891+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":94,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012364+0000","last_change":"2024-09-13T13:12:53.024521+0000","last_active":"2024-09-13T13:14:10.012364+0000","last_peered":"2024-09-13T13:14:10.012364+0000","last_clean":"2024-09-13T13:14:10.012364+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:14:10.012364+0000","last_undegraded":"2024-09-13T13:14:10.012364+0000","last_fullsized":"2024-09-13T13:14:10.012364+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:12:14.337224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:14.337224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:02.792645+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":593,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.996685+0000","last_change":"2024-09-13T13:12:43.176997+0000","last_active":"2024-09-13T13:14:04.996685+0000","last_peered":"2024-09-13T13:14:04.996685+0000","last_clean":"2024-09-13T13:14:04.996685+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:14:04.996685+0000","last_undegraded":"2024-09-13T13:14:04.996685+0000","last_fullsized":"2024-09-13T13:14:04.996685+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:43.176921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:43.176921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:07.803370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012882+0000","last_change":"2024-09-13T13:13:17.976677+0000","last_active":"2024-09-13T13:14:10.012882+0000","last_peered":"2024-09-13T13:14:10.012882+0000","last_clean":"2024-09-13T13:14:10.012882+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:14:10.012882+0000","last_undegraded":"2024-09-13T13:14:10.012882+0000","last_fullsized":"2024-09-13T13:14:10.012882+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:17.976607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:17.976607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:28.478702+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":609,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.003980+0000","last_change":"2024-09-13T13:11:47.852762+0000","last_active":"2024-09-13T13:14:09.003980+0000","last_peered":"2024-09-13T13:14:09.003980+0000","last_clean":"2024-09-13T13:14:09.003980+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:14:09.003980+0000","last_undegraded":"2024-09-13T13:14:09.003980+0000","last_fullsized":"2024-09-13T13:14:09.003980+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":622,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.005870+0000","last_change":"2024-09-13T13:13:52.301500+0000","last_active":"2024-09-13T13:14:09.005870+0000","last_peered":"2024-09-13T13:14:09.005870+0000","last_clean":"2024-09-13T13:14:09.005870+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:14:09.005870+0000","last_undegraded":"2024-09-13T13:14:09.005870+0000","last_fullsized":"2024-09-13T13:14:09.005870+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:52.301450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:52.301450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:21.335731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":657,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.003336+0000","last_change":"2024-09-13T13:10:39.460577+0000","last_active":"2024-09-13T13:14:09.003336+0000","last_peered":"2024-09-13T13:14:09.003336+0000","last_clean":"2024-09-13T13:14:09.003336+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:14:09.003336+0000","last_undegraded":"2024-09-13T13:14:09.003336+0000","last_fullsized":"2024-09-13T13:14:09.003336+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:10:39.460500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.460500+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:23.017525+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":84,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012274+0000","last_change":"2024-09-13T13:12:52.997865+0000","last_active":"2024-09-13T13:14:10.012274+0000","last_peered":"2024-09-13T13:14:10.012274+0000","last_clean":"2024-09-13T13:14:10.012274+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:14:10.012274+0000","last_undegraded":"2024-09-13T13:14:10.012274+0000","last_fullsized":"2024-09-13T13:14:10.012274+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:17.305608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:17.305608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:06.792355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":675,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017687+0000","last_change":"2024-09-13T13:13:41.518378+0000","last_active":"2024-09-13T13:14:10.017687+0000","last_peered":"2024-09-13T13:14:10.017687+0000","last_clean":"2024-09-13T13:14:10.017687+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:14:10.017687+0000","last_undegraded":"2024-09-13T13:14:10.017687+0000","last_fullsized":"2024-09-13T13:14:10.017687+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:41.518322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.518322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:09.555720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":94,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012965+0000","last_change":"2024-09-13T13:12:53.033861+0000","last_active":"2024-09-13T13:14:10.012965+0000","last_peered":"2024-09-13T13:14:10.012965+0000","last_clean":"2024-09-13T13:14:10.012965+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:14:10.012965+0000","last_undegraded":"2024-09-13T13:14:10.012965+0000","last_fullsized":"2024-09-13T13:14:10.012965+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:31.738685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.738685+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:33.976427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":612,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017522+0000","last_change":"2024-09-13T13:13:43.501368+0000","last_active":"2024-09-13T13:14:10.017522+0000","last_peered":"2024-09-13T13:14:10.017522+0000","last_clean":"2024-09-13T13:14:10.017522+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:14:10.017522+0000","last_undegraded":"2024-09-13T13:14:10.017522+0000","last_fullsized":"2024-09-13T13:14:10.017522+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:43.501206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:43.501206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:55.342802+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":605,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.010600+0000","last_change":"2024-09-13T13:13:40.720121+0000","last_active":"2024-09-13T13:14:09.010600+0000","last_peered":"2024-09-13T13:14:09.010600+0000","last_clean":"2024-09-13T13:14:09.010600+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:14:09.010600+0000","last_undegraded":"2024-09-13T13:14:09.010600+0000","last_fullsized":"2024-09-13T13:14:09.010600+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:40.719936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:40.719936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:09.971736+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":642,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017499+0000","last_change":"2024-09-13T13:12:52.951531+0000","last_active":"2024-09-13T13:14:10.017499+0000","last_peered":"2024-09-13T13:14:10.017499+0000","last_clean":"2024-09-13T13:14:10.017499+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:14:10.017499+0000","last_undegraded":"2024-09-13T13:14:10.017499+0000","last_fullsized":"2024-09-13T13:14:10.017499+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:16.989972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:16.989972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:33.371689+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":90,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013060+0000","last_change":"2024-09-13T13:12:53.028266+0000","last_active":"2024-09-13T13:14:10.013060+0000","last_peered":"2024-09-13T13:14:10.013060+0000","last_clean":"2024-09-13T13:14:10.013060+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:14:10.013060+0000","last_undegraded":"2024-09-13T13:14:10.013060+0000","last_fullsized":"2024-09-13T13:14:10.013060+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:11:53.120451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:53.120451+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:01.777114+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":633,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.023966+0000","last_change":"2024-09-13T13:13:54.122282+0000","last_active":"2024-09-13T13:14:10.023966+0000","last_peered":"2024-09-13T13:14:10.023966+0000","last_clean":"2024-09-13T13:14:10.023966+0000","last_became_active":"2024-09-13T13:12:52.999970+0000","last_became_peered":"2024-09-13T13:12:52.999970+0000","last_unstale":"2024-09-13T13:14:10.023966+0000","last_undegraded":"2024-09-13T13:14:10.023966+0000","last_fullsized":"2024-09-13T13:14:10.023966+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.122222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.122222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:17.226969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":584,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017161+0000","last_change":"2024-09-13T13:13:02.818644+0000","last_active":"2024-09-13T13:14:10.017161+0000","last_peered":"2024-09-13T13:14:10.017161+0000","last_clean":"2024-09-13T13:14:10.017161+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:14:10.017161+0000","last_undegraded":"2024-09-13T13:14:10.017161+0000","last_fullsized":"2024-09-13T13:14:10.017161+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:02.818543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:02.818543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:28.305833+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":88,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.014033+0000","last_change":"2024-09-13T13:12:53.014177+0000","last_active":"2024-09-13T13:14:10.014033+0000","last_peered":"2024-09-13T13:14:10.014033+0000","last_clean":"2024-09-13T13:14:10.014033+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:14:10.014033+0000","last_undegraded":"2024-09-13T13:14:10.014033+0000","last_fullsized":"2024-09-13T13:14:10.014033+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:28.804735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:28.804735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:37.904645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":623,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.010588+0000","last_change":"2024-09-13T13:13:09.953731+0000","last_active":"2024-09-13T13:14:09.010588+0000","last_peered":"2024-09-13T13:14:09.010588+0000","last_clean":"2024-09-13T13:14:09.010588+0000","last_became_active":"2024-09-13T13:12:53.008505+0000","last_became_peered":"2024-09-13T13:12:53.008505+0000","last_unstale":"2024-09-13T13:14:09.010588+0000","last_undegraded":"2024-09-13T13:14:09.010588+0000","last_fullsized":"2024-09-13T13:14:09.010588+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:09.953666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:09.953666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:17.969711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":660,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.022040+0000","last_change":"2024-09-13T13:12:40.125646+0000","last_active":"2024-09-13T13:14:10.022040+0000","last_peered":"2024-09-13T13:14:10.022040+0000","last_clean":"2024-09-13T13:14:10.022040+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:14:10.022040+0000","last_undegraded":"2024-09-13T13:14:10.022040+0000","last_fullsized":"2024-09-13T13:14:10.022040+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:12:40.125545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.125545+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:06.427906+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":623,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.997724+0000","last_change":"2024-09-13T13:13:20.716705+0000","last_active":"2024-09-13T13:14:04.997724+0000","last_peered":"2024-09-13T13:14:04.997724+0000","last_clean":"2024-09-13T13:14:04.997724+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:14:04.997724+0000","last_undegraded":"2024-09-13T13:14:04.997724+0000","last_fullsized":"2024-09-13T13:14:04.997724+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:20.716633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:20.716633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:46.465567+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":610,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.005510+0000","last_change":"2024-09-13T13:13:03.150524+0000","last_active":"2024-09-13T13:14:09.005510+0000","last_peered":"2024-09-13T13:14:09.005510+0000","last_clean":"2024-09-13T13:14:09.005510+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:14:09.005510+0000","last_undegraded":"2024-09-13T13:14:09.005510+0000","last_fullsized":"2024-09-13T13:14:09.005510+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:03.150442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.150442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:30.195272+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012719+0000","last_change":"2024-09-13T13:13:39.876364+0000","last_active":"2024-09-13T13:14:10.012719+0000","last_peered":"2024-09-13T13:14:10.012719+0000","last_clean":"2024-09-13T13:14:10.012719+0000","last_became_active":"2024-09-13T13:12:52.995055+0000","last_became_peered":"2024-09-13T13:12:52.995055+0000","last_unstale":"2024-09-13T13:14:10.012719+0000","last_undegraded":"2024-09-13T13:14:10.012719+0000","last_fullsized":"2024-09-13T13:14:10.012719+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:39.876294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:39.876294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:45.791176+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":676,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.004062+0000","last_change":"2024-09-13T13:13:41.164741+0000","last_active":"2024-09-13T13:14:09.004062+0000","last_peered":"2024-09-13T13:14:09.004062+0000","last_clean":"2024-09-13T13:14:09.004062+0000","last_became_active":"2024-09-13T13:12:52.953819+0000","last_became_peered":"2024-09-13T13:12:52.953819+0000","last_unstale":"2024-09-13T13:14:09.004062+0000","last_undegraded":"2024-09-13T13:14:09.004062+0000","last_fullsized":"2024-09-13T13:14:09.004062+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:13:41.164656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.164656+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:45.115584+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":78,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013530+0000","last_change":"2024-09-13T13:13:08.018161+0000","last_active":"2024-09-13T13:14:10.013530+0000","last_peered":"2024-09-13T13:14:10.013530+0000","last_clean":"2024-09-13T13:14:10.013530+0000","last_became_active":"2024-09-13T13:13:06.646781+0000","last_became_peered":"2024-09-13T13:13:06.646781+0000","last_unstale":"2024-09-13T13:14:10.013530+0000","last_undegraded":"2024-09-13T13:14:10.013530+0000","last_fullsized":"2024-09-13T13:14:10.013530+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:08.018089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:08.018089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:12.578734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":119,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017277+0000","last_change":"2024-09-13T13:12:42.975335+0000","last_active":"2024-09-13T13:14:10.017277+0000","last_peered":"2024-09-13T13:14:10.017277+0000","last_clean":"2024-09-13T13:14:10.017277+0000","last_became_active":"2024-09-13T13:12:39.954363+0000","last_became_peered":"2024-09-13T13:12:39.954363+0000","last_unstale":"2024-09-13T13:14:10.017277+0000","last_undegraded":"2024-09-13T13:14:10.017277+0000","last_fullsized":"2024-09-13T13:14:10.017277+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:12:42.975266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:42.975266+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:00.584923+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":624,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.996845+0000","last_change":"2024-09-13T13:13:53.796606+0000","last_active":"2024-09-13T13:14:04.996845+0000","last_peered":"2024-09-13T13:14:04.996845+0000","last_clean":"2024-09-13T13:14:04.996845+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:14:04.996845+0000","last_undegraded":"2024-09-13T13:14:04.996845+0000","last_fullsized":"2024-09-13T13:14:04.996845+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:53.796530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:53.796530+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:18.032974+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":644,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.011157+0000","last_change":"2024-09-13T13:13:14.875584+0000","last_active":"2024-09-13T13:14:09.011157+0000","last_peered":"2024-09-13T13:14:09.011157+0000","last_clean":"2024-09-13T13:14:09.011157+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:14:09.011157+0000","last_undegraded":"2024-09-13T13:14:09.011157+0000","last_fullsized":"2024-09-13T13:14:09.011157+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:13:14.875507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:14.875507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:18.943661+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"306'12","reported_seq":750,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.016405+0000","last_change":"2024-09-13T13:13:02.305723+0000","last_active":"2024-09-13T13:14:10.016405+0000","last_peered":"2024-09-13T13:14:10.016405+0000","last_clean":"2024-09-13T13:14:10.016405+0000","last_became_active":"2024-09-13T13:12:52.947886+0000","last_became_peered":"2024-09-13T13:12:52.947886+0000","last_unstale":"2024-09-13T13:14:10.016405+0000","last_undegraded":"2024-09-13T13:14:10.016405+0000","last_fullsized":"2024-09-13T13:14:10.016405+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'12","last_scrub_stamp":"2024-09-13T13:13:02.305656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:02.305656+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:19.160834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"342'1497","reported_seq":1524,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:13.078527+0000","last_change":"2024-09-13T13:13:59.149493+0000","last_active":"2024-09-13T13:14:13.078527+0000","last_peered":"2024-09-13T13:14:13.078527+0000","last_clean":"2024-09-13T13:14:13.078527+0000","last_became_active":"2024-09-13T13:13:59.149215+0000","last_became_peered":"2024-09-13T13:13:59.149215+0000","last_unstale":"2024-09-13T13:14:13.078527+0000","last_undegraded":"2024-09-13T13:14:13.078527+0000","last_fullsized":"2024-09-13T13:14:13.078527+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1497,"log_dups_size":0,"ondisk_log_size":1497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:25.361862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98107392,"num_objects":1497,"num_object_clones":0,"num_object_copies":4491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1497,"num_write_kb":95808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":694,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.998145+0000","last_change":"2024-09-13T13:12:53.022032+0000","last_active":"2024-09-13T13:14:04.998145+0000","last_peered":"2024-09-13T13:14:04.998145+0000","last_clean":"2024-09-13T13:14:04.998145+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:14:04.998145+0000","last_undegraded":"2024-09-13T13:14:04.998145+0000","last_fullsized":"2024-09-13T13:14:04.998145+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:11:53.007997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:53.007997+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:59.705246+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":619,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.003875+0000","last_change":"2024-09-13T13:10:30.552177+0000","last_active":"2024-09-13T13:14:09.003875+0000","last_peered":"2024-09-13T13:14:09.003875+0000","last_clean":"2024-09-13T13:14:09.003875+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:14:09.003875+0000","last_undegraded":"2024-09-13T13:14:09.003875+0000","last_fullsized":"2024-09-13T13:14:09.003875+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:30.552113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.552113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:21.064091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":105,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.014089+0000","last_change":"2024-09-13T13:12:53.034647+0000","last_active":"2024-09-13T13:14:10.014089+0000","last_peered":"2024-09-13T13:14:10.014089+0000","last_clean":"2024-09-13T13:14:10.014089+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:14:10.014089+0000","last_undegraded":"2024-09-13T13:14:10.014089+0000","last_fullsized":"2024-09-13T13:14:10.014089+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:10:07.726751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:07.726751+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:26.088519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":616,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017383+0000","last_change":"2024-09-13T13:13:57.546725+0000","last_active":"2024-09-13T13:14:10.017383+0000","last_peered":"2024-09-13T13:14:10.017383+0000","last_clean":"2024-09-13T13:14:10.017383+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:14:10.017383+0000","last_undegraded":"2024-09-13T13:14:10.017383+0000","last_fullsized":"2024-09-13T13:14:10.017383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.546671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.546671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:16.236847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"342'1491","reported_seq":1518,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:13.068353+0000","last_change":"2024-09-13T13:13:58.962128+0000","last_active":"2024-09-13T13:14:13.068353+0000","last_peered":"2024-09-13T13:14:13.068353+0000","last_clean":"2024-09-13T13:14:13.068353+0000","last_became_active":"2024-09-13T13:13:58.961386+0000","last_became_peered":"2024-09-13T13:13:58.961386+0000","last_unstale":"2024-09-13T13:14:13.068353+0000","last_undegraded":"2024-09-13T13:14:13.068353+0000","last_fullsized":"2024-09-13T13:14:13.068353+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1491,"log_dups_size":0,"ondisk_log_size":1491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:08.925772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97714176,"num_objects":1491,"num_object_clones":0,"num_object_copies":4473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1491,"num_write_kb":95424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":641,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.002750+0000","last_change":"2024-09-13T13:13:50.232646+0000","last_active":"2024-09-13T13:14:09.002750+0000","last_peered":"2024-09-13T13:14:09.002750+0000","last_clean":"2024-09-13T13:14:09.002750+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:14:09.002750+0000","last_undegraded":"2024-09-13T13:14:09.002750+0000","last_fullsized":"2024-09-13T13:14:09.002750+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:13:50.232588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.232588+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:03.575343+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":609,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.005874+0000","last_change":"2024-09-13T13:13:54.313641+0000","last_active":"2024-09-13T13:14:09.005874+0000","last_peered":"2024-09-13T13:14:09.005874+0000","last_clean":"2024-09-13T13:14:09.005874+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:14:09.005874+0000","last_undegraded":"2024-09-13T13:14:09.005874+0000","last_fullsized":"2024-09-13T13:14:09.005874+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:54.313570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.313570+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:12.826907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":84,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012718+0000","last_change":"2024-09-13T13:12:53.006667+0000","last_active":"2024-09-13T13:14:10.012718+0000","last_peered":"2024-09-13T13:14:10.012718+0000","last_clean":"2024-09-13T13:14:10.012718+0000","last_became_active":"2024-09-13T13:12:53.006199+0000","last_became_peered":"2024-09-13T13:12:53.006199+0000","last_unstale":"2024-09-13T13:14:10.012718+0000","last_undegraded":"2024-09-13T13:14:10.012718+0000","last_fullsized":"2024-09-13T13:14:10.012718+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:59.732082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013047+0000","last_change":"2024-09-13T13:13:44.750804+0000","last_active":"2024-09-13T13:14:10.013047+0000","last_peered":"2024-09-13T13:14:10.013047+0000","last_clean":"2024-09-13T13:14:10.013047+0000","last_became_active":"2024-09-13T13:12:53.001045+0000","last_became_peered":"2024-09-13T13:12:53.001045+0000","last_unstale":"2024-09-13T13:14:10.013047+0000","last_undegraded":"2024-09-13T13:14:10.013047+0000","last_fullsized":"2024-09-13T13:14:10.013047+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:44.750717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:44.750717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:48.950292+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"342'1146","reported_seq":1173,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:11.050151+0000","last_change":"2024-09-13T13:13:58.966300+0000","last_active":"2024-09-13T13:14:11.050151+0000","last_peered":"2024-09-13T13:14:11.050151+0000","last_clean":"2024-09-13T13:14:11.050151+0000","last_became_active":"2024-09-13T13:13:58.966015+0000","last_became_peered":"2024-09-13T13:13:58.966015+0000","last_unstale":"2024-09-13T13:14:11.050151+0000","last_undegraded":"2024-09-13T13:14:11.050151+0000","last_fullsized":"2024-09-13T13:14:11.050151+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1146,"log_dups_size":0,"ondisk_log_size":1146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:05.146946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75104256,"num_objects":1146,"num_object_clones":0,"num_object_copies":3438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1146,"num_write_kb":73344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":108,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012990+0000","last_change":"2024-09-13T13:13:50.716312+0000","last_active":"2024-09-13T13:14:10.012990+0000","last_peered":"2024-09-13T13:14:10.012990+0000","last_clean":"2024-09-13T13:14:10.012990+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:14:10.012990+0000","last_undegraded":"2024-09-13T13:14:10.012990+0000","last_fullsized":"2024-09-13T13:14:10.012990+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:13:50.716243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.716243+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:59.311921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":617,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.011467+0000","last_change":"2024-09-13T13:13:57.886950+0000","last_active":"2024-09-13T13:14:09.011467+0000","last_peered":"2024-09-13T13:14:09.011467+0000","last_clean":"2024-09-13T13:14:09.011467+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:14:09.011467+0000","last_undegraded":"2024-09-13T13:14:09.011467+0000","last_fullsized":"2024-09-13T13:14:09.011467+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:57.886879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.886879+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:05.700494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":792,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.002543+0000","last_change":"2024-09-13T13:13:55.316917+0000","last_active":"2024-09-13T13:14:09.002543+0000","last_peered":"2024-09-13T13:14:09.002543+0000","last_clean":"2024-09-13T13:14:09.002543+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:14:09.002543+0000","last_undegraded":"2024-09-13T13:14:09.002543+0000","last_fullsized":"2024-09-13T13:14:09.002543+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:13:55.316860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.316860+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:10.829853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":226,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.016212+0000","last_change":"2024-09-13T13:13:52.572068+0000","last_active":"2024-09-13T13:14:10.016212+0000","last_peered":"2024-09-13T13:14:10.016212+0000","last_clean":"2024-09-13T13:14:10.016212+0000","last_became_active":"2024-09-13T13:12:39.948363+0000","last_became_peered":"2024-09-13T13:12:39.948363+0000","last_unstale":"2024-09-13T13:14:10.016212+0000","last_undegraded":"2024-09-13T13:14:10.016212+0000","last_fullsized":"2024-09-13T13:14:10.016212+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:52.571999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:52.571999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:57.263831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"342'1515","reported_seq":1541,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:13.074198+0000","last_change":"2024-09-13T13:13:59.149331+0000","last_active":"2024-09-13T13:14:13.074198+0000","last_peered":"2024-09-13T13:14:13.074198+0000","last_clean":"2024-09-13T13:14:13.074198+0000","last_became_active":"2024-09-13T13:13:59.149046+0000","last_became_peered":"2024-09-13T13:13:59.149046+0000","last_unstale":"2024-09-13T13:14:13.074198+0000","last_undegraded":"2024-09-13T13:14:13.074198+0000","last_fullsized":"2024-09-13T13:14:13.074198+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:02.773024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99287040,"num_objects":1515,"num_object_clones":0,"num_object_copies":4545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1515,"num_write_kb":96960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":674,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.997104+0000","last_change":"2024-09-13T13:12:53.016127+0000","last_active":"2024-09-13T13:14:04.997104+0000","last_peered":"2024-09-13T13:14:04.997104+0000","last_clean":"2024-09-13T13:14:04.997104+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:14:04.997104+0000","last_undegraded":"2024-09-13T13:14:04.997104+0000","last_fullsized":"2024-09-13T13:14:04.997104+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:37.707437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":629,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017797+0000","last_change":"2024-09-13T13:13:12.664590+0000","last_active":"2024-09-13T13:14:10.017797+0000","last_peered":"2024-09-13T13:14:10.017797+0000","last_clean":"2024-09-13T13:14:10.017797+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:14:10.017797+0000","last_undegraded":"2024-09-13T13:14:10.017797+0000","last_fullsized":"2024-09-13T13:14:10.017797+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:12.664516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:12.664516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:19.332370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":766,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.010100+0000","last_change":"2024-09-13T13:10:10.134973+0000","last_active":"2024-09-13T13:14:09.010100+0000","last_peered":"2024-09-13T13:14:09.010100+0000","last_clean":"2024-09-13T13:14:09.010100+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:14:09.010100+0000","last_undegraded":"2024-09-13T13:14:09.010100+0000","last_fullsized":"2024-09-13T13:14:09.010100+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:10:10.134873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.134873+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:24.741279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":622,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.010857+0000","last_change":"2024-09-13T13:13:52.836892+0000","last_active":"2024-09-13T13:14:09.010857+0000","last_peered":"2024-09-13T13:14:09.010857+0000","last_clean":"2024-09-13T13:14:09.010857+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:14:09.010857+0000","last_undegraded":"2024-09-13T13:14:09.010857+0000","last_fullsized":"2024-09-13T13:14:09.010857+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:52.836457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:52.836457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:10.105271+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"342'1232","reported_seq":1259,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:11.063852+0000","last_change":"2024-09-13T13:13:58.966260+0000","last_active":"2024-09-13T13:14:11.063852+0000","last_peered":"2024-09-13T13:14:11.063852+0000","last_clean":"2024-09-13T13:14:11.063852+0000","last_became_active":"2024-09-13T13:13:58.965950+0000","last_became_peered":"2024-09-13T13:13:58.965950+0000","last_unstale":"2024-09-13T13:14:11.063852+0000","last_undegraded":"2024-09-13T13:14:11.063852+0000","last_fullsized":"2024-09-13T13:14:11.063852+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1232,"log_dups_size":0,"ondisk_log_size":1232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:18.220013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80740352,"num_objects":1232,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1232,"num_write_kb":78848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":673,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.010311+0000","last_change":"2024-09-13T13:13:45.757306+0000","last_active":"2024-09-13T13:14:09.010311+0000","last_peered":"2024-09-13T13:14:09.010311+0000","last_clean":"2024-09-13T13:14:09.010311+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:14:09.010311+0000","last_undegraded":"2024-09-13T13:14:09.010311+0000","last_fullsized":"2024-09-13T13:14:09.010311+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:13:45.751801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:45.751801+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:07.230721+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":594,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.997112+0000","last_change":"2024-09-13T13:13:18.686583+0000","last_active":"2024-09-13T13:14:04.997112+0000","last_peered":"2024-09-13T13:14:04.997112+0000","last_clean":"2024-09-13T13:14:04.997112+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:14:04.997112+0000","last_undegraded":"2024-09-13T13:14:04.997112+0000","last_fullsized":"2024-09-13T13:14:04.997112+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:18.686400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:18.686400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:21.962212+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":573,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.015498+0000","last_change":"2024-09-13T13:13:20.475316+0000","last_active":"2024-09-13T13:14:10.015498+0000","last_peered":"2024-09-13T13:14:10.015498+0000","last_clean":"2024-09-13T13:14:10.015498+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:14:10.015498+0000","last_undegraded":"2024-09-13T13:14:10.015498+0000","last_fullsized":"2024-09-13T13:14:10.015498+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:20.475262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:20.475262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:49.688681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":609,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.996577+0000","last_change":"2024-09-13T13:13:50.772738+0000","last_active":"2024-09-13T13:14:04.996577+0000","last_peered":"2024-09-13T13:14:04.996577+0000","last_clean":"2024-09-13T13:14:04.996577+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:14:04.996577+0000","last_undegraded":"2024-09-13T13:14:04.996577+0000","last_fullsized":"2024-09-13T13:14:04.996577+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:50.772689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.772689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:56.392159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"342'1219","reported_seq":1246,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:11.054294+0000","last_change":"2024-09-13T13:13:58.966631+0000","last_active":"2024-09-13T13:14:11.054294+0000","last_peered":"2024-09-13T13:14:11.054294+0000","last_clean":"2024-09-13T13:14:11.054294+0000","last_became_active":"2024-09-13T13:13:58.965789+0000","last_became_peered":"2024-09-13T13:13:58.965789+0000","last_unstale":"2024-09-13T13:14:11.054294+0000","last_undegraded":"2024-09-13T13:14:11.054294+0000","last_fullsized":"2024-09-13T13:14:11.054294+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1219,"log_dups_size":0,"ondisk_log_size":1219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:08.633800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79888384,"num_objects":1219,"num_object_clones":0,"num_object_copies":3657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1219,"num_write_kb":78016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"268'40","reported_seq":830,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.997009+0000","last_change":"2024-09-13T13:11:54.902633+0000","last_active":"2024-09-13T13:14:04.997009+0000","last_peered":"2024-09-13T13:14:04.997009+0000","last_clean":"2024-09-13T13:14:04.997009+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:14:04.997009+0000","last_undegraded":"2024-09-13T13:14:04.997009+0000","last_fullsized":"2024-09-13T13:14:04.997009+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'40","last_scrub_stamp":"2024-09-13T13:11:54.902575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:54.902575+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:01.766110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013150+0000","last_change":"2024-09-13T13:13:04.036073+0000","last_active":"2024-09-13T13:14:10.013150+0000","last_peered":"2024-09-13T13:14:10.013150+0000","last_clean":"2024-09-13T13:14:10.013150+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:14:10.013150+0000","last_undegraded":"2024-09-13T13:14:10.013150+0000","last_fullsized":"2024-09-13T13:14:10.013150+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.036016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.036016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:11.300206+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":624,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017422+0000","last_change":"2024-09-13T13:13:20.438191+0000","last_active":"2024-09-13T13:14:10.017422+0000","last_peered":"2024-09-13T13:14:10.017422+0000","last_clean":"2024-09-13T13:14:10.017422+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:14:10.017422+0000","last_undegraded":"2024-09-13T13:14:10.017422+0000","last_fullsized":"2024-09-13T13:14:10.017422+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:13:20.438142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:20.438142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:30.205583+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":624,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.010654+0000","last_change":"2024-09-13T13:13:43.786426+0000","last_active":"2024-09-13T13:14:09.010654+0000","last_peered":"2024-09-13T13:14:09.010654+0000","last_clean":"2024-09-13T13:14:09.010654+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:14:09.010654+0000","last_undegraded":"2024-09-13T13:14:09.010654+0000","last_fullsized":"2024-09-13T13:14:09.010654+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:43.786210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:43.786210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:58.722043+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":613,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.004517+0000","last_change":"2024-09-13T13:12:52.949134+0000","last_active":"2024-09-13T13:14:09.004517+0000","last_peered":"2024-09-13T13:14:09.004517+0000","last_clean":"2024-09-13T13:14:09.004517+0000","last_became_active":"2024-09-13T13:12:52.948742+0000","last_became_peered":"2024-09-13T13:12:52.948742+0000","last_unstale":"2024-09-13T13:14:09.004517+0000","last_undegraded":"2024-09-13T13:14:09.004517+0000","last_fullsized":"2024-09-13T13:14:09.004517+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:42.489264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.489264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:31.014690+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"342'1520","reported_seq":1547,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:13.344932+0000","last_change":"2024-09-13T13:13:59.149677+0000","last_active":"2024-09-13T13:14:13.344932+0000","last_peered":"2024-09-13T13:14:13.344932+0000","last_clean":"2024-09-13T13:14:13.344932+0000","last_became_active":"2024-09-13T13:13:59.149368+0000","last_became_peered":"2024-09-13T13:13:59.149368+0000","last_unstale":"2024-09-13T13:14:13.344932+0000","last_undegraded":"2024-09-13T13:14:13.344932+0000","last_fullsized":"2024-09-13T13:14:13.344932+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:05.289043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99614720,"num_objects":1520,"num_object_clones":0,"num_object_copies":4560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1520,"num_write_kb":97280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":631,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.998282+0000","last_change":"2024-09-13T13:10:29.799315+0000","last_active":"2024-09-13T13:14:04.998282+0000","last_peered":"2024-09-13T13:14:04.998282+0000","last_clean":"2024-09-13T13:14:04.998282+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:14:04.998282+0000","last_undegraded":"2024-09-13T13:14:04.998282+0000","last_fullsized":"2024-09-13T13:14:04.998282+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:10:29.799185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:29.799185+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:30.652713+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":629,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.996194+0000","last_change":"2024-09-13T13:13:17.676405+0000","last_active":"2024-09-13T13:14:04.996194+0000","last_peered":"2024-09-13T13:14:04.996194+0000","last_clean":"2024-09-13T13:14:04.996194+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:14:04.996194+0000","last_undegraded":"2024-09-13T13:14:04.996194+0000","last_fullsized":"2024-09-13T13:14:04.996194+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:17.676339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:17.676339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:46.604117+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":637,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.022604+0000","last_change":"2024-09-13T13:13:14.560620+0000","last_active":"2024-09-13T13:14:10.022604+0000","last_peered":"2024-09-13T13:14:10.022604+0000","last_clean":"2024-09-13T13:14:10.022604+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:14:10.022604+0000","last_undegraded":"2024-09-13T13:14:10.022604+0000","last_fullsized":"2024-09-13T13:14:10.022604+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:13:14.560555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:14.560555+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:28.903951+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":601,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.022498+0000","last_change":"2024-09-13T13:13:03.320855+0000","last_active":"2024-09-13T13:14:10.022498+0000","last_peered":"2024-09-13T13:14:10.022498+0000","last_clean":"2024-09-13T13:14:10.022498+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:14:10.022498+0000","last_undegraded":"2024-09-13T13:14:10.022498+0000","last_fullsized":"2024-09-13T13:14:10.022498+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:03.320778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.320778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:20.017270+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":234,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.016152+0000","last_change":"2024-09-13T13:12:41.332308+0000","last_active":"2024-09-13T13:14:10.016152+0000","last_peered":"2024-09-13T13:14:10.016152+0000","last_clean":"2024-09-13T13:14:10.016152+0000","last_became_active":"2024-09-13T13:12:40.120242+0000","last_became_peered":"2024-09-13T13:12:40.120242+0000","last_unstale":"2024-09-13T13:14:10.016152+0000","last_undegraded":"2024-09-13T13:14:10.016152+0000","last_fullsized":"2024-09-13T13:14:10.016152+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:41.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:41.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:07.494933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"341'961","reported_seq":986,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.256841+0000","last_change":"2024-09-13T13:13:59.149922+0000","last_active":"2024-09-13T13:14:09.256841+0000","last_peered":"2024-09-13T13:14:09.256841+0000","last_clean":"2024-09-13T13:14:09.256841+0000","last_became_active":"2024-09-13T13:13:59.149037+0000","last_became_peered":"2024-09-13T13:13:59.149037+0000","last_unstale":"2024-09-13T13:14:09.256841+0000","last_undegraded":"2024-09-13T13:14:09.256841+0000","last_fullsized":"2024-09-13T13:14:09.256841+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:19.556748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62980096,"num_objects":961,"num_object_clones":0,"num_object_copies":2883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":961,"num_write_kb":61504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":642,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.011297+0000","last_change":"2024-09-13T13:13:10.964903+0000","last_active":"2024-09-13T13:14:09.011297+0000","last_peered":"2024-09-13T13:14:09.011297+0000","last_clean":"2024-09-13T13:14:09.011297+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:14:09.011297+0000","last_undegraded":"2024-09-13T13:14:09.011297+0000","last_fullsized":"2024-09-13T13:14:09.011297+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:13:10.964809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.964809+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:37.366958+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":624,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.010527+0000","last_change":"2024-09-13T13:12:53.005196+0000","last_active":"2024-09-13T13:14:09.010527+0000","last_peered":"2024-09-13T13:14:09.010527+0000","last_clean":"2024-09-13T13:14:09.010527+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:14:09.010527+0000","last_undegraded":"2024-09-13T13:14:09.010527+0000","last_fullsized":"2024-09-13T13:14:09.010527+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:37.465261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:37.465261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:27.706020+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":233,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:07.186297+0000","last_change":"2024-09-13T13:13:12.633862+0000","last_active":"2024-09-13T13:14:07.186297+0000","last_peered":"2024-09-13T13:14:07.186297+0000","last_clean":"2024-09-13T13:14:07.186297+0000","last_became_active":"2024-09-13T13:12:39.947940+0000","last_became_peered":"2024-09-13T13:12:39.947940+0000","last_unstale":"2024-09-13T13:14:07.186297+0000","last_undegraded":"2024-09-13T13:14:07.186297+0000","last_fullsized":"2024-09-13T13:14:07.186297+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:13:12.633785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:12.633785+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:36.573968+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":95,"num_read_kb":90,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":597,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.018111+0000","last_change":"2024-09-13T13:13:11.703710+0000","last_active":"2024-09-13T13:14:10.018111+0000","last_peered":"2024-09-13T13:14:10.018111+0000","last_clean":"2024-09-13T13:14:10.018111+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:14:10.018111+0000","last_undegraded":"2024-09-13T13:14:10.018111+0000","last_fullsized":"2024-09-13T13:14:10.018111+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.703647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.703647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:21.004257+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":606,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.996582+0000","last_change":"2024-09-13T13:10:32.720742+0000","last_active":"2024-09-13T13:14:04.996582+0000","last_peered":"2024-09-13T13:14:04.996582+0000","last_clean":"2024-09-13T13:14:04.996582+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:14:04.996582+0000","last_undegraded":"2024-09-13T13:14:04.996582+0000","last_fullsized":"2024-09-13T13:14:04.996582+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:31.642159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"342'1155","reported_seq":1182,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:11.046105+0000","last_change":"2024-09-13T13:13:58.967163+0000","last_active":"2024-09-13T13:14:11.046105+0000","last_peered":"2024-09-13T13:14:11.046105+0000","last_clean":"2024-09-13T13:14:11.046105+0000","last_became_active":"2024-09-13T13:13:58.965962+0000","last_became_peered":"2024-09-13T13:13:58.965962+0000","last_unstale":"2024-09-13T13:14:11.046105+0000","last_undegraded":"2024-09-13T13:14:11.046105+0000","last_fullsized":"2024-09-13T13:14:11.046105+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1155,"log_dups_size":0,"ondisk_log_size":1155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:01.549226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75694080,"num_objects":1155,"num_object_clones":0,"num_object_copies":3465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1155,"num_write_kb":73920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"306'32","reported_seq":716,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.016041+0000","last_change":"2024-09-13T13:13:01.352416+0000","last_active":"2024-09-13T13:14:10.016041+0000","last_peered":"2024-09-13T13:14:10.016041+0000","last_clean":"2024-09-13T13:14:10.016041+0000","last_became_active":"2024-09-13T13:12:52.999924+0000","last_became_peered":"2024-09-13T13:12:52.999924+0000","last_unstale":"2024-09-13T13:14:10.016041+0000","last_undegraded":"2024-09-13T13:14:10.016041+0000","last_fullsized":"2024-09-13T13:14:10.016041+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:13:01.352337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.352337+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:03.437817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":614,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.003345+0000","last_change":"2024-09-13T13:11:54.028616+0000","last_active":"2024-09-13T13:14:09.003345+0000","last_peered":"2024-09-13T13:14:09.003345+0000","last_clean":"2024-09-13T13:14:09.003345+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:14:09.003345+0000","last_undegraded":"2024-09-13T13:14:09.003345+0000","last_fullsized":"2024-09-13T13:14:09.003345+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:54.028552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:54.028552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:03.452848+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":180,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.011784+0000","last_change":"2024-09-13T13:13:06.952227+0000","last_active":"2024-09-13T13:14:09.011784+0000","last_peered":"2024-09-13T13:14:09.011784+0000","last_clean":"2024-09-13T13:14:09.011784+0000","last_became_active":"2024-09-13T13:12:39.948765+0000","last_became_peered":"2024-09-13T13:12:39.948765+0000","last_unstale":"2024-09-13T13:14:09.011784+0000","last_undegraded":"2024-09-13T13:14:09.011784+0000","last_fullsized":"2024-09-13T13:14:09.011784+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:13:06.952132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.952132+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:21.389641+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":107,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.021757+0000","last_change":"2024-09-13T13:13:17.542667+0000","last_active":"2024-09-13T13:14:10.021757+0000","last_peered":"2024-09-13T13:14:10.021757+0000","last_clean":"2024-09-13T13:14:10.021757+0000","last_became_active":"2024-09-13T13:12:39.950496+0000","last_became_peered":"2024-09-13T13:12:39.950496+0000","last_unstale":"2024-09-13T13:14:10.021757+0000","last_undegraded":"2024-09-13T13:14:10.021757+0000","last_fullsized":"2024-09-13T13:14:10.021757+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:17.542599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:17.542599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:17.868103+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":631,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017989+0000","last_change":"2024-09-13T13:12:52.959776+0000","last_active":"2024-09-13T13:14:10.017989+0000","last_peered":"2024-09-13T13:14:10.017989+0000","last_clean":"2024-09-13T13:14:10.017989+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:14:10.017989+0000","last_undegraded":"2024-09-13T13:14:10.017989+0000","last_fullsized":"2024-09-13T13:14:10.017989+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:15.030577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:15.030577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:22.466810+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"342'1236","reported_seq":1263,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:11.017455+0000","last_change":"2024-09-13T13:13:58.965986+0000","last_active":"2024-09-13T13:14:11.017455+0000","last_peered":"2024-09-13T13:14:11.017455+0000","last_clean":"2024-09-13T13:14:11.017455+0000","last_became_active":"2024-09-13T13:13:58.965566+0000","last_became_peered":"2024-09-13T13:13:58.965566+0000","last_unstale":"2024-09-13T13:14:11.017455+0000","last_undegraded":"2024-09-13T13:14:11.017455+0000","last_fullsized":"2024-09-13T13:14:11.017455+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1236,"log_dups_size":0,"ondisk_log_size":1236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:59.817222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81002496,"num_objects":1236,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1236,"num_write_kb":79104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":104,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013970+0000","last_change":"2024-09-13T13:13:38.900634+0000","last_active":"2024-09-13T13:14:10.013970+0000","last_peered":"2024-09-13T13:14:10.013970+0000","last_clean":"2024-09-13T13:14:10.013970+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:14:10.013970+0000","last_undegraded":"2024-09-13T13:14:10.013970+0000","last_fullsized":"2024-09-13T13:14:10.013970+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:38.900566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:38.900566+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:53.707734+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":84,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012589+0000","last_change":"2024-09-13T13:12:53.007182+0000","last_active":"2024-09-13T13:14:10.012589+0000","last_peered":"2024-09-13T13:14:10.012589+0000","last_clean":"2024-09-13T13:14:10.012589+0000","last_became_active":"2024-09-13T13:12:53.006271+0000","last_became_peered":"2024-09-13T13:12:53.006271+0000","last_unstale":"2024-09-13T13:14:10.012589+0000","last_undegraded":"2024-09-13T13:14:10.012589+0000","last_fullsized":"2024-09-13T13:14:10.012589+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:01.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:01.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:22.070336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":645,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017997+0000","last_change":"2024-09-13T13:13:05.774388+0000","last_active":"2024-09-13T13:14:10.017997+0000","last_peered":"2024-09-13T13:14:10.017997+0000","last_clean":"2024-09-13T13:14:10.017997+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:14:10.017997+0000","last_undegraded":"2024-09-13T13:14:10.017997+0000","last_fullsized":"2024-09-13T13:14:10.017997+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:05.774328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:05.774328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:14.054120+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":601,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.003039+0000","last_change":"2024-09-13T13:10:34.526736+0000","last_active":"2024-09-13T13:14:09.003039+0000","last_peered":"2024-09-13T13:14:09.003039+0000","last_clean":"2024-09-13T13:14:09.003039+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:14:09.003039+0000","last_undegraded":"2024-09-13T13:14:09.003039+0000","last_fullsized":"2024-09-13T13:14:09.003039+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:34.526666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:34.526666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:22.041283+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"342'1564","reported_seq":1591,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:13.384977+0000","last_change":"2024-09-13T13:13:58.962261+0000","last_active":"2024-09-13T13:14:13.384977+0000","last_peered":"2024-09-13T13:14:13.384977+0000","last_clean":"2024-09-13T13:14:13.384977+0000","last_became_active":"2024-09-13T13:13:58.961825+0000","last_became_peered":"2024-09-13T13:13:58.961825+0000","last_unstale":"2024-09-13T13:14:13.384977+0000","last_undegraded":"2024-09-13T13:14:13.384977+0000","last_fullsized":"2024-09-13T13:14:13.384977+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1564,"log_dups_size":0,"ondisk_log_size":1564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:23.673189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102498304,"num_objects":1564,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1564,"num_write_kb":100096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":110,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.014496+0000","last_change":"2024-09-13T13:12:58.980713+0000","last_active":"2024-09-13T13:14:10.014496+0000","last_peered":"2024-09-13T13:14:10.014496+0000","last_clean":"2024-09-13T13:14:10.014496+0000","last_became_active":"2024-09-13T13:12:53.001687+0000","last_became_peered":"2024-09-13T13:12:53.001687+0000","last_unstale":"2024-09-13T13:14:10.014496+0000","last_undegraded":"2024-09-13T13:14:10.014496+0000","last_fullsized":"2024-09-13T13:14:10.014496+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:12:58.980652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.980652+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:18.706915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":619,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.015982+0000","last_change":"2024-09-13T13:13:16.379485+0000","last_active":"2024-09-13T13:14:10.015982+0000","last_peered":"2024-09-13T13:14:10.015982+0000","last_clean":"2024-09-13T13:14:10.015982+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:14:10.015982+0000","last_undegraded":"2024-09-13T13:14:10.015982+0000","last_fullsized":"2024-09-13T13:14:10.015982+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:16.379422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:16.379422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:18.114367+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":607,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.996684+0000","last_change":"2024-09-13T13:13:11.661426+0000","last_active":"2024-09-13T13:14:04.996684+0000","last_peered":"2024-09-13T13:14:04.996684+0000","last_clean":"2024-09-13T13:14:04.996684+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:14:04.996684+0000","last_undegraded":"2024-09-13T13:14:04.996684+0000","last_fullsized":"2024-09-13T13:14:04.996684+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.661364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.661364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:13.689135+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":615,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.022098+0000","last_change":"2024-09-13T13:13:54.649219+0000","last_active":"2024-09-13T13:14:10.022098+0000","last_peered":"2024-09-13T13:14:10.022098+0000","last_clean":"2024-09-13T13:14:10.022098+0000","last_became_active":"2024-09-13T13:12:52.938975+0000","last_became_peered":"2024-09-13T13:12:52.938975+0000","last_unstale":"2024-09-13T13:14:10.022098+0000","last_undegraded":"2024-09-13T13:14:10.022098+0000","last_fullsized":"2024-09-13T13:14:10.022098+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.649166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.649166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:13.711439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"340'880","reported_seq":903,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:08.944972+0000","last_change":"2024-09-13T13:13:58.963327+0000","last_active":"2024-09-13T13:14:08.944972+0000","last_peered":"2024-09-13T13:14:08.944972+0000","last_clean":"2024-09-13T13:14:08.944972+0000","last_became_active":"2024-09-13T13:13:58.963064+0000","last_became_peered":"2024-09-13T13:13:58.963064+0000","last_unstale":"2024-09-13T13:14:08.944972+0000","last_undegraded":"2024-09-13T13:14:08.944972+0000","last_fullsized":"2024-09-13T13:14:08.944972+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:12.065758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57671680,"num_objects":880,"num_object_clones":0,"num_object_copies":2640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":880,"num_write_kb":56320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":96,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013665+0000","last_change":"2024-09-13T13:12:53.018645+0000","last_active":"2024-09-13T13:14:10.013665+0000","last_peered":"2024-09-13T13:14:10.013665+0000","last_clean":"2024-09-13T13:14:10.013665+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:14:10.013665+0000","last_undegraded":"2024-09-13T13:14:10.013665+0000","last_fullsized":"2024-09-13T13:14:10.013665+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:09.739184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.739184+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:28.062380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":616,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.004478+0000","last_change":"2024-09-13T13:12:53.003530+0000","last_active":"2024-09-13T13:14:09.004478+0000","last_peered":"2024-09-13T13:14:09.004478+0000","last_clean":"2024-09-13T13:14:09.004478+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:14:09.004478+0000","last_undegraded":"2024-09-13T13:14:09.004478+0000","last_fullsized":"2024-09-13T13:14:09.004478+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.996346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.996346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:06.356979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":764,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.022899+0000","last_change":"2024-09-13T13:12:36.113757+0000","last_active":"2024-09-13T13:14:10.022899+0000","last_peered":"2024-09-13T13:14:10.022899+0000","last_clean":"2024-09-13T13:14:10.022899+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:14:10.022899+0000","last_undegraded":"2024-09-13T13:14:10.022899+0000","last_fullsized":"2024-09-13T13:14:10.022899+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:12:36.113709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:36.113709+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:07.451544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013952+0000","last_change":"2024-09-13T13:13:41.797728+0000","last_active":"2024-09-13T13:14:10.013952+0000","last_peered":"2024-09-13T13:14:10.013952+0000","last_clean":"2024-09-13T13:14:10.013952+0000","last_became_active":"2024-09-13T13:12:52.997283+0000","last_became_peered":"2024-09-13T13:12:52.997283+0000","last_unstale":"2024-09-13T13:14:10.013952+0000","last_undegraded":"2024-09-13T13:14:10.013952+0000","last_fullsized":"2024-09-13T13:14:10.013952+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.797639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.797639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:58.404714+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"342'1573","reported_seq":1600,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:13.393336+0000","last_change":"2024-09-13T13:13:58.963159+0000","last_active":"2024-09-13T13:14:13.393336+0000","last_peered":"2024-09-13T13:14:13.393336+0000","last_clean":"2024-09-13T13:14:13.393336+0000","last_became_active":"2024-09-13T13:13:58.962803+0000","last_became_peered":"2024-09-13T13:13:58.962803+0000","last_unstale":"2024-09-13T13:14:13.393336+0000","last_undegraded":"2024-09-13T13:14:13.393336+0000","last_fullsized":"2024-09-13T13:14:13.393336+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:24.258620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103088128,"num_objects":1573,"num_object_clones":0,"num_object_copies":4719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1573,"num_write_kb":100672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":648,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.010719+0000","last_change":"2024-09-13T13:13:56.850149+0000","last_active":"2024-09-13T13:14:09.010719+0000","last_peered":"2024-09-13T13:14:09.010719+0000","last_clean":"2024-09-13T13:14:09.010719+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:14:09.010719+0000","last_undegraded":"2024-09-13T13:14:09.010719+0000","last_fullsized":"2024-09-13T13:14:09.010719+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:56.850094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.850094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:59.311626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":639,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017594+0000","last_change":"2024-09-13T13:12:58.857634+0000","last_active":"2024-09-13T13:14:10.017594+0000","last_peered":"2024-09-13T13:14:10.017594+0000","last_clean":"2024-09-13T13:14:10.017594+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:14:10.017594+0000","last_undegraded":"2024-09-13T13:14:10.017594+0000","last_fullsized":"2024-09-13T13:14:10.017594+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:58.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:28.262215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":636,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.020963+0000","last_change":"2024-09-13T13:13:04.341912+0000","last_active":"2024-09-13T13:14:10.020963+0000","last_peered":"2024-09-13T13:14:10.020963+0000","last_clean":"2024-09-13T13:14:10.020963+0000","last_became_active":"2024-09-13T13:12:52.944725+0000","last_became_peered":"2024-09-13T13:12:52.944725+0000","last_unstale":"2024-09-13T13:14:10.020963+0000","last_undegraded":"2024-09-13T13:14:10.020963+0000","last_fullsized":"2024-09-13T13:14:10.020963+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.341673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.341673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:15.009663+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":598,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.010558+0000","last_change":"2024-09-13T13:13:41.762003+0000","last_active":"2024-09-13T13:14:09.010558+0000","last_peered":"2024-09-13T13:14:09.010558+0000","last_clean":"2024-09-13T13:14:09.010558+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:14:09.010558+0000","last_undegraded":"2024-09-13T13:14:09.010558+0000","last_fullsized":"2024-09-13T13:14:09.010558+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.761814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.761814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:08.371463+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"341'898","reported_seq":923,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.259155+0000","last_change":"2024-09-13T13:13:58.963548+0000","last_active":"2024-09-13T13:14:09.259155+0000","last_peered":"2024-09-13T13:14:09.259155+0000","last_clean":"2024-09-13T13:14:09.259155+0000","last_became_active":"2024-09-13T13:13:58.963160+0000","last_became_peered":"2024-09-13T13:13:58.963160+0000","last_unstale":"2024-09-13T13:14:09.259155+0000","last_undegraded":"2024-09-13T13:14:09.259155+0000","last_fullsized":"2024-09-13T13:14:09.259155+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":898,"log_dups_size":0,"ondisk_log_size":898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:04.981823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58851328,"num_objects":898,"num_object_clones":0,"num_object_copies":2694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":898,"num_write_kb":57472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":624,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.015602+0000","last_change":"2024-09-13T13:13:06.339762+0000","last_active":"2024-09-13T13:14:10.015602+0000","last_peered":"2024-09-13T13:14:10.015602+0000","last_clean":"2024-09-13T13:14:10.015602+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:14:10.015602+0000","last_undegraded":"2024-09-13T13:14:10.015602+0000","last_fullsized":"2024-09-13T13:14:10.015602+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:13:06.339664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.339664+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:08.487728+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":246,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017632+0000","last_change":"2024-09-13T13:13:52.445988+0000","last_active":"2024-09-13T13:14:10.017632+0000","last_peered":"2024-09-13T13:14:10.017632+0000","last_clean":"2024-09-13T13:14:10.017632+0000","last_became_active":"2024-09-13T13:12:39.954315+0000","last_became_peered":"2024-09-13T13:12:39.954315+0000","last_unstale":"2024-09-13T13:14:10.017632+0000","last_undegraded":"2024-09-13T13:14:10.017632+0000","last_fullsized":"2024-09-13T13:14:10.017632+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:52.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:52.445926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:54.728929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":613,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.023975+0000","last_change":"2024-09-13T13:13:11.484689+0000","last_active":"2024-09-13T13:14:10.023975+0000","last_peered":"2024-09-13T13:14:10.023975+0000","last_clean":"2024-09-13T13:14:10.023975+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:14:10.023975+0000","last_undegraded":"2024-09-13T13:14:10.023975+0000","last_fullsized":"2024-09-13T13:14:10.023975+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.484615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.484615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:37.563802+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":99,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.021360+0000","last_change":"2024-09-13T13:12:40.118611+0000","last_active":"2024-09-13T13:14:10.021360+0000","last_peered":"2024-09-13T13:14:10.021360+0000","last_clean":"2024-09-13T13:14:10.021360+0000","last_became_active":"2024-09-13T13:12:40.118050+0000","last_became_peered":"2024-09-13T13:12:40.118050+0000","last_unstale":"2024-09-13T13:14:10.021360+0000","last_undegraded":"2024-09-13T13:14:10.021360+0000","last_fullsized":"2024-09-13T13:14:10.021360+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:40.961744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.961744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:01.520492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"342'1208","reported_seq":1235,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:11.054944+0000","last_change":"2024-09-13T13:13:58.966955+0000","last_active":"2024-09-13T13:14:11.054944+0000","last_peered":"2024-09-13T13:14:11.054944+0000","last_clean":"2024-09-13T13:14:11.054944+0000","last_became_active":"2024-09-13T13:13:58.966675+0000","last_became_peered":"2024-09-13T13:13:58.966675+0000","last_unstale":"2024-09-13T13:14:11.054944+0000","last_undegraded":"2024-09-13T13:14:11.054944+0000","last_fullsized":"2024-09-13T13:14:11.054944+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1208,"log_dups_size":0,"ondisk_log_size":1208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:09.947796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79167488,"num_objects":1208,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1208,"num_write_kb":77312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":104,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013134+0000","last_change":"2024-09-13T13:13:20.064693+0000","last_active":"2024-09-13T13:14:10.013134+0000","last_peered":"2024-09-13T13:14:10.013134+0000","last_clean":"2024-09-13T13:14:10.013134+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:14:10.013134+0000","last_undegraded":"2024-09-13T13:14:10.013134+0000","last_fullsized":"2024-09-13T13:14:10.013134+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:13:20.064614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:20.064614+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:42.196256+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013868+0000","last_change":"2024-09-13T13:13:42.780315+0000","last_active":"2024-09-13T13:14:10.013868+0000","last_peered":"2024-09-13T13:14:10.013868+0000","last_clean":"2024-09-13T13:14:10.013868+0000","last_became_active":"2024-09-13T13:12:53.007561+0000","last_became_peered":"2024-09-13T13:12:53.007561+0000","last_unstale":"2024-09-13T13:14:10.013868+0000","last_undegraded":"2024-09-13T13:14:10.013868+0000","last_fullsized":"2024-09-13T13:14:10.013868+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:42.780247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:42.780247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:48.627353+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":598,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.021485+0000","last_change":"2024-09-13T13:13:51.608073+0000","last_active":"2024-09-13T13:14:10.021485+0000","last_peered":"2024-09-13T13:14:10.021485+0000","last_clean":"2024-09-13T13:14:10.021485+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:14:10.021485+0000","last_undegraded":"2024-09-13T13:14:10.021485+0000","last_fullsized":"2024-09-13T13:14:10.021485+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.608025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.608025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:00.485134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":582,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.004073+0000","last_change":"2024-09-13T13:11:17.667223+0000","last_active":"2024-09-13T13:14:09.004073+0000","last_peered":"2024-09-13T13:14:09.004073+0000","last_clean":"2024-09-13T13:14:09.004073+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:14:09.004073+0000","last_undegraded":"2024-09-13T13:14:09.004073+0000","last_fullsized":"2024-09-13T13:14:09.004073+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:17.667151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:17.667151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:02.403548+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"342'1576","reported_seq":1603,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:13.355288+0000","last_change":"2024-09-13T13:13:58.963039+0000","last_active":"2024-09-13T13:14:13.355288+0000","last_peered":"2024-09-13T13:14:13.355288+0000","last_clean":"2024-09-13T13:14:13.355288+0000","last_became_active":"2024-09-13T13:13:58.962773+0000","last_became_peered":"2024-09-13T13:13:58.962773+0000","last_unstale":"2024-09-13T13:14:13.355288+0000","last_undegraded":"2024-09-13T13:14:13.355288+0000","last_fullsized":"2024-09-13T13:14:13.355288+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1576,"log_dups_size":0,"ondisk_log_size":1576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:26.925464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103284736,"num_objects":1576,"num_object_clones":0,"num_object_copies":4728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1576,"num_write_kb":100864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":671,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.016975+0000","last_change":"2024-09-13T13:13:15.581950+0000","last_active":"2024-09-13T13:14:10.016975+0000","last_peered":"2024-09-13T13:14:10.016975+0000","last_clean":"2024-09-13T13:14:10.016975+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:14:10.016975+0000","last_undegraded":"2024-09-13T13:14:10.016975+0000","last_fullsized":"2024-09-13T13:14:10.016975+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:13:15.581882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:15.581882+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:19.155967+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":98,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013655+0000","last_change":"2024-09-13T13:13:01.972907+0000","last_active":"2024-09-13T13:14:10.013655+0000","last_peered":"2024-09-13T13:14:10.013655+0000","last_clean":"2024-09-13T13:14:10.013655+0000","last_became_active":"2024-09-13T13:12:53.006362+0000","last_became_peered":"2024-09-13T13:12:53.006362+0000","last_unstale":"2024-09-13T13:14:10.013655+0000","last_undegraded":"2024-09-13T13:14:10.013655+0000","last_fullsized":"2024-09-13T13:14:10.013655+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:13:01.972831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.972831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:25.555388+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"306'13","reported_seq":318,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.015358+0000","last_change":"2024-09-13T13:13:03.334319+0000","last_active":"2024-09-13T13:14:10.015358+0000","last_peered":"2024-09-13T13:14:10.015358+0000","last_clean":"2024-09-13T13:14:10.015358+0000","last_became_active":"2024-09-13T13:12:53.001189+0000","last_became_peered":"2024-09-13T13:12:53.001189+0000","last_unstale":"2024-09-13T13:14:10.015358+0000","last_undegraded":"2024-09-13T13:14:10.015358+0000","last_fullsized":"2024-09-13T13:14:10.015358+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'13","last_scrub_stamp":"2024-09-13T13:13:03.334012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.334012+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:32.766925+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":612,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017267+0000","last_change":"2024-09-13T13:13:55.500961+0000","last_active":"2024-09-13T13:14:10.017267+0000","last_peered":"2024-09-13T13:14:10.017267+0000","last_clean":"2024-09-13T13:14:10.017267+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:14:10.017267+0000","last_undegraded":"2024-09-13T13:14:10.017267+0000","last_fullsized":"2024-09-13T13:14:10.017267+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:55.500908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.500908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:07.523230+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":589,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.015747+0000","last_change":"2024-09-13T13:13:04.371082+0000","last_active":"2024-09-13T13:14:10.015747+0000","last_peered":"2024-09-13T13:14:10.015747+0000","last_clean":"2024-09-13T13:14:10.015747+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:14:10.015747+0000","last_undegraded":"2024-09-13T13:14:10.015747+0000","last_fullsized":"2024-09-13T13:14:10.015747+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.370881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.370881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:06.461544+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":126,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017749+0000","last_change":"2024-09-13T13:13:58.537515+0000","last_active":"2024-09-13T13:14:10.017749+0000","last_peered":"2024-09-13T13:14:10.017749+0000","last_clean":"2024-09-13T13:14:10.017749+0000","last_became_active":"2024-09-13T13:12:34.673528+0000","last_became_peered":"2024-09-13T13:12:34.673528+0000","last_unstale":"2024-09-13T13:14:10.017749+0000","last_undegraded":"2024-09-13T13:14:10.017749+0000","last_fullsized":"2024-09-13T13:14:10.017749+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:13:58.537459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:58.537459+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:20.801720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":620,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017978+0000","last_change":"2024-09-13T13:13:09.732763+0000","last_active":"2024-09-13T13:14:10.017978+0000","last_peered":"2024-09-13T13:14:10.017978+0000","last_clean":"2024-09-13T13:14:10.017978+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:14:10.017978+0000","last_undegraded":"2024-09-13T13:14:10.017978+0000","last_fullsized":"2024-09-13T13:14:10.017978+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:09.732693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:09.732693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:13.447045+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":633,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.024186+0000","last_change":"2024-09-13T13:12:53.001187+0000","last_active":"2024-09-13T13:14:10.024186+0000","last_peered":"2024-09-13T13:14:10.024186+0000","last_clean":"2024-09-13T13:14:10.024186+0000","last_became_active":"2024-09-13T13:12:53.000873+0000","last_became_peered":"2024-09-13T13:12:53.000873+0000","last_unstale":"2024-09-13T13:14:10.024186+0000","last_undegraded":"2024-09-13T13:14:10.024186+0000","last_fullsized":"2024-09-13T13:14:10.024186+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:43.842274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:43.842274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:11.655379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":622,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.011365+0000","last_change":"2024-09-13T13:13:12.952427+0000","last_active":"2024-09-13T13:14:09.011365+0000","last_peered":"2024-09-13T13:14:09.011365+0000","last_clean":"2024-09-13T13:14:09.011365+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:14:09.011365+0000","last_undegraded":"2024-09-13T13:14:09.011365+0000","last_fullsized":"2024-09-13T13:14:09.011365+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:12.952348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:12.952348+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:25.638147+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012762+0000","last_change":"2024-09-13T13:13:51.764793+0000","last_active":"2024-09-13T13:14:10.012762+0000","last_peered":"2024-09-13T13:14:10.012762+0000","last_clean":"2024-09-13T13:14:10.012762+0000","last_became_active":"2024-09-13T13:12:53.006579+0000","last_became_peered":"2024-09-13T13:12:53.006579+0000","last_unstale":"2024-09-13T13:14:10.012762+0000","last_undegraded":"2024-09-13T13:14:10.012762+0000","last_fullsized":"2024-09-13T13:14:10.012762+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.764717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.764717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:01.471924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":602,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.005674+0000","last_change":"2024-09-13T13:13:56.359731+0000","last_active":"2024-09-13T13:14:09.005674+0000","last_peered":"2024-09-13T13:14:09.005674+0000","last_clean":"2024-09-13T13:14:09.005674+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:14:09.005674+0000","last_undegraded":"2024-09-13T13:14:09.005674+0000","last_fullsized":"2024-09-13T13:14:09.005674+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:56.359684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.359684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:21.764264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":639,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.021233+0000","last_change":"2024-09-13T13:13:05.369910+0000","last_active":"2024-09-13T13:14:10.021233+0000","last_peered":"2024-09-13T13:14:10.021233+0000","last_clean":"2024-09-13T13:14:10.021233+0000","last_became_active":"2024-09-13T13:12:52.950504+0000","last_became_peered":"2024-09-13T13:12:52.950504+0000","last_unstale":"2024-09-13T13:14:10.021233+0000","last_undegraded":"2024-09-13T13:14:10.021233+0000","last_fullsized":"2024-09-13T13:14:10.021233+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:05.369824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:05.369824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:14.248796+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":637,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017693+0000","last_change":"2024-09-13T13:12:34.682535+0000","last_active":"2024-09-13T13:14:10.017693+0000","last_peered":"2024-09-13T13:14:10.017693+0000","last_clean":"2024-09-13T13:14:10.017693+0000","last_became_active":"2024-09-13T13:12:34.665746+0000","last_became_peered":"2024-09-13T13:12:34.665746+0000","last_unstale":"2024-09-13T13:14:10.017693+0000","last_undegraded":"2024-09-13T13:14:10.017693+0000","last_fullsized":"2024-09-13T13:14:10.017693+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:19.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:19.931466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:34.330831+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":601,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.010952+0000","last_change":"2024-09-13T13:13:05.981881+0000","last_active":"2024-09-13T13:14:09.010952+0000","last_peered":"2024-09-13T13:14:09.010952+0000","last_clean":"2024-09-13T13:14:09.010952+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:14:09.010952+0000","last_undegraded":"2024-09-13T13:14:09.010952+0000","last_fullsized":"2024-09-13T13:14:09.010952+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:05.981803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:05.981803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:06.678092+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":760,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.003143+0000","last_change":"2024-09-13T13:13:10.130755+0000","last_active":"2024-09-13T13:14:09.003143+0000","last_peered":"2024-09-13T13:14:09.003143+0000","last_clean":"2024-09-13T13:14:09.003143+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:14:09.003143+0000","last_undegraded":"2024-09-13T13:14:09.003143+0000","last_fullsized":"2024-09-13T13:14:09.003143+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:13:10.130680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.130680+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:38.263233+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013769+0000","last_change":"2024-09-13T13:13:11.956837+0000","last_active":"2024-09-13T13:14:10.013769+0000","last_peered":"2024-09-13T13:14:10.013769+0000","last_clean":"2024-09-13T13:14:10.013769+0000","last_became_active":"2024-09-13T13:12:52.946339+0000","last_became_peered":"2024-09-13T13:12:52.946339+0000","last_unstale":"2024-09-13T13:14:10.013769+0000","last_undegraded":"2024-09-13T13:14:10.013769+0000","last_fullsized":"2024-09-13T13:14:10.013769+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.956784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.956784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:24.331443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":636,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.003392+0000","last_change":"2024-09-13T13:13:01.121726+0000","last_active":"2024-09-13T13:14:09.003392+0000","last_peered":"2024-09-13T13:14:09.003392+0000","last_clean":"2024-09-13T13:14:09.003392+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:14:09.003392+0000","last_undegraded":"2024-09-13T13:14:09.003392+0000","last_fullsized":"2024-09-13T13:14:09.003392+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:13:01.121653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.121653+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:13.207486+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":618,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.024252+0000","last_change":"2024-09-13T13:13:57.105151+0000","last_active":"2024-09-13T13:14:10.024252+0000","last_peered":"2024-09-13T13:14:10.024252+0000","last_clean":"2024-09-13T13:14:10.024252+0000","last_became_active":"2024-09-13T13:12:52.999912+0000","last_became_peered":"2024-09-13T13:12:52.999912+0000","last_unstale":"2024-09-13T13:14:10.024252+0000","last_undegraded":"2024-09-13T13:14:10.024252+0000","last_fullsized":"2024-09-13T13:14:10.024252+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.105098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.105098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:13.380406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":779,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017601+0000","last_change":"2024-09-13T13:13:04.777566+0000","last_active":"2024-09-13T13:14:10.017601+0000","last_peered":"2024-09-13T13:14:10.017601+0000","last_clean":"2024-09-13T13:14:10.017601+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:14:10.017601+0000","last_undegraded":"2024-09-13T13:14:10.017601+0000","last_fullsized":"2024-09-13T13:14:10.017601+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:13:04.777515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.777515+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:25.054666+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":605,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.010837+0000","last_change":"2024-09-13T13:13:49.754588+0000","last_active":"2024-09-13T13:14:09.010837+0000","last_peered":"2024-09-13T13:14:09.010837+0000","last_clean":"2024-09-13T13:14:09.010837+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:14:09.010837+0000","last_undegraded":"2024-09-13T13:14:09.010837+0000","last_fullsized":"2024-09-13T13:14:09.010837+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:49.754540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:49.754540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:09.274489+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":104,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013217+0000","last_change":"2024-09-13T13:13:47.728092+0000","last_active":"2024-09-13T13:14:10.013217+0000","last_peered":"2024-09-13T13:14:10.013217+0000","last_clean":"2024-09-13T13:14:10.013217+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:14:10.013217+0000","last_undegraded":"2024-09-13T13:14:10.013217+0000","last_fullsized":"2024-09-13T13:14:10.013217+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:47.728010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:47.728010+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:08.916792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":84,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013751+0000","last_change":"2024-09-13T13:12:53.006302+0000","last_active":"2024-09-13T13:14:10.013751+0000","last_peered":"2024-09-13T13:14:10.013751+0000","last_clean":"2024-09-13T13:14:10.013751+0000","last_became_active":"2024-09-13T13:12:53.005988+0000","last_became_peered":"2024-09-13T13:12:53.005988+0000","last_unstale":"2024-09-13T13:14:10.013751+0000","last_undegraded":"2024-09-13T13:14:10.013751+0000","last_fullsized":"2024-09-13T13:14:10.013751+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:40.888083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.888083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:08.765290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":601,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.996852+0000","last_change":"2024-09-13T13:13:15.674968+0000","last_active":"2024-09-13T13:14:04.996852+0000","last_peered":"2024-09-13T13:14:04.996852+0000","last_clean":"2024-09-13T13:14:04.996852+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:14:04.996852+0000","last_undegraded":"2024-09-13T13:14:04.996852+0000","last_fullsized":"2024-09-13T13:14:04.996852+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:15.674898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:15.674898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:42.248042+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":598,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.003604+0000","last_change":"2024-09-13T13:12:59.145916+0000","last_active":"2024-09-13T13:14:09.003604+0000","last_peered":"2024-09-13T13:14:09.003604+0000","last_clean":"2024-09-13T13:14:09.003604+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:14:09.003604+0000","last_undegraded":"2024-09-13T13:14:09.003604+0000","last_fullsized":"2024-09-13T13:14:09.003604+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:59.145847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:59.145847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:16.083736+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":577,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017216+0000","last_change":"2024-09-13T13:12:39.874394+0000","last_active":"2024-09-13T13:14:10.017216+0000","last_peered":"2024-09-13T13:14:10.017216+0000","last_clean":"2024-09-13T13:14:10.017216+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:14:10.017216+0000","last_undegraded":"2024-09-13T13:14:10.017216+0000","last_fullsized":"2024-09-13T13:14:10.017216+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:39.874324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:39.874324+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:07.639902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":594,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.004443+0000","last_change":"2024-09-13T13:12:58.097405+0000","last_active":"2024-09-13T13:14:09.004443+0000","last_peered":"2024-09-13T13:14:09.004443+0000","last_clean":"2024-09-13T13:14:09.004443+0000","last_became_active":"2024-09-13T13:12:52.949094+0000","last_became_peered":"2024-09-13T13:12:52.949094+0000","last_unstale":"2024-09-13T13:14:09.004443+0000","last_undegraded":"2024-09-13T13:14:09.004443+0000","last_fullsized":"2024-09-13T13:14:09.004443+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:58.097353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.097353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:12.194897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":602,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.004231+0000","last_change":"2024-09-13T13:13:44.226432+0000","last_active":"2024-09-13T13:14:09.004231+0000","last_peered":"2024-09-13T13:14:09.004231+0000","last_clean":"2024-09-13T13:14:09.004231+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:14:09.004231+0000","last_undegraded":"2024-09-13T13:14:09.004231+0000","last_fullsized":"2024-09-13T13:14:09.004231+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:44.226240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:44.226240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:00.263002+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":597,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.010229+0000","last_change":"2024-09-13T13:10:39.477882+0000","last_active":"2024-09-13T13:14:09.010229+0000","last_peered":"2024-09-13T13:14:09.010229+0000","last_clean":"2024-09-13T13:14:09.010229+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:14:09.010229+0000","last_undegraded":"2024-09-13T13:14:09.010229+0000","last_fullsized":"2024-09-13T13:14:09.010229+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:39.477683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.477683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:37.708691+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":632,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.003159+0000","last_change":"2024-09-13T13:13:49.268080+0000","last_active":"2024-09-13T13:14:09.003159+0000","last_peered":"2024-09-13T13:14:09.003159+0000","last_clean":"2024-09-13T13:14:09.003159+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:14:09.003159+0000","last_undegraded":"2024-09-13T13:14:09.003159+0000","last_fullsized":"2024-09-13T13:14:09.003159+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:49.268018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:49.268018+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:12.973884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":620,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.018059+0000","last_change":"2024-09-13T13:13:44.485526+0000","last_active":"2024-09-13T13:14:10.018059+0000","last_peered":"2024-09-13T13:14:10.018059+0000","last_clean":"2024-09-13T13:14:10.018059+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:14:10.018059+0000","last_undegraded":"2024-09-13T13:14:10.018059+0000","last_fullsized":"2024-09-13T13:14:10.018059+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:44.485364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:44.485364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:47.391654+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":629,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.023746+0000","last_change":"2024-09-13T13:12:53.000159+0000","last_active":"2024-09-13T13:14:10.023746+0000","last_peered":"2024-09-13T13:14:10.023746+0000","last_clean":"2024-09-13T13:14:10.023746+0000","last_became_active":"2024-09-13T13:12:52.999748+0000","last_became_peered":"2024-09-13T13:12:52.999748+0000","last_unstale":"2024-09-13T13:14:10.023746+0000","last_undegraded":"2024-09-13T13:14:10.023746+0000","last_fullsized":"2024-09-13T13:14:10.023746+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:42.847961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:42.847961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:00.102694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":593,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.004382+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:14:09.004382+0000","last_peered":"2024-09-13T13:14:09.004382+0000","last_clean":"2024-09-13T13:14:09.004382+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:14:09.004382+0000","last_undegraded":"2024-09-13T13:14:09.004382+0000","last_fullsized":"2024-09-13T13:14:09.004382+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:07.339701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":634,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.002658+0000","last_change":"2024-09-13T13:13:09.131159+0000","last_active":"2024-09-13T13:14:09.002658+0000","last_peered":"2024-09-13T13:14:09.002658+0000","last_clean":"2024-09-13T13:14:09.002658+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:14:09.002658+0000","last_undegraded":"2024-09-13T13:14:09.002658+0000","last_fullsized":"2024-09-13T13:14:09.002658+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:13:09.130987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:09.130987+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:09.392149+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":625,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.011285+0000","last_change":"2024-09-13T13:13:44.750043+0000","last_active":"2024-09-13T13:14:09.011285+0000","last_peered":"2024-09-13T13:14:09.011285+0000","last_clean":"2024-09-13T13:14:09.011285+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:14:09.011285+0000","last_undegraded":"2024-09-13T13:14:09.011285+0000","last_fullsized":"2024-09-13T13:14:09.011285+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:44.749961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:44.749961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:01.447677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":535,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.015455+0000","last_change":"2024-09-13T13:13:14.358378+0000","last_active":"2024-09-13T13:14:10.015455+0000","last_peered":"2024-09-13T13:14:10.015455+0000","last_clean":"2024-09-13T13:14:10.015455+0000","last_became_active":"2024-09-13T13:12:53.001131+0000","last_became_peered":"2024-09-13T13:12:53.001131+0000","last_unstale":"2024-09-13T13:14:10.015455+0000","last_undegraded":"2024-09-13T13:14:10.015455+0000","last_fullsized":"2024-09-13T13:14:10.015455+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:14.358331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:14.358331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:15.273993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":592,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.020787+0000","last_change":"2024-09-13T13:12:52.947380+0000","last_active":"2024-09-13T13:14:10.020787+0000","last_peered":"2024-09-13T13:14:10.020787+0000","last_clean":"2024-09-13T13:14:10.020787+0000","last_became_active":"2024-09-13T13:12:52.946611+0000","last_became_peered":"2024-09-13T13:12:52.946611+0000","last_unstale":"2024-09-13T13:14:10.020787+0000","last_undegraded":"2024-09-13T13:14:10.020787+0000","last_fullsized":"2024-09-13T13:14:10.020787+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:08.704450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:08.704450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:03.458121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":611,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-13T13:14:04.997226+0000","last_change":"2024-09-13T13:13:08.622965+0000","last_active":"2024-09-13T13:14:04.997226+0000","last_peered":"2024-09-13T13:14:04.997226+0000","last_clean":"2024-09-13T13:14:04.997226+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:14:04.997226+0000","last_undegraded":"2024-09-13T13:14:04.997226+0000","last_fullsized":"2024-09-13T13:14:04.997226+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:08.622885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:08.622885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:15.445425+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":602,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.002897+0000","last_change":"2024-09-13T13:11:07.564094+0000","last_active":"2024-09-13T13:14:09.002897+0000","last_peered":"2024-09-13T13:14:09.002897+0000","last_clean":"2024-09-13T13:14:09.002897+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:14:09.002897+0000","last_undegraded":"2024-09-13T13:14:09.002897+0000","last_fullsized":"2024-09-13T13:14:09.002897+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:07.564025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:07.564025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:34.049577+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":611,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.016182+0000","last_change":"2024-09-13T13:11:49.264206+0000","last_active":"2024-09-13T13:14:10.016182+0000","last_peered":"2024-09-13T13:14:10.016182+0000","last_clean":"2024-09-13T13:14:10.016182+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:14:10.016182+0000","last_undegraded":"2024-09-13T13:14:10.016182+0000","last_fullsized":"2024-09-13T13:14:10.016182+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:49.264128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.264128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:22.505993+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":618,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-13T13:14:09.011065+0000","last_change":"2024-09-13T13:13:53.809209+0000","last_active":"2024-09-13T13:14:09.011065+0000","last_peered":"2024-09-13T13:14:09.011065+0000","last_clean":"2024-09-13T13:14:09.011065+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:14:09.011065+0000","last_undegraded":"2024-09-13T13:14:09.011065+0000","last_fullsized":"2024-09-13T13:14:09.011065+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:53.809160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:53.809160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:15.146128+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1354694656,"num_objects":20671,"num_object_clones":0,"num_object_copies":62013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20671,"num_write_kb":1322944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3937665024,"data_stored":3937665024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20671,"ondisk_log_size":20671,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":2508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":90,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201441,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":806680,"kb_used_data":388168,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92938472,"statfs":{"total":95995035648,"available":95168995328,"internally_reserved":0,"allocated":397484032,"data_stored":396954349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":428538314},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141051,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":863184,"kb_used_data":500856,"kb_used_omap":4,"kb_used_meta":362299,"kb_avail":92881968,"statfs":{"total":95995035648,"available":95111135232,"internally_reserved":0,"allocated":512876544,"data_stored":512361616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":370994861},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692013,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":931468,"kb_used_data":467132,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92813684,"statfs":{"total":95995035648,"available":95041212416,"internally_reserved":0,"allocated":478343168,"data_stored":477824069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920947,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1077280,"kb_used_data":753168,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92667872,"statfs":{"total":95995035648,"available":94891900928,"internally_reserved":0,"allocated":771244032,"data_stored":770704589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":331872710},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117177,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":805640,"kb_used_data":251064,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92939512,"statfs":{"total":95995035648,"available":95170060288,"internally_reserved":0,"allocated":257089536,"data_stored":256561503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":567867847},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411517,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901504,"kb_used_data":545456,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92843648,"statfs":{"total":95995035648,"available":95071895552,"internally_reserved":0,"allocated":558546944,"data_stored":558039583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":364575174},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575043,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":664860,"kb_used_data":164428,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93080292,"statfs":{"total":95995035648,"available":95314219008,"internally_reserved":0,"allocated":168374272,"data_stored":167825550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738570,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1102084,"kb_used_data":547508,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92643068,"statfs":{"total":95995035648,"available":94866501632,"internally_reserved":0,"allocated":560648192,"data_stored":560124812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":616366080,"data_stored":616366080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172883968,"data_stored":172883968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":608043008,"data_stored":608043008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":293470208,"data_stored":293470208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":789774336,"data_stored":789774336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":507379712,"data_stored":507379712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":551223296,"data_stored":551223296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":398524416,"data_stored":398524416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:14:20.289 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:14:20.295 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 16 15 34958 34943 136.483 84.625 0.00929066 0.00732148 2024-09-13T13:14:20.310 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:20.340 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:20.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:20.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:20.442 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:20.465 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:20.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:20.566 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:20.616 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:20.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:20.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:20.743 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:20.746 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:20.746 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:20.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:20.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:20 smithi167 ceph-mon[33178]: 3.a scrub starts 2024-09-13T13:14:20.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:20 smithi167 ceph-mon[33178]: 3.a scrub ok 2024-09-13T13:14:20.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:20 smithi167 ceph-mon[33178]: 5.4 scrub starts 2024-09-13T13:14:20.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:20 smithi167 ceph-mon[33178]: 5.4 scrub ok 2024-09-13T13:14:20.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:20 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:14:20.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:20 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:14:20.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:20 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:14:20.849 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:20.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[36682]: 3.a scrub starts 2024-09-13T13:14:20.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[36682]: 3.a scrub ok 2024-09-13T13:14:20.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[36682]: 5.4 scrub starts 2024-09-13T13:14:20.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[36682]: 5.4 scrub ok 2024-09-13T13:14:20.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:14:20.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:14:20.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:14:20.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[29471]: 3.a scrub starts 2024-09-13T13:14:20.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[29471]: 3.a scrub ok 2024-09-13T13:14:20.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[29471]: 5.4 scrub starts 2024-09-13T13:14:20.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[29471]: 5.4 scrub ok 2024-09-13T13:14:20.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:14:20.875 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:14:20.876 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:20 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:14:20.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:20.893 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:20.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:20.994 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:21.033 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:21.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:21.134 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:21.159 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:21.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:21.260 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:21.287 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:21.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:21.295 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 17 16 36193 36177 132.991 77.125 0.00719012 0.00750839 2024-09-13T13:14:21.389 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:21.429 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:21.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:21.524 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:14:21.524 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:14:21.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:21.560 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:21.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:21.662 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:21.693 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:21.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:21.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:21.788 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:14:21.788 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:14:21.788 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:14:21.789 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:14:21.789 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:14:21.789 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:14:21.789 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:14:21.789 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:14:21.789 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:14:21.789 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:14:21.789 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:14:21.789 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:14:21.790 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[36682]: 6.11 scrub starts 2024-09-13T13:14:21.790 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[36682]: 6.11 scrub ok 2024-09-13T13:14:21.790 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[36682]: pgmap v657: 148 pgs: 148 active+clean; 1.9 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 132 MiB/s wr, 2.10k op/s 2024-09-13T13:14:21.790 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[36682]: 4.d scrub starts 2024-09-13T13:14:21.790 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[36682]: 4.d scrub ok 2024-09-13T13:14:21.790 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[36682]: 4.8 scrub starts 2024-09-13T13:14:21.790 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[36682]: 4.8 scrub ok 2024-09-13T13:14:21.790 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1011349291' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:21.791 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3497902685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:21.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[29471]: 6.11 scrub starts 2024-09-13T13:14:21.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[29471]: 6.11 scrub ok 2024-09-13T13:14:21.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[29471]: pgmap v657: 148 pgs: 148 active+clean; 1.9 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 132 MiB/s wr, 2.10k op/s 2024-09-13T13:14:21.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[29471]: 4.d scrub starts 2024-09-13T13:14:21.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[29471]: 4.d scrub ok 2024-09-13T13:14:21.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[29471]: 4.8 scrub starts 2024-09-13T13:14:21.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[29471]: 4.8 scrub ok 2024-09-13T13:14:21.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1011349291' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:21.791 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:21 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3497902685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:21.796 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:21.797 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:14:21.797 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:14:21.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:21 smithi167 ceph-mon[33178]: 6.11 scrub starts 2024-09-13T13:14:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:21 smithi167 ceph-mon[33178]: 6.11 scrub ok 2024-09-13T13:14:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:21 smithi167 ceph-mon[33178]: pgmap v657: 148 pgs: 148 active+clean; 1.9 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 132 MiB/s wr, 2.10k op/s 2024-09-13T13:14:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:21 smithi167 ceph-mon[33178]: 4.d scrub starts 2024-09-13T13:14:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:21 smithi167 ceph-mon[33178]: 4.d scrub ok 2024-09-13T13:14:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:21 smithi167 ceph-mon[33178]: 4.8 scrub starts 2024-09-13T13:14:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:21 smithi167 ceph-mon[33178]: 4.8 scrub ok 2024-09-13T13:14:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:21 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1011349291' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:21 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3497902685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:21.826 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:21.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:21.835 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:14:21.835 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:14:21.881 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:14:21.928 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:21.967 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:21.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:22.024 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:14:22.026 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:14:22.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:22.106 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:22.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:22.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:22.208 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:22.242 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:22.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:22.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:22.295 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 18 16 37863 37847 131.4 104.375 0.00164531 0.00759011 2024-09-13T13:14:22.344 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:22.382 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:22.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:22.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:22.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:22.508 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:22.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:22.610 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:22.651 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:22.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:22.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:22.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[29471]: 5.1f scrub starts 2024-09-13T13:14:22.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[29471]: 5.1f scrub ok 2024-09-13T13:14:22.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[29471]: 4.1e scrub starts 2024-09-13T13:14:22.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[29471]: 4.1e scrub ok 2024-09-13T13:14:22.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:14:22.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:14:22.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1011349291' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:14:22.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[29471]: osdmap e343: 8 total, 8 up, 8 in 2024-09-13T13:14:22.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[36682]: 5.1f scrub starts 2024-09-13T13:14:22.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[36682]: 5.1f scrub ok 2024-09-13T13:14:22.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[36682]: 4.1e scrub starts 2024-09-13T13:14:22.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[36682]: 4.1e scrub ok 2024-09-13T13:14:22.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:14:22.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:14:22.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1011349291' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:14:22.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:22 smithi052 ceph-mon[36682]: osdmap e343: 8 total, 8 up, 8 in 2024-09-13T13:14:22.754 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:22.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:22.794 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:22.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:22.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:22 smithi167 ceph-mon[33178]: 5.1f scrub starts 2024-09-13T13:14:22.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:22 smithi167 ceph-mon[33178]: 5.1f scrub ok 2024-09-13T13:14:22.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:22 smithi167 ceph-mon[33178]: 4.1e scrub starts 2024-09-13T13:14:22.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:22 smithi167 ceph-mon[33178]: 4.1e scrub ok 2024-09-13T13:14:22.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:22 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:14:22.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:22 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:14:22.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:22 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1011349291' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:14:22.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:22 smithi167 ceph-mon[33178]: osdmap e343: 8 total, 8 up, 8 in 2024-09-13T13:14:22.896 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:22.905 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2024-09-13T13:14:21.886982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2222987803,"num_objects":34110,"num_object_clones":0,"num_object_copies":102330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34110,"num_whiteouts":0,"num_read":1346,"num_read_kb":1205,"num_write":34746,"num_write_kb":2171977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":594037,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34768,"ondisk_log_size":34768,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9915836,"kb_used_data":6278204,"kb_used_omap":15,"kb_used_meta":3637488,"kb_avail":740045380,"statfs":{"total":767960285184,"available":757806469120,"internally_reserved":0,"allocated":6428880896,"data_stored":6424638177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":3724788505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1212809216,"num_objects":18506,"num_object_clones":0,"num_object_copies":55518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18506,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":18506,"num_write_kb":1184384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003057"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":604,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.026623+0000","last_change":"2024-09-13T13:10:32.368663+0000","last_active":"2024-09-13T13:14:10.026623+0000","last_peered":"2024-09-13T13:14:10.026623+0000","last_clean":"2024-09-13T13:14:10.026623+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:14:10.026623+0000","last_undegraded":"2024-09-13T13:14:10.026623+0000","last_fullsized":"2024-09-13T13:14:10.026623+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.368588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.368588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:33.672891+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":94,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012364+0000","last_change":"2024-09-13T13:12:53.024521+0000","last_active":"2024-09-13T13:14:10.012364+0000","last_peered":"2024-09-13T13:14:10.012364+0000","last_clean":"2024-09-13T13:14:10.012364+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:14:10.012364+0000","last_undegraded":"2024-09-13T13:14:10.012364+0000","last_fullsized":"2024-09-13T13:14:10.012364+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:12:14.337224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:14.337224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:02.792645+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":605,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:12.865175+0000","last_change":"2024-09-13T13:14:12.865175+0000","last_active":"2024-09-13T13:14:12.865175+0000","last_peered":"2024-09-13T13:14:12.865175+0000","last_clean":"2024-09-13T13:14:12.865175+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:14:12.865175+0000","last_undegraded":"2024-09-13T13:14:12.865175+0000","last_fullsized":"2024-09-13T13:14:12.865175+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:12.865101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:12.865101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:17.166702+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012882+0000","last_change":"2024-09-13T13:13:17.976677+0000","last_active":"2024-09-13T13:14:10.012882+0000","last_peered":"2024-09-13T13:14:10.012882+0000","last_clean":"2024-09-13T13:14:10.012882+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:14:10.012882+0000","last_undegraded":"2024-09-13T13:14:10.012882+0000","last_fullsized":"2024-09-13T13:14:10.012882+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:17.976607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:17.976607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:28.478702+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":611,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.027186+0000","last_change":"2024-09-13T13:11:47.852762+0000","last_active":"2024-09-13T13:14:10.027186+0000","last_peered":"2024-09-13T13:14:10.027186+0000","last_clean":"2024-09-13T13:14:10.027186+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:14:10.027186+0000","last_undegraded":"2024-09-13T13:14:10.027186+0000","last_fullsized":"2024-09-13T13:14:10.027186+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:09.369830+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":624,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.023974+0000","last_change":"2024-09-13T13:13:52.301500+0000","last_active":"2024-09-13T13:14:10.023974+0000","last_peered":"2024-09-13T13:14:10.023974+0000","last_clean":"2024-09-13T13:14:10.023974+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:14:10.023974+0000","last_undegraded":"2024-09-13T13:14:10.023974+0000","last_fullsized":"2024-09-13T13:14:10.023974+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:52.301450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:52.301450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:21.335731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":659,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.020575+0000","last_change":"2024-09-13T13:10:39.460577+0000","last_active":"2024-09-13T13:14:10.020575+0000","last_peered":"2024-09-13T13:14:10.020575+0000","last_clean":"2024-09-13T13:14:10.020575+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:14:10.020575+0000","last_undegraded":"2024-09-13T13:14:10.020575+0000","last_fullsized":"2024-09-13T13:14:10.020575+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'15","last_scrub_stamp":"2024-09-13T13:10:39.460500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.460500+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:23.017525+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":84,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012274+0000","last_change":"2024-09-13T13:12:52.997865+0000","last_active":"2024-09-13T13:14:10.012274+0000","last_peered":"2024-09-13T13:14:10.012274+0000","last_clean":"2024-09-13T13:14:10.012274+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:14:10.012274+0000","last_undegraded":"2024-09-13T13:14:10.012274+0000","last_fullsized":"2024-09-13T13:14:10.012274+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:17.305608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:17.305608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:06.792355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":675,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017687+0000","last_change":"2024-09-13T13:13:41.518378+0000","last_active":"2024-09-13T13:14:10.017687+0000","last_peered":"2024-09-13T13:14:10.017687+0000","last_clean":"2024-09-13T13:14:10.017687+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:14:10.017687+0000","last_undegraded":"2024-09-13T13:14:10.017687+0000","last_fullsized":"2024-09-13T13:14:10.017687+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:41.518322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.518322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:09.555720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":94,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012965+0000","last_change":"2024-09-13T13:12:53.033861+0000","last_active":"2024-09-13T13:14:10.012965+0000","last_peered":"2024-09-13T13:14:10.012965+0000","last_clean":"2024-09-13T13:14:10.012965+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:14:10.012965+0000","last_undegraded":"2024-09-13T13:14:10.012965+0000","last_fullsized":"2024-09-13T13:14:10.012965+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:31.738685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.738685+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:33.976427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":612,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017522+0000","last_change":"2024-09-13T13:13:43.501368+0000","last_active":"2024-09-13T13:14:10.017522+0000","last_peered":"2024-09-13T13:14:10.017522+0000","last_clean":"2024-09-13T13:14:10.017522+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:14:10.017522+0000","last_undegraded":"2024-09-13T13:14:10.017522+0000","last_fullsized":"2024-09-13T13:14:10.017522+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:43.501206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:43.501206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:55.342802+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":607,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.027942+0000","last_change":"2024-09-13T13:13:40.720121+0000","last_active":"2024-09-13T13:14:10.027942+0000","last_peered":"2024-09-13T13:14:10.027942+0000","last_clean":"2024-09-13T13:14:10.027942+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:14:10.027942+0000","last_undegraded":"2024-09-13T13:14:10.027942+0000","last_fullsized":"2024-09-13T13:14:10.027942+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:40.719936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:40.719936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:09.971736+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":642,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017499+0000","last_change":"2024-09-13T13:12:52.951531+0000","last_active":"2024-09-13T13:14:10.017499+0000","last_peered":"2024-09-13T13:14:10.017499+0000","last_clean":"2024-09-13T13:14:10.017499+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:14:10.017499+0000","last_undegraded":"2024-09-13T13:14:10.017499+0000","last_fullsized":"2024-09-13T13:14:10.017499+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:16.989972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:16.989972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:33.371689+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":90,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013060+0000","last_change":"2024-09-13T13:12:53.028266+0000","last_active":"2024-09-13T13:14:10.013060+0000","last_peered":"2024-09-13T13:14:10.013060+0000","last_clean":"2024-09-13T13:14:10.013060+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:14:10.013060+0000","last_undegraded":"2024-09-13T13:14:10.013060+0000","last_fullsized":"2024-09-13T13:14:10.013060+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:11:53.120451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:53.120451+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:01.777114+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":633,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.023966+0000","last_change":"2024-09-13T13:13:54.122282+0000","last_active":"2024-09-13T13:14:10.023966+0000","last_peered":"2024-09-13T13:14:10.023966+0000","last_clean":"2024-09-13T13:14:10.023966+0000","last_became_active":"2024-09-13T13:12:52.999970+0000","last_became_peered":"2024-09-13T13:12:52.999970+0000","last_unstale":"2024-09-13T13:14:10.023966+0000","last_undegraded":"2024-09-13T13:14:10.023966+0000","last_fullsized":"2024-09-13T13:14:10.023966+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.122222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.122222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:17.226969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":584,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017161+0000","last_change":"2024-09-13T13:13:02.818644+0000","last_active":"2024-09-13T13:14:10.017161+0000","last_peered":"2024-09-13T13:14:10.017161+0000","last_clean":"2024-09-13T13:14:10.017161+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:14:10.017161+0000","last_undegraded":"2024-09-13T13:14:10.017161+0000","last_fullsized":"2024-09-13T13:14:10.017161+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:02.818543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:02.818543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:28.305833+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":88,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.014033+0000","last_change":"2024-09-13T13:12:53.014177+0000","last_active":"2024-09-13T13:14:10.014033+0000","last_peered":"2024-09-13T13:14:10.014033+0000","last_clean":"2024-09-13T13:14:10.014033+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:14:10.014033+0000","last_undegraded":"2024-09-13T13:14:10.014033+0000","last_fullsized":"2024-09-13T13:14:10.014033+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:28.804735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:28.804735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:37.904645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":633,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:18.624877+0000","last_change":"2024-09-13T13:14:18.624877+0000","last_active":"2024-09-13T13:14:18.624877+0000","last_peered":"2024-09-13T13:14:18.624877+0000","last_clean":"2024-09-13T13:14:18.624877+0000","last_became_active":"2024-09-13T13:12:53.008505+0000","last_became_peered":"2024-09-13T13:12:53.008505+0000","last_unstale":"2024-09-13T13:14:18.624877+0000","last_undegraded":"2024-09-13T13:14:18.624877+0000","last_fullsized":"2024-09-13T13:14:18.624877+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:18.624745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:18.624745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:43.578763+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":660,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.022040+0000","last_change":"2024-09-13T13:12:40.125646+0000","last_active":"2024-09-13T13:14:10.022040+0000","last_peered":"2024-09-13T13:14:10.022040+0000","last_clean":"2024-09-13T13:14:10.022040+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:14:10.022040+0000","last_undegraded":"2024-09-13T13:14:10.022040+0000","last_fullsized":"2024-09-13T13:14:10.022040+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:12:40.125545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.125545+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:06.427906+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":627,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013967+0000","last_change":"2024-09-13T13:13:20.716705+0000","last_active":"2024-09-13T13:14:10.013967+0000","last_peered":"2024-09-13T13:14:10.013967+0000","last_clean":"2024-09-13T13:14:10.013967+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:14:10.013967+0000","last_undegraded":"2024-09-13T13:14:10.013967+0000","last_fullsized":"2024-09-13T13:14:10.013967+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:20.716633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:20.716633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:46.465567+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":612,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.023196+0000","last_change":"2024-09-13T13:13:03.150524+0000","last_active":"2024-09-13T13:14:10.023196+0000","last_peered":"2024-09-13T13:14:10.023196+0000","last_clean":"2024-09-13T13:14:10.023196+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:14:10.023196+0000","last_undegraded":"2024-09-13T13:14:10.023196+0000","last_fullsized":"2024-09-13T13:14:10.023196+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:03.150442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.150442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:30.195272+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012719+0000","last_change":"2024-09-13T13:13:39.876364+0000","last_active":"2024-09-13T13:14:10.012719+0000","last_peered":"2024-09-13T13:14:10.012719+0000","last_clean":"2024-09-13T13:14:10.012719+0000","last_became_active":"2024-09-13T13:12:52.995055+0000","last_became_peered":"2024-09-13T13:12:52.995055+0000","last_unstale":"2024-09-13T13:14:10.012719+0000","last_undegraded":"2024-09-13T13:14:10.012719+0000","last_fullsized":"2024-09-13T13:14:10.012719+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:39.876294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:39.876294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:45.791176+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":678,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.019764+0000","last_change":"2024-09-13T13:13:41.164741+0000","last_active":"2024-09-13T13:14:10.019764+0000","last_peered":"2024-09-13T13:14:10.019764+0000","last_clean":"2024-09-13T13:14:10.019764+0000","last_became_active":"2024-09-13T13:12:52.953819+0000","last_became_peered":"2024-09-13T13:12:52.953819+0000","last_unstale":"2024-09-13T13:14:10.019764+0000","last_undegraded":"2024-09-13T13:14:10.019764+0000","last_fullsized":"2024-09-13T13:14:10.019764+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:13:41.164656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.164656+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:45.115584+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":86,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:15.986228+0000","last_change":"2024-09-13T13:14:15.986228+0000","last_active":"2024-09-13T13:14:15.986228+0000","last_peered":"2024-09-13T13:14:15.986228+0000","last_clean":"2024-09-13T13:14:15.986228+0000","last_became_active":"2024-09-13T13:13:06.646781+0000","last_became_peered":"2024-09-13T13:13:06.646781+0000","last_unstale":"2024-09-13T13:14:15.986228+0000","last_undegraded":"2024-09-13T13:14:15.986228+0000","last_fullsized":"2024-09-13T13:14:15.986228+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:15.986163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:15.986163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:25.157413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":119,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017277+0000","last_change":"2024-09-13T13:12:42.975335+0000","last_active":"2024-09-13T13:14:10.017277+0000","last_peered":"2024-09-13T13:14:10.017277+0000","last_clean":"2024-09-13T13:14:10.017277+0000","last_became_active":"2024-09-13T13:12:39.954363+0000","last_became_peered":"2024-09-13T13:12:39.954363+0000","last_unstale":"2024-09-13T13:14:10.017277+0000","last_undegraded":"2024-09-13T13:14:10.017277+0000","last_fullsized":"2024-09-13T13:14:10.017277+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:12:42.975266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:42.975266+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:00.584923+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":628,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012150+0000","last_change":"2024-09-13T13:13:53.796606+0000","last_active":"2024-09-13T13:14:10.012150+0000","last_peered":"2024-09-13T13:14:10.012150+0000","last_clean":"2024-09-13T13:14:10.012150+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:14:10.012150+0000","last_undegraded":"2024-09-13T13:14:10.012150+0000","last_fullsized":"2024-09-13T13:14:10.012150+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:53.796530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:53.796530+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:18.032974+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":646,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.027371+0000","last_change":"2024-09-13T13:13:14.875584+0000","last_active":"2024-09-13T13:14:10.027371+0000","last_peered":"2024-09-13T13:14:10.027371+0000","last_clean":"2024-09-13T13:14:10.027371+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:14:10.027371+0000","last_undegraded":"2024-09-13T13:14:10.027371+0000","last_fullsized":"2024-09-13T13:14:10.027371+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:13:14.875507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:14.875507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:18.943661+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"306'12","reported_seq":752,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:12.398801+0000","last_change":"2024-09-13T13:13:02.305723+0000","last_active":"2024-09-13T13:14:12.398801+0000","last_peered":"2024-09-13T13:14:12.398801+0000","last_clean":"2024-09-13T13:14:12.398801+0000","last_became_active":"2024-09-13T13:12:52.947886+0000","last_became_peered":"2024-09-13T13:12:52.947886+0000","last_unstale":"2024-09-13T13:14:12.398801+0000","last_undegraded":"2024-09-13T13:14:12.398801+0000","last_fullsized":"2024-09-13T13:14:12.398801+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'12","last_scrub_stamp":"2024-09-13T13:13:02.305656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:02.305656+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:19.160834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"342'2018","reported_seq":2045,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:18.072032+0000","last_change":"2024-09-13T13:13:59.149493+0000","last_active":"2024-09-13T13:14:18.072032+0000","last_peered":"2024-09-13T13:14:18.072032+0000","last_clean":"2024-09-13T13:14:18.072032+0000","last_became_active":"2024-09-13T13:13:59.149215+0000","last_became_peered":"2024-09-13T13:13:59.149215+0000","last_unstale":"2024-09-13T13:14:18.072032+0000","last_undegraded":"2024-09-13T13:14:18.072032+0000","last_fullsized":"2024-09-13T13:14:18.072032+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2018,"log_dups_size":0,"ondisk_log_size":2018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:25.361862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132251648,"num_objects":2018,"num_object_clones":0,"num_object_copies":6054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2018,"num_write_kb":129152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":698,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013531+0000","last_change":"2024-09-13T13:12:53.022032+0000","last_active":"2024-09-13T13:14:10.013531+0000","last_peered":"2024-09-13T13:14:10.013531+0000","last_clean":"2024-09-13T13:14:10.013531+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:14:10.013531+0000","last_undegraded":"2024-09-13T13:14:10.013531+0000","last_fullsized":"2024-09-13T13:14:10.013531+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:11:53.007997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:53.007997+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:59.705246+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":621,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.020142+0000","last_change":"2024-09-13T13:10:30.552177+0000","last_active":"2024-09-13T13:14:10.020142+0000","last_peered":"2024-09-13T13:14:10.020142+0000","last_clean":"2024-09-13T13:14:10.020142+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:14:10.020142+0000","last_undegraded":"2024-09-13T13:14:10.020142+0000","last_fullsized":"2024-09-13T13:14:10.020142+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:30.552113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:30.552113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:21.064091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":106,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:12.397895+0000","last_change":"2024-09-13T13:12:53.034647+0000","last_active":"2024-09-13T13:14:12.397895+0000","last_peered":"2024-09-13T13:14:12.397895+0000","last_clean":"2024-09-13T13:14:12.397895+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:14:12.397895+0000","last_undegraded":"2024-09-13T13:14:12.397895+0000","last_fullsized":"2024-09-13T13:14:12.397895+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:10:07.726751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:07.726751+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:26.088519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":616,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017383+0000","last_change":"2024-09-13T13:13:57.546725+0000","last_active":"2024-09-13T13:14:10.017383+0000","last_peered":"2024-09-13T13:14:10.017383+0000","last_clean":"2024-09-13T13:14:10.017383+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:14:10.017383+0000","last_undegraded":"2024-09-13T13:14:10.017383+0000","last_fullsized":"2024-09-13T13:14:10.017383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.546671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.546671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:16.236847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"342'1953","reported_seq":1980,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:18.063450+0000","last_change":"2024-09-13T13:13:58.962128+0000","last_active":"2024-09-13T13:14:18.063450+0000","last_peered":"2024-09-13T13:14:18.063450+0000","last_clean":"2024-09-13T13:14:18.063450+0000","last_became_active":"2024-09-13T13:13:58.961386+0000","last_became_peered":"2024-09-13T13:13:58.961386+0000","last_unstale":"2024-09-13T13:14:18.063450+0000","last_undegraded":"2024-09-13T13:14:18.063450+0000","last_fullsized":"2024-09-13T13:14:18.063450+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":1953,"log_dups_size":0,"ondisk_log_size":1953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:08.925772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127991808,"num_objects":1953,"num_object_clones":0,"num_object_copies":5859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1953,"num_write_kb":124992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":643,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.019911+0000","last_change":"2024-09-13T13:13:50.232646+0000","last_active":"2024-09-13T13:14:10.019911+0000","last_peered":"2024-09-13T13:14:10.019911+0000","last_clean":"2024-09-13T13:14:10.019911+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:14:10.019911+0000","last_undegraded":"2024-09-13T13:14:10.019911+0000","last_fullsized":"2024-09-13T13:14:10.019911+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:13:50.232588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.232588+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:03.575343+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":611,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.023547+0000","last_change":"2024-09-13T13:13:54.313641+0000","last_active":"2024-09-13T13:14:10.023547+0000","last_peered":"2024-09-13T13:14:10.023547+0000","last_clean":"2024-09-13T13:14:10.023547+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:14:10.023547+0000","last_undegraded":"2024-09-13T13:14:10.023547+0000","last_fullsized":"2024-09-13T13:14:10.023547+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:54.313570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.313570+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:12.826907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":84,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012718+0000","last_change":"2024-09-13T13:12:53.006667+0000","last_active":"2024-09-13T13:14:10.012718+0000","last_peered":"2024-09-13T13:14:10.012718+0000","last_clean":"2024-09-13T13:14:10.012718+0000","last_became_active":"2024-09-13T13:12:53.006199+0000","last_became_peered":"2024-09-13T13:12:53.006199+0000","last_unstale":"2024-09-13T13:14:10.012718+0000","last_undegraded":"2024-09-13T13:14:10.012718+0000","last_fullsized":"2024-09-13T13:14:10.012718+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:05:29.213949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:05:29.213949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:59.732082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013047+0000","last_change":"2024-09-13T13:13:44.750804+0000","last_active":"2024-09-13T13:14:10.013047+0000","last_peered":"2024-09-13T13:14:10.013047+0000","last_clean":"2024-09-13T13:14:10.013047+0000","last_became_active":"2024-09-13T13:12:53.001045+0000","last_became_peered":"2024-09-13T13:12:53.001045+0000","last_unstale":"2024-09-13T13:14:10.013047+0000","last_undegraded":"2024-09-13T13:14:10.013047+0000","last_fullsized":"2024-09-13T13:14:10.013047+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:44.750717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:44.750717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:48.950292+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"342'2223","reported_seq":2250,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:21.065722+0000","last_change":"2024-09-13T13:13:58.966300+0000","last_active":"2024-09-13T13:14:21.065722+0000","last_peered":"2024-09-13T13:14:21.065722+0000","last_clean":"2024-09-13T13:14:21.065722+0000","last_became_active":"2024-09-13T13:13:58.966015+0000","last_became_peered":"2024-09-13T13:13:58.966015+0000","last_unstale":"2024-09-13T13:14:21.065722+0000","last_undegraded":"2024-09-13T13:14:21.065722+0000","last_fullsized":"2024-09-13T13:14:21.065722+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:05.146946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145686528,"num_objects":2223,"num_object_clones":0,"num_object_copies":6669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2223,"num_write_kb":142272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":108,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012990+0000","last_change":"2024-09-13T13:13:50.716312+0000","last_active":"2024-09-13T13:14:10.012990+0000","last_peered":"2024-09-13T13:14:10.012990+0000","last_clean":"2024-09-13T13:14:10.012990+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:14:10.012990+0000","last_undegraded":"2024-09-13T13:14:10.012990+0000","last_fullsized":"2024-09-13T13:14:10.012990+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:13:50.716243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.716243+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:59.311921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":619,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.027621+0000","last_change":"2024-09-13T13:13:57.886950+0000","last_active":"2024-09-13T13:14:10.027621+0000","last_peered":"2024-09-13T13:14:10.027621+0000","last_clean":"2024-09-13T13:14:10.027621+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:14:10.027621+0000","last_undegraded":"2024-09-13T13:14:10.027621+0000","last_fullsized":"2024-09-13T13:14:10.027621+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:57.886879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.886879+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:05.700494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":796,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:12.399370+0000","last_change":"2024-09-13T13:13:55.316917+0000","last_active":"2024-09-13T13:14:12.399370+0000","last_peered":"2024-09-13T13:14:12.399370+0000","last_clean":"2024-09-13T13:14:12.399370+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:14:12.399370+0000","last_undegraded":"2024-09-13T13:14:12.399370+0000","last_fullsized":"2024-09-13T13:14:12.399370+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:13:55.316860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.316860+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:10.829853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":226,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.016212+0000","last_change":"2024-09-13T13:13:52.572068+0000","last_active":"2024-09-13T13:14:10.016212+0000","last_peered":"2024-09-13T13:14:10.016212+0000","last_clean":"2024-09-13T13:14:10.016212+0000","last_became_active":"2024-09-13T13:12:39.948363+0000","last_became_peered":"2024-09-13T13:12:39.948363+0000","last_unstale":"2024-09-13T13:14:10.016212+0000","last_undegraded":"2024-09-13T13:14:10.016212+0000","last_fullsized":"2024-09-13T13:14:10.016212+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:52.571999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:52.571999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:57.263831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"342'2038","reported_seq":2065,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:17.926210+0000","last_change":"2024-09-13T13:13:59.149331+0000","last_active":"2024-09-13T13:14:17.926210+0000","last_peered":"2024-09-13T13:14:17.926210+0000","last_clean":"2024-09-13T13:14:17.926210+0000","last_became_active":"2024-09-13T13:13:59.149046+0000","last_became_peered":"2024-09-13T13:13:59.149046+0000","last_unstale":"2024-09-13T13:14:17.926210+0000","last_undegraded":"2024-09-13T13:14:17.926210+0000","last_fullsized":"2024-09-13T13:14:17.926210+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2038,"log_dups_size":0,"ondisk_log_size":2038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:02.773024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133562368,"num_objects":2038,"num_object_clones":0,"num_object_copies":6114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":130432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":678,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012224+0000","last_change":"2024-09-13T13:12:53.016127+0000","last_active":"2024-09-13T13:14:10.012224+0000","last_peered":"2024-09-13T13:14:10.012224+0000","last_clean":"2024-09-13T13:14:10.012224+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:14:10.012224+0000","last_undegraded":"2024-09-13T13:14:10.012224+0000","last_fullsized":"2024-09-13T13:14:10.012224+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:37.707437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":629,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017797+0000","last_change":"2024-09-13T13:13:12.664590+0000","last_active":"2024-09-13T13:14:10.017797+0000","last_peered":"2024-09-13T13:14:10.017797+0000","last_clean":"2024-09-13T13:14:10.017797+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:14:10.017797+0000","last_undegraded":"2024-09-13T13:14:10.017797+0000","last_fullsized":"2024-09-13T13:14:10.017797+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:12.664516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:12.664516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:19.332370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":770,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:12.397335+0000","last_change":"2024-09-13T13:10:10.134973+0000","last_active":"2024-09-13T13:14:12.397335+0000","last_peered":"2024-09-13T13:14:12.397335+0000","last_clean":"2024-09-13T13:14:12.397335+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:14:12.397335+0000","last_undegraded":"2024-09-13T13:14:12.397335+0000","last_fullsized":"2024-09-13T13:14:12.397335+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:10:10.134873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:10:10.134873+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:24.741279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":624,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.028446+0000","last_change":"2024-09-13T13:13:52.836892+0000","last_active":"2024-09-13T13:14:10.028446+0000","last_peered":"2024-09-13T13:14:10.028446+0000","last_clean":"2024-09-13T13:14:10.028446+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:14:10.028446+0000","last_undegraded":"2024-09-13T13:14:10.028446+0000","last_fullsized":"2024-09-13T13:14:10.028446+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:52.836457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:52.836457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:10.105271+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"342'2291","reported_seq":2317,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:21.065179+0000","last_change":"2024-09-13T13:13:58.966260+0000","last_active":"2024-09-13T13:14:21.065179+0000","last_peered":"2024-09-13T13:14:21.065179+0000","last_clean":"2024-09-13T13:14:21.065179+0000","last_became_active":"2024-09-13T13:13:58.965950+0000","last_became_peered":"2024-09-13T13:13:58.965950+0000","last_unstale":"2024-09-13T13:14:21.065179+0000","last_undegraded":"2024-09-13T13:14:21.065179+0000","last_fullsized":"2024-09-13T13:14:21.065179+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2291,"log_dups_size":0,"ondisk_log_size":2291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:18.220013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150142976,"num_objects":2291,"num_object_clones":0,"num_object_copies":6873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":146624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":675,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.026377+0000","last_change":"2024-09-13T13:13:45.757306+0000","last_active":"2024-09-13T13:14:10.026377+0000","last_peered":"2024-09-13T13:14:10.026377+0000","last_clean":"2024-09-13T13:14:10.026377+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:14:10.026377+0000","last_undegraded":"2024-09-13T13:14:10.026377+0000","last_fullsized":"2024-09-13T13:14:10.026377+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:13:45.751801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:45.751801+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:07.230721+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":598,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012957+0000","last_change":"2024-09-13T13:13:18.686583+0000","last_active":"2024-09-13T13:14:10.012957+0000","last_peered":"2024-09-13T13:14:10.012957+0000","last_clean":"2024-09-13T13:14:10.012957+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:14:10.012957+0000","last_undegraded":"2024-09-13T13:14:10.012957+0000","last_fullsized":"2024-09-13T13:14:10.012957+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:18.686400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:18.686400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:21.962212+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":573,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.015498+0000","last_change":"2024-09-13T13:13:20.475316+0000","last_active":"2024-09-13T13:14:10.015498+0000","last_peered":"2024-09-13T13:14:10.015498+0000","last_clean":"2024-09-13T13:14:10.015498+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:14:10.015498+0000","last_undegraded":"2024-09-13T13:14:10.015498+0000","last_fullsized":"2024-09-13T13:14:10.015498+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:20.475262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:20.475262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:49.688681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":613,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.011864+0000","last_change":"2024-09-13T13:13:50.772738+0000","last_active":"2024-09-13T13:14:10.011864+0000","last_peered":"2024-09-13T13:14:10.011864+0000","last_clean":"2024-09-13T13:14:10.011864+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:14:10.011864+0000","last_undegraded":"2024-09-13T13:14:10.011864+0000","last_fullsized":"2024-09-13T13:14:10.011864+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:50.772689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.772689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:56.392159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"342'2237","reported_seq":2264,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:21.060352+0000","last_change":"2024-09-13T13:13:58.966631+0000","last_active":"2024-09-13T13:14:21.060352+0000","last_peered":"2024-09-13T13:14:21.060352+0000","last_clean":"2024-09-13T13:14:21.060352+0000","last_became_active":"2024-09-13T13:13:58.965789+0000","last_became_peered":"2024-09-13T13:13:58.965789+0000","last_unstale":"2024-09-13T13:14:21.060352+0000","last_undegraded":"2024-09-13T13:14:21.060352+0000","last_fullsized":"2024-09-13T13:14:21.060352+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2237,"log_dups_size":0,"ondisk_log_size":2237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:08.633800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146604032,"num_objects":2237,"num_object_clones":0,"num_object_copies":6711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2237,"num_write_kb":143168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"268'40","reported_seq":836,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:12.398795+0000","last_change":"2024-09-13T13:11:54.902633+0000","last_active":"2024-09-13T13:14:12.398795+0000","last_peered":"2024-09-13T13:14:12.398795+0000","last_clean":"2024-09-13T13:14:12.398795+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:14:12.398795+0000","last_undegraded":"2024-09-13T13:14:12.398795+0000","last_fullsized":"2024-09-13T13:14:12.398795+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'40","last_scrub_stamp":"2024-09-13T13:11:54.902575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:54.902575+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:01.766110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":100,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:14.899463+0000","last_change":"2024-09-13T13:14:14.899463+0000","last_active":"2024-09-13T13:14:14.899463+0000","last_peered":"2024-09-13T13:14:14.899463+0000","last_clean":"2024-09-13T13:14:14.899463+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:14:14.899463+0000","last_undegraded":"2024-09-13T13:14:14.899463+0000","last_fullsized":"2024-09-13T13:14:14.899463+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:14.899378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:14.899378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:30.197534+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":624,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017422+0000","last_change":"2024-09-13T13:13:20.438191+0000","last_active":"2024-09-13T13:14:10.017422+0000","last_peered":"2024-09-13T13:14:10.017422+0000","last_clean":"2024-09-13T13:14:10.017422+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:14:10.017422+0000","last_undegraded":"2024-09-13T13:14:10.017422+0000","last_fullsized":"2024-09-13T13:14:10.017422+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:13:20.438142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:20.438142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:30.205583+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":626,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.028446+0000","last_change":"2024-09-13T13:13:43.786426+0000","last_active":"2024-09-13T13:14:10.028446+0000","last_peered":"2024-09-13T13:14:10.028446+0000","last_clean":"2024-09-13T13:14:10.028446+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:14:10.028446+0000","last_undegraded":"2024-09-13T13:14:10.028446+0000","last_fullsized":"2024-09-13T13:14:10.028446+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:43.786210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:43.786210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:58.722043+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":615,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.023176+0000","last_change":"2024-09-13T13:12:52.949134+0000","last_active":"2024-09-13T13:14:10.023176+0000","last_peered":"2024-09-13T13:14:10.023176+0000","last_clean":"2024-09-13T13:14:10.023176+0000","last_became_active":"2024-09-13T13:12:52.948742+0000","last_became_peered":"2024-09-13T13:12:52.948742+0000","last_unstale":"2024-09-13T13:14:10.023176+0000","last_undegraded":"2024-09-13T13:14:10.023176+0000","last_fullsized":"2024-09-13T13:14:10.023176+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:42.489264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:42.489264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:31.014690+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"342'2002","reported_seq":2029,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:18.317112+0000","last_change":"2024-09-13T13:13:59.149677+0000","last_active":"2024-09-13T13:14:18.317112+0000","last_peered":"2024-09-13T13:14:18.317112+0000","last_clean":"2024-09-13T13:14:18.317112+0000","last_became_active":"2024-09-13T13:13:59.149368+0000","last_became_peered":"2024-09-13T13:13:59.149368+0000","last_unstale":"2024-09-13T13:14:18.317112+0000","last_undegraded":"2024-09-13T13:14:18.317112+0000","last_fullsized":"2024-09-13T13:14:18.317112+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2002,"log_dups_size":0,"ondisk_log_size":2002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:05.289043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131203072,"num_objects":2002,"num_object_clones":0,"num_object_copies":6006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2002,"num_write_kb":128128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":635,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013625+0000","last_change":"2024-09-13T13:10:29.799315+0000","last_active":"2024-09-13T13:14:10.013625+0000","last_peered":"2024-09-13T13:14:10.013625+0000","last_clean":"2024-09-13T13:14:10.013625+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:14:10.013625+0000","last_undegraded":"2024-09-13T13:14:10.013625+0000","last_fullsized":"2024-09-13T13:14:10.013625+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-13T13:10:29.799185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:29.799185+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:30.652713+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":633,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012911+0000","last_change":"2024-09-13T13:13:17.676405+0000","last_active":"2024-09-13T13:14:10.012911+0000","last_peered":"2024-09-13T13:14:10.012911+0000","last_clean":"2024-09-13T13:14:10.012911+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:14:10.012911+0000","last_undegraded":"2024-09-13T13:14:10.012911+0000","last_fullsized":"2024-09-13T13:14:10.012911+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:17.676339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:17.676339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:46.604117+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":637,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.022604+0000","last_change":"2024-09-13T13:13:14.560620+0000","last_active":"2024-09-13T13:14:10.022604+0000","last_peered":"2024-09-13T13:14:10.022604+0000","last_clean":"2024-09-13T13:14:10.022604+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:14:10.022604+0000","last_undegraded":"2024-09-13T13:14:10.022604+0000","last_fullsized":"2024-09-13T13:14:10.022604+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:13:14.560555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:14.560555+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:28.903951+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":601,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.022498+0000","last_change":"2024-09-13T13:13:03.320855+0000","last_active":"2024-09-13T13:14:10.022498+0000","last_peered":"2024-09-13T13:14:10.022498+0000","last_clean":"2024-09-13T13:14:10.022498+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:14:10.022498+0000","last_undegraded":"2024-09-13T13:14:10.022498+0000","last_fullsized":"2024-09-13T13:14:10.022498+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:03.320778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.320778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:20.017270+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":234,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.016152+0000","last_change":"2024-09-13T13:12:41.332308+0000","last_active":"2024-09-13T13:14:10.016152+0000","last_peered":"2024-09-13T13:14:10.016152+0000","last_clean":"2024-09-13T13:14:10.016152+0000","last_became_active":"2024-09-13T13:12:40.120242+0000","last_became_peered":"2024-09-13T13:12:40.120242+0000","last_unstale":"2024-09-13T13:14:10.016152+0000","last_undegraded":"2024-09-13T13:14:10.016152+0000","last_fullsized":"2024-09-13T13:14:10.016152+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:41.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:41.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:07.494933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"342'2128","reported_seq":2155,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:19.267422+0000","last_change":"2024-09-13T13:13:59.149922+0000","last_active":"2024-09-13T13:14:19.267422+0000","last_peered":"2024-09-13T13:14:19.267422+0000","last_clean":"2024-09-13T13:14:19.267422+0000","last_became_active":"2024-09-13T13:13:59.149037+0000","last_became_peered":"2024-09-13T13:13:59.149037+0000","last_unstale":"2024-09-13T13:14:19.267422+0000","last_undegraded":"2024-09-13T13:14:19.267422+0000","last_fullsized":"2024-09-13T13:14:19.267422+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2128,"log_dups_size":0,"ondisk_log_size":2128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:19.556748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139460608,"num_objects":2128,"num_object_clones":0,"num_object_copies":6384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2128,"num_write_kb":136192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":644,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.027453+0000","last_change":"2024-09-13T13:13:10.964903+0000","last_active":"2024-09-13T13:14:10.027453+0000","last_peered":"2024-09-13T13:14:10.027453+0000","last_clean":"2024-09-13T13:14:10.027453+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:14:10.027453+0000","last_undegraded":"2024-09-13T13:14:10.027453+0000","last_fullsized":"2024-09-13T13:14:10.027453+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:13:10.964809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.964809+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:37.366958+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":626,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.025929+0000","last_change":"2024-09-13T13:12:53.005196+0000","last_active":"2024-09-13T13:14:10.025929+0000","last_peered":"2024-09-13T13:14:10.025929+0000","last_clean":"2024-09-13T13:14:10.025929+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:14:10.025929+0000","last_undegraded":"2024-09-13T13:14:10.025929+0000","last_fullsized":"2024-09-13T13:14:10.025929+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:37.465261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:37.465261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:27.706020+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":247,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:17.201161+0000","last_change":"2024-09-13T13:13:12.633862+0000","last_active":"2024-09-13T13:14:17.201161+0000","last_peered":"2024-09-13T13:14:17.201161+0000","last_clean":"2024-09-13T13:14:17.201161+0000","last_became_active":"2024-09-13T13:12:39.947940+0000","last_became_peered":"2024-09-13T13:12:39.947940+0000","last_unstale":"2024-09-13T13:14:17.201161+0000","last_undegraded":"2024-09-13T13:14:17.201161+0000","last_fullsized":"2024-09-13T13:14:17.201161+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:13:12.633785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:12.633785+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:36.573968+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":105,"num_read_kb":100,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":597,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.018111+0000","last_change":"2024-09-13T13:13:11.703710+0000","last_active":"2024-09-13T13:14:10.018111+0000","last_peered":"2024-09-13T13:14:10.018111+0000","last_clean":"2024-09-13T13:14:10.018111+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:14:10.018111+0000","last_undegraded":"2024-09-13T13:14:10.018111+0000","last_fullsized":"2024-09-13T13:14:10.018111+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.703647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.703647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:21.004257+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":610,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012510+0000","last_change":"2024-09-13T13:10:32.720742+0000","last_active":"2024-09-13T13:14:10.012510+0000","last_peered":"2024-09-13T13:14:10.012510+0000","last_clean":"2024-09-13T13:14:10.012510+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:14:10.012510+0000","last_undegraded":"2024-09-13T13:14:10.012510+0000","last_fullsized":"2024-09-13T13:14:10.012510+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:31.642159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"342'2240","reported_seq":2267,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:21.051950+0000","last_change":"2024-09-13T13:13:58.967163+0000","last_active":"2024-09-13T13:14:21.051950+0000","last_peered":"2024-09-13T13:14:21.051950+0000","last_clean":"2024-09-13T13:14:21.051950+0000","last_became_active":"2024-09-13T13:13:58.965962+0000","last_became_peered":"2024-09-13T13:13:58.965962+0000","last_unstale":"2024-09-13T13:14:21.051950+0000","last_undegraded":"2024-09-13T13:14:21.051950+0000","last_fullsized":"2024-09-13T13:14:21.051950+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2240,"log_dups_size":0,"ondisk_log_size":2240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:01.549226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":2240,"num_object_clones":0,"num_object_copies":6720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2240,"num_write_kb":143360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"306'32","reported_seq":718,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:12.398574+0000","last_change":"2024-09-13T13:13:01.352416+0000","last_active":"2024-09-13T13:14:12.398574+0000","last_peered":"2024-09-13T13:14:12.398574+0000","last_clean":"2024-09-13T13:14:12.398574+0000","last_became_active":"2024-09-13T13:12:52.999924+0000","last_became_peered":"2024-09-13T13:12:52.999924+0000","last_unstale":"2024-09-13T13:14:12.398574+0000","last_undegraded":"2024-09-13T13:14:12.398574+0000","last_fullsized":"2024-09-13T13:14:12.398574+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:13:01.352337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.352337+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:03.437817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":616,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.020523+0000","last_change":"2024-09-13T13:11:54.028616+0000","last_active":"2024-09-13T13:14:10.020523+0000","last_peered":"2024-09-13T13:14:10.020523+0000","last_clean":"2024-09-13T13:14:10.020523+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:14:10.020523+0000","last_undegraded":"2024-09-13T13:14:10.020523+0000","last_fullsized":"2024-09-13T13:14:10.020523+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:54.028552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:54.028552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:03.452848+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":182,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.028640+0000","last_change":"2024-09-13T13:13:06.952227+0000","last_active":"2024-09-13T13:14:10.028640+0000","last_peered":"2024-09-13T13:14:10.028640+0000","last_clean":"2024-09-13T13:14:10.028640+0000","last_became_active":"2024-09-13T13:12:39.948765+0000","last_became_peered":"2024-09-13T13:12:39.948765+0000","last_unstale":"2024-09-13T13:14:10.028640+0000","last_undegraded":"2024-09-13T13:14:10.028640+0000","last_fullsized":"2024-09-13T13:14:10.028640+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:13:06.952132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.952132+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:21.389641+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":107,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.021757+0000","last_change":"2024-09-13T13:13:17.542667+0000","last_active":"2024-09-13T13:14:10.021757+0000","last_peered":"2024-09-13T13:14:10.021757+0000","last_clean":"2024-09-13T13:14:10.021757+0000","last_became_active":"2024-09-13T13:12:39.950496+0000","last_became_peered":"2024-09-13T13:12:39.950496+0000","last_unstale":"2024-09-13T13:14:10.021757+0000","last_undegraded":"2024-09-13T13:14:10.021757+0000","last_fullsized":"2024-09-13T13:14:10.021757+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:17.542599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:17.542599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:17.868103+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":631,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017989+0000","last_change":"2024-09-13T13:12:52.959776+0000","last_active":"2024-09-13T13:14:10.017989+0000","last_peered":"2024-09-13T13:14:10.017989+0000","last_clean":"2024-09-13T13:14:10.017989+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:14:10.017989+0000","last_undegraded":"2024-09-13T13:14:10.017989+0000","last_fullsized":"2024-09-13T13:14:10.017989+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:15.030577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:15.030577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:22.466810+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"342'2294","reported_seq":2321,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:21.044627+0000","last_change":"2024-09-13T13:13:58.965986+0000","last_active":"2024-09-13T13:14:21.044627+0000","last_peered":"2024-09-13T13:14:21.044627+0000","last_clean":"2024-09-13T13:14:21.044627+0000","last_became_active":"2024-09-13T13:13:58.965566+0000","last_became_peered":"2024-09-13T13:13:58.965566+0000","last_unstale":"2024-09-13T13:14:21.044627+0000","last_undegraded":"2024-09-13T13:14:21.044627+0000","last_fullsized":"2024-09-13T13:14:21.044627+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2294,"log_dups_size":0,"ondisk_log_size":2294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:59.817222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150339584,"num_objects":2294,"num_object_clones":0,"num_object_copies":6882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2294,"num_write_kb":146816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":104,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013970+0000","last_change":"2024-09-13T13:13:38.900634+0000","last_active":"2024-09-13T13:14:10.013970+0000","last_peered":"2024-09-13T13:14:10.013970+0000","last_clean":"2024-09-13T13:14:10.013970+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:14:10.013970+0000","last_undegraded":"2024-09-13T13:14:10.013970+0000","last_fullsized":"2024-09-13T13:14:10.013970+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:38.900566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:38.900566+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:53.707734+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":84,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012589+0000","last_change":"2024-09-13T13:12:53.007182+0000","last_active":"2024-09-13T13:14:10.012589+0000","last_peered":"2024-09-13T13:14:10.012589+0000","last_clean":"2024-09-13T13:14:10.012589+0000","last_became_active":"2024-09-13T13:12:53.006271+0000","last_became_peered":"2024-09-13T13:12:53.006271+0000","last_unstale":"2024-09-13T13:14:10.012589+0000","last_undegraded":"2024-09-13T13:14:10.012589+0000","last_fullsized":"2024-09-13T13:14:10.012589+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:01.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:01.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:22.070336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":653,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:14.630794+0000","last_change":"2024-09-13T13:14:14.630794+0000","last_active":"2024-09-13T13:14:14.630794+0000","last_peered":"2024-09-13T13:14:14.630794+0000","last_clean":"2024-09-13T13:14:14.630794+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:14:14.630794+0000","last_undegraded":"2024-09-13T13:14:14.630794+0000","last_fullsized":"2024-09-13T13:14:14.630794+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:14.630719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:14.630719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:40.328699+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":603,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.020246+0000","last_change":"2024-09-13T13:10:34.526736+0000","last_active":"2024-09-13T13:14:10.020246+0000","last_peered":"2024-09-13T13:14:10.020246+0000","last_clean":"2024-09-13T13:14:10.020246+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:14:10.020246+0000","last_undegraded":"2024-09-13T13:14:10.020246+0000","last_fullsized":"2024-09-13T13:14:10.020246+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:34.526666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:34.526666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:22.041283+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"342'2038","reported_seq":2065,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:18.364535+0000","last_change":"2024-09-13T13:13:58.962261+0000","last_active":"2024-09-13T13:14:18.364535+0000","last_peered":"2024-09-13T13:14:18.364535+0000","last_clean":"2024-09-13T13:14:18.364535+0000","last_became_active":"2024-09-13T13:13:58.961825+0000","last_became_peered":"2024-09-13T13:13:58.961825+0000","last_unstale":"2024-09-13T13:14:18.364535+0000","last_undegraded":"2024-09-13T13:14:18.364535+0000","last_fullsized":"2024-09-13T13:14:18.364535+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2038,"log_dups_size":0,"ondisk_log_size":2038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:23.673189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133562368,"num_objects":2038,"num_object_clones":0,"num_object_copies":6114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":130432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":110,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.014496+0000","last_change":"2024-09-13T13:12:58.980713+0000","last_active":"2024-09-13T13:14:10.014496+0000","last_peered":"2024-09-13T13:14:10.014496+0000","last_clean":"2024-09-13T13:14:10.014496+0000","last_became_active":"2024-09-13T13:12:53.001687+0000","last_became_peered":"2024-09-13T13:12:53.001687+0000","last_unstale":"2024-09-13T13:14:10.014496+0000","last_undegraded":"2024-09-13T13:14:10.014496+0000","last_fullsized":"2024-09-13T13:14:10.014496+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:12:58.980652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.980652+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:18.706915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":627,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:19.532275+0000","last_change":"2024-09-13T13:14:19.532275+0000","last_active":"2024-09-13T13:14:19.532275+0000","last_peered":"2024-09-13T13:14:19.532275+0000","last_clean":"2024-09-13T13:14:19.532275+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:14:19.532275+0000","last_undegraded":"2024-09-13T13:14:19.532275+0000","last_fullsized":"2024-09-13T13:14:19.532275+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:19.532200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:19.532200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:20.249973+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":619,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:14.857443+0000","last_change":"2024-09-13T13:14:14.857443+0000","last_active":"2024-09-13T13:14:14.857443+0000","last_peered":"2024-09-13T13:14:14.857443+0000","last_clean":"2024-09-13T13:14:14.857443+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:14:14.857443+0000","last_undegraded":"2024-09-13T13:14:14.857443+0000","last_fullsized":"2024-09-13T13:14:14.857443+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:14.857364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:14.857364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:24.043778+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":615,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.022098+0000","last_change":"2024-09-13T13:13:54.649219+0000","last_active":"2024-09-13T13:14:10.022098+0000","last_peered":"2024-09-13T13:14:10.022098+0000","last_clean":"2024-09-13T13:14:10.022098+0000","last_became_active":"2024-09-13T13:12:52.938975+0000","last_became_peered":"2024-09-13T13:12:52.938975+0000","last_unstale":"2024-09-13T13:14:10.022098+0000","last_undegraded":"2024-09-13T13:14:10.022098+0000","last_fullsized":"2024-09-13T13:14:10.022098+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.649166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.649166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:13.711439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"342'2093","reported_seq":2120,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:18.952064+0000","last_change":"2024-09-13T13:13:58.963327+0000","last_active":"2024-09-13T13:14:18.952064+0000","last_peered":"2024-09-13T13:14:18.952064+0000","last_clean":"2024-09-13T13:14:18.952064+0000","last_became_active":"2024-09-13T13:13:58.963064+0000","last_became_peered":"2024-09-13T13:13:58.963064+0000","last_unstale":"2024-09-13T13:14:18.952064+0000","last_undegraded":"2024-09-13T13:14:18.952064+0000","last_fullsized":"2024-09-13T13:14:18.952064+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2093,"log_dups_size":0,"ondisk_log_size":2093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:12.065758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137166848,"num_objects":2093,"num_object_clones":0,"num_object_copies":6279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2093,"num_write_kb":133952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":96,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013665+0000","last_change":"2024-09-13T13:12:53.018645+0000","last_active":"2024-09-13T13:14:10.013665+0000","last_peered":"2024-09-13T13:14:10.013665+0000","last_clean":"2024-09-13T13:14:10.013665+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:14:10.013665+0000","last_undegraded":"2024-09-13T13:14:10.013665+0000","last_fullsized":"2024-09-13T13:14:10.013665+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:09.739184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.739184+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:28.062380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":618,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.024951+0000","last_change":"2024-09-13T13:12:53.003530+0000","last_active":"2024-09-13T13:14:10.024951+0000","last_peered":"2024-09-13T13:14:10.024951+0000","last_clean":"2024-09-13T13:14:10.024951+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:14:10.024951+0000","last_undegraded":"2024-09-13T13:14:10.024951+0000","last_fullsized":"2024-09-13T13:14:10.024951+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.996346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.996346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:06.356979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":765,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:12.397422+0000","last_change":"2024-09-13T13:12:36.113757+0000","last_active":"2024-09-13T13:14:12.397422+0000","last_peered":"2024-09-13T13:14:12.397422+0000","last_clean":"2024-09-13T13:14:12.397422+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:14:12.397422+0000","last_undegraded":"2024-09-13T13:14:12.397422+0000","last_fullsized":"2024-09-13T13:14:12.397422+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:12:36.113709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:36.113709+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:07.451544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013952+0000","last_change":"2024-09-13T13:13:41.797728+0000","last_active":"2024-09-13T13:14:10.013952+0000","last_peered":"2024-09-13T13:14:10.013952+0000","last_clean":"2024-09-13T13:14:10.013952+0000","last_became_active":"2024-09-13T13:12:52.997283+0000","last_became_peered":"2024-09-13T13:12:52.997283+0000","last_unstale":"2024-09-13T13:14:10.013952+0000","last_undegraded":"2024-09-13T13:14:10.013952+0000","last_fullsized":"2024-09-13T13:14:10.013952+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.797639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.797639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:58.404714+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"342'2055","reported_seq":2082,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:18.400399+0000","last_change":"2024-09-13T13:13:58.963159+0000","last_active":"2024-09-13T13:14:18.400399+0000","last_peered":"2024-09-13T13:14:18.400399+0000","last_clean":"2024-09-13T13:14:18.400399+0000","last_became_active":"2024-09-13T13:13:58.962803+0000","last_became_peered":"2024-09-13T13:13:58.962803+0000","last_unstale":"2024-09-13T13:14:18.400399+0000","last_undegraded":"2024-09-13T13:14:18.400399+0000","last_fullsized":"2024-09-13T13:14:18.400399+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2055,"log_dups_size":0,"ondisk_log_size":2055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:24.258620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134676480,"num_objects":2055,"num_object_clones":0,"num_object_copies":6165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2055,"num_write_kb":131520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":650,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.026283+0000","last_change":"2024-09-13T13:13:56.850149+0000","last_active":"2024-09-13T13:14:10.026283+0000","last_peered":"2024-09-13T13:14:10.026283+0000","last_clean":"2024-09-13T13:14:10.026283+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:14:10.026283+0000","last_undegraded":"2024-09-13T13:14:10.026283+0000","last_fullsized":"2024-09-13T13:14:10.026283+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:56.850094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.850094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:59.311626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":639,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017594+0000","last_change":"2024-09-13T13:12:58.857634+0000","last_active":"2024-09-13T13:14:10.017594+0000","last_peered":"2024-09-13T13:14:10.017594+0000","last_clean":"2024-09-13T13:14:10.017594+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:14:10.017594+0000","last_undegraded":"2024-09-13T13:14:10.017594+0000","last_fullsized":"2024-09-13T13:14:10.017594+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:58.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:28.262215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":636,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.020963+0000","last_change":"2024-09-13T13:13:04.341912+0000","last_active":"2024-09-13T13:14:10.020963+0000","last_peered":"2024-09-13T13:14:10.020963+0000","last_clean":"2024-09-13T13:14:10.020963+0000","last_became_active":"2024-09-13T13:12:52.944725+0000","last_became_peered":"2024-09-13T13:12:52.944725+0000","last_unstale":"2024-09-13T13:14:10.020963+0000","last_undegraded":"2024-09-13T13:14:10.020963+0000","last_fullsized":"2024-09-13T13:14:10.020963+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.341673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.341673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:15.009663+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":600,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.025833+0000","last_change":"2024-09-13T13:13:41.762003+0000","last_active":"2024-09-13T13:14:10.025833+0000","last_peered":"2024-09-13T13:14:10.025833+0000","last_clean":"2024-09-13T13:14:10.025833+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:14:10.025833+0000","last_undegraded":"2024-09-13T13:14:10.025833+0000","last_fullsized":"2024-09-13T13:14:10.025833+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.761814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.761814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:08.371463+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"342'2014","reported_seq":2041,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:19.279109+0000","last_change":"2024-09-13T13:13:58.963548+0000","last_active":"2024-09-13T13:14:19.279109+0000","last_peered":"2024-09-13T13:14:19.279109+0000","last_clean":"2024-09-13T13:14:19.279109+0000","last_became_active":"2024-09-13T13:13:58.963160+0000","last_became_peered":"2024-09-13T13:13:58.963160+0000","last_unstale":"2024-09-13T13:14:19.279109+0000","last_undegraded":"2024-09-13T13:14:19.279109+0000","last_fullsized":"2024-09-13T13:14:19.279109+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2014,"log_dups_size":0,"ondisk_log_size":2014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:04.981823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131989504,"num_objects":2014,"num_object_clones":0,"num_object_copies":6042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2014,"num_write_kb":128896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":624,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.015602+0000","last_change":"2024-09-13T13:13:06.339762+0000","last_active":"2024-09-13T13:14:10.015602+0000","last_peered":"2024-09-13T13:14:10.015602+0000","last_clean":"2024-09-13T13:14:10.015602+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:14:10.015602+0000","last_undegraded":"2024-09-13T13:14:10.015602+0000","last_fullsized":"2024-09-13T13:14:10.015602+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:13:06.339664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.339664+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:08.487728+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":246,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017632+0000","last_change":"2024-09-13T13:13:52.445988+0000","last_active":"2024-09-13T13:14:10.017632+0000","last_peered":"2024-09-13T13:14:10.017632+0000","last_clean":"2024-09-13T13:14:10.017632+0000","last_became_active":"2024-09-13T13:12:39.954315+0000","last_became_peered":"2024-09-13T13:12:39.954315+0000","last_unstale":"2024-09-13T13:14:10.017632+0000","last_undegraded":"2024-09-13T13:14:10.017632+0000","last_fullsized":"2024-09-13T13:14:10.017632+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:52.445926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:52.445926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:54.728929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":613,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.023975+0000","last_change":"2024-09-13T13:13:11.484689+0000","last_active":"2024-09-13T13:14:10.023975+0000","last_peered":"2024-09-13T13:14:10.023975+0000","last_clean":"2024-09-13T13:14:10.023975+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:14:10.023975+0000","last_undegraded":"2024-09-13T13:14:10.023975+0000","last_fullsized":"2024-09-13T13:14:10.023975+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.484615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.484615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:37.563802+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":99,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.021360+0000","last_change":"2024-09-13T13:12:40.118611+0000","last_active":"2024-09-13T13:14:10.021360+0000","last_peered":"2024-09-13T13:14:10.021360+0000","last_clean":"2024-09-13T13:14:10.021360+0000","last_became_active":"2024-09-13T13:12:40.118050+0000","last_became_peered":"2024-09-13T13:12:40.118050+0000","last_unstale":"2024-09-13T13:14:10.021360+0000","last_undegraded":"2024-09-13T13:14:10.021360+0000","last_fullsized":"2024-09-13T13:14:10.021360+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:40.961744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.961744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:01.520492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"342'2251","reported_seq":2278,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:21.066024+0000","last_change":"2024-09-13T13:13:58.966955+0000","last_active":"2024-09-13T13:14:21.066024+0000","last_peered":"2024-09-13T13:14:21.066024+0000","last_clean":"2024-09-13T13:14:21.066024+0000","last_became_active":"2024-09-13T13:13:58.966675+0000","last_became_peered":"2024-09-13T13:13:58.966675+0000","last_unstale":"2024-09-13T13:14:21.066024+0000","last_undegraded":"2024-09-13T13:14:21.066024+0000","last_fullsized":"2024-09-13T13:14:21.066024+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:09.947796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147521536,"num_objects":2251,"num_object_clones":0,"num_object_copies":6753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2251,"num_write_kb":144064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":104,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013134+0000","last_change":"2024-09-13T13:13:20.064693+0000","last_active":"2024-09-13T13:14:10.013134+0000","last_peered":"2024-09-13T13:14:10.013134+0000","last_clean":"2024-09-13T13:14:10.013134+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:14:10.013134+0000","last_undegraded":"2024-09-13T13:14:10.013134+0000","last_fullsized":"2024-09-13T13:14:10.013134+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:13:20.064614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:20.064614+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:42.196256+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013868+0000","last_change":"2024-09-13T13:13:42.780315+0000","last_active":"2024-09-13T13:14:10.013868+0000","last_peered":"2024-09-13T13:14:10.013868+0000","last_clean":"2024-09-13T13:14:10.013868+0000","last_became_active":"2024-09-13T13:12:53.007561+0000","last_became_peered":"2024-09-13T13:12:53.007561+0000","last_unstale":"2024-09-13T13:14:10.013868+0000","last_undegraded":"2024-09-13T13:14:10.013868+0000","last_fullsized":"2024-09-13T13:14:10.013868+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:42.780247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:42.780247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:48.627353+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":598,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.021485+0000","last_change":"2024-09-13T13:13:51.608073+0000","last_active":"2024-09-13T13:14:10.021485+0000","last_peered":"2024-09-13T13:14:10.021485+0000","last_clean":"2024-09-13T13:14:10.021485+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:14:10.021485+0000","last_undegraded":"2024-09-13T13:14:10.021485+0000","last_fullsized":"2024-09-13T13:14:10.021485+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.608025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.608025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:00.485134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":584,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.024552+0000","last_change":"2024-09-13T13:11:17.667223+0000","last_active":"2024-09-13T13:14:10.024552+0000","last_peered":"2024-09-13T13:14:10.024552+0000","last_clean":"2024-09-13T13:14:10.024552+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:14:10.024552+0000","last_undegraded":"2024-09-13T13:14:10.024552+0000","last_fullsized":"2024-09-13T13:14:10.024552+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:17.667151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:17.667151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:02.403548+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"342'2036","reported_seq":2061,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:18.336058+0000","last_change":"2024-09-13T13:13:58.963039+0000","last_active":"2024-09-13T13:14:18.336058+0000","last_peered":"2024-09-13T13:14:18.336058+0000","last_clean":"2024-09-13T13:14:18.336058+0000","last_became_active":"2024-09-13T13:13:58.962773+0000","last_became_peered":"2024-09-13T13:13:58.962773+0000","last_unstale":"2024-09-13T13:14:18.336058+0000","last_undegraded":"2024-09-13T13:14:18.336058+0000","last_fullsized":"2024-09-13T13:14:18.336058+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":2036,"log_dups_size":0,"ondisk_log_size":2036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:26.925464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133431296,"num_objects":2036,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2036,"num_write_kb":130304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":671,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.016975+0000","last_change":"2024-09-13T13:13:15.581950+0000","last_active":"2024-09-13T13:14:10.016975+0000","last_peered":"2024-09-13T13:14:10.016975+0000","last_clean":"2024-09-13T13:14:10.016975+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:14:10.016975+0000","last_undegraded":"2024-09-13T13:14:10.016975+0000","last_fullsized":"2024-09-13T13:14:10.016975+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:13:15.581882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:15.581882+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:19.155967+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":98,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013655+0000","last_change":"2024-09-13T13:13:01.972907+0000","last_active":"2024-09-13T13:14:10.013655+0000","last_peered":"2024-09-13T13:14:10.013655+0000","last_clean":"2024-09-13T13:14:10.013655+0000","last_became_active":"2024-09-13T13:12:53.006362+0000","last_became_peered":"2024-09-13T13:12:53.006362+0000","last_unstale":"2024-09-13T13:14:10.013655+0000","last_undegraded":"2024-09-13T13:14:10.013655+0000","last_fullsized":"2024-09-13T13:14:10.013655+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:13:01.972831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.972831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:25.555388+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"306'13","reported_seq":320,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:12.398354+0000","last_change":"2024-09-13T13:13:03.334319+0000","last_active":"2024-09-13T13:14:12.398354+0000","last_peered":"2024-09-13T13:14:12.398354+0000","last_clean":"2024-09-13T13:14:12.398354+0000","last_became_active":"2024-09-13T13:12:53.001189+0000","last_became_peered":"2024-09-13T13:12:53.001189+0000","last_unstale":"2024-09-13T13:14:12.398354+0000","last_undegraded":"2024-09-13T13:14:12.398354+0000","last_fullsized":"2024-09-13T13:14:12.398354+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'13","last_scrub_stamp":"2024-09-13T13:13:03.334012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.334012+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:32.766925+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":612,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017267+0000","last_change":"2024-09-13T13:13:55.500961+0000","last_active":"2024-09-13T13:14:10.017267+0000","last_peered":"2024-09-13T13:14:10.017267+0000","last_clean":"2024-09-13T13:14:10.017267+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:14:10.017267+0000","last_undegraded":"2024-09-13T13:14:10.017267+0000","last_fullsized":"2024-09-13T13:14:10.017267+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:55.500908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.500908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:07.523230+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":589,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.015747+0000","last_change":"2024-09-13T13:13:04.371082+0000","last_active":"2024-09-13T13:14:10.015747+0000","last_peered":"2024-09-13T13:14:10.015747+0000","last_clean":"2024-09-13T13:14:10.015747+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:14:10.015747+0000","last_undegraded":"2024-09-13T13:14:10.015747+0000","last_fullsized":"2024-09-13T13:14:10.015747+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.370881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.370881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:06.461544+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":126,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017749+0000","last_change":"2024-09-13T13:13:58.537515+0000","last_active":"2024-09-13T13:14:10.017749+0000","last_peered":"2024-09-13T13:14:10.017749+0000","last_clean":"2024-09-13T13:14:10.017749+0000","last_became_active":"2024-09-13T13:12:34.673528+0000","last_became_peered":"2024-09-13T13:12:34.673528+0000","last_unstale":"2024-09-13T13:14:10.017749+0000","last_undegraded":"2024-09-13T13:14:10.017749+0000","last_fullsized":"2024-09-13T13:14:10.017749+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:13:58.537459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:58.537459+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:20.801720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":628,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:13.633412+0000","last_change":"2024-09-13T13:14:13.633412+0000","last_active":"2024-09-13T13:14:13.633412+0000","last_peered":"2024-09-13T13:14:13.633412+0000","last_clean":"2024-09-13T13:14:13.633412+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:14:13.633412+0000","last_undegraded":"2024-09-13T13:14:13.633412+0000","last_fullsized":"2024-09-13T13:14:13.633412+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:13.633215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:13.633215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:22.030001+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":641,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:14.100025+0000","last_change":"2024-09-13T13:14:14.100025+0000","last_active":"2024-09-13T13:14:14.100025+0000","last_peered":"2024-09-13T13:14:14.100025+0000","last_clean":"2024-09-13T13:14:14.100025+0000","last_became_active":"2024-09-13T13:12:53.000873+0000","last_became_peered":"2024-09-13T13:12:53.000873+0000","last_unstale":"2024-09-13T13:14:14.100025+0000","last_undegraded":"2024-09-13T13:14:14.100025+0000","last_fullsized":"2024-09-13T13:14:14.100025+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:14.099972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:14.099972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:20.120831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":624,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.027538+0000","last_change":"2024-09-13T13:13:12.952427+0000","last_active":"2024-09-13T13:14:10.027538+0000","last_peered":"2024-09-13T13:14:10.027538+0000","last_clean":"2024-09-13T13:14:10.027538+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:14:10.027538+0000","last_undegraded":"2024-09-13T13:14:10.027538+0000","last_fullsized":"2024-09-13T13:14:10.027538+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:12.952348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:12.952348+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:25.638147+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012762+0000","last_change":"2024-09-13T13:13:51.764793+0000","last_active":"2024-09-13T13:14:10.012762+0000","last_peered":"2024-09-13T13:14:10.012762+0000","last_clean":"2024-09-13T13:14:10.012762+0000","last_became_active":"2024-09-13T13:12:53.006579+0000","last_became_peered":"2024-09-13T13:12:53.006579+0000","last_unstale":"2024-09-13T13:14:10.012762+0000","last_undegraded":"2024-09-13T13:14:10.012762+0000","last_fullsized":"2024-09-13T13:14:10.012762+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.764717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.764717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:01.471924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":604,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.023342+0000","last_change":"2024-09-13T13:13:56.359731+0000","last_active":"2024-09-13T13:14:10.023342+0000","last_peered":"2024-09-13T13:14:10.023342+0000","last_clean":"2024-09-13T13:14:10.023342+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:14:10.023342+0000","last_undegraded":"2024-09-13T13:14:10.023342+0000","last_fullsized":"2024-09-13T13:14:10.023342+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:56.359684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.359684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:21.764264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":647,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:15.616158+0000","last_change":"2024-09-13T13:14:15.616158+0000","last_active":"2024-09-13T13:14:15.616158+0000","last_peered":"2024-09-13T13:14:15.616158+0000","last_clean":"2024-09-13T13:14:15.616158+0000","last_became_active":"2024-09-13T13:12:52.950504+0000","last_became_peered":"2024-09-13T13:12:52.950504+0000","last_unstale":"2024-09-13T13:14:15.616158+0000","last_undegraded":"2024-09-13T13:14:15.616158+0000","last_fullsized":"2024-09-13T13:14:15.616158+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:15.616112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:15.616112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:32.376467+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":637,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017693+0000","last_change":"2024-09-13T13:12:34.682535+0000","last_active":"2024-09-13T13:14:10.017693+0000","last_peered":"2024-09-13T13:14:10.017693+0000","last_clean":"2024-09-13T13:14:10.017693+0000","last_became_active":"2024-09-13T13:12:34.665746+0000","last_became_peered":"2024-09-13T13:12:34.665746+0000","last_unstale":"2024-09-13T13:14:10.017693+0000","last_undegraded":"2024-09-13T13:14:10.017693+0000","last_fullsized":"2024-09-13T13:14:10.017693+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:19.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:19.931466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:34.330831+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":603,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.029134+0000","last_change":"2024-09-13T13:13:05.981881+0000","last_active":"2024-09-13T13:14:10.029134+0000","last_peered":"2024-09-13T13:14:10.029134+0000","last_clean":"2024-09-13T13:14:10.029134+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:14:10.029134+0000","last_undegraded":"2024-09-13T13:14:10.029134+0000","last_fullsized":"2024-09-13T13:14:10.029134+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:05.981803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:05.981803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:06.678092+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":764,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:12.399174+0000","last_change":"2024-09-13T13:13:10.130755+0000","last_active":"2024-09-13T13:14:12.399174+0000","last_peered":"2024-09-13T13:14:12.399174+0000","last_clean":"2024-09-13T13:14:12.399174+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:14:12.399174+0000","last_undegraded":"2024-09-13T13:14:12.399174+0000","last_fullsized":"2024-09-13T13:14:12.399174+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:13:10.130680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.130680+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:38.263233+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":92,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013769+0000","last_change":"2024-09-13T13:13:11.956837+0000","last_active":"2024-09-13T13:14:10.013769+0000","last_peered":"2024-09-13T13:14:10.013769+0000","last_clean":"2024-09-13T13:14:10.013769+0000","last_became_active":"2024-09-13T13:12:52.946339+0000","last_became_peered":"2024-09-13T13:12:52.946339+0000","last_unstale":"2024-09-13T13:14:10.013769+0000","last_undegraded":"2024-09-13T13:14:10.013769+0000","last_fullsized":"2024-09-13T13:14:10.013769+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.956784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.956784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:24.331443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":646,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:16.609809+0000","last_change":"2024-09-13T13:14:16.609809+0000","last_active":"2024-09-13T13:14:16.609809+0000","last_peered":"2024-09-13T13:14:16.609809+0000","last_clean":"2024-09-13T13:14:16.609809+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:14:16.609809+0000","last_undegraded":"2024-09-13T13:14:16.609809+0000","last_fullsized":"2024-09-13T13:14:16.609809+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:14:16.609677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:16.609677+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:32.444192+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":618,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.024252+0000","last_change":"2024-09-13T13:13:57.105151+0000","last_active":"2024-09-13T13:14:10.024252+0000","last_peered":"2024-09-13T13:14:10.024252+0000","last_clean":"2024-09-13T13:14:10.024252+0000","last_became_active":"2024-09-13T13:12:52.999912+0000","last_became_peered":"2024-09-13T13:12:52.999912+0000","last_unstale":"2024-09-13T13:14:10.024252+0000","last_undegraded":"2024-09-13T13:14:10.024252+0000","last_fullsized":"2024-09-13T13:14:10.024252+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.105098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.105098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:13.380406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":780,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:12.398336+0000","last_change":"2024-09-13T13:13:04.777566+0000","last_active":"2024-09-13T13:14:12.398336+0000","last_peered":"2024-09-13T13:14:12.398336+0000","last_clean":"2024-09-13T13:14:12.398336+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:14:12.398336+0000","last_undegraded":"2024-09-13T13:14:12.398336+0000","last_fullsized":"2024-09-13T13:14:12.398336+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:13:04.777515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.777515+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:25.054666+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":607,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.026892+0000","last_change":"2024-09-13T13:13:49.754588+0000","last_active":"2024-09-13T13:14:10.026892+0000","last_peered":"2024-09-13T13:14:10.026892+0000","last_clean":"2024-09-13T13:14:10.026892+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:14:10.026892+0000","last_undegraded":"2024-09-13T13:14:10.026892+0000","last_fullsized":"2024-09-13T13:14:10.026892+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:49.754540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:49.754540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:09.274489+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":104,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013217+0000","last_change":"2024-09-13T13:13:47.728092+0000","last_active":"2024-09-13T13:14:10.013217+0000","last_peered":"2024-09-13T13:14:10.013217+0000","last_clean":"2024-09-13T13:14:10.013217+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:14:10.013217+0000","last_undegraded":"2024-09-13T13:14:10.013217+0000","last_fullsized":"2024-09-13T13:14:10.013217+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:47.728010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:47.728010+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:08.916792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":84,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.013751+0000","last_change":"2024-09-13T13:12:53.006302+0000","last_active":"2024-09-13T13:14:10.013751+0000","last_peered":"2024-09-13T13:14:10.013751+0000","last_clean":"2024-09-13T13:14:10.013751+0000","last_became_active":"2024-09-13T13:12:53.005988+0000","last_became_peered":"2024-09-13T13:12:53.005988+0000","last_unstale":"2024-09-13T13:14:10.013751+0000","last_undegraded":"2024-09-13T13:14:10.013751+0000","last_fullsized":"2024-09-13T13:14:10.013751+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:40.888083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.888083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:08.765290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":605,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.012077+0000","last_change":"2024-09-13T13:13:15.674968+0000","last_active":"2024-09-13T13:14:10.012077+0000","last_peered":"2024-09-13T13:14:10.012077+0000","last_clean":"2024-09-13T13:14:10.012077+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:14:10.012077+0000","last_undegraded":"2024-09-13T13:14:10.012077+0000","last_fullsized":"2024-09-13T13:14:10.012077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:15.674898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:15.674898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:42.248042+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":600,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.020293+0000","last_change":"2024-09-13T13:12:59.145916+0000","last_active":"2024-09-13T13:14:10.020293+0000","last_peered":"2024-09-13T13:14:10.020293+0000","last_clean":"2024-09-13T13:14:10.020293+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:14:10.020293+0000","last_undegraded":"2024-09-13T13:14:10.020293+0000","last_fullsized":"2024-09-13T13:14:10.020293+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:59.145847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:59.145847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:16.083736+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":577,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.017216+0000","last_change":"2024-09-13T13:12:39.874394+0000","last_active":"2024-09-13T13:14:10.017216+0000","last_peered":"2024-09-13T13:14:10.017216+0000","last_clean":"2024-09-13T13:14:10.017216+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:14:10.017216+0000","last_undegraded":"2024-09-13T13:14:10.017216+0000","last_fullsized":"2024-09-13T13:14:10.017216+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:39.874324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:39.874324+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:07.639902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":604,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:13.418389+0000","last_change":"2024-09-13T13:14:13.418389+0000","last_active":"2024-09-13T13:14:13.418389+0000","last_peered":"2024-09-13T13:14:13.418389+0000","last_clean":"2024-09-13T13:14:13.418389+0000","last_became_active":"2024-09-13T13:12:52.949094+0000","last_became_peered":"2024-09-13T13:12:52.949094+0000","last_unstale":"2024-09-13T13:14:13.418389+0000","last_undegraded":"2024-09-13T13:14:13.418389+0000","last_fullsized":"2024-09-13T13:14:13.418389+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:13.418196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:13.418196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:43.286027+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":604,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.024714+0000","last_change":"2024-09-13T13:13:44.226432+0000","last_active":"2024-09-13T13:14:10.024714+0000","last_peered":"2024-09-13T13:14:10.024714+0000","last_clean":"2024-09-13T13:14:10.024714+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:14:10.024714+0000","last_undegraded":"2024-09-13T13:14:10.024714+0000","last_fullsized":"2024-09-13T13:14:10.024714+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:44.226240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:44.226240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:00.263002+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":599,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.027767+0000","last_change":"2024-09-13T13:10:39.477882+0000","last_active":"2024-09-13T13:14:10.027767+0000","last_peered":"2024-09-13T13:14:10.027767+0000","last_clean":"2024-09-13T13:14:10.027767+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:14:10.027767+0000","last_undegraded":"2024-09-13T13:14:10.027767+0000","last_fullsized":"2024-09-13T13:14:10.027767+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:39.477683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.477683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:37.708691+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":634,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.020416+0000","last_change":"2024-09-13T13:13:49.268080+0000","last_active":"2024-09-13T13:14:10.020416+0000","last_peered":"2024-09-13T13:14:10.020416+0000","last_clean":"2024-09-13T13:14:10.020416+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:14:10.020416+0000","last_undegraded":"2024-09-13T13:14:10.020416+0000","last_fullsized":"2024-09-13T13:14:10.020416+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:49.268018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:49.268018+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:12.973884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":620,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.018059+0000","last_change":"2024-09-13T13:13:44.485526+0000","last_active":"2024-09-13T13:14:10.018059+0000","last_peered":"2024-09-13T13:14:10.018059+0000","last_clean":"2024-09-13T13:14:10.018059+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:14:10.018059+0000","last_undegraded":"2024-09-13T13:14:10.018059+0000","last_fullsized":"2024-09-13T13:14:10.018059+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:44.485364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:44.485364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:47.391654+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":629,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.023746+0000","last_change":"2024-09-13T13:12:53.000159+0000","last_active":"2024-09-13T13:14:10.023746+0000","last_peered":"2024-09-13T13:14:10.023746+0000","last_clean":"2024-09-13T13:14:10.023746+0000","last_became_active":"2024-09-13T13:12:52.999748+0000","last_became_peered":"2024-09-13T13:12:52.999748+0000","last_unstale":"2024-09-13T13:14:10.023746+0000","last_undegraded":"2024-09-13T13:14:10.023746+0000","last_fullsized":"2024-09-13T13:14:10.023746+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:42.847961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:42.847961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:00.102694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":595,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.024856+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:14:10.024856+0000","last_peered":"2024-09-13T13:14:10.024856+0000","last_clean":"2024-09-13T13:14:10.024856+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:14:10.024856+0000","last_undegraded":"2024-09-13T13:14:10.024856+0000","last_fullsized":"2024-09-13T13:14:10.024856+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:07.339701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":644,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:12.368507+0000","last_change":"2024-09-13T13:14:12.368507+0000","last_active":"2024-09-13T13:14:12.368507+0000","last_peered":"2024-09-13T13:14:12.368507+0000","last_clean":"2024-09-13T13:14:12.368507+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:14:12.368507+0000","last_undegraded":"2024-09-13T13:14:12.368507+0000","last_fullsized":"2024-09-13T13:14:12.368507+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:14:12.368430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:12.368430+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:32.625458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":627,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.026709+0000","last_change":"2024-09-13T13:13:44.750043+0000","last_active":"2024-09-13T13:14:10.026709+0000","last_peered":"2024-09-13T13:14:10.026709+0000","last_clean":"2024-09-13T13:14:10.026709+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:14:10.026709+0000","last_undegraded":"2024-09-13T13:14:10.026709+0000","last_fullsized":"2024-09-13T13:14:10.026709+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:44.749961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:44.749961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:01.447677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":543,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:16.457160+0000","last_change":"2024-09-13T13:14:16.457160+0000","last_active":"2024-09-13T13:14:16.457160+0000","last_peered":"2024-09-13T13:14:16.457160+0000","last_clean":"2024-09-13T13:14:16.457160+0000","last_became_active":"2024-09-13T13:12:53.001131+0000","last_became_peered":"2024-09-13T13:12:53.001131+0000","last_unstale":"2024-09-13T13:14:16.457160+0000","last_undegraded":"2024-09-13T13:14:16.457160+0000","last_fullsized":"2024-09-13T13:14:16.457160+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:16.456784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:16.456784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:42.903894+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":592,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.020787+0000","last_change":"2024-09-13T13:12:52.947380+0000","last_active":"2024-09-13T13:14:10.020787+0000","last_peered":"2024-09-13T13:14:10.020787+0000","last_clean":"2024-09-13T13:14:10.020787+0000","last_became_active":"2024-09-13T13:12:52.946611+0000","last_became_peered":"2024-09-13T13:12:52.946611+0000","last_unstale":"2024-09-13T13:14:10.020787+0000","last_undegraded":"2024-09-13T13:14:10.020787+0000","last_fullsized":"2024-09-13T13:14:10.020787+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:08.704450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:08.704450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:03.458121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":623,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:18.882969+0000","last_change":"2024-09-13T13:14:18.882969+0000","last_active":"2024-09-13T13:14:18.882969+0000","last_peered":"2024-09-13T13:14:18.882969+0000","last_clean":"2024-09-13T13:14:18.882969+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:14:18.882969+0000","last_undegraded":"2024-09-13T13:14:18.882969+0000","last_fullsized":"2024-09-13T13:14:18.882969+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:18.882903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:18.882903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:45.119719+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":604,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.020062+0000","last_change":"2024-09-13T13:11:07.564094+0000","last_active":"2024-09-13T13:14:10.020062+0000","last_peered":"2024-09-13T13:14:10.020062+0000","last_clean":"2024-09-13T13:14:10.020062+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:14:10.020062+0000","last_undegraded":"2024-09-13T13:14:10.020062+0000","last_fullsized":"2024-09-13T13:14:10.020062+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:07.564025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:07.564025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:34.049577+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":611,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.016182+0000","last_change":"2024-09-13T13:11:49.264206+0000","last_active":"2024-09-13T13:14:10.016182+0000","last_peered":"2024-09-13T13:14:10.016182+0000","last_clean":"2024-09-13T13:14:10.016182+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:14:10.016182+0000","last_undegraded":"2024-09-13T13:14:10.016182+0000","last_fullsized":"2024-09-13T13:14:10.016182+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:49.264128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.264128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:14:22.505993+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":620,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-13T13:14:10.027289+0000","last_change":"2024-09-13T13:13:53.809209+0000","last_active":"2024-09-13T13:14:10.027289+0000","last_peered":"2024-09-13T13:14:10.027289+0000","last_clean":"2024-09-13T13:14:10.027289+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:14:10.027289+0000","last_undegraded":"2024-09-13T13:14:10.027289+0000","last_fullsized":"2024-09-13T13:14:10.027289+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:53.809160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:53.809160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:15.146128+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2222391296,"num_objects":33911,"num_object_clones":0,"num_object_copies":101733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33911,"num_write_kb":2170304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6538002432,"data_stored":6538002432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33911,"ondisk_log_size":33911,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":2508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":105,"num_read_kb":100,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201442,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":927064,"kb_used_data":516872,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92818088,"statfs":{"total":95995035648,"available":95045722112,"internally_reserved":0,"allocated":529276928,"data_stored":528747245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":420018634},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141053,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1376136,"kb_used_data":987448,"kb_used_omap":4,"kb_used_meta":388667,"kb_avail":92369016,"statfs":{"total":95995035648,"available":94585872384,"internally_reserved":0,"allocated":1011146752,"data_stored":1010631824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":397995693},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692014,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1121932,"kb_used_data":631228,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92623220,"statfs":{"total":95995035648,"available":94846177280,"internally_reserved":0,"allocated":646377472,"data_stored":645858373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920948,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1367072,"kb_used_data":1024912,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92378080,"statfs":{"total":95995035648,"available":94595153920,"internally_reserved":0,"allocated":1049509888,"data_stored":1048970445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117179,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1212576,"kb_used_data":649680,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92532576,"statfs":{"total":95995035648,"available":94753357824,"internally_reserved":0,"allocated":665272320,"data_stored":664733597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":576387527},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411518,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1126528,"kb_used_data":756592,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92618624,"statfs":{"total":95995035648,"available":94841470976,"internally_reserved":0,"allocated":774750208,"data_stored":774242847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":378796486},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895156,"kb_used_data":398884,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92849996,"statfs":{"total":95995035648,"available":95078395904,"internally_reserved":0,"allocated":408457216,"data_stored":407897804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738572,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1889372,"kb_used_data":1312588,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91855780,"statfs":{"total":95995035648,"available":94060318720,"internally_reserved":0,"allocated":1344090112,"data_stored":1343556042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1379729408,"data_stored":1379729408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409337856,"data_stored":409337856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":796524544,"data_stored":796524544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":675545088,"data_stored":675545088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1050542080,"data_stored":1050542080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":662634496,"data_stored":662634496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1031405568,"data_stored":1031405568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532283392,"data_stored":532283392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:14:22.912 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T13:14:22.913 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:14:22.913 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-13T13:14:22.913 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-13T13:14:22.917 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:22.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:23.019 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:23.060 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:23.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:23.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:23.162 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:23.191 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:23.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:23.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:23.293 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:23.296 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 19 16 39521 39505 129.938 103.625 0.00432004 0.00769069 2024-09-13T13:14:23.331 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:23.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:23.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:23.433 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:23.455 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:23.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:23.556 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:23.596 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:23.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:23.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:23.698 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:23.738 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:23.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:23 smithi167 ceph-mon[33178]: 3.8 scrub starts 2024-09-13T13:14:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:23 smithi167 ceph-mon[33178]: 3.8 scrub ok 2024-09-13T13:14:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:23 smithi167 ceph-mon[33178]: pgmap v659: 148 pgs: 148 active+clean; 2.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-09-13T13:14:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:23 smithi167 ceph-mon[33178]: from='client.36143 v1:172.21.15.52:0/2820702764' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:14:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:23 smithi167 ceph-mon[33178]: osdmap e344: 8 total, 8 up, 8 in 2024-09-13T13:14:23.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:23.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:23.876 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:23.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:23.949 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:23 smithi052 ceph-mon[29471]: 3.8 scrub starts 2024-09-13T13:14:23.949 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:23 smithi052 ceph-mon[29471]: 3.8 scrub ok 2024-09-13T13:14:23.949 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:23 smithi052 ceph-mon[29471]: pgmap v659: 148 pgs: 148 active+clean; 2.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-09-13T13:14:23.949 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:23 smithi052 ceph-mon[29471]: from='client.36143 v1:172.21.15.52:0/2820702764' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:14:23.949 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:23 smithi052 ceph-mon[29471]: osdmap e344: 8 total, 8 up, 8 in 2024-09-13T13:14:23.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:23 smithi052 ceph-mon[36682]: 3.8 scrub starts 2024-09-13T13:14:23.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:23 smithi052 ceph-mon[36682]: 3.8 scrub ok 2024-09-13T13:14:23.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:23 smithi052 ceph-mon[36682]: pgmap v659: 148 pgs: 148 active+clean; 2.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-09-13T13:14:23.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:23 smithi052 ceph-mon[36682]: from='client.36143 v1:172.21.15.52:0/2820702764' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:14:23.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:23 smithi052 ceph-mon[36682]: osdmap e344: 8 total, 8 up, 8 in 2024-09-13T13:14:23.976 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:24.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:24.013 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:24.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:24.114 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:24.171 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:24.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:24.187 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:24.273 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:24.296 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:14:24.296016+0000 min lat: 0.0012696 max lat: 0.355388 avg lat: 0.00774525 2024-09-13T13:14:24.296 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:14:24.296 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 20 16 41307 41291 129.022 111.625 0.00590111 0.00774525 2024-09-13T13:14:24.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:24.310 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:24.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:24.412 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:24.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:24.435 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:24.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:24.536 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:24.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:24.568 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:24.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:24.610 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:24 smithi167 ceph-mon[33178]: 6.c scrub starts 2024-09-13T13:14:24.611 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:14:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:14:24.645 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:24 smithi052 ceph-mon[36682]: 6.c scrub starts 2024-09-13T13:14:24.645 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:24 smithi052 ceph-mon[36682]: 5.6 scrub starts 2024-09-13T13:14:24.646 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:24 smithi052 ceph-mon[36682]: 5.6 scrub ok 2024-09-13T13:14:24.646 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:24 smithi052 ceph-mon[29471]: 6.c scrub starts 2024-09-13T13:14:24.646 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:24 smithi052 ceph-mon[29471]: 5.6 scrub starts 2024-09-13T13:14:24.646 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:24 smithi052 ceph-mon[29471]: 5.6 scrub ok 2024-09-13T13:14:24.670 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:24.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:24.715 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:24.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:24.816 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:24.855 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:24.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:24.918 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:24 smithi167 ceph-mon[33178]: 5.6 scrub starts 2024-09-13T13:14:24.919 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:24 smithi167 ceph-mon[33178]: 5.6 scrub ok 2024-09-13T13:14:24.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:24.988 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:24.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:25.090 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:25.125 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:25.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:25.190 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:14:25.191 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:14:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 344, 2024-09-13T13:14:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: "fsid": "2b415b38-71cf-11ef-bceb-c7b262605968", 2024-09-13T13:14:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: "created": "2024-09-13T12:55:27.893882+0000", 2024-09-13T13:14:25.191 INFO:teuthology.orchestra.run.smithi052.stdout: "modified": "2024-09-13T13:14:22.957095+0000", 2024-09-13T13:14:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "last_up_change": "2024-09-13T13:04:02.281376+0000", 2024-09-13T13:14:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "last_in_change": "2024-09-13T13:12:52.437622+0000", 2024-09-13T13:14:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T13:14:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_num": 5799936, 2024-09-13T13:14:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_set": [ 2024-09-13T13:14:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "pglog_hardlimit", 2024-09-13T13:14:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "purged_snapdirs", 2024-09-13T13:14:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "recovery_deletes", 2024-09-13T13:14:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "sortbitwise" 2024-09-13T13:14:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:14:25.192 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_version": 19, 2024-09-13T13:14:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T13:14:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T13:14:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T13:14:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_snapshot": "", 2024-09-13T13:14:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_max": 10, 2024-09-13T13:14:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "max_osd": 8, 2024-09-13T13:14:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "require_min_compat_client": "luminous", 2024-09-13T13:14:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "min_compat_client": "luminous", 2024-09-13T13:14:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "require_osd_release": "squid", 2024-09-13T13:14:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_crimson": false, 2024-09-13T13:14:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "pools": [ 2024-09-13T13:14:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.193 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 1, 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": ".mgr", 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T12:58:31.271893+0000", 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:25.194 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 1, 2024-09-13T13:14:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 1, 2024-09-13T13:14:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 1, 2024-09-13T13:14:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 1, 2024-09-13T13:14:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 1, 2024-09-13T13:14:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:25.195 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "277", 2024-09-13T13:14:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:14:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:14:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:25.196 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:25.197 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:25.198 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "options": { 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_max": 32, 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_min": 1 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr": {} 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 7.8899998664855957, 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 7.8899998664855957, 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 0.37999999523162842, 2024-09-13T13:14:25.199 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 3, 2024-09-13T13:14:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 3, 2024-09-13T13:14:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 2, 2024-09-13T13:14:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "datapool", 2024-09-13T13:14:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:00.306747+0000", 2024-09-13T13:14:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 8193, 2024-09-13T13:14:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T13:14:25.200 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 3, 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 3, 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 3, 2024-09-13T13:14:25.201 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 3, 2024-09-13T13:14:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 3, 2024-09-13T13:14:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "63", 2024-09-13T13:14:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:25.202 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 2, 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 63, 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:25.203 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:25.204 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd": {} 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:25.205 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 2.6500000953674316, 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 2.6500000953674316, 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 3, 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": ".rgw.root", 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:01.103523+0000", 2024-09-13T13:14:25.206 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:14:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:14:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:25.207 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:14:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:14:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:14:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:14:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:14:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:25.208 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:25.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "61", 2024-09-13T13:14:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:14:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:14:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:25.210 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:25.211 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:25.212 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:14:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:14:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 1.75, 2024-09-13T13:14:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 1.75, 2024-09-13T13:14:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:14:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 1.75, 2024-09-13T13:14:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 1.75, 2024-09-13T13:14:25.213 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 4, 2024-09-13T13:14:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "default.rgw.log", 2024-09-13T13:14:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:02.249986+0000", 2024-09-13T13:14:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:14:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:14:25.214 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:25.215 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:14:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:14:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:14:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:14:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:14:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:25.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "63", 2024-09-13T13:14:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:14:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:14:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:25.217 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:25.218 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:25.219 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 2.25, 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 2.25, 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 2.25, 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 2.25, 2024-09-13T13:14:25.220 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 5, 2024-09-13T13:14:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "default.rgw.control", 2024-09-13T13:14:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:04.203764+0000", 2024-09-13T13:14:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:14:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:14:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:25.221 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:14:25.222 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "65", 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:25.223 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:25.224 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:25.225 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:14:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:14:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:25.226 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 1.5, 2024-09-13T13:14:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 1.5, 2024-09-13T13:14:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:14:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 1.5, 2024-09-13T13:14:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 1.5, 2024-09-13T13:14:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:25.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 6, 2024-09-13T13:14:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "default.rgw.meta", 2024-09-13T13:14:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:06.293676+0000", 2024-09-13T13:14:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:14:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:14:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:25.228 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:25.229 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "67", 2024-09-13T13:14:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:14:25.230 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:25.231 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:25.232 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "options": { 2024-09-13T13:14:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_bias": 4 2024-09-13T13:14:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:14:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:25.233 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 1.75, 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 1.75, 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 1.75, 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 1.75, 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 10, 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "unique_pool_3", 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:13:57.824428+0000", 2024-09-13T13:14:25.234 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 16, 2024-09-13T13:14:25.235 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 16, 2024-09-13T13:14:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 16, 2024-09-13T13:14:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 16, 2024-09-13T13:14:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 16, 2024-09-13T13:14:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "338", 2024-09-13T13:14:25.236 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:25.237 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:25.238 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "rados": {} 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.239 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 3, 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 3, 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 3, 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 3, 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: "osds": [ 2024-09-13T13:14:25.240 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.241 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 0, 2024-09-13T13:14:25.241 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "d52dca1a-94c1-4e9e-93f5-687c038cd7c7", 2024-09-13T13:14:25.241 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:25.241 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:25.241 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:25.241 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:25.241 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:25.241 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:25.241 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 8, 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 334, 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6801", 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:25.242 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6802", 2024-09-13T13:14:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:14:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6804", 2024-09-13T13:14:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:14:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6803", 2024-09-13T13:14:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:14:25.244 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6801/542298937", 2024-09-13T13:14:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6802/542298937", 2024-09-13T13:14:25.245 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6804/542298937", 2024-09-13T13:14:25.245 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6803/542298937", 2024-09-13T13:14:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 1, 2024-09-13T13:14:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "b40a3574-3c6f-4810-a651-8d64aee6801b", 2024-09-13T13:14:25.247 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 13, 2024-09-13T13:14:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 334, 2024-09-13T13:14:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.248 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6805", 2024-09-13T13:14:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:14:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6806", 2024-09-13T13:14:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:14:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.249 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6808", 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.250 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6807", 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6805/26289431", 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6806/26289431", 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6808/26289431", 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6807/26289431", 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 2, 2024-09-13T13:14:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "1e8e49a0-ccf8-4b7f-839b-104b64601408", 2024-09-13T13:14:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 18, 2024-09-13T13:14:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 334, 2024-09-13T13:14:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:25.252 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6809", 2024-09-13T13:14:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:14:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.253 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6810", 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6812", 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6811", 2024-09-13T13:14:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:14:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6809/3960240393", 2024-09-13T13:14:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6810/3960240393", 2024-09-13T13:14:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6812/3960240393", 2024-09-13T13:14:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6811/3960240393", 2024-09-13T13:14:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 3, 2024-09-13T13:14:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "9f6f1d22-be68-4d86-9d4b-86fd9e87d141", 2024-09-13T13:14:25.256 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 27, 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 309, 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.257 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6813", 2024-09-13T13:14:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:14:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6814", 2024-09-13T13:14:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:14:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6816", 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.259 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6815", 2024-09-13T13:14:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:14:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6813/2025522885", 2024-09-13T13:14:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6814/2025522885", 2024-09-13T13:14:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6816/2025522885", 2024-09-13T13:14:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6815/2025522885", 2024-09-13T13:14:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 4, 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "1326d850-caf4-484e-90a1-052f54f9f306", 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 33, 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 334, 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6800", 2024-09-13T13:14:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:14:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6801", 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6803", 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6802", 2024-09-13T13:14:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:14:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6800/2137138858", 2024-09-13T13:14:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6801/2137138858", 2024-09-13T13:14:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6803/2137138858", 2024-09-13T13:14:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6802/2137138858", 2024-09-13T13:14:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 5, 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2", 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 40, 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 334, 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6804", 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6805", 2024-09-13T13:14:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:14:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6807", 2024-09-13T13:14:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:14:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6806", 2024-09-13T13:14:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:14:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6804/2906884337", 2024-09-13T13:14:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6805/2906884337", 2024-09-13T13:14:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6807/2906884337", 2024-09-13T13:14:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6806/2906884337", 2024-09-13T13:14:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 6, 2024-09-13T13:14:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "2471907d-fcbd-44e0-9d29-54151ada9df0", 2024-09-13T13:14:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 48, 2024-09-13T13:14:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 102, 2024-09-13T13:14:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 118, 2024-09-13T13:14:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 334, 2024-09-13T13:14:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 103, 2024-09-13T13:14:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6808", 2024-09-13T13:14:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:14:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6809", 2024-09-13T13:14:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:14:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6811", 2024-09-13T13:14:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:14:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6810", 2024-09-13T13:14:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:14:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6808/143093129", 2024-09-13T13:14:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6809/143093129", 2024-09-13T13:14:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6811/143093129", 2024-09-13T13:14:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6810/143093129", 2024-09-13T13:14:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 7, 2024-09-13T13:14:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "97d2c6d7-7989-4c11-91e0-32ee87e76c6d", 2024-09-13T13:14:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 0.31549072265625, 2024-09-13T13:14:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 55, 2024-09-13T13:14:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 305, 2024-09-13T13:14:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6812", 2024-09-13T13:14:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:14:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6813", 2024-09-13T13:14:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:14:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6815", 2024-09-13T13:14:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:14:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6814", 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6812/403155155", 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6813/403155155", 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6815/403155155", 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6814/403155155", 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:14:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_xinfo": [ 2024-09-13T13:14:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 0, 2024-09-13T13:14:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:13:31.043416+0000", 2024-09-13T13:14:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 1, 2024-09-13T13:14:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:13:14.043919+0000", 2024-09-13T13:14:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 2, 2024-09-13T13:14:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:13:03.087185+0000", 2024-09-13T13:14:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 3, 2024-09-13T13:14:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:13:38.605406+0000", 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 4, 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:14:08.260279+0000", 2024-09-13T13:14:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 5, 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:13:38.414252+0000", 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 6, 2024-09-13T13:14:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:14:02.629672+0000", 2024-09-13T13:14:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 102 2024-09-13T13:14:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 7, 2024-09-13T13:14:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:13:42.542582+0000", 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_upmap": [], 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_upmap_items": [ 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.10", 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 1, 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 2 2024-09-13T13:14:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.12", 2024-09-13T13:14:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:14:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 1, 2024-09-13T13:14:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 5 2024-09-13T13:14:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.15", 2024-09-13T13:14:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:14:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 3, 2024-09-13T13:14:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 2 2024-09-13T13:14:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.17", 2024-09-13T13:14:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:14:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 3, 2024-09-13T13:14:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 5 2024-09-13T13:14:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.1f", 2024-09-13T13:14:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:14:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 1, 2024-09-13T13:14:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 2 2024-09-13T13:14:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:14:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_upmap_primaries": [], 2024-09-13T13:14:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_temp": [], 2024-09-13T13:14:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_temp": [], 2024-09-13T13:14:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: "blocklist": { 2024-09-13T13:14:25.293 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/2104786675": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:14:25.293 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:6800/910649698": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:14:25.293 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1471436716": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:14:25.293 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/4151877300": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:14:25.293 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/3080924817": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:14:25.293 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/4248579079": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:14:25.294 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/455730280": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:14:25.294 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1887735845": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:14:25.294 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:6800/1437783219": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:14:25.294 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:6800/326637643": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:14:25.294 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/991228238": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:14:25.294 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/3415403690": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:14:25.295 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1933060029": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:14:25.295 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1009267179": "2024-09-14T13:01:31.849627+0000" 2024-09-13T13:14:25.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.295 INFO:teuthology.orchestra.run.smithi052.stdout: "range_blocklist": {}, 2024-09-13T13:14:25.295 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profiles": { 2024-09-13T13:14:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: "default": { 2024-09-13T13:14:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: "crush-failure-domain": "osd", 2024-09-13T13:14:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: "k": "2", 2024-09-13T13:14:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: "m": "1", 2024-09-13T13:14:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: "plugin": "jerasure", 2024-09-13T13:14:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: "technique": "reed_sol_van" 2024-09-13T13:14:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps_queue": [], 2024-09-13T13:14:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: "new_removed_snaps": [], 2024-09-13T13:14:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: "new_purged_snaps": [], 2024-09-13T13:14:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_node_flags": {}, 2024-09-13T13:14:25.298 INFO:teuthology.orchestra.run.smithi052.stdout: "device_class_flags": {}, 2024-09-13T13:14:25.298 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_mode": { 2024-09-13T13:14:25.298 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_mode_enabled": false, 2024-09-13T13:14:25.298 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_bucket_count": 0, 2024-09-13T13:14:25.298 INFO:teuthology.orchestra.run.smithi052.stdout: "degraded_stretch_mode": 0, 2024-09-13T13:14:25.299 INFO:teuthology.orchestra.run.smithi052.stdout: "recovering_stretch_mode": 0, 2024-09-13T13:14:25.299 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_mode_bucket": 0 2024-09-13T13:14:25.299 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:25.299 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:14:25.301 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 21 16 42674 42658 126.946 85.4375 0.00971336 0.00787177 2024-09-13T13:14:25.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:25.302 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:25.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:25.304 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T13:14:25.404 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:25.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:25.545 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:25.596 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:25.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:25.638 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:14:25.698 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:25.748 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:25.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:25 smithi167 ceph-mon[33178]: 6.7 scrub starts 2024-09-13T13:14:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:25 smithi167 ceph-mon[33178]: 6.7 scrub ok 2024-09-13T13:14:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:25 smithi167 ceph-mon[33178]: 6.c scrub ok 2024-09-13T13:14:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:25 smithi167 ceph-mon[33178]: pgmap v661: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-13T13:14:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:25 smithi167 ceph-mon[33178]: 1.0 scrub starts 2024-09-13T13:14:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:25 smithi167 ceph-mon[33178]: 1.0 scrub ok 2024-09-13T13:14:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:25 smithi167 ceph-mon[33178]: 3.4 scrub starts 2024-09-13T13:14:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:25 smithi167 ceph-mon[33178]: 3.4 scrub ok 2024-09-13T13:14:25.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:25 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/891813601' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-13T13:14:25.849 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:25.891 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:25.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:25.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[36682]: 6.7 scrub starts 2024-09-13T13:14:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[36682]: 6.7 scrub ok 2024-09-13T13:14:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[36682]: 6.c scrub ok 2024-09-13T13:14:25.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[36682]: pgmap v661: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-13T13:14:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[36682]: 1.0 scrub starts 2024-09-13T13:14:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[36682]: 1.0 scrub ok 2024-09-13T13:14:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[36682]: 3.4 scrub starts 2024-09-13T13:14:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[36682]: 3.4 scrub ok 2024-09-13T13:14:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/891813601' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-13T13:14:25.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[29471]: 6.7 scrub starts 2024-09-13T13:14:25.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[29471]: 6.7 scrub ok 2024-09-13T13:14:25.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[29471]: 6.c scrub ok 2024-09-13T13:14:25.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[29471]: pgmap v661: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-13T13:14:25.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:25.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[29471]: 1.0 scrub starts 2024-09-13T13:14:25.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[29471]: 1.0 scrub ok 2024-09-13T13:14:25.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[29471]: 3.4 scrub starts 2024-09-13T13:14:25.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[29471]: 3.4 scrub ok 2024-09-13T13:14:25.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:25 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/891813601' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-13T13:14:25.993 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:26.026 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:26.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:26.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:26.076 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:14:26.129 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:26.162 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:26.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:26.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:26.296 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 22 15 44423 44408 126.147 109.375 0.0174159 0.00792159 2024-09-13T13:14:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:26.298 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:26.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:26.376 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:14:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:14:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:14:26.379 INFO:tasks.thrashosds.thrasher:j is {'epoch': 344, 'fsid': '2b415b38-71cf-11ef-bceb-c7b262605968', 'created': '2024-09-13T12:55:27.893882+0000', 'modified': '2024-09-13T13:14:22.957095+0000', 'last_up_change': '2024-09-13T13:04:02.281376+0000', 'last_in_change': '2024-09-13T13:12:52.437622+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T12:58:31.271893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '277', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-13T13:01:00.306747+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-13T13:01:01.103523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-13T13:01:02.249986+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-13T13:01:04.203764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-13T13:01:06.293676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-13T13:13:57.824428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '338', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}], 'osds': [{'osd': 0, 'uuid': 'd52dca1a-94c1-4e9e-93f5-687c038cd7c7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6801', 'nonce': 542298937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6802', 'nonce': 542298937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6804', 'nonce': 542298937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6803', 'nonce': 542298937}]}, 'public_addr': '172.21.15.52:6801/542298937', 'cluster_addr': '172.21.15.52:6802/542298937', 'heartbeat_back_addr': '172.21.15.52:6804/542298937', 'heartbeat_front_addr': '172.21.15.52:6803/542298937', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b40a3574-3c6f-4810-a651-8d64aee6801b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6805', 'nonce': 26289431}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6806', 'nonce': 26289431}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6808', 'nonce': 26289431}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6807', 'nonce': 26289431}]}, 'public_addr': '172.21.15.52:6805/26289431', 'cluster_addr': '172.21.15.52:6806/26289431', 'heartbeat_back_addr': '172.21.15.52:6808/26289431', 'heartbeat_front_addr': '172.21.15.52:6807/26289431', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1e8e49a0-ccf8-4b7f-839b-104b64601408', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6809', 'nonce': 3960240393}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6810', 'nonce': 3960240393}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6812', 'nonce': 3960240393}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6811', 'nonce': 3960240393}]}, 'public_addr': '172.21.15.52:6809/3960240393', 'cluster_addr': '172.21.15.52:6810/3960240393', 'heartbeat_back_addr': '172.21.15.52:6812/3960240393', 'heartbeat_front_addr': '172.21.15.52:6811/3960240393', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9f6f1d22-be68-4d86-9d4b-86fd9e87d141', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 309, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6813', 'nonce': 2025522885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6814', 'nonce': 2025522885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6816', 'nonce': 2025522885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6815', 'nonce': 2025522885}]}, 'public_addr': '172.21.15.52:6813/2025522885', 'cluster_addr': '172.21.15.52:6814/2025522885', 'heartbeat_back_addr': '172.21.15.52:6816/2025522885', 'heartbeat_front_addr': '172.21.15.52:6815/2025522885', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1326d850-caf4-484e-90a1-052f54f9f306', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6800', 'nonce': 2137138858}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 2137138858}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 2137138858}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6802', 'nonce': 2137138858}]}, 'public_addr': '172.21.15.167:6800/2137138858', 'cluster_addr': '172.21.15.167:6801/2137138858', 'heartbeat_back_addr': '172.21.15.167:6803/2137138858', 'heartbeat_front_addr': '172.21.15.167:6802/2137138858', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6804', 'nonce': 2906884337}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 2906884337}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 2906884337}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6806', 'nonce': 2906884337}]}, 'public_addr': '172.21.15.167:6804/2906884337', 'cluster_addr': '172.21.15.167:6805/2906884337', 'heartbeat_back_addr': '172.21.15.167:6807/2906884337', 'heartbeat_front_addr': '172.21.15.167:6806/2906884337', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2471907d-fcbd-44e0-9d29-54151ada9df0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 48, 'last_clean_end': 102, 'up_from': 118, 'up_thru': 334, 'down_at': 103, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6808', 'nonce': 143093129}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 143093129}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 143093129}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6810', 'nonce': 143093129}]}, 'public_addr': '172.21.15.167:6808/143093129', 'cluster_addr': '172.21.15.167:6809/143093129', 'heartbeat_back_addr': '172.21.15.167:6811/143093129', 'heartbeat_front_addr': '172.21.15.167:6810/143093129', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '97d2c6d7-7989-4c11-91e0-32ee87e76c6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.31549072265625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 305, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6812', 'nonce': 403155155}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 403155155}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 403155155}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6814', 'nonce': 403155155}]}, 'public_addr': '172.21.15.167:6812/403155155', 'cluster_addr': '172.21.15.167:6813/403155155', 'heartbeat_back_addr': '172.21.15.167:6815/403155155', 'heartbeat_front_addr': '172.21.15.167:6814/403155155', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:13:31.043416+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:13:14.043919+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:13:03.087185+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:13:38.605406+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:14:08.260279+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:13:38.414252+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:14:02.629672+0000', 'dead_epoch': 102}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:13:42.542582+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.15', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.52:0/2104786675': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:6800/910649698': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:0/1471436716': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:0/4151877300': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:0/3080924817': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:0/4248579079': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:0/455730280': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/1887735845': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:6800/1437783219': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:6800/326637643': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/991228238': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/3415403690': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/1933060029': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:0/1009267179': '2024-09-14T13:01:31.849627+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T13:14:26.379 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-13T13:14:26.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:26.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:26.427 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:26.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:26.527 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:26.528 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:26.582 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:26.601 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:26.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:26.604 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:26.703 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:26.744 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:26.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:26.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:26.846 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:26.888 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:26.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:26.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:26.991 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:27.030 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:27.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:27.132 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:27.156 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:27.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:27.257 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:27.296 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:27.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:27.298 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 23 16 46021 46005 125.002 99.8125 0.00454742 0.00799424 2024-09-13T13:14:27.398 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:27.437 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:27.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:27.538 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:27.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:27.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:27.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:27.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:27.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:27.761 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:14:27.762 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:14:27.805 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:27.808 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:14:27.809 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:14:27.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:27.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:27.846 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:14:27.847 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:14:27.847 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:27 smithi052 ceph-mon[36682]: pgmap v662: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-13T13:14:27.847 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:27 smithi052 ceph-mon[36682]: 3.9 scrub starts 2024-09-13T13:14:27.847 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:27 smithi052 ceph-mon[36682]: 3.9 scrub ok 2024-09-13T13:14:27.848 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:27 smithi052 ceph-mon[36682]: 5.3 scrub starts 2024-09-13T13:14:27.848 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:27 smithi052 ceph-mon[36682]: 5.3 scrub ok 2024-09-13T13:14:27.848 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:27 smithi052 ceph-mon[29471]: pgmap v662: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-13T13:14:27.848 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:27 smithi052 ceph-mon[29471]: 3.9 scrub starts 2024-09-13T13:14:27.848 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:27 smithi052 ceph-mon[29471]: 3.9 scrub ok 2024-09-13T13:14:27.848 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:27 smithi052 ceph-mon[29471]: 5.3 scrub starts 2024-09-13T13:14:27.848 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:27 smithi052 ceph-mon[29471]: 5.3 scrub ok 2024-09-13T13:14:27.883 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:14:27.883 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:14:27.924 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:14:27.924 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:14:27.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:27.966 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:14:27.966 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:14:27.969 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:27.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:27.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:28.008 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:14:28.009 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:14:28.044 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:14:28.044 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:14:28.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:27 smithi167 ceph-mon[33178]: pgmap v662: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-13T13:14:28.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:27 smithi167 ceph-mon[33178]: 3.9 scrub starts 2024-09-13T13:14:28.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:27 smithi167 ceph-mon[33178]: 3.9 scrub ok 2024-09-13T13:14:28.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:27 smithi167 ceph-mon[33178]: 5.3 scrub starts 2024-09-13T13:14:28.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:27 smithi167 ceph-mon[33178]: 5.3 scrub ok 2024-09-13T13:14:28.071 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:28.120 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:28.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:28.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:28.140 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:14:28.141 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:14:28.222 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:28.259 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:28.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:28.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:28.296 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 24 16 47648 47632 124.03 101.688 0.00825563 0.00805769 2024-09-13T13:14:28.362 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:28.401 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:28.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:28.506 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:28.542 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:28.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:28.599 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:14:28.644 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:28.683 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:28.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:28.785 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:28.823 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:28.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:28.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:28.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:28.963 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:28.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:28.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:28.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:28 smithi052 ceph-mon[29471]: 4.19 scrub starts 2024-09-13T13:14:28.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:28 smithi052 ceph-mon[29471]: 4.19 scrub ok 2024-09-13T13:14:28.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:28 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2311559909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:28.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:28 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4273175630' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:14:28.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:28 smithi052 ceph-mon[29471]: from='client.36155 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:14:28.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:28 smithi052 ceph-mon[36682]: 4.19 scrub starts 2024-09-13T13:14:28.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:28 smithi052 ceph-mon[36682]: 4.19 scrub ok 2024-09-13T13:14:28.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:28 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2311559909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:28.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:28 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4273175630' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:14:28.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:28 smithi052 ceph-mon[36682]: from='client.36155 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:14:29.066 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:29.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:28 smithi167 ceph-mon[33178]: 4.19 scrub starts 2024-09-13T13:14:29.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:28 smithi167 ceph-mon[33178]: 4.19 scrub ok 2024-09-13T13:14:29.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:28 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2311559909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:29.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:28 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4273175630' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:14:29.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:28 smithi167 ceph-mon[33178]: from='client.36155 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:14:29.103 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:29.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:29.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:29.168 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T13:14:29.205 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:29.256 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:29.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:29.297 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 25 16 49462 49446 123.599 113.375 0.0131785 0.0080851 2024-09-13T13:14:29.358 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:29.389 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:29.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:29.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:29.491 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:29.536 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:29.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:29.637 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:29.672 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:29.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:29 smithi052 ceph-mon[36682]: 3.5 scrub starts 2024-09-13T13:14:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:29 smithi052 ceph-mon[36682]: 3.5 scrub ok 2024-09-13T13:14:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:29 smithi052 ceph-mon[36682]: pgmap v663: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-09-13T13:14:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:29 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:29 smithi052 ceph-mon[36682]: from='client.36155 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:14:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:29 smithi052 ceph-mon[36682]: osdmap e345: 8 total, 8 up, 8 in 2024-09-13T13:14:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:29 smithi052 ceph-mon[29471]: 3.5 scrub starts 2024-09-13T13:14:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:29 smithi052 ceph-mon[29471]: 3.5 scrub ok 2024-09-13T13:14:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:29 smithi052 ceph-mon[29471]: pgmap v663: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-09-13T13:14:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:29 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:29 smithi052 ceph-mon[29471]: from='client.36155 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:14:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:29 smithi052 ceph-mon[29471]: osdmap e345: 8 total, 8 up, 8 in 2024-09-13T13:14:29.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:29.822 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:29.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:29.924 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:29.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:29.960 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:29.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:30.061 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:29 smithi167 ceph-mon[33178]: 3.5 scrub starts 2024-09-13T13:14:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:29 smithi167 ceph-mon[33178]: 3.5 scrub ok 2024-09-13T13:14:30.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:29 smithi167 ceph-mon[33178]: pgmap v663: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-09-13T13:14:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:29 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:29 smithi167 ceph-mon[33178]: from='client.36155 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:14:30.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:29 smithi167 ceph-mon[33178]: osdmap e345: 8 total, 8 up, 8 in 2024-09-13T13:14:30.099 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:30.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:30.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:30.238 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:30.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:30.297 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 26 15 51201 51186 123.027 108.75 0.00763626 0.00812257 2024-09-13T13:14:30.339 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:30.379 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:30.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:30.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:30.481 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:30.495 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:30.516 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:30.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:30.618 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:30.654 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:30.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:30.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:30 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:14:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:30 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:14:30.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:30 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:14:30.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:30 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:14:30.757 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:30.797 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:30.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:30.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:30.902 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:30.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:30.940 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:30.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:31.041 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:30 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:14:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:30 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:14:31.079 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:31.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:31.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:31.181 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:31.216 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:31.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:31.297 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 27 16 52987 52971 122.602 111.562 0.0110688 0.00815112 2024-09-13T13:14:31.319 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:31.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:31.358 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:31.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:31.380 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:14:31.380 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:14:31.380 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-13T13:14:31.381 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-13T13:14:31.459 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:31.510 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:31.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:31.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:31.637 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:31.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:31.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:31.731 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:31 smithi052 ceph-mon[36682]: pgmap v665: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-13T13:14:31.731 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:31 smithi052 ceph-mon[36682]: osdmap e346: 8 total, 8 up, 8 in 2024-09-13T13:14:31.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:31 smithi052 ceph-mon[29471]: pgmap v665: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-13T13:14:31.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:31 smithi052 ceph-mon[29471]: osdmap e346: 8 total, 8 up, 8 in 2024-09-13T13:14:31.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:31.781 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:31.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:31.807 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:14:31.807 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:14:31.850 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:14:31.882 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:31.941 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:31.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:32.043 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:32.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:31 smithi167 ceph-mon[33178]: pgmap v665: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-13T13:14:32.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:31 smithi167 ceph-mon[33178]: osdmap e346: 8 total, 8 up, 8 in 2024-09-13T13:14:32.074 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:32.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:32.176 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:32.225 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:32.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:32.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:32.301 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 28 16 54501 54485 121.603 94.625 0.00751618 0.00821795 2024-09-13T13:14:32.327 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:32.367 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:32.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:32.468 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:14:32.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:32.500 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:32.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:32.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:32.602 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:32.628 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:32.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:32.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:32.630 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:32.673 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T13:14:32.731 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:32.791 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:32.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:32.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:32.893 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:32.942 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:32.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:33.044 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:33.089 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:33.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:33.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:33.228 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:33.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:33.298 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 29 15 55977 55962 120.593 92.3125 0.00986908 0.00828743 2024-09-13T13:14:33.329 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:33.366 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:33.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:33.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:33.468 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:33.506 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:33.515 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:33.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:33.584 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:33.618 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:33.668 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:33.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:33.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:33 smithi052 ceph-mon[29471]: pgmap v667: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-13T13:14:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:33 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:14:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:33 smithi052 ceph-mon[36682]: pgmap v667: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-13T13:14:33.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:33 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:14:33.823 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:33.862 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:33.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:33.953 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:14:33.955 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:14:33.955 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 346, 2024-09-13T13:14:33.956 INFO:teuthology.orchestra.run.smithi052.stdout: "fsid": "2b415b38-71cf-11ef-bceb-c7b262605968", 2024-09-13T13:14:33.956 INFO:teuthology.orchestra.run.smithi052.stdout: "created": "2024-09-13T12:55:27.893882+0000", 2024-09-13T13:14:33.956 INFO:teuthology.orchestra.run.smithi052.stdout: "modified": "2024-09-13T13:14:30.590419+0000", 2024-09-13T13:14:33.956 INFO:teuthology.orchestra.run.smithi052.stdout: "last_up_change": "2024-09-13T13:04:02.281376+0000", 2024-09-13T13:14:33.956 INFO:teuthology.orchestra.run.smithi052.stdout: "last_in_change": "2024-09-13T13:12:52.437622+0000", 2024-09-13T13:14:33.956 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T13:14:33.956 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_num": 5801984, 2024-09-13T13:14:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_set": [ 2024-09-13T13:14:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: "noscrub", 2024-09-13T13:14:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: "pglog_hardlimit", 2024-09-13T13:14:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: "purged_snapdirs", 2024-09-13T13:14:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: "recovery_deletes", 2024-09-13T13:14:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: "sortbitwise" 2024-09-13T13:14:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:14:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_version": 19, 2024-09-13T13:14:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T13:14:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T13:14:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T13:14:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_snapshot": "", 2024-09-13T13:14:33.957 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_max": 10, 2024-09-13T13:14:33.958 INFO:teuthology.orchestra.run.smithi052.stdout: "max_osd": 8, 2024-09-13T13:14:33.958 INFO:teuthology.orchestra.run.smithi052.stdout: "require_min_compat_client": "luminous", 2024-09-13T13:14:33.958 INFO:teuthology.orchestra.run.smithi052.stdout: "min_compat_client": "luminous", 2024-09-13T13:14:33.958 INFO:teuthology.orchestra.run.smithi052.stdout: "require_osd_release": "squid", 2024-09-13T13:14:33.958 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_crimson": false, 2024-09-13T13:14:33.958 INFO:teuthology.orchestra.run.smithi052.stdout: "pools": [ 2024-09-13T13:14:33.958 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:33.958 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 1, 2024-09-13T13:14:33.959 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": ".mgr", 2024-09-13T13:14:33.959 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T12:58:31.271893+0000", 2024-09-13T13:14:33.959 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:14:33.959 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:14:33.959 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:33.959 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:33.960 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:33.960 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:33.960 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:33.960 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:33.960 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:33.961 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:33.961 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:33.961 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:33.961 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:33.962 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 1, 2024-09-13T13:14:33.962 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 1, 2024-09-13T13:14:33.962 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 1, 2024-09-13T13:14:33.962 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 1, 2024-09-13T13:14:33.962 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 1, 2024-09-13T13:14:33.963 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:33.963 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:33.963 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:33.963 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:33.963 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:33.963 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:33.964 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:33.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.964 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "277", 2024-09-13T13:14:33.964 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:33.964 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:33.965 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:33.965 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:33.965 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:33.965 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:14:33.965 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:14:33.965 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:33.966 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:33.966 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:33.966 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:33.966 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:33.966 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:33.967 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:33.967 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:33.967 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:33.967 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:33.967 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:33.967 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:33.968 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:33.968 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:33.968 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:33.968 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:33.968 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:33.969 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:33.969 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:33.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.969 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:33.969 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:33.969 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:33.970 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:33.970 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:33.970 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:33.970 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:33.970 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:33.971 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:33.971 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:33.971 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:33.971 INFO:teuthology.orchestra.run.smithi052.stdout: "options": { 2024-09-13T13:14:33.971 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_max": 32, 2024-09-13T13:14:33.971 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_min": 1 2024-09-13T13:14:33.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.972 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:33.972 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr": {} 2024-09-13T13:14:33.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.972 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:33.973 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:33.973 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 7.8899998664855957, 2024-09-13T13:14:33.973 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 7.8899998664855957, 2024-09-13T13:14:33.973 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 0.37999999523162842, 2024-09-13T13:14:33.973 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 3, 2024-09-13T13:14:33.973 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 3, 2024-09-13T13:14:33.974 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:33.974 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:33.974 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:33.974 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:33.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.974 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:33.975 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 2, 2024-09-13T13:14:33.975 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "datapool", 2024-09-13T13:14:33.975 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:00.306747+0000", 2024-09-13T13:14:33.975 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 8193, 2024-09-13T13:14:33.975 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T13:14:33.975 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:33.976 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:33.976 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:33.976 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:33.976 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:33.976 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:33.976 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:33.976 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:33.977 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:33.977 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:33.977 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:33.977 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 3, 2024-09-13T13:14:33.977 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 3, 2024-09-13T13:14:33.977 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 3, 2024-09-13T13:14:33.977 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 3, 2024-09-13T13:14:33.977 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 3, 2024-09-13T13:14:33.977 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:33.977 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:33.978 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:33.978 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:33.978 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:33.978 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:33.978 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:33.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.978 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "63", 2024-09-13T13:14:33.978 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:33.978 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:33.978 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:33.978 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:33.978 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:33.978 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 2, 2024-09-13T13:14:33.979 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 63, 2024-09-13T13:14:33.979 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:33.979 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:33.979 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:33.979 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:33.979 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:33.979 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:33.979 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:33.979 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:33.979 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:33.980 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:33.980 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:33.980 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:33.980 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:33.980 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:33.981 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:33.981 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:33.981 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:33.981 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:33.981 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:33.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.981 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:33.982 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:33.982 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:33.982 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:33.982 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:33.982 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:33.982 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:33.982 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:33.982 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:33.983 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:33.983 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:33.983 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:14:33.983 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:33.983 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd": {} 2024-09-13T13:14:33.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.983 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:33.983 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:33.983 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 2.6500000953674316, 2024-09-13T13:14:33.983 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 2.6500000953674316, 2024-09-13T13:14:33.983 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T13:14:33.983 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-13T13:14:33.983 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-13T13:14:33.984 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:33.984 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:33.984 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:33.984 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:33.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:33.984 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 3, 2024-09-13T13:14:33.984 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": ".rgw.root", 2024-09-13T13:14:33.984 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:01.103523+0000", 2024-09-13T13:14:33.985 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:14:33.985 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:14:33.985 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:33.985 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:33.985 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:33.985 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:33.985 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:33.985 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:33.985 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:33.985 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:33.986 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:33.987 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:33.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.987 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "61", 2024-09-13T13:14:33.987 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:33.987 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:33.987 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:33.987 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:33.987 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:33.987 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:14:33.987 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:14:33.987 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:33.987 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:33.987 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:33.988 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:33.988 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:33.988 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:33.988 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:33.988 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:33.988 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:33.988 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:33.988 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:33.988 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:33.988 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:33.989 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:33.989 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:33.989 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:33.989 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:33.989 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:33.989 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:33.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.989 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:33.990 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:33.990 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:33.990 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:33.990 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:33.990 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:33.990 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:33.990 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:33.990 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:33.990 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:33.991 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:33.991 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:14:33.991 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:33.991 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:14:33.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.991 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:33.991 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:33.991 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 1.75, 2024-09-13T13:14:33.992 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 1.75, 2024-09-13T13:14:33.992 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:14:33.992 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 1.75, 2024-09-13T13:14:33.992 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 1.75, 2024-09-13T13:14:33.992 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:33.992 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:33.992 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:33.992 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:33.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:33.993 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 4, 2024-09-13T13:14:33.993 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "default.rgw.log", 2024-09-13T13:14:33.993 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:02.249986+0000", 2024-09-13T13:14:33.993 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:14:33.993 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:14:33.994 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:33.994 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:33.994 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:33.994 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:33.994 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:33.994 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:33.994 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:33.994 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:33.995 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:33.995 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:33.995 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:33.995 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:14:33.995 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:14:33.995 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:14:33.995 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:14:33.996 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:14:33.996 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:33.996 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:33.996 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:33.996 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:33.996 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:33.996 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:33.996 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:33.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:33.997 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "63", 2024-09-13T13:14:33.997 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:33.997 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:33.997 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:33.997 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:33.997 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:33.997 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:14:33.998 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:14:33.998 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:33.998 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:33.998 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:33.998 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:33.998 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:33.998 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:33.998 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:33.999 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:33.999 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:33.999 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:33.999 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:33.999 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:33.999 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:33.999 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:34.000 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:34.000 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:34.000 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:34.000 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:34.000 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:34.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.000 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:34.000 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:34.001 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:34.001 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:34.001 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:34.001 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:34.001 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:34.001 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:34.001 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:34.001 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:34.001 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:34.002 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:14:34.002 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:34.002 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:14:34.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.002 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:34.002 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:34.002 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 2.25, 2024-09-13T13:14:34.002 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 2.25, 2024-09-13T13:14:34.003 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:14:34.003 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 2.25, 2024-09-13T13:14:34.003 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 2.25, 2024-09-13T13:14:34.003 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:34.003 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:34.003 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:34.003 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.004 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.004 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 5, 2024-09-13T13:14:34.004 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "default.rgw.control", 2024-09-13T13:14:34.004 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:04.203764+0000", 2024-09-13T13:14:34.004 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:14:34.004 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:14:34.004 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:34.005 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:34.005 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:34.005 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:34.005 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:34.005 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:34.005 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:14:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:14:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:14:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:14:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:14:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "65", 2024-09-13T13:14:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:14:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:14:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:34.009 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:34.009 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:34.009 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:34.009 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:34.009 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:34.009 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:34.009 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:34.009 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:34.010 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:34.010 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:34.010 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:34.010 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:34.010 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:34.010 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:34.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.010 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:34.011 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:34.011 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:34.011 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:34.011 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:34.011 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:34.011 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:34.011 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:34.011 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:34.011 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:34.012 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:34.012 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:14:34.012 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:34.012 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:14:34.012 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.012 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:34.012 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:34.012 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 1.5, 2024-09-13T13:14:34.013 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 1.5, 2024-09-13T13:14:34.013 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:14:34.013 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 1.5, 2024-09-13T13:14:34.013 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:34.015 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 1.5, 2024-09-13T13:14:34.015 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:34.015 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:34.015 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:34.016 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.016 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 6, 2024-09-13T13:14:34.016 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "default.rgw.meta", 2024-09-13T13:14:34.017 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:01:06.293676+0000", 2024-09-13T13:14:34.017 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:14:34.017 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:14:34.017 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:34.017 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:34.017 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:34.017 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:34.017 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:34.018 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:34.018 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:34.018 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:34.018 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:34.018 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:34.018 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:34.018 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 32, 2024-09-13T13:14:34.019 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 32, 2024-09-13T13:14:34.019 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 32, 2024-09-13T13:14:34.019 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 32, 2024-09-13T13:14:34.019 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 32, 2024-09-13T13:14:34.019 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:34.019 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:34.020 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:34.020 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:34.020 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:34.020 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:34.020 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:34.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.021 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "67", 2024-09-13T13:14:34.021 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:34.021 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:34.021 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:34.021 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:34.022 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:34.022 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:14:34.022 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:14:34.022 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:34.022 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:34.022 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:34.022 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:34.023 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:34.023 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:34.023 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:34.023 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:34.023 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:34.023 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:34.024 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:34.024 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:34.024 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:34.024 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:34.024 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:34.024 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:34.024 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:34.024 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:34.024 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:34.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.025 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:34.025 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:34.025 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:34.025 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:34.025 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:34.025 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:34.025 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:34.025 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:34.025 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:34.025 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:34.025 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:34.025 INFO:teuthology.orchestra.run.smithi052.stdout: "options": { 2024-09-13T13:14:34.026 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_bias": 4 2024-09-13T13:14:34.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.026 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:34.026 INFO:teuthology.orchestra.run.smithi052.stdout: "rgw": {} 2024-09-13T13:14:34.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.026 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:34.026 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:34.026 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 1.75, 2024-09-13T13:14:34.026 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 1.75, 2024-09-13T13:14:34.026 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:14:34.026 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 1.75, 2024-09-13T13:14:34.027 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 1.75, 2024-09-13T13:14:34.027 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:34.027 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:34.027 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:34.027 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.027 INFO:teuthology.orchestra.run.smithi052.stdout: "pool": 10, 2024-09-13T13:14:34.027 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": "unique_pool_3", 2024-09-13T13:14:34.027 INFO:teuthology.orchestra.run.smithi052.stdout: "create_time": "2024-09-13T13:13:57.824428+0000", 2024-09-13T13:14:34.027 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-09-13T13:14:34.027 INFO:teuthology.orchestra.run.smithi052.stdout: "flags_names": "hashpspool", 2024-09-13T13:14:34.027 INFO:teuthology.orchestra.run.smithi052.stdout: "type": 1, 2024-09-13T13:14:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 3, 2024-09-13T13:14:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": 2, 2024-09-13T13:14:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_rule": 0, 2024-09-13T13:14:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:14:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:14:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:14:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:14:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "is_stretch_pool": false, 2024-09-13T13:14:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "object_hash": 2, 2024-09-13T13:14:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:14:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": 16, 2024-09-13T13:14:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num": 16, 2024-09-13T13:14:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_placement_num_target": 16, 2024-09-13T13:14:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_target": 16, 2024-09-13T13:14:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num_pending": 16, 2024-09-13T13:14:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "last_pg_merge_meta": { 2024-09-13T13:14:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "source_pgid": "0.0", 2024-09-13T13:14:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "ready_epoch": 0, 2024-09-13T13:14:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_started": 0, 2024-09-13T13:14:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "last_epoch_clean": 0, 2024-09-13T13:14:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "source_version": "0'0", 2024-09-13T13:14:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "target_version": "0'0" 2024-09-13T13:14:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "last_change": "338", 2024-09-13T13:14:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend": "0", 2024-09-13T13:14:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:14:34.030 INFO:teuthology.orchestra.run.smithi052.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:14:34.030 INFO:teuthology.orchestra.run.smithi052.stdout: "auid": 0, 2024-09-13T13:14:34.030 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:14:34.030 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_seq": 0, 2024-09-13T13:14:34.030 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_epoch": 0, 2024-09-13T13:14:34.030 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_snaps": [], 2024-09-13T13:14:34.030 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps": "[]", 2024-09-13T13:14:34.030 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_bytes": 0, 2024-09-13T13:14:34.030 INFO:teuthology.orchestra.run.smithi052.stdout: "quota_max_objects": 0, 2024-09-13T13:14:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "tiers": [], 2024-09-13T13:14:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_of": -1, 2024-09-13T13:14:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "read_tier": -1, 2024-09-13T13:14:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "write_tier": -1, 2024-09-13T13:14:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_mode": "none", 2024-09-13T13:14:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_bytes": 0, 2024-09-13T13:14:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "target_max_objects": 0, 2024-09-13T13:14:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:14:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:14:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:14:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_flush_age": 0, 2024-09-13T13:14:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "cache_min_evict_age": 0, 2024-09-13T13:14:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profile": "", 2024-09-13T13:14:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_params": { 2024-09-13T13:14:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "none" 2024-09-13T13:14:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_period": 0, 2024-09-13T13:14:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_count": 0, 2024-09-13T13:14:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "use_gmt_hitset": true, 2024-09-13T13:14:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:14:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:14:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:14:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:14:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "grade_table": [], 2024-09-13T13:14:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "stripe_width": 0, 2024-09-13T13:14:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "expected_num_objects": 0, 2024-09-13T13:14:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "fast_read": false, 2024-09-13T13:14:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "options": {}, 2024-09-13T13:14:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "application_metadata": { 2024-09-13T13:14:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "rados": {} 2024-09-13T13:14:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "read_balance": { 2024-09-13T13:14:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "score_type": "Fair distribution", 2024-09-13T13:14:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "score_acting": 3, 2024-09-13T13:14:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "score_stable": 3, 2024-09-13T13:14:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "optimal_score": 1, 2024-09-13T13:14:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_acting": 3, 2024-09-13T13:14:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "raw_score_stable": 3, 2024-09-13T13:14:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity_weighted": 0.91443634033203125, 2024-09-13T13:14:34.034 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-13T13:14:34.034 INFO:teuthology.orchestra.run.smithi052.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-13T13:14:34.034 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.034 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.034 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:14:34.034 INFO:teuthology.orchestra.run.smithi052.stdout: "osds": [ 2024-09-13T13:14:34.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.034 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 0, 2024-09-13T13:14:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "d52dca1a-94c1-4e9e-93f5-687c038cd7c7", 2024-09-13T13:14:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 8, 2024-09-13T13:14:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 334, 2024-09-13T13:14:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6801", 2024-09-13T13:14:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:14:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6802", 2024-09-13T13:14:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:14:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6804", 2024-09-13T13:14:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:14:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6803", 2024-09-13T13:14:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 542298937 2024-09-13T13:14:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6801/542298937", 2024-09-13T13:14:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6802/542298937", 2024-09-13T13:14:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6804/542298937", 2024-09-13T13:14:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6803/542298937", 2024-09-13T13:14:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 1, 2024-09-13T13:14:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "b40a3574-3c6f-4810-a651-8d64aee6801b", 2024-09-13T13:14:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 13, 2024-09-13T13:14:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 334, 2024-09-13T13:14:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6805", 2024-09-13T13:14:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:14:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6806", 2024-09-13T13:14:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:14:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6808", 2024-09-13T13:14:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:14:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6807", 2024-09-13T13:14:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 26289431 2024-09-13T13:14:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6805/26289431", 2024-09-13T13:14:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6806/26289431", 2024-09-13T13:14:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6808/26289431", 2024-09-13T13:14:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6807/26289431", 2024-09-13T13:14:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 2, 2024-09-13T13:14:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "1e8e49a0-ccf8-4b7f-839b-104b64601408", 2024-09-13T13:14:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 18, 2024-09-13T13:14:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 334, 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6809", 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6810", 2024-09-13T13:14:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:14:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6812", 2024-09-13T13:14:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:14:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6811", 2024-09-13T13:14:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3960240393 2024-09-13T13:14:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6809/3960240393", 2024-09-13T13:14:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6810/3960240393", 2024-09-13T13:14:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6812/3960240393", 2024-09-13T13:14:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6811/3960240393", 2024-09-13T13:14:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 3, 2024-09-13T13:14:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "9f6f1d22-be68-4d86-9d4b-86fd9e87d141", 2024-09-13T13:14:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 27, 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 309, 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6813", 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:14:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6814", 2024-09-13T13:14:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:14:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6816", 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6815", 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2025522885 2024-09-13T13:14:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.52:6813/2025522885", 2024-09-13T13:14:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.52:6814/2025522885", 2024-09-13T13:14:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.52:6816/2025522885", 2024-09-13T13:14:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.52:6815/2025522885", 2024-09-13T13:14:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 4, 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "1326d850-caf4-484e-90a1-052f54f9f306", 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 33, 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 334, 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6800", 2024-09-13T13:14:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:14:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6801", 2024-09-13T13:14:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:14:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6803", 2024-09-13T13:14:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:14:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6802", 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2137138858 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6800/2137138858", 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6801/2137138858", 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6803/2137138858", 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6802/2137138858", 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 5, 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2", 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 40, 2024-09-13T13:14:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 334, 2024-09-13T13:14:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6804", 2024-09-13T13:14:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:14:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6805", 2024-09-13T13:14:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:14:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6807", 2024-09-13T13:14:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:14:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6806", 2024-09-13T13:14:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2906884337 2024-09-13T13:14:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6804/2906884337", 2024-09-13T13:14:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6805/2906884337", 2024-09-13T13:14:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6807/2906884337", 2024-09-13T13:14:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6806/2906884337", 2024-09-13T13:14:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 6, 2024-09-13T13:14:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "2471907d-fcbd-44e0-9d29-54151ada9df0", 2024-09-13T13:14:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 1, 2024-09-13T13:14:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 48, 2024-09-13T13:14:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 102, 2024-09-13T13:14:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 118, 2024-09-13T13:14:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 334, 2024-09-13T13:14:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 103, 2024-09-13T13:14:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6808", 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6809", 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6811", 2024-09-13T13:14:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:14:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6810", 2024-09-13T13:14:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 143093129 2024-09-13T13:14:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6808/143093129", 2024-09-13T13:14:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6809/143093129", 2024-09-13T13:14:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6811/143093129", 2024-09-13T13:14:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6810/143093129", 2024-09-13T13:14:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 7, 2024-09-13T13:14:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "uuid": "97d2c6d7-7989-4c11-91e0-32ee87e76c6d", 2024-09-13T13:14:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "up": 1, 2024-09-13T13:14:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "in": 1, 2024-09-13T13:14:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "weight": 1, 2024-09-13T13:14:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_affinity": 0.31549072265625, 2024-09-13T13:14:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_begin": 0, 2024-09-13T13:14:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "last_clean_end": 0, 2024-09-13T13:14:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "up_from": 55, 2024-09-13T13:14:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "up_thru": 305, 2024-09-13T13:14:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "down_at": 0, 2024-09-13T13:14:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: "lost_at": 0, 2024-09-13T13:14:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addrs": { 2024-09-13T13:14:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6812", 2024-09-13T13:14:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:14:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addrs": { 2024-09-13T13:14:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6813", 2024-09-13T13:14:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:14:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addrs": { 2024-09-13T13:14:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6815", 2024-09-13T13:14:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:14:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addrs": { 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.167:6814", 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 403155155 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "public_addr": "172.21.15.167:6812/403155155", 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_addr": "172.21.15.167:6813/403155155", 2024-09-13T13:14:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_back_addr": "172.21.15.167:6815/403155155", 2024-09-13T13:14:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "heartbeat_front_addr": "172.21.15.167:6814/403155155", 2024-09-13T13:14:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "state": [ 2024-09-13T13:14:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "exists", 2024-09-13T13:14:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "up" 2024-09-13T13:14:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:14:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_xinfo": [ 2024-09-13T13:14:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 0, 2024-09-13T13:14:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:13:31.043416+0000", 2024-09-13T13:14:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 1, 2024-09-13T13:14:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:13:14.043919+0000", 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 2, 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:14:29.611152+0000", 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 3, 2024-09-13T13:14:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:13:38.605406+0000", 2024-09-13T13:14:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 4, 2024-09-13T13:14:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:14:08.260279+0000", 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 5, 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:13:38.414252+0000", 2024-09-13T13:14:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 6, 2024-09-13T13:14:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:14:02.629672+0000", 2024-09-13T13:14:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 102 2024-09-13T13:14:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "osd": 7, 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: "down_stamp": "0.000000", 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_probability": 0, 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: "laggy_interval": 0, 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: "features": 4540719139924082687, 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: "old_weight": 0, 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: "last_purged_snaps_scrub": "2024-09-13T13:13:42.542582+0000", 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: "dead_epoch": 0 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_upmap": [], 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_upmap_items": [ 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.10", 2024-09-13T13:14:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:14:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 1, 2024-09-13T13:14:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 2 2024-09-13T13:14:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.12", 2024-09-13T13:14:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:14:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 1, 2024-09-13T13:14:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 5 2024-09-13T13:14:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.15", 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 3, 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 2 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.17", 2024-09-13T13:14:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:14:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 3, 2024-09-13T13:14:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 5 2024-09-13T13:14:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: "pgid": "4.1f", 2024-09-13T13:14:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: "mappings": [ 2024-09-13T13:14:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: "from": 1, 2024-09-13T13:14:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: "to": 2 2024-09-13T13:14:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:14:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_upmap_primaries": [], 2024-09-13T13:14:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_temp": [], 2024-09-13T13:14:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: "primary_temp": [], 2024-09-13T13:14:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: "blocklist": { 2024-09-13T13:14:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/2104786675": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:14:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:6800/910649698": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:14:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1471436716": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:14:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/4151877300": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:14:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/3080924817": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:14:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/4248579079": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:14:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/455730280": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:14:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1887735845": "2024-09-14T12:55:49.232470+0000", 2024-09-13T13:14:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:6800/1437783219": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:14:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:6800/326637643": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:14:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/991228238": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:14:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/3415403690": "2024-09-14T13:01:31.849627+0000", 2024-09-13T13:14:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1933060029": "2024-09-14T12:56:14.802531+0000", 2024-09-13T13:14:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: "172.21.15.52:0/1009267179": "2024-09-14T13:01:31.849627+0000" 2024-09-13T13:14:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: "range_blocklist": {}, 2024-09-13T13:14:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: "erasure_code_profiles": { 2024-09-13T13:14:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: "default": { 2024-09-13T13:14:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "crush-failure-domain": "osd", 2024-09-13T13:14:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "k": "2", 2024-09-13T13:14:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "m": "1", 2024-09-13T13:14:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "plugin": "jerasure", 2024-09-13T13:14:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "technique": "reed_sol_van" 2024-09-13T13:14:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "removed_snaps_queue": [], 2024-09-13T13:14:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "new_removed_snaps": [], 2024-09-13T13:14:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "new_purged_snaps": [], 2024-09-13T13:14:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_node_flags": {}, 2024-09-13T13:14:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "device_class_flags": {}, 2024-09-13T13:14:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_mode": { 2024-09-13T13:14:34.084 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_mode_enabled": false, 2024-09-13T13:14:34.084 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_bucket_count": 0, 2024-09-13T13:14:34.084 INFO:teuthology.orchestra.run.smithi052.stdout: "degraded_stretch_mode": 0, 2024-09-13T13:14:34.084 INFO:teuthology.orchestra.run.smithi052.stdout: "recovering_stretch_mode": 0, 2024-09-13T13:14:34.084 INFO:teuthology.orchestra.run.smithi052.stdout: "stretch_mode_bucket": 0 2024-09-13T13:14:34.084 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:34.084 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:14:34.085 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:33 smithi167 ceph-mon[33178]: pgmap v667: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-13T13:14:34.085 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:33 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:14:34.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:34.122 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:34.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:34.224 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:34.263 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:34.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:34.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:34.297 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 30 16 57488 57472 119.719 94.375 0.00629885 0.00834784 2024-09-13T13:14:34.365 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:34.401 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:34.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:34.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:34.503 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:34.566 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:34.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:34.629 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:14:34 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:14:34.668 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:34.697 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:34.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:34.799 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:34.840 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:34.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:34.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:34.942 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:34.978 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:34.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:34.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:34.983 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:34 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2600546984' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-13T13:14:34.984 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:34 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2600546984' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-13T13:14:35.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:34 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2600546984' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-13T13:14:35.081 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:35.118 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:35.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:35.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:35.220 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:35.259 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:35.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:35.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:35.298 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 31 16 59075 59059 119.056 99.1875 0.011313 0.00839456 2024-09-13T13:14:35.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:35.395 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:35.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:35.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:35.497 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:35.535 INFO:tasks.thrashosds.thrasher:j is {'epoch': 346, 'fsid': '2b415b38-71cf-11ef-bceb-c7b262605968', 'created': '2024-09-13T12:55:27.893882+0000', 'modified': '2024-09-13T13:14:30.590419+0000', 'last_up_change': '2024-09-13T13:04:02.281376+0000', 'last_in_change': '2024-09-13T13:12:52.437622+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T12:58:31.271893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '277', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-13T13:01:00.306747+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-13T13:01:01.103523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-13T13:01:02.249986+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-13T13:01:04.203764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-13T13:01:06.293676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-13T13:13:57.824428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '338', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9144363403320312, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}], 'osds': [{'osd': 0, 'uuid': 'd52dca1a-94c1-4e9e-93f5-687c038cd7c7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6801', 'nonce': 542298937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6802', 'nonce': 542298937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6804', 'nonce': 542298937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6803', 'nonce': 542298937}]}, 'public_addr': '172.21.15.52:6801/542298937', 'cluster_addr': '172.21.15.52:6802/542298937', 'heartbeat_back_addr': '172.21.15.52:6804/542298937', 'heartbeat_front_addr': '172.21.15.52:6803/542298937', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b40a3574-3c6f-4810-a651-8d64aee6801b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6805', 'nonce': 26289431}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6806', 'nonce': 26289431}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6808', 'nonce': 26289431}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6807', 'nonce': 26289431}]}, 'public_addr': '172.21.15.52:6805/26289431', 'cluster_addr': '172.21.15.52:6806/26289431', 'heartbeat_back_addr': '172.21.15.52:6808/26289431', 'heartbeat_front_addr': '172.21.15.52:6807/26289431', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1e8e49a0-ccf8-4b7f-839b-104b64601408', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6809', 'nonce': 3960240393}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6810', 'nonce': 3960240393}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6812', 'nonce': 3960240393}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6811', 'nonce': 3960240393}]}, 'public_addr': '172.21.15.52:6809/3960240393', 'cluster_addr': '172.21.15.52:6810/3960240393', 'heartbeat_back_addr': '172.21.15.52:6812/3960240393', 'heartbeat_front_addr': '172.21.15.52:6811/3960240393', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9f6f1d22-be68-4d86-9d4b-86fd9e87d141', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 309, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6813', 'nonce': 2025522885}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6814', 'nonce': 2025522885}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6816', 'nonce': 2025522885}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.52:6815', 'nonce': 2025522885}]}, 'public_addr': '172.21.15.52:6813/2025522885', 'cluster_addr': '172.21.15.52:6814/2025522885', 'heartbeat_back_addr': '172.21.15.52:6816/2025522885', 'heartbeat_front_addr': '172.21.15.52:6815/2025522885', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1326d850-caf4-484e-90a1-052f54f9f306', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6800', 'nonce': 2137138858}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 2137138858}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 2137138858}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6802', 'nonce': 2137138858}]}, 'public_addr': '172.21.15.167:6800/2137138858', 'cluster_addr': '172.21.15.167:6801/2137138858', 'heartbeat_back_addr': '172.21.15.167:6803/2137138858', 'heartbeat_front_addr': '172.21.15.167:6802/2137138858', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6804', 'nonce': 2906884337}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 2906884337}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 2906884337}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6806', 'nonce': 2906884337}]}, 'public_addr': '172.21.15.167:6804/2906884337', 'cluster_addr': '172.21.15.167:6805/2906884337', 'heartbeat_back_addr': '172.21.15.167:6807/2906884337', 'heartbeat_front_addr': '172.21.15.167:6806/2906884337', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2471907d-fcbd-44e0-9d29-54151ada9df0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 48, 'last_clean_end': 102, 'up_from': 118, 'up_thru': 334, 'down_at': 103, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6808', 'nonce': 143093129}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 143093129}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 143093129}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6810', 'nonce': 143093129}]}, 'public_addr': '172.21.15.167:6808/143093129', 'cluster_addr': '172.21.15.167:6809/143093129', 'heartbeat_back_addr': '172.21.15.167:6811/143093129', 'heartbeat_front_addr': '172.21.15.167:6810/143093129', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '97d2c6d7-7989-4c11-91e0-32ee87e76c6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.31549072265625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 305, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6812', 'nonce': 403155155}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 403155155}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 403155155}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6814', 'nonce': 403155155}]}, 'public_addr': '172.21.15.167:6812/403155155', 'cluster_addr': '172.21.15.167:6813/403155155', 'heartbeat_back_addr': '172.21.15.167:6815/403155155', 'heartbeat_front_addr': '172.21.15.167:6814/403155155', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:13:31.043416+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:13:14.043919+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:14:29.611152+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:13:38.605406+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:14:08.260279+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:13:38.414252+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:14:02.629672+0000', 'dead_epoch': 102}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:13:42.542582+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.15', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.52:0/2104786675': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:6800/910649698': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:0/1471436716': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:0/4151877300': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:0/3080924817': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:0/4248579079': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:0/455730280': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/1887735845': '2024-09-14T12:55:49.232470+0000', '172.21.15.52:6800/1437783219': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:6800/326637643': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/991228238': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/3415403690': '2024-09-14T13:01:31.849627+0000', '172.21.15.52:0/1933060029': '2024-09-14T12:56:14.802531+0000', '172.21.15.52:0/1009267179': '2024-09-14T13:01:31.849627+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T13:14:35.535 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-13T13:14:35.543 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:35.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:35.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:35.644 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:35.684 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:35.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:35.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:35.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:35 smithi052 ceph-mon[29471]: pgmap v668: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-13T13:14:35.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:35.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:35 smithi052 ceph-mon[36682]: pgmap v668: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-13T13:14:35.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:35.781 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:14:35.782 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:14:35.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:35.817 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:14:35.817 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:14:35.819 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:35.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:35.851 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:14:35.851 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:14:35.887 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:14:35.887 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:14:35.921 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:35.924 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:14:35.924 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:14:35.961 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:35.961 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:14:35.961 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:14:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:35.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:35.995 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:14:35.995 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:14:36.030 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:14:36.031 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:14:36.063 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:36.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:35 smithi167 ceph-mon[33178]: pgmap v668: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-13T13:14:36.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:36.109 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:36.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:36.211 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:36.252 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:36.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:36.298 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 32 16 60554 60538 118.224 92.4375 0.0249983 0.0084539 2024-09-13T13:14:36.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:36.383 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:36.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:36.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:36.485 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:14:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:14:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:14:36.486 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:36.494 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:36.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:36.518 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:36.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:36.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:36.660 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:36.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:36 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/339960341' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:36.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:36 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/339960341' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:36.761 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:36.800 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:36.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:36.902 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:36.931 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:36.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:36.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:37.033 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:37.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:36 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/339960341' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:37.079 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:37.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:37.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:37.181 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:37.223 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:37.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:37.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:37.298 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 33 16 62081 62065 117.533 95.4375 0.0106899 0.00850353 2024-09-13T13:14:37.325 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:37.367 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:37.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:37.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:37.469 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:37.510 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:37.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:37.612 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:37.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:37.653 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:37.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:37.660 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:14:37.672 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:14:37.672 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:14:37.672 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:14:37.672 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:14:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.25222.0:237928 8.6 8:692bc360:::benchmark_data_smithi052_101302_object109999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T13:14:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:09:38.068748+0000", 2024-09-13T13:14:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 299.59756247899998, 2024-09-13T13:14:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15122808900000001, 2024-09-13T13:14:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.25222", 2024-09-13T13:14:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:14:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 237928 2024-09-13T13:14:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068748+0000", 2024-09-13T13:14:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068748+0000", 2024-09-13T13:14:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068749+0000", 2024-09-13T13:14:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8599999999999996e-07 2024-09-13T13:14:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068753+0000", 2024-09-13T13:14:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9589999999999997e-06 2024-09-13T13:14:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068760+0000", 2024-09-13T13:14:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6860000000000006e-06 2024-09-13T13:14:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.068766+0000", 2024-09-13T13:14:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8050000000000003e-06 2024-09-13T13:14:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074343+0000", 2024-09-13T13:14:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055769210000000003 2024-09-13T13:14:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074450+0000", 2024-09-13T13:14:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107321 2024-09-13T13:14:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 6,7", 2024-09-13T13:14:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.074571+0000", 2024-09-13T13:14:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012104699999999999 2024-09-13T13:14:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.214148+0000", 2024-09-13T13:14:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.139576799 2024-09-13T13:14:37.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.678 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.678 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.678 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.217865+0000", 2024-09-13T13:14:37.678 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037168190000000001 2024-09-13T13:14:37.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.678 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.678 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.678 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.218583+0000", 2024-09-13T13:14:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00071750199999999996 2024-09-13T13:14:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.219928+0000", 2024-09-13T13:14:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0013456729999999999 2024-09-13T13:14:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:09:38.219976+0000", 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7571000000000002e-05 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:116255 9.8 e275/241 9:16d2e4bc:::benchmark_data_smithi052_128427_object116254:head v 275'7312, mlcod=275'7310)", 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:33.062682+0000", 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 184.603627921, 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.166173769, 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062682+0000", 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062682+0000", 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062685+0000", 2024-09-13T13:14:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3360000000000002e-06 2024-09-13T13:14:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062712+0000", 2024-09-13T13:14:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7667000000000001e-05 2024-09-13T13:14:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062729+0000", 2024-09-13T13:14:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6529e-05 2024-09-13T13:14:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062738+0000", 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6600000000000007e-06 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062830+0000", 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1796e-05 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062852+0000", 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2166000000000001e-05 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.228807+0000", 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.165954191 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.228856+0000", 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9424e-05 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:116256 9.8 e275/241 9:113e5c3c:::benchmark_data_smithi052_128427_object116255:head v 275'7313, mlcod=275'7310)", 2024-09-13T13:14:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:33.065529+0000", 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 184.60078142699999, 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.163368135, 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.065529+0000", 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.065529+0000", 2024-09-13T13:14:37.688 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.688 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.688 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.688 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.065531+0000", 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5069999999999999e-06 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.065558+0000", 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6798999999999999e-05 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.219272+0000", 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15371362499999999 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.219297+0000", 2024-09-13T13:14:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5775999999999999e-05 2024-09-13T13:14:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.221417+0000", 2024-09-13T13:14:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0021196100000000001 2024-09-13T13:14:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.221438+0000", 2024-09-13T13:14:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1094000000000002e-05 2024-09-13T13:14:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.228864+0000", 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0074256269999999997 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.228897+0000", 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3096999999999998e-05 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:116278 9.5 9:a96969c9:::benchmark_data_smithi052_128427_object116277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:33.068171+0000", 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 184.59813889200001, 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15898722100000001, 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 116278 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.068171+0000", 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.068171+0000", 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.068175+0000", 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3500000000000001e-06 2024-09-13T13:14:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.068203+0000", 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7854000000000001e-05 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.219251+0000", 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.151048722 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.219291+0000", 2024-09-13T13:14:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0120000000000002e-05 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.219402+0000", 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110684 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.219558+0000", 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015599599999999999 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:14:37.695 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.219730+0000", 2024-09-13T13:14:37.696 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017155800000000001 2024-09-13T13:14:37.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.696 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:37.696 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.225772+0000", 2024-09-13T13:14:37.696 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0060422519999999997 2024-09-13T13:14:37.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.696 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.696 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.226968+0000", 2024-09-13T13:14:37.696 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011956650000000001 2024-09-13T13:14:37.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.227079+0000", 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011116999999999999 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.227114+0000", 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5565000000000003e-05 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.227159+0000", 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4285e-05 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.697 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.698 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.698 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.698 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.698 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:169623 9.9 9:94682d60:::benchmark_data_smithi052_128427_object38954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:37.698 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.408270+0000", 2024-09-13T13:14:37.698 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 135.25804035600001, 2024-09-13T13:14:37.698 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.175397626, 2024-09-13T13:14:37.698 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.698 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 169623 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408270+0000", 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.699 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408270+0000", 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408272+0000", 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.136e-06 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408278+0000", 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7400000000000001e-06 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408289+0000", 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1712999999999999e-05 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408297+0000", 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8140000000000001e-06 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408367+0000", 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0221000000000004e-05 2024-09-13T13:14:37.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408487+0000", 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000119109 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408598+0000", 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011178799999999999 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413949+0000", 2024-09-13T13:14:37.702 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053510659999999998 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413997+0000", 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7862999999999999e-05 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583426+0000", 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16942890199999999 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.703 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583613+0000", 2024-09-13T13:14:37.704 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000186503 2024-09-13T13:14:37.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.704 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.704 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.705 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583667+0000", 2024-09-13T13:14:37.705 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4771000000000001e-05 2024-09-13T13:14:37.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.705 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.705 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.705 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:169631 9.9 9:9385c360:::benchmark_data_smithi052_128427_object38962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:37.705 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.413293+0000", 2024-09-13T13:14:37.705 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 135.25301752199999, 2024-09-13T13:14:37.705 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.17070595899999999, 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 169631 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413293+0000", 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.707 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.707 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.707 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413293+0000", 2024-09-13T13:14:37.707 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.707 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.707 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.707 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413295+0000", 2024-09-13T13:14:37.707 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.12e-06 2024-09-13T13:14:37.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.707 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.707 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.707 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413300+0000", 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.853e-06 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413318+0000", 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8155000000000002e-05 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413325+0000", 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0149999999999998e-06 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.708 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.414485+0000", 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001159818 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.414595+0000", 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110302 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.414702+0000", 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000106721 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.709 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583409+0000", 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16870703400000001 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583721+0000", 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00031262299999999999 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583768+0000", 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6217999999999997e-05 2024-09-13T13:14:37.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583929+0000", 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016142899999999999 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583999+0000", 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9671000000000004e-05 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.711 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:169632 9.9 9:93968084:::benchmark_data_smithi052_128427_object38963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.413626+0000", 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 135.252684211, 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.170605964, 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 169632 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.712 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413626+0000", 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413626+0000", 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413627+0000", 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.305e-06 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.713 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413632+0000", 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9339999999999997e-06 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413642+0000", 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6409999999999997e-06 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413650+0000", 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2600000000000005e-06 2024-09-13T13:14:37.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.715 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.715 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.715 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.418045+0000", 2024-09-13T13:14:37.715 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0043948060000000002 2024-09-13T13:14:37.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.715 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.715 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.715 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.418155+0000", 2024-09-13T13:14:37.715 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109876 2024-09-13T13:14:37.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.418269+0000", 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114432 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583710+0000", 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16544082800000001 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583726+0000", 2024-09-13T13:14:37.716 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6370000000000001e-05 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584039+0000", 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00031283400000000001 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584188+0000", 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014875399999999999 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584232+0000", 2024-09-13T13:14:37.717 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3924000000000002e-05 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:169635 9.9 9:99294624:::benchmark_data_smithi052_128427_object38966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.415308+0000", 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 135.25100236899999, 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.18454141299999999, 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:37.718 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 169635 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.415308+0000", 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.415308+0000", 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.719 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.720 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.720 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.720 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.415309+0000", 2024-09-13T13:14:37.720 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2309999999999999e-06 2024-09-13T13:14:37.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.720 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.720 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.720 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.415313+0000", 2024-09-13T13:14:37.720 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3930000000000001e-06 2024-09-13T13:14:37.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.720 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.720 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.720 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.415322+0000", 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8999999999999995e-06 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.415329+0000", 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6780000000000003e-06 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.423993+0000", 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0086634950000000002 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.721 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424112+0000", 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011977900000000001 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424232+0000", 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012014 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583731+0000", 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15949875199999999 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.722 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584324+0000", 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00059284099999999998 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.599742+0000", 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015417563 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.599801+0000", 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9497999999999999e-05 2024-09-13T13:14:37.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.599849+0000", 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8143000000000003e-05 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.15789.0:169631 9.9 e292/241 ondisk, result = 0)", 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.419312+0000", 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 135.24699782100001, 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16435757200000001, 2024-09-13T13:14:37.724 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.419312+0000", 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.419312+0000", 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.419313+0000", 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.419317+0000", 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9500000000000003e-06 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.419326+0000", 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.789e-06 2024-09-13T13:14:37.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.419332+0000", 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0780000000000001e-06 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583394+0000", 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16406172599999999 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583402+0000", 2024-09-13T13:14:37.727 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2220000000000002e-06 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583670+0000", 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000268006 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:169641 9.e e292/283 9:78334604:::benchmark_data_smithi052_128427_object38972:head v 292'10493, mlcod=292'10490)", 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.421631+0000", 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 135.24467946799999, 2024-09-13T13:14:37.728 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16167431299999999, 2024-09-13T13:14:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.421631+0000", 2024-09-13T13:14:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.421631+0000", 2024-09-13T13:14:37.729 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.421632+0000", 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6899999999999996e-07 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.421638+0000", 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.730 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.421647+0000", 2024-09-13T13:14:37.731 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0019999999999995e-06 2024-09-13T13:14:37.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.731 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.731 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.731 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.421653+0000", 2024-09-13T13:14:37.731 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4049999999999997e-06 2024-09-13T13:14:37.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.731 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.731 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.731 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.422643+0000", 2024-09-13T13:14:37.731 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00098952900000000006 2024-09-13T13:14:37.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.731 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.732 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.732 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.422662+0000", 2024-09-13T13:14:37.732 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9718999999999999e-05 2024-09-13T13:14:37.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.732 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.732 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.732 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583268+0000", 2024-09-13T13:14:37.732 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16060593500000001 2024-09-13T13:14:37.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.732 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.732 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583305+0000", 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6820999999999999e-05 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.15789.0:169632 9.9 e292/241 ondisk, result = 0)", 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.421955+0000", 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 135.24435479900001, 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16177659599999999, 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.733 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:14:37.734 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.734 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.734 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.421955+0000", 2024-09-13T13:14:37.734 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.734 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.734 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.421955+0000", 2024-09-13T13:14:37.734 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.735 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.735 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.421956+0000", 2024-09-13T13:14:37.735 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5499999999999996e-07 2024-09-13T13:14:37.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.735 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.735 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.421960+0000", 2024-09-13T13:14:37.735 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1010000000000002e-06 2024-09-13T13:14:37.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.735 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.735 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.421969+0000", 2024-09-13T13:14:37.735 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0930000000000003e-06 2024-09-13T13:14:37.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.736 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.736 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.736 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.421974+0000", 2024-09-13T13:14:37.736 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7309999999999998e-06 2024-09-13T13:14:37.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.736 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.736 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.736 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583698+0000", 2024-09-13T13:14:37.736 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16172352000000001 2024-09-13T13:14:37.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.736 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.736 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.737 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583704+0000", 2024-09-13T13:14:37.737 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6159999999999998e-06 2024-09-13T13:14:37.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.737 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.737 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.737 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583732+0000", 2024-09-13T13:14:37.737 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.758e-05 2024-09-13T13:14:37.737 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.737 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.737 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.15789.0:169631 9.9 e292/241 ondisk, result = 0)", 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.424216+0000", 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 135.24209465800001, 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.159787978, 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424216+0000", 2024-09-13T13:14:37.738 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424216+0000", 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424216+0000", 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1399999999999997e-07 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.739 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424219+0000", 2024-09-13T13:14:37.740 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7879999999999998e-06 2024-09-13T13:14:37.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.740 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.740 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424224+0000", 2024-09-13T13:14:37.740 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.288e-06 2024-09-13T13:14:37.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.740 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.740 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424229+0000", 2024-09-13T13:14:37.740 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7439999999999998e-06 2024-09-13T13:14:37.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583757+0000", 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.159527733 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583763+0000", 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3330000000000002e-06 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584004+0000", 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000240478 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.741 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.15789.0:169632 9.9 e292/241 ondisk, result = 0)", 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.424235+0000", 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 135.242075207, 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.160002115, 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.742 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424235+0000", 2024-09-13T13:14:37.743 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.743 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.743 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424235+0000", 2024-09-13T13:14:37.743 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.743 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.743 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424235+0000", 2024-09-13T13:14:37.743 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1399999999999998e-07 2024-09-13T13:14:37.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.744 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.744 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424237+0000", 2024-09-13T13:14:37.744 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.939e-06 2024-09-13T13:14:37.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.744 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.744 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424243+0000", 2024-09-13T13:14:37.744 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0469999999999999e-06 2024-09-13T13:14:37.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.744 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.744 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424247+0000", 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.788e-06 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584028+0000", 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15978078400000001 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584035+0000", 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6919999999999996e-06 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.745 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584237+0000", 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020255100000000001 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:169648 9.9 9:9769410b:::benchmark_data_smithi052_128427_object38979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.425234+0000", 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 135.24107602500001, 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.180310056, 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.746 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 169648 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425234+0000", 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425234+0000", 2024-09-13T13:14:37.747 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.748 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.748 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425236+0000", 2024-09-13T13:14:37.748 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9989999999999998e-06 2024-09-13T13:14:37.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.749 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.749 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.749 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425241+0000", 2024-09-13T13:14:37.749 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0329999999999999e-06 2024-09-13T13:14:37.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.749 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.749 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.749 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.582957+0000", 2024-09-13T13:14:37.749 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15771576000000001 2024-09-13T13:14:37.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.749 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.582975+0000", 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8532000000000001e-05 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584357+0000", 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001381714 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584454+0000", 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7256000000000005e-05 2024-09-13T13:14:37.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584554+0000", 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9856000000000001e-05 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.599901+0000", 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015346790000000001 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.599957+0000", 2024-09-13T13:14:37.751 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5668999999999998e-05 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.605439+0000", 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0054818260000000004 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.605491+0000", 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2278999999999999e-05 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.752 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.605544+0000", 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3341999999999999e-05 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.15789.0:169635 9.9 e292/241 ondisk, result = 0)", 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.426492+0000", 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 135.239817714, 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.157840118, 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:14:37.753 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.426492+0000", 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.426492+0000", 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.426494+0000", 2024-09-13T13:14:37.754 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3069999999999999e-06 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.426498+0000", 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6600000000000003e-06 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.582957+0000", 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15645851599999999 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.582975+0000", 2024-09-13T13:14:37.755 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8467000000000001e-05 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584265+0000", 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001289468 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584271+0000", 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2670000000000003e-06 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.756 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584333+0000", 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1433000000000006e-05 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:169650 9.c e292/283 9:365b7fc3:::benchmark_data_smithi052_128427_object38981:head v 292'10363, mlcod=292'10361)", 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.427761+0000", 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 135.23854873600001, 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15866117499999999, 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.757 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.427761+0000", 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.427761+0000", 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.427763+0000", 2024-09-13T13:14:37.758 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.424e-06 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.427770+0000", 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3000000000000004e-06 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583384+0000", 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.155614216 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.759 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583392+0000", 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3030000000000002e-06 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583567+0000", 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017553800000000001 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583588+0000", 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0784e-05 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.760 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.586387+0000", 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027985850000000001 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.586423+0000", 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6025000000000002e-05 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:252966 9.9 9:9988c3d9:::benchmark_data_smithi052_128427_object122297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-13T13:14:37.761 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:41.440579+0000", 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 56.22573139, 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.073608818000000006, 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 252966 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.762 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.440579+0000", 2024-09-13T13:14:37.763 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.763 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.763 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.440579+0000", 2024-09-13T13:14:37.763 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.763 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.763 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.440579+0000", 2024-09-13T13:14:37.763 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1099999999999995e-07 2024-09-13T13:14:37.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.440583+0000", 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.162e-06 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.440589+0000", 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1909999999999997e-06 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.440594+0000", 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5550000000000002e-06 2024-09-13T13:14:37.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.446996+0000", 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064021499999999997 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.447102+0000", 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000105993 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.447211+0000", 2024-09-13T13:14:37.765 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108927 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.453161+0000", 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0059499150000000001 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.455666+0000", 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025048660000000001 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.766 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.513537+0000", 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.057870589 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.514119+0000", 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00058232100000000001 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:41.514188+0000", 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8538e-05 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.767 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:39785 10.4 10:25780156:::benchmark_data_smithi052_153703_object39784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:23.428083+0000", 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 14.238227143, 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.083495345999999998, 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 39785 2024-09-13T13:14:37.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.428083+0000", 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.428083+0000", 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.769 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.428085+0000", 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.697e-06 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.428109+0000", 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4592000000000001e-05 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.428119+0000", 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4229999999999994e-06 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.770 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.771 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.428126+0000", 2024-09-13T13:14:37.771 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5399999999999998e-06 2024-09-13T13:14:37.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.771 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.771 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.771 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.428195+0000", 2024-09-13T13:14:37.771 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8849000000000004e-05 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.428312+0000", 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000117278 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.428436+0000", 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000123714 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:37.772 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.430139+0000", 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001702807 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.431441+0000", 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0013021129999999999 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.511487+0000", 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.080046193000000002 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.773 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.511532+0000", 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4551000000000001e-05 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:23.511578+0000", 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6588999999999997e-05 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:41376 10.4 10:2fcc1e87:::benchmark_data_smithi052_153703_object41375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:24.367467+0000", 2024-09-13T13:14:37.774 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 13.29884279, 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15987089299999999, 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 41376 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.367467+0000", 2024-09-13T13:14:37.775 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.367467+0000", 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.367469+0000", 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6270000000000001e-06 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.776 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.367494+0000", 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5000000000000001e-05 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.367738+0000", 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024389299999999999 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.367757+0000", 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8734000000000001e-05 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.777 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.368541+0000", 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00078479699999999995 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.368675+0000", 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000133979 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.368819+0000", 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014368399999999999 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.778 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.372415+0000", 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0035963779999999999 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.373258+0000", 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00084215500000000003 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.527263+0000", 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15400556800000001 2024-09-13T13:14:37.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.527305+0000", 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1378000000000003e-05 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.527338+0000", 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3699999999999999e-05 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.780 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:41379 10.4 10:2eb0096b:::benchmark_data_smithi052_153703_object41378:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:24.369543+0000", 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 13.296766764000001, 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16524090399999999, 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 41379 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:37.781 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.369543+0000", 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.369543+0000", 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.369545+0000", 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0310000000000001e-06 2024-09-13T13:14:37.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.369568+0000", 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2237000000000001e-05 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.369577+0000", 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.781e-06 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.369583+0000", 2024-09-13T13:14:37.783 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6019999999999999e-06 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.370590+0000", 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010069 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.370714+0000", 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012358000000000001 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:14:37.784 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.370798+0000", 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4515e-05 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.372419+0000", 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00162141 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.373399+0000", 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00097947599999999996 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:37.785 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.534685+0000", 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16128569500000001 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.534741+0000", 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6011000000000003e-05 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.534784+0000", 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3665999999999998e-05 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.787 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:37.787 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:37.787 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:14:37.787 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:37.824 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:37.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:37.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:37 smithi052 ceph-mon[29471]: pgmap v669: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.67k op/s 2024-09-13T13:14:37.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:37 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/201384678' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:37.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:37 smithi052 ceph-mon[29471]: from='client.26140 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:37.920 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:37 smithi052 ceph-mon[36682]: pgmap v669: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.67k op/s 2024-09-13T13:14:37.920 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:37 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/201384678' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:37.920 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:37 smithi052 ceph-mon[36682]: from='client.26140 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:37.926 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:37.966 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:37.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:37.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:38.068 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:37 smithi167 ceph-mon[33178]: pgmap v669: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.67k op/s 2024-09-13T13:14:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:37 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/201384678' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:38.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:37 smithi167 ceph-mon[33178]: from='client.26140 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:38.102 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:38.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:38.204 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:38.248 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:38.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:38.298 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 34 16 63533 63517 116.745 90.75 0.0129588 0.0085604 2024-09-13T13:14:38.350 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:38.393 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:38.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:38.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:38.495 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:38.530 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:38.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:38.631 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:38.674 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:38.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:38.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:38.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:38 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:38.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:38 smithi052 ceph-mon[36682]: from='client.26140 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:14:38.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:38 smithi052 ceph-mon[36682]: osdmap e347: 8 total, 8 up, 8 in 2024-09-13T13:14:38.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:38 smithi052 ceph-mon[36682]: osdmap e348: 8 total, 8 up, 8 in 2024-09-13T13:14:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:38 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:38 smithi052 ceph-mon[29471]: from='client.26140 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:14:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:38 smithi052 ceph-mon[29471]: osdmap e347: 8 total, 8 up, 8 in 2024-09-13T13:14:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:38 smithi052 ceph-mon[29471]: osdmap e348: 8 total, 8 up, 8 in 2024-09-13T13:14:38.777 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:38.811 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:38.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:38.913 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:38.917 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T13:14:38.949 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:38.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:38.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:39.051 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:39.060 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:14:39.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:38 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:39.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:38 smithi167 ceph-mon[33178]: from='client.26140 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:14:39.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:38 smithi167 ceph-mon[33178]: osdmap e347: 8 total, 8 up, 8 in 2024-09-13T13:14:39.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:38 smithi167 ceph-mon[33178]: osdmap e348: 8 total, 8 up, 8 in 2024-09-13T13:14:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:39.097 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:39.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:39.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:39.226 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:39.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:39.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:39.298 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 35 16 64976 64960 115.986 90.1875 0.00362615 0.00861571 2024-09-13T13:14:39.328 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:39.351 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:39.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:39.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:39.453 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:39.488 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:39.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:39.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:39.589 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:39.628 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:39.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:39.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:39.731 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:39.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:39.763 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:39.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:39.864 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:39.901 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:39.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:39.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:39.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:39 smithi052 ceph-mon[36682]: pgmap v671: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-13T13:14:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:39 smithi052 ceph-mon[29471]: pgmap v671: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-13T13:14:40.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:40.027 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:40.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:40.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:39 smithi167 ceph-mon[33178]: pgmap v671: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-13T13:14:40.129 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:40.166 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:40.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:40.268 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:40.301 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 36 16 66233 66217 114.937 78.5625 0.00912163 0.00869493 2024-09-13T13:14:40.307 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:40.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:40.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:40.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:40.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:40.431 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:40.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:40.532 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:40.535 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:14:40.536 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:14:40.536 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T13:14:40.536 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-13T13:14:40.536 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-13T13:14:40.566 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:40.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:40.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:40.668 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:40.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:40.735 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:40.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:40.836 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:40.875 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:40.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:40.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:40.977 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:40.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:40.999 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:40.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:41.100 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:41.131 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:41.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:41.233 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:41.271 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:41.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:41.301 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 37 16 67292 67276 113.619 66.1875 0.00999035 0.00879624 2024-09-13T13:14:41.373 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:41.417 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:41.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:41.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:41.558 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:41.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:41.660 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:41.696 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:41.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:41.798 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:41.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:41.845 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:41.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:41.942 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:41.946 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:41.955 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:41.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:41 smithi052 ceph-mon[36682]: pgmap v673: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-13T13:14:41.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:41 smithi052 ceph-mon[29471]: pgmap v673: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-13T13:14:42.018 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:42.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:42.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:41 smithi167 ceph-mon[33178]: pgmap v673: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-13T13:14:42.121 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:42.151 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:42.164 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:42.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:42.265 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:42.297 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:42.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:42.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:42.301 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 38 16 68636 68620 112.84 84 0.0102611 0.00885642 2024-09-13T13:14:42.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:42.432 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:42.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:42.533 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:42.573 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:42.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:42.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:42.705 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:42.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:42.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:42.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:42.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:42.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:42.947 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:42.990 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:42.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:43.092 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:43.142 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:43.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:43.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:43.244 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:43.280 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:43.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:43.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:43.301 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 39 16 70092 70076 112.28 91 0.00708952 0.00890196 2024-09-13T13:14:43.382 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:43.426 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:43.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:43.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:43.528 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:43.563 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:43.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:43.664 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:43.700 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:43.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:43.802 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:43.841 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:43.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:43.869 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:14:43.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:43 smithi052 ceph-mon[36682]: pgmap v674: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-13T13:14:43.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:43 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:14:43.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:43 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:14:43.876 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:43 smithi052 ceph-mon[29471]: pgmap v674: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-13T13:14:43.876 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:43 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:14:43.876 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:43 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:14:43.942 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:43.968 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:43.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:43.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:44.052 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:14:44.052 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:14:44.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.052 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:71036 10.2 10:4bf7b468:::benchmark_data_smithi052_153703_object71035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-13T13:14:44.052 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:44.042066+0000", 2024-09-13T13:14:44.052 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.007404178, 2024-09-13T13:14:44.052 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0074440289999999996, 2024-09-13T13:14:44.052 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:44.052 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 71036 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042066+0000", 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.053 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042066+0000", 2024-09-13T13:14:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042081+0000", 2024-09-13T13:14:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4929000000000001e-05 2024-09-13T13:14:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042187+0000", 2024-09-13T13:14:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000106209 2024-09-13T13:14:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042204+0000", 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7716000000000001e-05 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042214+0000", 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5249999999999994e-06 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042272+0000", 2024-09-13T13:14:44.055 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8255999999999999e-05 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042449+0000", 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000176613 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042631+0000", 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018207799999999999 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.057 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:71037 10.2 10:44b4e7e0:::benchmark_data_smithi052_153703_object71036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-13T13:14:44.057 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:44.042234+0000", 2024-09-13T13:14:44.057 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0072354689999999996, 2024-09-13T13:14:44.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0073583939999999999, 2024-09-13T13:14:44.057 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:44.057 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:44.057 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:14:44.057 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:44.057 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:44.057 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:44.057 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 71037 2024-09-13T13:14:44.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.058 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:44.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.058 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:44.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042234+0000", 2024-09-13T13:14:44.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:44.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.058 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:44.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042234+0000", 2024-09-13T13:14:44.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:44.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.059 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:44.059 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042235+0000", 2024-09-13T13:14:44.059 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5999999999999991e-07 2024-09-13T13:14:44.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:44.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042257+0000", 2024-09-13T13:14:44.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1347000000000001e-05 2024-09-13T13:14:44.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:44.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042265+0000", 2024-09-13T13:14:44.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3680000000000002e-06 2024-09-13T13:14:44.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:44.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042272+0000", 2024-09-13T13:14:44.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7850000000000003e-06 2024-09-13T13:14:44.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:44.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.045295+0000", 2024-09-13T13:14:44.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0030232060000000001 2024-09-13T13:14:44.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:44.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.045596+0000", 2024-09-13T13:14:44.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00030098400000000002 2024-09-13T13:14:44.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:14:44.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.046385+0000", 2024-09-13T13:14:44.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00078947299999999995 2024-09-13T13:14:44.062 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:44.062 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:44.062 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:44.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.063 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:71038 10.2 10.4b31a222 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-13T13:14:44.063 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:44.042281+0000", 2024-09-13T13:14:44.063 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.007188692, 2024-09-13T13:14:44.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0073828310000000003, 2024-09-13T13:14:44.063 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:44.063 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:44.063 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:14:44.063 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:44.063 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:44.064 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:44.064 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 71038 2024-09-13T13:14:44.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.064 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:44.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:44.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042281+0000", 2024-09-13T13:14:44.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:44.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:44.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042281+0000", 2024-09-13T13:14:44.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:44.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:44.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042282+0000", 2024-09-13T13:14:44.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2399999999999997e-07 2024-09-13T13:14:44.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:44.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042304+0000", 2024-09-13T13:14:44.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1933000000000002e-05 2024-09-13T13:14:44.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:44.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042311+0000", 2024-09-13T13:14:44.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2339999999999997e-06 2024-09-13T13:14:44.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:44.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.042318+0000", 2024-09-13T13:14:44.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4930000000000003e-06 2024-09-13T13:14:44.067 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:44.067 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:44.067 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:44.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.067 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.26071.0:71036 10.2 e349/334)", 2024-09-13T13:14:44.067 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:44.044961+0000", 2024-09-13T13:14:44.067 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0045088819999999996, 2024-09-13T13:14:44.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047441460000000003, 2024-09-13T13:14:44.068 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:44.068 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:44.068 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:14:44.068 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:44.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.068 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:44.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.044961+0000", 2024-09-13T13:14:44.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:44.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.068 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:44.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.044961+0000", 2024-09-13T13:14:44.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:44.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.069 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:44.069 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.044963+0000", 2024-09-13T13:14:44.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2249999999999999e-06 2024-09-13T13:14:44.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.069 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:44.069 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.044971+0000", 2024-09-13T13:14:44.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3549999999999999e-06 2024-09-13T13:14:44.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.069 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.044987+0000", 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6739e-05 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.044997+0000", 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4450000000000002e-06 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.26071.0:71037 10.2 e349/334)", 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:44.049513+0000", 2024-09-13T13:14:44.070 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 4294967295.9999571, 2024-09-13T13:14:44.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000235028, 2024-09-13T13:14:44.071 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:44.071 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:44.071 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:14:44.071 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:44.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:44.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.049513+0000", 2024-09-13T13:14:44.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:44.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:44.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.049513+0000", 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.049514+0000", 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.598e-06 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.049520+0000", 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0240000000000003e-06 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.049535+0000", 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4929000000000001e-05 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:44.049543+0000", 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.447e-06 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 5 2024-09-13T13:14:44.073 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:14:44.074 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:44.075 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:43 smithi167 ceph-mon[33178]: pgmap v674: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-13T13:14:44.075 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:43 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:14:44.075 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:43 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:14:44.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:44.126 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:44.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:44.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:44.255 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:44.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:44.302 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:14:44.301682+0000 min lat: 0.0012696 max lat: 0.355388 avg lat: 0.00897008 2024-09-13T13:14:44.302 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:14:44.302 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 40 16 71340 71324 111.423 78 0.0119532 0.00897008 2024-09-13T13:14:44.357 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:44.392 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:44.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:44.494 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:44.532 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:44.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:44.635 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:44.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:44.701 INFO:teuthology.orchestra.run.smithi052.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-13T13:14:44.705 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:14:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:14:44.777 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:44.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:44.815 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:44.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:44.917 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:44.956 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:44.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:44.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:44.964 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:44 smithi052 ceph-mon[36682]: osdmap e349: 8 total, 8 up, 8 in 2024-09-13T13:14:44.964 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/673418339' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:14:44.964 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:44 smithi052 ceph-mon[36682]: from='client.36185 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:14:44.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:44 smithi052 ceph-mon[29471]: osdmap e349: 8 total, 8 up, 8 in 2024-09-13T13:14:44.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/673418339' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:14:44.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:44 smithi052 ceph-mon[29471]: from='client.36185 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:14:45.059 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:45.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:44 smithi167 ceph-mon[33178]: osdmap e349: 8 total, 8 up, 8 in 2024-09-13T13:14:45.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/673418339' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:14:45.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:44 smithi167 ceph-mon[33178]: from='client.36185 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:14:45.100 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:45.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:45.202 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:45.242 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:45.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:45.256 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T13:14:45.302 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 41 16 72742 72726 110.842 87.625 0.016641 0.0090161 2024-09-13T13:14:45.344 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:45.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:45.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:45.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:45.483 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:45.535 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:45.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:45.637 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:45.675 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:45.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:45.776 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:45.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:45.809 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:45.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:45.873 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:45 smithi052 ceph-mon[36682]: pgmap v676: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 820 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-13T13:14:45.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:45 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:14:45.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:45 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:14:45.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:45 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:45.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:45 smithi052 ceph-mon[36682]: from='client.36185 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:14:45.874 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:45 smithi052 ceph-mon[36682]: osdmap e350: 8 total, 8 up, 8 in 2024-09-13T13:14:45.876 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:45 smithi052 ceph-mon[29471]: pgmap v676: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 820 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-13T13:14:45.876 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:45 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:14:45.876 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:45 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:14:45.876 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:45 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:45.876 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:45 smithi052 ceph-mon[29471]: from='client.36185 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:14:45.876 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:45 smithi052 ceph-mon[29471]: osdmap e350: 8 total, 8 up, 8 in 2024-09-13T13:14:45.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:45.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:45.944 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:45.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:46.045 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:46.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:45 smithi167 ceph-mon[33178]: pgmap v676: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 820 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-13T13:14:46.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:45 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:14:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:45 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:14:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:45 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:45 smithi167 ceph-mon[33178]: from='client.36185 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:14:46.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:45 smithi167 ceph-mon[33178]: osdmap e350: 8 total, 8 up, 8 in 2024-09-13T13:14:46.087 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:46.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:46.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:46.189 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:46.209 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:14:46.209 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:14:46.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:46.223 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:46.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:46.233 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:46.244 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:14:46.245 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:14:46.283 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:14:46.283 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:14:46.305 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 42 16 74075 74059 110.178 83.3125 0.0182247 0.0090711 2024-09-13T13:14:46.320 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:14:46.320 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:14:46.324 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:46.358 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:46.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:46.360 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:14:46.360 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:14:46.394 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:14:46.394 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:14:46.428 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:14:46.428 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:14:46.429 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:14:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:14:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:14:46.460 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:46.466 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:14:46.466 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:14:46.494 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:46.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:46.596 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:46.650 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:46.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:46.751 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:46.786 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:46.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:46.888 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:46.923 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:46.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:46.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:46 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4093735363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:46.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:46 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4093735363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:47.025 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:47.035 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:47.057 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:47.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:47.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:46 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4093735363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:47.159 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:47.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:47.209 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:47.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:47.305 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 43 15 75380 75365 109.514 81.625 0.00570284 0.00912594 2024-09-13T13:14:47.310 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:47.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:47.354 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:47.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:47.457 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:47.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:47.499 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:47.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:47.600 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:47.635 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:47.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:47.736 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:47.737 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:14:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:47.768 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:47.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:47.865 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:47 smithi052 ceph-mon[36682]: pgmap v678: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-13T13:14:47.866 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:47 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3884395062' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:14:47.866 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:47 smithi052 ceph-mon[36682]: from='client.36194 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:14:47.866 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:47 smithi052 ceph-mon[29471]: pgmap v678: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-13T13:14:47.866 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:47 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3884395062' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:14:47.866 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:47 smithi052 ceph-mon[29471]: from='client.36194 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:14:47.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:47.901 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:47.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:48.002 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:48.036 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:48.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:48.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:48.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:47 smithi167 ceph-mon[33178]: pgmap v678: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-13T13:14:48.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:47 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3884395062' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:14:48.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:47 smithi167 ceph-mon[33178]: from='client.36194 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:14:48.138 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:48.175 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:48.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:48.277 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:48.311 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 44 16 76701 76685 108.9 82.5 0.00781039 0.00917804 2024-09-13T13:14:48.315 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:48.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:48.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:48.387 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:14:48.387 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:14:48.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:48.453 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:48.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:48.555 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:48.597 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:48.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:48.698 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:48.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:48.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:48.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:48.838 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:48.871 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:48.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:48.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:48.973 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:48.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:48 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:48.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:48 smithi052 ceph-mon[36682]: from='client.36194 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:14:48.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:48 smithi052 ceph-mon[36682]: osdmap e351: 8 total, 8 up, 8 in 2024-09-13T13:14:48.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:48 smithi052 ceph-mon[36682]: osdmap e352: 8 total, 8 up, 8 in 2024-09-13T13:14:48.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:48 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:14:48.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:48 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:48.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:48 smithi052 ceph-mon[29471]: from='client.36194 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:14:48.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:48 smithi052 ceph-mon[29471]: osdmap e351: 8 total, 8 up, 8 in 2024-09-13T13:14:48.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:48 smithi052 ceph-mon[29471]: osdmap e352: 8 total, 8 up, 8 in 2024-09-13T13:14:48.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:48 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:14:49.019 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:49.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:49.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:49.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:48 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:14:49.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:48 smithi167 ceph-mon[33178]: from='client.36194 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:14:49.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:48 smithi167 ceph-mon[33178]: osdmap e351: 8 total, 8 up, 8 in 2024-09-13T13:14:49.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:48 smithi167 ceph-mon[33178]: osdmap e352: 8 total, 8 up, 8 in 2024-09-13T13:14:49.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:48 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:14:49.120 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:49.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:49.143 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:49.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:49.244 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:49.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:49.281 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:49.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:49.305 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 45 15 77893 77878 108.137 74.5625 0.0122344 0.00924245 2024-09-13T13:14:49.383 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:49.419 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:49.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:49.520 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:49.560 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:49.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:49.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:49.663 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:49.668 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:14:49.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:49.692 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:49.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:49.793 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:49.832 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:49.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:49.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:49.935 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:49.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:49 smithi052 ceph-mon[29471]: pgmap v680: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-13T13:14:49.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:49 smithi052 ceph-mon[29471]: 2.0 scrub starts 2024-09-13T13:14:49.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:49 smithi052 ceph-mon[29471]: 2.0 scrub ok 2024-09-13T13:14:49.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:49 smithi052 ceph-mon[36682]: pgmap v680: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-13T13:14:49.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:49 smithi052 ceph-mon[36682]: 2.0 scrub starts 2024-09-13T13:14:49.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:49 smithi052 ceph-mon[36682]: 2.0 scrub ok 2024-09-13T13:14:49.999 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:50.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:50.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:49 smithi167 ceph-mon[33178]: pgmap v680: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-13T13:14:50.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:49 smithi167 ceph-mon[33178]: 2.0 scrub starts 2024-09-13T13:14:50.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:49 smithi167 ceph-mon[33178]: 2.0 scrub ok 2024-09-13T13:14:50.101 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:50.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:50.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:50.234 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:50.271 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T13:14:50.289 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:50.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:50.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:50.307 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 46 16 79205 79189 107.564 81.9375 0.0143459 0.00929096 2024-09-13T13:14:50.361 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:50.391 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:50.422 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:50.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:50.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:50.524 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:50.560 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:50.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:50.662 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:50.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:50.694 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:50.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:50.795 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:50.831 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:50.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:50.922 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:14:50.922 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:14:50.922 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-13T13:14:50.922 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T13:14:50.922 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T13:14:50.923 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-13T13:14:50.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:50.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:50.962 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:50.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:51.036 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:51.049 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:50 smithi052 ceph-mon[36682]: 3.16 scrub starts 2024-09-13T13:14:51.049 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:50 smithi052 ceph-mon[36682]: 3.16 scrub ok 2024-09-13T13:14:51.049 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:50 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:14:51.049 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:50 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:14:51.049 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:14:51.050 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:14:51.051 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:50 smithi052 ceph-mon[29471]: 3.16 scrub starts 2024-09-13T13:14:51.051 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:50 smithi052 ceph-mon[29471]: 3.16 scrub ok 2024-09-13T13:14:51.051 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:50 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:14:51.052 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:50 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:14:51.063 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:51.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:50 smithi167 ceph-mon[33178]: 3.16 scrub starts 2024-09-13T13:14:51.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:50 smithi167 ceph-mon[33178]: 3.16 scrub ok 2024-09-13T13:14:51.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:50 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:14:51.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:50 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:14:51.088 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:14:51.088 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:14:51.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:51.090 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:51.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:51.124 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:14:51.124 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:14:51.163 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:14:51.163 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:14:51.191 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:51.197 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:14:51.198 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:14:51.236 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:14:51.236 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:14:51.237 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:51.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:51.277 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:14:51.277 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:14:51.307 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 47 16 80348 80332 106.795 71.4375 0.00740616 0.00935806 2024-09-13T13:14:51.311 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:14:51.311 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:14:51.339 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:51.389 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:51.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:51.490 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:51.528 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:51.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:51.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:51.630 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:51.652 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:14:51.667 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:51.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:51.769 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:51.820 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:51.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:51.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:51.922 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:51.960 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:51.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:51.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[36682]: pgmap v682: 148 pgs: 148 active+clean; 4.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T13:14:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[36682]: 5.12 scrub starts 2024-09-13T13:14:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[36682]: 5.12 scrub ok 2024-09-13T13:14:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[36682]: 5.2 scrub starts 2024-09-13T13:14:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[36682]: 5.2 scrub ok 2024-09-13T13:14:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[36682]: osdmap e353: 8 total, 8 up, 8 in 2024-09-13T13:14:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/107150210' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:51.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[29471]: pgmap v682: 148 pgs: 148 active+clean; 4.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T13:14:51.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[29471]: 5.12 scrub starts 2024-09-13T13:14:51.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[29471]: 5.12 scrub ok 2024-09-13T13:14:51.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[29471]: 5.2 scrub starts 2024-09-13T13:14:51.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[29471]: 5.2 scrub ok 2024-09-13T13:14:51.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[29471]: osdmap e353: 8 total, 8 up, 8 in 2024-09-13T13:14:51.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:51 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/107150210' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:52.063 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:52.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:51 smithi167 ceph-mon[33178]: pgmap v682: 148 pgs: 148 active+clean; 4.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T13:14:52.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:51 smithi167 ceph-mon[33178]: 5.12 scrub starts 2024-09-13T13:14:52.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:51 smithi167 ceph-mon[33178]: 5.12 scrub ok 2024-09-13T13:14:52.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:51 smithi167 ceph-mon[33178]: 5.2 scrub starts 2024-09-13T13:14:52.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:51 smithi167 ceph-mon[33178]: 5.2 scrub ok 2024-09-13T13:14:52.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:51 smithi167 ceph-mon[33178]: osdmap e353: 8 total, 8 up, 8 in 2024-09-13T13:14:52.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:51 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/107150210' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:14:52.100 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:52.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:52.202 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:52.226 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:52.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:52.307 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 48 16 81503 81487 106.074 72.1875 0.0125175 0.00942198 2024-09-13T13:14:52.328 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:52.353 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:52.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:52.455 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:52.497 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:52.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:52.598 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:52.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:52.623 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:52.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:52.724 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:52.761 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:52.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:52.863 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:52.899 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:52.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:53.001 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:53.038 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:53.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:53.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:52 smithi167 ceph-mon[33178]: pgmap v684: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-13T13:14:53.140 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:53.174 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:53.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:53.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:52 smithi052 ceph-mon[36682]: pgmap v684: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-13T13:14:53.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:52 smithi052 ceph-mon[29471]: pgmap v684: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-13T13:14:53.276 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:53.297 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:53.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:53.307 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 49 16 82707 82691 105.445 75.25 0.0360582 0.00947756 2024-09-13T13:14:53.399 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:53.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:53.437 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:53.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:53.538 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:53.575 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:53.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:53.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:53.677 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:53.706 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:53.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:53.808 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:53.851 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:53.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:53.952 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:53.958 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:53.966 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:14:53.966 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:14:53.966 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:14:53.966 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:14:53.966 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.966 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189885 9.4 9:25913ce2:::benchmark_data_smithi052_128427_object59216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-09-13T13:14:53.966 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:14:53.966 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 131.05198954900001, 2024-09-13T13:14:53.966 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.048349118, 2024-09-13T13:14:53.966 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:53.967 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:53.967 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:14:53.967 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:53.967 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:53.967 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:53.967 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189885 2024-09-13T13:14:53.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.967 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:53.967 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.967 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:53.967 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:14:53.967 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:53.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.968 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.968 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:53.968 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:14:53.968 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:53.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.968 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.968 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:53.968 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909609+0000", 2024-09-13T13:14:53.968 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0860000000000001e-06 2024-09-13T13:14:53.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.968 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.968 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:53.968 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909614+0000", 2024-09-13T13:14:53.969 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5199999999999999e-06 2024-09-13T13:14:53.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.969 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:53.969 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909627+0000", 2024-09-13T13:14:53.969 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3623e-05 2024-09-13T13:14:53.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.969 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:53.969 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909634+0000", 2024-09-13T13:14:53.969 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0210000000000002e-06 2024-09-13T13:14:53.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909697+0000", 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2769999999999997e-05 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909840+0000", 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014305399999999999 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3,4", 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909967+0000", 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012707300000000001 2024-09-13T13:14:53.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.971 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:53.971 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.913962+0000", 2024-09-13T13:14:53.971 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039947009999999998 2024-09-13T13:14:53.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.971 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:53.971 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.913977+0000", 2024-09-13T13:14:53.971 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.506e-05 2024-09-13T13:14:53.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.971 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.916641+0000", 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026644839999999999 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.937520+0000", 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.020878821999999998 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.937533+0000", 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2896e-05 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.973 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:53.973 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.957873+0000", 2024-09-13T13:14:53.973 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0203400570000001 2024-09-13T13:14:53.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.973 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:53.973 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.957957+0000", 2024-09-13T13:14:53.973 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3950999999999996e-05 2024-09-13T13:14:53.973 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:53.973 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:53.973 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:53.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189915 9.4 9:2fa1035b:::benchmark_data_smithi052_128427_object59246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 131.018655725, 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0238765249999999, 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189915 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:53.974 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.975 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:53.975 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:14:53.975 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:53.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.975 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:53.975 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:14:53.975 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:53.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942944+0000", 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.655e-06 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942948+0000", 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3220000000000003e-06 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942957+0000", 2024-09-13T13:14:53.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2609999999999999e-06 2024-09-13T13:14:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942964+0000", 2024-09-13T13:14:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6379999999999998e-06 2024-09-13T13:14:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.944838+0000", 2024-09-13T13:14:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018746419999999999 2024-09-13T13:14:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:14:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.944845+0000", 2024-09-13T13:14:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:14:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958435+0000", 2024-09-13T13:14:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0135895370000001 2024-09-13T13:14:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958549+0000", 2024-09-13T13:14:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114144 2024-09-13T13:14:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958677+0000", 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012798 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966592+0000", 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0079149440000000001 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966687+0000", 2024-09-13T13:14:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4741999999999995e-05 2024-09-13T13:14:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966725+0000", 2024-09-13T13:14:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7768000000000002e-05 2024-09-13T13:14:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966777+0000", 2024-09-13T13:14:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2605999999999997e-05 2024-09-13T13:14:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.981 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:53.981 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966818+0000", 2024-09-13T13:14:53.981 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1279999999999998e-05 2024-09-13T13:14:53.981 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:53.981 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:53.981 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189933 9.4 9:2eec938f:::benchmark_data_smithi052_128427_object59264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:14:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:14:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 131.00352712700001, 2024-09-13T13:14:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.008927224, 2024-09-13T13:14:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189933 2024-09-13T13:14:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:14:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:14:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.984 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:53.984 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958072+0000", 2024-09-13T13:14:53.985 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7e-06 2024-09-13T13:14:53.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.985 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958078+0000", 2024-09-13T13:14:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4990000000000002e-06 2024-09-13T13:14:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958089+0000", 2024-09-13T13:14:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0865e-05 2024-09-13T13:14:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958095+0000", 2024-09-13T13:14:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8279999999999997e-06 2024-09-13T13:14:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958183+0000", 2024-09-13T13:14:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7655e-05 2024-09-13T13:14:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:14:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958191+0000", 2024-09-13T13:14:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9529999999999999e-06 2024-09-13T13:14:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961131+0000", 2024-09-13T13:14:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0029402620000001 2024-09-13T13:14:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961259+0000", 2024-09-13T13:14:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012816999999999999 2024-09-13T13:14:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:14:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961395+0000", 2024-09-13T13:14:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013549800000000001 2024-09-13T13:14:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966691+0000", 2024-09-13T13:14:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0052965979999999996 2024-09-13T13:14:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966864+0000", 2024-09-13T13:14:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017277899999999999 2024-09-13T13:14:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966899+0000", 2024-09-13T13:14:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4377e-05 2024-09-13T13:14:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966955+0000", 2024-09-13T13:14:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6495e-05 2024-09-13T13:14:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966998+0000", 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2545e-05 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189960 9.4 9:2cd09825:::benchmark_data_smithi052_128427_object59291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 130.97948165299999, 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99033151900000005, 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:53.991 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:53.992 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:53.992 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:53.992 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:53.992 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:53.992 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189960 2024-09-13T13:14:53.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.992 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:53.992 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.992 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:53.992 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982118+0000", 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.993 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:53.994 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982123+0000", 2024-09-13T13:14:53.994 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8389999999999997e-06 2024-09-13T13:14:53.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.994 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:53.994 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982137+0000", 2024-09-13T13:14:53.994 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4433000000000001e-05 2024-09-13T13:14:53.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982145+0000", 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8290000000000008e-06 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982197+0000", 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2431e-05 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982205+0000", 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8630000000000008e-06 2024-09-13T13:14:53.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964055+0000", 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98184965300000004 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964156+0000", 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101055 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964261+0000", 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104961 2024-09-13T13:14:53.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972104+0000", 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0078433300000000008 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972147+0000", 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3022000000000002e-05 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972189+0000", 2024-09-13T13:14:53.997 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1727000000000002e-05 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972384+0000", 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019545200000000001 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972447+0000", 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3144000000000004e-05 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:53.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189972 9.4 9:28374290:::benchmark_data_smithi052_128427_object59303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 130.970909487, 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99084523700000005, 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189972 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:53.999 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.000 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.000 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.000 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:14:54.000 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.000 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.000 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.000 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:14:54.000 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.001 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.001 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.001 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990690+0000", 2024-09-13T13:14:54.001 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:14:54.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.001 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.001 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.001 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990694+0000", 2024-09-13T13:14:54.001 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6029999999999998e-06 2024-09-13T13:14:54.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.001 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.001 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.002 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990704+0000", 2024-09-13T13:14:54.002 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5060000000000001e-06 2024-09-13T13:14:54.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.002 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.002 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.002 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990710+0000", 2024-09-13T13:14:54.002 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:14:54.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.002 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.002 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990764+0000", 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4252000000000002e-05 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990773+0000", 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2379999999999997e-06 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967034+0000", 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97626137000000002 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.003 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.004 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.004 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967153+0000", 2024-09-13T13:14:54.004 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118855 2024-09-13T13:14:54.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.004 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.004 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:14:54.004 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967267+0000", 2024-09-13T13:14:54.004 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113855 2024-09-13T13:14:54.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.004 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.004 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.005 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972462+0000", 2024-09-13T13:14:54.005 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051953219999999996 2024-09-13T13:14:54.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.005 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.005 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.005 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.976022+0000", 2024-09-13T13:14:54.005 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003559563 2024-09-13T13:14:54.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.005 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.005 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981419+0000", 2024-09-13T13:14:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053977260000000003 2024-09-13T13:14:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981488+0000", 2024-09-13T13:14:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8950999999999998e-05 2024-09-13T13:14:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981533+0000", 2024-09-13T13:14:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4981999999999997e-05 2024-09-13T13:14:54.006 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202468 9.4 9:2e7e6063:::benchmark_data_smithi052_128427_object71799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 115.995689169, 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.030148343, 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:54.007 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:54.008 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202468 2024-09-13T13:14:54.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.008 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.008 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.008 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:14:54.008 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.008 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.008 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965910+0000", 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2810000000000001e-06 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965915+0000", 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0749999999999997e-06 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.009 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.010 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965926+0000", 2024-09-13T13:14:54.010 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1056999999999999e-05 2024-09-13T13:14:54.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.010 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.010 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.010 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965943+0000", 2024-09-13T13:14:54.010 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7178000000000001e-05 2024-09-13T13:14:54.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.010 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.010 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.010 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.966053+0000", 2024-09-13T13:14:54.010 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109768 2024-09-13T13:14:54.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.011 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.011 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:14:54.011 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.966062+0000", 2024-09-13T13:14:54.011 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4029999999999996e-06 2024-09-13T13:14:54.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.011 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.011 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.011 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.971764+0000", 2024-09-13T13:14:54.011 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0057021909999999 2024-09-13T13:14:54.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.011 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.971894+0000", 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000129987 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.972041+0000", 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000146398 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976575+0000", 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045341729999999998 2024-09-13T13:14:54.012 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.013 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.013 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.013 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976634+0000", 2024-09-13T13:14:54.013 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8996999999999998e-05 2024-09-13T13:14:54.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.013 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.013 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.013 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.995927+0000", 2024-09-13T13:14:54.013 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.019293457999999999 2024-09-13T13:14:54.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.014 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.014 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.014 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.996003+0000", 2024-09-13T13:14:54.015 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5221000000000004e-05 2024-09-13T13:14:54.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.016 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.016 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.996057+0000", 2024-09-13T13:14:54.016 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4156000000000001e-05 2024-09-13T13:14:54.016 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.016 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.016 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.016 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202476 9.4 9:205ce74f:::benchmark_data_smithi052_128427_object71807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:14:54.016 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 115.98871645600001, 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0253464960000001, 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202476 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.017 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972883+0000", 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.792e-06 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.018 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972888+0000", 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7380000000000002e-06 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972896+0000", 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9230000000000002e-06 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972903+0000", 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5310000000000004e-06 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.019 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.020 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.973002+0000", 2024-09-13T13:14:54.020 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8484000000000001e-05 2024-09-13T13:14:54.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.020 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:14:54.020 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.973012+0000", 2024-09-13T13:14:54.020 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0016e-05 2024-09-13T13:14:54.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.020 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.020 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974404+0000", 2024-09-13T13:14:54.020 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0013926820000001 2024-09-13T13:14:54.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974530+0000", 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012599 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974651+0000", 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012022400000000001 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976695+0000", 2024-09-13T13:14:54.021 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020448139999999998 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981646+0000", 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049504229999999998 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998145+0000", 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.016499092 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998183+0000", 2024-09-13T13:14:54.022 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8457999999999997e-05 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998228+0000", 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4329000000000002e-05 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202503 9.4 9:257a0b3a:::benchmark_data_smithi052_128427_object71834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 115.969668121, 2024-09-13T13:14:54.023 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.009335557, 2024-09-13T13:14:54.024 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.024 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.024 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.024 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:54.024 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:54.024 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:54.024 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202503 2024-09-13T13:14:54.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.024 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.024 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.024 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:14:54.024 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991936+0000", 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1149999999999999e-06 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.025 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991941+0000", 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.463e-06 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991954+0000", 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2533e-05 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991961+0000", 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1690000000000003e-06 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.026 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.992017+0000", 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5856999999999997e-05 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.992026+0000", 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976789+0000", 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98476255999999995 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.027 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976898+0000", 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109548 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.977048+0000", 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015011899999999999 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981632+0000", 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045837020000000003 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.029 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.029 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981701+0000", 2024-09-13T13:14:54.029 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8511999999999997e-05 2024-09-13T13:14:54.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.029 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:54.030 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:54.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:54.031 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.031 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001159+0000", 2024-09-13T13:14:54.031 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.019457953 2024-09-13T13:14:54.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.031 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.031 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001221+0000", 2024-09-13T13:14:54.031 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2229000000000003e-05 2024-09-13T13:14:54.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.031 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.031 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001265+0000", 2024-09-13T13:14:54.032 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4267999999999997e-05 2024-09-13T13:14:54.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.032 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.032 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202507 9.4 9:20f0d2e1:::benchmark_data_smithi052_128427_object71838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:14:54.032 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:14:54.032 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 115.967582978, 2024-09-13T13:14:54.032 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.007409886, 2024-09-13T13:14:54.032 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.032 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.032 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.033 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:54.033 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:54.033 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:54.033 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202507 2024-09-13T13:14:54.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.033 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.033 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.033 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:14:54.033 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.033 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994016+0000", 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.001e-06 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994020+0000", 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2320000000000003e-06 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994031+0000", 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1259e-05 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994036+0000", 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3580000000000002e-06 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994089+0000", 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2120000000000002e-05 2024-09-13T13:14:54.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:14:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994097+0000", 2024-09-13T13:14:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8000000000000004e-06 2024-09-13T13:14:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979607+0000", 2024-09-13T13:14:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98551001400000005 2024-09-13T13:14:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.036 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979708+0000", 2024-09-13T13:14:54.037 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000100108 2024-09-13T13:14:54.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.037 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:14:54.037 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979813+0000", 2024-09-13T13:14:54.037 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000105238 2024-09-13T13:14:54.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.037 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.037 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981689+0000", 2024-09-13T13:14:54.037 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018764299999999999 2024-09-13T13:14:54.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987043+0000", 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053537139999999999 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001312+0000", 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014268729000000001 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001378+0000", 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5980000000000002e-05 2024-09-13T13:14:54.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.039 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.039 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001424+0000", 2024-09-13T13:14:54.039 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6903000000000001e-05 2024-09-13T13:14:54.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.039 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.039 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202513 9.4 9:293cc3ac:::benchmark_data_smithi052_128427_object71844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:14:54.039 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:14:54.039 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 115.962127036, 2024-09-13T13:14:54.040 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0035790550000001, 2024-09-13T13:14:54.040 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.040 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.040 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.040 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:54.040 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:14:54.040 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:14:54.040 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202513 2024-09-13T13:14:54.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.040 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.040 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.040 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:14:54.041 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.041 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.041 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.041 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:14:54.041 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.041 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.041 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.041 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999472+0000", 2024-09-13T13:14:54.041 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.536e-06 2024-09-13T13:14:54.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.041 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.042 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.042 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999477+0000", 2024-09-13T13:14:54.042 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7260000000000002e-06 2024-09-13T13:14:54.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.042 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.042 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999492+0000", 2024-09-13T13:14:54.042 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5495999999999999e-05 2024-09-13T13:14:54.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.042 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.042 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999500+0000", 2024-09-13T13:14:54.042 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3100000000000003e-06 2024-09-13T13:14:54.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.043 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999557+0000", 2024-09-13T13:14:54.043 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7491999999999998e-05 2024-09-13T13:14:54.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:14:54.043 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999565+0000", 2024-09-13T13:14:54.043 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9359999999999999e-06 2024-09-13T13:14:54.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.043 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.044 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981724+0000", 2024-09-13T13:14:54.044 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98215919600000001 2024-09-13T13:14:54.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.044 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.044 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.044 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981838+0000", 2024-09-13T13:14:54.044 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114092 2024-09-13T13:14:54.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.044 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.044 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:14:54.044 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981966+0000", 2024-09-13T13:14:54.044 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012748300000000001 2024-09-13T13:14:54.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.045 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.045 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987109+0000", 2024-09-13T13:14:54.045 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.005142771 2024-09-13T13:14:54.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.045 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.045 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987186+0000", 2024-09-13T13:14:54.045 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7822000000000001e-05 2024-09-13T13:14:54.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.002955+0000", 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015768880999999998 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003010+0000", 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4959e-05 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003050+0000", 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9354999999999998e-05 2024-09-13T13:14:54.046 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.047 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.047 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.047 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.047 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:255488 9.6 e328/241 9:6d339f10:::benchmark_data_smithi052_128427_object124819:head v 328'15947, mlcod=328'15942)", 2024-09-13T13:14:54.047 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:13:43.495227+0000", 2024-09-13T13:14:54.047 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 70.466370198999996, 2024-09-13T13:14:54.047 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.091692534000000006, 2024-09-13T13:14:54.047 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.047 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.047 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.047 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.048 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.048 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:43.495227+0000", 2024-09-13T13:14:54.048 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.048 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.048 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:43.495227+0000", 2024-09-13T13:14:54.048 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.048 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:43.495228+0000", 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8400000000000004e-07 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:43.495233+0000", 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8359999999999999e-06 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:43.495240+0000", 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8759999999999999e-06 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:43.495247+0000", 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5279999999999998e-06 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:43.503249+0000", 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0080019570000000005 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:43.503266+0000", 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6405999999999998e-05 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:43.586849+0000", 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.083582839000000006 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:13:43.586920+0000", 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1408000000000005e-05 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.051 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:64966 10.2 10:425d4256:::benchmark_data_smithi052_153703_object64965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T13:14:54.052 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:39.291409+0000", 2024-09-13T13:14:54.052 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 14.670188261, 2024-09-13T13:14:54.052 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11422903299999999, 2024-09-13T13:14:54.052 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.052 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.052 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.052 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:54.052 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:54.052 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 64966 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291409+0000", 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291409+0000", 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291412+0000", 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7630000000000001e-06 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291438+0000", 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5956e-05 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291454+0000", 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5515e-05 2024-09-13T13:14:54.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291463+0000", 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1279999999999997e-06 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291532+0000", 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9577000000000001e-05 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291678+0000", 2024-09-13T13:14:54.055 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014602699999999999 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291815+0000", 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013668099999999999 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.298747+0000", 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0069321410000000002 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.056 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.301675+0000", 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029274309999999999 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.405525+0000", 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.103850845 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.405587+0000", 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1931e-05 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.057 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.405638+0000", 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1038000000000002e-05 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:64968 10.2 10:48ce5cf4:::benchmark_data_smithi052_153703_object64967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:39.291547+0000", 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 14.670050514, 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.119118166, 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.058 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.059 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:54.059 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:54.059 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:54.059 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 64968 2024-09-13T13:14:54.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.059 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.059 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.059 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291547+0000", 2024-09-13T13:14:54.059 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291547+0000", 2024-09-13T13:14:54.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291549+0000", 2024-09-13T13:14:54.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7e-06 2024-09-13T13:14:54.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291572+0000", 2024-09-13T13:14:54.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3722000000000001e-05 2024-09-13T13:14:54.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291584+0000", 2024-09-13T13:14:54.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1124000000000001e-05 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291589+0000", 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5600000000000001e-06 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.293879+0000", 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0022900920000000001 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.294001+0000", 2024-09-13T13:14:54.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000121694 2024-09-13T13:14:54.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:14:54.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.294208+0000", 2024-09-13T13:14:54.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020718099999999999 2024-09-13T13:14:54.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.301629+0000", 2024-09-13T13:14:54.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.007420935 2024-09-13T13:14:54.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.301681+0000", 2024-09-13T13:14:54.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1789000000000002e-05 2024-09-13T13:14:54.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.410571+0000", 2024-09-13T13:14:54.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.108890107 2024-09-13T13:14:54.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.410615+0000", 2024-09-13T13:14:54.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4155e-05 2024-09-13T13:14:54.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.410665+0000", 2024-09-13T13:14:54.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0107e-05 2024-09-13T13:14:54.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.065 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:64970 10.2 10:4c910a20:::benchmark_data_smithi052_153703_object64969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:39.291709+0000", 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 14.66988813, 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.120818178, 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 64970 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291709+0000", 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291709+0000", 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291711+0000", 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.266e-06 2024-09-13T13:14:54.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.068 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291735+0000", 2024-09-13T13:14:54.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4283000000000001e-05 2024-09-13T13:14:54.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.068 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291745+0000", 2024-09-13T13:14:54.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0468999999999999e-05 2024-09-13T13:14:54.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.068 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291753+0000", 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4669999999999998e-06 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.296536+0000", 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047835270000000001 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.296626+0000", 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0000000000000006e-05 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:14:54.069 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.296725+0000", 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8737e-05 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.301664+0000", 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049391289999999996 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.301686+0000", 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2092000000000001e-05 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.412397+0000", 2024-09-13T13:14:54.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11071099700000001 2024-09-13T13:14:54.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.412480+0000", 2024-09-13T13:14:54.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2267000000000004e-05 2024-09-13T13:14:54.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.412528+0000", 2024-09-13T13:14:54.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7944e-05 2024-09-13T13:14:54.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:64972 10.2 10:45880529:::benchmark_data_smithi052_153703_object64971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:39.292691+0000", 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 14.668906697000001, 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.120361115, 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:54.072 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:54.073 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 64972 2024-09-13T13:14:54.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.073 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.292691+0000", 2024-09-13T13:14:54.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.292691+0000", 2024-09-13T13:14:54.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.292692+0000", 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4020000000000001e-06 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.292718+0000", 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.563e-05 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.074 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.292730+0000", 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1809e-05 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.292738+0000", 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8390000000000007e-06 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.299230+0000", 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064920289999999999 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.075 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.299400+0000", 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017011899999999999 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.299498+0000", 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8152000000000005e-05 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.301710+0000", 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0022122750000000001 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.301794+0000", 2024-09-13T13:14:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3387999999999995e-05 2024-09-13T13:14:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.412922+0000", 2024-09-13T13:14:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.111128907 2024-09-13T13:14:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.413003+0000", 2024-09-13T13:14:54.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0685000000000004e-05 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.413052+0000", 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.888e-05 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:66914 10.0 e348/334 10:022d7c38:::benchmark_data_smithi052_153703_object66913:head v 348'4227, mlcod=348'4225)", 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:40.914459+0000", 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 13.047138209, 2024-09-13T13:14:54.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.174681006, 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.914459+0000", 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.914459+0000", 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.914461+0000", 2024-09-13T13:14:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0040000000000002e-06 2024-09-13T13:14:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.914582+0000", 2024-09-13T13:14:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012074399999999999 2024-09-13T13:14:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081614+0000", 2024-09-13T13:14:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.167032352 2024-09-13T13:14:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081630+0000", 2024-09-13T13:14:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5679999999999999e-05 2024-09-13T13:14:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081702+0000", 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1418e-05 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081726+0000", 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4161000000000001e-05 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.089092+0000", 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0073658389999999999 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.089140+0000", 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8807999999999998e-05 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:66935 10.2 10:4e89de86:::benchmark_data_smithi052_153703_object66934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:40.923262+0000", 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 13.038335992, 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16507603400000001, 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.083 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 66935 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.923262+0000", 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.923262+0000", 2024-09-13T13:14:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.923264+0000", 2024-09-13T13:14:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1490000000000001e-06 2024-09-13T13:14:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.923287+0000", 2024-09-13T13:14:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3023999999999998e-05 2024-09-13T13:14:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081630+0000", 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15834292599999999 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081641+0000", 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1077000000000001e-05 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081702+0000", 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1493000000000002e-05 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081872+0000", 2024-09-13T13:14:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016986900000000001 2024-09-13T13:14:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:14:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.082195+0000", 2024-09-13T13:14:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000323228 2024-09-13T13:14:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.088213+0000", 2024-09-13T13:14:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.006017924 2024-09-13T13:14:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.088226+0000", 2024-09-13T13:14:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2969000000000001e-05 2024-09-13T13:14:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.088264+0000", 2024-09-13T13:14:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7496999999999997e-05 2024-09-13T13:14:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.088301+0000", 2024-09-13T13:14:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7370999999999998e-05 2024-09-13T13:14:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.088338+0000", 2024-09-13T13:14:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6507000000000001e-05 2024-09-13T13:14:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:81541 10.2 10:45c156a8:::benchmark_data_smithi052_153703_object81540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:52.329984+0000", 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 1.6316139810000001, 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.050153510999999998, 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 81541 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.329984+0000", 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.329984+0000", 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.329986+0000", 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3549999999999999e-06 2024-09-13T13:14:54.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.330010+0000", 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3964e-05 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.330023+0000", 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2765000000000001e-05 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.330030+0000", 2024-09-13T13:14:54.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7559999999999999e-06 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.330098+0000", 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7689999999999997e-05 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.331103+0000", 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010048329999999999 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:14:54.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.331238+0000", 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000135339 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.334659+0000", 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0034208369999999999 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.334925+0000", 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00026583200000000003 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.379740+0000", 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.044815267999999998 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.379863+0000", 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000123096 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.380137+0000", 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027377599999999999 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.095 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:81874 10.2 10:4964b06f:::benchmark_data_smithi052_153703_object81873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-13T13:14:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:52.603893+0000", 2024-09-13T13:14:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 1.357704179, 2024-09-13T13:14:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.055214794999999997, 2024-09-13T13:14:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:54.096 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 81874 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.603893+0000", 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.603893+0000", 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.603896+0000", 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5600000000000001e-06 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.603922+0000", 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5984999999999999e-05 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.603948+0000", 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6173000000000002e-05 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.603958+0000", 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4050000000000006e-06 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.604020+0000", 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1999000000000001e-05 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.604217+0000", 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000197293 2024-09-13T13:14:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.604382+0000", 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016496999999999999 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.606879+0000", 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024977010000000002 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.608094+0000", 2024-09-13T13:14:54.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012144409999999999 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.658983+0000", 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.050889549999999999 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.659040+0000", 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6999000000000001e-05 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:52.659108+0000", 2024-09-13T13:14:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7719000000000007e-05 2024-09-13T13:14:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:82476 10.2 10:427afb92:::benchmark_data_smithi052_153703_object82475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T13:14:54.102 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:53.098823+0000", 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.86277506000000004, 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.048827611, 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 82476 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:54.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.098823+0000", 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.098823+0000", 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.098825+0000", 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.554e-06 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.098848+0000", 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2677999999999999e-05 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.098860+0000", 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2678e-05 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.098871+0000", 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0441e-05 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.098942+0000", 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1278000000000004e-05 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.099044+0000", 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010144799999999999 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.099126+0000", 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2264999999999999e-05 2024-09-13T13:14:54.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:14:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.105726+0000", 2024-09-13T13:14:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.006600402 2024-09-13T13:14:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.105776+0000", 2024-09-13T13:14:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9839999999999997e-05 2024-09-13T13:14:54.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:14:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.147524+0000", 2024-09-13T13:14:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.041748167000000003 2024-09-13T13:14:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:14:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.147598+0000", 2024-09-13T13:14:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3595000000000003e-05 2024-09-13T13:14:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:14:54.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:53.147650+0000", 2024-09-13T13:14:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2265000000000002e-05 2024-09-13T13:14:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:54.109 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:54.109 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:14:54.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:54.164 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:54.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:54.223 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:53 smithi052 ceph-mon[36682]: 5.1d scrub starts 2024-09-13T13:14:54.223 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:53 smithi052 ceph-mon[36682]: 5.1d scrub ok 2024-09-13T13:14:54.223 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:53 smithi052 ceph-mon[36682]: 3.e scrub starts 2024-09-13T13:14:54.223 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:53 smithi052 ceph-mon[36682]: 3.e scrub ok 2024-09-13T13:14:54.223 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:53 smithi052 ceph-mon[36682]: osdmap e354: 8 total, 8 up, 8 in 2024-09-13T13:14:54.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:53 smithi052 ceph-mon[29471]: 5.1d scrub starts 2024-09-13T13:14:54.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:54 smithi052 ceph-mon[29471]: 5.1d scrub ok 2024-09-13T13:14:54.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:54 smithi052 ceph-mon[29471]: 3.e scrub starts 2024-09-13T13:14:54.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:54 smithi052 ceph-mon[29471]: 3.e scrub ok 2024-09-13T13:14:54.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:54 smithi052 ceph-mon[29471]: osdmap e354: 8 total, 8 up, 8 in 2024-09-13T13:14:54.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:54.301 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:54.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:54.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:54.307 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 50 16 83937 83921 104.874 76.875 0.00845824 0.0095306 2024-09-13T13:14:54.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:53 smithi167 ceph-mon[33178]: 5.1d scrub starts 2024-09-13T13:14:54.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:53 smithi167 ceph-mon[33178]: 5.1d scrub ok 2024-09-13T13:14:54.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:53 smithi167 ceph-mon[33178]: 3.e scrub starts 2024-09-13T13:14:54.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:53 smithi167 ceph-mon[33178]: 3.e scrub ok 2024-09-13T13:14:54.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:53 smithi167 ceph-mon[33178]: osdmap e354: 8 total, 8 up, 8 in 2024-09-13T13:14:54.403 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:54.451 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:54.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:54.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:54.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:54.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:54.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:54.683 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:54.723 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:54.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:54.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:14:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:14:54.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:54.858 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:54.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:54.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:54.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:54.993 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:54.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:55.033 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:55.079 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:14:55.079 INFO:teuthology.orchestra.run.smithi052.stdout: "success": "" 2024-09-13T13:14:55.079 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:14:55.095 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:55.126 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:55.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:55.228 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:55.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:55 smithi052 ceph-mon[36682]: pgmap v686: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-13T13:14:55.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:55.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:55 smithi052 ceph-mon[29471]: pgmap v686: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-13T13:14:55.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:55.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:55.265 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:55.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:55.307 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 51 16 85179 85163 104.339 77.625 0.0204493 0.00957926 2024-09-13T13:14:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:55 smithi167 ceph-mon[33178]: pgmap v686: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-13T13:14:55.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:14:55.367 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:55.401 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:55.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:55.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:55.503 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:55.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:55.540 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:55.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:55.641 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:55.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:14:55.675 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T13:14:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:55.693 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:55.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:55.795 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:55.855 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:55.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:55.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:55.957 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:55.987 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:55.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:55.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:56.089 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:56.147 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:56.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:56.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:56 smithi052 ceph-mon[29471]: 3.d scrub starts 2024-09-13T13:14:56.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:56 smithi052 ceph-mon[29471]: 3.d scrub ok 2024-09-13T13:14:56.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:14:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:14:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:14:56.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:56 smithi052 ceph-mon[36682]: 3.d scrub starts 2024-09-13T13:14:56.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:56 smithi052 ceph-mon[36682]: 3.d scrub ok 2024-09-13T13:14:56.249 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:56.289 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:56.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:56.307 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 52 16 86350 86334 103.74 73.1875 0.0233906 0.00963415 2024-09-13T13:14:56.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:56 smithi167 ceph-mon[33178]: 3.d scrub starts 2024-09-13T13:14:56.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:56 smithi167 ceph-mon[33178]: 3.d scrub ok 2024-09-13T13:14:56.391 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:56.427 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:56.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:56.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:56.529 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:56.562 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:56.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:56.663 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:56.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:56.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:56.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:56.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:56.847 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:56.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:56.948 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:56.963 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:56.963 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:14:56.975 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:56.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:56.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:57.078 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:57.122 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:57.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:57.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:57.125 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:57 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:14:57.125 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:57 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:14:57.125 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:57 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:14:57.125 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:57 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:14:57.125 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:57 smithi052 ceph-mon[29471]: pgmap v687: 148 pgs: 148 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-13T13:14:57.126 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:57 smithi052 ceph-mon[29471]: osdmap e355: 8 total, 8 up, 8 in 2024-09-13T13:14:57.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:57 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:14:57.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:57 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:14:57.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:57 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:14:57.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:57 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:14:57.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:57 smithi052 ceph-mon[36682]: pgmap v687: 148 pgs: 148 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-13T13:14:57.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:57 smithi052 ceph-mon[36682]: osdmap e355: 8 total, 8 up, 8 in 2024-09-13T13:14:57.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:57.270 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:57.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:57.308 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 53 16 87547 87531 103.194 74.8125 0.0144069 0.00968567 2024-09-13T13:14:57.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:57 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:14:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:57 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:14:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:57 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:14:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:57 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:14:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:57 smithi167 ceph-mon[33178]: pgmap v687: 148 pgs: 148 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-13T13:14:57.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:57 smithi167 ceph-mon[33178]: osdmap e355: 8 total, 8 up, 8 in 2024-09-13T13:14:57.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:57.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:57.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:57.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:57.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:57.540 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:57.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:57.641 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:57.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:57.681 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:57.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:57.782 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:57.812 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:57.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:57.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:57.914 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:57.950 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:57.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:57.982 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:14:58.053 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:58.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:58.091 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:58.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:58.192 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:14:58.215 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:58.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:58.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:14:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[36682]: 6.2 scrub starts 2024-09-13T13:14:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[36682]: 6.2 scrub ok 2024-09-13T13:14:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/346012914' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:58.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[36682]: from='client.26158 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:58.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:14:58.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:14:58.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[36682]: from='client.26158 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:14:58.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[36682]: osdmap e356: 8 total, 8 up, 8 in 2024-09-13T13:14:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[29471]: 6.2 scrub starts 2024-09-13T13:14:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[29471]: 6.2 scrub ok 2024-09-13T13:14:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/346012914' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[29471]: from='client.26158 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:14:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:14:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[29471]: from='client.26158 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:14:58.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:58 smithi052 ceph-mon[29471]: osdmap e356: 8 total, 8 up, 8 in 2024-09-13T13:14:58.307 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 54 16 88679 88663 102.594 70.75 0.0127378 0.00974213 2024-09-13T13:14:58.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:14:58.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:58 smithi167 ceph-mon[33178]: 6.2 scrub starts 2024-09-13T13:14:58.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:58 smithi167 ceph-mon[33178]: 6.2 scrub ok 2024-09-13T13:14:58.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:58 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/346012914' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:58 smithi167 ceph-mon[33178]: from='client.26158 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:14:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:58 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:14:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:58 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:14:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:58 smithi167 ceph-mon[33178]: from='client.26158 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:14:58.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:58 smithi167 ceph-mon[33178]: osdmap e356: 8 total, 8 up, 8 in 2024-09-13T13:14:58.349 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:58.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:14:58.451 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:58.488 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:58.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:58.590 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:14:58.617 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:58.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:58.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:14:58.651 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:14:58.651 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:14:58.651 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.651 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:88188 10.8 e355/334 10:11dfd8fb:::benchmark_data_smithi052_153703_object88187:head v 355'5594, mlcod=355'5592)", 2024-09-13T13:14:58.651 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:57.870395+0000", 2024-09-13T13:14:58.651 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0075669259999999999, 2024-09-13T13:14:58.651 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0076061159999999996, 2024-09-13T13:14:58.651 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:58.651 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:58.651 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:14:58.651 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:58.652 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.652 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:58.652 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.870395+0000", 2024-09-13T13:14:58.652 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:58.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.652 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.652 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:58.652 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.870395+0000", 2024-09-13T13:14:58.652 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:58.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.652 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.652 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:58.652 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.870397+0000", 2024-09-13T13:14:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.392e-06 2024-09-13T13:14:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.870423+0000", 2024-09-13T13:14:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6055999999999999e-05 2024-09-13T13:14:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.870433+0000", 2024-09-13T13:14:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0225e-05 2024-09-13T13:14:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.870440+0000", 2024-09-13T13:14:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0249999999999997e-06 2024-09-13T13:14:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.870488+0000", 2024-09-13T13:14:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8208999999999999e-05 2024-09-13T13:14:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.871002+0000", 2024-09-13T13:14:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00051337900000000001 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:88197 10.e e355/334 10:77143113:::benchmark_data_smithi052_153703_object88196:head v 355'5446, mlcod=355'5444)", 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:57.874441+0000", 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.003521293, 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036240090000000001, 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:58.655 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.874441+0000", 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.874441+0000", 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.874442+0000", 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4589999999999999e-06 2024-09-13T13:14:58.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.874473+0000", 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0380999999999999e-05 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.874483+0000", 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0305e-05 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.874490+0000", 2024-09-13T13:14:58.657 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6050000000000003e-06 2024-09-13T13:14:58.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.658 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.658 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:58.658 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.874639+0000", 2024-09-13T13:14:58.658 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014908400000000001 2024-09-13T13:14:58.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.658 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.658 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:58.658 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.874657+0000", 2024-09-13T13:14:58.658 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8318999999999999e-05 2024-09-13T13:14:58.658 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:58.658 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:58.658 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:88198 10.9 e355/334 10:91725004:::benchmark_data_smithi052_153703_object88197:head v 355'5533, mlcod=355'5531)", 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:57.877017+0000", 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.00094528600000000004, 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011020730000000001, 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.877017+0000", 2024-09-13T13:14:58.659 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:58.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.660 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:14:58.660 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.877017+0000", 2024-09-13T13:14:58.660 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:14:58.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.660 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:14:58.660 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.877018+0000", 2024-09-13T13:14:58.660 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.187e-06 2024-09-13T13:14:58.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.660 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.877054+0000", 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5809999999999998e-05 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.877068+0000", 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4399000000000001e-05 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.877075+0000", 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0319999999999998e-06 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.877125+0000", 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0008e-05 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:57.877144+0000", 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8389999999999998e-05 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:14:58.662 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 3 2024-09-13T13:14:58.663 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:14:58.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:58.755 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:58.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:58.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:14:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:58.887 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:58.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:14:58.988 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:59.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:59.012 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:59.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:59.113 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:14:59.150 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:14:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:59.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:14:59.252 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:14:59.291 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:59.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:14:59.307 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 55 16 89749 89733 101.944 66.875 0.0113391 0.00980439 2024-09-13T13:14:59.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:14:59 smithi167 ceph-mon[33178]: pgmap v689: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 718 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-13T13:14:59.394 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:59.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:59.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:59.487 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:14:59 smithi052 ceph-mon[36682]: pgmap v689: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 718 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-13T13:14:59.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:14:59 smithi052 ceph-mon[29471]: pgmap v689: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 718 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-13T13:14:59.539 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:59.579 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:59.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:59.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:59.681 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:14:59.723 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:59.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:59.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:14:59.825 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:14:59.865 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:14:59.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:14:59.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:14:59.967 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:00.004 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:00.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:00.064 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T13:15:00.106 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:00.158 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:00.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:00.260 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:00.287 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:00.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:00.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:00.307 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 56 16 90961 90945 101.476 75.75 0.00987408 0.00984927 2024-09-13T13:15:00.346 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:15:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:15:00.387 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:15:00.387 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:15:00.388 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:00.415 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:00.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:00.428 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:15:00.428 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:15:00.467 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:15:00.467 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:15:00.500 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:15:00.500 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:15:00.517 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:00.537 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:15:00.538 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:15:00.555 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:00.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:00.572 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:15:00.572 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:15:00.604 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:15:00.604 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:15:00.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:00.690 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:00.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:00.791 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:00.827 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:00.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:00.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:00.966 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:00.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:00.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:01.068 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:01.112 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:01.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:01.214 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[29471]: 4.e scrub starts 2024-09-13T13:15:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[29471]: 4.e scrub ok 2024-09-13T13:15:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[29471]: 6.3 scrub starts 2024-09-13T13:15:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[29471]: 6.3 scrub ok 2024-09-13T13:15:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[29471]: 5.5 scrub starts 2024-09-13T13:15:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[29471]: 5.5 scrub ok 2024-09-13T13:15:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[29471]: pgmap v691: 148 pgs: 148 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-13T13:15:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2650620391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:01.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[36682]: 4.e scrub starts 2024-09-13T13:15:01.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[36682]: 4.e scrub ok 2024-09-13T13:15:01.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[36682]: 6.3 scrub starts 2024-09-13T13:15:01.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[36682]: 6.3 scrub ok 2024-09-13T13:15:01.241 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[36682]: 5.5 scrub starts 2024-09-13T13:15:01.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[36682]: 5.5 scrub ok 2024-09-13T13:15:01.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[36682]: pgmap v691: 148 pgs: 148 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-13T13:15:01.242 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:01 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2650620391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:01.248 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:01.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:01.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:01.307 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 57 16 92169 92153 101.021 75.5 0.0091175 0.00989418 2024-09-13T13:15:01.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:01.386 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:01.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:01.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:01.488 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:01.524 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:01.542 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:01.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:01.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:01.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:01 smithi167 ceph-mon[33178]: 4.e scrub starts 2024-09-13T13:15:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:01 smithi167 ceph-mon[33178]: 4.e scrub ok 2024-09-13T13:15:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:01 smithi167 ceph-mon[33178]: 6.3 scrub starts 2024-09-13T13:15:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:01 smithi167 ceph-mon[33178]: 6.3 scrub ok 2024-09-13T13:15:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:01 smithi167 ceph-mon[33178]: 5.5 scrub starts 2024-09-13T13:15:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:01 smithi167 ceph-mon[33178]: 5.5 scrub ok 2024-09-13T13:15:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:01 smithi167 ceph-mon[33178]: pgmap v691: 148 pgs: 148 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-13T13:15:01.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:01 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2650620391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:01.639 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:15:01.640 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T13:15:01.644 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:01.727 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:01.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:01.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:01.830 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:01.863 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:01.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:01.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:01.966 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:01.996 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:01.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:02.098 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:02.145 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:02.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:02.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:02.235 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:15:02.235 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:15:02.247 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:02.299 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:02.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:02.312 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 58 16 93309 93293 100.501 71.25 0.00932396 0.00994523 2024-09-13T13:15:02.401 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:02.441 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:02.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:02 smithi052 ceph-mon[29471]: 5.1 scrub starts 2024-09-13T13:15:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:02 smithi052 ceph-mon[29471]: 5.1 scrub ok 2024-09-13T13:15:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:02 smithi052 ceph-mon[29471]: 5.8 scrub starts 2024-09-13T13:15:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:02 smithi052 ceph-mon[29471]: 5.8 scrub ok 2024-09-13T13:15:02.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:02 smithi052 ceph-mon[29471]: 5.13 scrub starts 2024-09-13T13:15:02.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:02 smithi052 ceph-mon[29471]: 5.13 scrub ok 2024-09-13T13:15:02.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:02 smithi052 ceph-mon[36682]: 5.1 scrub starts 2024-09-13T13:15:02.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:02 smithi052 ceph-mon[36682]: 5.1 scrub ok 2024-09-13T13:15:02.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:02 smithi052 ceph-mon[36682]: 5.8 scrub starts 2024-09-13T13:15:02.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:02 smithi052 ceph-mon[36682]: 5.8 scrub ok 2024-09-13T13:15:02.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:02 smithi052 ceph-mon[36682]: 5.13 scrub starts 2024-09-13T13:15:02.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:02 smithi052 ceph-mon[36682]: 5.13 scrub ok 2024-09-13T13:15:02.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:02.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:02 smithi167 ceph-mon[33178]: 5.1 scrub starts 2024-09-13T13:15:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:02 smithi167 ceph-mon[33178]: 5.1 scrub ok 2024-09-13T13:15:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:02 smithi167 ceph-mon[33178]: 5.8 scrub starts 2024-09-13T13:15:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:02 smithi167 ceph-mon[33178]: 5.8 scrub ok 2024-09-13T13:15:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:02 smithi167 ceph-mon[33178]: 5.13 scrub starts 2024-09-13T13:15:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:02 smithi167 ceph-mon[33178]: 5.13 scrub ok 2024-09-13T13:15:02.579 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:02.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:02.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:02.680 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:02.701 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:15:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:02.716 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:02.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:02.817 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:02.865 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:02.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:02.966 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:03.030 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:03.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:03.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:03.066 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:03.115 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T13:15:03.132 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:03.160 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:03.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:03.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:03.262 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:03.294 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:03.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:03.312 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 59 16 94434 94418 99.9888 70.3125 0.0108963 0.00999609 2024-09-13T13:15:03.382 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:03 smithi052 ceph-mon[36682]: pgmap v692: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-13T13:15:03.383 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:03 smithi052 ceph-mon[29471]: pgmap v692: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-13T13:15:03.396 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:03.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:03.452 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:03.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:03.462 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:15:03.553 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:03.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:03 smithi167 ceph-mon[33178]: pgmap v692: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-13T13:15:03.598 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:03.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:03.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:03.673 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:03.700 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:03.726 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:03.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:03.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:03.828 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:03.863 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:03.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:03.965 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:03.998 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:03.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:03.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:04.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:04.134 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:04.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:04.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:04.236 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:04.294 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:04.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:04.312 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:15:04.312872+0000 min lat: 0.0012696 max lat: 0.355388 avg lat: 0.010056 2024-09-13T13:15:04.313 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:15:04.313 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 60 16 95465 95449 99.3949 64.4375 0.0159001 0.010056 2024-09-13T13:15:04.329 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[36682]: 4.f scrub starts 2024-09-13T13:15:04.330 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[36682]: 4.f scrub ok 2024-09-13T13:15:04.330 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[36682]: 10.0 scrub starts 2024-09-13T13:15:04.330 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[36682]: 6.8 scrub starts 2024-09-13T13:15:04.330 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[36682]: 6.8 scrub ok 2024-09-13T13:15:04.330 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[36682]: 6.b scrub starts 2024-09-13T13:15:04.330 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[36682]: 6.b scrub ok 2024-09-13T13:15:04.330 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:04.330 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:15:04.330 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[29471]: 4.f scrub starts 2024-09-13T13:15:04.330 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[29471]: 4.f scrub ok 2024-09-13T13:15:04.331 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[29471]: 10.0 scrub starts 2024-09-13T13:15:04.331 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[29471]: 6.8 scrub starts 2024-09-13T13:15:04.331 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[29471]: 6.8 scrub ok 2024-09-13T13:15:04.331 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[29471]: 6.b scrub starts 2024-09-13T13:15:04.331 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[29471]: 6.b scrub ok 2024-09-13T13:15:04.331 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:04.331 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:04 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:15:04.378 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:04 smithi167 ceph-mon[33178]: 4.f scrub starts 2024-09-13T13:15:04.378 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:04 smithi167 ceph-mon[33178]: 4.f scrub ok 2024-09-13T13:15:04.378 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:04 smithi167 ceph-mon[33178]: 10.0 scrub starts 2024-09-13T13:15:04.378 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:04 smithi167 ceph-mon[33178]: 6.8 scrub starts 2024-09-13T13:15:04.379 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:04 smithi167 ceph-mon[33178]: 6.8 scrub ok 2024-09-13T13:15:04.379 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:04 smithi167 ceph-mon[33178]: 6.b scrub starts 2024-09-13T13:15:04.379 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:04 smithi167 ceph-mon[33178]: 6.b scrub ok 2024-09-13T13:15:04.379 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:04 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:04.379 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:04 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:15:04.395 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:04.433 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:04.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:04.506 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:04.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:04.563 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:04.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:04.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:04.665 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:04.709 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:04.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:04.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:04.811 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:04.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:15:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:15:04.833 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:04.857 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:04.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:04.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:04.991 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:04.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:05.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:05.094 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T13:15:05.096 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:15:05.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:05.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:05.164 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:15:05.164 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:15:05.226 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:05.228 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[29471]: 4.1b scrub starts 2024-09-13T13:15:05.228 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[29471]: 4.1b scrub ok 2024-09-13T13:15:05.228 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[29471]: 5.10 scrub starts 2024-09-13T13:15:05.228 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[29471]: 5.10 scrub ok 2024-09-13T13:15:05.228 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[29471]: pgmap v693: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-13T13:15:05.228 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4156227758' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:15:05.228 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[29471]: 3.2 scrub starts 2024-09-13T13:15:05.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[29471]: 3.2 scrub ok 2024-09-13T13:15:05.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[29471]: 10.0 scrub ok 2024-09-13T13:15:05.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:05.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/582824917' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:15:05.229 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[36682]: 4.1b scrub starts 2024-09-13T13:15:05.229 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[36682]: 4.1b scrub ok 2024-09-13T13:15:05.229 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[36682]: 5.10 scrub starts 2024-09-13T13:15:05.229 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[36682]: 5.10 scrub ok 2024-09-13T13:15:05.229 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[36682]: pgmap v693: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-13T13:15:05.229 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4156227758' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:15:05.229 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[36682]: 3.2 scrub starts 2024-09-13T13:15:05.229 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[36682]: 3.2 scrub ok 2024-09-13T13:15:05.230 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[36682]: 10.0 scrub ok 2024-09-13T13:15:05.230 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:05.230 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:05 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/582824917' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:15:05.262 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:05.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:05.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:05.313 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 61 16 96522 96506 98.8485 66.0625 0.0136661 0.010112 2024-09-13T13:15:05.364 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:05.401 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:05.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:05.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:05.503 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:05.540 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:05.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:05.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:05 smithi167 ceph-mon[33178]: 4.1b scrub starts 2024-09-13T13:15:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:05 smithi167 ceph-mon[33178]: 4.1b scrub ok 2024-09-13T13:15:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:05 smithi167 ceph-mon[33178]: 5.10 scrub starts 2024-09-13T13:15:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:05 smithi167 ceph-mon[33178]: 5.10 scrub ok 2024-09-13T13:15:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:05 smithi167 ceph-mon[33178]: pgmap v693: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-13T13:15:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:05 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4156227758' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:15:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:05 smithi167 ceph-mon[33178]: 3.2 scrub starts 2024-09-13T13:15:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:05 smithi167 ceph-mon[33178]: 3.2 scrub ok 2024-09-13T13:15:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:05 smithi167 ceph-mon[33178]: 10.0 scrub ok 2024-09-13T13:15:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:05 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/582824917' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:15:05.643 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:05.677 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:05.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:05.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:05.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:05.815 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:05.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:05.917 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:05.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:05.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:06.054 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:06.084 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:06.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:06.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:06.186 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:06.218 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:06.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:06.313 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 62 16 97754 97738 98.4961 77 0.0138187 0.010148 2024-09-13T13:15:06.320 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:06.362 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:06.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:06.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:06.413 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:15:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:15:06.413 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[36682]: 4.4 scrub starts 2024-09-13T13:15:06.414 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[36682]: 4.4 scrub ok 2024-09-13T13:15:06.414 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[36682]: 4.18 scrub starts 2024-09-13T13:15:06.414 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[36682]: 4.18 scrub ok 2024-09-13T13:15:06.414 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:06.415 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4156227758' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:15:06.415 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/582824917' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:15:06.415 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[36682]: osdmap e357: 8 total, 8 up, 8 in 2024-09-13T13:15:06.415 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[29471]: 4.4 scrub starts 2024-09-13T13:15:06.415 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[29471]: 4.4 scrub ok 2024-09-13T13:15:06.416 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[29471]: 4.18 scrub starts 2024-09-13T13:15:06.416 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[29471]: 4.18 scrub ok 2024-09-13T13:15:06.416 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:06.416 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4156227758' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:15:06.416 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/582824917' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:15:06.416 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:06 smithi052 ceph-mon[29471]: osdmap e357: 8 total, 8 up, 8 in 2024-09-13T13:15:06.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:06.498 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:06.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:06.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:06.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:06 smithi167 ceph-mon[33178]: 4.4 scrub starts 2024-09-13T13:15:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:06 smithi167 ceph-mon[33178]: 4.4 scrub ok 2024-09-13T13:15:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:06 smithi167 ceph-mon[33178]: 4.18 scrub starts 2024-09-13T13:15:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:06 smithi167 ceph-mon[33178]: 4.18 scrub ok 2024-09-13T13:15:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:06 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:06 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4156227758' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:15:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:06 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/582824917' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:15:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:06 smithi167 ceph-mon[33178]: osdmap e357: 8 total, 8 up, 8 in 2024-09-13T13:15:06.600 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:06.637 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:06.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:06.739 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:06.775 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:06.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:06.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:06.877 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:06.908 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:06.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:06.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:07.010 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:07.054 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:07.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:07.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:07.156 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:07.204 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T13:15:07.211 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:07.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:07.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:07.313 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 63 16 98991 98975 98.1598 77.3125 0.0285202 0.0101821 2024-09-13T13:15:07.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:07.341 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:07.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:07.421 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:07 smithi052 ceph-mon[36682]: pgmap v695: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-13T13:15:07.421 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:07 smithi052 ceph-mon[29471]: pgmap v695: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-13T13:15:07.442 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:07.474 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:07.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:07.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:07.532 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:15:07.532 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:15:07.566 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:15:07.566 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:15:07.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:07 smithi167 ceph-mon[33178]: pgmap v695: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-13T13:15:07.575 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:07.600 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:15:07.600 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:15:07.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:07.621 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:07.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:07.632 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:15:07.633 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:15:07.673 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:15:07.674 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:15:07.708 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:15:07.709 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:15:07.721 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:07.748 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:15:07.748 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:15:07.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:07.763 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:07.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:07.785 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:15:07.785 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:15:07.865 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:07.902 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:07.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:08.003 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:08.046 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:08.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:08.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:08.148 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:08.201 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:08.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:08.211 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:15:08.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:08 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1497538180' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:08.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:08 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1497538180' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:08.243 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T13:15:08.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:08.315 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 64 16 100067 100051 97.6742 67.25 0.0142611 0.010233 2024-09-13T13:15:08.331 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:08.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:08.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:08.433 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:08.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:08.491 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:08.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:08.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:08 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1497538180' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:08.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:08.624 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:08.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:08.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:08.726 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:08.769 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:08.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:08.870 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:08.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:08.903 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:08.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:09.004 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:09.045 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:09.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:09.146 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:09.187 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:09.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:09.289 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:09.308 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:09.317 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 65 16 101178 101162 97.2377 69.4375 0.0193597 0.0102784 2024-09-13T13:15:09.348 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:09.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:09.354 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:09.394 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:09.449 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:09 smithi052 ceph-mon[29471]: pgmap v696: 148 pgs: 148 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-13T13:15:09.449 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:09 smithi052 ceph-mon[36682]: pgmap v696: 148 pgs: 148 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-13T13:15:09.450 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:09.484 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:09.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:09.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:09 smithi167 ceph-mon[33178]: pgmap v696: 148 pgs: 148 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-13T13:15:09.586 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:09.621 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:09.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:09.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:09.723 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:09.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:09.760 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:09.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:09.861 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:09.904 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:09.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:09.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:10.006 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:10.049 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:10.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:10.150 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:10.195 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:10.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:10.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:10.297 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:10.316 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 66 16 102253 102237 96.7822 67.1875 0.00819234 0.0103269 2024-09-13T13:15:10.337 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:10.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:10.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:10.353 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:15:10.353 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:15:10.353 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.353 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:102116 10.f e357/334 10:f218bf35:::benchmark_data_smithi052_153703_object102115:head v 357'6322, mlcod=357'6319)", 2024-09-13T13:15:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:10.172734+0000", 2024-09-13T13:15:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.022400333000000001, 2024-09-13T13:15:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.022428389, 2024-09-13T13:15:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:15:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:15:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:15:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:15:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:15:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.172734+0000", 2024-09-13T13:15:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:15:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.172734+0000", 2024-09-13T13:15:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:15:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.172737+0000", 2024-09-13T13:15:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.604e-06 2024-09-13T13:15:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:15:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.172794+0000", 2024-09-13T13:15:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7355999999999998e-05 2024-09-13T13:15:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:15:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.172814+0000", 2024-09-13T13:15:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0169999999999998e-05 2024-09-13T13:15:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:15:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.172827+0000", 2024-09-13T13:15:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2067e-05 2024-09-13T13:15:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:15:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.173080+0000", 2024-09-13T13:15:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00025355100000000003 2024-09-13T13:15:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:15:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.173147+0000", 2024-09-13T13:15:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6879999999999997e-05 2024-09-13T13:15:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:15:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.358 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.358 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:102104 10.5 e357/334 10:a584a43d:::benchmark_data_smithi052_153703_object102103:head v 357'6390, mlcod=357'6385)", 2024-09-13T13:15:10.358 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:10.176908+0000", 2024-09-13T13:15:10.358 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.018226618, 2024-09-13T13:15:10.358 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.018312166000000001, 2024-09-13T13:15:10.358 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:15:10.358 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:15:10.359 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:15:10.359 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:15:10.359 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.359 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:15:10.359 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.176908+0000", 2024-09-13T13:15:10.359 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.359 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:15:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.176908+0000", 2024-09-13T13:15:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:15:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.176910+0000", 2024-09-13T13:15:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5430000000000001e-06 2024-09-13T13:15:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:15:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.176946+0000", 2024-09-13T13:15:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6445000000000001e-05 2024-09-13T13:15:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:15:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.176953+0000", 2024-09-13T13:15:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6229999999999999e-06 2024-09-13T13:15:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:15:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.176958+0000", 2024-09-13T13:15:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1170000000000003e-06 2024-09-13T13:15:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:15:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.177588+0000", 2024-09-13T13:15:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00063066299999999995 2024-09-13T13:15:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:15:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.177646+0000", 2024-09-13T13:15:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7253000000000002e-05 2024-09-13T13:15:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:15:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:102122 10.f e357/334 10:f23c37c1:::benchmark_data_smithi052_153703_object102121:head v 357'6323, mlcod=357'6320)", 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:10.184908+0000", 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.010226948, 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010371016, 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.184908+0000", 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:15:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.184908+0000", 2024-09-13T13:15:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:15:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.184910+0000", 2024-09-13T13:15:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9010000000000001e-06 2024-09-13T13:15:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:15:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.185193+0000", 2024-09-13T13:15:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00028337499999999999 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.185209+0000", 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6351e-05 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.185218+0000", 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.473e-06 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.185424+0000", 2024-09-13T13:15:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000206026 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.185496+0000", 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1979999999999999e-05 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:102126 10.3 e357/334 10:c6ee7720:::benchmark_data_smithi052_153703_object102125:head v 357'6424, mlcod=357'6420)", 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:10.191177+0000", 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0039574240000000002, 2024-09-13T13:15:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041512670000000002, 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191177+0000", 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191177+0000", 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:15:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191185+0000", 2024-09-13T13:15:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6960000000000005e-06 2024-09-13T13:15:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:15:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191214+0000", 2024-09-13T13:15:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9544e-05 2024-09-13T13:15:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:15:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191233+0000", 2024-09-13T13:15:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8578999999999999e-05 2024-09-13T13:15:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:15:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191242+0000", 2024-09-13T13:15:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9490000000000002e-06 2024-09-13T13:15:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:15:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191375+0000", 2024-09-13T13:15:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013307999999999999 2024-09-13T13:15:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191403+0000", 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8104999999999999e-05 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:102120 10.4 e357/334 10:2be86339:::benchmark_data_smithi052_153703_object102119:head v 357'6277, mlcod=357'6275)", 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:10.191325+0000", 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0038093369999999999, 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0040538140000000002, 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:15:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:15:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:15:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:15:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:15:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191325+0000", 2024-09-13T13:15:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:15:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191325+0000", 2024-09-13T13:15:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:15:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191327+0000", 2024-09-13T13:15:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3120000000000001e-06 2024-09-13T13:15:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:15:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191350+0000", 2024-09-13T13:15:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3207999999999999e-05 2024-09-13T13:15:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:15:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191361+0000", 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1014e-05 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191368+0000", 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7950000000000002e-06 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191554+0000", 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000186443 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:15:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.191576+0000", 2024-09-13T13:15:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1642999999999999e-05 2024-09-13T13:15:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:15:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:102125 10.f e357/334)", 2024-09-13T13:15:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:10.192441+0000", 2024-09-13T13:15:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.002693455, 2024-09-13T13:15:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002984708, 2024-09-13T13:15:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:15:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:15:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:15:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:15:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:15:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.192441+0000", 2024-09-13T13:15:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:15:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.192441+0000", 2024-09-13T13:15:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:15:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.192443+0000", 2024-09-13T13:15:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.53e-06 2024-09-13T13:15:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:15:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.192482+0000", 2024-09-13T13:15:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9591000000000001e-05 2024-09-13T13:15:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:15:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.192496+0000", 2024-09-13T13:15:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3611000000000001e-05 2024-09-13T13:15:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:15:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.192504+0000", 2024-09-13T13:15:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1480000000000006e-06 2024-09-13T13:15:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:15:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:102129 10.f e357/334)", 2024-09-13T13:15:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:10.192518+0000", 2024-09-13T13:15:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.002616813, 2024-09-13T13:15:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029461510000000002, 2024-09-13T13:15:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:15:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:15:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:15:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:15:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:15:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.192518+0000", 2024-09-13T13:15:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:15:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.192518+0000", 2024-09-13T13:15:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:15:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.192519+0000", 2024-09-13T13:15:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8299999999999996e-07 2024-09-13T13:15:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:15:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.192549+0000", 2024-09-13T13:15:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0630000000000003e-05 2024-09-13T13:15:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:15:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.192557+0000", 2024-09-13T13:15:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3760000000000002e-06 2024-09-13T13:15:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:15:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.192562+0000", 2024-09-13T13:15:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7749999999999998e-06 2024-09-13T13:15:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:15:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:102098 10.5 e357/334 10:a94bdfab:::benchmark_data_smithi052_153703_object102097:head v 357'6389, mlcod=357'6385)", 2024-09-13T13:15:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:10.163757+0000", 2024-09-13T13:15:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.031377718999999998, 2024-09-13T13:15:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.031749343999999999, 2024-09-13T13:15:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:15:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:15:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:15:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:15:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:15:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.163757+0000", 2024-09-13T13:15:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:15:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.163757+0000", 2024-09-13T13:15:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:15:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:15:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.163759+0000", 2024-09-13T13:15:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.795e-06 2024-09-13T13:15:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:15:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.163788+0000", 2024-09-13T13:15:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9723e-05 2024-09-13T13:15:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:15:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.163798+0000", 2024-09-13T13:15:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6849999999999996e-06 2024-09-13T13:15:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:15:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.163807+0000", 2024-09-13T13:15:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0920000000000004e-06 2024-09-13T13:15:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:15:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.164041+0000", 2024-09-13T13:15:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00023375199999999999 2024-09-13T13:15:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:15:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:15:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:15:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:10.164090+0000", 2024-09-13T13:15:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9495e-05 2024-09-13T13:15:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:15:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:15:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:15:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 8 2024-09-13T13:15:10.385 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:15:10.440 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:10.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:10.472 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:10.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:10.573 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:10.615 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:10.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:10.717 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:10.754 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:10.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:10.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:10.855 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:10.894 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:10.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:10.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:10.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:11.026 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:11.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:11.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:11.128 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:11.159 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:11.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:11.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:11.172 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:15:11.187 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:15:11.220 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:11 smithi052 ceph-mon[36682]: pgmap v697: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T13:15:11.220 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2330480057' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:11.220 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:11 smithi052 ceph-mon[36682]: from='client.36224 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:11.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:11 smithi052 ceph-mon[29471]: pgmap v697: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T13:15:11.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2330480057' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:11.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:11 smithi052 ceph-mon[29471]: from='client.36224 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:11.262 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:11.294 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:11.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:11.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:11.316 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 67 16 103374 103358 96.3834 70.0625 0.0135783 0.0103706 2024-09-13T13:15:11.396 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:11.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:11.448 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:11.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:11.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:11 smithi167 ceph-mon[33178]: pgmap v697: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T13:15:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2330480057' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:11 smithi167 ceph-mon[33178]: from='client.36224 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:11.571 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T13:15:11.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:11.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:11.684 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:11.739 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:11.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:11.841 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:11.886 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:11.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:11.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:11.987 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:12.048 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:12.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:12.150 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:12.186 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:12.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:12.198 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T13:15:12.288 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:12.317 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 68 16 104383 104367 95.8935 63.0625 0.0218139 0.0104235 2024-09-13T13:15:12.332 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:12.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:12.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:12.434 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:12.472 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:12.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:12.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:12 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:12.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:12 smithi052 ceph-mon[36682]: from='client.36224 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:15:12.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:12 smithi052 ceph-mon[36682]: osdmap e358: 8 total, 8 up, 8 in 2024-09-13T13:15:12.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:12 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/717408143' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:15:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:12 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:12 smithi052 ceph-mon[29471]: from='client.36224 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:15:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:12 smithi052 ceph-mon[29471]: osdmap e358: 8 total, 8 up, 8 in 2024-09-13T13:15:12.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:12 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/717408143' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:15:12.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:12 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:12 smithi167 ceph-mon[33178]: from='client.36224 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:15:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:12 smithi167 ceph-mon[33178]: osdmap e358: 8 total, 8 up, 8 in 2024-09-13T13:15:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:12 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/717408143' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:15:12.574 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:12.618 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:12.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:12.720 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:12.756 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:12.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:12.860 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:12.891 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:12.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:12.993 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:13.027 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:13.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:13.129 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:13.166 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:13.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:13.268 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:13.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:13.297 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:13.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:13.312 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:13.318 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 69 16 105593 105577 95.5983 75.625 0.014137 0.0104554 2024-09-13T13:15:13.330 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T13:15:13.373 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:13.398 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:13.460 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:13.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:13.463 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:13 smithi052 ceph-mon[36682]: pgmap v699: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-13T13:15:13.463 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/717408143' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:15:13.463 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:13 smithi052 ceph-mon[36682]: osdmap e359: 8 total, 8 up, 8 in 2024-09-13T13:15:13.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:13 smithi052 ceph-mon[29471]: pgmap v699: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-13T13:15:13.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/717408143' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:15:13.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:13 smithi052 ceph-mon[29471]: osdmap e359: 8 total, 8 up, 8 in 2024-09-13T13:15:13.561 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:13 smithi167 ceph-mon[33178]: pgmap v699: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-13T13:15:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/717408143' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:15:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:13 smithi167 ceph-mon[33178]: osdmap e359: 8 total, 8 up, 8 in 2024-09-13T13:15:13.592 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:13.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:13.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:13.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:13.725 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:13.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:13.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:13.827 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:13.877 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:13.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:13.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:13.978 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:14.031 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:14.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:14.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:14.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:14.176 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:14.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:14.278 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:14.310 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:14.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:14.320 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 70 15 106725 106710 95.2442 70.8125 0.0076857 0.0104941 2024-09-13T13:15:14.411 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:14.446 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:14.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:14.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:14.466 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:15:14.467 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:15:14.509 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:15:14.510 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:15:14.548 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:14.550 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:15:14.550 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:15:14.578 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:14.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:14.585 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:15:14.586 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:15:14.602 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:15:14.623 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:15:14.623 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:15:14.663 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:15:14.663 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:15:14.681 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:14.705 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:15:14.706 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:15:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:14.722 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:14.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:14.739 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:15:14.740 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:15:14.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:15:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:15:14.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:14.855 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:14.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:14.957 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:15.000 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:15.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:15.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:15.102 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:15.126 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:15.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:15.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:15.228 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:15.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:15.266 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:15.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:15.318 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 71 16 107796 107780 94.8446 66.875 0.010105 0.0105389 2024-09-13T13:15:15.366 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:15.394 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:15.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:15 smithi052 ceph-mon[36682]: pgmap v701: 148 pgs: 148 active+clean; 6.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-09-13T13:15:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:15 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/89026751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:15 smithi052 ceph-mon[29471]: pgmap v701: 148 pgs: 148 active+clean; 6.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-09-13T13:15:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:15 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/89026751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:15.497 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:15.539 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:15.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:15.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:15 smithi167 ceph-mon[33178]: pgmap v701: 148 pgs: 148 active+clean; 6.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-09-13T13:15:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:15 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/89026751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:15.641 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:15.671 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:15.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:15.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:15.773 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:15.802 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:15.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:15.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:15.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:15.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:15.938 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:15.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:16.038 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:16.074 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:16.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:16.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:16.176 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:16.208 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:16.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:16.310 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:16.319 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 72 16 108952 108936 94.5296 72.25 0.0105364 0.0105741 2024-09-13T13:15:16.345 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:16.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:16.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:16.447 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:16.486 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:16.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:16.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:16 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:15:16.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:16 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:15:16.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:16 smithi052 ceph-mon[36682]: osdmap e360: 8 total, 8 up, 8 in 2024-09-13T13:15:16.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:16 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:15:16.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:16 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:15:16.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:16 smithi052 ceph-mon[29471]: osdmap e360: 8 total, 8 up, 8 in 2024-09-13T13:15:16.490 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:15:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:15:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:15:16.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:16 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:15:16.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:16 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:15:16.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:16 smithi167 ceph-mon[33178]: osdmap e360: 8 total, 8 up, 8 in 2024-09-13T13:15:16.589 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:16.624 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:15:16.624 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:15:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:16.627 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:16.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:16.728 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:16.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:16.775 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:16.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:16.876 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:16.914 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:16.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:16.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:17.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:17.049 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:17.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:17.151 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:17.193 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:17.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:17.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:17.319 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 73 16 110106 110090 94.2227 72.125 0.0098763 0.0106082 2024-09-13T13:15:17.332 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:17.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:17.434 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:17.465 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:17.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:17 smithi052 ceph-mon[36682]: pgmap v703: 148 pgs: 148 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 923 op/s 2024-09-13T13:15:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:17 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:15:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:17 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:15:17.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:17 smithi052 ceph-mon[36682]: osdmap e361: 8 total, 8 up, 8 in 2024-09-13T13:15:17.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:17 smithi052 ceph-mon[29471]: pgmap v703: 148 pgs: 148 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 923 op/s 2024-09-13T13:15:17.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:17 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:15:17.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:17 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:15:17.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:17 smithi052 ceph-mon[29471]: osdmap e361: 8 total, 8 up, 8 in 2024-09-13T13:15:17.566 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:17 smithi167 ceph-mon[33178]: pgmap v703: 148 pgs: 148 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 58 MiB/s wr, 923 op/s 2024-09-13T13:15:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:17 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:15:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:17 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:15:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:17 smithi167 ceph-mon[33178]: osdmap e361: 8 total, 8 up, 8 in 2024-09-13T13:15:17.599 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:17.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:17.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:17.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:17.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:17.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:17.841 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:17.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:17.880 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:17.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:17.981 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:18.014 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:18.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:18.116 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:18.121 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T13:15:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:18.150 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:18.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:18.251 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:18.299 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:18.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:18.321 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 74 16 111297 111281 93.9545 74.4375 0.00855367 0.0106387 2024-09-13T13:15:18.401 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:18.432 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:18.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:18.534 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:18.575 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:18.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:18.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:18.677 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:18.709 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:18.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:18.811 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:18.848 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:18.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:18.950 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:18.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:18.982 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:18.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:18.997 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:19.004 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:19.057 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:19.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:19.118 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:19.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:19.220 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:19.258 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:19.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:19.320 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 75 16 112436 112420 93.6509 71.1875 0.0079609 0.010673 2024-09-13T13:15:19.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:19.395 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:19.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:19.497 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:19.538 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:19.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:19.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:19.640 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:19.679 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:19.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:19.781 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:19 smithi052 ceph-mon[29471]: pgmap v705: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 921 op/s 2024-09-13T13:15:19.781 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:19 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:19.781 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:19 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:15:19.782 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:19 smithi052 ceph-mon[36682]: pgmap v705: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 921 op/s 2024-09-13T13:15:19.782 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:19 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:19.782 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:19 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:15:19.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:19.813 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:19.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:19.915 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:19.957 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:19.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:20.058 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:20.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:19 smithi167 ceph-mon[33178]: pgmap v705: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 921 op/s 2024-09-13T13:15:20.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:19 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:20.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:19 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:15:20.103 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:20.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:20.205 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:20.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:20.233 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:20.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:20.320 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 76 16 113526 113510 93.315 68.125 0.021205 0.010711 2024-09-13T13:15:20.334 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:20.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:20.383 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:20.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:20.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:20.513 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:20.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:20.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:20.614 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:20.644 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:15:20.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:20.655 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:20.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:20.659 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T13:15:20.757 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:20.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:20.791 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:20.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:20.862 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:20 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/485069407' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:15:20.862 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:20 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:15:20.862 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:20 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1361578878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:15:20.862 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:20 smithi052 ceph-mon[29471]: from='client.36248 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:15:20.863 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:20 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/485069407' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:15:20.863 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:20 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:15:20.863 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:20 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1361578878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:15:20.863 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:20 smithi052 ceph-mon[36682]: from='client.36248 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:15:20.892 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:20.939 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:15:20.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:20.946 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:20.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:21.029 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:15:21.029 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:15:21.047 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:21.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:20 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/485069407' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:15:21.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:20 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:15:21.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:20 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1361578878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:15:21.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:20 smithi167 ceph-mon[33178]: from='client.36248 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:15:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:21.118 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:21.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:21.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:21.257 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:21.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:21.320 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 77 16 114506 114490 92.8986 61.25 0.00896019 0.0107597 2024-09-13T13:15:21.357 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:21.391 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:21.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:21.492 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:21.514 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:21.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:21.615 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:21.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:21.653 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:21.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:21.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:21 smithi052 ceph-mon[36682]: pgmap v706: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-09-13T13:15:21.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:21 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:21.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:21 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/485069407' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:15:21.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:21 smithi052 ceph-mon[36682]: from='client.36248 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:15:21.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:21 smithi052 ceph-mon[36682]: osdmap e362: 8 total, 8 up, 8 in 2024-09-13T13:15:21.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:21 smithi052 ceph-mon[29471]: pgmap v706: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-09-13T13:15:21.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:21 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:21.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:21 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/485069407' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:15:21.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:21 smithi052 ceph-mon[29471]: from='client.36248 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:15:21.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:21 smithi052 ceph-mon[29471]: osdmap e362: 8 total, 8 up, 8 in 2024-09-13T13:15:21.754 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:21.791 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:21.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:21.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:21.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:21.924 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:21.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:22.026 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:22.061 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:22.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:22.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:21 smithi167 ceph-mon[33178]: pgmap v706: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-09-13T13:15:22.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:21 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:22.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:21 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/485069407' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:15:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:21 smithi167 ceph-mon[33178]: from='client.36248 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:15:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:21 smithi167 ceph-mon[33178]: osdmap e362: 8 total, 8 up, 8 in 2024-09-13T13:15:22.162 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:22.195 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:22.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:22.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:22.297 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:22.320 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 78 16 115529 115513 92.5274 63.9375 0.00718846 0.0107994 2024-09-13T13:15:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:22.325 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:22.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:22.426 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:22.462 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:22.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:22.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:22.565 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:22.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:22.603 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:22.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:22.704 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:22.739 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:22.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:22.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:22.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:22.880 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:22.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:22.982 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:23.045 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:23.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:23.150 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:23.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:23.190 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:23.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:23.291 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:23.320 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 79 16 116665 116649 92.2549 71 0.00604991 0.0108349 2024-09-13T13:15:23.321 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:23.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:23.423 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:23.458 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:23.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:23.559 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:23.591 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:23.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:23.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:23.693 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:23.724 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:23.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:23.826 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:23.858 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:23.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:23.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:23.930 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:23 smithi052 ceph-mon[36682]: pgmap v708: 148 pgs: 148 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 765 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-13T13:15:23.930 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:23 smithi052 ceph-mon[36682]: 6.16 scrub starts 2024-09-13T13:15:23.930 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:23 smithi052 ceph-mon[36682]: 6.16 scrub ok 2024-09-13T13:15:23.930 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:23 smithi052 ceph-mon[36682]: 10.d scrub starts 2024-09-13T13:15:23.931 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:23 smithi052 ceph-mon[36682]: 4.17 scrub starts 2024-09-13T13:15:23.931 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:23 smithi052 ceph-mon[36682]: 4.17 scrub ok 2024-09-13T13:15:23.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:23 smithi052 ceph-mon[29471]: pgmap v708: 148 pgs: 148 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 765 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-13T13:15:23.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:23 smithi052 ceph-mon[29471]: 6.16 scrub starts 2024-09-13T13:15:23.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:23 smithi052 ceph-mon[29471]: 6.16 scrub ok 2024-09-13T13:15:23.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:23 smithi052 ceph-mon[29471]: 10.d scrub starts 2024-09-13T13:15:23.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:23 smithi052 ceph-mon[29471]: 4.17 scrub starts 2024-09-13T13:15:23.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:23 smithi052 ceph-mon[29471]: 4.17 scrub ok 2024-09-13T13:15:23.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:23.971 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:23.986 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T13:15:23.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:23.988 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:23.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:24.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:15:24.067 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:23 smithi167 ceph-mon[33178]: pgmap v708: 148 pgs: 148 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 765 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-13T13:15:24.068 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:23 smithi167 ceph-mon[33178]: 6.16 scrub starts 2024-09-13T13:15:24.068 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:23 smithi167 ceph-mon[33178]: 6.16 scrub ok 2024-09-13T13:15:24.068 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:23 smithi167 ceph-mon[33178]: 10.d scrub starts 2024-09-13T13:15:24.068 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:23 smithi167 ceph-mon[33178]: 4.17 scrub starts 2024-09-13T13:15:24.068 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:23 smithi167 ceph-mon[33178]: 4.17 scrub ok 2024-09-13T13:15:24.088 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:24.138 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:24.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:24.240 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:24.289 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:24.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:24.322 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:15:24.321816+0000 min lat: 0.0012696 max lat: 0.355388 avg lat: 0.0108745 2024-09-13T13:15:24.322 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:15:24.322 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 80 16 117700 117684 91.9088 64.6875 0.0146962 0.0108745 2024-09-13T13:15:24.391 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:24.444 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:24.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:24.546 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:24.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:24.619 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:24.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:24.650 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:15:24.651 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:15:24.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:15:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:15:24.684 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:15:24.684 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:15:24.691 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:15:24.712 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:15:24.713 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:15:24.720 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:24.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[36682]: 3.19 scrub starts 2024-09-13T13:15:24.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[36682]: 3.19 scrub ok 2024-09-13T13:15:24.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[36682]: 5.a scrub starts 2024-09-13T13:15:24.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[36682]: 5.a scrub ok 2024-09-13T13:15:24.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[36682]: 3.a scrub starts 2024-09-13T13:15:24.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[36682]: 3.a scrub ok 2024-09-13T13:15:24.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/555079033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:24.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[29471]: 3.19 scrub starts 2024-09-13T13:15:24.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[29471]: 3.19 scrub ok 2024-09-13T13:15:24.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[29471]: 5.a scrub starts 2024-09-13T13:15:24.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[29471]: 5.a scrub ok 2024-09-13T13:15:24.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[29471]: 3.a scrub starts 2024-09-13T13:15:24.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[29471]: 3.a scrub ok 2024-09-13T13:15:24.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:24 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/555079033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:24.743 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:15:24.743 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:15:24.759 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:24.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:24.783 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:15:24.783 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:15:24.822 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:15:24.823 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:15:24.861 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:24.865 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:15:24.865 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:15:24.899 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:15:24.899 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:15:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:24.908 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:24.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:24.958 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:24 smithi167 ceph-mon[33178]: 3.19 scrub starts 2024-09-13T13:15:24.959 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:24 smithi167 ceph-mon[33178]: 3.19 scrub ok 2024-09-13T13:15:24.959 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:24 smithi167 ceph-mon[33178]: 5.a scrub starts 2024-09-13T13:15:24.959 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:24 smithi167 ceph-mon[33178]: 5.a scrub ok 2024-09-13T13:15:24.959 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:24 smithi167 ceph-mon[33178]: 3.a scrub starts 2024-09-13T13:15:24.959 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:24 smithi167 ceph-mon[33178]: 3.a scrub ok 2024-09-13T13:15:24.959 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:24 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/555079033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:25.009 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:25.049 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:25.050 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:15:25.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:25.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:25.151 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:25.204 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:25.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:25.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:25.305 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:25.321 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 81 16 118689 118673 91.5372 61.8125 0.00798934 0.0109197 2024-09-13T13:15:25.337 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:25.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:25.439 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:25.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:25.508 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:25.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:25.609 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:25.648 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:25.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:25.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:25.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:25 smithi167 ceph-mon[33178]: pgmap v709: 148 pgs: 148 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-13T13:15:25.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:25 smithi167 ceph-mon[33178]: 3.1e scrub starts 2024-09-13T13:15:25.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:25 smithi167 ceph-mon[33178]: 3.1e scrub ok 2024-09-13T13:15:25.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:25 smithi167 ceph-mon[33178]: 5.c scrub starts 2024-09-13T13:15:25.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:25 smithi167 ceph-mon[33178]: 5.c scrub ok 2024-09-13T13:15:25.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:25.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:25.807 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:25.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:25.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:25.901 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:15:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:15:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:119301 10.e 10:7a550843:::benchmark_data_smithi052_153703_object119300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-09-13T13:15:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:25.892631+0000", 2024-09-13T13:15:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0072917169999999996, 2024-09-13T13:15:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0073222779999999998, 2024-09-13T13:15:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:15:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 119301 2024-09-13T13:15:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.892631+0000", 2024-09-13T13:15:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.892631+0000", 2024-09-13T13:15:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.892633+0000", 2024-09-13T13:15:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8509999999999999e-06 2024-09-13T13:15:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.892696+0000", 2024-09-13T13:15:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3094000000000003e-05 2024-09-13T13:15:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.892713+0000", 2024-09-13T13:15:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7054999999999998e-05 2024-09-13T13:15:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.892722+0000", 2024-09-13T13:15:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7839999999999992e-06 2024-09-13T13:15:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.894617+0000", 2024-09-13T13:15:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001895355 2024-09-13T13:15:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.894751+0000", 2024-09-13T13:15:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013366699999999999 2024-09-13T13:15:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,6", 2024-09-13T13:15:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.894891+0000", 2024-09-13T13:15:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000139565 2024-09-13T13:15:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.897561+0000", 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026703740000000001 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.898586+0000", 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010246719999999999 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:119305 10.6 e362/334 10:63e45749:::benchmark_data_smithi052_153703_object119304:head v 362'7508, mlcod=362'7505)", 2024-09-13T13:15:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:25.898943+0000", 2024-09-13T13:15:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0009796099999999999, 2024-09-13T13:15:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00108658, 2024-09-13T13:15:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:15:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.898943+0000", 2024-09-13T13:15:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.898943+0000", 2024-09-13T13:15:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.898945+0000", 2024-09-13T13:15:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1540000000000001e-06 2024-09-13T13:15:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.898969+0000", 2024-09-13T13:15:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4669999999999999e-05 2024-09-13T13:15:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.898982+0000", 2024-09-13T13:15:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2274e-05 2024-09-13T13:15:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.898988+0000", 2024-09-13T13:15:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5540000000000003e-06 2024-09-13T13:15:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.899084+0000", 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5890000000000005e-05 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.899101+0000", 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6820000000000002e-05 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:119313 10.e 10:7f99b742:::benchmark_data_smithi052_153703_object119312:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-09-13T13:15:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:25.899380+0000", 2024-09-13T13:15:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00054289900000000001, 2024-09-13T13:15:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070035900000000001, 2024-09-13T13:15:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:15:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 119313 2024-09-13T13:15:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.899380+0000", 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.899380+0000", 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.899382+0000", 2024-09-13T13:15:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.88e-06 2024-09-13T13:15:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.899428+0000", 2024-09-13T13:15:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6366000000000003e-05 2024-09-13T13:15:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.899441+0000", 2024-09-13T13:15:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2339e-05 2024-09-13T13:15:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.899447+0000", 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6499999999999999e-06 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.899547+0000", 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9502999999999999e-05 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.899666+0000", 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118663 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,6", 2024-09-13T13:15:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:25.899803+0000", 2024-09-13T13:15:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000137284 2024-09-13T13:15:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:15:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 3 2024-09-13T13:15:25.916 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:15:25.916 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:25.953 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:25.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:25.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:25 smithi052 ceph-mon[29471]: pgmap v709: 148 pgs: 148 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-13T13:15:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:25 smithi052 ceph-mon[29471]: 3.1e scrub starts 2024-09-13T13:15:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:25 smithi052 ceph-mon[29471]: 3.1e scrub ok 2024-09-13T13:15:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:25 smithi052 ceph-mon[29471]: 5.c scrub starts 2024-09-13T13:15:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:25 smithi052 ceph-mon[29471]: 5.c scrub ok 2024-09-13T13:15:25.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:25 smithi052 ceph-mon[36682]: pgmap v709: 148 pgs: 148 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-13T13:15:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:25 smithi052 ceph-mon[36682]: 3.1e scrub starts 2024-09-13T13:15:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:25 smithi052 ceph-mon[36682]: 3.1e scrub ok 2024-09-13T13:15:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:25 smithi052 ceph-mon[36682]: 5.c scrub starts 2024-09-13T13:15:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:25 smithi052 ceph-mon[36682]: 5.c scrub ok 2024-09-13T13:15:25.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:26.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:26.084 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:26.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:26.173 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:26.187 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:26.212 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:26.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:26.258 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:26.314 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:26.323 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 82 16 119735 119719 91.2169 65.375 0.019687 0.0109575 2024-09-13T13:15:26.340 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:15:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:15:26.357 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:26.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:26.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:26.459 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:26.505 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:26.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:26.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:26.657 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:26.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:26.759 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:26.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:26.803 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:26.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:26.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T13:15:26.904 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:26.939 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:26.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:26.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[36682]: 5.17 scrub starts 2024-09-13T13:15:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[36682]: 5.17 scrub ok 2024-09-13T13:15:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[36682]: 10.d scrub ok 2024-09-13T13:15:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[36682]: 6.18 scrub starts 2024-09-13T13:15:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[36682]: 6.18 scrub ok 2024-09-13T13:15:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[36682]: 6.15 scrub starts 2024-09-13T13:15:26.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[36682]: 6.15 scrub ok 2024-09-13T13:15:26.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[29471]: 5.17 scrub starts 2024-09-13T13:15:26.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[29471]: 5.17 scrub ok 2024-09-13T13:15:26.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[29471]: 10.d scrub ok 2024-09-13T13:15:26.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[29471]: 6.18 scrub starts 2024-09-13T13:15:26.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[29471]: 6.18 scrub ok 2024-09-13T13:15:26.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[29471]: 6.15 scrub starts 2024-09-13T13:15:26.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:26 smithi052 ceph-mon[29471]: 6.15 scrub ok 2024-09-13T13:15:27.041 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:27.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:26 smithi167 ceph-mon[33178]: 5.17 scrub starts 2024-09-13T13:15:27.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:26 smithi167 ceph-mon[33178]: 5.17 scrub ok 2024-09-13T13:15:27.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:26 smithi167 ceph-mon[33178]: 10.d scrub ok 2024-09-13T13:15:27.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:26 smithi167 ceph-mon[33178]: 6.18 scrub starts 2024-09-13T13:15:27.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:26 smithi167 ceph-mon[33178]: 6.18 scrub ok 2024-09-13T13:15:27.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:26 smithi167 ceph-mon[33178]: 6.15 scrub starts 2024-09-13T13:15:27.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:26 smithi167 ceph-mon[33178]: 6.15 scrub ok 2024-09-13T13:15:27.074 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:27.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:27.177 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:27.215 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:27.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:27.317 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:27.323 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 83 16 120868 120852 90.971 70.8125 0.0143782 0.0109877 2024-09-13T13:15:27.358 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:27.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:27.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:27.459 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:27.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:27.489 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:27.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:27.590 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:27.618 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:27.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:27.719 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:27.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:27.741 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:27.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:27.842 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:27.903 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:27 smithi052 ceph-mon[36682]: pgmap v710: 148 pgs: 148 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 925 op/s 2024-09-13T13:15:27.903 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:27 smithi052 ceph-mon[36682]: 6.0 scrub starts 2024-09-13T13:15:27.903 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:27 smithi052 ceph-mon[36682]: 6.0 scrub ok 2024-09-13T13:15:27.904 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:27.904 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:27 smithi052 ceph-mon[29471]: pgmap v710: 148 pgs: 148 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 925 op/s 2024-09-13T13:15:27.904 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:27 smithi052 ceph-mon[29471]: 6.0 scrub starts 2024-09-13T13:15:27.904 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:27 smithi052 ceph-mon[29471]: 6.0 scrub ok 2024-09-13T13:15:27.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:27.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:28.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:28.031 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:28.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:28.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:28.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:27 smithi167 ceph-mon[33178]: pgmap v710: 148 pgs: 148 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 925 op/s 2024-09-13T13:15:28.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:27 smithi167 ceph-mon[33178]: 6.0 scrub starts 2024-09-13T13:15:28.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:27 smithi167 ceph-mon[33178]: 6.0 scrub ok 2024-09-13T13:15:28.111 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:15:28.133 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:28.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:28.233 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:28.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:28.325 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 84 16 121652 121636 90.4708 49 0.0132554 0.0110486 2024-09-13T13:15:28.334 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:28.392 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:28.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:28.493 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:28.531 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:28.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:28.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:28.633 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:28.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:28.656 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:28.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:28.685 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T13:15:28.685 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:15:28.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:28 smithi052 ceph-mon[29471]: 5.1e scrub starts 2024-09-13T13:15:28.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:28 smithi052 ceph-mon[29471]: 5.1e scrub ok 2024-09-13T13:15:28.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:28 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2517520420' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:28.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:28 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/826633383' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:15:28.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:28 smithi052 ceph-mon[36682]: 5.1e scrub starts 2024-09-13T13:15:28.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:28 smithi052 ceph-mon[36682]: 5.1e scrub ok 2024-09-13T13:15:28.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:28 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2517520420' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:28.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:28 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/826633383' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:15:28.757 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:28.788 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:28.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:28.890 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:28.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:28.930 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:28.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:29.031 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:29.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:28 smithi167 ceph-mon[33178]: 5.1e scrub starts 2024-09-13T13:15:29.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:28 smithi167 ceph-mon[33178]: 5.1e scrub ok 2024-09-13T13:15:29.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:28 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2517520420' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:29.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:28 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/826633383' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:15:29.075 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:29.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:29.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:29.178 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:29.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:29.221 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:29.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:29.322 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:29.325 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 85 16 122633 122617 90.1278 61.3125 0.0259745 0.0110906 2024-09-13T13:15:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:29.365 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:29.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:29.466 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:29.499 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:29.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:29.601 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:29.643 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:29.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[36682]: pgmap v711: 148 pgs: 148 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-13T13:15:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[36682]: 10.3 scrub starts 2024-09-13T13:15:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:15:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:15:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2517520420' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:15:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/826633383' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:15:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[36682]: osdmap e363: 8 total, 8 up, 8 in 2024-09-13T13:15:29.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[36682]: 5.1c scrub starts 2024-09-13T13:15:29.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[36682]: 5.1c scrub ok 2024-09-13T13:15:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[29471]: pgmap v711: 148 pgs: 148 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-13T13:15:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[29471]: 10.3 scrub starts 2024-09-13T13:15:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:15:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:15:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2517520420' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:15:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/826633383' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:15:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[29471]: osdmap e363: 8 total, 8 up, 8 in 2024-09-13T13:15:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[29471]: 5.1c scrub starts 2024-09-13T13:15:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:29 smithi052 ceph-mon[29471]: 5.1c scrub ok 2024-09-13T13:15:29.745 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:29.777 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:29.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:29.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:29 smithi167 ceph-mon[33178]: pgmap v711: 148 pgs: 148 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-13T13:15:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:29 smithi167 ceph-mon[33178]: 10.3 scrub starts 2024-09-13T13:15:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:29 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:15:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:29 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:15:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:29 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2517520420' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:15:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:29 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/826633383' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:15:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:29 smithi167 ceph-mon[33178]: osdmap e363: 8 total, 8 up, 8 in 2024-09-13T13:15:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:29 smithi167 ceph-mon[33178]: 5.1c scrub starts 2024-09-13T13:15:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:29 smithi167 ceph-mon[33178]: 5.1c scrub ok 2024-09-13T13:15:29.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:29.906 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:29.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:29.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:30.008 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:30.052 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:30.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:30.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:30.154 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:30.189 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:30.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:30.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:30.291 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:30.324 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 86 16 123672 123656 89.8349 64.9375 0.0131678 0.0111258 2024-09-13T13:15:30.337 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:30.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:30.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:30.369 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T13:15:30.439 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:30.455 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:30.466 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:30.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:30.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:30.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:30.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:30.633 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:30.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:30.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:30.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:30.762 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:30.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:30.863 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:30.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:30.923 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:30.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:30.940 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:30 smithi052 ceph-mon[36682]: 3.8 scrub starts 2024-09-13T13:15:30.941 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:30 smithi052 ceph-mon[36682]: 3.8 scrub ok 2024-09-13T13:15:30.941 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:30 smithi052 ceph-mon[36682]: 10.5 scrub starts 2024-09-13T13:15:30.941 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:30 smithi052 ceph-mon[29471]: 3.8 scrub starts 2024-09-13T13:15:30.941 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:30 smithi052 ceph-mon[29471]: 3.8 scrub ok 2024-09-13T13:15:30.941 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:30 smithi052 ceph-mon[29471]: 10.5 scrub starts 2024-09-13T13:15:31.025 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:31.051 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:31.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:30 smithi167 ceph-mon[33178]: 3.8 scrub starts 2024-09-13T13:15:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:30 smithi167 ceph-mon[33178]: 3.8 scrub ok 2024-09-13T13:15:31.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:30 smithi167 ceph-mon[33178]: 10.5 scrub starts 2024-09-13T13:15:31.153 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:31.210 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:31.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:31.312 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:31.324 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 87 16 124500 124484 89.3972 51.75 0.00982552 0.0111804 2024-09-13T13:15:31.345 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:31.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:31.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:31.447 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:31.483 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:31.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:31.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:31.531 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:15:31.585 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:31.639 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:31.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:31.744 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:31.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:31.795 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:31.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:31.860 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:15:31.860 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:15:31.896 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:31.905 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[36682]: pgmap v713: 148 pgs: 148 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T13:15:31.905 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[36682]: 3.1 scrub starts 2024-09-13T13:15:31.905 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[36682]: 3.1 scrub ok 2024-09-13T13:15:31.905 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:31.905 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:31.905 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:31.905 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:31.906 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[29471]: pgmap v713: 148 pgs: 148 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T13:15:31.907 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[29471]: 3.1 scrub starts 2024-09-13T13:15:31.907 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[29471]: 3.1 scrub ok 2024-09-13T13:15:31.907 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:31.907 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:31.907 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:31.907 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:31 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:31.911 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:15:31.911 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:15:31.926 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:15:31.941 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:31.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:31.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:31.952 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:15:31.953 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:15:31.954 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:31.990 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:15:31.991 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:15:32.019 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:31 smithi167 ceph-mon[33178]: pgmap v713: 148 pgs: 148 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-13T13:15:32.019 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:31 smithi167 ceph-mon[33178]: 3.1 scrub starts 2024-09-13T13:15:32.020 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:31 smithi167 ceph-mon[33178]: 3.1 scrub ok 2024-09-13T13:15:32.020 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:31 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:32.020 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:31 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:32.020 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:31 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:32.020 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:31 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:32.028 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:15:32.028 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:15:32.045 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:32.075 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:15:32.075 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:15:32.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:32.098 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:32.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:32.110 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:15:32.110 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:15:32.151 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:15:32.152 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:15:32.158 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:15:32.158 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:15:32.199 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:32.251 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:32.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:32.325 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 88 16 125332 125316 88.971 52 0.0141752 0.0112346 2024-09-13T13:15:32.352 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:32.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:32.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:32.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:32.483 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:32.521 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:32.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:32.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:32.623 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:32.656 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:32.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:32.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:32.758 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:32.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:32.787 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:32.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:32.874 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T13:15:32.888 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[29471]: 10.3 scrub ok 2024-09-13T13:15:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2007940167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:15:32.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:15:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[29471]: 5.e scrub starts 2024-09-13T13:15:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[29471]: 5.e scrub ok 2024-09-13T13:15:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[29471]: 5.7 scrub starts 2024-09-13T13:15:32.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[29471]: 5.7 scrub ok 2024-09-13T13:15:32.921 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[36682]: 10.3 scrub ok 2024-09-13T13:15:32.921 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2007940167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:32.921 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:15:32.921 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:15:32.922 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:32.922 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[36682]: 5.e scrub starts 2024-09-13T13:15:32.922 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[36682]: 5.e scrub ok 2024-09-13T13:15:32.922 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[36682]: 5.7 scrub starts 2024-09-13T13:15:32.922 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:32 smithi052 ceph-mon[36682]: 5.7 scrub ok 2024-09-13T13:15:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:32.937 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:32.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:33.038 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:33.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:32 smithi167 ceph-mon[33178]: 10.3 scrub ok 2024-09-13T13:15:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:32 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2007940167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:15:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:32 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:15:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:32 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:32 smithi167 ceph-mon[33178]: 5.e scrub starts 2024-09-13T13:15:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:32 smithi167 ceph-mon[33178]: 5.e scrub ok 2024-09-13T13:15:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:32 smithi167 ceph-mon[33178]: 5.7 scrub starts 2024-09-13T13:15:33.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:32 smithi167 ceph-mon[33178]: 5.7 scrub ok 2024-09-13T13:15:33.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:33.079 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:33.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:33.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:33.224 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:33.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:33.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:33.276 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:15:33.326 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 89 16 126337 126321 88.6771 62.8125 0.0136199 0.0112718 2024-09-13T13:15:33.326 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:33.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:33.372 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:33.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:33.387 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:33.474 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:33.513 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:33.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:33.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:33.616 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:33.648 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:33.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:33.754 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:33.786 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:33.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:33.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:33.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[29471]: pgmap v714: 148 pgs: 148 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-13T13:15:33.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[29471]: 10.5 scrub ok 2024-09-13T13:15:33.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2746810646' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:15:33.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[29471]: from='client.36266 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:15:33.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[29471]: 6.6 scrub starts 2024-09-13T13:15:33.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[29471]: 6.6 scrub ok 2024-09-13T13:15:33.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[29471]: 5.15 scrub starts 2024-09-13T13:15:33.851 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[29471]: 5.15 scrub ok 2024-09-13T13:15:33.851 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:33.851 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:15:33.851 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[36682]: pgmap v714: 148 pgs: 148 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-13T13:15:33.851 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[36682]: 10.5 scrub ok 2024-09-13T13:15:33.851 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2746810646' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:15:33.851 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[36682]: from='client.36266 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:15:33.851 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[36682]: 6.6 scrub starts 2024-09-13T13:15:33.851 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[36682]: 6.6 scrub ok 2024-09-13T13:15:33.851 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[36682]: 5.15 scrub starts 2024-09-13T13:15:33.852 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[36682]: 5.15 scrub ok 2024-09-13T13:15:33.852 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:33.852 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:33 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:15:33.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:33.919 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:33.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:34.016 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T13:15:34.018 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:33 smithi167 ceph-mon[33178]: pgmap v714: 148 pgs: 148 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-13T13:15:34.018 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:33 smithi167 ceph-mon[33178]: 10.5 scrub ok 2024-09-13T13:15:34.018 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:33 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2746810646' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:15:34.018 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:33 smithi167 ceph-mon[33178]: from='client.36266 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:15:34.018 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:33 smithi167 ceph-mon[33178]: 6.6 scrub starts 2024-09-13T13:15:34.019 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:33 smithi167 ceph-mon[33178]: 6.6 scrub ok 2024-09-13T13:15:34.019 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:33 smithi167 ceph-mon[33178]: 5.15 scrub starts 2024-09-13T13:15:34.019 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:33 smithi167 ceph-mon[33178]: 5.15 scrub ok 2024-09-13T13:15:34.019 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:33 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:34.019 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:33 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:15:34.021 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:34.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:34.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:34.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:34.161 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:34.173 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:15:34.173 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:15:34.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:34.202 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:34.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:34.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:34.326 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Total time run: 90.006 2024-09-13T13:15:34.326 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Total writes made: 127377 2024-09-13T13:15:34.326 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Write size: 65536 2024-09-13T13:15:34.326 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Object size: 65536 2024-09-13T13:15:34.326 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Bandwidth (MB/sec): 88.4504 2024-09-13T13:15:34.326 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev Bandwidth: 32.7675 2024-09-13T13:15:34.326 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max bandwidth (MB/sec): 215.5 2024-09-13T13:15:34.326 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min bandwidth (MB/sec): 49 2024-09-13T13:15:34.326 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Average IOPS: 1415 2024-09-13T13:15:34.326 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev IOPS: 524.28 2024-09-13T13:15:34.326 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max IOPS: 3448 2024-09-13T13:15:34.327 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min IOPS: 784 2024-09-13T13:15:34.327 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Average Latency(s): 0.0113018 2024-09-13T13:15:34.327 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev Latency(s): 0.0103216 2024-09-13T13:15:34.327 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max latency(s): 0.355388 2024-09-13T13:15:34.327 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min latency(s): 0.0012696 2024-09-13T13:15:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:34.328 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:34.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:34.332 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T13:15:34.428 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:34.462 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:34.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:34.563 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:34.605 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:34.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:34.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:34.707 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:34.755 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:34.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:34.761 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T13:15:34.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:15:34 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:15:34.857 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:34.900 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:34.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:35.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:35.013 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:15:35.036 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T13:15:35.058 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:35.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:35.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:35.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:35.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:35.302 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:35 smithi167 ceph-mon[33178]: 4.7 scrub starts 2024-09-13T13:15:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:35 smithi167 ceph-mon[33178]: 4.7 scrub ok 2024-09-13T13:15:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:35 smithi167 ceph-mon[33178]: 5.4 scrub starts 2024-09-13T13:15:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:35 smithi167 ceph-mon[33178]: 5.4 scrub ok 2024-09-13T13:15:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:35 smithi167 ceph-mon[33178]: pgmap v715: 148 pgs: 148 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 999 op/s 2024-09-13T13:15:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:35 smithi167 ceph-mon[33178]: from='client.36266 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:15:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:35 smithi167 ceph-mon[33178]: osdmap e364: 8 total, 8 up, 8 in 2024-09-13T13:15:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:35 smithi167 ceph-mon[33178]: 4.14 scrub starts 2024-09-13T13:15:35.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:35 smithi167 ceph-mon[33178]: 4.14 scrub ok 2024-09-13T13:15:35.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:35 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/313497537' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:15:35.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:35.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:35.348 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:35.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:35.351 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:15:35.369 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[36682]: 4.7 scrub starts 2024-09-13T13:15:35.370 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[36682]: 4.7 scrub ok 2024-09-13T13:15:35.370 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[36682]: 5.4 scrub starts 2024-09-13T13:15:35.370 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[36682]: 5.4 scrub ok 2024-09-13T13:15:35.370 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[36682]: pgmap v715: 148 pgs: 148 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 999 op/s 2024-09-13T13:15:35.370 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[36682]: from='client.36266 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:15:35.370 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[36682]: osdmap e364: 8 total, 8 up, 8 in 2024-09-13T13:15:35.371 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[36682]: 4.14 scrub starts 2024-09-13T13:15:35.371 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[36682]: 4.14 scrub ok 2024-09-13T13:15:35.371 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/313497537' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:15:35.371 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:35.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[29471]: 4.7 scrub starts 2024-09-13T13:15:35.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[29471]: 4.7 scrub ok 2024-09-13T13:15:35.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[29471]: 5.4 scrub starts 2024-09-13T13:15:35.371 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[29471]: 5.4 scrub ok 2024-09-13T13:15:35.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[29471]: pgmap v715: 148 pgs: 148 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 999 op/s 2024-09-13T13:15:35.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[29471]: from='client.36266 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:15:35.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[29471]: osdmap e364: 8 total, 8 up, 8 in 2024-09-13T13:15:35.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[29471]: 4.14 scrub starts 2024-09-13T13:15:35.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[29471]: 4.14 scrub ok 2024-09-13T13:15:35.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/313497537' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:15:35.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:35.440 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:15:35.450 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:35.500 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:35.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:35.602 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:35.649 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:35.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:35.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:35.751 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:35.787 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:35.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:35.891 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:35.916 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:35.957 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:35.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:35.961 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:36.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:36.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:36.092 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:36.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:36.193 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:36.237 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:36.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:36.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:36 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:36.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:36 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/313497537' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:15:36.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:36 smithi167 ceph-mon[33178]: osdmap e365: 8 total, 8 up, 8 in 2024-09-13T13:15:36.325 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:15:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:15:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:15:36.325 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:36 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:36.325 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:36 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/313497537' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:15:36.326 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:36 smithi052 ceph-mon[36682]: osdmap e365: 8 total, 8 up, 8 in 2024-09-13T13:15:36.326 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:36 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:36.326 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:36 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/313497537' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:15:36.326 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:36 smithi052 ceph-mon[29471]: osdmap e365: 8 total, 8 up, 8 in 2024-09-13T13:15:36.338 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:36.378 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:36.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:36.480 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:36.520 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:36.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:36.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:36.656 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:36.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:36.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:36.758 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:36.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:36.796 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:36.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:36.822 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:15:36.822 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:15:36.851 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:15:36.851 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:15:36.889 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:15:36.889 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:15:36.897 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:36.935 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:15:36.935 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:15:36.938 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:36.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:36.973 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:15:36.973 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:15:37.012 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:15:37.012 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:15:37.041 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:37.049 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:15:37.050 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:15:37.075 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:37.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:37.087 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:15:37.087 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:15:37.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:37.178 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:15:37.178 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:15:37.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.178 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:130647 10.9 10:985bcca3:::benchmark_data_smithi052_153703_object3267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-13T13:15:37.178 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:37.169636+0000", 2024-09-13T13:15:37.178 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.006847985, 2024-09-13T13:15:37.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068775559999999999, 2024-09-13T13:15:37.178 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 130647 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.169636+0000", 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:37.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.169636+0000", 2024-09-13T13:15:37.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:37.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:37.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.169639+0000", 2024-09-13T13:15:37.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3159999999999999e-06 2024-09-13T13:15:37.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:37.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.169644+0000", 2024-09-13T13:15:37.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4489999999999998e-06 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.169660+0000", 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5639000000000001e-05 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.169668+0000", 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7910000000000005e-06 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:37.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.170531+0000", 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00086296500000000004 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.170654+0000", 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123378 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.170767+0000", 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113096 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:37.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.174778+0000", 2024-09-13T13:15:37.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040112050000000003 2024-09-13T13:15:37.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:37.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.175996+0000", 2024-09-13T13:15:37.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012178410000000001 2024-09-13T13:15:37.183 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:37.183 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:37.183 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:37.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.183 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:130653 10.6 10:68bbf840:::benchmark_data_smithi052_153703_object3273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-13T13:15:37.184 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:37.175633+0000", 2024-09-13T13:15:37.184 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00085176800000000003, 2024-09-13T13:15:37.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097070499999999998, 2024-09-13T13:15:37.184 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:37.184 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:37.184 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:15:37.184 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:37.184 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 130653 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.175633+0000", 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.175633+0000", 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:37.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.175634+0000", 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.175640+0000", 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6180000000000003e-06 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.175655+0000", 2024-09-13T13:15:37.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4941e-05 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.175663+0000", 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0390000000000005e-06 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.175717+0000", 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3600999999999997e-05 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:37.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.175833+0000", 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115948 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.175942+0000", 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108891 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:130649 10.a e365/334 10:5c4f4a99:::benchmark_data_smithi052_153703_object3269:head v 365'8156, mlcod=365'8154)", 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:37.176553+0000", 2024-09-13T13:15:37.188 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4294967295.9999313, 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122247, 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.176553+0000", 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.176553+0000", 2024-09-13T13:15:37.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.176554+0000", 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018e-06 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.176559+0000", 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0030000000000002e-06 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:37.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.176568+0000", 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4319999999999999e-06 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.176574+0000", 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0939999999999996e-06 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.176618+0000", 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3689999999999997e-05 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:37.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:37.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:37.176637+0000", 2024-09-13T13:15:37.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9111e-05 2024-09-13T13:15:37.192 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:37.192 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:37.192 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:37.192 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:37.192 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:15:37.192 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 3 2024-09-13T13:15:37.192 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:15:37.228 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:37.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:37.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:37 smithi052 ceph-mon[29471]: pgmap v718: 148 pgs: 148 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 48 MiB/s wr, 778 op/s 2024-09-13T13:15:37.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:37 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1941169347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:37.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:37 smithi052 ceph-mon[36682]: pgmap v718: 148 pgs: 148 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 48 MiB/s wr, 778 op/s 2024-09-13T13:15:37.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:37 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1941169347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:37.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:37 smithi167 ceph-mon[33178]: pgmap v718: 148 pgs: 148 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 48 MiB/s wr, 778 op/s 2024-09-13T13:15:37.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:37 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1941169347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:37.330 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:37.365 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:37.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:37.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:37.467 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:37.502 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:37.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:37.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:37.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:37.638 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:37.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:37.739 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:37.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:37.777 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:37.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:37.826 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:15:37.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:37.911 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:37.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:38.010 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T13:15:38.012 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:38.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:38.051 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:38.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:38.067 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T13:15:38.152 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:38.189 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:38.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:38.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:38.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:38 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4217771188' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:15:38.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4217771188' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:15:38.291 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:38 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4217771188' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:15:38.326 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:38.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:38.396 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:15:38.428 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:38.474 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:38.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:38.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:38.621 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:38.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:38.723 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:38.755 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:38.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:38.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:38.857 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:38.892 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:38.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:38.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:38.994 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:39.024 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:39.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:39.126 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:39.155 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:39.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:39.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:39.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:39 smithi052 ceph-mon[36682]: pgmap v719: 148 pgs: 148 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 46 MiB/s wr, 831 op/s 2024-09-13T13:15:39.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:39 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4217771188' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:15:39.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:39 smithi052 ceph-mon[36682]: osdmap e366: 8 total, 8 up, 8 in 2024-09-13T13:15:39.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:39 smithi052 ceph-mon[29471]: pgmap v719: 148 pgs: 148 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 46 MiB/s wr, 831 op/s 2024-09-13T13:15:39.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:39 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4217771188' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:15:39.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:39 smithi052 ceph-mon[29471]: osdmap e366: 8 total, 8 up, 8 in 2024-09-13T13:15:39.258 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:39.296 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:39.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:39.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:39 smithi167 ceph-mon[33178]: pgmap v719: 148 pgs: 148 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 46 MiB/s wr, 831 op/s 2024-09-13T13:15:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:39 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4217771188' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:15:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:39 smithi167 ceph-mon[33178]: osdmap e366: 8 total, 8 up, 8 in 2024-09-13T13:15:39.398 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:39.421 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:39.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:39.422 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T13:15:39.461 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:15:39.462 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:15:39.462 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:15:39.462 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:15:39.462 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:15:39.492 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:39.522 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:39.544 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:39.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:39.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:39.646 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:39.670 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:39.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:39.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:39.771 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:39.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:39.823 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:39.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:39.923 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:39.966 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:39.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:40.068 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:40.095 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:40.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:40.113 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:40.198 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:40.235 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:40.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:40.337 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:40.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:40.372 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:40.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:40.474 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:40.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:40.509 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:40.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:40.610 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:40.640 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:40.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:40.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:40.650 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:15:40.743 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:40.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:40.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:40.918 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:40.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:40.995 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T13:15:41.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:41.075 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:41.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:41.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:41.093 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:15:41.117 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:41.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:41.226 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:41.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:41 smithi167 ceph-mon[33178]: pgmap v721: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.8 KiB/s rd, 26 MiB/s wr, 1.05k op/s 2024-09-13T13:15:41.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:41 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1848003167' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:41.327 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:41.368 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:41.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:41.399 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:15:41.435 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:41 smithi052 ceph-mon[29471]: pgmap v721: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.8 KiB/s rd, 26 MiB/s wr, 1.05k op/s 2024-09-13T13:15:41.435 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:41 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1848003167' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:41.435 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:41 smithi052 ceph-mon[36682]: pgmap v721: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.8 KiB/s rd, 26 MiB/s wr, 1.05k op/s 2024-09-13T13:15:41.435 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:41 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1848003167' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:41.471 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:41.508 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:41.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:41.610 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:41.671 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:41.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:41.773 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:41.809 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:41.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:41.910 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:41.947 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:41.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:42.045 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:15:42.045 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:15:42.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:15:42.045 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:15:42.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.045 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135962 8.3 e207/169 8:c288bb32:::benchmark_data_smithi052_101302_object8033:head v 207'8600, mlcod=207'8598)", 2024-09-13T13:15:42.045 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.798736+0000", 2024-09-13T13:15:42.045 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 453.243100416, 2024-09-13T13:15:42.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.214192363, 2024-09-13T13:15:42.046 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.046 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.046 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.046 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798736+0000", 2024-09-13T13:15:42.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798736+0000", 2024-09-13T13:15:42.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798738+0000", 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.542e-06 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798746+0000", 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7299999999999994e-06 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798754+0000", 2024-09-13T13:15:42.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7770000000000002e-06 2024-09-13T13:15:42.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798760+0000", 2024-09-13T13:15:42.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7769999999999999e-06 2024-09-13T13:15:42.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798809+0000", 2024-09-13T13:15:42.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8714000000000001e-05 2024-09-13T13:15:42.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.798822+0000", 2024-09-13T13:15:42.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3737000000000001e-05 2024-09-13T13:15:42.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012889+0000", 2024-09-13T13:15:42.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.21406626500000001 2024-09-13T13:15:42.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012928+0000", 2024-09-13T13:15:42.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9820999999999997e-05 2024-09-13T13:15:42.050 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.050 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.050 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.050 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:135964 8.8 8:1e05dc47:::benchmark_data_smithi052_101302_object8035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:15:42.050 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.800142+0000", 2024-09-13T13:15:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 453.24169432100001, 2024-09-13T13:15:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20392829400000001, 2024-09-13T13:15:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:15:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:15:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135964 2024-09-13T13:15:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800142+0000", 2024-09-13T13:15:42.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800142+0000", 2024-09-13T13:15:42.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800144+0000", 2024-09-13T13:15:42.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5719999999999999e-06 2024-09-13T13:15:42.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800149+0000", 2024-09-13T13:15:42.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2059999999999998e-06 2024-09-13T13:15:42.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800159+0000", 2024-09-13T13:15:42.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9110000000000005e-06 2024-09-13T13:15:42.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800165+0000", 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6640000000000001e-06 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800242+0000", 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6415999999999995e-05 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800369+0000", 2024-09-13T13:15:42.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000127238 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800490+0000", 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120827 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803447+0000", 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029568620000000002 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803750+0000", 2024-09-13T13:15:42.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030363800000000001 2024-09-13T13:15:42.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003963+0000", 2024-09-13T13:15:42.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200212532 2024-09-13T13:15:42.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004013+0000", 2024-09-13T13:15:42.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9765999999999998e-05 2024-09-13T13:15:42.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004070+0000", 2024-09-13T13:15:42.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7661999999999998e-05 2024-09-13T13:15:42.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.057 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.25222.0:135967 8.4 8:2cd9f03c:::benchmark_data_smithi052_101302_object8038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.800185+0000", 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 453.24165097899998, 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20402237600000001, 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.25222", 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/4147402633", 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135967 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.058 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800185+0000", 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800185+0000", 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800186+0000", 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8999999999999996e-07 2024-09-13T13:15:42.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800189+0000", 2024-09-13T13:15:42.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.689e-06 2024-09-13T13:15:42.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800194+0000", 2024-09-13T13:15:42.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1309999999999997e-06 2024-09-13T13:15:42.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.061 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:42.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800199+0000", 2024-09-13T13:15:42.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6959999999999996e-06 2024-09-13T13:15:42.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800250+0000", 2024-09-13T13:15:42.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1402e-05 2024-09-13T13:15:42.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800389+0000", 2024-09-13T13:15:42.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000138596 2024-09-13T13:15:42.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-09-13T13:15:42.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.800490+0000", 2024-09-13T13:15:42.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010134899999999999 2024-09-13T13:15:42.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803436+0000", 2024-09-13T13:15:42.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029464529999999999 2024-09-13T13:15:42.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004082+0000", 2024-09-13T13:15:42.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200645886 2024-09-13T13:15:42.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004122+0000", 2024-09-13T13:15:42.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9864000000000003e-05 2024-09-13T13:15:42.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004169+0000", 2024-09-13T13:15:42.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7052000000000003e-05 2024-09-13T13:15:42.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004208+0000", 2024-09-13T13:15:42.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8668000000000003e-05 2024-09-13T13:15:42.066 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.066 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.066 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.066 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135966 8.3 e207/169 8:cd1f5880:::benchmark_data_smithi052_101302_object8037:head v 207'8601, mlcod=207'8598)", 2024-09-13T13:15:42.067 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.802377+0000", 2024-09-13T13:15:42.067 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 453.23945972400003, 2024-09-13T13:15:42.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.21058527299999999, 2024-09-13T13:15:42.067 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.067 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.067 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.067 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802377+0000", 2024-09-13T13:15:42.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802377+0000", 2024-09-13T13:15:42.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802379+0000", 2024-09-13T13:15:42.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.827e-06 2024-09-13T13:15:42.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802390+0000", 2024-09-13T13:15:42.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1623999999999999e-05 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802402+0000", 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1564e-05 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802410+0000", 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2719999999999997e-06 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802483+0000", 2024-09-13T13:15:42.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2840000000000001e-05 2024-09-13T13:15:42.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.802503+0000", 2024-09-13T13:15:42.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0452e-05 2024-09-13T13:15:42.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012933+0000", 2024-09-13T13:15:42.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.210429965 2024-09-13T13:15:42.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012962+0000", 2024-09-13T13:15:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8728999999999999e-05 2024-09-13T13:15:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135968 8.2 e207/169 8:4f16015f:::benchmark_data_smithi052_101302_object8039:head v 207'8502, mlcod=207'8500)", 2024-09-13T13:15:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.803228+0000", 2024-09-13T13:15:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 453.23860870700003, 2024-09-13T13:15:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20068414800000001, 2024-09-13T13:15:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.072 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.072 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.072 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803228+0000", 2024-09-13T13:15:42.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803228+0000", 2024-09-13T13:15:42.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803229+0000", 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2399999999999996e-07 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803234+0000", 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6400000000000002e-06 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803244+0000", 2024-09-13T13:15:42.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6770000000000007e-06 2024-09-13T13:15:42.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803251+0000", 2024-09-13T13:15:42.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0149999999999998e-06 2024-09-13T13:15:42.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803344+0000", 2024-09-13T13:15:42.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2656000000000002e-05 2024-09-13T13:15:42.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803370+0000", 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6156000000000002e-05 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003858+0000", 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20048846400000001 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003912+0000", 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3616000000000003e-05 2024-09-13T13:15:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.076 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.076 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.076 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.25222.0:135964 8.8 e207/169 ondisk, result = 0)", 2024-09-13T13:15:42.076 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.803563+0000", 2024-09-13T13:15:42.076 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 453.23827335700003, 2024-09-13T13:15:42.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200512417, 2024-09-13T13:15:42.076 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.076 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.076 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:15:42.076 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803563+0000", 2024-09-13T13:15:42.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803563+0000", 2024-09-13T13:15:42.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.804856+0000", 2024-09-13T13:15:42.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001293055 2024-09-13T13:15:42.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.804861+0000", 2024-09-13T13:15:42.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5800000000000002e-06 2024-09-13T13:15:42.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003842+0000", 2024-09-13T13:15:42.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19898099699999999 2024-09-13T13:15:42.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003857+0000", 2024-09-13T13:15:42.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5021999999999999e-05 2024-09-13T13:15:42.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003949+0000", 2024-09-13T13:15:42.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1824999999999996e-05 2024-09-13T13:15:42.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003956+0000", 2024-09-13T13:15:42.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9070000000000007e-06 2024-09-13T13:15:42.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004075+0000", 2024-09-13T13:15:42.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011903100000000001 2024-09-13T13:15:42.080 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.080 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.080 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.080 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.25222.0:135967 8.4 e207/169 ondisk, result = 0)", 2024-09-13T13:15:42.080 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.803807+0000", 2024-09-13T13:15:42.080 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 453.238028977, 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.200283038, 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803807+0000", 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803807+0000", 2024-09-13T13:15:42.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803808+0000", 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.032e-06 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803812+0000", 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0069999999999999e-06 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.803820+0000", 2024-09-13T13:15:42.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5599999999999996e-06 2024-09-13T13:15:42.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003847+0000", 2024-09-13T13:15:42.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20002690300000001 2024-09-13T13:15:42.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004067+0000", 2024-09-13T13:15:42.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021978000000000001 2024-09-13T13:15:42.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004075+0000", 2024-09-13T13:15:42.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6430000000000001e-06 2024-09-13T13:15:42.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.004090+0000", 2024-09-13T13:15:42.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5113000000000001e-05 2024-09-13T13:15:42.084 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.084 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.084 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.25222.0:135970 8.3 e207/169 8:ce0cc5da:::benchmark_data_smithi052_101302_object8041:head v 207'8602, mlcod=207'8598)", 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:08:08.806702+0000", 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 453.23513394399998, 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20628901199999999, 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806702+0000", 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806702+0000", 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806703+0000", 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0109999999999999e-06 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:08.806711+0000", 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.52e-06 2024-09-13T13:15:42.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003876+0000", 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19716455899999999 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003884+0000", 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2269999999999993e-06 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003930+0000", 2024-09-13T13:15:42.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6368000000000001e-05 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.003955+0000", 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4374000000000001e-05 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012964+0000", 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090092720000000005 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:08:09.012991+0000", 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7681000000000001e-05 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:161588 9.6 9:6d59046d:::benchmark_data_smithi052_128427_object30919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:14.715303+0000", 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 207.32653296800001, 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.23500462999999999, 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.089 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 161588 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.715303+0000", 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.715303+0000", 2024-09-13T13:15:42.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.715305+0000", 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.888e-06 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.715311+0000", 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6710000000000004e-06 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.715323+0000", 2024-09-13T13:15:42.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1883e-05 2024-09-13T13:15:42.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.715330+0000", 2024-09-13T13:15:42.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2899999999999997e-06 2024-09-13T13:15:42.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.715388+0000", 2024-09-13T13:15:42.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.787e-05 2024-09-13T13:15:42.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.715499+0000", 2024-09-13T13:15:42.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110899 2024-09-13T13:15:42.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:15:42.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.715602+0000", 2024-09-13T13:15:42.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102973 2024-09-13T13:15:42.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.721397+0000", 2024-09-13T13:15:42.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057953950000000001 2024-09-13T13:15:42.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.721458+0000", 2024-09-13T13:15:42.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0857999999999999e-05 2024-09-13T13:15:42.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.950222+0000", 2024-09-13T13:15:42.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.228763515 2024-09-13T13:15:42.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.950264+0000", 2024-09-13T13:15:42.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1992e-05 2024-09-13T13:15:42.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.950308+0000", 2024-09-13T13:15:42.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4396e-05 2024-09-13T13:15:42.095 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.095 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.095 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.096 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:161591 9.6 9:60592e87:::benchmark_data_smithi052_128427_object30922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T13:15:42.096 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:14.716155+0000", 2024-09-13T13:15:42.096 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 207.32568103099999, 2024-09-13T13:15:42.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.239704054, 2024-09-13T13:15:42.096 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.096 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.096 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.096 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.096 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:15:42.096 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:15:42.096 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 161591 2024-09-13T13:15:42.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.097 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.716155+0000", 2024-09-13T13:15:42.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.716155+0000", 2024-09-13T13:15:42.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.716157+0000", 2024-09-13T13:15:42.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-09-13T13:15:42.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.716162+0000", 2024-09-13T13:15:42.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3299999999999998e-06 2024-09-13T13:15:42.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.716174+0000", 2024-09-13T13:15:42.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1871999999999999e-05 2024-09-13T13:15:42.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.716181+0000", 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4660000000000002e-06 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.718132+0000", 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019503120000000001 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.718248+0000", 2024-09-13T13:15:42.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115936 2024-09-13T13:15:42.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:15:42.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.718621+0000", 2024-09-13T13:15:42.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037309799999999998 2024-09-13T13:15:42.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.721671+0000", 2024-09-13T13:15:42.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030500369999999998 2024-09-13T13:15:42.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.726121+0000", 2024-09-13T13:15:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044502470000000001 2024-09-13T13:15:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.955753+0000", 2024-09-13T13:15:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.229631949 2024-09-13T13:15:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.955815+0000", 2024-09-13T13:15:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1821000000000003e-05 2024-09-13T13:15:42.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.955859+0000", 2024-09-13T13:15:42.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4567000000000002e-05 2024-09-13T13:15:42.102 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.102 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.102 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.102 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:161618 9.6 9:630a33e9:::benchmark_data_smithi052_128427_object30949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T13:15:42.102 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:14.740622+0000", 2024-09-13T13:15:42.102 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 207.30121476599999, 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.215360458, 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 161618 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.740622+0000", 2024-09-13T13:15:42.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.740622+0000", 2024-09-13T13:15:42.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.740622+0000", 2024-09-13T13:15:42.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0200000000000002e-07 2024-09-13T13:15:42.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.740624+0000", 2024-09-13T13:15:42.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2010000000000002e-06 2024-09-13T13:15:42.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.740629+0000", 2024-09-13T13:15:42.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5379999999999996e-06 2024-09-13T13:15:42.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.740633+0000", 2024-09-13T13:15:42.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8809999999999998e-06 2024-09-13T13:15:42.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.740679+0000", 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6427000000000001e-05 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.740826+0000", 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000146551 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.740945+0000", 2024-09-13T13:15:42.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011952500000000001 2024-09-13T13:15:42.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.743595+0000", 2024-09-13T13:15:42.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026497199999999999 2024-09-13T13:15:42.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.744513+0000", 2024-09-13T13:15:42.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00091797299999999999 2024-09-13T13:15:42.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.955921+0000", 2024-09-13T13:15:42.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.211408136 2024-09-13T13:15:42.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.955954+0000", 2024-09-13T13:15:42.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2459000000000002e-05 2024-09-13T13:15:42.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.955982+0000", 2024-09-13T13:15:42.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8544999999999999e-05 2024-09-13T13:15:42.109 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.109 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.110 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:161632 9.6 9:6d70ba42:::benchmark_data_smithi052_128427_object30963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T13:15:42.110 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:14.752447+0000", 2024-09-13T13:15:42.112 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 207.28938974499999, 2024-09-13T13:15:42.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20646440999999999, 2024-09-13T13:15:42.112 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.112 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.112 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.112 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.112 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:15:42.112 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:15:42.112 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 161632 2024-09-13T13:15:42.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.112 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.752447+0000", 2024-09-13T13:15:42.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.752447+0000", 2024-09-13T13:15:42.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.752448+0000", 2024-09-13T13:15:42.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4100000000000001e-06 2024-09-13T13:15:42.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.752452+0000", 2024-09-13T13:15:42.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3660000000000002e-06 2024-09-13T13:15:42.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.752460+0000", 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8390000000000007e-06 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.752469+0000", 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0149999999999992e-06 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.752532+0000", 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2428999999999995e-05 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.752635+0000", 2024-09-13T13:15:42.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010328900000000001 2024-09-13T13:15:42.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:15:42.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.752736+0000", 2024-09-13T13:15:42.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101449 2024-09-13T13:15:42.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.754945+0000", 2024-09-13T13:15:42.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022088699999999999 2024-09-13T13:15:42.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.755781+0000", 2024-09-13T13:15:42.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00083544000000000001 2024-09-13T13:15:42.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.958810+0000", 2024-09-13T13:15:42.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20302936699999999 2024-09-13T13:15:42.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.958866+0000", 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5411000000000002e-05 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.958911+0000", 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5525000000000002e-05 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:161655 9.6 9:613e50c2:::benchmark_data_smithi052_128427_object30986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:14.778599+0000", 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 207.263237506, 2024-09-13T13:15:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.18043693, 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 161655 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.778599+0000", 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.778599+0000", 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.778599+0000", 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.778602+0000", 2024-09-13T13:15:42.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9809999999999999e-06 2024-09-13T13:15:42.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.778606+0000", 2024-09-13T13:15:42.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1030000000000004e-06 2024-09-13T13:15:42.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.778610+0000", 2024-09-13T13:15:42.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4410000000000002e-06 2024-09-13T13:15:42.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.778651+0000", 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0680999999999999e-05 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.778762+0000", 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011178199999999999 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.778885+0000", 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122154 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.781117+0000", 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002232642 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.782060+0000", 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00094304600000000001 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.958950+0000", 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17689016499999999 2024-09-13T13:15:42.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.959002+0000", 2024-09-13T13:15:42.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000999999999999e-05 2024-09-13T13:15:42.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:14.959036+0000", 2024-09-13T13:15:42.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3435000000000001e-05 2024-09-13T13:15:42.124 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.124 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.124 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.125 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:163401 9.f 9:f3722156:::benchmark_data_smithi052_128427_object32732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-13T13:15:42.125 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:16.433229+0000", 2024-09-13T13:15:42.125 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 205.60860713700001, 2024-09-13T13:15:42.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16114342700000001, 2024-09-13T13:15:42.125 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.125 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.125 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:15:42.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:15:42.125 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 163401 2024-09-13T13:15:42.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.126 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.433229+0000", 2024-09-13T13:15:42.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.433229+0000", 2024-09-13T13:15:42.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.433230+0000", 2024-09-13T13:15:42.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1299999999999998e-07 2024-09-13T13:15:42.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.433238+0000", 2024-09-13T13:15:42.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4970000000000004e-06 2024-09-13T13:15:42.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.433245+0000", 2024-09-13T13:15:42.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7209999999999994e-06 2024-09-13T13:15:42.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.433251+0000", 2024-09-13T13:15:42.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4770000000000002e-06 2024-09-13T13:15:42.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.433508+0000", 2024-09-13T13:15:42.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025732899999999999 2024-09-13T13:15:42.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.433626+0000", 2024-09-13T13:15:42.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011791899999999999 2024-09-13T13:15:42.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:15:42.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.434179+0000", 2024-09-13T13:15:42.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00055292999999999998 2024-09-13T13:15:42.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.437751+0000", 2024-09-13T13:15:42.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035723119999999998 2024-09-13T13:15:42.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.441826+0000", 2024-09-13T13:15:42.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040743339999999998 2024-09-13T13:15:42.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.594249+0000", 2024-09-13T13:15:42.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.152423477 2024-09-13T13:15:42.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.594324+0000", 2024-09-13T13:15:42.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4865999999999998e-05 2024-09-13T13:15:42.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:16.594373+0000", 2024-09-13T13:15:42.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8652000000000001e-05 2024-09-13T13:15:42.131 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.131 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.131 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.131 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:41377 10.6 10:6e94fb43:::benchmark_data_smithi052_153703_object41376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T13:15:42.131 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:24.367635+0000", 2024-09-13T13:15:42.131 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 77.674201562999997, 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.15995672599999999, 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 41377 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.367635+0000", 2024-09-13T13:15:42.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.367635+0000", 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.367637+0000", 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.367e-06 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.367688+0000", 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1212999999999999e-05 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.367710+0000", 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1716999999999999e-05 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.367718+0000", 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6240000000000002e-06 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.367776+0000", 2024-09-13T13:15:42.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7948000000000002e-05 2024-09-13T13:15:42.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.367911+0000", 2024-09-13T13:15:42.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013577399999999999 2024-09-13T13:15:42.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:15:42.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.368043+0000", 2024-09-13T13:15:42.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131686 2024-09-13T13:15:42.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.370808+0000", 2024-09-13T13:15:42.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027647090000000002 2024-09-13T13:15:42.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.371979+0000", 2024-09-13T13:15:42.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001171477 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.527438+0000", 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.15545856599999999 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.527536+0000", 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8443000000000006e-05 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.527592+0000", 2024-09-13T13:15:42.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5201999999999998e-05 2024-09-13T13:15:42.138 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.138 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.138 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.138 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:41383 10.6 10:661679ce:::benchmark_data_smithi052_153703_object41382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T13:15:42.138 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:24.372586+0000", 2024-09-13T13:15:42.138 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 77.669250129000005, 2024-09-13T13:15:42.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.15864440799999999, 2024-09-13T13:15:42.138 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.138 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 41383 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372586+0000", 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372586+0000", 2024-09-13T13:15:42.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372588+0000", 2024-09-13T13:15:42.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0490000000000002e-06 2024-09-13T13:15:42.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372674+0000", 2024-09-13T13:15:42.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5304999999999997e-05 2024-09-13T13:15:42.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372692+0000", 2024-09-13T13:15:42.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8578999999999999e-05 2024-09-13T13:15:42.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372708+0000", 2024-09-13T13:15:42.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5403000000000001e-05 2024-09-13T13:15:42.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372754+0000", 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6297000000000001e-05 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372915+0000", 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016119800000000001 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.373067+0000", 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015170099999999999 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.377691+0000", 2024-09-13T13:15:42.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046246150000000003 2024-09-13T13:15:42.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.377758+0000", 2024-09-13T13:15:42.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6719999999999998e-05 2024-09-13T13:15:42.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.531118+0000", 2024-09-13T13:15:42.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.15336007600000001 2024-09-13T13:15:42.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.531184+0000", 2024-09-13T13:15:42.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5347999999999998e-05 2024-09-13T13:15:42.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.531231+0000", 2024-09-13T13:15:42.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7117000000000003e-05 2024-09-13T13:15:42.144 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.144 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.144 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:41384 10.6 10:682c3242:::benchmark_data_smithi052_153703_object41383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:24.372723+0000", 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 77.669113311999993, 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16083098700000001, 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 41384 2024-09-13T13:15:42.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.146 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372723+0000", 2024-09-13T13:15:42.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372723+0000", 2024-09-13T13:15:42.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372724+0000", 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6899999999999996e-07 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372756+0000", 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2206000000000003e-05 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372771+0000", 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5294e-05 2024-09-13T13:15:42.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.372779+0000", 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.17e-06 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.374351+0000", 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015725120000000001 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.374498+0000", 2024-09-13T13:15:42.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147369 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.374647+0000", 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014862999999999999 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.377745+0000", 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003097696 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.377764+0000", 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9143999999999998e-05 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.533400+0000", 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.15563613500000001 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.533464+0000", 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3672999999999998e-05 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.533554+0000", 2024-09-13T13:15:42.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0389000000000005e-05 2024-09-13T13:15:42.151 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.151 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.151 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.151 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:41386 10.6 10:63663e59:::benchmark_data_smithi052_153703_object41385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T13:15:42.151 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:24.373707+0000", 2024-09-13T13:15:42.151 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 77.668129037, 2024-09-13T13:15:42.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.160095248, 2024-09-13T13:15:42.151 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.151 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.152 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:42.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:42.152 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 41386 2024-09-13T13:15:42.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.152 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.153 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T13:15:42.154 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:42.154 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:15:42.154 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:15:42.154 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1848003167' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:15:42.154 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[29471]: osdmap e367: 8 total, 8 up, 8 in 2024-09-13T13:15:42.154 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2474159388' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:15:42.154 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[29471]: from='client.26227 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:15:42.155 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:42.155 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:15:42.155 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:15:42.155 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1848003167' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:15:42.155 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[36682]: osdmap e367: 8 total, 8 up, 8 in 2024-09-13T13:15:42.155 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2474159388' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:15:42.155 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:42 smithi052 ceph-mon[36682]: from='client.26227 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:15:42.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.373707+0000", 2024-09-13T13:15:42.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.373707+0000", 2024-09-13T13:15:42.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.373709+0000", 2024-09-13T13:15:42.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.471e-06 2024-09-13T13:15:42.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.373794+0000", 2024-09-13T13:15:42.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5583999999999999e-05 2024-09-13T13:15:42.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.373809+0000", 2024-09-13T13:15:42.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4532000000000001e-05 2024-09-13T13:15:42.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.373816+0000", 2024-09-13T13:15:42.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1169999999999998e-06 2024-09-13T13:15:42.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.376028+0000", 2024-09-13T13:15:42.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002211654 2024-09-13T13:15:42.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.376189+0000", 2024-09-13T13:15:42.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016165 2024-09-13T13:15:42.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:15:42.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.376349+0000", 2024-09-13T13:15:42.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015934900000000001 2024-09-13T13:15:42.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.377862+0000", 2024-09-13T13:15:42.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001513398 2024-09-13T13:15:42.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.377926+0000", 2024-09-13T13:15:42.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4249000000000001e-05 2024-09-13T13:15:42.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.533707+0000", 2024-09-13T13:15:42.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.155780852 2024-09-13T13:15:42.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.533759+0000", 2024-09-13T13:15:42.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1997999999999999e-05 2024-09-13T13:15:42.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.533803+0000", 2024-09-13T13:15:42.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3393999999999998e-05 2024-09-13T13:15:42.161 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.161 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.161 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.161 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:41400 10.6 10:6952781e:::benchmark_data_smithi052_153703_object41399:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T13:15:42.162 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:14:24.382685+0000", 2024-09-13T13:15:42.162 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 77.659151237000003, 2024-09-13T13:15:42.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151265974, 2024-09-13T13:15:42.162 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.162 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.162 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.162 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.162 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:42.162 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:42.162 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 41400 2024-09-13T13:15:42.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.162 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.382685+0000", 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.382685+0000", 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.382686+0000", 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2500000000000001e-06 2024-09-13T13:15:42.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.382769+0000", 2024-09-13T13:15:42.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2626999999999993e-05 2024-09-13T13:15:42.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.382779+0000", 2024-09-13T13:15:42.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0441e-05 2024-09-13T13:15:42.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.382807+0000", 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7474000000000002e-05 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.382862+0000", 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5510000000000002e-05 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.382965+0000", 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102853 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:15:42.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.383082+0000", 2024-09-13T13:15:42.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116279 2024-09-13T13:15:42.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.384547+0000", 2024-09-13T13:15:42.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014654170000000001 2024-09-13T13:15:42.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.384800+0000", 2024-09-13T13:15:42.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000252592 2024-09-13T13:15:42.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.533849+0000", 2024-09-13T13:15:42.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14904931099999999 2024-09-13T13:15:42.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.533908+0000", 2024-09-13T13:15:42.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9580999999999998e-05 2024-09-13T13:15:42.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:14:24.533951+0000", 2024-09-13T13:15:42.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2639000000000003e-05 2024-09-13T13:15:42.168 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.168 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.168 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.168 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:133029 10.9 10:97ab46ca:::benchmark_data_smithi052_153703_object5649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:15:42.168 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.308770+0000", 2024-09-13T13:15:42.168 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.7330664059999998, 2024-09-13T13:15:42.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.35692719899999997, 2024-09-13T13:15:42.168 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:42.168 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:42.168 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:15:42.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:42.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:42.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:42.169 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 133029 2024-09-13T13:15:42.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.169 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:42.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:42.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308770+0000", 2024-09-13T13:15:42.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:42.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308770+0000", 2024-09-13T13:15:42.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:42.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:42.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308772+0000", 2024-09-13T13:15:42.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3319999999999999e-06 2024-09-13T13:15:42.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:42.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308790+0000", 2024-09-13T13:15:42.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7960000000000001e-05 2024-09-13T13:15:42.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308804+0000", 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3628000000000001e-05 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308811+0000", 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2470000000000002e-06 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308877+0000", 2024-09-13T13:15:42.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5579000000000004e-05 2024-09-13T13:15:42.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:42.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308990+0000", 2024-09-13T13:15:42.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113732 2024-09-13T13:15:42.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:15:42.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309190+0000", 2024-09-13T13:15:42.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000199839 2024-09-13T13:15:42.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.311704+0000", 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025138460000000001 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.313823+0000", 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021192849999999998 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665572+0000", 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.35174855399999999 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665647+0000", 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5347999999999998e-05 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665697+0000", 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9849000000000003e-05 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:42.174 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:15:42.175 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:42 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:42.175 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:42 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:15:42.175 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:42 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:15:42.175 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:42 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1848003167' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:15:42.175 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:42 smithi167 ceph-mon[33178]: osdmap e367: 8 total, 8 up, 8 in 2024-09-13T13:15:42.175 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:42 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2474159388' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:15:42.175 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:42 smithi167 ceph-mon[33178]: from='client.26227 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:15:42.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:42.176 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:42.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:42.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:42.314 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:42.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:42.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:42.416 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:42.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:42.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:42.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:42.551 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:42.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:42.590 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:42.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:42.646 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:15:42.647 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:15:42.684 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:15:42.685 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:15:42.691 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:42.720 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:15:42.721 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:15:42.723 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:42.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:42.749 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:15:42.750 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:15:42.779 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:15:42.780 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:15:42.809 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:15:42.809 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:15:42.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:42.839 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:15:42.839 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:15:42.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:42.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:42.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:42.869 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:15:42.870 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:15:42.958 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:42.999 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:42.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:43.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:43.101 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:43.138 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:43.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:43.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:43.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T13:15:43.240 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:43.273 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:43.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:43.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:43.375 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:43.408 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:43.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:43.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:43.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:43 smithi052 ceph-mon[36682]: pgmap v723: 148 pgs: 148 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 889 B/s rd, 15 MiB/s wr, 1.06k op/s 2024-09-13T13:15:43.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:43 smithi052 ceph-mon[36682]: from='client.26227 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:15:43.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:43 smithi052 ceph-mon[36682]: osdmap e368: 8 total, 8 up, 8 in 2024-09-13T13:15:43.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/485696487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:43.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:43 smithi052 ceph-mon[29471]: pgmap v723: 148 pgs: 148 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 889 B/s rd, 15 MiB/s wr, 1.06k op/s 2024-09-13T13:15:43.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:43 smithi052 ceph-mon[29471]: from='client.26227 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:15:43.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:43 smithi052 ceph-mon[29471]: osdmap e368: 8 total, 8 up, 8 in 2024-09-13T13:15:43.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/485696487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:43.510 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:43.554 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:15:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:43 smithi167 ceph-mon[33178]: pgmap v723: 148 pgs: 148 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 889 B/s rd, 15 MiB/s wr, 1.06k op/s 2024-09-13T13:15:43.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:43 smithi167 ceph-mon[33178]: from='client.26227 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:15:43.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:43 smithi167 ceph-mon[33178]: osdmap e368: 8 total, 8 up, 8 in 2024-09-13T13:15:43.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/485696487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:43.576 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:43.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:43.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:43.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:43.710 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:43.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:43.811 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:43.834 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:43.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:43.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:43.936 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:43.972 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:43.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:44.074 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:44.114 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:44.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:44.178 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:15:44.215 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:44.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:44.251 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:44.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:44.352 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:44.378 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:44.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:44.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:44.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:44.516 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:44.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:44.618 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:44.654 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:44.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:44.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:44.756 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:44.786 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:44.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:44.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:15:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:15:44.888 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:44.925 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:44.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:44.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:45.027 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:45.061 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:45.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:45.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:45.163 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:45.197 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:45.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:45.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:45.332 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:45.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:45.434 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:45.455 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:45.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:45.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:45.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:45 smithi052 ceph-mon[36682]: pgmap v725: 148 pgs: 148 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 4.9 MiB/s wr, 1.28k op/s 2024-09-13T13:15:45.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:45 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:45.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:45 smithi052 ceph-mon[29471]: pgmap v725: 148 pgs: 148 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 4.9 MiB/s wr, 1.28k op/s 2024-09-13T13:15:45.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:45 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:45.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:45.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:45 smithi167 ceph-mon[33178]: pgmap v725: 148 pgs: 148 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 4.9 MiB/s wr, 1.28k op/s 2024-09-13T13:15:45.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:45 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:45.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:45.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:45.689 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:45.722 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:45.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:45.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:45.854 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:45.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:45.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:45.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:45.994 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:45.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:46.095 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:46.130 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:46.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:46.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:46.231 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:46.257 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:15:46.257 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:15:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:46.279 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:46.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:46.380 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:46.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:46.425 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:46.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:46.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:15:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:15:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:15:46.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:46.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:46.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:46.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:46.672 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:46.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:46.712 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:46.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:46.814 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:46.857 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:46.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:46.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:46.998 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:46.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:46.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:47.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:47.139 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:47.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:47.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T13:15:47.241 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:47.277 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:47.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:47.379 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:47.413 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:47.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:47.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:47.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:47 smithi052 ceph-mon[36682]: pgmap v726: 148 pgs: 148 active+clean; 7.2 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.7 MiB/s wr, 1.11k op/s 2024-09-13T13:15:47.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:47 smithi052 ceph-mon[29471]: pgmap v726: 148 pgs: 148 active+clean; 7.2 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.7 MiB/s wr, 1.11k op/s 2024-09-13T13:15:47.516 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:47.559 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:47.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:47.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:47 smithi167 ceph-mon[33178]: pgmap v726: 148 pgs: 148 active+clean; 7.2 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.7 MiB/s wr, 1.11k op/s 2024-09-13T13:15:47.635 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:15:47.660 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:47.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:47.701 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:47.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:47.802 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:47.844 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:47.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:47.945 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:47.952 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T13:15:47.989 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:47.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:47.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:48.020 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:48.091 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:48.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:48.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:48.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:48.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:48.267 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:48.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:48.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:48.288 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:15:48.288 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:15:48.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:48.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:48.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:48.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:48.529 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:48.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:48.631 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:48.670 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:48.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:48.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:48.772 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:48.807 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:48.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:48.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:48.909 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:48.955 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:48.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:49.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:49.090 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:49.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:49.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:49.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:49.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:49.278 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:49.321 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:49.373 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T13:15:49.382 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:49.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:49.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:49.483 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:49.521 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:49.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:49.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:49.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:49.667 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:49.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:49.709 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:15:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:49 smithi052 ceph-mon[36682]: pgmap v727: 148 pgs: 148 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 913 op/s 2024-09-13T13:15:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:49 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:49 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:15:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:49 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/778193900' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:15:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:49 smithi052 ceph-mon[36682]: from='client.26236 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:15:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:49 smithi052 ceph-mon[29471]: pgmap v727: 148 pgs: 148 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 913 op/s 2024-09-13T13:15:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:49 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:49 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:15:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:49 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/778193900' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:15:49.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:49 smithi052 ceph-mon[29471]: from='client.26236 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:15:49.769 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:49.771 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:15:49.779 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:15:49.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:49.818 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:49.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:49.919 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:49.953 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:49.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:49.953 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:49 smithi167 ceph-mon[33178]: pgmap v727: 148 pgs: 148 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 913 op/s 2024-09-13T13:15:49.954 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:49 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:15:49.954 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:49 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:15:49.954 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:49 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/778193900' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:15:49.954 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:49 smithi167 ceph-mon[33178]: from='client.26236 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:15:50.054 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:50.098 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:50.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:50.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:50.201 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:50.248 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:50.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:50.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:50.349 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:50.390 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:50.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:50.429 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:15:50.429 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:15:50.492 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:50.528 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:50.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:50.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:50.651 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:50.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:50.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:50.713 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T13:15:50.753 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:50.780 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:50.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:50.882 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:50.920 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:50.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:50.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:50.985 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:50.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:50.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[36682]: from='client.26236 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:15:50.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[36682]: osdmap e369: 8 total, 8 up, 8 in 2024-09-13T13:15:50.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2931234460' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:15:50.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[36682]: from='client.26242 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:15:50.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[36682]: 2.2 scrub starts 2024-09-13T13:15:50.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[36682]: 2.2 scrub ok 2024-09-13T13:15:50.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:50.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[29471]: from='client.26236 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:15:50.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[29471]: osdmap e369: 8 total, 8 up, 8 in 2024-09-13T13:15:50.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2931234460' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:15:50.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[29471]: from='client.26242 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:15:50.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[29471]: 2.2 scrub starts 2024-09-13T13:15:50.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:50 smithi052 ceph-mon[29471]: 2.2 scrub ok 2024-09-13T13:15:51.022 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:51.055 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:51.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:51.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:51.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:50 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:15:51.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:50 smithi167 ceph-mon[33178]: from='client.26236 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:15:51.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:50 smithi167 ceph-mon[33178]: osdmap e369: 8 total, 8 up, 8 in 2024-09-13T13:15:51.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:50 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2931234460' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:15:51.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:50 smithi167 ceph-mon[33178]: from='client.26242 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:15:51.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:50 smithi167 ceph-mon[33178]: 2.2 scrub starts 2024-09-13T13:15:51.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:50 smithi167 ceph-mon[33178]: 2.2 scrub ok 2024-09-13T13:15:51.159 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:51.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:51.182 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:51.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:51.282 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:51.314 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:51.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:51.416 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:51.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:51.456 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:51.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:51.486 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T13:15:51.487 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:15:51.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:51.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:51.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:51.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:51.686 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:51.730 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:51.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:51.831 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:51.869 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:51.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:51.970 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:51.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:51 smithi167 ceph-mon[33178]: pgmap v729: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-09-13T13:15:51.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:51 smithi167 ceph-mon[33178]: 4.19 scrub starts 2024-09-13T13:15:51.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:51 smithi167 ceph-mon[33178]: 4.19 scrub ok 2024-09-13T13:15:51.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:51 smithi167 ceph-mon[33178]: 4.1e scrub starts 2024-09-13T13:15:51.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:51 smithi167 ceph-mon[33178]: 4.1e scrub ok 2024-09-13T13:15:51.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:51 smithi167 ceph-mon[33178]: from='client.26242 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:15:51.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:51 smithi167 ceph-mon[33178]: osdmap e370: 8 total, 8 up, 8 in 2024-09-13T13:15:51.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:51 smithi167 ceph-mon[33178]: 6.1a scrub starts 2024-09-13T13:15:51.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:51 smithi167 ceph-mon[33178]: 6.1a scrub ok 2024-09-13T13:15:51.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:51 smithi167 ceph-mon[33178]: 6.11 scrub starts 2024-09-13T13:15:51.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:51 smithi167 ceph-mon[33178]: 6.11 scrub ok 2024-09-13T13:15:51.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:51 smithi167 ceph-mon[33178]: 5.16 scrub starts 2024-09-13T13:15:51.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:51 smithi167 ceph-mon[33178]: 5.16 scrub ok 2024-09-13T13:15:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[36682]: pgmap v729: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-09-13T13:15:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[36682]: 4.19 scrub starts 2024-09-13T13:15:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[36682]: 4.19 scrub ok 2024-09-13T13:15:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[36682]: 4.1e scrub starts 2024-09-13T13:15:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[36682]: 4.1e scrub ok 2024-09-13T13:15:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[36682]: from='client.26242 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:15:51.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[36682]: osdmap e370: 8 total, 8 up, 8 in 2024-09-13T13:15:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[36682]: 6.1a scrub starts 2024-09-13T13:15:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[36682]: 6.1a scrub ok 2024-09-13T13:15:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[36682]: 6.11 scrub starts 2024-09-13T13:15:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[36682]: 6.11 scrub ok 2024-09-13T13:15:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[36682]: 5.16 scrub starts 2024-09-13T13:15:51.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[36682]: 5.16 scrub ok 2024-09-13T13:15:51.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[29471]: pgmap v729: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-09-13T13:15:51.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[29471]: 4.19 scrub starts 2024-09-13T13:15:51.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[29471]: 4.19 scrub ok 2024-09-13T13:15:51.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[29471]: 4.1e scrub starts 2024-09-13T13:15:51.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[29471]: 4.1e scrub ok 2024-09-13T13:15:51.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[29471]: from='client.26242 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:15:51.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[29471]: osdmap e370: 8 total, 8 up, 8 in 2024-09-13T13:15:51.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[29471]: 6.1a scrub starts 2024-09-13T13:15:51.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[29471]: 6.1a scrub ok 2024-09-13T13:15:51.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[29471]: 6.11 scrub starts 2024-09-13T13:15:51.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[29471]: 6.11 scrub ok 2024-09-13T13:15:51.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[29471]: 5.16 scrub starts 2024-09-13T13:15:51.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:51 smithi052 ceph-mon[29471]: 5.16 scrub ok 2024-09-13T13:15:52.025 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:52.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:52.084 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:52.091 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:15:52.091 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:15:52.127 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:52.132 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:15:52.133 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:15:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:52.158 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:52.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:52.168 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:15:52.168 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:15:52.202 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:15:52.203 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:15:52.236 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:15:52.237 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:15:52.259 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:52.274 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:15:52.274 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:15:52.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:52.304 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:52.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:52.307 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:15:52.308 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:15:52.342 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:15:52.342 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:15:52.406 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:52.440 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:52.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:52.453 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T13:15:52.541 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:15:52.542 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:52.582 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:52.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:52.683 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:52.724 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:52.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:52.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:52.848 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:15:52.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:52.866 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:52.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:52.967 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:52 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:15:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:52 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:15:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:52 smithi052 ceph-mon[36682]: 5.3 scrub starts 2024-09-13T13:15:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:52 smithi052 ceph-mon[36682]: 5.3 scrub ok 2024-09-13T13:15:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:52 smithi052 ceph-mon[36682]: osdmap e371: 8 total, 8 up, 8 in 2024-09-13T13:15:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:52 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1359421516' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:52.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:52 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:15:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:52 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:15:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:52 smithi052 ceph-mon[29471]: 5.3 scrub starts 2024-09-13T13:15:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:52 smithi052 ceph-mon[29471]: 5.3 scrub ok 2024-09-13T13:15:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:52 smithi052 ceph-mon[29471]: osdmap e371: 8 total, 8 up, 8 in 2024-09-13T13:15:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:52 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1359421516' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:53.004 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:53.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:53.034 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:52 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:15:53.034 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:52 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:15:53.034 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:52 smithi167 ceph-mon[33178]: 5.3 scrub starts 2024-09-13T13:15:53.034 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:52 smithi167 ceph-mon[33178]: 5.3 scrub ok 2024-09-13T13:15:53.034 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:52 smithi167 ceph-mon[33178]: osdmap e371: 8 total, 8 up, 8 in 2024-09-13T13:15:53.034 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:52 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1359421516' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:53.107 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:53.157 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:53.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:53.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:53.259 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:53.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:53.296 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:53.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:53.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:53.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:53.432 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:53.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:53.482 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:15:53.482 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:15:53.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.483 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:148585 10.d 10:bced1570:::benchmark_data_smithi052_153703_object21205:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-13T13:15:53.483 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:53.471833+0000", 2024-09-13T13:15:53.483 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0086146309999999993, 2024-09-13T13:15:53.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086447880000000005, 2024-09-13T13:15:53.483 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:53.483 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:53.483 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:15:53.483 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:53.483 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:53.483 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:53.483 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 148585 2024-09-13T13:15:53.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.471833+0000", 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.471833+0000", 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:53.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.471835+0000", 2024-09-13T13:15:53.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6309999999999999e-06 2024-09-13T13:15:53.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:53.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.471839+0000", 2024-09-13T13:15:53.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4129999999999999e-06 2024-09-13T13:15:53.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.471847+0000", 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9929999999999995e-06 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.471853+0000", 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0390000000000001e-06 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472545+0000", 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00069216999999999998 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:53.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472729+0000", 2024-09-13T13:15:53.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018413199999999999 2024-09-13T13:15:53.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:15:53.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.475012+0000", 2024-09-13T13:15:53.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002282554 2024-09-13T13:15:53.488 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.488 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:53.488 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.488 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:148587 10.7 10:e4edde67:::benchmark_data_smithi052_153703_object21207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-13T13:15:53.489 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:53.472667+0000", 2024-09-13T13:15:53.489 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0077811410000000001, 2024-09-13T13:15:53.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078801470000000005, 2024-09-13T13:15:53.489 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:53.489 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:53.489 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:15:53.489 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:53.489 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:53.489 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:53.489 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 148587 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472667+0000", 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472667+0000", 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:53.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472668+0000", 2024-09-13T13:15:53.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.19e-06 2024-09-13T13:15:53.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:53.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472672+0000", 2024-09-13T13:15:53.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8179999999999997e-06 2024-09-13T13:15:53.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:53.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472679+0000", 2024-09-13T13:15:53.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.024e-06 2024-09-13T13:15:53.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:53.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472685+0000", 2024-09-13T13:15:53.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1260000000000003e-06 2024-09-13T13:15:53.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:53.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472863+0000", 2024-09-13T13:15:53.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000177932 2024-09-13T13:15:53.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:53.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472973+0000", 2024-09-13T13:15:53.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011061900000000001 2024-09-13T13:15:53.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:15:53.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.473086+0000", 2024-09-13T13:15:53.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011271499999999999 2024-09-13T13:15:53.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.494 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:53.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:148590 10.a 10:53f6382b:::benchmark_data_smithi052_153703_object21210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:53.472806+0000", 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0076417840000000004, 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078006459999999996, 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 148590 2024-09-13T13:15:53.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472806+0000", 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472806+0000", 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:53.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472807+0000", 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.054e-06 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472813+0000", 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2740000000000004e-06 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472820+0000", 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6939999999999998e-06 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:53.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472826+0000", 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5679999999999999e-06 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472870+0000", 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4382999999999998e-05 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.472961+0000", 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1118000000000004e-05 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:15:53.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.473058+0000", 2024-09-13T13:15:53.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7089000000000005e-05 2024-09-13T13:15:53.499 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.499 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:53.499 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.500 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:148591 10.7 10:e6b888ea:::benchmark_data_smithi052_153703_object21211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-13T13:15:53.500 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:53.473630+0000", 2024-09-13T13:15:53.500 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0068173319999999997, 2024-09-13T13:15:53.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070359919999999996, 2024-09-13T13:15:53.500 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:53.500 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:53.500 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:15:53.500 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:53.500 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:53.500 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 148591 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.473630+0000", 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.473630+0000", 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.473632+0000", 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1969999999999999e-06 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.473636+0000", 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4020000000000004e-06 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.473646+0000", 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0217000000000001e-05 2024-09-13T13:15:53.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.473653+0000", 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6019999999999996e-06 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.476883+0000", 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032297879999999999 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.476986+0000", 2024-09-13T13:15:53.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103084 2024-09-13T13:15:53.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:15:53.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.477092+0000", 2024-09-13T13:15:53.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106103 2024-09-13T13:15:53.504 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.504 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:53.504 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.504 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.26071.0:148587 10.7 e371/334)", 2024-09-13T13:15:53.504 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:53.476511+0000", 2024-09-13T13:15:53.504 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0039366499999999999, 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042086650000000003, 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "queued for pg", 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.476511+0000", 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.476511+0000", 2024-09-13T13:15:53.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.476513+0000", 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9659999999999999e-06 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.476518+0000", 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8130000000000003e-06 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.476528+0000", 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0526000000000001e-05 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.476535+0000", 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9140000000000002e-06 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:148596 10.0 10:02fe754b:::benchmark_data_smithi052_153703_object21216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:53.479405+0000", 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0010426389999999999, 2024-09-13T13:15:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013530700000000001, 2024-09-13T13:15:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:15:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:15:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:15:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:15:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 148596 2024-09-13T13:15:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.479405+0000", 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.479405+0000", 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.479407+0000", 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.272e-06 2024-09-13T13:15:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.479413+0000", 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5589999999999996e-06 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.479431+0000", 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7739000000000001e-05 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.479438+0000", 2024-09-13T13:15:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6079999999999998e-06 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.479497+0000", 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8499999999999999e-05 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.479598+0000", 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100888 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.479698+0000", 2024-09-13T13:15:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9868e-05 2024-09-13T13:15:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.26071.0:148591 10.7 e371/334)", 2024-09-13T13:15:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:53.480595+0000", 2024-09-13T13:15:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4294967295.9998531, 2024-09-13T13:15:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022414000000000001, 2024-09-13T13:15:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:15:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:15:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "queued for pg", 2024-09-13T13:15:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.480595+0000", 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.480595+0000", 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.480596+0000", 2024-09-13T13:15:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.528e-06 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.480601+0000", 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9509999999999997e-06 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.480609+0000", 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3529999999999995e-06 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:15:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:53.480616+0000", 2024-09-13T13:15:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1999999999999999e-06 2024-09-13T13:15:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:15:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:15:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:15:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 7 2024-09-13T13:15:53.515 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:15:53.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:53.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:53.569 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:53.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:53.656 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:15:53.659 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:15:53.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:53.705 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:53.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:53.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:53.807 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:53.852 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:53.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:53.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:53.954 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:53.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:53 smithi052 ceph-mon[36682]: pgmap v732: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.32k op/s 2024-09-13T13:15:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:53 smithi052 ceph-mon[36682]: 3.1b scrub starts 2024-09-13T13:15:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:53 smithi052 ceph-mon[36682]: 3.1b scrub ok 2024-09-13T13:15:53.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:53 smithi052 ceph-mon[29471]: pgmap v732: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.32k op/s 2024-09-13T13:15:53.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:53 smithi052 ceph-mon[29471]: 3.1b scrub starts 2024-09-13T13:15:53.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:53 smithi052 ceph-mon[29471]: 3.1b scrub ok 2024-09-13T13:15:53.990 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:53.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:54.038 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:53 smithi167 ceph-mon[33178]: pgmap v732: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.32k op/s 2024-09-13T13:15:54.039 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:53 smithi167 ceph-mon[33178]: 3.1b scrub starts 2024-09-13T13:15:54.039 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:53 smithi167 ceph-mon[33178]: 3.1b scrub ok 2024-09-13T13:15:54.091 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:54.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:54.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:54.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:54.265 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:54.281 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:54.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:54.382 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:54.428 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:54.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:54.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T13:15:54.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:54.534 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:54.573 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:54.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:54.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:54.713 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:54.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:54.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:15:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:15:54.815 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:54.848 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:15:54.852 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:54.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:54.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:54.954 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:54.975 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:54.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:55.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:54 smithi167 ceph-mon[33178]: from='client.36317 v1:172.21.15.52:0/3196325096' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:15:55.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:54 smithi167 ceph-mon[33178]: 10.6 scrub starts 2024-09-13T13:15:55.075 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:54 smithi052 ceph-mon[36682]: from='client.36317 v1:172.21.15.52:0/3196325096' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:15:55.075 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:54 smithi052 ceph-mon[36682]: 10.6 scrub starts 2024-09-13T13:15:55.076 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:54 smithi052 ceph-mon[29471]: from='client.36317 v1:172.21.15.52:0/3196325096' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:15:55.076 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:54 smithi052 ceph-mon[29471]: 10.6 scrub starts 2024-09-13T13:15:55.077 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:55.119 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:55.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:55.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:55.128 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":732,"stamp":"2024-09-13T13:15:51.930406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7238064691,"num_objects":110635,"num_object_clones":0,"num_object_copies":331905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110635,"num_whiteouts":0,"num_read":1437,"num_read_kb":1296,"num_write":145155,"num_write_kb":8153802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":594037,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91877,"ondisk_log_size":91877,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":25627640,"kb_used_data":21492708,"kb_used_omap":15,"kb_used_meta":4134512,"kb_avail":724333576,"statfs":{"total":767960285184,"available":741717581824,"internally_reserved":0,"allocated":22008532992,"data_stored":22005204941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":4233741081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-517734400,"num_objects":-7900,"num_object_clones":0,"num_object_copies":-23700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7900,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7900,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003661"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":657,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.113711+0000","last_change":"2024-09-13T13:10:32.368663+0000","last_active":"2024-09-13T13:15:42.113711+0000","last_peered":"2024-09-13T13:15:42.113711+0000","last_clean":"2024-09-13T13:15:42.113711+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:15:42.113711+0000","last_undegraded":"2024-09-13T13:15:42.113711+0000","last_fullsized":"2024-09-13T13:15:42.113711+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.368588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.368588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.615999+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":155,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.114433+0000","last_change":"2024-09-13T13:15:04.344174+0000","last_active":"2024-09-13T13:15:42.114433+0000","last_peered":"2024-09-13T13:15:42.114433+0000","last_clean":"2024-09-13T13:15:42.114433+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:15:42.114433+0000","last_undegraded":"2024-09-13T13:15:42.114433+0000","last_fullsized":"2024-09-13T13:15:42.114433+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:15:04.344092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.344092+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.313962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":658,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.122154+0000","last_change":"2024-09-13T13:14:12.865175+0000","last_active":"2024-09-13T13:15:42.122154+0000","last_peered":"2024-09-13T13:15:42.122154+0000","last_clean":"2024-09-13T13:15:42.122154+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:15:42.122154+0000","last_undegraded":"2024-09-13T13:15:42.122154+0000","last_fullsized":"2024-09-13T13:15:42.122154+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:12.865101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:12.865101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:17.268032+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":145,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.109937+0000","last_change":"2024-09-13T13:13:17.976677+0000","last_active":"2024-09-13T13:15:42.109937+0000","last_peered":"2024-09-13T13:15:42.109937+0000","last_clean":"2024-09-13T13:15:42.109937+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:15:42.109937+0000","last_undegraded":"2024-09-13T13:15:42.109937+0000","last_fullsized":"2024-09-13T13:15:42.109937+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:17.976607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:17.976607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":663,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.111960+0000","last_change":"2024-09-13T13:11:47.852762+0000","last_active":"2024-09-13T13:15:42.111960+0000","last_peered":"2024-09-13T13:15:42.111960+0000","last_clean":"2024-09-13T13:15:42.111960+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:15:42.111960+0000","last_undegraded":"2024-09-13T13:15:42.111960+0000","last_fullsized":"2024-09-13T13:15:42.111960+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:15.241902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":685,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.108749+0000","last_change":"2024-09-13T13:15:24.253436+0000","last_active":"2024-09-13T13:15:42.108749+0000","last_peered":"2024-09-13T13:15:42.108749+0000","last_clean":"2024-09-13T13:15:42.108749+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:15:42.108749+0000","last_undegraded":"2024-09-13T13:15:42.108749+0000","last_fullsized":"2024-09-13T13:15:42.108749+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:24.253369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:24.253369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:50.007360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":720,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.111481+0000","last_change":"2024-09-13T13:14:26.696961+0000","last_active":"2024-09-13T13:15:42.111481+0000","last_peered":"2024-09-13T13:15:42.111481+0000","last_clean":"2024-09-13T13:15:42.111481+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:15:42.111481+0000","last_undegraded":"2024-09-13T13:15:42.111481+0000","last_fullsized":"2024-09-13T13:15:42.111481+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:14:26.696863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:26.696863+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:45.303770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":137,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.114340+0000","last_change":"2024-09-13T13:12:52.997865+0000","last_active":"2024-09-13T13:15:42.114340+0000","last_peered":"2024-09-13T13:15:42.114340+0000","last_clean":"2024-09-13T13:15:42.114340+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:15:42.114340+0000","last_undegraded":"2024-09-13T13:15:42.114340+0000","last_fullsized":"2024-09-13T13:15:42.114340+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:17.305608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:17.305608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.322592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":727,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.116968+0000","last_change":"2024-09-13T13:13:41.518378+0000","last_active":"2024-09-13T13:15:42.116968+0000","last_peered":"2024-09-13T13:15:42.116968+0000","last_clean":"2024-09-13T13:15:42.116968+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:15:42.116968+0000","last_undegraded":"2024-09-13T13:15:42.116968+0000","last_fullsized":"2024-09-13T13:15:42.116968+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:41.518322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.518322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:12.339477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":147,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.110045+0000","last_change":"2024-09-13T13:12:53.033861+0000","last_active":"2024-09-13T13:15:42.110045+0000","last_peered":"2024-09-13T13:15:42.110045+0000","last_clean":"2024-09-13T13:15:42.110045+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:15:42.110045+0000","last_undegraded":"2024-09-13T13:15:42.110045+0000","last_fullsized":"2024-09-13T13:15:42.110045+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:31.738685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.738685+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.301470+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":664,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.117804+0000","last_change":"2024-09-13T13:13:43.501368+0000","last_active":"2024-09-13T13:15:42.117804+0000","last_peered":"2024-09-13T13:15:42.117804+0000","last_clean":"2024-09-13T13:15:42.117804+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:15:42.117804+0000","last_undegraded":"2024-09-13T13:15:42.117804+0000","last_fullsized":"2024-09-13T13:15:42.117804+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:43.501206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:43.501206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.419736+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":668,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.113242+0000","last_change":"2024-09-13T13:15:25.628234+0000","last_active":"2024-09-13T13:15:42.113242+0000","last_peered":"2024-09-13T13:15:42.113242+0000","last_clean":"2024-09-13T13:15:42.113242+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:15:42.113242+0000","last_undegraded":"2024-09-13T13:15:42.113242+0000","last_fullsized":"2024-09-13T13:15:42.113242+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.628156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.628156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:41.988194+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":694,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.116323+0000","last_change":"2024-09-13T13:12:52.951531+0000","last_active":"2024-09-13T13:15:42.116323+0000","last_peered":"2024-09-13T13:15:42.116323+0000","last_clean":"2024-09-13T13:15:42.116323+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:15:42.116323+0000","last_undegraded":"2024-09-13T13:15:42.116323+0000","last_fullsized":"2024-09-13T13:15:42.116323+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:16.989972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:16.989972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.371147+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":151,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.109214+0000","last_change":"2024-09-13T13:15:03.348187+0000","last_active":"2024-09-13T13:15:42.109214+0000","last_peered":"2024-09-13T13:15:42.109214+0000","last_clean":"2024-09-13T13:15:42.109214+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:15:42.109214+0000","last_undegraded":"2024-09-13T13:15:42.109214+0000","last_fullsized":"2024-09-13T13:15:42.109214+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:15:03.348115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.348115+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.237423+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":685,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.118046+0000","last_change":"2024-09-13T13:13:54.122282+0000","last_active":"2024-09-13T13:15:42.118046+0000","last_peered":"2024-09-13T13:15:42.118046+0000","last_clean":"2024-09-13T13:15:42.118046+0000","last_became_active":"2024-09-13T13:12:52.999970+0000","last_became_peered":"2024-09-13T13:12:52.999970+0000","last_unstale":"2024-09-13T13:15:42.118046+0000","last_undegraded":"2024-09-13T13:15:42.118046+0000","last_fullsized":"2024-09-13T13:15:42.118046+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.122222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.122222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:17.725218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":637,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.123101+0000","last_change":"2024-09-13T13:13:02.818644+0000","last_active":"2024-09-13T13:15:42.123101+0000","last_peered":"2024-09-13T13:15:42.123101+0000","last_clean":"2024-09-13T13:15:42.123101+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:15:42.123101+0000","last_undegraded":"2024-09-13T13:15:42.123101+0000","last_fullsized":"2024-09-13T13:15:42.123101+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:02.818543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:02.818543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":141,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.115169+0000","last_change":"2024-09-13T13:12:53.014177+0000","last_active":"2024-09-13T13:15:42.115169+0000","last_peered":"2024-09-13T13:15:42.115169+0000","last_clean":"2024-09-13T13:15:42.115169+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:15:42.115169+0000","last_undegraded":"2024-09-13T13:15:42.115169+0000","last_fullsized":"2024-09-13T13:15:42.115169+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:28.804735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:28.804735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":685,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.113927+0000","last_change":"2024-09-13T13:14:18.624877+0000","last_active":"2024-09-13T13:15:42.113927+0000","last_peered":"2024-09-13T13:15:42.113927+0000","last_clean":"2024-09-13T13:15:42.113927+0000","last_became_active":"2024-09-13T13:12:53.008505+0000","last_became_peered":"2024-09-13T13:12:53.008505+0000","last_unstale":"2024-09-13T13:15:42.113927+0000","last_undegraded":"2024-09-13T13:15:42.113927+0000","last_fullsized":"2024-09-13T13:15:42.113927+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:18.624745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:18.624745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:43.578763+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":712,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.110275+0000","last_change":"2024-09-13T13:12:40.125646+0000","last_active":"2024-09-13T13:15:42.110275+0000","last_peered":"2024-09-13T13:15:42.110275+0000","last_clean":"2024-09-13T13:15:42.110275+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:15:42.110275+0000","last_undegraded":"2024-09-13T13:15:42.110275+0000","last_fullsized":"2024-09-13T13:15:42.110275+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:12:40.125545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.125545+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.676016+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":687,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.123396+0000","last_change":"2024-09-13T13:14:52.196086+0000","last_active":"2024-09-13T13:15:42.123396+0000","last_peered":"2024-09-13T13:15:42.123396+0000","last_clean":"2024-09-13T13:15:42.123396+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:15:42.123396+0000","last_undegraded":"2024-09-13T13:15:42.123396+0000","last_fullsized":"2024-09-13T13:15:42.123396+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.196041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.502726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":665,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.111579+0000","last_change":"2024-09-13T13:13:03.150524+0000","last_active":"2024-09-13T13:15:42.111579+0000","last_peered":"2024-09-13T13:15:42.111579+0000","last_clean":"2024-09-13T13:15:42.111579+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:15:42.111579+0000","last_undegraded":"2024-09-13T13:15:42.111579+0000","last_fullsized":"2024-09-13T13:15:42.111579+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:03.150442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.150442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.295693+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":145,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.109815+0000","last_change":"2024-09-13T13:13:39.876364+0000","last_active":"2024-09-13T13:15:42.109815+0000","last_peered":"2024-09-13T13:15:42.109815+0000","last_clean":"2024-09-13T13:15:42.109815+0000","last_became_active":"2024-09-13T13:12:52.995055+0000","last_became_peered":"2024-09-13T13:12:52.995055+0000","last_unstale":"2024-09-13T13:15:42.109815+0000","last_undegraded":"2024-09-13T13:15:42.109815+0000","last_fullsized":"2024-09-13T13:15:42.109815+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:39.876294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:39.876294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:46.391189+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":730,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.108490+0000","last_change":"2024-09-13T13:13:41.164741+0000","last_active":"2024-09-13T13:15:42.108490+0000","last_peered":"2024-09-13T13:15:42.108490+0000","last_clean":"2024-09-13T13:15:42.108490+0000","last_became_active":"2024-09-13T13:12:52.953819+0000","last_became_peered":"2024-09-13T13:12:52.953819+0000","last_unstale":"2024-09-13T13:15:42.108490+0000","last_undegraded":"2024-09-13T13:15:42.108490+0000","last_fullsized":"2024-09-13T13:15:42.108490+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:13:41.164656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.164656+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:47.517448+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":147,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.114623+0000","last_change":"2024-09-13T13:15:29.208631+0000","last_active":"2024-09-13T13:15:42.114623+0000","last_peered":"2024-09-13T13:15:42.114623+0000","last_clean":"2024-09-13T13:15:42.114623+0000","last_became_active":"2024-09-13T13:13:06.646781+0000","last_became_peered":"2024-09-13T13:13:06.646781+0000","last_unstale":"2024-09-13T13:15:42.114623+0000","last_undegraded":"2024-09-13T13:15:42.114623+0000","last_fullsized":"2024-09-13T13:15:42.114623+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:29.208576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:29.208576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:34.495308+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":172,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.116125+0000","last_change":"2024-09-13T13:12:42.975335+0000","last_active":"2024-09-13T13:15:42.116125+0000","last_peered":"2024-09-13T13:15:42.116125+0000","last_clean":"2024-09-13T13:15:42.116125+0000","last_became_active":"2024-09-13T13:12:39.954363+0000","last_became_peered":"2024-09-13T13:12:39.954363+0000","last_unstale":"2024-09-13T13:15:42.116125+0000","last_undegraded":"2024-09-13T13:15:42.116125+0000","last_fullsized":"2024-09-13T13:15:42.116125+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:12:42.975266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:42.975266+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.483880+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":689,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.122297+0000","last_change":"2024-09-13T13:15:23.245116+0000","last_active":"2024-09-13T13:15:42.122297+0000","last_peered":"2024-09-13T13:15:42.122297+0000","last_clean":"2024-09-13T13:15:42.122297+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:15:42.122297+0000","last_undegraded":"2024-09-13T13:15:42.122297+0000","last_fullsized":"2024-09-13T13:15:42.122297+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:15:23.245047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.245047+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.758250+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":707,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.114272+0000","last_change":"2024-09-13T13:14:21.632726+0000","last_active":"2024-09-13T13:15:42.114272+0000","last_peered":"2024-09-13T13:15:42.114272+0000","last_clean":"2024-09-13T13:15:42.114272+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:15:42.114272+0000","last_undegraded":"2024-09-13T13:15:42.114272+0000","last_fullsized":"2024-09-13T13:15:42.114272+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:14:21.630012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:21.630012+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"306'12","reported_seq":835,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717573+0000","last_change":"2024-09-13T13:14:21.530680+0000","last_active":"2024-09-13T13:15:50.717573+0000","last_peered":"2024-09-13T13:15:50.717573+0000","last_clean":"2024-09-13T13:15:50.717573+0000","last_became_active":"2024-09-13T13:12:52.947886+0000","last_became_peered":"2024-09-13T13:12:52.947886+0000","last_unstale":"2024-09-13T13:15:50.717573+0000","last_undegraded":"2024-09-13T13:15:50.717573+0000","last_fullsized":"2024-09-13T13:15:50.717573+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'12","last_scrub_stamp":"2024-09-13T13:14:21.530612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:21.530612+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.292277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"368'8937","reported_seq":9016,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:48.140917+0000","last_change":"2024-09-13T13:13:59.149493+0000","last_active":"2024-09-13T13:15:48.140917+0000","last_peered":"2024-09-13T13:15:48.140917+0000","last_clean":"2024-09-13T13:15:48.140917+0000","last_became_active":"2024-09-13T13:13:59.149215+0000","last_became_peered":"2024-09-13T13:13:59.149215+0000","last_unstale":"2024-09-13T13:15:48.140917+0000","last_undegraded":"2024-09-13T13:15:48.140917+0000","last_fullsized":"2024-09-13T13:15:48.140917+0000","mapping_epoch":334,"log_start":"348'4300","ondisk_log_start":"348'4300","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461176832,"num_objects":7037,"num_object_clones":0,"num_object_copies":21111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8937,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":759,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.124077+0000","last_change":"2024-09-13T13:15:02.354689+0000","last_active":"2024-09-13T13:15:42.124077+0000","last_peered":"2024-09-13T13:15:42.124077+0000","last_clean":"2024-09-13T13:15:42.124077+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:15:42.124077+0000","last_undegraded":"2024-09-13T13:15:42.124077+0000","last_fullsized":"2024-09-13T13:15:42.124077+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:15:02.354591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:02.354591+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:22.985661+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":689,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.111261+0000","last_change":"2024-09-13T13:15:29.205122+0000","last_active":"2024-09-13T13:15:42.111261+0000","last_peered":"2024-09-13T13:15:42.111261+0000","last_clean":"2024-09-13T13:15:42.111261+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:15:42.111261+0000","last_undegraded":"2024-09-13T13:15:42.111261+0000","last_fullsized":"2024-09-13T13:15:42.111261+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:29.205058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:29.205058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.374619+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":185,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.417642+0000","last_change":"2024-09-13T13:15:32.212185+0000","last_active":"2024-09-13T13:15:42.417642+0000","last_peered":"2024-09-13T13:15:42.417642+0000","last_clean":"2024-09-13T13:15:42.417642+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:15:42.417642+0000","last_undegraded":"2024-09-13T13:15:42.417642+0000","last_fullsized":"2024-09-13T13:15:42.417642+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:15:32.212101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.212101+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:33.253768+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":669,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.116566+0000","last_change":"2024-09-13T13:13:57.546725+0000","last_active":"2024-09-13T13:15:42.116566+0000","last_peered":"2024-09-13T13:15:42.116566+0000","last_clean":"2024-09-13T13:15:42.116566+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:15:42.116566+0000","last_undegraded":"2024-09-13T13:15:42.116566+0000","last_fullsized":"2024-09-13T13:15:42.116566+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.546671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.546671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:18.300458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"368'8947","reported_seq":9026,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:48.144052+0000","last_change":"2024-09-13T13:13:58.962128+0000","last_active":"2024-09-13T13:15:48.144052+0000","last_peered":"2024-09-13T13:15:48.144052+0000","last_clean":"2024-09-13T13:15:48.144052+0000","last_became_active":"2024-09-13T13:13:58.961386+0000","last_became_peered":"2024-09-13T13:13:58.961386+0000","last_unstale":"2024-09-13T13:15:48.144052+0000","last_undegraded":"2024-09-13T13:15:48.144052+0000","last_fullsized":"2024-09-13T13:15:48.144052+0000","mapping_epoch":334,"log_start":"349'4400","ondisk_log_start":"349'4400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.366364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461307904,"num_objects":7039,"num_object_clones":0,"num_object_copies":21117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8947,"num_write_kb":511552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":696,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.111057+0000","last_change":"2024-09-13T13:13:50.232646+0000","last_active":"2024-09-13T13:15:42.111057+0000","last_peered":"2024-09-13T13:15:42.111057+0000","last_clean":"2024-09-13T13:15:42.111057+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:15:42.111057+0000","last_undegraded":"2024-09-13T13:15:42.111057+0000","last_fullsized":"2024-09-13T13:15:42.111057+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:13:50.232588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.232588+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.213810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":663,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.111881+0000","last_change":"2024-09-13T13:13:54.313641+0000","last_active":"2024-09-13T13:15:42.111881+0000","last_peered":"2024-09-13T13:15:42.111881+0000","last_clean":"2024-09-13T13:15:42.111881+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:15:42.111881+0000","last_undegraded":"2024-09-13T13:15:42.111881+0000","last_fullsized":"2024-09-13T13:15:42.111881+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:54.313570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.313570+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:18.294176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":145,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.108969+0000","last_change":"2024-09-13T13:15:00.323883+0000","last_active":"2024-09-13T13:15:42.108969+0000","last_peered":"2024-09-13T13:15:42.108969+0000","last_clean":"2024-09-13T13:15:42.108969+0000","last_became_active":"2024-09-13T13:12:53.006199+0000","last_became_peered":"2024-09-13T13:12:53.006199+0000","last_unstale":"2024-09-13T13:15:42.108969+0000","last_undegraded":"2024-09-13T13:15:42.108969+0000","last_fullsized":"2024-09-13T13:15:42.108969+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.323784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.323784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:00.914935+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":153,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.110088+0000","last_change":"2024-09-13T13:14:56.333854+0000","last_active":"2024-09-13T13:15:42.110088+0000","last_peered":"2024-09-13T13:15:42.110088+0000","last_clean":"2024-09-13T13:15:42.110088+0000","last_became_active":"2024-09-13T13:12:53.001045+0000","last_became_peered":"2024-09-13T13:12:53.001045+0000","last_unstale":"2024-09-13T13:15:42.110088+0000","last_undegraded":"2024-09-13T13:15:42.110088+0000","last_fullsized":"2024-09-13T13:15:42.110088+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:56.333778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:56.333778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:58.872471+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"370'9076","reported_seq":9159,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.130446+0000","last_change":"2024-09-13T13:13:58.966300+0000","last_active":"2024-09-13T13:15:51.130446+0000","last_peered":"2024-09-13T13:15:51.130446+0000","last_clean":"2024-09-13T13:15:51.130446+0000","last_became_active":"2024-09-13T13:13:58.966015+0000","last_became_peered":"2024-09-13T13:13:58.966015+0000","last_unstale":"2024-09-13T13:15:51.130446+0000","last_undegraded":"2024-09-13T13:15:51.130446+0000","last_fullsized":"2024-09-13T13:15:51.130446+0000","mapping_epoch":334,"log_start":"345'3000","ondisk_log_start":"345'3000","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.310606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449445888,"num_objects":6858,"num_object_clones":0,"num_object_copies":20574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9076,"num_write_kb":509888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":161,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.109133+0000","last_change":"2024-09-13T13:13:50.716312+0000","last_active":"2024-09-13T13:15:42.109133+0000","last_peered":"2024-09-13T13:15:42.109133+0000","last_clean":"2024-09-13T13:15:42.109133+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:15:42.109133+0000","last_undegraded":"2024-09-13T13:15:42.109133+0000","last_fullsized":"2024-09-13T13:15:42.109133+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:13:50.716243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.716243+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:10.352612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":671,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.114539+0000","last_change":"2024-09-13T13:13:57.886950+0000","last_active":"2024-09-13T13:15:42.114539+0000","last_peered":"2024-09-13T13:15:42.114539+0000","last_clean":"2024-09-13T13:15:42.114539+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:15:42.114539+0000","last_undegraded":"2024-09-13T13:15:42.114539+0000","last_fullsized":"2024-09-13T13:15:42.114539+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:57.886879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.886879+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:12.598128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":867,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.418525+0000","last_change":"2024-09-13T13:13:55.316917+0000","last_active":"2024-09-13T13:15:42.418525+0000","last_peered":"2024-09-13T13:15:42.418525+0000","last_clean":"2024-09-13T13:15:42.418525+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:15:42.418525+0000","last_undegraded":"2024-09-13T13:15:42.418525+0000","last_fullsized":"2024-09-13T13:15:42.418525+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:13:55.316860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.316860+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.251395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":291,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717331+0000","last_change":"2024-09-13T13:14:59.391483+0000","last_active":"2024-09-13T13:15:50.717331+0000","last_peered":"2024-09-13T13:15:50.717331+0000","last_clean":"2024-09-13T13:15:50.717331+0000","last_became_active":"2024-09-13T13:12:39.948363+0000","last_became_peered":"2024-09-13T13:12:39.948363+0000","last_unstale":"2024-09-13T13:15:50.717331+0000","last_undegraded":"2024-09-13T13:15:50.717331+0000","last_fullsized":"2024-09-13T13:15:50.717331+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.391418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.391418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:15.991030+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"368'8861","reported_seq":8940,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:48.162870+0000","last_change":"2024-09-13T13:13:59.149331+0000","last_active":"2024-09-13T13:15:48.162870+0000","last_peered":"2024-09-13T13:15:48.162870+0000","last_clean":"2024-09-13T13:15:48.162870+0000","last_became_active":"2024-09-13T13:13:59.149046+0000","last_became_peered":"2024-09-13T13:13:59.149046+0000","last_unstale":"2024-09-13T13:15:48.162870+0000","last_undegraded":"2024-09-13T13:15:48.162870+0000","last_fullsized":"2024-09-13T13:15:48.162870+0000","mapping_epoch":334,"log_start":"348'4300","ondisk_log_start":"348'4300","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:12.327645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450953216,"num_objects":6881,"num_object_clones":0,"num_object_copies":20643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8861,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":731,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.124992+0000","last_change":"2024-09-13T13:12:53.016127+0000","last_active":"2024-09-13T13:15:42.124992+0000","last_peered":"2024-09-13T13:15:42.124992+0000","last_clean":"2024-09-13T13:15:42.124992+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:15:42.124992+0000","last_undegraded":"2024-09-13T13:15:42.124992+0000","last_fullsized":"2024-09-13T13:15:42.124992+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.376603+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":690,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.117109+0000","last_change":"2024-09-13T13:14:27.622657+0000","last_active":"2024-09-13T13:15:42.117109+0000","last_peered":"2024-09-13T13:15:42.117109+0000","last_clean":"2024-09-13T13:15:42.117109+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:15:42.117109+0000","last_undegraded":"2024-09-13T13:15:42.117109+0000","last_fullsized":"2024-09-13T13:15:42.117109+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:27.622584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:27.622584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:51.144222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":849,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:44.559376+0000","last_change":"2024-09-13T13:14:26.606873+0000","last_active":"2024-09-13T13:15:44.559376+0000","last_peered":"2024-09-13T13:15:44.559376+0000","last_clean":"2024-09-13T13:15:44.559376+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:15:44.559376+0000","last_undegraded":"2024-09-13T13:15:44.559376+0000","last_fullsized":"2024-09-13T13:15:44.559376+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:14:26.606809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:26.606809+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":685,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.113575+0000","last_change":"2024-09-13T13:15:26.667525+0000","last_active":"2024-09-13T13:15:42.113575+0000","last_peered":"2024-09-13T13:15:42.113575+0000","last_clean":"2024-09-13T13:15:42.113575+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:15:42.113575+0000","last_undegraded":"2024-09-13T13:15:42.113575+0000","last_fullsized":"2024-09-13T13:15:42.113575+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:26.667455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:26.667455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.516125+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"370'9238","reported_seq":9403,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.124137+0000","last_change":"2024-09-13T13:15:25.514356+0000","last_active":"2024-09-13T13:15:51.124137+0000","last_peered":"2024-09-13T13:15:51.124137+0000","last_clean":"2024-09-13T13:15:51.124137+0000","last_became_active":"2024-09-13T13:13:58.965950+0000","last_became_peered":"2024-09-13T13:13:58.965950+0000","last_unstale":"2024-09-13T13:15:51.124137+0000","last_undegraded":"2024-09-13T13:15:51.124137+0000","last_fullsized":"2024-09-13T13:15:51.124137+0000","mapping_epoch":334,"log_start":"345'3200","ondisk_log_start":"345'3200","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'7503","last_scrub_stamp":"2024-09-13T13:15:25.514118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.514118+0000","objects_scrubbed":7440,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.146280+0000","scrub_duration":2270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449773592,"num_objects":6864,"num_object_clones":0,"num_object_copies":20592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6864,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9238,"num_write_kb":515201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":728,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.113749+0000","last_change":"2024-09-13T13:13:45.757306+0000","last_active":"2024-09-13T13:15:42.113749+0000","last_peered":"2024-09-13T13:15:42.113749+0000","last_clean":"2024-09-13T13:15:42.113749+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:15:42.113749+0000","last_undegraded":"2024-09-13T13:15:42.113749+0000","last_fullsized":"2024-09-13T13:15:42.113749+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:13:45.751801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:45.751801+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.559426+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":659,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.123080+0000","last_change":"2024-09-13T13:14:24.955591+0000","last_active":"2024-09-13T13:15:42.123080+0000","last_peered":"2024-09-13T13:15:42.123080+0000","last_clean":"2024-09-13T13:15:42.123080+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:15:42.123080+0000","last_undegraded":"2024-09-13T13:15:42.123080+0000","last_fullsized":"2024-09-13T13:15:42.123080+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:24.955516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:24.955516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:43.325029+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":638,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.716536+0000","last_change":"2024-09-13T13:14:50.408656+0000","last_active":"2024-09-13T13:15:50.716536+0000","last_peered":"2024-09-13T13:15:50.716536+0000","last_clean":"2024-09-13T13:15:50.716536+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:15:50.716536+0000","last_undegraded":"2024-09-13T13:15:50.716536+0000","last_fullsized":"2024-09-13T13:15:50.716536+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.408378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.408378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.441158+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":666,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.122105+0000","last_change":"2024-09-13T13:13:50.772738+0000","last_active":"2024-09-13T13:15:42.122105+0000","last_peered":"2024-09-13T13:15:42.122105+0000","last_clean":"2024-09-13T13:15:42.122105+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:15:42.122105+0000","last_undegraded":"2024-09-13T13:15:42.122105+0000","last_fullsized":"2024-09-13T13:15:42.122105+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:50.772689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.772689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.211300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"370'9140","reported_seq":9223,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.147457+0000","last_change":"2024-09-13T13:13:58.966631+0000","last_active":"2024-09-13T13:15:51.147457+0000","last_peered":"2024-09-13T13:15:51.147457+0000","last_clean":"2024-09-13T13:15:51.147457+0000","last_became_active":"2024-09-13T13:13:58.965789+0000","last_became_peered":"2024-09-13T13:13:58.965789+0000","last_unstale":"2024-09-13T13:15:51.147457+0000","last_undegraded":"2024-09-13T13:15:51.147457+0000","last_fullsized":"2024-09-13T13:15:51.147457+0000","mapping_epoch":334,"log_start":"345'3100","ondisk_log_start":"345'3100","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.272671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443285504,"num_objects":6764,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9140,"num_write_kb":508928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"268'40","reported_seq":915,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:44.042336+0000","last_change":"2024-09-13T13:15:04.288539+0000","last_active":"2024-09-13T13:15:44.042336+0000","last_peered":"2024-09-13T13:15:44.042336+0000","last_clean":"2024-09-13T13:15:44.042336+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:15:44.042336+0000","last_undegraded":"2024-09-13T13:15:44.042336+0000","last_fullsized":"2024-09-13T13:15:44.042336+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'40","last_scrub_stamp":"2024-09-13T13:15:04.288431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.288431+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.738711+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":153,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.109309+0000","last_change":"2024-09-13T13:14:14.899463+0000","last_active":"2024-09-13T13:15:42.109309+0000","last_peered":"2024-09-13T13:15:42.109309+0000","last_clean":"2024-09-13T13:15:42.109309+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:15:42.109309+0000","last_undegraded":"2024-09-13T13:15:42.109309+0000","last_fullsized":"2024-09-13T13:15:42.109309+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:14.899378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:14.899378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":677,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.117749+0000","last_change":"2024-09-13T13:13:20.438191+0000","last_active":"2024-09-13T13:15:42.117749+0000","last_peered":"2024-09-13T13:15:42.117749+0000","last_clean":"2024-09-13T13:15:42.117749+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:15:42.117749+0000","last_undegraded":"2024-09-13T13:15:42.117749+0000","last_fullsized":"2024-09-13T13:15:42.117749+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:13:20.438142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:20.438142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":687,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.121320+0000","last_change":"2024-09-13T13:14:59.670853+0000","last_active":"2024-09-13T13:15:42.121320+0000","last_peered":"2024-09-13T13:15:42.121320+0000","last_clean":"2024-09-13T13:15:42.121320+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:15:42.121320+0000","last_undegraded":"2024-09-13T13:15:42.121320+0000","last_fullsized":"2024-09-13T13:15:42.121320+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.670779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.670779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.298565+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":675,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.108482+0000","last_change":"2024-09-13T13:15:33.280401+0000","last_active":"2024-09-13T13:15:42.108482+0000","last_peered":"2024-09-13T13:15:42.108482+0000","last_clean":"2024-09-13T13:15:42.108482+0000","last_became_active":"2024-09-13T13:12:52.948742+0000","last_became_peered":"2024-09-13T13:12:52.948742+0000","last_unstale":"2024-09-13T13:15:42.108482+0000","last_undegraded":"2024-09-13T13:15:42.108482+0000","last_fullsized":"2024-09-13T13:15:42.108482+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:33.280352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.280352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:51.771913+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"368'8848","reported_seq":8927,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:48.402173+0000","last_change":"2024-09-13T13:13:59.149677+0000","last_active":"2024-09-13T13:15:48.402173+0000","last_peered":"2024-09-13T13:15:48.402173+0000","last_clean":"2024-09-13T13:15:48.402173+0000","last_became_active":"2024-09-13T13:13:59.149368+0000","last_became_peered":"2024-09-13T13:13:59.149368+0000","last_unstale":"2024-09-13T13:15:48.402173+0000","last_undegraded":"2024-09-13T13:15:48.402173+0000","last_fullsized":"2024-09-13T13:15:48.402173+0000","mapping_epoch":334,"log_start":"345'3200","ondisk_log_start":"345'3200","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5648,"log_dups_size":0,"ondisk_log_size":5648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:10.338929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451280896,"num_objects":6886,"num_object_clones":0,"num_object_copies":20658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8848,"num_write_kb":503488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":695,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.124177+0000","last_change":"2024-09-13T13:15:33.299837+0000","last_active":"2024-09-13T13:15:42.124177+0000","last_peered":"2024-09-13T13:15:42.124177+0000","last_clean":"2024-09-13T13:15:42.124177+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:15:42.124177+0000","last_undegraded":"2024-09-13T13:15:42.124177+0000","last_fullsized":"2024-09-13T13:15:42.124177+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'17","last_scrub_stamp":"2024-09-13T13:15:33.299746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.299746+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:37.843249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":686,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.122602+0000","last_change":"2024-09-13T13:13:17.676405+0000","last_active":"2024-09-13T13:15:42.122602+0000","last_peered":"2024-09-13T13:15:42.122602+0000","last_clean":"2024-09-13T13:15:42.122602+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:15:42.122602+0000","last_undegraded":"2024-09-13T13:15:42.122602+0000","last_fullsized":"2024-09-13T13:15:42.122602+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:17.676339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:17.676339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.177334+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":690,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.106885+0000","last_change":"2024-09-13T13:13:14.560620+0000","last_active":"2024-09-13T13:15:42.106885+0000","last_peered":"2024-09-13T13:15:42.106885+0000","last_clean":"2024-09-13T13:15:42.106885+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:15:42.106885+0000","last_undegraded":"2024-09-13T13:15:42.106885+0000","last_fullsized":"2024-09-13T13:15:42.106885+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:13:14.560555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:14.560555+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.776407+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":661,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.107362+0000","last_change":"2024-09-13T13:14:23.728707+0000","last_active":"2024-09-13T13:15:42.107362+0000","last_peered":"2024-09-13T13:15:42.107362+0000","last_clean":"2024-09-13T13:15:42.107362+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:15:42.107362+0000","last_undegraded":"2024-09-13T13:15:42.107362+0000","last_fullsized":"2024-09-13T13:15:42.107362+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:23.728632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:23.728632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:52.457538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":291,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717447+0000","last_change":"2024-09-13T13:12:41.332308+0000","last_active":"2024-09-13T13:15:50.717447+0000","last_peered":"2024-09-13T13:15:50.717447+0000","last_clean":"2024-09-13T13:15:50.717447+0000","last_became_active":"2024-09-13T13:12:40.120242+0000","last_became_peered":"2024-09-13T13:12:40.120242+0000","last_unstale":"2024-09-13T13:15:50.717447+0000","last_undegraded":"2024-09-13T13:15:50.717447+0000","last_fullsized":"2024-09-13T13:15:50.717447+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:41.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:41.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.372949+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"368'9117","reported_seq":9196,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:49.460020+0000","last_change":"2024-09-13T13:13:59.149922+0000","last_active":"2024-09-13T13:15:49.460020+0000","last_peered":"2024-09-13T13:15:49.460020+0000","last_clean":"2024-09-13T13:15:49.460020+0000","last_became_active":"2024-09-13T13:13:59.149037+0000","last_became_peered":"2024-09-13T13:13:59.149037+0000","last_unstale":"2024-09-13T13:15:49.460020+0000","last_undegraded":"2024-09-13T13:15:49.460020+0000","last_fullsized":"2024-09-13T13:15:49.460020+0000","mapping_epoch":334,"log_start":"346'3700","ondisk_log_start":"346'3700","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.573331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457244672,"num_objects":6977,"num_object_clones":0,"num_object_copies":20931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9117,"num_write_kb":515008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":696,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.114359+0000","last_change":"2024-09-13T13:13:10.964903+0000","last_active":"2024-09-13T13:15:42.114359+0000","last_peered":"2024-09-13T13:15:42.114359+0000","last_clean":"2024-09-13T13:15:42.114359+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:15:42.114359+0000","last_undegraded":"2024-09-13T13:15:42.114359+0000","last_fullsized":"2024-09-13T13:15:42.114359+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:13:10.964809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.964809+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:43.492892+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":687,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.113802+0000","last_change":"2024-09-13T13:15:30.590673+0000","last_active":"2024-09-13T13:15:42.113802+0000","last_peered":"2024-09-13T13:15:42.113802+0000","last_clean":"2024-09-13T13:15:42.113802+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:15:42.113802+0000","last_undegraded":"2024-09-13T13:15:42.113802+0000","last_fullsized":"2024-09-13T13:15:42.113802+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:30.590466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:30.590466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:49.834025+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":398,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:47.400970+0000","last_change":"2024-09-13T13:14:48.128237+0000","last_active":"2024-09-13T13:15:47.400970+0000","last_peered":"2024-09-13T13:15:47.400970+0000","last_clean":"2024-09-13T13:15:47.400970+0000","last_became_active":"2024-09-13T13:12:39.947940+0000","last_became_peered":"2024-09-13T13:12:39.947940+0000","last_unstale":"2024-09-13T13:15:47.400970+0000","last_undegraded":"2024-09-13T13:15:47.400970+0000","last_fullsized":"2024-09-13T13:15:47.400970+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:14:48.128156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:48.128156+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.088862+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":195,"num_read_kb":190,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":657,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.123454+0000","last_change":"2024-09-13T13:15:32.318497+0000","last_active":"2024-09-13T13:15:42.123454+0000","last_peered":"2024-09-13T13:15:42.123454+0000","last_clean":"2024-09-13T13:15:42.123454+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:15:42.123454+0000","last_undegraded":"2024-09-13T13:15:42.123454+0000","last_fullsized":"2024-09-13T13:15:42.123454+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:32.318413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.318413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:32.934946+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":663,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.122702+0000","last_change":"2024-09-13T13:10:32.720742+0000","last_active":"2024-09-13T13:15:42.122702+0000","last_peered":"2024-09-13T13:15:42.122702+0000","last_clean":"2024-09-13T13:15:42.122702+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:15:42.122702+0000","last_undegraded":"2024-09-13T13:15:42.122702+0000","last_fullsized":"2024-09-13T13:15:42.122702+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.305143+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"370'9094","reported_seq":9177,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.192529+0000","last_change":"2024-09-13T13:13:58.967163+0000","last_active":"2024-09-13T13:15:51.192529+0000","last_peered":"2024-09-13T13:15:51.192529+0000","last_clean":"2024-09-13T13:15:51.192529+0000","last_became_active":"2024-09-13T13:13:58.965962+0000","last_became_peered":"2024-09-13T13:13:58.965962+0000","last_unstale":"2024-09-13T13:15:51.192529+0000","last_undegraded":"2024-09-13T13:15:51.192529+0000","last_fullsized":"2024-09-13T13:15:51.192529+0000","mapping_epoch":334,"log_start":"345'3000","ondisk_log_start":"345'3000","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":6094,"log_dups_size":0,"ondisk_log_size":6094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:12.328620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448135168,"num_objects":6838,"num_object_clones":0,"num_object_copies":20514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9094,"num_write_kb":509824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"306'32","reported_seq":793,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717105+0000","last_change":"2024-09-13T13:13:01.352416+0000","last_active":"2024-09-13T13:15:50.717105+0000","last_peered":"2024-09-13T13:15:50.717105+0000","last_clean":"2024-09-13T13:15:50.717105+0000","last_became_active":"2024-09-13T13:12:52.999924+0000","last_became_peered":"2024-09-13T13:12:52.999924+0000","last_unstale":"2024-09-13T13:15:50.717105+0000","last_undegraded":"2024-09-13T13:15:50.717105+0000","last_fullsized":"2024-09-13T13:15:50.717105+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:13:01.352337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.352337+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.487539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":677,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.112082+0000","last_change":"2024-09-13T13:15:04.182566+0000","last_active":"2024-09-13T13:15:42.112082+0000","last_peered":"2024-09-13T13:15:42.112082+0000","last_clean":"2024-09-13T13:15:42.112082+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:15:42.112082+0000","last_undegraded":"2024-09-13T13:15:42.112082+0000","last_fullsized":"2024-09-13T13:15:42.112082+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:04.182499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.182499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.040803+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":243,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.113795+0000","last_change":"2024-09-13T13:14:24.630282+0000","last_active":"2024-09-13T13:15:42.113795+0000","last_peered":"2024-09-13T13:15:42.113795+0000","last_clean":"2024-09-13T13:15:42.113795+0000","last_became_active":"2024-09-13T13:12:39.948765+0000","last_became_peered":"2024-09-13T13:12:39.948765+0000","last_unstale":"2024-09-13T13:15:42.113795+0000","last_undegraded":"2024-09-13T13:15:42.113795+0000","last_fullsized":"2024-09-13T13:15:42.113795+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:14:24.630199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:14:24.630199+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:44.701869+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":176,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.110093+0000","last_change":"2024-09-13T13:15:33.709071+0000","last_active":"2024-09-13T13:15:42.110093+0000","last_peered":"2024-09-13T13:15:42.110093+0000","last_clean":"2024-09-13T13:15:42.110093+0000","last_became_active":"2024-09-13T13:12:39.950496+0000","last_became_peered":"2024-09-13T13:12:39.950496+0000","last_unstale":"2024-09-13T13:15:42.110093+0000","last_undegraded":"2024-09-13T13:15:42.110093+0000","last_fullsized":"2024-09-13T13:15:42.110093+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:33.708995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.708995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:57.976953+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":691,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.123298+0000","last_change":"2024-09-13T13:14:23.625418+0000","last_active":"2024-09-13T13:15:42.123298+0000","last_peered":"2024-09-13T13:15:42.123298+0000","last_clean":"2024-09-13T13:15:42.123298+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:15:42.123298+0000","last_undegraded":"2024-09-13T13:15:42.123298+0000","last_fullsized":"2024-09-13T13:15:42.123298+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:23.625281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:23.625281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:48.600459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"370'9294","reported_seq":9445,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.140396+0000","last_change":"2024-09-13T13:15:04.191246+0000","last_active":"2024-09-13T13:15:51.140396+0000","last_peered":"2024-09-13T13:15:51.140396+0000","last_clean":"2024-09-13T13:15:51.140396+0000","last_became_active":"2024-09-13T13:13:58.965566+0000","last_became_peered":"2024-09-13T13:13:58.965566+0000","last_unstale":"2024-09-13T13:15:51.140396+0000","last_undegraded":"2024-09-13T13:15:51.140396+0000","last_fullsized":"2024-09-13T13:15:51.140396+0000","mapping_epoch":334,"log_start":"345'3200","ondisk_log_start":"345'3200","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'6049","last_scrub_stamp":"2024-09-13T13:15:04.191100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.191100+0000","objects_scrubbed":5987,"log_size":6094,"log_dups_size":0,"ondisk_log_size":6094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:31.390342+0000","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451805184,"num_objects":6894,"num_object_clones":0,"num_object_copies":20682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9294,"num_write_kb":518016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":165,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.115075+0000","last_change":"2024-09-13T13:14:59.354034+0000","last_active":"2024-09-13T13:15:42.115075+0000","last_peered":"2024-09-13T13:15:42.115075+0000","last_clean":"2024-09-13T13:15:42.115075+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:15:42.115075+0000","last_undegraded":"2024-09-13T13:15:42.115075+0000","last_fullsized":"2024-09-13T13:15:42.115075+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:14:59.353691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.353691+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:10.915988+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":145,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.108896+0000","last_change":"2024-09-13T13:14:26.343833+0000","last_active":"2024-09-13T13:15:42.108896+0000","last_peered":"2024-09-13T13:15:42.108896+0000","last_clean":"2024-09-13T13:15:42.108896+0000","last_became_active":"2024-09-13T13:12:53.006271+0000","last_became_peered":"2024-09-13T13:12:53.006271+0000","last_unstale":"2024-09-13T13:15:42.108896+0000","last_undegraded":"2024-09-13T13:15:42.108896+0000","last_fullsized":"2024-09-13T13:15:42.108896+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:26.343754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:26.343754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":706,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.122955+0000","last_change":"2024-09-13T13:14:14.630794+0000","last_active":"2024-09-13T13:15:42.122955+0000","last_peered":"2024-09-13T13:15:42.122955+0000","last_clean":"2024-09-13T13:15:42.122955+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:15:42.122955+0000","last_undegraded":"2024-09-13T13:15:42.122955+0000","last_fullsized":"2024-09-13T13:15:42.122955+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:14.630719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:14.630719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":664,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.109502+0000","last_change":"2024-09-13T13:14:23.655250+0000","last_active":"2024-09-13T13:15:42.109502+0000","last_peered":"2024-09-13T13:15:42.109502+0000","last_clean":"2024-09-13T13:15:42.109502+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:15:42.109502+0000","last_undegraded":"2024-09-13T13:15:42.109502+0000","last_fullsized":"2024-09-13T13:15:42.109502+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:23.655072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:23.655072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"368'8936","reported_seq":9100,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:48.488867+0000","last_change":"2024-09-13T13:15:31.408104+0000","last_active":"2024-09-13T13:15:48.488867+0000","last_peered":"2024-09-13T13:15:48.488867+0000","last_clean":"2024-09-13T13:15:48.488867+0000","last_became_active":"2024-09-13T13:13:58.961825+0000","last_became_peered":"2024-09-13T13:13:58.961825+0000","last_unstale":"2024-09-13T13:15:48.488867+0000","last_undegraded":"2024-09-13T13:15:48.488867+0000","last_fullsized":"2024-09-13T13:15:48.488867+0000","mapping_epoch":334,"log_start":"342'2200","ondisk_log_start":"342'2200","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7793","last_scrub_stamp":"2024-09-13T13:15:31.408001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:31.408001+0000","objects_scrubbed":7714,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:37.609213+0000","scrub_duration":2771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458096640,"num_objects":6990,"num_object_clones":0,"num_object_copies":20970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8936,"num_write_kb":509632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":171,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.109593+0000","last_change":"2024-09-13T13:14:20.221364+0000","last_active":"2024-09-13T13:15:42.109593+0000","last_peered":"2024-09-13T13:15:42.109593+0000","last_clean":"2024-09-13T13:15:42.109593+0000","last_became_active":"2024-09-13T13:12:53.001687+0000","last_became_peered":"2024-09-13T13:12:53.001687+0000","last_unstale":"2024-09-13T13:15:42.109593+0000","last_undegraded":"2024-09-13T13:15:42.109593+0000","last_fullsized":"2024-09-13T13:15:42.109593+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:14:20.221286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:20.221286+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:42.472243+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":692,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.716760+0000","last_change":"2024-09-13T13:15:24.265548+0000","last_active":"2024-09-13T13:15:50.716760+0000","last_peered":"2024-09-13T13:15:50.716760+0000","last_clean":"2024-09-13T13:15:50.716760+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:15:50.716760+0000","last_undegraded":"2024-09-13T13:15:50.716760+0000","last_fullsized":"2024-09-13T13:15:50.716760+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:24.265477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:24.265477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:35.804220+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":680,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.122750+0000","last_change":"2024-09-13T13:15:24.264425+0000","last_active":"2024-09-13T13:15:42.122750+0000","last_peered":"2024-09-13T13:15:42.122750+0000","last_clean":"2024-09-13T13:15:42.122750+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:15:42.122750+0000","last_undegraded":"2024-09-13T13:15:42.122750+0000","last_fullsized":"2024-09-13T13:15:42.122750+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:24.264316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:24.264316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:32.724876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":668,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.107818+0000","last_change":"2024-09-13T13:13:54.649219+0000","last_active":"2024-09-13T13:15:42.107818+0000","last_peered":"2024-09-13T13:15:42.107818+0000","last_clean":"2024-09-13T13:15:42.107818+0000","last_became_active":"2024-09-13T13:12:52.938975+0000","last_became_peered":"2024-09-13T13:12:52.938975+0000","last_unstale":"2024-09-13T13:15:42.107818+0000","last_undegraded":"2024-09-13T13:15:42.107818+0000","last_fullsized":"2024-09-13T13:15:42.107818+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.649166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.649166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.656296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"368'9023","reported_seq":9101,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:49.011656+0000","last_change":"2024-09-13T13:13:58.963327+0000","last_active":"2024-09-13T13:15:49.011656+0000","last_peered":"2024-09-13T13:15:49.011656+0000","last_clean":"2024-09-13T13:15:49.011656+0000","last_became_active":"2024-09-13T13:13:58.963064+0000","last_became_peered":"2024-09-13T13:13:58.963064+0000","last_unstale":"2024-09-13T13:15:49.011656+0000","last_undegraded":"2024-09-13T13:15:49.011656+0000","last_fullsized":"2024-09-13T13:15:49.011656+0000","mapping_epoch":334,"log_start":"346'3700","ondisk_log_start":"346'3700","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5323,"log_dups_size":0,"ondisk_log_size":5323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.315173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459735040,"num_objects":7015,"num_object_clones":0,"num_object_copies":21045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9023,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":149,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.114760+0000","last_change":"2024-09-13T13:12:53.018645+0000","last_active":"2024-09-13T13:15:42.114760+0000","last_peered":"2024-09-13T13:15:42.114760+0000","last_clean":"2024-09-13T13:15:42.114760+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:15:42.114760+0000","last_undegraded":"2024-09-13T13:15:42.114760+0000","last_fullsized":"2024-09-13T13:15:42.114760+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:09.739184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.739184+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.221949+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":671,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.113103+0000","last_change":"2024-09-13T13:12:53.003530+0000","last_active":"2024-09-13T13:15:42.113103+0000","last_peered":"2024-09-13T13:15:42.113103+0000","last_clean":"2024-09-13T13:15:42.113103+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:15:42.113103+0000","last_undegraded":"2024-09-13T13:15:42.113103+0000","last_fullsized":"2024-09-13T13:15:42.113103+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.996346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.996346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:10.180309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":836,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.418061+0000","last_change":"2024-09-13T13:12:36.113757+0000","last_active":"2024-09-13T13:15:42.418061+0000","last_peered":"2024-09-13T13:15:42.418061+0000","last_clean":"2024-09-13T13:15:42.418061+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:15:42.418061+0000","last_undegraded":"2024-09-13T13:15:42.418061+0000","last_fullsized":"2024-09-13T13:15:42.418061+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:12:36.113709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:36.113709+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.715547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":145,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.114942+0000","last_change":"2024-09-13T13:13:41.797728+0000","last_active":"2024-09-13T13:15:42.114942+0000","last_peered":"2024-09-13T13:15:42.114942+0000","last_clean":"2024-09-13T13:15:42.114942+0000","last_became_active":"2024-09-13T13:12:52.997283+0000","last_became_peered":"2024-09-13T13:12:52.997283+0000","last_unstale":"2024-09-13T13:15:42.114942+0000","last_undegraded":"2024-09-13T13:15:42.114942+0000","last_fullsized":"2024-09-13T13:15:42.114942+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.797639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.797639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:06.323760+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"368'8942","reported_seq":9104,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:48.500922+0000","last_change":"2024-09-13T13:15:32.530404+0000","last_active":"2024-09-13T13:15:48.500922+0000","last_peered":"2024-09-13T13:15:48.500922+0000","last_clean":"2024-09-13T13:15:48.500922+0000","last_became_active":"2024-09-13T13:13:58.962803+0000","last_became_peered":"2024-09-13T13:13:58.962803+0000","last_unstale":"2024-09-13T13:15:48.500922+0000","last_undegraded":"2024-09-13T13:15:48.500922+0000","last_fullsized":"2024-09-13T13:15:48.500922+0000","mapping_epoch":334,"log_start":"342'2200","ondisk_log_start":"342'2200","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7822","last_scrub_stamp":"2024-09-13T13:15:32.530220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.530220+0000","objects_scrubbed":7744,"log_size":6742,"log_dups_size":0,"ondisk_log_size":6742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:48.786652+0000","scrub_duration":2862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453378048,"num_objects":6918,"num_object_clones":0,"num_object_copies":20754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8942,"num_write_kb":507520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":703,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.113608+0000","last_change":"2024-09-13T13:13:56.850149+0000","last_active":"2024-09-13T13:15:42.113608+0000","last_peered":"2024-09-13T13:15:42.113608+0000","last_clean":"2024-09-13T13:15:42.113608+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:15:42.113608+0000","last_undegraded":"2024-09-13T13:15:42.113608+0000","last_fullsized":"2024-09-13T13:15:42.113608+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:56.850094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.850094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.708354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":692,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.116438+0000","last_change":"2024-09-13T13:12:58.857634+0000","last_active":"2024-09-13T13:15:42.116438+0000","last_peered":"2024-09-13T13:15:42.116438+0000","last_clean":"2024-09-13T13:15:42.116438+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:15:42.116438+0000","last_undegraded":"2024-09-13T13:15:42.116438+0000","last_fullsized":"2024-09-13T13:15:42.116438+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:58.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":704,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.108214+0000","last_change":"2024-09-13T13:15:23.682529+0000","last_active":"2024-09-13T13:15:42.108214+0000","last_peered":"2024-09-13T13:15:42.108214+0000","last_clean":"2024-09-13T13:15:42.108214+0000","last_became_active":"2024-09-13T13:12:52.944725+0000","last_became_peered":"2024-09-13T13:12:52.944725+0000","last_unstale":"2024-09-13T13:15:42.108214+0000","last_undegraded":"2024-09-13T13:15:42.108214+0000","last_fullsized":"2024-09-13T13:15:42.108214+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:23.682463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.682463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.810934+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":652,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.113737+0000","last_change":"2024-09-13T13:13:41.762003+0000","last_active":"2024-09-13T13:15:42.113737+0000","last_peered":"2024-09-13T13:15:42.113737+0000","last_clean":"2024-09-13T13:15:42.113737+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:15:42.113737+0000","last_undegraded":"2024-09-13T13:15:42.113737+0000","last_fullsized":"2024-09-13T13:15:42.113737+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.761814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.761814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.545140+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"368'8861","reported_seq":8940,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:49.531116+0000","last_change":"2024-09-13T13:13:58.963548+0000","last_active":"2024-09-13T13:15:49.531116+0000","last_peered":"2024-09-13T13:15:49.531116+0000","last_clean":"2024-09-13T13:15:49.531116+0000","last_became_active":"2024-09-13T13:13:58.963160+0000","last_became_peered":"2024-09-13T13:13:58.963160+0000","last_unstale":"2024-09-13T13:15:49.531116+0000","last_undegraded":"2024-09-13T13:15:49.531116+0000","last_fullsized":"2024-09-13T13:15:49.531116+0000","mapping_epoch":334,"log_start":"346'3500","ondisk_log_start":"346'3500","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5361,"log_dups_size":0,"ondisk_log_size":5361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:11.607417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448069632,"num_objects":6837,"num_object_clones":0,"num_object_copies":20511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8861,"num_write_kb":502336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":681,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.716821+0000","last_change":"2024-09-13T13:13:06.339762+0000","last_active":"2024-09-13T13:15:50.716821+0000","last_peered":"2024-09-13T13:15:50.716821+0000","last_clean":"2024-09-13T13:15:50.716821+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:15:50.716821+0000","last_undegraded":"2024-09-13T13:15:50.716821+0000","last_fullsized":"2024-09-13T13:15:50.716821+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:13:06.339664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.339664+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:11.302026+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":307,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.116816+0000","last_change":"2024-09-13T13:14:55.471954+0000","last_active":"2024-09-13T13:15:42.116816+0000","last_peered":"2024-09-13T13:15:42.116816+0000","last_clean":"2024-09-13T13:15:42.116816+0000","last_became_active":"2024-09-13T13:12:39.954315+0000","last_became_peered":"2024-09-13T13:12:39.954315+0000","last_unstale":"2024-09-13T13:15:42.116816+0000","last_undegraded":"2024-09-13T13:15:42.116816+0000","last_fullsized":"2024-09-13T13:15:42.116816+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:55.471885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:55.471885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:01.346273+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":666,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.105795+0000","last_change":"2024-09-13T13:13:11.484689+0000","last_active":"2024-09-13T13:15:42.105795+0000","last_peered":"2024-09-13T13:15:42.105795+0000","last_clean":"2024-09-13T13:15:42.105795+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:15:42.105795+0000","last_undegraded":"2024-09-13T13:15:42.105795+0000","last_fullsized":"2024-09-13T13:15:42.105795+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.484615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.484615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:43.679169+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":160,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.106892+0000","last_change":"2024-09-13T13:15:02.627642+0000","last_active":"2024-09-13T13:15:42.106892+0000","last_peered":"2024-09-13T13:15:42.106892+0000","last_clean":"2024-09-13T13:15:42.106892+0000","last_became_active":"2024-09-13T13:12:40.118050+0000","last_became_peered":"2024-09-13T13:12:40.118050+0000","last_unstale":"2024-09-13T13:15:42.106892+0000","last_undegraded":"2024-09-13T13:15:42.106892+0000","last_fullsized":"2024-09-13T13:15:42.106892+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:02.627569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:02.627569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:32.377161+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"370'8971","reported_seq":9054,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.142020+0000","last_change":"2024-09-13T13:13:58.966955+0000","last_active":"2024-09-13T13:15:51.142020+0000","last_peered":"2024-09-13T13:15:51.142020+0000","last_clean":"2024-09-13T13:15:51.142020+0000","last_became_active":"2024-09-13T13:13:58.966675+0000","last_became_peered":"2024-09-13T13:13:58.966675+0000","last_unstale":"2024-09-13T13:15:51.142020+0000","last_undegraded":"2024-09-13T13:15:51.142020+0000","last_fullsized":"2024-09-13T13:15:51.142020+0000","mapping_epoch":334,"log_start":"344'2900","ondisk_log_start":"344'2900","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":6071,"log_dups_size":0,"ondisk_log_size":6071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.259461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433782784,"num_objects":6619,"num_object_clones":0,"num_object_copies":19857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8971,"num_write_kb":498880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":157,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.109267+0000","last_change":"2024-09-13T13:13:20.064693+0000","last_active":"2024-09-13T13:15:42.109267+0000","last_peered":"2024-09-13T13:15:42.109267+0000","last_clean":"2024-09-13T13:15:42.109267+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:15:42.109267+0000","last_undegraded":"2024-09-13T13:15:42.109267+0000","last_fullsized":"2024-09-13T13:15:42.109267+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:13:20.064614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:20.064614+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:44.303533+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":153,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.114846+0000","last_change":"2024-09-13T13:14:52.383027+0000","last_active":"2024-09-13T13:15:42.114846+0000","last_peered":"2024-09-13T13:15:42.114846+0000","last_clean":"2024-09-13T13:15:42.114846+0000","last_became_active":"2024-09-13T13:12:53.007561+0000","last_became_peered":"2024-09-13T13:12:53.007561+0000","last_unstale":"2024-09-13T13:15:42.114846+0000","last_undegraded":"2024-09-13T13:15:42.114846+0000","last_fullsized":"2024-09-13T13:15:42.114846+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.382755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.382755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:20.097418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":658,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.107035+0000","last_change":"2024-09-13T13:15:00.637152+0000","last_active":"2024-09-13T13:15:42.107035+0000","last_peered":"2024-09-13T13:15:42.107035+0000","last_clean":"2024-09-13T13:15:42.107035+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:15:42.107035+0000","last_undegraded":"2024-09-13T13:15:42.107035+0000","last_fullsized":"2024-09-13T13:15:42.107035+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.636898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.636898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.204507+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":645,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.112129+0000","last_change":"2024-09-13T13:15:03.223521+0000","last_active":"2024-09-13T13:15:42.112129+0000","last_peered":"2024-09-13T13:15:42.112129+0000","last_clean":"2024-09-13T13:15:42.112129+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:15:42.112129+0000","last_undegraded":"2024-09-13T13:15:42.112129+0000","last_fullsized":"2024-09-13T13:15:42.112129+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.223265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.223265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.540511+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"368'9035","reported_seq":9114,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:48.391578+0000","last_change":"2024-09-13T13:13:58.963039+0000","last_active":"2024-09-13T13:15:48.391578+0000","last_peered":"2024-09-13T13:15:48.391578+0000","last_clean":"2024-09-13T13:15:48.391578+0000","last_became_active":"2024-09-13T13:13:58.962773+0000","last_became_peered":"2024-09-13T13:13:58.962773+0000","last_unstale":"2024-09-13T13:15:48.391578+0000","last_undegraded":"2024-09-13T13:15:48.391578+0000","last_fullsized":"2024-09-13T13:15:48.391578+0000","mapping_epoch":334,"log_start":"346'3400","ondisk_log_start":"346'3400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459997184,"num_objects":7019,"num_object_clones":0,"num_object_copies":21057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9035,"num_write_kb":513728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":731,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.115858+0000","last_change":"2024-09-13T13:14:20.570758+0000","last_active":"2024-09-13T13:15:42.115858+0000","last_peered":"2024-09-13T13:15:42.115858+0000","last_clean":"2024-09-13T13:15:42.115858+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:15:42.115858+0000","last_undegraded":"2024-09-13T13:15:42.115858+0000","last_fullsized":"2024-09-13T13:15:42.115858+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:14:20.570698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:20.570698+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:44.140400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":151,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.120981+0000","last_change":"2024-09-13T13:13:01.972907+0000","last_active":"2024-09-13T13:15:42.120981+0000","last_peered":"2024-09-13T13:15:42.120981+0000","last_clean":"2024-09-13T13:15:42.120981+0000","last_became_active":"2024-09-13T13:12:53.006362+0000","last_became_peered":"2024-09-13T13:12:53.006362+0000","last_unstale":"2024-09-13T13:15:42.120981+0000","last_undegraded":"2024-09-13T13:15:42.120981+0000","last_fullsized":"2024-09-13T13:15:42.120981+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:13:01.972831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.972831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"306'13","reported_seq":395,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717428+0000","last_change":"2024-09-13T13:13:03.334319+0000","last_active":"2024-09-13T13:15:50.717428+0000","last_peered":"2024-09-13T13:15:50.717428+0000","last_clean":"2024-09-13T13:15:50.717428+0000","last_became_active":"2024-09-13T13:12:53.001189+0000","last_became_peered":"2024-09-13T13:12:53.001189+0000","last_unstale":"2024-09-13T13:15:50.717428+0000","last_undegraded":"2024-09-13T13:15:50.717428+0000","last_fullsized":"2024-09-13T13:15:50.717428+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'13","last_scrub_stamp":"2024-09-13T13:13:03.334012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.334012+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.265982+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":664,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.117494+0000","last_change":"2024-09-13T13:13:55.500961+0000","last_active":"2024-09-13T13:15:42.117494+0000","last_peered":"2024-09-13T13:15:42.117494+0000","last_clean":"2024-09-13T13:15:42.117494+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:15:42.117494+0000","last_undegraded":"2024-09-13T13:15:42.117494+0000","last_fullsized":"2024-09-13T13:15:42.117494+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:55.500908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.500908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:11.347558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":646,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.716998+0000","last_change":"2024-09-13T13:13:04.371082+0000","last_active":"2024-09-13T13:15:50.716998+0000","last_peered":"2024-09-13T13:15:50.716998+0000","last_clean":"2024-09-13T13:15:50.716998+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:15:50.716998+0000","last_undegraded":"2024-09-13T13:15:50.716998+0000","last_fullsized":"2024-09-13T13:15:50.716998+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.370881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.370881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.428165+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":187,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.118502+0000","last_change":"2024-09-13T13:15:23.319486+0000","last_active":"2024-09-13T13:15:42.118502+0000","last_peered":"2024-09-13T13:15:42.118502+0000","last_clean":"2024-09-13T13:15:42.118502+0000","last_became_active":"2024-09-13T13:12:34.673528+0000","last_became_peered":"2024-09-13T13:12:34.673528+0000","last_unstale":"2024-09-13T13:15:42.118502+0000","last_undegraded":"2024-09-13T13:15:42.118502+0000","last_fullsized":"2024-09-13T13:15:42.118502+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:15:23.319412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.319412+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:45.332563+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":681,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.118155+0000","last_change":"2024-09-13T13:14:13.633412+0000","last_active":"2024-09-13T13:15:42.118155+0000","last_peered":"2024-09-13T13:15:42.118155+0000","last_clean":"2024-09-13T13:15:42.118155+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:15:42.118155+0000","last_undegraded":"2024-09-13T13:15:42.118155+0000","last_fullsized":"2024-09-13T13:15:42.118155+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:13.633215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:13.633215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":702,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.118639+0000","last_change":"2024-09-13T13:15:25.787110+0000","last_active":"2024-09-13T13:15:42.118639+0000","last_peered":"2024-09-13T13:15:42.118639+0000","last_clean":"2024-09-13T13:15:42.118639+0000","last_became_active":"2024-09-13T13:12:53.000873+0000","last_became_peered":"2024-09-13T13:12:53.000873+0000","last_unstale":"2024-09-13T13:15:42.118639+0000","last_undegraded":"2024-09-13T13:15:42.118639+0000","last_fullsized":"2024-09-13T13:15:42.118639+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.787029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.787029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:31.829669+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":677,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.114446+0000","last_change":"2024-09-13T13:13:12.952427+0000","last_active":"2024-09-13T13:15:42.114446+0000","last_peered":"2024-09-13T13:15:42.114446+0000","last_clean":"2024-09-13T13:15:42.114446+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:15:42.114446+0000","last_undegraded":"2024-09-13T13:15:42.114446+0000","last_fullsized":"2024-09-13T13:15:42.114446+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:12.952348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:12.952348+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:37.576218+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":145,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.109045+0000","last_change":"2024-09-13T13:13:51.764793+0000","last_active":"2024-09-13T13:15:42.109045+0000","last_peered":"2024-09-13T13:15:42.109045+0000","last_clean":"2024-09-13T13:15:42.109045+0000","last_became_active":"2024-09-13T13:12:53.006579+0000","last_became_peered":"2024-09-13T13:12:53.006579+0000","last_unstale":"2024-09-13T13:15:42.109045+0000","last_undegraded":"2024-09-13T13:15:42.109045+0000","last_fullsized":"2024-09-13T13:15:42.109045+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.764717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.764717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:11.315764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":665,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.111688+0000","last_change":"2024-09-13T13:15:25.281502+0000","last_active":"2024-09-13T13:15:42.111688+0000","last_peered":"2024-09-13T13:15:42.111688+0000","last_clean":"2024-09-13T13:15:42.111688+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:15:42.111688+0000","last_undegraded":"2024-09-13T13:15:42.111688+0000","last_fullsized":"2024-09-13T13:15:42.111688+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.281295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.281295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.642193+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":700,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.109947+0000","last_change":"2024-09-13T13:14:15.616158+0000","last_active":"2024-09-13T13:15:42.109947+0000","last_peered":"2024-09-13T13:15:42.109947+0000","last_clean":"2024-09-13T13:15:42.109947+0000","last_became_active":"2024-09-13T13:12:52.950504+0000","last_became_peered":"2024-09-13T13:12:52.950504+0000","last_unstale":"2024-09-13T13:15:42.109947+0000","last_undegraded":"2024-09-13T13:15:42.109947+0000","last_fullsized":"2024-09-13T13:15:42.109947+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:15.616112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:15.616112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.725422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":690,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.116890+0000","last_change":"2024-09-13T13:12:34.682535+0000","last_active":"2024-09-13T13:15:42.116890+0000","last_peered":"2024-09-13T13:15:42.116890+0000","last_clean":"2024-09-13T13:15:42.116890+0000","last_became_active":"2024-09-13T13:12:34.665746+0000","last_became_peered":"2024-09-13T13:12:34.665746+0000","last_unstale":"2024-09-13T13:15:42.116890+0000","last_undegraded":"2024-09-13T13:15:42.116890+0000","last_fullsized":"2024-09-13T13:15:42.116890+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:19.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:19.931466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:41.387724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":656,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.117644+0000","last_change":"2024-09-13T13:13:05.981881+0000","last_active":"2024-09-13T13:15:42.117644+0000","last_peered":"2024-09-13T13:15:42.117644+0000","last_clean":"2024-09-13T13:15:42.117644+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:15:42.117644+0000","last_undegraded":"2024-09-13T13:15:42.117644+0000","last_fullsized":"2024-09-13T13:15:42.117644+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:05.981803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:05.981803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.697737+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":835,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.417556+0000","last_change":"2024-09-13T13:13:10.130755+0000","last_active":"2024-09-13T13:15:42.417556+0000","last_peered":"2024-09-13T13:15:42.417556+0000","last_clean":"2024-09-13T13:15:42.417556+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:15:42.417556+0000","last_undegraded":"2024-09-13T13:15:42.417556+0000","last_fullsized":"2024-09-13T13:15:42.417556+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:13:10.130680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.130680+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.267335+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":145,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.121082+0000","last_change":"2024-09-13T13:13:11.956837+0000","last_active":"2024-09-13T13:15:42.121082+0000","last_peered":"2024-09-13T13:15:42.121082+0000","last_clean":"2024-09-13T13:15:42.121082+0000","last_became_active":"2024-09-13T13:12:52.946339+0000","last_became_peered":"2024-09-13T13:12:52.946339+0000","last_unstale":"2024-09-13T13:15:42.121082+0000","last_undegraded":"2024-09-13T13:15:42.121082+0000","last_fullsized":"2024-09-13T13:15:42.121082+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.956784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.956784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":707,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.109862+0000","last_change":"2024-09-13T13:15:34.280338+0000","last_active":"2024-09-13T13:15:42.109862+0000","last_peered":"2024-09-13T13:15:42.109862+0000","last_clean":"2024-09-13T13:15:42.109862+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:15:42.109862+0000","last_undegraded":"2024-09-13T13:15:42.109862+0000","last_fullsized":"2024-09-13T13:15:42.109862+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:15:34.280253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:34.280253+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.301730+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":671,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.118373+0000","last_change":"2024-09-13T13:13:57.105151+0000","last_active":"2024-09-13T13:15:42.118373+0000","last_peered":"2024-09-13T13:15:42.118373+0000","last_clean":"2024-09-13T13:15:42.118373+0000","last_became_active":"2024-09-13T13:12:52.999912+0000","last_became_peered":"2024-09-13T13:12:52.999912+0000","last_unstale":"2024-09-13T13:15:42.118373+0000","last_undegraded":"2024-09-13T13:15:42.118373+0000","last_fullsized":"2024-09-13T13:15:42.118373+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.105098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.105098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.715266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":859,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.417761+0000","last_change":"2024-09-13T13:15:33.333062+0000","last_active":"2024-09-13T13:15:42.417761+0000","last_peered":"2024-09-13T13:15:42.417761+0000","last_clean":"2024-09-13T13:15:42.417761+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:15:42.417761+0000","last_undegraded":"2024-09-13T13:15:42.417761+0000","last_fullsized":"2024-09-13T13:15:42.417761+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:15:33.332981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.332981+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.941448+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":668,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.114041+0000","last_change":"2024-09-13T13:15:22.660262+0000","last_active":"2024-09-13T13:15:42.114041+0000","last_peered":"2024-09-13T13:15:42.114041+0000","last_clean":"2024-09-13T13:15:42.114041+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:15:42.114041+0000","last_undegraded":"2024-09-13T13:15:42.114041+0000","last_fullsized":"2024-09-13T13:15:42.114041+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:22.660050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:22.660050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:44.066070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":157,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.109392+0000","last_change":"2024-09-13T13:13:47.728092+0000","last_active":"2024-09-13T13:15:42.109392+0000","last_peered":"2024-09-13T13:15:42.109392+0000","last_clean":"2024-09-13T13:15:42.109392+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:15:42.109392+0000","last_undegraded":"2024-09-13T13:15:42.109392+0000","last_fullsized":"2024-09-13T13:15:42.109392+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:47.728010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:47.728010+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.358350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":137,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.114851+0000","last_change":"2024-09-13T13:12:53.006302+0000","last_active":"2024-09-13T13:15:42.114851+0000","last_peered":"2024-09-13T13:15:42.114851+0000","last_clean":"2024-09-13T13:15:42.114851+0000","last_became_active":"2024-09-13T13:12:53.005988+0000","last_became_peered":"2024-09-13T13:12:53.005988+0000","last_unstale":"2024-09-13T13:15:42.114851+0000","last_undegraded":"2024-09-13T13:15:42.114851+0000","last_fullsized":"2024-09-13T13:15:42.114851+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:40.888083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.888083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.372875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":666,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.124185+0000","last_change":"2024-09-13T13:14:50.185025+0000","last_active":"2024-09-13T13:15:42.124185+0000","last_peered":"2024-09-13T13:15:42.124185+0000","last_clean":"2024-09-13T13:15:42.124185+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:15:42.124185+0000","last_undegraded":"2024-09-13T13:15:42.124185+0000","last_fullsized":"2024-09-13T13:15:42.124185+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.184570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.184570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:56.711009+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":661,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.110897+0000","last_change":"2024-09-13T13:14:19.690490+0000","last_active":"2024-09-13T13:15:42.110897+0000","last_peered":"2024-09-13T13:15:42.110897+0000","last_clean":"2024-09-13T13:15:42.110897+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:15:42.110897+0000","last_undegraded":"2024-09-13T13:15:42.110897+0000","last_fullsized":"2024-09-13T13:15:42.110897+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:19.690414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:19.690414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:46.728053+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":629,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.117361+0000","last_change":"2024-09-13T13:12:39.874394+0000","last_active":"2024-09-13T13:15:42.117361+0000","last_peered":"2024-09-13T13:15:42.117361+0000","last_clean":"2024-09-13T13:15:42.117361+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:15:42.117361+0000","last_undegraded":"2024-09-13T13:15:42.117361+0000","last_fullsized":"2024-09-13T13:15:42.117361+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:39.874324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:39.874324+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.370889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":657,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.108589+0000","last_change":"2024-09-13T13:14:13.418389+0000","last_active":"2024-09-13T13:15:42.108589+0000","last_peered":"2024-09-13T13:15:42.108589+0000","last_clean":"2024-09-13T13:15:42.108589+0000","last_became_active":"2024-09-13T13:12:52.949094+0000","last_became_peered":"2024-09-13T13:12:52.949094+0000","last_unstale":"2024-09-13T13:15:42.108589+0000","last_undegraded":"2024-09-13T13:15:42.108589+0000","last_fullsized":"2024-09-13T13:15:42.108589+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:13.418196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:13.418196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:43.286027+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":665,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.112271+0000","last_change":"2024-09-13T13:15:01.269060+0000","last_active":"2024-09-13T13:15:42.112271+0000","last_peered":"2024-09-13T13:15:42.112271+0000","last_clean":"2024-09-13T13:15:42.112271+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:15:42.112271+0000","last_undegraded":"2024-09-13T13:15:42.112271+0000","last_fullsized":"2024-09-13T13:15:42.112271+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:01.268884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:01.268884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.329102+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":652,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.113082+0000","last_change":"2024-09-13T13:10:39.477882+0000","last_active":"2024-09-13T13:15:42.113082+0000","last_peered":"2024-09-13T13:15:42.113082+0000","last_clean":"2024-09-13T13:15:42.113082+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:15:42.113082+0000","last_undegraded":"2024-09-13T13:15:42.113082+0000","last_fullsized":"2024-09-13T13:15:42.113082+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:39.477683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.477683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.622029+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":687,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.111404+0000","last_change":"2024-09-13T13:13:49.268080+0000","last_active":"2024-09-13T13:15:42.111404+0000","last_peered":"2024-09-13T13:15:42.111404+0000","last_clean":"2024-09-13T13:15:42.111404+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:15:42.111404+0000","last_undegraded":"2024-09-13T13:15:42.111404+0000","last_fullsized":"2024-09-13T13:15:42.111404+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:49.268018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:49.268018+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.301185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":681,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.118232+0000","last_change":"2024-09-13T13:14:50.392659+0000","last_active":"2024-09-13T13:15:42.118232+0000","last_peered":"2024-09-13T13:15:42.118232+0000","last_clean":"2024-09-13T13:15:42.118232+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:15:42.118232+0000","last_undegraded":"2024-09-13T13:15:42.118232+0000","last_fullsized":"2024-09-13T13:15:42.118232+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.392444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.392444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.481736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":690,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.120807+0000","last_change":"2024-09-13T13:15:03.778214+0000","last_active":"2024-09-13T13:15:42.120807+0000","last_peered":"2024-09-13T13:15:42.120807+0000","last_clean":"2024-09-13T13:15:42.120807+0000","last_became_active":"2024-09-13T13:12:52.999748+0000","last_became_peered":"2024-09-13T13:12:52.999748+0000","last_unstale":"2024-09-13T13:15:42.120807+0000","last_undegraded":"2024-09-13T13:15:42.120807+0000","last_fullsized":"2024-09-13T13:15:42.120807+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.778170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.778170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:20.178488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":648,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.113011+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:15:42.113011+0000","last_peered":"2024-09-13T13:15:42.113011+0000","last_clean":"2024-09-13T13:15:42.113011+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:15:42.113011+0000","last_undegraded":"2024-09-13T13:15:42.113011+0000","last_fullsized":"2024-09-13T13:15:42.113011+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.231726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":697,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.110949+0000","last_change":"2024-09-13T13:14:12.368507+0000","last_active":"2024-09-13T13:15:42.110949+0000","last_peered":"2024-09-13T13:15:42.110949+0000","last_clean":"2024-09-13T13:15:42.110949+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:15:42.110949+0000","last_undegraded":"2024-09-13T13:15:42.110949+0000","last_fullsized":"2024-09-13T13:15:42.110949+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:14:12.368430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:12.368430+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.308538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":680,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.113813+0000","last_change":"2024-09-13T13:13:44.750043+0000","last_active":"2024-09-13T13:15:42.113813+0000","last_peered":"2024-09-13T13:15:42.113813+0000","last_clean":"2024-09-13T13:15:42.113813+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:15:42.113813+0000","last_undegraded":"2024-09-13T13:15:42.113813+0000","last_fullsized":"2024-09-13T13:15:42.113813+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:44.749961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:44.749961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.672770+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":600,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717516+0000","last_change":"2024-09-13T13:14:16.457160+0000","last_active":"2024-09-13T13:15:50.717516+0000","last_peered":"2024-09-13T13:15:50.717516+0000","last_clean":"2024-09-13T13:15:50.717516+0000","last_became_active":"2024-09-13T13:12:53.001131+0000","last_became_peered":"2024-09-13T13:12:53.001131+0000","last_unstale":"2024-09-13T13:15:50.717516+0000","last_undegraded":"2024-09-13T13:15:50.717516+0000","last_fullsized":"2024-09-13T13:15:50.717516+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:16.456784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:16.456784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:43.279313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":645,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.108056+0000","last_change":"2024-09-13T13:12:52.947380+0000","last_active":"2024-09-13T13:15:42.108056+0000","last_peered":"2024-09-13T13:15:42.108056+0000","last_clean":"2024-09-13T13:15:42.108056+0000","last_became_active":"2024-09-13T13:12:52.946611+0000","last_became_peered":"2024-09-13T13:12:52.946611+0000","last_unstale":"2024-09-13T13:15:42.108056+0000","last_undegraded":"2024-09-13T13:15:42.108056+0000","last_fullsized":"2024-09-13T13:15:42.108056+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:08.704450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:08.704450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.645492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":675,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.123173+0000","last_change":"2024-09-13T13:14:18.882969+0000","last_active":"2024-09-13T13:15:42.123173+0000","last_peered":"2024-09-13T13:15:42.123173+0000","last_clean":"2024-09-13T13:15:42.123173+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:15:42.123173+0000","last_undegraded":"2024-09-13T13:15:42.123173+0000","last_fullsized":"2024-09-13T13:15:42.123173+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:18.882903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:18.882903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:45.119719+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":657,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.111189+0000","last_change":"2024-09-13T13:11:07.564094+0000","last_active":"2024-09-13T13:15:42.111189+0000","last_peered":"2024-09-13T13:15:42.111189+0000","last_clean":"2024-09-13T13:15:42.111189+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:15:42.111189+0000","last_undegraded":"2024-09-13T13:15:42.111189+0000","last_fullsized":"2024-09-13T13:15:42.111189+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:07.564025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:07.564025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:35.266907+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":668,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717300+0000","last_change":"2024-09-13T13:11:49.264206+0000","last_active":"2024-09-13T13:15:50.717300+0000","last_peered":"2024-09-13T13:15:50.717300+0000","last_clean":"2024-09-13T13:15:50.717300+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:15:50.717300+0000","last_undegraded":"2024-09-13T13:15:50.717300+0000","last_fullsized":"2024-09-13T13:15:50.717300+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:49.264128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.264128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.230570+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":681,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T13:15:42.114194+0000","last_change":"2024-09-13T13:15:27.678822+0000","last_active":"2024-09-13T13:15:42.114194+0000","last_peered":"2024-09-13T13:15:42.114194+0000","last_clean":"2024-09-13T13:15:42.114194+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:15:42.114194+0000","last_undegraded":"2024-09-13T13:15:42.114194+0000","last_fullsized":"2024-09-13T13:15:42.114194+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:27.678752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:27.678752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:48.252436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":7237468184,"num_objects":110436,"num_object_clones":0,"num_object_copies":331308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110436,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144320,"num_write_kb":8152129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21813080064,"data_stored":21813067848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91020,"ondisk_log_size":91020,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":2508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":195,"num_read_kb":190,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201460,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2317660,"kb_used_data":1790860,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91427492,"statfs":{"total":95995035648,"available":93621751808,"internally_reserved":0,"allocated":1833840640,"data_stored":1833419537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":539425226},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141071,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3514272,"kb_used_data":3079612,"kb_used_omap":4,"kb_used_meta":434491,"kb_avail":90230880,"statfs":{"total":95995035648,"available":92396421120,"internally_reserved":0,"allocated":3153522688,"data_stored":3153170512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":444919469},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692032,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2788048,"kb_used_data":2243200,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":90957104,"statfs":{"total":95995035648,"available":93140074496,"internally_reserved":0,"allocated":2297036800,"data_stored":2296626281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920966,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4043752,"kb_used_data":3629272,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":89701400,"statfs":{"total":95995035648,"available":91854233600,"internally_reserved":0,"allocated":3716374528,"data_stored":3715939593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":424278470},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117197,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2838500,"kb_used_data":2225620,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":90906652,"statfs":{"total":95995035648,"available":93088411648,"internally_reserved":0,"allocated":2279034880,"data_stored":2278604737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":627571143},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411536,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3109764,"kb_used_data":2703732,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":90635388,"statfs":{"total":95995035648,"available":92810637312,"internally_reserved":0,"allocated":2768621568,"data_stored":2768222787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":415758790},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575063,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1891128,"kb_used_data":1358760,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":91854024,"statfs":{"total":95995035648,"available":94058520576,"internally_reserved":0,"allocated":1391370240,"data_stored":1390919408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738590,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5124516,"kb_used_data":4461652,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":88620636,"statfs":{"total":95995035648,"available":90747531264,"internally_reserved":0,"allocated":4568731648,"data_stored":4568302086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4527230976,"data_stored":4527226904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1378680832,"data_stored":1378680832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2745761792,"data_stored":2745761792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2254700544,"data_stored":2254700544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3684175872,"data_stored":3684171800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2273050624,"data_stored":2273050624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3122925568,"data_stored":3122921496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1826553856,"data_stored":1826553856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:15:55.137 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:15:55.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:55.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:55.244 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:55.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:55.345 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:55.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:55.390 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:55.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:55.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:55.534 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:55.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:55.635 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:55.667 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:55.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:55.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:55.770 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:55.784 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:55.799 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:55.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:55.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:55.847 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:15:55.902 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:55.946 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:55.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:55.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:55.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:55 smithi052 ceph-mon[29471]: pgmap v733: 148 pgs: 148 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-13T13:15:55.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:55.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:55 smithi052 ceph-mon[29471]: 1.0 scrub starts 2024-09-13T13:15:55.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:55 smithi052 ceph-mon[29471]: 1.0 scrub ok 2024-09-13T13:15:55.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:55 smithi052 ceph-mon[29471]: 5.f scrub starts 2024-09-13T13:15:55.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:55 smithi052 ceph-mon[29471]: 5.f scrub ok 2024-09-13T13:15:55.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:55 smithi052 ceph-mon[36682]: pgmap v733: 148 pgs: 148 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-13T13:15:55.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:55.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:55 smithi052 ceph-mon[36682]: 1.0 scrub starts 2024-09-13T13:15:55.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:55 smithi052 ceph-mon[36682]: 1.0 scrub ok 2024-09-13T13:15:55.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:55 smithi052 ceph-mon[36682]: 5.f scrub starts 2024-09-13T13:15:55.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:55 smithi052 ceph-mon[36682]: 5.f scrub ok 2024-09-13T13:15:56.048 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:56.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:55 smithi167 ceph-mon[33178]: pgmap v733: 148 pgs: 148 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-13T13:15:56.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:15:56.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:55 smithi167 ceph-mon[33178]: 1.0 scrub starts 2024-09-13T13:15:56.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:55 smithi167 ceph-mon[33178]: 1.0 scrub ok 2024-09-13T13:15:56.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:55 smithi167 ceph-mon[33178]: 5.f scrub starts 2024-09-13T13:15:56.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:55 smithi167 ceph-mon[33178]: 5.f scrub ok 2024-09-13T13:15:56.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:56.092 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:56.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:56.193 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:56.231 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:56.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:56.269 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:15:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:15:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:15:56.332 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:15:56.374 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:56.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:15:56.476 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:56.518 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:56.518 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:56.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:56.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:56.672 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:56.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:56.746 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:15:56.748 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:15:56.773 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:56.790 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:15:56.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:56.802 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:56.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:56.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:56.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:56.936 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:56.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:56 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1774540253' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:56 smithi052 ceph-mon[36682]: 10.6 scrub ok 2024-09-13T13:15:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:56 smithi052 ceph-mon[36682]: 4.8 scrub starts 2024-09-13T13:15:56.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:56 smithi052 ceph-mon[36682]: 4.8 scrub ok 2024-09-13T13:15:56.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:56 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1774540253' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:56.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:56 smithi052 ceph-mon[29471]: 10.6 scrub ok 2024-09-13T13:15:56.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:56 smithi052 ceph-mon[29471]: 4.8 scrub starts 2024-09-13T13:15:56.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:56 smithi052 ceph-mon[29471]: 4.8 scrub ok 2024-09-13T13:15:57.024 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:56 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1774540253' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:15:57.024 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:56 smithi167 ceph-mon[33178]: 10.6 scrub ok 2024-09-13T13:15:57.024 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:56 smithi167 ceph-mon[33178]: 4.8 scrub starts 2024-09-13T13:15:57.024 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:56 smithi167 ceph-mon[33178]: 4.8 scrub ok 2024-09-13T13:15:57.036 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:57.072 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:57.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:57.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:57.174 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:57.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:57.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:57.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:57.356 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:57.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:57.455 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:15:57.455 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:15:57.455 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:15:57.455 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:15:57.455 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:15:57.457 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:15:57.499 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:57.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:57.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:15:57.601 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:57.623 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:57.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:57.726 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:57.761 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:57.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:57.862 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:57.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:57.896 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:57.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:57.997 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:58.026 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:15:58.026 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:15:58.026 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:57 smithi052 ceph-mon[36682]: pgmap v734: 148 pgs: 148 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 823 B/s rd, 0 B/s wr, 920 op/s 2024-09-13T13:15:58.026 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:57 smithi052 ceph-mon[36682]: from='client.26263 v1:172.21.15.52:0/4071154752' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:15:58.026 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:57 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:15:58.026 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:57 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:15:58.027 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:57 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1774540253' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:15:58.027 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:57 smithi052 ceph-mon[36682]: osdmap e372: 8 total, 8 up, 8 in 2024-09-13T13:15:58.027 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:57 smithi052 ceph-mon[29471]: pgmap v734: 148 pgs: 148 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 823 B/s rd, 0 B/s wr, 920 op/s 2024-09-13T13:15:58.027 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:57 smithi052 ceph-mon[29471]: from='client.26263 v1:172.21.15.52:0/4071154752' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:15:58.027 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:57 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:15:58.027 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:57 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:15:58.027 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:57 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1774540253' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:15:58.028 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:57 smithi052 ceph-mon[29471]: osdmap e372: 8 total, 8 up, 8 in 2024-09-13T13:15:58.041 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:58.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:58.061 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:15:58.062 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:15:58.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:57 smithi167 ceph-mon[33178]: pgmap v734: 148 pgs: 148 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 823 B/s rd, 0 B/s wr, 920 op/s 2024-09-13T13:15:58.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:57 smithi167 ceph-mon[33178]: from='client.26263 v1:172.21.15.52:0/4071154752' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:15:58.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:57 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:15:58.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:57 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:15:58.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:57 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1774540253' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:15:58.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:57 smithi167 ceph-mon[33178]: osdmap e372: 8 total, 8 up, 8 in 2024-09-13T13:15:58.098 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:15:58.099 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:15:58.133 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:15:58.133 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:15:58.143 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:58.169 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:15:58.169 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:15:58.183 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:58.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:58.204 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:15:58.205 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:15:58.241 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:15:58.242 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:15:58.274 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:15:58.274 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:15:58.285 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:58.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:58.324 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:58.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:58.425 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:58.462 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:58.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:58.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:58.564 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:58.601 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:58.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:58.702 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:58.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:58.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:58.802 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":734,"stamp":"2024-09-13T13:15:55.931482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7008098867,"num_objects":107126,"num_object_clones":0,"num_object_copies":321378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107126,"num_whiteouts":0,"num_read":1442,"num_read_kb":1301,"num_write":148664,"num_write_kb":8153802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":594037,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91686,"ondisk_log_size":91686,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":24669528,"kb_used_data":20546864,"kb_used_omap":15,"kb_used_meta":4122032,"kb_avail":725291688,"statfs":{"total":767960285184,"available":742698688512,"internally_reserved":0,"allocated":21039988736,"data_stored":21036993563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":4220961561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-374931456,"num_objects":-5721,"num_object_clones":0,"num_object_copies":-17163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5721,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5721,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.220592"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":663,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.743085+0000","last_change":"2024-09-13T13:10:32.368663+0000","last_active":"2024-09-13T13:15:51.743085+0000","last_peered":"2024-09-13T13:15:51.743085+0000","last_clean":"2024-09-13T13:15:51.743085+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:15:51.743085+0000","last_undegraded":"2024-09-13T13:15:51.743085+0000","last_fullsized":"2024-09-13T13:15:51.743085+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.368588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.368588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.615999+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":161,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.727670+0000","last_change":"2024-09-13T13:15:04.344174+0000","last_active":"2024-09-13T13:15:51.727670+0000","last_peered":"2024-09-13T13:15:51.727670+0000","last_clean":"2024-09-13T13:15:51.727670+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:15:51.727670+0000","last_undegraded":"2024-09-13T13:15:51.727670+0000","last_fullsized":"2024-09-13T13:15:51.727670+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:15:04.344092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.344092+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.313962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":664,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.735457+0000","last_change":"2024-09-13T13:14:12.865175+0000","last_active":"2024-09-13T13:15:51.735457+0000","last_peered":"2024-09-13T13:15:51.735457+0000","last_clean":"2024-09-13T13:15:51.735457+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:15:51.735457+0000","last_undegraded":"2024-09-13T13:15:51.735457+0000","last_fullsized":"2024-09-13T13:15:51.735457+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:12.865101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:12.865101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:17.268032+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":159,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.729660+0000","last_change":"2024-09-13T13:15:51.244938+0000","last_active":"2024-09-13T13:15:51.729660+0000","last_peered":"2024-09-13T13:15:51.729660+0000","last_clean":"2024-09-13T13:15:51.729660+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:15:51.729660+0000","last_undegraded":"2024-09-13T13:15:51.729660+0000","last_fullsized":"2024-09-13T13:15:51.729660+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:19.110420+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":669,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734782+0000","last_change":"2024-09-13T13:11:47.852762+0000","last_active":"2024-09-13T13:15:51.734782+0000","last_peered":"2024-09-13T13:15:51.734782+0000","last_clean":"2024-09-13T13:15:51.734782+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:15:51.734782+0000","last_undegraded":"2024-09-13T13:15:51.734782+0000","last_fullsized":"2024-09-13T13:15:51.734782+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:15.241902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":691,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.738721+0000","last_change":"2024-09-13T13:15:24.253436+0000","last_active":"2024-09-13T13:15:51.738721+0000","last_peered":"2024-09-13T13:15:51.738721+0000","last_clean":"2024-09-13T13:15:51.738721+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:15:51.738721+0000","last_undegraded":"2024-09-13T13:15:51.738721+0000","last_fullsized":"2024-09-13T13:15:51.738721+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:24.253369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:24.253369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:50.007360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":734,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.737316+0000","last_change":"2024-09-13T13:15:50.330680+0000","last_active":"2024-09-13T13:15:51.737316+0000","last_peered":"2024-09-13T13:15:51.737316+0000","last_clean":"2024-09-13T13:15:51.737316+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:15:51.737316+0000","last_undegraded":"2024-09-13T13:15:51.737316+0000","last_fullsized":"2024-09-13T13:15:51.737316+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:15:50.330607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.330607+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:59.485090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":143,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.727617+0000","last_change":"2024-09-13T13:12:52.997865+0000","last_active":"2024-09-13T13:15:51.727617+0000","last_peered":"2024-09-13T13:15:51.727617+0000","last_clean":"2024-09-13T13:15:51.727617+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:15:51.727617+0000","last_undegraded":"2024-09-13T13:15:51.727617+0000","last_fullsized":"2024-09-13T13:15:51.727617+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:17.305608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:17.305608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.322592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":733,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734387+0000","last_change":"2024-09-13T13:13:41.518378+0000","last_active":"2024-09-13T13:15:51.734387+0000","last_peered":"2024-09-13T13:15:51.734387+0000","last_clean":"2024-09-13T13:15:51.734387+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:15:51.734387+0000","last_undegraded":"2024-09-13T13:15:51.734387+0000","last_fullsized":"2024-09-13T13:15:51.734387+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:41.518322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.518322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:12.339477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":153,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.729730+0000","last_change":"2024-09-13T13:12:53.033861+0000","last_active":"2024-09-13T13:15:51.729730+0000","last_peered":"2024-09-13T13:15:51.729730+0000","last_clean":"2024-09-13T13:15:51.729730+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:15:51.729730+0000","last_undegraded":"2024-09-13T13:15:51.729730+0000","last_fullsized":"2024-09-13T13:15:51.729730+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:31.738685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.738685+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.301470+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":670,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734936+0000","last_change":"2024-09-13T13:13:43.501368+0000","last_active":"2024-09-13T13:15:51.734936+0000","last_peered":"2024-09-13T13:15:51.734936+0000","last_clean":"2024-09-13T13:15:51.734936+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:15:51.734936+0000","last_undegraded":"2024-09-13T13:15:51.734936+0000","last_fullsized":"2024-09-13T13:15:51.734936+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:43.501206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:43.501206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.419736+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":674,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.743400+0000","last_change":"2024-09-13T13:15:25.628234+0000","last_active":"2024-09-13T13:15:51.743400+0000","last_peered":"2024-09-13T13:15:51.743400+0000","last_clean":"2024-09-13T13:15:51.743400+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:15:51.743400+0000","last_undegraded":"2024-09-13T13:15:51.743400+0000","last_fullsized":"2024-09-13T13:15:51.743400+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.628156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.628156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:41.988194+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":700,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.733735+0000","last_change":"2024-09-13T13:12:52.951531+0000","last_active":"2024-09-13T13:15:51.733735+0000","last_peered":"2024-09-13T13:15:51.733735+0000","last_clean":"2024-09-13T13:15:51.733735+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:15:51.733735+0000","last_undegraded":"2024-09-13T13:15:51.733735+0000","last_fullsized":"2024-09-13T13:15:51.733735+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:16.989972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:16.989972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.371147+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":157,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.727987+0000","last_change":"2024-09-13T13:15:03.348187+0000","last_active":"2024-09-13T13:15:51.727987+0000","last_peered":"2024-09-13T13:15:51.727987+0000","last_clean":"2024-09-13T13:15:51.727987+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:15:51.727987+0000","last_undegraded":"2024-09-13T13:15:51.727987+0000","last_fullsized":"2024-09-13T13:15:51.727987+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:15:03.348115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.348115+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.237423+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":691,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734506+0000","last_change":"2024-09-13T13:13:54.122282+0000","last_active":"2024-09-13T13:15:51.734506+0000","last_peered":"2024-09-13T13:15:51.734506+0000","last_clean":"2024-09-13T13:15:51.734506+0000","last_became_active":"2024-09-13T13:12:52.999970+0000","last_became_peered":"2024-09-13T13:12:52.999970+0000","last_unstale":"2024-09-13T13:15:51.734506+0000","last_undegraded":"2024-09-13T13:15:51.734506+0000","last_fullsized":"2024-09-13T13:15:51.734506+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.122222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.122222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:17.725218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":643,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.733236+0000","last_change":"2024-09-13T13:13:02.818644+0000","last_active":"2024-09-13T13:15:51.733236+0000","last_peered":"2024-09-13T13:15:51.733236+0000","last_clean":"2024-09-13T13:15:51.733236+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:15:51.733236+0000","last_undegraded":"2024-09-13T13:15:51.733236+0000","last_fullsized":"2024-09-13T13:15:51.733236+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:02.818543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:02.818543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.513000+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":147,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.729820+0000","last_change":"2024-09-13T13:12:53.014177+0000","last_active":"2024-09-13T13:15:51.729820+0000","last_peered":"2024-09-13T13:15:51.729820+0000","last_clean":"2024-09-13T13:15:51.729820+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:15:51.729820+0000","last_undegraded":"2024-09-13T13:15:51.729820+0000","last_fullsized":"2024-09-13T13:15:51.729820+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:28.804735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:28.804735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:42.251197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":699,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:52.658477+0000","last_change":"2024-09-13T13:15:52.658477+0000","last_active":"2024-09-13T13:15:52.658477+0000","last_peered":"2024-09-13T13:15:52.658477+0000","last_clean":"2024-09-13T13:15:52.658477+0000","last_became_active":"2024-09-13T13:12:53.008505+0000","last_became_peered":"2024-09-13T13:12:53.008505+0000","last_unstale":"2024-09-13T13:15:52.658477+0000","last_undegraded":"2024-09-13T13:15:52.658477+0000","last_fullsized":"2024-09-13T13:15:52.658477+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:52.658410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:52.658410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:05.062147+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":718,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.741356+0000","last_change":"2024-09-13T13:12:40.125646+0000","last_active":"2024-09-13T13:15:51.741356+0000","last_peered":"2024-09-13T13:15:51.741356+0000","last_clean":"2024-09-13T13:15:51.741356+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:15:51.741356+0000","last_undegraded":"2024-09-13T13:15:51.741356+0000","last_fullsized":"2024-09-13T13:15:51.741356+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:12:40.125545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.125545+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.676016+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":693,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.736325+0000","last_change":"2024-09-13T13:14:52.196086+0000","last_active":"2024-09-13T13:15:51.736325+0000","last_peered":"2024-09-13T13:15:51.736325+0000","last_clean":"2024-09-13T13:15:51.736325+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:15:51.736325+0000","last_undegraded":"2024-09-13T13:15:51.736325+0000","last_fullsized":"2024-09-13T13:15:51.736325+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.196041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.502726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":671,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.738730+0000","last_change":"2024-09-13T13:13:03.150524+0000","last_active":"2024-09-13T13:15:51.738730+0000","last_peered":"2024-09-13T13:15:51.738730+0000","last_clean":"2024-09-13T13:15:51.738730+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:15:51.738730+0000","last_undegraded":"2024-09-13T13:15:51.738730+0000","last_fullsized":"2024-09-13T13:15:51.738730+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:03.150442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.150442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.295693+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":151,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.729568+0000","last_change":"2024-09-13T13:13:39.876364+0000","last_active":"2024-09-13T13:15:51.729568+0000","last_peered":"2024-09-13T13:15:51.729568+0000","last_clean":"2024-09-13T13:15:51.729568+0000","last_became_active":"2024-09-13T13:12:52.995055+0000","last_became_peered":"2024-09-13T13:12:52.995055+0000","last_unstale":"2024-09-13T13:15:51.729568+0000","last_undegraded":"2024-09-13T13:15:51.729568+0000","last_fullsized":"2024-09-13T13:15:51.729568+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:39.876294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:39.876294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":736,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.739013+0000","last_change":"2024-09-13T13:13:41.164741+0000","last_active":"2024-09-13T13:15:51.739013+0000","last_peered":"2024-09-13T13:15:51.739013+0000","last_clean":"2024-09-13T13:15:51.739013+0000","last_became_active":"2024-09-13T13:12:52.953819+0000","last_became_peered":"2024-09-13T13:12:52.953819+0000","last_unstale":"2024-09-13T13:15:51.739013+0000","last_undegraded":"2024-09-13T13:15:51.739013+0000","last_fullsized":"2024-09-13T13:15:51.739013+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:13:41.164656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.164656+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:49.289309+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":153,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.727867+0000","last_change":"2024-09-13T13:15:29.208631+0000","last_active":"2024-09-13T13:15:51.727867+0000","last_peered":"2024-09-13T13:15:51.727867+0000","last_clean":"2024-09-13T13:15:51.727867+0000","last_became_active":"2024-09-13T13:13:06.646781+0000","last_became_peered":"2024-09-13T13:13:06.646781+0000","last_unstale":"2024-09-13T13:15:51.727867+0000","last_undegraded":"2024-09-13T13:15:51.727867+0000","last_fullsized":"2024-09-13T13:15:51.727867+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:29.208576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:29.208576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:34.495308+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":178,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.733268+0000","last_change":"2024-09-13T13:12:42.975335+0000","last_active":"2024-09-13T13:15:51.733268+0000","last_peered":"2024-09-13T13:15:51.733268+0000","last_clean":"2024-09-13T13:15:51.733268+0000","last_became_active":"2024-09-13T13:12:39.954363+0000","last_became_peered":"2024-09-13T13:12:39.954363+0000","last_unstale":"2024-09-13T13:15:51.733268+0000","last_undegraded":"2024-09-13T13:15:51.733268+0000","last_fullsized":"2024-09-13T13:15:51.733268+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:12:42.975266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:42.975266+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.483880+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":695,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.735601+0000","last_change":"2024-09-13T13:15:23.245116+0000","last_active":"2024-09-13T13:15:51.735601+0000","last_peered":"2024-09-13T13:15:51.735601+0000","last_clean":"2024-09-13T13:15:51.735601+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:15:51.735601+0000","last_undegraded":"2024-09-13T13:15:51.735601+0000","last_fullsized":"2024-09-13T13:15:51.735601+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:15:23.245047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.245047+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.758250+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":721,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.743942+0000","last_change":"2024-09-13T13:15:50.693943+0000","last_active":"2024-09-13T13:15:51.743942+0000","last_peered":"2024-09-13T13:15:51.743942+0000","last_clean":"2024-09-13T13:15:51.743942+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:15:51.743942+0000","last_undegraded":"2024-09-13T13:15:51.743942+0000","last_fullsized":"2024-09-13T13:15:51.743942+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:15:50.693753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.693753+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:56.425256+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"306'12","reported_seq":835,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717573+0000","last_change":"2024-09-13T13:14:21.530680+0000","last_active":"2024-09-13T13:15:50.717573+0000","last_peered":"2024-09-13T13:15:50.717573+0000","last_clean":"2024-09-13T13:15:50.717573+0000","last_became_active":"2024-09-13T13:12:52.947886+0000","last_became_peered":"2024-09-13T13:12:52.947886+0000","last_unstale":"2024-09-13T13:15:50.717573+0000","last_undegraded":"2024-09-13T13:15:50.717573+0000","last_fullsized":"2024-09-13T13:15:50.717573+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'12","last_scrub_stamp":"2024-09-13T13:14:21.530612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:21.530612+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.292277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"371'9301","reported_seq":9386,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:53.168314+0000","last_change":"2024-09-13T13:13:59.149493+0000","last_active":"2024-09-13T13:15:53.168314+0000","last_peered":"2024-09-13T13:15:53.168314+0000","last_clean":"2024-09-13T13:15:53.168314+0000","last_became_active":"2024-09-13T13:13:59.149215+0000","last_became_peered":"2024-09-13T13:13:59.149215+0000","last_unstale":"2024-09-13T13:15:53.168314+0000","last_undegraded":"2024-09-13T13:15:53.168314+0000","last_fullsized":"2024-09-13T13:15:53.168314+0000","mapping_epoch":334,"log_start":"350'4700","ondisk_log_start":"350'4700","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4601,"log_dups_size":0,"ondisk_log_size":4601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437321728,"num_objects":6673,"num_object_clones":0,"num_object_copies":20019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9301,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":765,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.735942+0000","last_change":"2024-09-13T13:15:02.354689+0000","last_active":"2024-09-13T13:15:51.735942+0000","last_peered":"2024-09-13T13:15:51.735942+0000","last_clean":"2024-09-13T13:15:51.735942+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:15:51.735942+0000","last_undegraded":"2024-09-13T13:15:51.735942+0000","last_fullsized":"2024-09-13T13:15:51.735942+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:15:02.354591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:02.354591+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:22.985661+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":695,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734678+0000","last_change":"2024-09-13T13:15:29.205122+0000","last_active":"2024-09-13T13:15:51.734678+0000","last_peered":"2024-09-13T13:15:51.734678+0000","last_clean":"2024-09-13T13:15:51.734678+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:15:51.734678+0000","last_undegraded":"2024-09-13T13:15:51.734678+0000","last_fullsized":"2024-09-13T13:15:51.734678+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:29.205058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:29.205058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.374619+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":192,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.729908+0000","last_change":"2024-09-13T13:15:32.212185+0000","last_active":"2024-09-13T13:15:51.729908+0000","last_peered":"2024-09-13T13:15:51.729908+0000","last_clean":"2024-09-13T13:15:51.729908+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:15:51.729908+0000","last_undegraded":"2024-09-13T13:15:51.729908+0000","last_fullsized":"2024-09-13T13:15:51.729908+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:15:32.212101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.212101+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:33.253768+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":675,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.733588+0000","last_change":"2024-09-13T13:13:57.546725+0000","last_active":"2024-09-13T13:15:51.733588+0000","last_peered":"2024-09-13T13:15:51.733588+0000","last_clean":"2024-09-13T13:15:51.733588+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:15:51.733588+0000","last_undegraded":"2024-09-13T13:15:51.733588+0000","last_fullsized":"2024-09-13T13:15:51.733588+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.546671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.546671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:18.300458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"371'9295","reported_seq":9380,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:53.162180+0000","last_change":"2024-09-13T13:13:58.962128+0000","last_active":"2024-09-13T13:15:53.162180+0000","last_peered":"2024-09-13T13:15:53.162180+0000","last_clean":"2024-09-13T13:15:53.162180+0000","last_became_active":"2024-09-13T13:13:58.961386+0000","last_became_peered":"2024-09-13T13:13:58.961386+0000","last_unstale":"2024-09-13T13:15:53.162180+0000","last_undegraded":"2024-09-13T13:15:53.162180+0000","last_fullsized":"2024-09-13T13:15:53.162180+0000","mapping_epoch":334,"log_start":"352'4700","ondisk_log_start":"352'4700","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4595,"log_dups_size":0,"ondisk_log_size":4595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.366364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438501376,"num_objects":6691,"num_object_clones":0,"num_object_copies":20073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9295,"num_write_kb":511552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":702,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734442+0000","last_change":"2024-09-13T13:13:50.232646+0000","last_active":"2024-09-13T13:15:51.734442+0000","last_peered":"2024-09-13T13:15:51.734442+0000","last_clean":"2024-09-13T13:15:51.734442+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:15:51.734442+0000","last_undegraded":"2024-09-13T13:15:51.734442+0000","last_fullsized":"2024-09-13T13:15:51.734442+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:13:50.232588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.232588+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.213810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":669,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.739030+0000","last_change":"2024-09-13T13:13:54.313641+0000","last_active":"2024-09-13T13:15:51.739030+0000","last_peered":"2024-09-13T13:15:51.739030+0000","last_clean":"2024-09-13T13:15:51.739030+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:15:51.739030+0000","last_undegraded":"2024-09-13T13:15:51.739030+0000","last_fullsized":"2024-09-13T13:15:51.739030+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:54.313570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.313570+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:18.294176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":151,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.727935+0000","last_change":"2024-09-13T13:15:00.323883+0000","last_active":"2024-09-13T13:15:51.727935+0000","last_peered":"2024-09-13T13:15:51.727935+0000","last_clean":"2024-09-13T13:15:51.727935+0000","last_became_active":"2024-09-13T13:12:53.006199+0000","last_became_peered":"2024-09-13T13:12:53.006199+0000","last_unstale":"2024-09-13T13:15:51.727935+0000","last_undegraded":"2024-09-13T13:15:51.727935+0000","last_fullsized":"2024-09-13T13:15:51.727935+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.323784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.323784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:00.914935+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":159,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.729806+0000","last_change":"2024-09-13T13:14:56.333854+0000","last_active":"2024-09-13T13:15:51.729806+0000","last_peered":"2024-09-13T13:15:51.729806+0000","last_clean":"2024-09-13T13:15:51.729806+0000","last_became_active":"2024-09-13T13:12:53.001045+0000","last_became_peered":"2024-09-13T13:12:53.001045+0000","last_unstale":"2024-09-13T13:15:51.729806+0000","last_undegraded":"2024-09-13T13:15:51.729806+0000","last_fullsized":"2024-09-13T13:15:51.729806+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:56.333778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:56.333778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:58.872471+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"370'9076","reported_seq":9159,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.130446+0000","last_change":"2024-09-13T13:13:58.966300+0000","last_active":"2024-09-13T13:15:51.130446+0000","last_peered":"2024-09-13T13:15:51.130446+0000","last_clean":"2024-09-13T13:15:51.130446+0000","last_became_active":"2024-09-13T13:13:58.966015+0000","last_became_peered":"2024-09-13T13:13:58.966015+0000","last_unstale":"2024-09-13T13:15:51.130446+0000","last_undegraded":"2024-09-13T13:15:51.130446+0000","last_fullsized":"2024-09-13T13:15:51.130446+0000","mapping_epoch":334,"log_start":"345'3000","ondisk_log_start":"345'3000","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.310606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449445888,"num_objects":6858,"num_object_clones":0,"num_object_copies":20574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9076,"num_write_kb":509888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":167,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.728085+0000","last_change":"2024-09-13T13:13:50.716312+0000","last_active":"2024-09-13T13:15:51.728085+0000","last_peered":"2024-09-13T13:15:51.728085+0000","last_clean":"2024-09-13T13:15:51.728085+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:15:51.728085+0000","last_undegraded":"2024-09-13T13:15:51.728085+0000","last_fullsized":"2024-09-13T13:15:51.728085+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:13:50.716243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.716243+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:10.352612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":677,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.747350+0000","last_change":"2024-09-13T13:13:57.886950+0000","last_active":"2024-09-13T13:15:51.747350+0000","last_peered":"2024-09-13T13:15:51.747350+0000","last_clean":"2024-09-13T13:15:51.747350+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:15:51.747350+0000","last_undegraded":"2024-09-13T13:15:51.747350+0000","last_fullsized":"2024-09-13T13:15:51.747350+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:57.886879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.886879+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:12.598128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":874,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734312+0000","last_change":"2024-09-13T13:13:55.316917+0000","last_active":"2024-09-13T13:15:51.734312+0000","last_peered":"2024-09-13T13:15:51.734312+0000","last_clean":"2024-09-13T13:15:51.734312+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:15:51.734312+0000","last_undegraded":"2024-09-13T13:15:51.734312+0000","last_fullsized":"2024-09-13T13:15:51.734312+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:13:55.316860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.316860+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.251395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":291,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717331+0000","last_change":"2024-09-13T13:14:59.391483+0000","last_active":"2024-09-13T13:15:50.717331+0000","last_peered":"2024-09-13T13:15:50.717331+0000","last_clean":"2024-09-13T13:15:50.717331+0000","last_became_active":"2024-09-13T13:12:39.948363+0000","last_became_peered":"2024-09-13T13:12:39.948363+0000","last_unstale":"2024-09-13T13:15:50.717331+0000","last_undegraded":"2024-09-13T13:15:50.717331+0000","last_fullsized":"2024-09-13T13:15:50.717331+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.391418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.391418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:15.991030+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"371'9172","reported_seq":9256,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:53.150291+0000","last_change":"2024-09-13T13:13:59.149331+0000","last_active":"2024-09-13T13:15:53.150291+0000","last_peered":"2024-09-13T13:15:53.150291+0000","last_clean":"2024-09-13T13:15:53.150291+0000","last_became_active":"2024-09-13T13:13:59.149046+0000","last_became_peered":"2024-09-13T13:13:59.149046+0000","last_unstale":"2024-09-13T13:15:53.150291+0000","last_undegraded":"2024-09-13T13:15:53.150291+0000","last_fullsized":"2024-09-13T13:15:53.150291+0000","mapping_epoch":334,"log_start":"350'4600","ondisk_log_start":"350'4600","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4572,"log_dups_size":0,"ondisk_log_size":4572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:12.327645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430571520,"num_objects":6570,"num_object_clones":0,"num_object_copies":19710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9172,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":737,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.736151+0000","last_change":"2024-09-13T13:12:53.016127+0000","last_active":"2024-09-13T13:15:51.736151+0000","last_peered":"2024-09-13T13:15:51.736151+0000","last_clean":"2024-09-13T13:15:51.736151+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:15:51.736151+0000","last_undegraded":"2024-09-13T13:15:51.736151+0000","last_fullsized":"2024-09-13T13:15:51.736151+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.376603+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":696,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734465+0000","last_change":"2024-09-13T13:14:27.622657+0000","last_active":"2024-09-13T13:15:51.734465+0000","last_peered":"2024-09-13T13:15:51.734465+0000","last_clean":"2024-09-13T13:15:51.734465+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:15:51.734465+0000","last_undegraded":"2024-09-13T13:15:51.734465+0000","last_fullsized":"2024-09-13T13:15:51.734465+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:27.622584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:27.622584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":864,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.745155+0000","last_change":"2024-09-13T13:15:51.663010+0000","last_active":"2024-09-13T13:15:51.745155+0000","last_peered":"2024-09-13T13:15:51.745155+0000","last_clean":"2024-09-13T13:15:51.745155+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:15:51.745155+0000","last_undegraded":"2024-09-13T13:15:51.745155+0000","last_fullsized":"2024-09-13T13:15:51.745155+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:15:51.662657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.662657+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:55.986754+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":691,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.743733+0000","last_change":"2024-09-13T13:15:26.667525+0000","last_active":"2024-09-13T13:15:51.743733+0000","last_peered":"2024-09-13T13:15:51.743733+0000","last_clean":"2024-09-13T13:15:51.743733+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:15:51.743733+0000","last_undegraded":"2024-09-13T13:15:51.743733+0000","last_fullsized":"2024-09-13T13:15:51.743733+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:26.667455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:26.667455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.516125+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"370'9238","reported_seq":9403,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.124137+0000","last_change":"2024-09-13T13:15:25.514356+0000","last_active":"2024-09-13T13:15:51.124137+0000","last_peered":"2024-09-13T13:15:51.124137+0000","last_clean":"2024-09-13T13:15:51.124137+0000","last_became_active":"2024-09-13T13:13:58.965950+0000","last_became_peered":"2024-09-13T13:13:58.965950+0000","last_unstale":"2024-09-13T13:15:51.124137+0000","last_undegraded":"2024-09-13T13:15:51.124137+0000","last_fullsized":"2024-09-13T13:15:51.124137+0000","mapping_epoch":334,"log_start":"345'3200","ondisk_log_start":"345'3200","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'7503","last_scrub_stamp":"2024-09-13T13:15:25.514118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.514118+0000","objects_scrubbed":7440,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.146280+0000","scrub_duration":2270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449773592,"num_objects":6864,"num_object_clones":0,"num_object_copies":20592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6864,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9238,"num_write_kb":515201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":734,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.745283+0000","last_change":"2024-09-13T13:13:45.757306+0000","last_active":"2024-09-13T13:15:51.745283+0000","last_peered":"2024-09-13T13:15:51.745283+0000","last_clean":"2024-09-13T13:15:51.745283+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:15:51.745283+0000","last_undegraded":"2024-09-13T13:15:51.745283+0000","last_fullsized":"2024-09-13T13:15:51.745283+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:13:45.751801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:45.751801+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.559426+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":665,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.735861+0000","last_change":"2024-09-13T13:14:24.955591+0000","last_active":"2024-09-13T13:15:51.735861+0000","last_peered":"2024-09-13T13:15:51.735861+0000","last_clean":"2024-09-13T13:15:51.735861+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:15:51.735861+0000","last_undegraded":"2024-09-13T13:15:51.735861+0000","last_fullsized":"2024-09-13T13:15:51.735861+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:24.955516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:24.955516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:45.361513+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":638,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.716536+0000","last_change":"2024-09-13T13:14:50.408656+0000","last_active":"2024-09-13T13:15:50.716536+0000","last_peered":"2024-09-13T13:15:50.716536+0000","last_clean":"2024-09-13T13:15:50.716536+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:15:50.716536+0000","last_undegraded":"2024-09-13T13:15:50.716536+0000","last_fullsized":"2024-09-13T13:15:50.716536+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.408378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.408378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.441158+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":672,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.735721+0000","last_change":"2024-09-13T13:13:50.772738+0000","last_active":"2024-09-13T13:15:51.735721+0000","last_peered":"2024-09-13T13:15:51.735721+0000","last_clean":"2024-09-13T13:15:51.735721+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:15:51.735721+0000","last_undegraded":"2024-09-13T13:15:51.735721+0000","last_fullsized":"2024-09-13T13:15:51.735721+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:50.772689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.772689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.211300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"370'9140","reported_seq":9223,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.147457+0000","last_change":"2024-09-13T13:13:58.966631+0000","last_active":"2024-09-13T13:15:51.147457+0000","last_peered":"2024-09-13T13:15:51.147457+0000","last_clean":"2024-09-13T13:15:51.147457+0000","last_became_active":"2024-09-13T13:13:58.965789+0000","last_became_peered":"2024-09-13T13:13:58.965789+0000","last_unstale":"2024-09-13T13:15:51.147457+0000","last_undegraded":"2024-09-13T13:15:51.147457+0000","last_fullsized":"2024-09-13T13:15:51.147457+0000","mapping_epoch":334,"log_start":"345'3100","ondisk_log_start":"345'3100","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.272671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443285504,"num_objects":6764,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9140,"num_write_kb":508928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"268'40","reported_seq":922,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.735820+0000","last_change":"2024-09-13T13:15:04.288539+0000","last_active":"2024-09-13T13:15:51.735820+0000","last_peered":"2024-09-13T13:15:51.735820+0000","last_clean":"2024-09-13T13:15:51.735820+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:15:51.735820+0000","last_undegraded":"2024-09-13T13:15:51.735820+0000","last_fullsized":"2024-09-13T13:15:51.735820+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'40","last_scrub_stamp":"2024-09-13T13:15:04.288431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.288431+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.738711+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":159,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.728076+0000","last_change":"2024-09-13T13:14:14.899463+0000","last_active":"2024-09-13T13:15:51.728076+0000","last_peered":"2024-09-13T13:15:51.728076+0000","last_clean":"2024-09-13T13:15:51.728076+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:15:51.728076+0000","last_undegraded":"2024-09-13T13:15:51.728076+0000","last_fullsized":"2024-09-13T13:15:51.728076+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:14.899378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:14.899378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":691,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.733534+0000","last_change":"2024-09-13T13:15:50.618476+0000","last_active":"2024-09-13T13:15:51.733534+0000","last_peered":"2024-09-13T13:15:51.733534+0000","last_clean":"2024-09-13T13:15:51.733534+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:15:51.733534+0000","last_undegraded":"2024-09-13T13:15:51.733534+0000","last_fullsized":"2024-09-13T13:15:51.733534+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:15:50.618426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.618426+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:58.477958+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":693,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.753434+0000","last_change":"2024-09-13T13:14:59.670853+0000","last_active":"2024-09-13T13:15:51.753434+0000","last_peered":"2024-09-13T13:15:51.753434+0000","last_clean":"2024-09-13T13:15:51.753434+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:15:51.753434+0000","last_undegraded":"2024-09-13T13:15:51.753434+0000","last_fullsized":"2024-09-13T13:15:51.753434+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.670779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.670779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.298565+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":681,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.738458+0000","last_change":"2024-09-13T13:15:33.280401+0000","last_active":"2024-09-13T13:15:51.738458+0000","last_peered":"2024-09-13T13:15:51.738458+0000","last_clean":"2024-09-13T13:15:51.738458+0000","last_became_active":"2024-09-13T13:12:52.948742+0000","last_became_peered":"2024-09-13T13:12:52.948742+0000","last_unstale":"2024-09-13T13:15:51.738458+0000","last_undegraded":"2024-09-13T13:15:51.738458+0000","last_fullsized":"2024-09-13T13:15:51.738458+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:33.280352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.280352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:51.771913+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"371'9174","reported_seq":9259,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:53.432494+0000","last_change":"2024-09-13T13:13:59.149677+0000","last_active":"2024-09-13T13:15:53.432494+0000","last_peered":"2024-09-13T13:15:53.432494+0000","last_clean":"2024-09-13T13:15:53.432494+0000","last_became_active":"2024-09-13T13:13:59.149368+0000","last_became_peered":"2024-09-13T13:13:59.149368+0000","last_unstale":"2024-09-13T13:15:53.432494+0000","last_undegraded":"2024-09-13T13:15:53.432494+0000","last_fullsized":"2024-09-13T13:15:53.432494+0000","mapping_epoch":334,"log_start":"346'3600","ondisk_log_start":"346'3600","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5574,"log_dups_size":0,"ondisk_log_size":5574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:10.338929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429916160,"num_objects":6560,"num_object_clones":0,"num_object_copies":19680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9174,"num_write_kb":503488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":701,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.736041+0000","last_change":"2024-09-13T13:15:33.299837+0000","last_active":"2024-09-13T13:15:51.736041+0000","last_peered":"2024-09-13T13:15:51.736041+0000","last_clean":"2024-09-13T13:15:51.736041+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:15:51.736041+0000","last_undegraded":"2024-09-13T13:15:51.736041+0000","last_fullsized":"2024-09-13T13:15:51.736041+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'17","last_scrub_stamp":"2024-09-13T13:15:33.299746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.299746+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:37.843249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":692,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.737605+0000","last_change":"2024-09-13T13:13:17.676405+0000","last_active":"2024-09-13T13:15:51.737605+0000","last_peered":"2024-09-13T13:15:51.737605+0000","last_clean":"2024-09-13T13:15:51.737605+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:15:51.737605+0000","last_undegraded":"2024-09-13T13:15:51.737605+0000","last_fullsized":"2024-09-13T13:15:51.737605+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:17.676339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:17.676339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.177334+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":696,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.743445+0000","last_change":"2024-09-13T13:13:14.560620+0000","last_active":"2024-09-13T13:15:51.743445+0000","last_peered":"2024-09-13T13:15:51.743445+0000","last_clean":"2024-09-13T13:15:51.743445+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:15:51.743445+0000","last_undegraded":"2024-09-13T13:15:51.743445+0000","last_fullsized":"2024-09-13T13:15:51.743445+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:13:14.560555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:14.560555+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.776407+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":667,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.748547+0000","last_change":"2024-09-13T13:14:23.728707+0000","last_active":"2024-09-13T13:15:51.748547+0000","last_peered":"2024-09-13T13:15:51.748547+0000","last_clean":"2024-09-13T13:15:51.748547+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:15:51.748547+0000","last_undegraded":"2024-09-13T13:15:51.748547+0000","last_fullsized":"2024-09-13T13:15:51.748547+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:23.728632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:23.728632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:52.457538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":291,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717447+0000","last_change":"2024-09-13T13:12:41.332308+0000","last_active":"2024-09-13T13:15:50.717447+0000","last_peered":"2024-09-13T13:15:50.717447+0000","last_clean":"2024-09-13T13:15:50.717447+0000","last_became_active":"2024-09-13T13:12:40.120242+0000","last_became_peered":"2024-09-13T13:12:40.120242+0000","last_unstale":"2024-09-13T13:15:50.717447+0000","last_undegraded":"2024-09-13T13:15:50.717447+0000","last_fullsized":"2024-09-13T13:15:50.717447+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:41.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:41.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.372949+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"371'9483","reported_seq":9568,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:54.546611+0000","last_change":"2024-09-13T13:13:59.149922+0000","last_active":"2024-09-13T13:15:54.546611+0000","last_peered":"2024-09-13T13:15:54.546611+0000","last_clean":"2024-09-13T13:15:54.546611+0000","last_became_active":"2024-09-13T13:13:59.149037+0000","last_became_peered":"2024-09-13T13:13:59.149037+0000","last_unstale":"2024-09-13T13:15:54.546611+0000","last_undegraded":"2024-09-13T13:15:54.546611+0000","last_fullsized":"2024-09-13T13:15:54.546611+0000","mapping_epoch":334,"log_start":"348'4100","ondisk_log_start":"348'4100","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5383,"log_dups_size":0,"ondisk_log_size":5383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.573331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433258496,"num_objects":6611,"num_object_clones":0,"num_object_copies":19833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":515008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":702,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.747179+0000","last_change":"2024-09-13T13:13:10.964903+0000","last_active":"2024-09-13T13:15:51.747179+0000","last_peered":"2024-09-13T13:15:51.747179+0000","last_clean":"2024-09-13T13:15:51.747179+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:15:51.747179+0000","last_undegraded":"2024-09-13T13:15:51.747179+0000","last_fullsized":"2024-09-13T13:15:51.747179+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:13:10.964809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.964809+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.681903+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":693,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.742969+0000","last_change":"2024-09-13T13:15:30.590673+0000","last_active":"2024-09-13T13:15:51.742969+0000","last_peered":"2024-09-13T13:15:51.742969+0000","last_clean":"2024-09-13T13:15:51.742969+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:15:51.742969+0000","last_undegraded":"2024-09-13T13:15:51.742969+0000","last_fullsized":"2024-09-13T13:15:51.742969+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:30.590466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:30.590466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:49.834025+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":409,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:52.409693+0000","last_change":"2024-09-13T13:14:48.128237+0000","last_active":"2024-09-13T13:15:52.409693+0000","last_peered":"2024-09-13T13:15:52.409693+0000","last_clean":"2024-09-13T13:15:52.409693+0000","last_became_active":"2024-09-13T13:12:39.947940+0000","last_became_peered":"2024-09-13T13:12:39.947940+0000","last_unstale":"2024-09-13T13:15:52.409693+0000","last_undegraded":"2024-09-13T13:15:52.409693+0000","last_fullsized":"2024-09-13T13:15:52.409693+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:14:48.128156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:48.128156+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.088862+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":200,"num_read_kb":195,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":663,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.735354+0000","last_change":"2024-09-13T13:15:32.318497+0000","last_active":"2024-09-13T13:15:51.735354+0000","last_peered":"2024-09-13T13:15:51.735354+0000","last_clean":"2024-09-13T13:15:51.735354+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:15:51.735354+0000","last_undegraded":"2024-09-13T13:15:51.735354+0000","last_fullsized":"2024-09-13T13:15:51.735354+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:32.318413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.318413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:32.934946+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":669,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.735542+0000","last_change":"2024-09-13T13:10:32.720742+0000","last_active":"2024-09-13T13:15:51.735542+0000","last_peered":"2024-09-13T13:15:51.735542+0000","last_clean":"2024-09-13T13:15:51.735542+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:15:51.735542+0000","last_undegraded":"2024-09-13T13:15:51.735542+0000","last_fullsized":"2024-09-13T13:15:51.735542+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.305143+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"370'9094","reported_seq":9177,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.192529+0000","last_change":"2024-09-13T13:13:58.967163+0000","last_active":"2024-09-13T13:15:51.192529+0000","last_peered":"2024-09-13T13:15:51.192529+0000","last_clean":"2024-09-13T13:15:51.192529+0000","last_became_active":"2024-09-13T13:13:58.965962+0000","last_became_peered":"2024-09-13T13:13:58.965962+0000","last_unstale":"2024-09-13T13:15:51.192529+0000","last_undegraded":"2024-09-13T13:15:51.192529+0000","last_fullsized":"2024-09-13T13:15:51.192529+0000","mapping_epoch":334,"log_start":"345'3000","ondisk_log_start":"345'3000","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":6094,"log_dups_size":0,"ondisk_log_size":6094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:12.328620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448135168,"num_objects":6838,"num_object_clones":0,"num_object_copies":20514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9094,"num_write_kb":509824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"306'32","reported_seq":793,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717105+0000","last_change":"2024-09-13T13:13:01.352416+0000","last_active":"2024-09-13T13:15:50.717105+0000","last_peered":"2024-09-13T13:15:50.717105+0000","last_clean":"2024-09-13T13:15:50.717105+0000","last_became_active":"2024-09-13T13:12:52.999924+0000","last_became_peered":"2024-09-13T13:12:52.999924+0000","last_unstale":"2024-09-13T13:15:50.717105+0000","last_undegraded":"2024-09-13T13:15:50.717105+0000","last_fullsized":"2024-09-13T13:15:50.717105+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:13:01.352337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.352337+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.487539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":683,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.737016+0000","last_change":"2024-09-13T13:15:04.182566+0000","last_active":"2024-09-13T13:15:51.737016+0000","last_peered":"2024-09-13T13:15:51.737016+0000","last_clean":"2024-09-13T13:15:51.737016+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:15:51.737016+0000","last_undegraded":"2024-09-13T13:15:51.737016+0000","last_fullsized":"2024-09-13T13:15:51.737016+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:04.182499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.182499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.040803+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":249,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.743928+0000","last_change":"2024-09-13T13:14:24.630282+0000","last_active":"2024-09-13T13:15:51.743928+0000","last_peered":"2024-09-13T13:15:51.743928+0000","last_clean":"2024-09-13T13:15:51.743928+0000","last_became_active":"2024-09-13T13:12:39.948765+0000","last_became_peered":"2024-09-13T13:12:39.948765+0000","last_unstale":"2024-09-13T13:15:51.743928+0000","last_undegraded":"2024-09-13T13:15:51.743928+0000","last_fullsized":"2024-09-13T13:15:51.743928+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:14:24.630199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:14:24.630199+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":182,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.741026+0000","last_change":"2024-09-13T13:15:33.709071+0000","last_active":"2024-09-13T13:15:51.741026+0000","last_peered":"2024-09-13T13:15:51.741026+0000","last_clean":"2024-09-13T13:15:51.741026+0000","last_became_active":"2024-09-13T13:12:39.950496+0000","last_became_peered":"2024-09-13T13:12:39.950496+0000","last_unstale":"2024-09-13T13:15:51.741026+0000","last_undegraded":"2024-09-13T13:15:51.741026+0000","last_fullsized":"2024-09-13T13:15:51.741026+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:33.708995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.708995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:57.976953+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":697,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.735189+0000","last_change":"2024-09-13T13:14:23.625418+0000","last_active":"2024-09-13T13:15:51.735189+0000","last_peered":"2024-09-13T13:15:51.735189+0000","last_clean":"2024-09-13T13:15:51.735189+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:15:51.735189+0000","last_undegraded":"2024-09-13T13:15:51.735189+0000","last_fullsized":"2024-09-13T13:15:51.735189+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:23.625281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:23.625281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"370'9294","reported_seq":9445,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.140396+0000","last_change":"2024-09-13T13:15:04.191246+0000","last_active":"2024-09-13T13:15:51.140396+0000","last_peered":"2024-09-13T13:15:51.140396+0000","last_clean":"2024-09-13T13:15:51.140396+0000","last_became_active":"2024-09-13T13:13:58.965566+0000","last_became_peered":"2024-09-13T13:13:58.965566+0000","last_unstale":"2024-09-13T13:15:51.140396+0000","last_undegraded":"2024-09-13T13:15:51.140396+0000","last_fullsized":"2024-09-13T13:15:51.140396+0000","mapping_epoch":334,"log_start":"345'3200","ondisk_log_start":"345'3200","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'6049","last_scrub_stamp":"2024-09-13T13:15:04.191100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.191100+0000","objects_scrubbed":5987,"log_size":6094,"log_dups_size":0,"ondisk_log_size":6094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:31.390342+0000","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451805184,"num_objects":6894,"num_object_clones":0,"num_object_copies":20682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9294,"num_write_kb":518016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":171,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.729832+0000","last_change":"2024-09-13T13:14:59.354034+0000","last_active":"2024-09-13T13:15:51.729832+0000","last_peered":"2024-09-13T13:15:51.729832+0000","last_clean":"2024-09-13T13:15:51.729832+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:15:51.729832+0000","last_undegraded":"2024-09-13T13:15:51.729832+0000","last_fullsized":"2024-09-13T13:15:51.729832+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:14:59.353691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.353691+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:10.915988+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":151,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.727858+0000","last_change":"2024-09-13T13:14:26.343833+0000","last_active":"2024-09-13T13:15:51.727858+0000","last_peered":"2024-09-13T13:15:51.727858+0000","last_clean":"2024-09-13T13:15:51.727858+0000","last_became_active":"2024-09-13T13:12:53.006271+0000","last_became_peered":"2024-09-13T13:12:53.006271+0000","last_unstale":"2024-09-13T13:15:51.727858+0000","last_undegraded":"2024-09-13T13:15:51.727858+0000","last_fullsized":"2024-09-13T13:15:51.727858+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:26.343754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:26.343754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":712,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734959+0000","last_change":"2024-09-13T13:14:14.630794+0000","last_active":"2024-09-13T13:15:51.734959+0000","last_peered":"2024-09-13T13:15:51.734959+0000","last_clean":"2024-09-13T13:15:51.734959+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:15:51.734959+0000","last_undegraded":"2024-09-13T13:15:51.734959+0000","last_fullsized":"2024-09-13T13:15:51.734959+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:14.630719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:14.630719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":670,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.738109+0000","last_change":"2024-09-13T13:14:23.655250+0000","last_active":"2024-09-13T13:15:51.738109+0000","last_peered":"2024-09-13T13:15:51.738109+0000","last_clean":"2024-09-13T13:15:51.738109+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:15:51.738109+0000","last_undegraded":"2024-09-13T13:15:51.738109+0000","last_fullsized":"2024-09-13T13:15:51.738109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:23.655072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:23.655072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:45.281222+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"371'9307","reported_seq":9477,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:53.468629+0000","last_change":"2024-09-13T13:15:31.408104+0000","last_active":"2024-09-13T13:15:53.468629+0000","last_peered":"2024-09-13T13:15:53.468629+0000","last_clean":"2024-09-13T13:15:53.468629+0000","last_became_active":"2024-09-13T13:13:58.961825+0000","last_became_peered":"2024-09-13T13:13:58.961825+0000","last_unstale":"2024-09-13T13:15:53.468629+0000","last_undegraded":"2024-09-13T13:15:53.468629+0000","last_fullsized":"2024-09-13T13:15:53.468629+0000","mapping_epoch":334,"log_start":"344'2600","ondisk_log_start":"344'2600","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7793","last_scrub_stamp":"2024-09-13T13:15:31.408001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:31.408001+0000","objects_scrubbed":7714,"log_size":6707,"log_dups_size":0,"ondisk_log_size":6707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:37.609213+0000","scrub_duration":2771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433782784,"num_objects":6619,"num_object_clones":0,"num_object_copies":19857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":509632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":177,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.728673+0000","last_change":"2024-09-13T13:14:20.221364+0000","last_active":"2024-09-13T13:15:51.728673+0000","last_peered":"2024-09-13T13:15:51.728673+0000","last_clean":"2024-09-13T13:15:51.728673+0000","last_became_active":"2024-09-13T13:12:53.001687+0000","last_became_peered":"2024-09-13T13:12:53.001687+0000","last_unstale":"2024-09-13T13:15:51.728673+0000","last_undegraded":"2024-09-13T13:15:51.728673+0000","last_fullsized":"2024-09-13T13:15:51.728673+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:14:20.221286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:20.221286+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":692,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.716760+0000","last_change":"2024-09-13T13:15:24.265548+0000","last_active":"2024-09-13T13:15:50.716760+0000","last_peered":"2024-09-13T13:15:50.716760+0000","last_clean":"2024-09-13T13:15:50.716760+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:15:50.716760+0000","last_undegraded":"2024-09-13T13:15:50.716760+0000","last_fullsized":"2024-09-13T13:15:50.716760+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:24.265477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:24.265477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:35.804220+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":686,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.735647+0000","last_change":"2024-09-13T13:15:24.264425+0000","last_active":"2024-09-13T13:15:51.735647+0000","last_peered":"2024-09-13T13:15:51.735647+0000","last_clean":"2024-09-13T13:15:51.735647+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:15:51.735647+0000","last_undegraded":"2024-09-13T13:15:51.735647+0000","last_fullsized":"2024-09-13T13:15:51.735647+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:24.264316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:24.264316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:32.724876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":674,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.748776+0000","last_change":"2024-09-13T13:13:54.649219+0000","last_active":"2024-09-13T13:15:51.748776+0000","last_peered":"2024-09-13T13:15:51.748776+0000","last_clean":"2024-09-13T13:15:51.748776+0000","last_became_active":"2024-09-13T13:12:52.938975+0000","last_became_peered":"2024-09-13T13:12:52.938975+0000","last_unstale":"2024-09-13T13:15:51.748776+0000","last_undegraded":"2024-09-13T13:15:51.748776+0000","last_fullsized":"2024-09-13T13:15:51.748776+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.649166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.649166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.656296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"371'9385","reported_seq":9470,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:54.042270+0000","last_change":"2024-09-13T13:13:58.963327+0000","last_active":"2024-09-13T13:15:54.042270+0000","last_peered":"2024-09-13T13:15:54.042270+0000","last_clean":"2024-09-13T13:15:54.042270+0000","last_became_active":"2024-09-13T13:13:58.963064+0000","last_became_peered":"2024-09-13T13:13:58.963064+0000","last_unstale":"2024-09-13T13:15:54.042270+0000","last_undegraded":"2024-09-13T13:15:54.042270+0000","last_fullsized":"2024-09-13T13:15:54.042270+0000","mapping_epoch":334,"log_start":"348'4100","ondisk_log_start":"348'4100","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5285,"log_dups_size":0,"ondisk_log_size":5285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.315173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436011008,"num_objects":6653,"num_object_clones":0,"num_object_copies":19959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9385,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":155,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.727944+0000","last_change":"2024-09-13T13:12:53.018645+0000","last_active":"2024-09-13T13:15:51.727944+0000","last_peered":"2024-09-13T13:15:51.727944+0000","last_clean":"2024-09-13T13:15:51.727944+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:15:51.727944+0000","last_undegraded":"2024-09-13T13:15:51.727944+0000","last_fullsized":"2024-09-13T13:15:51.727944+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:09.739184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.739184+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.221949+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":677,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.746442+0000","last_change":"2024-09-13T13:12:53.003530+0000","last_active":"2024-09-13T13:15:51.746442+0000","last_peered":"2024-09-13T13:15:51.746442+0000","last_clean":"2024-09-13T13:15:51.746442+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:15:51.746442+0000","last_undegraded":"2024-09-13T13:15:51.746442+0000","last_fullsized":"2024-09-13T13:15:51.746442+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.996346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.996346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:10.180309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":843,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.741617+0000","last_change":"2024-09-13T13:12:36.113757+0000","last_active":"2024-09-13T13:15:51.741617+0000","last_peered":"2024-09-13T13:15:51.741617+0000","last_clean":"2024-09-13T13:15:51.741617+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:15:51.741617+0000","last_undegraded":"2024-09-13T13:15:51.741617+0000","last_fullsized":"2024-09-13T13:15:51.741617+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:12:36.113709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:36.113709+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.715547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":151,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.729706+0000","last_change":"2024-09-13T13:13:41.797728+0000","last_active":"2024-09-13T13:15:51.729706+0000","last_peered":"2024-09-13T13:15:51.729706+0000","last_clean":"2024-09-13T13:15:51.729706+0000","last_became_active":"2024-09-13T13:12:52.997283+0000","last_became_peered":"2024-09-13T13:12:52.997283+0000","last_unstale":"2024-09-13T13:15:51.729706+0000","last_undegraded":"2024-09-13T13:15:51.729706+0000","last_fullsized":"2024-09-13T13:15:51.729706+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.797639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.797639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:06.323760+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"371'9306","reported_seq":9477,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:53.509090+0000","last_change":"2024-09-13T13:15:32.530404+0000","last_active":"2024-09-13T13:15:53.509090+0000","last_peered":"2024-09-13T13:15:53.509090+0000","last_clean":"2024-09-13T13:15:53.509090+0000","last_became_active":"2024-09-13T13:13:58.962803+0000","last_became_peered":"2024-09-13T13:13:58.962803+0000","last_unstale":"2024-09-13T13:15:53.509090+0000","last_undegraded":"2024-09-13T13:15:53.509090+0000","last_fullsized":"2024-09-13T13:15:53.509090+0000","mapping_epoch":334,"log_start":"344'2600","ondisk_log_start":"344'2600","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7822","last_scrub_stamp":"2024-09-13T13:15:32.530220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.530220+0000","objects_scrubbed":7744,"log_size":6706,"log_dups_size":0,"ondisk_log_size":6706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:48.786652+0000","scrub_duration":2862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429522944,"num_objects":6554,"num_object_clones":0,"num_object_copies":19662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9306,"num_write_kb":507520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":709,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.745201+0000","last_change":"2024-09-13T13:13:56.850149+0000","last_active":"2024-09-13T13:15:51.745201+0000","last_peered":"2024-09-13T13:15:51.745201+0000","last_clean":"2024-09-13T13:15:51.745201+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:15:51.745201+0000","last_undegraded":"2024-09-13T13:15:51.745201+0000","last_fullsized":"2024-09-13T13:15:51.745201+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:56.850094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.850094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.708354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":698,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.733858+0000","last_change":"2024-09-13T13:12:58.857634+0000","last_active":"2024-09-13T13:15:51.733858+0000","last_peered":"2024-09-13T13:15:51.733858+0000","last_clean":"2024-09-13T13:15:51.733858+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:15:51.733858+0000","last_undegraded":"2024-09-13T13:15:51.733858+0000","last_fullsized":"2024-09-13T13:15:51.733858+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:58.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:44.458422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":710,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.739561+0000","last_change":"2024-09-13T13:15:23.682529+0000","last_active":"2024-09-13T13:15:51.739561+0000","last_peered":"2024-09-13T13:15:51.739561+0000","last_clean":"2024-09-13T13:15:51.739561+0000","last_became_active":"2024-09-13T13:12:52.944725+0000","last_became_peered":"2024-09-13T13:12:52.944725+0000","last_unstale":"2024-09-13T13:15:51.739561+0000","last_undegraded":"2024-09-13T13:15:51.739561+0000","last_fullsized":"2024-09-13T13:15:51.739561+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:23.682463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.682463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.810934+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":658,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.742862+0000","last_change":"2024-09-13T13:13:41.762003+0000","last_active":"2024-09-13T13:15:51.742862+0000","last_peered":"2024-09-13T13:15:51.742862+0000","last_clean":"2024-09-13T13:15:51.742862+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:15:51.742862+0000","last_undegraded":"2024-09-13T13:15:51.742862+0000","last_fullsized":"2024-09-13T13:15:51.742862+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.761814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.761814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.545140+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"371'9207","reported_seq":9292,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:54.569790+0000","last_change":"2024-09-13T13:13:58.963548+0000","last_active":"2024-09-13T13:15:54.569790+0000","last_peered":"2024-09-13T13:15:54.569790+0000","last_clean":"2024-09-13T13:15:54.569790+0000","last_became_active":"2024-09-13T13:13:58.963160+0000","last_became_peered":"2024-09-13T13:13:58.963160+0000","last_unstale":"2024-09-13T13:15:54.569790+0000","last_undegraded":"2024-09-13T13:15:54.569790+0000","last_fullsized":"2024-09-13T13:15:54.569790+0000","mapping_epoch":334,"log_start":"346'3800","ondisk_log_start":"346'3800","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:11.607417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425394176,"num_objects":6491,"num_object_clones":0,"num_object_copies":19473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9207,"num_write_kb":502336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":681,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.716821+0000","last_change":"2024-09-13T13:13:06.339762+0000","last_active":"2024-09-13T13:15:50.716821+0000","last_peered":"2024-09-13T13:15:50.716821+0000","last_clean":"2024-09-13T13:15:50.716821+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:15:50.716821+0000","last_undegraded":"2024-09-13T13:15:50.716821+0000","last_fullsized":"2024-09-13T13:15:50.716821+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:13:06.339664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.339664+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:11.302026+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":313,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734235+0000","last_change":"2024-09-13T13:14:55.471954+0000","last_active":"2024-09-13T13:15:51.734235+0000","last_peered":"2024-09-13T13:15:51.734235+0000","last_clean":"2024-09-13T13:15:51.734235+0000","last_became_active":"2024-09-13T13:12:39.954315+0000","last_became_peered":"2024-09-13T13:12:39.954315+0000","last_unstale":"2024-09-13T13:15:51.734235+0000","last_undegraded":"2024-09-13T13:15:51.734235+0000","last_fullsized":"2024-09-13T13:15:51.734235+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:55.471885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:55.471885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:01.346273+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":672,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.744654+0000","last_change":"2024-09-13T13:13:11.484689+0000","last_active":"2024-09-13T13:15:51.744654+0000","last_peered":"2024-09-13T13:15:51.744654+0000","last_clean":"2024-09-13T13:15:51.744654+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:15:51.744654+0000","last_undegraded":"2024-09-13T13:15:51.744654+0000","last_fullsized":"2024-09-13T13:15:51.744654+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.484615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.484615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.740951+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":166,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.739402+0000","last_change":"2024-09-13T13:15:02.627642+0000","last_active":"2024-09-13T13:15:51.739402+0000","last_peered":"2024-09-13T13:15:51.739402+0000","last_clean":"2024-09-13T13:15:51.739402+0000","last_became_active":"2024-09-13T13:12:40.118050+0000","last_became_peered":"2024-09-13T13:12:40.118050+0000","last_unstale":"2024-09-13T13:15:51.739402+0000","last_undegraded":"2024-09-13T13:15:51.739402+0000","last_fullsized":"2024-09-13T13:15:51.739402+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:02.627569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:02.627569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:32.377161+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"370'8971","reported_seq":9054,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.142020+0000","last_change":"2024-09-13T13:13:58.966955+0000","last_active":"2024-09-13T13:15:51.142020+0000","last_peered":"2024-09-13T13:15:51.142020+0000","last_clean":"2024-09-13T13:15:51.142020+0000","last_became_active":"2024-09-13T13:13:58.966675+0000","last_became_peered":"2024-09-13T13:13:58.966675+0000","last_unstale":"2024-09-13T13:15:51.142020+0000","last_undegraded":"2024-09-13T13:15:51.142020+0000","last_fullsized":"2024-09-13T13:15:51.142020+0000","mapping_epoch":334,"log_start":"344'2900","ondisk_log_start":"344'2900","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":6071,"log_dups_size":0,"ondisk_log_size":6071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.259461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433782784,"num_objects":6619,"num_object_clones":0,"num_object_copies":19857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8971,"num_write_kb":498880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":163,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.728213+0000","last_change":"2024-09-13T13:13:20.064693+0000","last_active":"2024-09-13T13:15:51.728213+0000","last_peered":"2024-09-13T13:15:51.728213+0000","last_clean":"2024-09-13T13:15:51.728213+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:15:51.728213+0000","last_undegraded":"2024-09-13T13:15:51.728213+0000","last_fullsized":"2024-09-13T13:15:51.728213+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:13:20.064614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:20.064614+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":159,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.729701+0000","last_change":"2024-09-13T13:14:52.383027+0000","last_active":"2024-09-13T13:15:51.729701+0000","last_peered":"2024-09-13T13:15:51.729701+0000","last_clean":"2024-09-13T13:15:51.729701+0000","last_became_active":"2024-09-13T13:12:53.007561+0000","last_became_peered":"2024-09-13T13:12:53.007561+0000","last_unstale":"2024-09-13T13:15:51.729701+0000","last_undegraded":"2024-09-13T13:15:51.729701+0000","last_fullsized":"2024-09-13T13:15:51.729701+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.382755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.382755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:20.097418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":664,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.739700+0000","last_change":"2024-09-13T13:15:00.637152+0000","last_active":"2024-09-13T13:15:51.739700+0000","last_peered":"2024-09-13T13:15:51.739700+0000","last_clean":"2024-09-13T13:15:51.739700+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:15:51.739700+0000","last_undegraded":"2024-09-13T13:15:51.739700+0000","last_fullsized":"2024-09-13T13:15:51.739700+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.636898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.636898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.204507+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":651,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734905+0000","last_change":"2024-09-13T13:15:03.223521+0000","last_active":"2024-09-13T13:15:51.734905+0000","last_peered":"2024-09-13T13:15:51.734905+0000","last_clean":"2024-09-13T13:15:51.734905+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:15:51.734905+0000","last_undegraded":"2024-09-13T13:15:51.734905+0000","last_fullsized":"2024-09-13T13:15:51.734905+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.223265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.223265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.540511+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"371'9386","reported_seq":9469,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:53.401517+0000","last_change":"2024-09-13T13:13:58.963039+0000","last_active":"2024-09-13T13:15:53.401517+0000","last_peered":"2024-09-13T13:15:53.401517+0000","last_clean":"2024-09-13T13:15:53.401517+0000","last_became_active":"2024-09-13T13:13:58.962773+0000","last_became_peered":"2024-09-13T13:13:58.962773+0000","last_unstale":"2024-09-13T13:15:53.401517+0000","last_undegraded":"2024-09-13T13:15:53.401517+0000","last_fullsized":"2024-09-13T13:15:53.401517+0000","mapping_epoch":334,"log_start":"346'3800","ondisk_log_start":"346'3800","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5586,"log_dups_size":0,"ondisk_log_size":5586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436994048,"num_objects":6668,"num_object_clones":0,"num_object_copies":20004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9386,"num_write_kb":513728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":737,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.732819+0000","last_change":"2024-09-13T13:14:20.570758+0000","last_active":"2024-09-13T13:15:51.732819+0000","last_peered":"2024-09-13T13:15:51.732819+0000","last_clean":"2024-09-13T13:15:51.732819+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:15:51.732819+0000","last_undegraded":"2024-09-13T13:15:51.732819+0000","last_fullsized":"2024-09-13T13:15:51.732819+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:14:20.570698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:20.570698+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":157,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.728686+0000","last_change":"2024-09-13T13:13:01.972907+0000","last_active":"2024-09-13T13:15:51.728686+0000","last_peered":"2024-09-13T13:15:51.728686+0000","last_clean":"2024-09-13T13:15:51.728686+0000","last_became_active":"2024-09-13T13:12:53.006362+0000","last_became_peered":"2024-09-13T13:12:53.006362+0000","last_unstale":"2024-09-13T13:15:51.728686+0000","last_undegraded":"2024-09-13T13:15:51.728686+0000","last_fullsized":"2024-09-13T13:15:51.728686+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:13:01.972831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.972831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:46.279374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"306'13","reported_seq":395,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717428+0000","last_change":"2024-09-13T13:13:03.334319+0000","last_active":"2024-09-13T13:15:50.717428+0000","last_peered":"2024-09-13T13:15:50.717428+0000","last_clean":"2024-09-13T13:15:50.717428+0000","last_became_active":"2024-09-13T13:12:53.001189+0000","last_became_peered":"2024-09-13T13:12:53.001189+0000","last_unstale":"2024-09-13T13:15:50.717428+0000","last_undegraded":"2024-09-13T13:15:50.717428+0000","last_fullsized":"2024-09-13T13:15:50.717428+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'13","last_scrub_stamp":"2024-09-13T13:13:03.334012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.334012+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.265982+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":670,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.733222+0000","last_change":"2024-09-13T13:13:55.500961+0000","last_active":"2024-09-13T13:15:51.733222+0000","last_peered":"2024-09-13T13:15:51.733222+0000","last_clean":"2024-09-13T13:15:51.733222+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:15:51.733222+0000","last_undegraded":"2024-09-13T13:15:51.733222+0000","last_fullsized":"2024-09-13T13:15:51.733222+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:55.500908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.500908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:11.347558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":646,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.716998+0000","last_change":"2024-09-13T13:13:04.371082+0000","last_active":"2024-09-13T13:15:50.716998+0000","last_peered":"2024-09-13T13:15:50.716998+0000","last_clean":"2024-09-13T13:15:50.716998+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:15:50.716998+0000","last_undegraded":"2024-09-13T13:15:50.716998+0000","last_fullsized":"2024-09-13T13:15:50.716998+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.370881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.370881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.428165+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":193,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734752+0000","last_change":"2024-09-13T13:15:23.319486+0000","last_active":"2024-09-13T13:15:51.734752+0000","last_peered":"2024-09-13T13:15:51.734752+0000","last_clean":"2024-09-13T13:15:51.734752+0000","last_became_active":"2024-09-13T13:12:34.673528+0000","last_became_peered":"2024-09-13T13:12:34.673528+0000","last_unstale":"2024-09-13T13:15:51.734752+0000","last_undegraded":"2024-09-13T13:15:51.734752+0000","last_fullsized":"2024-09-13T13:15:51.734752+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:15:23.319412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.319412+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:45.332563+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":695,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734720+0000","last_change":"2024-09-13T13:15:51.633487+0000","last_active":"2024-09-13T13:15:51.734720+0000","last_peered":"2024-09-13T13:15:51.734720+0000","last_clean":"2024-09-13T13:15:51.734720+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:15:51.734720+0000","last_undegraded":"2024-09-13T13:15:51.734720+0000","last_fullsized":"2024-09-13T13:15:51.734720+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.633433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.633433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:56.064702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":708,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.735280+0000","last_change":"2024-09-13T13:15:25.787110+0000","last_active":"2024-09-13T13:15:51.735280+0000","last_peered":"2024-09-13T13:15:51.735280+0000","last_clean":"2024-09-13T13:15:51.735280+0000","last_became_active":"2024-09-13T13:12:53.000873+0000","last_became_peered":"2024-09-13T13:12:53.000873+0000","last_unstale":"2024-09-13T13:15:51.735280+0000","last_undegraded":"2024-09-13T13:15:51.735280+0000","last_fullsized":"2024-09-13T13:15:51.735280+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.787029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.787029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:31.829669+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":683,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.747268+0000","last_change":"2024-09-13T13:13:12.952427+0000","last_active":"2024-09-13T13:15:51.747268+0000","last_peered":"2024-09-13T13:15:51.747268+0000","last_clean":"2024-09-13T13:15:51.747268+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:15:51.747268+0000","last_undegraded":"2024-09-13T13:15:51.747268+0000","last_fullsized":"2024-09-13T13:15:51.747268+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:12.952348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:12.952348+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:37.576218+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":151,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.728012+0000","last_change":"2024-09-13T13:13:51.764793+0000","last_active":"2024-09-13T13:15:51.728012+0000","last_peered":"2024-09-13T13:15:51.728012+0000","last_clean":"2024-09-13T13:15:51.728012+0000","last_became_active":"2024-09-13T13:12:53.006579+0000","last_became_peered":"2024-09-13T13:12:53.006579+0000","last_unstale":"2024-09-13T13:15:51.728012+0000","last_undegraded":"2024-09-13T13:15:51.728012+0000","last_fullsized":"2024-09-13T13:15:51.728012+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.764717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.764717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:11.315764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":671,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.738893+0000","last_change":"2024-09-13T13:15:25.281502+0000","last_active":"2024-09-13T13:15:51.738893+0000","last_peered":"2024-09-13T13:15:51.738893+0000","last_clean":"2024-09-13T13:15:51.738893+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:15:51.738893+0000","last_undegraded":"2024-09-13T13:15:51.738893+0000","last_fullsized":"2024-09-13T13:15:51.738893+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.281295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.281295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.642193+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":706,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.740607+0000","last_change":"2024-09-13T13:14:15.616158+0000","last_active":"2024-09-13T13:15:51.740607+0000","last_peered":"2024-09-13T13:15:51.740607+0000","last_clean":"2024-09-13T13:15:51.740607+0000","last_became_active":"2024-09-13T13:12:52.950504+0000","last_became_peered":"2024-09-13T13:12:52.950504+0000","last_unstale":"2024-09-13T13:15:51.740607+0000","last_undegraded":"2024-09-13T13:15:51.740607+0000","last_fullsized":"2024-09-13T13:15:51.740607+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:15.616112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:15.616112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.725422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":696,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734307+0000","last_change":"2024-09-13T13:12:34.682535+0000","last_active":"2024-09-13T13:15:51.734307+0000","last_peered":"2024-09-13T13:15:51.734307+0000","last_clean":"2024-09-13T13:15:51.734307+0000","last_became_active":"2024-09-13T13:12:34.665746+0000","last_became_peered":"2024-09-13T13:12:34.665746+0000","last_unstale":"2024-09-13T13:15:51.734307+0000","last_undegraded":"2024-09-13T13:15:51.734307+0000","last_fullsized":"2024-09-13T13:15:51.734307+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:19.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:19.931466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:41.387724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":662,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.745690+0000","last_change":"2024-09-13T13:13:05.981881+0000","last_active":"2024-09-13T13:15:51.745690+0000","last_peered":"2024-09-13T13:15:51.745690+0000","last_clean":"2024-09-13T13:15:51.745690+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:15:51.745690+0000","last_undegraded":"2024-09-13T13:15:51.745690+0000","last_fullsized":"2024-09-13T13:15:51.745690+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:05.981803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:05.981803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.697737+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":842,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.738346+0000","last_change":"2024-09-13T13:13:10.130755+0000","last_active":"2024-09-13T13:15:51.738346+0000","last_peered":"2024-09-13T13:15:51.738346+0000","last_clean":"2024-09-13T13:15:51.738346+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:15:51.738346+0000","last_undegraded":"2024-09-13T13:15:51.738346+0000","last_fullsized":"2024-09-13T13:15:51.738346+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:13:10.130680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.130680+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.267335+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":151,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.728733+0000","last_change":"2024-09-13T13:13:11.956837+0000","last_active":"2024-09-13T13:15:51.728733+0000","last_peered":"2024-09-13T13:15:51.728733+0000","last_clean":"2024-09-13T13:15:51.728733+0000","last_became_active":"2024-09-13T13:12:52.946339+0000","last_became_peered":"2024-09-13T13:12:52.946339+0000","last_unstale":"2024-09-13T13:15:51.728733+0000","last_undegraded":"2024-09-13T13:15:51.728733+0000","last_fullsized":"2024-09-13T13:15:51.728733+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.956784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.956784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:44.263343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":713,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.738574+0000","last_change":"2024-09-13T13:15:34.280338+0000","last_active":"2024-09-13T13:15:51.738574+0000","last_peered":"2024-09-13T13:15:51.738574+0000","last_clean":"2024-09-13T13:15:51.738574+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:15:51.738574+0000","last_undegraded":"2024-09-13T13:15:51.738574+0000","last_fullsized":"2024-09-13T13:15:51.738574+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:15:34.280253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:34.280253+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.301730+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":677,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734889+0000","last_change":"2024-09-13T13:13:57.105151+0000","last_active":"2024-09-13T13:15:51.734889+0000","last_peered":"2024-09-13T13:15:51.734889+0000","last_clean":"2024-09-13T13:15:51.734889+0000","last_became_active":"2024-09-13T13:12:52.999912+0000","last_became_peered":"2024-09-13T13:12:52.999912+0000","last_unstale":"2024-09-13T13:15:51.734889+0000","last_undegraded":"2024-09-13T13:15:51.734889+0000","last_fullsized":"2024-09-13T13:15:51.734889+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.105098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.105098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.715266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":866,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734144+0000","last_change":"2024-09-13T13:15:33.333062+0000","last_active":"2024-09-13T13:15:51.734144+0000","last_peered":"2024-09-13T13:15:51.734144+0000","last_clean":"2024-09-13T13:15:51.734144+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:15:51.734144+0000","last_undegraded":"2024-09-13T13:15:51.734144+0000","last_fullsized":"2024-09-13T13:15:51.734144+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:15:33.332981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.332981+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.941448+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":674,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.743721+0000","last_change":"2024-09-13T13:15:22.660262+0000","last_active":"2024-09-13T13:15:51.743721+0000","last_peered":"2024-09-13T13:15:51.743721+0000","last_clean":"2024-09-13T13:15:51.743721+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:15:51.743721+0000","last_undegraded":"2024-09-13T13:15:51.743721+0000","last_fullsized":"2024-09-13T13:15:51.743721+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:22.660050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:22.660050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:44.066070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":163,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.728151+0000","last_change":"2024-09-13T13:13:47.728092+0000","last_active":"2024-09-13T13:15:51.728151+0000","last_peered":"2024-09-13T13:15:51.728151+0000","last_clean":"2024-09-13T13:15:51.728151+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:15:51.728151+0000","last_undegraded":"2024-09-13T13:15:51.728151+0000","last_fullsized":"2024-09-13T13:15:51.728151+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:47.728010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:47.728010+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.358350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":143,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.727996+0000","last_change":"2024-09-13T13:12:53.006302+0000","last_active":"2024-09-13T13:15:51.727996+0000","last_peered":"2024-09-13T13:15:51.727996+0000","last_clean":"2024-09-13T13:15:51.727996+0000","last_became_active":"2024-09-13T13:12:53.005988+0000","last_became_peered":"2024-09-13T13:12:53.005988+0000","last_unstale":"2024-09-13T13:15:51.727996+0000","last_undegraded":"2024-09-13T13:15:51.727996+0000","last_fullsized":"2024-09-13T13:15:51.727996+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:40.888083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.888083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.372875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":672,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.736015+0000","last_change":"2024-09-13T13:14:50.185025+0000","last_active":"2024-09-13T13:15:51.736015+0000","last_peered":"2024-09-13T13:15:51.736015+0000","last_clean":"2024-09-13T13:15:51.736015+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:15:51.736015+0000","last_undegraded":"2024-09-13T13:15:51.736015+0000","last_fullsized":"2024-09-13T13:15:51.736015+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.184570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.184570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:56.711009+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":675,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734844+0000","last_change":"2024-09-13T13:15:51.374176+0000","last_active":"2024-09-13T13:15:51.734844+0000","last_peered":"2024-09-13T13:15:51.734844+0000","last_clean":"2024-09-13T13:15:51.734844+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:15:51.734844+0000","last_undegraded":"2024-09-13T13:15:51.734844+0000","last_fullsized":"2024-09-13T13:15:51.734844+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.374109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.374109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:54.143731+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":635,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.733024+0000","last_change":"2024-09-13T13:12:39.874394+0000","last_active":"2024-09-13T13:15:51.733024+0000","last_peered":"2024-09-13T13:15:51.733024+0000","last_clean":"2024-09-13T13:15:51.733024+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:15:51.733024+0000","last_undegraded":"2024-09-13T13:15:51.733024+0000","last_fullsized":"2024-09-13T13:15:51.733024+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:39.874324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:39.874324+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.370889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":663,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.739162+0000","last_change":"2024-09-13T13:14:13.418389+0000","last_active":"2024-09-13T13:15:51.739162+0000","last_peered":"2024-09-13T13:15:51.739162+0000","last_clean":"2024-09-13T13:15:51.739162+0000","last_became_active":"2024-09-13T13:12:52.949094+0000","last_became_peered":"2024-09-13T13:12:52.949094+0000","last_unstale":"2024-09-13T13:15:51.739162+0000","last_undegraded":"2024-09-13T13:15:51.739162+0000","last_fullsized":"2024-09-13T13:15:51.739162+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:13.418196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:13.418196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.331557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":671,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.745296+0000","last_change":"2024-09-13T13:15:01.269060+0000","last_active":"2024-09-13T13:15:51.745296+0000","last_peered":"2024-09-13T13:15:51.745296+0000","last_clean":"2024-09-13T13:15:51.745296+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:15:51.745296+0000","last_undegraded":"2024-09-13T13:15:51.745296+0000","last_fullsized":"2024-09-13T13:15:51.745296+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:01.268884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:01.268884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.329102+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":658,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.743275+0000","last_change":"2024-09-13T13:10:39.477882+0000","last_active":"2024-09-13T13:15:51.743275+0000","last_peered":"2024-09-13T13:15:51.743275+0000","last_clean":"2024-09-13T13:15:51.743275+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:15:51.743275+0000","last_undegraded":"2024-09-13T13:15:51.743275+0000","last_fullsized":"2024-09-13T13:15:51.743275+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:39.477683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.477683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.622029+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":693,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.737121+0000","last_change":"2024-09-13T13:13:49.268080+0000","last_active":"2024-09-13T13:15:51.737121+0000","last_peered":"2024-09-13T13:15:51.737121+0000","last_clean":"2024-09-13T13:15:51.737121+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:15:51.737121+0000","last_undegraded":"2024-09-13T13:15:51.737121+0000","last_fullsized":"2024-09-13T13:15:51.737121+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:49.268018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:49.268018+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.301185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":687,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734958+0000","last_change":"2024-09-13T13:14:50.392659+0000","last_active":"2024-09-13T13:15:51.734958+0000","last_peered":"2024-09-13T13:15:51.734958+0000","last_clean":"2024-09-13T13:15:51.734958+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:15:51.734958+0000","last_undegraded":"2024-09-13T13:15:51.734958+0000","last_fullsized":"2024-09-13T13:15:51.734958+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.392444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.392444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.481736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":696,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.736273+0000","last_change":"2024-09-13T13:15:03.778214+0000","last_active":"2024-09-13T13:15:51.736273+0000","last_peered":"2024-09-13T13:15:51.736273+0000","last_clean":"2024-09-13T13:15:51.736273+0000","last_became_active":"2024-09-13T13:12:52.999748+0000","last_became_peered":"2024-09-13T13:12:52.999748+0000","last_unstale":"2024-09-13T13:15:51.736273+0000","last_undegraded":"2024-09-13T13:15:51.736273+0000","last_fullsized":"2024-09-13T13:15:51.736273+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.778170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.778170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:20.178488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":654,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.745797+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:15:51.745797+0000","last_peered":"2024-09-13T13:15:51.745797+0000","last_clean":"2024-09-13T13:15:51.745797+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:15:51.745797+0000","last_undegraded":"2024-09-13T13:15:51.745797+0000","last_fullsized":"2024-09-13T13:15:51.745797+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.231726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":703,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734359+0000","last_change":"2024-09-13T13:14:12.368507+0000","last_active":"2024-09-13T13:15:51.734359+0000","last_peered":"2024-09-13T13:15:51.734359+0000","last_clean":"2024-09-13T13:15:51.734359+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:15:51.734359+0000","last_undegraded":"2024-09-13T13:15:51.734359+0000","last_fullsized":"2024-09-13T13:15:51.734359+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:14:12.368430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:12.368430+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.308538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":686,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.743165+0000","last_change":"2024-09-13T13:13:44.750043+0000","last_active":"2024-09-13T13:15:51.743165+0000","last_peered":"2024-09-13T13:15:51.743165+0000","last_clean":"2024-09-13T13:15:51.743165+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:15:51.743165+0000","last_undegraded":"2024-09-13T13:15:51.743165+0000","last_fullsized":"2024-09-13T13:15:51.743165+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:44.749961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:44.749961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.672770+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":600,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717516+0000","last_change":"2024-09-13T13:14:16.457160+0000","last_active":"2024-09-13T13:15:50.717516+0000","last_peered":"2024-09-13T13:15:50.717516+0000","last_clean":"2024-09-13T13:15:50.717516+0000","last_became_active":"2024-09-13T13:12:53.001131+0000","last_became_peered":"2024-09-13T13:12:53.001131+0000","last_unstale":"2024-09-13T13:15:50.717516+0000","last_undegraded":"2024-09-13T13:15:50.717516+0000","last_fullsized":"2024-09-13T13:15:50.717516+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:16.456784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:16.456784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:43.279313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":651,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.739442+0000","last_change":"2024-09-13T13:12:52.947380+0000","last_active":"2024-09-13T13:15:51.739442+0000","last_peered":"2024-09-13T13:15:51.739442+0000","last_clean":"2024-09-13T13:15:51.739442+0000","last_became_active":"2024-09-13T13:12:52.946611+0000","last_became_peered":"2024-09-13T13:12:52.946611+0000","last_unstale":"2024-09-13T13:15:51.739442+0000","last_undegraded":"2024-09-13T13:15:51.739442+0000","last_fullsized":"2024-09-13T13:15:51.739442+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:08.704450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:08.704450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.645492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":681,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.735540+0000","last_change":"2024-09-13T13:14:18.882969+0000","last_active":"2024-09-13T13:15:51.735540+0000","last_peered":"2024-09-13T13:15:51.735540+0000","last_clean":"2024-09-13T13:15:51.735540+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:15:51.735540+0000","last_undegraded":"2024-09-13T13:15:51.735540+0000","last_fullsized":"2024-09-13T13:15:51.735540+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:18.882903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:18.882903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:46.394564+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":663,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734591+0000","last_change":"2024-09-13T13:11:07.564094+0000","last_active":"2024-09-13T13:15:51.734591+0000","last_peered":"2024-09-13T13:15:51.734591+0000","last_clean":"2024-09-13T13:15:51.734591+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:15:51.734591+0000","last_undegraded":"2024-09-13T13:15:51.734591+0000","last_fullsized":"2024-09-13T13:15:51.734591+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:07.564025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:07.564025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:35.266907+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":668,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-13T13:15:50.717300+0000","last_change":"2024-09-13T13:11:49.264206+0000","last_active":"2024-09-13T13:15:50.717300+0000","last_peered":"2024-09-13T13:15:50.717300+0000","last_clean":"2024-09-13T13:15:50.717300+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:15:50.717300+0000","last_undegraded":"2024-09-13T13:15:50.717300+0000","last_fullsized":"2024-09-13T13:15:50.717300+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:49.264128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.264128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.230570+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":687,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.743857+0000","last_change":"2024-09-13T13:15:27.678822+0000","last_active":"2024-09-13T13:15:51.743857+0000","last_peered":"2024-09-13T13:15:51.743857+0000","last_clean":"2024-09-13T13:15:51.743857+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:15:51.743857+0000","last_undegraded":"2024-09-13T13:15:51.743857+0000","last_fullsized":"2024-09-13T13:15:51.743857+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:27.678752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:27.678752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:48.252436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":7007502360,"num_objects":106927,"num_object_clones":0,"num_object_copies":320781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106927,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147829,"num_write_kb":8152129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20875915264,"data_stored":20875903048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90829,"ondisk_log_size":90829,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":2508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":200,"num_read_kb":195,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201461,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2221112,"kb_used_data":1706784,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":91524040,"statfs":{"total":95995035648,"available":93720616960,"internally_reserved":0,"allocated":1747746816,"data_stored":1747373267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":526645706},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141071,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3514272,"kb_used_data":3079612,"kb_used_omap":4,"kb_used_meta":434491,"kb_avail":90230880,"statfs":{"total":95995035648,"available":92396421120,"internally_reserved":0,"allocated":3153522688,"data_stored":3153170512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":444919469},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692033,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2663844,"kb_used_data":2127316,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91081308,"statfs":{"total":95995035648,"available":93267259392,"internally_reserved":0,"allocated":2178371584,"data_stored":2178008619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920967,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3829568,"kb_used_data":3419244,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":89915584,"statfs":{"total":95995035648,"available":92073558016,"internally_reserved":0,"allocated":3501305856,"data_stored":3500918475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":420018630},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117198,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2727996,"kb_used_data":2110952,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":91017156,"statfs":{"total":95995035648,"available":93201567744,"internally_reserved":0,"allocated":2161614848,"data_stored":2161232259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":631830983},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411537,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2983836,"kb_used_data":2569480,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":90761316,"statfs":{"total":95995035648,"available":92939587584,"internally_reserved":0,"allocated":2631147520,"data_stored":2630796293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":424278470},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1827340,"kb_used_data":1290812,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91917812,"statfs":{"total":95995035648,"available":94123839488,"internally_reserved":0,"allocated":1321791488,"data_stored":1321388210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738591,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4901560,"kb_used_data":4242664,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":88843592,"statfs":{"total":95995035648,"available":90975838208,"internally_reserved":0,"allocated":4344487936,"data_stored":4344105928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4297265152,"data_stored":4297261080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1310916608,"data_stored":1310916608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2607677440,"data_stored":2607677440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2138898432,"data_stored":2138898432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3500281856,"data_stored":3500277784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2159804416,"data_stored":2159804416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3122925568,"data_stored":3122921496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1738145792,"data_stored":1738145792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:15:58.808 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T13:15:58.809 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:15:58.809 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:15:58.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:58 smithi167 ceph-mon[33178]: 5.12 scrub starts 2024-09-13T13:15:58.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:58 smithi167 ceph-mon[33178]: 5.12 scrub ok 2024-09-13T13:15:58.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:58 smithi167 ceph-mon[33178]: 6.7 scrub starts 2024-09-13T13:15:58.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:58 smithi167 ceph-mon[33178]: 6.7 scrub ok 2024-09-13T13:15:58.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:58 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2674375188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:58.838 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:15:58.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:58.891 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:58.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:15:58.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:59.025 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:59.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:59.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:59.127 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:15:59.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:58 smithi052 ceph-mon[29471]: 5.12 scrub starts 2024-09-13T13:15:59.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:58 smithi052 ceph-mon[29471]: 5.12 scrub ok 2024-09-13T13:15:59.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:58 smithi052 ceph-mon[29471]: 6.7 scrub starts 2024-09-13T13:15:59.132 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:58 smithi052 ceph-mon[29471]: 6.7 scrub ok 2024-09-13T13:15:59.132 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:58 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2674375188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:59.132 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:58 smithi052 ceph-mon[36682]: 5.12 scrub starts 2024-09-13T13:15:59.132 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:58 smithi052 ceph-mon[36682]: 5.12 scrub ok 2024-09-13T13:15:59.132 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:58 smithi052 ceph-mon[36682]: 6.7 scrub starts 2024-09-13T13:15:59.132 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:58 smithi052 ceph-mon[36682]: 6.7 scrub ok 2024-09-13T13:15:59.132 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:58 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2674375188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:15:59.170 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T13:15:59.172 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:59.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:59.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:15:59.274 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:59.320 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:59.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:59.422 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:15:59.456 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:59.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:59.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:15:59.559 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:15:59.597 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:15:59.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:59.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:15:59.600 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:15:59.698 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:15:59.740 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:59.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:59.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:15:59.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:15:59.873 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:15:59.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:15:59.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:15:59.969 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:15:59.975 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:00.022 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:00.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:00.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:15:59 smithi167 ceph-mon[33178]: pgmap v736: 148 pgs: 148 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 708 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-13T13:16:00.113 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:15:59 smithi052 ceph-mon[36682]: pgmap v736: 148 pgs: 148 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 708 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-13T13:16:00.113 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:15:59 smithi052 ceph-mon[29471]: pgmap v736: 148 pgs: 148 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 708 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-13T13:16:00.124 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:00.159 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:00.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:00.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:00.231 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:16:00.231 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:16:00.261 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:00.299 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:00.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:00.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:00.401 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:00.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:00.438 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:00.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:00.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:00.572 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:00.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:00.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:00.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:00.706 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:00.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:00.807 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:00.851 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:00.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:00.949 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:16:00.952 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:16:00.952 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:00.983 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:00.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:00.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:01.049 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:00 smithi167 ceph-mon[33178]: 5.6 scrub starts 2024-09-13T13:16:01.049 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:00 smithi167 ceph-mon[33178]: 5.6 scrub ok 2024-09-13T13:16:01.049 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:00 smithi167 ceph-mon[33178]: 4.9 scrub starts 2024-09-13T13:16:01.049 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:00 smithi167 ceph-mon[33178]: 4.9 scrub ok 2024-09-13T13:16:01.085 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:01.097 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T13:16:01.122 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:01.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:01.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:01.158 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:00 smithi052 ceph-mon[36682]: 5.6 scrub starts 2024-09-13T13:16:01.158 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:00 smithi052 ceph-mon[36682]: 5.6 scrub ok 2024-09-13T13:16:01.158 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:00 smithi052 ceph-mon[36682]: 4.9 scrub starts 2024-09-13T13:16:01.158 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:00 smithi052 ceph-mon[36682]: 4.9 scrub ok 2024-09-13T13:16:01.159 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:00 smithi052 ceph-mon[29471]: 5.6 scrub starts 2024-09-13T13:16:01.159 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:00 smithi052 ceph-mon[29471]: 5.6 scrub ok 2024-09-13T13:16:01.159 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:00 smithi052 ceph-mon[29471]: 4.9 scrub starts 2024-09-13T13:16:01.159 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:00 smithi052 ceph-mon[29471]: 4.9 scrub ok 2024-09-13T13:16:01.225 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:01.251 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:01.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:01.353 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:01.406 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:01.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:01.487 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:16:01.507 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:01.539 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:01.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:01.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:01.640 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:01.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:01.689 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:01.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:01.744 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":737,"stamp":"2024-09-13T13:15:59.934804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6665673267,"num_objects":101901,"num_object_clones":0,"num_object_copies":305703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101901,"num_whiteouts":0,"num_read":1447,"num_read_kb":1306,"num_write":153889,"num_write_kb":8153802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":594037,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91711,"ondisk_log_size":91711,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":23713452,"kb_used_data":19562140,"kb_used_omap":15,"kb_used_meta":4151152,"kb_avail":726247764,"statfs":{"total":767960285184,"available":743677710336,"internally_reserved":0,"allocated":20031631360,"data_stored":20028587879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":4250780441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,5,8,7,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-572391424,"num_objects":-8734,"num_object_clones":0,"num_object_copies":-26202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8734,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8734,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.202398"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":665,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.803661+0000","last_change":"2024-09-13T13:10:32.368663+0000","last_active":"2024-09-13T13:15:56.803661+0000","last_peered":"2024-09-13T13:15:56.803661+0000","last_clean":"2024-09-13T13:15:56.803661+0000","last_became_active":"2024-09-13T13:01:02.182766+0000","last_became_peered":"2024-09-13T13:01:02.182766+0000","last_unstale":"2024-09-13T13:15:56.803661+0000","last_undegraded":"2024-09-13T13:15:56.803661+0000","last_fullsized":"2024-09-13T13:15:56.803661+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.368588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.368588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.615999+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":163,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800529+0000","last_change":"2024-09-13T13:15:04.344174+0000","last_active":"2024-09-13T13:15:56.800529+0000","last_peered":"2024-09-13T13:15:56.800529+0000","last_clean":"2024-09-13T13:15:56.800529+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:15:56.800529+0000","last_undegraded":"2024-09-13T13:15:56.800529+0000","last_fullsized":"2024-09-13T13:15:56.800529+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:15:04.344092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.344092+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.313962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":666,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.795323+0000","last_change":"2024-09-13T13:14:12.865175+0000","last_active":"2024-09-13T13:15:56.795323+0000","last_peered":"2024-09-13T13:15:56.795323+0000","last_clean":"2024-09-13T13:15:56.795323+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:15:56.795323+0000","last_undegraded":"2024-09-13T13:15:56.795323+0000","last_fullsized":"2024-09-13T13:15:56.795323+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:12.865101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:12.865101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:17.268032+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":161,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801537+0000","last_change":"2024-09-13T13:15:51.244938+0000","last_active":"2024-09-13T13:15:56.801537+0000","last_peered":"2024-09-13T13:15:56.801537+0000","last_clean":"2024-09-13T13:15:56.801537+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:15:56.801537+0000","last_undegraded":"2024-09-13T13:15:56.801537+0000","last_fullsized":"2024-09-13T13:15:56.801537+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:19.110420+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":671,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800551+0000","last_change":"2024-09-13T13:11:47.852762+0000","last_active":"2024-09-13T13:15:56.800551+0000","last_peered":"2024-09-13T13:15:56.800551+0000","last_clean":"2024-09-13T13:15:56.800551+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:15:56.800551+0000","last_undegraded":"2024-09-13T13:15:56.800551+0000","last_fullsized":"2024-09-13T13:15:56.800551+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:15.241902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":693,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800483+0000","last_change":"2024-09-13T13:15:24.253436+0000","last_active":"2024-09-13T13:15:56.800483+0000","last_peered":"2024-09-13T13:15:56.800483+0000","last_clean":"2024-09-13T13:15:56.800483+0000","last_became_active":"2024-09-13T13:04:02.756964+0000","last_became_peered":"2024-09-13T13:04:02.756964+0000","last_unstale":"2024-09-13T13:15:56.800483+0000","last_undegraded":"2024-09-13T13:15:56.800483+0000","last_fullsized":"2024-09-13T13:15:56.800483+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:24.253369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:24.253369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:50.007360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":736,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.799489+0000","last_change":"2024-09-13T13:15:50.330680+0000","last_active":"2024-09-13T13:15:56.799489+0000","last_peered":"2024-09-13T13:15:56.799489+0000","last_clean":"2024-09-13T13:15:56.799489+0000","last_became_active":"2024-09-13T13:01:04.187229+0000","last_became_peered":"2024-09-13T13:01:04.187229+0000","last_unstale":"2024-09-13T13:15:56.799489+0000","last_undegraded":"2024-09-13T13:15:56.799489+0000","last_fullsized":"2024-09-13T13:15:56.799489+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:15:50.330607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.330607+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:59.485090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":145,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800445+0000","last_change":"2024-09-13T13:12:52.997865+0000","last_active":"2024-09-13T13:15:56.800445+0000","last_peered":"2024-09-13T13:15:56.800445+0000","last_clean":"2024-09-13T13:15:56.800445+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:15:56.800445+0000","last_undegraded":"2024-09-13T13:15:56.800445+0000","last_fullsized":"2024-09-13T13:15:56.800445+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:17.305608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:17.305608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.322592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":735,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796892+0000","last_change":"2024-09-13T13:13:41.518378+0000","last_active":"2024-09-13T13:15:56.796892+0000","last_peered":"2024-09-13T13:15:56.796892+0000","last_clean":"2024-09-13T13:15:56.796892+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:15:56.796892+0000","last_undegraded":"2024-09-13T13:15:56.796892+0000","last_fullsized":"2024-09-13T13:15:56.796892+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:41.518322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.518322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:12.339477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":155,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801543+0000","last_change":"2024-09-13T13:12:53.033861+0000","last_active":"2024-09-13T13:15:56.801543+0000","last_peered":"2024-09-13T13:15:56.801543+0000","last_clean":"2024-09-13T13:15:56.801543+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:15:56.801543+0000","last_undegraded":"2024-09-13T13:15:56.801543+0000","last_fullsized":"2024-09-13T13:15:56.801543+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:31.738685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.738685+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.301470+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":672,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.797492+0000","last_change":"2024-09-13T13:13:43.501368+0000","last_active":"2024-09-13T13:15:56.797492+0000","last_peered":"2024-09-13T13:15:56.797492+0000","last_clean":"2024-09-13T13:15:56.797492+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:15:56.797492+0000","last_undegraded":"2024-09-13T13:15:56.797492+0000","last_fullsized":"2024-09-13T13:15:56.797492+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:43.501206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:43.501206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.419736+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":676,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.803362+0000","last_change":"2024-09-13T13:15:25.628234+0000","last_active":"2024-09-13T13:15:56.803362+0000","last_peered":"2024-09-13T13:15:56.803362+0000","last_clean":"2024-09-13T13:15:56.803362+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:15:56.803362+0000","last_undegraded":"2024-09-13T13:15:56.803362+0000","last_fullsized":"2024-09-13T13:15:56.803362+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.628156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.628156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:41.988194+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":702,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796213+0000","last_change":"2024-09-13T13:12:52.951531+0000","last_active":"2024-09-13T13:15:56.796213+0000","last_peered":"2024-09-13T13:15:56.796213+0000","last_clean":"2024-09-13T13:15:56.796213+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:15:56.796213+0000","last_undegraded":"2024-09-13T13:15:56.796213+0000","last_fullsized":"2024-09-13T13:15:56.796213+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:16.989972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:16.989972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.371147+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":159,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801693+0000","last_change":"2024-09-13T13:15:03.348187+0000","last_active":"2024-09-13T13:15:56.801693+0000","last_peered":"2024-09-13T13:15:56.801693+0000","last_clean":"2024-09-13T13:15:56.801693+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:15:56.801693+0000","last_undegraded":"2024-09-13T13:15:56.801693+0000","last_fullsized":"2024-09-13T13:15:56.801693+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:15:03.348115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.348115+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.237423+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":693,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800489+0000","last_change":"2024-09-13T13:13:54.122282+0000","last_active":"2024-09-13T13:15:56.800489+0000","last_peered":"2024-09-13T13:15:56.800489+0000","last_clean":"2024-09-13T13:15:56.800489+0000","last_became_active":"2024-09-13T13:12:52.999970+0000","last_became_peered":"2024-09-13T13:12:52.999970+0000","last_unstale":"2024-09-13T13:15:56.800489+0000","last_undegraded":"2024-09-13T13:15:56.800489+0000","last_fullsized":"2024-09-13T13:15:56.800489+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.122222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.122222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:17.725218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":645,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.798884+0000","last_change":"2024-09-13T13:13:02.818644+0000","last_active":"2024-09-13T13:15:56.798884+0000","last_peered":"2024-09-13T13:15:56.798884+0000","last_clean":"2024-09-13T13:15:56.798884+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:15:56.798884+0000","last_undegraded":"2024-09-13T13:15:56.798884+0000","last_fullsized":"2024-09-13T13:15:56.798884+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:02.818543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:02.818543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.513000+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":149,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801365+0000","last_change":"2024-09-13T13:12:53.014177+0000","last_active":"2024-09-13T13:15:56.801365+0000","last_peered":"2024-09-13T13:15:56.801365+0000","last_clean":"2024-09-13T13:15:56.801365+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:15:56.801365+0000","last_undegraded":"2024-09-13T13:15:56.801365+0000","last_fullsized":"2024-09-13T13:15:56.801365+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:28.804735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:28.804735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:42.251197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":701,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.807559+0000","last_change":"2024-09-13T13:15:52.658477+0000","last_active":"2024-09-13T13:15:56.807559+0000","last_peered":"2024-09-13T13:15:56.807559+0000","last_clean":"2024-09-13T13:15:56.807559+0000","last_became_active":"2024-09-13T13:12:53.008505+0000","last_became_peered":"2024-09-13T13:12:53.008505+0000","last_unstale":"2024-09-13T13:15:56.807559+0000","last_undegraded":"2024-09-13T13:15:56.807559+0000","last_fullsized":"2024-09-13T13:15:56.807559+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:52.658410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:52.658410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:05.062147+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":720,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.794710+0000","last_change":"2024-09-13T13:12:40.125646+0000","last_active":"2024-09-13T13:15:56.794710+0000","last_peered":"2024-09-13T13:15:56.794710+0000","last_clean":"2024-09-13T13:15:56.794710+0000","last_became_active":"2024-09-13T13:01:04.191614+0000","last_became_peered":"2024-09-13T13:01:04.191614+0000","last_unstale":"2024-09-13T13:15:56.794710+0000","last_undegraded":"2024-09-13T13:15:56.794710+0000","last_fullsized":"2024-09-13T13:15:56.794710+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:12:40.125545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.125545+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.676016+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":695,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796128+0000","last_change":"2024-09-13T13:14:52.196086+0000","last_active":"2024-09-13T13:15:56.796128+0000","last_peered":"2024-09-13T13:15:56.796128+0000","last_clean":"2024-09-13T13:15:56.796128+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:15:56.796128+0000","last_undegraded":"2024-09-13T13:15:56.796128+0000","last_fullsized":"2024-09-13T13:15:56.796128+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.196041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.502726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":673,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800891+0000","last_change":"2024-09-13T13:13:03.150524+0000","last_active":"2024-09-13T13:15:56.800891+0000","last_peered":"2024-09-13T13:15:56.800891+0000","last_clean":"2024-09-13T13:15:56.800891+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:15:56.800891+0000","last_undegraded":"2024-09-13T13:15:56.800891+0000","last_fullsized":"2024-09-13T13:15:56.800891+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:03.150442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.150442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.295693+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":153,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801141+0000","last_change":"2024-09-13T13:13:39.876364+0000","last_active":"2024-09-13T13:15:56.801141+0000","last_peered":"2024-09-13T13:15:56.801141+0000","last_clean":"2024-09-13T13:15:56.801141+0000","last_became_active":"2024-09-13T13:12:52.995055+0000","last_became_peered":"2024-09-13T13:12:52.995055+0000","last_unstale":"2024-09-13T13:15:56.801141+0000","last_undegraded":"2024-09-13T13:15:56.801141+0000","last_fullsized":"2024-09-13T13:15:56.801141+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:39.876294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:39.876294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":738,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800142+0000","last_change":"2024-09-13T13:13:41.164741+0000","last_active":"2024-09-13T13:15:56.800142+0000","last_peered":"2024-09-13T13:15:56.800142+0000","last_clean":"2024-09-13T13:15:56.800142+0000","last_became_active":"2024-09-13T13:12:52.953819+0000","last_became_peered":"2024-09-13T13:12:52.953819+0000","last_unstale":"2024-09-13T13:15:56.800142+0000","last_undegraded":"2024-09-13T13:15:56.800142+0000","last_fullsized":"2024-09-13T13:15:56.800142+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:13:41.164656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.164656+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:49.289309+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":155,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801989+0000","last_change":"2024-09-13T13:15:29.208631+0000","last_active":"2024-09-13T13:15:56.801989+0000","last_peered":"2024-09-13T13:15:56.801989+0000","last_clean":"2024-09-13T13:15:56.801989+0000","last_became_active":"2024-09-13T13:13:06.646781+0000","last_became_peered":"2024-09-13T13:13:06.646781+0000","last_unstale":"2024-09-13T13:15:56.801989+0000","last_undegraded":"2024-09-13T13:15:56.801989+0000","last_fullsized":"2024-09-13T13:15:56.801989+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:29.208576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:29.208576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:34.495308+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":180,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796030+0000","last_change":"2024-09-13T13:12:42.975335+0000","last_active":"2024-09-13T13:15:56.796030+0000","last_peered":"2024-09-13T13:15:56.796030+0000","last_clean":"2024-09-13T13:15:56.796030+0000","last_became_active":"2024-09-13T13:12:39.954363+0000","last_became_peered":"2024-09-13T13:12:39.954363+0000","last_unstale":"2024-09-13T13:15:56.796030+0000","last_undegraded":"2024-09-13T13:15:56.796030+0000","last_fullsized":"2024-09-13T13:15:56.796030+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:12:42.975266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:42.975266+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.483880+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":697,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.795494+0000","last_change":"2024-09-13T13:15:23.245116+0000","last_active":"2024-09-13T13:15:56.795494+0000","last_peered":"2024-09-13T13:15:56.795494+0000","last_clean":"2024-09-13T13:15:56.795494+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:15:56.795494+0000","last_undegraded":"2024-09-13T13:15:56.795494+0000","last_fullsized":"2024-09-13T13:15:56.795494+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:15:23.245047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.245047+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.758250+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":723,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.806812+0000","last_change":"2024-09-13T13:15:50.693943+0000","last_active":"2024-09-13T13:15:56.806812+0000","last_peered":"2024-09-13T13:15:56.806812+0000","last_clean":"2024-09-13T13:15:56.806812+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:15:56.806812+0000","last_undegraded":"2024-09-13T13:15:56.806812+0000","last_fullsized":"2024-09-13T13:15:56.806812+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:15:50.693753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.693753+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:56.425256+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"306'12","reported_seq":838,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.738229+0000","last_change":"2024-09-13T13:14:21.530680+0000","last_active":"2024-09-13T13:15:51.738229+0000","last_peered":"2024-09-13T13:15:51.738229+0000","last_clean":"2024-09-13T13:15:51.738229+0000","last_became_active":"2024-09-13T13:12:52.947886+0000","last_became_peered":"2024-09-13T13:12:52.947886+0000","last_unstale":"2024-09-13T13:15:51.738229+0000","last_undegraded":"2024-09-13T13:15:51.738229+0000","last_fullsized":"2024-09-13T13:15:51.738229+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'12","last_scrub_stamp":"2024-09-13T13:14:21.530612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:21.530612+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.292277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"372'9605","reported_seq":9692,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:58.157718+0000","last_change":"2024-09-13T13:13:59.149493+0000","last_active":"2024-09-13T13:15:58.157718+0000","last_peered":"2024-09-13T13:15:58.157718+0000","last_clean":"2024-09-13T13:15:58.157718+0000","last_became_active":"2024-09-13T13:13:59.149215+0000","last_became_peered":"2024-09-13T13:13:59.149215+0000","last_unstale":"2024-09-13T13:15:58.157718+0000","last_undegraded":"2024-09-13T13:15:58.157718+0000","last_fullsized":"2024-09-13T13:15:58.157718+0000","mapping_epoch":334,"log_start":"353'5000","ondisk_log_start":"353'5000","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4605,"log_dups_size":0,"ondisk_log_size":4605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417398784,"num_objects":6369,"num_object_clones":0,"num_object_copies":19107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9605,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":767,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796376+0000","last_change":"2024-09-13T13:15:02.354689+0000","last_active":"2024-09-13T13:15:56.796376+0000","last_peered":"2024-09-13T13:15:56.796376+0000","last_clean":"2024-09-13T13:15:56.796376+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:15:56.796376+0000","last_undegraded":"2024-09-13T13:15:56.796376+0000","last_fullsized":"2024-09-13T13:15:56.796376+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:15:02.354591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:02.354591+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:22.985661+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":697,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.798648+0000","last_change":"2024-09-13T13:15:29.205122+0000","last_active":"2024-09-13T13:15:56.798648+0000","last_peered":"2024-09-13T13:15:56.798648+0000","last_clean":"2024-09-13T13:15:56.798648+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:15:56.798648+0000","last_undegraded":"2024-09-13T13:15:56.798648+0000","last_fullsized":"2024-09-13T13:15:56.798648+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:29.205058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:29.205058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.374619+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":195,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801501+0000","last_change":"2024-09-13T13:15:32.212185+0000","last_active":"2024-09-13T13:15:56.801501+0000","last_peered":"2024-09-13T13:15:56.801501+0000","last_clean":"2024-09-13T13:15:56.801501+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:15:56.801501+0000","last_undegraded":"2024-09-13T13:15:56.801501+0000","last_fullsized":"2024-09-13T13:15:56.801501+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:15:32.212101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.212101+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:33.253768+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":677,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796461+0000","last_change":"2024-09-13T13:13:57.546725+0000","last_active":"2024-09-13T13:15:56.796461+0000","last_peered":"2024-09-13T13:15:56.796461+0000","last_clean":"2024-09-13T13:15:56.796461+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:15:56.796461+0000","last_undegraded":"2024-09-13T13:15:56.796461+0000","last_fullsized":"2024-09-13T13:15:56.796461+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.546671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.546671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:18.300458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"372'9599","reported_seq":9686,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:58.169573+0000","last_change":"2024-09-13T13:13:58.962128+0000","last_active":"2024-09-13T13:15:58.169573+0000","last_peered":"2024-09-13T13:15:58.169573+0000","last_clean":"2024-09-13T13:15:58.169573+0000","last_became_active":"2024-09-13T13:13:58.961386+0000","last_became_peered":"2024-09-13T13:13:58.961386+0000","last_unstale":"2024-09-13T13:15:58.169573+0000","last_undegraded":"2024-09-13T13:15:58.169573+0000","last_fullsized":"2024-09-13T13:15:58.169573+0000","mapping_epoch":334,"log_start":"353'5000","ondisk_log_start":"353'5000","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4599,"log_dups_size":0,"ondisk_log_size":4599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.366364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418578432,"num_objects":6387,"num_object_clones":0,"num_object_copies":19161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9599,"num_write_kb":511552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":704,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.798328+0000","last_change":"2024-09-13T13:13:50.232646+0000","last_active":"2024-09-13T13:15:56.798328+0000","last_peered":"2024-09-13T13:15:56.798328+0000","last_clean":"2024-09-13T13:15:56.798328+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:15:56.798328+0000","last_undegraded":"2024-09-13T13:15:56.798328+0000","last_fullsized":"2024-09-13T13:15:56.798328+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:13:50.232588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.232588+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.213810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":671,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801812+0000","last_change":"2024-09-13T13:13:54.313641+0000","last_active":"2024-09-13T13:15:56.801812+0000","last_peered":"2024-09-13T13:15:56.801812+0000","last_clean":"2024-09-13T13:15:56.801812+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:15:56.801812+0000","last_undegraded":"2024-09-13T13:15:56.801812+0000","last_fullsized":"2024-09-13T13:15:56.801812+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:54.313570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.313570+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:18.294176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":153,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.799314+0000","last_change":"2024-09-13T13:15:00.323883+0000","last_active":"2024-09-13T13:15:56.799314+0000","last_peered":"2024-09-13T13:15:56.799314+0000","last_clean":"2024-09-13T13:15:56.799314+0000","last_became_active":"2024-09-13T13:12:53.006199+0000","last_became_peered":"2024-09-13T13:12:53.006199+0000","last_unstale":"2024-09-13T13:15:56.799314+0000","last_undegraded":"2024-09-13T13:15:56.799314+0000","last_fullsized":"2024-09-13T13:15:56.799314+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.323784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.323784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:00.914935+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":161,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801653+0000","last_change":"2024-09-13T13:14:56.333854+0000","last_active":"2024-09-13T13:15:56.801653+0000","last_peered":"2024-09-13T13:15:56.801653+0000","last_clean":"2024-09-13T13:15:56.801653+0000","last_became_active":"2024-09-13T13:12:53.001045+0000","last_became_peered":"2024-09-13T13:12:53.001045+0000","last_unstale":"2024-09-13T13:15:56.801653+0000","last_undegraded":"2024-09-13T13:15:56.801653+0000","last_fullsized":"2024-09-13T13:15:56.801653+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:56.333778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:56.333778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:58.872471+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"371'9432","reported_seq":9517,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.216175+0000","last_change":"2024-09-13T13:13:58.966300+0000","last_active":"2024-09-13T13:15:56.216175+0000","last_peered":"2024-09-13T13:15:56.216175+0000","last_clean":"2024-09-13T13:15:56.216175+0000","last_became_active":"2024-09-13T13:13:58.966015+0000","last_became_peered":"2024-09-13T13:13:58.966015+0000","last_unstale":"2024-09-13T13:15:56.216175+0000","last_undegraded":"2024-09-13T13:15:56.216175+0000","last_fullsized":"2024-09-13T13:15:56.216175+0000","mapping_epoch":334,"log_start":"346'3400","ondisk_log_start":"346'3400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":6032,"log_dups_size":0,"ondisk_log_size":6032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.310606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426115072,"num_objects":6502,"num_object_clones":0,"num_object_copies":19506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9432,"num_write_kb":509888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":169,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.799534+0000","last_change":"2024-09-13T13:13:50.716312+0000","last_active":"2024-09-13T13:15:56.799534+0000","last_peered":"2024-09-13T13:15:56.799534+0000","last_clean":"2024-09-13T13:15:56.799534+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:15:56.799534+0000","last_undegraded":"2024-09-13T13:15:56.799534+0000","last_fullsized":"2024-09-13T13:15:56.799534+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:13:50.716243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.716243+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:10.352612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":679,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.807139+0000","last_change":"2024-09-13T13:13:57.886950+0000","last_active":"2024-09-13T13:15:56.807139+0000","last_peered":"2024-09-13T13:15:56.807139+0000","last_clean":"2024-09-13T13:15:56.807139+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:15:56.807139+0000","last_undegraded":"2024-09-13T13:15:56.807139+0000","last_fullsized":"2024-09-13T13:15:56.807139+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:57.886879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.886879+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:12.598128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":877,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.798173+0000","last_change":"2024-09-13T13:13:55.316917+0000","last_active":"2024-09-13T13:15:56.798173+0000","last_peered":"2024-09-13T13:15:56.798173+0000","last_clean":"2024-09-13T13:15:56.798173+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:15:56.798173+0000","last_undegraded":"2024-09-13T13:15:56.798173+0000","last_fullsized":"2024-09-13T13:15:56.798173+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:13:55.316860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.316860+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.251395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":293,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.737797+0000","last_change":"2024-09-13T13:14:59.391483+0000","last_active":"2024-09-13T13:15:51.737797+0000","last_peered":"2024-09-13T13:15:51.737797+0000","last_clean":"2024-09-13T13:15:51.737797+0000","last_became_active":"2024-09-13T13:12:39.948363+0000","last_became_peered":"2024-09-13T13:12:39.948363+0000","last_unstale":"2024-09-13T13:15:51.737797+0000","last_undegraded":"2024-09-13T13:15:51.737797+0000","last_fullsized":"2024-09-13T13:15:51.737797+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.391418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.391418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:15.991030+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"372'9524","reported_seq":9611,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:58.177135+0000","last_change":"2024-09-13T13:13:59.149331+0000","last_active":"2024-09-13T13:15:58.177135+0000","last_peered":"2024-09-13T13:15:58.177135+0000","last_clean":"2024-09-13T13:15:58.177135+0000","last_became_active":"2024-09-13T13:13:59.149046+0000","last_became_peered":"2024-09-13T13:13:59.149046+0000","last_unstale":"2024-09-13T13:15:58.177135+0000","last_undegraded":"2024-09-13T13:15:58.177135+0000","last_fullsized":"2024-09-13T13:15:58.177135+0000","mapping_epoch":334,"log_start":"352'4900","ondisk_log_start":"352'4900","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4624,"log_dups_size":0,"ondisk_log_size":4624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:12.327645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407502848,"num_objects":6218,"num_object_clones":0,"num_object_copies":18654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9524,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":739,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.795640+0000","last_change":"2024-09-13T13:12:53.016127+0000","last_active":"2024-09-13T13:15:56.795640+0000","last_peered":"2024-09-13T13:15:56.795640+0000","last_clean":"2024-09-13T13:15:56.795640+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:15:56.795640+0000","last_undegraded":"2024-09-13T13:15:56.795640+0000","last_fullsized":"2024-09-13T13:15:56.795640+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.376603+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":698,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796979+0000","last_change":"2024-09-13T13:14:27.622657+0000","last_active":"2024-09-13T13:15:56.796979+0000","last_peered":"2024-09-13T13:15:56.796979+0000","last_clean":"2024-09-13T13:15:56.796979+0000","last_became_active":"2024-09-13T13:01:02.181337+0000","last_became_peered":"2024-09-13T13:01:02.181337+0000","last_unstale":"2024-09-13T13:15:56.796979+0000","last_undegraded":"2024-09-13T13:15:56.796979+0000","last_fullsized":"2024-09-13T13:15:56.796979+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:27.622584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:27.622584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":867,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.804174+0000","last_change":"2024-09-13T13:15:51.663010+0000","last_active":"2024-09-13T13:15:56.804174+0000","last_peered":"2024-09-13T13:15:56.804174+0000","last_clean":"2024-09-13T13:15:56.804174+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:15:56.804174+0000","last_undegraded":"2024-09-13T13:15:56.804174+0000","last_fullsized":"2024-09-13T13:15:56.804174+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:15:51.662657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.662657+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:55.986754+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":693,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.803770+0000","last_change":"2024-09-13T13:15:26.667525+0000","last_active":"2024-09-13T13:15:56.803770+0000","last_peered":"2024-09-13T13:15:56.803770+0000","last_clean":"2024-09-13T13:15:56.803770+0000","last_became_active":"2024-09-13T13:01:08.210803+0000","last_became_peered":"2024-09-13T13:01:08.210803+0000","last_unstale":"2024-09-13T13:15:56.803770+0000","last_undegraded":"2024-09-13T13:15:56.803770+0000","last_fullsized":"2024-09-13T13:15:56.803770+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:26.667455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:26.667455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.516125+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"371'9576","reported_seq":9744,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.222706+0000","last_change":"2024-09-13T13:15:25.514356+0000","last_active":"2024-09-13T13:15:56.222706+0000","last_peered":"2024-09-13T13:15:56.222706+0000","last_clean":"2024-09-13T13:15:56.222706+0000","last_became_active":"2024-09-13T13:13:58.965950+0000","last_became_peered":"2024-09-13T13:13:58.965950+0000","last_unstale":"2024-09-13T13:15:56.222706+0000","last_undegraded":"2024-09-13T13:15:56.222706+0000","last_fullsized":"2024-09-13T13:15:56.222706+0000","mapping_epoch":334,"log_start":"346'3500","ondisk_log_start":"346'3500","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'7503","last_scrub_stamp":"2024-09-13T13:15:25.514118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.514118+0000","objects_scrubbed":7440,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.146280+0000","scrub_duration":2270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427622424,"num_objects":6526,"num_object_clones":0,"num_object_copies":19578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9576,"num_write_kb":515201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":736,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.807370+0000","last_change":"2024-09-13T13:13:45.757306+0000","last_active":"2024-09-13T13:15:56.807370+0000","last_peered":"2024-09-13T13:15:56.807370+0000","last_clean":"2024-09-13T13:15:56.807370+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:15:56.807370+0000","last_undegraded":"2024-09-13T13:15:56.807370+0000","last_fullsized":"2024-09-13T13:15:56.807370+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:13:45.751801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:45.751801+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.559426+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":667,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796286+0000","last_change":"2024-09-13T13:14:24.955591+0000","last_active":"2024-09-13T13:15:56.796286+0000","last_peered":"2024-09-13T13:15:56.796286+0000","last_clean":"2024-09-13T13:15:56.796286+0000","last_became_active":"2024-09-13T13:01:02.181815+0000","last_became_peered":"2024-09-13T13:01:02.181815+0000","last_unstale":"2024-09-13T13:15:56.796286+0000","last_undegraded":"2024-09-13T13:15:56.796286+0000","last_fullsized":"2024-09-13T13:15:56.796286+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:24.955516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:24.955516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:45.361513+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":640,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734044+0000","last_change":"2024-09-13T13:14:50.408656+0000","last_active":"2024-09-13T13:15:51.734044+0000","last_peered":"2024-09-13T13:15:51.734044+0000","last_clean":"2024-09-13T13:15:51.734044+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:15:51.734044+0000","last_undegraded":"2024-09-13T13:15:51.734044+0000","last_fullsized":"2024-09-13T13:15:51.734044+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.408378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.408378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.441158+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.792580+0000","last_change":"2024-09-13T13:13:50.772738+0000","last_active":"2024-09-13T13:15:56.792580+0000","last_peered":"2024-09-13T13:15:56.792580+0000","last_clean":"2024-09-13T13:15:56.792580+0000","last_became_active":"2024-09-13T13:04:02.749007+0000","last_became_peered":"2024-09-13T13:04:02.749007+0000","last_unstale":"2024-09-13T13:15:56.792580+0000","last_undegraded":"2024-09-13T13:15:56.792580+0000","last_fullsized":"2024-09-13T13:15:56.792580+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:50.772689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.772689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.211300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"371'9459","reported_seq":9544,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.191322+0000","last_change":"2024-09-13T13:13:58.966631+0000","last_active":"2024-09-13T13:15:56.191322+0000","last_peered":"2024-09-13T13:15:56.191322+0000","last_clean":"2024-09-13T13:15:56.191322+0000","last_became_active":"2024-09-13T13:13:58.965789+0000","last_became_peered":"2024-09-13T13:13:58.965789+0000","last_unstale":"2024-09-13T13:15:56.191322+0000","last_undegraded":"2024-09-13T13:15:56.191322+0000","last_fullsized":"2024-09-13T13:15:56.191322+0000","mapping_epoch":334,"log_start":"346'3400","ondisk_log_start":"346'3400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":6059,"log_dups_size":0,"ondisk_log_size":6059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.272671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422379520,"num_objects":6445,"num_object_clones":0,"num_object_copies":19335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9459,"num_write_kb":508928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"268'40","reported_seq":925,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796192+0000","last_change":"2024-09-13T13:15:04.288539+0000","last_active":"2024-09-13T13:15:56.796192+0000","last_peered":"2024-09-13T13:15:56.796192+0000","last_clean":"2024-09-13T13:15:56.796192+0000","last_became_active":"2024-09-13T13:01:04.197141+0000","last_became_peered":"2024-09-13T13:01:04.197141+0000","last_unstale":"2024-09-13T13:15:56.796192+0000","last_undegraded":"2024-09-13T13:15:56.796192+0000","last_fullsized":"2024-09-13T13:15:56.796192+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'40","last_scrub_stamp":"2024-09-13T13:15:04.288431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.288431+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.738711+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":161,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801720+0000","last_change":"2024-09-13T13:14:14.899463+0000","last_active":"2024-09-13T13:15:56.801720+0000","last_peered":"2024-09-13T13:15:56.801720+0000","last_clean":"2024-09-13T13:15:56.801720+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:15:56.801720+0000","last_undegraded":"2024-09-13T13:15:56.801720+0000","last_fullsized":"2024-09-13T13:15:56.801720+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:14.899378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:14.899378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":693,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796200+0000","last_change":"2024-09-13T13:15:50.618476+0000","last_active":"2024-09-13T13:15:56.796200+0000","last_peered":"2024-09-13T13:15:56.796200+0000","last_clean":"2024-09-13T13:15:56.796200+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:15:56.796200+0000","last_undegraded":"2024-09-13T13:15:56.796200+0000","last_fullsized":"2024-09-13T13:15:56.796200+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:15:50.618426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.618426+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:58.477958+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":695,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.804024+0000","last_change":"2024-09-13T13:14:59.670853+0000","last_active":"2024-09-13T13:15:56.804024+0000","last_peered":"2024-09-13T13:15:56.804024+0000","last_clean":"2024-09-13T13:15:56.804024+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:15:56.804024+0000","last_undegraded":"2024-09-13T13:15:56.804024+0000","last_fullsized":"2024-09-13T13:15:56.804024+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.670779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.670779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.298565+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":683,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800223+0000","last_change":"2024-09-13T13:15:33.280401+0000","last_active":"2024-09-13T13:15:56.800223+0000","last_peered":"2024-09-13T13:15:56.800223+0000","last_clean":"2024-09-13T13:15:56.800223+0000","last_became_active":"2024-09-13T13:12:52.948742+0000","last_became_peered":"2024-09-13T13:12:52.948742+0000","last_unstale":"2024-09-13T13:15:56.800223+0000","last_undegraded":"2024-09-13T13:15:56.800223+0000","last_fullsized":"2024-09-13T13:15:56.800223+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:33.280352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.280352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:51.771913+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"372'9510","reported_seq":9597,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:58.401123+0000","last_change":"2024-09-13T13:13:59.149677+0000","last_active":"2024-09-13T13:15:58.401123+0000","last_peered":"2024-09-13T13:15:58.401123+0000","last_clean":"2024-09-13T13:15:58.401123+0000","last_became_active":"2024-09-13T13:13:59.149368+0000","last_became_peered":"2024-09-13T13:13:59.149368+0000","last_unstale":"2024-09-13T13:15:58.401123+0000","last_undegraded":"2024-09-13T13:15:58.401123+0000","last_fullsized":"2024-09-13T13:15:58.401123+0000","mapping_epoch":334,"log_start":"346'3900","ondisk_log_start":"346'3900","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:10.338929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407896064,"num_objects":6224,"num_object_clones":0,"num_object_copies":18672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":503488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":703,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796403+0000","last_change":"2024-09-13T13:15:33.299837+0000","last_active":"2024-09-13T13:15:56.796403+0000","last_peered":"2024-09-13T13:15:56.796403+0000","last_clean":"2024-09-13T13:15:56.796403+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:15:56.796403+0000","last_undegraded":"2024-09-13T13:15:56.796403+0000","last_fullsized":"2024-09-13T13:15:56.796403+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'17","last_scrub_stamp":"2024-09-13T13:15:33.299746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.299746+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:37.843249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":694,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.793195+0000","last_change":"2024-09-13T13:13:17.676405+0000","last_active":"2024-09-13T13:15:56.793195+0000","last_peered":"2024-09-13T13:15:56.793195+0000","last_clean":"2024-09-13T13:15:56.793195+0000","last_became_active":"2024-09-13T13:04:02.759010+0000","last_became_peered":"2024-09-13T13:04:02.759010+0000","last_unstale":"2024-09-13T13:15:56.793195+0000","last_undegraded":"2024-09-13T13:15:56.793195+0000","last_fullsized":"2024-09-13T13:15:56.793195+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:17.676339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:17.676339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.177334+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":698,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.792764+0000","last_change":"2024-09-13T13:13:14.560620+0000","last_active":"2024-09-13T13:15:56.792764+0000","last_peered":"2024-09-13T13:15:56.792764+0000","last_clean":"2024-09-13T13:15:56.792764+0000","last_became_active":"2024-09-13T13:01:02.177165+0000","last_became_peered":"2024-09-13T13:01:02.177165+0000","last_unstale":"2024-09-13T13:15:56.792764+0000","last_undegraded":"2024-09-13T13:15:56.792764+0000","last_fullsized":"2024-09-13T13:15:56.792764+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:13:14.560555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:14.560555+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.776407+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00051541300000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":669,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.794551+0000","last_change":"2024-09-13T13:14:23.728707+0000","last_active":"2024-09-13T13:15:56.794551+0000","last_peered":"2024-09-13T13:15:56.794551+0000","last_clean":"2024-09-13T13:15:56.794551+0000","last_became_active":"2024-09-13T13:01:06.205495+0000","last_became_peered":"2024-09-13T13:01:06.205495+0000","last_unstale":"2024-09-13T13:15:56.794551+0000","last_undegraded":"2024-09-13T13:15:56.794551+0000","last_fullsized":"2024-09-13T13:15:56.794551+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:23.728632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:23.728632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":293,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.738083+0000","last_change":"2024-09-13T13:12:41.332308+0000","last_active":"2024-09-13T13:15:51.738083+0000","last_peered":"2024-09-13T13:15:51.738083+0000","last_clean":"2024-09-13T13:15:51.738083+0000","last_became_active":"2024-09-13T13:12:40.120242+0000","last_became_peered":"2024-09-13T13:12:40.120242+0000","last_unstale":"2024-09-13T13:15:51.738083+0000","last_undegraded":"2024-09-13T13:15:51.738083+0000","last_fullsized":"2024-09-13T13:15:51.738083+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:41.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:41.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.372949+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"372'9781","reported_seq":9868,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:59.563029+0000","last_change":"2024-09-13T13:13:59.149922+0000","last_active":"2024-09-13T13:15:59.563029+0000","last_peered":"2024-09-13T13:15:59.563029+0000","last_clean":"2024-09-13T13:15:59.563029+0000","last_became_active":"2024-09-13T13:13:59.149037+0000","last_became_peered":"2024-09-13T13:13:59.149037+0000","last_unstale":"2024-09-13T13:15:59.563029+0000","last_undegraded":"2024-09-13T13:15:59.563029+0000","last_fullsized":"2024-09-13T13:15:59.563029+0000","mapping_epoch":334,"log_start":"348'4400","ondisk_log_start":"348'4400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5381,"log_dups_size":0,"ondisk_log_size":5381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.573331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413728768,"num_objects":6313,"num_object_clones":0,"num_object_copies":18939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9781,"num_write_kb":515008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":704,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.806920+0000","last_change":"2024-09-13T13:13:10.964903+0000","last_active":"2024-09-13T13:15:56.806920+0000","last_peered":"2024-09-13T13:15:56.806920+0000","last_clean":"2024-09-13T13:15:56.806920+0000","last_became_active":"2024-09-13T13:01:04.192708+0000","last_became_peered":"2024-09-13T13:01:04.192708+0000","last_unstale":"2024-09-13T13:15:56.806920+0000","last_undegraded":"2024-09-13T13:15:56.806920+0000","last_fullsized":"2024-09-13T13:15:56.806920+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:13:10.964809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.964809+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.681903+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":695,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.803438+0000","last_change":"2024-09-13T13:15:30.590673+0000","last_active":"2024-09-13T13:15:56.803438+0000","last_peered":"2024-09-13T13:15:56.803438+0000","last_clean":"2024-09-13T13:15:56.803438+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:15:56.803438+0000","last_undegraded":"2024-09-13T13:15:56.803438+0000","last_fullsized":"2024-09-13T13:15:56.803438+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:30.590466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:30.590466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:49.834025+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":416,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:57.423220+0000","last_change":"2024-09-13T13:14:48.128237+0000","last_active":"2024-09-13T13:15:57.423220+0000","last_peered":"2024-09-13T13:15:57.423220+0000","last_clean":"2024-09-13T13:15:57.423220+0000","last_became_active":"2024-09-13T13:12:39.947940+0000","last_became_peered":"2024-09-13T13:12:39.947940+0000","last_unstale":"2024-09-13T13:15:57.423220+0000","last_undegraded":"2024-09-13T13:15:57.423220+0000","last_fullsized":"2024-09-13T13:15:57.423220+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:14:48.128156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:48.128156+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.088862+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":205,"num_read_kb":200,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":665,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.799919+0000","last_change":"2024-09-13T13:15:32.318497+0000","last_active":"2024-09-13T13:15:56.799919+0000","last_peered":"2024-09-13T13:15:56.799919+0000","last_clean":"2024-09-13T13:15:56.799919+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:15:56.799919+0000","last_undegraded":"2024-09-13T13:15:56.799919+0000","last_fullsized":"2024-09-13T13:15:56.799919+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:32.318413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.318413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:32.934946+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":671,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.795830+0000","last_change":"2024-09-13T13:10:32.720742+0000","last_active":"2024-09-13T13:15:56.795830+0000","last_peered":"2024-09-13T13:15:56.795830+0000","last_clean":"2024-09-13T13:15:56.795830+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:15:56.795830+0000","last_undegraded":"2024-09-13T13:15:56.795830+0000","last_fullsized":"2024-09-13T13:15:56.795830+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.305143+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"371'9416","reported_seq":9500,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.224100+0000","last_change":"2024-09-13T13:13:58.967163+0000","last_active":"2024-09-13T13:15:56.224100+0000","last_peered":"2024-09-13T13:15:56.224100+0000","last_clean":"2024-09-13T13:15:56.224100+0000","last_became_active":"2024-09-13T13:13:58.965962+0000","last_became_peered":"2024-09-13T13:13:58.965962+0000","last_unstale":"2024-09-13T13:15:56.224100+0000","last_undegraded":"2024-09-13T13:15:56.224100+0000","last_fullsized":"2024-09-13T13:15:56.224100+0000","mapping_epoch":334,"log_start":"346'3400","ondisk_log_start":"346'3400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:12.328620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427032576,"num_objects":6516,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9416,"num_write_kb":509824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"306'32","reported_seq":796,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.737923+0000","last_change":"2024-09-13T13:13:01.352416+0000","last_active":"2024-09-13T13:15:51.737923+0000","last_peered":"2024-09-13T13:15:51.737923+0000","last_clean":"2024-09-13T13:15:51.737923+0000","last_became_active":"2024-09-13T13:12:52.999924+0000","last_became_peered":"2024-09-13T13:12:52.999924+0000","last_unstale":"2024-09-13T13:15:51.737923+0000","last_undegraded":"2024-09-13T13:15:51.737923+0000","last_fullsized":"2024-09-13T13:15:51.737923+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:13:01.352337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.352337+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.487539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":685,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.799132+0000","last_change":"2024-09-13T13:15:04.182566+0000","last_active":"2024-09-13T13:15:56.799132+0000","last_peered":"2024-09-13T13:15:56.799132+0000","last_clean":"2024-09-13T13:15:56.799132+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:15:56.799132+0000","last_undegraded":"2024-09-13T13:15:56.799132+0000","last_fullsized":"2024-09-13T13:15:56.799132+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:04.182499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.182499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.040803+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":259,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.804008+0000","last_change":"2024-09-13T13:15:54.707061+0000","last_active":"2024-09-13T13:15:56.804008+0000","last_peered":"2024-09-13T13:15:56.804008+0000","last_clean":"2024-09-13T13:15:56.804008+0000","last_became_active":"2024-09-13T13:12:39.948765+0000","last_became_peered":"2024-09-13T13:12:39.948765+0000","last_unstale":"2024-09-13T13:15:56.804008+0000","last_undegraded":"2024-09-13T13:15:56.804008+0000","last_fullsized":"2024-09-13T13:15:56.804008+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:10.766426+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":184,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.794486+0000","last_change":"2024-09-13T13:15:33.709071+0000","last_active":"2024-09-13T13:15:56.794486+0000","last_peered":"2024-09-13T13:15:56.794486+0000","last_clean":"2024-09-13T13:15:56.794486+0000","last_became_active":"2024-09-13T13:12:39.950496+0000","last_became_peered":"2024-09-13T13:12:39.950496+0000","last_unstale":"2024-09-13T13:15:56.794486+0000","last_undegraded":"2024-09-13T13:15:56.794486+0000","last_fullsized":"2024-09-13T13:15:56.794486+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:33.708995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.708995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:57.976953+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":707,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:57.689044+0000","last_change":"2024-09-13T13:15:57.689044+0000","last_active":"2024-09-13T13:15:57.689044+0000","last_peered":"2024-09-13T13:15:57.689044+0000","last_clean":"2024-09-13T13:15:57.689044+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:15:57.689044+0000","last_undegraded":"2024-09-13T13:15:57.689044+0000","last_fullsized":"2024-09-13T13:15:57.689044+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:57.688955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:57.688955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:01.261821+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"371'9634","reported_seq":9787,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.217373+0000","last_change":"2024-09-13T13:15:04.191246+0000","last_active":"2024-09-13T13:15:56.217373+0000","last_peered":"2024-09-13T13:15:56.217373+0000","last_clean":"2024-09-13T13:15:56.217373+0000","last_became_active":"2024-09-13T13:13:58.965566+0000","last_became_peered":"2024-09-13T13:13:58.965566+0000","last_unstale":"2024-09-13T13:15:56.217373+0000","last_undegraded":"2024-09-13T13:15:56.217373+0000","last_fullsized":"2024-09-13T13:15:56.217373+0000","mapping_epoch":334,"log_start":"346'3600","ondisk_log_start":"346'3600","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'6049","last_scrub_stamp":"2024-09-13T13:15:04.191100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.191100+0000","objects_scrubbed":5987,"log_size":6034,"log_dups_size":0,"ondisk_log_size":6034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:31.390342+0000","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429522944,"num_objects":6554,"num_object_clones":0,"num_object_copies":19662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":518016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":173,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801287+0000","last_change":"2024-09-13T13:14:59.354034+0000","last_active":"2024-09-13T13:15:56.801287+0000","last_peered":"2024-09-13T13:15:56.801287+0000","last_clean":"2024-09-13T13:15:56.801287+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:15:56.801287+0000","last_undegraded":"2024-09-13T13:15:56.801287+0000","last_fullsized":"2024-09-13T13:15:56.801287+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:14:59.353691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.353691+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:10.915988+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":153,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.798758+0000","last_change":"2024-09-13T13:14:26.343833+0000","last_active":"2024-09-13T13:15:56.798758+0000","last_peered":"2024-09-13T13:15:56.798758+0000","last_clean":"2024-09-13T13:15:56.798758+0000","last_became_active":"2024-09-13T13:12:53.006271+0000","last_became_peered":"2024-09-13T13:12:53.006271+0000","last_unstale":"2024-09-13T13:15:56.798758+0000","last_undegraded":"2024-09-13T13:15:56.798758+0000","last_fullsized":"2024-09-13T13:15:56.798758+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:26.343754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:26.343754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":722,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.799665+0000","last_change":"2024-09-13T13:15:55.592241+0000","last_active":"2024-09-13T13:15:56.799665+0000","last_peered":"2024-09-13T13:15:56.799665+0000","last_clean":"2024-09-13T13:15:56.799665+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:15:56.799665+0000","last_undegraded":"2024-09-13T13:15:56.799665+0000","last_fullsized":"2024-09-13T13:15:56.799665+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:55.592172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:55.592172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:23.172298+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":672,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800044+0000","last_change":"2024-09-13T13:14:23.655250+0000","last_active":"2024-09-13T13:15:56.800044+0000","last_peered":"2024-09-13T13:15:56.800044+0000","last_clean":"2024-09-13T13:15:56.800044+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:15:56.800044+0000","last_undegraded":"2024-09-13T13:15:56.800044+0000","last_fullsized":"2024-09-13T13:15:56.800044+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:23.655072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:23.655072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:45.281222+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"372'9637","reported_seq":9809,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:58.512616+0000","last_change":"2024-09-13T13:15:31.408104+0000","last_active":"2024-09-13T13:15:58.512616+0000","last_peered":"2024-09-13T13:15:58.512616+0000","last_clean":"2024-09-13T13:15:58.512616+0000","last_became_active":"2024-09-13T13:13:58.961825+0000","last_became_peered":"2024-09-13T13:13:58.961825+0000","last_unstale":"2024-09-13T13:15:58.512616+0000","last_undegraded":"2024-09-13T13:15:58.512616+0000","last_fullsized":"2024-09-13T13:15:58.512616+0000","mapping_epoch":334,"log_start":"344'2900","ondisk_log_start":"344'2900","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7793","last_scrub_stamp":"2024-09-13T13:15:31.408001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:31.408001+0000","objects_scrubbed":7714,"log_size":6737,"log_dups_size":0,"ondisk_log_size":6737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:37.609213+0000","scrub_duration":2771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412155904,"num_objects":6289,"num_object_clones":0,"num_object_copies":18867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9637,"num_write_kb":509632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":179,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.802074+0000","last_change":"2024-09-13T13:14:20.221364+0000","last_active":"2024-09-13T13:15:56.802074+0000","last_peered":"2024-09-13T13:15:56.802074+0000","last_clean":"2024-09-13T13:15:56.802074+0000","last_became_active":"2024-09-13T13:12:53.001687+0000","last_became_peered":"2024-09-13T13:12:53.001687+0000","last_unstale":"2024-09-13T13:15:56.802074+0000","last_undegraded":"2024-09-13T13:15:56.802074+0000","last_fullsized":"2024-09-13T13:15:56.802074+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:14:20.221286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:20.221286+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":694,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734340+0000","last_change":"2024-09-13T13:15:24.265548+0000","last_active":"2024-09-13T13:15:51.734340+0000","last_peered":"2024-09-13T13:15:51.734340+0000","last_clean":"2024-09-13T13:15:51.734340+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:15:51.734340+0000","last_undegraded":"2024-09-13T13:15:51.734340+0000","last_fullsized":"2024-09-13T13:15:51.734340+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:24.265477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:24.265477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:35.804220+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":688,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.795929+0000","last_change":"2024-09-13T13:15:24.264425+0000","last_active":"2024-09-13T13:15:56.795929+0000","last_peered":"2024-09-13T13:15:56.795929+0000","last_clean":"2024-09-13T13:15:56.795929+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:15:56.795929+0000","last_undegraded":"2024-09-13T13:15:56.795929+0000","last_fullsized":"2024-09-13T13:15:56.795929+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:24.264316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:24.264316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:32.724876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":676,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.794605+0000","last_change":"2024-09-13T13:13:54.649219+0000","last_active":"2024-09-13T13:15:56.794605+0000","last_peered":"2024-09-13T13:15:56.794605+0000","last_clean":"2024-09-13T13:15:56.794605+0000","last_became_active":"2024-09-13T13:12:52.938975+0000","last_became_peered":"2024-09-13T13:12:52.938975+0000","last_unstale":"2024-09-13T13:15:56.794605+0000","last_undegraded":"2024-09-13T13:15:56.794605+0000","last_fullsized":"2024-09-13T13:15:56.794605+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.649166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.649166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.656296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"372'9730","reported_seq":9815,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:59.004372+0000","last_change":"2024-09-13T13:13:58.963327+0000","last_active":"2024-09-13T13:15:59.004372+0000","last_peered":"2024-09-13T13:15:59.004372+0000","last_clean":"2024-09-13T13:15:59.004372+0000","last_became_active":"2024-09-13T13:13:58.963064+0000","last_became_peered":"2024-09-13T13:13:58.963064+0000","last_unstale":"2024-09-13T13:15:59.004372+0000","last_undegraded":"2024-09-13T13:15:59.004372+0000","last_fullsized":"2024-09-13T13:15:59.004372+0000","mapping_epoch":334,"log_start":"348'4400","ondisk_log_start":"348'4400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5330,"log_dups_size":0,"ondisk_log_size":5330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.315173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413401088,"num_objects":6308,"num_object_clones":0,"num_object_copies":18924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9730,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":157,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.802113+0000","last_change":"2024-09-13T13:12:53.018645+0000","last_active":"2024-09-13T13:15:56.802113+0000","last_peered":"2024-09-13T13:15:56.802113+0000","last_clean":"2024-09-13T13:15:56.802113+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:15:56.802113+0000","last_undegraded":"2024-09-13T13:15:56.802113+0000","last_fullsized":"2024-09-13T13:15:56.802113+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:09.739184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.739184+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.221949+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":679,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800621+0000","last_change":"2024-09-13T13:12:53.003530+0000","last_active":"2024-09-13T13:15:56.800621+0000","last_peered":"2024-09-13T13:15:56.800621+0000","last_clean":"2024-09-13T13:15:56.800621+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:15:56.800621+0000","last_undegraded":"2024-09-13T13:15:56.800621+0000","last_fullsized":"2024-09-13T13:15:56.800621+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.996346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.996346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:10.180309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":846,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.794887+0000","last_change":"2024-09-13T13:12:36.113757+0000","last_active":"2024-09-13T13:15:56.794887+0000","last_peered":"2024-09-13T13:15:56.794887+0000","last_clean":"2024-09-13T13:15:56.794887+0000","last_became_active":"2024-09-13T13:01:06.205986+0000","last_became_peered":"2024-09-13T13:01:06.205986+0000","last_unstale":"2024-09-13T13:15:56.794887+0000","last_undegraded":"2024-09-13T13:15:56.794887+0000","last_fullsized":"2024-09-13T13:15:56.794887+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:12:36.113709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:36.113709+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.715547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":153,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801210+0000","last_change":"2024-09-13T13:13:41.797728+0000","last_active":"2024-09-13T13:15:56.801210+0000","last_peered":"2024-09-13T13:15:56.801210+0000","last_clean":"2024-09-13T13:15:56.801210+0000","last_became_active":"2024-09-13T13:12:52.997283+0000","last_became_peered":"2024-09-13T13:12:52.997283+0000","last_unstale":"2024-09-13T13:15:56.801210+0000","last_undegraded":"2024-09-13T13:15:56.801210+0000","last_fullsized":"2024-09-13T13:15:56.801210+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.797639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.797639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:06.323760+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"372'9621","reported_seq":9793,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:58.518337+0000","last_change":"2024-09-13T13:15:32.530404+0000","last_active":"2024-09-13T13:15:58.518337+0000","last_peered":"2024-09-13T13:15:58.518337+0000","last_clean":"2024-09-13T13:15:58.518337+0000","last_became_active":"2024-09-13T13:13:58.962803+0000","last_became_peered":"2024-09-13T13:13:58.962803+0000","last_unstale":"2024-09-13T13:15:58.518337+0000","last_undegraded":"2024-09-13T13:15:58.518337+0000","last_fullsized":"2024-09-13T13:15:58.518337+0000","mapping_epoch":334,"log_start":"344'2900","ondisk_log_start":"344'2900","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7822","last_scrub_stamp":"2024-09-13T13:15:32.530220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.530220+0000","objects_scrubbed":7744,"log_size":6721,"log_dups_size":0,"ondisk_log_size":6721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:48.786652+0000","scrub_duration":2862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408879104,"num_objects":6239,"num_object_clones":0,"num_object_copies":18717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9621,"num_write_kb":507520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":711,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.804246+0000","last_change":"2024-09-13T13:13:56.850149+0000","last_active":"2024-09-13T13:15:56.804246+0000","last_peered":"2024-09-13T13:15:56.804246+0000","last_clean":"2024-09-13T13:15:56.804246+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:15:56.804246+0000","last_undegraded":"2024-09-13T13:15:56.804246+0000","last_fullsized":"2024-09-13T13:15:56.804246+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:56.850094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.850094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.708354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":700,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796332+0000","last_change":"2024-09-13T13:12:58.857634+0000","last_active":"2024-09-13T13:15:56.796332+0000","last_peered":"2024-09-13T13:15:56.796332+0000","last_clean":"2024-09-13T13:15:56.796332+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:15:56.796332+0000","last_undegraded":"2024-09-13T13:15:56.796332+0000","last_fullsized":"2024-09-13T13:15:56.796332+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:58.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:44.458422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":712,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.794356+0000","last_change":"2024-09-13T13:15:23.682529+0000","last_active":"2024-09-13T13:15:56.794356+0000","last_peered":"2024-09-13T13:15:56.794356+0000","last_clean":"2024-09-13T13:15:56.794356+0000","last_became_active":"2024-09-13T13:12:52.944725+0000","last_became_peered":"2024-09-13T13:12:52.944725+0000","last_unstale":"2024-09-13T13:15:56.794356+0000","last_undegraded":"2024-09-13T13:15:56.794356+0000","last_fullsized":"2024-09-13T13:15:56.794356+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:23.682463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.682463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.810934+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":660,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.803382+0000","last_change":"2024-09-13T13:13:41.762003+0000","last_active":"2024-09-13T13:15:56.803382+0000","last_peered":"2024-09-13T13:15:56.803382+0000","last_clean":"2024-09-13T13:15:56.803382+0000","last_became_active":"2024-09-13T13:01:08.210843+0000","last_became_peered":"2024-09-13T13:01:08.210843+0000","last_unstale":"2024-09-13T13:15:56.803382+0000","last_undegraded":"2024-09-13T13:15:56.803382+0000","last_fullsized":"2024-09-13T13:15:56.803382+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.761814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.761814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.545140+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"372'9521","reported_seq":9607,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:59.563292+0000","last_change":"2024-09-13T13:13:58.963548+0000","last_active":"2024-09-13T13:15:59.563292+0000","last_peered":"2024-09-13T13:15:59.563292+0000","last_clean":"2024-09-13T13:15:59.563292+0000","last_became_active":"2024-09-13T13:13:58.963160+0000","last_became_peered":"2024-09-13T13:13:58.963160+0000","last_unstale":"2024-09-13T13:15:59.563292+0000","last_undegraded":"2024-09-13T13:15:59.563292+0000","last_fullsized":"2024-09-13T13:15:59.563292+0000","mapping_epoch":334,"log_start":"348'4100","ondisk_log_start":"348'4100","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5421,"log_dups_size":0,"ondisk_log_size":5421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:11.607417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404815872,"num_objects":6177,"num_object_clones":0,"num_object_copies":18531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9521,"num_write_kb":502336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":683,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734140+0000","last_change":"2024-09-13T13:13:06.339762+0000","last_active":"2024-09-13T13:15:51.734140+0000","last_peered":"2024-09-13T13:15:51.734140+0000","last_clean":"2024-09-13T13:15:51.734140+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:15:51.734140+0000","last_undegraded":"2024-09-13T13:15:51.734140+0000","last_fullsized":"2024-09-13T13:15:51.734140+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:13:06.339664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.339664+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:11.302026+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":315,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796709+0000","last_change":"2024-09-13T13:14:55.471954+0000","last_active":"2024-09-13T13:15:56.796709+0000","last_peered":"2024-09-13T13:15:56.796709+0000","last_clean":"2024-09-13T13:15:56.796709+0000","last_became_active":"2024-09-13T13:12:39.954315+0000","last_became_peered":"2024-09-13T13:12:39.954315+0000","last_unstale":"2024-09-13T13:15:56.796709+0000","last_undegraded":"2024-09-13T13:15:56.796709+0000","last_fullsized":"2024-09-13T13:15:56.796709+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:55.471885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:55.471885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:01.346273+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.794260+0000","last_change":"2024-09-13T13:13:11.484689+0000","last_active":"2024-09-13T13:15:56.794260+0000","last_peered":"2024-09-13T13:15:56.794260+0000","last_clean":"2024-09-13T13:15:56.794260+0000","last_became_active":"2024-09-13T13:01:06.203463+0000","last_became_peered":"2024-09-13T13:01:06.203463+0000","last_unstale":"2024-09-13T13:15:56.794260+0000","last_undegraded":"2024-09-13T13:15:56.794260+0000","last_fullsized":"2024-09-13T13:15:56.794260+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.484615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.484615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.740951+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":168,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.792336+0000","last_change":"2024-09-13T13:15:02.627642+0000","last_active":"2024-09-13T13:15:56.792336+0000","last_peered":"2024-09-13T13:15:56.792336+0000","last_clean":"2024-09-13T13:15:56.792336+0000","last_became_active":"2024-09-13T13:12:40.118050+0000","last_became_peered":"2024-09-13T13:12:40.118050+0000","last_unstale":"2024-09-13T13:15:56.792336+0000","last_undegraded":"2024-09-13T13:15:56.792336+0000","last_fullsized":"2024-09-13T13:15:56.792336+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:02.627569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:02.627569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:32.377161+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"371'9307","reported_seq":9392,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.203658+0000","last_change":"2024-09-13T13:13:58.966955+0000","last_active":"2024-09-13T13:15:56.203658+0000","last_peered":"2024-09-13T13:15:56.203658+0000","last_clean":"2024-09-13T13:15:56.203658+0000","last_became_active":"2024-09-13T13:13:58.966675+0000","last_became_peered":"2024-09-13T13:13:58.966675+0000","last_unstale":"2024-09-13T13:15:56.203658+0000","last_undegraded":"2024-09-13T13:15:56.203658+0000","last_fullsized":"2024-09-13T13:15:56.203658+0000","mapping_epoch":334,"log_start":"346'3300","ondisk_log_start":"346'3300","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":6007,"log_dups_size":0,"ondisk_log_size":6007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.259461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411762688,"num_objects":6283,"num_object_clones":0,"num_object_copies":18849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":498880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":165,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.799694+0000","last_change":"2024-09-13T13:13:20.064693+0000","last_active":"2024-09-13T13:15:56.799694+0000","last_peered":"2024-09-13T13:15:56.799694+0000","last_clean":"2024-09-13T13:15:56.799694+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:15:56.799694+0000","last_undegraded":"2024-09-13T13:15:56.799694+0000","last_fullsized":"2024-09-13T13:15:56.799694+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:13:20.064614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:20.064614+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":161,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801135+0000","last_change":"2024-09-13T13:14:52.383027+0000","last_active":"2024-09-13T13:15:56.801135+0000","last_peered":"2024-09-13T13:15:56.801135+0000","last_clean":"2024-09-13T13:15:56.801135+0000","last_became_active":"2024-09-13T13:12:53.007561+0000","last_became_peered":"2024-09-13T13:12:53.007561+0000","last_unstale":"2024-09-13T13:15:56.801135+0000","last_undegraded":"2024-09-13T13:15:56.801135+0000","last_fullsized":"2024-09-13T13:15:56.801135+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.382755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.382755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:20.097418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":666,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.792493+0000","last_change":"2024-09-13T13:15:00.637152+0000","last_active":"2024-09-13T13:15:56.792493+0000","last_peered":"2024-09-13T13:15:56.792493+0000","last_clean":"2024-09-13T13:15:56.792493+0000","last_became_active":"2024-09-13T13:01:06.209895+0000","last_became_peered":"2024-09-13T13:01:06.209895+0000","last_unstale":"2024-09-13T13:15:56.792493+0000","last_undegraded":"2024-09-13T13:15:56.792493+0000","last_fullsized":"2024-09-13T13:15:56.792493+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.636898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.636898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.204507+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":653,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800063+0000","last_change":"2024-09-13T13:15:03.223521+0000","last_active":"2024-09-13T13:15:56.800063+0000","last_peered":"2024-09-13T13:15:56.800063+0000","last_clean":"2024-09-13T13:15:56.800063+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:15:56.800063+0000","last_undegraded":"2024-09-13T13:15:56.800063+0000","last_fullsized":"2024-09-13T13:15:56.800063+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.223265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.223265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.540511+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"372'9702","reported_seq":9862,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:58.416284+0000","last_change":"2024-09-13T13:15:56.411481+0000","last_active":"2024-09-13T13:15:58.416284+0000","last_peered":"2024-09-13T13:15:58.416284+0000","last_clean":"2024-09-13T13:15:58.416284+0000","last_became_active":"2024-09-13T13:13:58.962773+0000","last_became_peered":"2024-09-13T13:13:58.962773+0000","last_unstale":"2024-09-13T13:15:58.416284+0000","last_undegraded":"2024-09-13T13:15:58.416284+0000","last_fullsized":"2024-09-13T13:15:58.416284+0000","mapping_epoch":334,"log_start":"348'4100","ondisk_log_start":"348'4100","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'9588","last_scrub_stamp":"2024-09-13T13:15:56.411320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:56.411320+0000","objects_scrubbed":6514,"log_size":5602,"log_dups_size":0,"ondisk_log_size":5602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:14.862198+0000","scrub_duration":1806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416284672,"num_objects":6352,"num_object_clones":0,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":513728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":747,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.795693+0000","last_change":"2024-09-13T13:15:56.631456+0000","last_active":"2024-09-13T13:15:56.795693+0000","last_peered":"2024-09-13T13:15:56.795693+0000","last_clean":"2024-09-13T13:15:56.795693+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:15:56.795693+0000","last_undegraded":"2024-09-13T13:15:56.795693+0000","last_fullsized":"2024-09-13T13:15:56.795693+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:15:56.631396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:56.631396+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:22.171338+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":159,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800972+0000","last_change":"2024-09-13T13:13:01.972907+0000","last_active":"2024-09-13T13:15:56.800972+0000","last_peered":"2024-09-13T13:15:56.800972+0000","last_clean":"2024-09-13T13:15:56.800972+0000","last_became_active":"2024-09-13T13:12:53.006362+0000","last_became_peered":"2024-09-13T13:12:53.006362+0000","last_unstale":"2024-09-13T13:15:56.800972+0000","last_undegraded":"2024-09-13T13:15:56.800972+0000","last_fullsized":"2024-09-13T13:15:56.800972+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:13:01.972831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.972831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:46.279374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"306'13","reported_seq":398,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.733489+0000","last_change":"2024-09-13T13:13:03.334319+0000","last_active":"2024-09-13T13:15:51.733489+0000","last_peered":"2024-09-13T13:15:51.733489+0000","last_clean":"2024-09-13T13:15:51.733489+0000","last_became_active":"2024-09-13T13:12:53.001189+0000","last_became_peered":"2024-09-13T13:12:53.001189+0000","last_unstale":"2024-09-13T13:15:51.733489+0000","last_undegraded":"2024-09-13T13:15:51.733489+0000","last_fullsized":"2024-09-13T13:15:51.733489+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'13","last_scrub_stamp":"2024-09-13T13:13:03.334012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.334012+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.265982+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":672,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.795913+0000","last_change":"2024-09-13T13:13:55.500961+0000","last_active":"2024-09-13T13:15:56.795913+0000","last_peered":"2024-09-13T13:15:56.795913+0000","last_clean":"2024-09-13T13:15:56.795913+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:15:56.795913+0000","last_undegraded":"2024-09-13T13:15:56.795913+0000","last_fullsized":"2024-09-13T13:15:56.795913+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:55.500908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.500908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:11.347558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":648,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.734092+0000","last_change":"2024-09-13T13:13:04.371082+0000","last_active":"2024-09-13T13:15:51.734092+0000","last_peered":"2024-09-13T13:15:51.734092+0000","last_clean":"2024-09-13T13:15:51.734092+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:15:51.734092+0000","last_undegraded":"2024-09-13T13:15:51.734092+0000","last_fullsized":"2024-09-13T13:15:51.734092+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.370881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.370881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.428165+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":195,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.799409+0000","last_change":"2024-09-13T13:15:23.319486+0000","last_active":"2024-09-13T13:15:56.799409+0000","last_peered":"2024-09-13T13:15:56.799409+0000","last_clean":"2024-09-13T13:15:56.799409+0000","last_became_active":"2024-09-13T13:12:34.673528+0000","last_became_peered":"2024-09-13T13:12:34.673528+0000","last_unstale":"2024-09-13T13:15:56.799409+0000","last_undegraded":"2024-09-13T13:15:56.799409+0000","last_fullsized":"2024-09-13T13:15:56.799409+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:15:23.319412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.319412+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:45.332563+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":697,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.797210+0000","last_change":"2024-09-13T13:15:51.633487+0000","last_active":"2024-09-13T13:15:56.797210+0000","last_peered":"2024-09-13T13:15:56.797210+0000","last_clean":"2024-09-13T13:15:56.797210+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:15:56.797210+0000","last_undegraded":"2024-09-13T13:15:56.797210+0000","last_fullsized":"2024-09-13T13:15:56.797210+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.633433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.633433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:56.064702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":710,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801377+0000","last_change":"2024-09-13T13:15:25.787110+0000","last_active":"2024-09-13T13:15:56.801377+0000","last_peered":"2024-09-13T13:15:56.801377+0000","last_clean":"2024-09-13T13:15:56.801377+0000","last_became_active":"2024-09-13T13:12:53.000873+0000","last_became_peered":"2024-09-13T13:12:53.000873+0000","last_unstale":"2024-09-13T13:15:56.801377+0000","last_undegraded":"2024-09-13T13:15:56.801377+0000","last_fullsized":"2024-09-13T13:15:56.801377+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.787029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.787029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:31.829669+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":685,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.807055+0000","last_change":"2024-09-13T13:13:12.952427+0000","last_active":"2024-09-13T13:15:56.807055+0000","last_peered":"2024-09-13T13:15:56.807055+0000","last_clean":"2024-09-13T13:15:56.807055+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:15:56.807055+0000","last_undegraded":"2024-09-13T13:15:56.807055+0000","last_fullsized":"2024-09-13T13:15:56.807055+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:12.952348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:12.952348+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:37.576218+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":153,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.799414+0000","last_change":"2024-09-13T13:13:51.764793+0000","last_active":"2024-09-13T13:15:56.799414+0000","last_peered":"2024-09-13T13:15:56.799414+0000","last_clean":"2024-09-13T13:15:56.799414+0000","last_became_active":"2024-09-13T13:12:53.006579+0000","last_became_peered":"2024-09-13T13:12:53.006579+0000","last_unstale":"2024-09-13T13:15:56.799414+0000","last_undegraded":"2024-09-13T13:15:56.799414+0000","last_fullsized":"2024-09-13T13:15:56.799414+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.764717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.764717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:11.315764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":673,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801590+0000","last_change":"2024-09-13T13:15:25.281502+0000","last_active":"2024-09-13T13:15:56.801590+0000","last_peered":"2024-09-13T13:15:56.801590+0000","last_clean":"2024-09-13T13:15:56.801590+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:15:56.801590+0000","last_undegraded":"2024-09-13T13:15:56.801590+0000","last_fullsized":"2024-09-13T13:15:56.801590+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.281295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.281295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.642193+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":708,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.794324+0000","last_change":"2024-09-13T13:14:15.616158+0000","last_active":"2024-09-13T13:15:56.794324+0000","last_peered":"2024-09-13T13:15:56.794324+0000","last_clean":"2024-09-13T13:15:56.794324+0000","last_became_active":"2024-09-13T13:12:52.950504+0000","last_became_peered":"2024-09-13T13:12:52.950504+0000","last_unstale":"2024-09-13T13:15:56.794324+0000","last_undegraded":"2024-09-13T13:15:56.794324+0000","last_fullsized":"2024-09-13T13:15:56.794324+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:15.616112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:15.616112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.725422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":698,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796797+0000","last_change":"2024-09-13T13:12:34.682535+0000","last_active":"2024-09-13T13:15:56.796797+0000","last_peered":"2024-09-13T13:15:56.796797+0000","last_clean":"2024-09-13T13:15:56.796797+0000","last_became_active":"2024-09-13T13:12:34.665746+0000","last_became_peered":"2024-09-13T13:12:34.665746+0000","last_unstale":"2024-09-13T13:15:56.796797+0000","last_undegraded":"2024-09-13T13:15:56.796797+0000","last_fullsized":"2024-09-13T13:15:56.796797+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:19.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:19.931466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:41.387724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":664,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.806720+0000","last_change":"2024-09-13T13:13:05.981881+0000","last_active":"2024-09-13T13:15:56.806720+0000","last_peered":"2024-09-13T13:15:56.806720+0000","last_clean":"2024-09-13T13:15:56.806720+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:15:56.806720+0000","last_undegraded":"2024-09-13T13:15:56.806720+0000","last_fullsized":"2024-09-13T13:15:56.806720+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:05.981803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:05.981803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:07.697737+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":845,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800198+0000","last_change":"2024-09-13T13:13:10.130755+0000","last_active":"2024-09-13T13:15:56.800198+0000","last_peered":"2024-09-13T13:15:56.800198+0000","last_clean":"2024-09-13T13:15:56.800198+0000","last_became_active":"2024-09-13T13:01:06.209896+0000","last_became_peered":"2024-09-13T13:01:06.209896+0000","last_unstale":"2024-09-13T13:15:56.800198+0000","last_undegraded":"2024-09-13T13:15:56.800198+0000","last_fullsized":"2024-09-13T13:15:56.800198+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:13:10.130680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.130680+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.267335+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":153,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801066+0000","last_change":"2024-09-13T13:13:11.956837+0000","last_active":"2024-09-13T13:15:56.801066+0000","last_peered":"2024-09-13T13:15:56.801066+0000","last_clean":"2024-09-13T13:15:56.801066+0000","last_became_active":"2024-09-13T13:12:52.946339+0000","last_became_peered":"2024-09-13T13:12:52.946339+0000","last_unstale":"2024-09-13T13:15:56.801066+0000","last_undegraded":"2024-09-13T13:15:56.801066+0000","last_fullsized":"2024-09-13T13:15:56.801066+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.956784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.956784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:44.263343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":715,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800340+0000","last_change":"2024-09-13T13:15:34.280338+0000","last_active":"2024-09-13T13:15:56.800340+0000","last_peered":"2024-09-13T13:15:56.800340+0000","last_clean":"2024-09-13T13:15:56.800340+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:15:56.800340+0000","last_undegraded":"2024-09-13T13:15:56.800340+0000","last_fullsized":"2024-09-13T13:15:56.800340+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:15:34.280253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:34.280253+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.301730+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":679,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801058+0000","last_change":"2024-09-13T13:13:57.105151+0000","last_active":"2024-09-13T13:15:56.801058+0000","last_peered":"2024-09-13T13:15:56.801058+0000","last_clean":"2024-09-13T13:15:56.801058+0000","last_became_active":"2024-09-13T13:12:52.999912+0000","last_became_peered":"2024-09-13T13:12:52.999912+0000","last_unstale":"2024-09-13T13:15:56.801058+0000","last_undegraded":"2024-09-13T13:15:56.801058+0000","last_fullsized":"2024-09-13T13:15:56.801058+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.105098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.105098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.715266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":869,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.796622+0000","last_change":"2024-09-13T13:15:33.333062+0000","last_active":"2024-09-13T13:15:56.796622+0000","last_peered":"2024-09-13T13:15:56.796622+0000","last_clean":"2024-09-13T13:15:56.796622+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:15:56.796622+0000","last_undegraded":"2024-09-13T13:15:56.796622+0000","last_fullsized":"2024-09-13T13:15:56.796622+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:15:33.332981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.332981+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.941448+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":676,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.804017+0000","last_change":"2024-09-13T13:15:22.660262+0000","last_active":"2024-09-13T13:15:56.804017+0000","last_peered":"2024-09-13T13:15:56.804017+0000","last_clean":"2024-09-13T13:15:56.804017+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:15:56.804017+0000","last_undegraded":"2024-09-13T13:15:56.804017+0000","last_fullsized":"2024-09-13T13:15:56.804017+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:22.660050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:22.660050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:44.066070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":165,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.801822+0000","last_change":"2024-09-13T13:13:47.728092+0000","last_active":"2024-09-13T13:15:56.801822+0000","last_peered":"2024-09-13T13:15:56.801822+0000","last_clean":"2024-09-13T13:15:56.801822+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:15:56.801822+0000","last_undegraded":"2024-09-13T13:15:56.801822+0000","last_fullsized":"2024-09-13T13:15:56.801822+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:47.728010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:47.728010+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.358350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":145,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.802288+0000","last_change":"2024-09-13T13:12:53.006302+0000","last_active":"2024-09-13T13:15:56.802288+0000","last_peered":"2024-09-13T13:15:56.802288+0000","last_clean":"2024-09-13T13:15:56.802288+0000","last_became_active":"2024-09-13T13:12:53.005988+0000","last_became_peered":"2024-09-13T13:12:53.005988+0000","last_unstale":"2024-09-13T13:15:56.802288+0000","last_undegraded":"2024-09-13T13:15:56.802288+0000","last_fullsized":"2024-09-13T13:15:56.802288+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:40.888083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.888083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.372875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":682,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:57.356507+0000","last_change":"2024-09-13T13:15:57.356507+0000","last_active":"2024-09-13T13:15:57.356507+0000","last_peered":"2024-09-13T13:15:57.356507+0000","last_clean":"2024-09-13T13:15:57.356507+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:15:57.356507+0000","last_undegraded":"2024-09-13T13:15:57.356507+0000","last_fullsized":"2024-09-13T13:15:57.356507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:57.356427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:57.356427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:20.233276+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":677,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.799245+0000","last_change":"2024-09-13T13:15:51.374176+0000","last_active":"2024-09-13T13:15:56.799245+0000","last_peered":"2024-09-13T13:15:56.799245+0000","last_clean":"2024-09-13T13:15:56.799245+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:15:56.799245+0000","last_undegraded":"2024-09-13T13:15:56.799245+0000","last_fullsized":"2024-09-13T13:15:56.799245+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.374109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.374109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:54.143731+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":637,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.795637+0000","last_change":"2024-09-13T13:12:39.874394+0000","last_active":"2024-09-13T13:15:56.795637+0000","last_peered":"2024-09-13T13:15:56.795637+0000","last_clean":"2024-09-13T13:15:56.795637+0000","last_became_active":"2024-09-13T13:04:02.756971+0000","last_became_peered":"2024-09-13T13:04:02.756971+0000","last_unstale":"2024-09-13T13:15:56.795637+0000","last_undegraded":"2024-09-13T13:15:56.795637+0000","last_fullsized":"2024-09-13T13:15:56.795637+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:39.874324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:39.874324+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.370889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":665,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800314+0000","last_change":"2024-09-13T13:14:13.418389+0000","last_active":"2024-09-13T13:15:56.800314+0000","last_peered":"2024-09-13T13:15:56.800314+0000","last_clean":"2024-09-13T13:15:56.800314+0000","last_became_active":"2024-09-13T13:12:52.949094+0000","last_became_peered":"2024-09-13T13:12:52.949094+0000","last_unstale":"2024-09-13T13:15:56.800314+0000","last_undegraded":"2024-09-13T13:15:56.800314+0000","last_fullsized":"2024-09-13T13:15:56.800314+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:13.418196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:13.418196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.331557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":673,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800393+0000","last_change":"2024-09-13T13:15:01.269060+0000","last_active":"2024-09-13T13:15:56.800393+0000","last_peered":"2024-09-13T13:15:56.800393+0000","last_clean":"2024-09-13T13:15:56.800393+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:15:56.800393+0000","last_undegraded":"2024-09-13T13:15:56.800393+0000","last_fullsized":"2024-09-13T13:15:56.800393+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:01.268884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:01.268884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:08.329102+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":660,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.803196+0000","last_change":"2024-09-13T13:10:39.477882+0000","last_active":"2024-09-13T13:15:56.803196+0000","last_peered":"2024-09-13T13:15:56.803196+0000","last_clean":"2024-09-13T13:15:56.803196+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:15:56.803196+0000","last_undegraded":"2024-09-13T13:15:56.803196+0000","last_fullsized":"2024-09-13T13:15:56.803196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:39.477683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.477683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.622029+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":695,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.799368+0000","last_change":"2024-09-13T13:13:49.268080+0000","last_active":"2024-09-13T13:15:56.799368+0000","last_peered":"2024-09-13T13:15:56.799368+0000","last_clean":"2024-09-13T13:15:56.799368+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:15:56.799368+0000","last_undegraded":"2024-09-13T13:15:56.799368+0000","last_fullsized":"2024-09-13T13:15:56.799368+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:49.268018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:49.268018+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.301185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":689,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.797283+0000","last_change":"2024-09-13T13:14:50.392659+0000","last_active":"2024-09-13T13:15:56.797283+0000","last_peered":"2024-09-13T13:15:56.797283+0000","last_clean":"2024-09-13T13:15:56.797283+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:15:56.797283+0000","last_undegraded":"2024-09-13T13:15:56.797283+0000","last_fullsized":"2024-09-13T13:15:56.797283+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.392444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.392444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.481736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":698,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800269+0000","last_change":"2024-09-13T13:15:03.778214+0000","last_active":"2024-09-13T13:15:56.800269+0000","last_peered":"2024-09-13T13:15:56.800269+0000","last_clean":"2024-09-13T13:15:56.800269+0000","last_became_active":"2024-09-13T13:12:52.999748+0000","last_became_peered":"2024-09-13T13:12:52.999748+0000","last_unstale":"2024-09-13T13:15:56.800269+0000","last_undegraded":"2024-09-13T13:15:56.800269+0000","last_fullsized":"2024-09-13T13:15:56.800269+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.778170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.778170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:20.178488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":656,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.800530+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:15:56.800530+0000","last_peered":"2024-09-13T13:15:56.800530+0000","last_clean":"2024-09-13T13:15:56.800530+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:15:56.800530+0000","last_undegraded":"2024-09-13T13:15:56.800530+0000","last_fullsized":"2024-09-13T13:15:56.800530+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.231726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":705,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.798232+0000","last_change":"2024-09-13T13:14:12.368507+0000","last_active":"2024-09-13T13:15:56.798232+0000","last_peered":"2024-09-13T13:15:56.798232+0000","last_clean":"2024-09-13T13:15:56.798232+0000","last_became_active":"2024-09-13T13:04:02.755364+0000","last_became_peered":"2024-09-13T13:04:02.755364+0000","last_unstale":"2024-09-13T13:15:56.798232+0000","last_undegraded":"2024-09-13T13:15:56.798232+0000","last_fullsized":"2024-09-13T13:15:56.798232+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:14:12.368430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:12.368430+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.308538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":688,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.803780+0000","last_change":"2024-09-13T13:13:44.750043+0000","last_active":"2024-09-13T13:15:56.803780+0000","last_peered":"2024-09-13T13:15:56.803780+0000","last_clean":"2024-09-13T13:15:56.803780+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:15:56.803780+0000","last_undegraded":"2024-09-13T13:15:56.803780+0000","last_fullsized":"2024-09-13T13:15:56.803780+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:44.749961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:44.749961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:09.672770+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":602,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.733536+0000","last_change":"2024-09-13T13:14:16.457160+0000","last_active":"2024-09-13T13:15:51.733536+0000","last_peered":"2024-09-13T13:15:51.733536+0000","last_clean":"2024-09-13T13:15:51.733536+0000","last_became_active":"2024-09-13T13:12:53.001131+0000","last_became_peered":"2024-09-13T13:12:53.001131+0000","last_unstale":"2024-09-13T13:15:51.733536+0000","last_undegraded":"2024-09-13T13:15:51.733536+0000","last_fullsized":"2024-09-13T13:15:51.733536+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:16.456784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:16.456784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:43.279313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":653,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.792145+0000","last_change":"2024-09-13T13:12:52.947380+0000","last_active":"2024-09-13T13:15:56.792145+0000","last_peered":"2024-09-13T13:15:56.792145+0000","last_clean":"2024-09-13T13:15:56.792145+0000","last_became_active":"2024-09-13T13:12:52.946611+0000","last_became_peered":"2024-09-13T13:12:52.946611+0000","last_unstale":"2024-09-13T13:15:56.792145+0000","last_undegraded":"2024-09-13T13:15:56.792145+0000","last_fullsized":"2024-09-13T13:15:56.792145+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:08.704450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:08.704450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:05.645492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":683,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.795855+0000","last_change":"2024-09-13T13:14:18.882969+0000","last_active":"2024-09-13T13:15:56.795855+0000","last_peered":"2024-09-13T13:15:56.795855+0000","last_clean":"2024-09-13T13:15:56.795855+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:15:56.795855+0000","last_undegraded":"2024-09-13T13:15:56.795855+0000","last_fullsized":"2024-09-13T13:15:56.795855+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:18.882903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:18.882903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:46.394564+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":665,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.798559+0000","last_change":"2024-09-13T13:11:07.564094+0000","last_active":"2024-09-13T13:15:56.798559+0000","last_peered":"2024-09-13T13:15:56.798559+0000","last_clean":"2024-09-13T13:15:56.798559+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:15:56.798559+0000","last_undegraded":"2024-09-13T13:15:56.798559+0000","last_fullsized":"2024-09-13T13:15:56.798559+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:07.564025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:07.564025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:35.266907+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":670,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-13T13:15:51.737978+0000","last_change":"2024-09-13T13:11:49.264206+0000","last_active":"2024-09-13T13:15:51.737978+0000","last_peered":"2024-09-13T13:15:51.737978+0000","last_clean":"2024-09-13T13:15:51.737978+0000","last_became_active":"2024-09-13T13:04:02.755205+0000","last_became_peered":"2024-09-13T13:04:02.755205+0000","last_unstale":"2024-09-13T13:15:51.737978+0000","last_undegraded":"2024-09-13T13:15:51.737978+0000","last_fullsized":"2024-09-13T13:15:51.737978+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:49.264128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.264128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.230570+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":689,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T13:15:56.804236+0000","last_change":"2024-09-13T13:15:27.678822+0000","last_active":"2024-09-13T13:15:56.804236+0000","last_peered":"2024-09-13T13:15:56.804236+0000","last_clean":"2024-09-13T13:15:56.804236+0000","last_became_active":"2024-09-13T13:01:06.211225+0000","last_became_peered":"2024-09-13T13:01:06.211225+0000","last_unstale":"2024-09-13T13:15:56.804236+0000","last_undegraded":"2024-09-13T13:15:56.804236+0000","last_fullsized":"2024-09-13T13:15:56.804236+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:27.678752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:27.678752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:48.252436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":6665076760,"num_objects":101702,"num_object_clones":0,"num_object_copies":305106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153054,"num_write_kb":8152129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19847327744,"data_stored":19847315528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90854,"ondisk_log_size":90854,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":2508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":205,"num_read_kb":200,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201462,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2142976,"kb_used_data":1620332,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":91602176,"statfs":{"total":95995035648,"available":93800628224,"internally_reserved":0,"allocated":1659219968,"data_stored":1658841045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":535165386},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141072,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3372900,"kb_used_data":2930068,"kb_used_omap":4,"kb_used_meta":442811,"kb_avail":90372252,"statfs":{"total":95995035648,"available":92541186048,"internally_reserved":0,"allocated":3000389632,"data_stored":3000026766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":453439149},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692034,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2563636,"kb_used_data":2022944,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91181516,"statfs":{"total":95995035648,"available":93369872384,"internally_reserved":0,"allocated":2071494656,"data_stored":2071126317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920968,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3703752,"kb_used_data":3285240,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":90041400,"statfs":{"total":95995035648,"available":92202393600,"internally_reserved":0,"allocated":3364085760,"data_stored":3363693005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":428538310},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117199,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2615172,"kb_used_data":2006452,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":91129980,"statfs":{"total":95995035648,"available":93317099520,"internally_reserved":0,"allocated":2054606848,"data_stored":2054218885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":623311303},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411538,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2849956,"kb_used_data":2443924,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":90895196,"statfs":{"total":95995035648,"available":93076680704,"internally_reserved":0,"allocated":2502578176,"data_stored":2502221575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":415758790},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575065,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1767068,"kb_used_data":1226376,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91978084,"statfs":{"total":95995035648,"available":94185558016,"internally_reserved":0,"allocated":1255809024,"data_stored":1255400372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738592,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4697992,"kb_used_data":4026804,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":89047160,"statfs":{"total":95995035648,"available":91184291840,"internally_reserved":0,"allocated":4123447296,"data_stored":4123059914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4087222272,"data_stored":4087218200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1243873280,"data_stored":1243873280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2480013312,"data_stored":2480013312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2031550464,"data_stored":2031550464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3330150400,"data_stored":3330146328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2052259840,"data_stored":2052259840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2969899008,"data_stored":2969894936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1652359168,"data_stored":1652359168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:16:01.791 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:01.800 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:01 smithi052 ceph-mon[29471]: pgmap v737: 148 pgs: 148 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-13T13:16:01.800 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:01 smithi052 ceph-mon[29471]: from='client.26272 v1:172.21.15.52:0/2841433617' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:16:01.801 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:01 smithi052 ceph-mon[29471]: 3.1a scrub starts 2024-09-13T13:16:01.801 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:01 smithi052 ceph-mon[29471]: 3.1a scrub ok 2024-09-13T13:16:01.844 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:01.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:01.921 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:01.942 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:16:01.946 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:01.999 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:02.004 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:02.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:02.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:02.062 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:01 smithi052 ceph-mon[36682]: pgmap v737: 148 pgs: 148 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-13T13:16:02.062 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:01 smithi052 ceph-mon[36682]: from='client.26272 v1:172.21.15.52:0/2841433617' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:16:02.062 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:01 smithi052 ceph-mon[36682]: 3.1a scrub starts 2024-09-13T13:16:02.062 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:01 smithi052 ceph-mon[36682]: 3.1a scrub ok 2024-09-13T13:16:02.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:01 smithi167 ceph-mon[33178]: pgmap v737: 148 pgs: 148 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-13T13:16:02.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:01 smithi167 ceph-mon[33178]: from='client.26272 v1:172.21.15.52:0/2841433617' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:16:02.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:01 smithi167 ceph-mon[33178]: 3.1a scrub starts 2024-09-13T13:16:02.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:01 smithi167 ceph-mon[33178]: 3.1a scrub ok 2024-09-13T13:16:02.106 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:02.174 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:02.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:02.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:02.276 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:02.309 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:02.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:02.410 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:02.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:02.458 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:02.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:02.560 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:02.589 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:02.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:02.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:02.690 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:02.713 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:16:02.714 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:16:02.716 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:02.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:02.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:02.819 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:02.854 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:02.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:02.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:02.956 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:02.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:02 smithi052 ceph-mon[36682]: pgmap v738: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-13T13:16:02.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:02 smithi052 ceph-mon[29471]: pgmap v738: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-13T13:16:02.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:02.992 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:02.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:03.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:02 smithi167 ceph-mon[33178]: pgmap v738: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-13T13:16:03.093 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:03.120 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:03.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:03.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:03.252 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:03.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:03.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:03.357 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:03.393 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:03.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:03.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:03.495 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:03.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:03.532 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:03.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:03.534 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T13:16:03.634 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:03.688 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:03.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:03.789 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:03.831 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:16:03.831 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:16:03.847 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:03.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:03.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:03.868 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:16:03.868 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:16:03.918 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:16:03.918 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:16:03.923 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:03.949 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:03.958 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:16:03.958 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:16:03.995 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:16:03.995 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:16:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:04.019 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:04.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:04.032 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:16:04.032 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:16:04.065 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:16:04.065 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:16:04.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:04 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:16:04.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:04 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:16:04.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:04 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:04.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:04 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:16:04.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:04 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1491765247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:04.099 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:04 smithi052 ceph-mon[29471]: osdmap e373: 8 total, 8 up, 8 in 2024-09-13T13:16:04.100 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:04 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:16:04.100 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:04 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:16:04.100 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:04 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:04.100 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:04 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:16:04.100 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:04 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1491765247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:04.100 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:04 smithi052 ceph-mon[36682]: osdmap e373: 8 total, 8 up, 8 in 2024-09-13T13:16:04.100 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:16:04.101 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:16:04.120 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:04.169 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:04.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:04.271 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:04.303 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:04.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:04.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:04 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:16:04.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:04 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:16:04.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:04 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:04 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:16:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:04 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1491765247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:04 smithi167 ceph-mon[33178]: osdmap e373: 8 total, 8 up, 8 in 2024-09-13T13:16:04.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:04.444 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:04.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:04.545 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:04.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:04.681 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:04.724 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:04.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:04.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:16:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:16:04.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:04.855 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:04.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:04.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:04.956 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:04.993 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:04.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:05.097 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:05.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:05.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:05.226 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:05.258 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:05.259 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:05 smithi052 ceph-mon[29471]: pgmap v740: 148 pgs: 148 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-13T13:16:05.259 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:05.259 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:05 smithi052 ceph-mon[36682]: pgmap v740: 148 pgs: 148 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-13T13:16:05.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:05.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:05.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:05.301 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:05.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:05 smithi167 ceph-mon[33178]: pgmap v740: 148 pgs: 148 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-13T13:16:05.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:05.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:05.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:05.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:05.501 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:05.547 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:05.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:05.648 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:05.689 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:05.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:05.790 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:05.821 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:05.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:05.923 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:05.967 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:16:05.967 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:16:05.967 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:16:05.967 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:16:05.968 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:16:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:05.972 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:05.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:06.031 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:16:06.073 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:06.105 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:06.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:06.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:06.207 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:06.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[36682]: 3.5 scrub starts 2024-09-13T13:16:06.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[36682]: 3.5 scrub ok 2024-09-13T13:16:06.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/384412481' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:16:06.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[36682]: from='client.36347 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:16:06.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[36682]: 10.1 scrub starts 2024-09-13T13:16:06.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[36682]: 4.0 scrub starts 2024-09-13T13:16:06.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[36682]: 4.0 scrub ok 2024-09-13T13:16:06.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:16:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:16:06.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[29471]: 3.5 scrub starts 2024-09-13T13:16:06.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[29471]: 3.5 scrub ok 2024-09-13T13:16:06.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/384412481' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:16:06.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[29471]: from='client.36347 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:16:06.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[29471]: 10.1 scrub starts 2024-09-13T13:16:06.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[29471]: 4.0 scrub starts 2024-09-13T13:16:06.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:06 smithi052 ceph-mon[29471]: 4.0 scrub ok 2024-09-13T13:16:06.242 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:06.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:06.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:06.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:06 smithi167 ceph-mon[33178]: 3.5 scrub starts 2024-09-13T13:16:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:06 smithi167 ceph-mon[33178]: 3.5 scrub ok 2024-09-13T13:16:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:06 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/384412481' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:16:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:06 smithi167 ceph-mon[33178]: from='client.36347 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:16:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:06 smithi167 ceph-mon[33178]: 10.1 scrub starts 2024-09-13T13:16:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:06 smithi167 ceph-mon[33178]: 4.0 scrub starts 2024-09-13T13:16:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:06 smithi167 ceph-mon[33178]: 4.0 scrub ok 2024-09-13T13:16:06.345 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:06.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:06.380 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:06.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:06.481 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:06.516 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:06.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:06.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:06.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:06.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:06.653 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:06.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:06.752 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:16:06.753 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:16:06.753 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-13T13:16:06.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd out 2 2024-09-13T13:16:06.754 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:06.782 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:06.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:06.801 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T13:16:06.883 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:06.908 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:06.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:07.010 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:07.016 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:07 smithi052 ceph-mon[29471]: 6.12 scrub starts 2024-09-13T13:16:07.017 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:07 smithi052 ceph-mon[29471]: 6.12 scrub ok 2024-09-13T13:16:07.017 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:07 smithi052 ceph-mon[29471]: pgmap v741: 148 pgs: 148 active+clean; 5.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 979 op/s 2024-09-13T13:16:07.017 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:07 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:07.017 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:07 smithi052 ceph-mon[29471]: from='client.36347 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:16:07.017 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:07 smithi052 ceph-mon[29471]: osdmap e374: 8 total, 8 up, 8 in 2024-09-13T13:16:07.098 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:07.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:07.101 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:16:07.200 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:07.247 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:07.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:07.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:07.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:07 smithi167 ceph-mon[33178]: 6.12 scrub starts 2024-09-13T13:16:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:07 smithi167 ceph-mon[33178]: 6.12 scrub ok 2024-09-13T13:16:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:07 smithi167 ceph-mon[33178]: pgmap v741: 148 pgs: 148 active+clean; 5.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 979 op/s 2024-09-13T13:16:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:07 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:07 smithi167 ceph-mon[33178]: from='client.36347 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:16:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:07 smithi167 ceph-mon[33178]: osdmap e374: 8 total, 8 up, 8 in 2024-09-13T13:16:07.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:07.375 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:07.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:07.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:07.477 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:07 smithi052 ceph-mon[36682]: 6.12 scrub starts 2024-09-13T13:16:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:07 smithi052 ceph-mon[36682]: 6.12 scrub ok 2024-09-13T13:16:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:07 smithi052 ceph-mon[36682]: pgmap v741: 148 pgs: 148 active+clean; 5.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 979 op/s 2024-09-13T13:16:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:07 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:07 smithi052 ceph-mon[36682]: from='client.36347 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:16:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:07 smithi052 ceph-mon[36682]: osdmap e374: 8 total, 8 up, 8 in 2024-09-13T13:16:07.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:07.506 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:07.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:07.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:07.633 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:07.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:07.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:07.730 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:07.735 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:07.741 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:07.761 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:07.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:07.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:07.803 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:07.864 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:07.897 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:07.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:07.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:08.036 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:08.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:08.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:08.138 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:08.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:08.170 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:08.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:08.271 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:08.307 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:08.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:08.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:08.410 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:08.437 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:08.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:08.539 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:08.576 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:16:08.576 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:16:08.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:16:08.577 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:16:08.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.577 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:116255 9.8 e275/241 9:16d2e4bc:::benchmark_data_smithi052_128427_object116254:head v 275'7312, mlcod=275'7310)", 2024-09-13T13:16:08.577 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:33.062682+0000", 2024-09-13T13:16:08.577 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 275.50887342300001, 2024-09-13T13:16:08.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.166173769, 2024-09-13T13:16:08.577 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.577 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.577 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.577 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062682+0000", 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062682+0000", 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062685+0000", 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3360000000000002e-06 2024-09-13T13:16:08.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062712+0000", 2024-09-13T13:16:08.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7667000000000001e-05 2024-09-13T13:16:08.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062729+0000", 2024-09-13T13:16:08.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6529e-05 2024-09-13T13:16:08.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062738+0000", 2024-09-13T13:16:08.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6600000000000007e-06 2024-09-13T13:16:08.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062830+0000", 2024-09-13T13:16:08.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1796e-05 2024-09-13T13:16:08.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.062852+0000", 2024-09-13T13:16:08.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2166000000000001e-05 2024-09-13T13:16:08.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.228807+0000", 2024-09-13T13:16:08.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.165954191 2024-09-13T13:16:08.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:33.228856+0000", 2024-09-13T13:16:08.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9424e-05 2024-09-13T13:16:08.581 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.582 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.582 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.582 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:169623 9.9 9:94682d60:::benchmark_data_smithi052_128427_object38954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:16:08.582 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.408270+0000", 2024-09-13T13:16:08.583 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 226.16328585799999, 2024-09-13T13:16:08.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.175397626, 2024-09-13T13:16:08.583 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.583 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.583 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.583 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.583 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:08.583 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:08.583 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 169623 2024-09-13T13:16:08.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.583 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408270+0000", 2024-09-13T13:16:08.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408270+0000", 2024-09-13T13:16:08.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408272+0000", 2024-09-13T13:16:08.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.136e-06 2024-09-13T13:16:08.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408278+0000", 2024-09-13T13:16:08.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7400000000000001e-06 2024-09-13T13:16:08.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408289+0000", 2024-09-13T13:16:08.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1712999999999999e-05 2024-09-13T13:16:08.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408297+0000", 2024-09-13T13:16:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8140000000000001e-06 2024-09-13T13:16:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408367+0000", 2024-09-13T13:16:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0221000000000004e-05 2024-09-13T13:16:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408487+0000", 2024-09-13T13:16:08.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000119109 2024-09-13T13:16:08.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:16:08.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.408598+0000", 2024-09-13T13:16:08.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011178799999999999 2024-09-13T13:16:08.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413949+0000", 2024-09-13T13:16:08.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053510659999999998 2024-09-13T13:16:08.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413997+0000", 2024-09-13T13:16:08.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7862999999999999e-05 2024-09-13T13:16:08.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583426+0000", 2024-09-13T13:16:08.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16942890199999999 2024-09-13T13:16:08.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.591 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583613+0000", 2024-09-13T13:16:08.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000186503 2024-09-13T13:16:08.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.591 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.591 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583667+0000", 2024-09-13T13:16:08.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4771000000000001e-05 2024-09-13T13:16:08.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.591 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.591 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:169631 9.9 9:9385c360:::benchmark_data_smithi052_128427_object38962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.413293+0000", 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 226.15826302400001, 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.17070595899999999, 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 169631 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.592 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413293+0000", 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413293+0000", 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413295+0000", 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.12e-06 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.593 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.594 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413300+0000", 2024-09-13T13:16:08.594 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.853e-06 2024-09-13T13:16:08.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.594 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413318+0000", 2024-09-13T13:16:08.594 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8155000000000002e-05 2024-09-13T13:16:08.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.594 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413325+0000", 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0149999999999998e-06 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.414485+0000", 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001159818 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.414595+0000", 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110302 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.595 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:16:08.596 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.414702+0000", 2024-09-13T13:16:08.596 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000106721 2024-09-13T13:16:08.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.596 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.597 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583409+0000", 2024-09-13T13:16:08.597 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16870703400000001 2024-09-13T13:16:08.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.597 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.597 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583721+0000", 2024-09-13T13:16:08.597 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00031262299999999999 2024-09-13T13:16:08.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.597 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.597 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583768+0000", 2024-09-13T13:16:08.597 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6217999999999997e-05 2024-09-13T13:16:08.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.598 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.598 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.598 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583929+0000", 2024-09-13T13:16:08.598 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016142899999999999 2024-09-13T13:16:08.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.598 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.598 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.598 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583999+0000", 2024-09-13T13:16:08.598 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9671000000000004e-05 2024-09-13T13:16:08.598 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.598 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.598 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.599 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.599 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:169632 9.9 9:93968084:::benchmark_data_smithi052_128427_object38963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:16:08.599 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.413626+0000", 2024-09-13T13:16:08.599 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 226.15792971299999, 2024-09-13T13:16:08.599 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.170605964, 2024-09-13T13:16:08.599 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.599 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.599 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.599 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.599 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:08.599 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:08.599 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 169632 2024-09-13T13:16:08.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.600 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.600 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.600 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413626+0000", 2024-09-13T13:16:08.600 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.600 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.600 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413626+0000", 2024-09-13T13:16:08.600 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413627+0000", 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.305e-06 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413632+0000", 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9339999999999997e-06 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413642+0000", 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6409999999999997e-06 2024-09-13T13:16:08.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.602 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.602 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.413650+0000", 2024-09-13T13:16:08.602 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2600000000000005e-06 2024-09-13T13:16:08.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.602 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.602 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.418045+0000", 2024-09-13T13:16:08.602 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0043948060000000002 2024-09-13T13:16:08.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.602 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.602 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.418155+0000", 2024-09-13T13:16:08.603 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109876 2024-09-13T13:16:08.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.603 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:16:08.603 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.418269+0000", 2024-09-13T13:16:08.603 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114432 2024-09-13T13:16:08.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.603 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.603 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583710+0000", 2024-09-13T13:16:08.603 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16544082800000001 2024-09-13T13:16:08.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583726+0000", 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6370000000000001e-05 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584039+0000", 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00031283400000000001 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584188+0000", 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014875399999999999 2024-09-13T13:16:08.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.605 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.605 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.605 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584232+0000", 2024-09-13T13:16:08.605 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3924000000000002e-05 2024-09-13T13:16:08.605 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.605 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.605 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.605 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.605 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:169635 9.9 9:99294624:::benchmark_data_smithi052_128427_object38966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:16:08.605 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.415308+0000", 2024-09-13T13:16:08.605 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 226.15624787100001, 2024-09-13T13:16:08.605 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.18454141299999999, 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 169635 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.415308+0000", 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.415308+0000", 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.415309+0000", 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2309999999999999e-06 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.415313+0000", 2024-09-13T13:16:08.607 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3930000000000001e-06 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.415322+0000", 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8999999999999995e-06 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.415329+0000", 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6780000000000003e-06 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.608 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.423993+0000", 2024-09-13T13:16:08.609 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0086634950000000002 2024-09-13T13:16:08.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.609 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.609 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.609 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424112+0000", 2024-09-13T13:16:08.609 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011977900000000001 2024-09-13T13:16:08.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.609 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.609 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:16:08.610 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.424232+0000", 2024-09-13T13:16:08.610 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012014 2024-09-13T13:16:08.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.610 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.610 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.610 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.583731+0000", 2024-09-13T13:16:08.610 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15949875199999999 2024-09-13T13:16:08.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.610 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.610 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.611 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584324+0000", 2024-09-13T13:16:08.611 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00059284099999999998 2024-09-13T13:16:08.611 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.611 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.611 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.611 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.599742+0000", 2024-09-13T13:16:08.611 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015417563 2024-09-13T13:16:08.611 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.611 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.611 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.611 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.599801+0000", 2024-09-13T13:16:08.611 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9497999999999999e-05 2024-09-13T13:16:08.612 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.612 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.612 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.612 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.599849+0000", 2024-09-13T13:16:08.612 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8143000000000003e-05 2024-09-13T13:16:08.612 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:169648 9.9 9:9769410b:::benchmark_data_smithi052_128427_object38979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:16:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:22.425234+0000", 2024-09-13T13:16:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 226.146321527, 2024-09-13T13:16:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.180310056, 2024-09-13T13:16:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 169648 2024-09-13T13:16:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425234+0000", 2024-09-13T13:16:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425234+0000", 2024-09-13T13:16:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425236+0000", 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9989999999999998e-06 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.425241+0000", 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0329999999999999e-06 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.582957+0000", 2024-09-13T13:16:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15771576000000001 2024-09-13T13:16:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.582975+0000", 2024-09-13T13:16:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8532000000000001e-05 2024-09-13T13:16:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584357+0000", 2024-09-13T13:16:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001381714 2024-09-13T13:16:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584454+0000", 2024-09-13T13:16:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7256000000000005e-05 2024-09-13T13:16:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,7", 2024-09-13T13:16:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.584554+0000", 2024-09-13T13:16:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9856000000000001e-05 2024-09-13T13:16:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.599901+0000", 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015346790000000001 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.599957+0000", 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5668999999999998e-05 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.605439+0000", 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0054818260000000004 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.605491+0000", 2024-09-13T13:16:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2278999999999999e-05 2024-09-13T13:16:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:22.605544+0000", 2024-09-13T13:16:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3341999999999999e-05 2024-09-13T13:16:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:41379 10.4 10:2eb0096b:::benchmark_data_smithi052_153703_object41378:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T13:16:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:24.369543+0000", 2024-09-13T13:16:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 104.202012266, 2024-09-13T13:16:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16524090399999999, 2024-09-13T13:16:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 41379 2024-09-13T13:16:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.621 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.621 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.369543+0000", 2024-09-13T13:16:08.621 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.621 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.621 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.621 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.369543+0000", 2024-09-13T13:16:08.621 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.621 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.369545+0000", 2024-09-13T13:16:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0310000000000001e-06 2024-09-13T13:16:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.369568+0000", 2024-09-13T13:16:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2237000000000001e-05 2024-09-13T13:16:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.369577+0000", 2024-09-13T13:16:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.781e-06 2024-09-13T13:16:08.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.623 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.623 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.623 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.369583+0000", 2024-09-13T13:16:08.623 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6019999999999999e-06 2024-09-13T13:16:08.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.623 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.623 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.623 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.370590+0000", 2024-09-13T13:16:08.623 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010069 2024-09-13T13:16:08.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.624 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.624 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.624 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.370714+0000", 2024-09-13T13:16:08.624 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012358000000000001 2024-09-13T13:16:08.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.624 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.624 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:16:08.624 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.370798+0000", 2024-09-13T13:16:08.624 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4515e-05 2024-09-13T13:16:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.372419+0000", 2024-09-13T13:16:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00162141 2024-09-13T13:16:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.373399+0000", 2024-09-13T13:16:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00097947599999999996 2024-09-13T13:16:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.626 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.626 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.534685+0000", 2024-09-13T13:16:08.627 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:08.627 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16128569500000001 2024-09-13T13:16:08.627 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.627 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.627 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.627 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.534741+0000", 2024-09-13T13:16:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6011000000000003e-05 2024-09-13T13:16:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:24.534784+0000", 2024-09-13T13:16:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3665999999999998e-05 2024-09-13T13:16:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133006 10.8 10:1945bf5a:::benchmark_data_smithi052_153703_object5626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:16:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.287772+0000", 2024-09-13T13:16:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.283783362000001, 2024-09-13T13:16:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.372937727, 2024-09-13T13:16:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133006 2024-09-13T13:16:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.287772+0000", 2024-09-13T13:16:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.287772+0000", 2024-09-13T13:16:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.287775+0000", 2024-09-13T13:16:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2960000000000001e-06 2024-09-13T13:16:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.287782+0000", 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8700000000000003e-06 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.287797+0000", 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5159e-05 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.287804+0000", 2024-09-13T13:16:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8059999999999995e-06 2024-09-13T13:16:08.632 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.632 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.632 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.632 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.288282+0000", 2024-09-13T13:16:08.632 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00047746099999999999 2024-09-13T13:16:08.632 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.633 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.633 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.633 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.288400+0000", 2024-09-13T13:16:08.633 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118169 2024-09-13T13:16:08.633 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.633 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.633 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:16:08.633 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.288514+0000", 2024-09-13T13:16:08.633 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114286 2024-09-13T13:16:08.633 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.633 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.633 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.633 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291531+0000", 2024-09-13T13:16:08.634 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0030161659999999998 2024-09-13T13:16:08.634 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.634 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.634 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.634 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302056+0000", 2024-09-13T13:16:08.634 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010525097000000001 2024-09-13T13:16:08.634 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.634 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.634 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.634 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.660629+0000", 2024-09-13T13:16:08.634 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.35857315699999998 2024-09-13T13:16:08.634 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.634 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.660674+0000", 2024-09-13T13:16:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4702e-05 2024-09-13T13:16:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.660710+0000", 2024-09-13T13:16:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6557999999999998e-05 2024-09-13T13:16:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133009 10.8 10:1c6c448b:::benchmark_data_smithi052_153703_object5629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:16:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.291955+0000", 2024-09-13T13:16:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.279600391999999, 2024-09-13T13:16:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.38164577399999999, 2024-09-13T13:16:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133009 2024-09-13T13:16:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.637 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.637 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.637 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291955+0000", 2024-09-13T13:16:08.637 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.637 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.637 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.637 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.637 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291955+0000", 2024-09-13T13:16:08.637 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.637 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.638 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.638 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.638 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291958+0000", 2024-09-13T13:16:08.638 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2089999999999999e-06 2024-09-13T13:16:08.638 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.638 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.638 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.638 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291964+0000", 2024-09-13T13:16:08.638 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0650000000000004e-06 2024-09-13T13:16:08.638 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.638 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.638 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.638 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291975+0000", 2024-09-13T13:16:08.639 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1916000000000001e-05 2024-09-13T13:16:08.639 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.639 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.639 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.639 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291983+0000", 2024-09-13T13:16:08.639 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0840000000000003e-06 2024-09-13T13:16:08.639 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.639 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.639 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.639 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.292037+0000", 2024-09-13T13:16:08.639 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4836999999999997e-05 2024-09-13T13:16:08.639 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.639 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.640 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.640 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.292157+0000", 2024-09-13T13:16:08.640 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000119206 2024-09-13T13:16:08.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.640 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.640 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:16:08.640 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.292287+0000", 2024-09-13T13:16:08.640 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013012500000000001 2024-09-13T13:16:08.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.640 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.296788+0000", 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045008069999999999 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302358+0000", 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055702039999999996 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.673484+0000", 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.371126542 2024-09-13T13:16:08.641 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.642 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.642 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.642 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.673549+0000", 2024-09-13T13:16:08.642 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4498000000000005e-05 2024-09-13T13:16:08.642 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.642 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.642 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.642 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.673601+0000", 2024-09-13T13:16:08.642 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2281000000000003e-05 2024-09-13T13:16:08.642 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.642 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.642 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.642 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.643 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:08.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:08.644 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133021 10.8 10:11f4c5d2:::benchmark_data_smithi052_153703_object5641:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:16:08.644 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.302866+0000", 2024-09-13T13:16:08.645 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.268689383999998, 2024-09-13T13:16:08.645 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.37089920399999998, 2024-09-13T13:16:08.645 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.645 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.645 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.645 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.645 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.645 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.645 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133021 2024-09-13T13:16:08.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.645 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.645 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.645 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.646 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302866+0000", 2024-09-13T13:16:08.646 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.646 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.646 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.646 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302866+0000", 2024-09-13T13:16:08.646 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.646 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.646 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.646 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302867+0000", 2024-09-13T13:16:08.646 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3200000000000004e-07 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302871+0000", 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4570000000000001e-06 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302876+0000", 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9349999999999999e-06 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.647 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302882+0000", 2024-09-13T13:16:08.648 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2669999999999997e-06 2024-09-13T13:16:08.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.648 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.648 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.648 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302927+0000", 2024-09-13T13:16:08.648 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5624000000000002e-05 2024-09-13T13:16:08.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.648 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.648 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.648 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.303052+0000", 2024-09-13T13:16:08.648 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012477699999999999 2024-09-13T13:16:08.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.648 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.649 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:16:08.649 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.303152+0000", 2024-09-13T13:16:08.649 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000100256 2024-09-13T13:16:08.649 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.649 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.649 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.649 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.310146+0000", 2024-09-13T13:16:08.649 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0069940749999999998 2024-09-13T13:16:08.649 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.649 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.649 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.649 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.314157+0000", 2024-09-13T13:16:08.649 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0040102250000000001 2024-09-13T13:16:08.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.650 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.650 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.650 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.673676+0000", 2024-09-13T13:16:08.650 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.35951962999999998 2024-09-13T13:16:08.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.650 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.650 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.650 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.673728+0000", 2024-09-13T13:16:08.650 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1481999999999999e-05 2024-09-13T13:16:08.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.650 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.650 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.651 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.673766+0000", 2024-09-13T13:16:08.651 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7744000000000003e-05 2024-09-13T13:16:08.651 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.651 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.651 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.651 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.651 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133022 10.8 10:1fafbb00:::benchmark_data_smithi052_153703_object5642:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:16:08.651 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.305318+0000", 2024-09-13T13:16:08.651 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.266237428, 2024-09-13T13:16:08.651 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.37413038399999998, 2024-09-13T13:16:08.651 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.651 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.652 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.652 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.652 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.652 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.652 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133022 2024-09-13T13:16:08.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.652 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.652 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.652 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.652 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.305318+0000", 2024-09-13T13:16:08.652 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.652 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.653 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.653 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.305318+0000", 2024-09-13T13:16:08.653 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.653 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.653 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.305321+0000", 2024-09-13T13:16:08.653 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4669999999999998e-06 2024-09-13T13:16:08.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.653 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.653 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.305326+0000", 2024-09-13T13:16:08.653 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.22e-06 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.305338+0000", 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1765e-05 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.305345+0000", 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5290000000000003e-06 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.654 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.306501+0000", 2024-09-13T13:16:08.655 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011562359999999999 2024-09-13T13:16:08.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.655 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.655 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.655 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.306616+0000", 2024-09-13T13:16:08.655 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114815 2024-09-13T13:16:08.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.655 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.655 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:16:08.655 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.306690+0000", 2024-09-13T13:16:08.655 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3603000000000007e-05 2024-09-13T13:16:08.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.655 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.314145+0000", 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0074549120000000002 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.314161+0000", 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6424999999999999e-05 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.679280+0000", 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.36511832700000002 2024-09-13T13:16:08.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.657 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.657 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.679402+0000", 2024-09-13T13:16:08.657 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000122399 2024-09-13T13:16:08.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.657 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.657 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.679449+0000", 2024-09-13T13:16:08.657 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6686e-05 2024-09-13T13:16:08.657 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.657 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.657 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133025 10.8 10:1f5a4099:::benchmark_data_smithi052_153703_object5645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.308577+0000", 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.262979127000001, 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.37100242, 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133025 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.658 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.659 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.659 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.659 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.308577+0000", 2024-09-13T13:16:08.659 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.659 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.659 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.659 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.308577+0000", 2024-09-13T13:16:08.659 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.659 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.659 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.659 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.308579+0000", 2024-09-13T13:16:08.660 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0439999999999998e-06 2024-09-13T13:16:08.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.660 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.660 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.308584+0000", 2024-09-13T13:16:08.660 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3850000000000001e-06 2024-09-13T13:16:08.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.660 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.660 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.308595+0000", 2024-09-13T13:16:08.660 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1073999999999999e-05 2024-09-13T13:16:08.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.308673+0000", 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8386000000000005e-05 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.310177+0000", 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0015036279999999999 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.310276+0000", 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9358000000000007e-05 2024-09-13T13:16:08.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.662 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.662 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:16:08.662 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.310380+0000", 2024-09-13T13:16:08.662 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103349 2024-09-13T13:16:08.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.662 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.662 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.662 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.316371+0000", 2024-09-13T13:16:08.662 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0059909020000000002 2024-09-13T13:16:08.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.662 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.662 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.662 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.321955+0000", 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055840250000000003 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.679487+0000", 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.35753211000000001 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.679545+0000", 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8440999999999998e-05 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.663 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.679579+0000", 2024-09-13T13:16:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3717999999999998e-05 2024-09-13T13:16:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133039 10.8 10:1d3cc177:::benchmark_data_smithi052_153703_object5659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:16:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.317059+0000", 2024-09-13T13:16:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.254496991, 2024-09-13T13:16:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.36894714499999998, 2024-09-13T13:16:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133039 2024-09-13T13:16:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317059+0000", 2024-09-13T13:16:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317059+0000", 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317081+0000", 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2044000000000001e-05 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317087+0000", 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2550000000000003e-06 2024-09-13T13:16:08.666 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317102+0000", 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5398000000000001e-05 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317110+0000", 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1339999999999998e-06 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317163+0000", 2024-09-13T13:16:08.667 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3170000000000001e-05 2024-09-13T13:16:08.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.668 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.668 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.668 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317278+0000", 2024-09-13T13:16:08.668 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000115739 2024-09-13T13:16:08.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.668 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.668 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:16:08.668 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317487+0000", 2024-09-13T13:16:08.668 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020869700000000001 2024-09-13T13:16:08.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.668 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.668 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.669 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.320968+0000", 2024-09-13T13:16:08.669 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0034804110000000001 2024-09-13T13:16:08.669 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.669 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.669 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.669 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.321999+0000", 2024-09-13T13:16:08.669 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010309749999999999 2024-09-13T13:16:08.669 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.669 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.669 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.669 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.685909+0000", 2024-09-13T13:16:08.669 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.36391061000000002 2024-09-13T13:16:08.669 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.685965+0000", 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5831999999999997e-05 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686006+0000", 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0880000000000002e-05 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.670 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.671 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133060 10.8 10:1fe5c61e:::benchmark_data_smithi052_153703_object5680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:16:08.671 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.336351+0000", 2024-09-13T13:16:08.671 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.235204333999999, 2024-09-13T13:16:08.671 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.349824423, 2024-09-13T13:16:08.671 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.671 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.671 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.671 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.671 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.671 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.671 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133060 2024-09-13T13:16:08.671 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.671 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336351+0000", 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336351+0000", 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336353+0000", 2024-09-13T13:16:08.672 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.796e-06 2024-09-13T13:16:08.673 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.673 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.673 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336358+0000", 2024-09-13T13:16:08.673 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.147e-06 2024-09-13T13:16:08.673 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.673 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.673 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336371+0000", 2024-09-13T13:16:08.673 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2359e-05 2024-09-13T13:16:08.673 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.673 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336378+0000", 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8510000000000001e-06 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336444+0000", 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6919000000000001e-05 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336557+0000", 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011230699999999999 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.674 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.675 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:16:08.675 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336668+0000", 2024-09-13T13:16:08.675 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000111459 2024-09-13T13:16:08.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.675 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.675 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.675 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.339885+0000", 2024-09-13T13:16:08.675 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0032169550000000001 2024-09-13T13:16:08.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.676 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.676 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.339996+0000", 2024-09-13T13:16:08.676 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110497 2024-09-13T13:16:08.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.676 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.676 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686049+0000", 2024-09-13T13:16:08.676 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34605333399999999 2024-09-13T13:16:08.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.676 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.677 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686137+0000", 2024-09-13T13:16:08.677 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8381999999999995e-05 2024-09-13T13:16:08.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.677 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.677 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.677 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686176+0000", 2024-09-13T13:16:08.677 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8417000000000002e-05 2024-09-13T13:16:08.677 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.677 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.677 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.677 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.677 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133065 10.8 10:12b65a36:::benchmark_data_smithi052_153703_object5685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.341093+0000", 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.230463035, 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34520810800000001, 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133065 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.678 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.679 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341093+0000", 2024-09-13T13:16:08.679 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.679 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.679 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.679 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341093+0000", 2024-09-13T13:16:08.679 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.679 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.679 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.679 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341094+0000", 2024-09-13T13:16:08.679 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.457e-06 2024-09-13T13:16:08.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.680 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.680 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.680 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341099+0000", 2024-09-13T13:16:08.680 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7690000000000004e-06 2024-09-13T13:16:08.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.680 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.680 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.680 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341110+0000", 2024-09-13T13:16:08.680 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1153e-05 2024-09-13T13:16:08.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.680 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.680 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.680 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341115+0000", 2024-09-13T13:16:08.681 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1909999999999999e-06 2024-09-13T13:16:08.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.681 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.681 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341158+0000", 2024-09-13T13:16:08.681 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2528999999999999e-05 2024-09-13T13:16:08.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.681 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.681 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341238+0000", 2024-09-13T13:16:08.681 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0149999999999997e-05 2024-09-13T13:16:08.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.682 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:16:08.682 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341308+0000", 2024-09-13T13:16:08.682 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9954000000000001e-05 2024-09-13T13:16:08.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.682 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.682 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.682 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.343428+0000", 2024-09-13T13:16:08.682 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0021204190000000001 2024-09-13T13:16:08.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.682 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.683 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.683 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.343759+0000", 2024-09-13T13:16:08.683 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033045600000000002 2024-09-13T13:16:08.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.683 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.683 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.683 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686214+0000", 2024-09-13T13:16:08.683 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34245503799999999 2024-09-13T13:16:08.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.683 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.683 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.683 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686267+0000", 2024-09-13T13:16:08.683 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3248000000000002e-05 2024-09-13T13:16:08.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.684 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.684 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.684 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686301+0000", 2024-09-13T13:16:08.684 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3744000000000001e-05 2024-09-13T13:16:08.684 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.684 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.684 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.684 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.684 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133068 10.8 10:107081bf:::benchmark_data_smithi052_153703_object5688:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:16:08.684 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.346485+0000", 2024-09-13T13:16:08.684 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.225070315, 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34413897900000001, 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133068 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346485+0000", 2024-09-13T13:16:08.685 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346485+0000", 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346487+0000", 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.767e-06 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.686 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346492+0000", 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8539999999999996e-06 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346502+0000", 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.533e-06 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346507+0000", 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1109999999999999e-06 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.687 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.688 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346558+0000", 2024-09-13T13:16:08.688 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1841000000000001e-05 2024-09-13T13:16:08.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.688 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.688 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.688 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346641+0000", 2024-09-13T13:16:08.688 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2157000000000006e-05 2024-09-13T13:16:08.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.688 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.688 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:16:08.688 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346718+0000", 2024-09-13T13:16:08.689 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7080000000000001e-05 2024-09-13T13:16:08.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.689 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.689 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.348847+0000", 2024-09-13T13:16:08.689 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0021295789999999999 2024-09-13T13:16:08.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.689 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.689 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.348979+0000", 2024-09-13T13:16:08.689 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013179000000000001 2024-09-13T13:16:08.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.690 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.690 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.690 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.690469+0000", 2024-09-13T13:16:08.690 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34149033299999998 2024-09-13T13:16:08.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.690 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.690 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.690 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.690568+0000", 2024-09-13T13:16:08.690 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8403e-05 2024-09-13T13:16:08.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.690 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.690 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.690624+0000", 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6530999999999998e-05 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:150470 10.8 10:1ddbf082:::benchmark_data_smithi052_153703_object23090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:55.067217+0000", 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 13.504338476000001, 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.206420829, 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.691 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.692 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.692 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.692 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.692 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 150470 2024-09-13T13:16:08.692 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.692 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.692 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.692 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.692 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.067217+0000", 2024-09-13T13:16:08.692 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.692 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.692 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.692 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.067217+0000", 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.067220+0000", 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2879999999999999e-06 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.067225+0000", 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.446e-06 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.693 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.694 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.694 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.067241+0000", 2024-09-13T13:16:08.694 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5648999999999999e-05 2024-09-13T13:16:08.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.694 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.694 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.067248+0000", 2024-09-13T13:16:08.694 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.396e-06 2024-09-13T13:16:08.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.694 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.694 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.067347+0000", 2024-09-13T13:16:08.694 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9400000000000004e-05 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.067456+0000", 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108386 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.067563+0000", 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107388 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.695 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.070370+0000", 2024-09-13T13:16:08.696 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0028072729999999999 2024-09-13T13:16:08.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.696 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.696 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.079263+0000", 2024-09-13T13:16:08.696 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0088922169999999991 2024-09-13T13:16:08.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.696 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.696 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.273517+0000", 2024-09-13T13:16:08.696 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.194254696 2024-09-13T13:16:08.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.697 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.697 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.273588+0000", 2024-09-13T13:16:08.697 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0480000000000003e-05 2024-09-13T13:16:08.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.697 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.697 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.273638+0000", 2024-09-13T13:16:08.697 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0210000000000002e-05 2024-09-13T13:16:08.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.697 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.698 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.698 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.698 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.698 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:150482 10.8 10:1e186be4:::benchmark_data_smithi052_153703_object23102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-13T13:16:08.698 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:55.080563+0000", 2024-09-13T13:16:08.698 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 13.490992339, 2024-09-13T13:16:08.698 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.20104017399999999, 2024-09-13T13:16:08.698 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.698 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.698 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.698 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.698 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.698 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 150482 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.080563+0000", 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.080563+0000", 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.699 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.080566+0000", 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6649999999999999e-06 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.080574+0000", 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9079999999999995e-06 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.080590+0000", 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6113e-05 2024-09-13T13:16:08.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.701 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.701 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.080599+0000", 2024-09-13T13:16:08.701 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2269999999999999e-06 2024-09-13T13:16:08.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.701 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.701 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.080675+0000", 2024-09-13T13:16:08.701 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5525000000000006e-05 2024-09-13T13:16:08.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.701 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.701 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.080817+0000", 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014243900000000001 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.080977+0000", 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000159612 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.083861+0000", 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002884508 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.702 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.703 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.087773+0000", 2024-09-13T13:16:08.703 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039116660000000003 2024-09-13T13:16:08.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.703 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.703 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.703 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.281547+0000", 2024-09-13T13:16:08.704 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19377368 2024-09-13T13:16:08.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.704 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.704 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.704 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.281578+0000", 2024-09-13T13:16:08.704 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1780999999999999e-05 2024-09-13T13:16:08.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.704 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.704 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.704 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.281604+0000", 2024-09-13T13:16:08.704 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5049999999999999e-05 2024-09-13T13:16:08.704 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.704 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.705 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.705 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:150497 10.8 10:115f525c:::benchmark_data_smithi052_153703_object23117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-13T13:16:08.705 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:55.095967+0000", 2024-09-13T13:16:08.705 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 13.475588215, 2024-09-13T13:16:08.705 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.196026266, 2024-09-13T13:16:08.705 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.705 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.705 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.705 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.705 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.705 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 150497 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.095967+0000", 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.095967+0000", 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.706 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.095971+0000", 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.805e-06 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.095981+0000", 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9830000000000008e-06 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.096007+0000", 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.55e-05 2024-09-13T13:16:08.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.708 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.708 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.096019+0000", 2024-09-13T13:16:08.708 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1911e-05 2024-09-13T13:16:08.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.708 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.708 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.096097+0000", 2024-09-13T13:16:08.708 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8046999999999993e-05 2024-09-13T13:16:08.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.708 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.708 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.096232+0000", 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013520000000000001 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.096368+0000", 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013591499999999999 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.100003+0000", 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036350240000000002 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.709 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.103912+0000", 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039089859999999997 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.291888+0000", 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.18797594300000001 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.291947+0000", 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8829999999999997e-05 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.711 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.711 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.291994+0000", 2024-09-13T13:16:08.711 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7122e-05 2024-09-13T13:16:08.711 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.711 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.711 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.711 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.711 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:150522 10.8 10:153c0f6d:::benchmark_data_smithi052_153703_object23142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-13T13:16:08.711 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:55.127462+0000", 2024-09-13T13:16:08.711 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 13.44409381, 2024-09-13T13:16:08.711 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16464926699999999, 2024-09-13T13:16:08.711 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:08.712 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:08.712 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:08.712 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:08.712 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:08.712 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:08.712 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 150522 2024-09-13T13:16:08.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.712 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:08.712 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.713 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:08.713 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.127462+0000", 2024-09-13T13:16:08.713 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.713 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.713 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:08.713 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.127462+0000", 2024-09-13T13:16:08.713 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:08.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.713 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.713 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.127464+0000", 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0660000000000002e-06 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.127470+0000", 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1330000000000004e-06 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.127483+0000", 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2636000000000001e-05 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.714 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.127491+0000", 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0779999999999996e-06 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.127563+0000", 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2519000000000001e-05 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.127732+0000", 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000168414 2024-09-13T13:16:08.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.127865+0000", 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000133497 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.130677+0000", 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002812246 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.131255+0000", 2024-09-13T13:16:08.716 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00057728800000000002 2024-09-13T13:16:08.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.717 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.717 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:08.717 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.292032+0000", 2024-09-13T13:16:08.717 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16077765999999999 2024-09-13T13:16:08.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.717 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.717 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:08.717 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.292078+0000", 2024-09-13T13:16:08.717 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5485000000000002e-05 2024-09-13T13:16:08.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:08.717 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:08.717 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:08.718 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:55.292111+0000", 2024-09-13T13:16:08.718 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3244999999999998e-05 2024-09-13T13:16:08.718 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.718 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.718 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.718 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:08.718 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:08.718 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:16:08.745 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:08.781 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:08.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:08.798 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:16:08.798 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:16:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:16:08.837 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:16:08.873 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:16:08.873 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:16:08.884 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:08.908 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:16:08.909 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:16:08.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:08.930 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:08.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:08.949 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:16:08.949 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:16:08.982 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:16:08.982 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:16:09.021 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:16:09.022 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:16:09.031 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:09.060 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:09.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:09.075 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:16:09.076 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:16:09.162 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:09.197 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:09.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:09.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:09.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:16:09.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:16:09.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[29471]: pgmap v743: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 980 op/s 2024-09-13T13:16:09.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:16:09.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:16:09.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[29471]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T13:16:09.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2244270160' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:16:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:16:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[36682]: pgmap v743: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 980 op/s 2024-09-13T13:16:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:16:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:16:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[36682]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T13:16:09.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:09 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2244270160' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:09.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:09 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:16:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:09 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:16:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:09 smithi167 ceph-mon[33178]: pgmap v743: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 980 op/s 2024-09-13T13:16:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:09 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:16:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:09 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:16:09.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:09 smithi167 ceph-mon[33178]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T13:16:09.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:09 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2244270160' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:09.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:09.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:09.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:09.431 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:09.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:09.465 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:09.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:09.566 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:09.591 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:09.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:09.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:09.601 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:16:09.693 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:09.741 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:09.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:09.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:09.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:09.877 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:09.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:09.979 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:10.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:10.005 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:10.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:10.106 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:10.137 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:10.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:10.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:10.239 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:10.279 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:10.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:10.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:10.381 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:10.423 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:10.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:10.514 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T13:16:10.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:10.557 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:10.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:10.658 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:10.692 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:10.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:10.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:10 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a[29467]: 2024-09-13T13:16:10.397+0000 7effa8af8640 -1 mon.a@0(leader).osd e375 definitely_dead 0 2024-09-13T13:16:10.793 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:10.836 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:10.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:10.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:10.958 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:10.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:11.057 INFO:teuthology.orchestra.run.smithi052.stderr:marked out osd.2. 2024-09-13T13:16:11.060 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:11.094 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:11.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:11.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:11.196 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:11.217 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:11.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:11.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:11.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:11 smithi052 ceph-mon[36682]: pgmap v745: 148 pgs: 148 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-13T13:16:11.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1602598538' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T13:16:11.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:11 smithi052 ceph-mon[36682]: from='client.36356 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T13:16:11.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:11 smithi052 ceph-mon[36682]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-13T13:16:11.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:11 smithi052 ceph-mon[29471]: pgmap v745: 148 pgs: 148 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-13T13:16:11.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1602598538' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T13:16:11.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:11 smithi052 ceph-mon[29471]: from='client.36356 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T13:16:11.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:11 smithi052 ceph-mon[29471]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-13T13:16:11.319 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:11.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:11 smithi167 ceph-mon[33178]: pgmap v745: 148 pgs: 148 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-13T13:16:11.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1602598538' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T13:16:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:11 smithi167 ceph-mon[33178]: from='client.36356 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T13:16:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:11 smithi167 ceph-mon[33178]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-13T13:16:11.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:11.341 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:11.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:11.343 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:11.443 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:11.466 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:11.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:11.567 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:11.590 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:11.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:11.691 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:11.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:11.714 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:11.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:11.815 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:11.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:11.840 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:11.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:11.941 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:11.964 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:11.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:11.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:12.065 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:12.097 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:12.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:12.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:12.191 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:12 smithi052 ceph-mon[29471]: from='client.36356 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-13T13:16:12.191 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:12 smithi052 ceph-mon[29471]: osdmap e376: 8 total, 8 up, 7 in 2024-09-13T13:16:12.191 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:12 smithi052 ceph-mon[36682]: from='client.36356 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-13T13:16:12.191 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:12 smithi052 ceph-mon[36682]: osdmap e376: 8 total, 8 up, 7 in 2024-09-13T13:16:12.199 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:12.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:12.222 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:12.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:12.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:12 smithi167 ceph-mon[33178]: from='client.36356 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-13T13:16:12.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:12 smithi167 ceph-mon[33178]: osdmap e376: 8 total, 8 up, 7 in 2024-09-13T13:16:12.324 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:12.346 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:12.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:12.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:12.448 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:12.472 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:12.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:12.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:12.489 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:12.574 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:12.596 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:12.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:12.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:12.698 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:12.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:12.722 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:12.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:12.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:12.849 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:12.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:12.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:12.930 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:16:12.930 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:16:12.950 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:12.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:12.974 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:12.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:13.058 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:16:13.074 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:13.113 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:13.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:13.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:13.161 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:16:13.184 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:13 smithi052 ceph-mon[36682]: pgmap v747: 148 pgs: 3 peering, 145 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-13T13:16:13.184 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:13 smithi052 ceph-mon[36682]: osdmap e377: 8 total, 8 up, 7 in 2024-09-13T13:16:13.184 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3039827794' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:13.185 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:13 smithi052 ceph-mon[36682]: from='client.36362 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:13.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:13 smithi052 ceph-mon[29471]: pgmap v747: 148 pgs: 3 peering, 145 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-13T13:16:13.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:13 smithi052 ceph-mon[29471]: osdmap e377: 8 total, 8 up, 7 in 2024-09-13T13:16:13.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3039827794' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:13.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:13 smithi052 ceph-mon[29471]: from='client.36362 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:13.214 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:13.245 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:13.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:13.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:13.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:13 smithi167 ceph-mon[33178]: pgmap v747: 148 pgs: 3 peering, 145 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-13T13:16:13.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:13 smithi167 ceph-mon[33178]: osdmap e377: 8 total, 8 up, 7 in 2024-09-13T13:16:13.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3039827794' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:13.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:13 smithi167 ceph-mon[33178]: from='client.36362 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:13.347 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:13.386 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:13.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:13.487 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:13.523 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:13.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:13.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:13.626 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:13.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:13.670 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:13.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:13.771 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:13.807 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:13.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:13.909 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:13.942 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:13.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:13.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:14.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:14.078 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:14.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:14.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:14.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:14.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:14.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:14.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:14 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:16:14.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:14 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:16:14.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:14 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:14.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:14 smithi167 ceph-mon[33178]: from='client.36362 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:16:14.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:14 smithi167 ceph-mon[33178]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 375'10495 2024-09-13T13:16:14.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:14 smithi167 ceph-mon[33178]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 375'10394 2024-09-13T13:16:14.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:14 smithi167 ceph-mon[33178]: osdmap e378: 8 total, 8 up, 7 in 2024-09-13T13:16:14.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:14 smithi167 ceph-mon[33178]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 375'10641 2024-09-13T13:16:14.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:14 smithi167 ceph-mon[33178]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 375'10516 2024-09-13T13:16:14.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:14 smithi167 ceph-mon[33178]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 375'10484 2024-09-13T13:16:14.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:14 smithi167 ceph-mon[33178]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 375'10440 2024-09-13T13:16:14.322 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:14.354 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:16:14.354 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:16:14.355 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:14.355 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[36682]: from='client.36362 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:16:14.355 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[36682]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 375'10495 2024-09-13T13:16:14.355 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[36682]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 375'10394 2024-09-13T13:16:14.355 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[36682]: osdmap e378: 8 total, 8 up, 7 in 2024-09-13T13:16:14.356 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[36682]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 375'10641 2024-09-13T13:16:14.356 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[36682]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 375'10516 2024-09-13T13:16:14.356 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[36682]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 375'10484 2024-09-13T13:16:14.356 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[36682]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 375'10440 2024-09-13T13:16:14.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:16:14.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:16:14.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:14.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[29471]: from='client.36362 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:16:14.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[29471]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 375'10495 2024-09-13T13:16:14.357 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[29471]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 375'10394 2024-09-13T13:16:14.358 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[29471]: osdmap e378: 8 total, 8 up, 7 in 2024-09-13T13:16:14.358 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[29471]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 375'10641 2024-09-13T13:16:14.358 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[29471]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 375'10516 2024-09-13T13:16:14.358 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[29471]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 375'10484 2024-09-13T13:16:14.358 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:14 smithi052 ceph-mon[29471]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 375'10440 2024-09-13T13:16:14.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:14.364 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:14.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:14.405 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T13:16:14.465 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:14.503 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:14.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:14.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:14.605 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:14.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:14.646 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:14.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:14.747 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:14.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:14.780 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:14.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:14.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:16:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:16:14.882 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:14.916 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:14.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:15.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:15.048 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:15.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:15.152 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:15.196 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:15.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:15.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:15.295 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:15.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:15.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:15 smithi167 ceph-mon[33178]: pgmap v750: 148 pgs: 2 active+remapped+backfilling, 3 peering, 1 active+remapped+backfill_toofull, 142 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 867 B/s rd, 0 B/s wr, 1.15k op/s; 16225/261903 objects misplaced (6.195%); 163 KiB/s, 5 objects/s recovering 2024-09-13T13:16:15.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:15 smithi167 ceph-mon[33178]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:16:15.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:15.330 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:15.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:15.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:15.432 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:15.433 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:15 smithi052 ceph-mon[29471]: pgmap v750: 148 pgs: 2 active+remapped+backfilling, 3 peering, 1 active+remapped+backfill_toofull, 142 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 867 B/s rd, 0 B/s wr, 1.15k op/s; 16225/261903 objects misplaced (6.195%); 163 KiB/s, 5 objects/s recovering 2024-09-13T13:16:15.433 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:15 smithi052 ceph-mon[29471]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:16:15.433 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:15.434 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:15 smithi052 ceph-mon[36682]: pgmap v750: 148 pgs: 2 active+remapped+backfilling, 3 peering, 1 active+remapped+backfill_toofull, 142 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 867 B/s rd, 0 B/s wr, 1.15k op/s; 16225/261903 objects misplaced (6.195%); 163 KiB/s, 5 objects/s recovering 2024-09-13T13:16:15.434 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:15 smithi052 ceph-mon[36682]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:16:15.434 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:15.467 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:15.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:15.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:15.569 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:15.602 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:15.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:15.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:15.704 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:15.737 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:15.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:15.839 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:15.874 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:15.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:15.976 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:16.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:16.012 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:16.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:16.076 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:16.113 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:16.151 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:16.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:16.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:16.152 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:16:16.153 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:16:16.181 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:16:16.182 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:16:16.216 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:16:16.217 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:16:16.244 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:16:16.244 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:16:16.252 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:16.282 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:16:16.282 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:16:16.291 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:16.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:16.319 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:16:16.320 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:16:16.320 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:16:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:16:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:16:16.360 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:16:16.360 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:16:16.389 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:16:16.390 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:16:16.393 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:16.427 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:16.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:16.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:16.529 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:16.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:16.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:16.670 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:16.707 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:16.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:16.809 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:16.844 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 4] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:16:16.845 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:16:16.845 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_3 2024-09-13T13:16:16.845 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_3 by 10 min 8 2024-09-13T13:16:16.845 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:16:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:16.847 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:16.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:16.881 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:16:16.881 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:16:16.881 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:16:16.881 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:16:16.882 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:16:16.948 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:16.994 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:16.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:17.096 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:17.135 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:17.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:17.237 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:17.276 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:17.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:17.279 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:16:17.354 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:17 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:16:17.354 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:17 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:16:17.355 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:17 smithi052 ceph-mon[36682]: pgmap v751: 148 pgs: 2 active+remapped+backfilling, 3 peering, 3 active+remapped+backfill_toofull, 140 active+clean; 5.3 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 685 op/s; 26809/259077 objects misplaced (10.348%); 160 KiB/s, 11 objects/s recovering 2024-09-13T13:16:17.355 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:17 smithi052 ceph-mon[36682]: osdmap e379: 8 total, 8 up, 7 in 2024-09-13T13:16:17.355 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:17 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3268132054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:17.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:17 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:16:17.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:17 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:16:17.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:17 smithi052 ceph-mon[29471]: pgmap v751: 148 pgs: 2 active+remapped+backfilling, 3 peering, 3 active+remapped+backfill_toofull, 140 active+clean; 5.3 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 685 op/s; 26809/259077 objects misplaced (10.348%); 160 KiB/s, 11 objects/s recovering 2024-09-13T13:16:17.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:17 smithi052 ceph-mon[29471]: osdmap e379: 8 total, 8 up, 7 in 2024-09-13T13:16:17.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:17 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3268132054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:17.379 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:17.430 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:17.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:17.532 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:17 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:16:17.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:17 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:16:17.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:17 smithi167 ceph-mon[33178]: pgmap v751: 148 pgs: 2 active+remapped+backfilling, 3 peering, 3 active+remapped+backfill_toofull, 140 active+clean; 5.3 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 685 op/s; 26809/259077 objects misplaced (10.348%); 160 KiB/s, 11 objects/s recovering 2024-09-13T13:16:17.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:17 smithi167 ceph-mon[33178]: osdmap e379: 8 total, 8 up, 7 in 2024-09-13T13:16:17.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:17 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3268132054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:17.572 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:17.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:17.673 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:17.705 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:17.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:17.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:17.807 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:17.847 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:17.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:17.949 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:17.984 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:17.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:18.086 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:18.127 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:18.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:18.228 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:18.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:18.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:18.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:18.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:18.382 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:18 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:16:18.382 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:18 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:16:18.382 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:18 smithi052 ceph-mon[36682]: osdmap e380: 8 total, 8 up, 7 in 2024-09-13T13:16:18.383 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:18 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:16:18.383 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:18 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:16:18.383 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:18 smithi052 ceph-mon[29471]: osdmap e380: 8 total, 8 up, 7 in 2024-09-13T13:16:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:18.416 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:18.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:18.425 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:18.437 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T13:16:18.472 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:18.516 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:18.558 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:18.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:18 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:16:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:18 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:16:18.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:18 smithi167 ceph-mon[33178]: osdmap e380: 8 total, 8 up, 7 in 2024-09-13T13:16:18.660 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:18.705 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:18.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:18.807 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:18.846 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:18.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:18.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:18.948 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:18.987 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:18.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:18.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:19.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:19.128 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:19.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:19.229 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:19.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:19.278 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:19.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:19.379 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:19.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:19.414 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:19.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:19.511 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:16:19.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:19.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:19.552 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:19.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:19.653 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:19.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:19.675 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:19.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:19.776 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:19.798 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:19.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:19.899 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:19.935 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:19.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:19.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:19 smithi052 ceph-mon[36682]: pgmap v754: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 5.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 695 B/s rd, 0 B/s wr, 594 op/s; 32056/254853 objects misplaced (12.578%); 163 KiB/s, 12 objects/s recovering 2024-09-13T13:16:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:19 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:19.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:19 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:16:19.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:19 smithi052 ceph-mon[29471]: pgmap v754: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 5.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 695 B/s rd, 0 B/s wr, 594 op/s; 32056/254853 objects misplaced (12.578%); 163 KiB/s, 12 objects/s recovering 2024-09-13T13:16:19.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:19 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:19.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:19 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:16:20.037 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:20.053 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:16:20.055 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:16:20.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:19 smithi167 ceph-mon[33178]: pgmap v754: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 5.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 695 B/s rd, 0 B/s wr, 594 op/s; 32056/254853 objects misplaced (12.578%); 163 KiB/s, 12 objects/s recovering 2024-09-13T13:16:20.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:19 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:20.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:19 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:16:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:20.093 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:20.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:20.194 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:20.233 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:20.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:20.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:20.335 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:20.365 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:20.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:20.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:20.467 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:20.501 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:20.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:20.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:20.603 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:20.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:20.635 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:20.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:20.736 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:20.766 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:20.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:20.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:20.868 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:20.898 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:20.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:21.000 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:21.028 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:20 smithi052 ceph-mon[29471]: from='client.16500 v1:172.21.15.52:0/3070240431' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:16:21.028 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:20 smithi052 ceph-mon[36682]: from='client.16500 v1:172.21.15.52:0/3070240431' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:16:21.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:21.038 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:21.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:21.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:20 smithi167 ceph-mon[33178]: from='client.16500 v1:172.21.15.52:0/3070240431' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:16:21.139 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:21.168 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:21.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:21.269 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:21.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:21.301 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:21.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:21.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:21.443 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:21.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:21.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:21.545 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:21.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:21.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:21.687 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:21.725 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:21.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:21.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:21.827 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:21.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:21.872 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:21.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:21.885 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:21.973 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:22.018 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:22.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:22.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:21 smithi167 ceph-mon[33178]: pgmap v755: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.04k op/s; 29122/243942 objects misplaced (11.938%); 10 MiB/s, 168 objects/s recovering 2024-09-13T13:16:22.119 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:22.158 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:22.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:22.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:22.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:21 smithi052 ceph-mon[36682]: pgmap v755: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.04k op/s; 29122/243942 objects misplaced (11.938%); 10 MiB/s, 168 objects/s recovering 2024-09-13T13:16:22.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:21 smithi052 ceph-mon[29471]: pgmap v755: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.04k op/s; 29122/243942 objects misplaced (11.938%); 10 MiB/s, 168 objects/s recovering 2024-09-13T13:16:22.260 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:22.286 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:22.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:22.388 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:22.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:22.411 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:22.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:22.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:22.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:22.538 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:22.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:22.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:22.676 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:22.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:22.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:22.778 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:22.797 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:16:22.818 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:22.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:22.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:22.920 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:22.961 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:22.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:22.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:23.063 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:23.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:22 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1760745210' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:16:23.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:22 smithi167 ceph-mon[33178]: from='client.36377 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:16:23.098 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:23.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:23.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:23.136 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:22 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1760745210' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:16:23.136 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:22 smithi052 ceph-mon[36682]: from='client.36377 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:16:23.137 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:22 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1760745210' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:16:23.137 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:22 smithi052 ceph-mon[29471]: from='client.36377 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:16:23.181 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":754,"stamp":"2024-09-13T13:16:17.940638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5554838067,"num_objects":84951,"num_object_clones":0,"num_object_copies":254853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":32056,"num_objects_unfound":0,"num_objects_dirty":84951,"num_whiteouts":0,"num_read":1461,"num_read_kb":1320,"num_write":170839,"num_write_kb":8153802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1578498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79055,"ondisk_log_size":79055,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":409,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":18248556,"kb_used_data":14421332,"kb_used_omap":13,"kb_used_meta":3827058,"kb_avail":637967508,"statfs":{"total":671965249536,"available":653278728192,"internally_reserved":0,"allocated":14767443968,"data_stored":14764509639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13997,"internal_metadata":3918907731},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-229113856,"num_objects":-3496,"num_object_clones":0,"num_object_copies":-10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":32056,"num_objects_unfound":0,"num_objects_dirty":-3496,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3496,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":984461,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.888493"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":684,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058254+0000","last_change":"2024-09-13T13:16:12.076433+0000","last_active":"2024-09-13T13:16:13.058254+0000","last_peered":"2024-09-13T13:16:13.058254+0000","last_clean":"2024-09-13T13:16:13.058254+0000","last_became_active":"2024-09-13T13:16:12.075424+0000","last_became_peered":"2024-09-13T13:16:12.075424+0000","last_unstale":"2024-09-13T13:16:13.058254+0000","last_undegraded":"2024-09-13T13:16:13.058254+0000","last_fullsized":"2024-09-13T13:16:13.058254+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.368588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.368588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":175,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055579+0000","last_change":"2024-09-13T13:15:04.344174+0000","last_active":"2024-09-13T13:16:13.055579+0000","last_peered":"2024-09-13T13:16:13.055579+0000","last_clean":"2024-09-13T13:16:13.055579+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:16:13.055579+0000","last_undegraded":"2024-09-13T13:16:13.055579+0000","last_fullsized":"2024-09-13T13:16:13.055579+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:15:04.344092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.344092+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":678,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055522+0000","last_change":"2024-09-13T13:14:12.865175+0000","last_active":"2024-09-13T13:16:13.055522+0000","last_peered":"2024-09-13T13:16:13.055522+0000","last_clean":"2024-09-13T13:16:13.055522+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:16:13.055522+0000","last_undegraded":"2024-09-13T13:16:13.055522+0000","last_fullsized":"2024-09-13T13:16:13.055522+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:12.865101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:12.865101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:17.268032+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":173,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057524+0000","last_change":"2024-09-13T13:15:51.244938+0000","last_active":"2024-09-13T13:16:13.057524+0000","last_peered":"2024-09-13T13:16:13.057524+0000","last_clean":"2024-09-13T13:16:13.057524+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:16:13.057524+0000","last_undegraded":"2024-09-13T13:16:13.057524+0000","last_fullsized":"2024-09-13T13:16:13.057524+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:19.110420+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":684,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075704+0000","last_change":"2024-09-13T13:11:47.852762+0000","last_active":"2024-09-13T13:16:13.075704+0000","last_peered":"2024-09-13T13:16:13.075704+0000","last_clean":"2024-09-13T13:16:13.075704+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:16:13.075704+0000","last_undegraded":"2024-09-13T13:16:13.075704+0000","last_fullsized":"2024-09-13T13:16:13.075704+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:15.241902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":712,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076637+0000","last_change":"2024-09-13T13:16:12.069748+0000","last_active":"2024-09-13T13:16:13.076637+0000","last_peered":"2024-09-13T13:16:13.076637+0000","last_clean":"2024-09-13T13:16:13.076637+0000","last_became_active":"2024-09-13T13:16:12.069442+0000","last_became_peered":"2024-09-13T13:16:12.069442+0000","last_unstale":"2024-09-13T13:16:13.076637+0000","last_undegraded":"2024-09-13T13:16:13.076637+0000","last_fullsized":"2024-09-13T13:16:13.076637+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:24.253369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:24.253369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:53.092969+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":771,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075833+0000","last_change":"2024-09-13T13:16:12.097652+0000","last_active":"2024-09-13T13:16:13.075833+0000","last_peered":"2024-09-13T13:16:13.075833+0000","last_clean":"2024-09-13T13:16:13.075833+0000","last_became_active":"2024-09-13T13:16:12.074619+0000","last_became_peered":"2024-09-13T13:16:12.074619+0000","last_unstale":"2024-09-13T13:16:13.075833+0000","last_undegraded":"2024-09-13T13:16:13.075833+0000","last_fullsized":"2024-09-13T13:16:13.075833+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:15:50.330607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.330607+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:09.731289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":157,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055489+0000","last_change":"2024-09-13T13:12:52.997865+0000","last_active":"2024-09-13T13:16:13.055489+0000","last_peered":"2024-09-13T13:16:13.055489+0000","last_clean":"2024-09-13T13:16:13.055489+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:16:13.055489+0000","last_undegraded":"2024-09-13T13:16:13.055489+0000","last_fullsized":"2024-09-13T13:16:13.055489+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:17.305608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:17.305608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:12.228270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":748,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057385+0000","last_change":"2024-09-13T13:13:41.518378+0000","last_active":"2024-09-13T13:16:13.057385+0000","last_peered":"2024-09-13T13:16:13.057385+0000","last_clean":"2024-09-13T13:16:13.057385+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:16:13.057385+0000","last_undegraded":"2024-09-13T13:16:13.057385+0000","last_fullsized":"2024-09-13T13:16:13.057385+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:41.518322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.518322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":167,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057524+0000","last_change":"2024-09-13T13:12:53.033861+0000","last_active":"2024-09-13T13:16:13.057524+0000","last_peered":"2024-09-13T13:16:13.057524+0000","last_clean":"2024-09-13T13:16:13.057524+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:16:13.057524+0000","last_undegraded":"2024-09-13T13:16:13.057524+0000","last_fullsized":"2024-09-13T13:16:13.057524+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'9","last_scrub_stamp":"2024-09-13T13:10:31.738685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:31.738685+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.301470+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":685,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.083933+0000","last_change":"2024-09-13T13:13:43.501368+0000","last_active":"2024-09-13T13:16:13.083933+0000","last_peered":"2024-09-13T13:16:13.083933+0000","last_clean":"2024-09-13T13:16:13.083933+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:16:13.083933+0000","last_undegraded":"2024-09-13T13:16:13.083933+0000","last_fullsized":"2024-09-13T13:16:13.083933+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:43.501206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:43.501206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:08.754967+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":688,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.072153+0000","last_change":"2024-09-13T13:15:25.628234+0000","last_active":"2024-09-13T13:16:13.072153+0000","last_peered":"2024-09-13T13:16:13.072153+0000","last_clean":"2024-09-13T13:16:13.072153+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:16:13.072153+0000","last_undegraded":"2024-09-13T13:16:13.072153+0000","last_fullsized":"2024-09-13T13:16:13.072153+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.628156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.628156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:41.988194+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":715,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056405+0000","last_change":"2024-09-13T13:12:52.951531+0000","last_active":"2024-09-13T13:16:13.056405+0000","last_peered":"2024-09-13T13:16:13.056405+0000","last_clean":"2024-09-13T13:16:13.056405+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:16:13.056405+0000","last_undegraded":"2024-09-13T13:16:13.056405+0000","last_fullsized":"2024-09-13T13:16:13.056405+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:16.989972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:16.989972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.371147+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":171,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056024+0000","last_change":"2024-09-13T13:15:03.348187+0000","last_active":"2024-09-13T13:16:13.056024+0000","last_peered":"2024-09-13T13:16:13.056024+0000","last_clean":"2024-09-13T13:16:13.056024+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:16:13.056024+0000","last_undegraded":"2024-09-13T13:16:13.056024+0000","last_fullsized":"2024-09-13T13:16:13.056024+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:15:03.348115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.348115+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.237423+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":706,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075391+0000","last_change":"2024-09-13T13:13:54.122282+0000","last_active":"2024-09-13T13:16:13.075391+0000","last_peered":"2024-09-13T13:16:13.075391+0000","last_clean":"2024-09-13T13:16:13.075391+0000","last_became_active":"2024-09-13T13:12:52.999970+0000","last_became_peered":"2024-09-13T13:12:52.999970+0000","last_unstale":"2024-09-13T13:16:13.075391+0000","last_undegraded":"2024-09-13T13:16:13.075391+0000","last_fullsized":"2024-09-13T13:16:13.075391+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.122222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.122222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:17.725218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":657,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.083326+0000","last_change":"2024-09-13T13:13:02.818644+0000","last_active":"2024-09-13T13:16:13.083326+0000","last_peered":"2024-09-13T13:16:13.083326+0000","last_clean":"2024-09-13T13:16:13.083326+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:16:13.083326+0000","last_undegraded":"2024-09-13T13:16:13.083326+0000","last_fullsized":"2024-09-13T13:16:13.083326+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:02.818543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:02.818543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.513000+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":161,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.070128+0000","last_change":"2024-09-13T13:12:53.014177+0000","last_active":"2024-09-13T13:16:13.070128+0000","last_peered":"2024-09-13T13:16:13.070128+0000","last_clean":"2024-09-13T13:16:13.070128+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:16:13.070128+0000","last_undegraded":"2024-09-13T13:16:13.070128+0000","last_fullsized":"2024-09-13T13:16:13.070128+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:10:28.804735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:28.804735+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:42.251197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":714,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.059139+0000","last_change":"2024-09-13T13:15:52.658477+0000","last_active":"2024-09-13T13:16:13.059139+0000","last_peered":"2024-09-13T13:16:13.059139+0000","last_clean":"2024-09-13T13:16:13.059139+0000","last_became_active":"2024-09-13T13:12:53.008505+0000","last_became_peered":"2024-09-13T13:12:53.008505+0000","last_unstale":"2024-09-13T13:16:13.059139+0000","last_undegraded":"2024-09-13T13:16:13.059139+0000","last_fullsized":"2024-09-13T13:16:13.059139+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:52.658410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:52.658410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:05.062147+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":532,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056608+0000","last_change":"2024-09-13T13:16:12.089425+0000","last_active":"2024-09-13T13:16:13.056608+0000","last_peered":"2024-09-13T13:16:13.056608+0000","last_clean":"2024-09-13T13:16:13.056608+0000","last_became_active":"2024-09-13T13:16:12.073838+0000","last_became_peered":"2024-09-13T13:16:12.073838+0000","last_unstale":"2024-09-13T13:16:13.056608+0000","last_undegraded":"2024-09-13T13:16:13.056608+0000","last_fullsized":"2024-09-13T13:16:13.056608+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:12:40.125545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.125545+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":708,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055506+0000","last_change":"2024-09-13T13:14:52.196086+0000","last_active":"2024-09-13T13:16:13.055506+0000","last_peered":"2024-09-13T13:16:13.055506+0000","last_clean":"2024-09-13T13:16:13.055506+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:16:13.055506+0000","last_undegraded":"2024-09-13T13:16:13.055506+0000","last_fullsized":"2024-09-13T13:16:13.055506+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.196041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.502726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":685,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076841+0000","last_change":"2024-09-13T13:13:03.150524+0000","last_active":"2024-09-13T13:16:13.076841+0000","last_peered":"2024-09-13T13:16:13.076841+0000","last_clean":"2024-09-13T13:16:13.076841+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:16:13.076841+0000","last_undegraded":"2024-09-13T13:16:13.076841+0000","last_fullsized":"2024-09-13T13:16:13.076841+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:03.150442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.150442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.295693+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":180,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057293+0000","last_change":"2024-09-13T13:16:12.077830+0000","last_active":"2024-09-13T13:16:13.057293+0000","last_peered":"2024-09-13T13:16:13.057293+0000","last_clean":"2024-09-13T13:16:13.057293+0000","last_became_active":"2024-09-13T13:16:12.077185+0000","last_became_peered":"2024-09-13T13:16:12.077185+0000","last_unstale":"2024-09-13T13:16:13.057293+0000","last_undegraded":"2024-09-13T13:16:13.057293+0000","last_fullsized":"2024-09-13T13:16:13.057293+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:01.242234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:01.242234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:21.091214+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":751,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076499+0000","last_change":"2024-09-13T13:13:41.164741+0000","last_active":"2024-09-13T13:16:13.076499+0000","last_peered":"2024-09-13T13:16:13.076499+0000","last_clean":"2024-09-13T13:16:13.076499+0000","last_became_active":"2024-09-13T13:12:52.953819+0000","last_became_peered":"2024-09-13T13:12:52.953819+0000","last_unstale":"2024-09-13T13:16:13.076499+0000","last_undegraded":"2024-09-13T13:16:13.076499+0000","last_fullsized":"2024-09-13T13:16:13.076499+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:13:41.164656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.164656+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:49.289309+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":174,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055884+0000","last_change":"2024-09-13T13:16:12.078175+0000","last_active":"2024-09-13T13:16:13.055884+0000","last_peered":"2024-09-13T13:16:13.055884+0000","last_clean":"2024-09-13T13:16:13.055884+0000","last_became_active":"2024-09-13T13:16:12.077603+0000","last_became_peered":"2024-09-13T13:16:12.077603+0000","last_unstale":"2024-09-13T13:16:13.055884+0000","last_undegraded":"2024-09-13T13:16:13.055884+0000","last_fullsized":"2024-09-13T13:16:13.055884+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:29.208576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:29.208576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:32.146934+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":203,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056161+0000","last_change":"2024-09-13T13:16:12.081390+0000","last_active":"2024-09-13T13:16:13.056161+0000","last_peered":"2024-09-13T13:16:13.056161+0000","last_clean":"2024-09-13T13:16:13.056161+0000","last_became_active":"2024-09-13T13:16:12.071603+0000","last_became_peered":"2024-09-13T13:16:12.071603+0000","last_unstale":"2024-09-13T13:16:13.056161+0000","last_undegraded":"2024-09-13T13:16:13.056161+0000","last_fullsized":"2024-09-13T13:16:13.056161+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:12:42.975266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:42.975266+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":709,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055960+0000","last_change":"2024-09-13T13:15:23.245116+0000","last_active":"2024-09-13T13:16:13.055960+0000","last_peered":"2024-09-13T13:16:13.055960+0000","last_clean":"2024-09-13T13:16:13.055960+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:16:13.055960+0000","last_undegraded":"2024-09-13T13:16:13.055960+0000","last_fullsized":"2024-09-13T13:16:13.055960+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:15:23.245047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.245047+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.758250+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":735,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058228+0000","last_change":"2024-09-13T13:15:50.693943+0000","last_active":"2024-09-13T13:16:13.058228+0000","last_peered":"2024-09-13T13:16:13.058228+0000","last_clean":"2024-09-13T13:16:13.058228+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:16:13.058228+0000","last_undegraded":"2024-09-13T13:16:13.058228+0000","last_fullsized":"2024-09-13T13:16:13.058228+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:15:50.693753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.693753+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:56.425256+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"306'12","reported_seq":858,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.080927+0000","last_change":"2024-09-13T13:14:21.530680+0000","last_active":"2024-09-13T13:16:16.080927+0000","last_peered":"2024-09-13T13:16:16.080927+0000","last_clean":"2024-09-13T13:16:16.080927+0000","last_became_active":"2024-09-13T13:12:52.947886+0000","last_became_peered":"2024-09-13T13:12:52.947886+0000","last_unstale":"2024-09-13T13:16:16.080927+0000","last_undegraded":"2024-09-13T13:16:16.080927+0000","last_fullsized":"2024-09-13T13:16:16.080927+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'12","last_scrub_stamp":"2024-09-13T13:14:21.530612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:21.530612+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.292277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"376'10571","reported_seq":10706,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057127+0000","last_change":"2024-09-13T13:16:06.040203+0000","last_active":"2024-09-13T13:16:13.057127+0000","last_peered":"2024-09-13T13:16:13.057127+0000","last_clean":"2024-09-13T13:16:13.057127+0000","last_became_active":"2024-09-13T13:13:59.149215+0000","last_became_peered":"2024-09-13T13:13:59.149215+0000","last_unstale":"2024-09-13T13:16:13.057127+0000","last_undegraded":"2024-09-13T13:16:13.057127+0000","last_fullsized":"2024-09-13T13:16:13.057127+0000","mapping_epoch":334,"log_start":"357'6000","ondisk_log_start":"357'6000","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":3199,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.029870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354091008,"num_objects":5403,"num_object_clones":0,"num_object_copies":16209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10571,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":779,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055679+0000","last_change":"2024-09-13T13:15:02.354689+0000","last_active":"2024-09-13T13:16:13.055679+0000","last_peered":"2024-09-13T13:16:13.055679+0000","last_clean":"2024-09-13T13:16:13.055679+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:16:13.055679+0000","last_undegraded":"2024-09-13T13:16:13.055679+0000","last_fullsized":"2024-09-13T13:16:13.055679+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:15:02.354591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:02.354591+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:22.985661+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":710,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075681+0000","last_change":"2024-09-13T13:15:29.205122+0000","last_active":"2024-09-13T13:16:13.075681+0000","last_peered":"2024-09-13T13:16:13.075681+0000","last_clean":"2024-09-13T13:16:13.075681+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:16:13.075681+0000","last_undegraded":"2024-09-13T13:16:13.075681+0000","last_fullsized":"2024-09-13T13:16:13.075681+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:29.205058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:29.205058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.374619+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":210,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.070171+0000","last_change":"2024-09-13T13:15:32.212185+0000","last_active":"2024-09-13T13:16:13.070171+0000","last_peered":"2024-09-13T13:16:13.070171+0000","last_clean":"2024-09-13T13:16:13.070171+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:16:13.070171+0000","last_undegraded":"2024-09-13T13:16:13.070171+0000","last_fullsized":"2024-09-13T13:16:13.070171+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:15:32.212101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.212101+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:33.253768+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":689,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056823+0000","last_change":"2024-09-13T13:13:57.546725+0000","last_active":"2024-09-13T13:16:13.056823+0000","last_peered":"2024-09-13T13:16:13.056823+0000","last_clean":"2024-09-13T13:16:13.056823+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:16:13.056823+0000","last_undegraded":"2024-09-13T13:16:13.056823+0000","last_fullsized":"2024-09-13T13:16:13.056823+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.546671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.546671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:18.300458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"378'10497","reported_seq":10605,"reported_epoch":378,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T13:16:13.219076+0000","last_change":"2024-09-13T13:16:13.193040+0000","last_active":"2024-09-13T13:16:13.219076+0000","last_peered":"2024-09-13T13:16:13.219076+0000","last_clean":"2024-09-13T13:16:11.048397+0000","last_became_active":"2024-09-13T13:16:13.156630+0000","last_became_peered":"2024-09-13T13:16:13.156630+0000","last_unstale":"2024-09-13T13:16:13.219076+0000","last_undegraded":"2024-09-13T13:16:13.219076+0000","last_fullsized":"2024-09-13T13:16:13.219076+0000","mapping_epoch":377,"log_start":"357'6210","ondisk_log_start":"357'6210","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359727104,"num_objects":5489,"num_object_clones":0,"num_object_copies":16467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5488,"num_objects_unfound":0,"num_objects_dirty":5489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10497,"num_write_kb":511552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,2],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"2,4,6","objects":5489}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":724,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075502+0000","last_change":"2024-09-13T13:16:05.289274+0000","last_active":"2024-09-13T13:16:13.075502+0000","last_peered":"2024-09-13T13:16:13.075502+0000","last_clean":"2024-09-13T13:16:13.075502+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:16:13.075502+0000","last_undegraded":"2024-09-13T13:16:13.075502+0000","last_fullsized":"2024-09-13T13:16:13.075502+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:16:05.289187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:05.289187+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:35.033329+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":684,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.077059+0000","last_change":"2024-09-13T13:13:54.313641+0000","last_active":"2024-09-13T13:16:13.077059+0000","last_peered":"2024-09-13T13:16:13.077059+0000","last_clean":"2024-09-13T13:16:13.077059+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:16:13.077059+0000","last_undegraded":"2024-09-13T13:16:13.077059+0000","last_fullsized":"2024-09-13T13:16:13.077059+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:54.313570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.313570+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:18.294176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":165,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058476+0000","last_change":"2024-09-13T13:15:00.323883+0000","last_active":"2024-09-13T13:16:13.058476+0000","last_peered":"2024-09-13T13:16:13.058476+0000","last_clean":"2024-09-13T13:16:13.058476+0000","last_became_active":"2024-09-13T13:12:53.006199+0000","last_became_peered":"2024-09-13T13:12:53.006199+0000","last_unstale":"2024-09-13T13:16:13.058476+0000","last_undegraded":"2024-09-13T13:16:13.058476+0000","last_fullsized":"2024-09-13T13:16:13.058476+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.323784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.323784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":180,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057649+0000","last_change":"2024-09-13T13:16:12.075699+0000","last_active":"2024-09-13T13:16:13.057649+0000","last_peered":"2024-09-13T13:16:13.057649+0000","last_clean":"2024-09-13T13:16:13.057649+0000","last_became_active":"2024-09-13T13:16:12.074280+0000","last_became_peered":"2024-09-13T13:16:12.074280+0000","last_unstale":"2024-09-13T13:16:13.057649+0000","last_undegraded":"2024-09-13T13:16:13.057649+0000","last_fullsized":"2024-09-13T13:16:13.057649+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:56.333778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:56.333778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"379'10687","reported_seq":10794,"reported_epoch":379,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T13:16:16.240478+0000","last_change":"2024-09-13T13:16:13.272305+0000","last_active":"2024-09-13T13:16:16.240478+0000","last_peered":"2024-09-13T13:16:16.240478+0000","last_clean":"2024-09-13T13:16:11.040381+0000","last_became_active":"2024-09-13T13:16:13.215045+0000","last_became_peered":"2024-09-13T13:16:13.215045+0000","last_unstale":"2024-09-13T13:16:16.240478+0000","last_undegraded":"2024-09-13T13:16:16.240478+0000","last_fullsized":"2024-09-13T13:16:16.240478+0000","mapping_epoch":377,"log_start":"355'5468","ondisk_log_start":"355'5468","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5219,"log_dups_size":0,"ondisk_log_size":5219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343867392,"num_objects":5247,"num_object_clones":0,"num_object_copies":15741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5247,"num_objects_unfound":0,"num_objects_dirty":5247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10687,"num_write_kb":509888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,2],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":5247}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":181,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058647+0000","last_change":"2024-09-13T13:13:50.716312+0000","last_active":"2024-09-13T13:16:13.058647+0000","last_peered":"2024-09-13T13:16:13.058647+0000","last_clean":"2024-09-13T13:16:13.058647+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:16:13.058647+0000","last_undegraded":"2024-09-13T13:16:13.058647+0000","last_fullsized":"2024-09-13T13:16:13.058647+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:13:50.716243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.716243+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":692,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058607+0000","last_change":"2024-09-13T13:13:57.886950+0000","last_active":"2024-09-13T13:16:13.058607+0000","last_peered":"2024-09-13T13:16:13.058607+0000","last_clean":"2024-09-13T13:16:13.058607+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:16:13.058607+0000","last_undegraded":"2024-09-13T13:16:13.058607+0000","last_fullsized":"2024-09-13T13:16:13.058607+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:57.886879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.886879+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":892,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075349+0000","last_change":"2024-09-13T13:13:55.316917+0000","last_active":"2024-09-13T13:16:13.075349+0000","last_peered":"2024-09-13T13:16:13.075349+0000","last_clean":"2024-09-13T13:16:13.075349+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:16:13.075349+0000","last_undegraded":"2024-09-13T13:16:13.075349+0000","last_fullsized":"2024-09-13T13:16:13.075349+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:13:55.316860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.316860+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.251395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":316,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.082448+0000","last_change":"2024-09-13T13:16:12.071941+0000","last_active":"2024-09-13T13:16:16.082448+0000","last_peered":"2024-09-13T13:16:16.082448+0000","last_clean":"2024-09-13T13:16:16.082448+0000","last_became_active":"2024-09-13T13:16:12.071629+0000","last_became_peered":"2024-09-13T13:16:12.071629+0000","last_unstale":"2024-09-13T13:16:16.082448+0000","last_undegraded":"2024-09-13T13:16:16.082448+0000","last_fullsized":"2024-09-13T13:16:16.082448+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.391418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.391418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:21.187177+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"376'10427","reported_seq":10527,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055821+0000","last_change":"2024-09-13T13:13:59.149331+0000","last_active":"2024-09-13T13:16:13.055821+0000","last_peered":"2024-09-13T13:16:13.055821+0000","last_clean":"2024-09-13T13:16:13.055821+0000","last_became_active":"2024-09-13T13:13:59.149046+0000","last_became_peered":"2024-09-13T13:13:59.149046+0000","last_unstale":"2024-09-13T13:16:13.055821+0000","last_undegraded":"2024-09-13T13:16:13.055821+0000","last_fullsized":"2024-09-13T13:16:13.055821+0000","mapping_epoch":334,"log_start":"356'5800","ondisk_log_start":"356'5800","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348323840,"num_objects":5315,"num_object_clones":0,"num_object_copies":15945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10427,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":751,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055960+0000","last_change":"2024-09-13T13:12:53.016127+0000","last_active":"2024-09-13T13:16:13.055960+0000","last_peered":"2024-09-13T13:16:13.055960+0000","last_clean":"2024-09-13T13:16:13.055960+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:16:13.055960+0000","last_undegraded":"2024-09-13T13:16:13.055960+0000","last_fullsized":"2024-09-13T13:16:13.055960+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.376603+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":725,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057474+0000","last_change":"2024-09-13T13:16:12.067616+0000","last_active":"2024-09-13T13:16:13.057474+0000","last_peered":"2024-09-13T13:16:13.057474+0000","last_clean":"2024-09-13T13:16:13.057474+0000","last_became_active":"2024-09-13T13:16:12.067157+0000","last_became_peered":"2024-09-13T13:16:12.067157+0000","last_unstale":"2024-09-13T13:16:13.057474+0000","last_undegraded":"2024-09-13T13:16:13.057474+0000","last_fullsized":"2024-09-13T13:16:13.057474+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:04.755828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:04.755828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:15.005053+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":882,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058846+0000","last_change":"2024-09-13T13:15:51.663010+0000","last_active":"2024-09-13T13:16:13.058846+0000","last_peered":"2024-09-13T13:16:13.058846+0000","last_clean":"2024-09-13T13:16:13.058846+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:16:13.058846+0000","last_undegraded":"2024-09-13T13:16:13.058846+0000","last_fullsized":"2024-09-13T13:16:13.058846+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:15:51.662657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.662657+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:55.986754+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":712,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.072283+0000","last_change":"2024-09-13T13:16:12.077285+0000","last_active":"2024-09-13T13:16:13.072283+0000","last_peered":"2024-09-13T13:16:13.072283+0000","last_clean":"2024-09-13T13:16:13.072283+0000","last_became_active":"2024-09-13T13:16:12.072760+0000","last_became_peered":"2024-09-13T13:16:12.072760+0000","last_unstale":"2024-09-13T13:16:13.072283+0000","last_undegraded":"2024-09-13T13:16:13.072283+0000","last_fullsized":"2024-09-13T13:16:13.072283+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:26.667455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:26.667455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.371296+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"379'10858","reported_seq":11043,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.241092+0000","last_change":"2024-09-13T13:15:25.514356+0000","last_active":"2024-09-13T13:16:16.241092+0000","last_peered":"2024-09-13T13:16:16.241092+0000","last_clean":"2024-09-13T13:16:16.241092+0000","last_became_active":"2024-09-13T13:13:58.965950+0000","last_became_peered":"2024-09-13T13:13:58.965950+0000","last_unstale":"2024-09-13T13:16:16.241092+0000","last_undegraded":"2024-09-13T13:16:16.241092+0000","last_fullsized":"2024-09-13T13:16:16.241092+0000","mapping_epoch":334,"log_start":"356'5668","ondisk_log_start":"356'5668","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'7503","last_scrub_stamp":"2024-09-13T13:15:25.514118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.514118+0000","objects_scrubbed":7440,"log_size":5190,"log_dups_size":0,"ondisk_log_size":5190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.146280+0000","scrub_duration":2270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343605272,"num_objects":5244,"num_object_clones":0,"num_object_copies":15732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5244,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10858,"num_write_kb":515201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":748,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058978+0000","last_change":"2024-09-13T13:13:45.757306+0000","last_active":"2024-09-13T13:16:13.058978+0000","last_peered":"2024-09-13T13:16:13.058978+0000","last_clean":"2024-09-13T13:16:13.058978+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:16:13.058978+0000","last_undegraded":"2024-09-13T13:16:13.058978+0000","last_fullsized":"2024-09-13T13:16:13.058978+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:13:45.751801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:45.751801+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.559426+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":686,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056751+0000","last_change":"2024-09-13T13:16:12.075290+0000","last_active":"2024-09-13T13:16:13.056751+0000","last_peered":"2024-09-13T13:16:13.056751+0000","last_clean":"2024-09-13T13:16:13.056751+0000","last_became_active":"2024-09-13T13:16:12.074272+0000","last_became_peered":"2024-09-13T13:16:12.074272+0000","last_unstale":"2024-09-13T13:16:13.056751+0000","last_undegraded":"2024-09-13T13:16:13.056751+0000","last_fullsized":"2024-09-13T13:16:13.056751+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:24.955516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:24.955516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":656,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.082742+0000","last_change":"2024-09-13T13:14:50.408656+0000","last_active":"2024-09-13T13:16:16.082742+0000","last_peered":"2024-09-13T13:16:16.082742+0000","last_clean":"2024-09-13T13:16:16.082742+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:16:16.082742+0000","last_undegraded":"2024-09-13T13:16:16.082742+0000","last_fullsized":"2024-09-13T13:16:16.082742+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.408378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.408378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":693,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055215+0000","last_change":"2024-09-13T13:16:12.068109+0000","last_active":"2024-09-13T13:16:13.055215+0000","last_peered":"2024-09-13T13:16:13.055215+0000","last_clean":"2024-09-13T13:16:13.055215+0000","last_became_active":"2024-09-13T13:16:12.067708+0000","last_became_peered":"2024-09-13T13:16:12.067708+0000","last_unstale":"2024-09-13T13:16:13.055215+0000","last_undegraded":"2024-09-13T13:16:13.055215+0000","last_fullsized":"2024-09-13T13:16:13.055215+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:50.772689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.772689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"379'10777","reported_seq":10879,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.230762+0000","last_change":"2024-09-13T13:13:58.966631+0000","last_active":"2024-09-13T13:16:16.230762+0000","last_peered":"2024-09-13T13:16:16.230762+0000","last_clean":"2024-09-13T13:16:16.230762+0000","last_became_active":"2024-09-13T13:13:58.965789+0000","last_became_peered":"2024-09-13T13:13:58.965789+0000","last_unstale":"2024-09-13T13:16:16.230762+0000","last_undegraded":"2024-09-13T13:16:16.230762+0000","last_fullsized":"2024-09-13T13:16:16.230762+0000","mapping_epoch":334,"log_start":"356'5563","ondisk_log_start":"356'5563","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5214,"log_dups_size":0,"ondisk_log_size":5214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336003072,"num_objects":5127,"num_object_clones":0,"num_object_copies":15381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10777,"num_write_kb":508928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"377'41","reported_seq":969,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056666+0000","last_change":"2024-09-13T13:16:12.095336+0000","last_active":"2024-09-13T13:16:13.056666+0000","last_peered":"2024-09-13T13:16:13.056666+0000","last_clean":"2024-09-13T13:16:13.056666+0000","last_became_active":"2024-09-13T13:16:12.072827+0000","last_became_peered":"2024-09-13T13:16:12.072827+0000","last_unstale":"2024-09-13T13:16:13.056666+0000","last_undegraded":"2024-09-13T13:16:13.056666+0000","last_fullsized":"2024-09-13T13:16:13.056666+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'40","last_scrub_stamp":"2024-09-13T13:15:04.288431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.288431+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:28.505547+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":173,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056131+0000","last_change":"2024-09-13T13:14:14.899463+0000","last_active":"2024-09-13T13:16:13.056131+0000","last_peered":"2024-09-13T13:16:13.056131+0000","last_clean":"2024-09-13T13:16:13.056131+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:16:13.056131+0000","last_undegraded":"2024-09-13T13:16:13.056131+0000","last_fullsized":"2024-09-13T13:16:13.056131+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:14.899378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:14.899378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":705,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056960+0000","last_change":"2024-09-13T13:15:50.618476+0000","last_active":"2024-09-13T13:16:13.056960+0000","last_peered":"2024-09-13T13:16:13.056960+0000","last_clean":"2024-09-13T13:16:13.056960+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:16:13.056960+0000","last_undegraded":"2024-09-13T13:16:13.056960+0000","last_fullsized":"2024-09-13T13:16:13.056960+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:15:50.618426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.618426+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:58.477958+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":707,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.077248+0000","last_change":"2024-09-13T13:14:59.670853+0000","last_active":"2024-09-13T13:16:13.077248+0000","last_peered":"2024-09-13T13:16:13.077248+0000","last_clean":"2024-09-13T13:16:13.077248+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:16:13.077248+0000","last_undegraded":"2024-09-13T13:16:13.077248+0000","last_fullsized":"2024-09-13T13:16:13.077248+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.670779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.670779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":695,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076463+0000","last_change":"2024-09-13T13:15:33.280401+0000","last_active":"2024-09-13T13:16:13.076463+0000","last_peered":"2024-09-13T13:16:13.076463+0000","last_clean":"2024-09-13T13:16:13.076463+0000","last_became_active":"2024-09-13T13:12:52.948742+0000","last_became_peered":"2024-09-13T13:12:52.948742+0000","last_unstale":"2024-09-13T13:16:13.076463+0000","last_undegraded":"2024-09-13T13:16:13.076463+0000","last_fullsized":"2024-09-13T13:16:13.076463+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:33.280352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.280352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:51.771913+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"378'10415","reported_seq":10549,"reported_epoch":378,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T13:16:13.441772+0000","last_change":"2024-09-13T13:16:13.279800+0000","last_active":"2024-09-13T13:16:13.441772+0000","last_peered":"2024-09-13T13:16:13.441772+0000","last_clean":"2024-09-13T13:16:11.028626+0000","last_became_active":"2024-09-13T13:16:13.213473+0000","last_became_peered":"2024-09-13T13:16:13.213473+0000","last_unstale":"2024-09-13T13:16:13.441772+0000","last_undegraded":"2024-09-13T13:16:13.441772+0000","last_fullsized":"2024-09-13T13:16:13.441772+0000","mapping_epoch":377,"log_start":"354'5394","ondisk_log_start":"354'5394","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348585984,"num_objects":5319,"num_object_clones":0,"num_object_copies":15957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5306,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10415,"num_write_kb":503488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,2],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":5319}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":715,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055741+0000","last_change":"2024-09-13T13:15:33.299837+0000","last_active":"2024-09-13T13:16:13.055741+0000","last_peered":"2024-09-13T13:16:13.055741+0000","last_clean":"2024-09-13T13:16:13.055741+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:16:13.055741+0000","last_undegraded":"2024-09-13T13:16:13.055741+0000","last_fullsized":"2024-09-13T13:16:13.055741+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'17","last_scrub_stamp":"2024-09-13T13:15:33.299746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.299746+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:37.843249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":713,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055282+0000","last_change":"2024-09-13T13:16:12.072959+0000","last_active":"2024-09-13T13:16:13.055282+0000","last_peered":"2024-09-13T13:16:13.055282+0000","last_clean":"2024-09-13T13:16:13.055282+0000","last_became_active":"2024-09-13T13:16:12.072367+0000","last_became_peered":"2024-09-13T13:16:12.072367+0000","last_unstale":"2024-09-13T13:16:13.055282+0000","last_undegraded":"2024-09-13T13:16:13.055282+0000","last_fullsized":"2024-09-13T13:16:13.055282+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:17.676339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:17.676339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":31,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075924+0000","last_change":"2024-09-13T13:16:12.092137+0000","last_active":"2024-09-13T13:16:13.075924+0000","last_peered":"2024-09-13T13:16:13.075924+0000","last_clean":"2024-09-13T13:16:13.075924+0000","last_became_active":"2024-09-13T13:16:12.074083+0000","last_became_peered":"2024-09-13T13:16:12.074083+0000","last_unstale":"2024-09-13T13:16:13.075924+0000","last_undegraded":"2024-09-13T13:16:13.075924+0000","last_fullsized":"2024-09-13T13:16:13.075924+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:13:14.560555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:14.560555+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:13.172641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056367+0000","last_change":"2024-09-13T13:16:12.070341+0000","last_active":"2024-09-13T13:16:13.056367+0000","last_peered":"2024-09-13T13:16:13.056367+0000","last_clean":"2024-09-13T13:16:13.056367+0000","last_became_active":"2024-09-13T13:16:12.069747+0000","last_became_peered":"2024-09-13T13:16:12.069747+0000","last_unstale":"2024-09-13T13:16:13.056367+0000","last_undegraded":"2024-09-13T13:16:13.056367+0000","last_fullsized":"2024-09-13T13:16:13.056367+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:59.809055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:59.809055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:24.864450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":309,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.085899+0000","last_change":"2024-09-13T13:12:41.332308+0000","last_active":"2024-09-13T13:16:16.085899+0000","last_peered":"2024-09-13T13:16:16.085899+0000","last_clean":"2024-09-13T13:16:16.085899+0000","last_became_active":"2024-09-13T13:12:40.120242+0000","last_became_peered":"2024-09-13T13:12:40.120242+0000","last_unstale":"2024-09-13T13:16:16.085899+0000","last_undegraded":"2024-09-13T13:16:16.085899+0000","last_fullsized":"2024-09-13T13:16:16.085899+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:41.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:41.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:14.436447+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"378'10757","reported_seq":10863,"reported_epoch":378,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T13:16:14.571249+0000","last_change":"2024-09-13T13:16:13.223583+0000","last_active":"2024-09-13T13:16:14.571249+0000","last_peered":"2024-09-13T13:16:14.571249+0000","last_clean":"2024-09-13T13:16:11.073420+0000","last_became_active":"2024-09-13T13:16:13.188743+0000","last_became_peered":"2024-09-13T13:16:13.188743+0000","last_unstale":"2024-09-13T13:16:14.571249+0000","last_undegraded":"2024-09-13T13:16:14.571249+0000","last_fullsized":"2024-09-13T13:16:14.571249+0000","mapping_epoch":377,"log_start":"357'6196","ondisk_log_start":"357'6196","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349765632,"num_objects":5337,"num_object_clones":0,"num_object_copies":16011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5337,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10757,"num_write_kb":515008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,2],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":5337}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":737,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058307+0000","last_change":"2024-09-13T13:16:12.094755+0000","last_active":"2024-09-13T13:16:13.058307+0000","last_peered":"2024-09-13T13:16:13.058307+0000","last_clean":"2024-09-13T13:16:13.058307+0000","last_became_active":"2024-09-13T13:16:12.074949+0000","last_became_peered":"2024-09-13T13:16:12.074949+0000","last_unstale":"2024-09-13T13:16:13.058307+0000","last_undegraded":"2024-09-13T13:16:13.058307+0000","last_fullsized":"2024-09-13T13:16:13.058307+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:13:10.964809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.964809+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":707,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057502+0000","last_change":"2024-09-13T13:15:30.590673+0000","last_active":"2024-09-13T13:16:13.057502+0000","last_peered":"2024-09-13T13:16:13.057502+0000","last_clean":"2024-09-13T13:16:13.057502+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:16:13.057502+0000","last_undegraded":"2024-09-13T13:16:13.057502+0000","last_fullsized":"2024-09-13T13:16:13.057502+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:30.590466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:30.590466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:49.834025+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":443,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055798+0000","last_change":"2024-09-13T13:14:48.128237+0000","last_active":"2024-09-13T13:16:13.055798+0000","last_peered":"2024-09-13T13:16:13.055798+0000","last_clean":"2024-09-13T13:16:13.055798+0000","last_became_active":"2024-09-13T13:12:39.947940+0000","last_became_peered":"2024-09-13T13:12:39.947940+0000","last_unstale":"2024-09-13T13:16:13.055798+0000","last_undegraded":"2024-09-13T13:16:13.055798+0000","last_fullsized":"2024-09-13T13:16:13.055798+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:14:48.128156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:48.128156+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:06.372754+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":220,"num_read_kb":215,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":677,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.063225+0000","last_change":"2024-09-13T13:15:32.318497+0000","last_active":"2024-09-13T13:16:13.063225+0000","last_peered":"2024-09-13T13:16:13.063225+0000","last_clean":"2024-09-13T13:16:13.063225+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:16:13.063225+0000","last_undegraded":"2024-09-13T13:16:13.063225+0000","last_fullsized":"2024-09-13T13:16:13.063225+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:32.318413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.318413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:32.934946+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":683,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055388+0000","last_change":"2024-09-13T13:10:32.720742+0000","last_active":"2024-09-13T13:16:13.055388+0000","last_peered":"2024-09-13T13:16:13.055388+0000","last_clean":"2024-09-13T13:16:13.055388+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:16:13.055388+0000","last_undegraded":"2024-09-13T13:16:13.055388+0000","last_fullsized":"2024-09-13T13:16:13.055388+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.305143+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"379'10717","reported_seq":10818,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.246303+0000","last_change":"2024-09-13T13:13:58.967163+0000","last_active":"2024-09-13T13:16:16.246303+0000","last_peered":"2024-09-13T13:16:16.246303+0000","last_clean":"2024-09-13T13:16:16.246303+0000","last_became_active":"2024-09-13T13:13:58.965962+0000","last_became_peered":"2024-09-13T13:13:58.965962+0000","last_unstale":"2024-09-13T13:16:16.246303+0000","last_undegraded":"2024-09-13T13:16:16.246303+0000","last_fullsized":"2024-09-13T13:16:16.246303+0000","mapping_epoch":334,"log_start":"356'5515","ondisk_log_start":"356'5515","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5202,"log_dups_size":0,"ondisk_log_size":5202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341770240,"num_objects":5215,"num_object_clones":0,"num_object_copies":15645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10717,"num_write_kb":509824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"306'32","reported_seq":816,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.085630+0000","last_change":"2024-09-13T13:13:01.352416+0000","last_active":"2024-09-13T13:16:16.085630+0000","last_peered":"2024-09-13T13:16:16.085630+0000","last_clean":"2024-09-13T13:16:16.085630+0000","last_became_active":"2024-09-13T13:12:52.999924+0000","last_became_peered":"2024-09-13T13:12:52.999924+0000","last_unstale":"2024-09-13T13:16:16.085630+0000","last_undegraded":"2024-09-13T13:16:16.085630+0000","last_fullsized":"2024-09-13T13:16:16.085630+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:13:01.352337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.352337+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:07.275239+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":697,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076043+0000","last_change":"2024-09-13T13:15:04.182566+0000","last_active":"2024-09-13T13:16:13.076043+0000","last_peered":"2024-09-13T13:16:13.076043+0000","last_clean":"2024-09-13T13:16:13.076043+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:16:13.076043+0000","last_undegraded":"2024-09-13T13:16:13.076043+0000","last_fullsized":"2024-09-13T13:16:13.076043+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:04.182499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.182499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.040803+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":271,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.072381+0000","last_change":"2024-09-13T13:15:54.707061+0000","last_active":"2024-09-13T13:16:13.072381+0000","last_peered":"2024-09-13T13:16:13.072381+0000","last_clean":"2024-09-13T13:16:13.072381+0000","last_became_active":"2024-09-13T13:12:39.948765+0000","last_became_peered":"2024-09-13T13:12:39.948765+0000","last_unstale":"2024-09-13T13:16:13.072381+0000","last_undegraded":"2024-09-13T13:16:13.072381+0000","last_fullsized":"2024-09-13T13:16:13.072381+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:10.766426+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":17,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056722+0000","last_change":"2024-09-13T13:16:12.074017+0000","last_active":"2024-09-13T13:16:13.056722+0000","last_peered":"2024-09-13T13:16:13.056722+0000","last_clean":"2024-09-13T13:16:13.056722+0000","last_became_active":"2024-09-13T13:16:12.073663+0000","last_became_peered":"2024-09-13T13:16:12.073663+0000","last_unstale":"2024-09-13T13:16:13.056722+0000","last_undegraded":"2024-09-13T13:16:13.056722+0000","last_fullsized":"2024-09-13T13:16:13.056722+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:33.708995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.708995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:34.926899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":719,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.063048+0000","last_change":"2024-09-13T13:15:57.689044+0000","last_active":"2024-09-13T13:16:13.063048+0000","last_peered":"2024-09-13T13:16:13.063048+0000","last_clean":"2024-09-13T13:16:13.063048+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:16:13.063048+0000","last_undegraded":"2024-09-13T13:16:13.063048+0000","last_fullsized":"2024-09-13T13:16:13.063048+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:57.688955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:57.688955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:01.261821+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"379'10923","reported_seq":11093,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.240924+0000","last_change":"2024-09-13T13:15:04.191246+0000","last_active":"2024-09-13T13:16:16.240924+0000","last_peered":"2024-09-13T13:16:16.240924+0000","last_clean":"2024-09-13T13:16:16.240924+0000","last_became_active":"2024-09-13T13:13:58.965566+0000","last_became_peered":"2024-09-13T13:13:58.965566+0000","last_unstale":"2024-09-13T13:16:16.240924+0000","last_undegraded":"2024-09-13T13:16:16.240924+0000","last_fullsized":"2024-09-13T13:16:16.240924+0000","mapping_epoch":334,"log_start":"356'5721","ondisk_log_start":"356'5721","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'6049","last_scrub_stamp":"2024-09-13T13:15:04.191100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.191100+0000","objects_scrubbed":5987,"log_size":5202,"log_dups_size":0,"ondisk_log_size":5202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:31.390342+0000","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345047040,"num_objects":5265,"num_object_clones":0,"num_object_copies":15795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10923,"num_write_kb":518016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":185,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.070021+0000","last_change":"2024-09-13T13:14:59.354034+0000","last_active":"2024-09-13T13:16:13.070021+0000","last_peered":"2024-09-13T13:16:13.070021+0000","last_clean":"2024-09-13T13:16:13.070021+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:16:13.070021+0000","last_undegraded":"2024-09-13T13:16:13.070021+0000","last_fullsized":"2024-09-13T13:16:13.070021+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:14:59.353691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.353691+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":172,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057727+0000","last_change":"2024-09-13T13:16:12.074151+0000","last_active":"2024-09-13T13:16:13.057727+0000","last_peered":"2024-09-13T13:16:13.057727+0000","last_clean":"2024-09-13T13:16:13.057727+0000","last_became_active":"2024-09-13T13:16:12.073474+0000","last_became_peered":"2024-09-13T13:16:12.073474+0000","last_unstale":"2024-09-13T13:16:13.057727+0000","last_undegraded":"2024-09-13T13:16:13.057727+0000","last_fullsized":"2024-09-13T13:16:13.057727+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:26.343754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:26.343754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":734,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.062986+0000","last_change":"2024-09-13T13:15:55.592241+0000","last_active":"2024-09-13T13:16:13.062986+0000","last_peered":"2024-09-13T13:16:13.062986+0000","last_clean":"2024-09-13T13:16:13.062986+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:16:13.062986+0000","last_undegraded":"2024-09-13T13:16:13.062986+0000","last_fullsized":"2024-09-13T13:16:13.062986+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:55.592172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:55.592172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:23.172298+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":684,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076249+0000","last_change":"2024-09-13T13:14:23.655250+0000","last_active":"2024-09-13T13:16:13.076249+0000","last_peered":"2024-09-13T13:16:13.076249+0000","last_clean":"2024-09-13T13:16:13.076249+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:16:13.076249+0000","last_undegraded":"2024-09-13T13:16:13.076249+0000","last_fullsized":"2024-09-13T13:16:13.076249+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:23.655072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:23.655072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:45.281222+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"378'10495","reported_seq":10684,"reported_epoch":378,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T13:16:13.421197+0000","last_change":"2024-09-13T13:16:13.281051+0000","last_active":"2024-09-13T13:16:13.421197+0000","last_peered":"2024-09-13T13:16:13.421197+0000","last_clean":"2024-09-13T13:16:11.023458+0000","last_became_active":"2024-09-13T13:16:13.223271+0000","last_became_peered":"2024-09-13T13:16:13.223271+0000","last_unstale":"2024-09-13T13:16:13.421197+0000","last_undegraded":"2024-09-13T13:16:13.421197+0000","last_fullsized":"2024-09-13T13:16:13.421197+0000","mapping_epoch":377,"log_start":"355'5484","ondisk_log_start":"355'5484","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7793","last_scrub_stamp":"2024-09-13T13:15:31.408001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:31.408001+0000","objects_scrubbed":7714,"log_size":5011,"log_dups_size":0,"ondisk_log_size":5011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355926016,"num_objects":5431,"num_object_clones":0,"num_object_copies":16293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5431,"num_objects_unfound":0,"num_objects_dirty":5431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10495,"num_write_kb":509632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,2],"avail_no_missing":["5","2","3"],"object_location_counts":[{"shards":"2,3,5","objects":5431}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":216,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058341+0000","last_change":"2024-09-13T13:16:12.090140+0000","last_active":"2024-09-13T13:16:13.058341+0000","last_peered":"2024-09-13T13:16:13.058341+0000","last_clean":"2024-09-13T13:16:13.058341+0000","last_became_active":"2024-09-13T13:16:12.075614+0000","last_became_peered":"2024-09-13T13:16:12.075614+0000","last_unstale":"2024-09-13T13:16:13.058341+0000","last_undegraded":"2024-09-13T13:16:13.058341+0000","last_fullsized":"2024-09-13T13:16:13.058341+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:14:20.221286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:20.221286+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":710,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.083027+0000","last_change":"2024-09-13T13:15:24.265548+0000","last_active":"2024-09-13T13:16:16.083027+0000","last_peered":"2024-09-13T13:16:16.083027+0000","last_clean":"2024-09-13T13:16:16.083027+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:16:16.083027+0000","last_undegraded":"2024-09-13T13:16:16.083027+0000","last_fullsized":"2024-09-13T13:16:16.083027+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:24.265477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:24.265477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:35.804220+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":700,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055568+0000","last_change":"2024-09-13T13:15:24.264425+0000","last_active":"2024-09-13T13:16:13.055568+0000","last_peered":"2024-09-13T13:16:13.055568+0000","last_clean":"2024-09-13T13:16:13.055568+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:16:13.055568+0000","last_undegraded":"2024-09-13T13:16:13.055568+0000","last_fullsized":"2024-09-13T13:16:13.055568+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:24.264316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:24.264316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:32.724876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":540,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076793+0000","last_change":"2024-09-13T13:16:12.067926+0000","last_active":"2024-09-13T13:16:13.076793+0000","last_peered":"2024-09-13T13:16:13.076793+0000","last_clean":"2024-09-13T13:16:13.076793+0000","last_became_active":"2024-09-13T13:16:12.067474+0000","last_became_peered":"2024-09-13T13:16:12.067474+0000","last_unstale":"2024-09-13T13:16:13.076793+0000","last_undegraded":"2024-09-13T13:16:13.076793+0000","last_fullsized":"2024-09-13T13:16:13.076793+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.649166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.649166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"378'10691","reported_seq":10791,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:14.075869+0000","last_change":"2024-09-13T13:13:58.963327+0000","last_active":"2024-09-13T13:16:14.075869+0000","last_peered":"2024-09-13T13:16:14.075869+0000","last_clean":"2024-09-13T13:16:14.075869+0000","last_became_active":"2024-09-13T13:13:58.963064+0000","last_became_peered":"2024-09-13T13:13:58.963064+0000","last_unstale":"2024-09-13T13:16:14.075869+0000","last_undegraded":"2024-09-13T13:16:14.075869+0000","last_fullsized":"2024-09-13T13:16:14.075869+0000","mapping_epoch":334,"log_start":"356'6014","ondisk_log_start":"356'6014","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4677,"log_dups_size":0,"ondisk_log_size":4677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350420992,"num_objects":5347,"num_object_clones":0,"num_object_copies":16041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10691,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":169,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057295+0000","last_change":"2024-09-13T13:12:53.018645+0000","last_active":"2024-09-13T13:16:13.057295+0000","last_peered":"2024-09-13T13:16:13.057295+0000","last_clean":"2024-09-13T13:16:13.057295+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:16:13.057295+0000","last_undegraded":"2024-09-13T13:16:13.057295+0000","last_fullsized":"2024-09-13T13:16:13.057295+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:09.739184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:09.739184+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:36.221949+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":691,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076171+0000","last_change":"2024-09-13T13:12:53.003530+0000","last_active":"2024-09-13T13:16:13.076171+0000","last_peered":"2024-09-13T13:16:13.076171+0000","last_clean":"2024-09-13T13:16:13.076171+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:16:13.076171+0000","last_undegraded":"2024-09-13T13:16:13.076171+0000","last_fullsized":"2024-09-13T13:16:13.076171+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.996346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.996346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:11.226277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"377'9","reported_seq":46,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056987+0000","last_change":"2024-09-13T13:16:12.089906+0000","last_active":"2024-09-13T13:16:13.056987+0000","last_peered":"2024-09-13T13:16:13.056987+0000","last_clean":"2024-09-13T13:16:13.056987+0000","last_became_active":"2024-09-13T13:16:12.079150+0000","last_became_peered":"2024-09-13T13:16:12.079150+0000","last_unstale":"2024-09-13T13:16:13.056987+0000","last_undegraded":"2024-09-13T13:16:13.056987+0000","last_fullsized":"2024-09-13T13:16:13.056987+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:12:36.113709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:36.113709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":172,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.069935+0000","last_change":"2024-09-13T13:16:12.076027+0000","last_active":"2024-09-13T13:16:13.069935+0000","last_peered":"2024-09-13T13:16:13.069935+0000","last_clean":"2024-09-13T13:16:13.069935+0000","last_became_active":"2024-09-13T13:16:12.075615+0000","last_became_peered":"2024-09-13T13:16:12.075615+0000","last_unstale":"2024-09-13T13:16:13.069935+0000","last_undegraded":"2024-09-13T13:16:13.069935+0000","last_fullsized":"2024-09-13T13:16:13.069935+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.797639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.797639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"378'10613","reported_seq":10806,"reported_epoch":378,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T13:16:14.615854+0000","last_change":"2024-09-13T13:16:13.280813+0000","last_active":"2024-09-13T13:16:14.615854+0000","last_peered":"2024-09-13T13:16:14.615854+0000","last_clean":"2024-09-13T13:16:11.023633+0000","last_became_active":"2024-09-13T13:16:13.217424+0000","last_became_peered":"2024-09-13T13:16:13.217424+0000","last_unstale":"2024-09-13T13:16:14.615854+0000","last_undegraded":"2024-09-13T13:16:14.615854+0000","last_fullsized":"2024-09-13T13:16:14.615854+0000","mapping_epoch":377,"log_start":"356'5971","ondisk_log_start":"356'5971","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7822","last_scrub_stamp":"2024-09-13T13:15:32.530220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.530220+0000","objects_scrubbed":7744,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343867392,"num_objects":5247,"num_object_clones":0,"num_object_copies":15741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5247,"num_objects_unfound":0,"num_objects_dirty":5247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":507520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,2],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2,3","objects":5247}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":723,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058894+0000","last_change":"2024-09-13T13:13:56.850149+0000","last_active":"2024-09-13T13:16:13.058894+0000","last_peered":"2024-09-13T13:16:13.058894+0000","last_clean":"2024-09-13T13:16:13.058894+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:16:13.058894+0000","last_undegraded":"2024-09-13T13:16:13.058894+0000","last_fullsized":"2024-09-13T13:16:13.058894+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:56.850094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.850094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:09.858841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":712,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056549+0000","last_change":"2024-09-13T13:12:58.857634+0000","last_active":"2024-09-13T13:16:13.056549+0000","last_peered":"2024-09-13T13:16:13.056549+0000","last_clean":"2024-09-13T13:16:13.056549+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:16:13.056549+0000","last_undegraded":"2024-09-13T13:16:13.056549+0000","last_fullsized":"2024-09-13T13:16:13.056549+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:58.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:44.458422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":560,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057344+0000","last_change":"2024-09-13T13:16:12.077831+0000","last_active":"2024-09-13T13:16:13.057344+0000","last_peered":"2024-09-13T13:16:13.057344+0000","last_clean":"2024-09-13T13:16:13.057344+0000","last_became_active":"2024-09-13T13:16:12.076962+0000","last_became_peered":"2024-09-13T13:16:12.076962+0000","last_unstale":"2024-09-13T13:16:13.057344+0000","last_undegraded":"2024-09-13T13:16:13.057344+0000","last_fullsized":"2024-09-13T13:16:13.057344+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:23.682463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.682463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:51.172109+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":679,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057397+0000","last_change":"2024-09-13T13:16:12.075127+0000","last_active":"2024-09-13T13:16:13.057397+0000","last_peered":"2024-09-13T13:16:13.057397+0000","last_clean":"2024-09-13T13:16:13.057397+0000","last_became_active":"2024-09-13T13:16:12.074784+0000","last_became_peered":"2024-09-13T13:16:12.074784+0000","last_unstale":"2024-09-13T13:16:13.057397+0000","last_undegraded":"2024-09-13T13:16:13.057397+0000","last_fullsized":"2024-09-13T13:16:13.057397+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.761814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.761814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"378'10436","reported_seq":10536,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:14.570652+0000","last_change":"2024-09-13T13:13:58.963548+0000","last_active":"2024-09-13T13:16:14.570652+0000","last_peered":"2024-09-13T13:16:14.570652+0000","last_clean":"2024-09-13T13:16:14.570652+0000","last_became_active":"2024-09-13T13:13:58.963160+0000","last_became_peered":"2024-09-13T13:13:58.963160+0000","last_unstale":"2024-09-13T13:16:14.570652+0000","last_undegraded":"2024-09-13T13:16:14.570652+0000","last_fullsized":"2024-09-13T13:16:14.570652+0000","mapping_epoch":334,"log_start":"356'5888","ondisk_log_start":"356'5888","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344850432,"num_objects":5262,"num_object_clones":0,"num_object_copies":15786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10436,"num_write_kb":502336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":699,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.082835+0000","last_change":"2024-09-13T13:13:06.339762+0000","last_active":"2024-09-13T13:16:16.082835+0000","last_peered":"2024-09-13T13:16:16.082835+0000","last_clean":"2024-09-13T13:16:16.082835+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:16:16.082835+0000","last_undegraded":"2024-09-13T13:16:16.082835+0000","last_fullsized":"2024-09-13T13:16:16.082835+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:13:06.339664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.339664+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:15.437483+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":327,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057210+0000","last_change":"2024-09-13T13:14:55.471954+0000","last_active":"2024-09-13T13:16:13.057210+0000","last_peered":"2024-09-13T13:16:13.057210+0000","last_clean":"2024-09-13T13:16:13.057210+0000","last_became_active":"2024-09-13T13:12:39.954315+0000","last_became_peered":"2024-09-13T13:12:39.954315+0000","last_unstale":"2024-09-13T13:16:13.057210+0000","last_undegraded":"2024-09-13T13:16:13.057210+0000","last_fullsized":"2024-09-13T13:16:13.057210+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:55.471885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:55.471885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058456+0000","last_change":"2024-09-13T13:16:12.076693+0000","last_active":"2024-09-13T13:16:13.058456+0000","last_peered":"2024-09-13T13:16:13.058456+0000","last_clean":"2024-09-13T13:16:13.058456+0000","last_became_active":"2024-09-13T13:16:12.076377+0000","last_became_peered":"2024-09-13T13:16:12.076377+0000","last_unstale":"2024-09-13T13:16:13.058456+0000","last_undegraded":"2024-09-13T13:16:13.058456+0000","last_fullsized":"2024-09-13T13:16:13.058456+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.484615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.484615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":17,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075533+0000","last_change":"2024-09-13T13:16:12.066346+0000","last_active":"2024-09-13T13:16:13.075533+0000","last_peered":"2024-09-13T13:16:13.075533+0000","last_clean":"2024-09-13T13:16:13.075533+0000","last_became_active":"2024-09-13T13:16:12.065848+0000","last_became_peered":"2024-09-13T13:16:12.065848+0000","last_unstale":"2024-09-13T13:16:13.075533+0000","last_undegraded":"2024-09-13T13:16:13.075533+0000","last_fullsized":"2024-09-13T13:16:13.075533+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:02.627569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:02.627569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"379'10554","reported_seq":10656,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.212974+0000","last_change":"2024-09-13T13:13:58.966955+0000","last_active":"2024-09-13T13:16:16.212974+0000","last_peered":"2024-09-13T13:16:16.212974+0000","last_clean":"2024-09-13T13:16:16.212974+0000","last_became_active":"2024-09-13T13:13:58.966675+0000","last_became_peered":"2024-09-13T13:13:58.966675+0000","last_unstale":"2024-09-13T13:16:16.212974+0000","last_undegraded":"2024-09-13T13:16:16.212974+0000","last_fullsized":"2024-09-13T13:16:16.212974+0000","mapping_epoch":334,"log_start":"355'5341","ondisk_log_start":"355'5341","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5213,"log_dups_size":0,"ondisk_log_size":5213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:16.259461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330039296,"num_objects":5036,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10554,"num_write_kb":498880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":185,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058749+0000","last_change":"2024-09-13T13:16:00.285064+0000","last_active":"2024-09-13T13:16:13.058749+0000","last_peered":"2024-09-13T13:16:13.058749+0000","last_clean":"2024-09-13T13:16:13.058749+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:16:13.058749+0000","last_undegraded":"2024-09-13T13:16:13.058749+0000","last_fullsized":"2024-09-13T13:16:13.058749+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:16:00.284984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:00.284984+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:27.689954+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":173,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.069843+0000","last_change":"2024-09-13T13:14:52.383027+0000","last_active":"2024-09-13T13:16:13.069843+0000","last_peered":"2024-09-13T13:16:13.069843+0000","last_clean":"2024-09-13T13:16:13.069843+0000","last_became_active":"2024-09-13T13:12:53.007561+0000","last_became_peered":"2024-09-13T13:12:53.007561+0000","last_unstale":"2024-09-13T13:16:13.069843+0000","last_undegraded":"2024-09-13T13:16:13.069843+0000","last_fullsized":"2024-09-13T13:16:13.069843+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.382755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.382755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:20.097418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.072093+0000","last_change":"2024-09-13T13:16:12.078026+0000","last_active":"2024-09-13T13:16:13.072093+0000","last_peered":"2024-09-13T13:16:13.072093+0000","last_clean":"2024-09-13T13:16:13.072093+0000","last_became_active":"2024-09-13T13:16:12.077714+0000","last_became_peered":"2024-09-13T13:16:12.077714+0000","last_unstale":"2024-09-13T13:16:13.072093+0000","last_undegraded":"2024-09-13T13:16:13.072093+0000","last_fullsized":"2024-09-13T13:16:13.072093+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.636898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.636898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":665,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075831+0000","last_change":"2024-09-13T13:15:03.223521+0000","last_active":"2024-09-13T13:16:13.075831+0000","last_peered":"2024-09-13T13:16:13.075831+0000","last_clean":"2024-09-13T13:16:13.075831+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:16:13.075831+0000","last_undegraded":"2024-09-13T13:16:13.075831+0000","last_fullsized":"2024-09-13T13:16:13.075831+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.223265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.223265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:30.540511+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"378'10586","reported_seq":10759,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.436739+0000","last_change":"2024-09-13T13:15:56.411481+0000","last_active":"2024-09-13T13:16:13.436739+0000","last_peered":"2024-09-13T13:16:13.436739+0000","last_clean":"2024-09-13T13:16:13.436739+0000","last_became_active":"2024-09-13T13:13:58.962773+0000","last_became_peered":"2024-09-13T13:13:58.962773+0000","last_unstale":"2024-09-13T13:16:13.436739+0000","last_undegraded":"2024-09-13T13:16:13.436739+0000","last_fullsized":"2024-09-13T13:16:13.436739+0000","mapping_epoch":334,"log_start":"356'5576","ondisk_log_start":"356'5576","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'9588","last_scrub_stamp":"2024-09-13T13:15:56.411320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:56.411320+0000","objects_scrubbed":6514,"log_size":5010,"log_dups_size":0,"ondisk_log_size":5010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:14.862198+0000","scrub_duration":1806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358350848,"num_objects":5468,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10586,"num_write_kb":513728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":760,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055959+0000","last_change":"2024-09-13T13:15:56.631456+0000","last_active":"2024-09-13T13:16:13.055959+0000","last_peered":"2024-09-13T13:16:13.055959+0000","last_clean":"2024-09-13T13:16:13.055959+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:16:13.055959+0000","last_undegraded":"2024-09-13T13:16:13.055959+0000","last_fullsized":"2024-09-13T13:16:13.055959+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:15:56.631396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:56.631396+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:22.171338+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":171,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056662+0000","last_change":"2024-09-13T13:13:01.972907+0000","last_active":"2024-09-13T13:16:13.056662+0000","last_peered":"2024-09-13T13:16:13.056662+0000","last_clean":"2024-09-13T13:16:13.056662+0000","last_became_active":"2024-09-13T13:12:53.006362+0000","last_became_peered":"2024-09-13T13:12:53.006362+0000","last_unstale":"2024-09-13T13:16:13.056662+0000","last_undegraded":"2024-09-13T13:16:13.056662+0000","last_fullsized":"2024-09-13T13:16:13.056662+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:13:01.972831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.972831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:46.279374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"306'13","reported_seq":418,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.079532+0000","last_change":"2024-09-13T13:13:03.334319+0000","last_active":"2024-09-13T13:16:16.079532+0000","last_peered":"2024-09-13T13:16:16.079532+0000","last_clean":"2024-09-13T13:16:16.079532+0000","last_became_active":"2024-09-13T13:12:53.001189+0000","last_became_peered":"2024-09-13T13:12:53.001189+0000","last_unstale":"2024-09-13T13:16:16.079532+0000","last_undegraded":"2024-09-13T13:16:16.079532+0000","last_fullsized":"2024-09-13T13:16:16.079532+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'13","last_scrub_stamp":"2024-09-13T13:13:03.334012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.334012+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:38.265982+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":684,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056599+0000","last_change":"2024-09-13T13:13:55.500961+0000","last_active":"2024-09-13T13:16:13.056599+0000","last_peered":"2024-09-13T13:16:13.056599+0000","last_clean":"2024-09-13T13:16:13.056599+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:16:13.056599+0000","last_undegraded":"2024-09-13T13:16:13.056599+0000","last_fullsized":"2024-09-13T13:16:13.056599+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:55.500908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.500908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":664,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.079851+0000","last_change":"2024-09-13T13:13:04.371082+0000","last_active":"2024-09-13T13:16:16.079851+0000","last_peered":"2024-09-13T13:16:16.079851+0000","last_clean":"2024-09-13T13:16:16.079851+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:16:16.079851+0000","last_undegraded":"2024-09-13T13:16:16.079851+0000","last_fullsized":"2024-09-13T13:16:16.079851+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.370881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.370881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:10.353265+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":207,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.062717+0000","last_change":"2024-09-13T13:15:23.319486+0000","last_active":"2024-09-13T13:16:13.062717+0000","last_peered":"2024-09-13T13:16:13.062717+0000","last_clean":"2024-09-13T13:16:13.062717+0000","last_became_active":"2024-09-13T13:12:34.673528+0000","last_became_peered":"2024-09-13T13:12:34.673528+0000","last_unstale":"2024-09-13T13:16:13.062717+0000","last_undegraded":"2024-09-13T13:16:13.062717+0000","last_fullsized":"2024-09-13T13:16:13.062717+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:15:23.319412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.319412+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:45.332563+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":709,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056420+0000","last_change":"2024-09-13T13:15:51.633487+0000","last_active":"2024-09-13T13:16:13.056420+0000","last_peered":"2024-09-13T13:16:13.056420+0000","last_clean":"2024-09-13T13:16:13.056420+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:16:13.056420+0000","last_undegraded":"2024-09-13T13:16:13.056420+0000","last_fullsized":"2024-09-13T13:16:13.056420+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.633433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.633433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:56.064702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":722,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075554+0000","last_change":"2024-09-13T13:15:25.787110+0000","last_active":"2024-09-13T13:16:13.075554+0000","last_peered":"2024-09-13T13:16:13.075554+0000","last_clean":"2024-09-13T13:16:13.075554+0000","last_became_active":"2024-09-13T13:12:53.000873+0000","last_became_peered":"2024-09-13T13:12:53.000873+0000","last_unstale":"2024-09-13T13:16:13.075554+0000","last_undegraded":"2024-09-13T13:16:13.075554+0000","last_fullsized":"2024-09-13T13:16:13.075554+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.787029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.787029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:31.829669+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":697,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.059076+0000","last_change":"2024-09-13T13:13:12.952427+0000","last_active":"2024-09-13T13:16:13.059076+0000","last_peered":"2024-09-13T13:16:13.059076+0000","last_clean":"2024-09-13T13:16:13.059076+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:16:13.059076+0000","last_undegraded":"2024-09-13T13:16:13.059076+0000","last_fullsized":"2024-09-13T13:16:13.059076+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:12.952348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:12.952348+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:37.576218+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":165,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058553+0000","last_change":"2024-09-13T13:13:51.764793+0000","last_active":"2024-09-13T13:16:13.058553+0000","last_peered":"2024-09-13T13:16:13.058553+0000","last_clean":"2024-09-13T13:16:13.058553+0000","last_became_active":"2024-09-13T13:12:53.006579+0000","last_became_peered":"2024-09-13T13:12:53.006579+0000","last_unstale":"2024-09-13T13:16:13.058553+0000","last_undegraded":"2024-09-13T13:16:13.058553+0000","last_fullsized":"2024-09-13T13:16:13.058553+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.764717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.764717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":685,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076947+0000","last_change":"2024-09-13T13:15:25.281502+0000","last_active":"2024-09-13T13:16:13.076947+0000","last_peered":"2024-09-13T13:16:13.076947+0000","last_clean":"2024-09-13T13:16:13.076947+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:16:13.076947+0000","last_undegraded":"2024-09-13T13:16:13.076947+0000","last_fullsized":"2024-09-13T13:16:13.076947+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.281295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.281295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.642193+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":563,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058183+0000","last_change":"2024-09-13T13:16:12.078548+0000","last_active":"2024-09-13T13:16:13.058183+0000","last_peered":"2024-09-13T13:16:13.058183+0000","last_clean":"2024-09-13T13:16:13.058183+0000","last_became_active":"2024-09-13T13:16:12.077562+0000","last_became_peered":"2024-09-13T13:16:12.077562+0000","last_unstale":"2024-09-13T13:16:13.058183+0000","last_undegraded":"2024-09-13T13:16:13.058183+0000","last_fullsized":"2024-09-13T13:16:13.058183+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:15.616112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:15.616112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":727,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057294+0000","last_change":"2024-09-13T13:16:12.084892+0000","last_active":"2024-09-13T13:16:13.057294+0000","last_peered":"2024-09-13T13:16:13.057294+0000","last_clean":"2024-09-13T13:16:13.057294+0000","last_became_active":"2024-09-13T13:16:12.069730+0000","last_became_peered":"2024-09-13T13:16:12.069730+0000","last_unstale":"2024-09-13T13:16:13.057294+0000","last_undegraded":"2024-09-13T13:16:13.057294+0000","last_fullsized":"2024-09-13T13:16:13.057294+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:19.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:19.931466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":676,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056609+0000","last_change":"2024-09-13T13:13:05.981881+0000","last_active":"2024-09-13T13:16:13.056609+0000","last_peered":"2024-09-13T13:16:13.056609+0000","last_clean":"2024-09-13T13:16:13.056609+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:16:13.056609+0000","last_undegraded":"2024-09-13T13:16:13.056609+0000","last_fullsized":"2024-09-13T13:16:13.056609+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:05.981803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:05.981803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:08.876559+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"377'9","reported_seq":871,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076314+0000","last_change":"2024-09-13T13:16:12.080560+0000","last_active":"2024-09-13T13:16:13.076314+0000","last_peered":"2024-09-13T13:16:13.076314+0000","last_clean":"2024-09-13T13:16:13.076314+0000","last_became_active":"2024-09-13T13:16:12.073710+0000","last_became_peered":"2024-09-13T13:16:12.073710+0000","last_unstale":"2024-09-13T13:16:13.076314+0000","last_undegraded":"2024-09-13T13:16:13.076314+0000","last_fullsized":"2024-09-13T13:16:13.076314+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:13:10.130680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.130680+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":172,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056769+0000","last_change":"2024-09-13T13:16:12.077256+0000","last_active":"2024-09-13T13:16:13.056769+0000","last_peered":"2024-09-13T13:16:13.056769+0000","last_clean":"2024-09-13T13:16:13.056769+0000","last_became_active":"2024-09-13T13:16:12.076912+0000","last_became_peered":"2024-09-13T13:16:12.076912+0000","last_unstale":"2024-09-13T13:16:13.056769+0000","last_undegraded":"2024-09-13T13:16:13.056769+0000","last_fullsized":"2024-09-13T13:16:13.056769+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.956784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.956784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":727,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076650+0000","last_change":"2024-09-13T13:15:34.280338+0000","last_active":"2024-09-13T13:16:13.076650+0000","last_peered":"2024-09-13T13:16:13.076650+0000","last_clean":"2024-09-13T13:16:13.076650+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:16:13.076650+0000","last_undegraded":"2024-09-13T13:16:13.076650+0000","last_fullsized":"2024-09-13T13:16:13.076650+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:15:34.280253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:34.280253+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.301730+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":698,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076359+0000","last_change":"2024-09-13T13:16:12.067154+0000","last_active":"2024-09-13T13:16:13.076359+0000","last_peered":"2024-09-13T13:16:13.076359+0000","last_clean":"2024-09-13T13:16:13.076359+0000","last_became_active":"2024-09-13T13:16:12.066808+0000","last_became_peered":"2024-09-13T13:16:12.066808+0000","last_unstale":"2024-09-13T13:16:13.076359+0000","last_undegraded":"2024-09-13T13:16:13.076359+0000","last_fullsized":"2024-09-13T13:16:13.076359+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.105098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.105098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":884,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057045+0000","last_change":"2024-09-13T13:15:33.333062+0000","last_active":"2024-09-13T13:16:13.057045+0000","last_peered":"2024-09-13T13:16:13.057045+0000","last_clean":"2024-09-13T13:16:13.057045+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:16:13.057045+0000","last_undegraded":"2024-09-13T13:16:13.057045+0000","last_fullsized":"2024-09-13T13:16:13.057045+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:15:33.332981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.332981+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:39.941448+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":688,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057826+0000","last_change":"2024-09-13T13:15:22.660262+0000","last_active":"2024-09-13T13:16:13.057826+0000","last_peered":"2024-09-13T13:16:13.057826+0000","last_clean":"2024-09-13T13:16:13.057826+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:16:13.057826+0000","last_undegraded":"2024-09-13T13:16:13.057826+0000","last_fullsized":"2024-09-13T13:16:13.057826+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:22.660050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:22.660050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:44.066070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":177,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.056224+0000","last_change":"2024-09-13T13:13:47.728092+0000","last_active":"2024-09-13T13:16:13.056224+0000","last_peered":"2024-09-13T13:16:13.056224+0000","last_clean":"2024-09-13T13:16:13.056224+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:16:13.056224+0000","last_undegraded":"2024-09-13T13:16:13.056224+0000","last_fullsized":"2024-09-13T13:16:13.056224+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:47.728010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:47.728010+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:13.358350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":157,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057988+0000","last_change":"2024-09-13T13:12:53.006302+0000","last_active":"2024-09-13T13:16:13.057988+0000","last_peered":"2024-09-13T13:16:13.057988+0000","last_clean":"2024-09-13T13:16:13.057988+0000","last_became_active":"2024-09-13T13:12:53.005988+0000","last_became_peered":"2024-09-13T13:12:53.005988+0000","last_unstale":"2024-09-13T13:16:13.057988+0000","last_undegraded":"2024-09-13T13:16:13.057988+0000","last_fullsized":"2024-09-13T13:16:13.057988+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:40.888083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.888083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":694,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055545+0000","last_change":"2024-09-13T13:15:57.356507+0000","last_active":"2024-09-13T13:16:13.055545+0000","last_peered":"2024-09-13T13:16:13.055545+0000","last_clean":"2024-09-13T13:16:13.055545+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:16:13.055545+0000","last_undegraded":"2024-09-13T13:16:13.055545+0000","last_fullsized":"2024-09-13T13:16:13.055545+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:57.356427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:57.356427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:20.233276+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":689,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075430+0000","last_change":"2024-09-13T13:15:51.374176+0000","last_active":"2024-09-13T13:16:13.075430+0000","last_peered":"2024-09-13T13:16:13.075430+0000","last_clean":"2024-09-13T13:16:13.075430+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:16:13.075430+0000","last_undegraded":"2024-09-13T13:16:13.075430+0000","last_fullsized":"2024-09-13T13:16:13.075430+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.374109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.374109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:54.143731+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":666,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055849+0000","last_change":"2024-09-13T13:16:12.087845+0000","last_active":"2024-09-13T13:16:13.055849+0000","last_peered":"2024-09-13T13:16:13.055849+0000","last_clean":"2024-09-13T13:16:13.055849+0000","last_became_active":"2024-09-13T13:16:12.072262+0000","last_became_peered":"2024-09-13T13:16:12.072262+0000","last_unstale":"2024-09-13T13:16:13.055849+0000","last_undegraded":"2024-09-13T13:16:13.055849+0000","last_fullsized":"2024-09-13T13:16:13.055849+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:39.874324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:39.874324+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":677,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076595+0000","last_change":"2024-09-13T13:14:13.418389+0000","last_active":"2024-09-13T13:16:13.076595+0000","last_peered":"2024-09-13T13:16:13.076595+0000","last_clean":"2024-09-13T13:16:13.076595+0000","last_became_active":"2024-09-13T13:12:52.949094+0000","last_became_peered":"2024-09-13T13:12:52.949094+0000","last_unstale":"2024-09-13T13:16:13.076595+0000","last_undegraded":"2024-09-13T13:16:13.076595+0000","last_fullsized":"2024-09-13T13:16:13.076595+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:13.418196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:13.418196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:47.331557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":685,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076066+0000","last_change":"2024-09-13T13:15:01.269060+0000","last_active":"2024-09-13T13:16:13.076066+0000","last_peered":"2024-09-13T13:16:13.076066+0000","last_clean":"2024-09-13T13:16:13.076066+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:16:13.076066+0000","last_undegraded":"2024-09-13T13:16:13.076066+0000","last_fullsized":"2024-09-13T13:16:13.076066+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:01.268884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:01.268884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":672,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.072024+0000","last_change":"2024-09-13T13:10:39.477882+0000","last_active":"2024-09-13T13:16:13.072024+0000","last_peered":"2024-09-13T13:16:13.072024+0000","last_clean":"2024-09-13T13:16:13.072024+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:16:13.072024+0000","last_undegraded":"2024-09-13T13:16:13.072024+0000","last_fullsized":"2024-09-13T13:16:13.072024+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:39.477683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.477683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:40.622029+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":707,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075642+0000","last_change":"2024-09-13T13:13:49.268080+0000","last_active":"2024-09-13T13:16:13.075642+0000","last_peered":"2024-09-13T13:16:13.075642+0000","last_clean":"2024-09-13T13:16:13.075642+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:16:13.075642+0000","last_undegraded":"2024-09-13T13:16:13.075642+0000","last_fullsized":"2024-09-13T13:16:13.075642+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:49.268018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:49.268018+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:19.301185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":701,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057134+0000","last_change":"2024-09-13T13:14:50.392659+0000","last_active":"2024-09-13T13:16:13.057134+0000","last_peered":"2024-09-13T13:16:13.057134+0000","last_clean":"2024-09-13T13:16:13.057134+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:16:13.057134+0000","last_undegraded":"2024-09-13T13:16:13.057134+0000","last_fullsized":"2024-09-13T13:16:13.057134+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.392444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.392444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":710,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.072955+0000","last_change":"2024-09-13T13:15:03.778214+0000","last_active":"2024-09-13T13:16:13.072955+0000","last_peered":"2024-09-13T13:16:13.072955+0000","last_clean":"2024-09-13T13:16:13.072955+0000","last_became_active":"2024-09-13T13:12:52.999748+0000","last_became_peered":"2024-09-13T13:12:52.999748+0000","last_unstale":"2024-09-13T13:16:13.072955+0000","last_undegraded":"2024-09-13T13:16:13.072955+0000","last_fullsized":"2024-09-13T13:16:13.072955+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.778170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.778170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:20.178488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":668,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.076147+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:16:13.076147+0000","last_peered":"2024-09-13T13:16:13.076147+0000","last_clean":"2024-09-13T13:16:13.076147+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:16:13.076147+0000","last_undegraded":"2024-09-13T13:16:13.076147+0000","last_fullsized":"2024-09-13T13:16:13.076147+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:14.231726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":730,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075363+0000","last_change":"2024-09-13T13:16:12.090990+0000","last_active":"2024-09-13T13:16:13.075363+0000","last_peered":"2024-09-13T13:16:13.075363+0000","last_clean":"2024-09-13T13:16:13.075363+0000","last_became_active":"2024-09-13T13:16:12.073702+0000","last_became_peered":"2024-09-13T13:16:12.073702+0000","last_unstale":"2024-09-13T13:16:13.075363+0000","last_undegraded":"2024-09-13T13:16:13.075363+0000","last_fullsized":"2024-09-13T13:16:13.075363+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:14:12.368430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:12.368430+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":700,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058431+0000","last_change":"2024-09-13T13:13:44.750043+0000","last_active":"2024-09-13T13:16:13.058431+0000","last_peered":"2024-09-13T13:16:13.058431+0000","last_clean":"2024-09-13T13:16:13.058431+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:16:13.058431+0000","last_undegraded":"2024-09-13T13:16:13.058431+0000","last_fullsized":"2024-09-13T13:16:13.058431+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:44.749961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:44.749961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":618,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.079620+0000","last_change":"2024-09-13T13:14:16.457160+0000","last_active":"2024-09-13T13:16:16.079620+0000","last_peered":"2024-09-13T13:16:16.079620+0000","last_clean":"2024-09-13T13:16:16.079620+0000","last_became_active":"2024-09-13T13:12:53.001131+0000","last_became_peered":"2024-09-13T13:12:53.001131+0000","last_unstale":"2024-09-13T13:16:16.079620+0000","last_undegraded":"2024-09-13T13:16:16.079620+0000","last_fullsized":"2024-09-13T13:16:16.079620+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:16.456784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:16.456784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:43.279313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":525,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.057222+0000","last_change":"2024-09-13T13:16:12.068100+0000","last_active":"2024-09-13T13:16:13.057222+0000","last_peered":"2024-09-13T13:16:13.057222+0000","last_clean":"2024-09-13T13:16:13.057222+0000","last_became_active":"2024-09-13T13:16:12.067757+0000","last_became_peered":"2024-09-13T13:16:12.067757+0000","last_unstale":"2024-09-13T13:16:13.057222+0000","last_undegraded":"2024-09-13T13:16:13.057222+0000","last_fullsized":"2024-09-13T13:16:13.057222+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:05.828300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:05.828300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:18.999851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":696,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.055853+0000","last_change":"2024-09-13T13:14:18.882969+0000","last_active":"2024-09-13T13:16:13.055853+0000","last_peered":"2024-09-13T13:16:13.055853+0000","last_clean":"2024-09-13T13:16:13.055853+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:16:13.055853+0000","last_undegraded":"2024-09-13T13:16:13.055853+0000","last_fullsized":"2024-09-13T13:16:13.055853+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:18.882903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:18.882903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:46.394564+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":677,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.075587+0000","last_change":"2024-09-13T13:11:07.564094+0000","last_active":"2024-09-13T13:16:13.075587+0000","last_peered":"2024-09-13T13:16:13.075587+0000","last_clean":"2024-09-13T13:16:13.075587+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:16:13.075587+0000","last_undegraded":"2024-09-13T13:16:13.075587+0000","last_fullsized":"2024-09-13T13:16:13.075587+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:07.564025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:07.564025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:35.266907+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":705,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-13T13:16:16.080251+0000","last_change":"2024-09-13T13:16:12.087075+0000","last_active":"2024-09-13T13:16:16.080251+0000","last_peered":"2024-09-13T13:16:16.080251+0000","last_clean":"2024-09-13T13:16:16.080251+0000","last_became_active":"2024-09-13T13:16:12.070277+0000","last_became_peered":"2024-09-13T13:16:12.070277+0000","last_unstale":"2024-09-13T13:16:16.080251+0000","last_undegraded":"2024-09-13T13:16:16.080251+0000","last_fullsized":"2024-09-13T13:16:16.080251+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:49.264128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.264128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:13.408862+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":708,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T13:16:13.058169+0000","last_change":"2024-09-13T13:16:12.074365+0000","last_active":"2024-09-13T13:16:13.058169+0000","last_peered":"2024-09-13T13:16:13.058169+0000","last_clean":"2024-09-13T13:16:13.058169+0000","last_became_active":"2024-09-13T13:16:12.072684+0000","last_became_peered":"2024-09-13T13:16:12.072684+0000","last_unstale":"2024-09-13T13:16:13.058169+0000","last_undegraded":"2024-09-13T13:16:13.058169+0000","last_fullsized":"2024-09-13T13:16:13.058169+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:27.678752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:27.678752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:16:29.249237+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5554241560,"num_objects":84752,"num_object_clones":0,"num_object_copies":254256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":32056,"num_objects_unfound":0,"num_objects_dirty":84752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170004,"num_write_kb":8152129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16799248384,"data_stored":16799236168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78195,"ondisk_log_size":78195,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":3526,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":684,"ondisk_log_size":684,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":220,"num_read_kb":215,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201465,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1914620,"kb_used_data":1378088,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91830532,"statfs":{"total":95995035648,"available":94034464768,"internally_reserved":0,"allocated":1411162112,"data_stored":1410757801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":549386698},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141076,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2838604,"kb_used_data":2359676,"kb_used_omap":4,"kb_used_meta":478907,"kb_avail":90906548,"statfs":{"total":95995035648,"available":93088305152,"internally_reserved":0,"allocated":2416308224,"data_stored":2415889403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":490401453},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692037,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2267640,"kb_used_data":1722784,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":91477512,"statfs":{"total":95995035648,"available":93672972288,"internally_reserved":0,"allocated":1764130816,"data_stored":1763733103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920971,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3211916,"kb_used_data":2779508,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":90533236,"statfs":{"total":95995035648,"available":92706033664,"internally_reserved":0,"allocated":2846216192,"data_stored":2845798049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":442759622},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117202,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2328472,"kb_used_data":1701700,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":91416680,"statfs":{"total":95995035648,"available":93610680320,"internally_reserved":0,"allocated":1742540800,"data_stored":1742114589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":641792455},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411541,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575068,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1583660,"kb_used_data":1047124,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92161492,"statfs":{"total":95995035648,"available":94373367808,"internally_reserved":0,"allocated":1072254976,"data_stored":1071811769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738595,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4103644,"kb_used_data":3432452,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":89641508,"statfs":{"total":95995035648,"available":91792904192,"internally_reserved":0,"allocated":3514830848,"data_stored":3514404925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3475968000,"data_stored":3475963928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1058537472,"data_stored":1058537472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2124283904,"data_stored":2124283904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1740570624,"data_stored":1740570624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2845118464,"data_stored":2845114392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1760821248,"data_stored":1760821248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2384859136,"data_stored":2384855064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1409089536,"data_stored":1409089536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:16:23.189 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-13T13:16:23.200 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:23.257 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:23.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:23.359 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:23.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:23.409 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:23.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:23.510 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:23.547 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:23.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:23.648 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:23.679 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:23.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:23.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:23.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:23.821 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:23.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:23.922 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:23.955 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:23.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:24.057 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:24.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:23 smithi167 ceph-mon[33178]: pgmap v756: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s; 28752/237507 objects misplaced (12.106%); 8.5 MiB/s, 141 objects/s recovering 2024-09-13T13:16:24.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:23 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:24.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:23 smithi167 ceph-mon[33178]: from='client.36377 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:16:24.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:23 smithi167 ceph-mon[33178]: osdmap e381: 8 total, 8 up, 7 in 2024-09-13T13:16:24.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:23 smithi167 ceph-mon[33178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:16:24.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:24.093 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:24.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:24.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:23 smithi052 ceph-mon[29471]: pgmap v756: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s; 28752/237507 objects misplaced (12.106%); 8.5 MiB/s, 141 objects/s recovering 2024-09-13T13:16:24.151 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:23 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:24.151 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:23 smithi052 ceph-mon[29471]: from='client.36377 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:16:24.151 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:23 smithi052 ceph-mon[29471]: osdmap e381: 8 total, 8 up, 7 in 2024-09-13T13:16:24.151 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:23 smithi052 ceph-mon[29471]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:16:24.151 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:23 smithi052 ceph-mon[36682]: pgmap v756: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s; 28752/237507 objects misplaced (12.106%); 8.5 MiB/s, 141 objects/s recovering 2024-09-13T13:16:24.151 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:23 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:24.151 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:23 smithi052 ceph-mon[36682]: from='client.36377 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:16:24.151 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:23 smithi052 ceph-mon[36682]: osdmap e381: 8 total, 8 up, 7 in 2024-09-13T13:16:24.151 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:23 smithi052 ceph-mon[36682]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:16:24.195 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:24.233 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:24.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:24.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:24.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:24.370 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:24.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:24.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:24.413 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:24.466 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:16:24.466 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:16:24.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:16:24.466 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:16:24.467 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.467 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189885 9.4 9:25913ce2:::benchmark_data_smithi052_128427_object59216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-09-13T13:16:24.467 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:16:24.467 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.552025128, 2024-09-13T13:16:24.467 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.048349118, 2024-09-13T13:16:24.467 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.467 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.467 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:16:24.467 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.467 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:24.467 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:24.467 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189885 2024-09-13T13:16:24.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.468 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909609+0000", 2024-09-13T13:16:24.469 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0860000000000001e-06 2024-09-13T13:16:24.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.469 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.469 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909614+0000", 2024-09-13T13:16:24.469 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5199999999999999e-06 2024-09-13T13:16:24.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.469 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.469 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909627+0000", 2024-09-13T13:16:24.469 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3623e-05 2024-09-13T13:16:24.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909634+0000", 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0210000000000002e-06 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909697+0000", 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2769999999999997e-05 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909840+0000", 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014305399999999999 2024-09-13T13:16:24.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3,4", 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909967+0000", 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012707300000000001 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.913962+0000", 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039947009999999998 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.913977+0000", 2024-09-13T13:16:24.471 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.506e-05 2024-09-13T13:16:24.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.472 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.472 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.916641+0000", 2024-09-13T13:16:24.472 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026644839999999999 2024-09-13T13:16:24.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.472 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.472 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.937520+0000", 2024-09-13T13:16:24.472 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.020878821999999998 2024-09-13T13:16:24.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.472 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:24.473 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.937533+0000", 2024-09-13T13:16:24.473 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2896e-05 2024-09-13T13:16:24.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.473 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.473 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.473 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.957873+0000", 2024-09-13T13:16:24.473 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0203400570000001 2024-09-13T13:16:24.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.957957+0000", 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3950999999999996e-05 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189915 9.4 9:2fa1035b:::benchmark_data_smithi052_128427_object59246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.51869130399999, 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0238765249999999, 2024-09-13T13:16:24.474 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189915 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942944+0000", 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.655e-06 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942948+0000", 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3220000000000003e-06 2024-09-13T13:16:24.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942957+0000", 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2609999999999999e-06 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942964+0000", 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6379999999999998e-06 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.944838+0000", 2024-09-13T13:16:24.477 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018746419999999999 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.944845+0000", 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958435+0000", 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0135895370000001 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.478 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958549+0000", 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114144 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958677+0000", 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012798 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966592+0000", 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0079149440000000001 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.479 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966687+0000", 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4741999999999995e-05 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966725+0000", 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7768000000000002e-05 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966777+0000", 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2605999999999997e-05 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.480 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966818+0000", 2024-09-13T13:16:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1279999999999998e-05 2024-09-13T13:16:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189933 9.4 9:2eec938f:::benchmark_data_smithi052_128427_object59264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:16:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:16:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.503562706, 2024-09-13T13:16:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.008927224, 2024-09-13T13:16:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189933 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:16:24.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958072+0000", 2024-09-13T13:16:24.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7e-06 2024-09-13T13:16:24.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958078+0000", 2024-09-13T13:16:24.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4990000000000002e-06 2024-09-13T13:16:24.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958089+0000", 2024-09-13T13:16:24.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0865e-05 2024-09-13T13:16:24.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958095+0000", 2024-09-13T13:16:24.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8279999999999997e-06 2024-09-13T13:16:24.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.485 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958183+0000", 2024-09-13T13:16:24.485 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7655e-05 2024-09-13T13:16:24.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.485 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:24.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958191+0000", 2024-09-13T13:16:24.485 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9529999999999999e-06 2024-09-13T13:16:24.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.485 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961131+0000", 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0029402620000001 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961259+0000", 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012816999999999999 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961395+0000", 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013549800000000001 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966691+0000", 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0052965979999999996 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966864+0000", 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017277899999999999 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966899+0000", 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4377e-05 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966955+0000", 2024-09-13T13:16:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6495e-05 2024-09-13T13:16:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966998+0000", 2024-09-13T13:16:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2545e-05 2024-09-13T13:16:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189960 9.4 9:2cd09825:::benchmark_data_smithi052_128427_object59291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.47951723200001, 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99033151900000005, 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189960 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982118+0000", 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982123+0000", 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8389999999999997e-06 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982137+0000", 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4433000000000001e-05 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982145+0000", 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8290000000000008e-06 2024-09-13T13:16:24.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982197+0000", 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2431e-05 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982205+0000", 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8630000000000008e-06 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964055+0000", 2024-09-13T13:16:24.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98184965300000004 2024-09-13T13:16:24.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.493 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.493 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964156+0000", 2024-09-13T13:16:24.493 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101055 2024-09-13T13:16:24.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.493 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:16:24.493 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964261+0000", 2024-09-13T13:16:24.493 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104961 2024-09-13T13:16:24.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.493 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972104+0000", 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0078433300000000008 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972147+0000", 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3022000000000002e-05 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972189+0000", 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1727000000000002e-05 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.495 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.495 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972384+0000", 2024-09-13T13:16:24.495 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:24.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019545200000000001 2024-09-13T13:16:24.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.497 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.497 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972447+0000", 2024-09-13T13:16:24.497 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3144000000000004e-05 2024-09-13T13:16:24.497 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.497 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.497 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.497 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189972 9.4 9:28374290:::benchmark_data_smithi052_128427_object59303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:16:24.497 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:16:24.497 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.47094506600001, 2024-09-13T13:16:24.497 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99084523700000005, 2024-09-13T13:16:24.497 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189972 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990690+0000", 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990694+0000", 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6029999999999998e-06 2024-09-13T13:16:24.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.500 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.500 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990704+0000", 2024-09-13T13:16:24.500 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5060000000000001e-06 2024-09-13T13:16:24.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.500 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.500 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990710+0000", 2024-09-13T13:16:24.500 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990764+0000", 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4252000000000002e-05 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990773+0000", 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2379999999999997e-06 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.501 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967034+0000", 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97626137000000002 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967153+0000", 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118855 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967267+0000", 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113855 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.502 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.503 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972462+0000", 2024-09-13T13:16:24.503 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051953219999999996 2024-09-13T13:16:24.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.503 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.503 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.976022+0000", 2024-09-13T13:16:24.503 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003559563 2024-09-13T13:16:24.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.504 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981419+0000", 2024-09-13T13:16:24.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053977260000000003 2024-09-13T13:16:24.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.504 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981488+0000", 2024-09-13T13:16:24.504 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8950999999999998e-05 2024-09-13T13:16:24.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.504 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.504 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981533+0000", 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4981999999999997e-05 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202468 9.4 9:2e7e6063:::benchmark_data_smithi052_128427_object71799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 206.49572474799999, 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.030148343, 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.505 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202468 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965910+0000", 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2810000000000001e-06 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965915+0000", 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0749999999999997e-06 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965926+0000", 2024-09-13T13:16:24.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1056999999999999e-05 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965943+0000", 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7178000000000001e-05 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.966053+0000", 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109768 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:24.508 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.966062+0000", 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4029999999999996e-06 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.971764+0000", 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0057021909999999 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.971894+0000", 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000129987 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:16:24.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.972041+0000", 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000146398 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976575+0000", 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045341729999999998 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976634+0000", 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8996999999999998e-05 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.995927+0000", 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.019293457999999999 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.996003+0000", 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5221000000000004e-05 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.996057+0000", 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4156000000000001e-05 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.512 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.512 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202476 9.4 9:205ce74f:::benchmark_data_smithi052_128427_object71807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:16:24.512 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:16:24.512 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 206.488752035, 2024-09-13T13:16:24.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0253464960000001, 2024-09-13T13:16:24.512 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.512 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.512 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.512 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.512 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:24.512 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:24.513 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202476 2024-09-13T13:16:24.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.513 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:16:24.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:16:24.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972883+0000", 2024-09-13T13:16:24.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.792e-06 2024-09-13T13:16:24.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972888+0000", 2024-09-13T13:16:24.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7380000000000002e-06 2024-09-13T13:16:24.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972896+0000", 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9230000000000002e-06 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972903+0000", 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5310000000000004e-06 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.973002+0000", 2024-09-13T13:16:24.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8484000000000001e-05 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.973012+0000", 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0016e-05 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974404+0000", 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0013926820000001 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974530+0000", 2024-09-13T13:16:24.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012599 2024-09-13T13:16:24.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:16:24.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974651+0000", 2024-09-13T13:16:24.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012022400000000001 2024-09-13T13:16:24.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976695+0000", 2024-09-13T13:16:24.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020448139999999998 2024-09-13T13:16:24.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981646+0000", 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049504229999999998 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998145+0000", 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.016499092 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998183+0000", 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8457999999999997e-05 2024-09-13T13:16:24.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998228+0000", 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4329000000000002e-05 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202503 9.4 9:257a0b3a:::benchmark_data_smithi052_128427_object71834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 206.4697037, 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.009335557, 2024-09-13T13:16:24.519 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.520 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.520 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.520 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.520 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:24.520 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:24.520 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202503 2024-09-13T13:16:24.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.520 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.520 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:16:24.520 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991936+0000", 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1149999999999999e-06 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991941+0000", 2024-09-13T13:16:24.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.463e-06 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991954+0000", 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2533e-05 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991961+0000", 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1690000000000003e-06 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.992017+0000", 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5856999999999997e-05 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.992026+0000", 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976789+0000", 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98476255999999995 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976898+0000", 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109548 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.977048+0000", 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015011899999999999 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981632+0000", 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045837020000000003 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981701+0000", 2024-09-13T13:16:24.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8511999999999997e-05 2024-09-13T13:16:24.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001159+0000", 2024-09-13T13:16:24.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.019457953 2024-09-13T13:16:24.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001221+0000", 2024-09-13T13:16:24.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2229000000000003e-05 2024-09-13T13:16:24.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001265+0000", 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4267999999999997e-05 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202507 9.4 9:20f0d2e1:::benchmark_data_smithi052_128427_object71838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 206.46761855700001, 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.007409886, 2024-09-13T13:16:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202507 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:16:24.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994016+0000", 2024-09-13T13:16:24.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.001e-06 2024-09-13T13:16:24.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994020+0000", 2024-09-13T13:16:24.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2320000000000003e-06 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994031+0000", 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1259e-05 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994036+0000", 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3580000000000002e-06 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994089+0000", 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2120000000000002e-05 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994097+0000", 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8000000000000004e-06 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979607+0000", 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98551001400000005 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979708+0000", 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000100108 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979813+0000", 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000105238 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981689+0000", 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018764299999999999 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987043+0000", 2024-09-13T13:16:24.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053537139999999999 2024-09-13T13:16:24.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001312+0000", 2024-09-13T13:16:24.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014268729000000001 2024-09-13T13:16:24.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001378+0000", 2024-09-13T13:16:24.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5980000000000002e-05 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001424+0000", 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6903000000000001e-05 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202513 9.4 9:293cc3ac:::benchmark_data_smithi052_128427_object71844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 206.46216261500001, 2024-09-13T13:16:24.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0035790550000001, 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202513 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999472+0000", 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.536e-06 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999477+0000", 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7260000000000002e-06 2024-09-13T13:16:24.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999492+0000", 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5495999999999999e-05 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999500+0000", 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3100000000000003e-06 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999557+0000", 2024-09-13T13:16:24.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7491999999999998e-05 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999565+0000", 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9359999999999999e-06 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981724+0000", 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98215919600000001 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981838+0000", 2024-09-13T13:16:24.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114092 2024-09-13T13:16:24.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:16:24.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981966+0000", 2024-09-13T13:16:24.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012748300000000001 2024-09-13T13:16:24.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987109+0000", 2024-09-13T13:16:24.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.005142771 2024-09-13T13:16:24.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:24.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987186+0000", 2024-09-13T13:16:24.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7822000000000001e-05 2024-09-13T13:16:24.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.002955+0000", 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015768880999999998 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003010+0000", 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4959e-05 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003050+0000", 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9354999999999998e-05 2024-09-13T13:16:24.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.542 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.542 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.542 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:64970 10.2 10:4c910a20:::benchmark_data_smithi052_153703_object64969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T13:16:24.542 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:39.291709+0000", 2024-09-13T13:16:24.542 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 105.169923709, 2024-09-13T13:16:24.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.120818178, 2024-09-13T13:16:24.542 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.542 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.542 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.542 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.542 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 64970 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291709+0000", 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291709+0000", 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291711+0000", 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.266e-06 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291735+0000", 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4283000000000001e-05 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291745+0000", 2024-09-13T13:16:24.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0468999999999999e-05 2024-09-13T13:16:24.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.291753+0000", 2024-09-13T13:16:24.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4669999999999998e-06 2024-09-13T13:16:24.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.296536+0000", 2024-09-13T13:16:24.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047835270000000001 2024-09-13T13:16:24.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.296626+0000", 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0000000000000006e-05 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.296725+0000", 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8737e-05 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.301664+0000", 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049391289999999996 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.301686+0000", 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2092000000000001e-05 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.412397+0000", 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.11071099700000001 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.412480+0000", 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2267000000000004e-05 2024-09-13T13:16:24.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.412528+0000", 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7944e-05 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:64972 10.2 10:45880529:::benchmark_data_smithi052_153703_object64971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:39.292691+0000", 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 105.168942276, 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.120361115, 2024-09-13T13:16:24.548 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 64972 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.292691+0000", 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.292691+0000", 2024-09-13T13:16:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.292692+0000", 2024-09-13T13:16:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4020000000000001e-06 2024-09-13T13:16:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.292718+0000", 2024-09-13T13:16:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.563e-05 2024-09-13T13:16:24.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.292730+0000", 2024-09-13T13:16:24.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1809e-05 2024-09-13T13:16:24.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.292738+0000", 2024-09-13T13:16:24.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8390000000000007e-06 2024-09-13T13:16:24.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.299230+0000", 2024-09-13T13:16:24.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064920289999999999 2024-09-13T13:16:24.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.299400+0000", 2024-09-13T13:16:24.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017011899999999999 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.299498+0000", 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8152000000000005e-05 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.301710+0000", 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0022122750000000001 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.301794+0000", 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3387999999999995e-05 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.412922+0000", 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.111128907 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.413003+0000", 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0685000000000004e-05 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.413052+0000", 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.888e-05 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:66914 10.0 e348/334 10:022d7c38:::benchmark_data_smithi052_153703_object66913:head v 348'4227, mlcod=348'4225)", 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:40.914459+0000", 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 103.54717378799999, 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.174681006, 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.914459+0000", 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.914459+0000", 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.556 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.914461+0000", 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0040000000000002e-06 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.914582+0000", 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012074399999999999 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081614+0000", 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.167032352 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081630+0000", 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5679999999999999e-05 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081702+0000", 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1418e-05 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081726+0000", 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4161000000000001e-05 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.089092+0000", 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0073658389999999999 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.089140+0000", 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8807999999999998e-05 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.559 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:66935 10.2 10:4e89de86:::benchmark_data_smithi052_153703_object66934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T13:16:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:40.923262+0000", 2024-09-13T13:16:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 103.538371571, 2024-09-13T13:16:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16507603400000001, 2024-09-13T13:16:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 66935 2024-09-13T13:16:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.923262+0000", 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.923262+0000", 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.923264+0000", 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1490000000000001e-06 2024-09-13T13:16:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.923287+0000", 2024-09-13T13:16:24.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3023999999999998e-05 2024-09-13T13:16:24.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081630+0000", 2024-09-13T13:16:24.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15834292599999999 2024-09-13T13:16:24.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081641+0000", 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1077000000000001e-05 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081702+0000", 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1493000000000002e-05 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.081872+0000", 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016986900000000001 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.082195+0000", 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000323228 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.088213+0000", 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.006017924 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.088226+0000", 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2969000000000001e-05 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.088264+0000", 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7496999999999997e-05 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.088301+0000", 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7370999999999998e-05 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:41.088338+0000", 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6507000000000001e-05 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.565 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:88880 10.0 e356/334 10:0c76401e:::benchmark_data_smithi052_153703_object88879:head v 356'5629, mlcod=356'5627)", 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:58.470860+0000", 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 85.990773189999999, 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13166224100000001, 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.470860+0000", 2024-09-13T13:16:24.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.470860+0000", 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.470862+0000", 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9870000000000002e-06 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.567 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.470897+0000", 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4944000000000003e-05 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.470910+0000", 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3334e-05 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.470918+0000", 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8800000000000008e-06 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.470996+0000", 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8230000000000001e-05 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.471022+0000", 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5789e-05 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.474240+0000", 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0032174780000000002 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.570 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.570 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.602522+0000", 2024-09-13T13:16:24.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.128282599 2024-09-13T13:16:24.570 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.570 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.570 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.570 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:88885 10.b e356/334 10:d639df6f:::benchmark_data_smithi052_153703_object88884:head v 356'5509, mlcod=356'5507)", 2024-09-13T13:16:24.570 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:58.474276+0000", 2024-09-13T13:16:24.570 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 85.987356715000004, 2024-09-13T13:16:24.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.13858297, 2024-09-13T13:16:24.570 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.474276+0000", 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.474276+0000", 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.474278+0000", 2024-09-13T13:16:24.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.438e-06 2024-09-13T13:16:24.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.474312+0000", 2024-09-13T13:16:24.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4109999999999997e-05 2024-09-13T13:16:24.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.602676+0000", 2024-09-13T13:16:24.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12836368500000001 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.602685+0000", 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6399999999999992e-06 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.602743+0000", 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7754999999999998e-05 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.602775+0000", 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1579999999999999e-05 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.612808+0000", 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010033528999999999 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.612859+0000", 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1233000000000002e-05 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:88900 10.2 10:4e424d93:::benchmark_data_smithi052_153703_object88899:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:58.484632+0000", 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 85.977000712000006, 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.126609104, 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 88900 2024-09-13T13:16:24.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.484632+0000", 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.484632+0000", 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.484636+0000", 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.185e-06 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.484665+0000", 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9773000000000001e-05 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.602493+0000", 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.117827591 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.602513+0000", 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9621000000000001e-05 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.602580+0000", 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7644999999999994e-05 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.602737+0000", 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015664300000000001 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.602884+0000", 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000147575 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.611062+0000", 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0081774680000000002 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.611110+0000", 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7876e-05 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.611155+0000", 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4690000000000001e-05 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.611191+0000", 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6183999999999998e-05 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.611242+0000", 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0853000000000003e-05 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.580 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:178641 10.2 10:4e16fcd3:::benchmark_data_smithi052_153703_object51261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:16:22.254059+0000", 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 2.207574095, 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19440316599999999, 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:24.581 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 178641 2024-09-13T13:16:24.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.582 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:24.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:24.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.254059+0000", 2024-09-13T13:16:24.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.254059+0000", 2024-09-13T13:16:24.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.254060+0000", 2024-09-13T13:16:24.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3909999999999999e-06 2024-09-13T13:16:24.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.254065+0000", 2024-09-13T13:16:24.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8489999999999997e-06 2024-09-13T13:16:24.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.254074+0000", 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5690000000000005e-06 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.254079+0000", 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3530000000000002e-06 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.254128+0000", 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8659000000000003e-05 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.254260+0000", 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000132459 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.254398+0000", 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013801199999999999 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.257292+0000", 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0028931780000000002 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.258306+0000", 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010149130000000001 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.448363+0000", 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.190056432 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.448417+0000", 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4162000000000001e-05 2024-09-13T13:16:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.448462+0000", 2024-09-13T13:16:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5188999999999997e-05 2024-09-13T13:16:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:178644 10.b e380/334 10:d5b84eb6:::benchmark_data_smithi052_153703_object51264:head v 380'11129, mlcod=380'11127)", 2024-09-13T13:16:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:16:22.257866+0000", 2024-09-13T13:16:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 2.2037676290000001, 2024-09-13T13:16:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19070036500000001, 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.257866+0000", 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.257866+0000", 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.257867+0000", 2024-09-13T13:16:24.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.561e-06 2024-09-13T13:16:24.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.257874+0000", 2024-09-13T13:16:24.590 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7580000000000003e-06 2024-09-13T13:16:24.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.590 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.590 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.257884+0000", 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0329999999999999e-05 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.257891+0000", 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.63e-06 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.257949+0000", 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7837999999999997e-05 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.591 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.257968+0000", 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8902e-05 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.448522+0000", 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19055423599999999 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.448566+0000", 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4110000000000003e-05 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.592 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop_reply(client.26071.0:178641 10.2 e380/334 ondisk, result = 0)", 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:16:22.259456+0000", 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 2.2021771389999998, 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.18901357899999999, 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:24.593 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.259456+0000", 2024-09-13T13:16:24.594 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.594 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:24.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.259456+0000", 2024-09-13T13:16:24.594 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:24.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.594 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:24.594 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.259457+0000", 2024-09-13T13:16:24.594 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.46e-06 2024-09-13T13:16:24.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.259462+0000", 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9150000000000004e-06 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.259473+0000", 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0627e-05 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.259480+0000", 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0879999999999997e-06 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.259548+0000", 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7804000000000003e-05 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.448345+0000", 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.18879678699999999 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.448470+0000", 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000124898 2024-09-13T13:16:24.596 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.597 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:24.597 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:24.597 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:16:24.683 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:24.719 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:24.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:24.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:24.789 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:16:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:16:24.821 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:24.859 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:24.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:24.960 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:25.006 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:25.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:25.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:25.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:24 smithi167 ceph-mon[33178]: 5.0 scrub starts 2024-09-13T13:16:25.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:24 smithi167 ceph-mon[33178]: 5.0 scrub ok 2024-09-13T13:16:25.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:24 smithi167 ceph-mon[33178]: 5.2 scrub starts 2024-09-13T13:16:25.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:24 smithi167 ceph-mon[33178]: 5.2 scrub ok 2024-09-13T13:16:25.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:25.139 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:25.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:25.223 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:24 smithi052 ceph-mon[36682]: 5.0 scrub starts 2024-09-13T13:16:25.223 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:24 smithi052 ceph-mon[36682]: 5.0 scrub ok 2024-09-13T13:16:25.223 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:24 smithi052 ceph-mon[36682]: 5.2 scrub starts 2024-09-13T13:16:25.223 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:24 smithi052 ceph-mon[36682]: 5.2 scrub ok 2024-09-13T13:16:25.223 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:24 smithi052 ceph-mon[29471]: 5.0 scrub starts 2024-09-13T13:16:25.223 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:24 smithi052 ceph-mon[29471]: 5.0 scrub ok 2024-09-13T13:16:25.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:24 smithi052 ceph-mon[29471]: 5.2 scrub starts 2024-09-13T13:16:25.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:24 smithi052 ceph-mon[29471]: 5.2 scrub ok 2024-09-13T13:16:25.241 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:25.276 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:25.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:25.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:25.378 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:25.391 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:16:25.392 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:16:25.418 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:25.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:25.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:25.427 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:16:25.428 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:16:25.458 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:16:25.458 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:16:25.488 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:16:25.489 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:16:25.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:25.524 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:16:25.524 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:16:25.553 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:25.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:25.557 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:16:25.557 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:16:25.593 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:16:25.593 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:16:25.624 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:16:25.624 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:16:25.655 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:25.692 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:25.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:25.796 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:25.840 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:25.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:25.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:25.982 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:25.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:26.063 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:25 smithi052 ceph-mon[36682]: pgmap v758: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 650 B/s rd, 0 B/s wr, 1.17k op/s; 26551/231471 objects misplaced (11.471%); 19 MiB/s, 306 objects/s recovering 2024-09-13T13:16:26.064 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:26.064 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:25 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2266037857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:26.064 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:25 smithi052 ceph-mon[29471]: pgmap v758: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 650 B/s rd, 0 B/s wr, 1.17k op/s; 26551/231471 objects misplaced (11.471%); 19 MiB/s, 306 objects/s recovering 2024-09-13T13:16:26.064 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:26.064 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:25 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2266037857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:26.065 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T13:16:26.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:25 smithi167 ceph-mon[33178]: pgmap v758: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 650 B/s rd, 0 B/s wr, 1.17k op/s; 26551/231471 objects misplaced (11.471%); 19 MiB/s, 306 objects/s recovering 2024-09-13T13:16:26.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:26.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:25 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2266037857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:26.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:26.122 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:26.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:26.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:26.203 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:16:26.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:26.258 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:26.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:26.359 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:26.397 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:26.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:26.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:26.431 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:16:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:16:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:16:26.499 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:26.546 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:26.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:26.649 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:26.681 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:26.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:26.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:26.782 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:26.849 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:26.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:26.868 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:26.938 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:26.952 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:26.985 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:26.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:27.019 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:26 smithi052 ceph-mon[29471]: 3.7 scrub starts 2024-09-13T13:16:27.019 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:26 smithi052 ceph-mon[29471]: 3.7 scrub ok 2024-09-13T13:16:27.019 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:26 smithi052 ceph-mon[29471]: 6.3 scrub starts 2024-09-13T13:16:27.019 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:26 smithi052 ceph-mon[29471]: 6.3 scrub ok 2024-09-13T13:16:27.019 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:26 smithi052 ceph-mon[36682]: 3.7 scrub starts 2024-09-13T13:16:27.019 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:26 smithi052 ceph-mon[36682]: 3.7 scrub ok 2024-09-13T13:16:27.019 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:26 smithi052 ceph-mon[36682]: 6.3 scrub starts 2024-09-13T13:16:27.020 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:26 smithi052 ceph-mon[36682]: 6.3 scrub ok 2024-09-13T13:16:27.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:26 smithi167 ceph-mon[33178]: 3.7 scrub starts 2024-09-13T13:16:27.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:26 smithi167 ceph-mon[33178]: 3.7 scrub ok 2024-09-13T13:16:27.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:26 smithi167 ceph-mon[33178]: 6.3 scrub starts 2024-09-13T13:16:27.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:26 smithi167 ceph-mon[33178]: 6.3 scrub ok 2024-09-13T13:16:27.087 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:27.119 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:27.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:27.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:27.259 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:27.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:27.361 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:27.395 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:27.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:27.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:27.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:27.540 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:27.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:27.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:27.627 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:16:27.627 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:16:27.641 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:27.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:27.685 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:27.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:27.786 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:27.825 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:27.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:27.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:27.960 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:27.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:28.061 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:28.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:27 smithi167 ceph-mon[33178]: pgmap v759: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 25916/227454 objects misplaced (11.394%); 17 MiB/s, 271 objects/s recovering 2024-09-13T13:16:28.093 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:28.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:28.191 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 4] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:16:28.191 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:16:28.191 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-13T13:16:28.191 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T13:16:28.191 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T13:16:28.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-13T13:16:28.195 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:28.230 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:28.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:28.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:27 smithi052 ceph-mon[36682]: pgmap v759: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 25916/227454 objects misplaced (11.394%); 17 MiB/s, 271 objects/s recovering 2024-09-13T13:16:28.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:27 smithi052 ceph-mon[29471]: pgmap v759: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.03k op/s; 25916/227454 objects misplaced (11.394%); 17 MiB/s, 271 objects/s recovering 2024-09-13T13:16:28.331 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:28.369 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:28.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:28.471 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:28.504 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:28.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:28.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:28.606 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:28.639 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:28.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:28.741 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:28.780 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:28.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:28.814 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:16:28.882 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:28.921 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:28.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:28.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:28.946 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:16:29.022 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:29.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:29.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:29.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:29.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:29.187 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:29.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:29.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:29 smithi052 ceph-mon[29471]: pgmap v760: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 25544/221175 objects misplaced (11.549%); 15 MiB/s, 240 objects/s recovering 2024-09-13T13:16:29.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:29 smithi052 ceph-mon[29471]: 4.11 scrub starts 2024-09-13T13:16:29.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:29 smithi052 ceph-mon[29471]: 4.11 scrub ok 2024-09-13T13:16:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:29 smithi052 ceph-mon[36682]: pgmap v760: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 25544/221175 objects misplaced (11.549%); 15 MiB/s, 240 objects/s recovering 2024-09-13T13:16:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:29 smithi052 ceph-mon[36682]: 4.11 scrub starts 2024-09-13T13:16:29.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:29 smithi052 ceph-mon[36682]: 4.11 scrub ok 2024-09-13T13:16:29.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:29.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:29 smithi167 ceph-mon[33178]: pgmap v760: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 25544/221175 objects misplaced (11.549%); 15 MiB/s, 240 objects/s recovering 2024-09-13T13:16:29.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:29 smithi167 ceph-mon[33178]: 4.11 scrub starts 2024-09-13T13:16:29.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:29 smithi167 ceph-mon[33178]: 4.11 scrub ok 2024-09-13T13:16:29.326 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:29.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:29.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:29.428 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:29.467 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:29.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:29.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:29.569 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:29.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:29.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:29.702 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:29.718 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:16:29.718 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T13:16:29.718 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:16:29.731 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:29.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:29.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:29.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:29.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:29.869 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:29.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:29.970 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:30.010 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:30.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:30.112 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:30.151 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:30.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:30.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:30.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:30.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:30.291 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:30.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:30.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:30 smithi167 ceph-mon[33178]: 10.2 scrub starts 2024-09-13T13:16:30.393 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:30.428 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:30.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:30.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:30 smithi052 ceph-mon[29471]: 10.2 scrub starts 2024-09-13T13:16:30.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:30 smithi052 ceph-mon[36682]: 10.2 scrub starts 2024-09-13T13:16:30.529 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:30.562 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:30.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:30.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:30.696 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:30.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:30.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:30.798 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:30.834 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:30.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:30.936 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:30.971 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:30.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:30.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:31.073 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:31.104 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:31.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:31.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:31.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:31.228 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:31.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:31.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:31.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:31 smithi167 ceph-mon[33178]: 6.1 scrub starts 2024-09-13T13:16:31.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:31 smithi167 ceph-mon[33178]: 6.1 scrub ok 2024-09-13T13:16:31.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:31 smithi167 ceph-mon[33178]: 10.2 scrub ok 2024-09-13T13:16:31.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:31 smithi167 ceph-mon[33178]: pgmap v761: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 141 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 22897/210159 objects misplaced (10.895%); 15 MiB/s, 236 objects/s recovering 2024-09-13T13:16:31.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:31 smithi167 ceph-mon[33178]: 5.19 scrub starts 2024-09-13T13:16:31.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:31 smithi167 ceph-mon[33178]: 5.19 scrub ok 2024-09-13T13:16:31.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:31 smithi167 ceph-mon[33178]: 10.d scrub starts 2024-09-13T13:16:31.330 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:31.348 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[29471]: 6.1 scrub starts 2024-09-13T13:16:31.348 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[29471]: 6.1 scrub ok 2024-09-13T13:16:31.348 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[29471]: 10.2 scrub ok 2024-09-13T13:16:31.348 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[29471]: pgmap v761: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 141 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 22897/210159 objects misplaced (10.895%); 15 MiB/s, 236 objects/s recovering 2024-09-13T13:16:31.348 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[29471]: 5.19 scrub starts 2024-09-13T13:16:31.348 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[29471]: 5.19 scrub ok 2024-09-13T13:16:31.348 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[29471]: 10.d scrub starts 2024-09-13T13:16:31.349 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[36682]: 6.1 scrub starts 2024-09-13T13:16:31.349 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[36682]: 6.1 scrub ok 2024-09-13T13:16:31.349 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[36682]: 10.2 scrub ok 2024-09-13T13:16:31.349 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[36682]: pgmap v761: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 141 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 22897/210159 objects misplaced (10.895%); 15 MiB/s, 236 objects/s recovering 2024-09-13T13:16:31.349 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[36682]: 5.19 scrub starts 2024-09-13T13:16:31.349 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[36682]: 5.19 scrub ok 2024-09-13T13:16:31.349 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:31 smithi052 ceph-mon[36682]: 10.d scrub starts 2024-09-13T13:16:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:31.363 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:31.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:31.464 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:31.510 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:31.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:31.612 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:31.644 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:31.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:31.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:31.662 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T13:16:31.688 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:31.747 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:31.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:31.795 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:31.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:31.897 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:31.940 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:31.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:32.041 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:16:32.041 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:32.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:32.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:32.161 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:32 smithi052 ceph-mon[36682]: 4.6 scrub starts 2024-09-13T13:16:32.161 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:32 smithi052 ceph-mon[36682]: 4.6 scrub ok 2024-09-13T13:16:32.161 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:32 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2242330825' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:32.162 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:32 smithi052 ceph-mon[29471]: 4.6 scrub starts 2024-09-13T13:16:32.162 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:32 smithi052 ceph-mon[29471]: 4.6 scrub ok 2024-09-13T13:16:32.162 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:32 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2242330825' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:32.201 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:32.240 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:32.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:32.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:32 smithi167 ceph-mon[33178]: 4.6 scrub starts 2024-09-13T13:16:32.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:32 smithi167 ceph-mon[33178]: 4.6 scrub ok 2024-09-13T13:16:32.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:32 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2242330825' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:32.343 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:32.381 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:32.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:32.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:32.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:32.506 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:16:32.506 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:16:32.506 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:16:32.506 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:16:32.506 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:16:32.506 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:16:32.507 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:16:32.507 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:16:32.507 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:16:32.507 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:16:32.507 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:16:32.507 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:16:32.529 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:32.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:32.531 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:16:32.531 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:16:32.570 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:16:32.570 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:16:32.583 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:32.631 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:32.672 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:32.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:32.774 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:32.810 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:32.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:32.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:32.912 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:32.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:32.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:32.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:33.058 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:33.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:33.095 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:33.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:33.196 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:33.227 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:33.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[36682]: 10.d scrub ok 2024-09-13T13:16:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[36682]: pgmap v762: 148 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 140 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 22556/203901 objects misplaced (11.062%); 15 MiB/s, 236 objects/s recovering 2024-09-13T13:16:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:16:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2242330825' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:16:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[36682]: osdmap e382: 8 total, 8 up, 7 in 2024-09-13T13:16:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[36682]: 10.0 scrub starts 2024-09-13T13:16:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2827558326' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:33.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:16:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[29471]: 10.d scrub ok 2024-09-13T13:16:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[29471]: pgmap v762: 148 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 140 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 22556/203901 objects misplaced (11.062%); 15 MiB/s, 236 objects/s recovering 2024-09-13T13:16:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:16:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2242330825' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:16:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[29471]: osdmap e382: 8 total, 8 up, 7 in 2024-09-13T13:16:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[29471]: 10.0 scrub starts 2024-09-13T13:16:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2827558326' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:33.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:33 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:16:33.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:33 smithi167 ceph-mon[33178]: 10.d scrub ok 2024-09-13T13:16:33.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:33 smithi167 ceph-mon[33178]: pgmap v762: 148 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 140 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 22556/203901 objects misplaced (11.062%); 15 MiB/s, 236 objects/s recovering 2024-09-13T13:16:33.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:33 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:16:33.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:33 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2242330825' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:16:33.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:33 smithi167 ceph-mon[33178]: osdmap e382: 8 total, 8 up, 7 in 2024-09-13T13:16:33.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:33 smithi167 ceph-mon[33178]: 10.0 scrub starts 2024-09-13T13:16:33.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:33 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2827558326' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:33.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:33 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:16:33.329 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:33.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:33.382 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:33.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:33.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:33.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:33.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:33.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:33.631 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:33.673 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:33.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:33.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:33.775 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:33.824 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:33.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:33.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:33.964 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:33.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:34.066 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:34.092 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:34.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:34.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:34.193 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:34.219 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:34.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:34 smithi167 ceph-mon[33178]: 3.17 scrub starts 2024-09-13T13:16:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:34 smithi167 ceph-mon[33178]: 3.17 scrub ok 2024-09-13T13:16:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:34 smithi167 ceph-mon[33178]: 6.15 scrub starts 2024-09-13T13:16:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:34 smithi167 ceph-mon[33178]: 6.15 scrub ok 2024-09-13T13:16:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:34 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:16:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:34 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:16:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:34 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:34 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:34.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:34 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:16:34.322 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:34.345 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[36682]: 3.17 scrub starts 2024-09-13T13:16:34.346 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[36682]: 3.17 scrub ok 2024-09-13T13:16:34.346 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[36682]: 6.15 scrub starts 2024-09-13T13:16:34.346 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[36682]: 6.15 scrub ok 2024-09-13T13:16:34.346 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:16:34.346 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:16:34.346 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:34.346 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:34.347 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:16:34.347 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[29471]: 3.17 scrub starts 2024-09-13T13:16:34.347 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[29471]: 3.17 scrub ok 2024-09-13T13:16:34.347 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[29471]: 6.15 scrub starts 2024-09-13T13:16:34.348 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[29471]: 6.15 scrub ok 2024-09-13T13:16:34.348 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:16:34.348 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:16:34.348 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:34.348 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:34.348 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:34 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:16:34.365 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:34.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:34.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:34.401 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:16:34.401 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:16:34.402 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:16:34.402 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:16:34.402 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:16:34.466 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:34.506 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:34.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:34.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:34.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:34.652 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:34.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:34.754 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:34.799 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:34.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:34.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:16:34 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:16:34.900 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:34.932 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:34.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:35.034 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:35.071 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:35.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:35.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:35.212 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:35.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:35.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:35 smithi052 ceph-mon[36682]: 10.0 scrub ok 2024-09-13T13:16:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:35 smithi052 ceph-mon[36682]: 10.4 scrub starts 2024-09-13T13:16:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:35 smithi052 ceph-mon[36682]: pgmap v764: 148 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 140 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 20701/197958 objects misplaced (10.457%); 14 MiB/s, 221 objects/s recovering 2024-09-13T13:16:35.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:35.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:35 smithi052 ceph-mon[29471]: 10.0 scrub ok 2024-09-13T13:16:35.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:35 smithi052 ceph-mon[29471]: 10.4 scrub starts 2024-09-13T13:16:35.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:35 smithi052 ceph-mon[29471]: pgmap v764: 148 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 140 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 20701/197958 objects misplaced (10.457%); 14 MiB/s, 221 objects/s recovering 2024-09-13T13:16:35.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:35.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:35.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:35 smithi167 ceph-mon[33178]: 10.0 scrub ok 2024-09-13T13:16:35.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:35 smithi167 ceph-mon[33178]: 10.4 scrub starts 2024-09-13T13:16:35.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:35 smithi167 ceph-mon[33178]: pgmap v764: 148 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 140 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 20701/197958 objects misplaced (10.457%); 14 MiB/s, 221 objects/s recovering 2024-09-13T13:16:35.323 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:35.359 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:35.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:35.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:35.391 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:16:35.461 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:35.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:35.498 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:35.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:35.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:35.632 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:35.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:35.734 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:35.771 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:35.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:35.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:35.815 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:16:35.874 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:35.908 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:35.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:36.009 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:36.047 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:36.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:36.119 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:36 smithi052 ceph-mon[36682]: 10.4 scrub ok 2024-09-13T13:16:36.119 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:36 smithi052 ceph-mon[36682]: 6.f scrub starts 2024-09-13T13:16:36.119 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:36 smithi052 ceph-mon[36682]: 6.f scrub ok 2024-09-13T13:16:36.120 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:36 smithi052 ceph-mon[29471]: 10.4 scrub ok 2024-09-13T13:16:36.120 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:36 smithi052 ceph-mon[29471]: 6.f scrub starts 2024-09-13T13:16:36.120 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:36 smithi052 ceph-mon[29471]: 6.f scrub ok 2024-09-13T13:16:36.149 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:36.153 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T13:16:36.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:36.173 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:36.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:36.193 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:36.274 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:36.311 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:36.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:36.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:36.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:36 smithi167 ceph-mon[33178]: 10.4 scrub ok 2024-09-13T13:16:36.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:36 smithi167 ceph-mon[33178]: 6.f scrub starts 2024-09-13T13:16:36.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:36 smithi167 ceph-mon[33178]: 6.f scrub ok 2024-09-13T13:16:36.379 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:16:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:16:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:16:36.393 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 4] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:16:36.393 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T13:16:36.413 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:36.449 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:36.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:36.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:36.551 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:36.587 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:36.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:36.689 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:36.726 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:36.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:36.829 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:36.866 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:36.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:36.968 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:37.003 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:37.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:37.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:37.105 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:37.149 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:37.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:37.251 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:37.291 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:37.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:37.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:37 smithi167 ceph-mon[33178]: pgmap v765: 148 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 140 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s; 20078/194133 objects misplaced (10.342%); 14 MiB/s, 221 objects/s recovering 2024-09-13T13:16:37.364 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:37 smithi052 ceph-mon[29471]: pgmap v765: 148 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 140 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s; 20078/194133 objects misplaced (10.342%); 14 MiB/s, 221 objects/s recovering 2024-09-13T13:16:37.365 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:37 smithi052 ceph-mon[36682]: pgmap v765: 148 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 140 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s; 20078/194133 objects misplaced (10.342%); 14 MiB/s, 221 objects/s recovering 2024-09-13T13:16:37.392 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:37.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:37.429 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:37.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:37.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:37.560 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:37.565 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:37.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:37.584 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:37.667 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:37.710 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:37.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:37.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:37.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:37.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:37.848 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:37.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:37.949 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:37.989 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:37.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:38.076 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:16:38.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:38.131 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:38.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:38.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:38.233 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:38.266 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:38.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:38 smithi167 ceph-mon[33178]: 3.4 scrub starts 2024-09-13T13:16:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:38 smithi167 ceph-mon[33178]: 3.4 scrub ok 2024-09-13T13:16:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:38 smithi167 ceph-mon[33178]: 3.18 scrub starts 2024-09-13T13:16:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:38 smithi167 ceph-mon[33178]: 3.18 scrub ok 2024-09-13T13:16:38.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:38 smithi167 ceph-mon[33178]: 3.a scrub starts 2024-09-13T13:16:38.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:38 smithi167 ceph-mon[33178]: 3.a scrub ok 2024-09-13T13:16:38.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:38 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1654823917' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:16:38.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:38.402 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:38.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:38.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[36682]: 3.4 scrub starts 2024-09-13T13:16:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[36682]: 3.4 scrub ok 2024-09-13T13:16:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[36682]: 3.18 scrub starts 2024-09-13T13:16:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[36682]: 3.18 scrub ok 2024-09-13T13:16:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[36682]: 3.a scrub starts 2024-09-13T13:16:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[36682]: 3.a scrub ok 2024-09-13T13:16:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1654823917' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:16:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[29471]: 3.4 scrub starts 2024-09-13T13:16:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[29471]: 3.4 scrub ok 2024-09-13T13:16:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[29471]: 3.18 scrub starts 2024-09-13T13:16:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[29471]: 3.18 scrub ok 2024-09-13T13:16:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[29471]: 3.a scrub starts 2024-09-13T13:16:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[29471]: 3.a scrub ok 2024-09-13T13:16:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1654823917' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:16:38.504 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:38.537 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:38.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:38.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:38.639 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:38.671 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:38.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:38.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:38.774 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:38.805 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:38.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:38.908 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:38.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:38.945 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:38.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:39.046 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:39.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:39.088 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:39.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:39.189 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:39.194 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:16:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:16:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:16:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:16:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112327 9.2 e275/241 9:464b334b:::benchmark_data_smithi052_128427_object112326:head v 275'7019, mlcod=275'7017)", 2024-09-13T13:16:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.572648+0000", 2024-09-13T13:16:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 309.619104483, 2024-09-13T13:16:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.103891961, 2024-09-13T13:16:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572648+0000", 2024-09-13T13:16:39.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572648+0000", 2024-09-13T13:16:39.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572650+0000", 2024-09-13T13:16:39.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.717e-06 2024-09-13T13:16:39.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572678+0000", 2024-09-13T13:16:39.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7985999999999999e-05 2024-09-13T13:16:39.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572690+0000", 2024-09-13T13:16:39.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2262999999999999e-05 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572699+0000", 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9600000000000006e-06 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572782+0000", 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3338999999999996e-05 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.572804+0000", 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1314999999999998e-05 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676479+0000", 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.103675184 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676540+0000", 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1197000000000004e-05 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.199 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.200 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112328 9.2 e275/241 9:45698ec9:::benchmark_data_smithi052_128427_object112327:head v 275'7020, mlcod=275'7017)", 2024-09-13T13:16:39.200 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.576519+0000", 2024-09-13T13:16:39.200 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 309.61523335800001, 2024-09-13T13:16:39.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10005291099999999, 2024-09-13T13:16:39.200 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.200 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.200 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.200 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.200 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576519+0000", 2024-09-13T13:16:39.201 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.201 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.201 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576519+0000", 2024-09-13T13:16:39.201 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.201 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.201 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576521+0000", 2024-09-13T13:16:39.201 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6080000000000001e-06 2024-09-13T13:16:39.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576544+0000", 2024-09-13T13:16:39.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3791999999999999e-05 2024-09-13T13:16:39.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576556+0000", 2024-09-13T13:16:39.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1671000000000001e-05 2024-09-13T13:16:39.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.576564+0000", 2024-09-13T13:16:39.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8290000000000008e-06 2024-09-13T13:16:39.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.578118+0000", 2024-09-13T13:16:39.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0015537820000000001 2024-09-13T13:16:39.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.578138+0000", 2024-09-13T13:16:39.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0152999999999998e-05 2024-09-13T13:16:39.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676545+0000", 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098407005000000006 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676572+0000", 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7070999999999999e-05 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112330 9.2 e275/241 9:4591f68b:::benchmark_data_smithi052_128427_object112329:head v 275'7021, mlcod=275'7017)", 2024-09-13T13:16:39.204 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.579400+0000", 2024-09-13T13:16:39.205 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 309.61235213700002, 2024-09-13T13:16:39.205 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10414786299999999, 2024-09-13T13:16:39.205 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.205 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.205 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.205 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.205 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.205 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579400+0000", 2024-09-13T13:16:39.205 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.205 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579400+0000", 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579402+0000", 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7e-06 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579446+0000", 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.439e-05 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579458+0000", 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1477999999999999e-05 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.579465+0000", 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.114e-06 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676607+0000", 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.097141839999999993 2024-09-13T13:16:39.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676630+0000", 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3332e-05 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.683496+0000", 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0068658679999999998 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.683548+0000", 2024-09-13T13:16:39.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2141000000000001e-05 2024-09-13T13:16:39.209 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.209 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.209 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.209 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112336 9.0 e275/241 9:041ef402:::benchmark_data_smithi052_128427_object112335:head v 275'7019, mlcod=275'7017)", 2024-09-13T13:16:39.209 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.580484+0000", 2024-09-13T13:16:39.209 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 309.61126842200002, 2024-09-13T13:16:39.210 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.10093202900000001, 2024-09-13T13:16:39.210 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.210 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.210 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.210 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.210 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.210 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580484+0000", 2024-09-13T13:16:39.210 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.210 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580484+0000", 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580486+0000", 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.53e-06 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580530+0000", 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4589999999999998e-05 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580541+0000", 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0779e-05 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580548+0000", 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3019999999999997e-06 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.580604+0000", 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6271999999999999e-05 2024-09-13T13:16:39.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.213 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.213 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676458+0000", 2024-09-13T13:16:39.213 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.095853820000000006 2024-09-13T13:16:39.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.213 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.213 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.681299+0000", 2024-09-13T13:16:39.213 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048409280000000004 2024-09-13T13:16:39.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.213 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.214 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.681416+0000", 2024-09-13T13:16:39.214 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116808 2024-09-13T13:16:39.214 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.214 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.214 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.214 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112346 9.6 e275/241 9:67a240c7:::benchmark_data_smithi052_128427_object112345:head v 275'7035, mlcod=275'7032)", 2024-09-13T13:16:39.214 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.587147+0000", 2024-09-13T13:16:39.214 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 309.60460587099999, 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.097407753, 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587147+0000", 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587147+0000", 2024-09-13T13:16:39.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587149+0000", 2024-09-13T13:16:39.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.075e-06 2024-09-13T13:16:39.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587225+0000", 2024-09-13T13:16:39.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5964000000000007e-05 2024-09-13T13:16:39.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.217 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.677080+0000", 2024-09-13T13:16:39.217 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.089855655000000006 2024-09-13T13:16:39.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.217 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.217 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.217 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.677087+0000", 2024-09-13T13:16:39.217 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0199999999999997e-06 2024-09-13T13:16:39.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.217 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.217 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.217 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.677138+0000", 2024-09-13T13:16:39.217 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0773000000000003e-05 2024-09-13T13:16:39.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.677157+0000", 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8910999999999999e-05 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.684462+0000", 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0073047269999999996 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.684554+0000", 2024-09-13T13:16:39.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2627999999999995e-05 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:112348 9.2 e275/241 9:414aa897:::benchmark_data_smithi052_128427_object112347:head v 275'7022, mlcod=275'7017)", 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:11:29.587656+0000", 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 309.60409671999997, 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.098073530000000006, 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587656+0000", 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587656+0000", 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587657+0000", 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.658e-06 2024-09-13T13:16:39.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.221 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.221 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.587683+0000", 2024-09-13T13:16:39.221 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5253999999999999e-05 2024-09-13T13:16:39.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.221 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.221 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676615+0000", 2024-09-13T13:16:39.221 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.088932251000000004 2024-09-13T13:16:39.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.221 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.676624+0000", 2024-09-13T13:16:39.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4150000000000005e-06 2024-09-13T13:16:39.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.222 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.679045+0000", 2024-09-13T13:16:39.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024209600000000002 2024-09-13T13:16:39.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.222 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.679070+0000", 2024-09-13T13:16:39.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4967e-05 2024-09-13T13:16:39.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.223 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.685690+0000", 2024-09-13T13:16:39.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0066195150000000003 2024-09-13T13:16:39.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.223 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:11:29.685729+0000", 2024-09-13T13:16:39.224 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9509999999999999e-05 2024-09-13T13:16:39.224 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.224 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.224 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:161588 9.6 e289/241 9:6d59046d:::benchmark_data_smithi052_128427_object30919:head v 289'10101, mlcod=289'10099)", 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.716022+0000", 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 264.475729978, 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.234028868, 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716022+0000", 2024-09-13T13:16:39.225 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.226 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.226 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716022+0000", 2024-09-13T13:16:39.226 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.226 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.226 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716024+0000", 2024-09-13T13:16:39.226 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7880000000000001e-06 2024-09-13T13:16:39.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.226 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716034+0000", 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5219999999999996e-06 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716214+0000", 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018008299999999999 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716459+0000", 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024492300000000003 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.228 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.228 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.716537+0000", 2024-09-13T13:16:39.228 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8596000000000004e-05 2024-09-13T13:16:39.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.228 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.228 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.228 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942849+0000", 2024-09-13T13:16:39.228 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.22631210600000001 2024-09-13T13:16:39.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.228 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.228 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.228 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.949963+0000", 2024-09-13T13:16:39.228 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0071133680000000001 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.950051+0000", 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8481999999999998e-05 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:161595 9.2 e289/241 9:4df2328c:::benchmark_data_smithi052_128427_object30926:head v 289'10015, mlcod=289'10013)", 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.717193+0000", 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 264.47455963800002, 2024-09-13T13:16:39.229 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.23484664699999999, 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.717193+0000", 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.717193+0000", 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.717194+0000", 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.579e-06 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.717201+0000", 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942820+0000", 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.225618547 2024-09-13T13:16:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942836+0000", 2024-09-13T13:16:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6680999999999998e-05 2024-09-13T13:16:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942951+0000", 2024-09-13T13:16:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011408000000000001 2024-09-13T13:16:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942972+0000", 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1250000000000002e-05 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.951998+0000", 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0090262640000000009 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.952039+0000", 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1378999999999998e-05 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:161591 9.6 e289/241 9:60592e87:::benchmark_data_smithi052_128427_object30922:head v 289'10102, mlcod=289'10099)", 2024-09-13T13:16:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.718993+0000", 2024-09-13T13:16:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 264.47275946799999, 2024-09-13T13:16:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.23666279500000001, 2024-09-13T13:16:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.718993+0000", 2024-09-13T13:16:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.718993+0000", 2024-09-13T13:16:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.718995+0000", 2024-09-13T13:16:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7990000000000001e-06 2024-09-13T13:16:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.719004+0000", 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3829999999999998e-06 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942844+0000", 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.22383993799999999 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.942855+0000", 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0499000000000001e-05 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.945174+0000", 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023197339999999999 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.945191+0000", 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7088e-05 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.955462+0000", 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010270646 2024-09-13T13:16:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.955656+0000", 2024-09-13T13:16:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019370800000000001 2024-09-13T13:16:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.15789.0:161608 9.0 e289/241 9:0a1fbe34:::benchmark_data_smithi052_128427_object30939:head v 289'10085, mlcod=289'10083)", 2024-09-13T13:16:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:14.734825+0000", 2024-09-13T13:16:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 264.45692765400003, 2024-09-13T13:16:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.21376037000000001, 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.734825+0000", 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.734825+0000", 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.240 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:39.241 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.734827+0000", 2024-09-13T13:16:39.241 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1610000000000001e-06 2024-09-13T13:16:39.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.242 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.242 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.734836+0000", 2024-09-13T13:16:39.242 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2879999999999998e-06 2024-09-13T13:16:39.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.242 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.242 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.943089+0000", 2024-09-13T13:16:39.242 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.208252828 2024-09-13T13:16:39.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.242 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.943096+0000", 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5429999999999999e-06 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.943141+0000", 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5708999999999999e-05 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.943164+0000", 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3146000000000002e-05 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.244 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.244 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.948488+0000", 2024-09-13T13:16:39.244 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053236239999999999 2024-09-13T13:16:39.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.244 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.244 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.244 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:14.948585+0000", 2024-09-13T13:16:39.244 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7071000000000006e-05 2024-09-13T13:16:39.244 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.244 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.244 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.244 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.245 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:65338 10.5 10:afcb50c5:::benchmark_data_smithi052_153703_object65337:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T13:16:39.245 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:39.641456+0000", 2024-09-13T13:16:39.245 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 119.55029593899999, 2024-09-13T13:16:39.245 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.091897169000000001, 2024-09-13T13:16:39.245 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.245 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.245 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.245 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:39.245 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:39.245 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:39.245 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 65338 2024-09-13T13:16:39.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.641456+0000", 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.641456+0000", 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.246 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.641458+0000", 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9420000000000002e-06 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.641493+0000", 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5002999999999997e-05 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.641505+0000", 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1775e-05 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.641513+0000", 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4059999999999999e-06 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.642550+0000", 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010372739999999999 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.642684+0000", 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000134227 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.248 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:16:39.249 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.642824+0000", 2024-09-13T13:16:39.249 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013940900000000001 2024-09-13T13:16:39.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.249 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:39.249 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.645077+0000", 2024-09-13T13:16:39.249 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0022537099999999999 2024-09-13T13:16:39.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.249 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.249 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.645704+0000", 2024-09-13T13:16:39.249 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00062725499999999996 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.733248+0000", 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.087543338999999998 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.733307+0000", 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9330000000000003e-05 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.733354+0000", 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6499000000000003e-05 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:65343 10.5 10:ac1ef633:::benchmark_data_smithi052_153703_object65342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:39.646928+0000", 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 119.54482471999999, 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.086710518, 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 65343 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.646928+0000", 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.646928+0000", 2024-09-13T13:16:39.252 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.646930+0000", 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2409999999999998e-06 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.646965+0000", 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4718000000000002e-05 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.253 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.647027+0000", 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2244999999999998e-05 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.647036+0000", 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6030000000000005e-06 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.647308+0000", 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027276999999999998 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.254 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.647462+0000", 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015327400000000001 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.719842+0000", 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.072380825999999995 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.722018+0000", 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0021753850000000002 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.722241+0000", 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000222728 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.733544+0000", 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.01130394 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.733599+0000", 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4514e-05 2024-09-13T13:16:39.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.257 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.257 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:39.733638+0000", 2024-09-13T13:16:39.257 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9273999999999997e-05 2024-09-13T13:16:39.257 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.257 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.257 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.257 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:66295 10.5 10:abeb546a:::benchmark_data_smithi052_153703_object66294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-13T13:16:39.257 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:40.342110+0000", 2024-09-13T13:16:39.257 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 118.849642448, 2024-09-13T13:16:39.257 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.12606862099999999, 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 66295 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.342110+0000", 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.259 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.259 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.342110+0000", 2024-09-13T13:16:39.259 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.259 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.259 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.342112+0000", 2024-09-13T13:16:39.259 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2330000000000001e-06 2024-09-13T13:16:39.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.259 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.259 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.342138+0000", 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5367e-05 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.342149+0000", 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1606e-05 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.342157+0000", 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4229999999999999e-06 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.260 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.346355+0000", 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041979959999999998 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.346546+0000", 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019147700000000001 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.346680+0000", 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000133869 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.262 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:39.262 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.350282+0000", 2024-09-13T13:16:39.262 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003602278 2024-09-13T13:16:39.262 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.262 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.262 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.262 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.356096+0000", 2024-09-13T13:16:39.262 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058141089999999996 2024-09-13T13:16:39.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.263 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.263 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.468075+0000", 2024-09-13T13:16:39.263 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.111978381 2024-09-13T13:16:39.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.263 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.263 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.468135+0000", 2024-09-13T13:16:39.263 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9880000000000003e-05 2024-09-13T13:16:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:40.468179+0000", 2024-09-13T13:16:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4002000000000003e-05 2024-09-13T13:16:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:89395 10.5 10:a13f1dbd:::benchmark_data_smithi052_153703_object89394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-13T13:16:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:14:58.961021+0000", 2024-09-13T13:16:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 100.23073168000001, 2024-09-13T13:16:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.086411634000000001, 2024-09-13T13:16:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 89395 2024-09-13T13:16:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.961021+0000", 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.961021+0000", 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.961023+0000", 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.79e-06 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.961046+0000", 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3042999999999999e-05 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.961056+0000", 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0222e-05 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.961063+0000", 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2289999999999997e-06 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.961198+0000", 2024-09-13T13:16:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013449600000000001 2024-09-13T13:16:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.961335+0000", 2024-09-13T13:16:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000137577 2024-09-13T13:16:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:16:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.961482+0000", 2024-09-13T13:16:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000147322 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.966235+0000", 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047521990000000004 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:58.967486+0000", 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012517699999999999 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:59.047332+0000", 2024-09-13T13:16:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.079845943000000003 2024-09-13T13:16:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:59.047390+0000", 2024-09-13T13:16:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7422e-05 2024-09-13T13:16:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:14:59.047432+0000", 2024-09-13T13:16:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2620999999999997e-05 2024-09-13T13:16:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:152851 10.3 10:c93b7c18:::benchmark_data_smithi052_153703_object25471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:57.507904+0000", 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 41.683848230999999, 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.175907164, 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 152851 2024-09-13T13:16:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.507904+0000", 2024-09-13T13:16:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.507904+0000", 2024-09-13T13:16:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.507906+0000", 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3400000000000001e-06 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.507910+0000", 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5950000000000001e-06 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.507919+0000", 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8149999999999995e-06 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.507925+0000", 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0329999999999997e-06 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.511805+0000", 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0038796450000000001 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.511947+0000", 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014200199999999999 2024-09-13T13:16:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.512102+0000", 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015531599999999999 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.526558+0000", 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014455641 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.526610+0000", 2024-09-13T13:16:39.275 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2791000000000003e-05 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.683714+0000", 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.157103881 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.683755+0000", 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0389000000000002e-05 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.683811+0000", 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6715999999999997e-05 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:152852 10.3 10:c496abdb:::benchmark_data_smithi052_153703_object25472:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:57.508577+0000", 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 41.683175744000003, 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.182504904, 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 152852 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.508577+0000", 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.508577+0000", 2024-09-13T13:16:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.508578+0000", 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.308e-06 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.508583+0000", 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5870000000000003e-06 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.508593+0000", 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0861e-05 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.508601+0000", 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7020000000000001e-06 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.516382+0000", 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0077809630000000001 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.516489+0000", 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107032 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.516596+0000", 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010684900000000001 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.526562+0000", 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.009966295 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.532079+0000", 2024-09-13T13:16:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055166030000000001 2024-09-13T13:16:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.690953+0000", 2024-09-13T13:16:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15887380600000001 2024-09-13T13:16:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.691040+0000", 2024-09-13T13:16:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7596000000000006e-05 2024-09-13T13:16:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.691082+0000", 2024-09-13T13:16:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1301999999999999e-05 2024-09-13T13:16:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:152854 10.3 10:c7f56f16:::benchmark_data_smithi052_153703_object25474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-13T13:16:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:57.512379+0000", 2024-09-13T13:16:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 41.679373666000004, 2024-09-13T13:16:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.17881715500000001, 2024-09-13T13:16:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 152854 2024-09-13T13:16:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.512379+0000", 2024-09-13T13:16:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.512379+0000", 2024-09-13T13:16:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.512381+0000", 2024-09-13T13:16:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9309999999999998e-06 2024-09-13T13:16:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.512387+0000", 2024-09-13T13:16:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3940000000000001e-06 2024-09-13T13:16:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.512397+0000", 2024-09-13T13:16:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.006e-05 2024-09-13T13:16:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.512404+0000", 2024-09-13T13:16:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0910000000000004e-06 2024-09-13T13:16:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.519902+0000", 2024-09-13T13:16:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0074977000000000004 2024-09-13T13:16:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.520030+0000", 2024-09-13T13:16:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000127568 2024-09-13T13:16:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:16:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.520142+0000", 2024-09-13T13:16:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112129 2024-09-13T13:16:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.526567+0000", 2024-09-13T13:16:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064250009999999996 2024-09-13T13:16:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.532121+0000", 2024-09-13T13:16:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055543750000000003 2024-09-13T13:16:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.691119+0000", 2024-09-13T13:16:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.15899749499999999 2024-09-13T13:16:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.691164+0000", 2024-09-13T13:16:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5002e-05 2024-09-13T13:16:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.691196+0000", 2024-09-13T13:16:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2409000000000001e-05 2024-09-13T13:16:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:152863 10.3 10:c343d15d:::benchmark_data_smithi052_153703_object25483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-13T13:16:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:57.518768+0000", 2024-09-13T13:16:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 41.672984352999997, 2024-09-13T13:16:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.18189238399999999, 2024-09-13T13:16:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 152863 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.518768+0000", 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.518768+0000", 2024-09-13T13:16:39.291 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.518771+0000", 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4509999999999999e-06 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.518777+0000", 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.054e-06 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.292 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.518790+0000", 2024-09-13T13:16:39.293 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3422e-05 2024-09-13T13:16:39.293 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.293 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.293 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.293 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.518800+0000", 2024-09-13T13:16:39.293 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5429999999999999e-06 2024-09-13T13:16:39.293 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.293 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.293 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.293 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.522981+0000", 2024-09-13T13:16:39.293 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041814740000000001 2024-09-13T13:16:39.293 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.293 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.523111+0000", 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000129713 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.523231+0000", 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000119974 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.527041+0000", 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00380986 2024-09-13T13:16:39.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.532153+0000", 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051122709999999998 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.700552+0000", 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.168399248 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.700615+0000", 2024-09-13T13:16:39.295 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3260000000000001e-05 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.700660+0000", 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5114000000000002e-05 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:152870 10.3 10:cf778e5e:::benchmark_data_smithi052_153703_object25490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:57.527164+0000", 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 41.664588408, 2024-09-13T13:16:39.296 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.17366701100000001, 2024-09-13T13:16:39.297 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.297 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.297 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.297 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:39.297 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:39.297 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:39.297 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 152870 2024-09-13T13:16:39.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.297 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.297 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.297 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.297 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.527164+0000", 2024-09-13T13:16:39.297 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.298 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.298 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.298 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.298 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:39.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:39.299 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.527164+0000", 2024-09-13T13:16:39.299 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.299 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.299 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.527166+0000", 2024-09-13T13:16:39.299 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3929999999999998e-06 2024-09-13T13:16:39.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.299 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.299 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.527172+0000", 2024-09-13T13:16:39.300 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7629999999999997e-06 2024-09-13T13:16:39.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.300 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.300 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.527186+0000", 2024-09-13T13:16:39.300 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3348e-05 2024-09-13T13:16:39.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.300 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.300 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.527193+0000", 2024-09-13T13:16:39.300 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4830000000000002e-06 2024-09-13T13:16:39.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.301 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.301 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.527429+0000", 2024-09-13T13:16:39.301 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00023570000000000001 2024-09-13T13:16:39.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.301 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.301 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.301 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.527565+0000", 2024-09-13T13:16:39.301 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000136351 2024-09-13T13:16:39.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.301 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.301 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:16:39.301 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.527685+0000", 2024-09-13T13:16:39.301 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120382 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.530947+0000", 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003261403 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.534158+0000", 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0032107120000000001 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.302 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.700736+0000", 2024-09-13T13:16:39.303 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16657850299999999 2024-09-13T13:16:39.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.303 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.303 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.303 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.700791+0000", 2024-09-13T13:16:39.303 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4669000000000001e-05 2024-09-13T13:16:39.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.303 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.303 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.303 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.700831+0000", 2024-09-13T13:16:39.303 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0303999999999999e-05 2024-09-13T13:16:39.303 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.303 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:152873 10.3 10:ce494d25:::benchmark_data_smithi052_153703_object25493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:57.534855+0000", 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 41.656897407000002, 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.16763587099999999, 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:39.304 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 152873 2024-09-13T13:16:39.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.305 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:39.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.305 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:39.305 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.534855+0000", 2024-09-13T13:16:39.305 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.305 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:39.305 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.534855+0000", 2024-09-13T13:16:39.305 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:39.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.534857+0000", 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5680000000000001e-06 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.534861+0000", 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6e-06 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.534874+0000", 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2503999999999999e-05 2024-09-13T13:16:39.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.534879+0000", 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.118e-06 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.534929+0000", 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0359999999999999e-05 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.535061+0000", 2024-09-13T13:16:39.307 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013173899999999999 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.535204+0000", 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000143311 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.537859+0000", 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026548380000000001 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.308 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.538419+0000", 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00055944499999999999 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.702292+0000", 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.163873028 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.702444+0000", 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015291799999999999 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:39.309 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:39.310 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:57.702491+0000", 2024-09-13T13:16:39.310 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6442e-05 2024-09-13T13:16:39.310 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.310 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.310 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.310 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:39.310 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:39.310 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:16:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:39 smithi167 ceph-mon[33178]: 4.16 scrub starts 2024-09-13T13:16:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:39 smithi167 ceph-mon[33178]: 4.16 scrub ok 2024-09-13T13:16:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:39 smithi167 ceph-mon[33178]: pgmap v766: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 141 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 19780/188583 objects misplaced (10.489%); 14 MiB/s, 221 objects/s recovering 2024-09-13T13:16:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:39 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:39 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1654823917' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:16:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:39 smithi167 ceph-mon[33178]: osdmap e383: 8 total, 8 up, 7 in 2024-09-13T13:16:39.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:39.435 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:39.436 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:39 smithi052 ceph-mon[36682]: 4.16 scrub starts 2024-09-13T13:16:39.436 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:39 smithi052 ceph-mon[36682]: 4.16 scrub ok 2024-09-13T13:16:39.436 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:39 smithi052 ceph-mon[36682]: pgmap v766: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 141 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 19780/188583 objects misplaced (10.489%); 14 MiB/s, 221 objects/s recovering 2024-09-13T13:16:39.436 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:39 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:39.436 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:39 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1654823917' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:16:39.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:39 smithi052 ceph-mon[36682]: osdmap e383: 8 total, 8 up, 7 in 2024-09-13T13:16:39.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:39 smithi052 ceph-mon[29471]: 4.16 scrub starts 2024-09-13T13:16:39.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:39 smithi052 ceph-mon[29471]: 4.16 scrub ok 2024-09-13T13:16:39.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:39 smithi052 ceph-mon[29471]: pgmap v766: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 141 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 19780/188583 objects misplaced (10.489%); 14 MiB/s, 221 objects/s recovering 2024-09-13T13:16:39.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:39 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:39.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:39 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1654823917' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:16:39.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:39 smithi052 ceph-mon[29471]: osdmap e383: 8 total, 8 up, 7 in 2024-09-13T13:16:39.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:39.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:39.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:39.577 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:39.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:39.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:39.680 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:39.683 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:16:39.683 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:16:39.683 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:16:39.683 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:16:39.683 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:16:39.684 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:16:39.684 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:16:39.684 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:16:39.684 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:16:39.684 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:16:39.684 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:16:39.684 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:16:39.684 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:16:39.684 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:16:39.684 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:16:39.685 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:16:39.695 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:39.711 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:39.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:39.813 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:39.850 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:39.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:39.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:39.952 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:39.991 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:39.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:39.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:40.093 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:40.129 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:40.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:40.231 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:40.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:40.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:40.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:40.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:40.411 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:40.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:40.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:40 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3536080757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:40.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:40 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3536080757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:40.512 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:40.548 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:40.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:40.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:40 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3536080757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:40.595 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T13:16:40.650 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:40.686 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:40.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:40.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:40.788 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:40.830 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:40.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:40.930 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:16:40.930 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:40.962 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:40.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:40.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:41.064 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:41.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:41.119 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:41.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:41.156 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:41 smithi052 ceph-mon[36682]: pgmap v768: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 17096/178884 objects misplaced (9.557%); 20 MiB/s, 326 objects/s recovering 2024-09-13T13:16:41.156 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:41 smithi052 ceph-mon[29471]: pgmap v768: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 17096/178884 objects misplaced (9.557%); 20 MiB/s, 326 objects/s recovering 2024-09-13T13:16:41.221 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:41.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:41.260 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:41.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:41.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:41.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:41.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:41.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:41.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:41.535 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:41.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:41.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:41.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:41 smithi167 ceph-mon[33178]: pgmap v768: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 17096/178884 objects misplaced (9.557%); 20 MiB/s, 326 objects/s recovering 2024-09-13T13:16:41.637 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:41.668 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:41.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:41.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:41.674 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:41.770 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:41.808 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:41.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:41.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:41.910 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:41.942 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:41.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:41.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:42.044 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:42.079 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:42.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:42.095 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T13:16:42.181 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:42.218 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:42.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:42.319 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:42.329 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:16:42.329 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:16:42.359 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:42.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:42.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:42.461 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:42.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/992896035' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:16:42.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:42 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/992896035' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:16:42.494 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:42.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:42.523 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:42.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:42 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/992896035' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:16:42.597 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:42.631 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:42.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:42.733 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:42.762 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:42.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:42.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:42.865 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:42.898 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:42.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:43.000 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:43.037 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:43.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:43.139 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:43.182 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:43.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:43.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:43.323 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:43.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:43.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:43.425 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:43.459 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:43.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:43.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:43.468 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:43 smithi052 ceph-mon[29471]: pgmap v769: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 3.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s; 16745/172587 objects misplaced (9.702%); 16 MiB/s, 263 objects/s recovering 2024-09-13T13:16:43.468 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/992896035' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:16:43.468 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:43 smithi052 ceph-mon[29471]: osdmap e384: 8 total, 8 up, 7 in 2024-09-13T13:16:43.468 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:43 smithi052 ceph-mon[36682]: pgmap v769: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 3.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s; 16745/172587 objects misplaced (9.702%); 16 MiB/s, 263 objects/s recovering 2024-09-13T13:16:43.468 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/992896035' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:16:43.468 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:43 smithi052 ceph-mon[36682]: osdmap e384: 8 total, 8 up, 7 in 2024-09-13T13:16:43.518 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T13:16:43.561 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:43.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:43 smithi167 ceph-mon[33178]: pgmap v769: 148 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 142 active+clean; 3.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s; 16745/172587 objects misplaced (9.702%); 16 MiB/s, 263 objects/s recovering 2024-09-13T13:16:43.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/992896035' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:16:43.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:43 smithi167 ceph-mon[33178]: osdmap e384: 8 total, 8 up, 7 in 2024-09-13T13:16:43.598 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:43.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:43.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:43.614 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:16:43.700 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:43.742 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:43.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:43.842 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:43.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:43.906 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:43.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:44.007 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:44.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:44.050 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:44.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:44.143 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T13:16:44.152 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:44.222 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:44.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:44.323 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:44.366 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:44.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:44.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:44.397 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:44.438 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:44.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:44.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:44 smithi052 ceph-mon[36682]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:16:44.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:44 smithi052 ceph-mon[29471]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:16:44.490 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:44 smithi167 ceph-mon[33178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:16:44.503 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:44.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:44.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:44.643 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:44.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:44.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:44.745 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:44.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:44.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:16:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:16:44.893 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:44.927 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:44.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:45.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:45.066 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:45.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:45.158 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:16:45.168 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:45.195 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:45.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:45.297 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:45.321 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:45.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:45.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:45.371 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:45 smithi052 ceph-mon[36682]: pgmap v771: 148 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 142 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.18k op/s; 15147/165837 objects misplaced (9.134%); 20 MiB/s, 312 objects/s recovering 2024-09-13T13:16:45.371 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:45 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2895547106' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:45.371 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:45 smithi052 ceph-mon[36682]: from='client.26341 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:45.371 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:45 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:45.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:45 smithi052 ceph-mon[29471]: pgmap v771: 148 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 142 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.18k op/s; 15147/165837 objects misplaced (9.134%); 20 MiB/s, 312 objects/s recovering 2024-09-13T13:16:45.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:45 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2895547106' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:45.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:45 smithi052 ceph-mon[29471]: from='client.26341 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:45.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:45 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:45.422 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:45.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:45.468 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:45.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:45.569 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:45.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:45 smithi167 ceph-mon[33178]: pgmap v771: 148 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 142 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.18k op/s; 15147/165837 objects misplaced (9.134%); 20 MiB/s, 312 objects/s recovering 2024-09-13T13:16:45.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:45 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2895547106' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:45 smithi167 ceph-mon[33178]: from='client.26341 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:45 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:45.575 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:16:45.575 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:16:45.575 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:16:45.575 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:16:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:16:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:16:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:16:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:16:45.576 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:16:45.576 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:16:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:16:45.577 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:16:45.586 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:16:45.587 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:16:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:45.610 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:45.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:45.620 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:16:45.620 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:16:45.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:45.749 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:45.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:45.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:45.851 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:45.880 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:45.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:45.982 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:46.006 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:46.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:46.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:46.108 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:46.155 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:46.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:46.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:46.187 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T13:16:46.257 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:46.304 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:46.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:46.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:46.384 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:16:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:16:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:16:46.384 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:46 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:46.384 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:46 smithi052 ceph-mon[36682]: from='client.26341 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:16:46.384 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:46 smithi052 ceph-mon[36682]: osdmap e385: 8 total, 8 up, 7 in 2024-09-13T13:16:46.385 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:46 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1541198123' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:46.385 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:46 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2655832038' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:16:46.385 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:46 smithi052 ceph-mon[36682]: from='client.36422 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:16:46.385 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:46 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:46.385 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:46 smithi052 ceph-mon[29471]: from='client.26341 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:16:46.385 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:46 smithi052 ceph-mon[29471]: osdmap e385: 8 total, 8 up, 7 in 2024-09-13T13:16:46.385 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:46 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1541198123' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:46.385 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:46 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2655832038' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:16:46.385 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:46 smithi052 ceph-mon[29471]: from='client.36422 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:16:46.406 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:46.440 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:46.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:46.543 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:46.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:46 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:46.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:46 smithi167 ceph-mon[33178]: from='client.26341 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:16:46.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:46 smithi167 ceph-mon[33178]: osdmap e385: 8 total, 8 up, 7 in 2024-09-13T13:16:46.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:46 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1541198123' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:46.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:46 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2655832038' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:16:46.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:46 smithi167 ceph-mon[33178]: from='client.36422 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:16:46.577 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:46.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:46.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:46.679 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:46.717 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:46.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:46.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:46.819 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:46.853 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:46.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:46.955 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:46.992 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:46.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:47.094 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:47.122 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:47.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:47.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:47.128 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T13:16:47.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:47.259 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:47.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:47.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:47.328 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:47.361 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:47.397 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:47.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:47.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:47 smithi052 ceph-mon[29471]: pgmap v773: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.14k op/s; 14136/161718 objects misplaced (8.741%); 22 MiB/s, 358 objects/s recovering 2024-09-13T13:16:47.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:47 smithi052 ceph-mon[29471]: from='client.36422 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:16:47.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:47 smithi052 ceph-mon[29471]: osdmap e386: 8 total, 8 up, 7 in 2024-09-13T13:16:47.429 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:47 smithi052 ceph-mon[36682]: pgmap v773: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.14k op/s; 14136/161718 objects misplaced (8.741%); 22 MiB/s, 358 objects/s recovering 2024-09-13T13:16:47.429 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:47 smithi052 ceph-mon[36682]: from='client.36422 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:16:47.429 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:47 smithi052 ceph-mon[36682]: osdmap e386: 8 total, 8 up, 7 in 2024-09-13T13:16:47.499 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:47.529 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:47.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:47.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:47.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:47 smithi167 ceph-mon[33178]: pgmap v773: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.14k op/s; 14136/161718 objects misplaced (8.741%); 22 MiB/s, 358 objects/s recovering 2024-09-13T13:16:47.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:47 smithi167 ceph-mon[33178]: from='client.36422 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:16:47.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:47 smithi167 ceph-mon[33178]: osdmap e386: 8 total, 8 up, 7 in 2024-09-13T13:16:47.631 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:47.656 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:47.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:47.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:47.757 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:47.782 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:47.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:47.884 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:47.910 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:47.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:48.012 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:48.035 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:48.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:48.137 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:48.165 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:48.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:48.267 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:48.297 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:48.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:48.399 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:48.430 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:48 smithi052 ceph-mon[36682]: osdmap e387: 8 total, 8 up, 7 in 2024-09-13T13:16:48.430 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:48 smithi052 ceph-mon[29471]: osdmap e387: 8 total, 8 up, 7 in 2024-09-13T13:16:48.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:48.446 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:48.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:48.487 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:16:48.487 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:16:48.487 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:16:48.487 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:16:48.487 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:16:48.504 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:48.547 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:48.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:48 smithi167 ceph-mon[33178]: osdmap e387: 8 total, 8 up, 7 in 2024-09-13T13:16:48.582 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:48.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:48.684 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:48.717 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:48.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:48.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:48.819 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:48.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:48.855 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:48.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:48.956 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:48.990 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:48.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:48.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:49.092 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:49.124 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:49.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:49.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:49.251 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T13:16:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:49.267 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:49.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:49.276 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:16:49.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:49.404 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:49.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:49.475 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:49 smithi052 ceph-mon[36682]: pgmap v776: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 874 B/s rd, 0 B/s wr, 941 op/s; 13434/156063 objects misplaced (8.608%); 19 MiB/s, 299 objects/s recovering 2024-09-13T13:16:49.475 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:49 smithi052 ceph-mon[36682]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T13:16:49.475 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:49 smithi052 ceph-mon[36682]: osdmap e388: 8 total, 8 up, 7 in 2024-09-13T13:16:49.475 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:49 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:49.475 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:49 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:16:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:49 smithi052 ceph-mon[29471]: pgmap v776: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 874 B/s rd, 0 B/s wr, 941 op/s; 13434/156063 objects misplaced (8.608%); 19 MiB/s, 299 objects/s recovering 2024-09-13T13:16:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:49 smithi052 ceph-mon[29471]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T13:16:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:49 smithi052 ceph-mon[29471]: osdmap e388: 8 total, 8 up, 7 in 2024-09-13T13:16:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:49 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:49 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:16:49.505 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:49.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:49.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:49.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:49.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:49 smithi167 ceph-mon[33178]: pgmap v776: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 874 B/s rd, 0 B/s wr, 941 op/s; 13434/156063 objects misplaced (8.608%); 19 MiB/s, 299 objects/s recovering 2024-09-13T13:16:49.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:49 smithi167 ceph-mon[33178]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T13:16:49.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:49 smithi167 ceph-mon[33178]: osdmap e388: 8 total, 8 up, 7 in 2024-09-13T13:16:49.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:49 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:16:49.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:49 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:16:49.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:49.659 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:49.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:49.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:49.788 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:49.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:49.843 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:49.850 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:16:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:49.890 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:49.918 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:49.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:49.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:50.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:50.048 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:50.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:50.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:50.178 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:50.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:50.269 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T13:16:50.280 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:50.307 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:50.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:50.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:50.386 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:50 smithi052 ceph-mon[36682]: osdmap e389: 8 total, 8 up, 7 in 2024-09-13T13:16:50.387 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:50 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4147103573' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:16:50.387 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:50 smithi052 ceph-mon[36682]: from='client.36434 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:16:50.387 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:50 smithi052 ceph-mon[29471]: osdmap e389: 8 total, 8 up, 7 in 2024-09-13T13:16:50.387 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:50 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4147103573' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:16:50.387 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:50 smithi052 ceph-mon[29471]: from='client.36434 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:16:50.410 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:50.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:50.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:50.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:50.552 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:50.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:50 smithi167 ceph-mon[33178]: osdmap e389: 8 total, 8 up, 7 in 2024-09-13T13:16:50.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:50 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4147103573' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:16:50.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:50 smithi167 ceph-mon[33178]: from='client.36434 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:16:50.587 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:50.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:50.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:50.689 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:50.721 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:50.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:50.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:50.853 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:16:50.854 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:16:50.854 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:16:50.854 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:16:50.854 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.854 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190217 9.c 9:3da1d190:::benchmark_data_smithi052_128427_object59548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:16:50.854 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:43.982184+0000", 2024-09-13T13:16:50.854 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 246.86865788899999, 2024-09-13T13:16:50.854 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99036358000000002, 2024-09-13T13:16:50.854 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.854 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.855 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.855 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.855 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.855 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.855 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190217 2024-09-13T13:16:50.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.855 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.855 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.855 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982184+0000", 2024-09-13T13:16:50.855 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.856 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982184+0000", 2024-09-13T13:16:50.856 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.856 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982185+0000", 2024-09-13T13:16:50.856 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5859999999999999e-06 2024-09-13T13:16:50.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.857 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982190+0000", 2024-09-13T13:16:50.857 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9749999999999998e-06 2024-09-13T13:16:50.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.857 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.857 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.857 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982200+0000", 2024-09-13T13:16:50.857 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2310000000000002e-06 2024-09-13T13:16:50.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.857 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.857 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.857 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982209+0000", 2024-09-13T13:16:50.857 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9220000000000003e-06 2024-09-13T13:16:50.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.858 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.858 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.858 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982540+0000", 2024-09-13T13:16:50.858 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033174499999999999 2024-09-13T13:16:50.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.858 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.858 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.858 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982550+0000", 2024-09-13T13:16:50.858 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5629999999999998e-06 2024-09-13T13:16:50.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.858 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.859 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.859 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.966848+0000", 2024-09-13T13:16:50.859 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.9842978 2024-09-13T13:16:50.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.859 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.859 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.966978+0000", 2024-09-13T13:16:50.859 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013044200000000001 2024-09-13T13:16:50.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.859 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:16:50.859 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.967121+0000", 2024-09-13T13:16:50.860 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000143096 2024-09-13T13:16:50.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.860 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.860 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.860 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972248+0000", 2024-09-13T13:16:50.860 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051265260000000002 2024-09-13T13:16:50.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.860 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.860 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.860 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972262+0000", 2024-09-13T13:16:50.860 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4037e-05 2024-09-13T13:16:50.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.861 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.861 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972452+0000", 2024-09-13T13:16:50.861 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019067699999999999 2024-09-13T13:16:50.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.861 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.861 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972517+0000", 2024-09-13T13:16:50.861 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4935000000000001e-05 2024-09-13T13:16:50.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.861 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972547+0000", 2024-09-13T13:16:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0045000000000001e-05 2024-09-13T13:16:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190219 9.c 9:30476813:::benchmark_data_smithi052_128427_object59550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:16:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:43.982580+0000", 2024-09-13T13:16:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 246.86826174699999, 2024-09-13T13:16:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.990151738, 2024-09-13T13:16:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.862 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190219 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982580+0000", 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.863 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982580+0000", 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982581+0000", 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0249999999999999e-06 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982585+0000", 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1189999999999999e-06 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.864 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982595+0000", 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0074999999999999e-05 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982601+0000", 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2149999999999998e-06 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982868+0000", 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00026633000000000001 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.865 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982876+0000", 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8900000000000007e-06 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.969658+0000", 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98678257999999996 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.969792+0000", 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013425500000000001 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.969926+0000", 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013390200000000001 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972556+0000", 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026292170000000001 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972600+0000", 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4697999999999998e-05 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.867 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.868 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.868 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972657+0000", 2024-09-13T13:16:50.868 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6453000000000003e-05 2024-09-13T13:16:50.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.868 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.868 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.868 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972711+0000", 2024-09-13T13:16:50.868 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.452e-05 2024-09-13T13:16:50.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:50.870 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.870 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.870 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972732+0000", 2024-09-13T13:16:50.870 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0458999999999998e-05 2024-09-13T13:16:50.870 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.870 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.870 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.870 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190252 9.c 9:3916637a:::benchmark_data_smithi052_128427_object59583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.013719+0000", 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 246.837123021, 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96423604299999999, 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190252 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.871 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.872 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.872 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013719+0000", 2024-09-13T13:16:50.872 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.872 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.872 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013719+0000", 2024-09-13T13:16:50.872 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.872 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.872 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013720+0000", 2024-09-13T13:16:50.872 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7039999999999999e-06 2024-09-13T13:16:50.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.873 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.873 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013726+0000", 2024-09-13T13:16:50.873 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3299999999999998e-06 2024-09-13T13:16:50.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.873 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.873 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013745+0000", 2024-09-13T13:16:50.873 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9623000000000002e-05 2024-09-13T13:16:50.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.873 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013753+0000", 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.498e-06 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.014204+0000", 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00045132700000000001 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.014212+0000", 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1510000000000004e-06 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972762+0000", 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.958549706 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972843+0000", 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1415000000000006e-05 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972915+0000", 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.875 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976630+0000", 2024-09-13T13:16:50.876 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037151879999999999 2024-09-13T13:16:50.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.876 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976739+0000", 2024-09-13T13:16:50.876 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109011 2024-09-13T13:16:50.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.876 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.977890+0000", 2024-09-13T13:16:50.876 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011513350000000001 2024-09-13T13:16:50.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.877 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.877 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.877 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.977916+0000", 2024-09-13T13:16:50.877 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5948e-05 2024-09-13T13:16:50.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.877 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.877 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.877 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.977955+0000", 2024-09-13T13:16:50.877 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8451999999999997e-05 2024-09-13T13:16:50.877 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.877 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.877 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190263 9.c 9:307a9428:::benchmark_data_smithi052_128427_object59594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:16:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.022090+0000", 2024-09-13T13:16:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 246.828752092, 2024-09-13T13:16:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95977093599999996, 2024-09-13T13:16:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190263 2024-09-13T13:16:50.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.879 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.879 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.879 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022090+0000", 2024-09-13T13:16:50.879 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.879 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.879 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022090+0000", 2024-09-13T13:16:50.879 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.880 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.880 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022092+0000", 2024-09-13T13:16:50.880 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8870000000000001e-06 2024-09-13T13:16:50.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.880 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.880 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022097+0000", 2024-09-13T13:16:50.880 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.31e-06 2024-09-13T13:16:50.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.880 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.881 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022108+0000", 2024-09-13T13:16:50.881 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0941000000000001e-05 2024-09-13T13:16:50.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.881 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.881 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.881 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022115+0000", 2024-09-13T13:16:50.881 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3050000000000004e-06 2024-09-13T13:16:50.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.881 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.881 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.881 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022210+0000", 2024-09-13T13:16:50.881 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5169999999999999e-05 2024-09-13T13:16:50.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.882 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.882 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.882 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022219+0000", 2024-09-13T13:16:50.882 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1500000000000005e-06 2024-09-13T13:16:50.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.882 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.882 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.882 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.974786+0000", 2024-09-13T13:16:50.882 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95256630799999997 2024-09-13T13:16:50.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.882 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.882 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.882 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.974884+0000", 2024-09-13T13:16:50.883 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8256000000000003e-05 2024-09-13T13:16:50.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.883 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:16:50.883 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.975003+0000", 2024-09-13T13:16:50.883 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011911100000000001 2024-09-13T13:16:50.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.883 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.883 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976679+0000", 2024-09-13T13:16:50.883 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016762019999999999 2024-09-13T13:16:50.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.884 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.884 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.978004+0000", 2024-09-13T13:16:50.884 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001324605 2024-09-13T13:16:50.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.884 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.884 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981764+0000", 2024-09-13T13:16:50.884 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003760556 2024-09-13T13:16:50.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.884 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.884 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981816+0000", 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1412999999999997e-05 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981861+0000", 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4722000000000003e-05 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190265 9.c 9:3c2318c0:::benchmark_data_smithi052_128427_object59596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.022643+0000", 2024-09-13T13:16:50.885 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 246.828199033, 2024-09-13T13:16:50.886 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95946595400000001, 2024-09-13T13:16:50.886 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.886 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.886 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.886 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.886 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.886 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.886 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190265 2024-09-13T13:16:50.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.886 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.886 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.887 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022643+0000", 2024-09-13T13:16:50.887 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.887 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.887 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022643+0000", 2024-09-13T13:16:50.887 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.887 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.887 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022644+0000", 2024-09-13T13:16:50.888 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.282e-06 2024-09-13T13:16:50.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.888 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.888 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022648+0000", 2024-09-13T13:16:50.888 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3189999999999996e-06 2024-09-13T13:16:50.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.888 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.888 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022657+0000", 2024-09-13T13:16:50.888 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1710000000000008e-06 2024-09-13T13:16:50.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.889 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.889 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022664+0000", 2024-09-13T13:16:50.889 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8600000000000004e-06 2024-09-13T13:16:50.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.889 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.889 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022843+0000", 2024-09-13T13:16:50.889 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017918 2024-09-13T13:16:50.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.889 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.890 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022851+0000", 2024-09-13T13:16:50.890 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5970000000000003e-06 2024-09-13T13:16:50.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.890 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.890 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976779+0000", 2024-09-13T13:16:50.890 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95392760200000004 2024-09-13T13:16:50.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.890 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.890 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976846+0000", 2024-09-13T13:16:50.890 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7539000000000005e-05 2024-09-13T13:16:50.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.891 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:16:50.891 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976913+0000", 2024-09-13T13:16:50.891 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6785999999999994e-05 2024-09-13T13:16:50.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.891 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.891 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981913+0000", 2024-09-13T13:16:50.891 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050001179999999996 2024-09-13T13:16:50.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.891 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.891 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981978+0000", 2024-09-13T13:16:50.892 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4917000000000002e-05 2024-09-13T13:16:50.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.892 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.892 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.892 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982009+0000", 2024-09-13T13:16:50.892 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0499999999999999e-05 2024-09-13T13:16:50.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.892 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.892 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.892 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982067+0000", 2024-09-13T13:16:50.892 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8573000000000003e-05 2024-09-13T13:16:50.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.892 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.893 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.893 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982109+0000", 2024-09-13T13:16:50.893 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1510000000000001e-05 2024-09-13T13:16:50.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.893 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.893 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190292 9.c 9:3187c4a7:::benchmark_data_smithi052_128427_object59623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:16:50.893 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.039937+0000", 2024-09-13T13:16:50.893 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 246.81090508099999, 2024-09-13T13:16:50.893 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95233577000000003, 2024-09-13T13:16:50.894 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.894 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.894 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.894 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.894 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.894 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.894 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190292 2024-09-13T13:16:50.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.894 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.894 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.895 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.895 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039937+0000", 2024-09-13T13:16:50.895 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.895 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.895 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039937+0000", 2024-09-13T13:16:50.895 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.895 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.895 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039938+0000", 2024-09-13T13:16:50.896 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.776e-06 2024-09-13T13:16:50.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.896 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.896 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039944+0000", 2024-09-13T13:16:50.896 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5949999999999998e-06 2024-09-13T13:16:50.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.896 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.896 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039959+0000", 2024-09-13T13:16:50.896 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4854e-05 2024-09-13T13:16:50.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.897 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.897 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039966+0000", 2024-09-13T13:16:50.897 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2239999999999998e-06 2024-09-13T13:16:50.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.897 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.897 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.897 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.040018+0000", 2024-09-13T13:16:50.897 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1578e-05 2024-09-13T13:16:50.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.897 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.897 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.897 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.040025+0000", 2024-09-13T13:16:50.898 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6690000000000006e-06 2024-09-13T13:16:50.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.898 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.898 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982498+0000", 2024-09-13T13:16:50.898 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.94247312100000002 2024-09-13T13:16:50.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.898 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.898 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982590+0000", 2024-09-13T13:16:50.898 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1283e-05 2024-09-13T13:16:50.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.899 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:16:50.899 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982692+0000", 2024-09-13T13:16:50.899 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102144 2024-09-13T13:16:50.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.899 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.899 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.987375+0000", 2024-09-13T13:16:50.899 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0046834329999999999 2024-09-13T13:16:50.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.899 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.899 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.987419+0000", 2024-09-13T13:16:50.899 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3282999999999999e-05 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992172+0000", 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047538839999999999 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992231+0000", 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8329999999999999e-05 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.900 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992272+0000", 2024-09-13T13:16:50.901 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1596e-05 2024-09-13T13:16:50.901 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.901 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.901 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.901 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.901 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190295 9.c 9:3895a0c0:::benchmark_data_smithi052_128427_object59626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:16:50.901 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.041745+0000", 2024-09-13T13:16:50.901 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 246.80909658300001, 2024-09-13T13:16:50.901 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96494661000000004, 2024-09-13T13:16:50.901 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.901 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.901 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190295 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041745+0000", 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041745+0000", 2024-09-13T13:16:50.902 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041746+0000", 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2619999999999999e-06 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041751+0000", 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.216e-06 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.903 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041759+0000", 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7129999999999994e-06 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041805+0000", 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5739999999999999e-05 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041899+0000", 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3598999999999996e-05 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.904 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.905 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041907+0000", 2024-09-13T13:16:50.905 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4169999999999992e-06 2024-09-13T13:16:50.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.905 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.905 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.984980+0000", 2024-09-13T13:16:50.905 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.94307334399999998 2024-09-13T13:16:50.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.905 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.905 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.985078+0000", 2024-09-13T13:16:50.905 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7633000000000005e-05 2024-09-13T13:16:50.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.985188+0000", 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110245 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989611+0000", 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0044224889999999999 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.994760+0000", 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051493700000000003 2024-09-13T13:16:50.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.907 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.907 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.006626+0000", 2024-09-13T13:16:50.907 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011865504000000001 2024-09-13T13:16:50.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.907 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.907 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.006651+0000", 2024-09-13T13:16:50.907 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5629000000000001e-05 2024-09-13T13:16:50.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.907 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.006692+0000", 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0448999999999998e-05 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190317 9.c 9:34b3085e:::benchmark_data_smithi052_128427_object59648:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.058087+0000", 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 246.792754992, 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95460188400000001, 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.908 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190317 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058087+0000", 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.909 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058087+0000", 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058088+0000", 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.192e-06 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058093+0000", 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7380000000000002e-06 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.910 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058102+0000", 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6590000000000002e-06 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058109+0000", 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9770000000000003e-06 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058175+0000", 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5343000000000001e-05 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.911 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058182+0000", 2024-09-13T13:16:50.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8059999999999995e-06 2024-09-13T13:16:50.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989661+0000", 2024-09-13T13:16:50.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.931478994 2024-09-13T13:16:50.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989768+0000", 2024-09-13T13:16:50.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010650900000000001 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989877+0000", 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109343 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.994834+0000", 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049566610000000002 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.994870+0000", 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6399999999999997e-05 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.012452+0000", 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.017581513999999999 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.012635+0000", 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018336900000000001 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.914 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.915 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.012689+0000", 2024-09-13T13:16:50.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3378999999999998e-05 2024-09-13T13:16:50.915 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.915 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.915 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.915 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190320 9.c 9:35ad6b23:::benchmark_data_smithi052_128427_object59651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:16:50.915 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.059653+0000", 2024-09-13T13:16:50.915 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 246.79118882099999, 2024-09-13T13:16:50.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95393949300000003, 2024-09-13T13:16:50.915 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.915 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.916 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.916 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.916 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.916 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.916 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190320 2024-09-13T13:16:50.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.916 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.916 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.916 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.916 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059653+0000", 2024-09-13T13:16:50.916 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059653+0000", 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059654+0000", 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1370000000000001e-06 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059658+0000", 2024-09-13T13:16:50.917 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.755e-06 2024-09-13T13:16:50.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.918 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.918 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059668+0000", 2024-09-13T13:16:50.918 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8709999999999992e-06 2024-09-13T13:16:50.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.918 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.918 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059675+0000", 2024-09-13T13:16:50.918 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5870000000000004e-06 2024-09-13T13:16:50.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059730+0000", 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5180999999999999e-05 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059739+0000", 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0569999999999999e-06 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992325+0000", 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.93258522899999996 2024-09-13T13:16:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992426+0000", 2024-09-13T13:16:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101094 2024-09-13T13:16:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:16:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992530+0000", 2024-09-13T13:16:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104051 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.999704+0000", 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0071743629999999996 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.004507+0000", 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048030340000000003 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.013507+0000", 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0089995439999999999 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.013555+0000", 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8572000000000002e-05 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.013592+0000", 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7017999999999997e-05 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202437 9.c 9:31d379dd:::benchmark_data_smithi052_128427_object71768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.936285+0000", 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 232.91455693399999, 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0668128299999999, 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202437 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936285+0000", 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936285+0000", 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936286+0000", 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5540000000000001e-06 2024-09-13T13:16:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936291+0000", 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0100000000000003e-06 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936305+0000", 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3562e-05 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936312+0000", 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3780000000000003e-06 2024-09-13T13:16:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936374+0000", 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1909999999999995e-05 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936528+0000", 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015417 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4,7", 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936642+0000", 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011353000000000001 2024-09-13T13:16:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.941575+0000", 2024-09-13T13:16:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049326969999999998 2024-09-13T13:16:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.947466+0000", 2024-09-13T13:16:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058910960000000002 2024-09-13T13:16:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.947475+0000", 2024-09-13T13:16:50.928 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6350000000000001e-06 2024-09-13T13:16:50.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:50.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:50.929 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.929 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.929 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.990524+0000", 2024-09-13T13:16:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0430484790000001 2024-09-13T13:16:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for missing object", 2024-09-13T13:16:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.990633+0000", 2024-09-13T13:16:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108872 2024-09-13T13:16:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003024+0000", 2024-09-13T13:16:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.012390988 2024-09-13T13:16:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003098+0000", 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3949000000000006e-05 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202453 9.c 9:3d8e06d1:::benchmark_data_smithi052_128427_object71784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.956183+0000", 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 232.89465871300001, 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.056296729, 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202453 2024-09-13T13:16:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956183+0000", 2024-09-13T13:16:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956183+0000", 2024-09-13T13:16:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956185+0000", 2024-09-13T13:16:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.784e-06 2024-09-13T13:16:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956190+0000", 2024-09-13T13:16:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:16:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956200+0000", 2024-09-13T13:16:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2900000000000008e-06 2024-09-13T13:16:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956206+0000", 2024-09-13T13:16:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1489999999999999e-06 2024-09-13T13:16:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957496+0000", 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012905519999999999 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957505+0000", 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9930000000000001e-06 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003310+0000", 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.045804368 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003406+0000", 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6335000000000005e-05 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003506+0000", 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000100295 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012143+0000", 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0086370389999999991 2024-09-13T13:16:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012233+0000", 2024-09-13T13:16:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9973000000000001e-05 2024-09-13T13:16:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012279+0000", 2024-09-13T13:16:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5966e-05 2024-09-13T13:16:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012376+0000", 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6220999999999999e-05 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012480+0000", 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010412900000000001 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202455 9.c 9:304f4878:::benchmark_data_smithi052_128427_object71786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:16:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 232.894623806, 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.071265975, 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202455 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:16:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5000000000000003e-07 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956221+0000", 2024-09-13T13:16:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.441e-06 2024-09-13T13:16:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956226+0000", 2024-09-13T13:16:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0089999999999998e-06 2024-09-13T13:16:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956230+0000", 2024-09-13T13:16:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5410000000000002e-06 2024-09-13T13:16:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957529+0000", 2024-09-13T13:16:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012988469999999999 2024-09-13T13:16:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957535+0000", 2024-09-13T13:16:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8109999999999999e-06 2024-09-13T13:16:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.005874+0000", 2024-09-13T13:16:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0483387770000001 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.005991+0000", 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116669 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.006107+0000", 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011618 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.943 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012522+0000", 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064157060000000002 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012562+0000", 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9756999999999999e-05 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.026940+0000", 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014377463 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.944 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027300+0000", 2024-09-13T13:16:50.945 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00036073000000000002 2024-09-13T13:16:50.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.945 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.945 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.945 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027484+0000", 2024-09-13T13:16:50.945 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000183494 2024-09-13T13:16:50.945 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.945 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.945 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.945 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.945 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202464 9.c 9:3602984d:::benchmark_data_smithi052_128427_object71795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:16:50.945 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:16:50.946 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 232.88955790599999, 2024-09-13T13:16:50.946 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.066828723, 2024-09-13T13:16:50.946 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.946 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.946 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.946 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.946 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.946 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.946 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202464 2024-09-13T13:16:50.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.946 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.946 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.946 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.947 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:16:50.947 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.947 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.947 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.947 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:16:50.947 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.947 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.947 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.947 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:16:50.947 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4099999999999999e-07 2024-09-13T13:16:50.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961286+0000", 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2010000000000002e-06 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961291+0000", 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5909999999999998e-06 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961295+0000", 2024-09-13T13:16:50.948 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1110000000000001e-06 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.962375+0000", 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00107939 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.962382+0000", 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5900000000000002e-06 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.949 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.008652+0000", 2024-09-13T13:16:50.950 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0462698880000001 2024-09-13T13:16:50.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.950 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.950 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.950 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.008749+0000", 2024-09-13T13:16:50.950 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7125000000000003e-05 2024-09-13T13:16:50.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.950 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.950 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:16:50.950 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.008858+0000", 2024-09-13T13:16:50.950 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010874699999999999 2024-09-13T13:16:50.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.950 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012711+0000", 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0038533619999999999 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012755+0000", 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3773000000000003e-05 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027802+0000", 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015047115 2024-09-13T13:16:50.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.952 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.952 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.952 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027905+0000", 2024-09-13T13:16:50.952 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103278 2024-09-13T13:16:50.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.952 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.952 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.952 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028113+0000", 2024-09-13T13:16:50.952 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000207111 2024-09-13T13:16:50.952 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.952 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.952 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.953 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.953 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.953 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202480 9.c 9:3e2ac5d3:::benchmark_data_smithi052_128427_object71811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:16:50.953 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.976129+0000", 2024-09-13T13:16:50.953 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 232.87471290600001, 2024-09-13T13:16:50.953 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.052223253, 2024-09-13T13:16:50.953 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.953 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.953 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.953 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.953 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.953 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.953 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202480 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976129+0000", 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976129+0000", 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.954 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976130+0000", 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3799999999999999e-06 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976135+0000", 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2809999999999999e-06 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976147+0000", 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1182000000000001e-05 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.955 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.956 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.956 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976154+0000", 2024-09-13T13:16:50.956 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4440000000000002e-06 2024-09-13T13:16:50.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.956 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.956 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.956 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976219+0000", 2024-09-13T13:16:50.956 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4414999999999999e-05 2024-09-13T13:16:50.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.956 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.956 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.956 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976227+0000", 2024-09-13T13:16:50.956 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012784+0000", 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.036556815 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012892+0000", 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010768999999999999 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:16:50.957 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.013010+0000", 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118244 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.018728+0000", 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057179079999999998 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.018768+0000", 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.985e-05 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.958 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.959 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028119+0000", 2024-09-13T13:16:50.959 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0093518460000000005 2024-09-13T13:16:50.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.959 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.959 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.959 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028244+0000", 2024-09-13T13:16:50.959 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012501799999999999 2024-09-13T13:16:50.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.959 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.959 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.959 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028352+0000", 2024-09-13T13:16:50.959 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107557 2024-09-13T13:16:50.960 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.960 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.960 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.960 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.960 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.960 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202488 9.c 9:3231b30e:::benchmark_data_smithi052_128427_object71819:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:16:50.960 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.980401+0000", 2024-09-13T13:16:50.960 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 232.870440433, 2024-09-13T13:16:50.960 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0481403709999999, 2024-09-13T13:16:50.960 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.960 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.960 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.960 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.961 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.961 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.961 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202488 2024-09-13T13:16:50.961 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.961 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.961 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.961 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.961 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980401+0000", 2024-09-13T13:16:50.961 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.961 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.961 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.962 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.962 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980401+0000", 2024-09-13T13:16:50.962 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.962 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.962 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.962 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.962 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980403+0000", 2024-09-13T13:16:50.962 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.339e-06 2024-09-13T13:16:50.962 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.962 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.962 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.962 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980408+0000", 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1909999999999999e-06 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980421+0000", 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3267e-05 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980428+0000", 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0310000000000001e-06 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.963 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.964 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980485+0000", 2024-09-13T13:16:50.964 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6527999999999998e-05 2024-09-13T13:16:50.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.964 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.964 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.964 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980493+0000", 2024-09-13T13:16:50.964 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2460000000000003e-06 2024-09-13T13:16:50.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.964 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.964 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.964 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.015559+0000", 2024-09-13T13:16:50.965 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0350661029999999 2024-09-13T13:16:50.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.965 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.965 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.965 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.015857+0000", 2024-09-13T13:16:50.965 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000297726 2024-09-13T13:16:50.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.965 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.965 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:16:50.965 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.016037+0000", 2024-09-13T13:16:50.965 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018051499999999999 2024-09-13T13:16:50.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.965 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.021797+0000", 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057596000000000001 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.021836+0000", 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9180000000000001e-05 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028359+0000", 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0065231589999999997 2024-09-13T13:16:50.966 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028449+0000", 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9874e-05 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028542+0000", 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2612e-05 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.967 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.968 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202524 9.c 9:322e12cc:::benchmark_data_smithi052_128427_object71855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:16:50.968 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:58.008117+0000", 2024-09-13T13:16:50.968 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 232.84272515999999, 2024-09-13T13:16:50.968 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0206121399999999, 2024-09-13T13:16:50.968 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.968 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.968 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:16:50.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:16:50.968 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:16:50.968 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202524 2024-09-13T13:16:50.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.968 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008117+0000", 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008117+0000", 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008118+0000", 2024-09-13T13:16:50.969 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3200000000000001e-06 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008123+0000", 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8189999999999999e-06 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008132+0000", 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6979999999999993e-06 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.970 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008141+0000", 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7620000000000001e-06 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008199+0000", 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7942999999999997e-05 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008208+0000", 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1069999999999994e-06 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.971 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.018947+0000", 2024-09-13T13:16:50.972 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0107389179999999 2024-09-13T13:16:50.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.972 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.972 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.019046+0000", 2024-09-13T13:16:50.972 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8896999999999999e-05 2024-09-13T13:16:50.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.972 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:16:50.972 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.019147+0000", 2024-09-13T13:16:50.972 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101001 2024-09-13T13:16:50.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.973 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.973 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.021878+0000", 2024-09-13T13:16:50.973 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002730985 2024-09-13T13:16:50.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.973 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:16:50.973 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.026889+0000", 2024-09-13T13:16:50.973 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050107529999999997 2024-09-13T13:16:50.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.973 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:16:50.973 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028548+0000", 2024-09-13T13:16:50.973 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016596950000000001 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028637+0000", 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8687e-05 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028729+0000", 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1555e-05 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:178616 10.f e380/377 10:f45131d6:::benchmark_data_smithi052_153703_object51236:head v 380'11130, mlcod=380'11128)", 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:16:22.234304+0000", 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 28.616538077000001, 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.33793278300000001, 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.234304+0000", 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.234304+0000", 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.234305+0000", 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.792e-06 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.234313+0000", 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7409999999999992e-06 2024-09-13T13:16:50.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.977 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.977 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.234324+0000", 2024-09-13T13:16:50.977 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.096e-05 2024-09-13T13:16:50.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.977 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.977 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.234332+0000", 2024-09-13T13:16:50.977 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5549999999999997e-06 2024-09-13T13:16:50.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.977 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.977 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.234376+0000", 2024-09-13T13:16:50.978 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3832999999999999e-05 2024-09-13T13:16:50.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.978 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.978 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.234391+0000", 2024-09-13T13:16:50.978 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5889e-05 2024-09-13T13:16:50.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.978 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.978 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.572210+0000", 2024-09-13T13:16:50.978 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.33781823900000002 2024-09-13T13:16:50.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.979 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.979 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.572236+0000", 2024-09-13T13:16:50.979 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6774000000000001e-05 2024-09-13T13:16:50.979 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.979 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.979 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.979 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.979 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:178620 10.f e380/377 10:f7cb9e30:::benchmark_data_smithi052_153703_object51240:head v 380'11131, mlcod=380'11128)", 2024-09-13T13:16:50.979 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:16:22.237173+0000", 2024-09-13T13:16:50.979 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 28.613669199, 2024-09-13T13:16:50.979 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34096315700000002, 2024-09-13T13:16:50.979 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237173+0000", 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237173+0000", 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.980 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.981 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237174+0000", 2024-09-13T13:16:50.981 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.373e-06 2024-09-13T13:16:50.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.981 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.981 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237182+0000", 2024-09-13T13:16:50.981 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9310000000000008e-06 2024-09-13T13:16:50.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.981 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.981 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237233+0000", 2024-09-13T13:16:50.981 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0778000000000001e-05 2024-09-13T13:16:50.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237240+0000", 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8120000000000008e-06 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237756+0000", 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000516009 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237780+0000", 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4009e-05 2024-09-13T13:16:50.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.578044+0000", 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34026351900000001 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.578136+0000", 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1725999999999996e-05 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.984 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:178624 10.f e380/377 10:f7e657e0:::benchmark_data_smithi052_153703_object51244:head v 380'11132, mlcod=380'11128)", 2024-09-13T13:16:50.984 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:16:22.243051+0000", 2024-09-13T13:16:50.984 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 28.607790494, 2024-09-13T13:16:50.984 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.339884729, 2024-09-13T13:16:50.984 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.984 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.984 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.984 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.984 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.984 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.243051+0000", 2024-09-13T13:16:50.984 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.985 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.985 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.985 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.243051+0000", 2024-09-13T13:16:50.985 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.985 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.985 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.985 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.243053+0000", 2024-09-13T13:16:50.985 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9230000000000001e-06 2024-09-13T13:16:50.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.985 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.985 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.988 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.243063+0000", 2024-09-13T13:16:50.988 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3719999999999994e-06 2024-09-13T13:16:50.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.988 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.988 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.989 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.568205+0000", 2024-09-13T13:16:50.989 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.32514227200000001 2024-09-13T13:16:50.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.989 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.989 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.989 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.568225+0000", 2024-09-13T13:16:50.989 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9792999999999998e-05 2024-09-13T13:16:50.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.989 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.989 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.989 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.570732+0000", 2024-09-13T13:16:50.989 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025074020000000002 2024-09-13T13:16:50.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.570747+0000", 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4547e-05 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.582885+0000", 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.012138322 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.582936+0000", 2024-09-13T13:16:50.990 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1097999999999998e-05 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:178627 10.3 e380/377 10:cad7953b:::benchmark_data_smithi052_153703_object51247:head v 380'11119, mlcod=380'11116)", 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:16:22.243080+0000", 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 28.60776173, 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.33339098499999997, 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:16:50.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.992 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:16:50.992 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.243080+0000", 2024-09-13T13:16:50.992 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.992 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.992 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:16:50.992 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.243080+0000", 2024-09-13T13:16:50.992 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:16:50.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.992 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.992 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:16:50.992 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.243081+0000", 2024-09-13T13:16:50.992 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.277e-06 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.243088+0000", 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0330000000000003e-06 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.568292+0000", 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.325203566 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:16:50.993 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.568320+0000", 2024-09-13T13:16:50.994 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8238999999999998e-05 2024-09-13T13:16:50.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.994 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:16:50.994 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.568526+0000", 2024-09-13T13:16:50.994 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020609 2024-09-13T13:16:50.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.994 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:16:50.995 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.568617+0000", 2024-09-13T13:16:50.995 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0767999999999996e-05 2024-09-13T13:16:50.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.995 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.995 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:16:50.995 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.576392+0000", 2024-09-13T13:16:50.995 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0077748579999999999 2024-09-13T13:16:50.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:16:50.995 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:16:50.996 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:16:50.996 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.576471+0000", 2024-09-13T13:16:50.996 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9153999999999995e-05 2024-09-13T13:16:50.996 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.996 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.996 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.996 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:16:50.996 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:16:50.996 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:16:51.029 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:51.061 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:51.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:51.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:51.202 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:51.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:51.304 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:51.338 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:51.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:51.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:51.440 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:51.474 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:51.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:51.478 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:51 smithi052 ceph-mon[36682]: pgmap v779: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 8531/148977 objects misplaced (5.726%); 72 MiB/s, 1.16k objects/s recovering 2024-09-13T13:16:51.478 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:51 smithi052 ceph-mon[36682]: from='client.36434 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:16:51.478 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:51 smithi052 ceph-mon[36682]: osdmap e390: 8 total, 8 up, 7 in 2024-09-13T13:16:51.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:51 smithi052 ceph-mon[29471]: pgmap v779: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 8531/148977 objects misplaced (5.726%); 72 MiB/s, 1.16k objects/s recovering 2024-09-13T13:16:51.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:51 smithi052 ceph-mon[29471]: from='client.36434 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:16:51.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:51 smithi052 ceph-mon[29471]: osdmap e390: 8 total, 8 up, 7 in 2024-09-13T13:16:51.527 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T13:16:51.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:51 smithi167 ceph-mon[33178]: pgmap v779: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 8531/148977 objects misplaced (5.726%); 72 MiB/s, 1.16k objects/s recovering 2024-09-13T13:16:51.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:51 smithi167 ceph-mon[33178]: from='client.36434 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:16:51.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:51 smithi167 ceph-mon[33178]: osdmap e390: 8 total, 8 up, 7 in 2024-09-13T13:16:51.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:51.612 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:51.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:51.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:51.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:51.753 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:51.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:51.854 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:51.887 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:51.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:51.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:51.989 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:52.027 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:52.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:52.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:52.129 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:52.161 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:52.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:52.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:52.263 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:52.284 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:16:52.310 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:52.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:52.412 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:52.453 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:52.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:52.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:52.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:52 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/467817882' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:16:52.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:52 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/467817882' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:16:52.554 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:52.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:52 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/467817882' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:16:52.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:52.579 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:52.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:52.680 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:52.715 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:52.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:52.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:52.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:52.859 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:52.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:52.961 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:52.998 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:52.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:52.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:53.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:53.135 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:53.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:53.237 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:53.274 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:53.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:53.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:53.376 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:53.412 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:53.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:53.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:53.551 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:53.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:53 smithi167 ceph-mon[33178]: pgmap v781: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 747 op/s; 7387/145383 objects misplaced (5.081%); 69 MiB/s, 1.10k objects/s recovering 2024-09-13T13:16:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:53 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:53 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/467817882' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:16:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:53 smithi167 ceph-mon[33178]: osdmap e391: 8 total, 8 up, 7 in 2024-09-13T13:16:53.652 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:53.698 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:53.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:53.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:53 smithi052 ceph-mon[36682]: pgmap v781: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 747 op/s; 7387/145383 objects misplaced (5.081%); 69 MiB/s, 1.10k objects/s recovering 2024-09-13T13:16:53.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:53 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:53.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:53 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/467817882' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:16:53.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:53 smithi052 ceph-mon[36682]: osdmap e391: 8 total, 8 up, 7 in 2024-09-13T13:16:53.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:53 smithi052 ceph-mon[29471]: pgmap v781: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 747 op/s; 7387/145383 objects misplaced (5.081%); 69 MiB/s, 1.10k objects/s recovering 2024-09-13T13:16:53.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:53 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:16:53.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/467817882' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:16:53.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:53 smithi052 ceph-mon[29471]: osdmap e391: 8 total, 8 up, 7 in 2024-09-13T13:16:53.799 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:53.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:53.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:53.938 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:53.971 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:53.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:54.072 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:54.106 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:54.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:54.209 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:54.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:54.239 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:54.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:54.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:54.378 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:54.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:54.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:54.479 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:54.514 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:16:54.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:54.517 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:54.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:54.537 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:54.592 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:54.618 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:54.647 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:54.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:54.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:54.749 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:54.786 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:54.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:54.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:16:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:16:54.887 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:54.939 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:54.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:55.041 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:55.049 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:16:55.077 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:55.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:55.179 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:55.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:55.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:55.322 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:55.358 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:55.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:55.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:55.461 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:55.501 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:55.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:55.540 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:55 smithi167 ceph-mon[33178]: 3.2 scrub starts 2024-09-13T13:16:55.540 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:55 smithi167 ceph-mon[33178]: 3.2 scrub ok 2024-09-13T13:16:55.540 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:55 smithi167 ceph-mon[33178]: pgmap v783: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 896 B/s rd, 0 B/s wr, 909 op/s; 6286/140499 objects misplaced (4.474%); 67 MiB/s, 1.08k objects/s recovering 2024-09-13T13:16:55.540 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:55 smithi167 ceph-mon[33178]: 5.1d scrub starts 2024-09-13T13:16:55.540 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:55 smithi167 ceph-mon[33178]: 5.1d scrub ok 2024-09-13T13:16:55.540 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:55.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:55 smithi052 ceph-mon[36682]: 3.2 scrub starts 2024-09-13T13:16:55.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:55 smithi052 ceph-mon[36682]: 3.2 scrub ok 2024-09-13T13:16:55.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:55 smithi052 ceph-mon[36682]: pgmap v783: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 896 B/s rd, 0 B/s wr, 909 op/s; 6286/140499 objects misplaced (4.474%); 67 MiB/s, 1.08k objects/s recovering 2024-09-13T13:16:55.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:55 smithi052 ceph-mon[36682]: 5.1d scrub starts 2024-09-13T13:16:55.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:55 smithi052 ceph-mon[36682]: 5.1d scrub ok 2024-09-13T13:16:55.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:55.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:55.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:55 smithi052 ceph-mon[29471]: 3.2 scrub starts 2024-09-13T13:16:55.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:55 smithi052 ceph-mon[29471]: 3.2 scrub ok 2024-09-13T13:16:55.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:55 smithi052 ceph-mon[29471]: pgmap v783: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 896 B/s rd, 0 B/s wr, 909 op/s; 6286/140499 objects misplaced (4.474%); 67 MiB/s, 1.08k objects/s recovering 2024-09-13T13:16:55.604 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:55 smithi052 ceph-mon[29471]: 5.1d scrub starts 2024-09-13T13:16:55.605 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:55 smithi052 ceph-mon[29471]: 5.1d scrub ok 2024-09-13T13:16:55.605 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:16:55.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:55.645 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:55.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:55.720 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:16:55.721 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:16:55.745 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:55.749 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:16:55.749 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:16:55.773 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:16:55.773 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:16:55.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:55.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:55.799 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:16:55.799 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:16:55.830 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:16:55.831 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:16:55.859 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:16:55.859 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:16:55.880 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:55.893 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:16:55.893 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:16:55.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:55.917 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:55.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:55.921 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:16:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:16:56.018 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:56.052 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:56.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:56.109 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:16:56.154 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:56.192 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:56.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:56.280 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T13:16:56.293 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:56.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:56.323 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:56.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:56.424 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:56.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:56.465 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:56.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:56.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:56 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1776529792' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:16:56.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:56 smithi052 ceph-mon[36682]: from='client.26353 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:16:56.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:56 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/182853710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:56.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:56 smithi052 ceph-mon[36682]: 4.f scrub starts 2024-09-13T13:16:56.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:56 smithi052 ceph-mon[36682]: 4.f scrub ok 2024-09-13T13:16:56.490 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:16:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:16:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:16:56.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:56 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1776529792' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:16:56.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:56 smithi052 ceph-mon[29471]: from='client.26353 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:16:56.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:56 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/182853710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:56.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:56 smithi052 ceph-mon[29471]: 4.f scrub starts 2024-09-13T13:16:56.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:56 smithi052 ceph-mon[29471]: 4.f scrub ok 2024-09-13T13:16:56.531 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:16:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:16:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:212201 10.d e391/334 10:bb7190c1:::benchmark_data_smithi052_153703_object84821:head v 391'13435, mlcod=391'13428)", 2024-09-13T13:16:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:55.622502+0000", 2024-09-13T13:16:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.059845167999999997, 2024-09-13T13:16:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059867264000000003, 2024-09-13T13:16:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:16:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:16:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:16:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:16:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:16:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.622502+0000", 2024-09-13T13:16:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:16:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.622502+0000", 2024-09-13T13:16:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:16:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.622503+0000", 2024-09-13T13:16:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1540000000000001e-06 2024-09-13T13:16:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:16:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.622509+0000", 2024-09-13T13:16:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4300000000000003e-06 2024-09-13T13:16:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:16:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.622518+0000", 2024-09-13T13:16:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2070000000000001e-06 2024-09-13T13:16:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:16:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.622525+0000", 2024-09-13T13:16:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1330000000000004e-06 2024-09-13T13:16:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:16:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.628295+0000", 2024-09-13T13:16:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057702600000000001 2024-09-13T13:16:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:16:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.628314+0000", 2024-09-13T13:16:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8745000000000001e-05 2024-09-13T13:16:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:16:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:212182 10.d e391/334 10:bb69604d:::benchmark_data_smithi052_153703_object84802:head v 391'13430, mlcod=391'13426)", 2024-09-13T13:16:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:55.597874+0000", 2024-09-13T13:16:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.084473196, 2024-09-13T13:16:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084526119999999996, 2024-09-13T13:16:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:16:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:16:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:16:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:16:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:16:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.597874+0000", 2024-09-13T13:16:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:16:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.597874+0000", 2024-09-13T13:16:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:16:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.597875+0000", 2024-09-13T13:16:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2750000000000001e-06 2024-09-13T13:16:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:16:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.597883+0000", 2024-09-13T13:16:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-09-13T13:16:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:16:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.597891+0000", 2024-09-13T13:16:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5189999999999992e-06 2024-09-13T13:16:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:16:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.597898+0000", 2024-09-13T13:16:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5769999999999999e-06 2024-09-13T13:16:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:16:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.598850+0000", 2024-09-13T13:16:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00095248800000000003 2024-09-13T13:16:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:16:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.598877+0000", 2024-09-13T13:16:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.69e-05 2024-09-13T13:16:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:16:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:212202 10.d e391/334 10:b18e5333:::benchmark_data_smithi052_153703_object84822:head v 391'13436, mlcod=391'13428)", 2024-09-13T13:16:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:55.629053+0000", 2024-09-13T13:16:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.053293949, 2024-09-13T13:16:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053372369000000003, 2024-09-13T13:16:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:16:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:16:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:16:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:16:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:16:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.629053+0000", 2024-09-13T13:16:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:16:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.629053+0000", 2024-09-13T13:16:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:16:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.629054+0000", 2024-09-13T13:16:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2799999999999995e-07 2024-09-13T13:16:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:16:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.629057+0000", 2024-09-13T13:16:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7610000000000001e-06 2024-09-13T13:16:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:16:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.629062+0000", 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8930000000000004e-06 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.629066+0000", 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7450000000000001e-06 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.634271+0000", 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0052044550000000002 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:16:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.634346+0000", 2024-09-13T13:16:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5945000000000006e-05 2024-09-13T13:16:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:16:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:212183 10.d e391/334 10:b30519a3:::benchmark_data_smithi052_153703_object84803:head v 391'13431, mlcod=391'13426)", 2024-09-13T13:16:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:55.600728+0000", 2024-09-13T13:16:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.081618905000000005, 2024-09-13T13:16:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.081721818000000002, 2024-09-13T13:16:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:16:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:16:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:16:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:16:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:16:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.600728+0000", 2024-09-13T13:16:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:16:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.600728+0000", 2024-09-13T13:16:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:16:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.600729+0000", 2024-09-13T13:16:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1659999999999999e-06 2024-09-13T13:16:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:16:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.600736+0000", 2024-09-13T13:16:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1380000000000001e-06 2024-09-13T13:16:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:16:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.600747+0000", 2024-09-13T13:16:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0339e-05 2024-09-13T13:16:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:16:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.600753+0000", 2024-09-13T13:16:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6969999999999996e-06 2024-09-13T13:16:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:16:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.602210+0000", 2024-09-13T13:16:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001457187 2024-09-13T13:16:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:16:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.602232+0000", 2024-09-13T13:16:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1565000000000001e-05 2024-09-13T13:16:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:16:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:212209 10.1 10:8d64caaa:::benchmark_data_smithi052_153703_object84829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-13T13:16:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:55.629228+0000", 2024-09-13T13:16:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.053118429000000002, 2024-09-13T13:16:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053250384999999997, 2024-09-13T13:16:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:16:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:16:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:16:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:16:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:16:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 212209 2024-09-13T13:16:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:16:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:16:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.629228+0000", 2024-09-13T13:16:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:16:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.629228+0000", 2024-09-13T13:16:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:16:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.629231+0000", 2024-09-13T13:16:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7920000000000001e-06 2024-09-13T13:16:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:16:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.629241+0000", 2024-09-13T13:16:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.577e-06 2024-09-13T13:16:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:16:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.629255+0000", 2024-09-13T13:16:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4635e-05 2024-09-13T13:16:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:16:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.629264+0000", 2024-09-13T13:16:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4600000000000003e-06 2024-09-13T13:16:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:16:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.632581+0000", 2024-09-13T13:16:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033167520000000001 2024-09-13T13:16:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:16:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.632699+0000", 2024-09-13T13:16:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118288 2024-09-13T13:16:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:16:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.632806+0000", 2024-09-13T13:16:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010686 2024-09-13T13:16:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:16:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.638780+0000", 2024-09-13T13:16:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00597446 2024-09-13T13:16:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:16:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:212210 10.1 10:8aeea8b9:::benchmark_data_smithi052_153703_object84830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-13T13:16:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:55.631695+0000", 2024-09-13T13:16:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.050652243, 2024-09-13T13:16:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050820911000000003, 2024-09-13T13:16:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:16:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:16:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:16:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:16:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:16:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:16:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 212210 2024-09-13T13:16:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:16:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:16:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.631695+0000", 2024-09-13T13:16:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:16:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.631695+0000", 2024-09-13T13:16:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:16:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.631697+0000", 2024-09-13T13:16:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2330000000000001e-06 2024-09-13T13:16:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:16:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.631703+0000", 2024-09-13T13:16:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4409999999999998e-06 2024-09-13T13:16:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:16:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.631717+0000", 2024-09-13T13:16:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3303e-05 2024-09-13T13:16:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:16:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.631725+0000", 2024-09-13T13:16:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5990000000000002e-06 2024-09-13T13:16:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:16:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.636308+0000", 2024-09-13T13:16:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045823280000000001 2024-09-13T13:16:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:16:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.636417+0000", 2024-09-13T13:16:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109739 2024-09-13T13:16:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:16:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.636535+0000", 2024-09-13T13:16:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118143 2024-09-13T13:16:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:16:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.639106+0000", 2024-09-13T13:16:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025703520000000001 2024-09-13T13:16:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:16:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:212212 10.d e391/334 10:bd579fd2:::benchmark_data_smithi052_153703_object84832:head v 391'13437, mlcod=391'13428)", 2024-09-13T13:16:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:55.636061+0000", 2024-09-13T13:16:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.046285730999999997, 2024-09-13T13:16:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046487319999999999, 2024-09-13T13:16:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:16:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:16:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:16:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:16:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:16:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.636061+0000", 2024-09-13T13:16:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:16:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.636061+0000", 2024-09-13T13:16:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:16:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.636063+0000", 2024-09-13T13:16:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.536e-06 2024-09-13T13:16:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:16:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.636071+0000", 2024-09-13T13:16:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0960000000000001e-06 2024-09-13T13:16:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:16:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.636080+0000", 2024-09-13T13:16:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:16:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:16:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.636088+0000", 2024-09-13T13:16:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4950000000000002e-06 2024-09-13T13:16:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:16:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.637645+0000", 2024-09-13T13:16:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015573850000000001 2024-09-13T13:16:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:16:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.637662+0000", 2024-09-13T13:16:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.677e-05 2024-09-13T13:16:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:16:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:212184 10.d e391/334 10:b1484e7a:::benchmark_data_smithi052_153703_object84804:head v 391'13432, mlcod=391'13426)", 2024-09-13T13:16:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:55.609164+0000", 2024-09-13T13:16:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.073182566000000004, 2024-09-13T13:16:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073412234000000007, 2024-09-13T13:16:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:16:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:16:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:16:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:16:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:16:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.609164+0000", 2024-09-13T13:16:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:16:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.609164+0000", 2024-09-13T13:16:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:16:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.609165+0000", 2024-09-13T13:16:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.004e-06 2024-09-13T13:16:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:16:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.609171+0000", 2024-09-13T13:16:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5470000000000004e-06 2024-09-13T13:16:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:16:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.609176+0000", 2024-09-13T13:16:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3890000000000004e-06 2024-09-13T13:16:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:16:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.609181+0000", 2024-09-13T13:16:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3320000000000002e-06 2024-09-13T13:16:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:16:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.609229+0000", 2024-09-13T13:16:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7960000000000002e-05 2024-09-13T13:16:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:16:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.609247+0000", 2024-09-13T13:16:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8743999999999999e-05 2024-09-13T13:16:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:16:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:212215 10.d e391/334)", 2024-09-13T13:16:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:55.637841+0000", 2024-09-13T13:16:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.044506165, 2024-09-13T13:16:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044757958, 2024-09-13T13:16:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:16:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:16:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "queued for pg", 2024-09-13T13:16:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:16:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:16:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.637841+0000", 2024-09-13T13:16:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:16:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.637841+0000", 2024-09-13T13:16:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:16:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.637842+0000", 2024-09-13T13:16:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1289999999999999e-06 2024-09-13T13:16:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:16:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.637847+0000", 2024-09-13T13:16:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2820000000000004e-06 2024-09-13T13:16:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:16:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.637855+0000", 2024-09-13T13:16:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8290000000000008e-06 2024-09-13T13:16:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:16:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.637861+0000", 2024-09-13T13:16:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0360000000000003e-06 2024-09-13T13:16:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:16:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.26071.0:212209 10.1 e391/334)", 2024-09-13T13:16:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:55.640412+0000", 2024-09-13T13:16:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.041934988999999999, 2024-09-13T13:16:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042205741999999997, 2024-09-13T13:16:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:16:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:16:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:16:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:16:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:16:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.640412+0000", 2024-09-13T13:16:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:16:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.640412+0000", 2024-09-13T13:16:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:16:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.640469+0000", 2024-09-13T13:16:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7321999999999998e-05 2024-09-13T13:16:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:16:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.640474+0000", 2024-09-13T13:16:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6600000000000003e-06 2024-09-13T13:16:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:16:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.640482+0000", 2024-09-13T13:16:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0469999999999994e-06 2024-09-13T13:16:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:16:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.640509+0000", 2024-09-13T13:16:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6894999999999999e-05 2024-09-13T13:16:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:16:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.640584+0000", 2024-09-13T13:16:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5289000000000004e-05 2024-09-13T13:16:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:16:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.26071.0:212210 10.1 e391/334)", 2024-09-13T13:16:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:55.640560+0000", 2024-09-13T13:16:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.041786871000000003, 2024-09-13T13:16:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042077822000000001, 2024-09-13T13:16:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:16:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:16:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "queued for pg", 2024-09-13T13:16:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:16:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:16:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.640560+0000", 2024-09-13T13:16:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:16:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.640560+0000", 2024-09-13T13:16:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:16:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.640561+0000", 2024-09-13T13:16:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3200000000000004e-07 2024-09-13T13:16:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:16:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.640564+0000", 2024-09-13T13:16:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4369999999999999e-06 2024-09-13T13:16:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:16:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.640571+0000", 2024-09-13T13:16:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4110000000000001e-06 2024-09-13T13:16:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:16:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.640576+0000", 2024-09-13T13:16:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6049999999999997e-06 2024-09-13T13:16:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:16:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:212190 10.d e391/334 10:b587dd2e:::benchmark_data_smithi052_153703_object84810:head v 391'13433, mlcod=391'13428)", 2024-09-13T13:16:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:55.616972+0000", 2024-09-13T13:16:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.065374587999999997, 2024-09-13T13:16:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065685560000000004, 2024-09-13T13:16:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:16:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:16:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:16:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:16:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:16:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.616972+0000", 2024-09-13T13:16:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:16:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.616972+0000", 2024-09-13T13:16:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:16:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.616973+0000", 2024-09-13T13:16:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9099999999999999e-07 2024-09-13T13:16:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:16:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.616976+0000", 2024-09-13T13:16:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4809999999999998e-06 2024-09-13T13:16:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:16:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.616981+0000", 2024-09-13T13:16:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9110000000000001e-06 2024-09-13T13:16:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:16:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.616985+0000", 2024-09-13T13:16:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1690000000000002e-06 2024-09-13T13:16:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:16:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.621049+0000", 2024-09-13T13:16:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040639309999999998 2024-09-13T13:16:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:16:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.621067+0000", 2024-09-13T13:16:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7808999999999998e-05 2024-09-13T13:16:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:16:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:212200 10.d e391/334 10:bc3f7514:::benchmark_data_smithi052_153703_object84820:head v 391'13434, mlcod=391'13428)", 2024-09-13T13:16:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:55.621154+0000", 2024-09-13T13:16:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.061192505000000001, 2024-09-13T13:16:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061527732000000002, 2024-09-13T13:16:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:16:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:16:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:16:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:16:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:16:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.621154+0000", 2024-09-13T13:16:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:16:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.621154+0000", 2024-09-13T13:16:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:16:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.621155+0000", 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9800000000000003e-07 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.621161+0000", 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.241e-06 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.621168+0000", 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7649999999999996e-06 2024-09-13T13:16:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:16:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.621173+0000", 2024-09-13T13:16:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7190000000000001e-06 2024-09-13T13:16:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:16:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.625012+0000", 2024-09-13T13:16:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038392349999999999 2024-09-13T13:16:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:16:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:16:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:16:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:55.625031+0000", 2024-09-13T13:16:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8408999999999999e-05 2024-09-13T13:16:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:16:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:16:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:16:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 13 2024-09-13T13:16:56.604 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:16:56.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:56.606 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:56 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1776529792' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:16:56.606 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:56 smithi167 ceph-mon[33178]: from='client.26353 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:16:56.606 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:56 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/182853710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:16:56.606 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:56 smithi167 ceph-mon[33178]: 4.f scrub starts 2024-09-13T13:16:56.606 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:56 smithi167 ceph-mon[33178]: 4.f scrub ok 2024-09-13T13:16:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:56.636 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:56.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:56.737 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:56.774 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:56.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:56.876 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:56.899 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:56.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:57.000 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:57.032 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:57.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:57.204 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:57.236 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:57.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:57.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:57.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:57.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:57.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:57.439 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:57 smithi052 ceph-mon[36682]: pgmap v784: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 765 B/s rd, 0 B/s wr, 698 op/s; 3891/137160 objects misplaced (2.837%); 62 MiB/s, 992 objects/s recovering 2024-09-13T13:16:57.439 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:57 smithi052 ceph-mon[36682]: from='client.26353 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:16:57.439 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:57 smithi052 ceph-mon[36682]: osdmap e392: 8 total, 8 up, 7 in 2024-09-13T13:16:57.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:57 smithi052 ceph-mon[29471]: pgmap v784: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 765 B/s rd, 0 B/s wr, 698 op/s; 3891/137160 objects misplaced (2.837%); 62 MiB/s, 992 objects/s recovering 2024-09-13T13:16:57.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:57 smithi052 ceph-mon[29471]: from='client.26353 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:16:57.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:57 smithi052 ceph-mon[29471]: osdmap e392: 8 total, 8 up, 7 in 2024-09-13T13:16:57.468 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T13:16:57.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:57.522 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:57.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:57.530 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:57 smithi167 ceph-mon[33178]: pgmap v784: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 765 B/s rd, 0 B/s wr, 698 op/s; 3891/137160 objects misplaced (2.837%); 62 MiB/s, 992 objects/s recovering 2024-09-13T13:16:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:57 smithi167 ceph-mon[33178]: from='client.26353 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:16:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:57 smithi167 ceph-mon[33178]: osdmap e392: 8 total, 8 up, 7 in 2024-09-13T13:16:57.623 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:57.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:57.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:57.770 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:57.809 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:57.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:57.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:16:57.911 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:57.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:57.949 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:57.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:58.050 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:58.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:58.084 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:58.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:58.170 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:16:58.184 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:58.210 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:58.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:58.312 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:58.334 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:58.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:58.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:58.436 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:58.462 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:58.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:58.564 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:58.590 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:58.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:58.691 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:58.714 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:58.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:58.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:58.794 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:16:58.816 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:16:58.859 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:58.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:16:58.961 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:16:58.986 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:58.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:16:59.041 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:16:59.041 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T13:16:59.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:59 smithi167 ceph-mon[33178]: pgmap v786: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 665 B/s rd, 0 B/s wr, 770 op/s; 3283/131223 objects misplaced (2.502%); 37 MiB/s, 598 objects/s recovering 2024-09-13T13:16:59.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:59 smithi167 ceph-mon[33178]: osdmap e393: 8 total, 8 up, 7 in 2024-09-13T13:16:59.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:59 smithi167 ceph-mon[33178]: 6.b scrub starts 2024-09-13T13:16:59.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:59 smithi167 ceph-mon[33178]: 6.b scrub ok 2024-09-13T13:16:59.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:59 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3576337173' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:59.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:59 smithi167 ceph-mon[33178]: from='client.26365 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:59.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:59 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2040044509' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:16:59.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:16:59 smithi167 ceph-mon[33178]: from='client.36464 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:16:59.087 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:16:59.109 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:59.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:16:59.169 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T13:16:59.211 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:59.257 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:59.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:59.358 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:16:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:59.411 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:59.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:16:59.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[29471]: pgmap v786: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 665 B/s rd, 0 B/s wr, 770 op/s; 3283/131223 objects misplaced (2.502%); 37 MiB/s, 598 objects/s recovering 2024-09-13T13:16:59.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[29471]: osdmap e393: 8 total, 8 up, 7 in 2024-09-13T13:16:59.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[29471]: 6.b scrub starts 2024-09-13T13:16:59.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[29471]: 6.b scrub ok 2024-09-13T13:16:59.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3576337173' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:59.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[29471]: from='client.26365 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:59.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2040044509' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:16:59.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[29471]: from='client.36464 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:16:59.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[36682]: pgmap v786: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 665 B/s rd, 0 B/s wr, 770 op/s; 3283/131223 objects misplaced (2.502%); 37 MiB/s, 598 objects/s recovering 2024-09-13T13:16:59.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[36682]: osdmap e393: 8 total, 8 up, 7 in 2024-09-13T13:16:59.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[36682]: 6.b scrub starts 2024-09-13T13:16:59.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[36682]: 6.b scrub ok 2024-09-13T13:16:59.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3576337173' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:59.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[36682]: from='client.26365 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:16:59.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2040044509' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:16:59.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:16:59 smithi052 ceph-mon[36682]: from='client.36464 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:16:59.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:16:59.558 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:59.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:59.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:16:59.618 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:16:59.660 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:16:59.689 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:59.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:16:59.793 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:16:59.825 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:16:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:59.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:16:59.927 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:16:59.959 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:16:59.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:16:59.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:00.061 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:00.090 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:00.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:00.192 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:00.222 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:00.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:00 smithi167 ceph-mon[33178]: 4.4 scrub starts 2024-09-13T13:17:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:00 smithi167 ceph-mon[33178]: 4.4 scrub ok 2024-09-13T13:17:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:00 smithi167 ceph-mon[33178]: 3.8 scrub starts 2024-09-13T13:17:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:00 smithi167 ceph-mon[33178]: 3.8 scrub ok 2024-09-13T13:17:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:00 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:17:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:00 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:17:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:00 smithi167 ceph-mon[33178]: from='client.26365 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:17:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:00 smithi167 ceph-mon[33178]: from='client.36464 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:17:00.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:00 smithi167 ceph-mon[33178]: osdmap e394: 8 total, 8 up, 7 in 2024-09-13T13:17:00.324 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:00.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:00.360 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:00.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:00.461 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:00.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[36682]: 4.4 scrub starts 2024-09-13T13:17:00.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[36682]: 4.4 scrub ok 2024-09-13T13:17:00.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[36682]: 3.8 scrub starts 2024-09-13T13:17:00.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[36682]: 3.8 scrub ok 2024-09-13T13:17:00.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:17:00.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:17:00.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[36682]: from='client.26365 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:17:00.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[36682]: from='client.36464 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:17:00.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[36682]: osdmap e394: 8 total, 8 up, 7 in 2024-09-13T13:17:00.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[29471]: 4.4 scrub starts 2024-09-13T13:17:00.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[29471]: 4.4 scrub ok 2024-09-13T13:17:00.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[29471]: 3.8 scrub starts 2024-09-13T13:17:00.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[29471]: 3.8 scrub ok 2024-09-13T13:17:00.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:17:00.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:17:00.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[29471]: from='client.26365 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:17:00.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[29471]: from='client.36464 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:17:00.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:00 smithi052 ceph-mon[29471]: osdmap e394: 8 total, 8 up, 7 in 2024-09-13T13:17:00.492 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:00.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:00.594 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:00.633 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:00.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:00.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:00.737 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:00.764 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:17:00.764 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:17:00.764 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:00.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:00.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:00.866 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:00.900 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:00.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:01.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:01.031 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:01.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:01.133 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:01.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:01.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:01.289 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:01.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:01.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:01 smithi167 ceph-mon[33178]: pgmap v789: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 985 op/s; 1603/122787 objects misplaced (1.306%); 44 MiB/s, 697 objects/s recovering 2024-09-13T13:17:01.391 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:01.394 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:01 smithi052 ceph-mon[36682]: pgmap v789: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 985 op/s; 1603/122787 objects misplaced (1.306%); 44 MiB/s, 697 objects/s recovering 2024-09-13T13:17:01.394 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:01 smithi052 ceph-mon[29471]: pgmap v789: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 985 op/s; 1603/122787 objects misplaced (1.306%); 44 MiB/s, 697 objects/s recovering 2024-09-13T13:17:01.414 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:01.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:01.510 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T13:17:01.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:01.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:01.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:01.673 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:01.696 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:01.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:01.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:01.798 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:01.799 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:17:01.821 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:01.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:01.923 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:01.945 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:01.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:01.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:02.048 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:02.075 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:02.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:02.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:02.208 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:02.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:02.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:02.282 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:02 smithi167 ceph-mon[33178]: 5.17 scrub starts 2024-09-13T13:17:02.282 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:02 smithi167 ceph-mon[33178]: 5.17 scrub ok 2024-09-13T13:17:02.282 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:02 smithi167 ceph-mon[33178]: osdmap e395: 8 total, 8 up, 7 in 2024-09-13T13:17:02.282 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:02 smithi167 ceph-mon[33178]: 3.19 scrub starts 2024-09-13T13:17:02.282 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:02 smithi167 ceph-mon[33178]: 3.19 scrub ok 2024-09-13T13:17:02.309 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:02.341 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:02.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:02.401 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:17:02.401 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:17:02.442 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:02.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:02.464 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:02.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:02 smithi052 ceph-mon[29471]: 5.17 scrub starts 2024-09-13T13:17:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:02 smithi052 ceph-mon[29471]: 5.17 scrub ok 2024-09-13T13:17:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:02 smithi052 ceph-mon[29471]: osdmap e395: 8 total, 8 up, 7 in 2024-09-13T13:17:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:02 smithi052 ceph-mon[29471]: 3.19 scrub starts 2024-09-13T13:17:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:02 smithi052 ceph-mon[29471]: 3.19 scrub ok 2024-09-13T13:17:02.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:02 smithi052 ceph-mon[36682]: 5.17 scrub starts 2024-09-13T13:17:02.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:02 smithi052 ceph-mon[36682]: 5.17 scrub ok 2024-09-13T13:17:02.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:02 smithi052 ceph-mon[36682]: osdmap e395: 8 total, 8 up, 7 in 2024-09-13T13:17:02.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:02 smithi052 ceph-mon[36682]: 3.19 scrub starts 2024-09-13T13:17:02.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:02 smithi052 ceph-mon[36682]: 3.19 scrub ok 2024-09-13T13:17:02.565 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:02.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:02.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:02.689 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:02.710 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:02.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:02.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:02.834 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:02.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:02.936 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:02.964 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:02.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:02.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:03.066 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:03.088 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:03.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:03.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:03.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T13:17:03.190 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:03.219 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:03.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:03.278 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:03.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:03.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:03 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:17:03.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:03 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:17:03.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:03 smithi167 ceph-mon[33178]: pgmap v791: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 901 B/s rd, 0 B/s wr, 1.11k op/s; 683/118275 objects misplaced (0.577%); 31 MiB/s, 497 objects/s recovering 2024-09-13T13:17:03.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:03 smithi167 ceph-mon[33178]: osdmap e396: 8 total, 8 up, 7 in 2024-09-13T13:17:03.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:03 smithi167 ceph-mon[33178]: 3.1c scrub starts 2024-09-13T13:17:03.322 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:03 smithi167 ceph-mon[33178]: 3.1c scrub ok 2024-09-13T13:17:03.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:03.364 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:03.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:03.407 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:03 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:17:03.407 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:03 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:17:03.407 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:03 smithi052 ceph-mon[36682]: pgmap v791: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 901 B/s rd, 0 B/s wr, 1.11k op/s; 683/118275 objects misplaced (0.577%); 31 MiB/s, 497 objects/s recovering 2024-09-13T13:17:03.407 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:03 smithi052 ceph-mon[36682]: osdmap e396: 8 total, 8 up, 7 in 2024-09-13T13:17:03.407 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:03 smithi052 ceph-mon[36682]: 3.1c scrub starts 2024-09-13T13:17:03.407 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:03 smithi052 ceph-mon[36682]: 3.1c scrub ok 2024-09-13T13:17:03.407 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:03 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:17:03.407 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:03 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:17:03.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:03 smithi052 ceph-mon[29471]: pgmap v791: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 901 B/s rd, 0 B/s wr, 1.11k op/s; 683/118275 objects misplaced (0.577%); 31 MiB/s, 497 objects/s recovering 2024-09-13T13:17:03.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:03 smithi052 ceph-mon[29471]: osdmap e396: 8 total, 8 up, 7 in 2024-09-13T13:17:03.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:03 smithi052 ceph-mon[29471]: 3.1c scrub starts 2024-09-13T13:17:03.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:03 smithi052 ceph-mon[29471]: 3.1c scrub ok 2024-09-13T13:17:03.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:03.515 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:03.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:03.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:03.589 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:17:03.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:03.666 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:03.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:03.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:03.768 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:03.799 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:03.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:03.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:03.901 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:03.929 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:03.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:04.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:04.057 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:04 smithi167 ceph-mon[33178]: 4.c scrub starts 2024-09-13T13:17:04.057 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:04 smithi167 ceph-mon[33178]: 4.c scrub ok 2024-09-13T13:17:04.058 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:04 smithi167 ceph-mon[33178]: 3.6 scrub starts 2024-09-13T13:17:04.058 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:04 smithi167 ceph-mon[33178]: 3.6 scrub ok 2024-09-13T13:17:04.068 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:04.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:04.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:04.170 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:04.198 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:04.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:04.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:04.259 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:17:04.259 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:17:04.300 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:04.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:04 smithi167 ceph-mon[33178]: 4.14 scrub starts 2024-09-13T13:17:04.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:04 smithi167 ceph-mon[33178]: 4.14 scrub ok 2024-09-13T13:17:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:04 smithi167 ceph-mon[33178]: osdmap e397: 8 total, 8 up, 7 in 2024-09-13T13:17:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:04 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:04.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:04 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:17:04.335 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:04.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:04.376 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[36682]: 4.c scrub starts 2024-09-13T13:17:04.376 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[36682]: 4.c scrub ok 2024-09-13T13:17:04.376 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[36682]: 3.6 scrub starts 2024-09-13T13:17:04.376 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[36682]: 3.6 scrub ok 2024-09-13T13:17:04.376 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[36682]: 4.14 scrub starts 2024-09-13T13:17:04.377 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[36682]: 4.14 scrub ok 2024-09-13T13:17:04.377 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[36682]: osdmap e397: 8 total, 8 up, 7 in 2024-09-13T13:17:04.377 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:04.377 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:17:04.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[29471]: 4.c scrub starts 2024-09-13T13:17:04.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[29471]: 4.c scrub ok 2024-09-13T13:17:04.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[29471]: 3.6 scrub starts 2024-09-13T13:17:04.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[29471]: 3.6 scrub ok 2024-09-13T13:17:04.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[29471]: 4.14 scrub starts 2024-09-13T13:17:04.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[29471]: 4.14 scrub ok 2024-09-13T13:17:04.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[29471]: osdmap e397: 8 total, 8 up, 7 in 2024-09-13T13:17:04.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:04.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:04 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:17:04.437 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:04.470 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:04.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:04.572 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:04.602 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:04.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:04.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:04.704 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:04.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:04.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:04.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:17:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:17:04.838 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:04.867 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:04.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:04.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:04.970 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:04.999 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:05.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:05.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:05.101 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:05.158 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:05.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:05.260 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:05.287 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:05.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:05.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:05.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:05 smithi167 ceph-mon[33178]: pgmap v794: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 737 op/s; 141/114966 objects misplaced (0.123%); 29 MiB/s, 461 objects/s recovering 2024-09-13T13:17:05.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:05 smithi167 ceph-mon[33178]: 4.15 scrub starts 2024-09-13T13:17:05.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:05 smithi167 ceph-mon[33178]: 4.15 scrub ok 2024-09-13T13:17:05.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:05.379 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:05 smithi052 ceph-mon[36682]: pgmap v794: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 737 op/s; 141/114966 objects misplaced (0.123%); 29 MiB/s, 461 objects/s recovering 2024-09-13T13:17:05.380 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:05 smithi052 ceph-mon[36682]: 4.15 scrub starts 2024-09-13T13:17:05.380 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:05 smithi052 ceph-mon[36682]: 4.15 scrub ok 2024-09-13T13:17:05.380 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:05.380 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:05 smithi052 ceph-mon[29471]: pgmap v794: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 737 op/s; 141/114966 objects misplaced (0.123%); 29 MiB/s, 461 objects/s recovering 2024-09-13T13:17:05.380 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:05 smithi052 ceph-mon[29471]: 4.15 scrub starts 2024-09-13T13:17:05.380 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:05 smithi052 ceph-mon[29471]: 4.15 scrub ok 2024-09-13T13:17:05.381 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:05.389 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:05.425 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:05.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:05.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:05.563 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:05.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:05.597 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:17:05.597 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:17:05.597 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:17:05.598 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:17:05.598 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:17:05.598 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:17:05.598 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:17:05.598 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:17:05.598 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:17:05.598 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:17:05.598 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:17:05.599 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:17:05.599 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:17:05.599 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:17:05.599 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:17:05.599 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:17:05.665 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:05.686 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:05.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:05.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:05.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:05.826 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:05.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:05.927 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:05.957 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:05.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:06.059 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:06.096 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:06.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:06.198 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:06.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:06.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:06.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:06 smithi167 ceph-mon[33178]: 6.6 scrub starts 2024-09-13T13:17:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:06 smithi167 ceph-mon[33178]: 6.6 scrub ok 2024-09-13T13:17:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:06 smithi167 ceph-mon[33178]: 6.1c scrub starts 2024-09-13T13:17:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:06 smithi167 ceph-mon[33178]: 6.1c scrub ok 2024-09-13T13:17:06.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:06 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2623459653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:06.321 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:06.352 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:06.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:06.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:06.437 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:17:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:17:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:17:06.438 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:06 smithi052 ceph-mon[36682]: 6.6 scrub starts 2024-09-13T13:17:06.438 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:06 smithi052 ceph-mon[36682]: 6.6 scrub ok 2024-09-13T13:17:06.438 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:06 smithi052 ceph-mon[36682]: 6.1c scrub starts 2024-09-13T13:17:06.439 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:06 smithi052 ceph-mon[36682]: 6.1c scrub ok 2024-09-13T13:17:06.439 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:06 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2623459653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:06.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:06 smithi052 ceph-mon[29471]: 6.6 scrub starts 2024-09-13T13:17:06.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:06 smithi052 ceph-mon[29471]: 6.6 scrub ok 2024-09-13T13:17:06.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:06 smithi052 ceph-mon[29471]: 6.1c scrub starts 2024-09-13T13:17:06.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:06 smithi052 ceph-mon[29471]: 6.1c scrub ok 2024-09-13T13:17:06.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:06 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2623459653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:06.455 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:06.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:06.491 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:06.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:06.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:06.633 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:06.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:06.735 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:06.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:06.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:06.873 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:06.906 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:06.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:06.998 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T13:17:07.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:07.028 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:07.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:07.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:07.130 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:07.151 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:07.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:07 smithi052 ceph-mon[36682]: 5.c scrub starts 2024-09-13T13:17:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:07 smithi052 ceph-mon[36682]: 5.c scrub ok 2024-09-13T13:17:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:07 smithi052 ceph-mon[36682]: 4.3 scrub starts 2024-09-13T13:17:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:07 smithi052 ceph-mon[36682]: 4.3 scrub ok 2024-09-13T13:17:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:07 smithi052 ceph-mon[36682]: pgmap v795: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 591 op/s; 141/112146 objects misplaced (0.126%); 15 MiB/s, 233 objects/s recovering 2024-09-13T13:17:07.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:07 smithi052 ceph-mon[29471]: 5.c scrub starts 2024-09-13T13:17:07.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:07 smithi052 ceph-mon[29471]: 5.c scrub ok 2024-09-13T13:17:07.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:07 smithi052 ceph-mon[29471]: 4.3 scrub starts 2024-09-13T13:17:07.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:07 smithi052 ceph-mon[29471]: 4.3 scrub ok 2024-09-13T13:17:07.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:07 smithi052 ceph-mon[29471]: pgmap v795: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 591 op/s; 141/112146 objects misplaced (0.126%); 15 MiB/s, 233 objects/s recovering 2024-09-13T13:17:07.252 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:07.274 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:07.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:07.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:07 smithi167 ceph-mon[33178]: 5.c scrub starts 2024-09-13T13:17:07.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:07 smithi167 ceph-mon[33178]: 5.c scrub ok 2024-09-13T13:17:07.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:07 smithi167 ceph-mon[33178]: 4.3 scrub starts 2024-09-13T13:17:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:07 smithi167 ceph-mon[33178]: 4.3 scrub ok 2024-09-13T13:17:07.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:07 smithi167 ceph-mon[33178]: pgmap v795: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 591 op/s; 141/112146 objects misplaced (0.126%); 15 MiB/s, 233 objects/s recovering 2024-09-13T13:17:07.376 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:07.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T13:17:07.401 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:07.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:07.503 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:07.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:07.536 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:07.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:07.625 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:07.637 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:07.678 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:07.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:07.780 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:07.808 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:07.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:07.825 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:17:07.910 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:07.969 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:07.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:08.071 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:08.112 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:08.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:08.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:08.214 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:08.215 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:08 smithi167 ceph-mon[33178]: 4.1a scrub starts 2024-09-13T13:17:08.215 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:08 smithi167 ceph-mon[33178]: 4.1a scrub ok 2024-09-13T13:17:08.215 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:08 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:17:08.216 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:08 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:17:08.216 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:17:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=sqlstore.transactions t=2024-09-13T13:17:08.084008115Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T13:17:08.253 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:08.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:08.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:08.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:08.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:08.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:08.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:08.482 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:17:08.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:08 smithi052 ceph-mon[36682]: 4.1a scrub starts 2024-09-13T13:17:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:08 smithi052 ceph-mon[36682]: 4.1a scrub ok 2024-09-13T13:17:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:08 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:17:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:08 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:17:08.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:08 smithi052 ceph-mon[29471]: 4.1a scrub starts 2024-09-13T13:17:08.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:08 smithi052 ceph-mon[29471]: 4.1a scrub ok 2024-09-13T13:17:08.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:08 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:17:08.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:08 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:17:08.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:08.516 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:17:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:17:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:17:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:17:08.516 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:17:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:08.534 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:08.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:08.636 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:08.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:08.678 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:08.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:08.779 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:08.818 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:08.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:08.854 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:08.919 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:08.940 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:08.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:09.041 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:09.071 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:09.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:09.080 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T13:17:09.173 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:09.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:09.235 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:09.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:09.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[36682]: 6.1e scrub starts 2024-09-13T13:17:09.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[36682]: 6.1e scrub ok 2024-09-13T13:17:09.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[36682]: 10.8 scrub starts 2024-09-13T13:17:09.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[36682]: pgmap v796: 148 pgs: 148 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 740 B/s rd, 0 B/s wr, 726 op/s; 14 MiB/s, 223 objects/s recovering 2024-09-13T13:17:09.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[36682]: osdmap e398: 8 total, 8 up, 7 in 2024-09-13T13:17:09.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[36682]: 10.8 scrub ok 2024-09-13T13:17:09.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2959662896' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:17:09.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[36682]: from='client.36473 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:17:09.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[29471]: 6.1e scrub starts 2024-09-13T13:17:09.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[29471]: 6.1e scrub ok 2024-09-13T13:17:09.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[29471]: 10.8 scrub starts 2024-09-13T13:17:09.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[29471]: pgmap v796: 148 pgs: 148 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 740 B/s rd, 0 B/s wr, 726 op/s; 14 MiB/s, 223 objects/s recovering 2024-09-13T13:17:09.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[29471]: osdmap e398: 8 total, 8 up, 7 in 2024-09-13T13:17:09.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[29471]: 10.8 scrub ok 2024-09-13T13:17:09.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2959662896' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:17:09.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:09 smithi052 ceph-mon[29471]: from='client.36473 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:17:09.336 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:09.366 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:09.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:09.469 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:09.498 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:09.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:09.507 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:17:09.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:09 smithi167 ceph-mon[33178]: 6.1e scrub starts 2024-09-13T13:17:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:09 smithi167 ceph-mon[33178]: 6.1e scrub ok 2024-09-13T13:17:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:09 smithi167 ceph-mon[33178]: 10.8 scrub starts 2024-09-13T13:17:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:09 smithi167 ceph-mon[33178]: pgmap v796: 148 pgs: 148 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 740 B/s rd, 0 B/s wr, 726 op/s; 14 MiB/s, 223 objects/s recovering 2024-09-13T13:17:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:09 smithi167 ceph-mon[33178]: osdmap e398: 8 total, 8 up, 7 in 2024-09-13T13:17:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:09 smithi167 ceph-mon[33178]: 10.8 scrub ok 2024-09-13T13:17:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:09 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2959662896' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:17:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:09 smithi167 ceph-mon[33178]: from='client.36473 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:17:09.601 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:09.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:09.637 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:09.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:09.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T13:17:09.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:09.770 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:09.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:09.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:09.872 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:09.918 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:09.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:09.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:09.987 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T13:17:10.019 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:10.076 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:10.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:10.161 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:17:10.177 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:10.216 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:10.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:10.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:10 smithi052 ceph-mon[29471]: 3.1e scrub starts 2024-09-13T13:17:10.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:10 smithi052 ceph-mon[29471]: 3.1e scrub ok 2024-09-13T13:17:10.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:10 smithi052 ceph-mon[29471]: from='client.36473 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:17:10.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:10 smithi052 ceph-mon[29471]: osdmap e399: 8 total, 8 up, 7 in 2024-09-13T13:17:10.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:10 smithi052 ceph-mon[29471]: 3.12 scrub starts 2024-09-13T13:17:10.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:10 smithi052 ceph-mon[29471]: 3.12 scrub ok 2024-09-13T13:17:10.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:10 smithi052 ceph-mon[36682]: 3.1e scrub starts 2024-09-13T13:17:10.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:10 smithi052 ceph-mon[36682]: 3.1e scrub ok 2024-09-13T13:17:10.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:10 smithi052 ceph-mon[36682]: from='client.36473 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:17:10.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:10 smithi052 ceph-mon[36682]: osdmap e399: 8 total, 8 up, 7 in 2024-09-13T13:17:10.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:10 smithi052 ceph-mon[36682]: 3.12 scrub starts 2024-09-13T13:17:10.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:10 smithi052 ceph-mon[36682]: 3.12 scrub ok 2024-09-13T13:17:10.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:10.321 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:10.355 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:10.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:10.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:10.457 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:10.461 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:10 smithi167 ceph-mon[33178]: 3.1e scrub starts 2024-09-13T13:17:10.461 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:10 smithi167 ceph-mon[33178]: 3.1e scrub ok 2024-09-13T13:17:10.461 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:10 smithi167 ceph-mon[33178]: from='client.36473 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:17:10.461 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:10 smithi167 ceph-mon[33178]: osdmap e399: 8 total, 8 up, 7 in 2024-09-13T13:17:10.461 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:10 smithi167 ceph-mon[33178]: 3.12 scrub starts 2024-09-13T13:17:10.461 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:10 smithi167 ceph-mon[33178]: 3.12 scrub ok 2024-09-13T13:17:10.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:10.516 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:10.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:10.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:10.659 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:10.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:10.760 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:10.779 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:17:10.779 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:17:10.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:17:10.779 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:17:10.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.779 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:133029 10.9 10:97ab46ca:::benchmark_data_smithi052_153703_object5649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:17:10.779 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.308770+0000", 2024-09-13T13:17:10.779 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 91.466577748999995, 2024-09-13T13:17:10.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.35692719899999997, 2024-09-13T13:17:10.780 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.780 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.780 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.780 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 133029 2024-09-13T13:17:10.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.780 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308770+0000", 2024-09-13T13:17:10.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308770+0000", 2024-09-13T13:17:10.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308772+0000", 2024-09-13T13:17:10.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3319999999999999e-06 2024-09-13T13:17:10.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308790+0000", 2024-09-13T13:17:10.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7960000000000001e-05 2024-09-13T13:17:10.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308804+0000", 2024-09-13T13:17:10.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3628000000000001e-05 2024-09-13T13:17:10.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308811+0000", 2024-09-13T13:17:10.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2470000000000002e-06 2024-09-13T13:17:10.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308877+0000", 2024-09-13T13:17:10.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5579000000000004e-05 2024-09-13T13:17:10.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.308990+0000", 2024-09-13T13:17:10.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113732 2024-09-13T13:17:10.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:17:10.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309190+0000", 2024-09-13T13:17:10.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000199839 2024-09-13T13:17:10.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.311704+0000", 2024-09-13T13:17:10.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025138460000000001 2024-09-13T13:17:10.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.313823+0000", 2024-09-13T13:17:10.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021192849999999998 2024-09-13T13:17:10.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665572+0000", 2024-09-13T13:17:10.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.35174855399999999 2024-09-13T13:17:10.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665647+0000", 2024-09-13T13:17:10.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5347999999999998e-05 2024-09-13T13:17:10.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665697+0000", 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9849000000000003e-05 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:178617 10.3 10:c6ec508e:::benchmark_data_smithi052_153703_object51237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:22.234413+0000", 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 48.540934739000001, 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.33432663499999998, 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.786 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.787 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 178617 2024-09-13T13:17:10.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.787 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.234413+0000", 2024-09-13T13:17:10.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.234413+0000", 2024-09-13T13:17:10.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.234415+0000", 2024-09-13T13:17:10.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0269999999999998e-06 2024-09-13T13:17:10.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.234421+0000", 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7960000000000001e-06 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.234432+0000", 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0883999999999999e-05 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.234439+0000", 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1930000000000004e-06 2024-09-13T13:17:10.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.234544+0000", 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104973 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.234673+0000", 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000128687 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3,4", 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.234810+0000", 2024-09-13T13:17:10.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013725900000000001 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.239216+0000", 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044061049999999996 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.241011+0000", 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017950900000000001 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.246670+0000", 2024-09-13T13:17:10.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056589029999999998 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.568619+0000", 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.321949186 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.568677+0000", 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7670000000000002e-05 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.568740+0000", 2024-09-13T13:17:10.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2861999999999995e-05 2024-09-13T13:17:10.793 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.793 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.793 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.793 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:178627 10.3 10:cad7953b:::benchmark_data_smithi052_153703_object51247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T13:17:10.793 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:22.241045+0000", 2024-09-13T13:17:10.793 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 48.534302683, 2024-09-13T13:17:10.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.33572172500000003, 2024-09-13T13:17:10.793 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.794 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.794 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 178627 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.241045+0000", 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.241045+0000", 2024-09-13T13:17:10.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.241047+0000", 2024-09-13T13:17:10.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.745e-06 2024-09-13T13:17:10.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.241052+0000", 2024-09-13T13:17:10.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5419999999999996e-06 2024-09-13T13:17:10.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.241063+0000", 2024-09-13T13:17:10.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0249e-05 2024-09-13T13:17:10.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.241070+0000", 2024-09-13T13:17:10.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.362e-06 2024-09-13T13:17:10.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.241157+0000", 2024-09-13T13:17:10.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6924999999999999e-05 2024-09-13T13:17:10.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.241274+0000", 2024-09-13T13:17:10.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116802 2024-09-13T13:17:10.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3,4", 2024-09-13T13:17:10.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.241396+0000", 2024-09-13T13:17:10.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012276499999999999 2024-09-13T13:17:10.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.244423+0000", 2024-09-13T13:17:10.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003027035 2024-09-13T13:17:10.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.244443+0000", 2024-09-13T13:17:10.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9782000000000001e-05 2024-09-13T13:17:10.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.246707+0000", 2024-09-13T13:17:10.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002263298 2024-09-13T13:17:10.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.576601+0000", 2024-09-13T13:17:10.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.32989454200000001 2024-09-13T13:17:10.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.576661+0000", 2024-09-13T13:17:10.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9533000000000001e-05 2024-09-13T13:17:10.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.576767+0000", 2024-09-13T13:17:10.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010614499999999999 2024-09-13T13:17:10.801 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.801 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.801 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.801 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:178637 10.3 10:c9981b0f:::benchmark_data_smithi052_153703_object51257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T13:17:10.801 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:22.251854+0000", 2024-09-13T13:17:10.801 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 48.523493295999998, 2024-09-13T13:17:10.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.32509291000000001, 2024-09-13T13:17:10.801 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 178637 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.251854+0000", 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.251854+0000", 2024-09-13T13:17:10.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.251857+0000", 2024-09-13T13:17:10.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3080000000000002e-06 2024-09-13T13:17:10.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.251862+0000", 2024-09-13T13:17:10.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7440000000000004e-06 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.251876+0000", 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4e-05 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.251884+0000", 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9480000000000008e-06 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.251986+0000", 2024-09-13T13:17:10.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010189 2024-09-13T13:17:10.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.252109+0000", 2024-09-13T13:17:10.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123149 2024-09-13T13:17:10.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3,4", 2024-09-13T13:17:10.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.252235+0000", 2024-09-13T13:17:10.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000125678 2024-09-13T13:17:10.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.259267+0000", 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070315639999999997 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.259327+0000", 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9806999999999998e-05 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.259346+0000", 2024-09-13T13:17:10.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9406e-05 2024-09-13T13:17:10.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.576809+0000", 2024-09-13T13:17:10.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31746343300000002 2024-09-13T13:17:10.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.576854+0000", 2024-09-13T13:17:10.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4472999999999999e-05 2024-09-13T13:17:10.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.576947+0000", 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3510000000000004e-05 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205092 10.9 10:9bc37bd5:::benchmark_data_smithi052_153703_object77712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.052783+0000", 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.722565041999999, 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2303267840000001, 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.808 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:17:10.809 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.809 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.809 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.809 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205092 2024-09-13T13:17:10.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.809 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.052783+0000", 2024-09-13T13:17:10.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.052783+0000", 2024-09-13T13:17:10.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.052783+0000", 2024-09-13T13:17:10.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7299999999999995e-07 2024-09-13T13:17:10.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.052791+0000", 2024-09-13T13:17:10.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6490000000000008e-06 2024-09-13T13:17:10.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.052797+0000", 2024-09-13T13:17:10.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0240000000000003e-06 2024-09-13T13:17:10.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.052802+0000", 2024-09-13T13:17:10.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3700000000000003e-06 2024-09-13T13:17:10.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.053393+0000", 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00059056899999999999 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.053495+0000", 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101637 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4,7", 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.053601+0000", 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106696 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.059189+0000", 2024-09-13T13:17:10.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055881150000000003 2024-09-13T13:17:10.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.059357+0000", 2024-09-13T13:17:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000167887 2024-09-13T13:17:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.059729+0000", 2024-09-13T13:17:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000371553 2024-09-13T13:17:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.192738+0000", 2024-09-13T13:17:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.133009234 2024-09-13T13:17:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.192748+0000", 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5829999999999996e-06 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.283042+0000", 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0902940800000001 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.283109+0000", 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7713999999999996e-05 2024-09-13T13:17:10.818 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.819 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.819 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205124 10.9 10:97a36986:::benchmark_data_smithi052_153703_object77744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.819 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.194292+0000", 2024-09-13T13:17:10.819 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.581056148000002, 2024-09-13T13:17:10.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1059571340000001, 2024-09-13T13:17:10.819 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.819 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.819 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.819 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.819 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205124 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194292+0000", 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194292+0000", 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194292+0000", 2024-09-13T13:17:10.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4200000000000001e-07 2024-09-13T13:17:10.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194295+0000", 2024-09-13T13:17:10.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2310000000000001e-06 2024-09-13T13:17:10.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194300+0000", 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6489999999999999e-06 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194305+0000", 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8119999999999998e-06 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194346+0000", 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1486000000000002e-05 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194355+0000", 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1819999999999995e-06 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.283537+0000", 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0891813960000001 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.283632+0000", 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4914000000000006e-05 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.283740+0000", 2024-09-13T13:17:10.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108137 2024-09-13T13:17:10.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.292289+0000", 2024-09-13T13:17:10.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085489859999999997 2024-09-13T13:17:10.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.292329+0000", 2024-09-13T13:17:10.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9690000000000001e-05 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300162+0000", 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078332079999999995 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300209+0000", 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6978000000000003e-05 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300249+0000", 2024-09-13T13:17:10.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0022999999999999e-05 2024-09-13T13:17:10.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.826 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.826 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205128 10.9 10:90438080:::benchmark_data_smithi052_153703_object77748:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.826 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.204172+0000", 2024-09-13T13:17:10.826 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.571175305000001, 2024-09-13T13:17:10.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0962660289999999, 2024-09-13T13:17:10.826 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.826 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.826 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.826 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:10.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:10.828 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.828 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.828 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205128 2024-09-13T13:17:10.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.828 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204172+0000", 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204172+0000", 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204174+0000", 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.747e-06 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204180+0000", 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1550000000000004e-06 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204195+0000", 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4477999999999999e-05 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204203+0000", 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8849999999999999e-06 2024-09-13T13:17:10.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204271+0000", 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8397000000000003e-05 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204281+0000", 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0349e-05 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.286598+0000", 2024-09-13T13:17:10.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.082317011 2024-09-13T13:17:10.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.286697+0000", 2024-09-13T13:17:10.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8559000000000003e-05 2024-09-13T13:17:10.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.286816+0000", 2024-09-13T13:17:10.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119269 2024-09-13T13:17:10.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300064+0000", 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013247765 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300261+0000", 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019662099999999999 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300310+0000", 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9104999999999998e-05 2024-09-13T13:17:10.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300397+0000", 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7416000000000005e-05 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300438+0000", 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1272000000000001e-05 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205146 10.9 10:93785db0:::benchmark_data_smithi052_153703_object77766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.218025+0000", 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.557323044, 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.082663522, 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205146 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218025+0000", 2024-09-13T13:17:10.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218025+0000", 2024-09-13T13:17:10.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218027+0000", 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.193e-06 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218033+0000", 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0549999999999996e-06 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218045+0000", 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1770000000000001e-05 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218052+0000", 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4580000000000004e-06 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218182+0000", 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012962600000000001 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218190+0000", 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7800000000000001e-06 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.289337+0000", 2024-09-13T13:17:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.071147294 2024-09-13T13:17:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.289501+0000", 2024-09-13T13:17:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016407999999999999 2024-09-13T13:17:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.289617+0000", 2024-09-13T13:17:10.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011618 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300266+0000", 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010648502000000001 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300541+0000", 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027551399999999998 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300600+0000", 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8829999999999997e-05 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300652+0000", 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2049999999999998e-05 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300688+0000", 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6189999999999997e-05 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.841 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.842 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205179 10.9 10:98229baf:::benchmark_data_smithi052_153703_object77799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.842 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.251737+0000", 2024-09-13T13:17:10.842 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.523610520999998, 2024-09-13T13:17:10.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0491762529999999, 2024-09-13T13:17:10.842 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.842 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.842 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.842 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.842 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.842 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.843 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205179 2024-09-13T13:17:10.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.843 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251737+0000", 2024-09-13T13:17:10.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251737+0000", 2024-09-13T13:17:10.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251739+0000", 2024-09-13T13:17:10.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.272e-06 2024-09-13T13:17:10.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251748+0000", 2024-09-13T13:17:10.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2479999999999996e-06 2024-09-13T13:17:10.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251763+0000", 2024-09-13T13:17:10.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5529e-05 2024-09-13T13:17:10.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251771+0000", 2024-09-13T13:17:10.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6580000000000002e-06 2024-09-13T13:17:10.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251881+0000", 2024-09-13T13:17:10.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110259 2024-09-13T13:17:10.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251889+0000", 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7929999999999998e-06 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.292357+0000", 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.04046767 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.292529+0000", 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017212400000000001 2024-09-13T13:17:10.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.292646+0000", 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011692999999999999 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300270+0000", 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.007624713 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300763+0000", 2024-09-13T13:17:10.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00049291299999999999 2024-09-13T13:17:10.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300836+0000", 2024-09-13T13:17:10.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2694000000000006e-05 2024-09-13T13:17:10.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300879+0000", 2024-09-13T13:17:10.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3108999999999997e-05 2024-09-13T13:17:10.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300913+0000", 2024-09-13T13:17:10.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4341000000000002e-05 2024-09-13T13:17:10.849 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.849 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.849 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.849 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205184 10.9 10:98702a24:::benchmark_data_smithi052_153703_object77804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.849 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.256046+0000", 2024-09-13T13:17:10.849 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.519301427999999, 2024-09-13T13:17:10.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0452002440000001, 2024-09-13T13:17:10.849 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.849 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205184 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256046+0000", 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256046+0000", 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256048+0000", 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.218e-06 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256054+0000", 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3210000000000001e-06 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256063+0000", 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3530000000000001e-06 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256070+0000", 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3779999999999998e-06 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256126+0000", 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6461e-05 2024-09-13T13:17:10.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256134+0000", 2024-09-13T13:17:10.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0679999999999997e-06 2024-09-13T13:17:10.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.295757+0000", 2024-09-13T13:17:10.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0396233669999999 2024-09-13T13:17:10.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.295880+0000", 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012273699999999999 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.296241+0000", 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036082900000000001 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300453+0000", 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042117359999999998 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300971+0000", 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051871199999999995 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.301033+0000", 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1320000000000002e-05 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.301208+0000", 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017559800000000001 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.301247+0000", 2024-09-13T13:17:10.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8145999999999997e-05 2024-09-13T13:17:10.856 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.856 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.856 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.856 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205219 10.9 10:91ed7462:::benchmark_data_smithi052_153703_object77839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.856 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.291958+0000", 2024-09-13T13:17:10.856 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.483389604999999, 2024-09-13T13:17:10.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0217932759999999, 2024-09-13T13:17:10.856 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.856 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205219 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.291958+0000", 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.291958+0000", 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.291959+0000", 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.311e-06 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.291964+0000", 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7720000000000002e-06 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.291980+0000", 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5466000000000001e-05 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.291986+0000", 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6989999999999997e-06 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.292049+0000", 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2577999999999996e-05 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.292056+0000", 2024-09-13T13:17:10.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3150000000000003e-06 2024-09-13T13:17:10.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.301362+0000", 2024-09-13T13:17:10.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.009306212 2024-09-13T13:17:10.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.301462+0000", 2024-09-13T13:17:10.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9495999999999997e-05 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.301583+0000", 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121166 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.313465+0000", 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011881844000000001 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.313508+0000", 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2598000000000001e-05 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.313668+0000", 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000160311 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.313712+0000", 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4039000000000003e-05 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.313751+0000", 2024-09-13T13:17:10.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9468999999999998e-05 2024-09-13T13:17:10.863 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.863 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.863 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.863 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205230 10.9 10:994b8581:::benchmark_data_smithi052_153703_object77850:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.863 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.301833+0000", 2024-09-13T13:17:10.863 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.473514992999998, 2024-09-13T13:17:10.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.012034592, 2024-09-13T13:17:10.863 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.863 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.864 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.864 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.864 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.864 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.864 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205230 2024-09-13T13:17:10.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.864 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.301833+0000", 2024-09-13T13:17:10.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.301833+0000", 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.301834+0000", 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.595e-06 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.301839+0000", 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7190000000000001e-06 2024-09-13T13:17:10.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.301848+0000", 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2369999999999998e-06 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.301855+0000", 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4740000000000002e-06 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.301899+0000", 2024-09-13T13:17:10.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4428999999999998e-05 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.301905+0000", 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5779999999999998e-06 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.307299+0000", 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.005394372 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.307409+0000", 2024-09-13T13:17:10.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109474 2024-09-13T13:17:10.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.307549+0000", 2024-09-13T13:17:10.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014016399999999999 2024-09-13T13:17:10.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.313545+0000", 2024-09-13T13:17:10.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059965490000000003 2024-09-13T13:17:10.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.313657+0000", 2024-09-13T13:17:10.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111465 2024-09-13T13:17:10.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.313758+0000", 2024-09-13T13:17:10.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010083500000000001 2024-09-13T13:17:10.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.313828+0000", 2024-09-13T13:17:10.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9924000000000003e-05 2024-09-13T13:17:10.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.313867+0000", 2024-09-13T13:17:10.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9777000000000002e-05 2024-09-13T13:17:10.870 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.870 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.870 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.871 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205259 10.9 10:9bdd94ce:::benchmark_data_smithi052_153703_object77879:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.871 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.338425+0000", 2024-09-13T13:17:10.871 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.436922999, 2024-09-13T13:17:10.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97841677299999996, 2024-09-13T13:17:10.871 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.872 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.872 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.872 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.872 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.872 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.872 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205259 2024-09-13T13:17:10.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.872 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.338425+0000", 2024-09-13T13:17:10.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.338425+0000", 2024-09-13T13:17:10.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.338427+0000", 2024-09-13T13:17:10.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9099999999999999e-06 2024-09-13T13:17:10.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.338436+0000", 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2860000000000005e-06 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.338450+0000", 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3849000000000001e-05 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.338458+0000", 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7600000000000002e-06 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.338525+0000", 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7367999999999996e-05 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.338533+0000", 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1289999999999996e-06 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.310290+0000", 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97175682299999999 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.310483+0000", 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.310591+0000", 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107629 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.313600+0000", 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003009112 2024-09-13T13:17:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.316739+0000", 2024-09-13T13:17:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031388039999999998 2024-09-13T13:17:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.316753+0000", 2024-09-13T13:17:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.431e-05 2024-09-13T13:17:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.316801+0000", 2024-09-13T13:17:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8155999999999998e-05 2024-09-13T13:17:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.316842+0000", 2024-09-13T13:17:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0422e-05 2024-09-13T13:17:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205266 10.9 10:995d074d:::benchmark_data_smithi052_153703_object77886:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.346697+0000", 2024-09-13T13:17:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.428651048999999, 2024-09-13T13:17:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98730642400000002, 2024-09-13T13:17:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205266 2024-09-13T13:17:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.346697+0000", 2024-09-13T13:17:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.346697+0000", 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.346699+0000", 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.277e-06 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.346705+0000", 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9200000000000001e-06 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.346716+0000", 2024-09-13T13:17:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0866000000000001e-05 2024-09-13T13:17:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.346723+0000", 2024-09-13T13:17:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3030000000000002e-06 2024-09-13T13:17:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.350730+0000", 2024-09-13T13:17:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040072170000000004 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.350739+0000", 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.313900+0000", 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96316096100000004 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.314000+0000", 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.925e-05 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.314107+0000", 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107463 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.322429+0000", 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0083218440000000001 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.322466+0000", 2024-09-13T13:17:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7335000000000001e-05 2024-09-13T13:17:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.333872+0000", 2024-09-13T13:17:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011406144 2024-09-13T13:17:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.333954+0000", 2024-09-13T13:17:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2189999999999997e-05 2024-09-13T13:17:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.334003+0000", 2024-09-13T13:17:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8618000000000001e-05 2024-09-13T13:17:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205271 10.9 10:995ed11d:::benchmark_data_smithi052_153703_object77891:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.351263+0000", 2024-09-13T13:17:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.424084370999999, 2024-09-13T13:17:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98670022999999996, 2024-09-13T13:17:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205271 2024-09-13T13:17:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.351263+0000", 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.351263+0000", 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.351265+0000", 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.635e-06 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.351270+0000", 2024-09-13T13:17:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5170000000000001e-06 2024-09-13T13:17:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.351278+0000", 2024-09-13T13:17:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2719999999999997e-06 2024-09-13T13:17:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.351285+0000", 2024-09-13T13:17:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.968e-06 2024-09-13T13:17:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.351336+0000", 2024-09-13T13:17:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0767999999999999e-05 2024-09-13T13:17:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.351343+0000", 2024-09-13T13:17:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3300000000000001e-06 2024-09-13T13:17:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.316871+0000", 2024-09-13T13:17:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.965528214 2024-09-13T13:17:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.316967+0000", 2024-09-13T13:17:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5449e-05 2024-09-13T13:17:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.317051+0000", 2024-09-13T13:17:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4420999999999997e-05 2024-09-13T13:17:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.334010+0000", 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016958595999999999 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.337748+0000", 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037383709999999999 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.337871+0000", 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012279399999999999 2024-09-13T13:17:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.337920+0000", 2024-09-13T13:17:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9555999999999998e-05 2024-09-13T13:17:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.337964+0000", 2024-09-13T13:17:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3339e-05 2024-09-13T13:17:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205285 10.9 10:9eaa858b:::benchmark_data_smithi052_153703_object77905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.371540+0000", 2024-09-13T13:17:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.403807796999999, 2024-09-13T13:17:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96653894500000004, 2024-09-13T13:17:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205285 2024-09-13T13:17:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.371540+0000", 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.371540+0000", 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.371541+0000", 2024-09-13T13:17:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.269e-06 2024-09-13T13:17:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.371548+0000", 2024-09-13T13:17:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1029999999999996e-06 2024-09-13T13:17:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.371562+0000", 2024-09-13T13:17:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3971000000000001e-05 2024-09-13T13:17:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.371569+0000", 2024-09-13T13:17:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4169999999999997e-06 2024-09-13T13:17:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.371615+0000", 2024-09-13T13:17:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6628000000000001e-05 2024-09-13T13:17:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.371622+0000", 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6549999999999998e-06 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.319620+0000", 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94799815099999996 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.319720+0000", 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010011800000000001 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.319840+0000", 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119472 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.337695+0000", 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017855324999999998 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.337803+0000", 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108041 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.337999+0000", 2024-09-13T13:17:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019593799999999999 2024-09-13T13:17:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.338047+0000", 2024-09-13T13:17:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7477e-05 2024-09-13T13:17:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.338079+0000", 2024-09-13T13:17:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2379999999999998e-05 2024-09-13T13:17:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205299 10.9 10:928048b9:::benchmark_data_smithi052_153703_object77919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.391348+0000", 2024-09-13T13:17:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.383999281000001, 2024-09-13T13:17:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94684358499999999, 2024-09-13T13:17:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205299 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.391348+0000", 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.391348+0000", 2024-09-13T13:17:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.391351+0000", 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.075e-06 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.391355+0000", 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3880000000000002e-06 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.391364+0000", 2024-09-13T13:17:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0420000000000008e-06 2024-09-13T13:17:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.391371+0000", 2024-09-13T13:17:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9469999999999997e-06 2024-09-13T13:17:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.391431+0000", 2024-09-13T13:17:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9728999999999997e-05 2024-09-13T13:17:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.391439+0000", 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1939999999999998e-06 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.322494+0000", 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93105474099999996 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.322584+0000", 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9986000000000002e-05 2024-09-13T13:17:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.322685+0000", 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101111 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.337700+0000", 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015015351999999999 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.337838+0000", 2024-09-13T13:17:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000137486 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.338114+0000", 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027632900000000002 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.338159+0000", 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5531000000000002e-05 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.338192+0000", 2024-09-13T13:17:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2673999999999999e-05 2024-09-13T13:17:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205359 10.9 10:9e80571c:::benchmark_data_smithi052_153703_object77979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.553246+0000", 2024-09-13T13:17:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.222101856999998, 2024-09-13T13:17:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.79013571100000002, 2024-09-13T13:17:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205359 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.553246+0000", 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.553246+0000", 2024-09-13T13:17:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.553248+0000", 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9309999999999998e-06 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.553253+0000", 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9069999999999998e-06 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.553264+0000", 2024-09-13T13:17:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1368e-05 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.553271+0000", 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1770000000000001e-06 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.553327+0000", 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6206999999999999e-05 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.553337+0000", 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9250000000000007e-06 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.325193+0000", 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.77185517100000001 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.325321+0000", 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000128851 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.325440+0000", 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118841 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.337705+0000", 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012264880000000001 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.338232+0000", 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000526503 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.343286+0000", 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050546610000000002 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.343338+0000", 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1761999999999997e-05 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.343382+0000", 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3526999999999998e-05 2024-09-13T13:17:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205364 10.9 10:902ecb36:::benchmark_data_smithi052_153703_object77984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.565520+0000", 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.209827279999999, 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78132959400000002, 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205364 2024-09-13T13:17:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.565520+0000", 2024-09-13T13:17:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.565520+0000", 2024-09-13T13:17:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.565523+0000", 2024-09-13T13:17:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3269999999999999e-06 2024-09-13T13:17:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.565532+0000", 2024-09-13T13:17:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9150000000000002e-06 2024-09-13T13:17:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.565548+0000", 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5841e-05 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.565556+0000", 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2339999999999994e-06 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.565683+0000", 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012740900000000001 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.565696+0000", 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2814e-05 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.328153+0000", 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.76245668200000005 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.328258+0000", 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010483400000000001 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.328364+0000", 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106147 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.337710+0000", 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0093462089999999994 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.342857+0000", 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0051467190000000001 2024-09-13T13:17:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.346750+0000", 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038934429999999999 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.346808+0000", 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8276000000000002e-05 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.346850+0000", 2024-09-13T13:17:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1743999999999999e-05 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205375 10.9 10:91d83f9e:::benchmark_data_smithi052_153703_object77995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.605830+0000", 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.169517595999999, 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.741225672, 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205375 2024-09-13T13:17:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.605830+0000", 2024-09-13T13:17:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.605830+0000", 2024-09-13T13:17:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.605831+0000", 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.605836+0000", 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5859999999999998e-06 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.605844+0000", 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4819999999999997e-06 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.605848+0000", 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.668e-06 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.605902+0000", 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3421999999999998e-05 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.605911+0000", 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8829999999999995e-06 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.330981+0000", 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.72507021000000005 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.331079+0000", 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7798000000000001e-05 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.331184+0000", 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105643 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.337758+0000", 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065741439999999996 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.342936+0000", 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0051775190000000002 2024-09-13T13:17:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.346907+0000", 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039714650000000004 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.347007+0000", 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100133 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.347056+0000", 2024-09-13T13:17:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8396000000000002e-05 2024-09-13T13:17:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:10.929 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:17:10.929 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:10.966 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:10.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:10.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:11.006 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:17:11.007 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:17:11.007 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:17:11.007 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:17:11.018 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:11.035 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:17:11.035 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:17:11.066 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:17:11.066 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:17:11.067 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:11.096 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:17:11.108 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:17:11.108 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:17:11.113 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:11.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:11.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:11.154 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:17:11.154 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:17:11.186 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:17:11.186 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:17:11.211 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:17:11.211 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:17:11.216 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:11.253 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:11.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:11.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:11.387 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:11.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[36682]: 3.14 scrub starts 2024-09-13T13:17:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[36682]: 3.14 scrub ok 2024-09-13T13:17:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[36682]: pgmap v799: 148 pgs: 148 active+clean; 2.0 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 993 op/s; 6.1 MiB/s, 98 objects/s recovering 2024-09-13T13:17:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[36682]: 6.11 scrub starts 2024-09-13T13:17:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[36682]: 6.11 scrub ok 2024-09-13T13:17:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/497067543' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:17:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/826940162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:11.489 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[29471]: 3.14 scrub starts 2024-09-13T13:17:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[29471]: 3.14 scrub ok 2024-09-13T13:17:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[29471]: pgmap v799: 148 pgs: 148 active+clean; 2.0 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 993 op/s; 6.1 MiB/s, 98 objects/s recovering 2024-09-13T13:17:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[29471]: 6.11 scrub starts 2024-09-13T13:17:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[29471]: 6.11 scrub ok 2024-09-13T13:17:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/497067543' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:17:11.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/826940162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:11.523 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:11.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:11 smithi167 ceph-mon[33178]: 3.14 scrub starts 2024-09-13T13:17:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:11 smithi167 ceph-mon[33178]: 3.14 scrub ok 2024-09-13T13:17:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:11 smithi167 ceph-mon[33178]: pgmap v799: 148 pgs: 148 active+clean; 2.0 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 993 op/s; 6.1 MiB/s, 98 objects/s recovering 2024-09-13T13:17:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:11 smithi167 ceph-mon[33178]: 6.11 scrub starts 2024-09-13T13:17:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:11 smithi167 ceph-mon[33178]: 6.11 scrub ok 2024-09-13T13:17:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/497067543' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:17:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/826940162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:11.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:11.661 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:11.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:11.762 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:11.794 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:11.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:11.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:11.866 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T13:17:11.896 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:11.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:11.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:11.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:12.053 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:12.085 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:12.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:12.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:12.186 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:12.235 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:12.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:12.301 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:17:12.337 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:12.366 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:12.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:12.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:12.440 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:12 smithi167 ceph-mon[33178]: 4.19 scrub starts 2024-09-13T13:17:12.440 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:12 smithi167 ceph-mon[33178]: 4.19 scrub ok 2024-09-13T13:17:12.440 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:12 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:12.440 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:12 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/497067543' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:17:12.440 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:12 smithi167 ceph-mon[33178]: osdmap e400: 8 total, 8 up, 7 in 2024-09-13T13:17:12.468 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:12 smithi052 ceph-mon[29471]: 4.19 scrub starts 2024-09-13T13:17:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:12 smithi052 ceph-mon[29471]: 4.19 scrub ok 2024-09-13T13:17:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:12 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:12 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/497067543' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:17:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:12 smithi052 ceph-mon[29471]: osdmap e400: 8 total, 8 up, 7 in 2024-09-13T13:17:12.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:12 smithi052 ceph-mon[36682]: 4.19 scrub starts 2024-09-13T13:17:12.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:12 smithi052 ceph-mon[36682]: 4.19 scrub ok 2024-09-13T13:17:12.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:12 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:12.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:12 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/497067543' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:17:12.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:12 smithi052 ceph-mon[36682]: osdmap e400: 8 total, 8 up, 7 in 2024-09-13T13:17:12.503 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:12.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:12.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:12.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:12.657 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:12.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:12.758 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:12.796 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:12.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:12.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:12.898 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:12.941 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:12.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:12.954 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:17:12.955 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:17:13.043 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:13.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:13.082 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:13.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:13.127 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T13:17:13.184 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:13.211 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:13.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:13.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:13.315 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:13.347 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:13.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:13.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:13.363 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:13 smithi052 ceph-mon[29471]: pgmap v801: 148 pgs: 148 active+clean; 1.9 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s; 1.5 MiB/s, 23 objects/s recovering 2024-09-13T13:17:13.363 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1510432351' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:17:13.363 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:13 smithi052 ceph-mon[29471]: from='client.36488 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:17:13.364 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:13 smithi052 ceph-mon[36682]: pgmap v801: 148 pgs: 148 active+clean; 1.9 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s; 1.5 MiB/s, 23 objects/s recovering 2024-09-13T13:17:13.364 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1510432351' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:17:13.364 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:13 smithi052 ceph-mon[36682]: from='client.36488 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:17:13.450 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:13.482 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:13.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:13 smithi167 ceph-mon[33178]: pgmap v801: 148 pgs: 148 active+clean; 1.9 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s; 1.5 MiB/s, 23 objects/s recovering 2024-09-13T13:17:13.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1510432351' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:17:13.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:13 smithi167 ceph-mon[33178]: from='client.36488 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:17:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:13.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:13.585 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:13.620 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:13.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:13.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:13.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T13:17:13.722 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:13.748 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:13.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:13.850 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:13.887 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:13.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:13.990 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:14.016 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:14.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:14.076 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T13:17:14.076 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:17:14.118 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:14.134 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:17:14.148 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:14.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:14.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:14.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:14.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:14.397 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:14.434 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:14.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:14.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:14.456 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:14 smithi052 ceph-mon[36682]: from='client.36488 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:17:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:14 smithi052 ceph-mon[36682]: osdmap e401: 8 total, 8 up, 7 in 2024-09-13T13:17:14.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:14 smithi052 ceph-mon[29471]: from='client.36488 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:17:14.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:14 smithi052 ceph-mon[29471]: osdmap e401: 8 total, 8 up, 7 in 2024-09-13T13:17:14.524 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:14 smithi167 ceph-mon[33178]: from='client.36488 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:17:14.524 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:14 smithi167 ceph-mon[33178]: osdmap e401: 8 total, 8 up, 7 in 2024-09-13T13:17:14.537 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:14.571 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:14.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:14.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:14.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:14.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:14.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:14.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:17:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:17:14.824 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:17:14.834 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:14.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:14.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:14.936 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:14.966 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:14.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:15.068 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:15.108 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:15.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:15.210 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:15.213 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:15.230 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:15.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:15.332 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:17:15.334 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:17:15.335 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:15.373 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:15.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:15.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:15.407 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:17:15.442 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:15 smithi052 ceph-mon[36682]: pgmap v803: 148 pgs: 148 active+clean; 1.7 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 869 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-13T13:17:15.442 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:15.442 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:15 smithi052 ceph-mon[29471]: pgmap v803: 148 pgs: 148 active+clean; 1.7 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 869 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-13T13:17:15.443 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:15.475 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:15.515 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:15.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:15.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:15 smithi167 ceph-mon[33178]: pgmap v803: 148 pgs: 148 active+clean; 1.7 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 869 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-13T13:17:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:15.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:15.657 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:15.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:15.759 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:15.781 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:15.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:15.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:15.883 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:15.915 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:15.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:16.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:16.045 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:16.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:16.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:16.147 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:16.188 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:16.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:16.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:16.332 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:16.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:16.434 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:16.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:16.466 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:16.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:16.474 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:16 smithi052 ceph-mon[36682]: from='client.16626 v1:172.21.15.52:0/569042121' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:17:16.474 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:17:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:17:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:17:16.475 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:16 smithi052 ceph-mon[29471]: from='client.16626 v1:172.21.15.52:0/569042121' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:17:16.530 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":803,"stamp":"2024-09-13T13:17:13.964254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1844714035,"num_objects":28339,"num_object_clones":0,"num_object_copies":85017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28339,"num_whiteouts":0,"num_read":1516,"num_read_kb":1375,"num_write":227451,"num_write_kb":8153802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19934,"num_bytes_recovered":1294210562,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78739,"ondisk_log_size":78739,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":9590920,"kb_used_data":5466560,"kb_used_omap":13,"kb_used_meta":4124146,"kb_avail":646625144,"statfs":{"total":671965249536,"available":662144147456,"internally_reserved":0,"allocated":5597757440,"data_stored":5594865181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13997,"internal_metadata":4223125843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,3,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-496369664,"num_objects":-7574,"num_object_clones":0,"num_object_copies":-22722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7574,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7574,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.890725"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":726,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.077249+0000","last_change":"2024-09-13T13:16:12.076433+0000","last_active":"2024-09-13T13:17:09.077249+0000","last_peered":"2024-09-13T13:17:09.077249+0000","last_clean":"2024-09-13T13:17:09.077249+0000","last_became_active":"2024-09-13T13:16:12.075424+0000","last_became_peered":"2024-09-13T13:16:12.075424+0000","last_unstale":"2024-09-13T13:17:09.077249+0000","last_undegraded":"2024-09-13T13:17:09.077249+0000","last_fullsized":"2024-09-13T13:17:09.077249+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.368588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.368588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:15.860241+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":221,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135044+0000","last_change":"2024-09-13T13:15:04.344174+0000","last_active":"2024-09-13T13:17:13.135044+0000","last_peered":"2024-09-13T13:17:13.135044+0000","last_clean":"2024-09-13T13:17:13.135044+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:17:13.135044+0000","last_undegraded":"2024-09-13T13:17:13.135044+0000","last_fullsized":"2024-09-13T13:17:13.135044+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:15:04.344092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.344092+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":728,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.094741+0000","last_change":"2024-09-13T13:16:30.163153+0000","last_active":"2024-09-13T13:17:09.094741+0000","last_peered":"2024-09-13T13:17:09.094741+0000","last_clean":"2024-09-13T13:17:09.094741+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:17:09.094741+0000","last_undegraded":"2024-09-13T13:17:09.094741+0000","last_fullsized":"2024-09-13T13:17:09.094741+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:30.163077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:30.163077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:54.764628+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":219,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131254+0000","last_change":"2024-09-13T13:15:51.244938+0000","last_active":"2024-09-13T13:17:13.131254+0000","last_peered":"2024-09-13T13:17:13.131254+0000","last_clean":"2024-09-13T13:17:13.131254+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:17:13.131254+0000","last_undegraded":"2024-09-13T13:17:13.131254+0000","last_fullsized":"2024-09-13T13:17:13.131254+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:19.110420+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":728,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.098610+0000","last_change":"2024-09-13T13:11:47.852762+0000","last_active":"2024-09-13T13:17:11.098610+0000","last_peered":"2024-09-13T13:17:11.098610+0000","last_clean":"2024-09-13T13:17:11.098610+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:17:11.098610+0000","last_undegraded":"2024-09-13T13:17:11.098610+0000","last_fullsized":"2024-09-13T13:17:11.098610+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:19.083537+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":765,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.099524+0000","last_change":"2024-09-13T13:17:08.938797+0000","last_active":"2024-09-13T13:17:11.099524+0000","last_peered":"2024-09-13T13:17:11.099524+0000","last_clean":"2024-09-13T13:17:11.099524+0000","last_became_active":"2024-09-13T13:16:12.069442+0000","last_became_peered":"2024-09-13T13:16:12.069442+0000","last_unstale":"2024-09-13T13:17:11.099524+0000","last_undegraded":"2024-09-13T13:17:11.099524+0000","last_fullsized":"2024-09-13T13:17:11.099524+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:08.938750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:08.938750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:13.091171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":823,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.098309+0000","last_change":"2024-09-13T13:17:11.044562+0000","last_active":"2024-09-13T13:17:11.098309+0000","last_peered":"2024-09-13T13:17:11.098309+0000","last_clean":"2024-09-13T13:17:11.098309+0000","last_became_active":"2024-09-13T13:16:12.074619+0000","last_became_peered":"2024-09-13T13:16:12.074619+0000","last_unstale":"2024-09-13T13:17:11.098309+0000","last_undegraded":"2024-09-13T13:17:11.098309+0000","last_fullsized":"2024-09-13T13:17:11.098309+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:17:11.044472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:11.044472+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:18.681524+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":203,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134963+0000","last_change":"2024-09-13T13:12:52.997865+0000","last_active":"2024-09-13T13:17:13.134963+0000","last_peered":"2024-09-13T13:17:13.134963+0000","last_clean":"2024-09-13T13:17:13.134963+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:17:13.134963+0000","last_undegraded":"2024-09-13T13:17:13.134963+0000","last_fullsized":"2024-09-13T13:17:13.134963+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:17.305608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:17.305608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:12.478035+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":794,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129425+0000","last_change":"2024-09-13T13:13:41.518378+0000","last_active":"2024-09-13T13:17:13.129425+0000","last_peered":"2024-09-13T13:17:13.129425+0000","last_clean":"2024-09-13T13:17:13.129425+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:17:13.129425+0000","last_undegraded":"2024-09-13T13:17:13.129425+0000","last_fullsized":"2024-09-13T13:17:13.129425+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:41.518322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.518322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":221,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131332+0000","last_change":"2024-09-13T13:17:06.406253+0000","last_active":"2024-09-13T13:17:13.131332+0000","last_peered":"2024-09-13T13:17:13.131332+0000","last_clean":"2024-09-13T13:17:13.131332+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:17:13.131332+0000","last_undegraded":"2024-09-13T13:17:13.131332+0000","last_fullsized":"2024-09-13T13:17:13.131332+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:06.406179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:06.406179+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:20.094337+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":731,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129488+0000","last_change":"2024-09-13T13:13:43.501368+0000","last_active":"2024-09-13T13:17:13.129488+0000","last_peered":"2024-09-13T13:17:13.129488+0000","last_clean":"2024-09-13T13:17:13.129488+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:17:13.129488+0000","last_undegraded":"2024-09-13T13:17:13.129488+0000","last_fullsized":"2024-09-13T13:17:13.129488+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:43.501206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:43.501206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":730,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.076659+0000","last_change":"2024-09-13T13:15:25.628234+0000","last_active":"2024-09-13T13:17:09.076659+0000","last_peered":"2024-09-13T13:17:09.076659+0000","last_clean":"2024-09-13T13:17:09.076659+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:17:09.076659+0000","last_undegraded":"2024-09-13T13:17:09.076659+0000","last_fullsized":"2024-09-13T13:17:09.076659+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.628156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.628156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":769,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129095+0000","last_change":"2024-09-13T13:17:02.211078+0000","last_active":"2024-09-13T13:17:13.129095+0000","last_peered":"2024-09-13T13:17:13.129095+0000","last_clean":"2024-09-13T13:17:13.129095+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:17:13.129095+0000","last_undegraded":"2024-09-13T13:17:13.129095+0000","last_fullsized":"2024-09-13T13:17:13.129095+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:02.211007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.211007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:19.487607+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":217,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134564+0000","last_change":"2024-09-13T13:15:03.348187+0000","last_active":"2024-09-13T13:17:13.134564+0000","last_peered":"2024-09-13T13:17:13.134564+0000","last_clean":"2024-09-13T13:17:13.134564+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:17:13.134564+0000","last_undegraded":"2024-09-13T13:17:13.134564+0000","last_fullsized":"2024-09-13T13:17:13.134564+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:15:03.348115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.348115+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":752,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.132534+0000","last_change":"2024-09-13T13:13:54.122282+0000","last_active":"2024-09-13T13:17:13.132534+0000","last_peered":"2024-09-13T13:17:13.132534+0000","last_clean":"2024-09-13T13:17:13.132534+0000","last_became_active":"2024-09-13T13:12:52.999970+0000","last_became_peered":"2024-09-13T13:12:52.999970+0000","last_unstale":"2024-09-13T13:17:13.132534+0000","last_undegraded":"2024-09-13T13:17:13.132534+0000","last_fullsized":"2024-09-13T13:17:13.132534+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.122222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.122222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:17.799545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":703,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129896+0000","last_change":"2024-09-13T13:13:02.818644+0000","last_active":"2024-09-13T13:17:13.129896+0000","last_peered":"2024-09-13T13:17:13.129896+0000","last_clean":"2024-09-13T13:17:13.129896+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:17:13.129896+0000","last_undegraded":"2024-09-13T13:17:13.129896+0000","last_fullsized":"2024-09-13T13:17:13.129896+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:02.818543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:02.818543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":216,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.133584+0000","last_change":"2024-09-13T13:17:07.444148+0000","last_active":"2024-09-13T13:17:13.133584+0000","last_peered":"2024-09-13T13:17:13.133584+0000","last_clean":"2024-09-13T13:17:13.133584+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:17:13.133584+0000","last_undegraded":"2024-09-13T13:17:13.133584+0000","last_fullsized":"2024-09-13T13:17:13.133584+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:07.443951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:07.443951+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:25.224293+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":756,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.078217+0000","last_change":"2024-09-13T13:15:52.658477+0000","last_active":"2024-09-13T13:17:09.078217+0000","last_peered":"2024-09-13T13:17:09.078217+0000","last_clean":"2024-09-13T13:17:09.078217+0000","last_became_active":"2024-09-13T13:12:53.008505+0000","last_became_peered":"2024-09-13T13:12:53.008505+0000","last_unstale":"2024-09-13T13:17:09.078217+0000","last_undegraded":"2024-09-13T13:17:09.078217+0000","last_fullsized":"2024-09-13T13:17:09.078217+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:52.658410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:52.658410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":574,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.095416+0000","last_change":"2024-09-13T13:16:12.089425+0000","last_active":"2024-09-13T13:17:09.095416+0000","last_peered":"2024-09-13T13:17:09.095416+0000","last_clean":"2024-09-13T13:17:09.095416+0000","last_became_active":"2024-09-13T13:16:12.073838+0000","last_became_peered":"2024-09-13T13:16:12.073838+0000","last_unstale":"2024-09-13T13:17:09.095416+0000","last_undegraded":"2024-09-13T13:17:09.095416+0000","last_fullsized":"2024-09-13T13:17:09.095416+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:12:40.125545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.125545+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:18.331517+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":758,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.096418+0000","last_change":"2024-09-13T13:16:54.229157+0000","last_active":"2024-09-13T13:17:09.096418+0000","last_peered":"2024-09-13T13:17:09.096418+0000","last_clean":"2024-09-13T13:17:09.096418+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:17:09.096418+0000","last_undegraded":"2024-09-13T13:17:09.096418+0000","last_fullsized":"2024-09-13T13:17:09.096418+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:54.229075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:54.229075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:13.235712+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":729,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.103089+0000","last_change":"2024-09-13T13:13:03.150524+0000","last_active":"2024-09-13T13:17:11.103089+0000","last_peered":"2024-09-13T13:17:11.103089+0000","last_clean":"2024-09-13T13:17:11.103089+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:17:11.103089+0000","last_undegraded":"2024-09-13T13:17:11.103089+0000","last_fullsized":"2024-09-13T13:17:11.103089+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:03.150442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.150442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:38.978974+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":226,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131100+0000","last_change":"2024-09-13T13:16:12.077830+0000","last_active":"2024-09-13T13:17:13.131100+0000","last_peered":"2024-09-13T13:17:13.131100+0000","last_clean":"2024-09-13T13:17:13.131100+0000","last_became_active":"2024-09-13T13:16:12.077185+0000","last_became_peered":"2024-09-13T13:16:12.077185+0000","last_unstale":"2024-09-13T13:17:13.131100+0000","last_undegraded":"2024-09-13T13:17:13.131100+0000","last_fullsized":"2024-09-13T13:17:13.131100+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:01.242234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:01.242234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:21.091214+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":795,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.099646+0000","last_change":"2024-09-13T13:13:41.164741+0000","last_active":"2024-09-13T13:17:11.099646+0000","last_peered":"2024-09-13T13:17:11.099646+0000","last_clean":"2024-09-13T13:17:11.099646+0000","last_became_active":"2024-09-13T13:12:52.953819+0000","last_became_peered":"2024-09-13T13:12:52.953819+0000","last_unstale":"2024-09-13T13:17:11.099646+0000","last_undegraded":"2024-09-13T13:17:11.099646+0000","last_fullsized":"2024-09-13T13:17:11.099646+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:13:41.164656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.164656+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:50.985657+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":220,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131202+0000","last_change":"2024-09-13T13:16:12.078175+0000","last_active":"2024-09-13T13:17:13.131202+0000","last_peered":"2024-09-13T13:17:13.131202+0000","last_clean":"2024-09-13T13:17:13.131202+0000","last_became_active":"2024-09-13T13:16:12.077603+0000","last_became_peered":"2024-09-13T13:16:12.077603+0000","last_unstale":"2024-09-13T13:17:13.131202+0000","last_undegraded":"2024-09-13T13:17:13.131202+0000","last_fullsized":"2024-09-13T13:17:13.131202+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:29.208576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:29.208576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":257,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.128853+0000","last_change":"2024-09-13T13:17:05.207522+0000","last_active":"2024-09-13T13:17:13.128853+0000","last_peered":"2024-09-13T13:17:13.128853+0000","last_clean":"2024-09-13T13:17:13.128853+0000","last_became_active":"2024-09-13T13:16:12.071603+0000","last_became_peered":"2024-09-13T13:16:12.071603+0000","last_unstale":"2024-09-13T13:17:13.128853+0000","last_undegraded":"2024-09-13T13:17:13.128853+0000","last_fullsized":"2024-09-13T13:17:13.128853+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:05.207445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.207445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:31.464107+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":759,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.094883+0000","last_change":"2024-09-13T13:17:01.162092+0000","last_active":"2024-09-13T13:17:09.094883+0000","last_peered":"2024-09-13T13:17:09.094883+0000","last_clean":"2024-09-13T13:17:09.094883+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:17:09.094883+0000","last_undegraded":"2024-09-13T13:17:09.094883+0000","last_fullsized":"2024-09-13T13:17:09.094883+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:17:01.162018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:01.162018+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:24.126509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":777,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.077265+0000","last_change":"2024-09-13T13:15:50.693943+0000","last_active":"2024-09-13T13:17:09.077265+0000","last_peered":"2024-09-13T13:17:09.077265+0000","last_clean":"2024-09-13T13:17:09.077265+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:17:09.077265+0000","last_undegraded":"2024-09-13T13:17:09.077265+0000","last_fullsized":"2024-09-13T13:17:09.077265+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:15:50.693753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.693753+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"306'12","reported_seq":912,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.108042+0000","last_change":"2024-09-13T13:14:21.530680+0000","last_active":"2024-09-13T13:17:11.108042+0000","last_peered":"2024-09-13T13:17:11.108042+0000","last_clean":"2024-09-13T13:17:11.108042+0000","last_became_active":"2024-09-13T13:12:52.947886+0000","last_became_peered":"2024-09-13T13:12:52.947886+0000","last_unstale":"2024-09-13T13:17:11.108042+0000","last_undegraded":"2024-09-13T13:17:11.108042+0000","last_fullsized":"2024-09-13T13:17:11.108042+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'12","last_scrub_stamp":"2024-09-13T13:14:21.530612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:21.530612+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:41.202009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"401'14295","reported_seq":14476,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.274384+0000","last_change":"2024-09-13T13:16:06.040203+0000","last_active":"2024-09-13T13:17:13.274384+0000","last_peered":"2024-09-13T13:17:13.274384+0000","last_clean":"2024-09-13T13:17:13.274384+0000","last_became_active":"2024-09-13T13:13:59.149215+0000","last_became_peered":"2024-09-13T13:13:59.149215+0000","last_unstale":"2024-09-13T13:17:13.274384+0000","last_undegraded":"2024-09-13T13:17:13.274384+0000","last_fullsized":"2024-09-13T13:17:13.274384+0000","mapping_epoch":334,"log_start":"372'9986","ondisk_log_start":"372'9986","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":3199,"log_size":4309,"log_dups_size":0,"ondisk_log_size":4309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110034944,"num_objects":1679,"num_object_clones":0,"num_object_copies":5037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14295,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":830,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.097119+0000","last_change":"2024-09-13T13:16:56.133395+0000","last_active":"2024-09-13T13:17:09.097119+0000","last_peered":"2024-09-13T13:17:09.097119+0000","last_clean":"2024-09-13T13:17:09.097119+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:17:09.097119+0000","last_undegraded":"2024-09-13T13:17:09.097119+0000","last_fullsized":"2024-09-13T13:17:09.097119+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:16:56.133314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:56.133314+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:01.815453+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":762,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.098355+0000","last_change":"2024-09-13T13:16:58.980151+0000","last_active":"2024-09-13T13:17:11.098355+0000","last_peered":"2024-09-13T13:17:11.098355+0000","last_clean":"2024-09-13T13:17:11.098355+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:17:11.098355+0000","last_undegraded":"2024-09-13T13:17:11.098355+0000","last_fullsized":"2024-09-13T13:17:11.098355+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:58.980098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.980098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:19.930402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":268,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.133691+0000","last_change":"2024-09-13T13:15:32.212185+0000","last_active":"2024-09-13T13:17:13.133691+0000","last_peered":"2024-09-13T13:17:13.133691+0000","last_clean":"2024-09-13T13:17:13.133691+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:17:13.133691+0000","last_undegraded":"2024-09-13T13:17:13.133691+0000","last_fullsized":"2024-09-13T13:17:13.133691+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:15:32.212101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.212101+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":736,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.128890+0000","last_change":"2024-09-13T13:13:57.546725+0000","last_active":"2024-09-13T13:17:13.128890+0000","last_peered":"2024-09-13T13:17:13.128890+0000","last_clean":"2024-09-13T13:17:13.128890+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:17:13.128890+0000","last_undegraded":"2024-09-13T13:17:13.128890+0000","last_fullsized":"2024-09-13T13:17:13.128890+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.546671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.546671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"401'14289","reported_seq":22766,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.273829+0000","last_change":"2024-09-13T13:16:50.311735+0000","last_active":"2024-09-13T13:17:13.273829+0000","last_peered":"2024-09-13T13:17:13.273829+0000","last_clean":"2024-09-13T13:17:13.273829+0000","last_became_active":"2024-09-13T13:16:50.276697+0000","last_became_peered":"2024-09-13T13:16:50.276697+0000","last_unstale":"2024-09-13T13:17:13.273829+0000","last_undegraded":"2024-09-13T13:17:13.273829+0000","last_fullsized":"2024-09-13T13:17:13.273829+0000","mapping_epoch":389,"log_start":"372'9910","ondisk_log_start":"372'9910","created":334,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111214592,"num_objects":1697,"num_object_clones":0,"num_object_copies":5091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14289,"num_write_kb":511552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4157,"num_bytes_recovered":272433152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":769,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.098124+0000","last_change":"2024-09-13T13:16:05.289274+0000","last_active":"2024-09-13T13:17:11.098124+0000","last_peered":"2024-09-13T13:17:11.098124+0000","last_clean":"2024-09-13T13:17:11.098124+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:17:11.098124+0000","last_undegraded":"2024-09-13T13:17:11.098124+0000","last_fullsized":"2024-09-13T13:17:11.098124+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:16:05.289187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:05.289187+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:35.033329+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":736,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.103461+0000","last_change":"2024-09-13T13:16:26.098415+0000","last_active":"2024-09-13T13:17:11.103461+0000","last_peered":"2024-09-13T13:17:11.103461+0000","last_clean":"2024-09-13T13:17:11.103461+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:17:11.103461+0000","last_undegraded":"2024-09-13T13:17:11.103461+0000","last_fullsized":"2024-09-13T13:17:11.103461+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:16:26.098365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:26.098365+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:54.098482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":212,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131737+0000","last_change":"2024-09-13T13:15:00.323883+0000","last_active":"2024-09-13T13:17:13.131737+0000","last_peered":"2024-09-13T13:17:13.131737+0000","last_clean":"2024-09-13T13:17:13.131737+0000","last_became_active":"2024-09-13T13:12:53.006199+0000","last_became_peered":"2024-09-13T13:12:53.006199+0000","last_unstale":"2024-09-13T13:17:13.131737+0000","last_undegraded":"2024-09-13T13:17:13.131737+0000","last_fullsized":"2024-09-13T13:17:13.131737+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.323784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.323784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":227,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131415+0000","last_change":"2024-09-13T13:16:12.075699+0000","last_active":"2024-09-13T13:17:13.131415+0000","last_peered":"2024-09-13T13:17:13.131415+0000","last_clean":"2024-09-13T13:17:13.131415+0000","last_became_active":"2024-09-13T13:16:12.074280+0000","last_became_peered":"2024-09-13T13:16:12.074280+0000","last_unstale":"2024-09-13T13:17:13.131415+0000","last_undegraded":"2024-09-13T13:17:13.131415+0000","last_fullsized":"2024-09-13T13:17:13.131415+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:56.333778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:56.333778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"400'14078","reported_seq":19909,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.315058+0000","last_change":"2024-09-13T13:17:03.085482+0000","last_active":"2024-09-13T13:17:11.315058+0000","last_peered":"2024-09-13T13:17:11.315058+0000","last_clean":"2024-09-13T13:17:11.315058+0000","last_became_active":"2024-09-13T13:17:03.059883+0000","last_became_peered":"2024-09-13T13:17:03.059883+0000","last_unstale":"2024-09-13T13:17:11.315058+0000","last_undegraded":"2024-09-13T13:17:11.315058+0000","last_fullsized":"2024-09-13T13:17:11.315058+0000","mapping_epoch":396,"log_start":"368'8868","ondisk_log_start":"368'8868","created":334,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5210,"log_dups_size":0,"ondisk_log_size":5210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":1856,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14078,"num_write_kb":509888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2833,"num_bytes_recovered":185663488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":228,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131823+0000","last_change":"2024-09-13T13:13:50.716312+0000","last_active":"2024-09-13T13:17:13.131823+0000","last_peered":"2024-09-13T13:17:13.131823+0000","last_clean":"2024-09-13T13:17:13.131823+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:17:13.131823+0000","last_undegraded":"2024-09-13T13:17:13.131823+0000","last_fullsized":"2024-09-13T13:17:13.131823+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:13:50.716243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.716243+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":742,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.077558+0000","last_change":"2024-09-13T13:17:02.704294+0000","last_active":"2024-09-13T13:17:09.077558+0000","last_peered":"2024-09-13T13:17:09.077558+0000","last_clean":"2024-09-13T13:17:09.077558+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:17:09.077558+0000","last_undegraded":"2024-09-13T13:17:09.077558+0000","last_fullsized":"2024-09-13T13:17:09.077558+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:17:02.704233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.704233+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:20.770762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":956,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.098033+0000","last_change":"2024-09-13T13:16:24.110415+0000","last_active":"2024-09-13T13:17:11.098033+0000","last_peered":"2024-09-13T13:17:11.098033+0000","last_clean":"2024-09-13T13:17:11.098033+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:17:11.098033+0000","last_undegraded":"2024-09-13T13:17:11.098033+0000","last_fullsized":"2024-09-13T13:17:11.098033+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:16:24.110369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:24.110369+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:42.263184+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":367,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.104569+0000","last_change":"2024-09-13T13:16:26.258774+0000","last_active":"2024-09-13T13:17:11.104569+0000","last_peered":"2024-09-13T13:17:11.104569+0000","last_clean":"2024-09-13T13:17:11.104569+0000","last_became_active":"2024-09-13T13:16:12.071629+0000","last_became_peered":"2024-09-13T13:16:12.071629+0000","last_unstale":"2024-09-13T13:17:11.104569+0000","last_undegraded":"2024-09-13T13:17:11.104569+0000","last_fullsized":"2024-09-13T13:17:11.104569+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:26.258619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:26.258619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:34.080748+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"401'14143","reported_seq":14288,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.269637+0000","last_change":"2024-09-13T13:13:59.149331+0000","last_active":"2024-09-13T13:17:13.269637+0000","last_peered":"2024-09-13T13:17:13.269637+0000","last_clean":"2024-09-13T13:17:13.269637+0000","last_became_active":"2024-09-13T13:13:59.149046+0000","last_became_peered":"2024-09-13T13:13:59.149046+0000","last_unstale":"2024-09-13T13:17:13.269637+0000","last_undegraded":"2024-09-13T13:17:13.269637+0000","last_fullsized":"2024-09-13T13:17:13.269637+0000","mapping_epoch":334,"log_start":"372'9842","ondisk_log_start":"372'9842","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4301,"log_dups_size":0,"ondisk_log_size":4301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104792064,"num_objects":1599,"num_object_clones":0,"num_object_copies":4797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14143,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":793,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.096641+0000","last_change":"2024-09-13T13:12:53.016127+0000","last_active":"2024-09-13T13:17:09.096641+0000","last_peered":"2024-09-13T13:17:09.096641+0000","last_clean":"2024-09-13T13:17:09.096641+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:17:09.096641+0000","last_undegraded":"2024-09-13T13:17:09.096641+0000","last_fullsized":"2024-09-13T13:17:09.096641+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:41.316459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":771,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129517+0000","last_change":"2024-09-13T13:16:12.067616+0000","last_active":"2024-09-13T13:17:13.129517+0000","last_peered":"2024-09-13T13:17:13.129517+0000","last_clean":"2024-09-13T13:17:13.129517+0000","last_became_active":"2024-09-13T13:16:12.067157+0000","last_became_peered":"2024-09-13T13:16:12.067157+0000","last_unstale":"2024-09-13T13:17:13.129517+0000","last_undegraded":"2024-09-13T13:17:13.129517+0000","last_fullsized":"2024-09-13T13:17:13.129517+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:04.755828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:04.755828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:15.005053+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":935,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.077972+0000","last_change":"2024-09-13T13:15:51.663010+0000","last_active":"2024-09-13T13:17:09.077972+0000","last_peered":"2024-09-13T13:17:09.077972+0000","last_clean":"2024-09-13T13:17:09.077972+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:17:09.077972+0000","last_undegraded":"2024-09-13T13:17:09.077972+0000","last_fullsized":"2024-09-13T13:17:09.077972+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:15:51.662657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.662657+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":754,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.077077+0000","last_change":"2024-09-13T13:16:12.077285+0000","last_active":"2024-09-13T13:17:09.077077+0000","last_peered":"2024-09-13T13:17:09.077077+0000","last_clean":"2024-09-13T13:17:09.077077+0000","last_became_active":"2024-09-13T13:16:12.072760+0000","last_became_peered":"2024-09-13T13:16:12.072760+0000","last_unstale":"2024-09-13T13:17:09.077077+0000","last_undegraded":"2024-09-13T13:17:09.077077+0000","last_fullsized":"2024-09-13T13:17:09.077077+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:26.667455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:26.667455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"400'14305","reported_seq":14583,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.345639+0000","last_change":"2024-09-13T13:16:31.945928+0000","last_active":"2024-09-13T13:17:11.345639+0000","last_peered":"2024-09-13T13:17:11.345639+0000","last_clean":"2024-09-13T13:17:11.345639+0000","last_became_active":"2024-09-13T13:13:58.965950+0000","last_became_peered":"2024-09-13T13:13:58.965950+0000","last_unstale":"2024-09-13T13:17:11.345639+0000","last_undegraded":"2024-09-13T13:17:11.345639+0000","last_fullsized":"2024-09-13T13:17:11.345639+0000","mapping_epoch":334,"log_start":"368'9068","ondisk_log_start":"368'9068","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'11986","last_scrub_stamp":"2024-09-13T13:16:31.945684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:16:31.945684+0000","objects_scrubbed":4166,"log_size":5237,"log_dups_size":0,"ondisk_log_size":5237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:51.306557+0000","scrub_duration":1677,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117702680,"num_objects":1797,"num_object_clones":0,"num_object_copies":5391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14305,"num_write_kb":515201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":798,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.078099+0000","last_change":"2024-09-13T13:17:05.726065+0000","last_active":"2024-09-13T13:17:09.078099+0000","last_peered":"2024-09-13T13:17:09.078099+0000","last_clean":"2024-09-13T13:17:09.078099+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:17:09.078099+0000","last_undegraded":"2024-09-13T13:17:09.078099+0000","last_fullsized":"2024-09-13T13:17:09.078099+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:17:05.726003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.726003+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:30.917098+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":736,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.095629+0000","last_change":"2024-09-13T13:16:36.283708+0000","last_active":"2024-09-13T13:17:09.095629+0000","last_peered":"2024-09-13T13:17:09.095629+0000","last_clean":"2024-09-13T13:17:09.095629+0000","last_became_active":"2024-09-13T13:16:12.074272+0000","last_became_peered":"2024-09-13T13:16:12.074272+0000","last_unstale":"2024-09-13T13:17:09.095629+0000","last_undegraded":"2024-09-13T13:17:09.095629+0000","last_fullsized":"2024-09-13T13:17:09.095629+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:36.283624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:36.283624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:05.724739+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":707,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.104612+0000","last_change":"2024-09-13T13:16:24.285416+0000","last_active":"2024-09-13T13:17:11.104612+0000","last_peered":"2024-09-13T13:17:11.104612+0000","last_clean":"2024-09-13T13:17:11.104612+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:17:11.104612+0000","last_undegraded":"2024-09-13T13:17:11.104612+0000","last_fullsized":"2024-09-13T13:17:11.104612+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:24.285346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:24.285346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:43.854597+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":743,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.094649+0000","last_change":"2024-09-13T13:16:29.167358+0000","last_active":"2024-09-13T13:17:09.094649+0000","last_peered":"2024-09-13T13:17:09.094649+0000","last_clean":"2024-09-13T13:17:09.094649+0000","last_became_active":"2024-09-13T13:16:12.067708+0000","last_became_peered":"2024-09-13T13:16:12.067708+0000","last_unstale":"2024-09-13T13:17:09.094649+0000","last_undegraded":"2024-09-13T13:17:09.094649+0000","last_fullsized":"2024-09-13T13:17:09.094649+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:29.167280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:29.167280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:44.607802+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"400'14130","reported_seq":14271,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.335384+0000","last_change":"2024-09-13T13:13:58.966631+0000","last_active":"2024-09-13T13:17:11.335384+0000","last_peered":"2024-09-13T13:17:11.335384+0000","last_clean":"2024-09-13T13:17:11.335384+0000","last_became_active":"2024-09-13T13:13:58.965789+0000","last_became_peered":"2024-09-13T13:13:58.965789+0000","last_unstale":"2024-09-13T13:17:11.335384+0000","last_undegraded":"2024-09-13T13:17:11.335384+0000","last_fullsized":"2024-09-13T13:17:11.335384+0000","mapping_epoch":334,"log_start":"368'8863","ondisk_log_start":"368'8863","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5267,"log_dups_size":0,"ondisk_log_size":5267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:19.436994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116260864,"num_objects":1774,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14130,"num_write_kb":508928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"377'41","reported_seq":1030,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.095502+0000","last_change":"2024-09-13T13:16:58.213866+0000","last_active":"2024-09-13T13:17:09.095502+0000","last_peered":"2024-09-13T13:17:09.095502+0000","last_clean":"2024-09-13T13:17:09.095502+0000","last_became_active":"2024-09-13T13:16:12.072827+0000","last_became_peered":"2024-09-13T13:16:12.072827+0000","last_unstale":"2024-09-13T13:17:09.095502+0000","last_undegraded":"2024-09-13T13:17:09.095502+0000","last_fullsized":"2024-09-13T13:17:09.095502+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'41","last_scrub_stamp":"2024-09-13T13:16:58.213805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.213805+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:10.010285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":220,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134653+0000","last_change":"2024-09-13T13:14:14.899463+0000","last_active":"2024-09-13T13:17:13.134653+0000","last_peered":"2024-09-13T13:17:13.134653+0000","last_clean":"2024-09-13T13:17:13.134653+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:17:13.134653+0000","last_undegraded":"2024-09-13T13:17:13.134653+0000","last_fullsized":"2024-09-13T13:17:13.134653+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:14.899378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:14.899378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":751,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.130075+0000","last_change":"2024-09-13T13:15:50.618476+0000","last_active":"2024-09-13T13:17:13.130075+0000","last_peered":"2024-09-13T13:17:13.130075+0000","last_clean":"2024-09-13T13:17:13.130075+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:17:13.130075+0000","last_undegraded":"2024-09-13T13:17:13.130075+0000","last_fullsized":"2024-09-13T13:17:13.130075+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:15:50.618426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.618426+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":749,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.077954+0000","last_change":"2024-09-13T13:14:59.670853+0000","last_active":"2024-09-13T13:17:09.077954+0000","last_peered":"2024-09-13T13:17:09.077954+0000","last_clean":"2024-09-13T13:17:09.077954+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:17:09.077954+0000","last_undegraded":"2024-09-13T13:17:09.077954+0000","last_fullsized":"2024-09-13T13:17:09.077954+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.670779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.670779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":748,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.099231+0000","last_change":"2024-09-13T13:17:05.016876+0000","last_active":"2024-09-13T13:17:11.099231+0000","last_peered":"2024-09-13T13:17:11.099231+0000","last_clean":"2024-09-13T13:17:11.099231+0000","last_became_active":"2024-09-13T13:12:52.948742+0000","last_became_peered":"2024-09-13T13:12:52.948742+0000","last_unstale":"2024-09-13T13:17:11.099231+0000","last_undegraded":"2024-09-13T13:17:11.099231+0000","last_fullsized":"2024-09-13T13:17:11.099231+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:05.016806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.016806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:07.820409+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"401'14141","reported_seq":22383,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.492606+0000","last_change":"2024-09-13T13:16:48.283334+0000","last_active":"2024-09-13T13:17:13.492606+0000","last_peered":"2024-09-13T13:17:13.492606+0000","last_clean":"2024-09-13T13:17:13.492606+0000","last_became_active":"2024-09-13T13:16:48.248459+0000","last_became_peered":"2024-09-13T13:16:48.248459+0000","last_unstale":"2024-09-13T13:17:13.492606+0000","last_undegraded":"2024-09-13T13:17:13.492606+0000","last_fullsized":"2024-09-13T13:17:13.492606+0000","mapping_epoch":387,"log_start":"371'9094","ondisk_log_start":"371'9094","created":334,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5047,"log_dups_size":0,"ondisk_log_size":5047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104398848,"num_objects":1593,"num_object_clones":0,"num_object_copies":4779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14141,"num_write_kb":503488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4040,"num_bytes_recovered":264765440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":758,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.097216+0000","last_change":"2024-09-13T13:15:33.299837+0000","last_active":"2024-09-13T13:17:09.097216+0000","last_peered":"2024-09-13T13:17:09.097216+0000","last_clean":"2024-09-13T13:17:09.097216+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:17:09.097216+0000","last_undegraded":"2024-09-13T13:17:09.097216+0000","last_fullsized":"2024-09-13T13:17:09.097216+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'17","last_scrub_stamp":"2024-09-13T13:15:33.299746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.299746+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":756,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.094980+0000","last_change":"2024-09-13T13:16:12.072959+0000","last_active":"2024-09-13T13:17:09.094980+0000","last_peered":"2024-09-13T13:17:09.094980+0000","last_clean":"2024-09-13T13:17:09.094980+0000","last_became_active":"2024-09-13T13:16:12.072367+0000","last_became_peered":"2024-09-13T13:16:12.072367+0000","last_unstale":"2024-09-13T13:17:09.094980+0000","last_undegraded":"2024-09-13T13:17:09.094980+0000","last_fullsized":"2024-09-13T13:17:09.094980+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:17.676339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:17.676339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:21.245559+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":75,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.098408+0000","last_change":"2024-09-13T13:16:12.092137+0000","last_active":"2024-09-13T13:17:11.098408+0000","last_peered":"2024-09-13T13:17:11.098408+0000","last_clean":"2024-09-13T13:17:11.098408+0000","last_became_active":"2024-09-13T13:16:12.074083+0000","last_became_peered":"2024-09-13T13:16:12.074083+0000","last_unstale":"2024-09-13T13:17:11.098408+0000","last_undegraded":"2024-09-13T13:17:11.098408+0000","last_fullsized":"2024-09-13T13:17:11.098408+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:13:14.560555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:14.560555+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:13.172641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":62,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129779+0000","last_change":"2024-09-13T13:16:12.070341+0000","last_active":"2024-09-13T13:17:13.129779+0000","last_peered":"2024-09-13T13:17:13.129779+0000","last_clean":"2024-09-13T13:17:13.129779+0000","last_became_active":"2024-09-13T13:16:12.069747+0000","last_became_peered":"2024-09-13T13:16:12.069747+0000","last_unstale":"2024-09-13T13:17:13.129779+0000","last_undegraded":"2024-09-13T13:17:13.129779+0000","last_fullsized":"2024-09-13T13:17:13.129779+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:59.809055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:59.809055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:24.864450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":352,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.105815+0000","last_change":"2024-09-13T13:12:41.332308+0000","last_active":"2024-09-13T13:17:11.105815+0000","last_peered":"2024-09-13T13:17:11.105815+0000","last_clean":"2024-09-13T13:17:11.105815+0000","last_became_active":"2024-09-13T13:12:40.120242+0000","last_became_peered":"2024-09-13T13:12:40.120242+0000","last_unstale":"2024-09-13T13:17:11.105815+0000","last_undegraded":"2024-09-13T13:17:11.105815+0000","last_fullsized":"2024-09-13T13:17:11.105815+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:41.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:41.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:14.436447+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"399'14209","reported_seq":20396,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.662305+0000","last_change":"2024-09-13T13:17:08.294281+0000","last_active":"2024-09-13T13:17:09.662305+0000","last_peered":"2024-09-13T13:17:09.662305+0000","last_clean":"2024-09-13T13:17:09.662305+0000","last_became_active":"2024-09-13T13:16:59.041685+0000","last_became_peered":"2024-09-13T13:16:59.041685+0000","last_unstale":"2024-09-13T13:17:09.662305+0000","last_undegraded":"2024-09-13T13:17:09.662305+0000","last_fullsized":"2024-09-13T13:17:09.662305+0000","mapping_epoch":393,"log_start":"371'9596","ondisk_log_start":"371'9596","created":334,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14102","last_scrub_stamp":"2024-09-13T13:17:08.294178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:17:08.294178+0000","objects_scrubbed":2017,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:13.042889+0000","scrub_duration":632,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123535360,"num_objects":1885,"num_object_clones":0,"num_object_copies":5655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14209,"num_write_kb":515008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2998,"num_bytes_recovered":196476928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":788,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.077362+0000","last_change":"2024-09-13T13:16:30.819216+0000","last_active":"2024-09-13T13:17:09.077362+0000","last_peered":"2024-09-13T13:17:09.077362+0000","last_clean":"2024-09-13T13:17:09.077362+0000","last_became_active":"2024-09-13T13:16:12.074949+0000","last_became_peered":"2024-09-13T13:16:12.074949+0000","last_unstale":"2024-09-13T13:17:09.077362+0000","last_undegraded":"2024-09-13T13:17:09.077362+0000","last_fullsized":"2024-09-13T13:17:09.077362+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:16:30.819129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:30.819129+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:50.347431+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":749,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.076952+0000","last_change":"2024-09-13T13:15:30.590673+0000","last_active":"2024-09-13T13:17:09.076952+0000","last_peered":"2024-09-13T13:17:09.076952+0000","last_clean":"2024-09-13T13:17:09.076952+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:17:09.076952+0000","last_undegraded":"2024-09-13T13:17:09.076952+0000","last_fullsized":"2024-09-13T13:17:09.076952+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:30.590466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:30.590466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":541,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.094830+0000","last_change":"2024-09-13T13:14:48.128237+0000","last_active":"2024-09-13T13:17:09.094830+0000","last_peered":"2024-09-13T13:17:09.094830+0000","last_clean":"2024-09-13T13:17:09.094830+0000","last_became_active":"2024-09-13T13:12:39.947940+0000","last_became_peered":"2024-09-13T13:12:39.947940+0000","last_unstale":"2024-09-13T13:17:09.094830+0000","last_undegraded":"2024-09-13T13:17:09.094830+0000","last_fullsized":"2024-09-13T13:17:09.094830+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:14:48.128156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:48.128156+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":275,"num_read_kb":270,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":724,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.130080+0000","last_change":"2024-09-13T13:15:32.318497+0000","last_active":"2024-09-13T13:17:13.130080+0000","last_peered":"2024-09-13T13:17:13.130080+0000","last_clean":"2024-09-13T13:17:13.130080+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:17:13.130080+0000","last_undegraded":"2024-09-13T13:17:13.130080+0000","last_fullsized":"2024-09-13T13:17:13.130080+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:32.318413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.318413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":725,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.095181+0000","last_change":"2024-09-13T13:10:32.720742+0000","last_active":"2024-09-13T13:17:09.095181+0000","last_peered":"2024-09-13T13:17:09.095181+0000","last_clean":"2024-09-13T13:17:09.095181+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:17:09.095181+0000","last_undegraded":"2024-09-13T13:17:09.095181+0000","last_fullsized":"2024-09-13T13:17:09.095181+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:39.284284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"400'14133","reported_seq":14277,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.330177+0000","last_change":"2024-09-13T13:13:58.967163+0000","last_active":"2024-09-13T13:17:11.330177+0000","last_peered":"2024-09-13T13:17:11.330177+0000","last_clean":"2024-09-13T13:17:11.330177+0000","last_became_active":"2024-09-13T13:13:58.965962+0000","last_became_peered":"2024-09-13T13:13:58.965962+0000","last_unstale":"2024-09-13T13:17:11.330177+0000","last_undegraded":"2024-09-13T13:17:11.330177+0000","last_fullsized":"2024-09-13T13:17:11.330177+0000","mapping_epoch":334,"log_start":"368'8915","ondisk_log_start":"368'8915","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5218,"log_dups_size":0,"ondisk_log_size":5218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:16.445030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117899264,"num_objects":1799,"num_object_clones":0,"num_object_copies":5397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14133,"num_write_kb":509824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"306'32","reported_seq":870,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.106168+0000","last_change":"2024-09-13T13:13:01.352416+0000","last_active":"2024-09-13T13:17:11.106168+0000","last_peered":"2024-09-13T13:17:11.106168+0000","last_clean":"2024-09-13T13:17:11.106168+0000","last_became_active":"2024-09-13T13:12:52.999924+0000","last_became_peered":"2024-09-13T13:12:52.999924+0000","last_unstale":"2024-09-13T13:17:11.106168+0000","last_undegraded":"2024-09-13T13:17:11.106168+0000","last_fullsized":"2024-09-13T13:17:11.106168+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:13:01.352337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.352337+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":749,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.098658+0000","last_change":"2024-09-13T13:16:53.951208+0000","last_active":"2024-09-13T13:17:11.098658+0000","last_peered":"2024-09-13T13:17:11.098658+0000","last_clean":"2024-09-13T13:17:11.098658+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:17:11.098658+0000","last_undegraded":"2024-09-13T13:17:11.098658+0000","last_fullsized":"2024-09-13T13:17:11.098658+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:53.951135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:53.951135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:55.102860+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":314,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.077331+0000","last_change":"2024-09-13T13:15:54.707061+0000","last_active":"2024-09-13T13:17:09.077331+0000","last_peered":"2024-09-13T13:17:09.077331+0000","last_clean":"2024-09-13T13:17:09.077331+0000","last_became_active":"2024-09-13T13:12:39.948765+0000","last_became_peered":"2024-09-13T13:12:39.948765+0000","last_unstale":"2024-09-13T13:17:09.077331+0000","last_undegraded":"2024-09-13T13:17:09.077331+0000","last_fullsized":"2024-09-13T13:17:09.077331+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:10.766426+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":63,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129012+0000","last_change":"2024-09-13T13:16:12.074017+0000","last_active":"2024-09-13T13:17:13.129012+0000","last_peered":"2024-09-13T13:17:13.129012+0000","last_clean":"2024-09-13T13:17:13.129012+0000","last_became_active":"2024-09-13T13:16:12.073663+0000","last_became_peered":"2024-09-13T13:16:12.073663+0000","last_unstale":"2024-09-13T13:17:13.129012+0000","last_undegraded":"2024-09-13T13:17:13.129012+0000","last_fullsized":"2024-09-13T13:17:13.129012+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:33.708995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.708995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":766,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129931+0000","last_change":"2024-09-13T13:15:57.689044+0000","last_active":"2024-09-13T13:17:13.129931+0000","last_peered":"2024-09-13T13:17:13.129931+0000","last_clean":"2024-09-13T13:17:13.129931+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:17:13.129931+0000","last_undegraded":"2024-09-13T13:17:13.129931+0000","last_fullsized":"2024-09-13T13:17:13.129931+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:57.688955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:57.688955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"400'14396","reported_seq":14659,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.280635+0000","last_change":"2024-09-13T13:16:33.808007+0000","last_active":"2024-09-13T13:17:11.280635+0000","last_peered":"2024-09-13T13:17:11.280635+0000","last_clean":"2024-09-13T13:17:11.280635+0000","last_became_active":"2024-09-13T13:13:58.965566+0000","last_became_peered":"2024-09-13T13:13:58.965566+0000","last_unstale":"2024-09-13T13:17:11.280635+0000","last_undegraded":"2024-09-13T13:17:11.280635+0000","last_fullsized":"2024-09-13T13:17:11.280635+0000","mapping_epoch":334,"log_start":"369'9221","ondisk_log_start":"369'9221","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'12107","last_scrub_stamp":"2024-09-13T13:16:33.807809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:16:33.807809+0000","objects_scrubbed":4136,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:55.361775+0000","scrub_duration":1617,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":1792,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14396,"num_write_kb":518016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":231,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.133478+0000","last_change":"2024-09-13T13:14:59.354034+0000","last_active":"2024-09-13T13:17:13.133478+0000","last_peered":"2024-09-13T13:17:13.133478+0000","last_clean":"2024-09-13T13:17:13.133478+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:17:13.133478+0000","last_undegraded":"2024-09-13T13:17:13.133478+0000","last_fullsized":"2024-09-13T13:17:13.133478+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:14:59.353691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.353691+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":219,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131720+0000","last_change":"2024-09-13T13:16:12.074151+0000","last_active":"2024-09-13T13:17:13.131720+0000","last_peered":"2024-09-13T13:17:13.131720+0000","last_clean":"2024-09-13T13:17:13.131720+0000","last_became_active":"2024-09-13T13:16:12.073474+0000","last_became_peered":"2024-09-13T13:16:12.073474+0000","last_unstale":"2024-09-13T13:17:13.131720+0000","last_undegraded":"2024-09-13T13:17:13.131720+0000","last_fullsized":"2024-09-13T13:17:13.131720+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:26.343754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:26.343754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":781,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129777+0000","last_change":"2024-09-13T13:15:55.592241+0000","last_active":"2024-09-13T13:17:13.129777+0000","last_peered":"2024-09-13T13:17:13.129777+0000","last_clean":"2024-09-13T13:17:13.129777+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:17:13.129777+0000","last_undegraded":"2024-09-13T13:17:13.129777+0000","last_fullsized":"2024-09-13T13:17:13.129777+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:55.592172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:55.592172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:23.172298+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":729,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.099394+0000","last_change":"2024-09-13T13:14:23.655250+0000","last_active":"2024-09-13T13:17:11.099394+0000","last_peered":"2024-09-13T13:17:11.099394+0000","last_clean":"2024-09-13T13:17:11.099394+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:17:11.099394+0000","last_undegraded":"2024-09-13T13:17:11.099394+0000","last_fullsized":"2024-09-13T13:17:11.099394+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:23.655072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:23.655072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:48.966053+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"401'14350","reported_seq":20184,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.491831+0000","last_change":"2024-09-13T13:17:02.078726+0000","last_active":"2024-09-13T13:17:13.491831+0000","last_peered":"2024-09-13T13:17:13.491831+0000","last_clean":"2024-09-13T13:17:13.491831+0000","last_became_active":"2024-09-13T13:17:02.055599+0000","last_became_peered":"2024-09-13T13:17:02.055599+0000","last_unstale":"2024-09-13T13:17:13.491831+0000","last_undegraded":"2024-09-13T13:17:13.491831+0000","last_fullsized":"2024-09-13T13:17:13.491831+0000","mapping_epoch":395,"log_start":"371'9284","ondisk_log_start":"371'9284","created":334,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7793","last_scrub_stamp":"2024-09-13T13:15:31.408001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:31.408001+0000","objects_scrubbed":7714,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103284736,"num_objects":1576,"num_object_clones":0,"num_object_copies":4728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14350,"num_write_kb":509632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2792,"num_bytes_recovered":182976512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":262,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134869+0000","last_change":"2024-09-13T13:16:12.090140+0000","last_active":"2024-09-13T13:17:13.134869+0000","last_peered":"2024-09-13T13:17:13.134869+0000","last_clean":"2024-09-13T13:17:13.134869+0000","last_became_active":"2024-09-13T13:16:12.075614+0000","last_became_peered":"2024-09-13T13:16:12.075614+0000","last_unstale":"2024-09-13T13:17:13.134869+0000","last_undegraded":"2024-09-13T13:17:13.134869+0000","last_fullsized":"2024-09-13T13:17:13.134869+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:14:20.221286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:20.221286+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":761,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.107096+0000","last_change":"2024-09-13T13:16:37.263033+0000","last_active":"2024-09-13T13:17:11.107096+0000","last_peered":"2024-09-13T13:17:11.107096+0000","last_clean":"2024-09-13T13:17:11.107096+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:17:11.107096+0000","last_undegraded":"2024-09-13T13:17:11.107096+0000","last_fullsized":"2024-09-13T13:17:11.107096+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:37.262977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:37.262977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:56.631589+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":751,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.095274+0000","last_change":"2024-09-13T13:17:05.238854+0000","last_active":"2024-09-13T13:17:09.095274+0000","last_peered":"2024-09-13T13:17:09.095274+0000","last_clean":"2024-09-13T13:17:09.095274+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:17:09.095274+0000","last_undegraded":"2024-09-13T13:17:09.095274+0000","last_fullsized":"2024-09-13T13:17:09.095274+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:05.238722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.238722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:12.615074+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":592,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.102985+0000","last_change":"2024-09-13T13:16:35.064359+0000","last_active":"2024-09-13T13:17:11.102985+0000","last_peered":"2024-09-13T13:17:11.102985+0000","last_clean":"2024-09-13T13:17:11.102985+0000","last_became_active":"2024-09-13T13:16:12.067474+0000","last_became_peered":"2024-09-13T13:16:12.067474+0000","last_unstale":"2024-09-13T13:17:11.102985+0000","last_undegraded":"2024-09-13T13:17:11.102985+0000","last_fullsized":"2024-09-13T13:17:11.102985+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:35.064188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:35.064188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:44.945474+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"399'14131","reported_seq":14323,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.094534+0000","last_change":"2024-09-13T13:16:29.564422+0000","last_active":"2024-09-13T13:17:09.094534+0000","last_peered":"2024-09-13T13:17:09.094534+0000","last_clean":"2024-09-13T13:17:09.094534+0000","last_became_active":"2024-09-13T13:13:58.963064+0000","last_became_peered":"2024-09-13T13:13:58.963064+0000","last_unstale":"2024-09-13T13:17:09.094534+0000","last_undegraded":"2024-09-13T13:17:09.094534+0000","last_fullsized":"2024-09-13T13:17:09.094534+0000","mapping_epoch":334,"log_start":"371'9514","ondisk_log_start":"371'9514","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'11857","last_scrub_stamp":"2024-09-13T13:16:29.563999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:16:29.563999+0000","objects_scrubbed":4241,"log_size":4617,"log_dups_size":0,"ondisk_log_size":4617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:58.196946+0000","scrub_duration":1422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124977152,"num_objects":1907,"num_object_clones":0,"num_object_copies":5721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14131,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":223,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131347+0000","last_change":"2024-09-13T13:17:02.421924+0000","last_active":"2024-09-13T13:17:13.131347+0000","last_peered":"2024-09-13T13:17:13.131347+0000","last_clean":"2024-09-13T13:17:13.131347+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:17:13.131347+0000","last_undegraded":"2024-09-13T13:17:13.131347+0000","last_fullsized":"2024-09-13T13:17:13.131347+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.421825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.421825+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:23.577428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":736,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.099188+0000","last_change":"2024-09-13T13:12:53.003530+0000","last_active":"2024-09-13T13:17:11.099188+0000","last_peered":"2024-09-13T13:17:11.099188+0000","last_clean":"2024-09-13T13:17:11.099188+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:17:11.099188+0000","last_undegraded":"2024-09-13T13:17:11.099188+0000","last_fullsized":"2024-09-13T13:17:11.099188+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.996346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.996346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:11.226277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"377'9","reported_seq":104,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129001+0000","last_change":"2024-09-13T13:16:12.089906+0000","last_active":"2024-09-13T13:17:13.129001+0000","last_peered":"2024-09-13T13:17:13.129001+0000","last_clean":"2024-09-13T13:17:13.129001+0000","last_became_active":"2024-09-13T13:16:12.079150+0000","last_became_peered":"2024-09-13T13:16:12.079150+0000","last_unstale":"2024-09-13T13:17:13.129001+0000","last_undegraded":"2024-09-13T13:17:13.129001+0000","last_fullsized":"2024-09-13T13:17:13.129001+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:12:36.113709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:36.113709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":218,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.133379+0000","last_change":"2024-09-13T13:16:12.076027+0000","last_active":"2024-09-13T13:17:13.133379+0000","last_peered":"2024-09-13T13:17:13.133379+0000","last_clean":"2024-09-13T13:17:13.133379+0000","last_became_active":"2024-09-13T13:16:12.075615+0000","last_became_peered":"2024-09-13T13:16:12.075615+0000","last_unstale":"2024-09-13T13:17:13.133379+0000","last_undegraded":"2024-09-13T13:17:13.133379+0000","last_fullsized":"2024-09-13T13:17:13.133379+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.797639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.797639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"399'13968","reported_seq":20055,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.600474+0000","last_change":"2024-09-13T13:16:59.076235+0000","last_active":"2024-09-13T13:17:09.600474+0000","last_peered":"2024-09-13T13:17:09.600474+0000","last_clean":"2024-09-13T13:17:09.600474+0000","last_became_active":"2024-09-13T13:16:59.041701+0000","last_became_peered":"2024-09-13T13:16:59.041701+0000","last_unstale":"2024-09-13T13:17:09.600474+0000","last_undegraded":"2024-09-13T13:17:09.600474+0000","last_fullsized":"2024-09-13T13:17:09.600474+0000","mapping_epoch":393,"log_start":"371'9371","ondisk_log_start":"371'9371","created":334,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7822","last_scrub_stamp":"2024-09-13T13:15:32.530220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.530220+0000","objects_scrubbed":7744,"log_size":4597,"log_dups_size":0,"ondisk_log_size":4597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123994112,"num_objects":1892,"num_object_clones":0,"num_object_copies":5676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13968,"num_write_kb":507520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2919,"num_bytes_recovered":191299584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":766,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.077981+0000","last_change":"2024-09-13T13:13:56.850149+0000","last_active":"2024-09-13T13:17:09.077981+0000","last_peered":"2024-09-13T13:17:09.077981+0000","last_clean":"2024-09-13T13:17:09.077981+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:17:09.077981+0000","last_undegraded":"2024-09-13T13:17:09.077981+0000","last_fullsized":"2024-09-13T13:17:09.077981+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:56.850094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.850094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:09.858841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":759,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129238+0000","last_change":"2024-09-13T13:12:58.857634+0000","last_active":"2024-09-13T13:17:13.129238+0000","last_peered":"2024-09-13T13:17:13.129238+0000","last_clean":"2024-09-13T13:17:13.129238+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:17:13.129238+0000","last_undegraded":"2024-09-13T13:17:13.129238+0000","last_fullsized":"2024-09-13T13:17:13.129238+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:58.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":606,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131177+0000","last_change":"2024-09-13T13:16:12.077831+0000","last_active":"2024-09-13T13:17:13.131177+0000","last_peered":"2024-09-13T13:17:13.131177+0000","last_clean":"2024-09-13T13:17:13.131177+0000","last_became_active":"2024-09-13T13:16:12.076962+0000","last_became_peered":"2024-09-13T13:16:12.076962+0000","last_unstale":"2024-09-13T13:17:13.131177+0000","last_undegraded":"2024-09-13T13:17:13.131177+0000","last_fullsized":"2024-09-13T13:17:13.131177+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:23.682463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.682463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":722,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.076850+0000","last_change":"2024-09-13T13:16:12.075127+0000","last_active":"2024-09-13T13:17:09.076850+0000","last_peered":"2024-09-13T13:17:09.076850+0000","last_clean":"2024-09-13T13:17:09.076850+0000","last_became_active":"2024-09-13T13:16:12.074784+0000","last_became_peered":"2024-09-13T13:16:12.074784+0000","last_unstale":"2024-09-13T13:17:09.076850+0000","last_undegraded":"2024-09-13T13:17:09.076850+0000","last_fullsized":"2024-09-13T13:17:09.076850+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.761814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.761814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"399'13793","reported_seq":13983,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.651402+0000","last_change":"2024-09-13T13:16:34.719611+0000","last_active":"2024-09-13T13:17:09.651402+0000","last_peered":"2024-09-13T13:17:09.651402+0000","last_clean":"2024-09-13T13:17:09.651402+0000","last_became_active":"2024-09-13T13:13:58.963160+0000","last_became_peered":"2024-09-13T13:13:58.963160+0000","last_unstale":"2024-09-13T13:17:09.651402+0000","last_undegraded":"2024-09-13T13:17:09.651402+0000","last_fullsized":"2024-09-13T13:17:09.651402+0000","mapping_epoch":334,"log_start":"371'9188","ondisk_log_start":"371'9188","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'11789","last_scrub_stamp":"2024-09-13T13:16:34.719499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:16:34.719499+0000","objects_scrubbed":3932,"log_size":4605,"log_dups_size":0,"ondisk_log_size":4605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:43.651072+0000","scrub_duration":908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124846080,"num_objects":1905,"num_object_clones":0,"num_object_copies":5715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13793,"num_write_kb":502336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":742,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.104706+0000","last_change":"2024-09-13T13:13:06.339762+0000","last_active":"2024-09-13T13:17:11.104706+0000","last_peered":"2024-09-13T13:17:11.104706+0000","last_clean":"2024-09-13T13:17:11.104706+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:17:11.104706+0000","last_undegraded":"2024-09-13T13:17:11.104706+0000","last_fullsized":"2024-09-13T13:17:11.104706+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:13:06.339664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.339664+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:15.437483+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":374,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129277+0000","last_change":"2024-09-13T13:14:55.471954+0000","last_active":"2024-09-13T13:17:13.129277+0000","last_peered":"2024-09-13T13:17:13.129277+0000","last_clean":"2024-09-13T13:17:13.129277+0000","last_became_active":"2024-09-13T13:12:39.954315+0000","last_became_peered":"2024-09-13T13:12:39.954315+0000","last_unstale":"2024-09-13T13:17:13.129277+0000","last_undegraded":"2024-09-13T13:17:13.129277+0000","last_fullsized":"2024-09-13T13:17:13.129277+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:55.471885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:55.471885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":58,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.077884+0000","last_change":"2024-09-13T13:16:12.076693+0000","last_active":"2024-09-13T13:17:09.077884+0000","last_peered":"2024-09-13T13:17:09.077884+0000","last_clean":"2024-09-13T13:17:09.077884+0000","last_became_active":"2024-09-13T13:16:12.076377+0000","last_became_peered":"2024-09-13T13:16:12.076377+0000","last_unstale":"2024-09-13T13:17:09.077884+0000","last_undegraded":"2024-09-13T13:17:09.077884+0000","last_fullsized":"2024-09-13T13:17:09.077884+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.484615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.484615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:18.832449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":61,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.098172+0000","last_change":"2024-09-13T13:16:12.066346+0000","last_active":"2024-09-13T13:17:11.098172+0000","last_peered":"2024-09-13T13:17:11.098172+0000","last_clean":"2024-09-13T13:17:11.098172+0000","last_became_active":"2024-09-13T13:16:12.065848+0000","last_became_peered":"2024-09-13T13:16:12.065848+0000","last_unstale":"2024-09-13T13:17:11.098172+0000","last_undegraded":"2024-09-13T13:17:11.098172+0000","last_fullsized":"2024-09-13T13:17:11.098172+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:02.627569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:02.627569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:41.973870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"400'13822","reported_seq":13966,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.318435+0000","last_change":"2024-09-13T13:13:58.966955+0000","last_active":"2024-09-13T13:17:11.318435+0000","last_peered":"2024-09-13T13:17:11.318435+0000","last_clean":"2024-09-13T13:17:11.318435+0000","last_became_active":"2024-09-13T13:13:58.966675+0000","last_became_peered":"2024-09-13T13:13:58.966675+0000","last_unstale":"2024-09-13T13:17:11.318435+0000","last_undegraded":"2024-09-13T13:17:11.318435+0000","last_fullsized":"2024-09-13T13:17:11.318435+0000","mapping_epoch":334,"log_start":"368'8641","ondisk_log_start":"368'8641","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:21.398151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115867648,"num_objects":1768,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13822,"num_write_kb":498880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":232,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131909+0000","last_change":"2024-09-13T13:16:00.285064+0000","last_active":"2024-09-13T13:17:13.131909+0000","last_peered":"2024-09-13T13:17:13.131909+0000","last_clean":"2024-09-13T13:17:13.131909+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:17:13.131909+0000","last_undegraded":"2024-09-13T13:17:13.131909+0000","last_fullsized":"2024-09-13T13:17:13.131909+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:16:00.284984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:00.284984+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:27.689954+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":219,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131335+0000","last_change":"2024-09-13T13:14:52.383027+0000","last_active":"2024-09-13T13:17:13.131335+0000","last_peered":"2024-09-13T13:17:13.131335+0000","last_clean":"2024-09-13T13:17:13.131335+0000","last_became_active":"2024-09-13T13:12:53.007561+0000","last_became_peered":"2024-09-13T13:12:53.007561+0000","last_unstale":"2024-09-13T13:17:13.131335+0000","last_undegraded":"2024-09-13T13:17:13.131335+0000","last_fullsized":"2024-09-13T13:17:13.131335+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.382755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.382755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":58,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.076552+0000","last_change":"2024-09-13T13:16:12.078026+0000","last_active":"2024-09-13T13:17:09.076552+0000","last_peered":"2024-09-13T13:17:09.076552+0000","last_clean":"2024-09-13T13:17:09.076552+0000","last_became_active":"2024-09-13T13:16:12.077714+0000","last_became_peered":"2024-09-13T13:16:12.077714+0000","last_unstale":"2024-09-13T13:17:09.076552+0000","last_undegraded":"2024-09-13T13:17:09.076552+0000","last_fullsized":"2024-09-13T13:17:09.076552+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.636898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.636898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":718,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.098794+0000","last_change":"2024-09-13T13:16:58.047398+0000","last_active":"2024-09-13T13:17:11.098794+0000","last_peered":"2024-09-13T13:17:11.098794+0000","last_clean":"2024-09-13T13:17:11.098794+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:17:11.098794+0000","last_undegraded":"2024-09-13T13:17:11.098794+0000","last_fullsized":"2024-09-13T13:17:11.098794+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:58.044139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.044139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:25.917931+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"401'14433","reported_seq":14652,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.489391+0000","last_change":"2024-09-13T13:15:56.411481+0000","last_active":"2024-09-13T13:17:13.489391+0000","last_peered":"2024-09-13T13:17:13.489391+0000","last_clean":"2024-09-13T13:17:13.489391+0000","last_became_active":"2024-09-13T13:13:58.962773+0000","last_became_peered":"2024-09-13T13:13:58.962773+0000","last_unstale":"2024-09-13T13:17:13.489391+0000","last_undegraded":"2024-09-13T13:17:13.489391+0000","last_fullsized":"2024-09-13T13:17:13.489391+0000","mapping_epoch":334,"log_start":"371'9376","ondisk_log_start":"371'9376","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'9588","last_scrub_stamp":"2024-09-13T13:15:56.411320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:56.411320+0000","objects_scrubbed":6514,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:14.862198+0000","scrub_duration":1806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106233856,"num_objects":1621,"num_object_clones":0,"num_object_copies":4863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14433,"num_write_kb":513728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":806,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.128894+0000","last_change":"2024-09-13T13:15:56.631456+0000","last_active":"2024-09-13T13:17:13.128894+0000","last_peered":"2024-09-13T13:17:13.128894+0000","last_clean":"2024-09-13T13:17:13.128894+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:17:13.128894+0000","last_undegraded":"2024-09-13T13:17:13.128894+0000","last_fullsized":"2024-09-13T13:17:13.128894+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:15:56.631396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:56.631396+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:22.171338+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":217,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131834+0000","last_change":"2024-09-13T13:13:01.972907+0000","last_active":"2024-09-13T13:17:13.131834+0000","last_peered":"2024-09-13T13:17:13.131834+0000","last_clean":"2024-09-13T13:17:13.131834+0000","last_became_active":"2024-09-13T13:12:53.006362+0000","last_became_peered":"2024-09-13T13:12:53.006362+0000","last_unstale":"2024-09-13T13:17:13.131834+0000","last_undegraded":"2024-09-13T13:17:13.131834+0000","last_fullsized":"2024-09-13T13:17:13.131834+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:13:01.972831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.972831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"306'13","reported_seq":472,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.103215+0000","last_change":"2024-09-13T13:13:03.334319+0000","last_active":"2024-09-13T13:17:11.103215+0000","last_peered":"2024-09-13T13:17:11.103215+0000","last_clean":"2024-09-13T13:17:11.103215+0000","last_became_active":"2024-09-13T13:12:53.001189+0000","last_became_peered":"2024-09-13T13:12:53.001189+0000","last_unstale":"2024-09-13T13:17:11.103215+0000","last_undegraded":"2024-09-13T13:17:11.103215+0000","last_fullsized":"2024-09-13T13:17:11.103215+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'13","last_scrub_stamp":"2024-09-13T13:13:03.334012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.334012+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:39.235043+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":731,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129756+0000","last_change":"2024-09-13T13:13:55.500961+0000","last_active":"2024-09-13T13:17:13.129756+0000","last_peered":"2024-09-13T13:17:13.129756+0000","last_clean":"2024-09-13T13:17:13.129756+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:17:13.129756+0000","last_undegraded":"2024-09-13T13:17:13.129756+0000","last_fullsized":"2024-09-13T13:17:13.129756+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:55.500908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.500908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":706,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.103526+0000","last_change":"2024-09-13T13:13:04.371082+0000","last_active":"2024-09-13T13:17:11.103526+0000","last_peered":"2024-09-13T13:17:11.103526+0000","last_clean":"2024-09-13T13:17:11.103526+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:17:11.103526+0000","last_undegraded":"2024-09-13T13:17:11.103526+0000","last_fullsized":"2024-09-13T13:17:11.103526+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.370881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.370881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":253,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129589+0000","last_change":"2024-09-13T13:15:23.319486+0000","last_active":"2024-09-13T13:17:13.129589+0000","last_peered":"2024-09-13T13:17:13.129589+0000","last_clean":"2024-09-13T13:17:13.129589+0000","last_became_active":"2024-09-13T13:12:34.673528+0000","last_became_peered":"2024-09-13T13:12:34.673528+0000","last_unstale":"2024-09-13T13:17:13.129589+0000","last_undegraded":"2024-09-13T13:17:13.129589+0000","last_fullsized":"2024-09-13T13:17:13.129589+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:15:23.319412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.319412+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":755,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129888+0000","last_change":"2024-09-13T13:15:51.633487+0000","last_active":"2024-09-13T13:17:13.129888+0000","last_peered":"2024-09-13T13:17:13.129888+0000","last_clean":"2024-09-13T13:17:13.129888+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:17:13.129888+0000","last_undegraded":"2024-09-13T13:17:13.129888+0000","last_fullsized":"2024-09-13T13:17:13.129888+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.633433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.633433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":776,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.133587+0000","last_change":"2024-09-13T13:16:33.024715+0000","last_active":"2024-09-13T13:17:13.133587+0000","last_peered":"2024-09-13T13:17:13.133587+0000","last_clean":"2024-09-13T13:17:13.133587+0000","last_became_active":"2024-09-13T13:12:53.000873+0000","last_became_peered":"2024-09-13T13:12:53.000873+0000","last_unstale":"2024-09-13T13:17:13.133587+0000","last_undegraded":"2024-09-13T13:17:13.133587+0000","last_fullsized":"2024-09-13T13:17:13.133587+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:33.024527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:33.024527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:47.416529+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":747,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.077462+0000","last_change":"2024-09-13T13:16:37.748574+0000","last_active":"2024-09-13T13:17:09.077462+0000","last_peered":"2024-09-13T13:17:09.077462+0000","last_clean":"2024-09-13T13:17:09.077462+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:17:09.077462+0000","last_undegraded":"2024-09-13T13:17:09.077462+0000","last_fullsized":"2024-09-13T13:17:09.077462+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:16:37.748500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:37.748500+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:52.503365+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":211,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131750+0000","last_change":"2024-09-13T13:13:51.764793+0000","last_active":"2024-09-13T13:17:13.131750+0000","last_peered":"2024-09-13T13:17:13.131750+0000","last_clean":"2024-09-13T13:17:13.131750+0000","last_became_active":"2024-09-13T13:12:53.006579+0000","last_became_peered":"2024-09-13T13:12:53.006579+0000","last_unstale":"2024-09-13T13:17:13.131750+0000","last_undegraded":"2024-09-13T13:17:13.131750+0000","last_fullsized":"2024-09-13T13:17:13.131750+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.764717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.764717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":737,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.103253+0000","last_change":"2024-09-13T13:17:00.968640+0000","last_active":"2024-09-13T13:17:11.103253+0000","last_peered":"2024-09-13T13:17:11.103253+0000","last_clean":"2024-09-13T13:17:11.103253+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:17:11.103253+0000","last_undegraded":"2024-09-13T13:17:11.103253+0000","last_fullsized":"2024-09-13T13:17:11.103253+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:00.968580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:00.968580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:22.774319+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":609,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131600+0000","last_change":"2024-09-13T13:16:12.078548+0000","last_active":"2024-09-13T13:17:13.131600+0000","last_peered":"2024-09-13T13:17:13.131600+0000","last_clean":"2024-09-13T13:17:13.131600+0000","last_became_active":"2024-09-13T13:16:12.077562+0000","last_became_peered":"2024-09-13T13:16:12.077562+0000","last_unstale":"2024-09-13T13:17:13.131600+0000","last_undegraded":"2024-09-13T13:17:13.131600+0000","last_fullsized":"2024-09-13T13:17:13.131600+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:15.616112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:15.616112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":781,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129327+0000","last_change":"2024-09-13T13:17:04.222896+0000","last_active":"2024-09-13T13:17:13.129327+0000","last_peered":"2024-09-13T13:17:13.129327+0000","last_clean":"2024-09-13T13:17:13.129327+0000","last_became_active":"2024-09-13T13:16:12.069730+0000","last_became_peered":"2024-09-13T13:16:12.069730+0000","last_unstale":"2024-09-13T13:17:13.129327+0000","last_undegraded":"2024-09-13T13:17:13.129327+0000","last_fullsized":"2024-09-13T13:17:13.129327+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:04.222760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:04.222760+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:29.524612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":726,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.634101+0000","last_change":"2024-09-13T13:17:09.634101+0000","last_active":"2024-09-13T13:17:09.634101+0000","last_peered":"2024-09-13T13:17:09.634101+0000","last_clean":"2024-09-13T13:17:09.634101+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:17:09.634101+0000","last_undegraded":"2024-09-13T13:17:09.634101+0000","last_fullsized":"2024-09-13T13:17:09.634101+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.634027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.634027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:29.941144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"377'9","reported_seq":927,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.099531+0000","last_change":"2024-09-13T13:16:12.080560+0000","last_active":"2024-09-13T13:17:11.099531+0000","last_peered":"2024-09-13T13:17:11.099531+0000","last_clean":"2024-09-13T13:17:11.099531+0000","last_became_active":"2024-09-13T13:16:12.073710+0000","last_became_peered":"2024-09-13T13:16:12.073710+0000","last_unstale":"2024-09-13T13:17:11.099531+0000","last_undegraded":"2024-09-13T13:17:11.099531+0000","last_fullsized":"2024-09-13T13:17:11.099531+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:13:10.130680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.130680+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:14.162003+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":218,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131922+0000","last_change":"2024-09-13T13:16:12.077256+0000","last_active":"2024-09-13T13:17:13.131922+0000","last_peered":"2024-09-13T13:17:13.131922+0000","last_clean":"2024-09-13T13:17:13.131922+0000","last_became_active":"2024-09-13T13:16:12.076912+0000","last_became_peered":"2024-09-13T13:16:12.076912+0000","last_unstale":"2024-09-13T13:17:13.131922+0000","last_undegraded":"2024-09-13T13:17:13.131922+0000","last_fullsized":"2024-09-13T13:17:13.131922+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.956784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.956784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":779,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.099811+0000","last_change":"2024-09-13T13:17:02.978088+0000","last_active":"2024-09-13T13:17:11.099811+0000","last_peered":"2024-09-13T13:17:11.099811+0000","last_clean":"2024-09-13T13:17:11.099811+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:17:11.099811+0000","last_undegraded":"2024-09-13T13:17:11.099811+0000","last_fullsized":"2024-09-13T13:17:11.099811+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.978003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.978003+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:17.628791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":744,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.132978+0000","last_change":"2024-09-13T13:16:12.067154+0000","last_active":"2024-09-13T13:17:13.132978+0000","last_peered":"2024-09-13T13:17:13.132978+0000","last_clean":"2024-09-13T13:17:13.132978+0000","last_became_active":"2024-09-13T13:16:12.066808+0000","last_became_peered":"2024-09-13T13:16:12.066808+0000","last_unstale":"2024-09-13T13:17:13.132978+0000","last_undegraded":"2024-09-13T13:17:13.132978+0000","last_fullsized":"2024-09-13T13:17:13.132978+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.105098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.105098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:13.844340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":942,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129098+0000","last_change":"2024-09-13T13:15:33.333062+0000","last_active":"2024-09-13T13:17:13.129098+0000","last_peered":"2024-09-13T13:17:13.129098+0000","last_clean":"2024-09-13T13:17:13.129098+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:17:13.129098+0000","last_undegraded":"2024-09-13T13:17:13.129098+0000","last_fullsized":"2024-09-13T13:17:13.129098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:15:33.332981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.332981+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":730,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.076975+0000","last_change":"2024-09-13T13:15:22.660262+0000","last_active":"2024-09-13T13:17:09.076975+0000","last_peered":"2024-09-13T13:17:09.076975+0000","last_clean":"2024-09-13T13:17:09.076975+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:17:09.076975+0000","last_undegraded":"2024-09-13T13:17:09.076975+0000","last_fullsized":"2024-09-13T13:17:09.076975+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:22.660050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:22.660050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":223,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134730+0000","last_change":"2024-09-13T13:13:47.728092+0000","last_active":"2024-09-13T13:17:13.134730+0000","last_peered":"2024-09-13T13:17:13.134730+0000","last_clean":"2024-09-13T13:17:13.134730+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:17:13.134730+0000","last_undegraded":"2024-09-13T13:17:13.134730+0000","last_fullsized":"2024-09-13T13:17:13.134730+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:47.728010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:47.728010+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":211,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131446+0000","last_change":"2024-09-13T13:17:09.431539+0000","last_active":"2024-09-13T13:17:13.131446+0000","last_peered":"2024-09-13T13:17:13.131446+0000","last_clean":"2024-09-13T13:17:13.131446+0000","last_became_active":"2024-09-13T13:12:53.005988+0000","last_became_peered":"2024-09-13T13:12:53.005988+0000","last_unstale":"2024-09-13T13:17:13.131446+0000","last_undegraded":"2024-09-13T13:17:13.131446+0000","last_fullsized":"2024-09-13T13:17:13.131446+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.431472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.431472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:30.094940+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":736,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.096509+0000","last_change":"2024-09-13T13:15:57.356507+0000","last_active":"2024-09-13T13:17:09.096509+0000","last_peered":"2024-09-13T13:17:09.096509+0000","last_clean":"2024-09-13T13:17:09.096509+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:17:09.096509+0000","last_undegraded":"2024-09-13T13:17:09.096509+0000","last_fullsized":"2024-09-13T13:17:09.096509+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:57.356427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:57.356427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:20.233276+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":741,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.097906+0000","last_change":"2024-09-13T13:17:09.989898+0000","last_active":"2024-09-13T13:17:11.097906+0000","last_peered":"2024-09-13T13:17:11.097906+0000","last_clean":"2024-09-13T13:17:11.097906+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:17:11.097906+0000","last_undegraded":"2024-09-13T13:17:11.097906+0000","last_fullsized":"2024-09-13T13:17:11.097906+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.989820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.989820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:38.033009+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":713,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129593+0000","last_change":"2024-09-13T13:16:12.087845+0000","last_active":"2024-09-13T13:17:13.129593+0000","last_peered":"2024-09-13T13:17:13.129593+0000","last_clean":"2024-09-13T13:17:13.129593+0000","last_became_active":"2024-09-13T13:16:12.072262+0000","last_became_peered":"2024-09-13T13:16:12.072262+0000","last_unstale":"2024-09-13T13:17:13.129593+0000","last_undegraded":"2024-09-13T13:17:13.129593+0000","last_fullsized":"2024-09-13T13:17:13.129593+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:39.874324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:39.874324+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":721,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.099792+0000","last_change":"2024-09-13T13:14:13.418389+0000","last_active":"2024-09-13T13:17:11.099792+0000","last_peered":"2024-09-13T13:17:11.099792+0000","last_clean":"2024-09-13T13:17:11.099792+0000","last_became_active":"2024-09-13T13:12:52.949094+0000","last_became_peered":"2024-09-13T13:12:52.949094+0000","last_unstale":"2024-09-13T13:17:11.099792+0000","last_undegraded":"2024-09-13T13:17:11.099792+0000","last_fullsized":"2024-09-13T13:17:11.099792+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:13.418196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:13.418196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:51.937954+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":729,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.098956+0000","last_change":"2024-09-13T13:15:01.269060+0000","last_active":"2024-09-13T13:17:11.098956+0000","last_peered":"2024-09-13T13:17:11.098956+0000","last_clean":"2024-09-13T13:17:11.098956+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:17:11.098956+0000","last_undegraded":"2024-09-13T13:17:11.098956+0000","last_fullsized":"2024-09-13T13:17:11.098956+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:01.268884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:01.268884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:44.996809+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":714,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.076491+0000","last_change":"2024-09-13T13:10:39.477882+0000","last_active":"2024-09-13T13:17:09.076491+0000","last_peered":"2024-09-13T13:17:09.076491+0000","last_clean":"2024-09-13T13:17:09.076491+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:17:09.076491+0000","last_undegraded":"2024-09-13T13:17:09.076491+0000","last_fullsized":"2024-09-13T13:17:09.076491+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:39.477683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.477683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:40.746995+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":759,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.098100+0000","last_change":"2024-09-13T13:16:28.056674+0000","last_active":"2024-09-13T13:17:11.098100+0000","last_peered":"2024-09-13T13:17:11.098100+0000","last_clean":"2024-09-13T13:17:11.098100+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:17:11.098100+0000","last_undegraded":"2024-09-13T13:17:11.098100+0000","last_fullsized":"2024-09-13T13:17:11.098100+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:16:28.056586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:28.056586+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:57.805987+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":747,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.130332+0000","last_change":"2024-09-13T13:14:50.392659+0000","last_active":"2024-09-13T13:17:13.130332+0000","last_peered":"2024-09-13T13:17:13.130332+0000","last_clean":"2024-09-13T13:17:13.130332+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:17:13.130332+0000","last_undegraded":"2024-09-13T13:17:13.130332+0000","last_fullsized":"2024-09-13T13:17:13.130332+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.392444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.392444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":756,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.132133+0000","last_change":"2024-09-13T13:15:03.778214+0000","last_active":"2024-09-13T13:17:13.132133+0000","last_peered":"2024-09-13T13:17:13.132133+0000","last_clean":"2024-09-13T13:17:13.132133+0000","last_became_active":"2024-09-13T13:12:52.999748+0000","last_became_peered":"2024-09-13T13:12:52.999748+0000","last_unstale":"2024-09-13T13:17:13.132133+0000","last_undegraded":"2024-09-13T13:17:13.132133+0000","last_fullsized":"2024-09-13T13:17:13.132133+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.778170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.778170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:20.863950+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":712,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.099101+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:17:11.099101+0000","last_peered":"2024-09-13T13:17:11.099101+0000","last_clean":"2024-09-13T13:17:11.099101+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:17:11.099101+0000","last_undegraded":"2024-09-13T13:17:11.099101+0000","last_fullsized":"2024-09-13T13:17:11.099101+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:17.141450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":774,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.098055+0000","last_change":"2024-09-13T13:16:12.090990+0000","last_active":"2024-09-13T13:17:11.098055+0000","last_peered":"2024-09-13T13:17:11.098055+0000","last_clean":"2024-09-13T13:17:11.098055+0000","last_became_active":"2024-09-13T13:16:12.073702+0000","last_became_peered":"2024-09-13T13:16:12.073702+0000","last_unstale":"2024-09-13T13:17:11.098055+0000","last_undegraded":"2024-09-13T13:17:11.098055+0000","last_fullsized":"2024-09-13T13:17:11.098055+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:14:12.368430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:12.368430+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:39.946451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":750,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.077336+0000","last_change":"2024-09-13T13:16:32.807018+0000","last_active":"2024-09-13T13:17:09.077336+0000","last_peered":"2024-09-13T13:17:09.077336+0000","last_clean":"2024-09-13T13:17:09.077336+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:17:09.077336+0000","last_undegraded":"2024-09-13T13:17:09.077336+0000","last_fullsized":"2024-09-13T13:17:09.077336+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:32.806957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:32.806957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:56.930095+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":660,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.103313+0000","last_change":"2024-09-13T13:14:16.457160+0000","last_active":"2024-09-13T13:17:11.103313+0000","last_peered":"2024-09-13T13:17:11.103313+0000","last_clean":"2024-09-13T13:17:11.103313+0000","last_became_active":"2024-09-13T13:12:53.001131+0000","last_became_peered":"2024-09-13T13:12:53.001131+0000","last_unstale":"2024-09-13T13:17:11.103313+0000","last_undegraded":"2024-09-13T13:17:11.103313+0000","last_fullsized":"2024-09-13T13:17:11.103313+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:16.456784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:16.456784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":571,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131009+0000","last_change":"2024-09-13T13:16:12.068100+0000","last_active":"2024-09-13T13:17:13.131009+0000","last_peered":"2024-09-13T13:17:13.131009+0000","last_clean":"2024-09-13T13:17:13.131009+0000","last_became_active":"2024-09-13T13:16:12.067757+0000","last_became_peered":"2024-09-13T13:16:12.067757+0000","last_unstale":"2024-09-13T13:17:13.131009+0000","last_undegraded":"2024-09-13T13:17:13.131009+0000","last_fullsized":"2024-09-13T13:17:13.131009+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:05.828300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:05.828300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:18.999851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":738,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.095130+0000","last_change":"2024-09-13T13:14:18.882969+0000","last_active":"2024-09-13T13:17:09.095130+0000","last_peered":"2024-09-13T13:17:09.095130+0000","last_clean":"2024-09-13T13:17:09.095130+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:17:09.095130+0000","last_undegraded":"2024-09-13T13:17:09.095130+0000","last_fullsized":"2024-09-13T13:17:09.095130+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:18.882903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:18.882903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:47.220508+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":729,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.098264+0000","last_change":"2024-09-13T13:16:36.980668+0000","last_active":"2024-09-13T13:17:11.098264+0000","last_peered":"2024-09-13T13:17:11.098264+0000","last_clean":"2024-09-13T13:17:11.098264+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:17:11.098264+0000","last_undegraded":"2024-09-13T13:17:11.098264+0000","last_fullsized":"2024-09-13T13:17:11.098264+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:36.980594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:36.980594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:50.040358+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":748,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-09-13T13:17:11.107814+0000","last_change":"2024-09-13T13:16:12.087075+0000","last_active":"2024-09-13T13:17:11.107814+0000","last_peered":"2024-09-13T13:17:11.107814+0000","last_clean":"2024-09-13T13:17:11.107814+0000","last_became_active":"2024-09-13T13:16:12.070277+0000","last_became_peered":"2024-09-13T13:16:12.070277+0000","last_unstale":"2024-09-13T13:17:11.107814+0000","last_undegraded":"2024-09-13T13:17:11.107814+0000","last_fullsized":"2024-09-13T13:17:11.107814+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:49.264128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.264128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:13.408862+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":750,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-13T13:17:09.077174+0000","last_change":"2024-09-13T13:16:12.074365+0000","last_active":"2024-09-13T13:17:09.077174+0000","last_peered":"2024-09-13T13:17:09.077174+0000","last_clean":"2024-09-13T13:17:09.077174+0000","last_became_active":"2024-09-13T13:16:12.072684+0000","last_became_peered":"2024-09-13T13:16:12.072684+0000","last_unstale":"2024-09-13T13:17:09.077174+0000","last_undegraded":"2024-09-13T13:17:09.077174+0000","last_fullsized":"2024-09-13T13:17:09.077174+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:27.678752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:27.678752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1844117528,"num_objects":28140,"num_object_clones":0,"num_object_copies":84420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226616,"num_write_kb":8152129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19739,"num_bytes_recovered":1293615104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6312243200,"data_stored":6312230984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77879,"ondisk_log_size":77879,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":3526,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":684,"ondisk_log_size":684,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":275,"num_read_kb":270,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201477,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1094392,"kb_used_data":527272,"kb_used_omap":1,"kb_used_meta":567038,"kb_avail":92650760,"statfs":{"total":95995035648,"available":94874378240,"internally_reserved":0,"allocated":539926528,"data_stored":539522926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":580647370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141087,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1326132,"kb_used_data":841636,"kb_used_omap":4,"kb_used_meta":484475,"kb_avail":92419020,"statfs":{"total":95995035648,"available":94637076480,"internally_reserved":0,"allocated":861835264,"data_stored":861427794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":496103085},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692049,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1307972,"kb_used_data":735340,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92437180,"statfs":{"total":95995035648,"available":94655672320,"internally_reserved":0,"allocated":752988160,"data_stored":752585703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920983,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1767176,"kb_used_data":1200116,"kb_used_omap":1,"kb_used_meta":567038,"kb_avail":91977976,"statfs":{"total":95995035648,"available":94185447424,"internally_reserved":0,"allocated":1228918784,"data_stored":1228501350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":580647366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117214,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1205004,"kb_used_data":556024,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92540148,"statfs":{"total":95995035648,"available":94761111552,"internally_reserved":0,"allocated":569368576,"data_stored":568948255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":664533447},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411553,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575079,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":967940,"kb_used_data":386992,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92777212,"statfs":{"total":95995035648,"available":95003865088,"internally_reserved":0,"allocated":396279808,"data_stored":395853405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738606,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1922304,"kb_used_data":1219180,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":91822848,"statfs":{"total":95995035648,"available":94026596352,"internally_reserved":0,"allocated":1248440320,"data_stored":1248025748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1245188096,"data_stored":1245184024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":384303104,"data_stored":384303104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":850722816,"data_stored":850722816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":548012032,"data_stored":548012032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1184763904,"data_stored":1184759832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":740425728,"data_stored":740425728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":825823232,"data_stored":825819160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":533004288,"data_stored":533004288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:17:16.537 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:17:16.538 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:16.568 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:16.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:16 smithi167 ceph-mon[33178]: from='client.16626 v1:172.21.15.52:0/569042121' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:17:16.610 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:16.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:16.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:16.711 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:16.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:16.743 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:16.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:16.844 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:16.877 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:16.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:16.979 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:17.009 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:17.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:17.111 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:17.149 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:17.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:17.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:17.152 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:17:17.152 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:17:17.162 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:17:17.162 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:17:17.164 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:17:17.195 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:17:17.195 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:17:17.226 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:17:17.226 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:17:17.251 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:17.273 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:17.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:17.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:17.282 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:17:17.282 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:17:17.317 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:17:17.317 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:17:17.345 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:17:17.345 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:17:17.374 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:17:17.374 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:17:17.375 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[36682]: pgmap v804: 148 pgs: 148 active+clean; 1.6 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-09-13T13:17:17.375 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:17:17.375 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:17:17.375 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[36682]: osdmap e402: 8 total, 8 up, 7 in 2024-09-13T13:17:17.375 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/84168190' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:17.375 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[36682]: from='client.26398 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:17.375 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/165446314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:17.375 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[29471]: pgmap v804: 148 pgs: 148 active+clean; 1.6 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-09-13T13:17:17.375 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:17:17.375 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:17:17.376 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[29471]: osdmap e402: 8 total, 8 up, 7 in 2024-09-13T13:17:17.376 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/84168190' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:17.376 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[29471]: from='client.26398 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:17.376 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:17 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/165446314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:17.376 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:17.415 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:17 smithi167 ceph-mon[33178]: pgmap v804: 148 pgs: 148 active+clean; 1.6 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-09-13T13:17:17.415 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:17 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:17:17.415 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:17 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:17:17.416 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:17 smithi167 ceph-mon[33178]: osdmap e402: 8 total, 8 up, 7 in 2024-09-13T13:17:17.416 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:17 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/84168190' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:17.416 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:17 smithi167 ceph-mon[33178]: from='client.26398 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:17.416 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:17 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/165446314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:17.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:17.419 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:17.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:17.521 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:17.553 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:17.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:17.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:17.581 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:17:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:17:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:17:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:17:17.582 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:17:17.654 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:17.682 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:17.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:17.784 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:17.823 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:17.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:17.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:17.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:17.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:17.961 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:17.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:18.062 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:18.093 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:18.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:18.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:18.195 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:18.224 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:18.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:18.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:18 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:18.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:18 smithi052 ceph-mon[36682]: from='client.26398 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:17:18.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:18 smithi052 ceph-mon[36682]: osdmap e403: 8 total, 8 up, 7 in 2024-09-13T13:17:18.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:18 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:17:18.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:18 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:17:18.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:18 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:18.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:18 smithi052 ceph-mon[29471]: from='client.26398 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:17:18.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:18 smithi052 ceph-mon[29471]: osdmap e403: 8 total, 8 up, 7 in 2024-09-13T13:17:18.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:18 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:17:18.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:18 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:17:18.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:18 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:18.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:18 smithi167 ceph-mon[33178]: from='client.26398 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:17:18.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:18 smithi167 ceph-mon[33178]: osdmap e403: 8 total, 8 up, 7 in 2024-09-13T13:17:18.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:18 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:17:18.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:18 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:17:18.325 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:18.346 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:18.365 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:18.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:18.467 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:18.502 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:18.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:18.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:18.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:18.646 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:18.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:18.748 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:18.780 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:18.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:18.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:18.829 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T13:17:18.881 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:18.943 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:18.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:18.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:19.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:19.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:19.091 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:19.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:19.192 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:19.246 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:19.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:19.270 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:17:19.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:19 smithi167 ceph-mon[33178]: pgmap v807: 148 pgs: 148 active+clean; 1.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 967 op/s 2024-09-13T13:17:19.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:19 smithi167 ceph-mon[33178]: osdmap e404: 8 total, 8 up, 7 in 2024-09-13T13:17:19.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:19 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:19.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:19 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:17:19.347 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:19.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:19.387 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:19.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:19.488 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:19.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:19 smithi052 ceph-mon[29471]: pgmap v807: 148 pgs: 148 active+clean; 1.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 967 op/s 2024-09-13T13:17:19.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:19 smithi052 ceph-mon[29471]: osdmap e404: 8 total, 8 up, 7 in 2024-09-13T13:17:19.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:19 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:19.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:19 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:17:19.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:19 smithi052 ceph-mon[36682]: pgmap v807: 148 pgs: 148 active+clean; 1.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 967 op/s 2024-09-13T13:17:19.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:19 smithi052 ceph-mon[36682]: osdmap e404: 8 total, 8 up, 7 in 2024-09-13T13:17:19.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:19 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:19.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:19 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:17:19.526 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:19.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:19.628 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:19.661 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:19.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:19.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:19.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:19.803 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:19.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:19.808 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:17:19.810 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:17:19.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:19.943 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:19.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:19.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:19.956 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:17:19.956 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:17:19.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:17:19.956 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:17:19.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.956 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:154796 9.3 9:c3ab54a2:::benchmark_data_smithi052_128427_object24127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T13:17:19.956 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:08.420648+0000", 2024-09-13T13:17:19.956 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 311.53255901599999, 2024-09-13T13:17:19.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16434322200000001, 2024-09-13T13:17:19.957 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:19.957 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:19.957 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:19.957 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:19.957 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:17:19.957 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:17:19.957 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154796 2024-09-13T13:17:19.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.957 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:19.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:19.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420648+0000", 2024-09-13T13:17:19.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:19.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420648+0000", 2024-09-13T13:17:19.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:19.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420649+0000", 2024-09-13T13:17:19.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2410000000000001e-06 2024-09-13T13:17:19.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:19.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420654+0000", 2024-09-13T13:17:19.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3000000000000003e-06 2024-09-13T13:17:19.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:19.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420662+0000", 2024-09-13T13:17:19.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4740000000000005e-06 2024-09-13T13:17:19.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:19.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420667+0000", 2024-09-13T13:17:19.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.797e-06 2024-09-13T13:17:19.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:19.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420714+0000", 2024-09-13T13:17:19.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6937999999999997e-05 2024-09-13T13:17:19.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:19.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420838+0000", 2024-09-13T13:17:19.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012402999999999999 2024-09-13T13:17:19.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-09-13T13:17:19.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.420949+0000", 2024-09-13T13:17:19.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110874 2024-09-13T13:17:19.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:19.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.423161+0000", 2024-09-13T13:17:19.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022121889999999998 2024-09-13T13:17:19.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:19.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.424593+0000", 2024-09-13T13:17:19.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014320030000000001 2024-09-13T13:17:19.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:19.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.584895+0000", 2024-09-13T13:17:19.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16030223599999999 2024-09-13T13:17:19.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:19.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.584952+0000", 2024-09-13T13:17:19.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6428000000000003e-05 2024-09-13T13:17:19.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:08.584991+0000", 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9712000000000002e-05 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:169641 9.e 9:78334604:::benchmark_data_smithi052_128427_object38972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:22.418468+0000", 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 297.534738963, 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.169452977, 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:19.963 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:19.964 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:19.964 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:19.964 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:17:19.964 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:17:19.964 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169641 2024-09-13T13:17:19.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.964 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:19.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:19.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418468+0000", 2024-09-13T13:17:19.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:19.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418468+0000", 2024-09-13T13:17:19.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:19.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418469+0000", 2024-09-13T13:17:19.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3099999999999996e-07 2024-09-13T13:17:19.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418472+0000", 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0730000000000001e-06 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418478+0000", 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6869999999999999e-06 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.418483+0000", 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.899e-06 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.421223+0000", 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027399949999999998 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.421339+0000", 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116055 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.421457+0000", 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118276 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:19.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.426437+0000", 2024-09-13T13:17:19.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049803579999999998 2024-09-13T13:17:19.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:19.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.429739+0000", 2024-09-13T13:17:19.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033021919999999998 2024-09-13T13:17:19.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:19.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.429814+0000", 2024-09-13T13:17:19.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4789999999999994e-05 2024-09-13T13:17:19.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:19.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587749+0000", 2024-09-13T13:17:19.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.15793521199999999 2024-09-13T13:17:19.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:19.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587804+0000", 2024-09-13T13:17:19.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4314999999999998e-05 2024-09-13T13:17:19.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:19.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587921+0000", 2024-09-13T13:17:19.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011729400000000001 2024-09-13T13:17:19.970 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.970 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:19.970 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.970 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15789.0:169651 9.e 9:73829da4:::benchmark_data_smithi052_128427_object38982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-13T13:17:19.970 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:12:22.425923+0000", 2024-09-13T13:17:19.970 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 297.52728440200002, 2024-09-13T13:17:19.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16214630499999999, 2024-09-13T13:17:19.970 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:19.970 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:19.971 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:19.971 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:19.971 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15789", 2024-09-13T13:17:19.971 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:17:19.971 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169651 2024-09-13T13:17:19.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.971 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:19.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:19.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425923+0000", 2024-09-13T13:17:19.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:19.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425923+0000", 2024-09-13T13:17:19.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:19.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425925+0000", 2024-09-13T13:17:19.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.153e-06 2024-09-13T13:17:19.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:19.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425930+0000", 2024-09-13T13:17:19.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1510000000000003e-06 2024-09-13T13:17:19.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:19.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425941+0000", 2024-09-13T13:17:19.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1219e-05 2024-09-13T13:17:19.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:19.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.425948+0000", 2024-09-13T13:17:19.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4590000000000003e-06 2024-09-13T13:17:19.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:19.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.426533+0000", 2024-09-13T13:17:19.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058506300000000003 2024-09-13T13:17:19.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.426626+0000", 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2878000000000001e-05 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1,4", 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.426723+0000", 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7369999999999998e-05 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.429801+0000", 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030778759999999998 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:19.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.429820+0000", 2024-09-13T13:17:19.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9456000000000001e-05 2024-09-13T13:17:19.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:19.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.559932+0000", 2024-09-13T13:17:19.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13011128799999999 2024-09-13T13:17:19.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:19.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587959+0000", 2024-09-13T13:17:19.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028027237 2024-09-13T13:17:19.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:19.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.587993+0000", 2024-09-13T13:17:19.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4285000000000001e-05 2024-09-13T13:17:19.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:12:22.588069+0000", 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5870000000000004e-05 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:133006 10.8 e366/334 10:1945bf5a:::benchmark_data_smithi052_153703_object5626:head v 366'8411, mlcod=366'8409)", 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.291495+0000", 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 100.661711638, 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.36882864700000001, 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:19.978 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:19.979 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:19.979 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:19.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:19.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291495+0000", 2024-09-13T13:17:19.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:19.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291495+0000", 2024-09-13T13:17:19.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:19.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291497+0000", 2024-09-13T13:17:19.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4109999999999999e-06 2024-09-13T13:17:19.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:19.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291503+0000", 2024-09-13T13:17:19.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2190000000000001e-06 2024-09-13T13:17:19.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:19.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291513+0000", 2024-09-13T13:17:19.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0451999999999999e-05 2024-09-13T13:17:19.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:19.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291520+0000", 2024-09-13T13:17:19.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.759e-06 2024-09-13T13:17:19.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:19.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291570+0000", 2024-09-13T13:17:19.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9979999999999999e-05 2024-09-13T13:17:19.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:19.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291585+0000", 2024-09-13T13:17:19.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4688999999999999e-05 2024-09-13T13:17:19.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:19.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660271+0000", 2024-09-13T13:17:19.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.36868583799999999 2024-09-13T13:17:19.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:19.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660324+0000", 2024-09-13T13:17:19.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3298999999999999e-05 2024-09-13T13:17:19.983 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.983 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:19.983 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.984 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:133009 10.8 e366/334 10:1c6c448b:::benchmark_data_smithi052_153703_object5629:head v 366'8412, mlcod=366'8409)", 2024-09-13T13:17:19.984 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.292418+0000", 2024-09-13T13:17:19.984 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 100.66078866300001, 2024-09-13T13:17:19.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.380843235, 2024-09-13T13:17:19.984 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:19.985 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:19.985 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:19.985 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:19.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:19.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292418+0000", 2024-09-13T13:17:19.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:19.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292418+0000", 2024-09-13T13:17:19.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:19.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292419+0000", 2024-09-13T13:17:19.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.569999999999999e-07 2024-09-13T13:17:19.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:19.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292425+0000", 2024-09-13T13:17:19.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2050000000000001e-06 2024-09-13T13:17:19.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:19.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660256+0000", 2024-09-13T13:17:19.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.36783123499999998 2024-09-13T13:17:19.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:19.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660266+0000", 2024-09-13T13:17:19.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0210000000000001e-05 2024-09-13T13:17:19.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:19.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660362+0000", 2024-09-13T13:17:19.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5999e-05 2024-09-13T13:17:19.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:19.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660382+0000", 2024-09-13T13:17:19.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0035999999999999e-05 2024-09-13T13:17:19.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:19.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.673210+0000", 2024-09-13T13:17:19.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012828381999999999 2024-09-13T13:17:19.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:19.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.673262+0000", 2024-09-13T13:17:19.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1211000000000001e-05 2024-09-13T13:17:19.989 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.989 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:19.989 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.990 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:133029 10.9 e366/334 10:97ab46ca:::benchmark_data_smithi052_153703_object5649:head v 366'8225, mlcod=366'8223)", 2024-09-13T13:17:19.990 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.309291+0000", 2024-09-13T13:17:19.990 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 100.643916276, 2024-09-13T13:17:19.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.356083341, 2024-09-13T13:17:19.990 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:19.990 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:19.990 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:19.990 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:19.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:19.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309291+0000", 2024-09-13T13:17:19.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:19.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309291+0000", 2024-09-13T13:17:19.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:19.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309293+0000", 2024-09-13T13:17:19.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9889999999999999e-06 2024-09-13T13:17:19.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309303+0000", 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0162e-05 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660206+0000", 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.35090332600000002 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660227+0000", 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0778999999999999e-05 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:19.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660332+0000", 2024-09-13T13:17:19.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104756 2024-09-13T13:17:19.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:19.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660360+0000", 2024-09-13T13:17:19.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8252999999999999e-05 2024-09-13T13:17:19.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:19.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665339+0000", 2024-09-13T13:17:19.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049790090000000004 2024-09-13T13:17:19.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:19.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665374+0000", 2024-09-13T13:17:19.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5067000000000002e-05 2024-09-13T13:17:19.994 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.994 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:19.994 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.994 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:150454 10.9 e371/334 10:9296a054:::benchmark_data_smithi052_153703_object23074:head v 371'9301, mlcod=371'9299)", 2024-09-13T13:17:19.994 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:55.061109+0000", 2024-09-13T13:17:19.995 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.892098042000001, 2024-09-13T13:17:19.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.204762101, 2024-09-13T13:17:19.995 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:19.995 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:19.995 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:19.995 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:19.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:19.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061109+0000", 2024-09-13T13:17:19.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:19.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061109+0000", 2024-09-13T13:17:19.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:19.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:19.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061111+0000", 2024-09-13T13:17:19.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5549999999999999e-06 2024-09-13T13:17:19.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:19.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061118+0000", 2024-09-13T13:17:19.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6490000000000008e-06 2024-09-13T13:17:19.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:19.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061130+0000", 2024-09-13T13:17:19.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2072000000000001e-05 2024-09-13T13:17:19.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:19.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061138+0000", 2024-09-13T13:17:19.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.824e-06 2024-09-13T13:17:19.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:19.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061201+0000", 2024-09-13T13:17:19.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3075000000000002e-05 2024-09-13T13:17:19.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:19.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061221+0000", 2024-09-13T13:17:19.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0017000000000002e-05 2024-09-13T13:17:19.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:19.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.066643+0000", 2024-09-13T13:17:19.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0054218249999999999 2024-09-13T13:17:19.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.265871+0000", 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.199228084 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:150456 10.1 e371/334 10:88b0c1b6:::benchmark_data_smithi052_153703_object23076:head v 371'9420, mlcod=371'9416)", 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:55.062401+0000", 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.890806154000003, 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.203845743, 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:19.999 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:20.000 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:20.000 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:20.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:20.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062401+0000", 2024-09-13T13:17:20.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:20.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062401+0000", 2024-09-13T13:17:20.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:20.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062402+0000", 2024-09-13T13:17:20.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.252e-06 2024-09-13T13:17:20.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:20.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062409+0000", 2024-09-13T13:17:20.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9700000000000002e-06 2024-09-13T13:17:20.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062419+0000", 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0084e-05 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062426+0000", 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0840000000000003e-06 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062491+0000", 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5096000000000002e-05 2024-09-13T13:17:20.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:20.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062509+0000", 2024-09-13T13:17:20.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7453e-05 2024-09-13T13:17:20.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:20.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.266168+0000", 2024-09-13T13:17:20.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20365898599999999 2024-09-13T13:17:20.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:20.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.266247+0000", 2024-09-13T13:17:20.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8818000000000003e-05 2024-09-13T13:17:20.004 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.004 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.004 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.005 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:150461 10.9 e371/334 10:9bef7284:::benchmark_data_smithi052_153703_object23081:head v 371'9302, mlcod=371'9299)", 2024-09-13T13:17:20.005 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:55.063189+0000", 2024-09-13T13:17:20.005 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.890018444999995, 2024-09-13T13:17:20.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20551095, 2024-09-13T13:17:20.005 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:20.005 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:20.005 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:20.005 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:20.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:20.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.063189+0000", 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.063189+0000", 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.063190+0000", 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2580000000000001e-06 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:20.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.063196+0000", 2024-09-13T13:17:20.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9959999999999999e-06 2024-09-13T13:17:20.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:20.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.063205+0000", 2024-09-13T13:17:20.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3170000000000008e-06 2024-09-13T13:17:20.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:20.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.063211+0000", 2024-09-13T13:17:20.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.913e-06 2024-09-13T13:17:20.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:20.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.064039+0000", 2024-09-13T13:17:20.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00082804099999999996 2024-09-13T13:17:20.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:20.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.064056+0000", 2024-09-13T13:17:20.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6917000000000001e-05 2024-09-13T13:17:20.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:20.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.268660+0000", 2024-09-13T13:17:20.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20460398499999999 2024-09-13T13:17:20.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:20.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.268700+0000", 2024-09-13T13:17:20.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9523000000000001e-05 2024-09-13T13:17:20.009 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.009 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.010 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.010 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:150463 10.1 e371/334 10:83d9251e:::benchmark_data_smithi052_153703_object23083:head v 371'9421, mlcod=371'9418)", 2024-09-13T13:17:20.010 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:55.065870+0000", 2024-09-13T13:17:20.010 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.887336687000001, 2024-09-13T13:17:20.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20289595699999999, 2024-09-13T13:17:20.010 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:20.010 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:20.010 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:20.010 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:20.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:20.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.065870+0000", 2024-09-13T13:17:20.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:20.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.065870+0000", 2024-09-13T13:17:20.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:20.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.065872+0000", 2024-09-13T13:17:20.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.248e-06 2024-09-13T13:17:20.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:20.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.065878+0000", 2024-09-13T13:17:20.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2910000000000004e-06 2024-09-13T13:17:20.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:20.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.065888+0000", 2024-09-13T13:17:20.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8390000000000002e-06 2024-09-13T13:17:20.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.065894+0000", 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5919999999999997e-06 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.065958+0000", 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3448999999999995e-05 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.065975+0000", 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7523000000000001e-05 2024-09-13T13:17:20.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.268703+0000", 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20272747499999999 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.268766+0000", 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3540000000000005e-05 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:150465 10.9 e371/334 10:9818483c:::benchmark_data_smithi052_153703_object23085:head v 371'9303, mlcod=371'9299)", 2024-09-13T13:17:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:55.066941+0000", 2024-09-13T13:17:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.886265762999997, 2024-09-13T13:17:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20225349500000001, 2024-09-13T13:17:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.066941+0000", 2024-09-13T13:17:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:20.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.066941+0000", 2024-09-13T13:17:20.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:20.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.066943+0000", 2024-09-13T13:17:20.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-09-13T13:17:20.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:20.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.066949+0000", 2024-09-13T13:17:20.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6669999999999999e-06 2024-09-13T13:17:20.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:20.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.265828+0000", 2024-09-13T13:17:20.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19887912199999999 2024-09-13T13:17:20.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:20.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.265842+0000", 2024-09-13T13:17:20.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3264e-05 2024-09-13T13:17:20.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:20.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.266004+0000", 2024-09-13T13:17:20.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016279000000000001 2024-09-13T13:17:20.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:20.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.266022+0000", 2024-09-13T13:17:20.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8040999999999999e-05 2024-09-13T13:17:20.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:20.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.269130+0000", 2024-09-13T13:17:20.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031072130000000002 2024-09-13T13:17:20.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:20.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.269195+0000", 2024-09-13T13:17:20.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5137999999999999e-05 2024-09-13T13:17:20.019 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.019 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.019 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.019 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:150470 10.8 e371/334 10:1ddbf082:::benchmark_data_smithi052_153703_object23090:head v 371'9514, mlcod=371'9512)", 2024-09-13T13:17:20.019 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:55.069046+0000", 2024-09-13T13:17:20.019 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.884160901000001, 2024-09-13T13:17:20.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.204381533, 2024-09-13T13:17:20.020 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:20.020 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:20.020 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:20.020 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:20.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:20.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.069046+0000", 2024-09-13T13:17:20.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:20.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.069046+0000", 2024-09-13T13:17:20.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:20.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.069048+0000", 2024-09-13T13:17:20.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3799999999999999e-06 2024-09-13T13:17:20.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:20.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.069055+0000", 2024-09-13T13:17:20.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8809999999999996e-06 2024-09-13T13:17:20.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:20.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.265825+0000", 2024-09-13T13:17:20.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.196769743 2024-09-13T13:17:20.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:20.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.265841+0000", 2024-09-13T13:17:20.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6076999999999999e-05 2024-09-13T13:17:20.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:20.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.265892+0000", 2024-09-13T13:17:20.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0534000000000001e-05 2024-09-13T13:17:20.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:20.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.265915+0000", 2024-09-13T13:17:20.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2928000000000001e-05 2024-09-13T13:17:20.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:20.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.273300+0000", 2024-09-13T13:17:20.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0073855659999999997 2024-09-13T13:17:20.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:20.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.273428+0000", 2024-09-13T13:17:20.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012742400000000001 2024-09-13T13:17:20.024 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.024 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.025 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.025 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:191396 10.9 e382/377 10:9fe2a3d5:::benchmark_data_smithi052_153703_object64016:head v 382'11888, mlcod=382'11886)", 2024-09-13T13:17:20.025 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:34.022668+0000", 2024-09-13T13:17:20.025 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 45.930539257, 2024-09-13T13:17:20.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.227577587, 2024-09-13T13:17:20.025 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:20.025 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:20.025 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:20.025 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:20.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:20.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022668+0000", 2024-09-13T13:17:20.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:20.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022668+0000", 2024-09-13T13:17:20.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:20.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022669+0000", 2024-09-13T13:17:20.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.519e-06 2024-09-13T13:17:20.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:20.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022677+0000", 2024-09-13T13:17:20.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3889999999999999e-06 2024-09-13T13:17:20.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:20.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022687+0000", 2024-09-13T13:17:20.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0190999999999999e-05 2024-09-13T13:17:20.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:20.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022693+0000", 2024-09-13T13:17:20.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5080000000000002e-06 2024-09-13T13:17:20.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:20.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022747+0000", 2024-09-13T13:17:20.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3820000000000003e-05 2024-09-13T13:17:20.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:20.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022769+0000", 2024-09-13T13:17:20.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1829000000000001e-05 2024-09-13T13:17:20.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:20.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.026913+0000", 2024-09-13T13:17:20.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041442939999999998 2024-09-13T13:17:20.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:20.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250245+0000", 2024-09-13T13:17:20.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.22333203700000001 2024-09-13T13:17:20.029 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.030 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.030 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.030 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:191388 10.c e382/334 10:3518f5d7:::benchmark_data_smithi052_153703_object64008:head v 382'11871, mlcod=382'11868)", 2024-09-13T13:17:20.030 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:34.028033+0000", 2024-09-13T13:17:20.030 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 45.925173794000003, 2024-09-13T13:17:20.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.237613568, 2024-09-13T13:17:20.030 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:20.030 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:20.030 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:20.030 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:20.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:20.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028033+0000", 2024-09-13T13:17:20.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:20.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028033+0000", 2024-09-13T13:17:20.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:20.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028035+0000", 2024-09-13T13:17:20.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8649999999999999e-06 2024-09-13T13:17:20.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:20.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028044+0000", 2024-09-13T13:17:20.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7630000000000006e-06 2024-09-13T13:17:20.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:20.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250215+0000", 2024-09-13T13:17:20.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.222171601 2024-09-13T13:17:20.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:20.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250226+0000", 2024-09-13T13:17:20.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0395e-05 2024-09-13T13:17:20.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:20.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250281+0000", 2024-09-13T13:17:20.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4833999999999997e-05 2024-09-13T13:17:20.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:20.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250301+0000", 2024-09-13T13:17:20.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0452999999999999e-05 2024-09-13T13:17:20.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:20.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.265605+0000", 2024-09-13T13:17:20.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01530372 2024-09-13T13:17:20.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:20.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.265647+0000", 2024-09-13T13:17:20.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1937000000000002e-05 2024-09-13T13:17:20.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.035 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.035 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:191418 10.9 e382/377 10:975b1348:::benchmark_data_smithi052_153703_object64038:head v 382'11889, mlcod=382'11886)", 2024-09-13T13:17:20.035 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:34.041411+0000", 2024-09-13T13:17:20.035 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 45.911796539000001, 2024-09-13T13:17:20.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.214851654, 2024-09-13T13:17:20.035 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:20.035 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:20.036 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:20.036 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:20.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:20.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041411+0000", 2024-09-13T13:17:20.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:20.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041411+0000", 2024-09-13T13:17:20.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:20.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041413+0000", 2024-09-13T13:17:20.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3819999999999998e-06 2024-09-13T13:17:20.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:20.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041424+0000", 2024-09-13T13:17:20.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1031e-05 2024-09-13T13:17:20.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:20.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250212+0000", 2024-09-13T13:17:20.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.208787785 2024-09-13T13:17:20.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:20.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250227+0000", 2024-09-13T13:17:20.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4887e-05 2024-09-13T13:17:20.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:20.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250280+0000", 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3733999999999997e-05 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250340+0000", 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9212000000000002e-05 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.256042+0000", 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057022210000000004 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:20.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.256262+0000", 2024-09-13T13:17:20.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022040199999999999 2024-09-13T13:17:20.040 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.040 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.040 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.040 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203978 10.9 e385/377 10:9bb10788:::benchmark_data_smithi052_153703_object76598:head v 385'12669, mlcod=385'12664)", 2024-09-13T13:17:20.040 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.815708+0000", 2024-09-13T13:17:20.040 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.137498696999998, 2024-09-13T13:17:20.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27159179500000002, 2024-09-13T13:17:20.040 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:20.040 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:20.041 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:20.041 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:20.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:20.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815708+0000", 2024-09-13T13:17:20.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815708+0000", 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815710+0000", 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8840000000000001e-06 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815718+0000", 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:17:20.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815731+0000", 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2853e-05 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815739+0000", 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6550000000000004e-06 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816390+0000", 2024-09-13T13:17:20.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00065173800000000001 2024-09-13T13:17:20.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:20.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816411+0000", 2024-09-13T13:17:20.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0727e-05 2024-09-13T13:17:20.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:20.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087219+0000", 2024-09-13T13:17:20.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27080745899999997 2024-09-13T13:17:20.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:20.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087300+0000", 2024-09-13T13:17:20.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1477999999999995e-05 2024-09-13T13:17:20.045 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.045 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.045 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.046 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203975 10.1 e385/334 10:87de9ad5:::benchmark_data_smithi052_153703_object76595:head v 385'12786, mlcod=385'12784)", 2024-09-13T13:17:20.046 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:17:20.046 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.136425289999998, 2024-09-13T13:17:20.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27055213299999997, 2024-09-13T13:17:20.046 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:20.046 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:20.046 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:20.046 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:20.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:20.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:17:20.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4199999999999995e-07 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816796+0000", 2024-09-13T13:17:20.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3298e-05 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816803+0000", 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8379999999999996e-06 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816808+0000", 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2020000000000003e-06 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:20.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079290+0000", 2024-09-13T13:17:20.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.26248217499999998 2024-09-13T13:17:20.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:20.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079309+0000", 2024-09-13T13:17:20.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8927e-05 2024-09-13T13:17:20.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:20.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087304+0000", 2024-09-13T13:17:20.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079953999999999997 2024-09-13T13:17:20.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087334+0000", 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9550999999999999e-05 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203977 10.c e385/334 10:35b89a8a:::benchmark_data_smithi052_153703_object76597:head v 385'12628, mlcod=385'12626)", 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.816846+0000", 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.136361024000003, 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27022401899999998, 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:20.050 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:20.051 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:20.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:20.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816846+0000", 2024-09-13T13:17:20.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:20.051 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:20.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816846+0000", 2024-09-13T13:17:20.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:20.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816847+0000", 2024-09-13T13:17:20.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9200000000000001e-07 2024-09-13T13:17:20.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:20.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816851+0000", 2024-09-13T13:17:20.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1989999999999999e-06 2024-09-13T13:17:20.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816857+0000", 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9610000000000002e-06 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816862+0000", 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8489999999999997e-06 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816923+0000", 2024-09-13T13:17:20.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1781999999999999e-05 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816940+0000", 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6844999999999999e-05 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087036+0000", 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27009603100000001 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:20.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087070+0000", 2024-09-13T13:17:20.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3760000000000002e-05 2024-09-13T13:17:20.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.056 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.056 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203979 10.c e385/334 10:3e03f71a:::benchmark_data_smithi052_153703_object76599:head v 385'12629, mlcod=385'12626)", 2024-09-13T13:17:20.056 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.820677+0000", 2024-09-13T13:17:20.056 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.132530025999998, 2024-09-13T13:17:20.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.26856709400000001, 2024-09-13T13:17:20.056 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:20.056 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:20.057 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:20.057 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:20.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:20.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820677+0000", 2024-09-13T13:17:20.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:20.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820677+0000", 2024-09-13T13:17:20.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820678+0000", 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4699999999999999e-06 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820696+0000", 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7331999999999999e-05 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079076+0000", 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.25837983199999998 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:20.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079091+0000", 2024-09-13T13:17:20.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5832999999999999e-05 2024-09-13T13:17:20.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:20.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.081590+0000", 2024-09-13T13:17:20.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002498545 2024-09-13T13:17:20.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:20.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.081623+0000", 2024-09-13T13:17:20.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3442000000000003e-05 2024-09-13T13:17:20.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:20.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.089192+0000", 2024-09-13T13:17:20.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075682340000000001 2024-09-13T13:17:20.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:20.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.089244+0000", 2024-09-13T13:17:20.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2405999999999999e-05 2024-09-13T13:17:20.060 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.060 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.060 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.061 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203982 10.9 e385/377 10:90b0e29e:::benchmark_data_smithi052_153703_object76602:head v 385'12670, mlcod=385'12666)", 2024-09-13T13:17:20.061 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.825124+0000", 2024-09-13T13:17:20.061 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.128082956, 2024-09-13T13:17:20.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.26226309199999998, 2024-09-13T13:17:20.061 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:20.061 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:20.061 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:20.061 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:20.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:20.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825124+0000", 2024-09-13T13:17:20.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:20.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825124+0000", 2024-09-13T13:17:20.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:20.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:20.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825126+0000", 2024-09-13T13:17:20.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.894e-06 2024-09-13T13:17:20.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:20.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825134+0000", 2024-09-13T13:17:20.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1960000000000008e-06 2024-09-13T13:17:20.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:20.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079730+0000", 2024-09-13T13:17:20.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.25459629499999997 2024-09-13T13:17:20.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:20.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079741+0000", 2024-09-13T13:17:20.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-09-13T13:17:20.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:20.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079895+0000", 2024-09-13T13:17:20.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154881 2024-09-13T13:17:20.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:20.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079919+0000", 2024-09-13T13:17:20.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3139e-05 2024-09-13T13:17:20.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:20.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087337+0000", 2024-09-13T13:17:20.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.007417964 2024-09-13T13:17:20.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:20.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:20.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:20.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087387+0000", 2024-09-13T13:17:20.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0612999999999998e-05 2024-09-13T13:17:20.066 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.066 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.066 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.066 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:20.066 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:20.066 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:17:20.079 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:20.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:20.181 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:20.218 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:20.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:20.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:20.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:20.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:20 smithi167 ceph-mon[33178]: from='client.16644 v1:172.21.15.52:0/3621480216' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:17:20.324 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:17:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:20.358 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:20.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:20.459 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:20.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:20 smithi052 ceph-mon[29471]: from='client.16644 v1:172.21.15.52:0/3621480216' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:17:20.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:20 smithi052 ceph-mon[36682]: from='client.16644 v1:172.21.15.52:0/3621480216' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:17:20.497 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:20.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:20.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:20.598 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:20.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:20.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:20.731 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:20.764 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:20.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:20.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:20.850 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":808,"stamp":"2024-09-13T13:17:18.167399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1608194611,"num_objects":24730,"num_object_clones":0,"num_object_copies":74190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24730,"num_whiteouts":0,"num_read":1521,"num_read_kb":1380,"num_write":231060,"num_write_kb":8153802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19934,"num_bytes_recovered":1294210562,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78748,"ondisk_log_size":78748,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":8961920,"kb_used_data":4854072,"kb_used_omap":13,"kb_used_meta":4107506,"kb_avail":647254144,"statfs":{"total":671965249536,"available":662788243456,"internally_reserved":0,"allocated":4970569728,"data_stored":4967622999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13997,"internal_metadata":4206086483},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-380370944,"num_objects":-5804,"num_object_clones":0,"num_object_copies":-17412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5804,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5804,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.045298"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":730,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134875+0000","last_change":"2024-09-13T13:16:12.076433+0000","last_active":"2024-09-13T13:17:13.134875+0000","last_peered":"2024-09-13T13:17:13.134875+0000","last_clean":"2024-09-13T13:17:13.134875+0000","last_became_active":"2024-09-13T13:16:12.075424+0000","last_became_peered":"2024-09-13T13:16:12.075424+0000","last_unstale":"2024-09-13T13:17:13.134875+0000","last_undegraded":"2024-09-13T13:17:13.134875+0000","last_fullsized":"2024-09-13T13:17:13.134875+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.368588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.368588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:15.860241+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":221,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135044+0000","last_change":"2024-09-13T13:15:04.344174+0000","last_active":"2024-09-13T13:17:13.135044+0000","last_peered":"2024-09-13T13:17:13.135044+0000","last_clean":"2024-09-13T13:17:13.135044+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:17:13.135044+0000","last_undegraded":"2024-09-13T13:17:13.135044+0000","last_fullsized":"2024-09-13T13:17:13.135044+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:15:04.344092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.344092+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":732,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.137700+0000","last_change":"2024-09-13T13:16:30.163153+0000","last_active":"2024-09-13T13:17:13.137700+0000","last_peered":"2024-09-13T13:17:13.137700+0000","last_clean":"2024-09-13T13:17:13.137700+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:17:13.137700+0000","last_undegraded":"2024-09-13T13:17:13.137700+0000","last_fullsized":"2024-09-13T13:17:13.137700+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:30.163077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:30.163077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:54.764628+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":219,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131254+0000","last_change":"2024-09-13T13:15:51.244938+0000","last_active":"2024-09-13T13:17:13.131254+0000","last_peered":"2024-09-13T13:17:13.131254+0000","last_clean":"2024-09-13T13:17:13.131254+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:17:13.131254+0000","last_undegraded":"2024-09-13T13:17:13.131254+0000","last_fullsized":"2024-09-13T13:17:13.131254+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:19.110420+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":734,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.163016+0000","last_change":"2024-09-13T13:11:47.852762+0000","last_active":"2024-09-13T13:17:17.163016+0000","last_peered":"2024-09-13T13:17:17.163016+0000","last_clean":"2024-09-13T13:17:17.163016+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:17:17.163016+0000","last_undegraded":"2024-09-13T13:17:17.163016+0000","last_fullsized":"2024-09-13T13:17:17.163016+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:19.083537+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":771,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.167192+0000","last_change":"2024-09-13T13:17:08.938797+0000","last_active":"2024-09-13T13:17:17.167192+0000","last_peered":"2024-09-13T13:17:17.167192+0000","last_clean":"2024-09-13T13:17:17.167192+0000","last_became_active":"2024-09-13T13:16:12.069442+0000","last_became_peered":"2024-09-13T13:16:12.069442+0000","last_unstale":"2024-09-13T13:17:17.167192+0000","last_undegraded":"2024-09-13T13:17:17.167192+0000","last_fullsized":"2024-09-13T13:17:17.167192+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:08.938750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:08.938750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:13.091171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":829,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.162144+0000","last_change":"2024-09-13T13:17:11.044562+0000","last_active":"2024-09-13T13:17:17.162144+0000","last_peered":"2024-09-13T13:17:17.162144+0000","last_clean":"2024-09-13T13:17:17.162144+0000","last_became_active":"2024-09-13T13:16:12.074619+0000","last_became_peered":"2024-09-13T13:16:12.074619+0000","last_unstale":"2024-09-13T13:17:17.162144+0000","last_undegraded":"2024-09-13T13:17:17.162144+0000","last_fullsized":"2024-09-13T13:17:17.162144+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:17:11.044472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:11.044472+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:18.681524+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":203,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134963+0000","last_change":"2024-09-13T13:12:52.997865+0000","last_active":"2024-09-13T13:17:13.134963+0000","last_peered":"2024-09-13T13:17:13.134963+0000","last_clean":"2024-09-13T13:17:13.134963+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:17:13.134963+0000","last_undegraded":"2024-09-13T13:17:13.134963+0000","last_fullsized":"2024-09-13T13:17:13.134963+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:17.305608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:17.305608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:12.478035+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":794,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129425+0000","last_change":"2024-09-13T13:13:41.518378+0000","last_active":"2024-09-13T13:17:13.129425+0000","last_peered":"2024-09-13T13:17:13.129425+0000","last_clean":"2024-09-13T13:17:13.129425+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:17:13.129425+0000","last_undegraded":"2024-09-13T13:17:13.129425+0000","last_fullsized":"2024-09-13T13:17:13.129425+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:41.518322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.518322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":221,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131332+0000","last_change":"2024-09-13T13:17:06.406253+0000","last_active":"2024-09-13T13:17:13.131332+0000","last_peered":"2024-09-13T13:17:13.131332+0000","last_clean":"2024-09-13T13:17:13.131332+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:17:13.131332+0000","last_undegraded":"2024-09-13T13:17:13.131332+0000","last_fullsized":"2024-09-13T13:17:13.131332+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:06.406179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:06.406179+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:20.094337+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":731,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129488+0000","last_change":"2024-09-13T13:13:43.501368+0000","last_active":"2024-09-13T13:17:13.129488+0000","last_peered":"2024-09-13T13:17:13.129488+0000","last_clean":"2024-09-13T13:17:13.129488+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:17:13.129488+0000","last_undegraded":"2024-09-13T13:17:13.129488+0000","last_fullsized":"2024-09-13T13:17:13.129488+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:43.501206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:43.501206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":734,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134339+0000","last_change":"2024-09-13T13:15:25.628234+0000","last_active":"2024-09-13T13:17:13.134339+0000","last_peered":"2024-09-13T13:17:13.134339+0000","last_clean":"2024-09-13T13:17:13.134339+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:17:13.134339+0000","last_undegraded":"2024-09-13T13:17:13.134339+0000","last_fullsized":"2024-09-13T13:17:13.134339+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:25.628156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:25.628156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":769,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129095+0000","last_change":"2024-09-13T13:17:02.211078+0000","last_active":"2024-09-13T13:17:13.129095+0000","last_peered":"2024-09-13T13:17:13.129095+0000","last_clean":"2024-09-13T13:17:13.129095+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:17:13.129095+0000","last_undegraded":"2024-09-13T13:17:13.129095+0000","last_fullsized":"2024-09-13T13:17:13.129095+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:02.211007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.211007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:19.487607+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":217,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134564+0000","last_change":"2024-09-13T13:15:03.348187+0000","last_active":"2024-09-13T13:17:13.134564+0000","last_peered":"2024-09-13T13:17:13.134564+0000","last_clean":"2024-09-13T13:17:13.134564+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:17:13.134564+0000","last_undegraded":"2024-09-13T13:17:13.134564+0000","last_fullsized":"2024-09-13T13:17:13.134564+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:15:03.348115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.348115+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":752,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.132534+0000","last_change":"2024-09-13T13:13:54.122282+0000","last_active":"2024-09-13T13:17:13.132534+0000","last_peered":"2024-09-13T13:17:13.132534+0000","last_clean":"2024-09-13T13:17:13.132534+0000","last_became_active":"2024-09-13T13:12:52.999970+0000","last_became_peered":"2024-09-13T13:12:52.999970+0000","last_unstale":"2024-09-13T13:17:13.132534+0000","last_undegraded":"2024-09-13T13:17:13.132534+0000","last_fullsized":"2024-09-13T13:17:13.132534+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.122222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.122222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:17.799545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":703,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129896+0000","last_change":"2024-09-13T13:13:02.818644+0000","last_active":"2024-09-13T13:17:13.129896+0000","last_peered":"2024-09-13T13:17:13.129896+0000","last_clean":"2024-09-13T13:17:13.129896+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:17:13.129896+0000","last_undegraded":"2024-09-13T13:17:13.129896+0000","last_fullsized":"2024-09-13T13:17:13.129896+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:02.818543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:02.818543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":216,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.133584+0000","last_change":"2024-09-13T13:17:07.444148+0000","last_active":"2024-09-13T13:17:13.133584+0000","last_peered":"2024-09-13T13:17:13.133584+0000","last_clean":"2024-09-13T13:17:13.133584+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:17:13.133584+0000","last_undegraded":"2024-09-13T13:17:13.133584+0000","last_fullsized":"2024-09-13T13:17:13.133584+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:07.443951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:07.443951+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:25.224293+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":760,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.136386+0000","last_change":"2024-09-13T13:15:52.658477+0000","last_active":"2024-09-13T13:17:13.136386+0000","last_peered":"2024-09-13T13:17:13.136386+0000","last_clean":"2024-09-13T13:17:13.136386+0000","last_became_active":"2024-09-13T13:12:53.008505+0000","last_became_peered":"2024-09-13T13:12:53.008505+0000","last_unstale":"2024-09-13T13:17:13.136386+0000","last_undegraded":"2024-09-13T13:17:13.136386+0000","last_fullsized":"2024-09-13T13:17:13.136386+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:52.658410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:52.658410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":578,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.138190+0000","last_change":"2024-09-13T13:16:12.089425+0000","last_active":"2024-09-13T13:17:13.138190+0000","last_peered":"2024-09-13T13:17:13.138190+0000","last_clean":"2024-09-13T13:17:13.138190+0000","last_became_active":"2024-09-13T13:16:12.073838+0000","last_became_peered":"2024-09-13T13:16:12.073838+0000","last_unstale":"2024-09-13T13:17:13.138190+0000","last_undegraded":"2024-09-13T13:17:13.138190+0000","last_fullsized":"2024-09-13T13:17:13.138190+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:12:40.125545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.125545+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:18.331517+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":762,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.138223+0000","last_change":"2024-09-13T13:16:54.229157+0000","last_active":"2024-09-13T13:17:13.138223+0000","last_peered":"2024-09-13T13:17:13.138223+0000","last_clean":"2024-09-13T13:17:13.138223+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:17:13.138223+0000","last_undegraded":"2024-09-13T13:17:13.138223+0000","last_fullsized":"2024-09-13T13:17:13.138223+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:54.229075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:54.229075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:13.235712+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":735,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.167455+0000","last_change":"2024-09-13T13:13:03.150524+0000","last_active":"2024-09-13T13:17:17.167455+0000","last_peered":"2024-09-13T13:17:17.167455+0000","last_clean":"2024-09-13T13:17:17.167455+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:17:17.167455+0000","last_undegraded":"2024-09-13T13:17:17.167455+0000","last_fullsized":"2024-09-13T13:17:17.167455+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:03.150442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.150442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:38.978974+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":226,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131100+0000","last_change":"2024-09-13T13:16:12.077830+0000","last_active":"2024-09-13T13:17:13.131100+0000","last_peered":"2024-09-13T13:17:13.131100+0000","last_clean":"2024-09-13T13:17:13.131100+0000","last_became_active":"2024-09-13T13:16:12.077185+0000","last_became_peered":"2024-09-13T13:16:12.077185+0000","last_unstale":"2024-09-13T13:17:13.131100+0000","last_undegraded":"2024-09-13T13:17:13.131100+0000","last_fullsized":"2024-09-13T13:17:13.131100+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:01.242234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:01.242234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:21.091214+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":801,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.168695+0000","last_change":"2024-09-13T13:13:41.164741+0000","last_active":"2024-09-13T13:17:17.168695+0000","last_peered":"2024-09-13T13:17:17.168695+0000","last_clean":"2024-09-13T13:17:17.168695+0000","last_became_active":"2024-09-13T13:12:52.953819+0000","last_became_peered":"2024-09-13T13:12:52.953819+0000","last_unstale":"2024-09-13T13:17:17.168695+0000","last_undegraded":"2024-09-13T13:17:17.168695+0000","last_fullsized":"2024-09-13T13:17:17.168695+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:13:41.164656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.164656+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:50.985657+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":220,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131202+0000","last_change":"2024-09-13T13:16:12.078175+0000","last_active":"2024-09-13T13:17:13.131202+0000","last_peered":"2024-09-13T13:17:13.131202+0000","last_clean":"2024-09-13T13:17:13.131202+0000","last_became_active":"2024-09-13T13:16:12.077603+0000","last_became_peered":"2024-09-13T13:16:12.077603+0000","last_unstale":"2024-09-13T13:17:13.131202+0000","last_undegraded":"2024-09-13T13:17:13.131202+0000","last_fullsized":"2024-09-13T13:17:13.131202+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:29.208576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:29.208576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":257,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.128853+0000","last_change":"2024-09-13T13:17:05.207522+0000","last_active":"2024-09-13T13:17:13.128853+0000","last_peered":"2024-09-13T13:17:13.128853+0000","last_clean":"2024-09-13T13:17:13.128853+0000","last_became_active":"2024-09-13T13:16:12.071603+0000","last_became_peered":"2024-09-13T13:16:12.071603+0000","last_unstale":"2024-09-13T13:17:13.128853+0000","last_undegraded":"2024-09-13T13:17:13.128853+0000","last_fullsized":"2024-09-13T13:17:13.128853+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:05.207445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.207445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:31.464107+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":763,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.137722+0000","last_change":"2024-09-13T13:17:01.162092+0000","last_active":"2024-09-13T13:17:13.137722+0000","last_peered":"2024-09-13T13:17:13.137722+0000","last_clean":"2024-09-13T13:17:13.137722+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:17:13.137722+0000","last_undegraded":"2024-09-13T13:17:13.137722+0000","last_fullsized":"2024-09-13T13:17:13.137722+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:17:01.162018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:01.162018+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:24.126509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":782,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135412+0000","last_change":"2024-09-13T13:15:50.693943+0000","last_active":"2024-09-13T13:17:13.135412+0000","last_peered":"2024-09-13T13:17:13.135412+0000","last_clean":"2024-09-13T13:17:13.135412+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:17:13.135412+0000","last_undegraded":"2024-09-13T13:17:13.135412+0000","last_fullsized":"2024-09-13T13:17:13.135412+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:15:50.693753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.693753+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"306'12","reported_seq":917,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.161512+0000","last_change":"2024-09-13T13:14:21.530680+0000","last_active":"2024-09-13T13:17:16.161512+0000","last_peered":"2024-09-13T13:17:16.161512+0000","last_clean":"2024-09-13T13:17:16.161512+0000","last_became_active":"2024-09-13T13:12:52.947886+0000","last_became_peered":"2024-09-13T13:12:52.947886+0000","last_unstale":"2024-09-13T13:17:16.161512+0000","last_undegraded":"2024-09-13T13:17:16.161512+0000","last_fullsized":"2024-09-13T13:17:16.161512+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'12","last_scrub_stamp":"2024-09-13T13:14:21.530612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:21.530612+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:41.202009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"401'14295","reported_seq":14476,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.274384+0000","last_change":"2024-09-13T13:16:06.040203+0000","last_active":"2024-09-13T13:17:13.274384+0000","last_peered":"2024-09-13T13:17:13.274384+0000","last_clean":"2024-09-13T13:17:13.274384+0000","last_became_active":"2024-09-13T13:13:59.149215+0000","last_became_peered":"2024-09-13T13:13:59.149215+0000","last_unstale":"2024-09-13T13:17:13.274384+0000","last_undegraded":"2024-09-13T13:17:13.274384+0000","last_fullsized":"2024-09-13T13:17:13.274384+0000","mapping_epoch":334,"log_start":"372'9986","ondisk_log_start":"372'9986","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":3199,"log_size":4309,"log_dups_size":0,"ondisk_log_size":4309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110034944,"num_objects":1679,"num_object_clones":0,"num_object_copies":5037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14295,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":834,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.138999+0000","last_change":"2024-09-13T13:16:56.133395+0000","last_active":"2024-09-13T13:17:13.138999+0000","last_peered":"2024-09-13T13:17:13.138999+0000","last_clean":"2024-09-13T13:17:13.138999+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:17:13.138999+0000","last_undegraded":"2024-09-13T13:17:13.138999+0000","last_fullsized":"2024-09-13T13:17:13.138999+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:16:56.133314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:56.133314+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:01.815453+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":768,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.162181+0000","last_change":"2024-09-13T13:16:58.980151+0000","last_active":"2024-09-13T13:17:17.162181+0000","last_peered":"2024-09-13T13:17:17.162181+0000","last_clean":"2024-09-13T13:17:17.162181+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:17:17.162181+0000","last_undegraded":"2024-09-13T13:17:17.162181+0000","last_fullsized":"2024-09-13T13:17:17.162181+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:58.980098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.980098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:19.930402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":268,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.133691+0000","last_change":"2024-09-13T13:15:32.212185+0000","last_active":"2024-09-13T13:17:13.133691+0000","last_peered":"2024-09-13T13:17:13.133691+0000","last_clean":"2024-09-13T13:17:13.133691+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:17:13.133691+0000","last_undegraded":"2024-09-13T13:17:13.133691+0000","last_fullsized":"2024-09-13T13:17:13.133691+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:15:32.212101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.212101+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":736,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.128890+0000","last_change":"2024-09-13T13:13:57.546725+0000","last_active":"2024-09-13T13:17:13.128890+0000","last_peered":"2024-09-13T13:17:13.128890+0000","last_clean":"2024-09-13T13:17:13.128890+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:17:13.128890+0000","last_undegraded":"2024-09-13T13:17:13.128890+0000","last_fullsized":"2024-09-13T13:17:13.128890+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.546671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.546671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"401'14289","reported_seq":22766,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.273829+0000","last_change":"2024-09-13T13:16:50.311735+0000","last_active":"2024-09-13T13:17:13.273829+0000","last_peered":"2024-09-13T13:17:13.273829+0000","last_clean":"2024-09-13T13:17:13.273829+0000","last_became_active":"2024-09-13T13:16:50.276697+0000","last_became_peered":"2024-09-13T13:16:50.276697+0000","last_unstale":"2024-09-13T13:17:13.273829+0000","last_undegraded":"2024-09-13T13:17:13.273829+0000","last_fullsized":"2024-09-13T13:17:13.273829+0000","mapping_epoch":389,"log_start":"372'9910","ondisk_log_start":"372'9910","created":334,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111214592,"num_objects":1697,"num_object_clones":0,"num_object_copies":5091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14289,"num_write_kb":511552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4157,"num_bytes_recovered":272433152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":775,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.161888+0000","last_change":"2024-09-13T13:16:05.289274+0000","last_active":"2024-09-13T13:17:17.161888+0000","last_peered":"2024-09-13T13:17:17.161888+0000","last_clean":"2024-09-13T13:17:17.161888+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:17:17.161888+0000","last_undegraded":"2024-09-13T13:17:17.161888+0000","last_fullsized":"2024-09-13T13:17:17.161888+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:16:05.289187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:05.289187+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:35.033329+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":742,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.167811+0000","last_change":"2024-09-13T13:16:26.098415+0000","last_active":"2024-09-13T13:17:17.167811+0000","last_peered":"2024-09-13T13:17:17.167811+0000","last_clean":"2024-09-13T13:17:17.167811+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:17:17.167811+0000","last_undegraded":"2024-09-13T13:17:17.167811+0000","last_fullsized":"2024-09-13T13:17:17.167811+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:16:26.098365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:26.098365+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:54.098482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":212,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131737+0000","last_change":"2024-09-13T13:15:00.323883+0000","last_active":"2024-09-13T13:17:13.131737+0000","last_peered":"2024-09-13T13:17:13.131737+0000","last_clean":"2024-09-13T13:17:13.131737+0000","last_became_active":"2024-09-13T13:12:53.006199+0000","last_became_peered":"2024-09-13T13:12:53.006199+0000","last_unstale":"2024-09-13T13:17:13.131737+0000","last_undegraded":"2024-09-13T13:17:13.131737+0000","last_fullsized":"2024-09-13T13:17:13.131737+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.323784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.323784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":227,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131415+0000","last_change":"2024-09-13T13:16:12.075699+0000","last_active":"2024-09-13T13:17:13.131415+0000","last_peered":"2024-09-13T13:17:13.131415+0000","last_clean":"2024-09-13T13:17:13.131415+0000","last_became_active":"2024-09-13T13:16:12.074280+0000","last_became_peered":"2024-09-13T13:16:12.074280+0000","last_unstale":"2024-09-13T13:17:13.131415+0000","last_undegraded":"2024-09-13T13:17:13.131415+0000","last_fullsized":"2024-09-13T13:17:13.131415+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:56.333778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:56.333778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"402'14420","reported_seq":20255,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.360500+0000","last_change":"2024-09-13T13:17:03.085482+0000","last_active":"2024-09-13T13:17:16.360500+0000","last_peered":"2024-09-13T13:17:16.360500+0000","last_clean":"2024-09-13T13:17:16.360500+0000","last_became_active":"2024-09-13T13:17:03.059883+0000","last_became_peered":"2024-09-13T13:17:03.059883+0000","last_unstale":"2024-09-13T13:17:16.360500+0000","last_undegraded":"2024-09-13T13:17:16.360500+0000","last_fullsized":"2024-09-13T13:17:16.360500+0000","mapping_epoch":396,"log_start":"371'9168","ondisk_log_start":"371'9168","created":334,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5252,"log_dups_size":0,"ondisk_log_size":5252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99221504,"num_objects":1514,"num_object_clones":0,"num_object_copies":4542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14420,"num_write_kb":509888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2833,"num_bytes_recovered":185663488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":228,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131823+0000","last_change":"2024-09-13T13:13:50.716312+0000","last_active":"2024-09-13T13:17:13.131823+0000","last_peered":"2024-09-13T13:17:13.131823+0000","last_clean":"2024-09-13T13:17:13.131823+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:17:13.131823+0000","last_undegraded":"2024-09-13T13:17:13.131823+0000","last_fullsized":"2024-09-13T13:17:13.131823+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:13:50.716243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.716243+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":746,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135537+0000","last_change":"2024-09-13T13:17:02.704294+0000","last_active":"2024-09-13T13:17:13.135537+0000","last_peered":"2024-09-13T13:17:13.135537+0000","last_clean":"2024-09-13T13:17:13.135537+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:17:13.135537+0000","last_undegraded":"2024-09-13T13:17:13.135537+0000","last_fullsized":"2024-09-13T13:17:13.135537+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:17:02.704233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.704233+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:20.770762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":964,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.436041+0000","last_change":"2024-09-13T13:16:24.110415+0000","last_active":"2024-09-13T13:17:17.436041+0000","last_peered":"2024-09-13T13:17:17.436041+0000","last_clean":"2024-09-13T13:17:17.436041+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:17:17.436041+0000","last_undegraded":"2024-09-13T13:17:17.436041+0000","last_fullsized":"2024-09-13T13:17:17.436041+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:16:24.110369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:24.110369+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:42.263184+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":371,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.161752+0000","last_change":"2024-09-13T13:16:26.258774+0000","last_active":"2024-09-13T13:17:16.161752+0000","last_peered":"2024-09-13T13:17:16.161752+0000","last_clean":"2024-09-13T13:17:16.161752+0000","last_became_active":"2024-09-13T13:16:12.071629+0000","last_became_peered":"2024-09-13T13:16:12.071629+0000","last_unstale":"2024-09-13T13:17:16.161752+0000","last_undegraded":"2024-09-13T13:17:16.161752+0000","last_fullsized":"2024-09-13T13:17:16.161752+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:26.258619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:26.258619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:34.080748+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"401'14143","reported_seq":14288,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.269637+0000","last_change":"2024-09-13T13:13:59.149331+0000","last_active":"2024-09-13T13:17:13.269637+0000","last_peered":"2024-09-13T13:17:13.269637+0000","last_clean":"2024-09-13T13:17:13.269637+0000","last_became_active":"2024-09-13T13:13:59.149046+0000","last_became_peered":"2024-09-13T13:13:59.149046+0000","last_unstale":"2024-09-13T13:17:13.269637+0000","last_undegraded":"2024-09-13T13:17:13.269637+0000","last_fullsized":"2024-09-13T13:17:13.269637+0000","mapping_epoch":334,"log_start":"372'9842","ondisk_log_start":"372'9842","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4301,"log_dups_size":0,"ondisk_log_size":4301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104792064,"num_objects":1599,"num_object_clones":0,"num_object_copies":4797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14143,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":797,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.139697+0000","last_change":"2024-09-13T13:12:53.016127+0000","last_active":"2024-09-13T13:17:13.139697+0000","last_peered":"2024-09-13T13:17:13.139697+0000","last_clean":"2024-09-13T13:17:13.139697+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:17:13.139697+0000","last_undegraded":"2024-09-13T13:17:13.139697+0000","last_fullsized":"2024-09-13T13:17:13.139697+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:41.316459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":771,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129517+0000","last_change":"2024-09-13T13:16:12.067616+0000","last_active":"2024-09-13T13:17:13.129517+0000","last_peered":"2024-09-13T13:17:13.129517+0000","last_clean":"2024-09-13T13:17:13.129517+0000","last_became_active":"2024-09-13T13:16:12.067157+0000","last_became_peered":"2024-09-13T13:16:12.067157+0000","last_unstale":"2024-09-13T13:17:13.129517+0000","last_undegraded":"2024-09-13T13:17:13.129517+0000","last_fullsized":"2024-09-13T13:17:13.129517+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:04.755828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:04.755828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:15.005053+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":940,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135704+0000","last_change":"2024-09-13T13:15:51.663010+0000","last_active":"2024-09-13T13:17:13.135704+0000","last_peered":"2024-09-13T13:17:13.135704+0000","last_clean":"2024-09-13T13:17:13.135704+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:17:13.135704+0000","last_undegraded":"2024-09-13T13:17:13.135704+0000","last_fullsized":"2024-09-13T13:17:13.135704+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:15:51.662657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.662657+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":759,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134684+0000","last_change":"2024-09-13T13:16:12.077285+0000","last_active":"2024-09-13T13:17:13.134684+0000","last_peered":"2024-09-13T13:17:13.134684+0000","last_clean":"2024-09-13T13:17:13.134684+0000","last_became_active":"2024-09-13T13:16:12.072760+0000","last_became_peered":"2024-09-13T13:16:12.072760+0000","last_unstale":"2024-09-13T13:17:13.134684+0000","last_undegraded":"2024-09-13T13:17:13.134684+0000","last_fullsized":"2024-09-13T13:17:13.134684+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:26.667455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:26.667455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"402'14666","reported_seq":14948,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.288662+0000","last_change":"2024-09-13T13:16:31.945928+0000","last_active":"2024-09-13T13:17:16.288662+0000","last_peered":"2024-09-13T13:17:16.288662+0000","last_clean":"2024-09-13T13:17:16.288662+0000","last_became_active":"2024-09-13T13:13:58.965950+0000","last_became_peered":"2024-09-13T13:13:58.965950+0000","last_unstale":"2024-09-13T13:17:16.288662+0000","last_undegraded":"2024-09-13T13:17:16.288662+0000","last_fullsized":"2024-09-13T13:17:16.288662+0000","mapping_epoch":334,"log_start":"371'9468","ondisk_log_start":"371'9468","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'11986","last_scrub_stamp":"2024-09-13T13:16:31.945684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:16:31.945684+0000","objects_scrubbed":4166,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:51.306557+0000","scrub_duration":1677,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94044184,"num_objects":1436,"num_object_clones":0,"num_object_copies":4308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1436,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14666,"num_write_kb":515201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":802,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135884+0000","last_change":"2024-09-13T13:17:05.726065+0000","last_active":"2024-09-13T13:17:13.135884+0000","last_peered":"2024-09-13T13:17:13.135884+0000","last_clean":"2024-09-13T13:17:13.135884+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:17:13.135884+0000","last_undegraded":"2024-09-13T13:17:13.135884+0000","last_fullsized":"2024-09-13T13:17:13.135884+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:17:05.726003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.726003+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:30.917098+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":740,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.138387+0000","last_change":"2024-09-13T13:16:36.283708+0000","last_active":"2024-09-13T13:17:13.138387+0000","last_peered":"2024-09-13T13:17:13.138387+0000","last_clean":"2024-09-13T13:17:13.138387+0000","last_became_active":"2024-09-13T13:16:12.074272+0000","last_became_peered":"2024-09-13T13:16:12.074272+0000","last_unstale":"2024-09-13T13:17:13.138387+0000","last_undegraded":"2024-09-13T13:17:13.138387+0000","last_fullsized":"2024-09-13T13:17:13.138387+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:36.283624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:36.283624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:05.724739+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":711,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.161875+0000","last_change":"2024-09-13T13:16:24.285416+0000","last_active":"2024-09-13T13:17:16.161875+0000","last_peered":"2024-09-13T13:17:16.161875+0000","last_clean":"2024-09-13T13:17:16.161875+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:17:16.161875+0000","last_undegraded":"2024-09-13T13:17:16.161875+0000","last_fullsized":"2024-09-13T13:17:16.161875+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:24.285346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:24.285346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:43.854597+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":747,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.137550+0000","last_change":"2024-09-13T13:16:29.167358+0000","last_active":"2024-09-13T13:17:13.137550+0000","last_peered":"2024-09-13T13:17:13.137550+0000","last_clean":"2024-09-13T13:17:13.137550+0000","last_became_active":"2024-09-13T13:16:12.067708+0000","last_became_peered":"2024-09-13T13:16:12.067708+0000","last_unstale":"2024-09-13T13:17:13.137550+0000","last_undegraded":"2024-09-13T13:17:13.137550+0000","last_fullsized":"2024-09-13T13:17:13.137550+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:29.167280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:29.167280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:44.607802+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"402'14468","reported_seq":14616,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.358988+0000","last_change":"2024-09-13T13:13:58.966631+0000","last_active":"2024-09-13T13:17:16.358988+0000","last_peered":"2024-09-13T13:17:16.358988+0000","last_clean":"2024-09-13T13:17:16.358988+0000","last_became_active":"2024-09-13T13:13:58.965789+0000","last_became_peered":"2024-09-13T13:13:58.965789+0000","last_unstale":"2024-09-13T13:17:16.358988+0000","last_undegraded":"2024-09-13T13:17:16.358988+0000","last_fullsized":"2024-09-13T13:17:16.358988+0000","mapping_epoch":334,"log_start":"371'9263","ondisk_log_start":"371'9263","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5205,"log_dups_size":0,"ondisk_log_size":5205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:19.436994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94109696,"num_objects":1436,"num_object_clones":0,"num_object_copies":4308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14468,"num_write_kb":508928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"377'41","reported_seq":1035,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.138293+0000","last_change":"2024-09-13T13:16:58.213866+0000","last_active":"2024-09-13T13:17:13.138293+0000","last_peered":"2024-09-13T13:17:13.138293+0000","last_clean":"2024-09-13T13:17:13.138293+0000","last_became_active":"2024-09-13T13:16:12.072827+0000","last_became_peered":"2024-09-13T13:16:12.072827+0000","last_unstale":"2024-09-13T13:17:13.138293+0000","last_undegraded":"2024-09-13T13:17:13.138293+0000","last_fullsized":"2024-09-13T13:17:13.138293+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'41","last_scrub_stamp":"2024-09-13T13:16:58.213805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.213805+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:10.010285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":220,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134653+0000","last_change":"2024-09-13T13:14:14.899463+0000","last_active":"2024-09-13T13:17:13.134653+0000","last_peered":"2024-09-13T13:17:13.134653+0000","last_clean":"2024-09-13T13:17:13.134653+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:17:13.134653+0000","last_undegraded":"2024-09-13T13:17:13.134653+0000","last_fullsized":"2024-09-13T13:17:13.134653+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:14.899378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:14.899378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":751,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.130075+0000","last_change":"2024-09-13T13:15:50.618476+0000","last_active":"2024-09-13T13:17:13.130075+0000","last_peered":"2024-09-13T13:17:13.130075+0000","last_clean":"2024-09-13T13:17:13.130075+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:17:13.130075+0000","last_undegraded":"2024-09-13T13:17:13.130075+0000","last_fullsized":"2024-09-13T13:17:13.130075+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:15:50.618426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.618426+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":753,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.136685+0000","last_change":"2024-09-13T13:14:59.670853+0000","last_active":"2024-09-13T13:17:13.136685+0000","last_peered":"2024-09-13T13:17:13.136685+0000","last_clean":"2024-09-13T13:17:13.136685+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:17:13.136685+0000","last_undegraded":"2024-09-13T13:17:13.136685+0000","last_fullsized":"2024-09-13T13:17:13.136685+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.670779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.670779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":754,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.166623+0000","last_change":"2024-09-13T13:17:05.016876+0000","last_active":"2024-09-13T13:17:17.166623+0000","last_peered":"2024-09-13T13:17:17.166623+0000","last_clean":"2024-09-13T13:17:17.166623+0000","last_became_active":"2024-09-13T13:12:52.948742+0000","last_became_peered":"2024-09-13T13:12:52.948742+0000","last_unstale":"2024-09-13T13:17:17.166623+0000","last_undegraded":"2024-09-13T13:17:17.166623+0000","last_fullsized":"2024-09-13T13:17:17.166623+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:05.016806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.016806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:07.820409+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"401'14141","reported_seq":22383,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.492606+0000","last_change":"2024-09-13T13:16:48.283334+0000","last_active":"2024-09-13T13:17:13.492606+0000","last_peered":"2024-09-13T13:17:13.492606+0000","last_clean":"2024-09-13T13:17:13.492606+0000","last_became_active":"2024-09-13T13:16:48.248459+0000","last_became_peered":"2024-09-13T13:16:48.248459+0000","last_unstale":"2024-09-13T13:17:13.492606+0000","last_undegraded":"2024-09-13T13:17:13.492606+0000","last_fullsized":"2024-09-13T13:17:13.492606+0000","mapping_epoch":387,"log_start":"371'9094","ondisk_log_start":"371'9094","created":334,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5047,"log_dups_size":0,"ondisk_log_size":5047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104398848,"num_objects":1593,"num_object_clones":0,"num_object_copies":4779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14141,"num_write_kb":503488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4040,"num_bytes_recovered":264765440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":762,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.139088+0000","last_change":"2024-09-13T13:15:33.299837+0000","last_active":"2024-09-13T13:17:13.139088+0000","last_peered":"2024-09-13T13:17:13.139088+0000","last_clean":"2024-09-13T13:17:13.139088+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:17:13.139088+0000","last_undegraded":"2024-09-13T13:17:13.139088+0000","last_fullsized":"2024-09-13T13:17:13.139088+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'17","last_scrub_stamp":"2024-09-13T13:15:33.299746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.299746+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":760,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.137552+0000","last_change":"2024-09-13T13:16:12.072959+0000","last_active":"2024-09-13T13:17:13.137552+0000","last_peered":"2024-09-13T13:17:13.137552+0000","last_clean":"2024-09-13T13:17:13.137552+0000","last_became_active":"2024-09-13T13:16:12.072367+0000","last_became_peered":"2024-09-13T13:16:12.072367+0000","last_unstale":"2024-09-13T13:17:13.137552+0000","last_undegraded":"2024-09-13T13:17:13.137552+0000","last_fullsized":"2024-09-13T13:17:13.137552+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:17.676339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:17.676339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:21.245559+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":81,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.162249+0000","last_change":"2024-09-13T13:16:12.092137+0000","last_active":"2024-09-13T13:17:17.162249+0000","last_peered":"2024-09-13T13:17:17.162249+0000","last_clean":"2024-09-13T13:17:17.162249+0000","last_became_active":"2024-09-13T13:16:12.074083+0000","last_became_peered":"2024-09-13T13:16:12.074083+0000","last_unstale":"2024-09-13T13:17:17.162249+0000","last_undegraded":"2024-09-13T13:17:17.162249+0000","last_fullsized":"2024-09-13T13:17:17.162249+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:13:14.560555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:14.560555+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:17.116977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":62,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129779+0000","last_change":"2024-09-13T13:16:12.070341+0000","last_active":"2024-09-13T13:17:13.129779+0000","last_peered":"2024-09-13T13:17:13.129779+0000","last_clean":"2024-09-13T13:17:13.129779+0000","last_became_active":"2024-09-13T13:16:12.069747+0000","last_became_peered":"2024-09-13T13:16:12.069747+0000","last_unstale":"2024-09-13T13:17:13.129779+0000","last_undegraded":"2024-09-13T13:17:13.129779+0000","last_fullsized":"2024-09-13T13:17:13.129779+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:59.809055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:59.809055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:24.864450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":356,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.162049+0000","last_change":"2024-09-13T13:12:41.332308+0000","last_active":"2024-09-13T13:17:16.162049+0000","last_peered":"2024-09-13T13:17:16.162049+0000","last_clean":"2024-09-13T13:17:16.162049+0000","last_became_active":"2024-09-13T13:12:40.120242+0000","last_became_peered":"2024-09-13T13:12:40.120242+0000","last_unstale":"2024-09-13T13:17:16.162049+0000","last_undegraded":"2024-09-13T13:17:16.162049+0000","last_fullsized":"2024-09-13T13:17:16.162049+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:41.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:41.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"401'14569","reported_seq":20760,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:14.612054+0000","last_change":"2024-09-13T13:17:08.294281+0000","last_active":"2024-09-13T13:17:14.612054+0000","last_peered":"2024-09-13T13:17:14.612054+0000","last_clean":"2024-09-13T13:17:14.612054+0000","last_became_active":"2024-09-13T13:16:59.041685+0000","last_became_peered":"2024-09-13T13:16:59.041685+0000","last_unstale":"2024-09-13T13:17:14.612054+0000","last_undegraded":"2024-09-13T13:17:14.612054+0000","last_fullsized":"2024-09-13T13:17:14.612054+0000","mapping_epoch":393,"log_start":"372'9996","ondisk_log_start":"372'9996","created":334,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14102","last_scrub_stamp":"2024-09-13T13:17:08.294178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:17:08.294178+0000","objects_scrubbed":2017,"log_size":4573,"log_dups_size":0,"ondisk_log_size":4573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:13.042889+0000","scrub_duration":632,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99942400,"num_objects":1525,"num_object_clones":0,"num_object_copies":4575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14569,"num_write_kb":515008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2998,"num_bytes_recovered":196476928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":792,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135445+0000","last_change":"2024-09-13T13:16:30.819216+0000","last_active":"2024-09-13T13:17:13.135445+0000","last_peered":"2024-09-13T13:17:13.135445+0000","last_clean":"2024-09-13T13:17:13.135445+0000","last_became_active":"2024-09-13T13:16:12.074949+0000","last_became_peered":"2024-09-13T13:16:12.074949+0000","last_unstale":"2024-09-13T13:17:13.135445+0000","last_undegraded":"2024-09-13T13:17:13.135445+0000","last_fullsized":"2024-09-13T13:17:13.135445+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:16:30.819129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:30.819129+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:50.347431+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":753,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135115+0000","last_change":"2024-09-13T13:15:30.590673+0000","last_active":"2024-09-13T13:17:13.135115+0000","last_peered":"2024-09-13T13:17:13.135115+0000","last_clean":"2024-09-13T13:17:13.135115+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:17:13.135115+0000","last_undegraded":"2024-09-13T13:17:13.135115+0000","last_fullsized":"2024-09-13T13:17:13.135115+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:30.590466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:30.590466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":550,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.137508+0000","last_change":"2024-09-13T13:14:48.128237+0000","last_active":"2024-09-13T13:17:13.137508+0000","last_peered":"2024-09-13T13:17:13.137508+0000","last_clean":"2024-09-13T13:17:13.137508+0000","last_became_active":"2024-09-13T13:12:39.947940+0000","last_became_peered":"2024-09-13T13:12:39.947940+0000","last_unstale":"2024-09-13T13:17:13.137508+0000","last_undegraded":"2024-09-13T13:17:13.137508+0000","last_fullsized":"2024-09-13T13:17:13.137508+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:14:48.128156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:48.128156+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":280,"num_read_kb":275,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":724,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.130080+0000","last_change":"2024-09-13T13:15:32.318497+0000","last_active":"2024-09-13T13:17:13.130080+0000","last_peered":"2024-09-13T13:17:13.130080+0000","last_clean":"2024-09-13T13:17:13.130080+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:17:13.130080+0000","last_undegraded":"2024-09-13T13:17:13.130080+0000","last_fullsized":"2024-09-13T13:17:13.130080+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:32.318413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.318413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":729,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.137921+0000","last_change":"2024-09-13T13:10:32.720742+0000","last_active":"2024-09-13T13:17:13.137921+0000","last_peered":"2024-09-13T13:17:13.137921+0000","last_clean":"2024-09-13T13:17:13.137921+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:17:13.137921+0000","last_undegraded":"2024-09-13T13:17:13.137921+0000","last_fullsized":"2024-09-13T13:17:13.137921+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:39.284284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"402'14494","reported_seq":14642,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.359407+0000","last_change":"2024-09-13T13:13:58.967163+0000","last_active":"2024-09-13T13:17:16.359407+0000","last_peered":"2024-09-13T13:17:16.359407+0000","last_clean":"2024-09-13T13:17:16.359407+0000","last_became_active":"2024-09-13T13:13:58.965962+0000","last_became_peered":"2024-09-13T13:13:58.965962+0000","last_unstale":"2024-09-13T13:17:16.359407+0000","last_undegraded":"2024-09-13T13:17:16.359407+0000","last_fullsized":"2024-09-13T13:17:16.359407+0000","mapping_epoch":334,"log_start":"371'9315","ondisk_log_start":"371'9315","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5179,"log_dups_size":0,"ondisk_log_size":5179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:16.445030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94240768,"num_objects":1438,"num_object_clones":0,"num_object_copies":4314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14494,"num_write_kb":509824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"306'32","reported_seq":875,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.161749+0000","last_change":"2024-09-13T13:13:01.352416+0000","last_active":"2024-09-13T13:17:16.161749+0000","last_peered":"2024-09-13T13:17:16.161749+0000","last_clean":"2024-09-13T13:17:16.161749+0000","last_became_active":"2024-09-13T13:12:52.999924+0000","last_became_peered":"2024-09-13T13:12:52.999924+0000","last_unstale":"2024-09-13T13:17:16.161749+0000","last_undegraded":"2024-09-13T13:17:16.161749+0000","last_fullsized":"2024-09-13T13:17:16.161749+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:13:01.352337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.352337+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:11.387458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":755,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.162986+0000","last_change":"2024-09-13T13:16:53.951208+0000","last_active":"2024-09-13T13:17:17.162986+0000","last_peered":"2024-09-13T13:17:17.162986+0000","last_clean":"2024-09-13T13:17:17.162986+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:17:17.162986+0000","last_undegraded":"2024-09-13T13:17:17.162986+0000","last_fullsized":"2024-09-13T13:17:17.162986+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:53.951135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:53.951135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:55.102860+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":318,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134781+0000","last_change":"2024-09-13T13:15:54.707061+0000","last_active":"2024-09-13T13:17:13.134781+0000","last_peered":"2024-09-13T13:17:13.134781+0000","last_clean":"2024-09-13T13:17:13.134781+0000","last_became_active":"2024-09-13T13:12:39.948765+0000","last_became_peered":"2024-09-13T13:12:39.948765+0000","last_unstale":"2024-09-13T13:17:13.134781+0000","last_undegraded":"2024-09-13T13:17:13.134781+0000","last_fullsized":"2024-09-13T13:17:13.134781+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":63,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129012+0000","last_change":"2024-09-13T13:16:12.074017+0000","last_active":"2024-09-13T13:17:13.129012+0000","last_peered":"2024-09-13T13:17:13.129012+0000","last_clean":"2024-09-13T13:17:13.129012+0000","last_became_active":"2024-09-13T13:16:12.073663+0000","last_became_peered":"2024-09-13T13:16:12.073663+0000","last_unstale":"2024-09-13T13:17:13.129012+0000","last_undegraded":"2024-09-13T13:17:13.129012+0000","last_fullsized":"2024-09-13T13:17:13.129012+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:33.708995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.708995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":766,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129931+0000","last_change":"2024-09-13T13:15:57.689044+0000","last_active":"2024-09-13T13:17:13.129931+0000","last_peered":"2024-09-13T13:17:13.129931+0000","last_clean":"2024-09-13T13:17:13.129931+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:17:13.129931+0000","last_undegraded":"2024-09-13T13:17:13.129931+0000","last_fullsized":"2024-09-13T13:17:13.129931+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:57.688955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:57.688955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"402'14734","reported_seq":15001,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.279557+0000","last_change":"2024-09-13T13:16:33.808007+0000","last_active":"2024-09-13T13:17:16.279557+0000","last_peered":"2024-09-13T13:17:16.279557+0000","last_clean":"2024-09-13T13:17:16.279557+0000","last_became_active":"2024-09-13T13:13:58.965566+0000","last_became_peered":"2024-09-13T13:13:58.965566+0000","last_unstale":"2024-09-13T13:17:16.279557+0000","last_undegraded":"2024-09-13T13:17:16.279557+0000","last_fullsized":"2024-09-13T13:17:16.279557+0000","mapping_epoch":334,"log_start":"371'9521","ondisk_log_start":"371'9521","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'12107","last_scrub_stamp":"2024-09-13T13:16:33.807809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:16:33.807809+0000","objects_scrubbed":4136,"log_size":5213,"log_dups_size":0,"ondisk_log_size":5213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:55.361775+0000","scrub_duration":1617,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95289344,"num_objects":1454,"num_object_clones":0,"num_object_copies":4362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14734,"num_write_kb":518016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":231,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.133478+0000","last_change":"2024-09-13T13:14:59.354034+0000","last_active":"2024-09-13T13:17:13.133478+0000","last_peered":"2024-09-13T13:17:13.133478+0000","last_clean":"2024-09-13T13:17:13.133478+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:17:13.133478+0000","last_undegraded":"2024-09-13T13:17:13.133478+0000","last_fullsized":"2024-09-13T13:17:13.133478+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:14:59.353691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.353691+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":219,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131720+0000","last_change":"2024-09-13T13:16:12.074151+0000","last_active":"2024-09-13T13:17:13.131720+0000","last_peered":"2024-09-13T13:17:13.131720+0000","last_clean":"2024-09-13T13:17:13.131720+0000","last_became_active":"2024-09-13T13:16:12.073474+0000","last_became_peered":"2024-09-13T13:16:12.073474+0000","last_unstale":"2024-09-13T13:17:13.131720+0000","last_undegraded":"2024-09-13T13:17:13.131720+0000","last_fullsized":"2024-09-13T13:17:13.131720+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:26.343754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:26.343754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":781,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129777+0000","last_change":"2024-09-13T13:15:55.592241+0000","last_active":"2024-09-13T13:17:13.129777+0000","last_peered":"2024-09-13T13:17:13.129777+0000","last_clean":"2024-09-13T13:17:13.129777+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:17:13.129777+0000","last_undegraded":"2024-09-13T13:17:13.129777+0000","last_fullsized":"2024-09-13T13:17:13.129777+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:55.592172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:55.592172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:23.172298+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":735,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.163347+0000","last_change":"2024-09-13T13:14:23.655250+0000","last_active":"2024-09-13T13:17:17.163347+0000","last_peered":"2024-09-13T13:17:17.163347+0000","last_clean":"2024-09-13T13:17:17.163347+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:17:17.163347+0000","last_undegraded":"2024-09-13T13:17:17.163347+0000","last_fullsized":"2024-09-13T13:17:17.163347+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:23.655072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:23.655072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:48.966053+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"401'14350","reported_seq":20184,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.491831+0000","last_change":"2024-09-13T13:17:02.078726+0000","last_active":"2024-09-13T13:17:13.491831+0000","last_peered":"2024-09-13T13:17:13.491831+0000","last_clean":"2024-09-13T13:17:13.491831+0000","last_became_active":"2024-09-13T13:17:02.055599+0000","last_became_peered":"2024-09-13T13:17:02.055599+0000","last_unstale":"2024-09-13T13:17:13.491831+0000","last_undegraded":"2024-09-13T13:17:13.491831+0000","last_fullsized":"2024-09-13T13:17:13.491831+0000","mapping_epoch":395,"log_start":"371'9284","ondisk_log_start":"371'9284","created":334,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7793","last_scrub_stamp":"2024-09-13T13:15:31.408001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:31.408001+0000","objects_scrubbed":7714,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103284736,"num_objects":1576,"num_object_clones":0,"num_object_copies":4728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14350,"num_write_kb":509632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2792,"num_bytes_recovered":182976512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":262,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134869+0000","last_change":"2024-09-13T13:16:12.090140+0000","last_active":"2024-09-13T13:17:13.134869+0000","last_peered":"2024-09-13T13:17:13.134869+0000","last_clean":"2024-09-13T13:17:13.134869+0000","last_became_active":"2024-09-13T13:16:12.075614+0000","last_became_peered":"2024-09-13T13:16:12.075614+0000","last_unstale":"2024-09-13T13:17:13.134869+0000","last_undegraded":"2024-09-13T13:17:13.134869+0000","last_fullsized":"2024-09-13T13:17:13.134869+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:14:20.221286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:20.221286+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":765,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.162196+0000","last_change":"2024-09-13T13:16:37.263033+0000","last_active":"2024-09-13T13:17:16.162196+0000","last_peered":"2024-09-13T13:17:16.162196+0000","last_clean":"2024-09-13T13:17:16.162196+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:17:16.162196+0000","last_undegraded":"2024-09-13T13:17:16.162196+0000","last_fullsized":"2024-09-13T13:17:16.162196+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:37.262977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:37.262977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:56.631589+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":755,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.138030+0000","last_change":"2024-09-13T13:17:05.238854+0000","last_active":"2024-09-13T13:17:13.138030+0000","last_peered":"2024-09-13T13:17:13.138030+0000","last_clean":"2024-09-13T13:17:13.138030+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:17:13.138030+0000","last_undegraded":"2024-09-13T13:17:13.138030+0000","last_fullsized":"2024-09-13T13:17:13.138030+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:05.238722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.238722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:12.615074+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":598,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.167364+0000","last_change":"2024-09-13T13:16:35.064359+0000","last_active":"2024-09-13T13:17:17.167364+0000","last_peered":"2024-09-13T13:17:17.167364+0000","last_clean":"2024-09-13T13:17:17.167364+0000","last_became_active":"2024-09-13T13:16:12.067474+0000","last_became_peered":"2024-09-13T13:16:12.067474+0000","last_unstale":"2024-09-13T13:17:17.167364+0000","last_undegraded":"2024-09-13T13:17:17.167364+0000","last_fullsized":"2024-09-13T13:17:17.167364+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:35.064188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:35.064188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:44.945474+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"401'14497","reported_seq":14693,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:14.058743+0000","last_change":"2024-09-13T13:16:29.564422+0000","last_active":"2024-09-13T13:17:14.058743+0000","last_peered":"2024-09-13T13:17:14.058743+0000","last_clean":"2024-09-13T13:17:14.058743+0000","last_became_active":"2024-09-13T13:13:58.963064+0000","last_became_peered":"2024-09-13T13:13:58.963064+0000","last_unstale":"2024-09-13T13:17:14.058743+0000","last_undegraded":"2024-09-13T13:17:14.058743+0000","last_fullsized":"2024-09-13T13:17:14.058743+0000","mapping_epoch":334,"log_start":"372'9814","ondisk_log_start":"372'9814","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'11857","last_scrub_stamp":"2024-09-13T13:16:29.563999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:16:29.563999+0000","objects_scrubbed":4241,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:58.196946+0000","scrub_duration":1422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100990976,"num_objects":1541,"num_object_clones":0,"num_object_copies":4623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14497,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":223,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131347+0000","last_change":"2024-09-13T13:17:02.421924+0000","last_active":"2024-09-13T13:17:13.131347+0000","last_peered":"2024-09-13T13:17:13.131347+0000","last_clean":"2024-09-13T13:17:13.131347+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:17:13.131347+0000","last_undegraded":"2024-09-13T13:17:13.131347+0000","last_fullsized":"2024-09-13T13:17:13.131347+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.421825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.421825+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:23.577428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":742,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.163511+0000","last_change":"2024-09-13T13:12:53.003530+0000","last_active":"2024-09-13T13:17:17.163511+0000","last_peered":"2024-09-13T13:17:17.163511+0000","last_clean":"2024-09-13T13:17:17.163511+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:17:17.163511+0000","last_undegraded":"2024-09-13T13:17:17.163511+0000","last_fullsized":"2024-09-13T13:17:17.163511+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.996346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.996346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:13.096545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"377'9","reported_seq":104,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129001+0000","last_change":"2024-09-13T13:16:12.089906+0000","last_active":"2024-09-13T13:17:13.129001+0000","last_peered":"2024-09-13T13:17:13.129001+0000","last_clean":"2024-09-13T13:17:13.129001+0000","last_became_active":"2024-09-13T13:16:12.079150+0000","last_became_peered":"2024-09-13T13:16:12.079150+0000","last_unstale":"2024-09-13T13:17:13.129001+0000","last_undegraded":"2024-09-13T13:17:13.129001+0000","last_fullsized":"2024-09-13T13:17:13.129001+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:12:36.113709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:36.113709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":218,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.133379+0000","last_change":"2024-09-13T13:16:12.076027+0000","last_active":"2024-09-13T13:17:13.133379+0000","last_peered":"2024-09-13T13:17:13.133379+0000","last_clean":"2024-09-13T13:17:13.133379+0000","last_became_active":"2024-09-13T13:16:12.075615+0000","last_became_peered":"2024-09-13T13:16:12.075615+0000","last_unstale":"2024-09-13T13:17:13.133379+0000","last_undegraded":"2024-09-13T13:17:13.133379+0000","last_fullsized":"2024-09-13T13:17:13.133379+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.797639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.797639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"401'14364","reported_seq":20455,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:14.621557+0000","last_change":"2024-09-13T13:16:59.076235+0000","last_active":"2024-09-13T13:17:14.621557+0000","last_peered":"2024-09-13T13:17:14.621557+0000","last_clean":"2024-09-13T13:17:14.621557+0000","last_became_active":"2024-09-13T13:16:59.041701+0000","last_became_peered":"2024-09-13T13:16:59.041701+0000","last_unstale":"2024-09-13T13:17:14.621557+0000","last_undegraded":"2024-09-13T13:17:14.621557+0000","last_fullsized":"2024-09-13T13:17:14.621557+0000","mapping_epoch":393,"log_start":"372'9771","ondisk_log_start":"372'9771","created":334,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7822","last_scrub_stamp":"2024-09-13T13:15:32.530220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.530220+0000","objects_scrubbed":7744,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98041856,"num_objects":1496,"num_object_clones":0,"num_object_copies":4488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14364,"num_write_kb":507520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2919,"num_bytes_recovered":191299584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":770,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135795+0000","last_change":"2024-09-13T13:13:56.850149+0000","last_active":"2024-09-13T13:17:13.135795+0000","last_peered":"2024-09-13T13:17:13.135795+0000","last_clean":"2024-09-13T13:17:13.135795+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:17:13.135795+0000","last_undegraded":"2024-09-13T13:17:13.135795+0000","last_fullsized":"2024-09-13T13:17:13.135795+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:56.850094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.850094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:11.629068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":759,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129238+0000","last_change":"2024-09-13T13:12:58.857634+0000","last_active":"2024-09-13T13:17:13.129238+0000","last_peered":"2024-09-13T13:17:13.129238+0000","last_clean":"2024-09-13T13:17:13.129238+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:17:13.129238+0000","last_undegraded":"2024-09-13T13:17:13.129238+0000","last_fullsized":"2024-09-13T13:17:13.129238+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:58.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":606,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131177+0000","last_change":"2024-09-13T13:16:12.077831+0000","last_active":"2024-09-13T13:17:13.131177+0000","last_peered":"2024-09-13T13:17:13.131177+0000","last_clean":"2024-09-13T13:17:13.131177+0000","last_became_active":"2024-09-13T13:16:12.076962+0000","last_became_peered":"2024-09-13T13:16:12.076962+0000","last_unstale":"2024-09-13T13:17:13.131177+0000","last_undegraded":"2024-09-13T13:17:13.131177+0000","last_fullsized":"2024-09-13T13:17:13.131177+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:23.682463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.682463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":726,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135028+0000","last_change":"2024-09-13T13:16:12.075127+0000","last_active":"2024-09-13T13:17:13.135028+0000","last_peered":"2024-09-13T13:17:13.135028+0000","last_clean":"2024-09-13T13:17:13.135028+0000","last_became_active":"2024-09-13T13:16:12.074784+0000","last_became_peered":"2024-09-13T13:16:12.074784+0000","last_unstale":"2024-09-13T13:17:13.135028+0000","last_undegraded":"2024-09-13T13:17:13.135028+0000","last_fullsized":"2024-09-13T13:17:13.135028+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.761814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.761814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:12.593132+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"401'14176","reported_seq":14369,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:14.611742+0000","last_change":"2024-09-13T13:16:34.719611+0000","last_active":"2024-09-13T13:17:14.611742+0000","last_peered":"2024-09-13T13:17:14.611742+0000","last_clean":"2024-09-13T13:17:14.611742+0000","last_became_active":"2024-09-13T13:13:58.963160+0000","last_became_peered":"2024-09-13T13:13:58.963160+0000","last_unstale":"2024-09-13T13:17:14.611742+0000","last_undegraded":"2024-09-13T13:17:14.611742+0000","last_fullsized":"2024-09-13T13:17:14.611742+0000","mapping_epoch":334,"log_start":"372'9588","ondisk_log_start":"372'9588","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'11789","last_scrub_stamp":"2024-09-13T13:16:34.719499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:16:34.719499+0000","objects_scrubbed":3932,"log_size":4588,"log_dups_size":0,"ondisk_log_size":4588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:43.651072+0000","scrub_duration":908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99745792,"num_objects":1522,"num_object_clones":0,"num_object_copies":4566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14176,"num_write_kb":502336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":746,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.161968+0000","last_change":"2024-09-13T13:13:06.339762+0000","last_active":"2024-09-13T13:17:16.161968+0000","last_peered":"2024-09-13T13:17:16.161968+0000","last_clean":"2024-09-13T13:17:16.161968+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:17:16.161968+0000","last_undegraded":"2024-09-13T13:17:16.161968+0000","last_fullsized":"2024-09-13T13:17:16.161968+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:13:06.339664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.339664+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":374,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129277+0000","last_change":"2024-09-13T13:14:55.471954+0000","last_active":"2024-09-13T13:17:13.129277+0000","last_peered":"2024-09-13T13:17:13.129277+0000","last_clean":"2024-09-13T13:17:13.129277+0000","last_became_active":"2024-09-13T13:12:39.954315+0000","last_became_peered":"2024-09-13T13:12:39.954315+0000","last_unstale":"2024-09-13T13:17:13.129277+0000","last_undegraded":"2024-09-13T13:17:13.129277+0000","last_fullsized":"2024-09-13T13:17:13.129277+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:55.471885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:55.471885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":62,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135554+0000","last_change":"2024-09-13T13:16:12.076693+0000","last_active":"2024-09-13T13:17:13.135554+0000","last_peered":"2024-09-13T13:17:13.135554+0000","last_clean":"2024-09-13T13:17:13.135554+0000","last_became_active":"2024-09-13T13:16:12.076377+0000","last_became_peered":"2024-09-13T13:16:12.076377+0000","last_unstale":"2024-09-13T13:17:13.135554+0000","last_undegraded":"2024-09-13T13:17:13.135554+0000","last_fullsized":"2024-09-13T13:17:13.135554+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.484615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.484615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:18.832449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":67,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.162000+0000","last_change":"2024-09-13T13:16:12.066346+0000","last_active":"2024-09-13T13:17:17.162000+0000","last_peered":"2024-09-13T13:17:17.162000+0000","last_clean":"2024-09-13T13:17:17.162000+0000","last_became_active":"2024-09-13T13:16:12.065848+0000","last_became_peered":"2024-09-13T13:16:12.065848+0000","last_unstale":"2024-09-13T13:17:17.162000+0000","last_undegraded":"2024-09-13T13:17:17.162000+0000","last_fullsized":"2024-09-13T13:17:17.162000+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:02.627569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:02.627569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:41.973870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"402'14186","reported_seq":14334,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.324570+0000","last_change":"2024-09-13T13:13:58.966955+0000","last_active":"2024-09-13T13:17:16.324570+0000","last_peered":"2024-09-13T13:17:16.324570+0000","last_clean":"2024-09-13T13:17:16.324570+0000","last_became_active":"2024-09-13T13:13:58.966675+0000","last_became_peered":"2024-09-13T13:13:58.966675+0000","last_unstale":"2024-09-13T13:17:16.324570+0000","last_undegraded":"2024-09-13T13:17:16.324570+0000","last_fullsized":"2024-09-13T13:17:16.324570+0000","mapping_epoch":334,"log_start":"369'8941","ondisk_log_start":"369'8941","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:21.398151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92012544,"num_objects":1404,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14186,"num_write_kb":498880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":232,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131909+0000","last_change":"2024-09-13T13:16:00.285064+0000","last_active":"2024-09-13T13:17:13.131909+0000","last_peered":"2024-09-13T13:17:13.131909+0000","last_clean":"2024-09-13T13:17:13.131909+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:17:13.131909+0000","last_undegraded":"2024-09-13T13:17:13.131909+0000","last_fullsized":"2024-09-13T13:17:13.131909+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:16:00.284984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:00.284984+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:27.689954+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":219,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131335+0000","last_change":"2024-09-13T13:14:52.383027+0000","last_active":"2024-09-13T13:17:13.131335+0000","last_peered":"2024-09-13T13:17:13.131335+0000","last_clean":"2024-09-13T13:17:13.131335+0000","last_became_active":"2024-09-13T13:12:53.007561+0000","last_became_peered":"2024-09-13T13:12:53.007561+0000","last_unstale":"2024-09-13T13:17:13.131335+0000","last_undegraded":"2024-09-13T13:17:13.131335+0000","last_fullsized":"2024-09-13T13:17:13.131335+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.382755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.382755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":62,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134227+0000","last_change":"2024-09-13T13:16:12.078026+0000","last_active":"2024-09-13T13:17:13.134227+0000","last_peered":"2024-09-13T13:17:13.134227+0000","last_clean":"2024-09-13T13:17:13.134227+0000","last_became_active":"2024-09-13T13:16:12.077714+0000","last_became_peered":"2024-09-13T13:16:12.077714+0000","last_unstale":"2024-09-13T13:17:13.134227+0000","last_undegraded":"2024-09-13T13:17:13.134227+0000","last_fullsized":"2024-09-13T13:17:13.134227+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.636898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.636898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":724,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.163115+0000","last_change":"2024-09-13T13:16:58.047398+0000","last_active":"2024-09-13T13:17:17.163115+0000","last_peered":"2024-09-13T13:17:17.163115+0000","last_clean":"2024-09-13T13:17:17.163115+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:17:17.163115+0000","last_undegraded":"2024-09-13T13:17:17.163115+0000","last_fullsized":"2024-09-13T13:17:17.163115+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:58.044139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.044139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:25.917931+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"401'14433","reported_seq":14652,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.489391+0000","last_change":"2024-09-13T13:15:56.411481+0000","last_active":"2024-09-13T13:17:13.489391+0000","last_peered":"2024-09-13T13:17:13.489391+0000","last_clean":"2024-09-13T13:17:13.489391+0000","last_became_active":"2024-09-13T13:13:58.962773+0000","last_became_peered":"2024-09-13T13:13:58.962773+0000","last_unstale":"2024-09-13T13:17:13.489391+0000","last_undegraded":"2024-09-13T13:17:13.489391+0000","last_fullsized":"2024-09-13T13:17:13.489391+0000","mapping_epoch":334,"log_start":"371'9376","ondisk_log_start":"371'9376","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'9588","last_scrub_stamp":"2024-09-13T13:15:56.411320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:56.411320+0000","objects_scrubbed":6514,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:14.862198+0000","scrub_duration":1806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106233856,"num_objects":1621,"num_object_clones":0,"num_object_copies":4863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14433,"num_write_kb":513728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":806,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.128894+0000","last_change":"2024-09-13T13:15:56.631456+0000","last_active":"2024-09-13T13:17:13.128894+0000","last_peered":"2024-09-13T13:17:13.128894+0000","last_clean":"2024-09-13T13:17:13.128894+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:17:13.128894+0000","last_undegraded":"2024-09-13T13:17:13.128894+0000","last_fullsized":"2024-09-13T13:17:13.128894+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:15:56.631396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:56.631396+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:22.171338+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":217,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131834+0000","last_change":"2024-09-13T13:13:01.972907+0000","last_active":"2024-09-13T13:17:13.131834+0000","last_peered":"2024-09-13T13:17:13.131834+0000","last_clean":"2024-09-13T13:17:13.131834+0000","last_became_active":"2024-09-13T13:12:53.006362+0000","last_became_peered":"2024-09-13T13:12:53.006362+0000","last_unstale":"2024-09-13T13:17:13.131834+0000","last_undegraded":"2024-09-13T13:17:13.131834+0000","last_fullsized":"2024-09-13T13:17:13.131834+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:13:01.972831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.972831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"306'13","reported_seq":477,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.160662+0000","last_change":"2024-09-13T13:13:03.334319+0000","last_active":"2024-09-13T13:17:16.160662+0000","last_peered":"2024-09-13T13:17:16.160662+0000","last_clean":"2024-09-13T13:17:16.160662+0000","last_became_active":"2024-09-13T13:12:53.001189+0000","last_became_peered":"2024-09-13T13:12:53.001189+0000","last_unstale":"2024-09-13T13:17:16.160662+0000","last_undegraded":"2024-09-13T13:17:16.160662+0000","last_fullsized":"2024-09-13T13:17:16.160662+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'13","last_scrub_stamp":"2024-09-13T13:13:03.334012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.334012+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:39.235043+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":731,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129756+0000","last_change":"2024-09-13T13:13:55.500961+0000","last_active":"2024-09-13T13:17:13.129756+0000","last_peered":"2024-09-13T13:17:13.129756+0000","last_clean":"2024-09-13T13:17:13.129756+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:17:13.129756+0000","last_undegraded":"2024-09-13T13:17:13.129756+0000","last_fullsized":"2024-09-13T13:17:13.129756+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:55.500908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.500908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":710,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.160670+0000","last_change":"2024-09-13T13:13:04.371082+0000","last_active":"2024-09-13T13:17:16.160670+0000","last_peered":"2024-09-13T13:17:16.160670+0000","last_clean":"2024-09-13T13:17:16.160670+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:17:16.160670+0000","last_undegraded":"2024-09-13T13:17:16.160670+0000","last_fullsized":"2024-09-13T13:17:16.160670+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.370881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.370881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:13.392940+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":253,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129589+0000","last_change":"2024-09-13T13:15:23.319486+0000","last_active":"2024-09-13T13:17:13.129589+0000","last_peered":"2024-09-13T13:17:13.129589+0000","last_clean":"2024-09-13T13:17:13.129589+0000","last_became_active":"2024-09-13T13:12:34.673528+0000","last_became_peered":"2024-09-13T13:12:34.673528+0000","last_unstale":"2024-09-13T13:17:13.129589+0000","last_undegraded":"2024-09-13T13:17:13.129589+0000","last_fullsized":"2024-09-13T13:17:13.129589+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:15:23.319412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.319412+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":755,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129888+0000","last_change":"2024-09-13T13:15:51.633487+0000","last_active":"2024-09-13T13:17:13.129888+0000","last_peered":"2024-09-13T13:17:13.129888+0000","last_clean":"2024-09-13T13:17:13.129888+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:17:13.129888+0000","last_undegraded":"2024-09-13T13:17:13.129888+0000","last_fullsized":"2024-09-13T13:17:13.129888+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.633433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.633433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":776,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.133587+0000","last_change":"2024-09-13T13:16:33.024715+0000","last_active":"2024-09-13T13:17:13.133587+0000","last_peered":"2024-09-13T13:17:13.133587+0000","last_clean":"2024-09-13T13:17:13.133587+0000","last_became_active":"2024-09-13T13:12:53.000873+0000","last_became_peered":"2024-09-13T13:12:53.000873+0000","last_unstale":"2024-09-13T13:17:13.133587+0000","last_undegraded":"2024-09-13T13:17:13.133587+0000","last_fullsized":"2024-09-13T13:17:13.133587+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:33.024527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:33.024527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:47.416529+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":751,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135464+0000","last_change":"2024-09-13T13:16:37.748574+0000","last_active":"2024-09-13T13:17:13.135464+0000","last_peered":"2024-09-13T13:17:13.135464+0000","last_clean":"2024-09-13T13:17:13.135464+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:17:13.135464+0000","last_undegraded":"2024-09-13T13:17:13.135464+0000","last_fullsized":"2024-09-13T13:17:13.135464+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:16:37.748500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:37.748500+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:52.503365+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":211,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131750+0000","last_change":"2024-09-13T13:13:51.764793+0000","last_active":"2024-09-13T13:17:13.131750+0000","last_peered":"2024-09-13T13:17:13.131750+0000","last_clean":"2024-09-13T13:17:13.131750+0000","last_became_active":"2024-09-13T13:12:53.006579+0000","last_became_peered":"2024-09-13T13:12:53.006579+0000","last_unstale":"2024-09-13T13:17:13.131750+0000","last_undegraded":"2024-09-13T13:17:13.131750+0000","last_fullsized":"2024-09-13T13:17:13.131750+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.764717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.764717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":743,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.167605+0000","last_change":"2024-09-13T13:17:00.968640+0000","last_active":"2024-09-13T13:17:17.167605+0000","last_peered":"2024-09-13T13:17:17.167605+0000","last_clean":"2024-09-13T13:17:17.167605+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:17:17.167605+0000","last_undegraded":"2024-09-13T13:17:17.167605+0000","last_fullsized":"2024-09-13T13:17:17.167605+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:00.968580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:00.968580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:22.774319+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":609,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131600+0000","last_change":"2024-09-13T13:16:12.078548+0000","last_active":"2024-09-13T13:17:13.131600+0000","last_peered":"2024-09-13T13:17:13.131600+0000","last_clean":"2024-09-13T13:17:13.131600+0000","last_became_active":"2024-09-13T13:16:12.077562+0000","last_became_peered":"2024-09-13T13:16:12.077562+0000","last_unstale":"2024-09-13T13:17:13.131600+0000","last_undegraded":"2024-09-13T13:17:13.131600+0000","last_fullsized":"2024-09-13T13:17:13.131600+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:15.616112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:15.616112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":781,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129327+0000","last_change":"2024-09-13T13:17:04.222896+0000","last_active":"2024-09-13T13:17:13.129327+0000","last_peered":"2024-09-13T13:17:13.129327+0000","last_clean":"2024-09-13T13:17:13.129327+0000","last_became_active":"2024-09-13T13:16:12.069730+0000","last_became_peered":"2024-09-13T13:16:12.069730+0000","last_unstale":"2024-09-13T13:17:13.129327+0000","last_undegraded":"2024-09-13T13:17:13.129327+0000","last_fullsized":"2024-09-13T13:17:13.129327+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:04.222760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:04.222760+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:29.524612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":730,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135024+0000","last_change":"2024-09-13T13:17:09.634101+0000","last_active":"2024-09-13T13:17:13.135024+0000","last_peered":"2024-09-13T13:17:13.135024+0000","last_clean":"2024-09-13T13:17:13.135024+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:17:13.135024+0000","last_undegraded":"2024-09-13T13:17:13.135024+0000","last_fullsized":"2024-09-13T13:17:13.135024+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.634027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.634027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:29.941144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"377'9","reported_seq":934,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.163494+0000","last_change":"2024-09-13T13:16:12.080560+0000","last_active":"2024-09-13T13:17:17.163494+0000","last_peered":"2024-09-13T13:17:17.163494+0000","last_clean":"2024-09-13T13:17:17.163494+0000","last_became_active":"2024-09-13T13:16:12.073710+0000","last_became_peered":"2024-09-13T13:16:12.073710+0000","last_unstale":"2024-09-13T13:17:17.163494+0000","last_undegraded":"2024-09-13T13:17:17.163494+0000","last_fullsized":"2024-09-13T13:17:17.163494+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:13:10.130680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:10.130680+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":218,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131922+0000","last_change":"2024-09-13T13:16:12.077256+0000","last_active":"2024-09-13T13:17:13.131922+0000","last_peered":"2024-09-13T13:17:13.131922+0000","last_clean":"2024-09-13T13:17:13.131922+0000","last_became_active":"2024-09-13T13:16:12.076912+0000","last_became_peered":"2024-09-13T13:16:12.076912+0000","last_unstale":"2024-09-13T13:17:13.131922+0000","last_undegraded":"2024-09-13T13:17:13.131922+0000","last_fullsized":"2024-09-13T13:17:13.131922+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.956784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.956784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":785,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.163789+0000","last_change":"2024-09-13T13:17:02.978088+0000","last_active":"2024-09-13T13:17:17.163789+0000","last_peered":"2024-09-13T13:17:17.163789+0000","last_clean":"2024-09-13T13:17:17.163789+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:17:17.163789+0000","last_undegraded":"2024-09-13T13:17:17.163789+0000","last_fullsized":"2024-09-13T13:17:17.163789+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.978003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.978003+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:17.628791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":744,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.132978+0000","last_change":"2024-09-13T13:16:12.067154+0000","last_active":"2024-09-13T13:17:13.132978+0000","last_peered":"2024-09-13T13:17:13.132978+0000","last_clean":"2024-09-13T13:17:13.132978+0000","last_became_active":"2024-09-13T13:16:12.066808+0000","last_became_peered":"2024-09-13T13:16:12.066808+0000","last_unstale":"2024-09-13T13:17:13.132978+0000","last_undegraded":"2024-09-13T13:17:13.132978+0000","last_fullsized":"2024-09-13T13:17:13.132978+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.105098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.105098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:13.844340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":942,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129098+0000","last_change":"2024-09-13T13:15:33.333062+0000","last_active":"2024-09-13T13:17:13.129098+0000","last_peered":"2024-09-13T13:17:13.129098+0000","last_clean":"2024-09-13T13:17:13.129098+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:17:13.129098+0000","last_undegraded":"2024-09-13T13:17:13.129098+0000","last_fullsized":"2024-09-13T13:17:13.129098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:15:33.332981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.332981+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":734,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135282+0000","last_change":"2024-09-13T13:15:22.660262+0000","last_active":"2024-09-13T13:17:13.135282+0000","last_peered":"2024-09-13T13:17:13.135282+0000","last_clean":"2024-09-13T13:17:13.135282+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:17:13.135282+0000","last_undegraded":"2024-09-13T13:17:13.135282+0000","last_fullsized":"2024-09-13T13:17:13.135282+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:22.660050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:22.660050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":223,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134730+0000","last_change":"2024-09-13T13:13:47.728092+0000","last_active":"2024-09-13T13:17:13.134730+0000","last_peered":"2024-09-13T13:17:13.134730+0000","last_clean":"2024-09-13T13:17:13.134730+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:17:13.134730+0000","last_undegraded":"2024-09-13T13:17:13.134730+0000","last_fullsized":"2024-09-13T13:17:13.134730+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:47.728010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:47.728010+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":211,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131446+0000","last_change":"2024-09-13T13:17:09.431539+0000","last_active":"2024-09-13T13:17:13.131446+0000","last_peered":"2024-09-13T13:17:13.131446+0000","last_clean":"2024-09-13T13:17:13.131446+0000","last_became_active":"2024-09-13T13:12:53.005988+0000","last_became_peered":"2024-09-13T13:12:53.005988+0000","last_unstale":"2024-09-13T13:17:13.131446+0000","last_undegraded":"2024-09-13T13:17:13.131446+0000","last_fullsized":"2024-09-13T13:17:13.131446+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.431472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.431472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:30.094940+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":740,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.139562+0000","last_change":"2024-09-13T13:15:57.356507+0000","last_active":"2024-09-13T13:17:13.139562+0000","last_peered":"2024-09-13T13:17:13.139562+0000","last_clean":"2024-09-13T13:17:13.139562+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:17:13.139562+0000","last_undegraded":"2024-09-13T13:17:13.139562+0000","last_fullsized":"2024-09-13T13:17:13.139562+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:57.356427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:57.356427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:20.233276+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":747,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.161764+0000","last_change":"2024-09-13T13:17:09.989898+0000","last_active":"2024-09-13T13:17:17.161764+0000","last_peered":"2024-09-13T13:17:17.161764+0000","last_clean":"2024-09-13T13:17:17.161764+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:17:17.161764+0000","last_undegraded":"2024-09-13T13:17:17.161764+0000","last_fullsized":"2024-09-13T13:17:17.161764+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.989820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.989820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:38.033009+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":713,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.129593+0000","last_change":"2024-09-13T13:16:12.087845+0000","last_active":"2024-09-13T13:17:13.129593+0000","last_peered":"2024-09-13T13:17:13.129593+0000","last_clean":"2024-09-13T13:17:13.129593+0000","last_became_active":"2024-09-13T13:16:12.072262+0000","last_became_peered":"2024-09-13T13:16:12.072262+0000","last_unstale":"2024-09-13T13:17:13.129593+0000","last_undegraded":"2024-09-13T13:17:13.129593+0000","last_fullsized":"2024-09-13T13:17:13.129593+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:12:39.874324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:39.874324+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":727,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.168833+0000","last_change":"2024-09-13T13:14:13.418389+0000","last_active":"2024-09-13T13:17:17.168833+0000","last_peered":"2024-09-13T13:17:17.168833+0000","last_clean":"2024-09-13T13:17:17.168833+0000","last_became_active":"2024-09-13T13:12:52.949094+0000","last_became_peered":"2024-09-13T13:12:52.949094+0000","last_unstale":"2024-09-13T13:17:17.168833+0000","last_undegraded":"2024-09-13T13:17:17.168833+0000","last_fullsized":"2024-09-13T13:17:17.168833+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:13.418196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:13.418196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:51.937954+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":735,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.163265+0000","last_change":"2024-09-13T13:15:01.269060+0000","last_active":"2024-09-13T13:17:17.163265+0000","last_peered":"2024-09-13T13:17:17.163265+0000","last_clean":"2024-09-13T13:17:17.163265+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:17:17.163265+0000","last_undegraded":"2024-09-13T13:17:17.163265+0000","last_fullsized":"2024-09-13T13:17:17.163265+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:01.268884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:01.268884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:44.996809+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":718,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134148+0000","last_change":"2024-09-13T13:10:39.477882+0000","last_active":"2024-09-13T13:17:13.134148+0000","last_peered":"2024-09-13T13:17:13.134148+0000","last_clean":"2024-09-13T13:17:13.134148+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:17:13.134148+0000","last_undegraded":"2024-09-13T13:17:13.134148+0000","last_fullsized":"2024-09-13T13:17:13.134148+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:39.477683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.477683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:40.746995+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":765,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.161900+0000","last_change":"2024-09-13T13:16:28.056674+0000","last_active":"2024-09-13T13:17:17.161900+0000","last_peered":"2024-09-13T13:17:17.161900+0000","last_clean":"2024-09-13T13:17:17.161900+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:17:17.161900+0000","last_undegraded":"2024-09-13T13:17:17.161900+0000","last_fullsized":"2024-09-13T13:17:17.161900+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:16:28.056586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:28.056586+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:57.805987+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":747,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.130332+0000","last_change":"2024-09-13T13:14:50.392659+0000","last_active":"2024-09-13T13:17:13.130332+0000","last_peered":"2024-09-13T13:17:13.130332+0000","last_clean":"2024-09-13T13:17:13.130332+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:17:13.130332+0000","last_undegraded":"2024-09-13T13:17:13.130332+0000","last_fullsized":"2024-09-13T13:17:13.130332+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.392444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.392444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":756,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.132133+0000","last_change":"2024-09-13T13:15:03.778214+0000","last_active":"2024-09-13T13:17:13.132133+0000","last_peered":"2024-09-13T13:17:13.132133+0000","last_clean":"2024-09-13T13:17:13.132133+0000","last_became_active":"2024-09-13T13:12:52.999748+0000","last_became_peered":"2024-09-13T13:12:52.999748+0000","last_unstale":"2024-09-13T13:17:13.132133+0000","last_undegraded":"2024-09-13T13:17:13.132133+0000","last_fullsized":"2024-09-13T13:17:13.132133+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.778170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.778170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:20.863950+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":718,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.163415+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:17:17.163415+0000","last_peered":"2024-09-13T13:17:17.163415+0000","last_clean":"2024-09-13T13:17:17.163415+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:17:17.163415+0000","last_undegraded":"2024-09-13T13:17:17.163415+0000","last_fullsized":"2024-09-13T13:17:17.163415+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":780,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.161713+0000","last_change":"2024-09-13T13:16:12.090990+0000","last_active":"2024-09-13T13:17:17.161713+0000","last_peered":"2024-09-13T13:17:17.161713+0000","last_clean":"2024-09-13T13:17:17.161713+0000","last_became_active":"2024-09-13T13:16:12.073702+0000","last_became_peered":"2024-09-13T13:16:12.073702+0000","last_unstale":"2024-09-13T13:17:17.161713+0000","last_undegraded":"2024-09-13T13:17:17.161713+0000","last_fullsized":"2024-09-13T13:17:17.161713+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:14:12.368430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:12.368430+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:39.946451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":754,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.134993+0000","last_change":"2024-09-13T13:16:32.807018+0000","last_active":"2024-09-13T13:17:13.134993+0000","last_peered":"2024-09-13T13:17:13.134993+0000","last_clean":"2024-09-13T13:17:13.134993+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:17:13.134993+0000","last_undegraded":"2024-09-13T13:17:13.134993+0000","last_fullsized":"2024-09-13T13:17:13.134993+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:32.806957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:32.806957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:56.930095+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":664,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.160748+0000","last_change":"2024-09-13T13:14:16.457160+0000","last_active":"2024-09-13T13:17:16.160748+0000","last_peered":"2024-09-13T13:17:16.160748+0000","last_clean":"2024-09-13T13:17:16.160748+0000","last_became_active":"2024-09-13T13:12:53.001131+0000","last_became_peered":"2024-09-13T13:12:53.001131+0000","last_unstale":"2024-09-13T13:17:16.160748+0000","last_undegraded":"2024-09-13T13:17:16.160748+0000","last_fullsized":"2024-09-13T13:17:16.160748+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:16.456784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:16.456784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":571,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.131009+0000","last_change":"2024-09-13T13:16:12.068100+0000","last_active":"2024-09-13T13:17:13.131009+0000","last_peered":"2024-09-13T13:17:13.131009+0000","last_clean":"2024-09-13T13:17:13.131009+0000","last_became_active":"2024-09-13T13:16:12.067757+0000","last_became_peered":"2024-09-13T13:16:12.067757+0000","last_unstale":"2024-09-13T13:17:13.131009+0000","last_undegraded":"2024-09-13T13:17:13.131009+0000","last_fullsized":"2024-09-13T13:17:13.131009+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:05.828300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:05.828300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:18.999851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":742,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.138034+0000","last_change":"2024-09-13T13:14:18.882969+0000","last_active":"2024-09-13T13:17:13.138034+0000","last_peered":"2024-09-13T13:17:13.138034+0000","last_clean":"2024-09-13T13:17:13.138034+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:17:13.138034+0000","last_undegraded":"2024-09-13T13:17:13.138034+0000","last_fullsized":"2024-09-13T13:17:13.138034+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:18.882903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:18.882903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:47.220508+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":735,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-13T13:17:17.162091+0000","last_change":"2024-09-13T13:16:36.980668+0000","last_active":"2024-09-13T13:17:17.162091+0000","last_peered":"2024-09-13T13:17:17.162091+0000","last_clean":"2024-09-13T13:17:17.162091+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:17:17.162091+0000","last_undegraded":"2024-09-13T13:17:17.162091+0000","last_fullsized":"2024-09-13T13:17:17.162091+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:36.980594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:36.980594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:50.040358+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":752,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-13T13:17:16.161217+0000","last_change":"2024-09-13T13:16:12.087075+0000","last_active":"2024-09-13T13:17:16.161217+0000","last_peered":"2024-09-13T13:17:16.161217+0000","last_clean":"2024-09-13T13:17:16.161217+0000","last_became_active":"2024-09-13T13:16:12.070277+0000","last_became_peered":"2024-09-13T13:16:12.070277+0000","last_unstale":"2024-09-13T13:17:16.161217+0000","last_undegraded":"2024-09-13T13:17:16.161217+0000","last_fullsized":"2024-09-13T13:17:16.161217+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:49.264128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.264128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:14.366634+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":755,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-13T13:17:13.135397+0000","last_change":"2024-09-13T13:16:12.074365+0000","last_active":"2024-09-13T13:17:13.135397+0000","last_peered":"2024-09-13T13:17:13.135397+0000","last_clean":"2024-09-13T13:17:13.135397+0000","last_became_active":"2024-09-13T13:16:12.072684+0000","last_became_peered":"2024-09-13T13:16:12.072684+0000","last_unstale":"2024-09-13T13:17:13.135397+0000","last_undegraded":"2024-09-13T13:17:13.135397+0000","last_fullsized":"2024-09-13T13:17:13.135397+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:27.678752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:27.678752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1607598104,"num_objects":24531,"num_object_clones":0,"num_object_copies":73593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24531,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":230225,"num_write_kb":8152129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19739,"num_bytes_recovered":1293615104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5662781440,"data_stored":5662769224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77888,"ondisk_log_size":77888,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":3526,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":684,"ondisk_log_size":684,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":280,"num_read_kb":275,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201477,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1094392,"kb_used_data":527272,"kb_used_omap":1,"kb_used_meta":567038,"kb_avail":92650760,"statfs":{"total":95995035648,"available":94874378240,"internally_reserved":0,"allocated":539926528,"data_stored":539522926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":580647370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141088,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1145816,"kb_used_data":652936,"kb_used_omap":4,"kb_used_meta":492795,"kb_avail":92599336,"statfs":{"total":95995035648,"available":94821720064,"internally_reserved":0,"allocated":668606464,"data_stored":668182669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":504622765},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692049,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1307972,"kb_used_data":735340,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92437180,"statfs":{"total":95995035648,"available":94655672320,"internally_reserved":0,"allocated":752988160,"data_stored":752585703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920983,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1767176,"kb_used_data":1200116,"kb_used_omap":1,"kb_used_meta":567038,"kb_avail":91977976,"statfs":{"total":95995035648,"available":94185447424,"internally_reserved":0,"allocated":1228918784,"data_stored":1228501350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":580647366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117215,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1079844,"kb_used_data":439184,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92665308,"statfs":{"total":95995035648,"available":94889275392,"internally_reserved":0,"allocated":449724416,"data_stored":449293231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":656013767},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411554,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575080,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":882088,"kb_used_data":313620,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92863064,"statfs":{"total":95995035648,"available":95091777536,"internally_reserved":0,"allocated":321146880,"data_stored":320704110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738607,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1684632,"kb_used_data":985604,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":92060520,"statfs":{"total":95995035648,"available":94269972480,"internally_reserved":0,"allocated":1009258496,"data_stored":1008833010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1004867584,"data_stored":1004863512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":310444032,"data_stored":310444032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":791740416,"data_stored":791740416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":431030272,"data_stored":431030272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1184763904,"data_stored":1184759832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":740425728,"data_stored":740425728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":666505216,"data_stored":666501144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":533004288,"data_stored":533004288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:17:20.856 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T13:17:20.856 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:17:20.856 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-13T13:17:20.857 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd in 2 2024-09-13T13:17:20.866 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:20.904 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:20.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:20.925 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:17:20.976 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:21.006 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:21.065 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:21.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:21.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T13:17:21.168 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:21.213 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:21.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:21.315 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:21.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:21 smithi167 ceph-mon[33178]: pgmap v809: 148 pgs: 148 active+clean; 1.3 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-09-13T13:17:21.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:21 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:17:21.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:21 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:17:21.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:21 smithi167 ceph-mon[33178]: osdmap e405: 8 total, 8 up, 7 in 2024-09-13T13:17:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:21.356 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:21.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:21.457 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:21.458 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:21 smithi052 ceph-mon[36682]: pgmap v809: 148 pgs: 148 active+clean; 1.3 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-09-13T13:17:21.458 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:21 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:17:21.458 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:21 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:17:21.458 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:21 smithi052 ceph-mon[36682]: osdmap e405: 8 total, 8 up, 7 in 2024-09-13T13:17:21.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:21 smithi052 ceph-mon[29471]: pgmap v809: 148 pgs: 148 active+clean; 1.3 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-09-13T13:17:21.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:21 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:17:21.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:21 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:17:21.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:21 smithi052 ceph-mon[29471]: osdmap e405: 8 total, 8 up, 7 in 2024-09-13T13:17:21.503 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:21.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:21.562 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:17:21.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:21.639 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:21.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:21.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:21.741 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:21.749 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:21.760 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:21.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:21.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:21.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:21.880 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:21.920 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:21.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:22.021 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:22.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:22.057 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:22.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:22.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:22.192 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:17:22.202 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:22.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:22.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:22.281 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:17:22.281 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:17:22.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.281 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:238468 10.b 10:d95b95cd:::benchmark_data_smithi052_153703_object111088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-13T13:17:22.282 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:22.266052+0000", 2024-09-13T13:17:22.282 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.01282292, 2024-09-13T13:17:22.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012851420000000001, 2024-09-13T13:17:22.282 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:22.282 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:22.282 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:17:22.282 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:22.282 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:22.282 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:22.282 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 238468 2024-09-13T13:17:22.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.282 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:22.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:22.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.266052+0000", 2024-09-13T13:17:22.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:22.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.266052+0000", 2024-09-13T13:17:22.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:22.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.266053+0000", 2024-09-13T13:17:22.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5820000000000001e-06 2024-09-13T13:17:22.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:22.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.266059+0000", 2024-09-13T13:17:22.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1939999999999997e-06 2024-09-13T13:17:22.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:22.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.266069+0000", 2024-09-13T13:17:22.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0550999999999999e-05 2024-09-13T13:17:22.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:22.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.266077+0000", 2024-09-13T13:17:22.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9140000000000008e-06 2024-09-13T13:17:22.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:22.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.266154+0000", 2024-09-13T13:17:22.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7265e-05 2024-09-13T13:17:22.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:22.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.266262+0000", 2024-09-13T13:17:22.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010767100000000001 2024-09-13T13:17:22.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:17:22.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.266372+0000", 2024-09-13T13:17:22.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011055 2024-09-13T13:17:22.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:22.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.272227+0000", 2024-09-13T13:17:22.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058545469999999999 2024-09-13T13:17:22.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:22.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.272274+0000", 2024-09-13T13:17:22.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6687000000000002e-05 2024-09-13T13:17:22.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.287 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:22.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.287 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.26071.0:238458 10.7 e406/334)", 2024-09-13T13:17:22.287 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:22.268806+0000", 2024-09-13T13:17:22.287 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.010068475, 2024-09-13T13:17:22.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010169368, 2024-09-13T13:17:22.287 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:22.287 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:22.288 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "queued for pg", 2024-09-13T13:17:22.288 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:22.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:22.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.268806+0000", 2024-09-13T13:17:22.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:22.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.268806+0000", 2024-09-13T13:17:22.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:22.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.268807+0000", 2024-09-13T13:17:22.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9299999999999998e-07 2024-09-13T13:17:22.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:22.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.268809+0000", 2024-09-13T13:17:22.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4499999999999998e-06 2024-09-13T13:17:22.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:22.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.268814+0000", 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4120000000000003e-06 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.268817+0000", 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5769999999999998e-06 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.26071.0:238460 10.f e406/396)", 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:22.272425+0000", 2024-09-13T13:17:22.291 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.006449672, 2024-09-13T13:17:22.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065870690000000001, 2024-09-13T13:17:22.292 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:22.292 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:22.292 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "queued for pg", 2024-09-13T13:17:22.292 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:22.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:22.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.272425+0000", 2024-09-13T13:17:22.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:22.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.272425+0000", 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.272426+0000", 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6899999999999996e-07 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.272429+0000", 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6040000000000001e-06 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:22.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.272436+0000", 2024-09-13T13:17:22.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8900000000000001e-06 2024-09-13T13:17:22.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:22.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.272442+0000", 2024-09-13T13:17:22.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3510000000000001e-06 2024-09-13T13:17:22.294 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.294 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:22.294 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.294 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:238475 10.0 10:0add5654:::benchmark_data_smithi052_153703_object111095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:22.273119+0000", 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0057552200000000001, 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059322050000000003, 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 238475 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273119+0000", 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273119+0000", 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273121+0000", 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.635e-06 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:22.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273126+0000", 2024-09-13T13:17:22.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0599999999999998e-06 2024-09-13T13:17:22.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:22.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273135+0000", 2024-09-13T13:17:22.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6810000000000004e-06 2024-09-13T13:17:22.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:22.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273141+0000", 2024-09-13T13:17:22.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1639999999999998e-06 2024-09-13T13:17:22.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:22.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.275013+0000", 2024-09-13T13:17:22.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018716780000000001 2024-09-13T13:17:22.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:22.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.275129+0000", 2024-09-13T13:17:22.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116363 2024-09-13T13:17:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:17:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.275249+0000", 2024-09-13T13:17:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120084 2024-09-13T13:17:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:238476 10.0 10:0c2859f9:::benchmark_data_smithi052_153703_object111096:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-13T13:17:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:22.273159+0000", 2024-09-13T13:17:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0057157780000000004, 2024-09-13T13:17:22.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059516109999999999, 2024-09-13T13:17:22.301 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:22.301 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:22.301 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:17:22.301 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:22.301 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:22.301 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:22.301 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 238476 2024-09-13T13:17:22.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.301 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:22.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:22.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273159+0000", 2024-09-13T13:17:22.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:22.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273159+0000", 2024-09-13T13:17:22.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:22.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273160+0000", 2024-09-13T13:17:22.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2699999999999998e-07 2024-09-13T13:17:22.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:22.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273163+0000", 2024-09-13T13:17:22.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1300000000000001e-06 2024-09-13T13:17:22.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:22.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273170+0000", 2024-09-13T13:17:22.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8789999999999997e-06 2024-09-13T13:17:22.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:22.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273175+0000", 2024-09-13T13:17:22.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1320000000000002e-06 2024-09-13T13:17:22.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:22.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.278797+0000", 2024-09-13T13:17:22.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056220580000000001 2024-09-13T13:17:22.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:22.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.278916+0000", 2024-09-13T13:17:22.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118986 2024-09-13T13:17:22.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:17:22.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.279027+0000", 2024-09-13T13:17:22.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111014 2024-09-13T13:17:22.305 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.305 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:22.305 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.305 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.26071.0:238464 10.7 e406/334)", 2024-09-13T13:17:22.305 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:22.278681+0000", 2024-09-13T13:17:22.305 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00019359999999999999, 2024-09-13T13:17:22.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00048136400000000002, 2024-09-13T13:17:22.306 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:22.306 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:22.306 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "queued for pg", 2024-09-13T13:17:22.306 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:22.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:22.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.278681+0000", 2024-09-13T13:17:22.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:22.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.278681+0000", 2024-09-13T13:17:22.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.278683+0000", 2024-09-13T13:17:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5719999999999999e-06 2024-09-13T13:17:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.278689+0000", 2024-09-13T13:17:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9370000000000001e-06 2024-09-13T13:17:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:22.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.278707+0000", 2024-09-13T13:17:22.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8189000000000002e-05 2024-09-13T13:17:22.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:22.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.278715+0000", 2024-09-13T13:17:22.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3489999999999992e-06 2024-09-13T13:17:22.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.308 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:22.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.309 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:238458 10.7 10:e1d4c4e0:::benchmark_data_smithi052_153703_object111078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-13T13:17:22.309 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:22.257604+0000", 2024-09-13T13:17:22.309 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.021270753999999999, 2024-09-13T13:17:22.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021599230000000001, 2024-09-13T13:17:22.309 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:22.309 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:22.309 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:17:22.309 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:22.309 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:22.309 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:22.309 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 238458 2024-09-13T13:17:22.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.310 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:22.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:22.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.257604+0000", 2024-09-13T13:17:22.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:22.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.257604+0000", 2024-09-13T13:17:22.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:22.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.257605+0000", 2024-09-13T13:17:22.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1800000000000005e-07 2024-09-13T13:17:22.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:22.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.257608+0000", 2024-09-13T13:17:22.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.196e-06 2024-09-13T13:17:22.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:22.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.257613+0000", 2024-09-13T13:17:22.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5609999999999998e-06 2024-09-13T13:17:22.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:22.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.257618+0000", 2024-09-13T13:17:22.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399999999999996e-06 2024-09-13T13:17:22.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:22.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263544+0000", 2024-09-13T13:17:22.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059261219999999998 2024-09-13T13:17:22.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:22.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.265433+0000", 2024-09-13T13:17:22.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001889195 2024-09-13T13:17:22.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:17:22.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.265536+0000", 2024-09-13T13:17:22.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010302300000000001 2024-09-13T13:17:22.315 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.315 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:22.315 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.316 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:238460 10.f 10:fe751228:::benchmark_data_smithi052_153703_object111080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-13T13:17:22.316 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:22.259598+0000", 2024-09-13T13:17:22.316 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.019276661, 2024-09-13T13:17:22.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019664335000000002, 2024-09-13T13:17:22.316 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:22.316 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:22.316 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:17:22.316 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:22.316 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:22.316 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:22.317 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 238460 2024-09-13T13:17:22.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.317 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:22.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:22.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.259598+0000", 2024-09-13T13:17:22.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:22.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.259598+0000", 2024-09-13T13:17:22.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:22.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.259600+0000", 2024-09-13T13:17:22.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.505e-06 2024-09-13T13:17:22.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:22.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.259604+0000", 2024-09-13T13:17:22.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1570000000000002e-06 2024-09-13T13:17:22.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:22.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.259615+0000", 2024-09-13T13:17:22.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1732e-05 2024-09-13T13:17:22.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:22.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.259622+0000", 2024-09-13T13:17:22.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.99e-06 2024-09-13T13:17:22.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:22.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263883+0000", 2024-09-13T13:17:22.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042601929999999998 2024-09-13T13:17:22.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:22.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263977+0000", 2024-09-13T13:17:22.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4767999999999998e-05 2024-09-13T13:17:22.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:17:22.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.264075+0000", 2024-09-13T13:17:22.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7745999999999996e-05 2024-09-13T13:17:22.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:22.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.275574+0000", 2024-09-13T13:17:22.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011499343 2024-09-13T13:17:22.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.321 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:22.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.321 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:238462 10.a 10:54b15734:::benchmark_data_smithi052_153703_object111082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:22.263142+0000", 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.015732546, 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016182370000000001, 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 238462 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:22.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263142+0000", 2024-09-13T13:17:22.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:22.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263142+0000", 2024-09-13T13:17:22.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:22.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263144+0000", 2024-09-13T13:17:22.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.708e-06 2024-09-13T13:17:22.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:22.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263149+0000", 2024-09-13T13:17:22.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2730000000000002e-06 2024-09-13T13:17:22.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:22.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263159+0000", 2024-09-13T13:17:22.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6080000000000002e-06 2024-09-13T13:17:22.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:22.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263164+0000", 2024-09-13T13:17:22.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8230000000000002e-06 2024-09-13T13:17:22.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:22.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263211+0000", 2024-09-13T13:17:22.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7068999999999999e-05 2024-09-13T13:17:22.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:22.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263327+0000", 2024-09-13T13:17:22.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116035 2024-09-13T13:17:22.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:17:22.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263455+0000", 2024-09-13T13:17:22.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012862099999999999 2024-09-13T13:17:22.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:22.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.268906+0000", 2024-09-13T13:17:22.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0054507069999999999 2024-09-13T13:17:22.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:22.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273147+0000", 2024-09-13T13:17:22.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042408419999999999 2024-09-13T13:17:22.327 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.327 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:22.327 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.328 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:238463 10.0 10:0fc6c98c:::benchmark_data_smithi052_153703_object111083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-13T13:17:22.328 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:22.263181+0000", 2024-09-13T13:17:22.328 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.015693287, 2024-09-13T13:17:22.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016209049999999999, 2024-09-13T13:17:22.328 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:22.328 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:22.328 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:17:22.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:22.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:22.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:22.328 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 238463 2024-09-13T13:17:22.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.329 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:22.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:22.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263181+0000", 2024-09-13T13:17:22.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:22.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263181+0000", 2024-09-13T13:17:22.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:22.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263182+0000", 2024-09-13T13:17:22.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4099999999999998e-07 2024-09-13T13:17:22.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:22.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263185+0000", 2024-09-13T13:17:22.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7039999999999999e-06 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263191+0000", 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2600000000000002e-06 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263196+0000", 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2750000000000003e-06 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:22.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263667+0000", 2024-09-13T13:17:22.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00047034699999999997 2024-09-13T13:17:22.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:22.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263769+0000", 2024-09-13T13:17:22.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102353 2024-09-13T13:17:22.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:17:22.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263882+0000", 2024-09-13T13:17:22.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112713 2024-09-13T13:17:22.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:22.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.274394+0000", 2024-09-13T13:17:22.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010512755 2024-09-13T13:17:22.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:22.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.274977+0000", 2024-09-13T13:17:22.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058268800000000004 2024-09-13T13:17:22.333 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.334 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:22.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.334 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:238464 10.7 10:efa34f29:::benchmark_data_smithi052_153703_object111084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-13T13:17:22.334 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:22.263207+0000", 2024-09-13T13:17:22.334 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.015667538000000002, 2024-09-13T13:17:22.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016248511, 2024-09-13T13:17:22.334 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:22.334 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:22.334 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 238464 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263207+0000", 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:22.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263207+0000", 2024-09-13T13:17:22.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:22.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263208+0000", 2024-09-13T13:17:22.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1900000000000003e-07 2024-09-13T13:17:22.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:22.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263210+0000", 2024-09-13T13:17:22.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5129999999999999e-06 2024-09-13T13:17:22.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:22.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263215+0000", 2024-09-13T13:17:22.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5220000000000001e-06 2024-09-13T13:17:22.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:22.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263220+0000", 2024-09-13T13:17:22.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9010000000000002e-06 2024-09-13T13:17:22.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:22.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.275229+0000", 2024-09-13T13:17:22.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012009505 2024-09-13T13:17:22.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:22.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.275327+0000", 2024-09-13T13:17:22.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7553000000000006e-05 2024-09-13T13:17:22.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:17:22.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.275429+0000", 2024-09-13T13:17:22.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102079 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:238466 10.d 10:b73b73d1:::benchmark_data_smithi052_153703_object111086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:22.263227+0000", 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.015648038, 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016285366999999999, 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:22.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:22.340 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 238466 2024-09-13T13:17:22.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.340 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:22.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:22.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263227+0000", 2024-09-13T13:17:22.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:22.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263227+0000", 2024-09-13T13:17:22.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:22.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:22.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263227+0000", 2024-09-13T13:17:22.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0500000000000004e-07 2024-09-13T13:17:22.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:22.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263230+0000", 2024-09-13T13:17:22.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4140000000000001e-06 2024-09-13T13:17:22.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263235+0000", 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9599999999999999e-06 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263239+0000", 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5399999999999997e-06 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263283+0000", 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3979e-05 2024-09-13T13:17:22.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:22.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263378+0000", 2024-09-13T13:17:22.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4928999999999999e-05 2024-09-13T13:17:22.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:17:22.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.263472+0000", 2024-09-13T13:17:22.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4102000000000001e-05 2024-09-13T13:17:22.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:22.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273187+0000", 2024-09-13T13:17:22.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097145189999999996 2024-09-13T13:17:22.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:22.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:22.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:22.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:22.273227+0000", 2024-09-13T13:17:22.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0351e-05 2024-09-13T13:17:22.344 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.345 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:22.345 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.345 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:22.345 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:17:22.345 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 12 2024-09-13T13:17:22.345 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:17:22.345 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:22.377 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:22.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:22.479 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:22.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:22 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:17:22.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:22 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:17:22.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:22 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4243831907' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:17:22.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:22 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:17:22.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:22 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:17:22.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:22 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4243831907' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:17:22.510 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:22.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:22.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:22.559 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:22 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:17:22.559 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:22 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:17:22.559 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:22 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4243831907' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:17:22.612 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:22.651 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:22.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:22.752 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:22.792 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:22.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:22.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:22.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:22.927 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:22.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:23.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:23.070 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:23.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:23.171 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:23.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:23.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:23.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:23.288 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:17:23.289 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:17:23.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[29471]: pgmap v811: 148 pgs: 148 active+clean; 1.2 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 879 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-13T13:17:23.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:23.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4243831907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:17:23.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[29471]: osdmap e406: 8 total, 8 up, 7 in 2024-09-13T13:17:23.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[29471]: 3.0 scrub starts 2024-09-13T13:17:23.290 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[29471]: 3.0 scrub ok 2024-09-13T13:17:23.290 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:17:23.290 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:17:23.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[36682]: pgmap v811: 148 pgs: 148 active+clean; 1.2 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 879 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-13T13:17:23.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:23.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4243831907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:17:23.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[36682]: osdmap e406: 8 total, 8 up, 7 in 2024-09-13T13:17:23.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[36682]: 3.0 scrub starts 2024-09-13T13:17:23.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[36682]: 3.0 scrub ok 2024-09-13T13:17:23.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:17:23.290 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:17:23.316 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:17:23.316 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:17:23.321 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:23.341 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:17:23.342 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:17:23.347 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:23.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:23.358 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:17:23.358 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:17:23.395 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:17:23.395 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:17:23.430 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:17:23.430 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:17:23.449 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:23.467 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:17:23.467 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:17:23.489 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:23.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:23.495 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:17:23.495 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:17:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:23 smithi167 ceph-mon[33178]: pgmap v811: 148 pgs: 148 active+clean; 1.2 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 879 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-13T13:17:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:23 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:23 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4243831907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:17:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:23 smithi167 ceph-mon[33178]: osdmap e406: 8 total, 8 up, 7 in 2024-09-13T13:17:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:23 smithi167 ceph-mon[33178]: 3.0 scrub starts 2024-09-13T13:17:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:23 smithi167 ceph-mon[33178]: 3.0 scrub ok 2024-09-13T13:17:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:23 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:17:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:23 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:17:23.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:23.625 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:23.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:23.727 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:23.765 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:23.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:23.818 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T13:17:23.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:23.913 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:23.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:24.015 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:24.059 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:24.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:24.121 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:23 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a[29467]: 2024-09-13T13:17:23.813+0000 7effa8af8640 -1 mon.a@0(leader).osd e407 definitely_dead 0 2024-09-13T13:17:24.162 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:24.191 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:24.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:24.222 INFO:teuthology.orchestra.run.smithi052.stderr:marked in osd.2. 2024-09-13T13:17:24.270 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:17:24.292 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:24.312 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:24.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:24.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:24.414 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:24.437 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:24.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:24.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:17:24.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:17:24.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[29471]: osdmap e407: 8 total, 8 up, 7 in 2024-09-13T13:17:24.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3527266273' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:24.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[29471]: 5.1e scrub starts 2024-09-13T13:17:24.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[29471]: 5.1e scrub ok 2024-09-13T13:17:24.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3044263469' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T13:17:24.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:17:24.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:17:24.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[36682]: osdmap e407: 8 total, 8 up, 7 in 2024-09-13T13:17:24.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3527266273' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:24.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[36682]: 5.1e scrub starts 2024-09-13T13:17:24.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[36682]: 5.1e scrub ok 2024-09-13T13:17:24.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:24 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3044263469' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T13:17:24.534 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:24 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:17:24.535 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:24 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:17:24.535 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:24 smithi167 ceph-mon[33178]: osdmap e407: 8 total, 8 up, 7 in 2024-09-13T13:17:24.535 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:24 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3527266273' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:24.535 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:24 smithi167 ceph-mon[33178]: 5.1e scrub starts 2024-09-13T13:17:24.535 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:24 smithi167 ceph-mon[33178]: 5.1e scrub ok 2024-09-13T13:17:24.535 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:24 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3044263469' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T13:17:24.539 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:24.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:24.563 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:24.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:24.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:24.686 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:24.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:24.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:24.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:24.810 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:24.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:24.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:24.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:17:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:17:24.905 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-13T13:17:24.912 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:24.934 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:24.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:25.036 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:25.046 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:17:25.058 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:25.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:25.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:25.184 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:25.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:25.286 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:25.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:25.308 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:25.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:25.342 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:25.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:25.431 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:25.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:25.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:25 smithi052 ceph-mon[36682]: pgmap v814: 148 pgs: 148 active+clean; 1.0 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 882 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-13T13:17:25.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:25 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3044263469' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-13T13:17:25.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:25 smithi052 ceph-mon[36682]: osdmap e408: 8 total, 8 up, 8 in 2024-09-13T13:17:25.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:25 smithi052 ceph-mon[36682]: 10.b scrub starts 2024-09-13T13:17:25.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:25.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:25 smithi052 ceph-mon[36682]: 10.b scrub ok 2024-09-13T13:17:25.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:25 smithi052 ceph-mon[29471]: pgmap v814: 148 pgs: 148 active+clean; 1.0 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 882 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-13T13:17:25.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:25 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3044263469' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-13T13:17:25.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:25 smithi052 ceph-mon[29471]: osdmap e408: 8 total, 8 up, 8 in 2024-09-13T13:17:25.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:25 smithi052 ceph-mon[29471]: 10.b scrub starts 2024-09-13T13:17:25.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:25.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:25 smithi052 ceph-mon[29471]: 10.b scrub ok 2024-09-13T13:17:25.534 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:25.555 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:25.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:25.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:25 smithi167 ceph-mon[33178]: pgmap v814: 148 pgs: 148 active+clean; 1.0 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 882 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-13T13:17:25.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:25 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3044263469' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-13T13:17:25.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:25 smithi167 ceph-mon[33178]: osdmap e408: 8 total, 8 up, 8 in 2024-09-13T13:17:25.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:25 smithi167 ceph-mon[33178]: 10.b scrub starts 2024-09-13T13:17:25.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:25.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:25 smithi167 ceph-mon[33178]: 10.b scrub ok 2024-09-13T13:17:25.657 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:25.679 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:25.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:25.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:25.805 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:25.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:25.906 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:25.929 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:25.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:25.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:26.031 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:26.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:26.057 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:26.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:26.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:26.166 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:17:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:26.183 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:26.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:26.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:26.306 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:26.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:26.403 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:26 smithi167 ceph-mon[33178]: 10.c scrub starts 2024-09-13T13:17:26.403 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:26 smithi167 ceph-mon[33178]: 10.c scrub ok 2024-09-13T13:17:26.403 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:26 smithi167 ceph-mon[33178]: 6.19 scrub starts 2024-09-13T13:17:26.403 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:26 smithi167 ceph-mon[33178]: 6.19 scrub ok 2024-09-13T13:17:26.403 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:26 smithi167 ceph-mon[33178]: osdmap e409: 8 total, 8 up, 8 in 2024-09-13T13:17:26.403 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:26 smithi167 ceph-mon[33178]: 2.0 scrub starts 2024-09-13T13:17:26.403 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:26 smithi167 ceph-mon[33178]: 2.0 scrub ok 2024-09-13T13:17:26.403 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:26 smithi167 ceph-mon[33178]: 3.1f scrub starts 2024-09-13T13:17:26.403 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:26 smithi167 ceph-mon[33178]: 3.1f scrub ok 2024-09-13T13:17:26.403 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:26 smithi167 ceph-mon[33178]: osdmap e410: 8 total, 8 up, 8 in 2024-09-13T13:17:26.404 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:17:26.404 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:17:26.408 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:26.439 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:26.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:26.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:26.488 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:17:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:17:26.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[29471]: 10.c scrub starts 2024-09-13T13:17:26.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[29471]: 10.c scrub ok 2024-09-13T13:17:26.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[29471]: 6.19 scrub starts 2024-09-13T13:17:26.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[29471]: 6.19 scrub ok 2024-09-13T13:17:26.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[29471]: osdmap e409: 8 total, 8 up, 8 in 2024-09-13T13:17:26.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[29471]: 2.0 scrub starts 2024-09-13T13:17:26.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[29471]: 2.0 scrub ok 2024-09-13T13:17:26.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[29471]: 3.1f scrub starts 2024-09-13T13:17:26.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[29471]: 3.1f scrub ok 2024-09-13T13:17:26.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[29471]: osdmap e410: 8 total, 8 up, 8 in 2024-09-13T13:17:26.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[36682]: 10.c scrub starts 2024-09-13T13:17:26.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[36682]: 10.c scrub ok 2024-09-13T13:17:26.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[36682]: 6.19 scrub starts 2024-09-13T13:17:26.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[36682]: 6.19 scrub ok 2024-09-13T13:17:26.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[36682]: osdmap e409: 8 total, 8 up, 8 in 2024-09-13T13:17:26.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[36682]: 2.0 scrub starts 2024-09-13T13:17:26.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[36682]: 2.0 scrub ok 2024-09-13T13:17:26.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[36682]: 3.1f scrub starts 2024-09-13T13:17:26.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[36682]: 3.1f scrub ok 2024-09-13T13:17:26.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:26 smithi052 ceph-mon[36682]: osdmap e410: 8 total, 8 up, 8 in 2024-09-13T13:17:26.541 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:26.547 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:26.592 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:26.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:26.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:26.694 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:26.733 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:26.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:26.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:26.835 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:26.869 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:26.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:26.970 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:27.007 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:27.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:27.108 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:27.144 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T13:17:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:27.146 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:27.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:27.247 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:27.257 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:17:27.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:27.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:27.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:27.372 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:27 smithi167 ceph-mon[33178]: pgmap v817: 148 pgs: 6 peering, 142 active+clean; 1000 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 748 op/s 2024-09-13T13:17:27.372 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:27 smithi167 ceph-mon[33178]: 4.12 scrub starts 2024-09-13T13:17:27.372 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:27 smithi167 ceph-mon[33178]: 4.12 scrub ok 2024-09-13T13:17:27.372 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:27 smithi167 ceph-mon[33178]: 10.e starting backfill to osd.2 from (367'8510,388'12876] MIN to 407'15028 2024-09-13T13:17:27.372 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:27 smithi167 ceph-mon[33178]: 10.9 starting backfill to osd.2 from (363'7694,386'12738] MIN to 407'14879 2024-09-13T13:17:27.372 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:27 smithi167 ceph-mon[33178]: 10.3 starting backfill to osd.2 from (367'8484,394'13583] MIN to 407'15084 2024-09-13T13:17:27.372 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:27 smithi167 ceph-mon[33178]: 10.f starting backfill to osd.2 from (366'8268,395'13517] MIN to 407'15008 2024-09-13T13:17:27.373 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:27 smithi167 ceph-mon[33178]: 10.5 starting backfill to osd.2 from (368'8771,392'13367] MIN to 407'15006 2024-09-13T13:17:27.373 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:27 smithi167 ceph-mon[33178]: 10.8 starting backfill to osd.2 from (368'8996,392'13568] MIN to 407'15221 2024-09-13T13:17:27.373 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:27 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1651054121' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:27.373 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:27 smithi167 ceph-mon[33178]: from='client.36530 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:27.396 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:27.397 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:17:27.398 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:17:27.426 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:17:27.426 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:17:27.441 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:27.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:27.458 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:17:27.458 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:17:27.488 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:17:27.488 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:17:27.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[29471]: pgmap v817: 148 pgs: 6 peering, 142 active+clean; 1000 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 748 op/s 2024-09-13T13:17:27.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[29471]: 4.12 scrub starts 2024-09-13T13:17:27.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[29471]: 4.12 scrub ok 2024-09-13T13:17:27.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[29471]: 10.e starting backfill to osd.2 from (367'8510,388'12876] MIN to 407'15028 2024-09-13T13:17:27.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[29471]: 10.9 starting backfill to osd.2 from (363'7694,386'12738] MIN to 407'14879 2024-09-13T13:17:27.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[29471]: 10.3 starting backfill to osd.2 from (367'8484,394'13583] MIN to 407'15084 2024-09-13T13:17:27.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[29471]: 10.f starting backfill to osd.2 from (366'8268,395'13517] MIN to 407'15008 2024-09-13T13:17:27.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[29471]: 10.5 starting backfill to osd.2 from (368'8771,392'13367] MIN to 407'15006 2024-09-13T13:17:27.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[29471]: 10.8 starting backfill to osd.2 from (368'8996,392'13568] MIN to 407'15221 2024-09-13T13:17:27.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1651054121' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:27.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[29471]: from='client.36530 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:27.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[36682]: pgmap v817: 148 pgs: 6 peering, 142 active+clean; 1000 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 748 op/s 2024-09-13T13:17:27.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[36682]: 4.12 scrub starts 2024-09-13T13:17:27.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[36682]: 4.12 scrub ok 2024-09-13T13:17:27.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[36682]: 10.e starting backfill to osd.2 from (367'8510,388'12876] MIN to 407'15028 2024-09-13T13:17:27.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[36682]: 10.9 starting backfill to osd.2 from (363'7694,386'12738] MIN to 407'14879 2024-09-13T13:17:27.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[36682]: 10.3 starting backfill to osd.2 from (367'8484,394'13583] MIN to 407'15084 2024-09-13T13:17:27.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[36682]: 10.f starting backfill to osd.2 from (366'8268,395'13517] MIN to 407'15008 2024-09-13T13:17:27.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[36682]: 10.5 starting backfill to osd.2 from (368'8771,392'13367] MIN to 407'15006 2024-09-13T13:17:27.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[36682]: 10.8 starting backfill to osd.2 from (368'8996,392'13568] MIN to 407'15221 2024-09-13T13:17:27.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1651054121' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:27.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:27 smithi052 ceph-mon[36682]: from='client.36530 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:27.527 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:17:27.527 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:17:27.543 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:27.570 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:17:27.570 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:17:27.580 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:27.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:27.605 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:17:27.605 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:17:27.636 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:17:27.636 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:17:27.651 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:17:27.682 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:27.712 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:27.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:27.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:27.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:27.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:27.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:27.989 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:27.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:27.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:28.091 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:28.120 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:28.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:28.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:28.222 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:28.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:28.257 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:28.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:28.308 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:17:28.309 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:17:28.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:17:28.309 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:17:28.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.309 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:178616 10.f 10:f45131d6:::benchmark_data_smithi052_153703_object51236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T13:17:28.309 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:22.233047+0000", 2024-09-13T13:17:28.309 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.072230181999998, 2024-09-13T13:17:28.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.33948704099999999, 2024-09-13T13:17:28.309 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.309 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.309 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.309 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.310 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 178616 2024-09-13T13:17:28.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.310 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.233047+0000", 2024-09-13T13:17:28.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.233047+0000", 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.233049+0000", 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7540000000000001e-06 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.233053+0000", 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3919999999999996e-06 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.233061+0000", 2024-09-13T13:17:28.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5750000000000003e-06 2024-09-13T13:17:28.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.233067+0000", 2024-09-13T13:17:28.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4749999999999998e-06 2024-09-13T13:17:28.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.233124+0000", 2024-09-13T13:17:28.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6629000000000003e-05 2024-09-13T13:17:28.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.233241+0000", 2024-09-13T13:17:28.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011769000000000001 2024-09-13T13:17:28.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3,4", 2024-09-13T13:17:28.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.233359+0000", 2024-09-13T13:17:28.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117318 2024-09-13T13:17:28.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.236774+0000", 2024-09-13T13:17:28.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034149089999999998 2024-09-13T13:17:28.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.244748+0000", 2024-09-13T13:17:28.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079744689999999997 2024-09-13T13:17:28.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.244826+0000", 2024-09-13T13:17:28.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7869999999999998e-05 2024-09-13T13:17:28.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.572389+0000", 2024-09-13T13:17:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.32756340099999998 2024-09-13T13:17:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.572447+0000", 2024-09-13T13:17:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7142000000000003e-05 2024-09-13T13:17:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.572534+0000", 2024-09-13T13:17:28.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7416999999999993e-05 2024-09-13T13:17:28.316 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.316 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.316 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.316 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:178620 10.f 10:f7cb9e30:::benchmark_data_smithi052_153703_object51240:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T13:17:28.316 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:22.236293+0000", 2024-09-13T13:17:28.317 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.068984512, 2024-09-13T13:17:28.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.34223717999999997, 2024-09-13T13:17:28.317 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.317 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.317 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.317 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.317 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.317 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.317 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 178620 2024-09-13T13:17:28.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.317 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.236293+0000", 2024-09-13T13:17:28.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.236293+0000", 2024-09-13T13:17:28.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.236294+0000", 2024-09-13T13:17:28.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7829999999999999e-06 2024-09-13T13:17:28.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.236300+0000", 2024-09-13T13:17:28.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2469999999999999e-06 2024-09-13T13:17:28.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.236310+0000", 2024-09-13T13:17:28.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0215e-05 2024-09-13T13:17:28.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.236317+0000", 2024-09-13T13:17:28.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2880000000000004e-06 2024-09-13T13:17:28.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.236810+0000", 2024-09-13T13:17:28.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00049279700000000001 2024-09-13T13:17:28.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.236917+0000", 2024-09-13T13:17:28.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010739300000000001 2024-09-13T13:17:28.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3,4", 2024-09-13T13:17:28.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.237034+0000", 2024-09-13T13:17:28.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011684999999999999 2024-09-13T13:17:28.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.244864+0000", 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078294119999999991 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.244882+0000", 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8703e-05 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.246808+0000", 2024-09-13T13:17:28.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019260220000000001 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.578308+0000", 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.33150000200000002 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.578416+0000", 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108001 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.578530+0000", 2024-09-13T13:17:28.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113467 2024-09-13T13:17:28.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.324 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.324 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:178624 10.f 10:f7e657e0:::benchmark_data_smithi052_153703_object51244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T13:17:28.324 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:22.239639+0000", 2024-09-13T13:17:28.324 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.065637972000005, 2024-09-13T13:17:28.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.34519390100000003, 2024-09-13T13:17:28.324 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.324 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.325 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.325 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 178624 2024-09-13T13:17:28.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.325 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.239639+0000", 2024-09-13T13:17:28.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.239639+0000", 2024-09-13T13:17:28.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.239640+0000", 2024-09-13T13:17:28.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1999999999999999e-07 2024-09-13T13:17:28.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.239643+0000", 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8449999999999999e-06 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.239648+0000", 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2950000000000001e-06 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.239653+0000", 2024-09-13T13:17:28.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8210000000000001e-06 2024-09-13T13:17:28.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.241099+0000", 2024-09-13T13:17:28.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014458889999999999 2024-09-13T13:17:28.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.241206+0000", 2024-09-13T13:17:28.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010716 2024-09-13T13:17:28.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3,4", 2024-09-13T13:17:28.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.241321+0000", 2024-09-13T13:17:28.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115016 2024-09-13T13:17:28.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.244888+0000", 2024-09-13T13:17:28.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003567626 2024-09-13T13:17:28.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.246370+0000", 2024-09-13T13:17:28.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001481986 2024-09-13T13:17:28.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.246849+0000", 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00047833199999999999 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.584661+0000", 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.33781224599999998 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.584719+0000", 2024-09-13T13:17:28.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7637999999999999e-05 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.584833+0000", 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011432700000000001 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:178643 10.f 10:f916b983:::benchmark_data_smithi052_153703_object51263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:22.254856+0000", 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.050420629000001, 2024-09-13T13:17:28.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.33035033400000002, 2024-09-13T13:17:28.332 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.332 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.332 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.332 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.332 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.332 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.332 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 178643 2024-09-13T13:17:28.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.332 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.254856+0000", 2024-09-13T13:17:28.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.254856+0000", 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.254858+0000", 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2240000000000001e-06 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.254862+0000", 2024-09-13T13:17:28.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6260000000000003e-06 2024-09-13T13:17:28.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.254878+0000", 2024-09-13T13:17:28.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5964000000000002e-05 2024-09-13T13:17:28.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.254885+0000", 2024-09-13T13:17:28.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7549999999999997e-06 2024-09-13T13:17:28.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.255058+0000", 2024-09-13T13:17:28.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000173387 2024-09-13T13:17:28.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.255175+0000", 2024-09-13T13:17:28.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117069 2024-09-13T13:17:28.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3,4", 2024-09-13T13:17:28.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.255299+0000", 2024-09-13T13:17:28.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123041 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.258614+0000", 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033152490000000001 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.259062+0000", 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00044821900000000003 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.259665+0000", 2024-09-13T13:17:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00060347899999999997 2024-09-13T13:17:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.584870+0000", 2024-09-13T13:17:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.32520485100000002 2024-09-13T13:17:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.585100+0000", 2024-09-13T13:17:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022988199999999999 2024-09-13T13:17:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.585207+0000", 2024-09-13T13:17:28.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106588 2024-09-13T13:17:28.338 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.338 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.338 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.338 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:178651 10.7 10:e6d19818:::benchmark_data_smithi052_153703_object51271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T13:17:28.338 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:22.267341+0000", 2024-09-13T13:17:28.338 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.037935646999998, 2024-09-13T13:17:28.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31789864899999998, 2024-09-13T13:17:28.338 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.339 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.339 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.340 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.340 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.340 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.340 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 178651 2024-09-13T13:17:28.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.340 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.267341+0000", 2024-09-13T13:17:28.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.267341+0000", 2024-09-13T13:17:28.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.267343+0000", 2024-09-13T13:17:28.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7010000000000001e-06 2024-09-13T13:17:28.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.267348+0000", 2024-09-13T13:17:28.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1000000000000003e-06 2024-09-13T13:17:28.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.267359+0000", 2024-09-13T13:17:28.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0801999999999999e-05 2024-09-13T13:17:28.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.267366+0000", 2024-09-13T13:17:28.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1910000000000003e-06 2024-09-13T13:17:28.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.267433+0000", 2024-09-13T13:17:28.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7008999999999995e-05 2024-09-13T13:17:28.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.267551+0000", 2024-09-13T13:17:28.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117718 2024-09-13T13:17:28.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:17:28.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.267665+0000", 2024-09-13T13:17:28.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114268 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.270728+0000", 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030624570000000002 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.270952+0000", 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022468 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.585036+0000", 2024-09-13T13:17:28.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31408324799999998 2024-09-13T13:17:28.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.585188+0000", 2024-09-13T13:17:28.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015240499999999999 2024-09-13T13:17:28.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.585240+0000", 2024-09-13T13:17:28.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2070000000000001e-05 2024-09-13T13:17:28.345 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.345 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.346 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.346 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:178652 10.f 10:f5b33db1:::benchmark_data_smithi052_153703_object51272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T13:17:28.346 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:22.271454+0000", 2024-09-13T13:17:28.346 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.033822752000006, 2024-09-13T13:17:28.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31424675800000001, 2024-09-13T13:17:28.346 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.346 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.346 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.346 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.346 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 178652 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.271454+0000", 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.271454+0000", 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.271456+0000", 2024-09-13T13:17:28.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.99e-06 2024-09-13T13:17:28.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.271465+0000", 2024-09-13T13:17:28.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0999999999999993e-06 2024-09-13T13:17:28.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.271479+0000", 2024-09-13T13:17:28.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3134e-05 2024-09-13T13:17:28.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.271486+0000", 2024-09-13T13:17:28.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:17:28.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.271543+0000", 2024-09-13T13:17:28.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6796999999999998e-05 2024-09-13T13:17:28.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.271677+0000", 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013422900000000001 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3,4", 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.271824+0000", 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014656599999999999 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.274689+0000", 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028652740000000001 2024-09-13T13:17:28.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.274733+0000", 2024-09-13T13:17:28.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3772e-05 2024-09-13T13:17:28.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.274744+0000", 2024-09-13T13:17:28.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1378e-05 2024-09-13T13:17:28.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.585247+0000", 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31050324099999999 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.585592+0000", 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000344385 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:22.585701+0000", 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109412 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.352 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:221659 10.f 10:f8bff29a:::benchmark_data_smithi052_153703_object94279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:06.976121+0000", 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.329156398999999, 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.40928983600000002, 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 221659 2024-09-13T13:17:28.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.354 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:06.976121+0000", 2024-09-13T13:17:28.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:06.976121+0000", 2024-09-13T13:17:28.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:06.976122+0000", 2024-09-13T13:17:28.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1069999999999999e-06 2024-09-13T13:17:28.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:06.976128+0000", 2024-09-13T13:17:28.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1029999999999998e-06 2024-09-13T13:17:28.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:06.976137+0000", 2024-09-13T13:17:28.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8910000000000001e-06 2024-09-13T13:17:28.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:06.976143+0000", 2024-09-13T13:17:28.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5180000000000001e-06 2024-09-13T13:17:28.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.097099+0000", 2024-09-13T13:17:28.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.120955655 2024-09-13T13:17:28.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.097222+0000", 2024-09-13T13:17:28.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012321899999999999 2024-09-13T13:17:28.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:17:28.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.097350+0000", 2024-09-13T13:17:28.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000127928 2024-09-13T13:17:28.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.100620+0000", 2024-09-13T13:17:28.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003269522 2024-09-13T13:17:28.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385263+0000", 2024-09-13T13:17:28.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.284643545 2024-09-13T13:17:28.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385310+0000", 2024-09-13T13:17:28.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7227e-05 2024-09-13T13:17:28.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385363+0000", 2024-09-13T13:17:28.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2893999999999999e-05 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385411+0000", 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7227e-05 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:221668 10.7 10:e440f161:::benchmark_data_smithi052_153703_object94288:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:07.087332+0000", 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.217945381, 2024-09-13T13:17:28.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.30121268800000001, 2024-09-13T13:17:28.360 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.360 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.360 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.360 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 221668 2024-09-13T13:17:28.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.360 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.087332+0000", 2024-09-13T13:17:28.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.087332+0000", 2024-09-13T13:17:28.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.087335+0000", 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0359999999999998e-06 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.087347+0000", 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1864e-05 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093541+0000", 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061942339999999999 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093552+0000", 2024-09-13T13:17:28.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1236999999999999e-05 2024-09-13T13:17:28.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.100885+0000", 2024-09-13T13:17:28.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0073328389999999999 2024-09-13T13:17:28.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.100985+0000", 2024-09-13T13:17:28.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010002500000000001 2024-09-13T13:17:28.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:17:28.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101603+0000", 2024-09-13T13:17:28.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00061768399999999996 2024-09-13T13:17:28.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385276+0000", 2024-09-13T13:17:28.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.28367386 2024-09-13T13:17:28.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.388394+0000", 2024-09-13T13:17:28.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031179290000000002 2024-09-13T13:17:28.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.388449+0000", 2024-09-13T13:17:28.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4642000000000003e-05 2024-09-13T13:17:28.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.388500+0000", 2024-09-13T13:17:28.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0688e-05 2024-09-13T13:17:28.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.388544+0000", 2024-09-13T13:17:28.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4650000000000001e-05 2024-09-13T13:17:28.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.366 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:221670 10.7 10:ef63c662:::benchmark_data_smithi052_153703_object94290:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:07.093602+0000", 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.211674810000002, 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.30132297400000002, 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 221670 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093602+0000", 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093602+0000", 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093603+0000", 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7400000000000003e-07 2024-09-13T13:17:28.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093605+0000", 2024-09-13T13:17:28.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.441e-06 2024-09-13T13:17:28.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093610+0000", 2024-09-13T13:17:28.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9989999999999999e-06 2024-09-13T13:17:28.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093615+0000", 2024-09-13T13:17:28.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2300000000000002e-06 2024-09-13T13:17:28.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385399+0000", 2024-09-13T13:17:28.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.29178483300000002 2024-09-13T13:17:28.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385493+0000", 2024-09-13T13:17:28.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4025999999999997e-05 2024-09-13T13:17:28.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:17:28.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385563+0000", 2024-09-13T13:17:28.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9151000000000003e-05 2024-09-13T13:17:28.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.394643+0000", 2024-09-13T13:17:28.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090805550000000006 2024-09-13T13:17:28.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.394655+0000", 2024-09-13T13:17:28.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.164e-05 2024-09-13T13:17:28.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.394726+0000", 2024-09-13T13:17:28.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1674000000000005e-05 2024-09-13T13:17:28.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.394878+0000", 2024-09-13T13:17:28.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000151123 2024-09-13T13:17:28.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.394925+0000", 2024-09-13T13:17:28.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7728000000000001e-05 2024-09-13T13:17:28.373 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.373 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.373 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.373 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:221672 10.d 10:bf0fff57:::benchmark_data_smithi052_153703_object94292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-13T13:17:28.373 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:07.093620+0000", 2024-09-13T13:17:28.373 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.211657408000001, 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.30704841399999999, 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 221672 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093620+0000", 2024-09-13T13:17:28.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093620+0000", 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093620+0000", 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8299999999999997e-07 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093623+0000", 2024-09-13T13:17:28.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3910000000000001e-06 2024-09-13T13:17:28.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093627+0000", 2024-09-13T13:17:28.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8770000000000001e-06 2024-09-13T13:17:28.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093632+0000", 2024-09-13T13:17:28.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4789999999999998e-06 2024-09-13T13:17:28.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.096206+0000", 2024-09-13T13:17:28.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025745910000000002 2024-09-13T13:17:28.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.096304+0000", 2024-09-13T13:17:28.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7590000000000006e-05 2024-09-13T13:17:28.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:17:28.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.096407+0000", 2024-09-13T13:17:28.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102668 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.102890+0000", 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064835379999999996 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.388397+0000", 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.285506433 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400583+0000", 2024-09-13T13:17:28.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012186582 2024-09-13T13:17:28.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400628+0000", 2024-09-13T13:17:28.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4901000000000003e-05 2024-09-13T13:17:28.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400668+0000", 2024-09-13T13:17:28.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9881e-05 2024-09-13T13:17:28.379 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.379 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.380 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.380 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:221674 10.d 10:bbc45f7f:::benchmark_data_smithi052_153703_object94294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-13T13:17:28.380 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:07.093655+0000", 2024-09-13T13:17:28.380 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.211621783999998, 2024-09-13T13:17:28.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.307167511, 2024-09-13T13:17:28.380 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.380 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.380 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.380 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.380 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.380 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.381 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 221674 2024-09-13T13:17:28.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.381 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093655+0000", 2024-09-13T13:17:28.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093655+0000", 2024-09-13T13:17:28.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093656+0000", 2024-09-13T13:17:28.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3000000000000001e-07 2024-09-13T13:17:28.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093659+0000", 2024-09-13T13:17:28.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.875e-06 2024-09-13T13:17:28.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093663+0000", 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.093668+0000", 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2950000000000004e-06 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099392+0000", 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057243750000000003 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099486+0000", 2024-09-13T13:17:28.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3665999999999994e-05 2024-09-13T13:17:28.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.384 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:28.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:17:28.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099663+0000", 2024-09-13T13:17:28.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000177634 2024-09-13T13:17:28.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400487+0000", 2024-09-13T13:17:28.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.30082373299999998 2024-09-13T13:17:28.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400531+0000", 2024-09-13T13:17:28.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3859000000000001e-05 2024-09-13T13:17:28.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400708+0000", 2024-09-13T13:17:28.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000176978 2024-09-13T13:17:28.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400754+0000", 2024-09-13T13:17:28.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6400000000000003e-05 2024-09-13T13:17:28.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400823+0000", 2024-09-13T13:17:28.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8472000000000004e-05 2024-09-13T13:17:28.387 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.388 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.388 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.388 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:221679 10.7 10:ee5b0b67:::benchmark_data_smithi052_153703_object94299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-13T13:17:28.388 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:07.099547+0000", 2024-09-13T13:17:28.388 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.20572962, 2024-09-13T13:17:28.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.30510512099999998, 2024-09-13T13:17:28.388 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.388 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.388 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.388 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.388 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.389 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.389 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 221679 2024-09-13T13:17:28.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.389 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099547+0000", 2024-09-13T13:17:28.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099547+0000", 2024-09-13T13:17:28.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099549+0000", 2024-09-13T13:17:28.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.823e-06 2024-09-13T13:17:28.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099554+0000", 2024-09-13T13:17:28.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5929999999999999e-06 2024-09-13T13:17:28.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099562+0000", 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1310000000000006e-06 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099568+0000", 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3230000000000003e-06 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.389035+0000", 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.28946655999999998 2024-09-13T13:17:28.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.391696+0000", 2024-09-13T13:17:28.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026611370000000001 2024-09-13T13:17:28.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:17:28.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.391840+0000", 2024-09-13T13:17:28.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144356 2024-09-13T13:17:28.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.403568+0000", 2024-09-13T13:17:28.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01172718 2024-09-13T13:17:28.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.403737+0000", 2024-09-13T13:17:28.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016938899999999999 2024-09-13T13:17:28.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.404403+0000", 2024-09-13T13:17:28.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066559799999999999 2024-09-13T13:17:28.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.404546+0000", 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000143196 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.404653+0000", 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106835 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.394 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.26071.0:221672 10.d e397/334 ondisk, result = 0)", 2024-09-13T13:17:28.395 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:07.099721+0000", 2024-09-13T13:17:28.395 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.205555660999998, 2024-09-13T13:17:28.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.28544172800000001, 2024-09-13T13:17:28.395 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.395 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.395 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:17:28.395 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099721+0000", 2024-09-13T13:17:28.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099721+0000", 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099723+0000", 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.64e-06 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099727+0000", 2024-09-13T13:17:28.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7610000000000001e-06 2024-09-13T13:17:28.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099736+0000", 2024-09-13T13:17:28.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5670000000000001e-06 2024-09-13T13:17:28.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.099743+0000", 2024-09-13T13:17:28.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3960000000000003e-06 2024-09-13T13:17:28.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.102878+0000", 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031354429999999999 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.102885+0000", 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6529999999999997e-06 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385163+0000", 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.282278268 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.398 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:221683 10.d 10:bebf4492:::benchmark_data_smithi052_153703_object94303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-13T13:17:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:07.100681+0000", 2024-09-13T13:17:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.204596240000001, 2024-09-13T13:17:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.30028538500000002, 2024-09-13T13:17:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.399 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.400 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 221683 2024-09-13T13:17:28.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.400 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.100681+0000", 2024-09-13T13:17:28.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.100681+0000", 2024-09-13T13:17:28.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.100682+0000", 2024-09-13T13:17:28.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2330000000000001e-06 2024-09-13T13:17:28.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.100687+0000", 2024-09-13T13:17:28.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5379999999999996e-06 2024-09-13T13:17:28.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.100694+0000", 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1969999999999999e-06 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.100700+0000", 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1639999999999998e-06 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385232+0000", 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.284532071 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385363+0000", 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131397 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385444+0000", 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0741999999999994e-05 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400543+0000", 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015098623 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400861+0000", 2024-09-13T13:17:28.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031844499999999999 2024-09-13T13:17:28.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400895+0000", 2024-09-13T13:17:28.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4192000000000001e-05 2024-09-13T13:17:28.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400935+0000", 2024-09-13T13:17:28.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9583999999999999e-05 2024-09-13T13:17:28.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400966+0000", 2024-09-13T13:17:28.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1198999999999997e-05 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:221684 10.0 10:0c9cdd71:::benchmark_data_smithi052_153703_object94304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:07.101007+0000", 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.204269973999999, 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.29981641999999997, 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.406 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.407 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.407 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 221684 2024-09-13T13:17:28.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.407 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101007+0000", 2024-09-13T13:17:28.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101007+0000", 2024-09-13T13:17:28.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101008+0000", 2024-09-13T13:17:28.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1790000000000001e-06 2024-09-13T13:17:28.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101013+0000", 2024-09-13T13:17:28.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2300000000000002e-06 2024-09-13T13:17:28.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101020+0000", 2024-09-13T13:17:28.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1539999999999996e-06 2024-09-13T13:17:28.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101026+0000", 2024-09-13T13:17:28.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2040000000000002e-06 2024-09-13T13:17:28.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101273+0000", 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024681699999999998 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101371+0000", 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8513999999999999e-05 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101474+0000", 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103063 2024-09-13T13:17:28.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.388098+0000", 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.28662410500000002 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.392190+0000", 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040920970000000003 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400736+0000", 2024-09-13T13:17:28.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.008545647 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400781+0000", 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5395000000000002e-05 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400824+0000", 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2014999999999997e-05 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.26071.0:221659 10.f e397/396 ondisk, result = 0)", 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:07.101146+0000", 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.204131106999998, 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.284131615, 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101146+0000", 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101146+0000", 2024-09-13T13:17:28.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101147+0000", 2024-09-13T13:17:28.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2999999999999999e-07 2024-09-13T13:17:28.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101151+0000", 2024-09-13T13:17:28.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2189999999999998e-06 2024-09-13T13:17:28.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101159+0000", 2024-09-13T13:17:28.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5410000000000003e-06 2024-09-13T13:17:28.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101164+0000", 2024-09-13T13:17:28.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4700000000000001e-06 2024-09-13T13:17:28.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385247+0000", 2024-09-13T13:17:28.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.28408242900000003 2024-09-13T13:17:28.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385256+0000", 2024-09-13T13:17:28.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2390000000000008e-06 2024-09-13T13:17:28.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385278+0000", 2024-09-13T13:17:28.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1787e-05 2024-09-13T13:17:28.417 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.417 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.417 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.417 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.26071.0:221659 10.f e397/396 ondisk, result = 0)", 2024-09-13T13:17:28.417 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:07.101219+0000", 2024-09-13T13:17:28.417 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.204057788, 2024-09-13T13:17:28.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.284196112, 2024-09-13T13:17:28.417 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.417 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.417 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:17:28.417 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101219+0000", 2024-09-13T13:17:28.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101219+0000", 2024-09-13T13:17:28.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101220+0000", 2024-09-13T13:17:28.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.09e-07 2024-09-13T13:17:28.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101224+0000", 2024-09-13T13:17:28.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1339999999999997e-06 2024-09-13T13:17:28.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101231+0000", 2024-09-13T13:17:28.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5860000000000001e-06 2024-09-13T13:17:28.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101237+0000", 2024-09-13T13:17:28.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0700000000000003e-06 2024-09-13T13:17:28.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385298+0000", 2024-09-13T13:17:28.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.28406071999999999 2024-09-13T13:17:28.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385305+0000", 2024-09-13T13:17:28.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7360000000000001e-06 2024-09-13T13:17:28.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385415+0000", 2024-09-13T13:17:28.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109957 2024-09-13T13:17:28.421 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.421 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.421 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.421 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:221685 10.d 10:bef5378d:::benchmark_data_smithi052_153703_object94305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-13T13:17:28.421 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:07.101219+0000", 2024-09-13T13:17:28.421 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.204057598999999, 2024-09-13T13:17:28.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.29992391800000001, 2024-09-13T13:17:28.422 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.422 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.422 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.422 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 221685 2024-09-13T13:17:28.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.422 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101219+0000", 2024-09-13T13:17:28.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101219+0000", 2024-09-13T13:17:28.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101221+0000", 2024-09-13T13:17:28.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.28e-06 2024-09-13T13:17:28.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101226+0000", 2024-09-13T13:17:28.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8389999999999997e-06 2024-09-13T13:17:28.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101238+0000", 2024-09-13T13:17:28.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2295e-05 2024-09-13T13:17:28.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.101245+0000", 2024-09-13T13:17:28.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0589999999999997e-06 2024-09-13T13:17:28.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.389145+0000", 2024-09-13T13:17:28.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.28790028299999998 2024-09-13T13:17:28.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.391759+0000", 2024-09-13T13:17:28.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026132809999999999 2024-09-13T13:17:28.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-09-13T13:17:28.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.392390+0000", 2024-09-13T13:17:28.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000631479 2024-09-13T13:17:28.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400548+0000", 2024-09-13T13:17:28.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0081577079999999996 2024-09-13T13:17:28.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.401012+0000", 2024-09-13T13:17:28.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00046474899999999998 2024-09-13T13:17:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.401062+0000", 2024-09-13T13:17:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9675000000000001e-05 2024-09-13T13:17:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.401109+0000", 2024-09-13T13:17:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6924e-05 2024-09-13T13:17:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.401143+0000", 2024-09-13T13:17:28.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4345999999999999e-05 2024-09-13T13:17:28.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.428 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.428 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.26071.0:221674 10.d e397/334 ondisk, result = 0)", 2024-09-13T13:17:28.428 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:07.103855+0000", 2024-09-13T13:17:28.428 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.201422125000001, 2024-09-13T13:17:28.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.29669365199999997, 2024-09-13T13:17:28.428 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.428 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.103855+0000", 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.103855+0000", 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.103856+0000", 2024-09-13T13:17:28.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5570000000000001e-06 2024-09-13T13:17:28.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.103870+0000", 2024-09-13T13:17:28.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3217999999999999e-05 2024-09-13T13:17:28.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385152+0000", 2024-09-13T13:17:28.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.28128270799999999 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385171+0000", 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8779000000000001e-05 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400519+0000", 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015347444999999999 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400526+0000", 2024-09-13T13:17:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:17:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.400549+0000", 2024-09-13T13:17:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3078000000000002e-05 2024-09-13T13:17:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.432 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:221689 10.a 10:5332e3a1:::benchmark_data_smithi052_153703_object94309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-13T13:17:28.433 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:07.105122+0000", 2024-09-13T13:17:28.433 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.200154617999999, 2024-09-13T13:17:28.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.29604418300000002, 2024-09-13T13:17:28.433 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:17:28.433 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:17:28.433 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:17:28.433 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:17:28.433 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:17:28.433 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:28.433 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 221689 2024-09-13T13:17:28.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.433 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:17:28.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.105122+0000", 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.105122+0000", 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.105124+0000", 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0470000000000001e-06 2024-09-13T13:17:28.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.105130+0000", 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7030000000000003e-06 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385232+0000", 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.280101339 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385242+0000", 2024-09-13T13:17:28.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0181e-05 2024-09-13T13:17:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:17:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385298+0000", 2024-09-13T13:17:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5989000000000002e-05 2024-09-13T13:17:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:17:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385401+0000", 2024-09-13T13:17:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103552 2024-09-13T13:17:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.385500+0000", 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9185000000000007e-05 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.394644+0000", 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091436030000000001 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.394686+0000", 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1838999999999997e-05 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:17:28.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.401084+0000", 2024-09-13T13:17:28.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006397916 2024-09-13T13:17:28.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:17:28.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.401126+0000", 2024-09-13T13:17:28.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2014000000000001e-05 2024-09-13T13:17:28.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:17:28.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:17:28.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:17:28.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:07.401167+0000", 2024-09-13T13:17:28.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0815000000000001e-05 2024-09-13T13:17:28.439 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.439 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.439 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.439 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:17:28.439 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:17:28.439 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:17:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:28.465 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:28.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:28.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[29471]: 5.14 scrub starts 2024-09-13T13:17:28.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[29471]: 5.14 scrub ok 2024-09-13T13:17:28.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:17:28.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:17:28.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[29471]: from='client.36530 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:17:28.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[29471]: osdmap e411: 8 total, 8 up, 8 in 2024-09-13T13:17:28.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/193187756' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:28.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[36682]: 5.14 scrub starts 2024-09-13T13:17:28.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[36682]: 5.14 scrub ok 2024-09-13T13:17:28.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:17:28.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:17:28.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[36682]: from='client.36530 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:17:28.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[36682]: osdmap e411: 8 total, 8 up, 8 in 2024-09-13T13:17:28.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:28 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/193187756' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:28.566 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:28.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:28 smithi167 ceph-mon[33178]: 5.14 scrub starts 2024-09-13T13:17:28.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:28 smithi167 ceph-mon[33178]: 5.14 scrub ok 2024-09-13T13:17:28.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:28 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:17:28.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:28 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:17:28.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:28 smithi167 ceph-mon[33178]: from='client.36530 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:17:28.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:28 smithi167 ceph-mon[33178]: osdmap e411: 8 total, 8 up, 8 in 2024-09-13T13:17:28.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:28 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/193187756' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:28.591 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:28.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:28.692 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:28.720 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:28.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:28.821 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:28.848 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:28.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:28.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:28.950 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:28.979 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:28.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:29.081 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:29.114 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:29.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:29.216 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:29.250 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:29.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:29.352 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:29.373 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:29.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:29.475 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:29.498 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:29.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:29.503 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T13:17:29.526 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:29 smithi052 ceph-mon[36682]: pgmap v820: 148 pgs: 1 activating+remapped, 6 peering, 141 active+clean; 926 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 538 op/s; 926/45000 objects misplaced (2.058%); 27 B/s, 1 objects/s recovering 2024-09-13T13:17:29.526 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:29 smithi052 ceph-mon[36682]: osdmap e412: 8 total, 8 up, 8 in 2024-09-13T13:17:29.527 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:29 smithi052 ceph-mon[29471]: pgmap v820: 148 pgs: 1 activating+remapped, 6 peering, 141 active+clean; 926 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 538 op/s; 926/45000 objects misplaced (2.058%); 27 B/s, 1 objects/s recovering 2024-09-13T13:17:29.527 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:29 smithi052 ceph-mon[29471]: osdmap e412: 8 total, 8 up, 8 in 2024-09-13T13:17:29.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:29 smithi167 ceph-mon[33178]: pgmap v820: 148 pgs: 1 activating+remapped, 6 peering, 141 active+clean; 926 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 538 op/s; 926/45000 objects misplaced (2.058%); 27 B/s, 1 objects/s recovering 2024-09-13T13:17:29.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:29 smithi167 ceph-mon[33178]: osdmap e412: 8 total, 8 up, 8 in 2024-09-13T13:17:29.600 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:29.621 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:29.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:29.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:29.722 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:29.743 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:29.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:29.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:29.764 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:29.846 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:29.868 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:29.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:29.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:29.906 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:17:29.906 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:17:29.906 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-13T13:17:29.907 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T13:17:29.907 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T13:17:29.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-13T13:17:29.969 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:29.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:29.992 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:29.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:30.093 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:30.124 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:30.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:30.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:30.189 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:17:30.224 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:17:30.226 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:30.245 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:17:30.245 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:17:30.245 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:17:30.245 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:242835 10.8 10.3ed2e048 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:17:30.245 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:17:30.151285+0000", 2024-09-13T13:17:30.245 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.092545453999999999, 2024-09-13T13:17:30.245 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.092569525999999999, 2024-09-13T13:17:30.245 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:17:30.245 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:17:30.245 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for peered", 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 242835 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151285+0000", 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:17:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151285+0000", 2024-09-13T13:17:30.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:17:30.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:17:30.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:17:30.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:17:30.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151288+0000", 2024-09-13T13:17:30.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2560000000000001e-06 2024-09-13T13:17:30.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:17:30.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:17:30.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:17:30.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151294+0000", 2024-09-13T13:17:30.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1199999999999999e-06 2024-09-13T13:17:30.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:17:30.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:17:30.248 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:17:30.248 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151308+0000", 2024-09-13T13:17:30.248 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4329e-05 2024-09-13T13:17:30.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:17:30.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:17:30.248 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:17:30.248 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151316+0000", 2024-09-13T13:17:30.248 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5610000000000001e-06 2024-09-13T13:17:30.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:17:30.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:17:30.249 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:17:30.249 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151379+0000", 2024-09-13T13:17:30.249 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3212000000000004e-05 2024-09-13T13:17:30.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:17:30.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:17:30.249 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:17:30.249 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151388+0000", 2024-09-13T13:17:30.249 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2129999999999997e-06 2024-09-13T13:17:30.249 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:17:30.249 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:17:30.249 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:17:30.249 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:17:30.249 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:17:30.250 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 1 2024-09-13T13:17:30.250 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:17:30.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:30.251 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:30.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:30.352 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:30.374 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:30.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:30.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:30.476 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:30.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:30 smithi052 ceph-mon[36682]: 3.13 scrub starts 2024-09-13T13:17:30.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:30 smithi052 ceph-mon[36682]: 3.13 scrub ok 2024-09-13T13:17:30.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:30 smithi052 ceph-mon[36682]: osdmap e413: 8 total, 8 up, 8 in 2024-09-13T13:17:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:30 smithi052 ceph-mon[29471]: 3.13 scrub starts 2024-09-13T13:17:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:30 smithi052 ceph-mon[29471]: 3.13 scrub ok 2024-09-13T13:17:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:30 smithi052 ceph-mon[29471]: osdmap e413: 8 total, 8 up, 8 in 2024-09-13T13:17:30.499 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:30.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:30.569 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:30 smithi167 ceph-mon[33178]: 3.13 scrub starts 2024-09-13T13:17:30.569 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:30 smithi167 ceph-mon[33178]: 3.13 scrub ok 2024-09-13T13:17:30.569 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:30 smithi167 ceph-mon[33178]: osdmap e413: 8 total, 8 up, 8 in 2024-09-13T13:17:30.601 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:30.623 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:30.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:30.677 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:17:30.678 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T13:17:30.678 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:17:30.724 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:30.747 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:30.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:30.848 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:30.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:30.870 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:30.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:30.966 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:17:30.972 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:30.995 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:30.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:31.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T13:17:31.039 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:31.097 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:31.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:31.147 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:31.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:31.248 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:31.303 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:31.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:31.405 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:31.441 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:31.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:31.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:31.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[36682]: pgmap v822: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 activating+remapped, 1 active+remapped+backfill_toofull, 143 active+clean; 840 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 542 op/s; 2998/40872 objects misplaced (7.335%); 6.7 MiB/s, 114 objects/s recovering 2024-09-13T13:17:31.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[36682]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:17:31.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[36682]: 5.d scrub starts 2024-09-13T13:17:31.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[36682]: 5.d scrub ok 2024-09-13T13:17:31.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[36682]: 6.9 scrub starts 2024-09-13T13:17:31.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[36682]: 6.9 scrub ok 2024-09-13T13:17:31.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[36682]: osdmap e414: 8 total, 8 up, 8 in 2024-09-13T13:17:31.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[29471]: pgmap v822: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 activating+remapped, 1 active+remapped+backfill_toofull, 143 active+clean; 840 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 542 op/s; 2998/40872 objects misplaced (7.335%); 6.7 MiB/s, 114 objects/s recovering 2024-09-13T13:17:31.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[29471]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:17:31.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[29471]: 5.d scrub starts 2024-09-13T13:17:31.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[29471]: 5.d scrub ok 2024-09-13T13:17:31.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[29471]: 6.9 scrub starts 2024-09-13T13:17:31.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[29471]: 6.9 scrub ok 2024-09-13T13:17:31.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:31 smithi052 ceph-mon[29471]: osdmap e414: 8 total, 8 up, 8 in 2024-09-13T13:17:31.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:31.569 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:31.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:31.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:31.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:31 smithi167 ceph-mon[33178]: pgmap v822: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 activating+remapped, 1 active+remapped+backfill_toofull, 143 active+clean; 840 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 542 op/s; 2998/40872 objects misplaced (7.335%); 6.7 MiB/s, 114 objects/s recovering 2024-09-13T13:17:31.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:31 smithi167 ceph-mon[33178]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:17:31.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:31 smithi167 ceph-mon[33178]: 5.d scrub starts 2024-09-13T13:17:31.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:31 smithi167 ceph-mon[33178]: 5.d scrub ok 2024-09-13T13:17:31.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:31 smithi167 ceph-mon[33178]: 6.9 scrub starts 2024-09-13T13:17:31.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:31 smithi167 ceph-mon[33178]: 6.9 scrub ok 2024-09-13T13:17:31.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:31 smithi167 ceph-mon[33178]: osdmap e414: 8 total, 8 up, 8 in 2024-09-13T13:17:31.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:31.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:31.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:31.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:31.803 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:31.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:31.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:31.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:31.945 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:31.981 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:31.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:31.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:32.083 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:32.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:32.120 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:32.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:32.221 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:32.258 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:32.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:32.360 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:32.392 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:32.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:32.494 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:32.532 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:32.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:32.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:32 smithi167 ceph-mon[33178]: 5.1b scrub starts 2024-09-13T13:17:32.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:32 smithi167 ceph-mon[33178]: 5.1b scrub ok 2024-09-13T13:17:32.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:32.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:32.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:32.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:32.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:32 smithi052 ceph-mon[36682]: 5.1b scrub starts 2024-09-13T13:17:32.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:32 smithi052 ceph-mon[36682]: 5.1b scrub ok 2024-09-13T13:17:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:32 smithi052 ceph-mon[29471]: 5.1b scrub starts 2024-09-13T13:17:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:32 smithi052 ceph-mon[29471]: 5.1b scrub ok 2024-09-13T13:17:32.771 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:32.808 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:32.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:32.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:32.953 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:32.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:33.054 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:33.093 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:33.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:33.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:33.194 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:33.240 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:33.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:33.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:33.378 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:33.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:33.481 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:33.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:33 smithi052 ceph-mon[29471]: pgmap v825: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 144 active+clean; 779 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 499 op/s; 2072/37941 objects misplaced (5.461%); 9.8 MiB/s, 165 objects/s recovering 2024-09-13T13:17:33.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:33 smithi052 ceph-mon[29471]: 4.7 scrub starts 2024-09-13T13:17:33.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:33 smithi052 ceph-mon[29471]: 4.7 scrub ok 2024-09-13T13:17:33.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:33 smithi052 ceph-mon[29471]: 6.18 scrub starts 2024-09-13T13:17:33.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:33 smithi052 ceph-mon[29471]: 6.18 scrub ok 2024-09-13T13:17:33.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:33 smithi052 ceph-mon[36682]: pgmap v825: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 144 active+clean; 779 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 499 op/s; 2072/37941 objects misplaced (5.461%); 9.8 MiB/s, 165 objects/s recovering 2024-09-13T13:17:33.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:33 smithi052 ceph-mon[36682]: 4.7 scrub starts 2024-09-13T13:17:33.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:33 smithi052 ceph-mon[36682]: 4.7 scrub ok 2024-09-13T13:17:33.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:33 smithi052 ceph-mon[36682]: 6.18 scrub starts 2024-09-13T13:17:33.494 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:33 smithi052 ceph-mon[36682]: 6.18 scrub ok 2024-09-13T13:17:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:33.527 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:33.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:33.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:33 smithi167 ceph-mon[33178]: pgmap v825: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 144 active+clean; 779 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 499 op/s; 2072/37941 objects misplaced (5.461%); 9.8 MiB/s, 165 objects/s recovering 2024-09-13T13:17:33.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:33 smithi167 ceph-mon[33178]: 4.7 scrub starts 2024-09-13T13:17:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:33 smithi167 ceph-mon[33178]: 4.7 scrub ok 2024-09-13T13:17:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:33 smithi167 ceph-mon[33178]: 6.18 scrub starts 2024-09-13T13:17:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:33 smithi167 ceph-mon[33178]: 6.18 scrub ok 2024-09-13T13:17:33.628 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:33.664 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:33.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:33.766 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:33.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:33.799 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:33.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:33.900 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:33.925 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:33.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:33.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:34.026 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:34.056 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:34.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:34.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:34.195 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:34.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:34.297 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:34.338 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:34.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:34.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:34.374 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:17:34.440 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:34.478 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:34.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:34.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:34.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:17:34 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:17:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:34 smithi167 ceph-mon[33178]: 5.12 scrub starts 2024-09-13T13:17:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:34 smithi167 ceph-mon[33178]: 5.12 scrub ok 2024-09-13T13:17:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:34 smithi167 ceph-mon[33178]: 6.16 scrub starts 2024-09-13T13:17:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:34 smithi167 ceph-mon[33178]: 6.16 scrub ok 2024-09-13T13:17:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:34 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2547158821' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:17:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:34 smithi167 ceph-mon[33178]: from='client.36542 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:17:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:34 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:34 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:17:34.580 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:34.623 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:34.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:34.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:34.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[36682]: 5.12 scrub starts 2024-09-13T13:17:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[36682]: 5.12 scrub ok 2024-09-13T13:17:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[36682]: 6.16 scrub starts 2024-09-13T13:17:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[36682]: 6.16 scrub ok 2024-09-13T13:17:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2547158821' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:17:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[36682]: from='client.36542 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:17:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:17:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[29471]: 5.12 scrub starts 2024-09-13T13:17:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[29471]: 5.12 scrub ok 2024-09-13T13:17:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[29471]: 6.16 scrub starts 2024-09-13T13:17:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[29471]: 6.16 scrub ok 2024-09-13T13:17:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2547158821' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:17:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[29471]: from='client.36542 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:17:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:34 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:17:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:34.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:34.860 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:34.901 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:34.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:35.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:35.055 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:35.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:35.156 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:35.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:35.197 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:35.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:35.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:35.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:35.343 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:35.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:35.444 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:35.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:35.489 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:35.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:35.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:35 smithi167 ceph-mon[33178]: pgmap v826: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 146 active+clean; 669 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 685 op/s; 558/32676 objects misplaced (1.708%); 12 MiB/s, 202 objects/s recovering 2024-09-13T13:17:35.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:35 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:35.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:35 smithi167 ceph-mon[33178]: from='client.36542 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:17:35.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:35 smithi167 ceph-mon[33178]: osdmap e415: 8 total, 8 up, 8 in 2024-09-13T13:17:35.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:35.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:35 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:17:35.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:35 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:35.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:35 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:35.590 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:35.622 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:35.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:35.724 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:35.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[29471]: pgmap v826: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 146 active+clean; 669 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 685 op/s; 558/32676 objects misplaced (1.708%); 12 MiB/s, 202 objects/s recovering 2024-09-13T13:17:35.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:35.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[29471]: from='client.36542 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:17:35.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[29471]: osdmap e415: 8 total, 8 up, 8 in 2024-09-13T13:17:35.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:35.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:17:35.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:35.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:35.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[36682]: pgmap v826: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 146 active+clean; 669 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 685 op/s; 558/32676 objects misplaced (1.708%); 12 MiB/s, 202 objects/s recovering 2024-09-13T13:17:35.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:35.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[36682]: from='client.36542 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:17:35.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[36682]: osdmap e415: 8 total, 8 up, 8 in 2024-09-13T13:17:35.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:35.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:17:35.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:35.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:35 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:35.764 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:35.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:35.866 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:35.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:35.899 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:35.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:36.000 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:36.041 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:36.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:36.142 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:36.181 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:36.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:36.283 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:36.312 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:36.314 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:36.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:36.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:36.333 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:36.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:36.443 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:17:36.444 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:17:36.444 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T13:17:36.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:17:36.456 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:17:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:17:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:17:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:36.482 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:36.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:36.583 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:36.646 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:36.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:36.748 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:36.777 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:36.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:36.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:36.879 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:36.924 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:36.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:36.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:37.026 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:37.065 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:37.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:37.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:37.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:37.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:37.302 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:37.338 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:37.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:37.440 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:37.473 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:37.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:37.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:37.575 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:37.616 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:37.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:37.626 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:17:37.626 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:17:37.626 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:17:37.627 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:17:37.627 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:17:37.718 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:37.757 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:37.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:37.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:37 smithi167 ceph-mon[33178]: pgmap v828: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 618 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 591 op/s; 558/30228 objects misplaced (1.846%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T13:17:37.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:37 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:37 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:37.859 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:37.920 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:37.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:37.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:37.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:37 smithi052 ceph-mon[29471]: pgmap v828: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 618 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 591 op/s; 558/30228 objects misplaced (1.846%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T13:17:37.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:37 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:37.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:37 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:37.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:37 smithi052 ceph-mon[36682]: pgmap v828: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 618 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 591 op/s; 558/30228 objects misplaced (1.846%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T13:17:37.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:37 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:37.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:37 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:38.023 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:38.058 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:38.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:38.160 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:38.192 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:38.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:38.294 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:38.318 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:17:38.337 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:38.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:38.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:38.406 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:38.439 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:38.468 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T13:17:38.497 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:38.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:38.599 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:38.652 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:38.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:38.753 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:38.779 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:38.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:38.881 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:38.912 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:38.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:39.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:39.049 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:39.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:39.077 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:17:39.077 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:17:39.112 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:17:39.112 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:17:39.149 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:17:39.150 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:17:39.151 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:39.186 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:17:39.186 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:17:39.196 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:39.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:39.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:39.221 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:17:39.221 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:17:39.254 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:17:39.254 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:17:39.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:39 smithi052 ceph-mon[36682]: pgmap v829: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 434 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 819 op/s; 558/21408 objects misplaced (2.607%); 5.4 MiB/s, 85 objects/s recovering 2024-09-13T13:17:39.260 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:39 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:17:39.261 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:39 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:17:39.261 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:39 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:39.261 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:39 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3484334306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:39.262 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:39 smithi052 ceph-mon[29471]: pgmap v829: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 434 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 819 op/s; 558/21408 objects misplaced (2.607%); 5.4 MiB/s, 85 objects/s recovering 2024-09-13T13:17:39.262 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:39 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:17:39.262 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:39 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:17:39.262 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:39 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:39.262 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:39 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3484334306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:39.287 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:17:39.287 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:17:39.297 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:39.319 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:17:39.320 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:17:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:39 smithi167 ceph-mon[33178]: pgmap v829: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 434 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 819 op/s; 558/21408 objects misplaced (2.607%); 5.4 MiB/s, 85 objects/s recovering 2024-09-13T13:17:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:39 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:17:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:39 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:17:39.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:39 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:39.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:39 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3484334306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:39.345 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:39.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:39.447 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:39.483 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:39.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:39.585 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:39.623 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:39.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:39.725 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:39.756 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:39.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:39.764 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:39.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:39.865 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:39.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:39.905 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:39.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:39.947 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:17:39.953 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:17:40.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:40.043 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:40.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:40.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:40.145 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:40.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:40.181 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:40.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:40.282 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:40.321 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:40.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:40.423 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:40.465 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:40.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:40.567 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:40.607 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:40.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:40.708 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:40.744 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:40.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:40.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:40.846 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:40.888 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:40.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:40.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:40.990 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:41.027 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:41.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:41.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:41.129 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:41.167 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:41.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:41.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:41.257 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Removed 127377 objects 2024-09-13T13:17:41.257 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Clean up completed and total clean up time :126.925 2024-09-13T13:17:41.270 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:41.291 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:41.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:41.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:41 smithi167 ceph-mon[33178]: from='client.36554 v1:172.21.15.52:0/4008134159' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:17:41.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:41 smithi167 ceph-mon[33178]: pgmap v830: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 252 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s; 214/12678 objects misplaced (1.688%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T13:17:41.393 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:41.415 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:41.416 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:41 smithi052 ceph-mon[36682]: from='client.36554 v1:172.21.15.52:0/4008134159' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:17:41.416 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:41 smithi052 ceph-mon[36682]: pgmap v830: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 252 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s; 214/12678 objects misplaced (1.688%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T13:17:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:41.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:41.417 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:41 smithi052 ceph-mon[29471]: from='client.36554 v1:172.21.15.52:0/4008134159' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:17:41.417 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:41 smithi052 ceph-mon[29471]: pgmap v830: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 252 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s; 214/12678 objects misplaced (1.688%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T13:17:41.518 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:41.539 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:41.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:41.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:41.641 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:41.662 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:41.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:41.736 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":829,"stamp":"2024-09-13T13:17:37.972148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":455154227,"num_objects":7136,"num_object_clones":0,"num_object_copies":21408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":558,"num_objects_unfound":0,"num_objects_dirty":7136,"num_whiteouts":0,"num_read":1541,"num_read_kb":1400,"num_write":248654,"num_write_kb":8153802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21160,"num_bytes_recovered":1372002885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85645,"ondisk_log_size":85645,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":450,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6523444,"kb_used_data":1762536,"kb_used_omap":15,"kb_used_meta":4760624,"kb_avail":743437772,"statfs":{"total":767960285184,"available":761280278528,"internally_reserved":0,"allocated":1804836864,"data_stored":1801726889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":4874879769},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,10,12,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-425197568,"num_objects":-6488,"num_object_clones":0,"num_object_copies":-19464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2440,"num_objects_unfound":0,"num_objects_dirty":-6488,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6488,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":44630016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.927150"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":773,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.308410+0000","last_change":"2024-09-13T13:17:25.583614+0000","last_active":"2024-09-13T13:17:34.308410+0000","last_peered":"2024-09-13T13:17:34.308410+0000","last_clean":"2024-09-13T13:17:34.308410+0000","last_became_active":"2024-09-13T13:17:25.335900+0000","last_became_peered":"2024-09-13T13:17:25.335900+0000","last_unstale":"2024-09-13T13:17:34.308410+0000","last_undegraded":"2024-09-13T13:17:34.308410+0000","last_fullsized":"2024-09-13T13:17:34.308410+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:25.583569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:25.583569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:30.790877+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":247,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.049363+0000","last_change":"2024-09-13T13:15:04.344174+0000","last_active":"2024-09-13T13:17:31.049363+0000","last_peered":"2024-09-13T13:17:31.049363+0000","last_clean":"2024-09-13T13:17:31.049363+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:17:31.049363+0000","last_undegraded":"2024-09-13T13:17:31.049363+0000","last_fullsized":"2024-09-13T13:17:31.049363+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:15:04.344092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:04.344092+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":758,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.078094+0000","last_change":"2024-09-13T13:16:30.163153+0000","last_active":"2024-09-13T13:17:31.078094+0000","last_peered":"2024-09-13T13:17:31.078094+0000","last_clean":"2024-09-13T13:17:31.078094+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:17:31.078094+0000","last_undegraded":"2024-09-13T13:17:31.078094+0000","last_fullsized":"2024-09-13T13:17:31.078094+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:30.163077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:30.163077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:54.764628+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":245,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.049616+0000","last_change":"2024-09-13T13:15:51.244938+0000","last_active":"2024-09-13T13:17:31.049616+0000","last_peered":"2024-09-13T13:17:31.049616+0000","last_clean":"2024-09-13T13:17:31.049616+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:17:31.049616+0000","last_undegraded":"2024-09-13T13:17:31.049616+0000","last_fullsized":"2024-09-13T13:17:31.049616+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":758,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.301770+0000","last_change":"2024-09-13T13:11:47.852762+0000","last_active":"2024-09-13T13:17:34.301770+0000","last_peered":"2024-09-13T13:17:34.301770+0000","last_clean":"2024-09-13T13:17:34.301770+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:17:34.301770+0000","last_undegraded":"2024-09-13T13:17:34.301770+0000","last_fullsized":"2024-09-13T13:17:34.301770+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:19.106080+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":802,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.302642+0000","last_change":"2024-09-13T13:17:25.243843+0000","last_active":"2024-09-13T13:17:34.302642+0000","last_peered":"2024-09-13T13:17:34.302642+0000","last_clean":"2024-09-13T13:17:34.302642+0000","last_became_active":"2024-09-13T13:17:25.243371+0000","last_became_peered":"2024-09-13T13:17:25.243371+0000","last_unstale":"2024-09-13T13:17:34.302642+0000","last_undegraded":"2024-09-13T13:17:34.302642+0000","last_fullsized":"2024-09-13T13:17:34.302642+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:08.938750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:08.938750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:17.155030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":876,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.301630+0000","last_change":"2024-09-13T13:17:25.262965+0000","last_active":"2024-09-13T13:17:34.301630+0000","last_peered":"2024-09-13T13:17:34.301630+0000","last_clean":"2024-09-13T13:17:34.301630+0000","last_became_active":"2024-09-13T13:17:25.245131+0000","last_became_peered":"2024-09-13T13:17:25.245131+0000","last_unstale":"2024-09-13T13:17:34.301630+0000","last_undegraded":"2024-09-13T13:17:34.301630+0000","last_fullsized":"2024-09-13T13:17:34.301630+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:17:11.044472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:11.044472+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:29.751940+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":229,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.049248+0000","last_change":"2024-09-13T13:12:52.997865+0000","last_active":"2024-09-13T13:17:31.049248+0000","last_peered":"2024-09-13T13:17:31.049248+0000","last_clean":"2024-09-13T13:17:31.049248+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:17:31.049248+0000","last_undegraded":"2024-09-13T13:17:31.049248+0000","last_fullsized":"2024-09-13T13:17:31.049248+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:17.305608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:17.305608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:12.478035+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":820,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047867+0000","last_change":"2024-09-13T13:13:41.518378+0000","last_active":"2024-09-13T13:17:31.047867+0000","last_peered":"2024-09-13T13:17:31.047867+0000","last_clean":"2024-09-13T13:17:31.047867+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:17:31.047867+0000","last_undegraded":"2024-09-13T13:17:31.047867+0000","last_fullsized":"2024-09-13T13:17:31.047867+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:41.518322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.518322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":247,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050325+0000","last_change":"2024-09-13T13:17:06.406253+0000","last_active":"2024-09-13T13:17:31.050325+0000","last_peered":"2024-09-13T13:17:31.050325+0000","last_clean":"2024-09-13T13:17:31.050325+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:17:31.050325+0000","last_undegraded":"2024-09-13T13:17:31.050325+0000","last_fullsized":"2024-09-13T13:17:31.050325+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:06.406179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:06.406179+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:20.094337+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":765,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.089557+0000","last_change":"2024-09-13T13:17:31.089557+0000","last_active":"2024-09-13T13:17:31.089557+0000","last_peered":"2024-09-13T13:17:31.089557+0000","last_clean":"2024-09-13T13:17:31.089557+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:17:31.089557+0000","last_undegraded":"2024-09-13T13:17:31.089557+0000","last_fullsized":"2024-09-13T13:17:31.089557+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:31.089505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:31.089505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:50.762733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":770,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.314882+0000","last_change":"2024-09-13T13:17:32.559966+0000","last_active":"2024-09-13T13:17:34.314882+0000","last_peered":"2024-09-13T13:17:34.314882+0000","last_clean":"2024-09-13T13:17:34.314882+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:17:34.314882+0000","last_undegraded":"2024-09-13T13:17:34.314882+0000","last_fullsized":"2024-09-13T13:17:34.314882+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:32.559884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:32.559884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:34.281038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":795,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047651+0000","last_change":"2024-09-13T13:17:02.211078+0000","last_active":"2024-09-13T13:17:31.047651+0000","last_peered":"2024-09-13T13:17:31.047651+0000","last_clean":"2024-09-13T13:17:31.047651+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:17:31.047651+0000","last_undegraded":"2024-09-13T13:17:31.047651+0000","last_fullsized":"2024-09-13T13:17:31.047651+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:02.211007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.211007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:19.487607+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":243,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.049824+0000","last_change":"2024-09-13T13:15:03.348187+0000","last_active":"2024-09-13T13:17:31.049824+0000","last_peered":"2024-09-13T13:17:31.049824+0000","last_clean":"2024-09-13T13:17:31.049824+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:17:31.049824+0000","last_undegraded":"2024-09-13T13:17:31.049824+0000","last_fullsized":"2024-09-13T13:17:31.049824+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:15:03.348115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.348115+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":778,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.049625+0000","last_change":"2024-09-13T13:13:54.122282+0000","last_active":"2024-09-13T13:17:31.049625+0000","last_peered":"2024-09-13T13:17:31.049625+0000","last_clean":"2024-09-13T13:17:31.049625+0000","last_became_active":"2024-09-13T13:12:52.999970+0000","last_became_peered":"2024-09-13T13:12:52.999970+0000","last_unstale":"2024-09-13T13:17:31.049625+0000","last_undegraded":"2024-09-13T13:17:31.049625+0000","last_fullsized":"2024-09-13T13:17:31.049625+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.122222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.122222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:18.050234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":737,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047363+0000","last_change":"2024-09-13T13:17:25.133282+0000","last_active":"2024-09-13T13:17:31.047363+0000","last_peered":"2024-09-13T13:17:31.047363+0000","last_clean":"2024-09-13T13:17:31.047363+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:17:31.047363+0000","last_undegraded":"2024-09-13T13:17:31.047363+0000","last_fullsized":"2024-09-13T13:17:31.047363+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:25.133232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:25.133232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:25.990650+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":242,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.051129+0000","last_change":"2024-09-13T13:17:07.444148+0000","last_active":"2024-09-13T13:17:31.051129+0000","last_peered":"2024-09-13T13:17:31.051129+0000","last_clean":"2024-09-13T13:17:31.051129+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:17:31.051129+0000","last_undegraded":"2024-09-13T13:17:31.051129+0000","last_fullsized":"2024-09-13T13:17:31.051129+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:07.443951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:07.443951+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:25.224293+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":788,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.309083+0000","last_change":"2024-09-13T13:15:52.658477+0000","last_active":"2024-09-13T13:17:34.309083+0000","last_peered":"2024-09-13T13:17:34.309083+0000","last_clean":"2024-09-13T13:17:34.309083+0000","last_became_active":"2024-09-13T13:12:53.008505+0000","last_became_peered":"2024-09-13T13:12:53.008505+0000","last_unstale":"2024-09-13T13:17:34.309083+0000","last_undegraded":"2024-09-13T13:17:34.309083+0000","last_fullsized":"2024-09-13T13:17:34.309083+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:52.658410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:52.658410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":40,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.308331+0000","last_change":"2024-09-13T13:17:25.267260+0000","last_active":"2024-09-13T13:17:34.308331+0000","last_peered":"2024-09-13T13:17:34.308331+0000","last_clean":"2024-09-13T13:17:34.308331+0000","last_became_active":"2024-09-13T13:17:25.245658+0000","last_became_peered":"2024-09-13T13:17:25.245658+0000","last_unstale":"2024-09-13T13:17:34.308331+0000","last_undegraded":"2024-09-13T13:17:34.308331+0000","last_fullsized":"2024-09-13T13:17:34.308331+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:12:40.125545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:12:40.125545+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":788,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.077279+0000","last_change":"2024-09-13T13:16:54.229157+0000","last_active":"2024-09-13T13:17:31.077279+0000","last_peered":"2024-09-13T13:17:31.077279+0000","last_clean":"2024-09-13T13:17:31.077279+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:17:31.077279+0000","last_undegraded":"2024-09-13T13:17:31.077279+0000","last_fullsized":"2024-09-13T13:17:31.077279+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:54.229075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:54.229075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:13.235712+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":759,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.302282+0000","last_change":"2024-09-13T13:13:03.150524+0000","last_active":"2024-09-13T13:17:34.302282+0000","last_peered":"2024-09-13T13:17:34.302282+0000","last_clean":"2024-09-13T13:17:34.302282+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:17:34.302282+0000","last_undegraded":"2024-09-13T13:17:34.302282+0000","last_fullsized":"2024-09-13T13:17:34.302282+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:03.150442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.150442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:38.978974+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":259,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.049539+0000","last_change":"2024-09-13T13:17:25.245798+0000","last_active":"2024-09-13T13:17:31.049539+0000","last_peered":"2024-09-13T13:17:31.049539+0000","last_clean":"2024-09-13T13:17:31.049539+0000","last_became_active":"2024-09-13T13:17:25.244839+0000","last_became_peered":"2024-09-13T13:17:25.244839+0000","last_unstale":"2024-09-13T13:17:31.049539+0000","last_undegraded":"2024-09-13T13:17:31.049539+0000","last_fullsized":"2024-09-13T13:17:31.049539+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:01.242234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:01.242234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":825,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.304680+0000","last_change":"2024-09-13T13:13:41.164741+0000","last_active":"2024-09-13T13:17:34.304680+0000","last_peered":"2024-09-13T13:17:34.304680+0000","last_clean":"2024-09-13T13:17:34.304680+0000","last_became_active":"2024-09-13T13:12:52.953819+0000","last_became_peered":"2024-09-13T13:12:52.953819+0000","last_unstale":"2024-09-13T13:17:34.304680+0000","last_undegraded":"2024-09-13T13:17:34.304680+0000","last_fullsized":"2024-09-13T13:17:34.304680+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:13:41.164656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.164656+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:50.985657+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":253,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.049754+0000","last_change":"2024-09-13T13:17:25.245574+0000","last_active":"2024-09-13T13:17:31.049754+0000","last_peered":"2024-09-13T13:17:31.049754+0000","last_clean":"2024-09-13T13:17:31.049754+0000","last_became_active":"2024-09-13T13:17:25.245239+0000","last_became_peered":"2024-09-13T13:17:25.245239+0000","last_unstale":"2024-09-13T13:17:31.049754+0000","last_undegraded":"2024-09-13T13:17:31.049754+0000","last_fullsized":"2024-09-13T13:17:31.049754+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:29.208576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:29.208576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":294,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047426+0000","last_change":"2024-09-13T13:17:25.339695+0000","last_active":"2024-09-13T13:17:31.047426+0000","last_peered":"2024-09-13T13:17:31.047426+0000","last_clean":"2024-09-13T13:17:31.047426+0000","last_became_active":"2024-09-13T13:17:25.329235+0000","last_became_peered":"2024-09-13T13:17:25.329235+0000","last_unstale":"2024-09-13T13:17:31.047426+0000","last_undegraded":"2024-09-13T13:17:31.047426+0000","last_fullsized":"2024-09-13T13:17:31.047426+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:05.207445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.207445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:20.420990+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":789,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.078256+0000","last_change":"2024-09-13T13:17:01.162092+0000","last_active":"2024-09-13T13:17:31.078256+0000","last_peered":"2024-09-13T13:17:31.078256+0000","last_clean":"2024-09-13T13:17:31.078256+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:17:31.078256+0000","last_undegraded":"2024-09-13T13:17:31.078256+0000","last_fullsized":"2024-09-13T13:17:31.078256+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:17:01.162018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:01.162018+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:24.126509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":810,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.315760+0000","last_change":"2024-09-13T13:15:50.693943+0000","last_active":"2024-09-13T13:17:34.315760+0000","last_peered":"2024-09-13T13:17:34.315760+0000","last_clean":"2024-09-13T13:17:34.315760+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:17:34.315760+0000","last_undegraded":"2024-09-13T13:17:34.315760+0000","last_fullsized":"2024-09-13T13:17:34.315760+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:15:50.693753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.693753+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"306'12","reported_seq":947,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.306725+0000","last_change":"2024-09-13T13:14:21.530680+0000","last_active":"2024-09-13T13:17:34.306725+0000","last_peered":"2024-09-13T13:17:34.306725+0000","last_clean":"2024-09-13T13:17:34.306725+0000","last_became_active":"2024-09-13T13:12:52.947886+0000","last_became_peered":"2024-09-13T13:12:52.947886+0000","last_unstale":"2024-09-13T13:17:34.306725+0000","last_undegraded":"2024-09-13T13:17:34.306725+0000","last_fullsized":"2024-09-13T13:17:34.306725+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'12","last_scrub_stamp":"2024-09-13T13:14:21.530612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:21.530612+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:41.202009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"414'15395","reported_seq":15602,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:33.307141+0000","last_change":"2024-09-13T13:16:06.040203+0000","last_active":"2024-09-13T13:17:33.307141+0000","last_peered":"2024-09-13T13:17:33.307141+0000","last_clean":"2024-09-13T13:17:33.307141+0000","last_became_active":"2024-09-13T13:13:59.149215+0000","last_became_peered":"2024-09-13T13:13:59.149215+0000","last_unstale":"2024-09-13T13:17:33.307141+0000","last_undegraded":"2024-09-13T13:17:33.307141+0000","last_fullsized":"2024-09-13T13:17:33.307141+0000","mapping_epoch":334,"log_start":"380'10986","ondisk_log_start":"380'10986","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":3199,"log_size":4409,"log_dups_size":0,"ondisk_log_size":4409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37945344,"num_objects":579,"num_object_clones":0,"num_object_copies":1737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15395,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":860,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.078628+0000","last_change":"2024-09-13T13:16:56.133395+0000","last_active":"2024-09-13T13:17:31.078628+0000","last_peered":"2024-09-13T13:17:31.078628+0000","last_clean":"2024-09-13T13:17:31.078628+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:17:31.078628+0000","last_undegraded":"2024-09-13T13:17:31.078628+0000","last_fullsized":"2024-09-13T13:17:31.078628+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:16:56.133314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:56.133314+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:01.815453+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":792,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.301553+0000","last_change":"2024-09-13T13:16:58.980151+0000","last_active":"2024-09-13T13:17:34.301553+0000","last_peered":"2024-09-13T13:17:34.301553+0000","last_clean":"2024-09-13T13:17:34.301553+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:17:34.301553+0000","last_undegraded":"2024-09-13T13:17:34.301553+0000","last_fullsized":"2024-09-13T13:17:34.301553+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:58.980098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.980098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:19.930402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":299,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.051252+0000","last_change":"2024-09-13T13:15:32.212185+0000","last_active":"2024-09-13T13:17:31.051252+0000","last_peered":"2024-09-13T13:17:31.051252+0000","last_clean":"2024-09-13T13:17:31.051252+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:17:31.051252+0000","last_undegraded":"2024-09-13T13:17:31.051252+0000","last_fullsized":"2024-09-13T13:17:31.051252+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:15:32.212101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.212101+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":762,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047433+0000","last_change":"2024-09-13T13:13:57.546725+0000","last_active":"2024-09-13T13:17:31.047433+0000","last_peered":"2024-09-13T13:17:31.047433+0000","last_clean":"2024-09-13T13:17:31.047433+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:17:31.047433+0000","last_undegraded":"2024-09-13T13:17:31.047433+0000","last_fullsized":"2024-09-13T13:17:31.047433+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.546671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.546671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"414'15342","reported_seq":24430,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:33.303582+0000","last_change":"2024-09-13T13:17:31.075144+0000","last_active":"2024-09-13T13:17:33.303582+0000","last_peered":"2024-09-13T13:17:33.303582+0000","last_clean":"2024-09-13T13:17:33.303582+0000","last_became_active":"2024-09-13T13:17:31.053706+0000","last_became_peered":"2024-09-13T13:17:31.053706+0000","last_unstale":"2024-09-13T13:17:33.303582+0000","last_undegraded":"2024-09-13T13:17:33.303582+0000","last_fullsized":"2024-09-13T13:17:33.303582+0000","mapping_epoch":413,"log_start":"380'10910","ondisk_log_start":"380'10910","created":334,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":4432,"log_dups_size":0,"ondisk_log_size":4432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42205184,"num_objects":644,"num_object_clones":0,"num_object_copies":1932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15342,"num_write_kb":511552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4441,"num_bytes_recovered":291045376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":799,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.301325+0000","last_change":"2024-09-13T13:16:05.289274+0000","last_active":"2024-09-13T13:17:34.301325+0000","last_peered":"2024-09-13T13:17:34.301325+0000","last_clean":"2024-09-13T13:17:34.301325+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:17:34.301325+0000","last_undegraded":"2024-09-13T13:17:34.301325+0000","last_fullsized":"2024-09-13T13:17:34.301325+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:16:05.289187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:05.289187+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:35.033329+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":766,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.302619+0000","last_change":"2024-09-13T13:16:26.098415+0000","last_active":"2024-09-13T13:17:34.302619+0000","last_peered":"2024-09-13T13:17:34.302619+0000","last_clean":"2024-09-13T13:17:34.302619+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:17:34.302619+0000","last_undegraded":"2024-09-13T13:17:34.302619+0000","last_fullsized":"2024-09-13T13:17:34.302619+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:16:26.098365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:26.098365+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:54.098482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":238,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.049852+0000","last_change":"2024-09-13T13:15:00.323883+0000","last_active":"2024-09-13T13:17:31.049852+0000","last_peered":"2024-09-13T13:17:31.049852+0000","last_clean":"2024-09-13T13:17:31.049852+0000","last_became_active":"2024-09-13T13:12:53.006199+0000","last_became_peered":"2024-09-13T13:12:53.006199+0000","last_unstale":"2024-09-13T13:17:31.049852+0000","last_undegraded":"2024-09-13T13:17:31.049852+0000","last_fullsized":"2024-09-13T13:17:31.049852+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.323784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.323784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":260,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050412+0000","last_change":"2024-09-13T13:17:25.244632+0000","last_active":"2024-09-13T13:17:31.050412+0000","last_peered":"2024-09-13T13:17:31.050412+0000","last_clean":"2024-09-13T13:17:31.050412+0000","last_became_active":"2024-09-13T13:17:25.244171+0000","last_became_peered":"2024-09-13T13:17:25.244171+0000","last_unstale":"2024-09-13T13:17:31.050412+0000","last_undegraded":"2024-09-13T13:17:31.050412+0000","last_fullsized":"2024-09-13T13:17:31.050412+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:56.333778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:56.333778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"415'15560","reported_seq":21908,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:36.363052+0000","last_change":"2024-09-13T13:17:30.094681+0000","last_active":"2024-09-13T13:17:36.363052+0000","last_peered":"2024-09-13T13:17:36.363052+0000","last_clean":"2024-09-13T13:17:36.363052+0000","last_became_active":"2024-09-13T13:17:30.062909+0000","last_became_peered":"2024-09-13T13:17:30.062909+0000","last_unstale":"2024-09-13T13:17:36.363052+0000","last_undegraded":"2024-09-13T13:17:36.363052+0000","last_fullsized":"2024-09-13T13:17:36.363052+0000","mapping_epoch":412,"log_start":"372'9768","ondisk_log_start":"372'9768","created":334,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5792,"log_dups_size":0,"ondisk_log_size":5792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24510464,"num_objects":374,"num_object_clones":0,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15560,"num_write_kb":509888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3068,"num_bytes_recovered":201064448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":254,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.051068+0000","last_change":"2024-09-13T13:13:50.716312+0000","last_active":"2024-09-13T13:17:31.051068+0000","last_peered":"2024-09-13T13:17:31.051068+0000","last_clean":"2024-09-13T13:17:31.051068+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:17:31.051068+0000","last_undegraded":"2024-09-13T13:17:31.051068+0000","last_fullsized":"2024-09-13T13:17:31.051068+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:13:50.716243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.716243+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:19.545172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":774,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.316074+0000","last_change":"2024-09-13T13:17:02.704294+0000","last_active":"2024-09-13T13:17:34.316074+0000","last_peered":"2024-09-13T13:17:34.316074+0000","last_clean":"2024-09-13T13:17:34.316074+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:17:34.316074+0000","last_undegraded":"2024-09-13T13:17:34.316074+0000","last_fullsized":"2024-09-13T13:17:34.316074+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:17:02.704233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.704233+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:20.770762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":992,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.301187+0000","last_change":"2024-09-13T13:16:24.110415+0000","last_active":"2024-09-13T13:17:34.301187+0000","last_peered":"2024-09-13T13:17:34.301187+0000","last_clean":"2024-09-13T13:17:34.301187+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:17:34.301187+0000","last_undegraded":"2024-09-13T13:17:34.301187+0000","last_fullsized":"2024-09-13T13:17:34.301187+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:16:24.110369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:24.110369+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:42.263184+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":404,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.306420+0000","last_change":"2024-09-13T13:17:25.244548+0000","last_active":"2024-09-13T13:17:34.306420+0000","last_peered":"2024-09-13T13:17:34.306420+0000","last_clean":"2024-09-13T13:17:34.306420+0000","last_became_active":"2024-09-13T13:17:25.244188+0000","last_became_peered":"2024-09-13T13:17:25.244188+0000","last_unstale":"2024-09-13T13:17:34.306420+0000","last_undegraded":"2024-09-13T13:17:34.306420+0000","last_fullsized":"2024-09-13T13:17:34.306420+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:26.258619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:26.258619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:49.226535+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"414'15172","reported_seq":15360,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:33.278801+0000","last_change":"2024-09-13T13:17:24.530110+0000","last_active":"2024-09-13T13:17:33.278801+0000","last_peered":"2024-09-13T13:17:33.278801+0000","last_clean":"2024-09-13T13:17:33.278801+0000","last_became_active":"2024-09-13T13:13:59.149046+0000","last_became_peered":"2024-09-13T13:13:59.149046+0000","last_unstale":"2024-09-13T13:17:33.278801+0000","last_undegraded":"2024-09-13T13:17:33.278801+0000","last_fullsized":"2024-09-13T13:17:33.278801+0000","mapping_epoch":334,"log_start":"380'10742","ondisk_log_start":"380'10742","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'14865","last_scrub_stamp":"2024-09-13T13:17:24.529929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:17:24.529929+0000","objects_scrubbed":877,"log_size":4430,"log_dups_size":0,"ondisk_log_size":4430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:46.795002+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37355520,"num_objects":570,"num_object_clones":0,"num_object_copies":1710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15172,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":823,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.079785+0000","last_change":"2024-09-13T13:12:53.016127+0000","last_active":"2024-09-13T13:17:31.079785+0000","last_peered":"2024-09-13T13:17:31.079785+0000","last_clean":"2024-09-13T13:17:31.079785+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:17:31.079785+0000","last_undegraded":"2024-09-13T13:17:31.079785+0000","last_fullsized":"2024-09-13T13:17:31.079785+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:41.316459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":804,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047873+0000","last_change":"2024-09-13T13:17:25.331470+0000","last_active":"2024-09-13T13:17:31.047873+0000","last_peered":"2024-09-13T13:17:31.047873+0000","last_clean":"2024-09-13T13:17:31.047873+0000","last_became_active":"2024-09-13T13:17:25.331166+0000","last_became_peered":"2024-09-13T13:17:25.331166+0000","last_unstale":"2024-09-13T13:17:31.047873+0000","last_undegraded":"2024-09-13T13:17:31.047873+0000","last_fullsized":"2024-09-13T13:17:31.047873+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:04.755828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:04.755828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":972,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.308652+0000","last_change":"2024-09-13T13:15:51.663010+0000","last_active":"2024-09-13T13:17:34.308652+0000","last_peered":"2024-09-13T13:17:34.308652+0000","last_clean":"2024-09-13T13:17:34.308652+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:17:34.308652+0000","last_undegraded":"2024-09-13T13:17:34.308652+0000","last_fullsized":"2024-09-13T13:17:34.308652+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:15:51.662657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.662657+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":794,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.315193+0000","last_change":"2024-09-13T13:17:25.254962+0000","last_active":"2024-09-13T13:17:34.315193+0000","last_peered":"2024-09-13T13:17:34.315193+0000","last_clean":"2024-09-13T13:17:34.315193+0000","last_became_active":"2024-09-13T13:17:25.252712+0000","last_became_peered":"2024-09-13T13:17:25.252712+0000","last_unstale":"2024-09-13T13:17:34.315193+0000","last_undegraded":"2024-09-13T13:17:34.315193+0000","last_fullsized":"2024-09-13T13:17:34.315193+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:26.667455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:26.667455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"415'15730","reported_seq":16038,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:36.331315+0000","last_change":"2024-09-13T13:16:31.945928+0000","last_active":"2024-09-13T13:17:36.331315+0000","last_peered":"2024-09-13T13:17:36.331315+0000","last_clean":"2024-09-13T13:17:36.331315+0000","last_became_active":"2024-09-13T13:13:58.965950+0000","last_became_peered":"2024-09-13T13:13:58.965950+0000","last_unstale":"2024-09-13T13:17:36.331315+0000","last_undegraded":"2024-09-13T13:17:36.331315+0000","last_fullsized":"2024-09-13T13:17:36.331315+0000","mapping_epoch":334,"log_start":"372'9968","ondisk_log_start":"372'9968","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'11986","last_scrub_stamp":"2024-09-13T13:16:31.945684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:16:31.945684+0000","objects_scrubbed":4166,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:51.306557+0000","scrub_duration":1677,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24313880,"num_objects":372,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":372,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15730,"num_write_kb":515201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":830,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.308878+0000","last_change":"2024-09-13T13:17:05.726065+0000","last_active":"2024-09-13T13:17:34.308878+0000","last_peered":"2024-09-13T13:17:34.308878+0000","last_clean":"2024-09-13T13:17:34.308878+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:17:34.308878+0000","last_undegraded":"2024-09-13T13:17:34.308878+0000","last_fullsized":"2024-09-13T13:17:34.308878+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:17:05.726003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.726003+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:30.917098+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":773,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.076908+0000","last_change":"2024-09-13T13:17:25.333121+0000","last_active":"2024-09-13T13:17:31.076908+0000","last_peered":"2024-09-13T13:17:31.076908+0000","last_clean":"2024-09-13T13:17:31.076908+0000","last_became_active":"2024-09-13T13:17:25.332589+0000","last_became_peered":"2024-09-13T13:17:25.332589+0000","last_unstale":"2024-09-13T13:17:31.076908+0000","last_undegraded":"2024-09-13T13:17:31.076908+0000","last_fullsized":"2024-09-13T13:17:31.076908+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:36.283624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:36.283624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:39.801833+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":737,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.305902+0000","last_change":"2024-09-13T13:16:24.285416+0000","last_active":"2024-09-13T13:17:34.305902+0000","last_peered":"2024-09-13T13:17:34.305902+0000","last_clean":"2024-09-13T13:17:34.305902+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:17:34.305902+0000","last_undegraded":"2024-09-13T13:17:34.305902+0000","last_fullsized":"2024-09-13T13:17:34.305902+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:24.285346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:24.285346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:43.854597+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":780,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.077999+0000","last_change":"2024-09-13T13:17:25.244827+0000","last_active":"2024-09-13T13:17:31.077999+0000","last_peered":"2024-09-13T13:17:31.077999+0000","last_clean":"2024-09-13T13:17:31.077999+0000","last_became_active":"2024-09-13T13:17:25.244517+0000","last_became_peered":"2024-09-13T13:17:25.244517+0000","last_unstale":"2024-09-13T13:17:31.077999+0000","last_undegraded":"2024-09-13T13:17:31.077999+0000","last_fullsized":"2024-09-13T13:17:31.077999+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:29.167280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:29.167280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:58.017086+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"415'15595","reported_seq":15769,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:36.341082+0000","last_change":"2024-09-13T13:13:58.966631+0000","last_active":"2024-09-13T13:17:36.341082+0000","last_peered":"2024-09-13T13:17:36.341082+0000","last_clean":"2024-09-13T13:17:36.341082+0000","last_became_active":"2024-09-13T13:13:58.965789+0000","last_became_peered":"2024-09-13T13:13:58.965789+0000","last_unstale":"2024-09-13T13:17:36.341082+0000","last_undegraded":"2024-09-13T13:17:36.341082+0000","last_fullsized":"2024-09-13T13:17:36.341082+0000","mapping_epoch":334,"log_start":"372'9863","ondisk_log_start":"372'9863","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5732,"log_dups_size":0,"ondisk_log_size":5732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:34.394775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20250624,"num_objects":309,"num_object_clones":0,"num_object_copies":927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15595,"num_write_kb":508928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"411'42","reported_seq":1094,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.076809+0000","last_change":"2024-09-13T13:17:25.270001+0000","last_active":"2024-09-13T13:17:31.076809+0000","last_peered":"2024-09-13T13:17:31.076809+0000","last_clean":"2024-09-13T13:17:31.076809+0000","last_became_active":"2024-09-13T13:17:25.245371+0000","last_became_peered":"2024-09-13T13:17:25.245371+0000","last_unstale":"2024-09-13T13:17:31.076809+0000","last_undegraded":"2024-09-13T13:17:31.076809+0000","last_fullsized":"2024-09-13T13:17:31.076809+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'41","last_scrub_stamp":"2024-09-13T13:16:58.213805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.213805+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:00.328586+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":246,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050019+0000","last_change":"2024-09-13T13:14:14.899463+0000","last_active":"2024-09-13T13:17:31.050019+0000","last_peered":"2024-09-13T13:17:31.050019+0000","last_clean":"2024-09-13T13:17:31.050019+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:17:31.050019+0000","last_undegraded":"2024-09-13T13:17:31.050019+0000","last_fullsized":"2024-09-13T13:17:31.050019+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:14.899378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:14.899378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":777,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047669+0000","last_change":"2024-09-13T13:15:50.618476+0000","last_active":"2024-09-13T13:17:31.047669+0000","last_peered":"2024-09-13T13:17:31.047669+0000","last_clean":"2024-09-13T13:17:31.047669+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:17:31.047669+0000","last_undegraded":"2024-09-13T13:17:31.047669+0000","last_fullsized":"2024-09-13T13:17:31.047669+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:15:50.618426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:50.618426+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":781,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.309068+0000","last_change":"2024-09-13T13:14:59.670853+0000","last_active":"2024-09-13T13:17:34.309068+0000","last_peered":"2024-09-13T13:17:34.309068+0000","last_clean":"2024-09-13T13:17:34.309068+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:17:34.309068+0000","last_undegraded":"2024-09-13T13:17:34.309068+0000","last_fullsized":"2024-09-13T13:17:34.309068+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.670779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.670779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:17.617943+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":778,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.302398+0000","last_change":"2024-09-13T13:17:05.016876+0000","last_active":"2024-09-13T13:17:34.302398+0000","last_peered":"2024-09-13T13:17:34.302398+0000","last_clean":"2024-09-13T13:17:34.302398+0000","last_became_active":"2024-09-13T13:12:52.948742+0000","last_became_peered":"2024-09-13T13:12:52.948742+0000","last_unstale":"2024-09-13T13:17:34.302398+0000","last_undegraded":"2024-09-13T13:17:34.302398+0000","last_fullsized":"2024-09-13T13:17:34.302398+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:05.016806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.016806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:07.820409+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"414'15176","reported_seq":23450,"reported_epoch":414,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T13:17:33.541346+0000","last_change":"2024-09-13T13:17:26.427452+0000","last_active":"2024-09-13T13:17:33.541346+0000","last_peered":"2024-09-13T13:17:33.541346+0000","last_clean":"2024-09-13T13:17:24.223242+0000","last_became_active":"2024-09-13T13:17:26.387085+0000","last_became_peered":"2024-09-13T13:17:26.387085+0000","last_unstale":"2024-09-13T13:17:33.541346+0000","last_undegraded":"2024-09-13T13:17:33.541346+0000","last_fullsized":"2024-09-13T13:17:33.541346+0000","mapping_epoch":409,"log_start":"372'9794","ondisk_log_start":"372'9794","created":334,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36569088,"num_objects":558,"num_object_clones":0,"num_object_copies":1674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":558,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15176,"num_write_kb":503488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4040,"num_bytes_recovered":264765440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,7,4],"avail_no_missing":["5","4","7"],"object_location_counts":[{"shards":"4,5,7","objects":558}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":796,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:32.264550+0000","last_change":"2024-09-13T13:17:32.264550+0000","last_active":"2024-09-13T13:17:32.264550+0000","last_peered":"2024-09-13T13:17:32.264550+0000","last_clean":"2024-09-13T13:17:32.264550+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:17:32.264550+0000","last_undegraded":"2024-09-13T13:17:32.264550+0000","last_fullsized":"2024-09-13T13:17:32.264550+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'17","last_scrub_stamp":"2024-09-13T13:17:32.263873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:32.263873+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:01.566837+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":801,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.081525+0000","last_change":"2024-09-13T13:17:25.246782+0000","last_active":"2024-09-13T13:17:31.081525+0000","last_peered":"2024-09-13T13:17:31.081525+0000","last_clean":"2024-09-13T13:17:31.081525+0000","last_became_active":"2024-09-13T13:17:25.246400+0000","last_became_peered":"2024-09-13T13:17:25.246400+0000","last_unstale":"2024-09-13T13:17:31.081525+0000","last_undegraded":"2024-09-13T13:17:31.081525+0000","last_fullsized":"2024-09-13T13:17:31.081525+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:22.306820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:22.306820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:30.705035+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":28,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.307561+0000","last_change":"2024-09-13T13:17:25.258763+0000","last_active":"2024-09-13T13:17:34.307561+0000","last_peered":"2024-09-13T13:17:34.307561+0000","last_clean":"2024-09-13T13:17:34.307561+0000","last_became_active":"2024-09-13T13:17:25.252664+0000","last_became_peered":"2024-09-13T13:17:25.252664+0000","last_unstale":"2024-09-13T13:17:34.307561+0000","last_undegraded":"2024-09-13T13:17:34.307561+0000","last_fullsized":"2024-09-13T13:17:34.307561+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:13:14.560555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:14.560555+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":24,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.307954+0000","last_change":"2024-09-13T13:17:25.333004+0000","last_active":"2024-09-13T13:17:34.307954+0000","last_peered":"2024-09-13T13:17:34.307954+0000","last_clean":"2024-09-13T13:17:34.307954+0000","last_became_active":"2024-09-13T13:17:25.332669+0000","last_became_peered":"2024-09-13T13:17:25.332669+0000","last_unstale":"2024-09-13T13:17:34.307954+0000","last_undegraded":"2024-09-13T13:17:34.307954+0000","last_fullsized":"2024-09-13T13:17:34.307954+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:59.809055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:59.809055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":382,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.306293+0000","last_change":"2024-09-13T13:12:41.332308+0000","last_active":"2024-09-13T13:17:34.306293+0000","last_peered":"2024-09-13T13:17:34.306293+0000","last_clean":"2024-09-13T13:17:34.306293+0000","last_became_active":"2024-09-13T13:12:40.120242+0000","last_became_peered":"2024-09-13T13:12:40.120242+0000","last_unstale":"2024-09-13T13:17:34.306293+0000","last_undegraded":"2024-09-13T13:17:34.306293+0000","last_fullsized":"2024-09-13T13:17:34.306293+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:41.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:41.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:16.331681+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"415'15633","reported_seq":22326,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.652625+0000","last_change":"2024-09-13T13:17:31.085611+0000","last_active":"2024-09-13T13:17:34.652625+0000","last_peered":"2024-09-13T13:17:34.652625+0000","last_clean":"2024-09-13T13:17:34.652625+0000","last_became_active":"2024-09-13T13:17:31.053870+0000","last_became_peered":"2024-09-13T13:17:31.053870+0000","last_unstale":"2024-09-13T13:17:34.652625+0000","last_undegraded":"2024-09-13T13:17:34.652625+0000","last_fullsized":"2024-09-13T13:17:34.652625+0000","mapping_epoch":413,"log_start":"375'10596","ondisk_log_start":"375'10596","created":334,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14102","last_scrub_stamp":"2024-09-13T13:17:08.294178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:17:08.294178+0000","objects_scrubbed":2017,"log_size":5037,"log_dups_size":0,"ondisk_log_size":5037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:26.880456+0000","scrub_duration":632,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":461,"num_object_clones":0,"num_object_copies":1383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15633,"num_write_kb":515008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3225,"num_bytes_recovered":211288064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":841,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.315851+0000","last_change":"2024-09-13T13:17:25.272220+0000","last_active":"2024-09-13T13:17:34.315851+0000","last_peered":"2024-09-13T13:17:34.315851+0000","last_clean":"2024-09-13T13:17:34.315851+0000","last_became_active":"2024-09-13T13:17:25.253120+0000","last_became_peered":"2024-09-13T13:17:25.253120+0000","last_unstale":"2024-09-13T13:17:34.315851+0000","last_undegraded":"2024-09-13T13:17:34.315851+0000","last_fullsized":"2024-09-13T13:17:34.315851+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:16:30.819129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:30.819129+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:59.327195+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":781,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.308151+0000","last_change":"2024-09-13T13:15:30.590673+0000","last_active":"2024-09-13T13:17:34.308151+0000","last_peered":"2024-09-13T13:17:34.308151+0000","last_clean":"2024-09-13T13:17:34.308151+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:17:34.308151+0000","last_undegraded":"2024-09-13T13:17:34.308151+0000","last_fullsized":"2024-09-13T13:17:34.308151+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:30.590466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:15:30.590466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":604,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:32.572506+0000","last_change":"2024-09-13T13:17:25.325347+0000","last_active":"2024-09-13T13:17:32.572506+0000","last_peered":"2024-09-13T13:17:32.572506+0000","last_clean":"2024-09-13T13:17:32.572506+0000","last_became_active":"2024-09-13T13:12:39.947940+0000","last_became_peered":"2024-09-13T13:12:39.947940+0000","last_unstale":"2024-09-13T13:17:32.572506+0000","last_undegraded":"2024-09-13T13:17:32.572506+0000","last_fullsized":"2024-09-13T13:17:32.572506+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:17:25.325275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:25.325275+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:51.942457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":300,"num_read_kb":295,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":750,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.048304+0000","last_change":"2024-09-13T13:15:32.318497+0000","last_active":"2024-09-13T13:17:31.048304+0000","last_peered":"2024-09-13T13:17:31.048304+0000","last_clean":"2024-09-13T13:17:31.048304+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:17:31.048304+0000","last_undegraded":"2024-09-13T13:17:31.048304+0000","last_fullsized":"2024-09-13T13:17:31.048304+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:32.318413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.318413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":755,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.076345+0000","last_change":"2024-09-13T13:10:32.720742+0000","last_active":"2024-09-13T13:17:31.076345+0000","last_peered":"2024-09-13T13:17:31.076345+0000","last_clean":"2024-09-13T13:17:31.076345+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:17:31.076345+0000","last_undegraded":"2024-09-13T13:17:31.076345+0000","last_fullsized":"2024-09-13T13:17:31.076345+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:32.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:32.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:39.284284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"415'15570","reported_seq":15761,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:36.341222+0000","last_change":"2024-09-13T13:17:24.545205+0000","last_active":"2024-09-13T13:17:36.341222+0000","last_peered":"2024-09-13T13:17:36.341222+0000","last_clean":"2024-09-13T13:17:36.341222+0000","last_became_active":"2024-09-13T13:13:58.965962+0000","last_became_peered":"2024-09-13T13:13:58.965962+0000","last_unstale":"2024-09-13T13:17:36.341222+0000","last_undegraded":"2024-09-13T13:17:36.341222+0000","last_fullsized":"2024-09-13T13:17:36.341222+0000","mapping_epoch":334,"log_start":"372'9815","ondisk_log_start":"372'9815","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'15022","last_scrub_stamp":"2024-09-13T13:17:24.545144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:17:24.545144+0000","objects_scrubbed":910,"log_size":5755,"log_dups_size":0,"ondisk_log_size":5755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:50.743272+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23724032,"num_objects":362,"num_object_clones":0,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15570,"num_write_kb":509824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"306'32","reported_seq":905,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.306082+0000","last_change":"2024-09-13T13:13:01.352416+0000","last_active":"2024-09-13T13:17:34.306082+0000","last_peered":"2024-09-13T13:17:34.306082+0000","last_clean":"2024-09-13T13:17:34.306082+0000","last_became_active":"2024-09-13T13:12:52.999924+0000","last_became_peered":"2024-09-13T13:12:52.999924+0000","last_unstale":"2024-09-13T13:17:34.306082+0000","last_undegraded":"2024-09-13T13:17:34.306082+0000","last_fullsized":"2024-09-13T13:17:34.306082+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:13:01.352337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.352337+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:11.387458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":779,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.302249+0000","last_change":"2024-09-13T13:16:53.951208+0000","last_active":"2024-09-13T13:17:34.302249+0000","last_peered":"2024-09-13T13:17:34.302249+0000","last_clean":"2024-09-13T13:17:34.302249+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:17:34.302249+0000","last_undegraded":"2024-09-13T13:17:34.302249+0000","last_fullsized":"2024-09-13T13:17:34.302249+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:53.951135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:53.951135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:55.102860+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":346,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.315391+0000","last_change":"2024-09-13T13:15:54.707061+0000","last_active":"2024-09-13T13:17:34.315391+0000","last_peered":"2024-09-13T13:17:34.315391+0000","last_clean":"2024-09-13T13:17:34.315391+0000","last_became_active":"2024-09-13T13:12:39.948765+0000","last_became_peered":"2024-09-13T13:12:39.948765+0000","last_unstale":"2024-09-13T13:17:34.315391+0000","last_undegraded":"2024-09-13T13:17:34.315391+0000","last_fullsized":"2024-09-13T13:17:34.315391+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":24,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.308233+0000","last_change":"2024-09-13T13:17:25.328398+0000","last_active":"2024-09-13T13:17:34.308233+0000","last_peered":"2024-09-13T13:17:34.308233+0000","last_clean":"2024-09-13T13:17:34.308233+0000","last_became_active":"2024-09-13T13:17:25.328065+0000","last_became_peered":"2024-09-13T13:17:25.328065+0000","last_unstale":"2024-09-13T13:17:34.308233+0000","last_undegraded":"2024-09-13T13:17:34.308233+0000","last_fullsized":"2024-09-13T13:17:34.308233+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:33.708995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.708995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":792,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.048235+0000","last_change":"2024-09-13T13:15:57.689044+0000","last_active":"2024-09-13T13:17:31.048235+0000","last_peered":"2024-09-13T13:17:31.048235+0000","last_clean":"2024-09-13T13:17:31.048235+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:17:31.048235+0000","last_undegraded":"2024-09-13T13:17:31.048235+0000","last_fullsized":"2024-09-13T13:17:31.048235+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:57.688955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:57.688955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"415'15810","reported_seq":16103,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:36.346292+0000","last_change":"2024-09-13T13:16:33.808007+0000","last_active":"2024-09-13T13:17:36.346292+0000","last_peered":"2024-09-13T13:17:36.346292+0000","last_clean":"2024-09-13T13:17:36.346292+0000","last_became_active":"2024-09-13T13:13:58.965566+0000","last_became_peered":"2024-09-13T13:13:58.965566+0000","last_unstale":"2024-09-13T13:17:36.346292+0000","last_undegraded":"2024-09-13T13:17:36.346292+0000","last_fullsized":"2024-09-13T13:17:36.346292+0000","mapping_epoch":334,"log_start":"372'10021","ondisk_log_start":"372'10021","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'12107","last_scrub_stamp":"2024-09-13T13:16:33.807809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:16:33.807809+0000","objects_scrubbed":4136,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:55.361775+0000","scrub_duration":1617,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24772608,"num_objects":378,"num_object_clones":0,"num_object_copies":1134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15810,"num_write_kb":518016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":258,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050889+0000","last_change":"2024-09-13T13:14:59.354034+0000","last_active":"2024-09-13T13:17:31.050889+0000","last_peered":"2024-09-13T13:17:31.050889+0000","last_clean":"2024-09-13T13:17:31.050889+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:17:31.050889+0000","last_undegraded":"2024-09-13T13:17:31.050889+0000","last_fullsized":"2024-09-13T13:17:31.050889+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:14:59.353691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.353691+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":252,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.049714+0000","last_change":"2024-09-13T13:17:25.245204+0000","last_active":"2024-09-13T13:17:31.049714+0000","last_peered":"2024-09-13T13:17:31.049714+0000","last_clean":"2024-09-13T13:17:31.049714+0000","last_became_active":"2024-09-13T13:17:25.244810+0000","last_became_peered":"2024-09-13T13:17:25.244810+0000","last_unstale":"2024-09-13T13:17:31.049714+0000","last_undegraded":"2024-09-13T13:17:31.049714+0000","last_fullsized":"2024-09-13T13:17:31.049714+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:26.343754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:26.343754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":807,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.048207+0000","last_change":"2024-09-13T13:15:55.592241+0000","last_active":"2024-09-13T13:17:31.048207+0000","last_peered":"2024-09-13T13:17:31.048207+0000","last_clean":"2024-09-13T13:17:31.048207+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:17:31.048207+0000","last_undegraded":"2024-09-13T13:17:31.048207+0000","last_fullsized":"2024-09-13T13:17:31.048207+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:55.592172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:55.592172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":759,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.301730+0000","last_change":"2024-09-13T13:14:23.655250+0000","last_active":"2024-09-13T13:17:34.301730+0000","last_peered":"2024-09-13T13:17:34.301730+0000","last_clean":"2024-09-13T13:17:34.301730+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:17:34.301730+0000","last_undegraded":"2024-09-13T13:17:34.301730+0000","last_fullsized":"2024-09-13T13:17:34.301730+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:23.655072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:23.655072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:48.966053+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"415'15694","reported_seq":21993,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:37.887901+0000","last_change":"2024-09-13T13:17:31.096108+0000","last_active":"2024-09-13T13:17:37.887901+0000","last_peered":"2024-09-13T13:17:37.887901+0000","last_clean":"2024-09-13T13:17:37.887901+0000","last_became_active":"2024-09-13T13:17:31.054238+0000","last_became_peered":"2024-09-13T13:17:31.054238+0000","last_unstale":"2024-09-13T13:17:37.887901+0000","last_undegraded":"2024-09-13T13:17:37.887901+0000","last_fullsized":"2024-09-13T13:17:37.887901+0000","mapping_epoch":413,"log_start":"372'9984","ondisk_log_start":"372'9984","created":334,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7793","last_scrub_stamp":"2024-09-13T13:15:31.408001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:31.408001+0000","objects_scrubbed":7714,"log_size":5710,"log_dups_size":0,"ondisk_log_size":5710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15204352,"num_objects":232,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15694,"num_write_kb":509632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3003,"num_bytes_recovered":196739072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":314,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050293+0000","last_change":"2024-09-13T13:17:25.265759+0000","last_active":"2024-09-13T13:17:31.050293+0000","last_peered":"2024-09-13T13:17:31.050293+0000","last_clean":"2024-09-13T13:17:31.050293+0000","last_became_active":"2024-09-13T13:17:25.244686+0000","last_became_peered":"2024-09-13T13:17:25.244686+0000","last_unstale":"2024-09-13T13:17:31.050293+0000","last_undegraded":"2024-09-13T13:17:31.050293+0000","last_fullsized":"2024-09-13T13:17:31.050293+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:14:20.221286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:20.221286+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":791,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.306423+0000","last_change":"2024-09-13T13:16:37.263033+0000","last_active":"2024-09-13T13:17:34.306423+0000","last_peered":"2024-09-13T13:17:34.306423+0000","last_clean":"2024-09-13T13:17:34.306423+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:17:34.306423+0000","last_undegraded":"2024-09-13T13:17:34.306423+0000","last_fullsized":"2024-09-13T13:17:34.306423+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:37.262977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:37.262977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:56.631589+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":781,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.076527+0000","last_change":"2024-09-13T13:17:05.238854+0000","last_active":"2024-09-13T13:17:31.076527+0000","last_peered":"2024-09-13T13:17:31.076527+0000","last_clean":"2024-09-13T13:17:31.076527+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:17:31.076527+0000","last_undegraded":"2024-09-13T13:17:31.076527+0000","last_fullsized":"2024-09-13T13:17:31.076527+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:05.238722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.238722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:12.615074+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":24,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.312345+0000","last_change":"2024-09-13T13:17:25.243030+0000","last_active":"2024-09-13T13:17:34.312345+0000","last_peered":"2024-09-13T13:17:34.312345+0000","last_clean":"2024-09-13T13:17:34.312345+0000","last_became_active":"2024-09-13T13:17:25.242366+0000","last_became_peered":"2024-09-13T13:17:25.242366+0000","last_unstale":"2024-09-13T13:17:34.312345+0000","last_undegraded":"2024-09-13T13:17:34.312345+0000","last_fullsized":"2024-09-13T13:17:34.312345+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:35.064188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:35.064188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:56.790301+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"414'15547","reported_seq":15771,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.093191+0000","last_change":"2024-09-13T13:16:29.564422+0000","last_active":"2024-09-13T13:17:34.093191+0000","last_peered":"2024-09-13T13:17:34.093191+0000","last_clean":"2024-09-13T13:17:34.093191+0000","last_became_active":"2024-09-13T13:13:58.963064+0000","last_became_peered":"2024-09-13T13:13:58.963064+0000","last_unstale":"2024-09-13T13:17:34.093191+0000","last_undegraded":"2024-09-13T13:17:34.093191+0000","last_fullsized":"2024-09-13T13:17:34.093191+0000","mapping_epoch":334,"log_start":"375'10514","ondisk_log_start":"375'10514","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'11857","last_scrub_stamp":"2024-09-13T13:16:29.563999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:16:29.563999+0000","objects_scrubbed":4241,"log_size":5033,"log_dups_size":0,"ondisk_log_size":5033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:58.196946+0000","scrub_duration":1422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":491,"num_object_clones":0,"num_object_copies":1473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15547,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":250,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050798+0000","last_change":"2024-09-13T13:17:02.421924+0000","last_active":"2024-09-13T13:17:31.050798+0000","last_peered":"2024-09-13T13:17:31.050798+0000","last_clean":"2024-09-13T13:17:31.050798+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:17:31.050798+0000","last_undegraded":"2024-09-13T13:17:31.050798+0000","last_fullsized":"2024-09-13T13:17:31.050798+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.421825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.421825+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:23.577428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":766,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.303123+0000","last_change":"2024-09-13T13:12:53.003530+0000","last_active":"2024-09-13T13:17:34.303123+0000","last_peered":"2024-09-13T13:17:34.303123+0000","last_clean":"2024-09-13T13:17:34.303123+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:17:34.303123+0000","last_undegraded":"2024-09-13T13:17:34.303123+0000","last_fullsized":"2024-09-13T13:17:34.303123+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.996346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.996346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:13.096545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"411'10","reported_seq":39,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.313302+0000","last_change":"2024-09-13T13:17:30.560167+0000","last_active":"2024-09-13T13:17:34.313302+0000","last_peered":"2024-09-13T13:17:34.313302+0000","last_clean":"2024-09-13T13:17:34.313302+0000","last_became_active":"2024-09-13T13:17:25.333196+0000","last_became_peered":"2024-09-13T13:17:25.333196+0000","last_unstale":"2024-09-13T13:17:34.313302+0000","last_undegraded":"2024-09-13T13:17:34.313302+0000","last_fullsized":"2024-09-13T13:17:34.313302+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'10","last_scrub_stamp":"2024-09-13T13:17:30.560099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:30.560099+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:31.255268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":252,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050782+0000","last_change":"2024-09-13T13:17:25.252990+0000","last_active":"2024-09-13T13:17:31.050782+0000","last_peered":"2024-09-13T13:17:31.050782+0000","last_clean":"2024-09-13T13:17:31.050782+0000","last_became_active":"2024-09-13T13:17:25.252583+0000","last_became_peered":"2024-09-13T13:17:25.252583+0000","last_unstale":"2024-09-13T13:17:31.050782+0000","last_undegraded":"2024-09-13T13:17:31.050782+0000","last_fullsized":"2024-09-13T13:17:31.050782+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:41.797639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.797639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"415'15635","reported_seq":22236,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:37.849145+0000","last_change":"2024-09-13T13:17:30.075160+0000","last_active":"2024-09-13T13:17:37.849145+0000","last_peered":"2024-09-13T13:17:37.849145+0000","last_clean":"2024-09-13T13:17:37.849145+0000","last_became_active":"2024-09-13T13:17:30.052501+0000","last_became_peered":"2024-09-13T13:17:30.052501+0000","last_unstale":"2024-09-13T13:17:37.849145+0000","last_undegraded":"2024-09-13T13:17:37.849145+0000","last_fullsized":"2024-09-13T13:17:37.849145+0000","mapping_epoch":412,"log_start":"375'10371","ondisk_log_start":"375'10371","created":334,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'7822","last_scrub_stamp":"2024-09-13T13:15:32.530220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.530220+0000","objects_scrubbed":7744,"log_size":5264,"log_dups_size":0,"ondisk_log_size":5264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14745600,"num_objects":225,"num_object_clones":0,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15635,"num_write_kb":507520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3151,"num_bytes_recovered":206503936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":798,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.308748+0000","last_change":"2024-09-13T13:13:56.850149+0000","last_active":"2024-09-13T13:17:34.308748+0000","last_peered":"2024-09-13T13:17:34.308748+0000","last_clean":"2024-09-13T13:17:34.308748+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:17:34.308748+0000","last_undegraded":"2024-09-13T13:17:34.308748+0000","last_fullsized":"2024-09-13T13:17:34.308748+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:56.850094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.850094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:11.629068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":785,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047911+0000","last_change":"2024-09-13T13:12:58.857634+0000","last_active":"2024-09-13T13:17:31.047911+0000","last_peered":"2024-09-13T13:17:31.047911+0000","last_clean":"2024-09-13T13:17:31.047911+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:17:31.047911+0000","last_undegraded":"2024-09-13T13:17:31.047911+0000","last_fullsized":"2024-09-13T13:17:31.047911+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:58.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:22.160768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":24,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.307952+0000","last_change":"2024-09-13T13:17:25.238946+0000","last_active":"2024-09-13T13:17:34.307952+0000","last_peered":"2024-09-13T13:17:34.307952+0000","last_clean":"2024-09-13T13:17:34.307952+0000","last_became_active":"2024-09-13T13:17:25.238627+0000","last_became_peered":"2024-09-13T13:17:25.238627+0000","last_unstale":"2024-09-13T13:17:34.307952+0000","last_undegraded":"2024-09-13T13:17:34.307952+0000","last_fullsized":"2024-09-13T13:17:34.307952+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:23.682463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.682463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":769,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.308048+0000","last_change":"2024-09-13T13:17:30.598783+0000","last_active":"2024-09-13T13:17:34.308048+0000","last_peered":"2024-09-13T13:17:34.308048+0000","last_clean":"2024-09-13T13:17:34.308048+0000","last_became_active":"2024-09-13T13:17:25.252822+0000","last_became_peered":"2024-09-13T13:17:25.252822+0000","last_unstale":"2024-09-13T13:17:34.308048+0000","last_undegraded":"2024-09-13T13:17:34.308048+0000","last_fullsized":"2024-09-13T13:17:34.308048+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:30.598736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:30.598736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:32.689780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"415'15238","reported_seq":15458,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.642369+0000","last_change":"2024-09-13T13:16:34.719611+0000","last_active":"2024-09-13T13:17:34.642369+0000","last_peered":"2024-09-13T13:17:34.642369+0000","last_clean":"2024-09-13T13:17:34.642369+0000","last_became_active":"2024-09-13T13:13:58.963160+0000","last_became_peered":"2024-09-13T13:13:58.963160+0000","last_unstale":"2024-09-13T13:17:34.642369+0000","last_undegraded":"2024-09-13T13:17:34.642369+0000","last_fullsized":"2024-09-13T13:17:34.642369+0000","mapping_epoch":334,"log_start":"375'10188","ondisk_log_start":"375'10188","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'11789","last_scrub_stamp":"2024-09-13T13:16:34.719499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:16:34.719499+0000","objects_scrubbed":3932,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:43.651072+0000","scrub_duration":908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":460,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15238,"num_write_kb":502336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":772,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.306004+0000","last_change":"2024-09-13T13:13:06.339762+0000","last_active":"2024-09-13T13:17:34.306004+0000","last_peered":"2024-09-13T13:17:34.306004+0000","last_clean":"2024-09-13T13:17:34.306004+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:17:34.306004+0000","last_undegraded":"2024-09-13T13:17:34.306004+0000","last_fullsized":"2024-09-13T13:17:34.306004+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:13:06.339664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.339664+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:19.414813+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":400,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047748+0000","last_change":"2024-09-13T13:14:55.471954+0000","last_active":"2024-09-13T13:17:31.047748+0000","last_peered":"2024-09-13T13:17:31.047748+0000","last_clean":"2024-09-13T13:17:31.047748+0000","last_became_active":"2024-09-13T13:12:39.954315+0000","last_became_peered":"2024-09-13T13:12:39.954315+0000","last_unstale":"2024-09-13T13:17:31.047748+0000","last_undegraded":"2024-09-13T13:17:31.047748+0000","last_fullsized":"2024-09-13T13:17:31.047748+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:55.471885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:55.471885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":24,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.308733+0000","last_change":"2024-09-13T13:17:25.333336+0000","last_active":"2024-09-13T13:17:34.308733+0000","last_peered":"2024-09-13T13:17:34.308733+0000","last_clean":"2024-09-13T13:17:34.308733+0000","last_became_active":"2024-09-13T13:17:25.332964+0000","last_became_peered":"2024-09-13T13:17:25.332964+0000","last_unstale":"2024-09-13T13:17:34.308733+0000","last_undegraded":"2024-09-13T13:17:34.308733+0000","last_fullsized":"2024-09-13T13:17:34.308733+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.484615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.484615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":24,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.308274+0000","last_change":"2024-09-13T13:17:25.243438+0000","last_active":"2024-09-13T13:17:34.308274+0000","last_peered":"2024-09-13T13:17:34.308274+0000","last_clean":"2024-09-13T13:17:34.308274+0000","last_became_active":"2024-09-13T13:17:25.243114+0000","last_became_peered":"2024-09-13T13:17:25.243114+0000","last_unstale":"2024-09-13T13:17:34.308274+0000","last_undegraded":"2024-09-13T13:17:34.308274+0000","last_fullsized":"2024-09-13T13:17:34.308274+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:02.627569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:02.627569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"415'15248","reported_seq":15422,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:36.340801+0000","last_change":"2024-09-13T13:13:58.966955+0000","last_active":"2024-09-13T13:17:36.340801+0000","last_peered":"2024-09-13T13:17:36.340801+0000","last_clean":"2024-09-13T13:17:36.340801+0000","last_became_active":"2024-09-13T13:13:58.966675+0000","last_became_peered":"2024-09-13T13:13:58.966675+0000","last_unstale":"2024-09-13T13:17:36.340801+0000","last_undegraded":"2024-09-13T13:17:36.340801+0000","last_fullsized":"2024-09-13T13:17:36.340801+0000","mapping_epoch":334,"log_start":"372'9441","ondisk_log_start":"372'9441","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.948994+0000","objects_scrubbed":0,"log_size":5807,"log_dups_size":0,"ondisk_log_size":5807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22413312,"num_objects":342,"num_object_clones":0,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15248,"num_write_kb":498880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":258,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.051157+0000","last_change":"2024-09-13T13:16:00.285064+0000","last_active":"2024-09-13T13:17:31.051157+0000","last_peered":"2024-09-13T13:17:31.051157+0000","last_clean":"2024-09-13T13:17:31.051157+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:17:31.051157+0000","last_undegraded":"2024-09-13T13:17:31.051157+0000","last_fullsized":"2024-09-13T13:17:31.051157+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:16:00.284984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:00.284984+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":246,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050695+0000","last_change":"2024-09-13T13:14:52.383027+0000","last_active":"2024-09-13T13:17:31.050695+0000","last_peered":"2024-09-13T13:17:31.050695+0000","last_clean":"2024-09-13T13:17:31.050695+0000","last_became_active":"2024-09-13T13:12:53.007561+0000","last_became_peered":"2024-09-13T13:12:53.007561+0000","last_unstale":"2024-09-13T13:17:31.050695+0000","last_undegraded":"2024-09-13T13:17:31.050695+0000","last_fullsized":"2024-09-13T13:17:31.050695+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.382755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.382755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":24,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.308190+0000","last_change":"2024-09-13T13:17:25.252865+0000","last_active":"2024-09-13T13:17:34.308190+0000","last_peered":"2024-09-13T13:17:34.308190+0000","last_clean":"2024-09-13T13:17:34.308190+0000","last_became_active":"2024-09-13T13:17:25.252535+0000","last_became_peered":"2024-09-13T13:17:25.252535+0000","last_unstale":"2024-09-13T13:17:34.308190+0000","last_undegraded":"2024-09-13T13:17:34.308190+0000","last_fullsized":"2024-09-13T13:17:34.308190+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:00.636898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:00.636898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":748,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.302754+0000","last_change":"2024-09-13T13:16:58.047398+0000","last_active":"2024-09-13T13:17:34.302754+0000","last_peered":"2024-09-13T13:17:34.302754+0000","last_clean":"2024-09-13T13:17:34.302754+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:17:34.302754+0000","last_undegraded":"2024-09-13T13:17:34.302754+0000","last_fullsized":"2024-09-13T13:17:34.302754+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:58.044139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.044139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:25.917931+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"414'15474","reported_seq":15719,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:33.537106+0000","last_change":"2024-09-13T13:15:56.411481+0000","last_active":"2024-09-13T13:17:33.537106+0000","last_peered":"2024-09-13T13:17:33.537106+0000","last_clean":"2024-09-13T13:17:33.537106+0000","last_became_active":"2024-09-13T13:13:58.962773+0000","last_became_peered":"2024-09-13T13:13:58.962773+0000","last_unstale":"2024-09-13T13:17:33.537106+0000","last_undegraded":"2024-09-13T13:17:33.537106+0000","last_fullsized":"2024-09-13T13:17:33.537106+0000","mapping_epoch":334,"log_start":"373'10076","ondisk_log_start":"373'10076","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'9588","last_scrub_stamp":"2024-09-13T13:15:56.411320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:13:57.948994+0000","last_clean_scrub_stamp":"2024-09-13T13:15:56.411320+0000","objects_scrubbed":6514,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38010880,"num_objects":580,"num_object_clones":0,"num_object_copies":1740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15474,"num_write_kb":513728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":832,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047249+0000","last_change":"2024-09-13T13:15:56.631456+0000","last_active":"2024-09-13T13:17:31.047249+0000","last_peered":"2024-09-13T13:17:31.047249+0000","last_clean":"2024-09-13T13:17:31.047249+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:17:31.047249+0000","last_undegraded":"2024-09-13T13:17:31.047249+0000","last_fullsized":"2024-09-13T13:17:31.047249+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:15:56.631396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:56.631396+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":244,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050217+0000","last_change":"2024-09-13T13:13:01.972907+0000","last_active":"2024-09-13T13:17:31.050217+0000","last_peered":"2024-09-13T13:17:31.050217+0000","last_clean":"2024-09-13T13:17:31.050217+0000","last_became_active":"2024-09-13T13:12:53.006362+0000","last_became_peered":"2024-09-13T13:12:53.006362+0000","last_unstale":"2024-09-13T13:17:31.050217+0000","last_undegraded":"2024-09-13T13:17:31.050217+0000","last_fullsized":"2024-09-13T13:17:31.050217+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:13:01.972831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.972831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:15.537697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"306'13","reported_seq":507,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.305881+0000","last_change":"2024-09-13T13:13:03.334319+0000","last_active":"2024-09-13T13:17:34.305881+0000","last_peered":"2024-09-13T13:17:34.305881+0000","last_clean":"2024-09-13T13:17:34.305881+0000","last_became_active":"2024-09-13T13:12:53.001189+0000","last_became_peered":"2024-09-13T13:12:53.001189+0000","last_unstale":"2024-09-13T13:17:34.305881+0000","last_undegraded":"2024-09-13T13:17:34.305881+0000","last_fullsized":"2024-09-13T13:17:34.305881+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'13","last_scrub_stamp":"2024-09-13T13:13:03.334012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:03.334012+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:39.235043+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":757,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047367+0000","last_change":"2024-09-13T13:13:55.500961+0000","last_active":"2024-09-13T13:17:31.047367+0000","last_peered":"2024-09-13T13:17:31.047367+0000","last_clean":"2024-09-13T13:17:31.047367+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:17:31.047367+0000","last_undegraded":"2024-09-13T13:17:31.047367+0000","last_fullsized":"2024-09-13T13:17:31.047367+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:55.500908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:55.500908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":736,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.305813+0000","last_change":"2024-09-13T13:13:04.371082+0000","last_active":"2024-09-13T13:17:34.305813+0000","last_peered":"2024-09-13T13:17:34.305813+0000","last_clean":"2024-09-13T13:17:34.305813+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:17:34.305813+0000","last_undegraded":"2024-09-13T13:17:34.305813+0000","last_fullsized":"2024-09-13T13:17:34.305813+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.370881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.370881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:13.392940+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":280,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.048092+0000","last_change":"2024-09-13T13:15:23.319486+0000","last_active":"2024-09-13T13:17:31.048092+0000","last_peered":"2024-09-13T13:17:31.048092+0000","last_clean":"2024-09-13T13:17:31.048092+0000","last_became_active":"2024-09-13T13:12:34.673528+0000","last_became_peered":"2024-09-13T13:12:34.673528+0000","last_unstale":"2024-09-13T13:17:31.048092+0000","last_undegraded":"2024-09-13T13:17:31.048092+0000","last_fullsized":"2024-09-13T13:17:31.048092+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:15:23.319412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:23.319412+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":781,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.048064+0000","last_change":"2024-09-13T13:15:51.633487+0000","last_active":"2024-09-13T13:17:31.048064+0000","last_peered":"2024-09-13T13:17:31.048064+0000","last_clean":"2024-09-13T13:17:31.048064+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:17:31.048064+0000","last_undegraded":"2024-09-13T13:17:31.048064+0000","last_fullsized":"2024-09-13T13:17:31.048064+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.633433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.633433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":802,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050458+0000","last_change":"2024-09-13T13:16:33.024715+0000","last_active":"2024-09-13T13:17:31.050458+0000","last_peered":"2024-09-13T13:17:31.050458+0000","last_clean":"2024-09-13T13:17:31.050458+0000","last_became_active":"2024-09-13T13:12:53.000873+0000","last_became_peered":"2024-09-13T13:12:53.000873+0000","last_unstale":"2024-09-13T13:17:31.050458+0000","last_undegraded":"2024-09-13T13:17:31.050458+0000","last_fullsized":"2024-09-13T13:17:31.050458+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:33.024527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:33.024527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:47.416529+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":779,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.315911+0000","last_change":"2024-09-13T13:16:37.748574+0000","last_active":"2024-09-13T13:17:34.315911+0000","last_peered":"2024-09-13T13:17:34.315911+0000","last_clean":"2024-09-13T13:17:34.315911+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:17:34.315911+0000","last_undegraded":"2024-09-13T13:17:34.315911+0000","last_fullsized":"2024-09-13T13:17:34.315911+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:16:37.748500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:37.748500+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:52.503365+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":238,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.051019+0000","last_change":"2024-09-13T13:13:51.764793+0000","last_active":"2024-09-13T13:17:31.051019+0000","last_peered":"2024-09-13T13:17:31.051019+0000","last_clean":"2024-09-13T13:17:31.051019+0000","last_became_active":"2024-09-13T13:12:53.006579+0000","last_became_peered":"2024-09-13T13:12:53.006579+0000","last_unstale":"2024-09-13T13:17:31.051019+0000","last_undegraded":"2024-09-13T13:17:31.051019+0000","last_fullsized":"2024-09-13T13:17:31.051019+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.764717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.764717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:20.540563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":767,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.302428+0000","last_change":"2024-09-13T13:17:00.968640+0000","last_active":"2024-09-13T13:17:34.302428+0000","last_peered":"2024-09-13T13:17:34.302428+0000","last_clean":"2024-09-13T13:17:34.302428+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:17:34.302428+0000","last_undegraded":"2024-09-13T13:17:34.302428+0000","last_fullsized":"2024-09-13T13:17:34.302428+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:00.968580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:00.968580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:22.774319+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":24,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.308429+0000","last_change":"2024-09-13T13:17:25.246094+0000","last_active":"2024-09-13T13:17:34.308429+0000","last_peered":"2024-09-13T13:17:34.308429+0000","last_clean":"2024-09-13T13:17:34.308429+0000","last_became_active":"2024-09-13T13:17:25.244421+0000","last_became_peered":"2024-09-13T13:17:25.244421+0000","last_unstale":"2024-09-13T13:17:34.308429+0000","last_undegraded":"2024-09-13T13:17:34.308429+0000","last_fullsized":"2024-09-13T13:17:34.308429+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:15.616112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:15.616112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":807,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047763+0000","last_change":"2024-09-13T13:17:04.222896+0000","last_active":"2024-09-13T13:17:31.047763+0000","last_peered":"2024-09-13T13:17:31.047763+0000","last_clean":"2024-09-13T13:17:31.047763+0000","last_became_active":"2024-09-13T13:16:12.069730+0000","last_became_peered":"2024-09-13T13:16:12.069730+0000","last_unstale":"2024-09-13T13:17:31.047763+0000","last_undegraded":"2024-09-13T13:17:31.047763+0000","last_fullsized":"2024-09-13T13:17:31.047763+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:04.222760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:04.222760+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:29.524612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":759,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.309501+0000","last_change":"2024-09-13T13:17:09.634101+0000","last_active":"2024-09-13T13:17:34.309501+0000","last_peered":"2024-09-13T13:17:34.309501+0000","last_clean":"2024-09-13T13:17:34.309501+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:17:34.309501+0000","last_undegraded":"2024-09-13T13:17:34.309501+0000","last_fullsized":"2024-09-13T13:17:34.309501+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.634027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.634027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:29.941144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"411'10","reported_seq":982,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:37.440367+0000","last_change":"2024-09-13T13:17:27.165835+0000","last_active":"2024-09-13T13:17:37.440367+0000","last_peered":"2024-09-13T13:17:37.440367+0000","last_clean":"2024-09-13T13:17:37.440367+0000","last_became_active":"2024-09-13T13:17:25.245468+0000","last_became_peered":"2024-09-13T13:17:25.245468+0000","last_unstale":"2024-09-13T13:17:37.440367+0000","last_undegraded":"2024-09-13T13:17:37.440367+0000","last_fullsized":"2024-09-13T13:17:37.440367+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'9","last_scrub_stamp":"2024-09-13T13:17:27.165657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:27.165657+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:41.073300+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":251,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050410+0000","last_change":"2024-09-13T13:17:25.333913+0000","last_active":"2024-09-13T13:17:31.050410+0000","last_peered":"2024-09-13T13:17:31.050410+0000","last_clean":"2024-09-13T13:17:31.050410+0000","last_became_active":"2024-09-13T13:17:25.333587+0000","last_became_peered":"2024-09-13T13:17:25.333587+0000","last_unstale":"2024-09-13T13:17:31.050410+0000","last_undegraded":"2024-09-13T13:17:31.050410+0000","last_fullsized":"2024-09-13T13:17:31.050410+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:11.956784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:11.956784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":810,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.302112+0000","last_change":"2024-09-13T13:17:02.978088+0000","last_active":"2024-09-13T13:17:34.302112+0000","last_peered":"2024-09-13T13:17:34.302112+0000","last_clean":"2024-09-13T13:17:34.302112+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:17:34.302112+0000","last_undegraded":"2024-09-13T13:17:34.302112+0000","last_fullsized":"2024-09-13T13:17:34.302112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.978003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.978003+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:17.628791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":786,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050249+0000","last_change":"2024-09-13T13:17:28.963214+0000","last_active":"2024-09-13T13:17:31.050249+0000","last_peered":"2024-09-13T13:17:31.050249+0000","last_clean":"2024-09-13T13:17:31.050249+0000","last_became_active":"2024-09-13T13:17:25.252500+0000","last_became_peered":"2024-09-13T13:17:25.252500+0000","last_unstale":"2024-09-13T13:17:31.050249+0000","last_undegraded":"2024-09-13T13:17:31.050249+0000","last_fullsized":"2024-09-13T13:17:31.050249+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:28.963152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:28.963152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:50.378740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":972,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047705+0000","last_change":"2024-09-13T13:15:33.333062+0000","last_active":"2024-09-13T13:17:31.047705+0000","last_peered":"2024-09-13T13:17:31.047705+0000","last_clean":"2024-09-13T13:17:31.047705+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:17:31.047705+0000","last_undegraded":"2024-09-13T13:17:31.047705+0000","last_fullsized":"2024-09-13T13:17:31.047705+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:15:33.332981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:33.332981+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":770,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.315531+0000","last_change":"2024-09-13T13:17:33.554954+0000","last_active":"2024-09-13T13:17:34.315531+0000","last_peered":"2024-09-13T13:17:34.315531+0000","last_clean":"2024-09-13T13:17:34.315531+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:17:34.315531+0000","last_undegraded":"2024-09-13T13:17:34.315531+0000","last_fullsized":"2024-09-13T13:17:34.315531+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:33.554870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:33.554870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:03.321466+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":250,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050633+0000","last_change":"2024-09-13T13:13:47.728092+0000","last_active":"2024-09-13T13:17:31.050633+0000","last_peered":"2024-09-13T13:17:31.050633+0000","last_clean":"2024-09-13T13:17:31.050633+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:17:31.050633+0000","last_undegraded":"2024-09-13T13:17:31.050633+0000","last_fullsized":"2024-09-13T13:17:31.050633+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:13:47.728010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:47.728010+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":238,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.051010+0000","last_change":"2024-09-13T13:17:09.431539+0000","last_active":"2024-09-13T13:17:31.051010+0000","last_peered":"2024-09-13T13:17:31.051010+0000","last_clean":"2024-09-13T13:17:31.051010+0000","last_became_active":"2024-09-13T13:12:53.005988+0000","last_became_peered":"2024-09-13T13:12:53.005988+0000","last_unstale":"2024-09-13T13:17:31.051010+0000","last_undegraded":"2024-09-13T13:17:31.051010+0000","last_fullsized":"2024-09-13T13:17:31.051010+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.431472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.431472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:30.094940+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":775,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:33.279005+0000","last_change":"2024-09-13T13:17:33.279005+0000","last_active":"2024-09-13T13:17:33.279005+0000","last_peered":"2024-09-13T13:17:33.279005+0000","last_clean":"2024-09-13T13:17:33.279005+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:17:33.279005+0000","last_undegraded":"2024-09-13T13:17:33.279005+0000","last_fullsized":"2024-09-13T13:17:33.279005+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:33.278909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:33.278909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:55.488227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":772,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.301305+0000","last_change":"2024-09-13T13:17:09.989898+0000","last_active":"2024-09-13T13:17:34.301305+0000","last_peered":"2024-09-13T13:17:34.301305+0000","last_clean":"2024-09-13T13:17:34.301305+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:17:34.301305+0000","last_undegraded":"2024-09-13T13:17:34.301305+0000","last_fullsized":"2024-09-13T13:17:34.301305+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.989820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.989820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:38.033009+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":764,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.047384+0000","last_change":"2024-09-13T13:17:26.125163+0000","last_active":"2024-09-13T13:17:31.047384+0000","last_peered":"2024-09-13T13:17:31.047384+0000","last_clean":"2024-09-13T13:17:31.047384+0000","last_became_active":"2024-09-13T13:17:25.331530+0000","last_became_peered":"2024-09-13T13:17:25.331530+0000","last_unstale":"2024-09-13T13:17:31.047384+0000","last_undegraded":"2024-09-13T13:17:31.047384+0000","last_fullsized":"2024-09-13T13:17:31.047384+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:26.125088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:26.125088+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:50.498858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":752,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.304809+0000","last_change":"2024-09-13T13:14:13.418389+0000","last_active":"2024-09-13T13:17:34.304809+0000","last_peered":"2024-09-13T13:17:34.304809+0000","last_clean":"2024-09-13T13:17:34.304809+0000","last_became_active":"2024-09-13T13:12:52.949094+0000","last_became_peered":"2024-09-13T13:12:52.949094+0000","last_unstale":"2024-09-13T13:17:34.304809+0000","last_undegraded":"2024-09-13T13:17:34.304809+0000","last_fullsized":"2024-09-13T13:17:34.304809+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:13.418196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:13.418196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:51.937954+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":760,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.302945+0000","last_change":"2024-09-13T13:15:01.269060+0000","last_active":"2024-09-13T13:17:34.302945+0000","last_peered":"2024-09-13T13:17:34.302945+0000","last_clean":"2024-09-13T13:17:34.302945+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:17:34.302945+0000","last_undegraded":"2024-09-13T13:17:34.302945+0000","last_fullsized":"2024-09-13T13:17:34.302945+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:01.268884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:01.268884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:44.996809+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":747,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.314711+0000","last_change":"2024-09-13T13:10:39.477882+0000","last_active":"2024-09-13T13:17:34.314711+0000","last_peered":"2024-09-13T13:17:34.314711+0000","last_clean":"2024-09-13T13:17:34.314711+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:17:34.314711+0000","last_undegraded":"2024-09-13T13:17:34.314711+0000","last_fullsized":"2024-09-13T13:17:34.314711+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:10:39.477683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:10:39.477683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:40.746995+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":790,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.301445+0000","last_change":"2024-09-13T13:16:28.056674+0000","last_active":"2024-09-13T13:17:34.301445+0000","last_peered":"2024-09-13T13:17:34.301445+0000","last_clean":"2024-09-13T13:17:34.301445+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:17:34.301445+0000","last_undegraded":"2024-09-13T13:17:34.301445+0000","last_fullsized":"2024-09-13T13:17:34.301445+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:16:28.056586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:28.056586+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:57.805987+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":773,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.048100+0000","last_change":"2024-09-13T13:14:50.392659+0000","last_active":"2024-09-13T13:17:31.048100+0000","last_peered":"2024-09-13T13:17:31.048100+0000","last_clean":"2024-09-13T13:17:31.048100+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:17:31.048100+0000","last_undegraded":"2024-09-13T13:17:31.048100+0000","last_fullsized":"2024-09-13T13:17:31.048100+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.392444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.392444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":783,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.050594+0000","last_change":"2024-09-13T13:15:03.778214+0000","last_active":"2024-09-13T13:17:31.050594+0000","last_peered":"2024-09-13T13:17:31.050594+0000","last_clean":"2024-09-13T13:17:31.050594+0000","last_became_active":"2024-09-13T13:12:52.999748+0000","last_became_peered":"2024-09-13T13:12:52.999748+0000","last_unstale":"2024-09-13T13:17:31.050594+0000","last_undegraded":"2024-09-13T13:17:31.050594+0000","last_fullsized":"2024-09-13T13:17:31.050594+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.778170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.778170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:21.058865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":743,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.303080+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:17:34.303080+0000","last_peered":"2024-09-13T13:17:34.303080+0000","last_clean":"2024-09-13T13:17:34.303080+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:17:34.303080+0000","last_undegraded":"2024-09-13T13:17:34.303080+0000","last_fullsized":"2024-09-13T13:17:34.303080+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:18.075084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":805,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.301231+0000","last_change":"2024-09-13T13:16:12.090990+0000","last_active":"2024-09-13T13:17:34.301231+0000","last_peered":"2024-09-13T13:17:34.301231+0000","last_clean":"2024-09-13T13:17:34.301231+0000","last_became_active":"2024-09-13T13:16:12.073702+0000","last_became_peered":"2024-09-13T13:16:12.073702+0000","last_unstale":"2024-09-13T13:17:34.301231+0000","last_undegraded":"2024-09-13T13:17:34.301231+0000","last_fullsized":"2024-09-13T13:17:34.301231+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:14:12.368430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:14:12.368430+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:39.946451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":782,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.308498+0000","last_change":"2024-09-13T13:16:32.807018+0000","last_active":"2024-09-13T13:17:34.308498+0000","last_peered":"2024-09-13T13:17:34.308498+0000","last_clean":"2024-09-13T13:17:34.308498+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:17:34.308498+0000","last_undegraded":"2024-09-13T13:17:34.308498+0000","last_fullsized":"2024-09-13T13:17:34.308498+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:32.806957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:32.806957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:56.930095+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":690,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.305980+0000","last_change":"2024-09-13T13:14:16.457160+0000","last_active":"2024-09-13T13:17:34.305980+0000","last_peered":"2024-09-13T13:17:34.305980+0000","last_clean":"2024-09-13T13:17:34.305980+0000","last_became_active":"2024-09-13T13:12:53.001131+0000","last_became_peered":"2024-09-13T13:12:53.001131+0000","last_unstale":"2024-09-13T13:17:34.305980+0000","last_undegraded":"2024-09-13T13:17:34.305980+0000","last_fullsized":"2024-09-13T13:17:34.305980+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:16.456784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:16.456784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":24,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.307845+0000","last_change":"2024-09-13T13:17:25.242638+0000","last_active":"2024-09-13T13:17:34.307845+0000","last_peered":"2024-09-13T13:17:34.307845+0000","last_clean":"2024-09-13T13:17:34.307845+0000","last_became_active":"2024-09-13T13:17:25.242337+0000","last_became_peered":"2024-09-13T13:17:25.242337+0000","last_unstale":"2024-09-13T13:17:34.307845+0000","last_undegraded":"2024-09-13T13:17:34.307845+0000","last_fullsized":"2024-09-13T13:17:34.307845+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:05.828300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:05.828300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":768,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-09-13T13:17:31.077026+0000","last_change":"2024-09-13T13:14:18.882969+0000","last_active":"2024-09-13T13:17:31.077026+0000","last_peered":"2024-09-13T13:17:31.077026+0000","last_clean":"2024-09-13T13:17:31.077026+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:17:31.077026+0000","last_undegraded":"2024-09-13T13:17:31.077026+0000","last_fullsized":"2024-09-13T13:17:31.077026+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:18.882903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:18.882903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:47.220508+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":759,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.301470+0000","last_change":"2024-09-13T13:16:36.980668+0000","last_active":"2024-09-13T13:17:34.301470+0000","last_peered":"2024-09-13T13:17:34.301470+0000","last_clean":"2024-09-13T13:17:34.301470+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:17:34.301470+0000","last_undegraded":"2024-09-13T13:17:34.301470+0000","last_fullsized":"2024-09-13T13:17:34.301470+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:36.980594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:36.980594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:17:50.040358+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":778,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.306229+0000","last_change":"2024-09-13T13:16:12.087075+0000","last_active":"2024-09-13T13:17:34.306229+0000","last_peered":"2024-09-13T13:17:34.306229+0000","last_clean":"2024-09-13T13:17:34.306229+0000","last_became_active":"2024-09-13T13:16:12.070277+0000","last_became_peered":"2024-09-13T13:16:12.070277+0000","last_unstale":"2024-09-13T13:17:34.306229+0000","last_undegraded":"2024-09-13T13:17:34.306229+0000","last_fullsized":"2024-09-13T13:17:34.306229+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:49.264128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.264128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:14.366634+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":798,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-13T13:17:34.315677+0000","last_change":"2024-09-13T13:17:25.253423+0000","last_active":"2024-09-13T13:17:34.315677+0000","last_peered":"2024-09-13T13:17:34.315677+0000","last_clean":"2024-09-13T13:17:34.315677+0000","last_became_active":"2024-09-13T13:17:25.252637+0000","last_became_peered":"2024-09-13T13:17:25.252637+0000","last_unstale":"2024-09-13T13:17:34.315677+0000","last_undegraded":"2024-09-13T13:17:34.315677+0000","last_fullsized":"2024-09-13T13:17:34.315677+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:23.547758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:23.547758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:29.345548+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":454557720,"num_objects":6937,"num_object_clones":0,"num_object_copies":20811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":558,"num_objects_unfound":0,"num_objects_dirty":6937,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":247819,"num_write_kb":8152129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20928,"num_bytes_recovered":1371406336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654534144,"data_stored":1654521928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84782,"ondisk_log_size":84782,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":4214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":685,"ondisk_log_size":685,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":300,"num_read_kb":295,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201481,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":747540,"kb_used_data":192960,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92997612,"statfs":{"total":95995035648,"available":95229554688,"internally_reserved":0,"allocated":197591040,"data_stored":197177775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":567867850},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141092,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":679308,"kb_used_data":194812,"kb_used_omap":4,"kb_used_meta":484475,"kb_avail":93065844,"statfs":{"total":95995035648,"available":95299424256,"internally_reserved":0,"allocated":199487488,"data_stored":199138310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":496103085},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692053,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":803716,"kb_used_data":186676,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92941436,"statfs":{"total":95995035648,"available":95172030464,"internally_reserved":0,"allocated":191156224,"data_stored":190753543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920987,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1054428,"kb_used_data":463756,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92690724,"statfs":{"total":95995035648,"available":94915301376,"internally_reserved":0,"allocated":474886144,"data_stored":474459047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":604830150},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117219,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740196,"kb_used_data":95380,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93004956,"statfs":{"total":95995035648,"available":95237074944,"internally_reserved":0,"allocated":97669120,"data_stored":97296233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":660273607},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411558,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":736504,"kb_used_data":209704,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":93008648,"statfs":{"total":95995035648,"available":95240855552,"internally_reserved":0,"allocated":214736896,"data_stored":214407329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":539425222},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":692848,"kb_used_data":111904,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93052304,"statfs":{"total":95995035648,"available":95285559296,"internally_reserved":0,"allocated":114589696,"data_stored":114142716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738611,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1068904,"kb_used_data":307344,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92676248,"statfs":{"total":95995035648,"available":94900477952,"internally_reserved":0,"allocated":314720256,"data_stored":314351936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305074176,"data_stored":305070104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102105088,"data_stored":102105088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":205324288,"data_stored":205324288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78381056,"data_stored":78381056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":429723648,"data_stored":429719576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":178388992,"data_stored":178388992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":166203392,"data_stored":166199320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":189333504,"data_stored":189333504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:17:41.742 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T13:17:41.743 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-13T13:17:41.764 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:41.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:41.808 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:41.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:41.908 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:41.933 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:41.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:41.982 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:42.015 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:42.035 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:42.057 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:42.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:42.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:42.159 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:42.184 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:42.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:42.286 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:42.311 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:42.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:42.412 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:42.434 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:42.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:42.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:42.536 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:42.558 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:42.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:42.659 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:42.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:42.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:42.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:17:42.721 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:17:42.721 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:17:42.784 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:42.807 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:42.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:42.909 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:42.933 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:42.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:42.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:43.035 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:43.058 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:43.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:43.077 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:17:43.160 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:43.187 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:43.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:43 smithi052 ceph-mon[29471]: pgmap v831: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 119 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 214/6267 objects misplaced (3.415%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T13:17:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/774606996' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:43.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:43 smithi052 ceph-mon[29471]: from='client.36560 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:43 smithi052 ceph-mon[36682]: pgmap v831: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 119 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 214/6267 objects misplaced (3.415%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T13:17:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/774606996' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:43.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:43 smithi052 ceph-mon[36682]: from='client.36560 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:43.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:43.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:43.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:43.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:43.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:43 smithi167 ceph-mon[33178]: pgmap v831: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 119 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 214/6267 objects misplaced (3.415%); 2.8 MiB/s, 44 objects/s recovering 2024-09-13T13:17:43.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/774606996' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:43.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:43 smithi167 ceph-mon[33178]: from='client.36560 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:17:43.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:43.441 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:43.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:43.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:43.569 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:43.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:43.671 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:43.696 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:43.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:43.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:43.708 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T13:17:43.797 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:43.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:43.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:43.901 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:43.925 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:43.949 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:43.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:44.051 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:44.077 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:44.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:44.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:44.079 INFO:teuthology.orchestra.run.smithi052.stderr:set pool 1 pg_num to 11 2024-09-13T13:17:44.179 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:44.204 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:44.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:44.305 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:44.333 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:44.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:44.435 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:44.457 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:44 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:44.457 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:44 smithi052 ceph-mon[36682]: from='client.36560 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:17:44.457 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:44 smithi052 ceph-mon[36682]: osdmap e416: 8 total, 8 up, 8 in 2024-09-13T13:17:44.458 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3472469640' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T13:17:44.458 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:44 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:44.458 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:44 smithi052 ceph-mon[29471]: from='client.36560 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:17:44.458 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:44 smithi052 ceph-mon[29471]: osdmap e416: 8 total, 8 up, 8 in 2024-09-13T13:17:44.458 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3472469640' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T13:17:44.458 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:44.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:44.551 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:44 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:44.551 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:44 smithi167 ceph-mon[33178]: from='client.36560 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:17:44.551 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:44 smithi167 ceph-mon[33178]: osdmap e416: 8 total, 8 up, 8 in 2024-09-13T13:17:44.551 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3472469640' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T13:17:44.561 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:44.583 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:44.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:44.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:44.685 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:44.708 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:44.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:44.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:44.810 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:44.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:17:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:17:44.827 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-13T13:17:44.827 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-13T13:17:44.836 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:44.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:44.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:44.938 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:44.961 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:44.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:44.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:45.031 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:17:45.031 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:17:45.053 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:17:45.053 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:17:45.063 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:45.072 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:17:45.072 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:17:45.086 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:45.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:45.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:45.090 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:17:45.092 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:17:45.108 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:17:45.109 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:17:45.125 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:17:45.126 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:17:45.142 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:17:45.142 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:17:45.159 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:17:45.160 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:17:45.188 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:45.211 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:45.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:45.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:45.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:45.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:45.340 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:45.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:45.441 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:45.463 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:45 smithi052 ceph-mon[36682]: pgmap v833: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 25 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 529 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-13T13:17:45.463 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:45 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3472469640' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T13:17:45.463 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:45 smithi052 ceph-mon[36682]: osdmap e417: 8 total, 8 up, 8 in 2024-09-13T13:17:45.464 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:45 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:45.464 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:45 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1670862622' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:45.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:45 smithi052 ceph-mon[29471]: pgmap v833: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 25 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 529 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-13T13:17:45.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:45 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3472469640' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T13:17:45.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:45 smithi052 ceph-mon[29471]: osdmap e417: 8 total, 8 up, 8 in 2024-09-13T13:17:45.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:45 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:45.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:45 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1670862622' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:45.467 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:45.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:45.568 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:45 smithi167 ceph-mon[33178]: pgmap v833: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 25 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 529 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-13T13:17:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:45 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3472469640' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T13:17:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:45 smithi167 ceph-mon[33178]: osdmap e417: 8 total, 8 up, 8 in 2024-09-13T13:17:45.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:45 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:45.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:45 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1670862622' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:45.594 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:45.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:45.695 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:45.721 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:45.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:45.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:45.731 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:45.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:45.849 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:45.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:45.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:45.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:45.979 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:45.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:46.080 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:46.108 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:46.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:46.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:46.122 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:46.210 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:46.236 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:46.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:46.338 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:46.363 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:46.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:46.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:46.385 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:17:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:17:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:17:46.385 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:46 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T13:17:46.386 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:46 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T13:17:46.386 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:46 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T13:17:46.386 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:46 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T13:17:46.465 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:46.491 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:46.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:46.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:46 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T13:17:46.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:46 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T13:17:46.593 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:46.619 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:46.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:46.721 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:46.747 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:46.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:46.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:46.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:46.875 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:46.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:46.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:46.877 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:17:46.877 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:17:46.877 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 0 2024-09-13T13:17:46.877 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:17:46.976 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:46.999 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:47.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:47.101 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:47.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:47.127 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:47.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:47.228 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:47.256 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:47.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:47.358 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:47.361 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:47 smithi052 ceph-mon[36682]: pgmap v835: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 868 op/s 2024-09-13T13:17:47.361 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:47 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T13:17:47.361 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:47 smithi052 ceph-mon[36682]: osdmap e418: 8 total, 8 up, 8 in 2024-09-13T13:17:47.361 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:47 smithi052 ceph-mon[29471]: pgmap v835: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 868 op/s 2024-09-13T13:17:47.361 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:47 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T13:17:47.361 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:47 smithi052 ceph-mon[29471]: osdmap e418: 8 total, 8 up, 8 in 2024-09-13T13:17:47.383 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:47.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:47.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:47.495 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:17:47.509 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:47.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:47.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:47 smithi167 ceph-mon[33178]: pgmap v835: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 868 op/s 2024-09-13T13:17:47.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:47 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T13:17:47.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:47 smithi167 ceph-mon[33178]: osdmap e418: 8 total, 8 up, 8 in 2024-09-13T13:17:47.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:47.637 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:47.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:47.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:47.762 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:47.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:47.863 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:47.885 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:47.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:47.987 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:48.010 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T13:17:48.014 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:48.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:48.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:48.038 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'unique_pool_3' removed 2024-09-13T13:17:48.116 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:48.157 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:17:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:48.196 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:48.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[36682]: osdmap e419: 8 total, 8 up, 8 in 2024-09-13T13:17:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1813812184' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:17:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[36682]: 1.0 scrub starts 2024-09-13T13:17:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T13:17:48.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T13:17:48.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1813812184' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:17:48.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T13:17:48.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[36682]: osdmap e420: 8 total, 8 up, 8 in 2024-09-13T13:17:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[29471]: osdmap e419: 8 total, 8 up, 8 in 2024-09-13T13:17:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1813812184' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:17:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[29471]: 1.0 scrub starts 2024-09-13T13:17:48.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T13:17:48.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T13:17:48.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1813812184' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:17:48.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T13:17:48.241 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:48 smithi052 ceph-mon[29471]: osdmap e420: 8 total, 8 up, 8 in 2024-09-13T13:17:48.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:48.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:48.319 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:48.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:48.420 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:48.442 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:48.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:48.472 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:48 smithi167 ceph-mon[33178]: osdmap e419: 8 total, 8 up, 8 in 2024-09-13T13:17:48.472 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:48 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1813812184' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:17:48.472 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:48 smithi167 ceph-mon[33178]: 1.0 scrub starts 2024-09-13T13:17:48.472 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:48 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T13:17:48.472 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:48 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T13:17:48.472 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:48 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1813812184' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:17:48.472 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:48 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T13:17:48.472 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:48 smithi167 ceph-mon[33178]: osdmap e420: 8 total, 8 up, 8 in 2024-09-13T13:17:48.543 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:48.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:48.668 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:48.691 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:48.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:48.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:48.793 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:48.816 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:48.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:48.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:48.818 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T13:17:48.819 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T13:17:48.819 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-13T13:17:48.819 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-13T13:17:48.917 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:48.920 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:48.927 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:48.931 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:48.939 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:48.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:49.041 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:49.070 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:49.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:49.171 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:49.193 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:49.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:49.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:49.316 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:49.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:49.344 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:49 smithi052 ceph-mon[36682]: pgmap v838: 158 pgs: 10 unknown, 1 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 293 op/s 2024-09-13T13:17:49.344 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:49 smithi052 ceph-mon[36682]: 1.0 scrub starts 2024-09-13T13:17:49.344 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:49 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:49.345 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:49 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:17:49.345 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:49 smithi052 ceph-mon[36682]: osdmap e421: 8 total, 8 up, 8 in 2024-09-13T13:17:49.345 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:49 smithi052 ceph-mon[29471]: pgmap v838: 158 pgs: 10 unknown, 1 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 293 op/s 2024-09-13T13:17:49.345 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:49 smithi052 ceph-mon[29471]: 1.0 scrub starts 2024-09-13T13:17:49.345 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:49 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:49.345 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:49 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:17:49.345 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:49 smithi052 ceph-mon[29471]: osdmap e421: 8 total, 8 up, 8 in 2024-09-13T13:17:49.419 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:49.441 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:49.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:49.543 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:49.566 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:49.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:49.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:49 smithi167 ceph-mon[33178]: pgmap v838: 158 pgs: 10 unknown, 1 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 293 op/s 2024-09-13T13:17:49.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:49 smithi167 ceph-mon[33178]: 1.0 scrub starts 2024-09-13T13:17:49.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:49 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:17:49.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:49 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:17:49.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:49 smithi167 ceph-mon[33178]: osdmap e421: 8 total, 8 up, 8 in 2024-09-13T13:17:49.668 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:49.694 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:49.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:49.795 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:49.821 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:49.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:49.828 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:17:49.828 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T13:17:49.898 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:17:49.898 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:17:49.898 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:17:49.898 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:17:49.899 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:17:49.899 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:17:49.899 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:17:49.899 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:17:49.899 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:17:49.899 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:17:49.899 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:17:49.899 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:17:49.899 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:17:49.900 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:17:49.900 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:17:49.900 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:17:49.922 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:49.944 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:49.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:50.046 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:50.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:50.071 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:50.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:50.115 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:17:50.172 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:50.196 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:50.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:50.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:50.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:50.323 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:50.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:50.424 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:50.446 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:50.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:50.451 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[36682]: 1.3 scrub starts 2024-09-13T13:17:50.451 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[36682]: 1.a scrub starts 2024-09-13T13:17:50.452 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[36682]: 1.7 scrub starts 2024-09-13T13:17:50.452 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[36682]: 1.2 scrub starts 2024-09-13T13:17:50.452 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3481801299' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:50.452 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/421595212' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:17:50.452 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[36682]: from='client.36590 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:17:50.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[29471]: 1.3 scrub starts 2024-09-13T13:17:50.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[29471]: 1.a scrub starts 2024-09-13T13:17:50.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[29471]: 1.7 scrub starts 2024-09-13T13:17:50.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[29471]: 1.2 scrub starts 2024-09-13T13:17:50.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3481801299' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:50.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/421595212' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:17:50.454 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:50 smithi052 ceph-mon[29471]: from='client.36590 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:17:50.496 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:50 smithi167 ceph-mon[33178]: 1.3 scrub starts 2024-09-13T13:17:50.496 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:50 smithi167 ceph-mon[33178]: 1.a scrub starts 2024-09-13T13:17:50.496 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:50 smithi167 ceph-mon[33178]: 1.7 scrub starts 2024-09-13T13:17:50.496 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:50 smithi167 ceph-mon[33178]: 1.2 scrub starts 2024-09-13T13:17:50.496 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:50 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3481801299' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:50.496 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:50 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/421595212' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:17:50.496 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:50 smithi167 ceph-mon[33178]: from='client.36590 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:17:50.548 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:50.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:50.570 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:50.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:50.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:50.692 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:50.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:50.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:50.794 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:50.815 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:50.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:50.917 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:50.940 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:50.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:51.041 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:51.068 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:51.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:51.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:51.170 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:51.191 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:51.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:51.232 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:17:51.232 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:17:51.293 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:51.314 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:51.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:51.415 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:51.436 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:51.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:51.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:51.455 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[29471]: 1.5 scrub starts 2024-09-13T13:17:51.455 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[29471]: pgmap v841: 142 pgs: 4 peering, 4 unknown, 134 active+clean; 373 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:17:51.455 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[29471]: 1.9 scrub starts 2024-09-13T13:17:51.455 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[29471]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T13:17:51.455 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:51.455 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[29471]: from='client.36590 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:17:51.455 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[29471]: osdmap e422: 8 total, 8 up, 8 in 2024-09-13T13:17:51.455 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[29471]: 1.1 scrub starts 2024-09-13T13:17:51.456 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[29471]: 1.6 scrub starts 2024-09-13T13:17:51.456 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[36682]: 1.5 scrub starts 2024-09-13T13:17:51.456 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[36682]: pgmap v841: 142 pgs: 4 peering, 4 unknown, 134 active+clean; 373 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:17:51.456 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[36682]: 1.9 scrub starts 2024-09-13T13:17:51.456 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[36682]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T13:17:51.456 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:51.456 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[36682]: from='client.36590 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:17:51.456 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[36682]: osdmap e422: 8 total, 8 up, 8 in 2024-09-13T13:17:51.456 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[36682]: 1.1 scrub starts 2024-09-13T13:17:51.456 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:51 smithi052 ceph-mon[36682]: 1.6 scrub starts 2024-09-13T13:17:51.538 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:51.559 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:51.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:51.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:51 smithi167 ceph-mon[33178]: 1.5 scrub starts 2024-09-13T13:17:51.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:51 smithi167 ceph-mon[33178]: pgmap v841: 142 pgs: 4 peering, 4 unknown, 134 active+clean; 373 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:17:51.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:51 smithi167 ceph-mon[33178]: 1.9 scrub starts 2024-09-13T13:17:51.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:51 smithi167 ceph-mon[33178]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T13:17:51.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:51 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:17:51.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:51 smithi167 ceph-mon[33178]: from='client.36590 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:17:51.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:51 smithi167 ceph-mon[33178]: osdmap e422: 8 total, 8 up, 8 in 2024-09-13T13:17:51.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:51 smithi167 ceph-mon[33178]: 1.1 scrub starts 2024-09-13T13:17:51.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:51 smithi167 ceph-mon[33178]: 1.6 scrub starts 2024-09-13T13:17:51.661 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:51.683 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:51.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:51.785 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:51.806 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:51.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:51.908 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:51.916 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T13:17:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:51.930 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:51.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:52.031 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:52.054 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:52.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:52.155 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:52.178 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:52.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:52.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:52 smithi052 ceph-mon[36682]: 1.4 scrub starts 2024-09-13T13:17:52.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:52 smithi052 ceph-mon[36682]: 1.8 scrub starts 2024-09-13T13:17:52.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:52 smithi052 ceph-mon[36682]: 1.8 scrub ok 2024-09-13T13:17:52.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:52 smithi052 ceph-mon[29471]: 1.4 scrub starts 2024-09-13T13:17:52.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:52 smithi052 ceph-mon[29471]: 1.8 scrub starts 2024-09-13T13:17:52.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:52 smithi052 ceph-mon[29471]: 1.8 scrub ok 2024-09-13T13:17:52.280 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:52.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:52.302 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:52.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:52.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:52 smithi167 ceph-mon[33178]: 1.4 scrub starts 2024-09-13T13:17:52.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:52 smithi167 ceph-mon[33178]: 1.8 scrub starts 2024-09-13T13:17:52.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:52 smithi167 ceph-mon[33178]: 1.8 scrub ok 2024-09-13T13:17:52.403 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:52.425 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:52.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:52.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:52.526 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:52.552 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:52.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:52.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:52.591 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:52.591 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:52.625 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:52.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:52.675 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:52.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:52.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:52.777 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:52.798 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:52.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:52.900 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:52.960 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:52.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:53.061 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:53.083 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:53.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:53.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:53.089 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:17:53.125 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:53 smithi052 ceph-mon[36682]: pgmap v843: 142 pgs: 4 peering, 2 unknown, 136 active+clean; 478 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:17:53.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:53 smithi052 ceph-mon[36682]: 6.1d scrub starts 2024-09-13T13:17:53.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:53 smithi052 ceph-mon[36682]: 6.1d scrub ok 2024-09-13T13:17:53.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:53 smithi052 ceph-mon[36682]: 4.16 scrub starts 2024-09-13T13:17:53.126 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:53 smithi052 ceph-mon[36682]: 4.16 scrub ok 2024-09-13T13:17:53.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:53 smithi052 ceph-mon[29471]: pgmap v843: 142 pgs: 4 peering, 2 unknown, 136 active+clean; 478 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:17:53.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:53 smithi052 ceph-mon[29471]: 6.1d scrub starts 2024-09-13T13:17:53.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:53 smithi052 ceph-mon[29471]: 6.1d scrub ok 2024-09-13T13:17:53.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:53 smithi052 ceph-mon[29471]: 4.16 scrub starts 2024-09-13T13:17:53.127 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:53 smithi052 ceph-mon[29471]: 4.16 scrub ok 2024-09-13T13:17:53.185 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:53.228 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:53.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:53.330 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:53.354 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:53.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:53.456 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:53.479 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:53.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:53.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:53 smithi167 ceph-mon[33178]: pgmap v843: 142 pgs: 4 peering, 2 unknown, 136 active+clean; 478 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:17:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:53 smithi167 ceph-mon[33178]: 6.1d scrub starts 2024-09-13T13:17:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:53 smithi167 ceph-mon[33178]: 6.1d scrub ok 2024-09-13T13:17:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:53 smithi167 ceph-mon[33178]: 4.16 scrub starts 2024-09-13T13:17:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:53 smithi167 ceph-mon[33178]: 4.16 scrub ok 2024-09-13T13:17:53.580 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:53.603 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:53.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:53.704 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:53.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:53.727 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:53.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:53.828 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:53.850 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:53.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:53.952 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:53.975 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:53.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:54.077 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:54.100 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:54.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:54.129 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'unique_pool_4' created 2024-09-13T13:17:54.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[36682]: 3.3 scrub starts 2024-09-13T13:17:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[36682]: 3.3 scrub ok 2024-09-13T13:17:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[36682]: 5.8 scrub starts 2024-09-13T13:17:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[36682]: 5.8 scrub ok 2024-09-13T13:17:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[36682]: 3.4 scrub starts 2024-09-13T13:17:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[36682]: 3.4 scrub ok 2024-09-13T13:17:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[36682]: 6.15 scrub starts 2024-09-13T13:17:54.221 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[36682]: 6.15 scrub ok 2024-09-13T13:17:54.222 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3263950377' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T13:17:54.222 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[36682]: from='client.36596 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T13:17:54.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[29471]: 3.3 scrub starts 2024-09-13T13:17:54.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[29471]: 3.3 scrub ok 2024-09-13T13:17:54.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[29471]: 5.8 scrub starts 2024-09-13T13:17:54.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[29471]: 5.8 scrub ok 2024-09-13T13:17:54.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[29471]: 3.4 scrub starts 2024-09-13T13:17:54.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[29471]: 3.4 scrub ok 2024-09-13T13:17:54.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[29471]: 6.15 scrub starts 2024-09-13T13:17:54.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[29471]: 6.15 scrub ok 2024-09-13T13:17:54.222 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3263950377' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T13:17:54.223 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:54 smithi052 ceph-mon[29471]: from='client.36596 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T13:17:54.223 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:54.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:54.325 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:54.351 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:54.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:54.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:54.453 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:54.477 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:54.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:54.554 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:54 smithi167 ceph-mon[33178]: 3.3 scrub starts 2024-09-13T13:17:54.554 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:54 smithi167 ceph-mon[33178]: 3.3 scrub ok 2024-09-13T13:17:54.554 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:54 smithi167 ceph-mon[33178]: 5.8 scrub starts 2024-09-13T13:17:54.554 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:54 smithi167 ceph-mon[33178]: 5.8 scrub ok 2024-09-13T13:17:54.554 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:54 smithi167 ceph-mon[33178]: 3.4 scrub starts 2024-09-13T13:17:54.555 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:54 smithi167 ceph-mon[33178]: 3.4 scrub ok 2024-09-13T13:17:54.555 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:54 smithi167 ceph-mon[33178]: 6.15 scrub starts 2024-09-13T13:17:54.555 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:54 smithi167 ceph-mon[33178]: 6.15 scrub ok 2024-09-13T13:17:54.555 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:54 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3263950377' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T13:17:54.555 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:54 smithi167 ceph-mon[33178]: from='client.36596 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T13:17:54.557 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:17:54.578 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:54.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:54.623 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:54.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:54.724 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:54.747 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:54.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:54.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:54.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:17:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:17:54.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:54.871 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:54.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:54.959 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-13T13:17:54.974 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:54.997 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:54.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:55.098 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:55.121 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:55.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:55.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:55.130 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T13:17:55.198 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:55 smithi052 ceph-mon[36682]: pgmap v844: 142 pgs: 142 active+clean; 787 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T13:17:55.198 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:55 smithi052 ceph-mon[36682]: from='client.36596 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T13:17:55.198 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:55 smithi052 ceph-mon[36682]: osdmap e423: 8 total, 8 up, 8 in 2024-09-13T13:17:55.198 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:55 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/993496370' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:17:55.198 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:55 smithi052 ceph-mon[36682]: from='client.36602 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:17:55.199 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:55.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:55 smithi052 ceph-mon[29471]: pgmap v844: 142 pgs: 142 active+clean; 787 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T13:17:55.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:55 smithi052 ceph-mon[29471]: from='client.36596 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T13:17:55.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:55 smithi052 ceph-mon[29471]: osdmap e423: 8 total, 8 up, 8 in 2024-09-13T13:17:55.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:55 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/993496370' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:17:55.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:55 smithi052 ceph-mon[29471]: from='client.36602 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:17:55.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:55.223 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:55.244 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:55.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:55.346 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:55.369 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:55.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:55.471 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:55.495 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:55.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:55.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:55 smithi167 ceph-mon[33178]: pgmap v844: 142 pgs: 142 active+clean; 787 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T13:17:55.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:55 smithi167 ceph-mon[33178]: from='client.36596 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T13:17:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:55 smithi167 ceph-mon[33178]: osdmap e423: 8 total, 8 up, 8 in 2024-09-13T13:17:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:55 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/993496370' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:17:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:55 smithi167 ceph-mon[33178]: from='client.36602 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:17:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:17:55.596 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:55.613 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:17:55.613 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:17:55.614 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:55.621 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:55.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:55.723 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:55.746 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:55.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:55.848 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:55.870 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:55.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:55.972 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:55.995 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:55.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:56.096 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:56.120 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T13:17:56.120 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:56.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:56.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:56.223 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:56.245 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:56.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:56.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:56.347 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:56.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:56.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:56.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:56.467 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:17:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:17:56.467 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[36682]: 3.9 scrub starts 2024-09-13T13:17:56.467 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[36682]: 3.9 scrub ok 2024-09-13T13:17:56.467 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[36682]: 5.a scrub starts 2024-09-13T13:17:56.467 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[36682]: 5.a scrub ok 2024-09-13T13:17:56.468 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[36682]: from='client.36602 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:17:56.468 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[36682]: osdmap e424: 8 total, 8 up, 8 in 2024-09-13T13:17:56.468 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[36682]: 5.7 scrub starts 2024-09-13T13:17:56.468 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[36682]: 5.7 scrub ok 2024-09-13T13:17:56.468 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[36682]: 5.11 scrub starts 2024-09-13T13:17:56.468 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[36682]: 5.11 scrub ok 2024-09-13T13:17:56.468 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[29471]: 3.9 scrub starts 2024-09-13T13:17:56.468 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[29471]: 3.9 scrub ok 2024-09-13T13:17:56.468 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[29471]: 5.a scrub starts 2024-09-13T13:17:56.468 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[29471]: 5.a scrub ok 2024-09-13T13:17:56.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[29471]: from='client.36602 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:17:56.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[29471]: osdmap e424: 8 total, 8 up, 8 in 2024-09-13T13:17:56.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[29471]: 5.7 scrub starts 2024-09-13T13:17:56.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[29471]: 5.7 scrub ok 2024-09-13T13:17:56.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[29471]: 5.11 scrub starts 2024-09-13T13:17:56.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:56 smithi052 ceph-mon[29471]: 5.11 scrub ok 2024-09-13T13:17:56.471 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:17:56.494 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:56.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:56.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:17:56.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:56 smithi167 ceph-mon[33178]: 3.9 scrub starts 2024-09-13T13:17:56.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:56 smithi167 ceph-mon[33178]: 3.9 scrub ok 2024-09-13T13:17:56.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:56 smithi167 ceph-mon[33178]: 5.a scrub starts 2024-09-13T13:17:56.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:56 smithi167 ceph-mon[33178]: 5.a scrub ok 2024-09-13T13:17:56.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:56 smithi167 ceph-mon[33178]: from='client.36602 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:17:56.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:56 smithi167 ceph-mon[33178]: osdmap e424: 8 total, 8 up, 8 in 2024-09-13T13:17:56.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:56 smithi167 ceph-mon[33178]: 5.7 scrub starts 2024-09-13T13:17:56.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:56 smithi167 ceph-mon[33178]: 5.7 scrub ok 2024-09-13T13:17:56.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:56 smithi167 ceph-mon[33178]: 5.11 scrub starts 2024-09-13T13:17:56.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:56 smithi167 ceph-mon[33178]: 5.11 scrub ok 2024-09-13T13:17:56.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:56.618 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:56.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:56.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:56.634 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:56.648 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:56.719 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:56.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:56.742 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:56.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:56.839 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:17:56.840 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:17:56.843 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:56.846 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:17:56.846 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:17:56.861 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:17:56.862 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:17:56.866 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:56.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:56.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:56.878 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:17:56.878 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:17:56.895 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:17:56.895 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:17:56.912 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:17:56.913 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:17:56.929 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:17:56.929 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:17:56.946 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:17:56.946 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:17:56.968 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:17:56.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:56.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:17:57.092 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:57.116 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:57.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:57.217 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:57.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:57.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:57.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:57.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:57.363 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:57.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:57.385 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[36682]: pgmap v847: 158 pgs: 1 creating+peering, 15 unknown, 142 active+clean; 787 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 74 KiB/s wr, 10 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-13T13:17:57.385 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[36682]: 4.1d scrub starts 2024-09-13T13:17:57.386 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[36682]: 4.1d scrub ok 2024-09-13T13:17:57.386 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[36682]: 5.19 scrub starts 2024-09-13T13:17:57.386 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[36682]: 5.19 scrub ok 2024-09-13T13:17:57.386 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[36682]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:17:57.386 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[36682]: osdmap e425: 8 total, 8 up, 8 in 2024-09-13T13:17:57.386 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/623671538' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:57.386 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[29471]: pgmap v847: 158 pgs: 1 creating+peering, 15 unknown, 142 active+clean; 787 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 74 KiB/s wr, 10 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-13T13:17:57.386 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[29471]: 4.1d scrub starts 2024-09-13T13:17:57.386 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[29471]: 4.1d scrub ok 2024-09-13T13:17:57.386 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[29471]: 5.19 scrub starts 2024-09-13T13:17:57.386 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[29471]: 5.19 scrub ok 2024-09-13T13:17:57.387 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[29471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:17:57.387 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[29471]: osdmap e425: 8 total, 8 up, 8 in 2024-09-13T13:17:57.387 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:57 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/623671538' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:57.390 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T13:17:57.465 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:57.487 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:57.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:57.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:57 smithi167 ceph-mon[33178]: pgmap v847: 158 pgs: 1 creating+peering, 15 unknown, 142 active+clean; 787 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 74 KiB/s wr, 10 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-13T13:17:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:57 smithi167 ceph-mon[33178]: 4.1d scrub starts 2024-09-13T13:17:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:57 smithi167 ceph-mon[33178]: 4.1d scrub ok 2024-09-13T13:17:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:57 smithi167 ceph-mon[33178]: 5.19 scrub starts 2024-09-13T13:17:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:57 smithi167 ceph-mon[33178]: 5.19 scrub ok 2024-09-13T13:17:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:57 smithi167 ceph-mon[33178]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T13:17:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:57 smithi167 ceph-mon[33178]: osdmap e425: 8 total, 8 up, 8 in 2024-09-13T13:17:57.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:57 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/623671538' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:17:57.589 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:57.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:57.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:57.713 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:57.737 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:57.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:57.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:57.838 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:57.861 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:57.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:57.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:57.963 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:57.988 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:57.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:57.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:58.090 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:58.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:58.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:58.134 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:58 smithi052 ceph-mon[29471]: 3.15 scrub starts 2024-09-13T13:17:58.134 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:58 smithi052 ceph-mon[29471]: 3.15 scrub ok 2024-09-13T13:17:58.134 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:58 smithi052 ceph-mon[29471]: 5.15 scrub starts 2024-09-13T13:17:58.134 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:58 smithi052 ceph-mon[29471]: 5.15 scrub ok 2024-09-13T13:17:58.216 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:58.240 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:58.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:58.342 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:58.365 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:58.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:58.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:58.407 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:58 smithi052 ceph-mon[36682]: 3.15 scrub starts 2024-09-13T13:17:58.407 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:58 smithi052 ceph-mon[36682]: 3.15 scrub ok 2024-09-13T13:17:58.407 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:58 smithi052 ceph-mon[36682]: 5.15 scrub starts 2024-09-13T13:17:58.407 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:58 smithi052 ceph-mon[36682]: 5.15 scrub ok 2024-09-13T13:17:58.407 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:58 smithi052 ceph-mon[36682]: 3.17 scrub starts 2024-09-13T13:17:58.407 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:58 smithi052 ceph-mon[36682]: 3.17 scrub ok 2024-09-13T13:17:58.407 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:58 smithi052 ceph-mon[29471]: 3.17 scrub starts 2024-09-13T13:17:58.407 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:58 smithi052 ceph-mon[29471]: 3.17 scrub ok 2024-09-13T13:17:58.437 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:17:58.467 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:58.562 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:58.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:58 smithi167 ceph-mon[33178]: 3.15 scrub starts 2024-09-13T13:17:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:58 smithi167 ceph-mon[33178]: 3.15 scrub ok 2024-09-13T13:17:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:58 smithi167 ceph-mon[33178]: 5.15 scrub starts 2024-09-13T13:17:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:58 smithi167 ceph-mon[33178]: 5.15 scrub ok 2024-09-13T13:17:58.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:58 smithi167 ceph-mon[33178]: 3.17 scrub starts 2024-09-13T13:17:58.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:58 smithi167 ceph-mon[33178]: 3.17 scrub ok 2024-09-13T13:17:58.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:58.686 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:58.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:58.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:58.813 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:58.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:58.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:58.914 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:58.937 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:58.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:58.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:17:59.039 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:59.064 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:59.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:59.141 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-13T13:17:59.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:17:59.190 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:59.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:17:59.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:59 smithi052 ceph-mon[29471]: pgmap v849: 158 pgs: 1 creating+peering, 6 unknown, 151 active+clean; 787 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 25 KiB/s wr, 4 op/s; 62 KiB/s, 0 objects/s recovering 2024-09-13T13:17:59.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:59 smithi052 ceph-mon[29471]: 5.2 scrub starts 2024-09-13T13:17:59.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:59 smithi052 ceph-mon[29471]: 5.2 scrub ok 2024-09-13T13:17:59.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:59 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/800036089' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:17:59.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:17:59 smithi052 ceph-mon[29471]: from='client.36614 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:17:59.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:59 smithi052 ceph-mon[36682]: pgmap v849: 158 pgs: 1 creating+peering, 6 unknown, 151 active+clean; 787 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 25 KiB/s wr, 4 op/s; 62 KiB/s, 0 objects/s recovering 2024-09-13T13:17:59.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:59 smithi052 ceph-mon[36682]: 5.2 scrub starts 2024-09-13T13:17:59.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:59 smithi052 ceph-mon[36682]: 5.2 scrub ok 2024-09-13T13:17:59.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:59 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/800036089' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:17:59.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:17:59 smithi052 ceph-mon[36682]: from='client.36614 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:17:59.292 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:17:59.316 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:59.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:17:59.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:17:59.440 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:59.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:59.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:17:59.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:17:59.566 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:17:59.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:59.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:17:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:59 smithi167 ceph-mon[33178]: pgmap v849: 158 pgs: 1 creating+peering, 6 unknown, 151 active+clean; 787 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 25 KiB/s wr, 4 op/s; 62 KiB/s, 0 objects/s recovering 2024-09-13T13:17:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:59 smithi167 ceph-mon[33178]: 5.2 scrub starts 2024-09-13T13:17:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:59 smithi167 ceph-mon[33178]: 5.2 scrub ok 2024-09-13T13:17:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:59 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/800036089' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:17:59.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:17:59 smithi167 ceph-mon[33178]: from='client.36614 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T13:17:59.668 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:59.692 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:59.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:59.793 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:17:59.816 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:59.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:17:59.918 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:17:59.942 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:17:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:17:59.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:00.043 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:00.069 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:00.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:00.171 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:00.193 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:00.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:00.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:00.295 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:00.315 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:00.320 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:00.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:00.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:00.422 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:00.436 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: 5.1 scrub starts 2024-09-13T13:18:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: 5.1 scrub ok 2024-09-13T13:18:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: 5.4 scrub starts 2024-09-13T13:18:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: 5.4 scrub ok 2024-09-13T13:18:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: from='client.36614 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:18:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: osdmap e426: 8 total, 8 up, 8 in 2024-09-13T13:18:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: 6.1 scrub starts 2024-09-13T13:18:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: 6.1 scrub ok 2024-09-13T13:18:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: 4.17 scrub starts 2024-09-13T13:18:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: 4.17 scrub ok 2024-09-13T13:18:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: 6.3 scrub starts 2024-09-13T13:18:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: 6.3 scrub ok 2024-09-13T13:18:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: 6.2 scrub starts 2024-09-13T13:18:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[36682]: 6.2 scrub ok 2024-09-13T13:18:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: 5.1 scrub starts 2024-09-13T13:18:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: 5.1 scrub ok 2024-09-13T13:18:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: 5.4 scrub starts 2024-09-13T13:18:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: 5.4 scrub ok 2024-09-13T13:18:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: from='client.36614 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:18:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: osdmap e426: 8 total, 8 up, 8 in 2024-09-13T13:18:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: 6.1 scrub starts 2024-09-13T13:18:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: 6.1 scrub ok 2024-09-13T13:18:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: 4.17 scrub starts 2024-09-13T13:18:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: 4.17 scrub ok 2024-09-13T13:18:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: 6.3 scrub starts 2024-09-13T13:18:00.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: 6.3 scrub ok 2024-09-13T13:18:00.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: 6.2 scrub starts 2024-09-13T13:18:00.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:00 smithi052 ceph-mon[29471]: 6.2 scrub ok 2024-09-13T13:18:00.439 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:18:00.448 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:00.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:00.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:00.550 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:00.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: 5.1 scrub starts 2024-09-13T13:18:00.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: 5.1 scrub ok 2024-09-13T13:18:00.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: 5.4 scrub starts 2024-09-13T13:18:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: 5.4 scrub ok 2024-09-13T13:18:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: from='client.36614 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T13:18:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: osdmap e426: 8 total, 8 up, 8 in 2024-09-13T13:18:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: 6.1 scrub starts 2024-09-13T13:18:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: 6.1 scrub ok 2024-09-13T13:18:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: 4.17 scrub starts 2024-09-13T13:18:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: 4.17 scrub ok 2024-09-13T13:18:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: 6.3 scrub starts 2024-09-13T13:18:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: 6.3 scrub ok 2024-09-13T13:18:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: 6.2 scrub starts 2024-09-13T13:18:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:00 smithi167 ceph-mon[33178]: 6.2 scrub ok 2024-09-13T13:18:00.573 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:00.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:00.675 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:00.702 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:00.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:00.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:00.813 DEBUG:teuthology.orchestra.run.smithi052:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-13T13:18:00.816 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T13:18:00.816 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T13:18:00.828 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:18:00.828 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:18:00.828 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 0 2024-09-13T13:18:00.828 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:18:00.831 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:00.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:00.871 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:hints = 1 2024-09-13T13:18:00.871 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T13:18:00.872 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Object prefix: benchmark_data_smithi052_182575 2024-09-13T13:18:00.872 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:18:00.872 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 0 0 0 0 0 0 - 0 2024-09-13T13:18:00.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:00.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:00.968 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:00.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:01.069 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:01.119 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:01.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:01.155 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T13:18:01.162 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:18:01.221 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:01.264 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:01.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:01.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:01 smithi167 ceph-mon[33178]: 5.6 scrub starts 2024-09-13T13:18:01.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:01 smithi167 ceph-mon[33178]: 5.6 scrub ok 2024-09-13T13:18:01.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:01 smithi167 ceph-mon[33178]: pgmap v851: 158 pgs: 158 active+clean; 787 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 874 B/s rd, 0 op/s 2024-09-13T13:18:01.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:01 smithi167 ceph-mon[33178]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:18:01.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:01 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2254559605' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:01.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:01 smithi167 ceph-mon[33178]: from='client.36620 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:01.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:01 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3202028916' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:18:01.366 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:01.408 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:01.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:01.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:01.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[36682]: 5.6 scrub starts 2024-09-13T13:18:01.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[36682]: 5.6 scrub ok 2024-09-13T13:18:01.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[36682]: pgmap v851: 158 pgs: 158 active+clean; 787 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 874 B/s rd, 0 op/s 2024-09-13T13:18:01.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[36682]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:18:01.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2254559605' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[36682]: from='client.36620 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:01.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3202028916' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:18:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[29471]: 5.6 scrub starts 2024-09-13T13:18:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[29471]: 5.6 scrub ok 2024-09-13T13:18:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[29471]: pgmap v851: 158 pgs: 158 active+clean; 787 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 874 B/s rd, 0 op/s 2024-09-13T13:18:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[29471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T13:18:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2254559605' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[29471]: from='client.36620 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:01.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:01 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3202028916' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:18:01.510 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:01.553 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:01.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:01.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:01.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:01.683 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:01.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:01.703 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T13:18:01.785 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:01.815 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:01.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:01.872 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 1 16 3364 3348 209.19 209.25 0.00267879 0.00475398 2024-09-13T13:18:01.917 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:01.949 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:01.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:01.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:02.051 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:02.081 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:02.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:02.176 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[29471]: 4.4 scrub starts 2024-09-13T13:18:02.176 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:18:02.176 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:18:02.182 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:02.215 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:02.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:02.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:02.348 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:02.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:02.450 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:02.482 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:02.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:02.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[29471]: from='client.36620 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:18:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3202028916' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:18:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[29471]: 4.4 scrub ok 2024-09-13T13:18:02.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[29471]: osdmap e427: 8 total, 8 up, 8 in 2024-09-13T13:18:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[36682]: 4.4 scrub starts 2024-09-13T13:18:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:18:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:18:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[36682]: from='client.36620 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:18:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3202028916' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:18:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[36682]: 4.4 scrub ok 2024-09-13T13:18:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:02 smithi052 ceph-mon[36682]: osdmap e427: 8 total, 8 up, 8 in 2024-09-13T13:18:02.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:02 smithi167 ceph-mon[33178]: 4.4 scrub starts 2024-09-13T13:18:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:02 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:18:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:02 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:18:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:02 smithi167 ceph-mon[33178]: from='client.36620 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:18:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:02 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3202028916' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:18:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:02 smithi167 ceph-mon[33178]: 4.4 scrub ok 2024-09-13T13:18:02.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:02 smithi167 ceph-mon[33178]: osdmap e427: 8 total, 8 up, 8 in 2024-09-13T13:18:02.584 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:02.614 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:02.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:02.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:02.716 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:02.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:02.760 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:02.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:02.862 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:02.872 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 2 16 6604 6588 205.826 202.5 0.0055687 0.00484635 2024-09-13T13:18:02.902 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:02.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:03.004 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:03.044 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:03.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:03.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:03.146 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:03.186 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:03.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:03.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:03.194 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:03.243 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T13:18:03.288 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:03.375 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:03.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:03.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:03.477 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:03.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:03 smithi052 ceph-mon[36682]: pgmap v853: 158 pgs: 158 active+clean; 20 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 3.2 MiB/s wr, 51 op/s 2024-09-13T13:18:03.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:03 smithi052 ceph-mon[29471]: pgmap v853: 158 pgs: 158 active+clean; 20 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 3.2 MiB/s wr, 51 op/s 2024-09-13T13:18:03.508 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:03.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:03 smithi167 ceph-mon[33178]: pgmap v853: 158 pgs: 158 active+clean; 20 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 3.2 MiB/s wr, 51 op/s 2024-09-13T13:18:03.610 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:03.635 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:03.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:03.737 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:03.782 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:03.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:03.872 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 3 15 9265 9250 192.672 166.375 0.00952027 0.00518083 2024-09-13T13:18:03.884 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:03.928 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:03.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:03.991 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:03.998 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:18:03.998 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:18:04.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:04.039 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:18:04.039 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:18:04.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:04.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:04.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:04.077 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:18:04.077 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:18:04.115 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:18:04.115 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:18:04.150 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:18:04.151 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:18:04.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:04.187 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:18:04.188 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:18:04.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:04.189 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:04.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:04.227 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:18:04.227 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:18:04.264 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:18:04.264 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:18:04.290 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:04.329 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:04.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:04.431 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:04.460 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:04.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[36682]: 5.13 scrub starts 2024-09-13T13:18:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[36682]: 5.13 scrub ok 2024-09-13T13:18:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[36682]: 3.a scrub starts 2024-09-13T13:18:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[36682]: 3.a scrub ok 2024-09-13T13:18:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:18:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:18:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4157828514' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[29471]: 5.13 scrub starts 2024-09-13T13:18:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[29471]: 5.13 scrub ok 2024-09-13T13:18:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[29471]: 3.a scrub starts 2024-09-13T13:18:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[29471]: 3.a scrub ok 2024-09-13T13:18:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:18:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:18:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:04 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4157828514' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:04.562 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:04.564 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:04 smithi167 ceph-mon[33178]: 5.13 scrub starts 2024-09-13T13:18:04.564 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:04 smithi167 ceph-mon[33178]: 5.13 scrub ok 2024-09-13T13:18:04.564 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:04 smithi167 ceph-mon[33178]: 3.a scrub starts 2024-09-13T13:18:04.564 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:04 smithi167 ceph-mon[33178]: 3.a scrub ok 2024-09-13T13:18:04.564 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:04 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:18:04.565 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:04 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:18:04.565 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:04 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4157828514' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:04.609 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:04.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:04.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:04.711 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:04.751 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:04.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:04.784 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:18:04.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:18:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:18:04.853 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:04.872 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 4 16 11968 11952 186.715 168.875 0.00437448 0.00534676 2024-09-13T13:18:04.897 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:04.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:05.000 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:05.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:05.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:05.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:05.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:05.159 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:05.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:05.261 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:05.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:05.292 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:05.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:05.393 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:05.418 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:05 smithi052 ceph-mon[29471]: pgmap v854: 158 pgs: 158 active+clean; 270 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 652 B/s rd, 34 MiB/s wr, 549 op/s 2024-09-13T13:18:05.418 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:05 smithi052 ceph-mon[29471]: 4.6 scrub starts 2024-09-13T13:18:05.418 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:05.418 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:05 smithi052 ceph-mon[29471]: 4.6 scrub ok 2024-09-13T13:18:05.418 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:05 smithi052 ceph-mon[36682]: pgmap v854: 158 pgs: 158 active+clean; 270 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 652 B/s rd, 34 MiB/s wr, 549 op/s 2024-09-13T13:18:05.418 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:05 smithi052 ceph-mon[36682]: 4.6 scrub starts 2024-09-13T13:18:05.418 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:05.419 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:05 smithi052 ceph-mon[36682]: 4.6 scrub ok 2024-09-13T13:18:05.421 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:05.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:05.523 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:05.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:05.553 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:05.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:05 smithi167 ceph-mon[33178]: pgmap v854: 158 pgs: 158 active+clean; 270 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 652 B/s rd, 34 MiB/s wr, 549 op/s 2024-09-13T13:18:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:05 smithi167 ceph-mon[33178]: 4.6 scrub starts 2024-09-13T13:18:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:05 smithi167 ceph-mon[33178]: 4.6 scrub ok 2024-09-13T13:18:05.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:05.682 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:05.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:05.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:05.784 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:05.825 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:05.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:05.872 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 5 16 14475 14459 180.707 156.688 0.00521207 0.00552343 2024-09-13T13:18:05.906 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:18:05.906 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:18:05.927 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:05.957 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:05.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:05.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:06.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:06.089 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:06.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:06.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:06.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:06.221 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:06.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:06.326 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:06.356 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:06.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:06.458 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:06.487 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:06.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:06.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:06.488 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:18:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:18:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:18:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:06 smithi052 ceph-mon[36682]: 4.f scrub starts 2024-09-13T13:18:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:06 smithi052 ceph-mon[36682]: 4.f scrub ok 2024-09-13T13:18:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:06 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:18:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:06 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:18:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:06 smithi052 ceph-mon[36682]: 5.16 scrub starts 2024-09-13T13:18:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:06 smithi052 ceph-mon[36682]: 5.16 scrub ok 2024-09-13T13:18:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:06 smithi052 ceph-mon[29471]: 4.f scrub starts 2024-09-13T13:18:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:06 smithi052 ceph-mon[29471]: 4.f scrub ok 2024-09-13T13:18:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:06 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:18:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:06 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:18:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:06 smithi052 ceph-mon[29471]: 5.16 scrub starts 2024-09-13T13:18:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:06 smithi052 ceph-mon[29471]: 5.16 scrub ok 2024-09-13T13:18:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:06 smithi167 ceph-mon[33178]: 4.f scrub starts 2024-09-13T13:18:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:06 smithi167 ceph-mon[33178]: 4.f scrub ok 2024-09-13T13:18:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:06 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:18:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:06 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:18:06.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:06 smithi167 ceph-mon[33178]: 5.16 scrub starts 2024-09-13T13:18:06.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:06 smithi167 ceph-mon[33178]: 5.16 scrub ok 2024-09-13T13:18:06.589 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:06.622 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:06.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:06.724 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:06.765 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:06.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:06.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:06.867 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:06.872 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 6 16 16953 16937 176.4 154.875 0.00290367 0.0056602 2024-09-13T13:18:06.905 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:06.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:07.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:07.033 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:07.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:07.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:07.035 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:07.135 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:07.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:07.168 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:07.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:07 smithi052 ceph-mon[36682]: pgmap v855: 158 pgs: 158 active+clean; 430 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 860 op/s 2024-09-13T13:18:07.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:07 smithi052 ceph-mon[36682]: osdmap e428: 8 total, 8 up, 8 in 2024-09-13T13:18:07.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:07 smithi052 ceph-mon[29471]: pgmap v855: 158 pgs: 158 active+clean; 430 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 860 op/s 2024-09-13T13:18:07.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:07 smithi052 ceph-mon[29471]: osdmap e428: 8 total, 8 up, 8 in 2024-09-13T13:18:07.269 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:07.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:07.304 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:07.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:07.405 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:07.441 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:07.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:07.543 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:07.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:07 smithi167 ceph-mon[33178]: pgmap v855: 158 pgs: 158 active+clean; 430 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 860 op/s 2024-09-13T13:18:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:07 smithi167 ceph-mon[33178]: osdmap e428: 8 total, 8 up, 8 in 2024-09-13T13:18:07.574 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:07.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:07.676 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:07.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:07.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:07.739 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T13:18:07.832 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:07.873 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 7 16 19488 19472 173.819 158.438 0.00970649 0.00574528 2024-09-13T13:18:07.879 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:07.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:07.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:07.982 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:08.003 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:08.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:08.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:08.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:08.140 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:08.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:08.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:08.194 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:08.206 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:08 smithi052 ceph-mon[29471]: 2.2 scrub starts 2024-09-13T13:18:08.206 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:08 smithi052 ceph-mon[29471]: 2.2 scrub ok 2024-09-13T13:18:08.242 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:08.308 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:08.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:08.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:08.410 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:08.453 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:08.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:08.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:08 smithi052 ceph-mon[36682]: 2.2 scrub starts 2024-09-13T13:18:08.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:08 smithi052 ceph-mon[36682]: 2.2 scrub ok 2024-09-13T13:18:08.555 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:08.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:08 smithi167 ceph-mon[33178]: 2.2 scrub starts 2024-09-13T13:18:08.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:08 smithi167 ceph-mon[33178]: 2.2 scrub ok 2024-09-13T13:18:08.579 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:08.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:08.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:08.681 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:08.711 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:08.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:08.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:08.799 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:08.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:08.842 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:08.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:08.873 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 8 16 21465 21449 167.537 123.562 0.0364342 0.00595395 2024-09-13T13:18:08.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:08.972 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:08.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:09.039 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:18:09.074 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:09.100 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:09.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:09.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:09.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:09.230 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:09.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:09.232 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T13:18:09.332 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:09.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[29471]: 6.12 scrub starts 2024-09-13T13:18:09.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[29471]: 6.12 scrub ok 2024-09-13T13:18:09.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[29471]: pgmap v857: 158 pgs: 158 active+clean; 899 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-13T13:18:09.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[29471]: 6.7 scrub starts 2024-09-13T13:18:09.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[29471]: 6.7 scrub ok 2024-09-13T13:18:09.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1398410589' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:18:09.343 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[29471]: from='client.26533 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:18:09.343 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[36682]: 6.12 scrub starts 2024-09-13T13:18:09.343 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[36682]: 6.12 scrub ok 2024-09-13T13:18:09.343 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[36682]: pgmap v857: 158 pgs: 158 active+clean; 899 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-13T13:18:09.343 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[36682]: 6.7 scrub starts 2024-09-13T13:18:09.343 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[36682]: 6.7 scrub ok 2024-09-13T13:18:09.343 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1398410589' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:18:09.343 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:09 smithi052 ceph-mon[36682]: from='client.26533 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:18:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:09.380 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:09.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:09.397 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:18:09.397 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:18:09.481 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:09.509 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:09.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:09.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:09.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:09 smithi167 ceph-mon[33178]: 6.12 scrub starts 2024-09-13T13:18:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:09 smithi167 ceph-mon[33178]: 6.12 scrub ok 2024-09-13T13:18:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:09 smithi167 ceph-mon[33178]: pgmap v857: 158 pgs: 158 active+clean; 899 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-13T13:18:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:09 smithi167 ceph-mon[33178]: 6.7 scrub starts 2024-09-13T13:18:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:09 smithi167 ceph-mon[33178]: 6.7 scrub ok 2024-09-13T13:18:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:09 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1398410589' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:18:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:09 smithi167 ceph-mon[33178]: from='client.26533 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:18:09.612 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:09.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:09.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:09.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:09.745 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:09.786 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:09.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:09.873 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 9 16 23352 23336 162.025 117.938 0.0026774 0.00614204 2024-09-13T13:18:09.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:09.916 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:09.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:09.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:10.018 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:10.051 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:10.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:10.153 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:10.175 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:10.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:10.228 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:18:10.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:10.307 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:10.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:10.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:10.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:10.439 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:10.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[36682]: 4.18 scrub starts 2024-09-13T13:18:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[36682]: 4.18 scrub ok 2024-09-13T13:18:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:18:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:18:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[36682]: from='client.26533 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:18:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[36682]: osdmap e429: 8 total, 8 up, 8 in 2024-09-13T13:18:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/710472082' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:18:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[29471]: 4.18 scrub starts 2024-09-13T13:18:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[29471]: 4.18 scrub ok 2024-09-13T13:18:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:18:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:18:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[29471]: from='client.26533 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:18:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[29471]: osdmap e429: 8 total, 8 up, 8 in 2024-09-13T13:18:10.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:10 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/710472082' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:18:10.541 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:10.562 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:10.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:10.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:10 smithi167 ceph-mon[33178]: 4.18 scrub starts 2024-09-13T13:18:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:10 smithi167 ceph-mon[33178]: 4.18 scrub ok 2024-09-13T13:18:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:10 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:18:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:10 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:18:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:10 smithi167 ceph-mon[33178]: from='client.26533 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:18:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:10 smithi167 ceph-mon[33178]: osdmap e429: 8 total, 8 up, 8 in 2024-09-13T13:18:10.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:10 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/710472082' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:18:10.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:10.706 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:10.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:10.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:10.808 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:10.848 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:10.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:10.873 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 10 16 25354 25338 158.334 125.125 0.00186244 0.00629484 2024-09-13T13:18:10.950 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:10.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:10.983 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:10.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:11.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:11.113 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:11.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:11.215 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:11.243 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:11.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:11.345 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:11.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:11.387 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:11.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:11.488 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:11 smithi052 ceph-mon[36682]: 4.e scrub starts 2024-09-13T13:18:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:11 smithi052 ceph-mon[36682]: 4.e scrub ok 2024-09-13T13:18:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:11 smithi052 ceph-mon[36682]: pgmap v859: 158 pgs: 158 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-13T13:18:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:11 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:18:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/710472082' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:18:11.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:11 smithi052 ceph-mon[36682]: osdmap e430: 8 total, 8 up, 8 in 2024-09-13T13:18:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:11 smithi052 ceph-mon[29471]: 4.e scrub starts 2024-09-13T13:18:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:11 smithi052 ceph-mon[29471]: 4.e scrub ok 2024-09-13T13:18:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:11 smithi052 ceph-mon[29471]: pgmap v859: 158 pgs: 158 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-13T13:18:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:11 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:18:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/710472082' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:18:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:11 smithi052 ceph-mon[29471]: osdmap e430: 8 total, 8 up, 8 in 2024-09-13T13:18:11.510 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:11.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:11 smithi167 ceph-mon[33178]: 4.e scrub starts 2024-09-13T13:18:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:11 smithi167 ceph-mon[33178]: 4.e scrub ok 2024-09-13T13:18:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:11 smithi167 ceph-mon[33178]: pgmap v859: 158 pgs: 158 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-13T13:18:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:11 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:18:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/710472082' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:18:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:11 smithi167 ceph-mon[33178]: osdmap e430: 8 total, 8 up, 8 in 2024-09-13T13:18:11.612 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:11.657 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:11.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:11.758 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:11.779 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:11.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:11.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:11.873 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 11 16 27243 27227 154.672 118.062 0.00934981 0.00645936 2024-09-13T13:18:11.881 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:11.919 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:11.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:12.022 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:12.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:12.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:12.152 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:12.190 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:12.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:12.292 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:12.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:12.329 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:12.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:12.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:12.460 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:12.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:12.561 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:12.603 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:12.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:12.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:12.705 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:12.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:12.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:12.837 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:12.871 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:12.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:12.873 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 12 16 29347 29331 152.74 131.5 0.00669311 0.00654135 2024-09-13T13:18:12.973 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:13.014 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:13.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:13.116 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:13.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:13.149 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:13.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:13.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:13.272 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:13.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:13.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:13.374 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:13.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:13.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:13.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:13.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:13.524 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:13.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:13 smithi167 ceph-mon[33178]: pgmap v861: 158 pgs: 158 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 154 MiB/s wr, 2.46k op/s 2024-09-13T13:18:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:13 smithi167 ceph-mon[33178]: osdmap e431: 8 total, 8 up, 8 in 2024-09-13T13:18:13.626 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:13.665 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:13.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:13.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:13 smithi052 ceph-mon[36682]: pgmap v861: 158 pgs: 158 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 154 MiB/s wr, 2.46k op/s 2024-09-13T13:18:13.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:13 smithi052 ceph-mon[36682]: osdmap e431: 8 total, 8 up, 8 in 2024-09-13T13:18:13.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:13 smithi052 ceph-mon[29471]: pgmap v861: 158 pgs: 158 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 154 MiB/s wr, 2.46k op/s 2024-09-13T13:18:13.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:13 smithi052 ceph-mon[29471]: osdmap e431: 8 total, 8 up, 8 in 2024-09-13T13:18:13.767 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:13.796 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:13.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:13.873 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 13 15 30847 30832 148.207 93.8125 0.00672407 0.00672585 2024-09-13T13:18:13.898 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:13.919 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:13.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:13.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:14.021 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:14.061 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:14.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:14.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:14.163 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:14.197 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:14.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:14.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:14.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:14.332 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:14.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:14.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:14.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:14.468 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:14.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:14.569 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:14.591 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:14.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:14.693 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:14.732 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:14.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:14.834 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:14.863 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:14.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:14.873 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 14 16 32529 32513 145.125 105.062 0.00454773 0.00688048 2024-09-13T13:18:14.965 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:15.007 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:15.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:15.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:15.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:18:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:18:15.109 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:15.131 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:15.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:15.233 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:15.272 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:15.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:15.373 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:15.404 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:15.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:15.505 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:15.544 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:15.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:15 smithi167 ceph-mon[33178]: pgmap v863: 158 pgs: 158 active+clean; 1.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 133 MiB/s wr, 2.12k op/s 2024-09-13T13:18:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:15.613 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:15 smithi052 ceph-mon[36682]: pgmap v863: 158 pgs: 158 active+clean; 1.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 133 MiB/s wr, 2.12k op/s 2024-09-13T13:18:15.613 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:15.613 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:15 smithi052 ceph-mon[29471]: pgmap v863: 158 pgs: 158 active+clean; 1.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 133 MiB/s wr, 2.12k op/s 2024-09-13T13:18:15.614 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:15.646 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:15.674 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:15.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:15.775 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:15.798 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:15.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:15.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:15.874 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 15 16 34008 33992 141.612 92.4375 0.00219261 0.00698065 2024-09-13T13:18:15.901 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:15.931 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:15.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:15.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:16.033 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:16.078 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:16.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:16.179 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:16.209 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:16.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:16.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:16.311 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:16.336 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:16.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:16.437 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:16.471 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:16.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:16.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:16.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:18:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:18:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:18:16.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:16.601 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:16.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:16.703 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:16.731 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T13:18:16.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:16.732 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:16.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:16.833 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:16.875 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 16 16 35706 35690 139.393 106.125 0.0110745 0.00716736 2024-09-13T13:18:16.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:16.913 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:16.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:17.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:17.042 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:17.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:17.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:17.144 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:17.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:17.192 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:17.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:17.293 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:17.336 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:17.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:17.438 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:17.463 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:17.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:17.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:17 smithi052 ceph-mon[36682]: pgmap v864: 158 pgs: 158 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 759 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T13:18:17.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:17 smithi052 ceph-mon[29471]: pgmap v864: 158 pgs: 158 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 759 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T13:18:17.565 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:17 smithi167 ceph-mon[33178]: pgmap v864: 158 pgs: 158 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 759 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-13T13:18:17.592 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:17.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:17.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:17.693 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:17.725 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:17.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:17.827 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:17.858 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:17.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:17.874 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 17 16 37580 37564 138.082 117.125 0.00787452 0.00723628 2024-09-13T13:18:17.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:17.992 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:17.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:17.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:18.082 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T13:18:18.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:18.120 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:18.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:18.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:18.128 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:18.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:18.248 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:18.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:18.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:18.379 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:18.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:18.481 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:18.512 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:18.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:18.613 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:18.660 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:18.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:18.763 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:18.810 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:18.811 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:18.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:18.854 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:18:18.854 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:18:18.874 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 18 15 39027 39012 135.438 90.5 0.0084623 0.00737904 2024-09-13T13:18:18.887 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:18:18.887 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:18:18.911 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:18.923 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:18:18.923 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:18:18.924 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:18.945 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:18.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:18.958 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:18:18.958 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:18:18.994 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:18:18.994 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:18:19.030 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:18:19.030 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:18:19.047 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:19.066 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:18:19.067 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:18:19.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:19.078 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:19.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:19.102 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:18:19.102 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:18:19.179 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:19.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:19.222 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:19.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:19 smithi052 ceph-mon[36682]: pgmap v865: 158 pgs: 158 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-09-13T13:18:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:19 smithi052 ceph-mon[36682]: osdmap e432: 8 total, 8 up, 8 in 2024-09-13T13:18:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:19 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1276204801' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:19 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:18:19.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:19 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:18:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:19 smithi052 ceph-mon[29471]: pgmap v865: 158 pgs: 158 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-09-13T13:18:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:19 smithi052 ceph-mon[29471]: osdmap e432: 8 total, 8 up, 8 in 2024-09-13T13:18:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:19 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1276204801' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:19 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:18:19.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:19 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:18:19.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:19 smithi167 ceph-mon[33178]: pgmap v865: 158 pgs: 158 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-09-13T13:18:19.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:19 smithi167 ceph-mon[33178]: osdmap e432: 8 total, 8 up, 8 in 2024-09-13T13:18:19.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:19 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1276204801' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:19.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:19 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:18:19.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:19 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:18:19.323 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:19.362 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:19.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:19.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:19.493 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:19.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:19.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:19.595 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:19.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:18:19.621 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:19.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:19.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:19.723 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:19.748 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:19.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:19.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:19.850 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:19.875 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 19 16 40646 40630 133.628 101.125 0.0142322 0.00747691 2024-09-13T13:18:19.895 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:19.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:19.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:19.997 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:20.026 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:20.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:20.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:20.127 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:20.171 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:20.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:20.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:20 smithi052 ceph-mon[36682]: 1.a scrub starts 2024-09-13T13:18:20.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:20 smithi052 ceph-mon[36682]: 1.0 scrub starts 2024-09-13T13:18:20.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:20 smithi052 ceph-mon[29471]: 1.a scrub starts 2024-09-13T13:18:20.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:20 smithi052 ceph-mon[29471]: 1.0 scrub starts 2024-09-13T13:18:20.274 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:20.318 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:20.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:20.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:20.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:20 smithi167 ceph-mon[33178]: 1.a scrub starts 2024-09-13T13:18:20.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:20 smithi167 ceph-mon[33178]: 1.0 scrub starts 2024-09-13T13:18:20.420 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:20.462 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:20.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:20.564 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:20.611 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:20.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:20.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:20.713 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:20.752 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:20.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:20.854 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:20.877 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:18:20.876943+0000 min lat: 0.00125067 max lat: 0.222583 avg lat: 0.00758145 2024-09-13T13:18:20.877 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:18:20.877 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 20 16 42204 42188 131.802 97.375 0.00778839 0.00758145 2024-09-13T13:18:20.893 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:20.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:20.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:20.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:21.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:21.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:21.072 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T13:18:21.131 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:21.176 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:21.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:21.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:21.231 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:21 smithi052 ceph-mon[36682]: pgmap v867: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-13T13:18:21.231 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:21 smithi052 ceph-mon[36682]: 1.3 scrub starts 2024-09-13T13:18:21.231 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:21 smithi052 ceph-mon[36682]: 1.7 scrub starts 2024-09-13T13:18:21.232 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:21 smithi052 ceph-mon[36682]: 1.1 scrub starts 2024-09-13T13:18:21.232 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:21 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1201716789' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:18:21.232 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:21 smithi052 ceph-mon[36682]: 1.2 scrub starts 2024-09-13T13:18:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:21 smithi052 ceph-mon[29471]: pgmap v867: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-13T13:18:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:21 smithi052 ceph-mon[29471]: 1.3 scrub starts 2024-09-13T13:18:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:21 smithi052 ceph-mon[29471]: 1.7 scrub starts 2024-09-13T13:18:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:21 smithi052 ceph-mon[29471]: 1.1 scrub starts 2024-09-13T13:18:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:21 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1201716789' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:18:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:21 smithi052 ceph-mon[29471]: 1.2 scrub starts 2024-09-13T13:18:21.233 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:18:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:18:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:42648 11.a 11:507663bf:::benchmark_data_smithi052_182575_object42647:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-13T13:18:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:18:21.224599+0000", 2024-09-13T13:18:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0053843220000000004, 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0054162899999999998, 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 42648 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.224599+0000", 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:18:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.224599+0000", 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.224601+0000", 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.804e-06 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.224621+0000", 2024-09-13T13:18:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9527000000000001e-05 2024-09-13T13:18:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:18:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.224632+0000", 2024-09-13T13:18:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1479e-05 2024-09-13T13:18:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:18:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.224638+0000", 2024-09-13T13:18:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5160000000000002e-06 2024-09-13T13:18:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:18:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.224723+0000", 2024-09-13T13:18:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5605000000000005e-05 2024-09-13T13:18:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:18:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.224884+0000", 2024-09-13T13:18:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016078800000000001 2024-09-13T13:18:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.225119+0000", 2024-09-13T13:18:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00023455000000000001 2024-09-13T13:18:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:18:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.227729+0000", 2024-09-13T13:18:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026102230000000001 2024-09-13T13:18:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.229739+0000", 2024-09-13T13:18:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002010408 2024-09-13T13:18:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:18:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:18:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:18:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:42653 11.8 11:1ccbf112:::benchmark_data_smithi052_182575_object42652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-13T13:18:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:18:21.226766+0000", 2024-09-13T13:18:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0032181459999999999, 2024-09-13T13:18:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033323839999999999, 2024-09-13T13:18:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:18:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:18:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:18:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:18:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:18:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:18:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 42653 2024-09-13T13:18:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:18:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:18:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.226766+0000", 2024-09-13T13:18:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:18:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:18:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.226766+0000", 2024-09-13T13:18:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:18:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:18:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.226767+0000", 2024-09-13T13:18:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6199999999999999e-06 2024-09-13T13:18:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:18:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.226790+0000", 2024-09-13T13:18:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3153999999999999e-05 2024-09-13T13:18:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:18:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.226801+0000", 2024-09-13T13:18:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0678e-05 2024-09-13T13:18:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:18:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.226808+0000", 2024-09-13T13:18:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9800000000000001e-06 2024-09-13T13:18:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:18:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.226892+0000", 2024-09-13T13:18:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4101e-05 2024-09-13T13:18:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:18:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.227039+0000", 2024-09-13T13:18:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014696300000000001 2024-09-13T13:18:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.227165+0000", 2024-09-13T13:18:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012577899999999999 2024-09-13T13:18:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:18:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.229398+0000", 2024-09-13T13:18:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0022334519999999999 2024-09-13T13:18:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:18:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:18:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:18:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.36635.0:42651 11.1 e433/423 11:83907fc1:::benchmark_data_smithi052_182575_object42650:head v 433'2622, mlcod=433'2620)", 2024-09-13T13:18:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:18:21.228254+0000", 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.001729789, 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001945238, 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.228254+0000", 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.228254+0000", 2024-09-13T13:18:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.228255+0000", 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0669999999999999e-06 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.228290+0000", 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4700999999999999e-05 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:18:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.228302+0000", 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2479e-05 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.228308+0000", 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1850000000000001e-06 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.228379+0000", 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0956000000000003e-05 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:18:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.228397+0000", 2024-09-13T13:18:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7812000000000002e-05 2024-09-13T13:18:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:18:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:18:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:18:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:42658 11.b 11:d03e0b55:::benchmark_data_smithi052_182575_object42657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-13T13:18:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:18:21.229217+0000", 2024-09-13T13:18:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.00076698800000000002, 2024-09-13T13:18:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001038485, 2024-09-13T13:18:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:18:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:18:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:18:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:18:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:18:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:18:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 42658 2024-09-13T13:18:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:18:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:18:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.229217+0000", 2024-09-13T13:18:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:18:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:18:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.229217+0000", 2024-09-13T13:18:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:18:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:18:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.229219+0000", 2024-09-13T13:18:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.75e-06 2024-09-13T13:18:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:18:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.229241+0000", 2024-09-13T13:18:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2246999999999999e-05 2024-09-13T13:18:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:18:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.229251+0000", 2024-09-13T13:18:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0677e-05 2024-09-13T13:18:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:18:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.229258+0000", 2024-09-13T13:18:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7649999999999996e-06 2024-09-13T13:18:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:18:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.229315+0000", 2024-09-13T13:18:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6645999999999999e-05 2024-09-13T13:18:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:18:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.229512+0000", 2024-09-13T13:18:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000196608 2024-09-13T13:18:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:18:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:18:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:18:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:21.229625+0000", 2024-09-13T13:18:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011359299999999999 2024-09-13T13:18:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:18:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:18:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:18:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:18:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:18:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 4 2024-09-13T13:18:21.254 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:18:21.278 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:21.314 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:21.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:21.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:21.436 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:21 smithi167 ceph-mon[33178]: pgmap v867: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-13T13:18:21.436 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:21 smithi167 ceph-mon[33178]: 1.3 scrub starts 2024-09-13T13:18:21.436 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:21 smithi167 ceph-mon[33178]: 1.7 scrub starts 2024-09-13T13:18:21.436 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:21 smithi167 ceph-mon[33178]: 1.1 scrub starts 2024-09-13T13:18:21.436 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:21 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1201716789' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:18:21.436 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:21 smithi167 ceph-mon[33178]: 1.2 scrub starts 2024-09-13T13:18:21.449 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:21.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:21.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:21.551 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:21.590 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:21.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:21.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:21.691 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:21.738 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:21.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:21.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:21.840 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:21.877 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 21 16 43771 43755 130.189 97.9375 0.0100172 0.00767595 2024-09-13T13:18:21.879 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:21.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:21.982 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:22.024 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:22.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:22.034 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T13:18:22.126 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:22.173 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:22.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:22.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[29471]: 1.5 scrub starts 2024-09-13T13:18:22.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:18:22.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:18:22.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[29471]: 1.9 scrub starts 2024-09-13T13:18:22.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1201716789' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:18:22.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[29471]: osdmap e433: 8 total, 8 up, 8 in 2024-09-13T13:18:22.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[29471]: 1.6 scrub starts 2024-09-13T13:18:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[36682]: 1.5 scrub starts 2024-09-13T13:18:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:18:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:18:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[36682]: 1.9 scrub starts 2024-09-13T13:18:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1201716789' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:18:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[36682]: osdmap e433: 8 total, 8 up, 8 in 2024-09-13T13:18:22.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:22 smithi052 ceph-mon[36682]: 1.6 scrub starts 2024-09-13T13:18:22.240 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:22.274 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:22.300 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:22.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:22.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:22.438 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:22.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:22.534 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:18:22.538 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:22.567 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:22.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:22.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:22 smithi167 ceph-mon[33178]: 1.5 scrub starts 2024-09-13T13:18:22.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:22 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:18:22.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:22 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:18:22.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:22 smithi167 ceph-mon[33178]: 1.9 scrub starts 2024-09-13T13:18:22.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:22 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1201716789' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:18:22.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:22 smithi167 ceph-mon[33178]: osdmap e433: 8 total, 8 up, 8 in 2024-09-13T13:18:22.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:22 smithi167 ceph-mon[33178]: 1.6 scrub starts 2024-09-13T13:18:22.669 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:22.698 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:22.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:22.800 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:22.829 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:22.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:22.877 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 22 16 45263 45247 128.51 93.25 0.00712767 0.00777587 2024-09-13T13:18:22.931 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:22.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:22.960 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:22.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:23.016 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T13:18:23.062 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:23.092 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:23.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:23.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:23.194 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:23.224 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:23.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:23.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:23 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:18:23.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:23 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:18:23.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:23 smithi052 ceph-mon[29471]: 1.4 scrub starts 2024-09-13T13:18:23.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:23 smithi052 ceph-mon[29471]: pgmap v869: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-13T13:18:23.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:23 smithi052 ceph-mon[29471]: osdmap e434: 8 total, 8 up, 8 in 2024-09-13T13:18:23.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:23 smithi052 ceph-mon[29471]: osdmap e435: 8 total, 8 up, 8 in 2024-09-13T13:18:23.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:23 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:18:23.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:23 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:18:23.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:23 smithi052 ceph-mon[36682]: 1.4 scrub starts 2024-09-13T13:18:23.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:23 smithi052 ceph-mon[36682]: pgmap v869: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-13T13:18:23.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:23 smithi052 ceph-mon[36682]: osdmap e434: 8 total, 8 up, 8 in 2024-09-13T13:18:23.240 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:23 smithi052 ceph-mon[36682]: osdmap e435: 8 total, 8 up, 8 in 2024-09-13T13:18:23.325 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:23.381 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:23.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:23.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:23.483 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:23.540 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:23.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:23.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:23 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:18:23.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:23 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:18:23.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:23 smithi167 ceph-mon[33178]: 1.4 scrub starts 2024-09-13T13:18:23.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:23 smithi167 ceph-mon[33178]: pgmap v869: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-13T13:18:23.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:23 smithi167 ceph-mon[33178]: osdmap e434: 8 total, 8 up, 8 in 2024-09-13T13:18:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:23 smithi167 ceph-mon[33178]: osdmap e435: 8 total, 8 up, 8 in 2024-09-13T13:18:23.582 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:23.587 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:23.598 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:23.643 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:23.687 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:23.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:23.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:23.789 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:23.823 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:23.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:23.877 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 23 16 46622 46606 126.615 84.9375 0.0163542 0.00789183 2024-09-13T13:18:23.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:23.953 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:23.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:24.056 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:24.100 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:24.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:24.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:24.126 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:18:24.202 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:24.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:24.231 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:24.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:24.332 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:24.370 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:24.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:24.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:24.387 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:24 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1585626083' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:24.387 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:24 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1585626083' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:24.473 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:24.499 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:24.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:24.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:24.560 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:18:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:18:24.560 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:18:24.561 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:18:24.561 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:18:24.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:24 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1585626083' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:24.599 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:18:24.599 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:18:24.599 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:18:24.600 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:18:24.600 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:18:24.600 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:18:24.600 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:18:24.600 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:18:24.601 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:24.621 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:18:24.621 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:18:24.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:24.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:24.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:24.654 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:18:24.654 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:18:24.692 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:18:24.692 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:18:24.725 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:18:24.726 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:18:24.731 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:24.760 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:24.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:24.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:24.863 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:24.877 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 24 16 48081 48065 125.139 91.1875 0.00322756 0.00798523 2024-09-13T13:18:24.894 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:24.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:24.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:24.996 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:25.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:25.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:18:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:18:25.141 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:25.181 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:25.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:25.284 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:25.324 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:25.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:25.426 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:25.463 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:25.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:25.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:18:25.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:18:25.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[29471]: pgmap v872: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 862 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-13T13:18:25.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:18:25.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1585626083' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:18:25.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[29471]: osdmap e436: 8 total, 8 up, 8 in 2024-09-13T13:18:25.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3655220980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:25.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:25.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:18:25.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:18:25.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[36682]: pgmap v872: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 862 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-13T13:18:25.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:18:25.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1585626083' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:18:25.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[36682]: osdmap e436: 8 total, 8 up, 8 in 2024-09-13T13:18:25.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3655220980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:25.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:25.566 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:25.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:25 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:18:25.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:25 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:18:25.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:25 smithi167 ceph-mon[33178]: pgmap v872: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 862 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-13T13:18:25.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:25 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:18:25.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:25 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1585626083' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:18:25.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:25 smithi167 ceph-mon[33178]: osdmap e436: 8 total, 8 up, 8 in 2024-09-13T13:18:25.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:25 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3655220980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:25.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:25.598 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:25.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:25.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:25.700 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:25.741 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:25.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:25.843 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:25.873 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:25.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:25.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:25.877 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 25 15 49758 49743 124.328 104.875 0.00658136 0.00803822 2024-09-13T13:18:25.974 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:26.013 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:26.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:26.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:26.115 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:26.162 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:26.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:26.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:18:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:18:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:18:26.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:26.289 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:26.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:26.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:26.392 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:26.422 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:26.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:26.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:26.524 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:26.556 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:26.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:26.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:26.687 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:26.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:26.788 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:26.829 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:26.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:26.877 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 26 16 51391 51375 123.469 102 0.0182315 0.00809369 2024-09-13T13:18:26.932 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:26.954 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:26.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:26.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:27.056 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:27.058 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T13:18:27.093 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:27.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:27.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:27.195 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:27.251 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:27.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:27.322 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:27 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:18:27.322 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:27 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:18:27.323 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:27 smithi052 ceph-mon[36682]: pgmap v874: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-13T13:18:27.323 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:27 smithi052 ceph-mon[36682]: osdmap e437: 8 total, 8 up, 8 in 2024-09-13T13:18:27.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:27 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:18:27.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:27 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:18:27.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:27 smithi052 ceph-mon[29471]: pgmap v874: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-13T13:18:27.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:27 smithi052 ceph-mon[29471]: osdmap e437: 8 total, 8 up, 8 in 2024-09-13T13:18:27.353 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:27.380 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:27.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:27.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:27.482 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:27.534 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:27.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:27.558 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:27.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:27 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:18:27.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:27 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:18:27.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:27 smithi167 ceph-mon[33178]: pgmap v874: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-13T13:18:27.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:27 smithi167 ceph-mon[33178]: osdmap e437: 8 total, 8 up, 8 in 2024-09-13T13:18:27.636 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:27.662 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:18:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:27.667 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:27.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:27.768 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:27.825 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:27.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:27.878 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 27 16 52709 52693 121.944 82.375 0.010065 0.00819428 2024-09-13T13:18:27.928 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:27.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:27.973 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:27.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:28.075 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:28.103 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:28.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:28.160 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T13:18:28.205 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:28.243 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:28.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:28.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:28.345 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:28.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:28.387 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:28.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:28.405 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:18:28.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:28 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2420228368' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:18:28.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:28 smithi052 ceph-mon[29471]: from='client.36674 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:18:28.469 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:28 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2420228368' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:18:28.470 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:28 smithi052 ceph-mon[36682]: from='client.36674 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:18:28.489 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:28.538 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:28.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:28.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:28 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2420228368' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:18:28.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:28 smithi167 ceph-mon[33178]: from='client.36674 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:18:28.640 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:28.664 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:28.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:28.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:28.766 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:28.833 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:18:28.833 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:18:28.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:28.834 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:28.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:28.878 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 28 15 54102 54087 120.701 87.125 0.0124951 0.00828001 2024-09-13T13:18:28.935 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:28.959 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:28.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:28.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:29.061 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:29.097 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:29.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:29.199 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:29.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:29.267 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:29.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:29.367 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:29.404 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:29.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:29.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:29.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:29 smithi052 ceph-mon[29471]: pgmap v876: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 871 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-13T13:18:29.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:29 smithi052 ceph-mon[29471]: from='client.36674 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:18:29.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:29 smithi052 ceph-mon[29471]: osdmap e438: 8 total, 8 up, 8 in 2024-09-13T13:18:29.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:29 smithi052 ceph-mon[36682]: pgmap v876: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 871 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-13T13:18:29.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:29 smithi052 ceph-mon[36682]: from='client.36674 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:18:29.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:29 smithi052 ceph-mon[36682]: osdmap e438: 8 total, 8 up, 8 in 2024-09-13T13:18:29.506 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:29.552 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:29.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:29.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:29 smithi167 ceph-mon[33178]: pgmap v876: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 871 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-13T13:18:29.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:29 smithi167 ceph-mon[33178]: from='client.36674 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:18:29.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:29 smithi167 ceph-mon[33178]: osdmap e438: 8 total, 8 up, 8 in 2024-09-13T13:18:29.653 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:29.691 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:29.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:29.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:29.793 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:29.835 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:29.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:29.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:29.878 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 29 16 55613 55597 119.793 94.375 0.00318759 0.00834236 2024-09-13T13:18:29.938 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:29.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:29.967 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:29.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:30.068 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:30.091 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T13:18:30.098 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:30.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:30.199 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:30.255 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:30.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:30.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:30.357 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:30.389 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:30.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:30.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:30.517 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:30.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:30.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:30.644 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:30.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:30.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:18:30.746 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:30.762 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:30.779 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:30.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:30.817 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:30.881 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:30.882 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 30 16 57044 57028 118.776 89.4375 0.0215696 0.0084133 2024-09-13T13:18:30.924 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:30.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:30.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:31.026 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:31.059 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:31.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:31.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:31.101 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:18:31.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:31.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:31.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:31.302 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:31.328 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:31.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:31.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:31.425 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:31 smithi167 ceph-mon[33178]: pgmap v878: 158 pgs: 158 active+clean; 3.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-13T13:18:31.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:31.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:31 smithi052 ceph-mon[29471]: pgmap v878: 158 pgs: 158 active+clean; 3.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-13T13:18:31.438 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:31 smithi052 ceph-mon[36682]: pgmap v878: 158 pgs: 158 active+clean; 3.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-13T13:18:31.445 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:31.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:31.456 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:31.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:31.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:31.587 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:31.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:31.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:31.688 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:31.697 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:18:31.697 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:18:31.738 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:31.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:31.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:31.868 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:31.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:31.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:31.879 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 31 16 58399 58383 117.677 84.6875 0.00424457 0.00849202 2024-09-13T13:18:31.970 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:32.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:32.011 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:32.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:32.113 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:32.153 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:32.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:32.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:32.168 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:18:32.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:32 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2852991880' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:18:32.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:32 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2852991880' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:18:32.255 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:32.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:32.301 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:32.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:32.374 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:18:32.374 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:18:32.403 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:32.408 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:18:32.408 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:18:32.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:32.444 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:32.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:32.454 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:18:32.454 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:18:32.491 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:18:32.491 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:18:32.525 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:18:32.525 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:18:32.545 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:32.559 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:18:32.560 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:18:32.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:32 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2852991880' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:18:32.574 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:32.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:32.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:32.595 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:18:32.596 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:18:32.631 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:18:32.632 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:18:32.676 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:32.699 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:32.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:32.800 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:32.832 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:32.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:32.880 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 32 16 59577 59561 116.3 73.625 0.0109256 0.00859291 2024-09-13T13:18:32.932 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:32.973 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:32.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:33.076 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:33.114 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:33.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:33.183 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T13:18:33.216 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:33.266 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:33.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:33.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:33.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:33.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[29471]: pgmap v879: 158 pgs: 158 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-13T13:18:33.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:18:33.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2852991880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:18:33.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[29471]: osdmap e439: 8 total, 8 up, 8 in 2024-09-13T13:18:33.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2220845388' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:33.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[29471]: 5.1c scrub starts 2024-09-13T13:18:33.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[29471]: 5.1c scrub ok 2024-09-13T13:18:33.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2477376785' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:18:33.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[29471]: 3.1f scrub starts 2024-09-13T13:18:33.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[29471]: 3.1f scrub ok 2024-09-13T13:18:33.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[36682]: pgmap v879: 158 pgs: 158 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-13T13:18:33.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:18:33.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2852991880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:18:33.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[36682]: osdmap e439: 8 total, 8 up, 8 in 2024-09-13T13:18:33.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2220845388' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:33.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[36682]: 5.1c scrub starts 2024-09-13T13:18:33.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[36682]: 5.1c scrub ok 2024-09-13T13:18:33.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2477376785' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:18:33.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[36682]: 3.1f scrub starts 2024-09-13T13:18:33.374 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:33 smithi052 ceph-mon[36682]: 3.1f scrub ok 2024-09-13T13:18:33.392 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:33.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:33.494 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:33.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:33.519 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:33.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:33.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:33 smithi167 ceph-mon[33178]: pgmap v879: 158 pgs: 158 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-13T13:18:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:33 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:18:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:33 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2852991880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:18:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:33 smithi167 ceph-mon[33178]: osdmap e439: 8 total, 8 up, 8 in 2024-09-13T13:18:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:33 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2220845388' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:33 smithi167 ceph-mon[33178]: 5.1c scrub starts 2024-09-13T13:18:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:33 smithi167 ceph-mon[33178]: 5.1c scrub ok 2024-09-13T13:18:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:33 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2477376785' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:18:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:33 smithi167 ceph-mon[33178]: 3.1f scrub starts 2024-09-13T13:18:33.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:33 smithi167 ceph-mon[33178]: 3.1f scrub ok 2024-09-13T13:18:33.620 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:33.641 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:33.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:33.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:33.743 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:33.765 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:33.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:33.866 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:33.879 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 33 16 60538 60522 114.596 60.0625 0.00735655 0.00872144 2024-09-13T13:18:33.904 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T13:18:33.923 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:33.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:33.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:34.025 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:34.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:34.057 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:34.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:34.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:34.184 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:34.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:34.286 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:34.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:34.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:34.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:34.460 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:34.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:34.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2477376785' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:18:34.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[36682]: osdmap e440: 8 total, 8 up, 8 in 2024-09-13T13:18:34.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[36682]: 4.3 scrub starts 2024-09-13T13:18:34.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]} : dispatch 2024-09-13T13:18:34.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-13T13:18:34.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]} : dispatch 2024-09-13T13:18:34.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-13T13:18:34.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[36682]: 4.11 scrub starts 2024-09-13T13:18:34.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[36682]: 4.11 scrub ok 2024-09-13T13:18:34.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:18:34.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2477376785' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:18:34.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[29471]: osdmap e440: 8 total, 8 up, 8 in 2024-09-13T13:18:34.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[29471]: 4.3 scrub starts 2024-09-13T13:18:34.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]} : dispatch 2024-09-13T13:18:34.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-13T13:18:34.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]} : dispatch 2024-09-13T13:18:34.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-13T13:18:34.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[29471]: 4.11 scrub starts 2024-09-13T13:18:34.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[29471]: 4.11 scrub ok 2024-09-13T13:18:34.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:34 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:18:34.562 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:34.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:34 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2477376785' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:18:34.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:34 smithi167 ceph-mon[33178]: osdmap e440: 8 total, 8 up, 8 in 2024-09-13T13:18:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:34 smithi167 ceph-mon[33178]: 4.3 scrub starts 2024-09-13T13:18:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:34 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]} : dispatch 2024-09-13T13:18:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:34 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-13T13:18:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:34 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]} : dispatch 2024-09-13T13:18:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:34 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-13T13:18:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:34 smithi167 ceph-mon[33178]: 4.11 scrub starts 2024-09-13T13:18:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:34 smithi167 ceph-mon[33178]: 4.11 scrub ok 2024-09-13T13:18:34.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:34 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:18:34.595 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:34.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:34.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T13:18:34.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:34.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:34.834 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:34.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:34.873 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:34.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:34.880 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 34 16 61806 61790 113.556 79.25 0.00685968 0.00880056 2024-09-13T13:18:34.974 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:34.995 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:18:35.012 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:35.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:35.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:18:34 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:18:35.077 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:18:35.114 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:35.166 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:35.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:35.267 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:35.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:35.313 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:35.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:35.414 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:35.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:35.458 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:35.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:35.458 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:35 smithi167 ceph-mon[33178]: 4.3 scrub ok 2024-09-13T13:18:35.458 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:35 smithi167 ceph-mon[33178]: 3.0 scrub starts 2024-09-13T13:18:35.458 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:35 smithi167 ceph-mon[33178]: 3.0 scrub ok 2024-09-13T13:18:35.459 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:35 smithi167 ceph-mon[33178]: pgmap v882: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-13T13:18:35.459 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:35 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]': finished 2024-09-13T13:18:35.459 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:35 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T13:18:35.459 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:35 smithi167 ceph-mon[33178]: osdmap e441: 8 total, 8 up, 8 in 2024-09-13T13:18:35.459 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:35.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[36682]: 4.3 scrub ok 2024-09-13T13:18:35.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[36682]: 3.0 scrub starts 2024-09-13T13:18:35.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[36682]: 3.0 scrub ok 2024-09-13T13:18:35.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[36682]: pgmap v882: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-13T13:18:35.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]': finished 2024-09-13T13:18:35.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T13:18:35.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[36682]: osdmap e441: 8 total, 8 up, 8 in 2024-09-13T13:18:35.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:35.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[29471]: 4.3 scrub ok 2024-09-13T13:18:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[29471]: 3.0 scrub starts 2024-09-13T13:18:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[29471]: 3.0 scrub ok 2024-09-13T13:18:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[29471]: pgmap v882: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-13T13:18:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 0]}]': finished 2024-09-13T13:18:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T13:18:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[29471]: osdmap e441: 8 total, 8 up, 8 in 2024-09-13T13:18:35.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:35.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:35.590 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:18:35.590 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:18:35.591 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:18:35.591 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:18:35.591 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:18:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:35.596 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:35.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:35.697 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:35.738 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:35.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:35.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:35.840 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:35.880 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 35 16 63094 63078 112.612 80.5 0.00880227 0.00887572 2024-09-13T13:18:35.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:35.882 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:35.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:35.983 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:36.005 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:36.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:36.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:36.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:36.144 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:36.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:36.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:36.245 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:36.283 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:36.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:36.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:36.293 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T13:18:36.385 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:36.446 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:36.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:36.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:36 smithi052 ceph-mon[29471]: 6.19 scrub starts 2024-09-13T13:18:36.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:36 smithi052 ceph-mon[29471]: 6.19 scrub ok 2024-09-13T13:18:36.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:36 smithi052 ceph-mon[29471]: osdmap e442: 8 total, 8 up, 8 in 2024-09-13T13:18:36.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:36 smithi052 ceph-mon[36682]: 6.19 scrub starts 2024-09-13T13:18:36.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:36 smithi052 ceph-mon[36682]: 6.19 scrub ok 2024-09-13T13:18:36.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:36 smithi052 ceph-mon[36682]: osdmap e442: 8 total, 8 up, 8 in 2024-09-13T13:18:36.490 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:18:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:18:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:18:36.547 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:36.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:36 smithi167 ceph-mon[33178]: 6.19 scrub starts 2024-09-13T13:18:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:36 smithi167 ceph-mon[33178]: 6.19 scrub ok 2024-09-13T13:18:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:36 smithi167 ceph-mon[33178]: osdmap e442: 8 total, 8 up, 8 in 2024-09-13T13:18:36.588 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:36.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:36.661 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:18:36.689 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:36.728 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:36.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:36.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:36.830 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:36.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:36.852 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:36.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:36.880 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 36 16 64233 64217 111.461 71.1875 0.0026495 0.00894716 2024-09-13T13:18:36.953 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:36.988 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:36.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:36.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:37.089 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:37.127 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:37.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:37.229 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:37.269 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:37.270 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:18:37.270 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:18:37.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:18:37.271 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:18:37.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.271 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:206414 10.e 10:70abf7d9:::benchmark_data_smithi052_153703_object79034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T13:18:37.271 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:49.280088+0000", 2024-09-13T13:18:37.271 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.985853623, 2024-09-13T13:18:37.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0346179959999999, 2024-09-13T13:18:37.271 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.271 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.271 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.271 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.271 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.271 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 206414 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.280088+0000", 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.280088+0000", 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.280091+0000", 2024-09-13T13:18:37.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.672e-06 2024-09-13T13:18:37.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.280096+0000", 2024-09-13T13:18:37.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5180000000000004e-06 2024-09-13T13:18:37.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.280110+0000", 2024-09-13T13:18:37.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3704e-05 2024-09-13T13:18:37.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.280128+0000", 2024-09-13T13:18:37.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7929000000000001e-05 2024-09-13T13:18:37.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.280207+0000", 2024-09-13T13:18:37.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9552e-05 2024-09-13T13:18:37.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.280215+0000", 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8869999999999992e-06 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.308762+0000", 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0285465789999999 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.308887+0000", 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012524300000000001 2024-09-13T13:18:37.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.309011+0000", 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124067 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.314547+0000", 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055356939999999999 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.314598+0000", 2024-09-13T13:18:37.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0701999999999997e-05 2024-09-13T13:18:37.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.314612+0000", 2024-09-13T13:18:37.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4851000000000001e-05 2024-09-13T13:18:37.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.314662+0000", 2024-09-13T13:18:37.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0015000000000001e-05 2024-09-13T13:18:37.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.314706+0000", 2024-09-13T13:18:37.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3582999999999999e-05 2024-09-13T13:18:37.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.278 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.278 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:206460 10.e 10:7d1ed710:::benchmark_data_smithi052_153703_object79080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T13:18:37.278 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:49.317771+0000", 2024-09-13T13:18:37.278 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.948170578, 2024-09-13T13:18:37.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.009569028, 2024-09-13T13:18:37.278 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 206460 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.317771+0000", 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.317771+0000", 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.317773+0000", 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.48e-06 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.317778+0000", 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.04e-06 2024-09-13T13:18:37.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.317817+0000", 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8980000000000003e-05 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.317823+0000", 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1650000000000003e-06 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.317876+0000", 2024-09-13T13:18:37.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3582999999999998e-05 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.317928+0000", 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2182000000000003e-05 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.311983+0000", 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99405455200000004 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.312110+0000", 2024-09-13T13:18:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012652799999999999 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.312249+0000", 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013902399999999999 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.317191+0000", 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049422889999999999 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.317228+0000", 2024-09-13T13:18:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7623000000000002e-05 2024-09-13T13:18:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327254+0000", 2024-09-13T13:18:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010025053000000001 2024-09-13T13:18:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327294+0000", 2024-09-13T13:18:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0442999999999999e-05 2024-09-13T13:18:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327340+0000", 2024-09-13T13:18:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6085999999999999e-05 2024-09-13T13:18:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:206481 10.e 10:74da50e1:::benchmark_data_smithi052_153703_object79101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T13:18:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:49.337001+0000", 2024-09-13T13:18:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.92894061600001, 2024-09-13T13:18:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99056515199999995, 2024-09-13T13:18:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 206481 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337001+0000", 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337001+0000", 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337003+0000", 2024-09-13T13:18:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7090000000000001e-06 2024-09-13T13:18:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337009+0000", 2024-09-13T13:18:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9170000000000003e-06 2024-09-13T13:18:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337018+0000", 2024-09-13T13:18:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9609999999999994e-06 2024-09-13T13:18:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337023+0000", 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0000000000000004e-06 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337067+0000", 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4700000000000002e-05 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337076+0000", 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5490000000000006e-06 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.314749+0000", 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97767297900000005 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.314864+0000", 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115127 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.314949+0000", 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5117999999999994e-05 2024-09-13T13:18:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.322086+0000", 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0071365710000000004 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.322125+0000", 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9001999999999997e-05 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327476+0000", 2024-09-13T13:18:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0053515309999999996 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327524+0000", 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8312000000000001e-05 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327566+0000", 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1675999999999999e-05 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:206482 10.e 10:7dc5835b:::benchmark_data_smithi052_153703_object79102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:49.337036+0000", 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.92890599499999, 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99062451100000004, 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 206482 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337036+0000", 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337036+0000", 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337036+0000", 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4300000000000002e-07 2024-09-13T13:18:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337039+0000", 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8849999999999999e-06 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337045+0000", 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5960000000000003e-06 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337050+0000", 2024-09-13T13:18:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2639999999999999e-06 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337097+0000", 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7274000000000002e-05 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337103+0000", 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7740000000000001e-06 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.317258+0000", 2024-09-13T13:18:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98015457699999997 2024-09-13T13:18:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.317355+0000", 2024-09-13T13:18:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7233999999999998e-05 2024-09-13T13:18:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.317483+0000", 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012800800000000001 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.322162+0000", 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00467929 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327168+0000", 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050052830000000001 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327576+0000", 2024-09-13T13:18:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040891199999999999 2024-09-13T13:18:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:37.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327624+0000", 2024-09-13T13:18:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7596000000000003e-05 2024-09-13T13:18:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327660+0000", 2024-09-13T13:18:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6075000000000003e-05 2024-09-13T13:18:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:206483 10.e 10:7ff492a4:::benchmark_data_smithi052_153703_object79103:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:49.337057+0000", 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.928884568, 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99067974400000003, 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 206483 2024-09-13T13:18:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337057+0000", 2024-09-13T13:18:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337057+0000", 2024-09-13T13:18:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337058+0000", 2024-09-13T13:18:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6000000000000004e-07 2024-09-13T13:18:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337060+0000", 2024-09-13T13:18:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3590000000000002e-06 2024-09-13T13:18:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337065+0000", 2024-09-13T13:18:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8330000000000001e-06 2024-09-13T13:18:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337069+0000", 2024-09-13T13:18:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2729999999999996e-06 2024-09-13T13:18:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337123+0000", 2024-09-13T13:18:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4273e-05 2024-09-13T13:18:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.337129+0000", 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2399999999999998e-06 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.319695+0000", 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98256676300000001 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.319791+0000", 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5414999999999994e-05 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.319889+0000", 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8509000000000002e-05 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327396+0000", 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075065230000000002 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327463+0000", 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6748999999999994e-05 2024-09-13T13:18:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327667+0000", 2024-09-13T13:18:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000204728 2024-09-13T13:18:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327704+0000", 2024-09-13T13:18:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6371000000000001e-05 2024-09-13T13:18:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327737+0000", 2024-09-13T13:18:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3148000000000002e-05 2024-09-13T13:18:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:206490 10.e 10:71ca6bf6:::benchmark_data_smithi052_153703_object79110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T13:18:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:49.341540+0000", 2024-09-13T13:18:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.924402093, 2024-09-13T13:18:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98642488699999997, 2024-09-13T13:18:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 206490 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.341540+0000", 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.341540+0000", 2024-09-13T13:18:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.341541+0000", 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.866e-06 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.341546+0000", 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9980000000000003e-06 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.341616+0000", 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0020999999999999e-05 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.341623+0000", 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8789999999999997e-06 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.342408+0000", 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00078464599999999995 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.342416+0000", 2024-09-13T13:18:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9619999999999993e-06 2024-09-13T13:18:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.322192+0000", 2024-09-13T13:18:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97977646600000001 2024-09-13T13:18:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.322298+0000", 2024-09-13T13:18:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010553899999999999 2024-09-13T13:18:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.322429+0000", 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013128799999999999 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327745+0000", 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0053158859999999997 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327819+0000", 2024-09-13T13:18:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3467000000000007e-05 2024-09-13T13:18:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327875+0000", 2024-09-13T13:18:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6598999999999998e-05 2024-09-13T13:18:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327925+0000", 2024-09-13T13:18:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9990000000000001e-05 2024-09-13T13:18:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327964+0000", 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9279999999999997e-05 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:206504 10.e 10:721317dc:::benchmark_data_smithi052_153703_object79124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:49.353129+0000", 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.912812329, 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98057226900000005, 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 206504 2024-09-13T13:18:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.353129+0000", 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.353129+0000", 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.353131+0000", 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4279999999999999e-06 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.353135+0000", 2024-09-13T13:18:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:18:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.353145+0000", 2024-09-13T13:18:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-09-13T13:18:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.353151+0000", 2024-09-13T13:18:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5470000000000002e-06 2024-09-13T13:18:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.353201+0000", 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9978000000000002e-05 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.353208+0000", 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9720000000000003e-06 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.324657+0000", 2024-09-13T13:18:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.971448965 2024-09-13T13:18:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.324768+0000", 2024-09-13T13:18:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110742 2024-09-13T13:18:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.324893+0000", 2024-09-13T13:18:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012538800000000001 2024-09-13T13:18:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.327761+0000", 2024-09-13T13:18:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002867372 2024-09-13T13:18:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.328024+0000", 2024-09-13T13:18:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026365699999999999 2024-09-13T13:18:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.333609+0000", 2024-09-13T13:18:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055843639999999996 2024-09-13T13:18:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.333661+0000", 2024-09-13T13:18:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2311000000000001e-05 2024-09-13T13:18:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.333702+0000", 2024-09-13T13:18:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0435000000000002e-05 2024-09-13T13:18:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:206514 10.e 10:73b88039:::benchmark_data_smithi052_153703_object79134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T13:18:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:49.362882+0000", 2024-09-13T13:18:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.903059136, 2024-09-13T13:18:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0040074729999999, 2024-09-13T13:18:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 206514 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.362882+0000", 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.362882+0000", 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.362883+0000", 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2099999999999995e-07 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.362887+0000", 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2160000000000002e-06 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.362896+0000", 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4609999999999997e-06 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.362902+0000", 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.755e-06 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.362956+0000", 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4595999999999997e-05 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.362964+0000", 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5519999999999999e-06 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.328061+0000", 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96509717800000006 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.328175+0000", 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011381400000000001 2024-09-13T13:18:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.328295+0000", 2024-09-13T13:18:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120343 2024-09-13T13:18:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.341547+0000", 2024-09-13T13:18:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013252002000000001 2024-09-13T13:18:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.344243+0000", 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026956649999999999 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.366798+0000", 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022555594000000002 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.366850+0000", 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1209999999999999e-05 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.366890+0000", 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0265999999999997e-05 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:206542 10.e 10:7886fb96:::benchmark_data_smithi052_153703_object79162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:49.397708+0000", 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.86823313799999, 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96932137500000004, 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 206542 2024-09-13T13:18:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.397708+0000", 2024-09-13T13:18:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.397708+0000", 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.397711+0000", 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.046e-06 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.397719+0000", 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9980000000000003e-06 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.397729+0000", 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0268000000000001e-05 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.397736+0000", 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2749999999999998e-06 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.397866+0000", 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012970399999999999 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.397875+0000", 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1500000000000005e-06 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.331043+0000", 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93316820099999997 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.331150+0000", 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010652700000000001 2024-09-13T13:18:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.331261+0000", 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111681 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.341553+0000", 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010291509000000001 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.344285+0000", 2024-09-13T13:18:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027320280000000001 2024-09-13T13:18:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.366956+0000", 2024-09-13T13:18:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022670749 2024-09-13T13:18:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.366998+0000", 2024-09-13T13:18:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2732999999999999e-05 2024-09-13T13:18:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.367030+0000", 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1506e-05 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:206543 10.e 10:742def16:::benchmark_data_smithi052_153703_object79163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:49.398241+0000", 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.86770055300001, 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96895406699999997, 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 206543 2024-09-13T13:18:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.398241+0000", 2024-09-13T13:18:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.398241+0000", 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.398242+0000", 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.249e-06 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.398247+0000", 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9849999999999997e-06 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.398268+0000", 2024-09-13T13:18:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0780000000000001e-05 2024-09-13T13:18:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.398276+0000", 2024-09-13T13:18:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2390000000000002e-06 2024-09-13T13:18:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.398385+0000", 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108954 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.398392+0000", 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6449999999999999e-06 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.333733+0000", 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93534119100000002 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.333838+0000", 2024-09-13T13:18:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010483 2024-09-13T13:18:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.333948+0000", 2024-09-13T13:18:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010967999999999999 2024-09-13T13:18:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.341558+0000", 2024-09-13T13:18:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0076100509999999996 2024-09-13T13:18:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.366924+0000", 2024-09-13T13:18:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025366696000000001 2024-09-13T13:18:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.367091+0000", 2024-09-13T13:18:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016682200000000001 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.367130+0000", 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8729000000000001e-05 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.367195+0000", 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5215999999999994e-05 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:206562 10.e 10:79323631:::benchmark_data_smithi052_153703_object79182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T13:18:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:49.422324+0000", 2024-09-13T13:18:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.843617842, 2024-09-13T13:18:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94500799300000005, 2024-09-13T13:18:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 206562 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.422324+0000", 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.422324+0000", 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.422326+0000", 2024-09-13T13:18:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.753e-06 2024-09-13T13:18:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.422331+0000", 2024-09-13T13:18:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1490000000000002e-06 2024-09-13T13:18:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.422341+0000", 2024-09-13T13:18:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0706e-05 2024-09-13T13:18:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.422348+0000", 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.425458+0000", 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031097149999999999 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.425466+0000", 2024-09-13T13:18:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8599999999999993e-06 2024-09-13T13:18:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.336322+0000", 2024-09-13T13:18:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91085616899999999 2024-09-13T13:18:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.336430+0000", 2024-09-13T13:18:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108496 2024-09-13T13:18:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.336544+0000", 2024-09-13T13:18:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113338 2024-09-13T13:18:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.344161+0000", 2024-09-13T13:18:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0076175230000000002 2024-09-13T13:18:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.367061+0000", 2024-09-13T13:18:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022900203000000001 2024-09-13T13:18:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.367260+0000", 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019870200000000001 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.367301+0000", 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1196999999999999e-05 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.367332+0000", 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0422000000000001e-05 2024-09-13T13:18:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:206567 10.e 10:7b5fb777:::benchmark_data_smithi052_153703_object79187:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:49.427130+0000", 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.83881178599999, 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94888725900000004, 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 206567 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.427130+0000", 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.427130+0000", 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.427132+0000", 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.84e-06 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.427137+0000", 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4079999999999997e-06 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.427149+0000", 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1477999999999999e-05 2024-09-13T13:18:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.427156+0000", 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5650000000000004e-06 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.427207+0000", 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0778000000000001e-05 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.427215+0000", 2024-09-13T13:18:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.208e-06 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.338983+0000", 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91176749700000004 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.339092+0000", 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109487 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.339208+0000", 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115751 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.367229+0000", 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028021167 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.367398+0000", 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016894600000000001 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.375938+0000", 2024-09-13T13:18:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085399310000000006 2024-09-13T13:18:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.375979+0000", 2024-09-13T13:18:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0788999999999999e-05 2024-09-13T13:18:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.376017+0000", 2024-09-13T13:18:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8414000000000002e-05 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:206597 10.e 10:70e570c6:::benchmark_data_smithi052_153703_object79217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:49.472030+0000", 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.793911917, 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.90406112999999999, 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 206597 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.472030+0000", 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.472030+0000", 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.472032+0000", 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0329999999999998e-06 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.472038+0000", 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5969999999999997e-06 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.472049+0000", 2024-09-13T13:18:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0919e-05 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.472077+0000", 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7474000000000002e-05 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.472124+0000", 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7008999999999997e-05 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:49.472132+0000", 2024-09-13T13:18:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1319999999999994e-06 2024-09-13T13:18:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.341588+0000", 2024-09-13T13:18:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.86945650399999996 2024-09-13T13:18:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.341694+0000", 2024-09-13T13:18:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010537199999999999 2024-09-13T13:18:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.341820+0000", 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012587899999999999 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.367366+0000", 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025546696000000001 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.367456+0000", 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0013999999999996e-05 2024-09-13T13:18:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.376024+0000", 2024-09-13T13:18:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085677170000000007 2024-09-13T13:18:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.376059+0000", 2024-09-13T13:18:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4866000000000001e-05 2024-09-13T13:18:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:50.376091+0000", 2024-09-13T13:18:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1918000000000002e-05 2024-09-13T13:18:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242771 10.e 10:7314505a:::benchmark_data_smithi052_153703_object115391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:18:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:30.086247+0000", 2024-09-13T13:18:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.17969411, 2024-09-13T13:18:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.991725737, 2024-09-13T13:18:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242771 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.086247+0000", 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.086247+0000", 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.086250+0000", 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2330000000000001e-06 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.086256+0000", 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6479999999999997e-06 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.086270+0000", 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3478999999999999e-05 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.086278+0000", 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8399999999999995e-06 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.086333+0000", 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4889999999999998e-05 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.086339+0000", 2024-09-13T13:18:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1670000000000004e-06 2024-09-13T13:18:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.073029+0000", 2024-09-13T13:18:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98669026800000004 2024-09-13T13:18:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.073129+0000", 2024-09-13T13:18:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100226 2024-09-13T13:18:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,6", 2024-09-13T13:18:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.073209+0000", 2024-09-13T13:18:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9812000000000001e-05 2024-09-13T13:18:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.077807+0000", 2024-09-13T13:18:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045978429999999999 2024-09-13T13:18:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.077853+0000", 2024-09-13T13:18:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6050999999999997e-05 2024-09-13T13:18:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.077906+0000", 2024-09-13T13:18:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2750000000000001e-05 2024-09-13T13:18:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.077949+0000", 2024-09-13T13:18:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2898999999999997e-05 2024-09-13T13:18:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.077973+0000", 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4630999999999998e-05 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242781 10.e 10:761d7a8b:::benchmark_data_smithi052_153703_object115401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:30.095811+0000", 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.170130510999996, 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98869639600000003, 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242781 2024-09-13T13:18:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.095811+0000", 2024-09-13T13:18:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.095811+0000", 2024-09-13T13:18:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.095812+0000", 2024-09-13T13:18:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.299e-06 2024-09-13T13:18:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.095817+0000", 2024-09-13T13:18:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5839999999999996e-06 2024-09-13T13:18:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.095827+0000", 2024-09-13T13:18:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9189999999999994e-06 2024-09-13T13:18:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.095833+0000", 2024-09-13T13:18:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4629999999999998e-06 2024-09-13T13:18:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.095892+0000", 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8468000000000003e-05 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.095900+0000", 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2779999999999993e-06 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.075312+0000", 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97941169800000005 2024-09-13T13:18:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.075377+0000", 2024-09-13T13:18:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5425000000000005e-05 2024-09-13T13:18:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,6", 2024-09-13T13:18:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.075457+0000", 2024-09-13T13:18:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9348e-05 2024-09-13T13:18:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.078055+0000", 2024-09-13T13:18:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002598604 2024-09-13T13:18:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.079694+0000", 2024-09-13T13:18:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016392710000000001 2024-09-13T13:18:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.084404+0000", 2024-09-13T13:18:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047091759999999998 2024-09-13T13:18:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.084460+0000", 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6150000000000003e-05 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.084507+0000", 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7713000000000002e-05 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242783 10.e 10:7f3603ee:::benchmark_data_smithi052_153703_object115403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:18:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:30.098588+0000", 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.167354033999999, 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98835710700000001, 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242783 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.098588+0000", 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.098588+0000", 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.098589+0000", 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.435e-06 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.098594+0000", 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.006e-06 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.098605+0000", 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1002000000000001e-05 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.098612+0000", 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1180000000000003e-06 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.098689+0000", 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7337999999999997e-05 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.098698+0000", 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4889999999999995e-06 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.076662+0000", 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97796415800000003 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.076730+0000", 2024-09-13T13:18:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7428999999999994e-05 2024-09-13T13:18:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,6", 2024-09-13T13:18:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.076804+0000", 2024-09-13T13:18:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4435000000000001e-05 2024-09-13T13:18:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.079709+0000", 2024-09-13T13:18:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029046699999999998 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.079741+0000", 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2548000000000001e-05 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.086853+0000", 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0071114389999999998 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.086903+0000", 2024-09-13T13:18:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0744000000000001e-05 2024-09-13T13:18:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.086945+0000", 2024-09-13T13:18:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1295999999999999e-05 2024-09-13T13:18:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242803 10.e 10:7411783c:::benchmark_data_smithi052_153703_object115423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:18:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:30.122040+0000", 2024-09-13T13:18:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.143901920999994, 2024-09-13T13:18:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97130097599999998, 2024-09-13T13:18:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242803 2024-09-13T13:18:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.122040+0000", 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.122040+0000", 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.122041+0000", 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3710000000000001e-06 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.122046+0000", 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7779999999999998e-06 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.122062+0000", 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5812e-05 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.122070+0000", 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3569999999999998e-06 2024-09-13T13:18:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.122138+0000", 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8287000000000005e-05 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.122146+0000", 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0709999999999995e-06 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.078080+0000", 2024-09-13T13:18:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95593386000000002 2024-09-13T13:18:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.078183+0000", 2024-09-13T13:18:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102994 2024-09-13T13:18:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,6", 2024-09-13T13:18:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.078307+0000", 2024-09-13T13:18:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123449 2024-09-13T13:18:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.079774+0000", 2024-09-13T13:18:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014670270000000001 2024-09-13T13:18:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.079780+0000", 2024-09-13T13:18:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3380000000000001e-06 2024-09-13T13:18:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.093243+0000", 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013462729 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.093306+0000", 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3101000000000005e-05 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.093341+0000", 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4802000000000003e-05 2024-09-13T13:18:37.399 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242832 10.e 10:7a9a4534:::benchmark_data_smithi052_153703_object115452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:30.151157+0000", 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.114784529000005, 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94228700799999998, 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.400 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.401 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242832 2024-09-13T13:18:37.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.401 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.151157+0000", 2024-09-13T13:18:37.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.151157+0000", 2024-09-13T13:18:37.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.151158+0000", 2024-09-13T13:18:37.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2330000000000001e-06 2024-09-13T13:18:37.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.151163+0000", 2024-09-13T13:18:37.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4769999999999997e-06 2024-09-13T13:18:37.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.151171+0000", 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3489999999999992e-06 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.151176+0000", 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5050000000000001e-06 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.151219+0000", 2024-09-13T13:18:37.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2871999999999999e-05 2024-09-13T13:18:37.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.151225+0000", 2024-09-13T13:18:37.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1739999999999997e-06 2024-09-13T13:18:37.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.079810+0000", 2024-09-13T13:18:37.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92858507499999998 2024-09-13T13:18:37.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.079868+0000", 2024-09-13T13:18:37.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8678000000000003e-05 2024-09-13T13:18:37.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,6", 2024-09-13T13:18:37.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.079929+0000", 2024-09-13T13:18:37.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0739999999999998e-05 2024-09-13T13:18:37.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.084565+0000", 2024-09-13T13:18:37.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004635915 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.084578+0000", 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3301e-05 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.093393+0000", 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088142610000000003 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.093422+0000", 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9371999999999999e-05 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.093444+0000", 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2056e-05 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242840 10.e 10:78c303a5:::benchmark_data_smithi052_153703_object115460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:30.155111+0000", 2024-09-13T13:18:37.407 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.110830819, 2024-09-13T13:18:37.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93842112, 2024-09-13T13:18:37.408 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.408 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.408 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.408 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.408 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.408 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.408 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242840 2024-09-13T13:18:37.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.408 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.155111+0000", 2024-09-13T13:18:37.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.155111+0000", 2024-09-13T13:18:37.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.155113+0000", 2024-09-13T13:18:37.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7179999999999999e-06 2024-09-13T13:18:37.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.155117+0000", 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4599999999999996e-06 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.155127+0000", 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9369999999999999e-06 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.155132+0000", 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7060000000000003e-06 2024-09-13T13:18:37.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.155180+0000", 2024-09-13T13:18:37.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8253e-05 2024-09-13T13:18:37.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.155193+0000", 2024-09-13T13:18:37.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2894999999999999e-05 2024-09-13T13:18:37.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.081247+0000", 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92605472300000002 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.081329+0000", 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1057999999999995e-05 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,6", 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.081401+0000", 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2231999999999996e-05 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.084638+0000", 2024-09-13T13:18:37.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032371840000000002 2024-09-13T13:18:37.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.086978+0000", 2024-09-13T13:18:37.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023399369999999998 2024-09-13T13:18:37.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.093477+0000", 2024-09-13T13:18:37.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064986699999999998 2024-09-13T13:18:37.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.093511+0000", 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4915e-05 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.093532+0000", 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0432e-05 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242848 10.e 10:7777d3f6:::benchmark_data_smithi052_153703_object115468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:30.176934+0000", 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.089007573000004, 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91668958099999998, 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242848 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:37.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.176934+0000", 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.176934+0000", 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.176936+0000", 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:18:37.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:37.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.176941+0000", 2024-09-13T13:18:37.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0509999999999996e-06 2024-09-13T13:18:37.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:37.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.176954+0000", 2024-09-13T13:18:37.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3570000000000001e-05 2024-09-13T13:18:37.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:37.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.176962+0000", 2024-09-13T13:18:37.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8490000000000006e-06 2024-09-13T13:18:37.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.177004+0000", 2024-09-13T13:18:37.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1857000000000003e-05 2024-09-13T13:18:37.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:18:37.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.177009+0000", 2024-09-13T13:18:37.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4700000000000004e-06 2024-09-13T13:18:37.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:37.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.082961+0000", 2024-09-13T13:18:37.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.90595211799999997 2024-09-13T13:18:37.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:37.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.083037+0000", 2024-09-13T13:18:37.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6611000000000002e-05 2024-09-13T13:18:37.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,6", 2024-09-13T13:18:37.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.083112+0000", 2024-09-13T13:18:37.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5053000000000001e-05 2024-09-13T13:18:37.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.086809+0000", 2024-09-13T13:18:37.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036962470000000002 2024-09-13T13:18:37.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:37.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.086983+0000", 2024-09-13T13:18:37.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017475300000000001 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.093563+0000", 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065798130000000003 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.093589+0000", 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5437999999999999e-05 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:31.093624+0000", 2024-09-13T13:18:37.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4971000000000001e-05 2024-09-13T13:18:37.422 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.422 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.422 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.422 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:37.422 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:37.422 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:18:37.422 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:37.423 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:37 smithi167 ceph-mon[33178]: pgmap v885: 158 pgs: 1 peering, 157 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T13:18:37.423 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:37 smithi167 ceph-mon[33178]: 6.0 scrub starts 2024-09-13T13:18:37.423 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:37 smithi167 ceph-mon[33178]: 6.0 scrub ok 2024-09-13T13:18:37.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:37.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:37.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:37.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:37.587 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:37.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:37.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:37.689 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:37.721 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:37.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:37.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:37.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:37 smithi052 ceph-mon[36682]: pgmap v885: 158 pgs: 1 peering, 157 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T13:18:37.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:37 smithi052 ceph-mon[36682]: 6.0 scrub starts 2024-09-13T13:18:37.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:37 smithi052 ceph-mon[36682]: 6.0 scrub ok 2024-09-13T13:18:37.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:37 smithi052 ceph-mon[29471]: pgmap v885: 158 pgs: 1 peering, 157 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-13T13:18:37.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:37 smithi052 ceph-mon[29471]: 6.0 scrub starts 2024-09-13T13:18:37.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:37 smithi052 ceph-mon[29471]: 6.0 scrub ok 2024-09-13T13:18:37.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:37.856 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:37.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:37.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:37.880 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 37 16 65470 65454 110.538 77.3125 0.0152821 0.00904163 2024-09-13T13:18:37.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:37.984 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:37.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:37.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:38.046 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:38.057 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:38.086 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:38.137 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:38.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:38.241 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:38.288 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:38.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:38.388 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:38.430 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:38.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:38.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T13:18:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[36682]: 4.0 scrub starts 2024-09-13T13:18:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[36682]: 4.0 scrub ok 2024-09-13T13:18:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:18:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:18:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[36682]: osdmap e443: 8 total, 8 up, 8 in 2024-09-13T13:18:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[36682]: 3.1 scrub starts 2024-09-13T13:18:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[36682]: 3.1 scrub ok 2024-09-13T13:18:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[36682]: 4.1c scrub starts 2024-09-13T13:18:38.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[36682]: 4.1c scrub ok 2024-09-13T13:18:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[29471]: 4.0 scrub starts 2024-09-13T13:18:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[29471]: 4.0 scrub ok 2024-09-13T13:18:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:18:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:18:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[29471]: osdmap e443: 8 total, 8 up, 8 in 2024-09-13T13:18:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[29471]: 3.1 scrub starts 2024-09-13T13:18:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[29471]: 3.1 scrub ok 2024-09-13T13:18:38.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[29471]: 4.1c scrub starts 2024-09-13T13:18:38.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:38 smithi052 ceph-mon[29471]: 4.1c scrub ok 2024-09-13T13:18:38.532 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:38.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:38 smithi167 ceph-mon[33178]: 4.0 scrub starts 2024-09-13T13:18:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:38 smithi167 ceph-mon[33178]: 4.0 scrub ok 2024-09-13T13:18:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:38 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:18:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:38 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:18:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:38 smithi167 ceph-mon[33178]: osdmap e443: 8 total, 8 up, 8 in 2024-09-13T13:18:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:38 smithi167 ceph-mon[33178]: 3.1 scrub starts 2024-09-13T13:18:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:38 smithi167 ceph-mon[33178]: 3.1 scrub ok 2024-09-13T13:18:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:38 smithi167 ceph-mon[33178]: 4.1c scrub starts 2024-09-13T13:18:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:38 smithi167 ceph-mon[33178]: 4.1c scrub ok 2024-09-13T13:18:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:38.573 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:38.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:38.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:38.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:38.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:38.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:38.813 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:18:38.845 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:38.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:38.880 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 38 16 66771 66755 109.769 81.3125 0.00465679 0.00910507 2024-09-13T13:18:38.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:38.982 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:38.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:39.083 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:39.126 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:39.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:39.227 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:39.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:39.277 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:39.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:39.378 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:39.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:39.422 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:39.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:39.466 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:18:39.466 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:18:39.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.466 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:67469 11.7 11:e54943f2:::benchmark_data_smithi052_182575_object67468:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e443)", 2024-09-13T13:18:39.466 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:39.442144+0000", 2024-09-13T13:18:39.466 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.022280048, 2024-09-13T13:18:39.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022310487, 2024-09-13T13:18:39.466 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:39.466 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:39.466 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:18:39.467 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:39.467 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:18:39.467 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:18:39.467 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 67469 2024-09-13T13:18:39.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.467 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:39.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:39.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.442144+0000", 2024-09-13T13:18:39.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:39.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:39.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.442144+0000", 2024-09-13T13:18:39.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:39.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:39.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.442146+0000", 2024-09-13T13:18:39.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9410000000000001e-06 2024-09-13T13:18:39.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:39.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.442190+0000", 2024-09-13T13:18:39.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4286000000000003e-05 2024-09-13T13:18:39.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.442207+0000", 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6560999999999999e-05 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.442215+0000", 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0239999999999997e-06 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.442276+0000", 2024-09-13T13:18:39.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1289000000000002e-05 2024-09-13T13:18:39.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:39.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.442431+0000", 2024-09-13T13:18:39.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154356 2024-09-13T13:18:39.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:18:39.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.442617+0000", 2024-09-13T13:18:39.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018651299999999999 2024-09-13T13:18:39.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:39.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.447907+0000", 2024-09-13T13:18:39.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0052896610000000002 2024-09-13T13:18:39.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:39.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.453189+0000", 2024-09-13T13:18:39.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0052823569999999997 2024-09-13T13:18:39.471 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:39.471 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:39.471 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:39.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:67473 11.7 11:e5b4613f:::benchmark_data_smithi052_182575_object67472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e443)", 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:39.448103+0000", 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.016320734, 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016402869, 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 67473 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:39.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:39.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.448103+0000", 2024-09-13T13:18:39.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:39.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:39.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.448103+0000", 2024-09-13T13:18:39.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:39.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:39.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.448106+0000", 2024-09-13T13:18:39.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3029999999999998e-06 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.448143+0000", 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6838999999999997e-05 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.448159+0000", 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.609e-05 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.448168+0000", 2024-09-13T13:18:39.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9979999999999992e-06 2024-09-13T13:18:39.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:39.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.448247+0000", 2024-09-13T13:18:39.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9413999999999996e-05 2024-09-13T13:18:39.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:39.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.448410+0000", 2024-09-13T13:18:39.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000162406 2024-09-13T13:18:39.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:18:39.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.448562+0000", 2024-09-13T13:18:39.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015288500000000001 2024-09-13T13:18:39.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:18:39.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.451034+0000", 2024-09-13T13:18:39.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002471254 2024-09-13T13:18:39.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:39.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:39.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:18:39.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:39.453664+0000", 2024-09-13T13:18:39.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026300749999999999 2024-09-13T13:18:39.477 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:39.477 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:39.477 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:39.477 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:39.477 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:18:39.477 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 2 2024-09-13T13:18:39.477 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:18:39.524 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:39.560 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:39.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:39.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:39 smithi167 ceph-mon[33178]: pgmap v887: 158 pgs: 1 peering, 157 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 45 B/s, 1 objects/s recovering 2024-09-13T13:18:39.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:39 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:18:39.661 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:39.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:39.710 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:39.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:39.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:39 smithi052 ceph-mon[36682]: pgmap v887: 158 pgs: 1 peering, 157 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 45 B/s, 1 objects/s recovering 2024-09-13T13:18:39.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:39 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:18:39.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:39 smithi052 ceph-mon[29471]: pgmap v887: 158 pgs: 1 peering, 157 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 45 B/s, 1 objects/s recovering 2024-09-13T13:18:39.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:39 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:18:39.812 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:39.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:39.857 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:39.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:39.880 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 39 16 67846 67830 108.677 67.1875 0.0256369 0.00919654 2024-09-13T13:18:39.958 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:39.980 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:18:40.004 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:40.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:40.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:40.107 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:40.155 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:40.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:40.256 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:40.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:40.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:40.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:40.319 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T13:18:40.386 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:40.397 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:40 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:18:40.397 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:40 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:18:40.398 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:40 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:18:40.398 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:40 smithi052 ceph-mon[29471]: 5.b scrub starts 2024-09-13T13:18:40.398 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:40 smithi052 ceph-mon[29471]: 5.b scrub ok 2024-09-13T13:18:40.398 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:40 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:18:40.398 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:40 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:18:40.398 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:40 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:18:40.399 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:40 smithi052 ceph-mon[36682]: 5.b scrub starts 2024-09-13T13:18:40.399 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:40 smithi052 ceph-mon[36682]: 5.b scrub ok 2024-09-13T13:18:40.435 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:40.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:40.537 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:40.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:40 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:18:40.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:40 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:18:40.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:40 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:18:40.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:40 smithi167 ceph-mon[33178]: 5.b scrub starts 2024-09-13T13:18:40.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:40 smithi167 ceph-mon[33178]: 5.b scrub ok 2024-09-13T13:18:40.579 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:40.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:40.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:40.681 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:40.720 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:40.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:40.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:40.738 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:18:40.822 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:40.862 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:40.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:40.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:40.880 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:18:40.880683+0000 min lat: 0.00125067 max lat: 0.321739 avg lat: 0.00927385 2024-09-13T13:18:40.880 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:18:40.880 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 40 16 68997 68981 107.758 71.9375 0.00690533 0.00927385 2024-09-13T13:18:40.963 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:40.991 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:40.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:41.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:41.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:41.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:41.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:41.226 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:41.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:41.252 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:41.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:41.354 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:41.372 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:18:41.372 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:18:41.372 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:18:41.373 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:18:41.373 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:18:41.389 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:41.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:41.491 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:41.531 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:41.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:41.633 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:41.662 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:41.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:41.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:41 smithi052 ceph-mon[36682]: pgmap v888: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 96 B/s, 2 objects/s recovering 2024-09-13T13:18:41.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:41 smithi052 ceph-mon[36682]: 5.3 scrub starts 2024-09-13T13:18:41.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:41 smithi052 ceph-mon[36682]: 5.3 scrub ok 2024-09-13T13:18:41.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:41 smithi052 ceph-mon[36682]: 6.17 scrub starts 2024-09-13T13:18:41.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:41 smithi052 ceph-mon[36682]: 6.17 scrub ok 2024-09-13T13:18:41.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:41 smithi052 ceph-mon[29471]: pgmap v888: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 96 B/s, 2 objects/s recovering 2024-09-13T13:18:41.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:41 smithi052 ceph-mon[29471]: 5.3 scrub starts 2024-09-13T13:18:41.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:41 smithi052 ceph-mon[29471]: 5.3 scrub ok 2024-09-13T13:18:41.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:41 smithi052 ceph-mon[29471]: 6.17 scrub starts 2024-09-13T13:18:41.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:41 smithi052 ceph-mon[29471]: 6.17 scrub ok 2024-09-13T13:18:41.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:41.790 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:41.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:41.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:41.880 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 41 16 70142 70126 106.875 71.5625 0.00728252 0.00935143 2024-09-13T13:18:41.892 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:41.943 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:41.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:42.044 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:41 smithi167 ceph-mon[33178]: pgmap v888: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 96 B/s, 2 objects/s recovering 2024-09-13T13:18:42.044 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:41 smithi167 ceph-mon[33178]: 5.3 scrub starts 2024-09-13T13:18:42.044 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:41 smithi167 ceph-mon[33178]: 5.3 scrub ok 2024-09-13T13:18:42.044 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:41 smithi167 ceph-mon[33178]: 6.17 scrub starts 2024-09-13T13:18:42.044 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:41 smithi167 ceph-mon[33178]: 6.17 scrub ok 2024-09-13T13:18:42.045 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:42.084 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:42.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:42.090 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T13:18:42.184 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:42.219 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:42.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:42.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:42.321 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:42.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:42.351 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:42.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:42.451 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:18:42.452 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:42.486 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:42.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:42.577 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:42.588 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:42.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:42.614 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:42.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:42.716 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:42.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:42.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:42.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:42.839 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:42.864 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:42 smithi167 ceph-mon[33178]: 5.9 scrub starts 2024-09-13T13:18:42.865 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:42 smithi167 ceph-mon[33178]: 5.9 scrub ok 2024-09-13T13:18:42.865 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:42 smithi167 ceph-mon[33178]: 4.1e scrub starts 2024-09-13T13:18:42.865 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:42 smithi167 ceph-mon[33178]: 4.1e scrub ok 2024-09-13T13:18:42.865 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:42 smithi167 ceph-mon[33178]: 6.e scrub starts 2024-09-13T13:18:42.865 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:42 smithi167 ceph-mon[33178]: 6.e scrub ok 2024-09-13T13:18:42.882 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 42 16 71200 71184 105.904 66.125 0.0148212 0.00943682 2024-09-13T13:18:42.888 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:42.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:42.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:42.899 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:42 smithi052 ceph-mon[29471]: 5.9 scrub starts 2024-09-13T13:18:42.899 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:42 smithi052 ceph-mon[29471]: 5.9 scrub ok 2024-09-13T13:18:42.900 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:42 smithi052 ceph-mon[29471]: 4.1e scrub starts 2024-09-13T13:18:42.900 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:42 smithi052 ceph-mon[29471]: 4.1e scrub ok 2024-09-13T13:18:42.900 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:42 smithi052 ceph-mon[29471]: 6.e scrub starts 2024-09-13T13:18:42.900 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:42 smithi052 ceph-mon[29471]: 6.e scrub ok 2024-09-13T13:18:42.900 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:42 smithi052 ceph-mon[36682]: 5.9 scrub starts 2024-09-13T13:18:42.900 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:42 smithi052 ceph-mon[36682]: 5.9 scrub ok 2024-09-13T13:18:42.900 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:42 smithi052 ceph-mon[36682]: 4.1e scrub starts 2024-09-13T13:18:42.900 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:42 smithi052 ceph-mon[36682]: 4.1e scrub ok 2024-09-13T13:18:42.901 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:42 smithi052 ceph-mon[36682]: 6.e scrub starts 2024-09-13T13:18:42.901 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:42 smithi052 ceph-mon[36682]: 6.e scrub ok 2024-09-13T13:18:42.990 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:43.027 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:43.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:43.129 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:43.163 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:43.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:43.248 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:18:43.249 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:18:43.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:43.288 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:18:43.288 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:18:43.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:43.290 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:43.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:43.330 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:18:43.332 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:18:43.367 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:18:43.367 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:18:43.391 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:43.401 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:18:43.401 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:18:43.426 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:43.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:43.473 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:18:43.473 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:18:43.504 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:18:43.504 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:18:43.529 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:43.543 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:18:43.544 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:18:43.559 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:43.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:43.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:43.628 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T13:18:43.641 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:18:43.661 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:43.694 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:18:43.694 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:18:43.703 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:43.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:43.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[36682]: 6.1f scrub starts 2024-09-13T13:18:43.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[36682]: 6.1f scrub ok 2024-09-13T13:18:43.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[36682]: pgmap v889: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 74 B/s, 1 objects/s recovering 2024-09-13T13:18:43.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/448941295' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:18:43.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1676089161' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:43.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[36682]: from='client.36701 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:43.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1601438227' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:43.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[29471]: 6.1f scrub starts 2024-09-13T13:18:43.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[29471]: 6.1f scrub ok 2024-09-13T13:18:43.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[29471]: pgmap v889: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 74 B/s, 1 objects/s recovering 2024-09-13T13:18:43.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/448941295' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:18:43.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1676089161' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:43.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[29471]: from='client.36701 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:43.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1601438227' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:43.806 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:43.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:43.846 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:43.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:43.881 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 43 16 72386 72370 105.165 74.125 0.00939697 0.00950441 2024-09-13T13:18:43.948 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:43.982 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:43.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:44.038 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:43 smithi167 ceph-mon[33178]: 6.1f scrub starts 2024-09-13T13:18:44.038 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:43 smithi167 ceph-mon[33178]: 6.1f scrub ok 2024-09-13T13:18:44.038 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:43 smithi167 ceph-mon[33178]: pgmap v889: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 74 B/s, 1 objects/s recovering 2024-09-13T13:18:44.039 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/448941295' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:18:44.039 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1676089161' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:44.039 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:43 smithi167 ceph-mon[33178]: from='client.36701 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:44.039 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1601438227' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:44.083 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:44.126 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:44.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:44.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:44.228 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:44.263 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:44.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:44.365 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:44.397 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:44.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:44.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:44.499 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:44.527 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:44.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:44.577 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T13:18:44.629 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:44.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:44.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:44.777 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:44.815 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:44.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:44.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:44.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:18:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:18:44.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:44 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:18:44.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:44 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:18:44.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/448941295' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:18:44.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:44 smithi167 ceph-mon[33178]: from='client.36701 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:18:44.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:44 smithi167 ceph-mon[33178]: osdmap e444: 8 total, 8 up, 8 in 2024-09-13T13:18:44.881 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 44 16 73703 73687 104.645 82.3125 0.0135002 0.00955142 2024-09-13T13:18:44.918 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:44.957 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:44.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:44.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:44.981 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:18:44.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:44 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:18:44.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:44 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:18:44.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/448941295' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:18:44.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:44 smithi052 ceph-mon[36682]: from='client.36701 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:18:44.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:44 smithi052 ceph-mon[36682]: osdmap e444: 8 total, 8 up, 8 in 2024-09-13T13:18:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:44 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:18:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:44 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:18:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/448941295' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:18:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:44 smithi052 ceph-mon[29471]: from='client.36701 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:18:44.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:44 smithi052 ceph-mon[29471]: osdmap e444: 8 total, 8 up, 8 in 2024-09-13T13:18:45.059 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:45.087 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:45.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:45.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:45.189 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:45.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:45.215 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:45.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:45.316 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:45.345 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:45.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:45.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:45.365 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T13:18:45.446 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:45.479 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:45.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:45.579 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:18:45.579 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:18:45.582 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:45.626 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:45.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:45.728 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:45.765 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:45.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:45 smithi167 ceph-mon[33178]: 6.a scrub starts 2024-09-13T13:18:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:45 smithi167 ceph-mon[33178]: 6.a scrub ok 2024-09-13T13:18:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:45 smithi167 ceph-mon[33178]: pgmap v891: 158 pgs: 158 active+clean; 4.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.29k op/s; 72 B/s, 1 objects/s recovering 2024-09-13T13:18:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:45 smithi167 ceph-mon[33178]: 6.10 scrub starts 2024-09-13T13:18:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:45 smithi167 ceph-mon[33178]: 6.10 scrub ok 2024-09-13T13:18:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:45 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:45 smithi167 ceph-mon[33178]: 2.1 scrub starts 2024-09-13T13:18:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:45 smithi167 ceph-mon[33178]: 2.1 scrub ok 2024-09-13T13:18:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:45 smithi167 ceph-mon[33178]: 4.13 scrub starts 2024-09-13T13:18:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:45 smithi167 ceph-mon[33178]: 4.13 scrub ok 2024-09-13T13:18:45.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:45.883 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 45 16 74939 74923 104.033 77.25 0.0134809 0.0096069 2024-09-13T13:18:45.893 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:45.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:45.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:45.915 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[36682]: 6.a scrub starts 2024-09-13T13:18:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[36682]: 6.a scrub ok 2024-09-13T13:18:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[36682]: pgmap v891: 158 pgs: 158 active+clean; 4.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.29k op/s; 72 B/s, 1 objects/s recovering 2024-09-13T13:18:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[36682]: 6.10 scrub starts 2024-09-13T13:18:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[36682]: 6.10 scrub ok 2024-09-13T13:18:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:45.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[36682]: 2.1 scrub starts 2024-09-13T13:18:45.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[36682]: 2.1 scrub ok 2024-09-13T13:18:45.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[36682]: 4.13 scrub starts 2024-09-13T13:18:45.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[36682]: 4.13 scrub ok 2024-09-13T13:18:45.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[29471]: 6.a scrub starts 2024-09-13T13:18:45.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[29471]: 6.a scrub ok 2024-09-13T13:18:45.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[29471]: pgmap v891: 158 pgs: 158 active+clean; 4.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.29k op/s; 72 B/s, 1 objects/s recovering 2024-09-13T13:18:45.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[29471]: 6.10 scrub starts 2024-09-13T13:18:45.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[29471]: 6.10 scrub ok 2024-09-13T13:18:45.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:45.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[29471]: 2.1 scrub starts 2024-09-13T13:18:45.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[29471]: 2.1 scrub ok 2024-09-13T13:18:45.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[29471]: 4.13 scrub starts 2024-09-13T13:18:45.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:45 smithi052 ceph-mon[29471]: 4.13 scrub ok 2024-09-13T13:18:45.995 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:46.042 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:46.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:46.144 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:46.181 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:46.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:46.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:46.283 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:46.324 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:46.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:46.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T13:18:46.399 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:18:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:18:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:18:46.426 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:46.469 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:46.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:46.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:46.570 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:46.622 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:46.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:46.677 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:18:46.700 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:18:46.724 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:46.757 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:46.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:46.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:46.859 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:46.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:46.883 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 46 15 76064 76049 103.301 70.375 0.0132345 0.00967417 2024-09-13T13:18:46.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:46.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:46.984 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:46 smithi052 ceph-mon[36682]: 3.1b scrub starts 2024-09-13T13:18:46.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:46 smithi052 ceph-mon[36682]: 3.1b scrub ok 2024-09-13T13:18:46.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:46 smithi052 ceph-mon[36682]: 6.14 scrub starts 2024-09-13T13:18:46.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:46 smithi052 ceph-mon[36682]: 6.14 scrub ok 2024-09-13T13:18:46.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:46 smithi052 ceph-mon[36682]: 3.1a scrub starts 2024-09-13T13:18:46.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:46 smithi052 ceph-mon[36682]: 3.1a scrub ok 2024-09-13T13:18:46.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:46 smithi052 ceph-mon[29471]: 3.1b scrub starts 2024-09-13T13:18:46.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:46 smithi052 ceph-mon[29471]: 3.1b scrub ok 2024-09-13T13:18:46.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:46 smithi052 ceph-mon[29471]: 6.14 scrub starts 2024-09-13T13:18:46.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:46 smithi052 ceph-mon[29471]: 6.14 scrub ok 2024-09-13T13:18:46.992 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:46 smithi052 ceph-mon[29471]: 3.1a scrub starts 2024-09-13T13:18:46.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:46 smithi052 ceph-mon[29471]: 3.1a scrub ok 2024-09-13T13:18:46.998 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:46 smithi167 ceph-mon[33178]: 3.1b scrub starts 2024-09-13T13:18:46.998 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:46 smithi167 ceph-mon[33178]: 3.1b scrub ok 2024-09-13T13:18:46.999 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:46 smithi167 ceph-mon[33178]: 6.14 scrub starts 2024-09-13T13:18:46.999 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:46 smithi167 ceph-mon[33178]: 6.14 scrub ok 2024-09-13T13:18:46.999 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:46 smithi167 ceph-mon[33178]: 3.1a scrub starts 2024-09-13T13:18:46.999 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:46 smithi167 ceph-mon[33178]: 3.1a scrub ok 2024-09-13T13:18:47.012 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:47.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:47.114 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:47.138 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:47.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:47.146 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:47.241 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:47.275 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:47.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:47.319 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:18:47.320 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:18:47.376 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:47.415 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:47.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:47.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:47.517 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:47.565 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:47.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:47.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:47.667 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:47.670 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T13:18:47.720 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:47.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:47.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:47.822 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:47.843 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:18:47.853 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:47.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:47.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:47.882 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[36682]: pgmap v892: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 66 B/s, 1 objects/s recovering 2024-09-13T13:18:47.882 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[36682]: 4.10 scrub starts 2024-09-13T13:18:47.882 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[36682]: 4.10 scrub ok 2024-09-13T13:18:47.882 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3679727298' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:18:47.882 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[36682]: from='client.36713 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:18:47.882 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[36682]: 6.4 scrub starts 2024-09-13T13:18:47.882 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[36682]: 6.4 scrub ok 2024-09-13T13:18:47.883 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[29471]: pgmap v892: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 66 B/s, 1 objects/s recovering 2024-09-13T13:18:47.883 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[29471]: 4.10 scrub starts 2024-09-13T13:18:47.883 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[29471]: 4.10 scrub ok 2024-09-13T13:18:47.883 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3679727298' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:18:47.883 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[29471]: from='client.36713 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:18:47.883 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[29471]: 6.4 scrub starts 2024-09-13T13:18:47.883 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:47 smithi052 ceph-mon[29471]: 6.4 scrub ok 2024-09-13T13:18:47.883 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 47 16 77189 77173 102.598 70.25 0.00834189 0.00974064 2024-09-13T13:18:47.955 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:48.006 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:48.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:48.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:48.036 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:47 smithi167 ceph-mon[33178]: pgmap v892: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 66 B/s, 1 objects/s recovering 2024-09-13T13:18:48.036 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:47 smithi167 ceph-mon[33178]: 4.10 scrub starts 2024-09-13T13:18:48.036 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:47 smithi167 ceph-mon[33178]: 4.10 scrub ok 2024-09-13T13:18:48.037 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:47 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3679727298' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:18:48.037 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:47 smithi167 ceph-mon[33178]: from='client.36713 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:18:48.037 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:47 smithi167 ceph-mon[33178]: 6.4 scrub starts 2024-09-13T13:18:48.037 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:47 smithi167 ceph-mon[33178]: 6.4 scrub ok 2024-09-13T13:18:48.081 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T13:18:48.108 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:48.154 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:48.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:48.227 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:18:48.228 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:18:48.256 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:48.266 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:18:48.266 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:18:48.283 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:48.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:48.314 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:18:48.315 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:18:48.350 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:18:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:18:48.384 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:48.393 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:18:48.393 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:18:48.415 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:48.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:48.425 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:18:48.433 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:18:48.433 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:18:48.473 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:18:48.474 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:18:48.508 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:18:48.508 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:18:48.517 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:48.563 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:48.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:48.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:48.708 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:48.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:48 smithi052 ceph-mon[36682]: from='client.36713 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:18:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:48 smithi052 ceph-mon[36682]: osdmap e445: 8 total, 8 up, 8 in 2024-09-13T13:18:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:48 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3208074417' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:48 smithi052 ceph-mon[36682]: 4.d scrub starts 2024-09-13T13:18:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:48 smithi052 ceph-mon[36682]: 4.d scrub ok 2024-09-13T13:18:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:48 smithi052 ceph-mon[29471]: from='client.36713 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:18:48.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:48 smithi052 ceph-mon[29471]: osdmap e445: 8 total, 8 up, 8 in 2024-09-13T13:18:48.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:48 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3208074417' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:48.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:48 smithi052 ceph-mon[29471]: 4.d scrub starts 2024-09-13T13:18:48.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:48 smithi052 ceph-mon[29471]: 4.d scrub ok 2024-09-13T13:18:48.810 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:48.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:48 smithi167 ceph-mon[33178]: from='client.36713 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:18:48.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:48 smithi167 ceph-mon[33178]: osdmap e445: 8 total, 8 up, 8 in 2024-09-13T13:18:48.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:48 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3208074417' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:48.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:48 smithi167 ceph-mon[33178]: 4.d scrub starts 2024-09-13T13:18:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:48 smithi167 ceph-mon[33178]: 4.d scrub ok 2024-09-13T13:18:48.848 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:48.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:48.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:48.883 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 48 16 78171 78155 101.739 61.375 0.00818519 0.00982308 2024-09-13T13:18:48.894 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T13:18:48.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:18:48.950 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:49.031 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:49.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:49.052 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:18:49.052 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:18:49.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:18:49.052 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:18:49.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.052 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:133006 10.8 e366/334 10:1945bf5a:::benchmark_data_smithi052_153703_object5626:head v 366'8411, mlcod=366'8409)", 2024-09-13T13:18:49.052 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.291495+0000", 2024-09-13T13:18:49.052 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.75780027100001, 2024-09-13T13:18:49.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.36882864700000001, 2024-09-13T13:18:49.052 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.052 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.053 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.053 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291495+0000", 2024-09-13T13:18:49.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291495+0000", 2024-09-13T13:18:49.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291497+0000", 2024-09-13T13:18:49.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4109999999999999e-06 2024-09-13T13:18:49.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291503+0000", 2024-09-13T13:18:49.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2190000000000001e-06 2024-09-13T13:18:49.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291513+0000", 2024-09-13T13:18:49.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0451999999999999e-05 2024-09-13T13:18:49.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291520+0000", 2024-09-13T13:18:49.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.759e-06 2024-09-13T13:18:49.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291570+0000", 2024-09-13T13:18:49.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9979999999999999e-05 2024-09-13T13:18:49.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291585+0000", 2024-09-13T13:18:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4688999999999999e-05 2024-09-13T13:18:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660271+0000", 2024-09-13T13:18:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.36868583799999999 2024-09-13T13:18:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660324+0000", 2024-09-13T13:18:49.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3298999999999999e-05 2024-09-13T13:18:49.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.057 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.057 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:133009 10.8 e366/334 10:1c6c448b:::benchmark_data_smithi052_153703_object5629:head v 366'8412, mlcod=366'8409)", 2024-09-13T13:18:49.057 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.292418+0000", 2024-09-13T13:18:49.057 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.756877296, 2024-09-13T13:18:49.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.380843235, 2024-09-13T13:18:49.057 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.057 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292418+0000", 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292418+0000", 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292419+0000", 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.569999999999999e-07 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292425+0000", 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2050000000000001e-06 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660256+0000", 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.36783123499999998 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660266+0000", 2024-09-13T13:18:49.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0210000000000001e-05 2024-09-13T13:18:49.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660362+0000", 2024-09-13T13:18:49.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5999e-05 2024-09-13T13:18:49.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660382+0000", 2024-09-13T13:18:49.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0035999999999999e-05 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.673210+0000", 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012828381999999999 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.673262+0000", 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1211000000000001e-05 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.062 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:133029 10.9 e366/334 10:97ab46ca:::benchmark_data_smithi052_153703_object5649:head v 366'8225, mlcod=366'8223)", 2024-09-13T13:18:49.062 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.309291+0000", 2024-09-13T13:18:49.062 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.74000490899999, 2024-09-13T13:18:49.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.356083341, 2024-09-13T13:18:49.062 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.062 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.062 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.062 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309291+0000", 2024-09-13T13:18:49.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309291+0000", 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309293+0000", 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9889999999999999e-06 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309303+0000", 2024-09-13T13:18:49.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0162e-05 2024-09-13T13:18:49.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660206+0000", 2024-09-13T13:18:49.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.35090332600000002 2024-09-13T13:18:49.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660227+0000", 2024-09-13T13:18:49.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0778999999999999e-05 2024-09-13T13:18:49.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660332+0000", 2024-09-13T13:18:49.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104756 2024-09-13T13:18:49.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660360+0000", 2024-09-13T13:18:49.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8252999999999999e-05 2024-09-13T13:18:49.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665339+0000", 2024-09-13T13:18:49.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049790090000000004 2024-09-13T13:18:49.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665374+0000", 2024-09-13T13:18:49.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5067000000000002e-05 2024-09-13T13:18:49.066 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.066 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.066 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.066 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:150454 10.9 e371/334 10:9296a054:::benchmark_data_smithi052_153703_object23074:head v 371'9301, mlcod=371'9299)", 2024-09-13T13:18:49.066 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:55.061109+0000", 2024-09-13T13:18:49.066 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 173.98818667500001, 2024-09-13T13:18:49.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.204762101, 2024-09-13T13:18:49.067 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.067 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.068 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.068 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061109+0000", 2024-09-13T13:18:49.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061109+0000", 2024-09-13T13:18:49.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061111+0000", 2024-09-13T13:18:49.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5549999999999999e-06 2024-09-13T13:18:49.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061118+0000", 2024-09-13T13:18:49.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6490000000000008e-06 2024-09-13T13:18:49.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061130+0000", 2024-09-13T13:18:49.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2072000000000001e-05 2024-09-13T13:18:49.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061138+0000", 2024-09-13T13:18:49.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.824e-06 2024-09-13T13:18:49.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061201+0000", 2024-09-13T13:18:49.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3075000000000002e-05 2024-09-13T13:18:49.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.061221+0000", 2024-09-13T13:18:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0017000000000002e-05 2024-09-13T13:18:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.066643+0000", 2024-09-13T13:18:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0054218249999999999 2024-09-13T13:18:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.265871+0000", 2024-09-13T13:18:49.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.199228084 2024-09-13T13:18:49.072 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.072 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.072 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.072 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:150456 10.1 e371/334 10:88b0c1b6:::benchmark_data_smithi052_153703_object23076:head v 371'9420, mlcod=371'9416)", 2024-09-13T13:18:49.072 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:55.062401+0000", 2024-09-13T13:18:49.072 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 173.98689478700001, 2024-09-13T13:18:49.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.203845743, 2024-09-13T13:18:49.072 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.072 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062401+0000", 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062401+0000", 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062402+0000", 2024-09-13T13:18:49.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.252e-06 2024-09-13T13:18:49.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062409+0000", 2024-09-13T13:18:49.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9700000000000002e-06 2024-09-13T13:18:49.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062419+0000", 2024-09-13T13:18:49.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0084e-05 2024-09-13T13:18:49.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062426+0000", 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0840000000000003e-06 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062491+0000", 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5096000000000002e-05 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.062509+0000", 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7453e-05 2024-09-13T13:18:49.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.266168+0000", 2024-09-13T13:18:49.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20365898599999999 2024-09-13T13:18:49.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.266247+0000", 2024-09-13T13:18:49.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8818000000000003e-05 2024-09-13T13:18:49.078 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.078 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.078 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:150461 10.9 e371/334 10:9bef7284:::benchmark_data_smithi052_153703_object23081:head v 371'9302, mlcod=371'9299)", 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:55.063189+0000", 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 173.986107078, 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20551095, 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.063189+0000", 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.063189+0000", 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.063190+0000", 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2580000000000001e-06 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.063196+0000", 2024-09-13T13:18:49.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9959999999999999e-06 2024-09-13T13:18:49.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.063205+0000", 2024-09-13T13:18:49.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3170000000000008e-06 2024-09-13T13:18:49.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.063211+0000", 2024-09-13T13:18:49.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.913e-06 2024-09-13T13:18:49.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.064039+0000", 2024-09-13T13:18:49.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00082804099999999996 2024-09-13T13:18:49.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.064056+0000", 2024-09-13T13:18:49.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6917000000000001e-05 2024-09-13T13:18:49.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.268660+0000", 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20460398499999999 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.268700+0000", 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9523000000000001e-05 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.083 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:150470 10.8 e371/334 10:1ddbf082:::benchmark_data_smithi052_153703_object23090:head v 371'9514, mlcod=371'9512)", 2024-09-13T13:18:49.084 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:55.069046+0000", 2024-09-13T13:18:49.084 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 173.980249534, 2024-09-13T13:18:49.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.204381533, 2024-09-13T13:18:49.084 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.084 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.084 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.084 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.069046+0000", 2024-09-13T13:18:49.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.069046+0000", 2024-09-13T13:18:49.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.069048+0000", 2024-09-13T13:18:49.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3799999999999999e-06 2024-09-13T13:18:49.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.069055+0000", 2024-09-13T13:18:49.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8809999999999996e-06 2024-09-13T13:18:49.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.265825+0000", 2024-09-13T13:18:49.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.196769743 2024-09-13T13:18:49.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.265841+0000", 2024-09-13T13:18:49.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6076999999999999e-05 2024-09-13T13:18:49.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.265892+0000", 2024-09-13T13:18:49.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0534000000000001e-05 2024-09-13T13:18:49.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.265915+0000", 2024-09-13T13:18:49.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2928000000000001e-05 2024-09-13T13:18:49.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.273300+0000", 2024-09-13T13:18:49.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0073855659999999997 2024-09-13T13:18:49.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:55.273428+0000", 2024-09-13T13:18:49.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012742400000000001 2024-09-13T13:18:49.088 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.088 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.088 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.088 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:191396 10.9 e382/377 10:9fe2a3d5:::benchmark_data_smithi052_153703_object64016:head v 382'11888, mlcod=382'11886)", 2024-09-13T13:18:49.088 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:34.022668+0000", 2024-09-13T13:18:49.088 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 135.02662788999999, 2024-09-13T13:18:49.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.227577587, 2024-09-13T13:18:49.089 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.089 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.089 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.089 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022668+0000", 2024-09-13T13:18:49.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022668+0000", 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022669+0000", 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.519e-06 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022677+0000", 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3889999999999999e-06 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022687+0000", 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0190999999999999e-05 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022693+0000", 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5080000000000002e-06 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022747+0000", 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3820000000000003e-05 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022769+0000", 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1829000000000001e-05 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.026913+0000", 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041442939999999998 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250245+0000", 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.22333203700000001 2024-09-13T13:18:49.092 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.093 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.093 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.093 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:191388 10.c e382/334 10:3518f5d7:::benchmark_data_smithi052_153703_object64008:head v 382'11871, mlcod=382'11868)", 2024-09-13T13:18:49.093 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:34.028033+0000", 2024-09-13T13:18:49.093 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 135.02126242700001, 2024-09-13T13:18:49.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.237613568, 2024-09-13T13:18:49.093 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.093 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.093 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.093 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028033+0000", 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028033+0000", 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028035+0000", 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8649999999999999e-06 2024-09-13T13:18:49.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028044+0000", 2024-09-13T13:18:49.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7630000000000006e-06 2024-09-13T13:18:49.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250215+0000", 2024-09-13T13:18:49.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.222171601 2024-09-13T13:18:49.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250226+0000", 2024-09-13T13:18:49.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0395e-05 2024-09-13T13:18:49.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250281+0000", 2024-09-13T13:18:49.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4833999999999997e-05 2024-09-13T13:18:49.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250301+0000", 2024-09-13T13:18:49.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0452999999999999e-05 2024-09-13T13:18:49.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.265605+0000", 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01530372 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.265647+0000", 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1937000000000002e-05 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.097 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:191418 10.9 e382/377 10:975b1348:::benchmark_data_smithi052_153703_object64038:head v 382'11889, mlcod=382'11886)", 2024-09-13T13:18:49.098 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:34.041411+0000", 2024-09-13T13:18:49.098 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 135.00788517199999, 2024-09-13T13:18:49.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.214851654, 2024-09-13T13:18:49.098 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.098 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.098 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.098 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041411+0000", 2024-09-13T13:18:49.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041411+0000", 2024-09-13T13:18:49.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041413+0000", 2024-09-13T13:18:49.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3819999999999998e-06 2024-09-13T13:18:49.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041424+0000", 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1031e-05 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250212+0000", 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.208787785 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250227+0000", 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4887e-05 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250280+0000", 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3733999999999997e-05 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250340+0000", 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9212000000000002e-05 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.256042+0000", 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057022210000000004 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.256262+0000", 2024-09-13T13:18:49.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022040199999999999 2024-09-13T13:18:49.102 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.102 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.102 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.102 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203978 10.9 e385/377 10:9bb10788:::benchmark_data_smithi052_153703_object76598:head v 385'12669, mlcod=385'12664)", 2024-09-13T13:18:49.102 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.815708+0000", 2024-09-13T13:18:49.102 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.23358733000001, 2024-09-13T13:18:49.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27159179500000002, 2024-09-13T13:18:49.102 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.103 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.103 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.103 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815708+0000", 2024-09-13T13:18:49.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815708+0000", 2024-09-13T13:18:49.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815710+0000", 2024-09-13T13:18:49.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8840000000000001e-06 2024-09-13T13:18:49.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815718+0000", 2024-09-13T13:18:49.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:18:49.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815731+0000", 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2853e-05 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815739+0000", 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6550000000000004e-06 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816390+0000", 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00065173800000000001 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816411+0000", 2024-09-13T13:18:49.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0727e-05 2024-09-13T13:18:49.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087219+0000", 2024-09-13T13:18:49.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27080745899999997 2024-09-13T13:18:49.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087300+0000", 2024-09-13T13:18:49.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1477999999999995e-05 2024-09-13T13:18:49.106 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.107 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.107 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.107 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203975 10.1 e385/334 10:87de9ad5:::benchmark_data_smithi052_153703_object76595:head v 385'12786, mlcod=385'12784)", 2024-09-13T13:18:49.107 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:18:49.107 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.232513923, 2024-09-13T13:18:49.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27055213299999997, 2024-09-13T13:18:49.107 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.107 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.107 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.107 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:18:49.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:18:49.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:18:49.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4199999999999995e-07 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816796+0000", 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3298e-05 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816803+0000", 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8379999999999996e-06 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816808+0000", 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2020000000000003e-06 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079290+0000", 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.26248217499999998 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079309+0000", 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8927e-05 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087304+0000", 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079953999999999997 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087334+0000", 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9550999999999999e-05 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203977 10.c e385/334 10:35b89a8a:::benchmark_data_smithi052_153703_object76597:head v 385'12628, mlcod=385'12626)", 2024-09-13T13:18:49.111 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.816846+0000", 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.232449657, 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27022401899999998, 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816846+0000", 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816846+0000", 2024-09-13T13:18:49.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816847+0000", 2024-09-13T13:18:49.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9200000000000001e-07 2024-09-13T13:18:49.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816851+0000", 2024-09-13T13:18:49.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1989999999999999e-06 2024-09-13T13:18:49.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816857+0000", 2024-09-13T13:18:49.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9610000000000002e-06 2024-09-13T13:18:49.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816862+0000", 2024-09-13T13:18:49.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8489999999999997e-06 2024-09-13T13:18:49.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816923+0000", 2024-09-13T13:18:49.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1781999999999999e-05 2024-09-13T13:18:49.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816940+0000", 2024-09-13T13:18:49.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6844999999999999e-05 2024-09-13T13:18:49.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087036+0000", 2024-09-13T13:18:49.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27009603100000001 2024-09-13T13:18:49.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087070+0000", 2024-09-13T13:18:49.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3760000000000002e-05 2024-09-13T13:18:49.116 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.116 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.117 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.117 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203979 10.c e385/334 10:3e03f71a:::benchmark_data_smithi052_153703_object76599:head v 385'12629, mlcod=385'12626)", 2024-09-13T13:18:49.117 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.820677+0000", 2024-09-13T13:18:49.117 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.22861865900001, 2024-09-13T13:18:49.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.26856709400000001, 2024-09-13T13:18:49.117 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.117 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.117 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.117 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820677+0000", 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820677+0000", 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820678+0000", 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4699999999999999e-06 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820696+0000", 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7331999999999999e-05 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079076+0000", 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.25837983199999998 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079091+0000", 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5832999999999999e-05 2024-09-13T13:18:49.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.081590+0000", 2024-09-13T13:18:49.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002498545 2024-09-13T13:18:49.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.081623+0000", 2024-09-13T13:18:49.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3442000000000003e-05 2024-09-13T13:18:49.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.089192+0000", 2024-09-13T13:18:49.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075682340000000001 2024-09-13T13:18:49.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.089244+0000", 2024-09-13T13:18:49.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2405999999999999e-05 2024-09-13T13:18:49.121 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.121 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.121 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.121 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203982 10.9 e385/377 10:90b0e29e:::benchmark_data_smithi052_153703_object76602:head v 385'12670, mlcod=385'12666)", 2024-09-13T13:18:49.121 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.825124+0000", 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.22417158899999, 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.26226309199999998, 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825124+0000", 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825124+0000", 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825126+0000", 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.894e-06 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825134+0000", 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1960000000000008e-06 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079730+0000", 2024-09-13T13:18:49.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.25459629499999997 2024-09-13T13:18:49.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079741+0000", 2024-09-13T13:18:49.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-09-13T13:18:49.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079895+0000", 2024-09-13T13:18:49.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154881 2024-09-13T13:18:49.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079919+0000", 2024-09-13T13:18:49.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3139e-05 2024-09-13T13:18:49.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087337+0000", 2024-09-13T13:18:49.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.007417964 2024-09-13T13:18:49.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087387+0000", 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0612999999999998e-05 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:60365 11.0 e440/423 11:088590ea:::benchmark_data_smithi052_182575_object60364:head v 440'3937, mlcod=440'3934)", 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.475832+0000", 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.573463265999999, 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31868556799999997, 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.126 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475832+0000", 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475832+0000", 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475834+0000", 2024-09-13T13:18:49.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.643e-06 2024-09-13T13:18:49.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475869+0000", 2024-09-13T13:18:49.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4582000000000002e-05 2024-09-13T13:18:49.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475879+0000", 2024-09-13T13:18:49.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0244000000000001e-05 2024-09-13T13:18:49.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475886+0000", 2024-09-13T13:18:49.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8759999999999999e-06 2024-09-13T13:18:49.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475941+0000", 2024-09-13T13:18:49.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4975999999999997e-05 2024-09-13T13:18:49.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475960+0000", 2024-09-13T13:18:49.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9381999999999998e-05 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480677+0000", 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047167650000000004 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794518+0000", 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31384109999999998 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:60369 11.0 e440/423 11:00a94002:::benchmark_data_smithi052_182575_object60368:head v 440'3938, mlcod=440'3935)", 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.478361+0000", 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.570934250000001, 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31619510000000001, 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478361+0000", 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478361+0000", 2024-09-13T13:18:49.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478363+0000", 2024-09-13T13:18:49.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3540000000000001e-06 2024-09-13T13:18:49.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478408+0000", 2024-09-13T13:18:49.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5260999999999999e-05 2024-09-13T13:18:49.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478417+0000", 2024-09-13T13:18:49.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7690000000000002e-06 2024-09-13T13:18:49.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478423+0000", 2024-09-13T13:18:49.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.331e-06 2024-09-13T13:18:49.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478857+0000", 2024-09-13T13:18:49.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043396400000000001 2024-09-13T13:18:49.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478875+0000", 2024-09-13T13:18:49.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7611000000000001e-05 2024-09-13T13:18:49.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794529+0000", 2024-09-13T13:18:49.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31565414899999999 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794557+0000", 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7661000000000002e-05 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:60375 11.5 e440/423 11:ab976586:::benchmark_data_smithi052_182575_object60374:head v 440'3806, mlcod=440'3804)", 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.480358+0000", 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.568937305, 2024-09-13T13:18:49.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31425154100000002, 2024-09-13T13:18:49.136 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.136 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.136 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.136 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480358+0000", 2024-09-13T13:18:49.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480358+0000", 2024-09-13T13:18:49.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480360+0000", 2024-09-13T13:18:49.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.139e-06 2024-09-13T13:18:49.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480381+0000", 2024-09-13T13:18:49.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1444e-05 2024-09-13T13:18:49.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480390+0000", 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480396+0000", 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2389999999999999e-06 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480450+0000", 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3616000000000003e-05 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480467+0000", 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7686e-05 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794579+0000", 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31411161500000001 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794610+0000", 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1027e-05 2024-09-13T13:18:49.139 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:60372 11.0 e440/423 11:055cf520:::benchmark_data_smithi052_182575_object60371:head v 440'3939, mlcod=440'3935)", 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.480901+0000", 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.568394199, 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31806362700000002, 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480901+0000", 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480901+0000", 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480903+0000", 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1969999999999999e-06 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480964+0000", 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1058999999999999e-05 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794586+0000", 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31362191099999998 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794593+0000", 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7139999999999993e-06 2024-09-13T13:18:49.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794668+0000", 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4833000000000007e-05 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794690+0000", 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2263e-05 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.798931+0000", 2024-09-13T13:18:49.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042401610000000001 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.798965+0000", 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4489000000000001e-05 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "replica_scrub(pg: 4.3,from:0'0,to:268'29,epoch:440/61,start:4:c0000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.481428+0000", 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.567867651, 2024-09-13T13:18:49.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.313186768, 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.481428+0000", 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.481428+0000", 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:49.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:49.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.481429+0000", 2024-09-13T13:18:49.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.178e-06 2024-09-13T13:18:49.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:49.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.481435+0000", 2024-09-13T13:18:49.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6860000000000003e-06 2024-09-13T13:18:49.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:49.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794453+0000", 2024-09-13T13:18:49.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31301767800000002 2024-09-13T13:18:49.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:49.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794474+0000", 2024-09-13T13:18:49.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1761999999999999e-05 2024-09-13T13:18:49.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:49.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794570+0000", 2024-09-13T13:18:49.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6155000000000003e-05 2024-09-13T13:18:49.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:49.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794584+0000", 2024-09-13T13:18:49.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3613e-05 2024-09-13T13:18:49.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:49.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:49.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:18:49.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794615+0000", 2024-09-13T13:18:49.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0695999999999999e-05 2024-09-13T13:18:49.148 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.149 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.149 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.149 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:49.149 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:49.149 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:18:49.149 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:49.192 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:49.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:49.294 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:49.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:49.334 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:49.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:49.435 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:49.469 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:49.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:49.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:49.571 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:49.606 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:49.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:49.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:49.808 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:49.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:49.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:49.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:49.885 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 49 16 79305 79289 101.106 70.875 0.00996545 0.00988584 2024-09-13T13:18:49.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:49.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[36682]: pgmap v894: 158 pgs: 158 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-13T13:18:49.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[36682]: 5.0 scrub starts 2024-09-13T13:18:49.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[36682]: 5.0 scrub ok 2024-09-13T13:18:49.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[36682]: 6.8 scrub starts 2024-09-13T13:18:49.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[36682]: 6.8 scrub ok 2024-09-13T13:18:49.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:18:49.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:18:49.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[36682]: 6.1a scrub starts 2024-09-13T13:18:49.950 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[36682]: 6.1a scrub ok 2024-09-13T13:18:49.951 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[29471]: pgmap v894: 158 pgs: 158 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-13T13:18:49.951 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[29471]: 5.0 scrub starts 2024-09-13T13:18:49.951 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[29471]: 5.0 scrub ok 2024-09-13T13:18:49.951 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[29471]: 6.8 scrub starts 2024-09-13T13:18:49.951 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[29471]: 6.8 scrub ok 2024-09-13T13:18:49.951 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:18:49.951 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:18:49.952 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[29471]: 6.1a scrub starts 2024-09-13T13:18:49.952 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:49 smithi052 ceph-mon[29471]: 6.1a scrub ok 2024-09-13T13:18:49.964 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:49.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:49.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:50.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:50.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:49 smithi167 ceph-mon[33178]: pgmap v894: 158 pgs: 158 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-13T13:18:50.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:49 smithi167 ceph-mon[33178]: 5.0 scrub starts 2024-09-13T13:18:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:49 smithi167 ceph-mon[33178]: 5.0 scrub ok 2024-09-13T13:18:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:49 smithi167 ceph-mon[33178]: 6.8 scrub starts 2024-09-13T13:18:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:49 smithi167 ceph-mon[33178]: 6.8 scrub ok 2024-09-13T13:18:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:49 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:18:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:49 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:18:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:49 smithi167 ceph-mon[33178]: 6.1a scrub starts 2024-09-13T13:18:50.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:49 smithi167 ceph-mon[33178]: 6.1a scrub ok 2024-09-13T13:18:50.095 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:50.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:50.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:50.197 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:50.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:50.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:50.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:50.322 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:50.363 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:50.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:50.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:50.413 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:50.466 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:50.491 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:50.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:50.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:50.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:50.615 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:50.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:50.717 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:50.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:50.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:50.829 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:50 smithi052 ceph-mon[29471]: 6.c scrub starts 2024-09-13T13:18:50.829 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:50 smithi052 ceph-mon[29471]: 6.c scrub ok 2024-09-13T13:18:50.829 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:50 smithi052 ceph-mon[29471]: 1.7 scrub starts 2024-09-13T13:18:50.829 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:50 smithi052 ceph-mon[29471]: 1.7 scrub ok 2024-09-13T13:18:50.829 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:50 smithi052 ceph-mon[36682]: 6.c scrub starts 2024-09-13T13:18:50.829 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:50 smithi052 ceph-mon[36682]: 6.c scrub ok 2024-09-13T13:18:50.829 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:50 smithi052 ceph-mon[36682]: 1.7 scrub starts 2024-09-13T13:18:50.829 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:50 smithi052 ceph-mon[36682]: 1.7 scrub ok 2024-09-13T13:18:50.840 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:50.885 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:50.885 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 50 16 80209 80193 100.214 56.5 0.0170398 0.00997335 2024-09-13T13:18:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:50.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:50.987 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:51.019 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:51.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:51.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:51.028 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:50 smithi167 ceph-mon[33178]: 6.c scrub starts 2024-09-13T13:18:51.028 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:50 smithi167 ceph-mon[33178]: 6.c scrub ok 2024-09-13T13:18:51.028 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:50 smithi167 ceph-mon[33178]: 1.7 scrub starts 2024-09-13T13:18:51.028 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:50 smithi167 ceph-mon[33178]: 1.7 scrub ok 2024-09-13T13:18:51.121 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:51.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:51.165 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:51.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:51.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:51.271 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T13:18:51.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:51.301 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:51.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:51.369 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:18:51.377 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:18:51.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:51.440 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:51.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:51.542 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:51.579 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:51.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:51.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:51.681 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:51.686 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:18:51.710 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:51.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:51.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:51.715 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:18:51.768 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[36682]: 1.a scrub starts 2024-09-13T13:18:51.769 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[36682]: 1.a scrub ok 2024-09-13T13:18:51.769 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[36682]: pgmap v895: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T13:18:51.769 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[36682]: 1.0 scrub starts 2024-09-13T13:18:51.769 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[36682]: 1.0 scrub ok 2024-09-13T13:18:51.769 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1376127347' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:18:51.769 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[36682]: from='client.36722 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:18:51.769 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[36682]: 1.3 scrub starts 2024-09-13T13:18:51.769 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[36682]: 1.3 scrub ok 2024-09-13T13:18:51.769 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[36682]: 1.5 scrub starts 2024-09-13T13:18:51.769 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[36682]: 1.5 scrub ok 2024-09-13T13:18:51.769 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[36682]: 1.6 scrub starts 2024-09-13T13:18:51.770 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[36682]: 1.6 scrub ok 2024-09-13T13:18:51.770 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[29471]: 1.a scrub starts 2024-09-13T13:18:51.770 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[29471]: 1.a scrub ok 2024-09-13T13:18:51.770 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[29471]: pgmap v895: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T13:18:51.770 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[29471]: 1.0 scrub starts 2024-09-13T13:18:51.770 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[29471]: 1.0 scrub ok 2024-09-13T13:18:51.770 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1376127347' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:18:51.770 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[29471]: from='client.36722 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:18:51.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[29471]: 1.3 scrub starts 2024-09-13T13:18:51.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[29471]: 1.3 scrub ok 2024-09-13T13:18:51.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[29471]: 1.5 scrub starts 2024-09-13T13:18:51.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[29471]: 1.5 scrub ok 2024-09-13T13:18:51.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[29471]: 1.6 scrub starts 2024-09-13T13:18:51.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:51 smithi052 ceph-mon[29471]: 1.6 scrub ok 2024-09-13T13:18:51.809 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:18:51.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:51.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:51.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:51.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:51.885 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 51 16 81329 81313 99.6216 70 0.0137574 0.0100331 2024-09-13T13:18:51.945 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:51.966 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:51.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:52.068 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:52.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:51 smithi167 ceph-mon[33178]: 1.a scrub starts 2024-09-13T13:18:52.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:51 smithi167 ceph-mon[33178]: 1.a scrub ok 2024-09-13T13:18:52.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:51 smithi167 ceph-mon[33178]: pgmap v895: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-13T13:18:52.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:51 smithi167 ceph-mon[33178]: 1.0 scrub starts 2024-09-13T13:18:52.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:51 smithi167 ceph-mon[33178]: 1.0 scrub ok 2024-09-13T13:18:52.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:51 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1376127347' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:18:52.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:51 smithi167 ceph-mon[33178]: from='client.36722 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:18:52.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:51 smithi167 ceph-mon[33178]: 1.3 scrub starts 2024-09-13T13:18:52.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:51 smithi167 ceph-mon[33178]: 1.3 scrub ok 2024-09-13T13:18:52.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:51 smithi167 ceph-mon[33178]: 1.5 scrub starts 2024-09-13T13:18:52.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:51 smithi167 ceph-mon[33178]: 1.5 scrub ok 2024-09-13T13:18:52.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:51 smithi167 ceph-mon[33178]: 1.6 scrub starts 2024-09-13T13:18:52.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:51 smithi167 ceph-mon[33178]: 1.6 scrub ok 2024-09-13T13:18:52.097 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:52.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:52.199 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:52.249 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":895,"stamp":"2024-09-13T13:18:50.003706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5069621909,"num_objects":77544,"num_object_clones":0,"num_object_copies":232632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77544,"num_whiteouts":0,"num_read":1605,"num_read_kb":1454,"num_write":78169,"num_write_kb":4951541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":871137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77098,"ondisk_log_size":77098,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":475,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19988372,"kb_used_data":14819396,"kb_used_omap":15,"kb_used_meta":5168816,"kb_avail":729972844,"statfs":{"total":767960285184,"available":747492192256,"internally_reserved":0,"allocated":15175061504,"data_stored":15170864255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":5292868377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":686882816,"num_objects":10481,"num_object_clones":0,"num_object_copies":31443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10481,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10481,"num_write_kb":670784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006971"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":842,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.675693+0000","last_change":"2024-09-13T13:18:32.440533+0000","last_active":"2024-09-13T13:18:47.675693+0000","last_peered":"2024-09-13T13:18:47.675693+0000","last_clean":"2024-09-13T13:18:47.675693+0000","last_became_active":"2024-09-13T13:17:25.335900+0000","last_became_peered":"2024-09-13T13:17:25.335900+0000","last_unstale":"2024-09-13T13:18:47.675693+0000","last_undegraded":"2024-09-13T13:18:47.675693+0000","last_fullsized":"2024-09-13T13:18:47.675693+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:32.440448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:32.440448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:57.101135+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":318,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.659840+0000","last_change":"2024-09-13T13:18:08.548626+0000","last_active":"2024-09-13T13:18:47.659840+0000","last_peered":"2024-09-13T13:18:47.659840+0000","last_clean":"2024-09-13T13:18:47.659840+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:18:47.659840+0000","last_undegraded":"2024-09-13T13:18:47.659840+0000","last_fullsized":"2024-09-13T13:18:47.659840+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:18:08.548556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:08.548556+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:22.274215+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":829,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.675566+0000","last_change":"2024-09-13T13:17:56.109579+0000","last_active":"2024-09-13T13:18:47.675566+0000","last_peered":"2024-09-13T13:18:47.675566+0000","last_clean":"2024-09-13T13:18:47.675566+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:18:47.675566+0000","last_undegraded":"2024-09-13T13:18:47.675566+0000","last_fullsized":"2024-09-13T13:18:47.675566+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:56.109526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:56.109526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:00.485890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":308,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660172+0000","last_change":"2024-09-13T13:15:51.244938+0000","last_active":"2024-09-13T13:18:47.660172+0000","last_peered":"2024-09-13T13:18:47.660172+0000","last_clean":"2024-09-13T13:18:47.660172+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:18:47.660172+0000","last_undegraded":"2024-09-13T13:18:47.660172+0000","last_fullsized":"2024-09-13T13:18:47.660172+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:51.244870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:51.244870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":819,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.666135+0000","last_change":"2024-09-13T13:11:47.852762+0000","last_active":"2024-09-13T13:18:47.666135+0000","last_peered":"2024-09-13T13:18:47.666135+0000","last_clean":"2024-09-13T13:18:47.666135+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:18:47.666135+0000","last_undegraded":"2024-09-13T13:18:47.666135+0000","last_fullsized":"2024-09-13T13:18:47.666135+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:26.862207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":862,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.667712+0000","last_change":"2024-09-13T13:17:25.243843+0000","last_active":"2024-09-13T13:18:47.667712+0000","last_peered":"2024-09-13T13:18:47.667712+0000","last_clean":"2024-09-13T13:18:47.667712+0000","last_became_active":"2024-09-13T13:17:25.243371+0000","last_became_peered":"2024-09-13T13:17:25.243371+0000","last_unstale":"2024-09-13T13:18:47.667712+0000","last_undegraded":"2024-09-13T13:18:47.667712+0000","last_fullsized":"2024-09-13T13:18:47.667712+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:08.938750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:08.938750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":937,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.668195+0000","last_change":"2024-09-13T13:17:25.262965+0000","last_active":"2024-09-13T13:18:47.668195+0000","last_peered":"2024-09-13T13:18:47.668195+0000","last_clean":"2024-09-13T13:18:47.668195+0000","last_became_active":"2024-09-13T13:17:25.245131+0000","last_became_peered":"2024-09-13T13:17:25.245131+0000","last_unstale":"2024-09-13T13:18:47.668195+0000","last_undegraded":"2024-09-13T13:18:47.668195+0000","last_fullsized":"2024-09-13T13:18:47.668195+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:17:11.044472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:11.044472+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":292,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.659835+0000","last_change":"2024-09-13T13:12:52.997865+0000","last_active":"2024-09-13T13:18:47.659835+0000","last_peered":"2024-09-13T13:18:47.659835+0000","last_clean":"2024-09-13T13:18:47.659835+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:18:47.659835+0000","last_undegraded":"2024-09-13T13:18:47.659835+0000","last_fullsized":"2024-09-13T13:18:47.659835+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:17.305608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:17.305608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.504193+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":883,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664546+0000","last_change":"2024-09-13T13:13:41.518378+0000","last_active":"2024-09-13T13:18:47.664546+0000","last_peered":"2024-09-13T13:18:47.664546+0000","last_clean":"2024-09-13T13:18:47.664546+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:18:47.664546+0000","last_undegraded":"2024-09-13T13:18:47.664546+0000","last_fullsized":"2024-09-13T13:18:47.664546+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:41.518322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.518322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":310,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660217+0000","last_change":"2024-09-13T13:17:06.406253+0000","last_active":"2024-09-13T13:18:47.660217+0000","last_peered":"2024-09-13T13:18:47.660217+0000","last_clean":"2024-09-13T13:18:47.660217+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:18:47.660217+0000","last_undegraded":"2024-09-13T13:18:47.660217+0000","last_fullsized":"2024-09-13T13:18:47.660217+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:06.406179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:06.406179+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":828,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.663941+0000","last_change":"2024-09-13T13:17:31.089557+0000","last_active":"2024-09-13T13:18:47.663941+0000","last_peered":"2024-09-13T13:18:47.663941+0000","last_clean":"2024-09-13T13:18:47.663941+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:18:47.663941+0000","last_undegraded":"2024-09-13T13:18:47.663941+0000","last_fullsized":"2024-09-13T13:18:47.663941+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:31.089505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:31.089505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:50.762733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":831,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.671952+0000","last_change":"2024-09-13T13:17:32.559966+0000","last_active":"2024-09-13T13:18:47.671952+0000","last_peered":"2024-09-13T13:18:47.671952+0000","last_clean":"2024-09-13T13:18:47.671952+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:18:47.671952+0000","last_undegraded":"2024-09-13T13:18:47.671952+0000","last_fullsized":"2024-09-13T13:18:47.671952+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:32.559884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:32.559884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":858,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664274+0000","last_change":"2024-09-13T13:17:02.211078+0000","last_active":"2024-09-13T13:18:47.664274+0000","last_peered":"2024-09-13T13:18:47.664274+0000","last_clean":"2024-09-13T13:18:47.664274+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:18:47.664274+0000","last_undegraded":"2024-09-13T13:18:47.664274+0000","last_fullsized":"2024-09-13T13:18:47.664274+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:02.211007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.211007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:26.585228+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":306,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660305+0000","last_change":"2024-09-13T13:15:03.348187+0000","last_active":"2024-09-13T13:18:47.660305+0000","last_peered":"2024-09-13T13:18:47.660305+0000","last_clean":"2024-09-13T13:18:47.660305+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:18:47.660305+0000","last_undegraded":"2024-09-13T13:18:47.660305+0000","last_fullsized":"2024-09-13T13:18:47.660305+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:15:03.348115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.348115+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:10.491929+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":841,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672896+0000","last_change":"2024-09-13T13:13:54.122282+0000","last_active":"2024-09-13T13:18:47.672896+0000","last_peered":"2024-09-13T13:18:47.672896+0000","last_clean":"2024-09-13T13:18:47.672896+0000","last_became_active":"2024-09-13T13:12:52.999970+0000","last_became_peered":"2024-09-13T13:12:52.999970+0000","last_unstale":"2024-09-13T13:18:47.672896+0000","last_undegraded":"2024-09-13T13:18:47.672896+0000","last_fullsized":"2024-09-13T13:18:47.672896+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.122222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.122222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:19.834956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":808,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.663873+0000","last_change":"2024-09-13T13:18:34.846273+0000","last_active":"2024-09-13T13:18:47.663873+0000","last_peered":"2024-09-13T13:18:47.663873+0000","last_clean":"2024-09-13T13:18:47.663873+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:18:47.663873+0000","last_undegraded":"2024-09-13T13:18:47.663873+0000","last_fullsized":"2024-09-13T13:18:47.663873+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:34.846222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:34.846222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:51.829884+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":304,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660792+0000","last_change":"2024-09-13T13:17:07.444148+0000","last_active":"2024-09-13T13:18:47.660792+0000","last_peered":"2024-09-13T13:18:47.660792+0000","last_clean":"2024-09-13T13:18:47.660792+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:18:47.660792+0000","last_undegraded":"2024-09-13T13:18:47.660792+0000","last_fullsized":"2024-09-13T13:18:47.660792+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:07.443951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:07.443951+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":857,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672540+0000","last_change":"2024-09-13T13:18:45.471190+0000","last_active":"2024-09-13T13:18:47.672540+0000","last_peered":"2024-09-13T13:18:47.672540+0000","last_clean":"2024-09-13T13:18:47.672540+0000","last_became_active":"2024-09-13T13:12:53.008505+0000","last_became_peered":"2024-09-13T13:12:53.008505+0000","last_unstale":"2024-09-13T13:18:47.672540+0000","last_undegraded":"2024-09-13T13:18:47.672540+0000","last_fullsized":"2024-09-13T13:18:47.672540+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:45.470790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.470790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:15.409504+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":108,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.661030+0000","last_change":"2024-09-13T13:18:37.714594+0000","last_active":"2024-09-13T13:18:47.661030+0000","last_peered":"2024-09-13T13:18:47.661030+0000","last_clean":"2024-09-13T13:18:47.661030+0000","last_became_active":"2024-09-13T13:17:25.245658+0000","last_became_peered":"2024-09-13T13:17:25.245658+0000","last_unstale":"2024-09-13T13:18:47.661030+0000","last_undegraded":"2024-09-13T13:18:47.661030+0000","last_fullsized":"2024-09-13T13:18:47.661030+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:18:37.714503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:37.714503+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:41.615700+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":851,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.686133+0000","last_change":"2024-09-13T13:16:54.229157+0000","last_active":"2024-09-13T13:18:47.686133+0000","last_peered":"2024-09-13T13:18:47.686133+0000","last_clean":"2024-09-13T13:18:47.686133+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:18:47.686133+0000","last_undegraded":"2024-09-13T13:18:47.686133+0000","last_fullsized":"2024-09-13T13:18:47.686133+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:54.229075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:54.229075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:17.035251+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":828,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.667660+0000","last_change":"2024-09-13T13:18:41.949104+0000","last_active":"2024-09-13T13:18:47.667660+0000","last_peered":"2024-09-13T13:18:47.667660+0000","last_clean":"2024-09-13T13:18:47.667660+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:18:47.667660+0000","last_undegraded":"2024-09-13T13:18:47.667660+0000","last_fullsized":"2024-09-13T13:18:47.667660+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:41.949029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.949029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:49.617995+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":330,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660115+0000","last_change":"2024-09-13T13:18:46.181826+0000","last_active":"2024-09-13T13:18:47.660115+0000","last_peered":"2024-09-13T13:18:47.660115+0000","last_clean":"2024-09-13T13:18:47.660115+0000","last_became_active":"2024-09-13T13:17:25.244839+0000","last_became_peered":"2024-09-13T13:17:25.244839+0000","last_unstale":"2024-09-13T13:18:47.660115+0000","last_undegraded":"2024-09-13T13:18:47.660115+0000","last_fullsized":"2024-09-13T13:18:47.660115+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:46.181741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.181741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:01.669320+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":913,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.667099+0000","last_change":"2024-09-13T13:18:35.281750+0000","last_active":"2024-09-13T13:18:47.667099+0000","last_peered":"2024-09-13T13:18:47.667099+0000","last_clean":"2024-09-13T13:18:47.667099+0000","last_became_active":"2024-09-13T13:18:35.239702+0000","last_became_peered":"2024-09-13T13:18:35.239702+0000","last_unstale":"2024-09-13T13:18:47.667099+0000","last_undegraded":"2024-09-13T13:18:47.667099+0000","last_fullsized":"2024-09-13T13:18:47.667099+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:17:56.001763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:56.001763+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:56.286987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":324,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660336+0000","last_change":"2024-09-13T13:18:32.362014+0000","last_active":"2024-09-13T13:18:47.660336+0000","last_peered":"2024-09-13T13:18:47.660336+0000","last_clean":"2024-09-13T13:18:47.660336+0000","last_became_active":"2024-09-13T13:17:25.245239+0000","last_became_peered":"2024-09-13T13:17:25.245239+0000","last_unstale":"2024-09-13T13:18:47.660336+0000","last_undegraded":"2024-09-13T13:18:47.660336+0000","last_fullsized":"2024-09-13T13:18:47.660336+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:32.361971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:32.361971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:43.233668+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":357,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664024+0000","last_change":"2024-09-13T13:17:25.339695+0000","last_active":"2024-09-13T13:18:47.664024+0000","last_peered":"2024-09-13T13:18:47.664024+0000","last_clean":"2024-09-13T13:18:47.664024+0000","last_became_active":"2024-09-13T13:17:25.329235+0000","last_became_peered":"2024-09-13T13:17:25.329235+0000","last_unstale":"2024-09-13T13:18:47.664024+0000","last_undegraded":"2024-09-13T13:18:47.664024+0000","last_fullsized":"2024-09-13T13:18:47.664024+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:05.207445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.207445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:29.581260+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":852,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.675698+0000","last_change":"2024-09-13T13:17:01.162092+0000","last_active":"2024-09-13T13:18:47.675698+0000","last_peered":"2024-09-13T13:18:47.675698+0000","last_clean":"2024-09-13T13:18:47.675698+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:18:47.675698+0000","last_undegraded":"2024-09-13T13:18:47.675698+0000","last_fullsized":"2024-09-13T13:18:47.675698+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:17:01.162018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:01.162018+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:25.064905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":878,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.677131+0000","last_change":"2024-09-13T13:18:41.472174+0000","last_active":"2024-09-13T13:18:47.677131+0000","last_peered":"2024-09-13T13:18:47.677131+0000","last_clean":"2024-09-13T13:18:47.677131+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:18:47.677131+0000","last_undegraded":"2024-09-13T13:18:47.677131+0000","last_fullsized":"2024-09-13T13:18:47.677131+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:18:41.472008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.472008+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:56.597353+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"306'12","reported_seq":1020,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:43.638641+0000","last_change":"2024-09-13T13:14:21.530680+0000","last_active":"2024-09-13T13:18:43.638641+0000","last_peered":"2024-09-13T13:18:43.638641+0000","last_clean":"2024-09-13T13:18:43.638641+0000","last_became_active":"2024-09-13T13:12:52.947886+0000","last_became_peered":"2024-09-13T13:12:52.947886+0000","last_unstale":"2024-09-13T13:18:43.638641+0000","last_undegraded":"2024-09-13T13:18:43.638641+0000","last_fullsized":"2024-09-13T13:18:43.638641+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'12","last_scrub_stamp":"2024-09-13T13:14:21.530612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:21.530612+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:47.451147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"445'5083","reported_seq":5138,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:49.181699+0000","last_change":"2024-09-13T13:17:55.200504+0000","last_active":"2024-09-13T13:18:49.181699+0000","last_peered":"2024-09-13T13:18:49.181699+0000","last_clean":"2024-09-13T13:18:49.181699+0000","last_became_active":"2024-09-13T13:17:55.200162+0000","last_became_peered":"2024-09-13T13:17:55.200162+0000","last_unstale":"2024-09-13T13:18:49.181699+0000","last_undegraded":"2024-09-13T13:18:49.181699+0000","last_fullsized":"2024-09-13T13:18:49.181699+0000","mapping_epoch":423,"log_start":"427'300","ondisk_log_start":"427'300","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:09.157335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333119488,"num_objects":5083,"num_object_clones":0,"num_object_copies":15249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5083,"num_write_kb":325312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":66,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.663710+0000","last_change":"2024-09-13T13:18:21.084865+0000","last_active":"2024-09-13T13:18:47.663710+0000","last_peered":"2024-09-13T13:18:47.663710+0000","last_clean":"2024-09-13T13:18:47.663710+0000","last_became_active":"2024-09-13T13:17:49.052966+0000","last_became_peered":"2024-09-13T13:17:49.052966+0000","last_unstale":"2024-09-13T13:18:47.663710+0000","last_undegraded":"2024-09-13T13:18:47.663710+0000","last_fullsized":"2024-09-13T13:18:47.663710+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:18:49.460502+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":930,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.687761+0000","last_change":"2024-09-13T13:18:05.102318+0000","last_active":"2024-09-13T13:18:47.687761+0000","last_peered":"2024-09-13T13:18:47.687761+0000","last_clean":"2024-09-13T13:18:47.687761+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:18:47.687761+0000","last_undegraded":"2024-09-13T13:18:47.687761+0000","last_fullsized":"2024-09-13T13:18:47.687761+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:18:05.102223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:05.102223+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:29.926605+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":853,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.667157+0000","last_change":"2024-09-13T13:16:58.980151+0000","last_active":"2024-09-13T13:18:47.667157+0000","last_peered":"2024-09-13T13:18:47.667157+0000","last_clean":"2024-09-13T13:18:47.667157+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:18:47.667157+0000","last_undegraded":"2024-09-13T13:18:47.667157+0000","last_fullsized":"2024-09-13T13:18:47.667157+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:58.980098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.980098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":376,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660838+0000","last_change":"2024-09-13T13:15:32.212185+0000","last_active":"2024-09-13T13:18:47.660838+0000","last_peered":"2024-09-13T13:18:47.660838+0000","last_clean":"2024-09-13T13:18:47.660838+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:18:47.660838+0000","last_undegraded":"2024-09-13T13:18:47.660838+0000","last_fullsized":"2024-09-13T13:18:47.660838+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:15:32.212101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.212101+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:25.305722+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":824,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664120+0000","last_change":"2024-09-13T13:13:57.546725+0000","last_active":"2024-09-13T13:18:47.664120+0000","last_peered":"2024-09-13T13:18:47.664120+0000","last_clean":"2024-09-13T13:18:47.664120+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:18:47.664120+0000","last_undegraded":"2024-09-13T13:18:47.664120+0000","last_fullsized":"2024-09-13T13:18:47.664120+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.546671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.546671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"445'4959","reported_seq":5014,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:49.214248+0000","last_change":"2024-09-13T13:17:55.200709+0000","last_active":"2024-09-13T13:18:49.214248+0000","last_peered":"2024-09-13T13:18:49.214248+0000","last_clean":"2024-09-13T13:18:49.214248+0000","last_became_active":"2024-09-13T13:17:55.200283+0000","last_became_peered":"2024-09-13T13:17:55.200283+0000","last_unstale":"2024-09-13T13:18:49.214248+0000","last_undegraded":"2024-09-13T13:18:49.214248+0000","last_fullsized":"2024-09-13T13:18:49.214248+0000","mapping_epoch":423,"log_start":"427'100","ondisk_log_start":"427'100","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:00.586409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324993024,"num_objects":4959,"num_object_clones":0,"num_object_copies":14877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4959,"num_write_kb":317376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":66,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660537+0000","last_change":"2024-09-13T13:18:21.073578+0000","last_active":"2024-09-13T13:18:47.660537+0000","last_peered":"2024-09-13T13:18:47.660537+0000","last_clean":"2024-09-13T13:18:47.660537+0000","last_became_active":"2024-09-13T13:17:49.050655+0000","last_became_peered":"2024-09-13T13:17:49.050655+0000","last_unstale":"2024-09-13T13:18:47.660537+0000","last_undegraded":"2024-09-13T13:18:47.660537+0000","last_fullsized":"2024-09-13T13:18:47.660537+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:18:50.478380+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":867,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.666860+0000","last_change":"2024-09-13T13:18:36.922068+0000","last_active":"2024-09-13T13:18:47.666860+0000","last_peered":"2024-09-13T13:18:47.666860+0000","last_clean":"2024-09-13T13:18:47.666860+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:18:47.666860+0000","last_undegraded":"2024-09-13T13:18:47.666860+0000","last_fullsized":"2024-09-13T13:18:47.666860+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:18:36.921964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:36.921964+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:40.263721+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":827,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.667958+0000","last_change":"2024-09-13T13:16:26.098415+0000","last_active":"2024-09-13T13:18:47.667958+0000","last_peered":"2024-09-13T13:18:47.667958+0000","last_clean":"2024-09-13T13:18:47.667958+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:18:47.667958+0000","last_undegraded":"2024-09-13T13:18:47.667958+0000","last_fullsized":"2024-09-13T13:18:47.667958+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:16:26.098365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:26.098365+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:30.772726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":308,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.659907+0000","last_change":"2024-09-13T13:17:58.389822+0000","last_active":"2024-09-13T13:18:47.659907+0000","last_peered":"2024-09-13T13:18:47.659907+0000","last_clean":"2024-09-13T13:18:47.659907+0000","last_became_active":"2024-09-13T13:12:53.006199+0000","last_became_peered":"2024-09-13T13:12:53.006199+0000","last_unstale":"2024-09-13T13:18:47.659907+0000","last_undegraded":"2024-09-13T13:18:47.659907+0000","last_fullsized":"2024-09-13T13:18:47.659907+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:58.389763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:58.389763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:19.447538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":330,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660255+0000","last_change":"2024-09-13T13:17:59.373240+0000","last_active":"2024-09-13T13:18:47.660255+0000","last_peered":"2024-09-13T13:18:47.660255+0000","last_clean":"2024-09-13T13:18:47.660255+0000","last_became_active":"2024-09-13T13:17:25.244171+0000","last_became_peered":"2024-09-13T13:17:25.244171+0000","last_unstale":"2024-09-13T13:18:47.660255+0000","last_undegraded":"2024-09-13T13:18:47.660255+0000","last_fullsized":"2024-09-13T13:18:47.660255+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.373194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.373194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:20.574739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"445'4853","reported_seq":4908,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:48.215040+0000","last_change":"2024-09-13T13:17:55.200309+0000","last_active":"2024-09-13T13:18:48.215040+0000","last_peered":"2024-09-13T13:18:48.215040+0000","last_clean":"2024-09-13T13:18:48.215040+0000","last_became_active":"2024-09-13T13:17:55.200037+0000","last_became_peered":"2024-09-13T13:17:55.200037+0000","last_unstale":"2024-09-13T13:18:48.215040+0000","last_undegraded":"2024-09-13T13:18:48.215040+0000","last_fullsized":"2024-09-13T13:18:48.215040+0000","mapping_epoch":423,"log_start":"427'200","ondisk_log_start":"427'200","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4653,"log_dups_size":0,"ondisk_log_size":4653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:01.812502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318046208,"num_objects":4853,"num_object_clones":0,"num_object_copies":14559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4853,"num_write_kb":310592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":71,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.666367+0000","last_change":"2024-09-13T13:18:22.127461+0000","last_active":"2024-09-13T13:18:47.666367+0000","last_peered":"2024-09-13T13:18:47.666367+0000","last_clean":"2024-09-13T13:18:47.666367+0000","last_became_active":"2024-09-13T13:17:49.053135+0000","last_became_peered":"2024-09-13T13:17:49.053135+0000","last_unstale":"2024-09-13T13:18:47.666367+0000","last_undegraded":"2024-09-13T13:18:47.666367+0000","last_fullsized":"2024-09-13T13:18:47.666367+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:18:51.943236+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":316,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.659997+0000","last_change":"2024-09-13T13:13:50.716312+0000","last_active":"2024-09-13T13:18:47.659997+0000","last_peered":"2024-09-13T13:18:47.659997+0000","last_clean":"2024-09-13T13:18:47.659997+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:18:47.659997+0000","last_undegraded":"2024-09-13T13:18:47.659997+0000","last_fullsized":"2024-09-13T13:18:47.659997+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:13:50.716243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.716243+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:22.408438+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":835,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.678161+0000","last_change":"2024-09-13T13:17:02.704294+0000","last_active":"2024-09-13T13:18:47.678161+0000","last_peered":"2024-09-13T13:18:47.678161+0000","last_clean":"2024-09-13T13:18:47.678161+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:18:47.678161+0000","last_undegraded":"2024-09-13T13:18:47.678161+0000","last_fullsized":"2024-09-13T13:18:47.678161+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:17:02.704233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.704233+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:21.560467+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":1066,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.666674+0000","last_change":"2024-09-13T13:16:24.110415+0000","last_active":"2024-09-13T13:18:47.666674+0000","last_peered":"2024-09-13T13:18:47.666674+0000","last_clean":"2024-09-13T13:18:47.666674+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:18:47.666674+0000","last_undegraded":"2024-09-13T13:18:47.666674+0000","last_fullsized":"2024-09-13T13:18:47.666674+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:16:24.110369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:24.110369+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":470,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:43.643136+0000","last_change":"2024-09-13T13:17:59.355700+0000","last_active":"2024-09-13T13:18:43.643136+0000","last_peered":"2024-09-13T13:18:43.643136+0000","last_clean":"2024-09-13T13:18:43.643136+0000","last_became_active":"2024-09-13T13:17:25.244188+0000","last_became_peered":"2024-09-13T13:17:25.244188+0000","last_unstale":"2024-09-13T13:18:43.643136+0000","last_undegraded":"2024-09-13T13:18:43.643136+0000","last_fullsized":"2024-09-13T13:18:43.643136+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.355639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.355639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:00.304064+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"445'4716","reported_seq":4771,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.866354+0000","last_change":"2024-09-13T13:17:55.131505+0000","last_active":"2024-09-13T13:18:47.866354+0000","last_peered":"2024-09-13T13:18:47.866354+0000","last_clean":"2024-09-13T13:18:47.866354+0000","last_became_active":"2024-09-13T13:17:55.131230+0000","last_became_peered":"2024-09-13T13:17:55.131230+0000","last_unstale":"2024-09-13T13:18:47.866354+0000","last_undegraded":"2024-09-13T13:18:47.866354+0000","last_fullsized":"2024-09-13T13:18:47.866354+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4716,"log_dups_size":0,"ondisk_log_size":4716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:59.022573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309067776,"num_objects":4716,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4716,"num_write_kb":301824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":140,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:43.638417+0000","last_change":"2024-09-13T13:18:21.082474+0000","last_active":"2024-09-13T13:18:43.638417+0000","last_peered":"2024-09-13T13:18:43.638417+0000","last_clean":"2024-09-13T13:18:43.638417+0000","last_became_active":"2024-09-13T13:17:49.052271+0000","last_became_peered":"2024-09-13T13:17:49.052271+0000","last_unstale":"2024-09-13T13:18:43.638417+0000","last_undegraded":"2024-09-13T13:18:43.638417+0000","last_fullsized":"2024-09-13T13:18:43.638417+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:18:50.297944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":886,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.676309+0000","last_change":"2024-09-13T13:12:53.016127+0000","last_active":"2024-09-13T13:18:47.676309+0000","last_peered":"2024-09-13T13:18:47.676309+0000","last_clean":"2024-09-13T13:18:47.676309+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:18:47.676309+0000","last_undegraded":"2024-09-13T13:18:47.676309+0000","last_fullsized":"2024-09-13T13:18:47.676309+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:48.110144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":867,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664634+0000","last_change":"2024-09-13T13:17:25.331470+0000","last_active":"2024-09-13T13:18:47.664634+0000","last_peered":"2024-09-13T13:18:47.664634+0000","last_clean":"2024-09-13T13:18:47.664634+0000","last_became_active":"2024-09-13T13:17:25.331166+0000","last_became_peered":"2024-09-13T13:17:25.331166+0000","last_unstale":"2024-09-13T13:18:47.664634+0000","last_undegraded":"2024-09-13T13:18:47.664634+0000","last_fullsized":"2024-09-13T13:18:47.664634+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:04.755828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:04.755828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.345827+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":1056,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672120+0000","last_change":"2024-09-13T13:18:40.464456+0000","last_active":"2024-09-13T13:18:47.672120+0000","last_peered":"2024-09-13T13:18:47.672120+0000","last_clean":"2024-09-13T13:18:47.672120+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:18:47.672120+0000","last_undegraded":"2024-09-13T13:18:47.672120+0000","last_fullsized":"2024-09-13T13:18:47.672120+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:18:40.464398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:40.464398+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:58.756596+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":862,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672270+0000","last_change":"2024-09-13T13:18:36.466657+0000","last_active":"2024-09-13T13:18:47.672270+0000","last_peered":"2024-09-13T13:18:47.672270+0000","last_clean":"2024-09-13T13:18:47.672270+0000","last_became_active":"2024-09-13T13:17:25.252712+0000","last_became_peered":"2024-09-13T13:17:25.252712+0000","last_unstale":"2024-09-13T13:18:47.672270+0000","last_undegraded":"2024-09-13T13:18:47.672270+0000","last_fullsized":"2024-09-13T13:18:47.672270+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:36.466575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:36.466575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:56.925016+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"445'4929","reported_seq":4984,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:49.225105+0000","last_change":"2024-09-13T13:17:55.140405+0000","last_active":"2024-09-13T13:18:49.225105+0000","last_peered":"2024-09-13T13:18:49.225105+0000","last_clean":"2024-09-13T13:18:49.225105+0000","last_became_active":"2024-09-13T13:17:55.140134+0000","last_became_peered":"2024-09-13T13:17:55.140134+0000","last_unstale":"2024-09-13T13:18:49.225105+0000","last_undegraded":"2024-09-13T13:18:49.225105+0000","last_fullsized":"2024-09-13T13:18:49.225105+0000","mapping_epoch":423,"log_start":"427'100","ondisk_log_start":"427'100","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4829,"log_dups_size":0,"ondisk_log_size":4829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.372449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323026944,"num_objects":4929,"num_object_clones":0,"num_object_copies":14787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4929,"num_write_kb":315456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":140,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:43.638136+0000","last_change":"2024-09-13T13:18:22.118884+0000","last_active":"2024-09-13T13:18:43.638136+0000","last_peered":"2024-09-13T13:18:43.638136+0000","last_clean":"2024-09-13T13:18:43.638136+0000","last_became_active":"2024-09-13T13:17:49.055888+0000","last_became_peered":"2024-09-13T13:17:49.055888+0000","last_unstale":"2024-09-13T13:18:43.638136+0000","last_undegraded":"2024-09-13T13:18:43.638136+0000","last_fullsized":"2024-09-13T13:18:43.638136+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:18:51.297294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":899,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672319+0000","last_change":"2024-09-13T13:18:33.796717+0000","last_active":"2024-09-13T13:18:47.672319+0000","last_peered":"2024-09-13T13:18:47.672319+0000","last_clean":"2024-09-13T13:18:47.672319+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:18:47.672319+0000","last_undegraded":"2024-09-13T13:18:47.672319+0000","last_fullsized":"2024-09-13T13:18:47.672319+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:18:33.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.796644+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:36.008576+0000","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":844,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.678179+0000","last_change":"2024-09-13T13:17:53.069585+0000","last_active":"2024-09-13T13:18:47.678179+0000","last_peered":"2024-09-13T13:18:47.678179+0000","last_clean":"2024-09-13T13:18:47.678179+0000","last_became_active":"2024-09-13T13:17:25.332589+0000","last_became_peered":"2024-09-13T13:17:25.332589+0000","last_unstale":"2024-09-13T13:18:47.678179+0000","last_undegraded":"2024-09-13T13:18:47.678179+0000","last_fullsized":"2024-09-13T13:18:47.678179+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:53.069538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:53.069538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:19.459005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":804,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:43.638330+0000","last_change":"2024-09-13T13:17:58.379729+0000","last_active":"2024-09-13T13:18:43.638330+0000","last_peered":"2024-09-13T13:18:43.638330+0000","last_clean":"2024-09-13T13:18:43.638330+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:18:43.638330+0000","last_undegraded":"2024-09-13T13:18:43.638330+0000","last_fullsized":"2024-09-13T13:18:43.638330+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:58.379674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:58.379674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:16.089064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":851,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.675459+0000","last_change":"2024-09-13T13:17:59.178198+0000","last_active":"2024-09-13T13:18:47.675459+0000","last_peered":"2024-09-13T13:18:47.675459+0000","last_clean":"2024-09-13T13:18:47.675459+0000","last_became_active":"2024-09-13T13:17:25.244517+0000","last_became_peered":"2024-09-13T13:17:25.244517+0000","last_unstale":"2024-09-13T13:18:47.675459+0000","last_undegraded":"2024-09-13T13:18:47.675459+0000","last_fullsized":"2024-09-13T13:18:47.675459+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.178129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.178129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.299906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"445'4716","reported_seq":4771,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.902427+0000","last_change":"2024-09-13T13:17:55.146287+0000","last_active":"2024-09-13T13:18:47.902427+0000","last_peered":"2024-09-13T13:18:47.902427+0000","last_clean":"2024-09-13T13:18:47.902427+0000","last_became_active":"2024-09-13T13:17:55.145616+0000","last_became_peered":"2024-09-13T13:17:55.145616+0000","last_unstale":"2024-09-13T13:18:47.902427+0000","last_undegraded":"2024-09-13T13:18:47.902427+0000","last_fullsized":"2024-09-13T13:18:47.902427+0000","mapping_epoch":423,"log_start":"427'200","ondisk_log_start":"427'200","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4516,"log_dups_size":0,"ondisk_log_size":4516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:20.121854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309067776,"num_objects":4716,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4716,"num_write_kb":301824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":66,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.663654+0000","last_change":"2024-09-13T13:18:21.084798+0000","last_active":"2024-09-13T13:18:47.663654+0000","last_peered":"2024-09-13T13:18:47.663654+0000","last_clean":"2024-09-13T13:18:47.663654+0000","last_became_active":"2024-09-13T13:17:49.054449+0000","last_became_peered":"2024-09-13T13:17:49.054449+0000","last_unstale":"2024-09-13T13:18:47.663654+0000","last_undegraded":"2024-09-13T13:18:47.663654+0000","last_fullsized":"2024-09-13T13:18:47.663654+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:18:50.453894+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"443'43","reported_seq":43,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672509+0000","last_change":"2024-09-13T13:18:35.332086+0000","last_active":"2024-09-13T13:18:47.672509+0000","last_peered":"2024-09-13T13:18:47.672509+0000","last_clean":"2024-09-13T13:18:47.672509+0000","last_became_active":"2024-09-13T13:18:35.244311+0000","last_became_peered":"2024-09-13T13:18:35.244311+0000","last_unstale":"2024-09-13T13:18:47.672509+0000","last_undegraded":"2024-09-13T13:18:47.672509+0000","last_fullsized":"2024-09-13T13:18:47.672509+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'42","last_scrub_stamp":"2024-09-13T13:18:01.158345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:01.158345+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:27.731963+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":316,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660315+0000","last_change":"2024-09-13T13:17:52.422920+0000","last_active":"2024-09-13T13:18:47.660315+0000","last_peered":"2024-09-13T13:18:47.660315+0000","last_clean":"2024-09-13T13:18:47.660315+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:18:47.660315+0000","last_undegraded":"2024-09-13T13:18:47.660315+0000","last_fullsized":"2024-09-13T13:18:47.660315+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:52.422872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.422872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:02.132631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":848,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664074+0000","last_change":"2024-09-13T13:18:07.306441+0000","last_active":"2024-09-13T13:18:47.664074+0000","last_peered":"2024-09-13T13:18:47.664074+0000","last_clean":"2024-09-13T13:18:47.664074+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:18:47.664074+0000","last_undegraded":"2024-09-13T13:18:47.664074+0000","last_fullsized":"2024-09-13T13:18:47.664074+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:18:07.306366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:07.306366+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:17.328610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":842,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672663+0000","last_change":"2024-09-13T13:14:59.670853+0000","last_active":"2024-09-13T13:18:47.672663+0000","last_peered":"2024-09-13T13:18:47.672663+0000","last_clean":"2024-09-13T13:18:47.672663+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:18:47.672663+0000","last_undegraded":"2024-09-13T13:18:47.672663+0000","last_fullsized":"2024-09-13T13:18:47.672663+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.670779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.670779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:18.561846+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":839,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.667456+0000","last_change":"2024-09-13T13:17:05.016876+0000","last_active":"2024-09-13T13:18:47.667456+0000","last_peered":"2024-09-13T13:18:47.667456+0000","last_clean":"2024-09-13T13:18:47.667456+0000","last_became_active":"2024-09-13T13:12:52.948742+0000","last_became_peered":"2024-09-13T13:12:52.948742+0000","last_unstale":"2024-09-13T13:18:47.667456+0000","last_undegraded":"2024-09-13T13:18:47.667456+0000","last_fullsized":"2024-09-13T13:18:47.667456+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:05.016806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.016806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"445'4912","reported_seq":4967,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.946758+0000","last_change":"2024-09-13T13:17:55.132888+0000","last_active":"2024-09-13T13:18:47.946758+0000","last_peered":"2024-09-13T13:18:47.946758+0000","last_clean":"2024-09-13T13:18:47.946758+0000","last_became_active":"2024-09-13T13:17:55.132636+0000","last_became_peered":"2024-09-13T13:17:55.132636+0000","last_unstale":"2024-09-13T13:18:47.946758+0000","last_undegraded":"2024-09-13T13:18:47.946758+0000","last_fullsized":"2024-09-13T13:18:47.946758+0000","mapping_epoch":423,"log_start":"427'400","ondisk_log_start":"427'400","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4512,"log_dups_size":0,"ondisk_log_size":4512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.085485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321912832,"num_objects":4912,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":314368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":428,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.679141+0000","last_change":"2024-09-13T13:18:21.088014+0000","last_active":"2024-09-13T13:18:47.679141+0000","last_peered":"2024-09-13T13:18:47.679141+0000","last_clean":"2024-09-13T13:18:47.679141+0000","last_became_active":"2024-09-13T13:17:49.057194+0000","last_became_peered":"2024-09-13T13:17:49.057194+0000","last_unstale":"2024-09-13T13:18:47.679141+0000","last_undegraded":"2024-09-13T13:18:47.679141+0000","last_fullsized":"2024-09-13T13:18:47.679141+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:18:50.548532+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":859,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.687883+0000","last_change":"2024-09-13T13:17:32.264550+0000","last_active":"2024-09-13T13:18:47.687883+0000","last_peered":"2024-09-13T13:18:47.687883+0000","last_clean":"2024-09-13T13:18:47.687883+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:18:47.687883+0000","last_undegraded":"2024-09-13T13:18:47.687883+0000","last_fullsized":"2024-09-13T13:18:47.687883+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'17","last_scrub_stamp":"2024-09-13T13:17:32.263873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:32.263873+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:01.566837+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":871,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.684321+0000","last_change":"2024-09-13T13:18:33.892926+0000","last_active":"2024-09-13T13:18:47.684321+0000","last_peered":"2024-09-13T13:18:47.684321+0000","last_clean":"2024-09-13T13:18:47.684321+0000","last_became_active":"2024-09-13T13:17:25.246400+0000","last_became_peered":"2024-09-13T13:17:25.246400+0000","last_unstale":"2024-09-13T13:18:47.684321+0000","last_undegraded":"2024-09-13T13:18:47.684321+0000","last_fullsized":"2024-09-13T13:18:47.684321+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:33.892597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.892597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:39.873900+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":96,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660542+0000","last_change":"2024-09-13T13:18:44.719355+0000","last_active":"2024-09-13T13:18:47.660542+0000","last_peered":"2024-09-13T13:18:47.660542+0000","last_clean":"2024-09-13T13:18:47.660542+0000","last_became_active":"2024-09-13T13:17:25.252664+0000","last_became_peered":"2024-09-13T13:17:25.252664+0000","last_unstale":"2024-09-13T13:18:47.660542+0000","last_undegraded":"2024-09-13T13:18:47.660542+0000","last_fullsized":"2024-09-13T13:18:47.660542+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:18:44.719269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:44.719269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:12.759919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":92,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664343+0000","last_change":"2024-09-13T13:17:59.698099+0000","last_active":"2024-09-13T13:18:47.664343+0000","last_peered":"2024-09-13T13:18:47.664343+0000","last_clean":"2024-09-13T13:18:47.664343+0000","last_became_active":"2024-09-13T13:17:25.332669+0000","last_became_peered":"2024-09-13T13:17:25.332669+0000","last_unstale":"2024-09-13T13:18:47.664343+0000","last_undegraded":"2024-09-13T13:18:47.664343+0000","last_fullsized":"2024-09-13T13:18:47.664343+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.698047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.698047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.194640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":440,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:43.642156+0000","last_change":"2024-09-13T13:12:41.332308+0000","last_active":"2024-09-13T13:18:43.642156+0000","last_peered":"2024-09-13T13:18:43.642156+0000","last_clean":"2024-09-13T13:18:43.642156+0000","last_became_active":"2024-09-13T13:12:40.120242+0000","last_became_peered":"2024-09-13T13:12:40.120242+0000","last_unstale":"2024-09-13T13:18:43.642156+0000","last_undegraded":"2024-09-13T13:18:43.642156+0000","last_fullsized":"2024-09-13T13:18:43.642156+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:41.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:41.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:25.287809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"445'4824","reported_seq":4879,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.879875+0000","last_change":"2024-09-13T13:17:55.143535+0000","last_active":"2024-09-13T13:18:47.879875+0000","last_peered":"2024-09-13T13:18:47.879875+0000","last_clean":"2024-09-13T13:18:47.879875+0000","last_became_active":"2024-09-13T13:17:55.143252+0000","last_became_peered":"2024-09-13T13:17:55.143252+0000","last_unstale":"2024-09-13T13:18:47.879875+0000","last_undegraded":"2024-09-13T13:18:47.879875+0000","last_fullsized":"2024-09-13T13:18:47.879875+0000","mapping_epoch":423,"log_start":"427'300","ondisk_log_start":"427'300","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4524,"log_dups_size":0,"ondisk_log_size":4524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:14.112065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316145664,"num_objects":4824,"num_object_clones":0,"num_object_copies":14472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":308736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":66,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.677400+0000","last_change":"2024-09-13T13:18:21.083672+0000","last_active":"2024-09-13T13:18:47.677400+0000","last_peered":"2024-09-13T13:18:47.677400+0000","last_clean":"2024-09-13T13:18:47.677400+0000","last_became_active":"2024-09-13T13:17:49.049531+0000","last_became_peered":"2024-09-13T13:17:49.049531+0000","last_unstale":"2024-09-13T13:18:47.677400+0000","last_undegraded":"2024-09-13T13:18:47.677400+0000","last_fullsized":"2024-09-13T13:18:47.677400+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:18:50.094646+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":910,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.677188+0000","last_change":"2024-09-13T13:18:04.569761+0000","last_active":"2024-09-13T13:18:47.677188+0000","last_peered":"2024-09-13T13:18:47.677188+0000","last_clean":"2024-09-13T13:18:47.677188+0000","last_became_active":"2024-09-13T13:17:25.253120+0000","last_became_peered":"2024-09-13T13:17:25.253120+0000","last_unstale":"2024-09-13T13:18:47.677188+0000","last_undegraded":"2024-09-13T13:18:47.677188+0000","last_fullsized":"2024-09-13T13:18:47.677188+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:18:04.569670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:04.569670+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:14.912474+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":850,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.673133+0000","last_change":"2024-09-13T13:18:37.465516+0000","last_active":"2024-09-13T13:18:47.673133+0000","last_peered":"2024-09-13T13:18:47.673133+0000","last_clean":"2024-09-13T13:18:47.673133+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:18:47.673133+0000","last_undegraded":"2024-09-13T13:18:47.673133+0000","last_fullsized":"2024-09-13T13:18:47.673133+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:37.465371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:37.465371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:41.600910+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":742,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.811892+0000","last_change":"2024-09-13T13:17:25.325347+0000","last_active":"2024-09-13T13:18:47.811892+0000","last_peered":"2024-09-13T13:18:47.811892+0000","last_clean":"2024-09-13T13:18:47.811892+0000","last_became_active":"2024-09-13T13:12:39.947940+0000","last_became_peered":"2024-09-13T13:12:39.947940+0000","last_unstale":"2024-09-13T13:18:47.811892+0000","last_undegraded":"2024-09-13T13:18:47.811892+0000","last_fullsized":"2024-09-13T13:18:47.811892+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:17:25.325275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:25.325275+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:51.942457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":370,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":821,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.665161+0000","last_change":"2024-09-13T13:17:55.293423+0000","last_active":"2024-09-13T13:18:47.665161+0000","last_peered":"2024-09-13T13:18:47.665161+0000","last_clean":"2024-09-13T13:18:47.665161+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:18:47.665161+0000","last_undegraded":"2024-09-13T13:18:47.665161+0000","last_fullsized":"2024-09-13T13:18:47.665161+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:55.293375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:55.293375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:20.691046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":826,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.677729+0000","last_change":"2024-09-13T13:18:46.991251+0000","last_active":"2024-09-13T13:18:47.677729+0000","last_peered":"2024-09-13T13:18:47.677729+0000","last_clean":"2024-09-13T13:18:47.677729+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:18:47.677729+0000","last_undegraded":"2024-09-13T13:18:47.677729+0000","last_fullsized":"2024-09-13T13:18:47.677729+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:46.991076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.991076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:06.527735+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"445'4853","reported_seq":4908,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.934573+0000","last_change":"2024-09-13T13:17:55.142415+0000","last_active":"2024-09-13T13:18:47.934573+0000","last_peered":"2024-09-13T13:18:47.934573+0000","last_clean":"2024-09-13T13:18:47.934573+0000","last_became_active":"2024-09-13T13:17:55.142112+0000","last_became_peered":"2024-09-13T13:17:55.142112+0000","last_unstale":"2024-09-13T13:18:47.934573+0000","last_undegraded":"2024-09-13T13:18:47.934573+0000","last_fullsized":"2024-09-13T13:18:47.934573+0000","mapping_epoch":423,"log_start":"427'300","ondisk_log_start":"427'300","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4553,"log_dups_size":0,"ondisk_log_size":4553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:01.465992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318046208,"num_objects":4853,"num_object_clones":0,"num_object_copies":14559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4853,"num_write_kb":310592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"306'32","reported_seq":978,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:43.641781+0000","last_change":"2024-09-13T13:13:01.352416+0000","last_active":"2024-09-13T13:18:43.641781+0000","last_peered":"2024-09-13T13:18:43.641781+0000","last_clean":"2024-09-13T13:18:43.641781+0000","last_became_active":"2024-09-13T13:12:52.999924+0000","last_became_peered":"2024-09-13T13:12:52.999924+0000","last_unstale":"2024-09-13T13:18:43.641781+0000","last_undegraded":"2024-09-13T13:18:43.641781+0000","last_fullsized":"2024-09-13T13:18:43.641781+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:13:01.352337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.352337+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.222564+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":840,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.680634+0000","last_change":"2024-09-13T13:16:53.951208+0000","last_active":"2024-09-13T13:18:47.680634+0000","last_peered":"2024-09-13T13:18:47.680634+0000","last_clean":"2024-09-13T13:18:47.680634+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:18:47.680634+0000","last_undegraded":"2024-09-13T13:18:47.680634+0000","last_fullsized":"2024-09-13T13:18:47.680634+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:53.951135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:53.951135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:31.816447+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":426,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672490+0000","last_change":"2024-09-13T13:18:21.086890+0000","last_active":"2024-09-13T13:18:47.672490+0000","last_peered":"2024-09-13T13:18:47.672490+0000","last_clean":"2024-09-13T13:18:47.672490+0000","last_became_active":"2024-09-13T13:17:47.097787+0000","last_became_peered":"2024-09-13T13:17:47.097787+0000","last_unstale":"2024-09-13T13:18:47.672490+0000","last_undegraded":"2024-09-13T13:18:47.672490+0000","last_fullsized":"2024-09-13T13:18:47.672490+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:18:49.582263+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":92,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660948+0000","last_change":"2024-09-13T13:17:58.664229+0000","last_active":"2024-09-13T13:18:47.660948+0000","last_peered":"2024-09-13T13:18:47.660948+0000","last_clean":"2024-09-13T13:18:47.660948+0000","last_became_active":"2024-09-13T13:17:25.328065+0000","last_became_peered":"2024-09-13T13:17:25.328065+0000","last_unstale":"2024-09-13T13:18:47.660948+0000","last_undegraded":"2024-09-13T13:18:47.660948+0000","last_fullsized":"2024-09-13T13:18:47.660948+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:58.664177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:58.664177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:07.154650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":863,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.665104+0000","last_change":"2024-09-13T13:18:08.354464+0000","last_active":"2024-09-13T13:18:47.665104+0000","last_peered":"2024-09-13T13:18:47.665104+0000","last_clean":"2024-09-13T13:18:47.665104+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:18:47.665104+0000","last_undegraded":"2024-09-13T13:18:47.665104+0000","last_fullsized":"2024-09-13T13:18:47.665104+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:08.354394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:08.354394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:09.946043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"444'4638","reported_seq":4691,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:46.381154+0000","last_change":"2024-09-13T13:17:55.144368+0000","last_active":"2024-09-13T13:18:46.381154+0000","last_peered":"2024-09-13T13:18:46.381154+0000","last_clean":"2024-09-13T13:18:46.381154+0000","last_became_active":"2024-09-13T13:17:55.144129+0000","last_became_peered":"2024-09-13T13:17:55.144129+0000","last_unstale":"2024-09-13T13:18:46.381154+0000","last_undegraded":"2024-09-13T13:18:46.381154+0000","last_fullsized":"2024-09-13T13:18:46.381154+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4638,"log_dups_size":0,"ondisk_log_size":4638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:17.686405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303955968,"num_objects":4638,"num_object_clones":0,"num_object_copies":13914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4638,"num_write_kb":296832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":328,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660733+0000","last_change":"2024-09-13T13:18:09.522990+0000","last_active":"2024-09-13T13:18:47.660733+0000","last_peered":"2024-09-13T13:18:47.660733+0000","last_clean":"2024-09-13T13:18:47.660733+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:18:47.660733+0000","last_undegraded":"2024-09-13T13:18:47.660733+0000","last_fullsized":"2024-09-13T13:18:47.660733+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:09.522908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:09.522908+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.176687+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":322,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.659882+0000","last_change":"2024-09-13T13:17:54.465421+0000","last_active":"2024-09-13T13:18:47.659882+0000","last_peered":"2024-09-13T13:18:47.659882+0000","last_clean":"2024-09-13T13:18:47.659882+0000","last_became_active":"2024-09-13T13:17:25.244810+0000","last_became_peered":"2024-09-13T13:17:25.244810+0000","last_unstale":"2024-09-13T13:18:47.659882+0000","last_undegraded":"2024-09-13T13:18:47.659882+0000","last_fullsized":"2024-09-13T13:18:47.659882+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.465363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.465363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:23.531768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":869,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.665073+0000","last_change":"2024-09-13T13:15:55.592241+0000","last_active":"2024-09-13T13:18:47.665073+0000","last_peered":"2024-09-13T13:18:47.665073+0000","last_clean":"2024-09-13T13:18:47.665073+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:18:47.665073+0000","last_undegraded":"2024-09-13T13:18:47.665073+0000","last_fullsized":"2024-09-13T13:18:47.665073+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:55.592172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:55.592172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:21.490551+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":819,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.666537+0000","last_change":"2024-09-13T13:14:23.655250+0000","last_active":"2024-09-13T13:18:47.666537+0000","last_peered":"2024-09-13T13:18:47.666537+0000","last_clean":"2024-09-13T13:18:47.666537+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:18:47.666537+0000","last_undegraded":"2024-09-13T13:18:47.666537+0000","last_fullsized":"2024-09-13T13:18:47.666537+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:23.655072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:14:23.655072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:49.001657+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"445'4846","reported_seq":4901,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.888040+0000","last_change":"2024-09-13T13:17:55.139372+0000","last_active":"2024-09-13T13:18:47.888040+0000","last_peered":"2024-09-13T13:18:47.888040+0000","last_clean":"2024-09-13T13:18:47.888040+0000","last_became_active":"2024-09-13T13:17:55.139080+0000","last_became_peered":"2024-09-13T13:17:55.139080+0000","last_unstale":"2024-09-13T13:18:47.888040+0000","last_undegraded":"2024-09-13T13:18:47.888040+0000","last_fullsized":"2024-09-13T13:18:47.888040+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:12.883286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317587456,"num_objects":4846,"num_object_clones":0,"num_object_copies":14538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4846,"num_write_kb":310144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":75,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.663620+0000","last_change":"2024-09-13T13:17:51.212127+0000","last_active":"2024-09-13T13:18:47.663620+0000","last_peered":"2024-09-13T13:18:47.663620+0000","last_clean":"2024-09-13T13:18:47.663620+0000","last_became_active":"2024-09-13T13:17:49.053269+0000","last_became_peered":"2024-09-13T13:17:49.053269+0000","last_unstale":"2024-09-13T13:18:47.663620+0000","last_undegraded":"2024-09-13T13:18:47.663620+0000","last_fullsized":"2024-09-13T13:18:47.663620+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:17:51.212071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:17:51.212071+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:10.098094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":384,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:48.229223+0000","last_change":"2024-09-13T13:18:48.229223+0000","last_active":"2024-09-13T13:18:48.229223+0000","last_peered":"2024-09-13T13:18:48.229223+0000","last_clean":"2024-09-13T13:18:48.229223+0000","last_became_active":"2024-09-13T13:17:25.244686+0000","last_became_peered":"2024-09-13T13:17:25.244686+0000","last_unstale":"2024-09-13T13:18:48.229223+0000","last_undegraded":"2024-09-13T13:18:48.229223+0000","last_fullsized":"2024-09-13T13:18:48.229223+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:18:48.229160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:48.229160+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:59.840762+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":857,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:43.638705+0000","last_change":"2024-09-13T13:18:03.351698+0000","last_active":"2024-09-13T13:18:43.638705+0000","last_peered":"2024-09-13T13:18:43.638705+0000","last_clean":"2024-09-13T13:18:43.638705+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:18:43.638705+0000","last_undegraded":"2024-09-13T13:18:43.638705+0000","last_fullsized":"2024-09-13T13:18:43.638705+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:03.351653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:03.351653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:11.465761+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":843,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.677830+0000","last_change":"2024-09-13T13:17:05.238854+0000","last_active":"2024-09-13T13:18:47.677830+0000","last_peered":"2024-09-13T13:18:47.677830+0000","last_clean":"2024-09-13T13:18:47.677830+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:18:47.677830+0000","last_undegraded":"2024-09-13T13:18:47.677830+0000","last_fullsized":"2024-09-13T13:18:47.677830+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:05.238722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.238722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.936311+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":84,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.661234+0000","last_change":"2024-09-13T13:17:25.243030+0000","last_active":"2024-09-13T13:18:47.661234+0000","last_peered":"2024-09-13T13:18:47.661234+0000","last_clean":"2024-09-13T13:18:47.661234+0000","last_became_active":"2024-09-13T13:17:25.242366+0000","last_became_peered":"2024-09-13T13:17:25.242366+0000","last_unstale":"2024-09-13T13:18:47.661234+0000","last_undegraded":"2024-09-13T13:18:47.661234+0000","last_fullsized":"2024-09-13T13:18:47.661234+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:35.064188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:35.064188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.698744+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"444'4799","reported_seq":4852,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:46.372873+0000","last_change":"2024-09-13T13:17:55.200332+0000","last_active":"2024-09-13T13:18:46.372873+0000","last_peered":"2024-09-13T13:18:46.372873+0000","last_clean":"2024-09-13T13:18:46.372873+0000","last_became_active":"2024-09-13T13:17:55.199999+0000","last_became_peered":"2024-09-13T13:17:55.199999+0000","last_unstale":"2024-09-13T13:18:46.372873+0000","last_undegraded":"2024-09-13T13:18:46.372873+0000","last_fullsized":"2024-09-13T13:18:46.372873+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:09.224261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314507264,"num_objects":4799,"num_object_clones":0,"num_object_copies":14397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4799,"num_write_kb":307136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":66,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.667529+0000","last_change":"2024-09-13T13:18:21.093184+0000","last_active":"2024-09-13T13:18:47.667529+0000","last_peered":"2024-09-13T13:18:47.667529+0000","last_clean":"2024-09-13T13:18:47.667529+0000","last_became_active":"2024-09-13T13:17:49.053160+0000","last_became_peered":"2024-09-13T13:17:49.053160+0000","last_unstale":"2024-09-13T13:18:47.667529+0000","last_undegraded":"2024-09-13T13:18:47.667529+0000","last_fullsized":"2024-09-13T13:18:47.667529+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:18:50.989284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":312,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660560+0000","last_change":"2024-09-13T13:17:02.421924+0000","last_active":"2024-09-13T13:18:47.660560+0000","last_peered":"2024-09-13T13:18:47.660560+0000","last_clean":"2024-09-13T13:18:47.660560+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:18:47.660560+0000","last_undegraded":"2024-09-13T13:18:47.660560+0000","last_fullsized":"2024-09-13T13:18:47.660560+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.421825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.421825+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":826,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.666415+0000","last_change":"2024-09-13T13:12:53.003530+0000","last_active":"2024-09-13T13:18:47.666415+0000","last_peered":"2024-09-13T13:18:47.666415+0000","last_clean":"2024-09-13T13:18:47.666415+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:18:47.666415+0000","last_undegraded":"2024-09-13T13:18:47.666415+0000","last_fullsized":"2024-09-13T13:18:47.666415+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.996346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.996346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:15.070878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"411'10","reported_seq":113,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.661505+0000","last_change":"2024-09-13T13:17:30.560167+0000","last_active":"2024-09-13T13:18:47.661505+0000","last_peered":"2024-09-13T13:18:47.661505+0000","last_clean":"2024-09-13T13:18:47.661505+0000","last_became_active":"2024-09-13T13:17:25.333196+0000","last_became_peered":"2024-09-13T13:17:25.333196+0000","last_unstale":"2024-09-13T13:18:47.661505+0000","last_undegraded":"2024-09-13T13:18:47.661505+0000","last_fullsized":"2024-09-13T13:18:47.661505+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'10","last_scrub_stamp":"2024-09-13T13:17:30.560099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:30.560099+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":322,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660663+0000","last_change":"2024-09-13T13:18:42.196223+0000","last_active":"2024-09-13T13:18:47.660663+0000","last_peered":"2024-09-13T13:18:47.660663+0000","last_clean":"2024-09-13T13:18:47.660663+0000","last_became_active":"2024-09-13T13:17:25.252583+0000","last_became_peered":"2024-09-13T13:17:25.252583+0000","last_unstale":"2024-09-13T13:18:47.660663+0000","last_undegraded":"2024-09-13T13:18:47.660663+0000","last_fullsized":"2024-09-13T13:18:47.660663+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:42.196120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:42.196120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:02.244520+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"444'4614","reported_seq":4667,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:46.393027+0000","last_change":"2024-09-13T13:17:55.200305+0000","last_active":"2024-09-13T13:18:46.393027+0000","last_peered":"2024-09-13T13:18:46.393027+0000","last_clean":"2024-09-13T13:18:46.393027+0000","last_became_active":"2024-09-13T13:17:55.200036+0000","last_became_peered":"2024-09-13T13:17:55.200036+0000","last_unstale":"2024-09-13T13:18:46.393027+0000","last_undegraded":"2024-09-13T13:18:46.393027+0000","last_fullsized":"2024-09-13T13:18:46.393027+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4614,"log_dups_size":0,"ondisk_log_size":4614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.345647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302383104,"num_objects":4614,"num_object_clones":0,"num_object_copies":13842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4614,"num_write_kb":295296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":858,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672225+0000","last_change":"2024-09-13T13:13:56.850149+0000","last_active":"2024-09-13T13:18:47.672225+0000","last_peered":"2024-09-13T13:18:47.672225+0000","last_clean":"2024-09-13T13:18:47.672225+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:18:47.672225+0000","last_undegraded":"2024-09-13T13:18:47.672225+0000","last_fullsized":"2024-09-13T13:18:47.672225+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:56.850094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.850094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.496112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":847,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664407+0000","last_change":"2024-09-13T13:12:58.857634+0000","last_active":"2024-09-13T13:18:47.664407+0000","last_peered":"2024-09-13T13:18:47.664407+0000","last_clean":"2024-09-13T13:18:47.664407+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:18:47.664407+0000","last_undegraded":"2024-09-13T13:18:47.664407+0000","last_fullsized":"2024-09-13T13:18:47.664407+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:58.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:24.527045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":92,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.665737+0000","last_change":"2024-09-13T13:17:54.620568+0000","last_active":"2024-09-13T13:18:47.665737+0000","last_peered":"2024-09-13T13:18:47.665737+0000","last_clean":"2024-09-13T13:18:47.665737+0000","last_became_active":"2024-09-13T13:17:25.238627+0000","last_became_peered":"2024-09-13T13:17:25.238627+0000","last_unstale":"2024-09-13T13:18:47.665737+0000","last_undegraded":"2024-09-13T13:18:47.665737+0000","last_fullsized":"2024-09-13T13:18:47.665737+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.620524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.620524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:08.712954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":830,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.673040+0000","last_change":"2024-09-13T13:17:30.598783+0000","last_active":"2024-09-13T13:18:47.673040+0000","last_peered":"2024-09-13T13:18:47.673040+0000","last_clean":"2024-09-13T13:18:47.673040+0000","last_became_active":"2024-09-13T13:17:25.252822+0000","last_became_peered":"2024-09-13T13:17:25.252822+0000","last_unstale":"2024-09-13T13:18:47.673040+0000","last_undegraded":"2024-09-13T13:18:47.673040+0000","last_fullsized":"2024-09-13T13:18:47.673040+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:30.598736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:30.598736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"445'4934","reported_seq":4989,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:48.516872+0000","last_change":"2024-09-13T13:17:55.200461+0000","last_active":"2024-09-13T13:18:48.516872+0000","last_peered":"2024-09-13T13:18:48.516872+0000","last_clean":"2024-09-13T13:18:48.516872+0000","last_became_active":"2024-09-13T13:17:55.200186+0000","last_became_peered":"2024-09-13T13:17:55.200186+0000","last_unstale":"2024-09-13T13:18:48.516872+0000","last_undegraded":"2024-09-13T13:18:48.516872+0000","last_fullsized":"2024-09-13T13:18:48.516872+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4934,"log_dups_size":0,"ondisk_log_size":4934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.535572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323354624,"num_objects":4934,"num_object_clones":0,"num_object_copies":14802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4934,"num_write_kb":315776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":830,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:43.638486+0000","last_change":"2024-09-13T13:13:06.339762+0000","last_active":"2024-09-13T13:18:43.638486+0000","last_peered":"2024-09-13T13:18:43.638486+0000","last_clean":"2024-09-13T13:18:43.638486+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:18:43.638486+0000","last_undegraded":"2024-09-13T13:18:43.638486+0000","last_fullsized":"2024-09-13T13:18:43.638486+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:13:06.339664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.339664+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:26.319604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":462,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664426+0000","last_change":"2024-09-13T13:14:55.471954+0000","last_active":"2024-09-13T13:18:47.664426+0000","last_peered":"2024-09-13T13:18:47.664426+0000","last_clean":"2024-09-13T13:18:47.664426+0000","last_became_active":"2024-09-13T13:12:39.954315+0000","last_became_peered":"2024-09-13T13:12:39.954315+0000","last_unstale":"2024-09-13T13:18:47.664426+0000","last_undegraded":"2024-09-13T13:18:47.664426+0000","last_fullsized":"2024-09-13T13:18:47.664426+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:55.471885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:55.471885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":92,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660885+0000","last_change":"2024-09-13T13:18:39.692048+0000","last_active":"2024-09-13T13:18:47.660885+0000","last_peered":"2024-09-13T13:18:47.660885+0000","last_clean":"2024-09-13T13:18:47.660885+0000","last_became_active":"2024-09-13T13:17:25.332964+0000","last_became_peered":"2024-09-13T13:17:25.332964+0000","last_unstale":"2024-09-13T13:18:47.660885+0000","last_undegraded":"2024-09-13T13:18:47.660885+0000","last_fullsized":"2024-09-13T13:18:47.660885+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:39.691978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:39.691978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:08.687543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":84,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660204+0000","last_change":"2024-09-13T13:17:25.243438+0000","last_active":"2024-09-13T13:18:47.660204+0000","last_peered":"2024-09-13T13:18:47.660204+0000","last_clean":"2024-09-13T13:18:47.660204+0000","last_became_active":"2024-09-13T13:17:25.243114+0000","last_became_peered":"2024-09-13T13:17:25.243114+0000","last_unstale":"2024-09-13T13:18:47.660204+0000","last_undegraded":"2024-09-13T13:18:47.660204+0000","last_fullsized":"2024-09-13T13:18:47.660204+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:02.627569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:15:02.627569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"445'4909","reported_seq":4964,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:49.691870+0000","last_change":"2024-09-13T13:17:55.140187+0000","last_active":"2024-09-13T13:18:49.691870+0000","last_peered":"2024-09-13T13:18:49.691870+0000","last_clean":"2024-09-13T13:18:49.691870+0000","last_became_active":"2024-09-13T13:17:55.139876+0000","last_became_peered":"2024-09-13T13:17:55.139876+0000","last_unstale":"2024-09-13T13:18:49.691870+0000","last_undegraded":"2024-09-13T13:18:49.691870+0000","last_fullsized":"2024-09-13T13:18:49.691870+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4909,"log_dups_size":0,"ondisk_log_size":4909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:02.705289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321716224,"num_objects":4909,"num_object_clones":0,"num_object_copies":14727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4909,"num_write_kb":314176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":320,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660083+0000","last_change":"2024-09-13T13:16:00.285064+0000","last_active":"2024-09-13T13:18:47.660083+0000","last_peered":"2024-09-13T13:18:47.660083+0000","last_clean":"2024-09-13T13:18:47.660083+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:18:47.660083+0000","last_undegraded":"2024-09-13T13:18:47.660083+0000","last_fullsized":"2024-09-13T13:18:47.660083+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:16:00.284984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:00.284984+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":308,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660622+0000","last_change":"2024-09-13T13:14:52.383027+0000","last_active":"2024-09-13T13:18:47.660622+0000","last_peered":"2024-09-13T13:18:47.660622+0000","last_clean":"2024-09-13T13:18:47.660622+0000","last_became_active":"2024-09-13T13:12:53.007561+0000","last_became_peered":"2024-09-13T13:12:53.007561+0000","last_unstale":"2024-09-13T13:18:47.660622+0000","last_undegraded":"2024-09-13T13:18:47.660622+0000","last_fullsized":"2024-09-13T13:18:47.660622+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.382755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.382755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:15.521717+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":92,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.659382+0000","last_change":"2024-09-13T13:17:52.607482+0000","last_active":"2024-09-13T13:18:47.659382+0000","last_peered":"2024-09-13T13:18:47.659382+0000","last_clean":"2024-09-13T13:18:47.659382+0000","last_became_active":"2024-09-13T13:17:25.252535+0000","last_became_peered":"2024-09-13T13:17:25.252535+0000","last_unstale":"2024-09-13T13:18:47.659382+0000","last_undegraded":"2024-09-13T13:18:47.659382+0000","last_fullsized":"2024-09-13T13:18:47.659382+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:52.607404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.607404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:58.714721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":808,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.666247+0000","last_change":"2024-09-13T13:16:58.047398+0000","last_active":"2024-09-13T13:18:47.666247+0000","last_peered":"2024-09-13T13:18:47.666247+0000","last_clean":"2024-09-13T13:18:47.666247+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:18:47.666247+0000","last_undegraded":"2024-09-13T13:18:47.666247+0000","last_fullsized":"2024-09-13T13:18:47.666247+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:58.044139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.044139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"445'4759","reported_seq":4813,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:48.499282+0000","last_change":"2024-09-13T13:17:55.137957+0000","last_active":"2024-09-13T13:18:48.499282+0000","last_peered":"2024-09-13T13:18:48.499282+0000","last_clean":"2024-09-13T13:18:48.499282+0000","last_became_active":"2024-09-13T13:17:55.137667+0000","last_became_peered":"2024-09-13T13:17:55.137667+0000","last_unstale":"2024-09-13T13:18:48.499282+0000","last_undegraded":"2024-09-13T13:18:48.499282+0000","last_fullsized":"2024-09-13T13:18:48.499282+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4759,"log_dups_size":0,"ondisk_log_size":4759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:03.830885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311885824,"num_objects":4759,"num_object_clones":0,"num_object_copies":14277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4759,"num_write_kb":304576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":895,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.663762+0000","last_change":"2024-09-13T13:15:56.631456+0000","last_active":"2024-09-13T13:18:47.663762+0000","last_peered":"2024-09-13T13:18:47.663762+0000","last_clean":"2024-09-13T13:18:47.663762+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:18:47.663762+0000","last_undegraded":"2024-09-13T13:18:47.663762+0000","last_fullsized":"2024-09-13T13:18:47.663762+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:15:56.631396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:56.631396+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:18.472573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":306,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660659+0000","last_change":"2024-09-13T13:13:01.972907+0000","last_active":"2024-09-13T13:18:47.660659+0000","last_peered":"2024-09-13T13:18:47.660659+0000","last_clean":"2024-09-13T13:18:47.660659+0000","last_became_active":"2024-09-13T13:12:53.006362+0000","last_became_peered":"2024-09-13T13:12:53.006362+0000","last_unstale":"2024-09-13T13:18:47.660659+0000","last_undegraded":"2024-09-13T13:18:47.660659+0000","last_fullsized":"2024-09-13T13:18:47.660659+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:13:01.972831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.972831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:17.528749+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"306'13","reported_seq":587,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:43.638427+0000","last_change":"2024-09-13T13:18:41.356034+0000","last_active":"2024-09-13T13:18:43.638427+0000","last_peered":"2024-09-13T13:18:43.638427+0000","last_clean":"2024-09-13T13:18:43.638427+0000","last_became_active":"2024-09-13T13:12:53.001189+0000","last_became_peered":"2024-09-13T13:12:53.001189+0000","last_unstale":"2024-09-13T13:18:43.638427+0000","last_undegraded":"2024-09-13T13:18:43.638427+0000","last_fullsized":"2024-09-13T13:18:43.638427+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'13","last_scrub_stamp":"2024-09-13T13:18:41.355851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.355851+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:59.453980+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":828,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.663893+0000","last_change":"2024-09-13T13:18:43.850216+0000","last_active":"2024-09-13T13:18:47.663893+0000","last_peered":"2024-09-13T13:18:47.663893+0000","last_clean":"2024-09-13T13:18:47.663893+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:18:47.663893+0000","last_undegraded":"2024-09-13T13:18:47.663893+0000","last_fullsized":"2024-09-13T13:18:47.663893+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:43.850163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:43.850163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:11.330472+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":795,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:43.640016+0000","last_change":"2024-09-13T13:13:04.371082+0000","last_active":"2024-09-13T13:18:43.640016+0000","last_peered":"2024-09-13T13:18:43.640016+0000","last_clean":"2024-09-13T13:18:43.640016+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:18:43.640016+0000","last_undegraded":"2024-09-13T13:18:43.640016+0000","last_fullsized":"2024-09-13T13:18:43.640016+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.370881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.370881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:22.297512+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":350,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664020+0000","last_change":"2024-09-13T13:17:59.217373+0000","last_active":"2024-09-13T13:18:47.664020+0000","last_peered":"2024-09-13T13:18:47.664020+0000","last_clean":"2024-09-13T13:18:47.664020+0000","last_became_active":"2024-09-13T13:12:34.673528+0000","last_became_peered":"2024-09-13T13:12:34.673528+0000","last_unstale":"2024-09-13T13:18:47.664020+0000","last_undegraded":"2024-09-13T13:18:47.664020+0000","last_fullsized":"2024-09-13T13:18:47.664020+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:17:59.217309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.217309+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:28.238775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":852,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664321+0000","last_change":"2024-09-13T13:18:05.302935+0000","last_active":"2024-09-13T13:18:47.664321+0000","last_peered":"2024-09-13T13:18:47.664321+0000","last_clean":"2024-09-13T13:18:47.664321+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:18:47.664321+0000","last_undegraded":"2024-09-13T13:18:47.664321+0000","last_fullsized":"2024-09-13T13:18:47.664321+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:05.302882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:05.302882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.921840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":873,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.673156+0000","last_change":"2024-09-13T13:17:53.194243+0000","last_active":"2024-09-13T13:18:47.673156+0000","last_peered":"2024-09-13T13:18:47.673156+0000","last_clean":"2024-09-13T13:18:47.673156+0000","last_became_active":"2024-09-13T13:12:53.000873+0000","last_became_peered":"2024-09-13T13:12:53.000873+0000","last_unstale":"2024-09-13T13:18:47.673156+0000","last_undegraded":"2024-09-13T13:18:47.673156+0000","last_fullsized":"2024-09-13T13:18:47.673156+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:53.194191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:53.194191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:19.153631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":848,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.678142+0000","last_change":"2024-09-13T13:17:52.518556+0000","last_active":"2024-09-13T13:18:47.678142+0000","last_peered":"2024-09-13T13:18:47.678142+0000","last_clean":"2024-09-13T13:18:47.678142+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:18:47.678142+0000","last_undegraded":"2024-09-13T13:18:47.678142+0000","last_fullsized":"2024-09-13T13:18:47.678142+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:52.518502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.518502+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:12.668389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":300,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.659978+0000","last_change":"2024-09-13T13:13:51.764793+0000","last_active":"2024-09-13T13:18:47.659978+0000","last_peered":"2024-09-13T13:18:47.659978+0000","last_clean":"2024-09-13T13:18:47.659978+0000","last_became_active":"2024-09-13T13:12:53.006579+0000","last_became_peered":"2024-09-13T13:12:53.006579+0000","last_unstale":"2024-09-13T13:18:47.659978+0000","last_undegraded":"2024-09-13T13:18:47.659978+0000","last_fullsized":"2024-09-13T13:18:47.659978+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.764717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.764717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:24.315631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":828,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.667751+0000","last_change":"2024-09-13T13:17:00.968640+0000","last_active":"2024-09-13T13:18:47.667751+0000","last_peered":"2024-09-13T13:18:47.667751+0000","last_clean":"2024-09-13T13:18:47.667751+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:18:47.667751+0000","last_undegraded":"2024-09-13T13:18:47.667751+0000","last_fullsized":"2024-09-13T13:18:47.667751+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:00.968580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:00.968580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":92,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.661117+0000","last_change":"2024-09-13T13:18:45.736249+0000","last_active":"2024-09-13T13:18:47.661117+0000","last_peered":"2024-09-13T13:18:47.661117+0000","last_clean":"2024-09-13T13:18:47.661117+0000","last_became_active":"2024-09-13T13:17:25.244421+0000","last_became_peered":"2024-09-13T13:17:25.244421+0000","last_unstale":"2024-09-13T13:18:47.661117+0000","last_undegraded":"2024-09-13T13:18:47.661117+0000","last_fullsized":"2024-09-13T13:18:47.661117+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:45.736080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.736080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:56.355923+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":870,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664474+0000","last_change":"2024-09-13T13:17:04.222896+0000","last_active":"2024-09-13T13:18:47.664474+0000","last_peered":"2024-09-13T13:18:47.664474+0000","last_clean":"2024-09-13T13:18:47.664474+0000","last_became_active":"2024-09-13T13:16:12.069730+0000","last_became_peered":"2024-09-13T13:16:12.069730+0000","last_unstale":"2024-09-13T13:18:47.664474+0000","last_undegraded":"2024-09-13T13:18:47.664474+0000","last_fullsized":"2024-09-13T13:18:47.664474+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:04.222760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:04.222760+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":819,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.680329+0000","last_change":"2024-09-13T13:17:09.634101+0000","last_active":"2024-09-13T13:18:47.680329+0000","last_peered":"2024-09-13T13:18:47.680329+0000","last_clean":"2024-09-13T13:18:47.680329+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:18:47.680329+0000","last_undegraded":"2024-09-13T13:18:47.680329+0000","last_fullsized":"2024-09-13T13:18:47.680329+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.634027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.634027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:30.479119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"411'10","reported_seq":1056,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.666627+0000","last_change":"2024-09-13T13:17:27.165835+0000","last_active":"2024-09-13T13:18:47.666627+0000","last_peered":"2024-09-13T13:18:47.666627+0000","last_clean":"2024-09-13T13:18:47.666627+0000","last_became_active":"2024-09-13T13:17:25.245468+0000","last_became_peered":"2024-09-13T13:17:25.245468+0000","last_unstale":"2024-09-13T13:18:47.666627+0000","last_undegraded":"2024-09-13T13:18:47.666627+0000","last_fullsized":"2024-09-13T13:18:47.666627+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'9","last_scrub_stamp":"2024-09-13T13:17:27.165657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:27.165657+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":322,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660688+0000","last_change":"2024-09-13T13:18:41.148239+0000","last_active":"2024-09-13T13:18:47.660688+0000","last_peered":"2024-09-13T13:18:47.660688+0000","last_clean":"2024-09-13T13:18:47.660688+0000","last_became_active":"2024-09-13T13:17:25.333587+0000","last_became_peered":"2024-09-13T13:17:25.333587+0000","last_unstale":"2024-09-13T13:18:47.660688+0000","last_undegraded":"2024-09-13T13:18:47.660688+0000","last_fullsized":"2024-09-13T13:18:47.660688+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:41.147987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.147987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:48.324808+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":870,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.667999+0000","last_change":"2024-09-13T13:17:02.978088+0000","last_active":"2024-09-13T13:18:47.667999+0000","last_peered":"2024-09-13T13:18:47.667999+0000","last_clean":"2024-09-13T13:18:47.667999+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:18:47.667999+0000","last_undegraded":"2024-09-13T13:18:47.667999+0000","last_fullsized":"2024-09-13T13:18:47.667999+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.978003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.978003+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":848,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.673273+0000","last_change":"2024-09-13T13:17:28.963214+0000","last_active":"2024-09-13T13:18:47.673273+0000","last_peered":"2024-09-13T13:18:47.673273+0000","last_clean":"2024-09-13T13:18:47.673273+0000","last_became_active":"2024-09-13T13:17:25.252500+0000","last_became_peered":"2024-09-13T13:17:25.252500+0000","last_unstale":"2024-09-13T13:18:47.673273+0000","last_undegraded":"2024-09-13T13:18:47.673273+0000","last_fullsized":"2024-09-13T13:18:47.673273+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:28.963152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:28.963152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:50.378740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":1058,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664370+0000","last_change":"2024-09-13T13:17:57.251817+0000","last_active":"2024-09-13T13:18:47.664370+0000","last_peered":"2024-09-13T13:18:47.664370+0000","last_clean":"2024-09-13T13:18:47.664370+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:18:47.664370+0000","last_undegraded":"2024-09-13T13:18:47.664370+0000","last_fullsized":"2024-09-13T13:18:47.664370+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:17:57.251734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:57.251734+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:03.487958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":831,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.674399+0000","last_change":"2024-09-13T13:17:33.554954+0000","last_active":"2024-09-13T13:18:47.674399+0000","last_peered":"2024-09-13T13:18:47.674399+0000","last_clean":"2024-09-13T13:18:47.674399+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:18:47.674399+0000","last_undegraded":"2024-09-13T13:18:47.674399+0000","last_fullsized":"2024-09-13T13:18:47.674399+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:33.554870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:33.554870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:03.321466+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":320,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660411+0000","last_change":"2024-09-13T13:18:45.145089+0000","last_active":"2024-09-13T13:18:47.660411+0000","last_peered":"2024-09-13T13:18:47.660411+0000","last_clean":"2024-09-13T13:18:47.660411+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:18:47.660411+0000","last_undegraded":"2024-09-13T13:18:47.660411+0000","last_fullsized":"2024-09-13T13:18:47.660411+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:45.145010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.145010+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:04.145291+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":300,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.660445+0000","last_change":"2024-09-13T13:17:09.431539+0000","last_active":"2024-09-13T13:18:47.660445+0000","last_peered":"2024-09-13T13:18:47.660445+0000","last_clean":"2024-09-13T13:18:47.660445+0000","last_became_active":"2024-09-13T13:12:53.005988+0000","last_became_peered":"2024-09-13T13:12:53.005988+0000","last_unstale":"2024-09-13T13:18:47.660445+0000","last_undegraded":"2024-09-13T13:18:47.660445+0000","last_fullsized":"2024-09-13T13:18:47.660445+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.431472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.431472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":837,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.676173+0000","last_change":"2024-09-13T13:17:33.279005+0000","last_active":"2024-09-13T13:18:47.676173+0000","last_peered":"2024-09-13T13:18:47.676173+0000","last_clean":"2024-09-13T13:18:47.676173+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:18:47.676173+0000","last_undegraded":"2024-09-13T13:18:47.676173+0000","last_fullsized":"2024-09-13T13:18:47.676173+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:33.278909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:33.278909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:55.488227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":832,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.667393+0000","last_change":"2024-09-13T13:17:09.989898+0000","last_active":"2024-09-13T13:18:47.667393+0000","last_peered":"2024-09-13T13:18:47.667393+0000","last_clean":"2024-09-13T13:18:47.667393+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:18:47.667393+0000","last_undegraded":"2024-09-13T13:18:47.667393+0000","last_fullsized":"2024-09-13T13:18:47.667393+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.989820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.989820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":826,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.663839+0000","last_change":"2024-09-13T13:17:26.125163+0000","last_active":"2024-09-13T13:18:47.663839+0000","last_peered":"2024-09-13T13:18:47.663839+0000","last_clean":"2024-09-13T13:18:47.663839+0000","last_became_active":"2024-09-13T13:17:25.331530+0000","last_became_peered":"2024-09-13T13:17:25.331530+0000","last_unstale":"2024-09-13T13:18:47.663839+0000","last_undegraded":"2024-09-13T13:18:47.663839+0000","last_fullsized":"2024-09-13T13:18:47.663839+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:26.125088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:26.125088+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:50.498858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":820,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.667237+0000","last_change":"2024-09-13T13:17:56.965925+0000","last_active":"2024-09-13T13:18:47.667237+0000","last_peered":"2024-09-13T13:18:47.667237+0000","last_clean":"2024-09-13T13:18:47.667237+0000","last_became_active":"2024-09-13T13:12:52.949094+0000","last_became_peered":"2024-09-13T13:12:52.949094+0000","last_unstale":"2024-09-13T13:18:47.667237+0000","last_undegraded":"2024-09-13T13:18:47.667237+0000","last_fullsized":"2024-09-13T13:18:47.667237+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:56.965870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:56.965870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:02.221904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":828,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.666270+0000","last_change":"2024-09-13T13:18:03.037546+0000","last_active":"2024-09-13T13:18:47.666270+0000","last_peered":"2024-09-13T13:18:47.666270+0000","last_clean":"2024-09-13T13:18:47.666270+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:18:47.666270+0000","last_undegraded":"2024-09-13T13:18:47.666270+0000","last_fullsized":"2024-09-13T13:18:47.666270+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:03.037472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:03.037472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.016351+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":815,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.671769+0000","last_change":"2024-09-13T13:18:44.505493+0000","last_active":"2024-09-13T13:18:47.671769+0000","last_peered":"2024-09-13T13:18:47.671769+0000","last_clean":"2024-09-13T13:18:47.671769+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:18:47.671769+0000","last_undegraded":"2024-09-13T13:18:47.671769+0000","last_fullsized":"2024-09-13T13:18:47.671769+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:44.505427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:44.505427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:54.043312+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":858,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.667998+0000","last_change":"2024-09-13T13:18:33.848957+0000","last_active":"2024-09-13T13:18:47.667998+0000","last_peered":"2024-09-13T13:18:47.667998+0000","last_clean":"2024-09-13T13:18:47.667998+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:18:47.667998+0000","last_undegraded":"2024-09-13T13:18:47.667998+0000","last_fullsized":"2024-09-13T13:18:47.667998+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:33.848726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.848726+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:59.409198+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":836,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.664398+0000","last_change":"2024-09-13T13:14:50.392659+0000","last_active":"2024-09-13T13:18:47.664398+0000","last_peered":"2024-09-13T13:18:47.664398+0000","last_clean":"2024-09-13T13:18:47.664398+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:18:47.664398+0000","last_undegraded":"2024-09-13T13:18:47.664398+0000","last_fullsized":"2024-09-13T13:18:47.664398+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.392444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.392444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:48.280356+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":845,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672525+0000","last_change":"2024-09-13T13:15:03.778214+0000","last_active":"2024-09-13T13:18:47.672525+0000","last_peered":"2024-09-13T13:18:47.672525+0000","last_clean":"2024-09-13T13:18:47.672525+0000","last_became_active":"2024-09-13T13:12:52.999748+0000","last_became_peered":"2024-09-13T13:12:52.999748+0000","last_unstale":"2024-09-13T13:18:47.672525+0000","last_undegraded":"2024-09-13T13:18:47.672525+0000","last_fullsized":"2024-09-13T13:18:47.672525+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.778170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.778170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:24.797615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":803,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.666351+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:18:47.666351+0000","last_peered":"2024-09-13T13:18:47.666351+0000","last_clean":"2024-09-13T13:18:47.666351+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:18:47.666351+0000","last_undegraded":"2024-09-13T13:18:47.666351+0000","last_fullsized":"2024-09-13T13:18:47.666351+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:23.884576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":873,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.666767+0000","last_change":"2024-09-13T13:18:46.033316+0000","last_active":"2024-09-13T13:18:47.666767+0000","last_peered":"2024-09-13T13:18:47.666767+0000","last_clean":"2024-09-13T13:18:47.666767+0000","last_became_active":"2024-09-13T13:16:12.073702+0000","last_became_peered":"2024-09-13T13:16:12.073702+0000","last_unstale":"2024-09-13T13:18:47.666767+0000","last_undegraded":"2024-09-13T13:18:47.666767+0000","last_fullsized":"2024-09-13T13:18:47.666767+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:18:46.033238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.033238+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:01.052128+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":851,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.675792+0000","last_change":"2024-09-13T13:17:57.579827+0000","last_active":"2024-09-13T13:18:47.675792+0000","last_peered":"2024-09-13T13:18:47.675792+0000","last_clean":"2024-09-13T13:18:47.675792+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:18:47.675792+0000","last_undegraded":"2024-09-13T13:18:47.675792+0000","last_fullsized":"2024-09-13T13:18:47.675792+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:57.579778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:57.579778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:11.821230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":757,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:43.638529+0000","last_change":"2024-09-13T13:17:55.331435+0000","last_active":"2024-09-13T13:18:43.638529+0000","last_peered":"2024-09-13T13:18:43.638529+0000","last_clean":"2024-09-13T13:18:43.638529+0000","last_became_active":"2024-09-13T13:12:53.001131+0000","last_became_peered":"2024-09-13T13:12:53.001131+0000","last_unstale":"2024-09-13T13:18:43.638529+0000","last_undegraded":"2024-09-13T13:18:43.638529+0000","last_fullsized":"2024-09-13T13:18:43.638529+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:55.331393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:55.331393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:14.150805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":92,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.665597+0000","last_change":"2024-09-13T13:18:07.682986+0000","last_active":"2024-09-13T13:18:47.665597+0000","last_peered":"2024-09-13T13:18:47.665597+0000","last_clean":"2024-09-13T13:18:47.665597+0000","last_became_active":"2024-09-13T13:17:25.242337+0000","last_became_peered":"2024-09-13T13:17:25.242337+0000","last_unstale":"2024-09-13T13:18:47.665597+0000","last_undegraded":"2024-09-13T13:18:47.665597+0000","last_fullsized":"2024-09-13T13:18:47.665597+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:07.682648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:07.682648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:11.815648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":839,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.677000+0000","last_change":"2024-09-13T13:17:52.052009+0000","last_active":"2024-09-13T13:18:47.677000+0000","last_peered":"2024-09-13T13:18:47.677000+0000","last_clean":"2024-09-13T13:18:47.677000+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:18:47.677000+0000","last_undegraded":"2024-09-13T13:18:47.677000+0000","last_fullsized":"2024-09-13T13:18:47.677000+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:52.051961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.051961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:18.556468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":820,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.667016+0000","last_change":"2024-09-13T13:16:36.980668+0000","last_active":"2024-09-13T13:18:47.667016+0000","last_peered":"2024-09-13T13:18:47.667016+0000","last_clean":"2024-09-13T13:18:47.667016+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:18:47.667016+0000","last_undegraded":"2024-09-13T13:18:47.667016+0000","last_fullsized":"2024-09-13T13:18:47.667016+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:36.980594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:36.980594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.050858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":837,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-13T13:18:43.638511+0000","last_change":"2024-09-13T13:16:12.087075+0000","last_active":"2024-09-13T13:18:43.638511+0000","last_peered":"2024-09-13T13:18:43.638511+0000","last_clean":"2024-09-13T13:18:43.638511+0000","last_became_active":"2024-09-13T13:16:12.070277+0000","last_became_peered":"2024-09-13T13:16:12.070277+0000","last_unstale":"2024-09-13T13:18:43.638511+0000","last_undegraded":"2024-09-13T13:18:43.638511+0000","last_fullsized":"2024-09-13T13:18:43.638511+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:49.264128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.264128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:15.253268+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":858,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.675754+0000","last_change":"2024-09-13T13:17:25.253423+0000","last_active":"2024-09-13T13:18:47.675754+0000","last_peered":"2024-09-13T13:18:47.675754+0000","last_clean":"2024-09-13T13:18:47.675754+0000","last_became_active":"2024-09-13T13:17:25.252637+0000","last_became_peered":"2024-09-13T13:17:25.252637+0000","last_unstale":"2024-09-13T13:18:47.675754+0000","last_undegraded":"2024-09-13T13:18:47.675754+0000","last_fullsized":"2024-09-13T13:18:47.675754+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:23.547758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:23.547758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:29.521458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5068816384,"num_objects":77344,"num_object_clones":0,"num_object_copies":232032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77344,"num_write_kb":4950016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15236202496,"data_stored":15236202496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75444,"ondisk_log_size":75444,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":4792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":686,"ondisk_log_size":686,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":375,"num_read_kb":370,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":799386,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":199,"num_write":103,"num_write_kb":1480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":864378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201496,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2465156,"kb_used_data":1870320,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91279996,"statfs":{"total":95995035648,"available":93470715904,"internally_reserved":0,"allocated":1915207680,"data_stored":1914685884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":609089994},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141106,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3009188,"kb_used_data":2412244,"kb_used_omap":4,"kb_used_meta":596923,"kb_avail":90735964,"statfs":{"total":95995035648,"available":92913627136,"internally_reserved":0,"allocated":2470137856,"data_stored":2469636210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":611249837},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692068,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1881216,"kb_used_data":1240556,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":91863936,"statfs":{"total":95995035648,"available":94068670464,"internally_reserved":0,"allocated":1270329344,"data_stored":1269811028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921002,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1852560,"kb_used_data":1229948,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91892592,"statfs":{"total":95995035648,"available":94098014208,"internally_reserved":0,"allocated":1259466752,"data_stored":1258931644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":637532614},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117233,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2881480,"kb_used_data":2142264,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":90863672,"statfs":{"total":95995035648,"available":93044400128,"internally_reserved":0,"allocated":2193678336,"data_stored":2193141805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":756939207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411572,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2398772,"kb_used_data":1848352,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":91346380,"statfs":{"total":95995035648,"available":93538693120,"internally_reserved":0,"allocated":1892712448,"data_stored":1892205870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":563608006},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575099,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3145804,"kb_used_data":2510716,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":90599348,"statfs":{"total":95995035648,"available":92773732352,"internally_reserved":0,"allocated":2570973184,"data_stored":2570438119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738626,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2354196,"kb_used_data":1564996,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":91390956,"statfs":{"total":95995035648,"available":93584338944,"internally_reserved":0,"allocated":1602555904,"data_stored":1602013695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":808122821},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1608253440,"data_stored":1608253440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2578186240,"data_stored":2578186240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1889402880,"data_stored":1889402880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2222391296,"data_stored":2222391296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1259601920,"data_stored":1259601920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1279983616,"data_stored":1279983616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2472869888,"data_stored":2472869888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1925513216,"data_stored":1925513216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:18:52.258 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:18:52.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:52.260 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:52.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:52.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:52.389 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:52.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:52.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:52.491 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:52.538 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:52.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:52.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:52.640 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:52.670 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:52.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:52.771 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:52.802 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:52.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:52.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:52.885 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 52 16 82291 82275 98.862 60.125 0.0140423 0.0101102 2024-09-13T13:18:52.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:52.934 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:52.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:52.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:52.955 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:52 smithi167 ceph-mon[33178]: 1.1 scrub starts 2024-09-13T13:18:52.956 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:52 smithi167 ceph-mon[33178]: 1.1 scrub ok 2024-09-13T13:18:52.956 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:52 smithi167 ceph-mon[33178]: 1.9 scrub starts 2024-09-13T13:18:52.956 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:52 smithi167 ceph-mon[33178]: 1.9 scrub ok 2024-09-13T13:18:52.956 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:52 smithi167 ceph-mon[33178]: from='client.36725 v1:172.21.15.52:0/2803775368' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:18:52.956 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:52 smithi167 ceph-mon[33178]: 1.2 scrub starts 2024-09-13T13:18:52.956 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:52 smithi167 ceph-mon[33178]: 1.2 scrub ok 2024-09-13T13:18:52.956 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:52 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:18:52.956 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:52 smithi167 ceph-mon[33178]: from='client.36722 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:18:52.956 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:52 smithi167 ceph-mon[33178]: osdmap e446: 8 total, 8 up, 8 in 2024-09-13T13:18:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[36682]: 1.1 scrub starts 2024-09-13T13:18:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[36682]: 1.1 scrub ok 2024-09-13T13:18:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[36682]: 1.9 scrub starts 2024-09-13T13:18:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[36682]: 1.9 scrub ok 2024-09-13T13:18:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[36682]: from='client.36725 v1:172.21.15.52:0/2803775368' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:18:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[36682]: 1.2 scrub starts 2024-09-13T13:18:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[36682]: 1.2 scrub ok 2024-09-13T13:18:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:18:52.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[36682]: from='client.36722 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:18:52.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[36682]: osdmap e446: 8 total, 8 up, 8 in 2024-09-13T13:18:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[29471]: 1.1 scrub starts 2024-09-13T13:18:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[29471]: 1.1 scrub ok 2024-09-13T13:18:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[29471]: 1.9 scrub starts 2024-09-13T13:18:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[29471]: 1.9 scrub ok 2024-09-13T13:18:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[29471]: from='client.36725 v1:172.21.15.52:0/2803775368' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:18:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[29471]: 1.2 scrub starts 2024-09-13T13:18:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[29471]: 1.2 scrub ok 2024-09-13T13:18:52.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:18:52.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[29471]: from='client.36722 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:18:52.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:52 smithi052 ceph-mon[29471]: osdmap e446: 8 total, 8 up, 8 in 2024-09-13T13:18:53.036 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:53.067 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:53.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:53.168 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:53.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:53.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:53.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:53.302 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:53.345 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:53.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:53.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:53.448 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:53.490 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:53.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:53.562 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:18:53.562 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:18:53.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.563 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:82987 11.8 e447/423 11:10788a93:::benchmark_data_smithi052_182575_object82986:head v 447'5259, mlcod=447'5254)", 2024-09-13T13:18:53.563 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:53.560407+0000", 2024-09-13T13:18:53.563 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00095029399999999997, 2024-09-13T13:18:53.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097506999999999997, 2024-09-13T13:18:53.563 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:53.563 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:53.563 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:18:53.563 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:53.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:53.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560407+0000", 2024-09-13T13:18:53.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:53.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:53.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560407+0000", 2024-09-13T13:18:53.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:53.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560408+0000", 2024-09-13T13:18:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.119e-06 2024-09-13T13:18:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560518+0000", 2024-09-13T13:18:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109833 2024-09-13T13:18:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560529+0000", 2024-09-13T13:18:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1273e-05 2024-09-13T13:18:53.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:53.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560536+0000", 2024-09-13T13:18:53.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.472e-06 2024-09-13T13:18:53.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:53.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560579+0000", 2024-09-13T13:18:53.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3256000000000001e-05 2024-09-13T13:18:53.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:53.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560591+0000", 2024-09-13T13:18:53.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2301e-05 2024-09-13T13:18:53.567 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:53.567 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:53.567 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:53.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.568 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:82986 11.a e447/423 11:57461b43:::benchmark_data_smithi052_182575_object82985:head v 447'5224, mlcod=447'5222)", 2024-09-13T13:18:53.568 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:53.560550+0000", 2024-09-13T13:18:53.568 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00080729599999999999, 2024-09-13T13:18:53.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088723199999999995, 2024-09-13T13:18:53.568 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:53.568 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:53.568 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:18:53.568 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:53.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:53.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560550+0000", 2024-09-13T13:18:53.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:53.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:53.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560550+0000", 2024-09-13T13:18:53.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:53.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:53.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560551+0000", 2024-09-13T13:18:53.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.68e-07 2024-09-13T13:18:53.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:53.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560581+0000", 2024-09-13T13:18:53.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0839000000000001e-05 2024-09-13T13:18:53.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:53.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560588+0000", 2024-09-13T13:18:53.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5189999999999998e-06 2024-09-13T13:18:53.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:53.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560593+0000", 2024-09-13T13:18:53.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9230000000000001e-06 2024-09-13T13:18:53.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:53.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560640+0000", 2024-09-13T13:18:53.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6799999999999999e-05 2024-09-13T13:18:53.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:53.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560658+0000", 2024-09-13T13:18:53.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8743999999999999e-05 2024-09-13T13:18:53.572 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:53.572 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:53.572 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:53.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.572 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:82988 11.d e447/423 11:b1da2873:::benchmark_data_smithi052_182575_object82987:head v 447'5072, mlcod=447'5070)", 2024-09-13T13:18:53.572 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:53.560839+0000", 2024-09-13T13:18:53.572 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00051814299999999999, 2024-09-13T13:18:53.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064544600000000004, 2024-09-13T13:18:53.573 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:18:53.573 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:18:53.573 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:18:53.573 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:18:53.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:18:53.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560839+0000", 2024-09-13T13:18:53.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:53.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:18:53.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560839+0000", 2024-09-13T13:18:53.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:18:53.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:18:53.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560845+0000", 2024-09-13T13:18:53.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9630000000000003e-06 2024-09-13T13:18:53.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:18:53.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560886+0000", 2024-09-13T13:18:53.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1334000000000001e-05 2024-09-13T13:18:53.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:18:53.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560897+0000", 2024-09-13T13:18:53.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0621999999999999e-05 2024-09-13T13:18:53.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:18:53.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560905+0000", 2024-09-13T13:18:53.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5680000000000002e-06 2024-09-13T13:18:53.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:18:53.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560962+0000", 2024-09-13T13:18:53.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7688000000000001e-05 2024-09-13T13:18:53.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:18:53.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:18:53.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:18:53.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:53.560981+0000", 2024-09-13T13:18:53.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.889e-05 2024-09-13T13:18:53.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:53.582 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:18:53.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:53.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:18:53.582 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:18:53.582 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 3 2024-09-13T13:18:53.582 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:18:53.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:53.614 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:53.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:53.716 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:53.756 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:53.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:53.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:53 smithi167 ceph-mon[33178]: pgmap v897: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-13T13:18:53.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:53 smithi167 ceph-mon[33178]: 1.4 scrub starts 2024-09-13T13:18:53.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:53 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:18:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:53 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:18:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:53 smithi167 ceph-mon[33178]: osdmap e447: 8 total, 8 up, 8 in 2024-09-13T13:18:53.857 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:53.886 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 53 16 83269 83253 98.1489 61.125 0.0086223 0.010183 2024-09-13T13:18:53.898 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:53.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:53.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:53.999 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:54.039 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:54.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:54.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:54.077 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:53 smithi052 ceph-mon[29471]: pgmap v897: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-13T13:18:54.077 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:53 smithi052 ceph-mon[29471]: 1.4 scrub starts 2024-09-13T13:18:54.077 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:53 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:18:54.078 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:53 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:18:54.078 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:53 smithi052 ceph-mon[29471]: osdmap e447: 8 total, 8 up, 8 in 2024-09-13T13:18:54.078 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:53 smithi052 ceph-mon[36682]: pgmap v897: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-13T13:18:54.078 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:53 smithi052 ceph-mon[36682]: 1.4 scrub starts 2024-09-13T13:18:54.078 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:53 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:18:54.078 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:53 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:18:54.078 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:53 smithi052 ceph-mon[36682]: osdmap e447: 8 total, 8 up, 8 in 2024-09-13T13:18:54.139 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:54.142 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:54.207 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:54.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:54.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:54.209 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:54.310 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:54.364 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:54.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:54.466 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:54.513 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:54.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:54.615 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:54.653 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:54.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:54.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:54.755 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:54.788 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:54.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:54.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:18:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:18:54.886 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 54 16 84410 84394 97.6519 71.3125 0.0120092 0.0102356 2024-09-13T13:18:54.889 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:54.930 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T13:18:54.932 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:54.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:55.005 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:18:55.005 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:18:55.034 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:55.044 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:18:55.044 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:18:55.079 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:55.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:55.081 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:18:55.081 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:18:55.132 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:18:55.132 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:18:55.167 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:18:55.167 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:18:55.182 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:55.207 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:18:55.207 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:18:55.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:55.221 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:55.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:55.250 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:18:55.251 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:18:55.283 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:18:55.284 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:18:55.322 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:55.362 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:18:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:55.383 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:55.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:55.484 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:55.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:55.576 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:18:55.579 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:18:55.620 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:55.651 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:55.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:55.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:55.794 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:55.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:55.830 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:55 smithi052 ceph-mon[36682]: pgmap v899: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-13T13:18:55.830 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:55.830 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:55 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/712789874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:55.830 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:55 smithi052 ceph-mon[29471]: pgmap v899: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-13T13:18:55.830 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:55.830 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:55 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/712789874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:55.886 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 55 16 85495 85479 97.1093 67.8125 0.017545 0.0102926 2024-09-13T13:18:55.896 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:55.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:55.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:56.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:56.062 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:56.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:56.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:56.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:55 smithi167 ceph-mon[33178]: pgmap v899: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-13T13:18:56.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:18:56.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:55 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/712789874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:18:56.131 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:18:56.165 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:56.195 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:56.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:56.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:56.296 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:56.335 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:56.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:56.398 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:18:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:18:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:18:56.436 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:56.456 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":899,"stamp":"2024-09-13T13:18:54.005356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5322858674,"num_objects":81412,"num_object_clones":0,"num_object_copies":244236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81412,"num_whiteouts":0,"num_read":1544,"num_read_kb":1334,"num_write":81977,"num_write_kb":5198269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":549118,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79367,"ondisk_log_size":79367,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20751732,"kb_used_data":15535588,"kb_used_omap":15,"kb_used_meta":5215984,"kb_avail":729209484,"statfs":{"total":767960285184,"available":746710511616,"internally_reserved":0,"allocated":15908442112,"data_stored":15904201705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":5341168409},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":563942941,"num_objects":8609,"num_object_clones":0,"num_object_copies":25827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8609,"num_whiteouts":0,"num_read":-56,"num_read_kb":-115,"num_write":8549,"num_write_kb":550152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-322019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.345722"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":842,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.675693+0000","last_change":"2024-09-13T13:18:32.440533+0000","last_active":"2024-09-13T13:18:47.675693+0000","last_peered":"2024-09-13T13:18:47.675693+0000","last_clean":"2024-09-13T13:18:47.675693+0000","last_became_active":"2024-09-13T13:17:25.335900+0000","last_became_peered":"2024-09-13T13:17:25.335900+0000","last_unstale":"2024-09-13T13:18:47.675693+0000","last_undegraded":"2024-09-13T13:18:47.675693+0000","last_fullsized":"2024-09-13T13:18:47.675693+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:32.440448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:32.440448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:57.101135+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":322,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.065687+0000","last_change":"2024-09-13T13:18:08.548626+0000","last_active":"2024-09-13T13:18:53.065687+0000","last_peered":"2024-09-13T13:18:53.065687+0000","last_clean":"2024-09-13T13:18:53.065687+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:18:53.065687+0000","last_undegraded":"2024-09-13T13:18:53.065687+0000","last_fullsized":"2024-09-13T13:18:53.065687+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:18:08.548556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:08.548556+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:22.274215+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":829,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.675566+0000","last_change":"2024-09-13T13:17:56.109579+0000","last_active":"2024-09-13T13:18:47.675566+0000","last_peered":"2024-09-13T13:18:47.675566+0000","last_clean":"2024-09-13T13:18:47.675566+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:18:47.675566+0000","last_undegraded":"2024-09-13T13:18:47.675566+0000","last_fullsized":"2024-09-13T13:18:47.675566+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:56.109526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:56.109526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:00.485890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":320,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067366+0000","last_change":"2024-09-13T13:18:49.194573+0000","last_active":"2024-09-13T13:18:53.067366+0000","last_peered":"2024-09-13T13:18:53.067366+0000","last_clean":"2024-09-13T13:18:53.067366+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:18:53.067366+0000","last_undegraded":"2024-09-13T13:18:53.067366+0000","last_fullsized":"2024-09-13T13:18:53.067366+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:49.194510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:49.194510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:02.508037+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":821,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.714987+0000","last_change":"2024-09-13T13:11:47.852762+0000","last_active":"2024-09-13T13:18:51.714987+0000","last_peered":"2024-09-13T13:18:51.714987+0000","last_clean":"2024-09-13T13:18:51.714987+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:18:51.714987+0000","last_undegraded":"2024-09-13T13:18:51.714987+0000","last_fullsized":"2024-09-13T13:18:51.714987+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:26.862207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":864,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.715848+0000","last_change":"2024-09-13T13:17:25.243843+0000","last_active":"2024-09-13T13:18:51.715848+0000","last_peered":"2024-09-13T13:18:51.715848+0000","last_clean":"2024-09-13T13:18:51.715848+0000","last_became_active":"2024-09-13T13:17:25.243371+0000","last_became_peered":"2024-09-13T13:17:25.243371+0000","last_unstale":"2024-09-13T13:18:51.715848+0000","last_undegraded":"2024-09-13T13:18:51.715848+0000","last_fullsized":"2024-09-13T13:18:51.715848+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:08.938750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:08.938750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":939,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.716804+0000","last_change":"2024-09-13T13:17:25.262965+0000","last_active":"2024-09-13T13:18:51.716804+0000","last_peered":"2024-09-13T13:18:51.716804+0000","last_clean":"2024-09-13T13:18:51.716804+0000","last_became_active":"2024-09-13T13:17:25.245131+0000","last_became_peered":"2024-09-13T13:17:25.245131+0000","last_unstale":"2024-09-13T13:18:51.716804+0000","last_undegraded":"2024-09-13T13:18:51.716804+0000","last_fullsized":"2024-09-13T13:18:51.716804+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:17:11.044472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:11.044472+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":296,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.065617+0000","last_change":"2024-09-13T13:12:52.997865+0000","last_active":"2024-09-13T13:18:53.065617+0000","last_peered":"2024-09-13T13:18:53.065617+0000","last_clean":"2024-09-13T13:18:53.065617+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:18:53.065617+0000","last_undegraded":"2024-09-13T13:18:53.065617+0000","last_fullsized":"2024-09-13T13:18:53.065617+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:17.305608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:17.305608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.504193+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":887,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069482+0000","last_change":"2024-09-13T13:13:41.518378+0000","last_active":"2024-09-13T13:18:53.069482+0000","last_peered":"2024-09-13T13:18:53.069482+0000","last_clean":"2024-09-13T13:18:53.069482+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:18:53.069482+0000","last_undegraded":"2024-09-13T13:18:53.069482+0000","last_fullsized":"2024-09-13T13:18:53.069482+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:41.518322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.518322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:52.806617+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":314,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067453+0000","last_change":"2024-09-13T13:17:06.406253+0000","last_active":"2024-09-13T13:18:53.067453+0000","last_peered":"2024-09-13T13:18:53.067453+0000","last_clean":"2024-09-13T13:18:53.067453+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:18:53.067453+0000","last_undegraded":"2024-09-13T13:18:53.067453+0000","last_fullsized":"2024-09-13T13:18:53.067453+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:06.406179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:06.406179+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":832,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069781+0000","last_change":"2024-09-13T13:17:31.089557+0000","last_active":"2024-09-13T13:18:53.069781+0000","last_peered":"2024-09-13T13:18:53.069781+0000","last_clean":"2024-09-13T13:18:53.069781+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:18:53.069781+0000","last_undegraded":"2024-09-13T13:18:53.069781+0000","last_fullsized":"2024-09-13T13:18:53.069781+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:31.089505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:31.089505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":831,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.671952+0000","last_change":"2024-09-13T13:17:32.559966+0000","last_active":"2024-09-13T13:18:47.671952+0000","last_peered":"2024-09-13T13:18:47.671952+0000","last_clean":"2024-09-13T13:18:47.671952+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:18:47.671952+0000","last_undegraded":"2024-09-13T13:18:47.671952+0000","last_fullsized":"2024-09-13T13:18:47.671952+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:32.559884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:32.559884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":862,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069163+0000","last_change":"2024-09-13T13:17:02.211078+0000","last_active":"2024-09-13T13:18:53.069163+0000","last_peered":"2024-09-13T13:18:53.069163+0000","last_clean":"2024-09-13T13:18:53.069163+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:18:53.069163+0000","last_undegraded":"2024-09-13T13:18:53.069163+0000","last_fullsized":"2024-09-13T13:18:53.069163+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:02.211007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.211007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:26.585228+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":310,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068501+0000","last_change":"2024-09-13T13:15:03.348187+0000","last_active":"2024-09-13T13:18:53.068501+0000","last_peered":"2024-09-13T13:18:53.068501+0000","last_clean":"2024-09-13T13:18:53.068501+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:18:53.068501+0000","last_undegraded":"2024-09-13T13:18:53.068501+0000","last_fullsized":"2024-09-13T13:18:53.068501+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:15:03.348115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.348115+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:10.491929+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":845,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.081938+0000","last_change":"2024-09-13T13:13:54.122282+0000","last_active":"2024-09-13T13:18:53.081938+0000","last_peered":"2024-09-13T13:18:53.081938+0000","last_clean":"2024-09-13T13:18:53.081938+0000","last_became_active":"2024-09-13T13:12:52.999970+0000","last_became_peered":"2024-09-13T13:12:52.999970+0000","last_unstale":"2024-09-13T13:18:53.081938+0000","last_undegraded":"2024-09-13T13:18:53.081938+0000","last_fullsized":"2024-09-13T13:18:53.081938+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.122222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.122222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:19.834956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":812,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068496+0000","last_change":"2024-09-13T13:18:34.846273+0000","last_active":"2024-09-13T13:18:53.068496+0000","last_peered":"2024-09-13T13:18:53.068496+0000","last_clean":"2024-09-13T13:18:53.068496+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:18:53.068496+0000","last_undegraded":"2024-09-13T13:18:53.068496+0000","last_fullsized":"2024-09-13T13:18:53.068496+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:34.846222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:34.846222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:51.829884+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":309,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069398+0000","last_change":"2024-09-13T13:17:07.444148+0000","last_active":"2024-09-13T13:18:53.069398+0000","last_peered":"2024-09-13T13:18:53.069398+0000","last_clean":"2024-09-13T13:18:53.069398+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:18:53.069398+0000","last_undegraded":"2024-09-13T13:18:53.069398+0000","last_fullsized":"2024-09-13T13:18:53.069398+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:07.443951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:07.443951+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":857,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672540+0000","last_change":"2024-09-13T13:18:45.471190+0000","last_active":"2024-09-13T13:18:47.672540+0000","last_peered":"2024-09-13T13:18:47.672540+0000","last_clean":"2024-09-13T13:18:47.672540+0000","last_became_active":"2024-09-13T13:12:53.008505+0000","last_became_peered":"2024-09-13T13:12:53.008505+0000","last_unstale":"2024-09-13T13:18:47.672540+0000","last_undegraded":"2024-09-13T13:18:47.672540+0000","last_fullsized":"2024-09-13T13:18:47.672540+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:45.470790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.470790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:15.409504+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":110,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.728385+0000","last_change":"2024-09-13T13:18:37.714594+0000","last_active":"2024-09-13T13:18:51.728385+0000","last_peered":"2024-09-13T13:18:51.728385+0000","last_clean":"2024-09-13T13:18:51.728385+0000","last_became_active":"2024-09-13T13:17:25.245658+0000","last_became_peered":"2024-09-13T13:17:25.245658+0000","last_unstale":"2024-09-13T13:18:51.728385+0000","last_undegraded":"2024-09-13T13:18:51.728385+0000","last_fullsized":"2024-09-13T13:18:51.728385+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:18:37.714503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:37.714503+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:41.615700+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":851,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.686133+0000","last_change":"2024-09-13T13:16:54.229157+0000","last_active":"2024-09-13T13:18:47.686133+0000","last_peered":"2024-09-13T13:18:47.686133+0000","last_clean":"2024-09-13T13:18:47.686133+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:18:47.686133+0000","last_undegraded":"2024-09-13T13:18:47.686133+0000","last_fullsized":"2024-09-13T13:18:47.686133+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:54.229075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:54.229075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:17.035251+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":830,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.715093+0000","last_change":"2024-09-13T13:18:41.949104+0000","last_active":"2024-09-13T13:18:51.715093+0000","last_peered":"2024-09-13T13:18:51.715093+0000","last_clean":"2024-09-13T13:18:51.715093+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:18:51.715093+0000","last_undegraded":"2024-09-13T13:18:51.715093+0000","last_fullsized":"2024-09-13T13:18:51.715093+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:41.949029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.949029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:49.617995+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":334,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067224+0000","last_change":"2024-09-13T13:18:46.181826+0000","last_active":"2024-09-13T13:18:53.067224+0000","last_peered":"2024-09-13T13:18:53.067224+0000","last_clean":"2024-09-13T13:18:53.067224+0000","last_became_active":"2024-09-13T13:17:25.244839+0000","last_became_peered":"2024-09-13T13:17:25.244839+0000","last_unstale":"2024-09-13T13:18:53.067224+0000","last_undegraded":"2024-09-13T13:18:53.067224+0000","last_fullsized":"2024-09-13T13:18:53.067224+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:46.181741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.181741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:01.669320+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":915,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.714723+0000","last_change":"2024-09-13T13:18:35.281750+0000","last_active":"2024-09-13T13:18:51.714723+0000","last_peered":"2024-09-13T13:18:51.714723+0000","last_clean":"2024-09-13T13:18:51.714723+0000","last_became_active":"2024-09-13T13:18:35.239702+0000","last_became_peered":"2024-09-13T13:18:35.239702+0000","last_unstale":"2024-09-13T13:18:51.714723+0000","last_undegraded":"2024-09-13T13:18:51.714723+0000","last_fullsized":"2024-09-13T13:18:51.714723+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:17:56.001763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:56.001763+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:56.286987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":328,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067860+0000","last_change":"2024-09-13T13:18:32.362014+0000","last_active":"2024-09-13T13:18:53.067860+0000","last_peered":"2024-09-13T13:18:53.067860+0000","last_clean":"2024-09-13T13:18:53.067860+0000","last_became_active":"2024-09-13T13:17:25.245239+0000","last_became_peered":"2024-09-13T13:17:25.245239+0000","last_unstale":"2024-09-13T13:18:53.067860+0000","last_undegraded":"2024-09-13T13:18:53.067860+0000","last_fullsized":"2024-09-13T13:18:53.067860+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:32.361971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:32.361971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:43.233668+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":361,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068911+0000","last_change":"2024-09-13T13:17:25.339695+0000","last_active":"2024-09-13T13:18:53.068911+0000","last_peered":"2024-09-13T13:18:53.068911+0000","last_clean":"2024-09-13T13:18:53.068911+0000","last_became_active":"2024-09-13T13:17:25.329235+0000","last_became_peered":"2024-09-13T13:17:25.329235+0000","last_unstale":"2024-09-13T13:18:53.068911+0000","last_undegraded":"2024-09-13T13:18:53.068911+0000","last_fullsized":"2024-09-13T13:18:53.068911+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:05.207445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.207445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:29.581260+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":852,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.675698+0000","last_change":"2024-09-13T13:17:01.162092+0000","last_active":"2024-09-13T13:18:47.675698+0000","last_peered":"2024-09-13T13:18:47.675698+0000","last_clean":"2024-09-13T13:18:47.675698+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:18:47.675698+0000","last_undegraded":"2024-09-13T13:18:47.675698+0000","last_fullsized":"2024-09-13T13:18:47.675698+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:17:01.162018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:01.162018+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:25.064905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":878,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.677131+0000","last_change":"2024-09-13T13:18:41.472174+0000","last_active":"2024-09-13T13:18:47.677131+0000","last_peered":"2024-09-13T13:18:47.677131+0000","last_clean":"2024-09-13T13:18:47.677131+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:18:47.677131+0000","last_undegraded":"2024-09-13T13:18:47.677131+0000","last_fullsized":"2024-09-13T13:18:47.677131+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:18:41.472008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.472008+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:56.597353+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"306'12","reported_seq":1023,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.662980+0000","last_change":"2024-09-13T13:14:21.530680+0000","last_active":"2024-09-13T13:18:47.662980+0000","last_peered":"2024-09-13T13:18:47.662980+0000","last_clean":"2024-09-13T13:18:47.662980+0000","last_became_active":"2024-09-13T13:12:52.947886+0000","last_became_peered":"2024-09-13T13:12:52.947886+0000","last_unstale":"2024-09-13T13:18:47.662980+0000","last_undegraded":"2024-09-13T13:18:47.662980+0000","last_fullsized":"2024-09-13T13:18:47.662980+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'12","last_scrub_stamp":"2024-09-13T13:14:21.530612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:21.530612+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"445'5083","reported_seq":5138,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:49.181699+0000","last_change":"2024-09-13T13:17:55.200504+0000","last_active":"2024-09-13T13:18:49.181699+0000","last_peered":"2024-09-13T13:18:49.181699+0000","last_clean":"2024-09-13T13:18:49.181699+0000","last_became_active":"2024-09-13T13:17:55.200162+0000","last_became_peered":"2024-09-13T13:17:55.200162+0000","last_unstale":"2024-09-13T13:18:49.181699+0000","last_undegraded":"2024-09-13T13:18:49.181699+0000","last_fullsized":"2024-09-13T13:18:49.181699+0000","mapping_epoch":423,"log_start":"427'300","ondisk_log_start":"427'300","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:09.157335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333119488,"num_objects":5083,"num_object_clones":0,"num_object_copies":15249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5083,"num_write_kb":325312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":78,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068957+0000","last_change":"2024-09-13T13:18:49.869928+0000","last_active":"2024-09-13T13:18:53.068957+0000","last_peered":"2024-09-13T13:18:53.068957+0000","last_clean":"2024-09-13T13:18:53.068957+0000","last_became_active":"2024-09-13T13:17:49.052966+0000","last_became_peered":"2024-09-13T13:17:49.052966+0000","last_unstale":"2024-09-13T13:18:53.068957+0000","last_undegraded":"2024-09-13T13:18:53.068957+0000","last_fullsized":"2024-09-13T13:18:53.068957+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:49.869851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:49.869851+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:57.610765+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":930,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.687761+0000","last_change":"2024-09-13T13:18:05.102318+0000","last_active":"2024-09-13T13:18:47.687761+0000","last_peered":"2024-09-13T13:18:47.687761+0000","last_clean":"2024-09-13T13:18:47.687761+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:18:47.687761+0000","last_undegraded":"2024-09-13T13:18:47.687761+0000","last_fullsized":"2024-09-13T13:18:47.687761+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:18:05.102223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:05.102223+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:29.926605+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":855,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.717660+0000","last_change":"2024-09-13T13:16:58.980151+0000","last_active":"2024-09-13T13:18:51.717660+0000","last_peered":"2024-09-13T13:18:51.717660+0000","last_clean":"2024-09-13T13:18:51.717660+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:18:51.717660+0000","last_undegraded":"2024-09-13T13:18:51.717660+0000","last_fullsized":"2024-09-13T13:18:51.717660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:58.980098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.980098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":381,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069453+0000","last_change":"2024-09-13T13:15:32.212185+0000","last_active":"2024-09-13T13:18:53.069453+0000","last_peered":"2024-09-13T13:18:53.069453+0000","last_clean":"2024-09-13T13:18:53.069453+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:18:53.069453+0000","last_undegraded":"2024-09-13T13:18:53.069453+0000","last_fullsized":"2024-09-13T13:18:53.069453+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:15:32.212101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.212101+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:25.305722+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":828,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068950+0000","last_change":"2024-09-13T13:13:57.546725+0000","last_active":"2024-09-13T13:18:53.068950+0000","last_peered":"2024-09-13T13:18:53.068950+0000","last_clean":"2024-09-13T13:18:53.068950+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:18:53.068950+0000","last_undegraded":"2024-09-13T13:18:53.068950+0000","last_fullsized":"2024-09-13T13:18:53.068950+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.546671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.546671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"445'4959","reported_seq":5014,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:49.214248+0000","last_change":"2024-09-13T13:17:55.200709+0000","last_active":"2024-09-13T13:18:49.214248+0000","last_peered":"2024-09-13T13:18:49.214248+0000","last_clean":"2024-09-13T13:18:49.214248+0000","last_became_active":"2024-09-13T13:17:55.200283+0000","last_became_peered":"2024-09-13T13:17:55.200283+0000","last_unstale":"2024-09-13T13:18:49.214248+0000","last_undegraded":"2024-09-13T13:18:49.214248+0000","last_fullsized":"2024-09-13T13:18:49.214248+0000","mapping_epoch":423,"log_start":"427'100","ondisk_log_start":"427'100","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:00.586409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324993024,"num_objects":4959,"num_object_clones":0,"num_object_copies":14877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4959,"num_write_kb":317376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":78,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070398+0000","last_change":"2024-09-13T13:18:51.159546+0000","last_active":"2024-09-13T13:18:53.070398+0000","last_peered":"2024-09-13T13:18:53.070398+0000","last_clean":"2024-09-13T13:18:53.070398+0000","last_became_active":"2024-09-13T13:17:49.050655+0000","last_became_peered":"2024-09-13T13:17:49.050655+0000","last_unstale":"2024-09-13T13:18:53.070398+0000","last_undegraded":"2024-09-13T13:18:53.070398+0000","last_fullsized":"2024-09-13T13:18:53.070398+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.159496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.159496+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:01.476223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":869,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.716523+0000","last_change":"2024-09-13T13:18:36.922068+0000","last_active":"2024-09-13T13:18:51.716523+0000","last_peered":"2024-09-13T13:18:51.716523+0000","last_clean":"2024-09-13T13:18:51.716523+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:18:51.716523+0000","last_undegraded":"2024-09-13T13:18:51.716523+0000","last_fullsized":"2024-09-13T13:18:51.716523+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:18:36.921964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:36.921964+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:40.263721+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":829,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.715399+0000","last_change":"2024-09-13T13:16:26.098415+0000","last_active":"2024-09-13T13:18:51.715399+0000","last_peered":"2024-09-13T13:18:51.715399+0000","last_clean":"2024-09-13T13:18:51.715399+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:18:51.715399+0000","last_undegraded":"2024-09-13T13:18:51.715399+0000","last_fullsized":"2024-09-13T13:18:51.715399+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:16:26.098365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:26.098365+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:30.772726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":312,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.066930+0000","last_change":"2024-09-13T13:17:58.389822+0000","last_active":"2024-09-13T13:18:53.066930+0000","last_peered":"2024-09-13T13:18:53.066930+0000","last_clean":"2024-09-13T13:18:53.066930+0000","last_became_active":"2024-09-13T13:12:53.006199+0000","last_became_peered":"2024-09-13T13:12:53.006199+0000","last_unstale":"2024-09-13T13:18:53.066930+0000","last_undegraded":"2024-09-13T13:18:53.066930+0000","last_fullsized":"2024-09-13T13:18:53.066930+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:58.389763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:58.389763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:19.447538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":334,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067540+0000","last_change":"2024-09-13T13:17:59.373240+0000","last_active":"2024-09-13T13:18:53.067540+0000","last_peered":"2024-09-13T13:18:53.067540+0000","last_clean":"2024-09-13T13:18:53.067540+0000","last_became_active":"2024-09-13T13:17:25.244171+0000","last_became_peered":"2024-09-13T13:17:25.244171+0000","last_unstale":"2024-09-13T13:18:53.067540+0000","last_undegraded":"2024-09-13T13:18:53.067540+0000","last_fullsized":"2024-09-13T13:18:53.067540+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.373194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.373194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:20.574739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"447'5175","reported_seq":5234,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.255590+0000","last_change":"2024-09-13T13:17:55.200309+0000","last_active":"2024-09-13T13:18:53.255590+0000","last_peered":"2024-09-13T13:18:53.255590+0000","last_clean":"2024-09-13T13:18:53.255590+0000","last_became_active":"2024-09-13T13:17:55.200037+0000","last_became_peered":"2024-09-13T13:17:55.200037+0000","last_unstale":"2024-09-13T13:18:53.255590+0000","last_undegraded":"2024-09-13T13:18:53.255590+0000","last_fullsized":"2024-09-13T13:18:53.255590+0000","mapping_epoch":423,"log_start":"427'500","ondisk_log_start":"427'500","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4675,"log_dups_size":0,"ondisk_log_size":4675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:01.812502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339148800,"num_objects":5175,"num_object_clones":0,"num_object_copies":15525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":331200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":76,"reported_epoch":446,"state":"active+clean+scrubbing","last_fresh":"2024-09-13T13:18:52.041719+0000","last_change":"2024-09-13T13:18:52.041559+0000","last_active":"2024-09-13T13:18:52.041719+0000","last_peered":"2024-09-13T13:18:52.041719+0000","last_clean":"2024-09-13T13:18:52.041719+0000","last_became_active":"2024-09-13T13:17:49.053135+0000","last_became_peered":"2024-09-13T13:17:49.053135+0000","last_unstale":"2024-09-13T13:18:52.041719+0000","last_undegraded":"2024-09-13T13:18:52.041719+0000","last_fullsized":"2024-09-13T13:18:52.041719+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":320,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067090+0000","last_change":"2024-09-13T13:13:50.716312+0000","last_active":"2024-09-13T13:18:53.067090+0000","last_peered":"2024-09-13T13:18:53.067090+0000","last_clean":"2024-09-13T13:18:53.067090+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:18:53.067090+0000","last_undegraded":"2024-09-13T13:18:53.067090+0000","last_fullsized":"2024-09-13T13:18:53.067090+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:13:50.716243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.716243+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:22.408438+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":835,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.678161+0000","last_change":"2024-09-13T13:17:02.704294+0000","last_active":"2024-09-13T13:18:47.678161+0000","last_peered":"2024-09-13T13:18:47.678161+0000","last_clean":"2024-09-13T13:18:47.678161+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:18:47.678161+0000","last_undegraded":"2024-09-13T13:18:47.678161+0000","last_fullsized":"2024-09-13T13:18:47.678161+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:17:02.704233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.704233+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:21.560467+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":1077,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.716384+0000","last_change":"2024-09-13T13:18:48.032272+0000","last_active":"2024-09-13T13:18:51.716384+0000","last_peered":"2024-09-13T13:18:51.716384+0000","last_clean":"2024-09-13T13:18:51.716384+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:18:51.716384+0000","last_undegraded":"2024-09-13T13:18:51.716384+0000","last_fullsized":"2024-09-13T13:18:51.716384+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:18:48.032199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:48.032199+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:17.576150+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":472,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.662176+0000","last_change":"2024-09-13T13:17:59.355700+0000","last_active":"2024-09-13T13:18:47.662176+0000","last_peered":"2024-09-13T13:18:47.662176+0000","last_clean":"2024-09-13T13:18:47.662176+0000","last_became_active":"2024-09-13T13:17:25.244188+0000","last_became_peered":"2024-09-13T13:17:25.244188+0000","last_unstale":"2024-09-13T13:18:47.662176+0000","last_undegraded":"2024-09-13T13:18:47.662176+0000","last_fullsized":"2024-09-13T13:18:47.662176+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.355639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.355639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:00.304064+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"446'5039","reported_seq":5096,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:52.925773+0000","last_change":"2024-09-13T13:17:55.131505+0000","last_active":"2024-09-13T13:18:52.925773+0000","last_peered":"2024-09-13T13:18:52.925773+0000","last_clean":"2024-09-13T13:18:52.925773+0000","last_became_active":"2024-09-13T13:17:55.131230+0000","last_became_peered":"2024-09-13T13:17:55.131230+0000","last_unstale":"2024-09-13T13:18:52.925773+0000","last_undegraded":"2024-09-13T13:18:52.925773+0000","last_fullsized":"2024-09-13T13:18:52.925773+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:59.022573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330235904,"num_objects":5039,"num_object_clones":0,"num_object_copies":15117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5039,"num_write_kb":322496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":150,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:50.404870+0000","last_change":"2024-09-13T13:18:50.404870+0000","last_active":"2024-09-13T13:18:50.404870+0000","last_peered":"2024-09-13T13:18:50.404870+0000","last_clean":"2024-09-13T13:18:50.404870+0000","last_became_active":"2024-09-13T13:17:49.052271+0000","last_became_peered":"2024-09-13T13:17:49.052271+0000","last_unstale":"2024-09-13T13:18:50.404870+0000","last_undegraded":"2024-09-13T13:18:50.404870+0000","last_fullsized":"2024-09-13T13:18:50.404870+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:50.404821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:50.404821+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:55.364511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":886,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.676309+0000","last_change":"2024-09-13T13:12:53.016127+0000","last_active":"2024-09-13T13:18:47.676309+0000","last_peered":"2024-09-13T13:18:47.676309+0000","last_clean":"2024-09-13T13:18:47.676309+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:18:47.676309+0000","last_undegraded":"2024-09-13T13:18:47.676309+0000","last_fullsized":"2024-09-13T13:18:47.676309+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:48.110144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":871,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069564+0000","last_change":"2024-09-13T13:17:25.331470+0000","last_active":"2024-09-13T13:18:53.069564+0000","last_peered":"2024-09-13T13:18:53.069564+0000","last_clean":"2024-09-13T13:18:53.069564+0000","last_became_active":"2024-09-13T13:17:25.331166+0000","last_became_peered":"2024-09-13T13:17:25.331166+0000","last_unstale":"2024-09-13T13:18:53.069564+0000","last_undegraded":"2024-09-13T13:18:53.069564+0000","last_fullsized":"2024-09-13T13:18:53.069564+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:04.755828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:04.755828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.345827+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":1056,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672120+0000","last_change":"2024-09-13T13:18:40.464456+0000","last_active":"2024-09-13T13:18:47.672120+0000","last_peered":"2024-09-13T13:18:47.672120+0000","last_clean":"2024-09-13T13:18:47.672120+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:18:47.672120+0000","last_undegraded":"2024-09-13T13:18:47.672120+0000","last_fullsized":"2024-09-13T13:18:47.672120+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:18:40.464398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:40.464398+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:58.756596+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":862,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672270+0000","last_change":"2024-09-13T13:18:36.466657+0000","last_active":"2024-09-13T13:18:47.672270+0000","last_peered":"2024-09-13T13:18:47.672270+0000","last_clean":"2024-09-13T13:18:47.672270+0000","last_became_active":"2024-09-13T13:17:25.252712+0000","last_became_peered":"2024-09-13T13:17:25.252712+0000","last_unstale":"2024-09-13T13:18:47.672270+0000","last_undegraded":"2024-09-13T13:18:47.672270+0000","last_fullsized":"2024-09-13T13:18:47.672270+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:36.466575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:36.466575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:56.925016+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"445'4929","reported_seq":4984,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:49.225105+0000","last_change":"2024-09-13T13:17:55.140405+0000","last_active":"2024-09-13T13:18:49.225105+0000","last_peered":"2024-09-13T13:18:49.225105+0000","last_clean":"2024-09-13T13:18:49.225105+0000","last_became_active":"2024-09-13T13:17:55.140134+0000","last_became_peered":"2024-09-13T13:17:55.140134+0000","last_unstale":"2024-09-13T13:18:49.225105+0000","last_undegraded":"2024-09-13T13:18:49.225105+0000","last_fullsized":"2024-09-13T13:18:49.225105+0000","mapping_epoch":423,"log_start":"427'100","ondisk_log_start":"427'100","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4829,"log_dups_size":0,"ondisk_log_size":4829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.372449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323026944,"num_objects":4929,"num_object_clones":0,"num_object_copies":14787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4929,"num_write_kb":315456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":150,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.388593+0000","last_change":"2024-09-13T13:18:51.388593+0000","last_active":"2024-09-13T13:18:51.388593+0000","last_peered":"2024-09-13T13:18:51.388593+0000","last_clean":"2024-09-13T13:18:51.388593+0000","last_became_active":"2024-09-13T13:17:49.055888+0000","last_became_peered":"2024-09-13T13:17:49.055888+0000","last_unstale":"2024-09-13T13:18:51.388593+0000","last_undegraded":"2024-09-13T13:18:51.388593+0000","last_fullsized":"2024-09-13T13:18:51.388593+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.388502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.388502+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:16.343178+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":899,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672319+0000","last_change":"2024-09-13T13:18:33.796717+0000","last_active":"2024-09-13T13:18:47.672319+0000","last_peered":"2024-09-13T13:18:47.672319+0000","last_clean":"2024-09-13T13:18:47.672319+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:18:47.672319+0000","last_undegraded":"2024-09-13T13:18:47.672319+0000","last_fullsized":"2024-09-13T13:18:47.672319+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:18:33.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.796644+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:36.008576+0000","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":844,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.678179+0000","last_change":"2024-09-13T13:17:53.069585+0000","last_active":"2024-09-13T13:18:47.678179+0000","last_peered":"2024-09-13T13:18:47.678179+0000","last_clean":"2024-09-13T13:18:47.678179+0000","last_became_active":"2024-09-13T13:17:25.332589+0000","last_became_peered":"2024-09-13T13:17:25.332589+0000","last_unstale":"2024-09-13T13:18:47.678179+0000","last_undegraded":"2024-09-13T13:18:47.678179+0000","last_fullsized":"2024-09-13T13:18:47.678179+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:53.069538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:53.069538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:19.459005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":806,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.661926+0000","last_change":"2024-09-13T13:17:58.379729+0000","last_active":"2024-09-13T13:18:47.661926+0000","last_peered":"2024-09-13T13:18:47.661926+0000","last_clean":"2024-09-13T13:18:47.661926+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:18:47.661926+0000","last_undegraded":"2024-09-13T13:18:47.661926+0000","last_fullsized":"2024-09-13T13:18:47.661926+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:58.379674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:58.379674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:16.089064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":851,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.675459+0000","last_change":"2024-09-13T13:17:59.178198+0000","last_active":"2024-09-13T13:18:47.675459+0000","last_peered":"2024-09-13T13:18:47.675459+0000","last_clean":"2024-09-13T13:18:47.675459+0000","last_became_active":"2024-09-13T13:17:25.244517+0000","last_became_peered":"2024-09-13T13:17:25.244517+0000","last_unstale":"2024-09-13T13:18:47.675459+0000","last_undegraded":"2024-09-13T13:18:47.675459+0000","last_fullsized":"2024-09-13T13:18:47.675459+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.178129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.178129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.299906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"446'5021","reported_seq":5078,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:52.931745+0000","last_change":"2024-09-13T13:17:55.146287+0000","last_active":"2024-09-13T13:18:52.931745+0000","last_peered":"2024-09-13T13:18:52.931745+0000","last_clean":"2024-09-13T13:18:52.931745+0000","last_became_active":"2024-09-13T13:17:55.145616+0000","last_became_peered":"2024-09-13T13:17:55.145616+0000","last_unstale":"2024-09-13T13:18:52.931745+0000","last_undegraded":"2024-09-13T13:18:52.931745+0000","last_fullsized":"2024-09-13T13:18:52.931745+0000","mapping_epoch":423,"log_start":"427'500","ondisk_log_start":"427'500","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:20.121854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329056256,"num_objects":5021,"num_object_clones":0,"num_object_copies":15063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":321344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":78,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068352+0000","last_change":"2024-09-13T13:18:50.856493+0000","last_active":"2024-09-13T13:18:53.068352+0000","last_peered":"2024-09-13T13:18:53.068352+0000","last_clean":"2024-09-13T13:18:53.068352+0000","last_became_active":"2024-09-13T13:17:49.054449+0000","last_became_peered":"2024-09-13T13:17:49.054449+0000","last_unstale":"2024-09-13T13:18:53.068352+0000","last_undegraded":"2024-09-13T13:18:53.068352+0000","last_fullsized":"2024-09-13T13:18:53.068352+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:50.856437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:50.856437+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:03.097517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"443'43","reported_seq":43,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672509+0000","last_change":"2024-09-13T13:18:35.332086+0000","last_active":"2024-09-13T13:18:47.672509+0000","last_peered":"2024-09-13T13:18:47.672509+0000","last_clean":"2024-09-13T13:18:47.672509+0000","last_became_active":"2024-09-13T13:18:35.244311+0000","last_became_peered":"2024-09-13T13:18:35.244311+0000","last_unstale":"2024-09-13T13:18:47.672509+0000","last_undegraded":"2024-09-13T13:18:47.672509+0000","last_fullsized":"2024-09-13T13:18:47.672509+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'42","last_scrub_stamp":"2024-09-13T13:18:01.158345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:01.158345+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:27.731963+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":320,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068594+0000","last_change":"2024-09-13T13:17:52.422920+0000","last_active":"2024-09-13T13:18:53.068594+0000","last_peered":"2024-09-13T13:18:53.068594+0000","last_clean":"2024-09-13T13:18:53.068594+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:18:53.068594+0000","last_undegraded":"2024-09-13T13:18:53.068594+0000","last_fullsized":"2024-09-13T13:18:53.068594+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:52.422872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.422872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:02.132631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":852,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069427+0000","last_change":"2024-09-13T13:18:07.306441+0000","last_active":"2024-09-13T13:18:53.069427+0000","last_peered":"2024-09-13T13:18:53.069427+0000","last_clean":"2024-09-13T13:18:53.069427+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:18:53.069427+0000","last_undegraded":"2024-09-13T13:18:53.069427+0000","last_fullsized":"2024-09-13T13:18:53.069427+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:18:07.306366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:07.306366+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:17.328610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":842,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672663+0000","last_change":"2024-09-13T13:14:59.670853+0000","last_active":"2024-09-13T13:18:47.672663+0000","last_peered":"2024-09-13T13:18:47.672663+0000","last_clean":"2024-09-13T13:18:47.672663+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:18:47.672663+0000","last_undegraded":"2024-09-13T13:18:47.672663+0000","last_fullsized":"2024-09-13T13:18:47.672663+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.670779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.670779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:18.561846+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":841,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.715607+0000","last_change":"2024-09-13T13:17:05.016876+0000","last_active":"2024-09-13T13:18:51.715607+0000","last_peered":"2024-09-13T13:18:51.715607+0000","last_clean":"2024-09-13T13:18:51.715607+0000","last_became_active":"2024-09-13T13:12:52.948742+0000","last_became_peered":"2024-09-13T13:12:52.948742+0000","last_unstale":"2024-09-13T13:18:51.715607+0000","last_undegraded":"2024-09-13T13:18:51.715607+0000","last_fullsized":"2024-09-13T13:18:51.715607+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:05.016806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.016806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"446'5220","reported_seq":5277,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:52.929445+0000","last_change":"2024-09-13T13:17:55.132888+0000","last_active":"2024-09-13T13:18:52.929445+0000","last_peered":"2024-09-13T13:18:52.929445+0000","last_clean":"2024-09-13T13:18:52.929445+0000","last_became_active":"2024-09-13T13:17:55.132636+0000","last_became_peered":"2024-09-13T13:17:55.132636+0000","last_unstale":"2024-09-13T13:18:52.929445+0000","last_undegraded":"2024-09-13T13:18:52.929445+0000","last_fullsized":"2024-09-13T13:18:52.929445+0000","mapping_epoch":423,"log_start":"427'700","ondisk_log_start":"427'700","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4520,"log_dups_size":0,"ondisk_log_size":4520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.085485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342097920,"num_objects":5220,"num_object_clones":0,"num_object_copies":15660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5220,"num_write_kb":334080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":428,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.679141+0000","last_change":"2024-09-13T13:18:21.088014+0000","last_active":"2024-09-13T13:18:47.679141+0000","last_peered":"2024-09-13T13:18:47.679141+0000","last_clean":"2024-09-13T13:18:47.679141+0000","last_became_active":"2024-09-13T13:17:49.057194+0000","last_became_peered":"2024-09-13T13:17:49.057194+0000","last_unstale":"2024-09-13T13:18:47.679141+0000","last_undegraded":"2024-09-13T13:18:47.679141+0000","last_fullsized":"2024-09-13T13:18:47.679141+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:18:50.548532+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":859,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.687883+0000","last_change":"2024-09-13T13:17:32.264550+0000","last_active":"2024-09-13T13:18:47.687883+0000","last_peered":"2024-09-13T13:18:47.687883+0000","last_clean":"2024-09-13T13:18:47.687883+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:18:47.687883+0000","last_undegraded":"2024-09-13T13:18:47.687883+0000","last_fullsized":"2024-09-13T13:18:47.687883+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'17","last_scrub_stamp":"2024-09-13T13:17:32.263873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:32.263873+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:01.566837+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":871,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.684321+0000","last_change":"2024-09-13T13:18:33.892926+0000","last_active":"2024-09-13T13:18:47.684321+0000","last_peered":"2024-09-13T13:18:47.684321+0000","last_clean":"2024-09-13T13:18:47.684321+0000","last_became_active":"2024-09-13T13:17:25.246400+0000","last_became_peered":"2024-09-13T13:17:25.246400+0000","last_unstale":"2024-09-13T13:18:47.684321+0000","last_undegraded":"2024-09-13T13:18:47.684321+0000","last_fullsized":"2024-09-13T13:18:47.684321+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:33.892597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.892597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:39.873900+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":98,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.719556+0000","last_change":"2024-09-13T13:18:44.719355+0000","last_active":"2024-09-13T13:18:51.719556+0000","last_peered":"2024-09-13T13:18:51.719556+0000","last_clean":"2024-09-13T13:18:51.719556+0000","last_became_active":"2024-09-13T13:17:25.252664+0000","last_became_peered":"2024-09-13T13:17:25.252664+0000","last_unstale":"2024-09-13T13:18:51.719556+0000","last_undegraded":"2024-09-13T13:18:51.719556+0000","last_fullsized":"2024-09-13T13:18:51.719556+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:18:44.719269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:44.719269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:12.759919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":94,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.723658+0000","last_change":"2024-09-13T13:17:59.698099+0000","last_active":"2024-09-13T13:18:51.723658+0000","last_peered":"2024-09-13T13:18:51.723658+0000","last_clean":"2024-09-13T13:18:51.723658+0000","last_became_active":"2024-09-13T13:17:25.332669+0000","last_became_peered":"2024-09-13T13:17:25.332669+0000","last_unstale":"2024-09-13T13:18:51.723658+0000","last_undegraded":"2024-09-13T13:18:51.723658+0000","last_fullsized":"2024-09-13T13:18:51.723658+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.698047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.698047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.194640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":442,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.662458+0000","last_change":"2024-09-13T13:12:41.332308+0000","last_active":"2024-09-13T13:18:47.662458+0000","last_peered":"2024-09-13T13:18:47.662458+0000","last_clean":"2024-09-13T13:18:47.662458+0000","last_became_active":"2024-09-13T13:12:40.120242+0000","last_became_peered":"2024-09-13T13:12:40.120242+0000","last_unstale":"2024-09-13T13:18:47.662458+0000","last_undegraded":"2024-09-13T13:18:47.662458+0000","last_fullsized":"2024-09-13T13:18:47.662458+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:41.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:41.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:25.287809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"446'5152","reported_seq":5209,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:52.913820+0000","last_change":"2024-09-13T13:17:55.143535+0000","last_active":"2024-09-13T13:18:52.913820+0000","last_peered":"2024-09-13T13:18:52.913820+0000","last_clean":"2024-09-13T13:18:52.913820+0000","last_became_active":"2024-09-13T13:17:55.143252+0000","last_became_peered":"2024-09-13T13:17:55.143252+0000","last_unstale":"2024-09-13T13:18:52.913820+0000","last_undegraded":"2024-09-13T13:18:52.913820+0000","last_fullsized":"2024-09-13T13:18:52.913820+0000","mapping_epoch":423,"log_start":"427'600","ondisk_log_start":"427'600","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4552,"log_dups_size":0,"ondisk_log_size":4552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:14.112065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337641472,"num_objects":5152,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5152,"num_write_kb":329728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":66,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.677400+0000","last_change":"2024-09-13T13:18:21.083672+0000","last_active":"2024-09-13T13:18:47.677400+0000","last_peered":"2024-09-13T13:18:47.677400+0000","last_clean":"2024-09-13T13:18:47.677400+0000","last_became_active":"2024-09-13T13:17:49.049531+0000","last_became_peered":"2024-09-13T13:17:49.049531+0000","last_unstale":"2024-09-13T13:18:47.677400+0000","last_undegraded":"2024-09-13T13:18:47.677400+0000","last_fullsized":"2024-09-13T13:18:47.677400+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:18:50.094646+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":910,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.677188+0000","last_change":"2024-09-13T13:18:04.569761+0000","last_active":"2024-09-13T13:18:47.677188+0000","last_peered":"2024-09-13T13:18:47.677188+0000","last_clean":"2024-09-13T13:18:47.677188+0000","last_became_active":"2024-09-13T13:17:25.253120+0000","last_became_peered":"2024-09-13T13:17:25.253120+0000","last_unstale":"2024-09-13T13:18:47.677188+0000","last_undegraded":"2024-09-13T13:18:47.677188+0000","last_fullsized":"2024-09-13T13:18:47.677188+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:18:04.569670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:04.569670+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:14.912474+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":850,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.673133+0000","last_change":"2024-09-13T13:18:37.465516+0000","last_active":"2024-09-13T13:18:47.673133+0000","last_peered":"2024-09-13T13:18:47.673133+0000","last_clean":"2024-09-13T13:18:47.673133+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:18:47.673133+0000","last_undegraded":"2024-09-13T13:18:47.673133+0000","last_fullsized":"2024-09-13T13:18:47.673133+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:37.465371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:37.465371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:41.600910+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":742,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.811892+0000","last_change":"2024-09-13T13:17:25.325347+0000","last_active":"2024-09-13T13:18:47.811892+0000","last_peered":"2024-09-13T13:18:47.811892+0000","last_clean":"2024-09-13T13:18:47.811892+0000","last_became_active":"2024-09-13T13:12:39.947940+0000","last_became_peered":"2024-09-13T13:12:39.947940+0000","last_unstale":"2024-09-13T13:18:47.811892+0000","last_undegraded":"2024-09-13T13:18:47.811892+0000","last_fullsized":"2024-09-13T13:18:47.811892+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:17:25.325275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:25.325275+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:51.942457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":370,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":825,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070455+0000","last_change":"2024-09-13T13:17:55.293423+0000","last_active":"2024-09-13T13:18:53.070455+0000","last_peered":"2024-09-13T13:18:53.070455+0000","last_clean":"2024-09-13T13:18:53.070455+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:18:53.070455+0000","last_undegraded":"2024-09-13T13:18:53.070455+0000","last_fullsized":"2024-09-13T13:18:53.070455+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:55.293375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:55.293375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:20.691046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":826,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.677729+0000","last_change":"2024-09-13T13:18:46.991251+0000","last_active":"2024-09-13T13:18:47.677729+0000","last_peered":"2024-09-13T13:18:47.677729+0000","last_clean":"2024-09-13T13:18:47.677729+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:18:47.677729+0000","last_undegraded":"2024-09-13T13:18:47.677729+0000","last_fullsized":"2024-09-13T13:18:47.677729+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:46.991076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.991076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:06.527735+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"446'5186","reported_seq":5242,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:52.947314+0000","last_change":"2024-09-13T13:17:55.142415+0000","last_active":"2024-09-13T13:18:52.947314+0000","last_peered":"2024-09-13T13:18:52.947314+0000","last_clean":"2024-09-13T13:18:52.947314+0000","last_became_active":"2024-09-13T13:17:55.142112+0000","last_became_peered":"2024-09-13T13:17:55.142112+0000","last_unstale":"2024-09-13T13:18:52.947314+0000","last_undegraded":"2024-09-13T13:18:52.947314+0000","last_fullsized":"2024-09-13T13:18:52.947314+0000","mapping_epoch":423,"log_start":"427'700","ondisk_log_start":"427'700","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4486,"log_dups_size":0,"ondisk_log_size":4486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:01.465992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339869696,"num_objects":5186,"num_object_clones":0,"num_object_copies":15558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5186,"num_write_kb":331904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"306'32","reported_seq":981,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.662227+0000","last_change":"2024-09-13T13:13:01.352416+0000","last_active":"2024-09-13T13:18:47.662227+0000","last_peered":"2024-09-13T13:18:47.662227+0000","last_clean":"2024-09-13T13:18:47.662227+0000","last_became_active":"2024-09-13T13:12:52.999924+0000","last_became_peered":"2024-09-13T13:12:52.999924+0000","last_unstale":"2024-09-13T13:18:47.662227+0000","last_undegraded":"2024-09-13T13:18:47.662227+0000","last_fullsized":"2024-09-13T13:18:47.662227+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:13:01.352337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.352337+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.222564+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":842,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.714659+0000","last_change":"2024-09-13T13:16:53.951208+0000","last_active":"2024-09-13T13:18:51.714659+0000","last_peered":"2024-09-13T13:18:51.714659+0000","last_clean":"2024-09-13T13:18:51.714659+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:18:51.714659+0000","last_undegraded":"2024-09-13T13:18:51.714659+0000","last_fullsized":"2024-09-13T13:18:51.714659+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:53.951135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:53.951135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:31.816447+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":426,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672490+0000","last_change":"2024-09-13T13:18:21.086890+0000","last_active":"2024-09-13T13:18:47.672490+0000","last_peered":"2024-09-13T13:18:47.672490+0000","last_clean":"2024-09-13T13:18:47.672490+0000","last_became_active":"2024-09-13T13:17:47.097787+0000","last_became_peered":"2024-09-13T13:17:47.097787+0000","last_unstale":"2024-09-13T13:18:47.672490+0000","last_undegraded":"2024-09-13T13:18:47.672490+0000","last_fullsized":"2024-09-13T13:18:47.672490+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:18:49.582263+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":94,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.727849+0000","last_change":"2024-09-13T13:17:58.664229+0000","last_active":"2024-09-13T13:18:51.727849+0000","last_peered":"2024-09-13T13:18:51.727849+0000","last_clean":"2024-09-13T13:18:51.727849+0000","last_became_active":"2024-09-13T13:17:25.328065+0000","last_became_peered":"2024-09-13T13:17:25.328065+0000","last_unstale":"2024-09-13T13:18:51.727849+0000","last_undegraded":"2024-09-13T13:18:51.727849+0000","last_fullsized":"2024-09-13T13:18:51.727849+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:58.664177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:58.664177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:07.154650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":867,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070321+0000","last_change":"2024-09-13T13:18:08.354464+0000","last_active":"2024-09-13T13:18:53.070321+0000","last_peered":"2024-09-13T13:18:53.070321+0000","last_clean":"2024-09-13T13:18:53.070321+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:18:53.070321+0000","last_undegraded":"2024-09-13T13:18:53.070321+0000","last_fullsized":"2024-09-13T13:18:53.070321+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:08.354394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:08.354394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:09.946043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"445'4964","reported_seq":5019,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.364550+0000","last_change":"2024-09-13T13:17:55.144368+0000","last_active":"2024-09-13T13:18:51.364550+0000","last_peered":"2024-09-13T13:18:51.364550+0000","last_clean":"2024-09-13T13:18:51.364550+0000","last_became_active":"2024-09-13T13:17:55.144129+0000","last_became_peered":"2024-09-13T13:17:55.144129+0000","last_unstale":"2024-09-13T13:18:51.364550+0000","last_undegraded":"2024-09-13T13:18:51.364550+0000","last_fullsized":"2024-09-13T13:18:51.364550+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:17.686405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325320704,"num_objects":4964,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4964,"num_write_kb":317696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":332,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069146+0000","last_change":"2024-09-13T13:18:09.522990+0000","last_active":"2024-09-13T13:18:53.069146+0000","last_peered":"2024-09-13T13:18:53.069146+0000","last_clean":"2024-09-13T13:18:53.069146+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:18:53.069146+0000","last_undegraded":"2024-09-13T13:18:53.069146+0000","last_fullsized":"2024-09-13T13:18:53.069146+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:09.522908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:09.522908+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.176687+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":326,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.066840+0000","last_change":"2024-09-13T13:17:54.465421+0000","last_active":"2024-09-13T13:18:53.066840+0000","last_peered":"2024-09-13T13:18:53.066840+0000","last_clean":"2024-09-13T13:18:53.066840+0000","last_became_active":"2024-09-13T13:17:25.244810+0000","last_became_peered":"2024-09-13T13:17:25.244810+0000","last_unstale":"2024-09-13T13:18:53.066840+0000","last_undegraded":"2024-09-13T13:18:53.066840+0000","last_fullsized":"2024-09-13T13:18:53.066840+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.465363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.465363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:23.531768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":873,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070187+0000","last_change":"2024-09-13T13:15:55.592241+0000","last_active":"2024-09-13T13:18:53.070187+0000","last_peered":"2024-09-13T13:18:53.070187+0000","last_clean":"2024-09-13T13:18:53.070187+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:18:53.070187+0000","last_undegraded":"2024-09-13T13:18:53.070187+0000","last_fullsized":"2024-09-13T13:18:53.070187+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:55.592172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:55.592172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:21.490551+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":829,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.714480+0000","last_change":"2024-09-13T13:18:49.072017+0000","last_active":"2024-09-13T13:18:51.714480+0000","last_peered":"2024-09-13T13:18:51.714480+0000","last_clean":"2024-09-13T13:18:51.714480+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:18:51.714480+0000","last_undegraded":"2024-09-13T13:18:51.714480+0000","last_fullsized":"2024-09-13T13:18:51.714480+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:49.071942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:49.071942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:16.825126+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"446'5155","reported_seq":5212,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:52.906524+0000","last_change":"2024-09-13T13:17:55.139372+0000","last_active":"2024-09-13T13:18:52.906524+0000","last_peered":"2024-09-13T13:18:52.906524+0000","last_clean":"2024-09-13T13:18:52.906524+0000","last_became_active":"2024-09-13T13:17:55.139080+0000","last_became_peered":"2024-09-13T13:17:55.139080+0000","last_unstale":"2024-09-13T13:18:52.906524+0000","last_undegraded":"2024-09-13T13:18:52.906524+0000","last_fullsized":"2024-09-13T13:18:52.906524+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5155,"log_dups_size":0,"ondisk_log_size":5155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:12.883286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337838080,"num_objects":5155,"num_object_clones":0,"num_object_copies":15465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5155,"num_write_kb":329920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":79,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068092+0000","last_change":"2024-09-13T13:17:51.212127+0000","last_active":"2024-09-13T13:18:53.068092+0000","last_peered":"2024-09-13T13:18:53.068092+0000","last_clean":"2024-09-13T13:18:53.068092+0000","last_became_active":"2024-09-13T13:17:49.053269+0000","last_became_peered":"2024-09-13T13:17:49.053269+0000","last_unstale":"2024-09-13T13:18:53.068092+0000","last_undegraded":"2024-09-13T13:18:53.068092+0000","last_fullsized":"2024-09-13T13:18:53.068092+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:17:51.212071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:17:51.212071+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:10.098094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":388,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070699+0000","last_change":"2024-09-13T13:18:48.229223+0000","last_active":"2024-09-13T13:18:53.070699+0000","last_peered":"2024-09-13T13:18:53.070699+0000","last_clean":"2024-09-13T13:18:53.070699+0000","last_became_active":"2024-09-13T13:17:25.244686+0000","last_became_peered":"2024-09-13T13:17:25.244686+0000","last_unstale":"2024-09-13T13:18:53.070699+0000","last_undegraded":"2024-09-13T13:18:53.070699+0000","last_fullsized":"2024-09-13T13:18:53.070699+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:18:48.229160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:48.229160+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:59.840762+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":859,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.662770+0000","last_change":"2024-09-13T13:18:03.351698+0000","last_active":"2024-09-13T13:18:47.662770+0000","last_peered":"2024-09-13T13:18:47.662770+0000","last_clean":"2024-09-13T13:18:47.662770+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:18:47.662770+0000","last_undegraded":"2024-09-13T13:18:47.662770+0000","last_fullsized":"2024-09-13T13:18:47.662770+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:03.351653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:03.351653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:11.465761+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":843,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.677830+0000","last_change":"2024-09-13T13:17:05.238854+0000","last_active":"2024-09-13T13:18:47.677830+0000","last_peered":"2024-09-13T13:18:47.677830+0000","last_clean":"2024-09-13T13:18:47.677830+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:18:47.677830+0000","last_undegraded":"2024-09-13T13:18:47.677830+0000","last_fullsized":"2024-09-13T13:18:47.677830+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:05.238722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.238722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.936311+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":86,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.720603+0000","last_change":"2024-09-13T13:17:25.243030+0000","last_active":"2024-09-13T13:18:51.720603+0000","last_peered":"2024-09-13T13:18:51.720603+0000","last_clean":"2024-09-13T13:18:51.720603+0000","last_became_active":"2024-09-13T13:17:25.242366+0000","last_became_peered":"2024-09-13T13:17:25.242366+0000","last_unstale":"2024-09-13T13:18:51.720603+0000","last_undegraded":"2024-09-13T13:18:51.720603+0000","last_fullsized":"2024-09-13T13:18:51.720603+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:35.064188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:35.064188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.698744+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"445'5129","reported_seq":5183,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.394641+0000","last_change":"2024-09-13T13:17:55.200332+0000","last_active":"2024-09-13T13:18:51.394641+0000","last_peered":"2024-09-13T13:18:51.394641+0000","last_clean":"2024-09-13T13:18:51.394641+0000","last_became_active":"2024-09-13T13:17:55.199999+0000","last_became_peered":"2024-09-13T13:17:55.199999+0000","last_unstale":"2024-09-13T13:18:51.394641+0000","last_undegraded":"2024-09-13T13:18:51.394641+0000","last_fullsized":"2024-09-13T13:18:51.394641+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5129,"log_dups_size":0,"ondisk_log_size":5129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:09.224261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336134144,"num_objects":5129,"num_object_clones":0,"num_object_copies":15387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5129,"num_write_kb":328256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":76,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.714595+0000","last_change":"2024-09-13T13:18:51.111528+0000","last_active":"2024-09-13T13:18:51.714595+0000","last_peered":"2024-09-13T13:18:51.714595+0000","last_clean":"2024-09-13T13:18:51.714595+0000","last_became_active":"2024-09-13T13:17:49.053160+0000","last_became_peered":"2024-09-13T13:17:49.053160+0000","last_unstale":"2024-09-13T13:18:51.714595+0000","last_undegraded":"2024-09-13T13:18:51.714595+0000","last_fullsized":"2024-09-13T13:18:51.714595+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.111449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.111449+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:00.573449+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":316,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067981+0000","last_change":"2024-09-13T13:17:02.421924+0000","last_active":"2024-09-13T13:18:53.067981+0000","last_peered":"2024-09-13T13:18:53.067981+0000","last_clean":"2024-09-13T13:18:53.067981+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:18:53.067981+0000","last_undegraded":"2024-09-13T13:18:53.067981+0000","last_fullsized":"2024-09-13T13:18:53.067981+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.421825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.421825+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":828,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.715307+0000","last_change":"2024-09-13T13:12:53.003530+0000","last_active":"2024-09-13T13:18:51.715307+0000","last_peered":"2024-09-13T13:18:51.715307+0000","last_clean":"2024-09-13T13:18:51.715307+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:18:51.715307+0000","last_undegraded":"2024-09-13T13:18:51.715307+0000","last_fullsized":"2024-09-13T13:18:51.715307+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.996346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.996346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:15.070878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"411'10","reported_seq":116,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.720706+0000","last_change":"2024-09-13T13:17:30.560167+0000","last_active":"2024-09-13T13:18:51.720706+0000","last_peered":"2024-09-13T13:18:51.720706+0000","last_clean":"2024-09-13T13:18:51.720706+0000","last_became_active":"2024-09-13T13:17:25.333196+0000","last_became_peered":"2024-09-13T13:17:25.333196+0000","last_unstale":"2024-09-13T13:18:51.720706+0000","last_undegraded":"2024-09-13T13:18:51.720706+0000","last_fullsized":"2024-09-13T13:18:51.720706+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'10","last_scrub_stamp":"2024-09-13T13:17:30.560099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:30.560099+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":326,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069049+0000","last_change":"2024-09-13T13:18:42.196223+0000","last_active":"2024-09-13T13:18:53.069049+0000","last_peered":"2024-09-13T13:18:53.069049+0000","last_clean":"2024-09-13T13:18:53.069049+0000","last_became_active":"2024-09-13T13:17:25.252583+0000","last_became_peered":"2024-09-13T13:17:25.252583+0000","last_unstale":"2024-09-13T13:18:53.069049+0000","last_undegraded":"2024-09-13T13:18:53.069049+0000","last_fullsized":"2024-09-13T13:18:53.069049+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:42.196120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:42.196120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:02.244520+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"445'4953","reported_seq":5008,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.395817+0000","last_change":"2024-09-13T13:17:55.200305+0000","last_active":"2024-09-13T13:18:51.395817+0000","last_peered":"2024-09-13T13:18:51.395817+0000","last_clean":"2024-09-13T13:18:51.395817+0000","last_became_active":"2024-09-13T13:17:55.200036+0000","last_became_peered":"2024-09-13T13:17:55.200036+0000","last_unstale":"2024-09-13T13:18:51.395817+0000","last_undegraded":"2024-09-13T13:18:51.395817+0000","last_fullsized":"2024-09-13T13:18:51.395817+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4953,"log_dups_size":0,"ondisk_log_size":4953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.345647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324599808,"num_objects":4953,"num_object_clones":0,"num_object_copies":14859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4953,"num_write_kb":316992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":858,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.672225+0000","last_change":"2024-09-13T13:13:56.850149+0000","last_active":"2024-09-13T13:18:47.672225+0000","last_peered":"2024-09-13T13:18:47.672225+0000","last_clean":"2024-09-13T13:18:47.672225+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:18:47.672225+0000","last_undegraded":"2024-09-13T13:18:47.672225+0000","last_fullsized":"2024-09-13T13:18:47.672225+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:56.850094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.850094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.496112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":851,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069291+0000","last_change":"2024-09-13T13:12:58.857634+0000","last_active":"2024-09-13T13:18:53.069291+0000","last_peered":"2024-09-13T13:18:53.069291+0000","last_clean":"2024-09-13T13:18:53.069291+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:18:53.069291+0000","last_undegraded":"2024-09-13T13:18:53.069291+0000","last_fullsized":"2024-09-13T13:18:53.069291+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:58.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:24.527045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":94,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.719808+0000","last_change":"2024-09-13T13:17:54.620568+0000","last_active":"2024-09-13T13:18:51.719808+0000","last_peered":"2024-09-13T13:18:51.719808+0000","last_clean":"2024-09-13T13:18:51.719808+0000","last_became_active":"2024-09-13T13:17:25.238627+0000","last_became_peered":"2024-09-13T13:17:25.238627+0000","last_unstale":"2024-09-13T13:18:51.719808+0000","last_undegraded":"2024-09-13T13:18:51.719808+0000","last_fullsized":"2024-09-13T13:18:51.719808+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.620524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.620524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:08.712954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":830,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.673040+0000","last_change":"2024-09-13T13:17:30.598783+0000","last_active":"2024-09-13T13:18:47.673040+0000","last_peered":"2024-09-13T13:18:47.673040+0000","last_clean":"2024-09-13T13:18:47.673040+0000","last_became_active":"2024-09-13T13:17:25.252822+0000","last_became_peered":"2024-09-13T13:17:25.252822+0000","last_unstale":"2024-09-13T13:18:47.673040+0000","last_undegraded":"2024-09-13T13:18:47.673040+0000","last_fullsized":"2024-09-13T13:18:47.673040+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:30.598736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:30.598736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"447'5258","reported_seq":5316,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.542192+0000","last_change":"2024-09-13T13:17:55.200461+0000","last_active":"2024-09-13T13:18:53.542192+0000","last_peered":"2024-09-13T13:18:53.542192+0000","last_clean":"2024-09-13T13:18:53.542192+0000","last_became_active":"2024-09-13T13:17:55.200186+0000","last_became_peered":"2024-09-13T13:17:55.200186+0000","last_unstale":"2024-09-13T13:18:53.542192+0000","last_undegraded":"2024-09-13T13:18:53.542192+0000","last_fullsized":"2024-09-13T13:18:53.542192+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5258,"log_dups_size":0,"ondisk_log_size":5258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.535572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344588288,"num_objects":5258,"num_object_clones":0,"num_object_copies":15774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5258,"num_write_kb":336512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":832,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.662367+0000","last_change":"2024-09-13T13:13:06.339762+0000","last_active":"2024-09-13T13:18:47.662367+0000","last_peered":"2024-09-13T13:18:47.662367+0000","last_clean":"2024-09-13T13:18:47.662367+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:18:47.662367+0000","last_undegraded":"2024-09-13T13:18:47.662367+0000","last_fullsized":"2024-09-13T13:18:47.662367+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:13:06.339664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.339664+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:26.319604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":466,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069313+0000","last_change":"2024-09-13T13:14:55.471954+0000","last_active":"2024-09-13T13:18:53.069313+0000","last_peered":"2024-09-13T13:18:53.069313+0000","last_clean":"2024-09-13T13:18:53.069313+0000","last_became_active":"2024-09-13T13:12:39.954315+0000","last_became_peered":"2024-09-13T13:12:39.954315+0000","last_unstale":"2024-09-13T13:18:53.069313+0000","last_undegraded":"2024-09-13T13:18:53.069313+0000","last_fullsized":"2024-09-13T13:18:53.069313+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:55.471885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:55.471885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":94,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.719830+0000","last_change":"2024-09-13T13:18:39.692048+0000","last_active":"2024-09-13T13:18:51.719830+0000","last_peered":"2024-09-13T13:18:51.719830+0000","last_clean":"2024-09-13T13:18:51.719830+0000","last_became_active":"2024-09-13T13:17:25.332964+0000","last_became_peered":"2024-09-13T13:17:25.332964+0000","last_unstale":"2024-09-13T13:18:51.719830+0000","last_undegraded":"2024-09-13T13:18:51.719830+0000","last_fullsized":"2024-09-13T13:18:51.719830+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:39.691978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:39.691978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:08.687543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":94,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.720094+0000","last_change":"2024-09-13T13:18:48.717531+0000","last_active":"2024-09-13T13:18:51.720094+0000","last_peered":"2024-09-13T13:18:51.720094+0000","last_clean":"2024-09-13T13:18:51.720094+0000","last_became_active":"2024-09-13T13:17:25.243114+0000","last_became_peered":"2024-09-13T13:17:25.243114+0000","last_unstale":"2024-09-13T13:18:51.720094+0000","last_undegraded":"2024-09-13T13:18:51.720094+0000","last_fullsized":"2024-09-13T13:18:51.720094+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:48.717456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:48.717456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:18.104517+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"445'4909","reported_seq":4964,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:49.691870+0000","last_change":"2024-09-13T13:17:55.140187+0000","last_active":"2024-09-13T13:18:49.691870+0000","last_peered":"2024-09-13T13:18:49.691870+0000","last_clean":"2024-09-13T13:18:49.691870+0000","last_became_active":"2024-09-13T13:17:55.139876+0000","last_became_peered":"2024-09-13T13:17:55.139876+0000","last_unstale":"2024-09-13T13:18:49.691870+0000","last_undegraded":"2024-09-13T13:18:49.691870+0000","last_fullsized":"2024-09-13T13:18:49.691870+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4909,"log_dups_size":0,"ondisk_log_size":4909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:02.705289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321716224,"num_objects":4909,"num_object_clones":0,"num_object_copies":14727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4909,"num_write_kb":314176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":324,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067223+0000","last_change":"2024-09-13T13:16:00.285064+0000","last_active":"2024-09-13T13:18:53.067223+0000","last_peered":"2024-09-13T13:18:53.067223+0000","last_clean":"2024-09-13T13:18:53.067223+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:18:53.067223+0000","last_undegraded":"2024-09-13T13:18:53.067223+0000","last_fullsized":"2024-09-13T13:18:53.067223+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:16:00.284984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:00.284984+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:52.129677+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":312,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068957+0000","last_change":"2024-09-13T13:14:52.383027+0000","last_active":"2024-09-13T13:18:53.068957+0000","last_peered":"2024-09-13T13:18:53.068957+0000","last_clean":"2024-09-13T13:18:53.068957+0000","last_became_active":"2024-09-13T13:12:53.007561+0000","last_became_peered":"2024-09-13T13:12:53.007561+0000","last_unstale":"2024-09-13T13:18:53.068957+0000","last_undegraded":"2024-09-13T13:18:53.068957+0000","last_fullsized":"2024-09-13T13:18:53.068957+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.382755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.382755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:15.521717+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":94,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.720020+0000","last_change":"2024-09-13T13:17:52.607482+0000","last_active":"2024-09-13T13:18:51.720020+0000","last_peered":"2024-09-13T13:18:51.720020+0000","last_clean":"2024-09-13T13:18:51.720020+0000","last_became_active":"2024-09-13T13:17:25.252535+0000","last_became_peered":"2024-09-13T13:17:25.252535+0000","last_unstale":"2024-09-13T13:18:51.720020+0000","last_undegraded":"2024-09-13T13:18:51.720020+0000","last_fullsized":"2024-09-13T13:18:51.720020+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:52.607404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.607404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:58.714721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":810,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.715113+0000","last_change":"2024-09-13T13:16:58.047398+0000","last_active":"2024-09-13T13:18:51.715113+0000","last_peered":"2024-09-13T13:18:51.715113+0000","last_clean":"2024-09-13T13:18:51.715113+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:18:51.715113+0000","last_undegraded":"2024-09-13T13:18:51.715113+0000","last_fullsized":"2024-09-13T13:18:51.715113+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:58.044139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.044139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"447'5081","reported_seq":5138,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.670115+0000","last_change":"2024-09-13T13:17:55.137957+0000","last_active":"2024-09-13T13:18:53.670115+0000","last_peered":"2024-09-13T13:18:53.670115+0000","last_clean":"2024-09-13T13:18:53.670115+0000","last_became_active":"2024-09-13T13:17:55.137667+0000","last_became_peered":"2024-09-13T13:17:55.137667+0000","last_unstale":"2024-09-13T13:18:53.670115+0000","last_undegraded":"2024-09-13T13:18:53.670115+0000","last_fullsized":"2024-09-13T13:18:53.670115+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5081,"log_dups_size":0,"ondisk_log_size":5081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:03.830885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332988416,"num_objects":5081,"num_object_clones":0,"num_object_copies":15243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5081,"num_write_kb":325184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":899,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068271+0000","last_change":"2024-09-13T13:15:56.631456+0000","last_active":"2024-09-13T13:18:53.068271+0000","last_peered":"2024-09-13T13:18:53.068271+0000","last_clean":"2024-09-13T13:18:53.068271+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:18:53.068271+0000","last_undegraded":"2024-09-13T13:18:53.068271+0000","last_fullsized":"2024-09-13T13:18:53.068271+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:15:56.631396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:56.631396+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:18.472573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":310,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069959+0000","last_change":"2024-09-13T13:13:01.972907+0000","last_active":"2024-09-13T13:18:53.069959+0000","last_peered":"2024-09-13T13:18:53.069959+0000","last_clean":"2024-09-13T13:18:53.069959+0000","last_became_active":"2024-09-13T13:12:53.006362+0000","last_became_peered":"2024-09-13T13:12:53.006362+0000","last_unstale":"2024-09-13T13:18:53.069959+0000","last_undegraded":"2024-09-13T13:18:53.069959+0000","last_fullsized":"2024-09-13T13:18:53.069959+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:13:01.972831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.972831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:17.528749+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"306'13","reported_seq":590,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.665454+0000","last_change":"2024-09-13T13:18:41.356034+0000","last_active":"2024-09-13T13:18:47.665454+0000","last_peered":"2024-09-13T13:18:47.665454+0000","last_clean":"2024-09-13T13:18:47.665454+0000","last_became_active":"2024-09-13T13:12:53.001189+0000","last_became_peered":"2024-09-13T13:12:53.001189+0000","last_unstale":"2024-09-13T13:18:47.665454+0000","last_undegraded":"2024-09-13T13:18:47.665454+0000","last_fullsized":"2024-09-13T13:18:47.665454+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'13","last_scrub_stamp":"2024-09-13T13:18:41.355851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.355851+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:59.453980+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":832,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069191+0000","last_change":"2024-09-13T13:18:43.850216+0000","last_active":"2024-09-13T13:18:53.069191+0000","last_peered":"2024-09-13T13:18:53.069191+0000","last_clean":"2024-09-13T13:18:53.069191+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:18:53.069191+0000","last_undegraded":"2024-09-13T13:18:53.069191+0000","last_fullsized":"2024-09-13T13:18:53.069191+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:43.850163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:43.850163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:11.330472+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":797,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.662840+0000","last_change":"2024-09-13T13:13:04.371082+0000","last_active":"2024-09-13T13:18:47.662840+0000","last_peered":"2024-09-13T13:18:47.662840+0000","last_clean":"2024-09-13T13:18:47.662840+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:18:47.662840+0000","last_undegraded":"2024-09-13T13:18:47.662840+0000","last_fullsized":"2024-09-13T13:18:47.662840+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.370881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.370881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:22.297512+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":354,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068907+0000","last_change":"2024-09-13T13:17:59.217373+0000","last_active":"2024-09-13T13:18:53.068907+0000","last_peered":"2024-09-13T13:18:53.068907+0000","last_clean":"2024-09-13T13:18:53.068907+0000","last_became_active":"2024-09-13T13:12:34.673528+0000","last_became_peered":"2024-09-13T13:12:34.673528+0000","last_unstale":"2024-09-13T13:18:53.068907+0000","last_undegraded":"2024-09-13T13:18:53.068907+0000","last_fullsized":"2024-09-13T13:18:53.068907+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:17:59.217309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.217309+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:28.238775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":856,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069584+0000","last_change":"2024-09-13T13:18:05.302935+0000","last_active":"2024-09-13T13:18:53.069584+0000","last_peered":"2024-09-13T13:18:53.069584+0000","last_clean":"2024-09-13T13:18:53.069584+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:18:53.069584+0000","last_undegraded":"2024-09-13T13:18:53.069584+0000","last_fullsized":"2024-09-13T13:18:53.069584+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:05.302882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:05.302882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.921840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":877,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.082313+0000","last_change":"2024-09-13T13:17:53.194243+0000","last_active":"2024-09-13T13:18:53.082313+0000","last_peered":"2024-09-13T13:18:53.082313+0000","last_clean":"2024-09-13T13:18:53.082313+0000","last_became_active":"2024-09-13T13:12:53.000873+0000","last_became_peered":"2024-09-13T13:12:53.000873+0000","last_unstale":"2024-09-13T13:18:53.082313+0000","last_undegraded":"2024-09-13T13:18:53.082313+0000","last_fullsized":"2024-09-13T13:18:53.082313+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:53.194191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:53.194191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:19.153631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":848,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.678142+0000","last_change":"2024-09-13T13:17:52.518556+0000","last_active":"2024-09-13T13:18:47.678142+0000","last_peered":"2024-09-13T13:18:47.678142+0000","last_clean":"2024-09-13T13:18:47.678142+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:18:47.678142+0000","last_undegraded":"2024-09-13T13:18:47.678142+0000","last_fullsized":"2024-09-13T13:18:47.678142+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:52.518502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.518502+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:12.668389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":304,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067012+0000","last_change":"2024-09-13T13:13:51.764793+0000","last_active":"2024-09-13T13:18:53.067012+0000","last_peered":"2024-09-13T13:18:53.067012+0000","last_clean":"2024-09-13T13:18:53.067012+0000","last_became_active":"2024-09-13T13:12:53.006579+0000","last_became_peered":"2024-09-13T13:12:53.006579+0000","last_unstale":"2024-09-13T13:18:53.067012+0000","last_undegraded":"2024-09-13T13:18:53.067012+0000","last_fullsized":"2024-09-13T13:18:53.067012+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.764717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.764717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:24.315631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":830,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.715189+0000","last_change":"2024-09-13T13:17:00.968640+0000","last_active":"2024-09-13T13:18:51.715189+0000","last_peered":"2024-09-13T13:18:51.715189+0000","last_clean":"2024-09-13T13:18:51.715189+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:18:51.715189+0000","last_undegraded":"2024-09-13T13:18:51.715189+0000","last_fullsized":"2024-09-13T13:18:51.715189+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:00.968580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:00.968580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":94,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.728868+0000","last_change":"2024-09-13T13:18:45.736249+0000","last_active":"2024-09-13T13:18:51.728868+0000","last_peered":"2024-09-13T13:18:51.728868+0000","last_clean":"2024-09-13T13:18:51.728868+0000","last_became_active":"2024-09-13T13:17:25.244421+0000","last_became_peered":"2024-09-13T13:17:25.244421+0000","last_unstale":"2024-09-13T13:18:51.728868+0000","last_undegraded":"2024-09-13T13:18:51.728868+0000","last_fullsized":"2024-09-13T13:18:51.728868+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:45.736080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.736080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:56.355923+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":874,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069391+0000","last_change":"2024-09-13T13:17:04.222896+0000","last_active":"2024-09-13T13:18:53.069391+0000","last_peered":"2024-09-13T13:18:53.069391+0000","last_clean":"2024-09-13T13:18:53.069391+0000","last_became_active":"2024-09-13T13:16:12.069730+0000","last_became_peered":"2024-09-13T13:16:12.069730+0000","last_unstale":"2024-09-13T13:18:53.069391+0000","last_undegraded":"2024-09-13T13:18:53.069391+0000","last_fullsized":"2024-09-13T13:18:53.069391+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:04.222760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:04.222760+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":819,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.680329+0000","last_change":"2024-09-13T13:17:09.634101+0000","last_active":"2024-09-13T13:18:47.680329+0000","last_peered":"2024-09-13T13:18:47.680329+0000","last_clean":"2024-09-13T13:18:47.680329+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:18:47.680329+0000","last_undegraded":"2024-09-13T13:18:47.680329+0000","last_fullsized":"2024-09-13T13:18:47.680329+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.634027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.634027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:30.479119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"411'10","reported_seq":1059,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.714622+0000","last_change":"2024-09-13T13:17:27.165835+0000","last_active":"2024-09-13T13:18:51.714622+0000","last_peered":"2024-09-13T13:18:51.714622+0000","last_clean":"2024-09-13T13:18:51.714622+0000","last_became_active":"2024-09-13T13:17:25.245468+0000","last_became_peered":"2024-09-13T13:17:25.245468+0000","last_unstale":"2024-09-13T13:18:51.714622+0000","last_undegraded":"2024-09-13T13:18:51.714622+0000","last_fullsized":"2024-09-13T13:18:51.714622+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'9","last_scrub_stamp":"2024-09-13T13:17:27.165657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:27.165657+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":326,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070021+0000","last_change":"2024-09-13T13:18:41.148239+0000","last_active":"2024-09-13T13:18:53.070021+0000","last_peered":"2024-09-13T13:18:53.070021+0000","last_clean":"2024-09-13T13:18:53.070021+0000","last_became_active":"2024-09-13T13:17:25.333587+0000","last_became_peered":"2024-09-13T13:17:25.333587+0000","last_unstale":"2024-09-13T13:18:53.070021+0000","last_undegraded":"2024-09-13T13:18:53.070021+0000","last_fullsized":"2024-09-13T13:18:53.070021+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:41.147987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.147987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:48.324808+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":872,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.715043+0000","last_change":"2024-09-13T13:17:02.978088+0000","last_active":"2024-09-13T13:18:51.715043+0000","last_peered":"2024-09-13T13:18:51.715043+0000","last_clean":"2024-09-13T13:18:51.715043+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:18:51.715043+0000","last_undegraded":"2024-09-13T13:18:51.715043+0000","last_fullsized":"2024-09-13T13:18:51.715043+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.978003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.978003+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":852,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.082334+0000","last_change":"2024-09-13T13:17:28.963214+0000","last_active":"2024-09-13T13:18:53.082334+0000","last_peered":"2024-09-13T13:18:53.082334+0000","last_clean":"2024-09-13T13:18:53.082334+0000","last_became_active":"2024-09-13T13:17:25.252500+0000","last_became_peered":"2024-09-13T13:17:25.252500+0000","last_unstale":"2024-09-13T13:18:53.082334+0000","last_undegraded":"2024-09-13T13:18:53.082334+0000","last_fullsized":"2024-09-13T13:18:53.082334+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:28.963152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:28.963152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":1063,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069092+0000","last_change":"2024-09-13T13:17:57.251817+0000","last_active":"2024-09-13T13:18:53.069092+0000","last_peered":"2024-09-13T13:18:53.069092+0000","last_clean":"2024-09-13T13:18:53.069092+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:18:53.069092+0000","last_undegraded":"2024-09-13T13:18:53.069092+0000","last_fullsized":"2024-09-13T13:18:53.069092+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:17:57.251734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:57.251734+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:03.487958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":831,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.674399+0000","last_change":"2024-09-13T13:17:33.554954+0000","last_active":"2024-09-13T13:18:47.674399+0000","last_peered":"2024-09-13T13:18:47.674399+0000","last_clean":"2024-09-13T13:18:47.674399+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:18:47.674399+0000","last_undegraded":"2024-09-13T13:18:47.674399+0000","last_fullsized":"2024-09-13T13:18:47.674399+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:33.554870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:33.554870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:03.321466+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":324,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068753+0000","last_change":"2024-09-13T13:18:45.145089+0000","last_active":"2024-09-13T13:18:53.068753+0000","last_peered":"2024-09-13T13:18:53.068753+0000","last_clean":"2024-09-13T13:18:53.068753+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:18:53.068753+0000","last_undegraded":"2024-09-13T13:18:53.068753+0000","last_fullsized":"2024-09-13T13:18:53.068753+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:45.145010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.145010+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:04.145291+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":304,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068059+0000","last_change":"2024-09-13T13:17:09.431539+0000","last_active":"2024-09-13T13:18:53.068059+0000","last_peered":"2024-09-13T13:18:53.068059+0000","last_clean":"2024-09-13T13:18:53.068059+0000","last_became_active":"2024-09-13T13:12:53.005988+0000","last_became_peered":"2024-09-13T13:12:53.005988+0000","last_unstale":"2024-09-13T13:18:53.068059+0000","last_undegraded":"2024-09-13T13:18:53.068059+0000","last_fullsized":"2024-09-13T13:18:53.068059+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.431472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.431472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":837,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.676173+0000","last_change":"2024-09-13T13:17:33.279005+0000","last_active":"2024-09-13T13:18:47.676173+0000","last_peered":"2024-09-13T13:18:47.676173+0000","last_clean":"2024-09-13T13:18:47.676173+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:18:47.676173+0000","last_undegraded":"2024-09-13T13:18:47.676173+0000","last_fullsized":"2024-09-13T13:18:47.676173+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:33.278909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:33.278909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:55.488227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":834,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.714447+0000","last_change":"2024-09-13T13:17:09.989898+0000","last_active":"2024-09-13T13:18:51.714447+0000","last_peered":"2024-09-13T13:18:51.714447+0000","last_clean":"2024-09-13T13:18:51.714447+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:18:51.714447+0000","last_undegraded":"2024-09-13T13:18:51.714447+0000","last_fullsized":"2024-09-13T13:18:51.714447+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.989820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.989820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":831,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069090+0000","last_change":"2024-09-13T13:17:26.125163+0000","last_active":"2024-09-13T13:18:53.069090+0000","last_peered":"2024-09-13T13:18:53.069090+0000","last_clean":"2024-09-13T13:18:53.069090+0000","last_became_active":"2024-09-13T13:17:25.331530+0000","last_became_peered":"2024-09-13T13:17:25.331530+0000","last_unstale":"2024-09-13T13:18:53.069090+0000","last_undegraded":"2024-09-13T13:18:53.069090+0000","last_fullsized":"2024-09-13T13:18:53.069090+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:26.125088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:26.125088+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":822,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.714911+0000","last_change":"2024-09-13T13:17:56.965925+0000","last_active":"2024-09-13T13:18:51.714911+0000","last_peered":"2024-09-13T13:18:51.714911+0000","last_clean":"2024-09-13T13:18:51.714911+0000","last_became_active":"2024-09-13T13:12:52.949094+0000","last_became_peered":"2024-09-13T13:12:52.949094+0000","last_unstale":"2024-09-13T13:18:51.714911+0000","last_undegraded":"2024-09-13T13:18:51.714911+0000","last_fullsized":"2024-09-13T13:18:51.714911+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:56.965870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:56.965870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:02.221904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":830,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.715209+0000","last_change":"2024-09-13T13:18:03.037546+0000","last_active":"2024-09-13T13:18:51.715209+0000","last_peered":"2024-09-13T13:18:51.715209+0000","last_clean":"2024-09-13T13:18:51.715209+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:18:51.715209+0000","last_undegraded":"2024-09-13T13:18:51.715209+0000","last_fullsized":"2024-09-13T13:18:51.715209+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:03.037472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:03.037472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.016351+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":815,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.671769+0000","last_change":"2024-09-13T13:18:44.505493+0000","last_active":"2024-09-13T13:18:47.671769+0000","last_peered":"2024-09-13T13:18:47.671769+0000","last_clean":"2024-09-13T13:18:47.671769+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:18:47.671769+0000","last_undegraded":"2024-09-13T13:18:47.671769+0000","last_fullsized":"2024-09-13T13:18:47.671769+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:44.505427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:44.505427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:54.043312+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":860,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.714691+0000","last_change":"2024-09-13T13:18:33.848957+0000","last_active":"2024-09-13T13:18:51.714691+0000","last_peered":"2024-09-13T13:18:51.714691+0000","last_clean":"2024-09-13T13:18:51.714691+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:18:51.714691+0000","last_undegraded":"2024-09-13T13:18:51.714691+0000","last_fullsized":"2024-09-13T13:18:51.714691+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:33.848726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.848726+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:59.409198+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":840,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069657+0000","last_change":"2024-09-13T13:14:50.392659+0000","last_active":"2024-09-13T13:18:53.069657+0000","last_peered":"2024-09-13T13:18:53.069657+0000","last_clean":"2024-09-13T13:18:53.069657+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:18:53.069657+0000","last_undegraded":"2024-09-13T13:18:53.069657+0000","last_fullsized":"2024-09-13T13:18:53.069657+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.392444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.392444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":849,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.081631+0000","last_change":"2024-09-13T13:15:03.778214+0000","last_active":"2024-09-13T13:18:53.081631+0000","last_peered":"2024-09-13T13:18:53.081631+0000","last_clean":"2024-09-13T13:18:53.081631+0000","last_became_active":"2024-09-13T13:12:52.999748+0000","last_became_peered":"2024-09-13T13:12:52.999748+0000","last_unstale":"2024-09-13T13:18:53.081631+0000","last_undegraded":"2024-09-13T13:18:53.081631+0000","last_fullsized":"2024-09-13T13:18:53.081631+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.778170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.778170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:24.797615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":805,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.715329+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:18:51.715329+0000","last_peered":"2024-09-13T13:18:51.715329+0000","last_clean":"2024-09-13T13:18:51.715329+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:18:51.715329+0000","last_undegraded":"2024-09-13T13:18:51.715329+0000","last_fullsized":"2024-09-13T13:18:51.715329+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:23.884576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":875,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.716433+0000","last_change":"2024-09-13T13:18:46.033316+0000","last_active":"2024-09-13T13:18:51.716433+0000","last_peered":"2024-09-13T13:18:51.716433+0000","last_clean":"2024-09-13T13:18:51.716433+0000","last_became_active":"2024-09-13T13:16:12.073702+0000","last_became_peered":"2024-09-13T13:16:12.073702+0000","last_unstale":"2024-09-13T13:18:51.716433+0000","last_undegraded":"2024-09-13T13:18:51.716433+0000","last_fullsized":"2024-09-13T13:18:51.716433+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:18:46.033238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.033238+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:01.052128+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":851,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.675792+0000","last_change":"2024-09-13T13:17:57.579827+0000","last_active":"2024-09-13T13:18:47.675792+0000","last_peered":"2024-09-13T13:18:47.675792+0000","last_clean":"2024-09-13T13:18:47.675792+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:18:47.675792+0000","last_undegraded":"2024-09-13T13:18:47.675792+0000","last_fullsized":"2024-09-13T13:18:47.675792+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:57.579778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:57.579778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:11.821230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":759,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.665487+0000","last_change":"2024-09-13T13:17:55.331435+0000","last_active":"2024-09-13T13:18:47.665487+0000","last_peered":"2024-09-13T13:18:47.665487+0000","last_clean":"2024-09-13T13:18:47.665487+0000","last_became_active":"2024-09-13T13:12:53.001131+0000","last_became_peered":"2024-09-13T13:12:53.001131+0000","last_unstale":"2024-09-13T13:18:47.665487+0000","last_undegraded":"2024-09-13T13:18:47.665487+0000","last_fullsized":"2024-09-13T13:18:47.665487+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:55.331393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:55.331393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:14.150805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":94,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.719720+0000","last_change":"2024-09-13T13:18:07.682986+0000","last_active":"2024-09-13T13:18:51.719720+0000","last_peered":"2024-09-13T13:18:51.719720+0000","last_clean":"2024-09-13T13:18:51.719720+0000","last_became_active":"2024-09-13T13:17:25.242337+0000","last_became_peered":"2024-09-13T13:17:25.242337+0000","last_unstale":"2024-09-13T13:18:51.719720+0000","last_undegraded":"2024-09-13T13:18:51.719720+0000","last_fullsized":"2024-09-13T13:18:51.719720+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:07.682648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:07.682648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:11.815648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":839,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.677000+0000","last_change":"2024-09-13T13:17:52.052009+0000","last_active":"2024-09-13T13:18:47.677000+0000","last_peered":"2024-09-13T13:18:47.677000+0000","last_clean":"2024-09-13T13:18:47.677000+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:18:47.677000+0000","last_undegraded":"2024-09-13T13:18:47.677000+0000","last_fullsized":"2024-09-13T13:18:47.677000+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:52.051961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.051961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:18.556468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":822,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-13T13:18:51.716659+0000","last_change":"2024-09-13T13:16:36.980668+0000","last_active":"2024-09-13T13:18:51.716659+0000","last_peered":"2024-09-13T13:18:51.716659+0000","last_clean":"2024-09-13T13:18:51.716659+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:18:51.716659+0000","last_undegraded":"2024-09-13T13:18:51.716659+0000","last_fullsized":"2024-09-13T13:18:51.716659+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:36.980594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:36.980594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.050858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":839,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.662881+0000","last_change":"2024-09-13T13:16:12.087075+0000","last_active":"2024-09-13T13:18:47.662881+0000","last_peered":"2024-09-13T13:18:47.662881+0000","last_clean":"2024-09-13T13:18:47.662881+0000","last_became_active":"2024-09-13T13:16:12.070277+0000","last_became_peered":"2024-09-13T13:16:12.070277+0000","last_unstale":"2024-09-13T13:18:47.662881+0000","last_undegraded":"2024-09-13T13:18:47.662881+0000","last_fullsized":"2024-09-13T13:18:47.662881+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:49.264128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.264128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:15.253268+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":858,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-13T13:18:47.675754+0000","last_change":"2024-09-13T13:17:25.253423+0000","last_active":"2024-09-13T13:18:47.675754+0000","last_peered":"2024-09-13T13:18:47.675754+0000","last_clean":"2024-09-13T13:18:47.675754+0000","last_became_active":"2024-09-13T13:17:25.252637+0000","last_became_peered":"2024-09-13T13:17:25.252637+0000","last_unstale":"2024-09-13T13:18:47.675754+0000","last_undegraded":"2024-09-13T13:18:47.675754+0000","last_fullsized":"2024-09-13T13:18:47.675754+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:23.547758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:23.547758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:29.521458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5322375168,"num_objects":81213,"num_object_clones":0,"num_object_copies":243639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81213,"num_write_kb":5197632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15977349120,"data_stored":15977349120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77713,"ondisk_log_size":77713,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":4792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":686,"ondisk_log_size":686,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":375,"num_read_kb":370,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":477367,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":79,"num_write":42,"num_write_kb":592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":542359,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201497,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2602124,"kb_used_data":1997564,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91143028,"statfs":{"total":95995035648,"available":93330460672,"internally_reserved":0,"allocated":2045505536,"data_stored":2044978337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":619051466},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141107,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3193008,"kb_used_data":2583584,"kb_used_omap":4,"kb_used_meta":609403,"kb_avail":90552144,"statfs":{"total":95995035648,"available":92725395456,"internally_reserved":0,"allocated":2645590016,"data_stored":2645082975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":624029357},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692069,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1965900,"kb_used_data":1321080,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":91779252,"statfs":{"total":95995035648,"available":93981954048,"internally_reserved":0,"allocated":1352785920,"data_stored":1352262201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921003,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1934760,"kb_used_data":1307988,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":91810392,"statfs":{"total":95995035648,"available":94013841408,"internally_reserved":0,"allocated":1339379712,"data_stored":1338833848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":641792454},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117234,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3036132,"kb_used_data":2288592,"kb_used_omap":1,"kb_used_meta":747518,"kb_avail":90709020,"statfs":{"total":95995035648,"available":92886036480,"internally_reserved":0,"allocated":2343518208,"data_stored":2342970879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":765458887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411573,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2519808,"kb_used_data":1961068,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":91225344,"statfs":{"total":95995035648,"available":93414752256,"internally_reserved":0,"allocated":2008133632,"data_stored":2007621651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":572127686},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575099,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3145804,"kb_used_data":2510716,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":90599348,"statfs":{"total":95995035648,"available":92773732352,"internally_reserved":0,"allocated":2570973184,"data_stored":2570438119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738626,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2354196,"kb_used_data":1564996,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":91390956,"statfs":{"total":95995035648,"available":93584338944,"internally_reserved":0,"allocated":1602555904,"data_stored":1602013695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":808122821},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1608253440,"data_stored":1608253440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2578186240,"data_stored":2578186240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2013986816,"data_stored":2013986816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2367750144,"data_stored":2367750144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1342242816,"data_stored":1342242816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1364787200,"data_stored":1364787200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2645426176,"data_stored":2645426176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2056716288,"data_stored":2056716288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:18:56.463 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T13:18:56.463 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:18:56.464 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-09-13T13:18:56.464 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:18:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:56.484 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:56.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:56.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:56.614 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:56.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:56.715 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:56.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:56.756 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:56.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:56.768 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:18:56.768 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:18:56.857 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:18:56.886 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 56 15 86609 86594 96.6197 69.6875 0.014735 0.010345 2024-09-13T13:18:56.890 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:56.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:18:56.992 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:57.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:57.036 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:57.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:57.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:56 smithi167 ceph-mon[33178]: from='client.16857 v1:172.21.15.52:0/1963081795' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:18:57.102 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:56 smithi052 ceph-mon[36682]: from='client.16857 v1:172.21.15.52:0/1963081795' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:18:57.102 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:56 smithi052 ceph-mon[29471]: from='client.16857 v1:172.21.15.52:0/1963081795' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:18:57.137 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:57.214 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:57.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:57.307 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:57.316 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:18:57.372 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:57.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:57.474 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:57.522 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:57.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:57.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T13:18:57.623 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:57.672 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:57.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:57.774 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:57.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:57 smithi167 ceph-mon[33178]: pgmap v900: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 959 op/s 2024-09-13T13:18:57.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:57.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:57.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:57.886 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 57 16 87721 87705 96.1426 69.4375 0.0101615 0.0103958 2024-09-13T13:18:57.945 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:57.983 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:57.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:57.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:57.988 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:57 smithi052 ceph-mon[36682]: pgmap v900: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 959 op/s 2024-09-13T13:18:57.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:57 smithi052 ceph-mon[29471]: pgmap v900: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 959 op/s 2024-09-13T13:18:58.000 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:18:58.085 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:58.125 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:58.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:58.228 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:58.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:58.282 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:58.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:58.383 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:58.418 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:58.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:58.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:58.520 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:58.564 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:58.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:18:58.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:18:58.667 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:18:58.667 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:18:58.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:58.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:18:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:18:58.734 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:18:58.805 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:58.838 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:58.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:58.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:58.886 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 58 16 88716 88700 95.5572 62.1875 0.0183271 0.0104598 2024-09-13T13:18:58.940 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:58.980 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:58.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:59.082 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:18:59.107 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:18:59.128 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:59.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:18:59.160 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:18:59.230 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:59.258 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:59.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:59.360 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:18:59.387 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:59.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:18:59.467 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:59 smithi167 ceph-mon[33178]: pgmap v901: 158 pgs: 158 active+clean; 5.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.01k op/s 2024-09-13T13:18:59.467 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:59 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4070476665' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:59.468 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:59 smithi167 ceph-mon[33178]: from='client.36740 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:59.468 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:18:59 smithi167 ceph-mon[33178]: from='client.26638 v1:172.21.15.52:0/2323094777' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:18:59.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:59 smithi052 ceph-mon[36682]: pgmap v901: 158 pgs: 158 active+clean; 5.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.01k op/s 2024-09-13T13:18:59.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:59 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4070476665' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:59.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:59 smithi052 ceph-mon[36682]: from='client.36740 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:59.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:18:59 smithi052 ceph-mon[36682]: from='client.26638 v1:172.21.15.52:0/2323094777' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:18:59.490 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:18:59.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:59 smithi052 ceph-mon[29471]: pgmap v901: 158 pgs: 158 active+clean; 5.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.01k op/s 2024-09-13T13:18:59.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:59 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4070476665' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:59.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:59 smithi052 ceph-mon[29471]: from='client.36740 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:18:59.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:18:59 smithi052 ceph-mon[29471]: from='client.26638 v1:172.21.15.52:0/2323094777' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:18:59.513 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T13:18:59.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:18:59.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:59.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:18:59.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:18:59.675 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:59.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:18:59.777 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:18:59.822 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:59.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:59.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:18:59.887 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 59 15 89824 89809 95.1115 69.3125 0.0310325 0.0105087 2024-09-13T13:18:59.924 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:18:59.979 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:18:59.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:18:59.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:00.076 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":901,"stamp":"2024-09-13T13:18:58.006560+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5606861761,"num_objects":85747,"num_object_clones":0,"num_object_copies":257241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85747,"num_whiteouts":0,"num_read":1518,"num_read_kb":1280,"num_write":86281,"num_write_kb":5475329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":388109,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81303,"ondisk_log_size":81303,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21508632,"kb_used_data":16282756,"kb_used_omap":15,"kb_used_meta":5225712,"kb_avail":728452584,"statfs":{"total":767960285184,"available":745935446016,"internally_reserved":0,"allocated":16673542144,"data_stored":16669258763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15591,"internal_metadata":5351129881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,6,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":537239852,"num_objects":8203,"num_object_clones":0,"num_object_copies":24609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8203,"num_whiteouts":0,"num_read":-87,"num_read_kb":-174,"num_write":8112,"num_write_kb":523788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-483028,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002854"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":846,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.073358+0000","last_change":"2024-09-13T13:18:32.440533+0000","last_active":"2024-09-13T13:18:53.073358+0000","last_peered":"2024-09-13T13:18:53.073358+0000","last_clean":"2024-09-13T13:18:53.073358+0000","last_became_active":"2024-09-13T13:17:25.335900+0000","last_became_peered":"2024-09-13T13:17:25.335900+0000","last_unstale":"2024-09-13T13:18:53.073358+0000","last_undegraded":"2024-09-13T13:18:53.073358+0000","last_fullsized":"2024-09-13T13:18:53.073358+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:32.440448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:32.440448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:57.101135+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"267'15","reported_seq":322,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.065687+0000","last_change":"2024-09-13T13:18:08.548626+0000","last_active":"2024-09-13T13:18:53.065687+0000","last_peered":"2024-09-13T13:18:53.065687+0000","last_clean":"2024-09-13T13:18:53.065687+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:18:53.065687+0000","last_undegraded":"2024-09-13T13:18:53.065687+0000","last_fullsized":"2024-09-13T13:18:53.065687+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:18:08.548556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:08.548556+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:22.274215+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":833,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.078359+0000","last_change":"2024-09-13T13:17:56.109579+0000","last_active":"2024-09-13T13:18:53.078359+0000","last_peered":"2024-09-13T13:18:53.078359+0000","last_clean":"2024-09-13T13:18:53.078359+0000","last_became_active":"2024-09-13T13:01:06.207532+0000","last_became_peered":"2024-09-13T13:01:06.207532+0000","last_unstale":"2024-09-13T13:18:53.078359+0000","last_undegraded":"2024-09-13T13:18:53.078359+0000","last_fullsized":"2024-09-13T13:18:53.078359+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:56.109526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:56.109526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:00.485890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":320,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067366+0000","last_change":"2024-09-13T13:18:49.194573+0000","last_active":"2024-09-13T13:18:53.067366+0000","last_peered":"2024-09-13T13:18:53.067366+0000","last_clean":"2024-09-13T13:18:53.067366+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:18:53.067366+0000","last_undegraded":"2024-09-13T13:18:53.067366+0000","last_fullsized":"2024-09-13T13:18:53.067366+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:49.194510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:49.194510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:02.508037+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":823,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085709+0000","last_change":"2024-09-13T13:11:47.852762+0000","last_active":"2024-09-13T13:18:53.085709+0000","last_peered":"2024-09-13T13:18:53.085709+0000","last_clean":"2024-09-13T13:18:53.085709+0000","last_became_active":"2024-09-13T13:04:02.760281+0000","last_became_peered":"2024-09-13T13:04:02.760281+0000","last_unstale":"2024-09-13T13:18:53.085709+0000","last_undegraded":"2024-09-13T13:18:53.085709+0000","last_fullsized":"2024-09-13T13:18:53.085709+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:26.862207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":866,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085384+0000","last_change":"2024-09-13T13:17:25.243843+0000","last_active":"2024-09-13T13:18:53.085384+0000","last_peered":"2024-09-13T13:18:53.085384+0000","last_clean":"2024-09-13T13:18:53.085384+0000","last_became_active":"2024-09-13T13:17:25.243371+0000","last_became_peered":"2024-09-13T13:17:25.243371+0000","last_unstale":"2024-09-13T13:18:53.085384+0000","last_undegraded":"2024-09-13T13:18:53.085384+0000","last_fullsized":"2024-09-13T13:18:53.085384+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:08.938750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:08.938750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"269'27","reported_seq":941,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.086747+0000","last_change":"2024-09-13T13:17:25.262965+0000","last_active":"2024-09-13T13:18:53.086747+0000","last_peered":"2024-09-13T13:18:53.086747+0000","last_clean":"2024-09-13T13:18:53.086747+0000","last_became_active":"2024-09-13T13:17:25.245131+0000","last_became_peered":"2024-09-13T13:17:25.245131+0000","last_unstale":"2024-09-13T13:18:53.086747+0000","last_undegraded":"2024-09-13T13:18:53.086747+0000","last_fullsized":"2024-09-13T13:18:53.086747+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:17:11.044472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:11.044472+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":296,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.065617+0000","last_change":"2024-09-13T13:12:52.997865+0000","last_active":"2024-09-13T13:18:53.065617+0000","last_peered":"2024-09-13T13:18:53.065617+0000","last_clean":"2024-09-13T13:18:53.065617+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:18:53.065617+0000","last_undegraded":"2024-09-13T13:18:53.065617+0000","last_fullsized":"2024-09-13T13:18:53.065617+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:17.305608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:12:17.305608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.504193+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":887,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069482+0000","last_change":"2024-09-13T13:13:41.518378+0000","last_active":"2024-09-13T13:18:53.069482+0000","last_peered":"2024-09-13T13:18:53.069482+0000","last_clean":"2024-09-13T13:18:53.069482+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:18:53.069482+0000","last_undegraded":"2024-09-13T13:18:53.069482+0000","last_fullsized":"2024-09-13T13:18:53.069482+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:13:41.518322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:41.518322+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:52.806617+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"268'15","reported_seq":314,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067453+0000","last_change":"2024-09-13T13:17:06.406253+0000","last_active":"2024-09-13T13:18:53.067453+0000","last_peered":"2024-09-13T13:18:53.067453+0000","last_clean":"2024-09-13T13:18:53.067453+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:18:53.067453+0000","last_undegraded":"2024-09-13T13:18:53.067453+0000","last_fullsized":"2024-09-13T13:18:53.067453+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:06.406179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:06.406179+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":832,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069781+0000","last_change":"2024-09-13T13:17:31.089557+0000","last_active":"2024-09-13T13:18:53.069781+0000","last_peered":"2024-09-13T13:18:53.069781+0000","last_clean":"2024-09-13T13:18:53.069781+0000","last_became_active":"2024-09-13T13:01:06.205507+0000","last_became_peered":"2024-09-13T13:01:06.205507+0000","last_unstale":"2024-09-13T13:18:53.069781+0000","last_undegraded":"2024-09-13T13:18:53.069781+0000","last_fullsized":"2024-09-13T13:18:53.069781+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:31.089505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:31.089505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":835,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069648+0000","last_change":"2024-09-13T13:17:32.559966+0000","last_active":"2024-09-13T13:18:53.069648+0000","last_peered":"2024-09-13T13:18:53.069648+0000","last_clean":"2024-09-13T13:18:53.069648+0000","last_became_active":"2024-09-13T13:01:08.209751+0000","last_became_peered":"2024-09-13T13:01:08.209751+0000","last_unstale":"2024-09-13T13:18:53.069648+0000","last_undegraded":"2024-09-13T13:18:53.069648+0000","last_fullsized":"2024-09-13T13:18:53.069648+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:32.559884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:32.559884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":862,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069163+0000","last_change":"2024-09-13T13:17:02.211078+0000","last_active":"2024-09-13T13:18:53.069163+0000","last_peered":"2024-09-13T13:18:53.069163+0000","last_clean":"2024-09-13T13:18:53.069163+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:18:53.069163+0000","last_undegraded":"2024-09-13T13:18:53.069163+0000","last_fullsized":"2024-09-13T13:18:53.069163+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:02.211007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.211007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:26.585228+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"268'7","reported_seq":310,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068501+0000","last_change":"2024-09-13T13:15:03.348187+0000","last_active":"2024-09-13T13:18:53.068501+0000","last_peered":"2024-09-13T13:18:53.068501+0000","last_clean":"2024-09-13T13:18:53.068501+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:18:53.068501+0000","last_undegraded":"2024-09-13T13:18:53.068501+0000","last_fullsized":"2024-09-13T13:18:53.068501+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:15:03.348115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.348115+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:10.491929+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":845,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.081938+0000","last_change":"2024-09-13T13:13:54.122282+0000","last_active":"2024-09-13T13:18:53.081938+0000","last_peered":"2024-09-13T13:18:53.081938+0000","last_clean":"2024-09-13T13:18:53.081938+0000","last_became_active":"2024-09-13T13:12:52.999970+0000","last_became_peered":"2024-09-13T13:12:52.999970+0000","last_unstale":"2024-09-13T13:18:53.081938+0000","last_undegraded":"2024-09-13T13:18:53.081938+0000","last_fullsized":"2024-09-13T13:18:53.081938+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:54.122222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:13:54.122222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:19.834956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":812,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068496+0000","last_change":"2024-09-13T13:18:34.846273+0000","last_active":"2024-09-13T13:18:53.068496+0000","last_peered":"2024-09-13T13:18:53.068496+0000","last_clean":"2024-09-13T13:18:53.068496+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:18:53.068496+0000","last_undegraded":"2024-09-13T13:18:53.068496+0000","last_fullsized":"2024-09-13T13:18:53.068496+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:34.846222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:34.846222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:51.829884+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":309,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069398+0000","last_change":"2024-09-13T13:17:07.444148+0000","last_active":"2024-09-13T13:18:53.069398+0000","last_peered":"2024-09-13T13:18:53.069398+0000","last_clean":"2024-09-13T13:18:53.069398+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:18:53.069398+0000","last_undegraded":"2024-09-13T13:18:53.069398+0000","last_fullsized":"2024-09-13T13:18:53.069398+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:07.443951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:07.443951+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":861,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.071264+0000","last_change":"2024-09-13T13:18:45.471190+0000","last_active":"2024-09-13T13:18:53.071264+0000","last_peered":"2024-09-13T13:18:53.071264+0000","last_clean":"2024-09-13T13:18:53.071264+0000","last_became_active":"2024-09-13T13:12:53.008505+0000","last_became_peered":"2024-09-13T13:12:53.008505+0000","last_unstale":"2024-09-13T13:18:53.071264+0000","last_undegraded":"2024-09-13T13:18:53.071264+0000","last_fullsized":"2024-09-13T13:18:53.071264+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:45.470790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.470790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:15.409504+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"268'27","reported_seq":112,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.074258+0000","last_change":"2024-09-13T13:18:37.714594+0000","last_active":"2024-09-13T13:18:53.074258+0000","last_peered":"2024-09-13T13:18:53.074258+0000","last_clean":"2024-09-13T13:18:53.074258+0000","last_became_active":"2024-09-13T13:17:25.245658+0000","last_became_peered":"2024-09-13T13:17:25.245658+0000","last_unstale":"2024-09-13T13:18:53.074258+0000","last_undegraded":"2024-09-13T13:18:53.074258+0000","last_fullsized":"2024-09-13T13:18:53.074258+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:18:37.714503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:37.714503+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:41.615700+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":855,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.078439+0000","last_change":"2024-09-13T13:16:54.229157+0000","last_active":"2024-09-13T13:18:53.078439+0000","last_peered":"2024-09-13T13:18:53.078439+0000","last_clean":"2024-09-13T13:18:53.078439+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:18:53.078439+0000","last_undegraded":"2024-09-13T13:18:53.078439+0000","last_fullsized":"2024-09-13T13:18:53.078439+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:54.229075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:16:54.229075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:17.035251+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":832,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085314+0000","last_change":"2024-09-13T13:18:41.949104+0000","last_active":"2024-09-13T13:18:53.085314+0000","last_peered":"2024-09-13T13:18:53.085314+0000","last_clean":"2024-09-13T13:18:53.085314+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:18:53.085314+0000","last_undegraded":"2024-09-13T13:18:53.085314+0000","last_fullsized":"2024-09-13T13:18:53.085314+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:41.949029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.949029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:49.617995+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":334,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067224+0000","last_change":"2024-09-13T13:18:46.181826+0000","last_active":"2024-09-13T13:18:53.067224+0000","last_peered":"2024-09-13T13:18:53.067224+0000","last_clean":"2024-09-13T13:18:53.067224+0000","last_became_active":"2024-09-13T13:17:25.244839+0000","last_became_peered":"2024-09-13T13:17:25.244839+0000","last_unstale":"2024-09-13T13:18:53.067224+0000","last_undegraded":"2024-09-13T13:18:53.067224+0000","last_fullsized":"2024-09-13T13:18:53.067224+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:46.181741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.181741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:01.669320+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"269'18","reported_seq":917,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085237+0000","last_change":"2024-09-13T13:18:35.281750+0000","last_active":"2024-09-13T13:18:53.085237+0000","last_peered":"2024-09-13T13:18:53.085237+0000","last_clean":"2024-09-13T13:18:53.085237+0000","last_became_active":"2024-09-13T13:18:35.239702+0000","last_became_peered":"2024-09-13T13:18:35.239702+0000","last_unstale":"2024-09-13T13:18:53.085237+0000","last_undegraded":"2024-09-13T13:18:53.085237+0000","last_fullsized":"2024-09-13T13:18:53.085237+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:17:56.001763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:56.001763+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:56.286987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":328,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067860+0000","last_change":"2024-09-13T13:18:32.362014+0000","last_active":"2024-09-13T13:18:53.067860+0000","last_peered":"2024-09-13T13:18:53.067860+0000","last_clean":"2024-09-13T13:18:53.067860+0000","last_became_active":"2024-09-13T13:17:25.245239+0000","last_became_peered":"2024-09-13T13:17:25.245239+0000","last_unstale":"2024-09-13T13:18:53.067860+0000","last_undegraded":"2024-09-13T13:18:53.067860+0000","last_fullsized":"2024-09-13T13:18:53.067860+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:32.361971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:32.361971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:43.233668+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":361,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068911+0000","last_change":"2024-09-13T13:17:25.339695+0000","last_active":"2024-09-13T13:18:53.068911+0000","last_peered":"2024-09-13T13:18:53.068911+0000","last_clean":"2024-09-13T13:18:53.068911+0000","last_became_active":"2024-09-13T13:17:25.329235+0000","last_became_peered":"2024-09-13T13:17:25.329235+0000","last_unstale":"2024-09-13T13:18:53.068911+0000","last_undegraded":"2024-09-13T13:18:53.068911+0000","last_fullsized":"2024-09-13T13:18:53.068911+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:05.207445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.207445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:29.581260+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":856,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.078516+0000","last_change":"2024-09-13T13:17:01.162092+0000","last_active":"2024-09-13T13:18:53.078516+0000","last_peered":"2024-09-13T13:18:53.078516+0000","last_clean":"2024-09-13T13:18:53.078516+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:18:53.078516+0000","last_undegraded":"2024-09-13T13:18:53.078516+0000","last_fullsized":"2024-09-13T13:18:53.078516+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:17:01.162018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:01.162018+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:25.064905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"268'20","reported_seq":882,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.073568+0000","last_change":"2024-09-13T13:18:41.472174+0000","last_active":"2024-09-13T13:18:53.073568+0000","last_peered":"2024-09-13T13:18:53.073568+0000","last_clean":"2024-09-13T13:18:53.073568+0000","last_became_active":"2024-09-13T13:01:04.207651+0000","last_became_peered":"2024-09-13T13:01:04.207651+0000","last_unstale":"2024-09-13T13:18:53.073568+0000","last_undegraded":"2024-09-13T13:18:53.073568+0000","last_fullsized":"2024-09-13T13:18:53.073568+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:18:41.472008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.472008+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:56.597353+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"306'12","reported_seq":1028,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085679+0000","last_change":"2024-09-13T13:14:21.530680+0000","last_active":"2024-09-13T13:18:53.085679+0000","last_peered":"2024-09-13T13:18:53.085679+0000","last_clean":"2024-09-13T13:18:53.085679+0000","last_became_active":"2024-09-13T13:12:52.947886+0000","last_became_peered":"2024-09-13T13:12:52.947886+0000","last_unstale":"2024-09-13T13:18:53.085679+0000","last_undegraded":"2024-09-13T13:18:53.085679+0000","last_fullsized":"2024-09-13T13:18:53.085679+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'12","last_scrub_stamp":"2024-09-13T13:14:21.530612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:21.530612+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"447'5418","reported_seq":5477,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:54.165805+0000","last_change":"2024-09-13T13:17:55.200504+0000","last_active":"2024-09-13T13:18:54.165805+0000","last_peered":"2024-09-13T13:18:54.165805+0000","last_clean":"2024-09-13T13:18:54.165805+0000","last_became_active":"2024-09-13T13:17:55.200162+0000","last_became_peered":"2024-09-13T13:17:55.200162+0000","last_unstale":"2024-09-13T13:18:54.165805+0000","last_undegraded":"2024-09-13T13:18:54.165805+0000","last_fullsized":"2024-09-13T13:18:54.165805+0000","mapping_epoch":423,"log_start":"427'600","ondisk_log_start":"427'600","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4818,"log_dups_size":0,"ondisk_log_size":4818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:09.157335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355074048,"num_objects":5418,"num_object_clones":0,"num_object_copies":16254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5418,"num_write_kb":346752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":78,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068957+0000","last_change":"2024-09-13T13:18:49.869928+0000","last_active":"2024-09-13T13:18:53.068957+0000","last_peered":"2024-09-13T13:18:53.068957+0000","last_clean":"2024-09-13T13:18:53.068957+0000","last_became_active":"2024-09-13T13:17:49.052966+0000","last_became_peered":"2024-09-13T13:17:49.052966+0000","last_unstale":"2024-09-13T13:18:53.068957+0000","last_undegraded":"2024-09-13T13:18:53.068957+0000","last_fullsized":"2024-09-13T13:18:53.068957+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:49.869851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:49.869851+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:57.610765+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"269'27","reported_seq":934,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.080564+0000","last_change":"2024-09-13T13:18:05.102318+0000","last_active":"2024-09-13T13:18:53.080564+0000","last_peered":"2024-09-13T13:18:53.080564+0000","last_clean":"2024-09-13T13:18:53.080564+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:18:53.080564+0000","last_undegraded":"2024-09-13T13:18:53.080564+0000","last_fullsized":"2024-09-13T13:18:53.080564+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:18:05.102223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:05.102223+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:29.926605+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":857,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.095160+0000","last_change":"2024-09-13T13:16:58.980151+0000","last_active":"2024-09-13T13:18:53.095160+0000","last_peered":"2024-09-13T13:18:53.095160+0000","last_clean":"2024-09-13T13:18:53.095160+0000","last_became_active":"2024-09-13T13:01:02.186447+0000","last_became_peered":"2024-09-13T13:01:02.186447+0000","last_unstale":"2024-09-13T13:18:53.095160+0000","last_undegraded":"2024-09-13T13:18:53.095160+0000","last_fullsized":"2024-09-13T13:18:53.095160+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:58.980098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.980098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":381,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069453+0000","last_change":"2024-09-13T13:15:32.212185+0000","last_active":"2024-09-13T13:18:53.069453+0000","last_peered":"2024-09-13T13:18:53.069453+0000","last_clean":"2024-09-13T13:18:53.069453+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:18:53.069453+0000","last_undegraded":"2024-09-13T13:18:53.069453+0000","last_fullsized":"2024-09-13T13:18:53.069453+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:15:32.212101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:32.212101+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:25.305722+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":828,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068950+0000","last_change":"2024-09-13T13:13:57.546725+0000","last_active":"2024-09-13T13:18:53.068950+0000","last_peered":"2024-09-13T13:18:53.068950+0000","last_clean":"2024-09-13T13:18:53.068950+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:18:53.068950+0000","last_undegraded":"2024-09-13T13:18:53.068950+0000","last_fullsized":"2024-09-13T13:18:53.068950+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:57.546671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:13:57.546671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"447'5313","reported_seq":5372,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:54.221334+0000","last_change":"2024-09-13T13:17:55.200709+0000","last_active":"2024-09-13T13:18:54.221334+0000","last_peered":"2024-09-13T13:18:54.221334+0000","last_clean":"2024-09-13T13:18:54.221334+0000","last_became_active":"2024-09-13T13:17:55.200283+0000","last_became_peered":"2024-09-13T13:17:55.200283+0000","last_unstale":"2024-09-13T13:18:54.221334+0000","last_undegraded":"2024-09-13T13:18:54.221334+0000","last_fullsized":"2024-09-13T13:18:54.221334+0000","mapping_epoch":423,"log_start":"427'500","ondisk_log_start":"427'500","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4813,"log_dups_size":0,"ondisk_log_size":4813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:00.586409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348192768,"num_objects":5313,"num_object_clones":0,"num_object_copies":15939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5313,"num_write_kb":340032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":78,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070398+0000","last_change":"2024-09-13T13:18:51.159546+0000","last_active":"2024-09-13T13:18:53.070398+0000","last_peered":"2024-09-13T13:18:53.070398+0000","last_clean":"2024-09-13T13:18:53.070398+0000","last_became_active":"2024-09-13T13:17:49.050655+0000","last_became_peered":"2024-09-13T13:17:49.050655+0000","last_unstale":"2024-09-13T13:18:53.070398+0000","last_undegraded":"2024-09-13T13:18:53.070398+0000","last_fullsized":"2024-09-13T13:18:53.070398+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.159496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.159496+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:01.476223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"268'30","reported_seq":871,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.094878+0000","last_change":"2024-09-13T13:18:36.922068+0000","last_active":"2024-09-13T13:18:53.094878+0000","last_peered":"2024-09-13T13:18:53.094878+0000","last_clean":"2024-09-13T13:18:53.094878+0000","last_became_active":"2024-09-13T13:01:04.208884+0000","last_became_peered":"2024-09-13T13:01:04.208884+0000","last_unstale":"2024-09-13T13:18:53.094878+0000","last_undegraded":"2024-09-13T13:18:53.094878+0000","last_fullsized":"2024-09-13T13:18:53.094878+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:18:36.921964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:36.921964+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:40.263721+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":831,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085715+0000","last_change":"2024-09-13T13:16:26.098415+0000","last_active":"2024-09-13T13:18:53.085715+0000","last_peered":"2024-09-13T13:18:53.085715+0000","last_clean":"2024-09-13T13:18:53.085715+0000","last_became_active":"2024-09-13T13:01:02.180681+0000","last_became_peered":"2024-09-13T13:01:02.180681+0000","last_unstale":"2024-09-13T13:18:53.085715+0000","last_undegraded":"2024-09-13T13:18:53.085715+0000","last_fullsized":"2024-09-13T13:18:53.085715+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:16:26.098365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:26.098365+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:30.772726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":312,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.066930+0000","last_change":"2024-09-13T13:17:58.389822+0000","last_active":"2024-09-13T13:18:53.066930+0000","last_peered":"2024-09-13T13:18:53.066930+0000","last_clean":"2024-09-13T13:18:53.066930+0000","last_became_active":"2024-09-13T13:12:53.006199+0000","last_became_peered":"2024-09-13T13:12:53.006199+0000","last_unstale":"2024-09-13T13:18:53.066930+0000","last_undegraded":"2024-09-13T13:18:53.066930+0000","last_fullsized":"2024-09-13T13:18:53.066930+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:58.389763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:58.389763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:19.447538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":334,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067540+0000","last_change":"2024-09-13T13:17:59.373240+0000","last_active":"2024-09-13T13:18:53.067540+0000","last_peered":"2024-09-13T13:18:53.067540+0000","last_clean":"2024-09-13T13:18:53.067540+0000","last_became_active":"2024-09-13T13:17:25.244171+0000","last_became_peered":"2024-09-13T13:17:25.244171+0000","last_unstale":"2024-09-13T13:18:53.067540+0000","last_undegraded":"2024-09-13T13:18:53.067540+0000","last_fullsized":"2024-09-13T13:18:53.067540+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.373194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.373194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:20.574739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"447'5175","reported_seq":5234,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.255590+0000","last_change":"2024-09-13T13:17:55.200309+0000","last_active":"2024-09-13T13:18:53.255590+0000","last_peered":"2024-09-13T13:18:53.255590+0000","last_clean":"2024-09-13T13:18:53.255590+0000","last_became_active":"2024-09-13T13:17:55.200037+0000","last_became_peered":"2024-09-13T13:17:55.200037+0000","last_unstale":"2024-09-13T13:18:53.255590+0000","last_undegraded":"2024-09-13T13:18:53.255590+0000","last_fullsized":"2024-09-13T13:18:53.255590+0000","mapping_epoch":423,"log_start":"427'500","ondisk_log_start":"427'500","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4675,"log_dups_size":0,"ondisk_log_size":4675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:01.812502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339148800,"num_objects":5175,"num_object_clones":0,"num_object_copies":15525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":331200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":78,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.080669+0000","last_change":"2024-09-13T13:18:53.080669+0000","last_active":"2024-09-13T13:18:53.080669+0000","last_peered":"2024-09-13T13:18:53.080669+0000","last_clean":"2024-09-13T13:18:53.080669+0000","last_became_active":"2024-09-13T13:17:49.053135+0000","last_became_peered":"2024-09-13T13:17:49.053135+0000","last_unstale":"2024-09-13T13:18:53.080669+0000","last_undegraded":"2024-09-13T13:18:53.080669+0000","last_fullsized":"2024-09-13T13:18:53.080669+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:15:54.707006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:15:54.707006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:19:22.041857+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"268'28","reported_seq":320,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067090+0000","last_change":"2024-09-13T13:13:50.716312+0000","last_active":"2024-09-13T13:18:53.067090+0000","last_peered":"2024-09-13T13:18:53.067090+0000","last_clean":"2024-09-13T13:18:53.067090+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:18:53.067090+0000","last_undegraded":"2024-09-13T13:18:53.067090+0000","last_fullsized":"2024-09-13T13:18:53.067090+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:13:50.716243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:50.716243+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:22.408438+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":839,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.074667+0000","last_change":"2024-09-13T13:17:02.704294+0000","last_active":"2024-09-13T13:18:53.074667+0000","last_peered":"2024-09-13T13:18:53.074667+0000","last_clean":"2024-09-13T13:18:53.074667+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:18:53.074667+0000","last_undegraded":"2024-09-13T13:18:53.074667+0000","last_fullsized":"2024-09-13T13:18:53.074667+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:17:02.704233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.704233+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:21.560467+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":1080,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.094741+0000","last_change":"2024-09-13T13:18:48.032272+0000","last_active":"2024-09-13T13:18:53.094741+0000","last_peered":"2024-09-13T13:18:53.094741+0000","last_clean":"2024-09-13T13:18:53.094741+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:18:53.094741+0000","last_undegraded":"2024-09-13T13:18:53.094741+0000","last_fullsized":"2024-09-13T13:18:53.094741+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:18:48.032199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:48.032199+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:17.576150+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":477,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085764+0000","last_change":"2024-09-13T13:17:59.355700+0000","last_active":"2024-09-13T13:18:53.085764+0000","last_peered":"2024-09-13T13:18:53.085764+0000","last_clean":"2024-09-13T13:18:53.085764+0000","last_became_active":"2024-09-13T13:17:25.244188+0000","last_became_peered":"2024-09-13T13:17:25.244188+0000","last_unstale":"2024-09-13T13:18:53.085764+0000","last_undegraded":"2024-09-13T13:18:53.085764+0000","last_fullsized":"2024-09-13T13:18:53.085764+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.355639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.355639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:00.304064+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"447'5362","reported_seq":5421,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:57.904460+0000","last_change":"2024-09-13T13:17:55.131505+0000","last_active":"2024-09-13T13:18:57.904460+0000","last_peered":"2024-09-13T13:18:57.904460+0000","last_clean":"2024-09-13T13:18:57.904460+0000","last_became_active":"2024-09-13T13:17:55.131230+0000","last_became_peered":"2024-09-13T13:17:55.131230+0000","last_unstale":"2024-09-13T13:18:57.904460+0000","last_undegraded":"2024-09-13T13:18:57.904460+0000","last_fullsized":"2024-09-13T13:18:57.904460+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5362,"log_dups_size":0,"ondisk_log_size":5362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:59.022573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351404032,"num_objects":5362,"num_object_clones":0,"num_object_copies":16086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5362,"num_write_kb":343168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":154,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085431+0000","last_change":"2024-09-13T13:18:50.404870+0000","last_active":"2024-09-13T13:18:53.085431+0000","last_peered":"2024-09-13T13:18:53.085431+0000","last_clean":"2024-09-13T13:18:53.085431+0000","last_became_active":"2024-09-13T13:17:49.052271+0000","last_became_peered":"2024-09-13T13:17:49.052271+0000","last_unstale":"2024-09-13T13:18:53.085431+0000","last_undegraded":"2024-09-13T13:18:53.085431+0000","last_fullsized":"2024-09-13T13:18:53.085431+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:50.404821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:50.404821+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:55.364511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"268'20","reported_seq":890,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.079078+0000","last_change":"2024-09-13T13:12:53.016127+0000","last_active":"2024-09-13T13:18:53.079078+0000","last_peered":"2024-09-13T13:18:53.079078+0000","last_clean":"2024-09-13T13:18:53.079078+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:18:53.079078+0000","last_undegraded":"2024-09-13T13:18:53.079078+0000","last_fullsized":"2024-09-13T13:18:53.079078+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'10","last_scrub_stamp":"2024-09-13T13:10:40.651349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:10:40.651349+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:53.042997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":871,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069564+0000","last_change":"2024-09-13T13:17:25.331470+0000","last_active":"2024-09-13T13:18:53.069564+0000","last_peered":"2024-09-13T13:18:53.069564+0000","last_clean":"2024-09-13T13:18:53.069564+0000","last_became_active":"2024-09-13T13:17:25.331166+0000","last_became_peered":"2024-09-13T13:17:25.331166+0000","last_unstale":"2024-09-13T13:18:53.069564+0000","last_undegraded":"2024-09-13T13:18:53.069564+0000","last_fullsized":"2024-09-13T13:18:53.069564+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:04.755828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:04.755828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.345827+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":1061,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070839+0000","last_change":"2024-09-13T13:18:40.464456+0000","last_active":"2024-09-13T13:18:53.070839+0000","last_peered":"2024-09-13T13:18:53.070839+0000","last_clean":"2024-09-13T13:18:53.070839+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:18:53.070839+0000","last_undegraded":"2024-09-13T13:18:53.070839+0000","last_fullsized":"2024-09-13T13:18:53.070839+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:18:40.464398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:40.464398+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:58.756596+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":866,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069781+0000","last_change":"2024-09-13T13:18:36.466657+0000","last_active":"2024-09-13T13:18:53.069781+0000","last_peered":"2024-09-13T13:18:53.069781+0000","last_clean":"2024-09-13T13:18:53.069781+0000","last_became_active":"2024-09-13T13:17:25.252712+0000","last_became_peered":"2024-09-13T13:17:25.252712+0000","last_unstale":"2024-09-13T13:18:53.069781+0000","last_undegraded":"2024-09-13T13:18:53.069781+0000","last_fullsized":"2024-09-13T13:18:53.069781+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:36.466575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:36.466575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:56.925016+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"447'5248","reported_seq":5307,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:54.187847+0000","last_change":"2024-09-13T13:17:55.140405+0000","last_active":"2024-09-13T13:18:54.187847+0000","last_peered":"2024-09-13T13:18:54.187847+0000","last_clean":"2024-09-13T13:18:54.187847+0000","last_became_active":"2024-09-13T13:17:55.140134+0000","last_became_peered":"2024-09-13T13:17:55.140134+0000","last_unstale":"2024-09-13T13:18:54.187847+0000","last_undegraded":"2024-09-13T13:18:54.187847+0000","last_fullsized":"2024-09-13T13:18:54.187847+0000","mapping_epoch":423,"log_start":"427'400","ondisk_log_start":"427'400","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4848,"log_dups_size":0,"ondisk_log_size":4848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.372449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343932928,"num_objects":5248,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5248,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":154,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085293+0000","last_change":"2024-09-13T13:18:51.388593+0000","last_active":"2024-09-13T13:18:53.085293+0000","last_peered":"2024-09-13T13:18:53.085293+0000","last_clean":"2024-09-13T13:18:53.085293+0000","last_became_active":"2024-09-13T13:17:49.055888+0000","last_became_peered":"2024-09-13T13:17:49.055888+0000","last_unstale":"2024-09-13T13:18:53.085293+0000","last_undegraded":"2024-09-13T13:18:53.085293+0000","last_fullsized":"2024-09-13T13:18:53.085293+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.388502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.388502+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:16.343178+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"268'29","reported_seq":903,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.071050+0000","last_change":"2024-09-13T13:18:33.796717+0000","last_active":"2024-09-13T13:18:53.071050+0000","last_peered":"2024-09-13T13:18:53.071050+0000","last_clean":"2024-09-13T13:18:53.071050+0000","last_became_active":"2024-09-13T13:01:04.207502+0000","last_became_peered":"2024-09-13T13:01:04.207502+0000","last_unstale":"2024-09-13T13:18:53.071050+0000","last_undegraded":"2024-09-13T13:18:53.071050+0000","last_fullsized":"2024-09-13T13:18:53.071050+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:18:33.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.796644+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:36.008576+0000","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":848,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.078989+0000","last_change":"2024-09-13T13:17:53.069585+0000","last_active":"2024-09-13T13:18:53.078989+0000","last_peered":"2024-09-13T13:18:53.078989+0000","last_clean":"2024-09-13T13:18:53.078989+0000","last_became_active":"2024-09-13T13:17:25.332589+0000","last_became_peered":"2024-09-13T13:17:25.332589+0000","last_unstale":"2024-09-13T13:18:53.078989+0000","last_undegraded":"2024-09-13T13:18:53.078989+0000","last_fullsized":"2024-09-13T13:18:53.078989+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:53.069538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:53.069538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:19.459005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":810,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085960+0000","last_change":"2024-09-13T13:17:58.379729+0000","last_active":"2024-09-13T13:18:53.085960+0000","last_peered":"2024-09-13T13:18:53.085960+0000","last_clean":"2024-09-13T13:18:53.085960+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:18:53.085960+0000","last_undegraded":"2024-09-13T13:18:53.085960+0000","last_fullsized":"2024-09-13T13:18:53.085960+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:58.379674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:58.379674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:16.089064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":855,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.078265+0000","last_change":"2024-09-13T13:17:59.178198+0000","last_active":"2024-09-13T13:18:53.078265+0000","last_peered":"2024-09-13T13:18:53.078265+0000","last_clean":"2024-09-13T13:18:53.078265+0000","last_became_active":"2024-09-13T13:17:25.244517+0000","last_became_peered":"2024-09-13T13:17:25.244517+0000","last_unstale":"2024-09-13T13:18:53.078265+0000","last_undegraded":"2024-09-13T13:18:53.078265+0000","last_fullsized":"2024-09-13T13:18:53.078265+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.178129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.178129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.299906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"447'5317","reported_seq":5376,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:57.902501+0000","last_change":"2024-09-13T13:17:55.146287+0000","last_active":"2024-09-13T13:18:57.902501+0000","last_peered":"2024-09-13T13:18:57.902501+0000","last_clean":"2024-09-13T13:18:57.902501+0000","last_became_active":"2024-09-13T13:17:55.145616+0000","last_became_peered":"2024-09-13T13:17:55.145616+0000","last_unstale":"2024-09-13T13:18:57.902501+0000","last_undegraded":"2024-09-13T13:18:57.902501+0000","last_fullsized":"2024-09-13T13:18:57.902501+0000","mapping_epoch":423,"log_start":"427'800","ondisk_log_start":"427'800","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4517,"log_dups_size":0,"ondisk_log_size":4517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:20.121854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348454912,"num_objects":5317,"num_object_clones":0,"num_object_copies":15951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":340288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":78,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068352+0000","last_change":"2024-09-13T13:18:50.856493+0000","last_active":"2024-09-13T13:18:53.068352+0000","last_peered":"2024-09-13T13:18:53.068352+0000","last_clean":"2024-09-13T13:18:53.068352+0000","last_became_active":"2024-09-13T13:17:49.054449+0000","last_became_peered":"2024-09-13T13:17:49.054449+0000","last_unstale":"2024-09-13T13:18:53.068352+0000","last_undegraded":"2024-09-13T13:18:53.068352+0000","last_fullsized":"2024-09-13T13:18:53.068352+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:50.856437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:50.856437+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:03.097517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"443'43","reported_seq":48,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070467+0000","last_change":"2024-09-13T13:18:35.332086+0000","last_active":"2024-09-13T13:18:53.070467+0000","last_peered":"2024-09-13T13:18:53.070467+0000","last_clean":"2024-09-13T13:18:53.070467+0000","last_became_active":"2024-09-13T13:18:35.244311+0000","last_became_peered":"2024-09-13T13:18:35.244311+0000","last_unstale":"2024-09-13T13:18:53.070467+0000","last_undegraded":"2024-09-13T13:18:53.070467+0000","last_fullsized":"2024-09-13T13:18:53.070467+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'42","last_scrub_stamp":"2024-09-13T13:18:01.158345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:01.158345+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:27.731963+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":320,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068594+0000","last_change":"2024-09-13T13:17:52.422920+0000","last_active":"2024-09-13T13:18:53.068594+0000","last_peered":"2024-09-13T13:18:53.068594+0000","last_clean":"2024-09-13T13:18:53.068594+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:18:53.068594+0000","last_undegraded":"2024-09-13T13:18:53.068594+0000","last_fullsized":"2024-09-13T13:18:53.068594+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:52.422872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.422872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:02.132631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":852,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069427+0000","last_change":"2024-09-13T13:18:07.306441+0000","last_active":"2024-09-13T13:18:53.069427+0000","last_peered":"2024-09-13T13:18:53.069427+0000","last_clean":"2024-09-13T13:18:53.069427+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:18:53.069427+0000","last_undegraded":"2024-09-13T13:18:53.069427+0000","last_fullsized":"2024-09-13T13:18:53.069427+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:18:07.306366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:07.306366+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:17.328610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":846,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.071532+0000","last_change":"2024-09-13T13:14:59.670853+0000","last_active":"2024-09-13T13:18:53.071532+0000","last_peered":"2024-09-13T13:18:53.071532+0000","last_clean":"2024-09-13T13:18:53.071532+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:18:53.071532+0000","last_undegraded":"2024-09-13T13:18:53.071532+0000","last_fullsized":"2024-09-13T13:18:53.071532+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:59.670779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:14:59.670779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:18.561846+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":843,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085121+0000","last_change":"2024-09-13T13:17:05.016876+0000","last_active":"2024-09-13T13:18:53.085121+0000","last_peered":"2024-09-13T13:18:53.085121+0000","last_clean":"2024-09-13T13:18:53.085121+0000","last_became_active":"2024-09-13T13:12:52.948742+0000","last_became_peered":"2024-09-13T13:12:52.948742+0000","last_unstale":"2024-09-13T13:18:53.085121+0000","last_undegraded":"2024-09-13T13:18:53.085121+0000","last_fullsized":"2024-09-13T13:18:53.085121+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:05.016806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.016806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"447'5559","reported_seq":5618,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:57.952890+0000","last_change":"2024-09-13T13:17:55.132888+0000","last_active":"2024-09-13T13:18:57.952890+0000","last_peered":"2024-09-13T13:18:57.952890+0000","last_clean":"2024-09-13T13:18:57.952890+0000","last_became_active":"2024-09-13T13:17:55.132636+0000","last_became_peered":"2024-09-13T13:17:55.132636+0000","last_unstale":"2024-09-13T13:18:57.952890+0000","last_undegraded":"2024-09-13T13:18:57.952890+0000","last_fullsized":"2024-09-13T13:18:57.952890+0000","mapping_epoch":423,"log_start":"428'1000","ondisk_log_start":"428'1000","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.085485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364314624,"num_objects":5559,"num_object_clones":0,"num_object_copies":16677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":355776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":440,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070180+0000","last_change":"2024-09-13T13:18:51.529311+0000","last_active":"2024-09-13T13:18:53.070180+0000","last_peered":"2024-09-13T13:18:53.070180+0000","last_clean":"2024-09-13T13:18:53.070180+0000","last_became_active":"2024-09-13T13:17:49.057194+0000","last_became_peered":"2024-09-13T13:17:49.057194+0000","last_unstale":"2024-09-13T13:18:53.070180+0000","last_undegraded":"2024-09-13T13:18:53.070180+0000","last_fullsized":"2024-09-13T13:18:53.070180+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.529089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.529089+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:58.783716+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"268'17","reported_seq":863,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.080662+0000","last_change":"2024-09-13T13:17:32.264550+0000","last_active":"2024-09-13T13:18:53.080662+0000","last_peered":"2024-09-13T13:18:53.080662+0000","last_clean":"2024-09-13T13:18:53.080662+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:18:53.080662+0000","last_undegraded":"2024-09-13T13:18:53.080662+0000","last_fullsized":"2024-09-13T13:18:53.080662+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'17","last_scrub_stamp":"2024-09-13T13:17:32.263873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:32.263873+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:01.566837+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":875,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.080821+0000","last_change":"2024-09-13T13:18:33.892926+0000","last_active":"2024-09-13T13:18:53.080821+0000","last_peered":"2024-09-13T13:18:53.080821+0000","last_clean":"2024-09-13T13:18:53.080821+0000","last_became_active":"2024-09-13T13:17:25.246400+0000","last_became_peered":"2024-09-13T13:17:25.246400+0000","last_unstale":"2024-09-13T13:18:53.080821+0000","last_undegraded":"2024-09-13T13:18:53.080821+0000","last_fullsized":"2024-09-13T13:18:53.080821+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:33.892597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.892597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:39.873900+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":100,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.073062+0000","last_change":"2024-09-13T13:18:44.719355+0000","last_active":"2024-09-13T13:18:53.073062+0000","last_peered":"2024-09-13T13:18:53.073062+0000","last_clean":"2024-09-13T13:18:53.073062+0000","last_became_active":"2024-09-13T13:17:25.252664+0000","last_became_peered":"2024-09-13T13:17:25.252664+0000","last_unstale":"2024-09-13T13:18:53.073062+0000","last_undegraded":"2024-09-13T13:18:53.073062+0000","last_fullsized":"2024-09-13T13:18:53.073062+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:18:44.719269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:44.719269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:12.759919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":96,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.076091+0000","last_change":"2024-09-13T13:17:59.698099+0000","last_active":"2024-09-13T13:18:53.076091+0000","last_peered":"2024-09-13T13:18:53.076091+0000","last_clean":"2024-09-13T13:18:53.076091+0000","last_became_active":"2024-09-13T13:17:25.332669+0000","last_became_peered":"2024-09-13T13:17:25.332669+0000","last_unstale":"2024-09-13T13:18:53.076091+0000","last_undegraded":"2024-09-13T13:18:53.076091+0000","last_fullsized":"2024-09-13T13:18:53.076091+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.698047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.698047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.194640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":447,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.086889+0000","last_change":"2024-09-13T13:12:41.332308+0000","last_active":"2024-09-13T13:18:53.086889+0000","last_peered":"2024-09-13T13:18:53.086889+0000","last_clean":"2024-09-13T13:18:53.086889+0000","last_became_active":"2024-09-13T13:12:40.120242+0000","last_became_peered":"2024-09-13T13:12:40.120242+0000","last_unstale":"2024-09-13T13:18:53.086889+0000","last_undegraded":"2024-09-13T13:18:53.086889+0000","last_fullsized":"2024-09-13T13:18:53.086889+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:41.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:12:41.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:25.287809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"447'5479","reported_seq":5538,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:57.946158+0000","last_change":"2024-09-13T13:17:55.143535+0000","last_active":"2024-09-13T13:18:57.946158+0000","last_peered":"2024-09-13T13:18:57.946158+0000","last_clean":"2024-09-13T13:18:57.946158+0000","last_became_active":"2024-09-13T13:17:55.143252+0000","last_became_peered":"2024-09-13T13:17:55.143252+0000","last_unstale":"2024-09-13T13:18:57.946158+0000","last_undegraded":"2024-09-13T13:18:57.946158+0000","last_fullsized":"2024-09-13T13:18:57.946158+0000","mapping_epoch":423,"log_start":"428'1000","ondisk_log_start":"428'1000","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4479,"log_dups_size":0,"ondisk_log_size":4479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:14.112065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359071744,"num_objects":5479,"num_object_clones":0,"num_object_copies":16437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5479,"num_write_kb":350656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":78,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.080479+0000","last_change":"2024-09-13T13:18:51.048420+0000","last_active":"2024-09-13T13:18:53.080479+0000","last_peered":"2024-09-13T13:18:53.080479+0000","last_clean":"2024-09-13T13:18:53.080479+0000","last_became_active":"2024-09-13T13:17:49.049531+0000","last_became_peered":"2024-09-13T13:17:49.049531+0000","last_unstale":"2024-09-13T13:18:53.080479+0000","last_undegraded":"2024-09-13T13:18:53.080479+0000","last_fullsized":"2024-09-13T13:18:53.080479+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.048347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.048347+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:56.866592+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"268'24","reported_seq":914,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.073660+0000","last_change":"2024-09-13T13:18:04.569761+0000","last_active":"2024-09-13T13:18:53.073660+0000","last_peered":"2024-09-13T13:18:53.073660+0000","last_clean":"2024-09-13T13:18:53.073660+0000","last_became_active":"2024-09-13T13:17:25.253120+0000","last_became_peered":"2024-09-13T13:17:25.253120+0000","last_unstale":"2024-09-13T13:18:53.073660+0000","last_undegraded":"2024-09-13T13:18:53.073660+0000","last_fullsized":"2024-09-13T13:18:53.073660+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:18:04.569670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:04.569670+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:14.912474+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":854,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070863+0000","last_change":"2024-09-13T13:18:37.465516+0000","last_active":"2024-09-13T13:18:53.070863+0000","last_peered":"2024-09-13T13:18:53.070863+0000","last_clean":"2024-09-13T13:18:53.070863+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:18:53.070863+0000","last_undegraded":"2024-09-13T13:18:53.070863+0000","last_fullsized":"2024-09-13T13:18:53.070863+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:37.465371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:37.465371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:41.600910+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":751,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.080079+0000","last_change":"2024-09-13T13:17:25.325347+0000","last_active":"2024-09-13T13:18:53.080079+0000","last_peered":"2024-09-13T13:18:53.080079+0000","last_clean":"2024-09-13T13:18:53.080079+0000","last_became_active":"2024-09-13T13:12:39.947940+0000","last_became_peered":"2024-09-13T13:12:39.947940+0000","last_unstale":"2024-09-13T13:18:53.080079+0000","last_undegraded":"2024-09-13T13:18:53.080079+0000","last_fullsized":"2024-09-13T13:18:53.080079+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:17:25.325275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:25.325275+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":380,"num_read_kb":375,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":825,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070455+0000","last_change":"2024-09-13T13:17:55.293423+0000","last_active":"2024-09-13T13:18:53.070455+0000","last_peered":"2024-09-13T13:18:53.070455+0000","last_clean":"2024-09-13T13:18:53.070455+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:18:53.070455+0000","last_undegraded":"2024-09-13T13:18:53.070455+0000","last_fullsized":"2024-09-13T13:18:53.070455+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:55.293375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:55.293375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:20.691046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":830,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.078139+0000","last_change":"2024-09-13T13:18:46.991251+0000","last_active":"2024-09-13T13:18:53.078139+0000","last_peered":"2024-09-13T13:18:53.078139+0000","last_clean":"2024-09-13T13:18:53.078139+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:18:53.078139+0000","last_undegraded":"2024-09-13T13:18:53.078139+0000","last_fullsized":"2024-09-13T13:18:53.078139+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:46.991076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.991076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:06.527735+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"447'5540","reported_seq":5599,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:57.929680+0000","last_change":"2024-09-13T13:17:55.142415+0000","last_active":"2024-09-13T13:18:57.929680+0000","last_peered":"2024-09-13T13:18:57.929680+0000","last_clean":"2024-09-13T13:18:57.929680+0000","last_became_active":"2024-09-13T13:17:55.142112+0000","last_became_peered":"2024-09-13T13:17:55.142112+0000","last_unstale":"2024-09-13T13:18:57.929680+0000","last_undegraded":"2024-09-13T13:18:57.929680+0000","last_fullsized":"2024-09-13T13:18:57.929680+0000","mapping_epoch":423,"log_start":"428'1000","ondisk_log_start":"428'1000","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:01.465992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363069440,"num_objects":5540,"num_object_clones":0,"num_object_copies":16620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5540,"num_write_kb":354560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"306'32","reported_seq":986,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085279+0000","last_change":"2024-09-13T13:13:01.352416+0000","last_active":"2024-09-13T13:18:53.085279+0000","last_peered":"2024-09-13T13:18:53.085279+0000","last_clean":"2024-09-13T13:18:53.085279+0000","last_became_active":"2024-09-13T13:12:52.999924+0000","last_became_peered":"2024-09-13T13:12:52.999924+0000","last_unstale":"2024-09-13T13:18:53.085279+0000","last_undegraded":"2024-09-13T13:18:53.085279+0000","last_fullsized":"2024-09-13T13:18:53.085279+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:13:01.352337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.352337+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.222564+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":844,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.086800+0000","last_change":"2024-09-13T13:16:53.951208+0000","last_active":"2024-09-13T13:18:53.086800+0000","last_peered":"2024-09-13T13:18:53.086800+0000","last_clean":"2024-09-13T13:18:53.086800+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:18:53.086800+0000","last_undegraded":"2024-09-13T13:18:53.086800+0000","last_fullsized":"2024-09-13T13:18:53.086800+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:53.951135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:53.951135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:31.816447+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":438,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069938+0000","last_change":"2024-09-13T13:18:50.486006+0000","last_active":"2024-09-13T13:18:53.069938+0000","last_peered":"2024-09-13T13:18:53.069938+0000","last_clean":"2024-09-13T13:18:53.069938+0000","last_became_active":"2024-09-13T13:17:47.097787+0000","last_became_peered":"2024-09-13T13:17:47.097787+0000","last_unstale":"2024-09-13T13:18:53.069938+0000","last_undegraded":"2024-09-13T13:18:53.069938+0000","last_fullsized":"2024-09-13T13:18:53.069938+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:50.485924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:50.485924+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:56.721606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":96,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.074174+0000","last_change":"2024-09-13T13:17:58.664229+0000","last_active":"2024-09-13T13:18:53.074174+0000","last_peered":"2024-09-13T13:18:53.074174+0000","last_clean":"2024-09-13T13:18:53.074174+0000","last_became_active":"2024-09-13T13:17:25.328065+0000","last_became_peered":"2024-09-13T13:17:25.328065+0000","last_unstale":"2024-09-13T13:18:53.074174+0000","last_undegraded":"2024-09-13T13:18:53.074174+0000","last_fullsized":"2024-09-13T13:18:53.074174+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:58.664177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:58.664177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:07.154650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":867,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070321+0000","last_change":"2024-09-13T13:18:08.354464+0000","last_active":"2024-09-13T13:18:53.070321+0000","last_peered":"2024-09-13T13:18:53.070321+0000","last_clean":"2024-09-13T13:18:53.070321+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:18:53.070321+0000","last_undegraded":"2024-09-13T13:18:53.070321+0000","last_fullsized":"2024-09-13T13:18:53.070321+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:08.354394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:08.354394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:09.946043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"447'5305","reported_seq":5364,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:56.344764+0000","last_change":"2024-09-13T13:17:55.144368+0000","last_active":"2024-09-13T13:18:56.344764+0000","last_peered":"2024-09-13T13:18:56.344764+0000","last_clean":"2024-09-13T13:18:56.344764+0000","last_became_active":"2024-09-13T13:17:55.144129+0000","last_became_peered":"2024-09-13T13:17:55.144129+0000","last_unstale":"2024-09-13T13:18:56.344764+0000","last_undegraded":"2024-09-13T13:18:56.344764+0000","last_fullsized":"2024-09-13T13:18:56.344764+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5305,"log_dups_size":0,"ondisk_log_size":5305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:17.686405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347668480,"num_objects":5305,"num_object_clones":0,"num_object_copies":15915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5305,"num_write_kb":339520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"268'19","reported_seq":332,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069146+0000","last_change":"2024-09-13T13:18:09.522990+0000","last_active":"2024-09-13T13:18:53.069146+0000","last_peered":"2024-09-13T13:18:53.069146+0000","last_clean":"2024-09-13T13:18:53.069146+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:18:53.069146+0000","last_undegraded":"2024-09-13T13:18:53.069146+0000","last_fullsized":"2024-09-13T13:18:53.069146+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:09.522908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:09.522908+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.176687+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":326,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.066840+0000","last_change":"2024-09-13T13:17:54.465421+0000","last_active":"2024-09-13T13:18:53.066840+0000","last_peered":"2024-09-13T13:18:53.066840+0000","last_clean":"2024-09-13T13:18:53.066840+0000","last_became_active":"2024-09-13T13:17:25.244810+0000","last_became_peered":"2024-09-13T13:17:25.244810+0000","last_unstale":"2024-09-13T13:18:53.066840+0000","last_undegraded":"2024-09-13T13:18:53.066840+0000","last_fullsized":"2024-09-13T13:18:53.066840+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.465363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.465363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:23.531768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":873,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070187+0000","last_change":"2024-09-13T13:15:55.592241+0000","last_active":"2024-09-13T13:18:53.070187+0000","last_peered":"2024-09-13T13:18:53.070187+0000","last_clean":"2024-09-13T13:18:53.070187+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:18:53.070187+0000","last_undegraded":"2024-09-13T13:18:53.070187+0000","last_fullsized":"2024-09-13T13:18:53.070187+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:55.592172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:55.592172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:21.490551+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":831,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.084239+0000","last_change":"2024-09-13T13:18:49.072017+0000","last_active":"2024-09-13T13:18:53.084239+0000","last_peered":"2024-09-13T13:18:53.084239+0000","last_clean":"2024-09-13T13:18:53.084239+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:18:53.084239+0000","last_undegraded":"2024-09-13T13:18:53.084239+0000","last_fullsized":"2024-09-13T13:18:53.084239+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:49.071942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:49.071942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:16.825126+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"447'5506","reported_seq":5564,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:57.933389+0000","last_change":"2024-09-13T13:17:55.139372+0000","last_active":"2024-09-13T13:18:57.933389+0000","last_peered":"2024-09-13T13:18:57.933389+0000","last_clean":"2024-09-13T13:18:57.933389+0000","last_became_active":"2024-09-13T13:17:55.139080+0000","last_became_peered":"2024-09-13T13:17:55.139080+0000","last_unstale":"2024-09-13T13:18:57.933389+0000","last_undegraded":"2024-09-13T13:18:57.933389+0000","last_fullsized":"2024-09-13T13:18:57.933389+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:12.883286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360841216,"num_objects":5506,"num_object_clones":0,"num_object_copies":16518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5506,"num_write_kb":352384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":79,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068092+0000","last_change":"2024-09-13T13:17:51.212127+0000","last_active":"2024-09-13T13:18:53.068092+0000","last_peered":"2024-09-13T13:18:53.068092+0000","last_clean":"2024-09-13T13:18:53.068092+0000","last_became_active":"2024-09-13T13:17:49.053269+0000","last_became_peered":"2024-09-13T13:17:49.053269+0000","last_unstale":"2024-09-13T13:18:53.068092+0000","last_undegraded":"2024-09-13T13:18:53.068092+0000","last_fullsized":"2024-09-13T13:18:53.068092+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:17:51.212071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:17:51.212071+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:10.098094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"268'31","reported_seq":388,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070699+0000","last_change":"2024-09-13T13:18:48.229223+0000","last_active":"2024-09-13T13:18:53.070699+0000","last_peered":"2024-09-13T13:18:53.070699+0000","last_clean":"2024-09-13T13:18:53.070699+0000","last_became_active":"2024-09-13T13:17:25.244686+0000","last_became_peered":"2024-09-13T13:17:25.244686+0000","last_unstale":"2024-09-13T13:18:53.070699+0000","last_undegraded":"2024-09-13T13:18:53.070699+0000","last_fullsized":"2024-09-13T13:18:53.070699+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:18:48.229160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:48.229160+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:59.840762+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":863,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.086237+0000","last_change":"2024-09-13T13:18:03.351698+0000","last_active":"2024-09-13T13:18:53.086237+0000","last_peered":"2024-09-13T13:18:53.086237+0000","last_clean":"2024-09-13T13:18:53.086237+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:18:53.086237+0000","last_undegraded":"2024-09-13T13:18:53.086237+0000","last_fullsized":"2024-09-13T13:18:53.086237+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:03.351653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:03.351653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:11.465761+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":847,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.078193+0000","last_change":"2024-09-13T13:17:05.238854+0000","last_active":"2024-09-13T13:18:53.078193+0000","last_peered":"2024-09-13T13:18:53.078193+0000","last_clean":"2024-09-13T13:18:53.078193+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:18:53.078193+0000","last_undegraded":"2024-09-13T13:18:53.078193+0000","last_fullsized":"2024-09-13T13:18:53.078193+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:05.238722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:05.238722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.936311+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":88,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.073535+0000","last_change":"2024-09-13T13:17:25.243030+0000","last_active":"2024-09-13T13:18:53.073535+0000","last_peered":"2024-09-13T13:18:53.073535+0000","last_clean":"2024-09-13T13:18:53.073535+0000","last_became_active":"2024-09-13T13:17:25.242366+0000","last_became_peered":"2024-09-13T13:17:25.242366+0000","last_unstale":"2024-09-13T13:18:53.073535+0000","last_undegraded":"2024-09-13T13:18:53.073535+0000","last_fullsized":"2024-09-13T13:18:53.073535+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:35.064188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:35.064188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.698744+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"447'5470","reported_seq":5529,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:56.359731+0000","last_change":"2024-09-13T13:17:55.200332+0000","last_active":"2024-09-13T13:18:56.359731+0000","last_peered":"2024-09-13T13:18:56.359731+0000","last_clean":"2024-09-13T13:18:56.359731+0000","last_became_active":"2024-09-13T13:17:55.199999+0000","last_became_peered":"2024-09-13T13:17:55.199999+0000","last_unstale":"2024-09-13T13:18:56.359731+0000","last_undegraded":"2024-09-13T13:18:56.359731+0000","last_fullsized":"2024-09-13T13:18:56.359731+0000","mapping_epoch":423,"log_start":"427'100","ondisk_log_start":"427'100","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:09.224261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358481920,"num_objects":5470,"num_object_clones":0,"num_object_copies":16410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5470,"num_write_kb":350080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":78,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.080712+0000","last_change":"2024-09-13T13:18:51.111528+0000","last_active":"2024-09-13T13:18:53.080712+0000","last_peered":"2024-09-13T13:18:53.080712+0000","last_clean":"2024-09-13T13:18:53.080712+0000","last_became_active":"2024-09-13T13:17:49.053160+0000","last_became_peered":"2024-09-13T13:17:49.053160+0000","last_unstale":"2024-09-13T13:18:53.080712+0000","last_undegraded":"2024-09-13T13:18:53.080712+0000","last_fullsized":"2024-09-13T13:18:53.080712+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.111449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.111449+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:00.573449+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"268'20","reported_seq":316,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067981+0000","last_change":"2024-09-13T13:17:02.421924+0000","last_active":"2024-09-13T13:18:53.067981+0000","last_peered":"2024-09-13T13:18:53.067981+0000","last_clean":"2024-09-13T13:18:53.067981+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:18:53.067981+0000","last_undegraded":"2024-09-13T13:18:53.067981+0000","last_fullsized":"2024-09-13T13:18:53.067981+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.421825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.421825+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":830,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.086533+0000","last_change":"2024-09-13T13:12:53.003530+0000","last_active":"2024-09-13T13:18:53.086533+0000","last_peered":"2024-09-13T13:18:53.086533+0000","last_clean":"2024-09-13T13:18:53.086533+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:18:53.086533+0000","last_undegraded":"2024-09-13T13:18:53.086533+0000","last_fullsized":"2024-09-13T13:18:53.086533+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.996346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.996346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:15.070878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"411'10","reported_seq":119,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.073690+0000","last_change":"2024-09-13T13:17:30.560167+0000","last_active":"2024-09-13T13:18:53.073690+0000","last_peered":"2024-09-13T13:18:53.073690+0000","last_clean":"2024-09-13T13:18:53.073690+0000","last_became_active":"2024-09-13T13:17:25.333196+0000","last_became_peered":"2024-09-13T13:17:25.333196+0000","last_unstale":"2024-09-13T13:18:53.073690+0000","last_undegraded":"2024-09-13T13:18:53.073690+0000","last_fullsized":"2024-09-13T13:18:53.073690+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'10","last_scrub_stamp":"2024-09-13T13:17:30.560099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:30.560099+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:55.601708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":326,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069049+0000","last_change":"2024-09-13T13:18:42.196223+0000","last_active":"2024-09-13T13:18:53.069049+0000","last_peered":"2024-09-13T13:18:53.069049+0000","last_clean":"2024-09-13T13:18:53.069049+0000","last_became_active":"2024-09-13T13:17:25.252583+0000","last_became_peered":"2024-09-13T13:17:25.252583+0000","last_unstale":"2024-09-13T13:18:53.069049+0000","last_undegraded":"2024-09-13T13:18:53.069049+0000","last_fullsized":"2024-09-13T13:18:53.069049+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:42.196120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:42.196120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:02.244520+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"447'5283","reported_seq":5342,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:56.380613+0000","last_change":"2024-09-13T13:17:55.200305+0000","last_active":"2024-09-13T13:18:56.380613+0000","last_peered":"2024-09-13T13:18:56.380613+0000","last_clean":"2024-09-13T13:18:56.380613+0000","last_became_active":"2024-09-13T13:17:55.200036+0000","last_became_peered":"2024-09-13T13:17:55.200036+0000","last_unstale":"2024-09-13T13:18:56.380613+0000","last_undegraded":"2024-09-13T13:18:56.380613+0000","last_fullsized":"2024-09-13T13:18:56.380613+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5283,"log_dups_size":0,"ondisk_log_size":5283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.345647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346226688,"num_objects":5283,"num_object_clones":0,"num_object_copies":15849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5283,"num_write_kb":338112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"268'15","reported_seq":862,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070951+0000","last_change":"2024-09-13T13:13:56.850149+0000","last_active":"2024-09-13T13:18:53.070951+0000","last_peered":"2024-09-13T13:18:53.070951+0000","last_clean":"2024-09-13T13:18:53.070951+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:18:53.070951+0000","last_undegraded":"2024-09-13T13:18:53.070951+0000","last_fullsized":"2024-09-13T13:18:53.070951+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:13:56.850094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:56.850094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.496112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":851,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069291+0000","last_change":"2024-09-13T13:12:58.857634+0000","last_active":"2024-09-13T13:18:53.069291+0000","last_peered":"2024-09-13T13:18:53.069291+0000","last_clean":"2024-09-13T13:18:53.069291+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:18:53.069291+0000","last_undegraded":"2024-09-13T13:18:53.069291+0000","last_fullsized":"2024-09-13T13:18:53.069291+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:12:58.857585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:12:58.857585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:24.527045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":96,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.073068+0000","last_change":"2024-09-13T13:17:54.620568+0000","last_active":"2024-09-13T13:18:53.073068+0000","last_peered":"2024-09-13T13:18:53.073068+0000","last_clean":"2024-09-13T13:18:53.073068+0000","last_became_active":"2024-09-13T13:17:25.238627+0000","last_became_peered":"2024-09-13T13:17:25.238627+0000","last_unstale":"2024-09-13T13:18:53.073068+0000","last_undegraded":"2024-09-13T13:18:53.073068+0000","last_fullsized":"2024-09-13T13:18:53.073068+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.620524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.620524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:08.712954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":834,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070711+0000","last_change":"2024-09-13T13:17:30.598783+0000","last_active":"2024-09-13T13:18:53.070711+0000","last_peered":"2024-09-13T13:18:53.070711+0000","last_clean":"2024-09-13T13:18:53.070711+0000","last_became_active":"2024-09-13T13:17:25.252822+0000","last_became_peered":"2024-09-13T13:17:25.252822+0000","last_unstale":"2024-09-13T13:18:53.070711+0000","last_undegraded":"2024-09-13T13:18:53.070711+0000","last_fullsized":"2024-09-13T13:18:53.070711+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:30.598736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:30.598736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"447'5258","reported_seq":5316,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.542192+0000","last_change":"2024-09-13T13:17:55.200461+0000","last_active":"2024-09-13T13:18:53.542192+0000","last_peered":"2024-09-13T13:18:53.542192+0000","last_clean":"2024-09-13T13:18:53.542192+0000","last_became_active":"2024-09-13T13:17:55.200186+0000","last_became_peered":"2024-09-13T13:17:55.200186+0000","last_unstale":"2024-09-13T13:18:53.542192+0000","last_undegraded":"2024-09-13T13:18:53.542192+0000","last_fullsized":"2024-09-13T13:18:53.542192+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5258,"log_dups_size":0,"ondisk_log_size":5258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.535572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344588288,"num_objects":5258,"num_object_clones":0,"num_object_copies":15774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5258,"num_write_kb":336512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"269'35","reported_seq":836,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.086138+0000","last_change":"2024-09-13T13:13:06.339762+0000","last_active":"2024-09-13T13:18:53.086138+0000","last_peered":"2024-09-13T13:18:53.086138+0000","last_clean":"2024-09-13T13:18:53.086138+0000","last_became_active":"2024-09-13T13:04:02.756772+0000","last_became_peered":"2024-09-13T13:04:02.756772+0000","last_unstale":"2024-09-13T13:18:53.086138+0000","last_undegraded":"2024-09-13T13:18:53.086138+0000","last_fullsized":"2024-09-13T13:18:53.086138+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:13:06.339664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:13:06.339664+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:26.319604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":466,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069313+0000","last_change":"2024-09-13T13:14:55.471954+0000","last_active":"2024-09-13T13:18:53.069313+0000","last_peered":"2024-09-13T13:18:53.069313+0000","last_clean":"2024-09-13T13:18:53.069313+0000","last_became_active":"2024-09-13T13:12:39.954315+0000","last_became_peered":"2024-09-13T13:12:39.954315+0000","last_unstale":"2024-09-13T13:18:53.069313+0000","last_undegraded":"2024-09-13T13:18:53.069313+0000","last_fullsized":"2024-09-13T13:18:53.069313+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:55.471885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:55.471885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":96,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.072685+0000","last_change":"2024-09-13T13:18:39.692048+0000","last_active":"2024-09-13T13:18:53.072685+0000","last_peered":"2024-09-13T13:18:53.072685+0000","last_clean":"2024-09-13T13:18:53.072685+0000","last_became_active":"2024-09-13T13:17:25.332964+0000","last_became_peered":"2024-09-13T13:17:25.332964+0000","last_unstale":"2024-09-13T13:18:53.072685+0000","last_undegraded":"2024-09-13T13:18:53.072685+0000","last_fullsized":"2024-09-13T13:18:53.072685+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:39.691978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:39.691978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:08.687543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":96,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.072560+0000","last_change":"2024-09-13T13:18:48.717531+0000","last_active":"2024-09-13T13:18:53.072560+0000","last_peered":"2024-09-13T13:18:53.072560+0000","last_clean":"2024-09-13T13:18:53.072560+0000","last_became_active":"2024-09-13T13:17:25.243114+0000","last_became_peered":"2024-09-13T13:17:25.243114+0000","last_unstale":"2024-09-13T13:18:53.072560+0000","last_undegraded":"2024-09-13T13:18:53.072560+0000","last_fullsized":"2024-09-13T13:18:53.072560+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:48.717456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:48.717456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:18.104517+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"447'5235","reported_seq":5294,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:54.726785+0000","last_change":"2024-09-13T13:17:55.140187+0000","last_active":"2024-09-13T13:18:54.726785+0000","last_peered":"2024-09-13T13:18:54.726785+0000","last_clean":"2024-09-13T13:18:54.726785+0000","last_became_active":"2024-09-13T13:17:55.139876+0000","last_became_peered":"2024-09-13T13:17:55.139876+0000","last_unstale":"2024-09-13T13:18:54.726785+0000","last_undegraded":"2024-09-13T13:18:54.726785+0000","last_fullsized":"2024-09-13T13:18:54.726785+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:02.705289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343080960,"num_objects":5235,"num_object_clones":0,"num_object_copies":15705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5235,"num_write_kb":335040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"267'18","reported_seq":324,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067223+0000","last_change":"2024-09-13T13:16:00.285064+0000","last_active":"2024-09-13T13:18:53.067223+0000","last_peered":"2024-09-13T13:18:53.067223+0000","last_clean":"2024-09-13T13:18:53.067223+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:18:53.067223+0000","last_undegraded":"2024-09-13T13:18:53.067223+0000","last_fullsized":"2024-09-13T13:18:53.067223+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:16:00.284984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:16:00.284984+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:52.129677+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":312,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068957+0000","last_change":"2024-09-13T13:14:52.383027+0000","last_active":"2024-09-13T13:18:53.068957+0000","last_peered":"2024-09-13T13:18:53.068957+0000","last_clean":"2024-09-13T13:18:53.068957+0000","last_became_active":"2024-09-13T13:12:53.007561+0000","last_became_peered":"2024-09-13T13:12:53.007561+0000","last_unstale":"2024-09-13T13:18:53.068957+0000","last_undegraded":"2024-09-13T13:18:53.068957+0000","last_fullsized":"2024-09-13T13:18:53.068957+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:52.382755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:52.382755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:15.521717+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":96,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.072473+0000","last_change":"2024-09-13T13:17:52.607482+0000","last_active":"2024-09-13T13:18:53.072473+0000","last_peered":"2024-09-13T13:18:53.072473+0000","last_clean":"2024-09-13T13:18:53.072473+0000","last_became_active":"2024-09-13T13:17:25.252535+0000","last_became_peered":"2024-09-13T13:17:25.252535+0000","last_unstale":"2024-09-13T13:18:53.072473+0000","last_undegraded":"2024-09-13T13:18:53.072473+0000","last_fullsized":"2024-09-13T13:18:53.072473+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:52.607404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.607404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:58.714721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":812,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.086042+0000","last_change":"2024-09-13T13:16:58.047398+0000","last_active":"2024-09-13T13:18:53.086042+0000","last_peered":"2024-09-13T13:18:53.086042+0000","last_clean":"2024-09-13T13:18:53.086042+0000","last_became_active":"2024-09-13T13:01:08.211154+0000","last_became_peered":"2024-09-13T13:01:08.211154+0000","last_unstale":"2024-09-13T13:18:53.086042+0000","last_undegraded":"2024-09-13T13:18:53.086042+0000","last_fullsized":"2024-09-13T13:18:53.086042+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:58.044139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:58.044139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"447'5081","reported_seq":5138,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.670115+0000","last_change":"2024-09-13T13:17:55.137957+0000","last_active":"2024-09-13T13:18:53.670115+0000","last_peered":"2024-09-13T13:18:53.670115+0000","last_clean":"2024-09-13T13:18:53.670115+0000","last_became_active":"2024-09-13T13:17:55.137667+0000","last_became_peered":"2024-09-13T13:17:55.137667+0000","last_unstale":"2024-09-13T13:18:53.670115+0000","last_undegraded":"2024-09-13T13:18:53.670115+0000","last_fullsized":"2024-09-13T13:18:53.670115+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5081,"log_dups_size":0,"ondisk_log_size":5081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:03.830885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332988416,"num_objects":5081,"num_object_clones":0,"num_object_copies":15243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5081,"num_write_kb":325184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"268'27","reported_seq":899,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068271+0000","last_change":"2024-09-13T13:15:56.631456+0000","last_active":"2024-09-13T13:18:53.068271+0000","last_peered":"2024-09-13T13:18:53.068271+0000","last_clean":"2024-09-13T13:18:53.068271+0000","last_became_active":"2024-09-13T13:04:02.753872+0000","last_became_peered":"2024-09-13T13:04:02.753872+0000","last_unstale":"2024-09-13T13:18:53.068271+0000","last_undegraded":"2024-09-13T13:18:53.068271+0000","last_fullsized":"2024-09-13T13:18:53.068271+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:15:56.631396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:15:56.631396+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:18.472573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":310,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069959+0000","last_change":"2024-09-13T13:13:01.972907+0000","last_active":"2024-09-13T13:18:53.069959+0000","last_peered":"2024-09-13T13:18:53.069959+0000","last_clean":"2024-09-13T13:18:53.069959+0000","last_became_active":"2024-09-13T13:12:53.006362+0000","last_became_peered":"2024-09-13T13:12:53.006362+0000","last_unstale":"2024-09-13T13:18:53.069959+0000","last_undegraded":"2024-09-13T13:18:53.069959+0000","last_fullsized":"2024-09-13T13:18:53.069959+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:13:01.972831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:01.972831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:17.528749+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"306'13","reported_seq":595,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085572+0000","last_change":"2024-09-13T13:18:41.356034+0000","last_active":"2024-09-13T13:18:53.085572+0000","last_peered":"2024-09-13T13:18:53.085572+0000","last_clean":"2024-09-13T13:18:53.085572+0000","last_became_active":"2024-09-13T13:12:53.001189+0000","last_became_peered":"2024-09-13T13:12:53.001189+0000","last_unstale":"2024-09-13T13:18:53.085572+0000","last_undegraded":"2024-09-13T13:18:53.085572+0000","last_fullsized":"2024-09-13T13:18:53.085572+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'13","last_scrub_stamp":"2024-09-13T13:18:41.355851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.355851+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:59.453980+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":832,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069191+0000","last_change":"2024-09-13T13:18:43.850216+0000","last_active":"2024-09-13T13:18:53.069191+0000","last_peered":"2024-09-13T13:18:53.069191+0000","last_clean":"2024-09-13T13:18:53.069191+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:18:53.069191+0000","last_undegraded":"2024-09-13T13:18:53.069191+0000","last_fullsized":"2024-09-13T13:18:53.069191+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:43.850163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:43.850163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:11.330472+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":801,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085081+0000","last_change":"2024-09-13T13:13:04.371082+0000","last_active":"2024-09-13T13:18:53.085081+0000","last_peered":"2024-09-13T13:18:53.085081+0000","last_clean":"2024-09-13T13:18:53.085081+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:18:53.085081+0000","last_undegraded":"2024-09-13T13:18:53.085081+0000","last_fullsized":"2024-09-13T13:18:53.085081+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:04.370881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:04.370881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:22.297512+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"268'12","reported_seq":354,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068907+0000","last_change":"2024-09-13T13:17:59.217373+0000","last_active":"2024-09-13T13:18:53.068907+0000","last_peered":"2024-09-13T13:18:53.068907+0000","last_clean":"2024-09-13T13:18:53.068907+0000","last_became_active":"2024-09-13T13:12:34.673528+0000","last_became_peered":"2024-09-13T13:12:34.673528+0000","last_unstale":"2024-09-13T13:18:53.068907+0000","last_undegraded":"2024-09-13T13:18:53.068907+0000","last_fullsized":"2024-09-13T13:18:53.068907+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:17:59.217309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.217309+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:28.238775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":856,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069584+0000","last_change":"2024-09-13T13:18:05.302935+0000","last_active":"2024-09-13T13:18:53.069584+0000","last_peered":"2024-09-13T13:18:53.069584+0000","last_clean":"2024-09-13T13:18:53.069584+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:18:53.069584+0000","last_undegraded":"2024-09-13T13:18:53.069584+0000","last_fullsized":"2024-09-13T13:18:53.069584+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:05.302882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:05.302882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:05.921840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":877,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.082313+0000","last_change":"2024-09-13T13:17:53.194243+0000","last_active":"2024-09-13T13:18:53.082313+0000","last_peered":"2024-09-13T13:18:53.082313+0000","last_clean":"2024-09-13T13:18:53.082313+0000","last_became_active":"2024-09-13T13:12:53.000873+0000","last_became_peered":"2024-09-13T13:12:53.000873+0000","last_unstale":"2024-09-13T13:18:53.082313+0000","last_undegraded":"2024-09-13T13:18:53.082313+0000","last_fullsized":"2024-09-13T13:18:53.082313+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:53.194191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:53.194191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:19.153631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"268'15","reported_seq":852,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.073907+0000","last_change":"2024-09-13T13:17:52.518556+0000","last_active":"2024-09-13T13:18:53.073907+0000","last_peered":"2024-09-13T13:18:53.073907+0000","last_clean":"2024-09-13T13:18:53.073907+0000","last_became_active":"2024-09-13T13:01:04.208545+0000","last_became_peered":"2024-09-13T13:01:04.208545+0000","last_unstale":"2024-09-13T13:18:53.073907+0000","last_undegraded":"2024-09-13T13:18:53.073907+0000","last_fullsized":"2024-09-13T13:18:53.073907+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:52.518502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.518502+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:12.668389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":304,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.067012+0000","last_change":"2024-09-13T13:13:51.764793+0000","last_active":"2024-09-13T13:18:53.067012+0000","last_peered":"2024-09-13T13:18:53.067012+0000","last_clean":"2024-09-13T13:18:53.067012+0000","last_became_active":"2024-09-13T13:12:53.006579+0000","last_became_peered":"2024-09-13T13:12:53.006579+0000","last_unstale":"2024-09-13T13:18:53.067012+0000","last_undegraded":"2024-09-13T13:18:53.067012+0000","last_fullsized":"2024-09-13T13:18:53.067012+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:13:51.764717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:13:51.764717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:24.315631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":832,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085507+0000","last_change":"2024-09-13T13:17:00.968640+0000","last_active":"2024-09-13T13:18:53.085507+0000","last_peered":"2024-09-13T13:18:53.085507+0000","last_clean":"2024-09-13T13:18:53.085507+0000","last_became_active":"2024-09-13T13:01:06.209968+0000","last_became_peered":"2024-09-13T13:01:06.209968+0000","last_unstale":"2024-09-13T13:18:53.085507+0000","last_undegraded":"2024-09-13T13:18:53.085507+0000","last_fullsized":"2024-09-13T13:18:53.085507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:00.968580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:00.968580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":96,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.074338+0000","last_change":"2024-09-13T13:18:45.736249+0000","last_active":"2024-09-13T13:18:53.074338+0000","last_peered":"2024-09-13T13:18:53.074338+0000","last_clean":"2024-09-13T13:18:53.074338+0000","last_became_active":"2024-09-13T13:17:25.244421+0000","last_became_peered":"2024-09-13T13:17:25.244421+0000","last_unstale":"2024-09-13T13:18:53.074338+0000","last_undegraded":"2024-09-13T13:18:53.074338+0000","last_fullsized":"2024-09-13T13:18:53.074338+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:45.736080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.736080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:56.355923+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"268'15","reported_seq":874,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069391+0000","last_change":"2024-09-13T13:17:04.222896+0000","last_active":"2024-09-13T13:18:53.069391+0000","last_peered":"2024-09-13T13:18:53.069391+0000","last_clean":"2024-09-13T13:18:53.069391+0000","last_became_active":"2024-09-13T13:16:12.069730+0000","last_became_peered":"2024-09-13T13:16:12.069730+0000","last_unstale":"2024-09-13T13:18:53.069391+0000","last_undegraded":"2024-09-13T13:18:53.069391+0000","last_fullsized":"2024-09-13T13:18:53.069391+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:04.222760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:04.222760+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":823,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.071732+0000","last_change":"2024-09-13T13:17:09.634101+0000","last_active":"2024-09-13T13:18:53.071732+0000","last_peered":"2024-09-13T13:18:53.071732+0000","last_clean":"2024-09-13T13:18:53.071732+0000","last_became_active":"2024-09-13T13:01:02.185122+0000","last_became_peered":"2024-09-13T13:01:02.185122+0000","last_unstale":"2024-09-13T13:18:53.071732+0000","last_undegraded":"2024-09-13T13:18:53.071732+0000","last_fullsized":"2024-09-13T13:18:53.071732+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.634027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.634027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:30.479119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"411'10","reported_seq":1062,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.084365+0000","last_change":"2024-09-13T13:17:27.165835+0000","last_active":"2024-09-13T13:18:53.084365+0000","last_peered":"2024-09-13T13:18:53.084365+0000","last_clean":"2024-09-13T13:18:53.084365+0000","last_became_active":"2024-09-13T13:17:25.245468+0000","last_became_peered":"2024-09-13T13:17:25.245468+0000","last_unstale":"2024-09-13T13:18:53.084365+0000","last_undegraded":"2024-09-13T13:18:53.084365+0000","last_fullsized":"2024-09-13T13:18:53.084365+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'9","last_scrub_stamp":"2024-09-13T13:17:27.165657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:27.165657+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":326,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.070021+0000","last_change":"2024-09-13T13:18:41.148239+0000","last_active":"2024-09-13T13:18:53.070021+0000","last_peered":"2024-09-13T13:18:53.070021+0000","last_clean":"2024-09-13T13:18:53.070021+0000","last_became_active":"2024-09-13T13:17:25.333587+0000","last_became_peered":"2024-09-13T13:17:25.333587+0000","last_unstale":"2024-09-13T13:18:53.070021+0000","last_undegraded":"2024-09-13T13:18:53.070021+0000","last_fullsized":"2024-09-13T13:18:53.070021+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:41.147987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.147987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:48.324808+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"268'20","reported_seq":874,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.084568+0000","last_change":"2024-09-13T13:17:02.978088+0000","last_active":"2024-09-13T13:18:53.084568+0000","last_peered":"2024-09-13T13:18:53.084568+0000","last_clean":"2024-09-13T13:18:53.084568+0000","last_became_active":"2024-09-13T13:01:04.208001+0000","last_became_peered":"2024-09-13T13:01:04.208001+0000","last_unstale":"2024-09-13T13:18:53.084568+0000","last_undegraded":"2024-09-13T13:18:53.084568+0000","last_fullsized":"2024-09-13T13:18:53.084568+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.978003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.978003+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":852,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.082334+0000","last_change":"2024-09-13T13:17:28.963214+0000","last_active":"2024-09-13T13:18:53.082334+0000","last_peered":"2024-09-13T13:18:53.082334+0000","last_clean":"2024-09-13T13:18:53.082334+0000","last_became_active":"2024-09-13T13:17:25.252500+0000","last_became_peered":"2024-09-13T13:17:25.252500+0000","last_unstale":"2024-09-13T13:18:53.082334+0000","last_undegraded":"2024-09-13T13:18:53.082334+0000","last_fullsized":"2024-09-13T13:18:53.082334+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:28.963152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:28.963152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":1063,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069092+0000","last_change":"2024-09-13T13:17:57.251817+0000","last_active":"2024-09-13T13:18:53.069092+0000","last_peered":"2024-09-13T13:18:53.069092+0000","last_clean":"2024-09-13T13:18:53.069092+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:18:53.069092+0000","last_undegraded":"2024-09-13T13:18:53.069092+0000","last_fullsized":"2024-09-13T13:18:53.069092+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:17:57.251734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:57.251734+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:03.487958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":835,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.071846+0000","last_change":"2024-09-13T13:17:33.554954+0000","last_active":"2024-09-13T13:18:53.071846+0000","last_peered":"2024-09-13T13:18:53.071846+0000","last_clean":"2024-09-13T13:18:53.071846+0000","last_became_active":"2024-09-13T13:01:08.214723+0000","last_became_peered":"2024-09-13T13:01:08.214723+0000","last_unstale":"2024-09-13T13:18:53.071846+0000","last_undegraded":"2024-09-13T13:18:53.071846+0000","last_fullsized":"2024-09-13T13:18:53.071846+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:33.554870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:33.554870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:03.321466+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"268'19","reported_seq":324,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068753+0000","last_change":"2024-09-13T13:18:45.145089+0000","last_active":"2024-09-13T13:18:53.068753+0000","last_peered":"2024-09-13T13:18:53.068753+0000","last_clean":"2024-09-13T13:18:53.068753+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:18:53.068753+0000","last_undegraded":"2024-09-13T13:18:53.068753+0000","last_fullsized":"2024-09-13T13:18:53.068753+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:45.145010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.145010+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:04.145291+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":304,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.068059+0000","last_change":"2024-09-13T13:17:09.431539+0000","last_active":"2024-09-13T13:18:53.068059+0000","last_peered":"2024-09-13T13:18:53.068059+0000","last_clean":"2024-09-13T13:18:53.068059+0000","last_became_active":"2024-09-13T13:12:53.005988+0000","last_became_peered":"2024-09-13T13:12:53.005988+0000","last_unstale":"2024-09-13T13:18:53.068059+0000","last_undegraded":"2024-09-13T13:18:53.068059+0000","last_fullsized":"2024-09-13T13:18:53.068059+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.431472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.431472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":841,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.078938+0000","last_change":"2024-09-13T13:17:33.279005+0000","last_active":"2024-09-13T13:18:53.078938+0000","last_peered":"2024-09-13T13:18:53.078938+0000","last_clean":"2024-09-13T13:18:53.078938+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:18:53.078938+0000","last_undegraded":"2024-09-13T13:18:53.078938+0000","last_fullsized":"2024-09-13T13:18:53.078938+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:33.278909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:33.278909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:18:55.488227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":836,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.086272+0000","last_change":"2024-09-13T13:17:09.989898+0000","last_active":"2024-09-13T13:18:53.086272+0000","last_peered":"2024-09-13T13:18:53.086272+0000","last_clean":"2024-09-13T13:18:53.086272+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:18:53.086272+0000","last_undegraded":"2024-09-13T13:18:53.086272+0000","last_fullsized":"2024-09-13T13:18:53.086272+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.989820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.989820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"268'15","reported_seq":831,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069090+0000","last_change":"2024-09-13T13:17:26.125163+0000","last_active":"2024-09-13T13:18:53.069090+0000","last_peered":"2024-09-13T13:18:53.069090+0000","last_clean":"2024-09-13T13:18:53.069090+0000","last_became_active":"2024-09-13T13:17:25.331530+0000","last_became_peered":"2024-09-13T13:17:25.331530+0000","last_unstale":"2024-09-13T13:18:53.069090+0000","last_undegraded":"2024-09-13T13:18:53.069090+0000","last_fullsized":"2024-09-13T13:18:53.069090+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:26.125088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:26.125088+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":824,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085377+0000","last_change":"2024-09-13T13:17:56.965925+0000","last_active":"2024-09-13T13:18:53.085377+0000","last_peered":"2024-09-13T13:18:53.085377+0000","last_clean":"2024-09-13T13:18:53.085377+0000","last_became_active":"2024-09-13T13:12:52.949094+0000","last_became_peered":"2024-09-13T13:12:52.949094+0000","last_unstale":"2024-09-13T13:18:53.085377+0000","last_undegraded":"2024-09-13T13:18:53.085377+0000","last_fullsized":"2024-09-13T13:18:53.085377+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:56.965870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:56.965870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:02.221904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":832,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.086139+0000","last_change":"2024-09-13T13:18:03.037546+0000","last_active":"2024-09-13T13:18:53.086139+0000","last_peered":"2024-09-13T13:18:53.086139+0000","last_clean":"2024-09-13T13:18:53.086139+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:18:53.086139+0000","last_undegraded":"2024-09-13T13:18:53.086139+0000","last_fullsized":"2024-09-13T13:18:53.086139+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:03.037472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:03.037472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.016351+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":819,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069618+0000","last_change":"2024-09-13T13:18:44.505493+0000","last_active":"2024-09-13T13:18:53.069618+0000","last_peered":"2024-09-13T13:18:53.069618+0000","last_clean":"2024-09-13T13:18:53.069618+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:18:53.069618+0000","last_undegraded":"2024-09-13T13:18:53.069618+0000","last_fullsized":"2024-09-13T13:18:53.069618+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:44.505427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:44.505427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:54.043312+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"268'19","reported_seq":862,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.086447+0000","last_change":"2024-09-13T13:18:33.848957+0000","last_active":"2024-09-13T13:18:53.086447+0000","last_peered":"2024-09-13T13:18:53.086447+0000","last_clean":"2024-09-13T13:18:53.086447+0000","last_became_active":"2024-09-13T13:04:02.755308+0000","last_became_peered":"2024-09-13T13:04:02.755308+0000","last_unstale":"2024-09-13T13:18:53.086447+0000","last_undegraded":"2024-09-13T13:18:53.086447+0000","last_fullsized":"2024-09-13T13:18:53.086447+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:33.848726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.848726+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:59.409198+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":840,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.069657+0000","last_change":"2024-09-13T13:14:50.392659+0000","last_active":"2024-09-13T13:18:53.069657+0000","last_peered":"2024-09-13T13:18:53.069657+0000","last_clean":"2024-09-13T13:18:53.069657+0000","last_became_active":"2024-09-13T13:01:02.185099+0000","last_became_peered":"2024-09-13T13:01:02.185099+0000","last_unstale":"2024-09-13T13:18:53.069657+0000","last_undegraded":"2024-09-13T13:18:53.069657+0000","last_fullsized":"2024-09-13T13:18:53.069657+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:14:50.392444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:14:50.392444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":849,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.081631+0000","last_change":"2024-09-13T13:15:03.778214+0000","last_active":"2024-09-13T13:18:53.081631+0000","last_peered":"2024-09-13T13:18:53.081631+0000","last_clean":"2024-09-13T13:18:53.081631+0000","last_became_active":"2024-09-13T13:12:52.999748+0000","last_became_peered":"2024-09-13T13:12:52.999748+0000","last_unstale":"2024-09-13T13:18:53.081631+0000","last_undegraded":"2024-09-13T13:18:53.081631+0000","last_fullsized":"2024-09-13T13:18:53.081631+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:15:03.778170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:15:03.778170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:24.797615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":807,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.086393+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:18:53.086393+0000","last_peered":"2024-09-13T13:18:53.086393+0000","last_clean":"2024-09-13T13:18:53.086393+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:18:53.086393+0000","last_undegraded":"2024-09-13T13:18:53.086393+0000","last_fullsized":"2024-09-13T13:18:53.086393+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:23.884576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"267'8","reported_seq":877,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.094790+0000","last_change":"2024-09-13T13:18:46.033316+0000","last_active":"2024-09-13T13:18:53.094790+0000","last_peered":"2024-09-13T13:18:53.094790+0000","last_clean":"2024-09-13T13:18:53.094790+0000","last_became_active":"2024-09-13T13:16:12.073702+0000","last_became_peered":"2024-09-13T13:16:12.073702+0000","last_unstale":"2024-09-13T13:18:53.094790+0000","last_undegraded":"2024-09-13T13:18:53.094790+0000","last_fullsized":"2024-09-13T13:18:53.094790+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:18:46.033238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.033238+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:20:01.052128+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":855,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.073445+0000","last_change":"2024-09-13T13:17:57.579827+0000","last_active":"2024-09-13T13:18:53.073445+0000","last_peered":"2024-09-13T13:18:53.073445+0000","last_clean":"2024-09-13T13:18:53.073445+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:18:53.073445+0000","last_undegraded":"2024-09-13T13:18:53.073445+0000","last_fullsized":"2024-09-13T13:18:53.073445+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:57.579778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:57.579778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:11.821230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":763,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085670+0000","last_change":"2024-09-13T13:17:55.331435+0000","last_active":"2024-09-13T13:18:53.085670+0000","last_peered":"2024-09-13T13:18:53.085670+0000","last_clean":"2024-09-13T13:18:53.085670+0000","last_became_active":"2024-09-13T13:12:53.001131+0000","last_became_peered":"2024-09-13T13:12:53.001131+0000","last_unstale":"2024-09-13T13:18:53.085670+0000","last_undegraded":"2024-09-13T13:18:53.085670+0000","last_fullsized":"2024-09-13T13:18:53.085670+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:55.331393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:55.331393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:14.150805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":96,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.072974+0000","last_change":"2024-09-13T13:18:07.682986+0000","last_active":"2024-09-13T13:18:53.072974+0000","last_peered":"2024-09-13T13:18:53.072974+0000","last_clean":"2024-09-13T13:18:53.072974+0000","last_became_active":"2024-09-13T13:17:25.242337+0000","last_became_peered":"2024-09-13T13:17:25.242337+0000","last_unstale":"2024-09-13T13:18:53.072974+0000","last_undegraded":"2024-09-13T13:18:53.072974+0000","last_fullsized":"2024-09-13T13:18:53.072974+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:07.682648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:07.682648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:11.815648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":843,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.078139+0000","last_change":"2024-09-13T13:17:52.052009+0000","last_active":"2024-09-13T13:18:53.078139+0000","last_peered":"2024-09-13T13:18:53.078139+0000","last_clean":"2024-09-13T13:18:53.078139+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:18:53.078139+0000","last_undegraded":"2024-09-13T13:18:53.078139+0000","last_fullsized":"2024-09-13T13:18:53.078139+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:52.051961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.051961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:18.556468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":824,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.095028+0000","last_change":"2024-09-13T13:16:36.980668+0000","last_active":"2024-09-13T13:18:53.095028+0000","last_peered":"2024-09-13T13:18:53.095028+0000","last_clean":"2024-09-13T13:18:53.095028+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:18:53.095028+0000","last_undegraded":"2024-09-13T13:18:53.095028+0000","last_fullsized":"2024-09-13T13:18:53.095028+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:36.980594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:36.980594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:13.050858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"268'19","reported_seq":843,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.085541+0000","last_change":"2024-09-13T13:16:12.087075+0000","last_active":"2024-09-13T13:18:53.085541+0000","last_peered":"2024-09-13T13:18:53.085541+0000","last_clean":"2024-09-13T13:18:53.085541+0000","last_became_active":"2024-09-13T13:16:12.070277+0000","last_became_peered":"2024-09-13T13:16:12.070277+0000","last_unstale":"2024-09-13T13:18:53.085541+0000","last_undegraded":"2024-09-13T13:18:53.085541+0000","last_fullsized":"2024-09-13T13:18:53.085541+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:11:49.264128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:11:49.264128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:15.253268+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":862,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-13T13:18:53.071989+0000","last_change":"2024-09-13T13:17:25.253423+0000","last_active":"2024-09-13T13:18:53.071989+0000","last_peered":"2024-09-13T13:18:53.071989+0000","last_clean":"2024-09-13T13:18:53.071989+0000","last_became_active":"2024-09-13T13:17:25.252637+0000","last_became_peered":"2024-09-13T13:17:25.252637+0000","last_unstale":"2024-09-13T13:18:53.071989+0000","last_undegraded":"2024-09-13T13:18:53.071989+0000","last_fullsized":"2024-09-13T13:18:53.071989+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:23.547758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:23.547758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:19:29.521458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5606539264,"num_objects":85549,"num_object_clones":0,"num_object_copies":256647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85549,"num_write_kb":5475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16712204288,"data_stored":16712204288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79649,"ondisk_log_size":79649,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":4792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":686,"ondisk_log_size":686,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":380,"num_read_kb":375,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201497,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2602124,"kb_used_data":1997564,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91143028,"statfs":{"total":95995035648,"available":93330460672,"internally_reserved":0,"allocated":2045505536,"data_stored":2044978337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":619051466},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141108,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3356808,"kb_used_data":2755704,"kb_used_omap":4,"kb_used_meta":601083,"kb_avail":90388344,"statfs":{"total":95995035648,"available":92557664256,"internally_reserved":0,"allocated":2821840896,"data_stored":2821323099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":615509677},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692069,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1965900,"kb_used_data":1321080,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":91779252,"statfs":{"total":95995035648,"available":93981954048,"internally_reserved":0,"allocated":1352785920,"data_stored":1352262201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921003,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1934760,"kb_used_data":1307988,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":91810392,"statfs":{"total":95995035648,"available":94013841408,"internally_reserved":0,"allocated":1339379712,"data_stored":1338833848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":641792454},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117235,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3222060,"kb_used_data":2464796,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":90523092,"statfs":{"total":95995035648,"available":92695646208,"internally_reserved":0,"allocated":2523951104,"data_stored":2523398422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":775420359},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411574,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2656264,"kb_used_data":2089208,"kb_used_omap":1,"kb_used_meta":567038,"kb_avail":91088888,"statfs":{"total":95995035648,"available":93275021312,"internally_reserved":0,"allocated":2139348992,"data_stored":2138831658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":580647366},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575100,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3311852,"kb_used_data":2680916,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":90433300,"statfs":{"total":95995035648,"available":92603699200,"internally_reserved":0,"allocated":2745257984,"data_stored":2744712163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738627,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2458864,"kb_used_data":1665500,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":91286288,"statfs":{"total":95995035648,"available":93477158912,"internally_reserved":0,"allocated":1705472000,"data_stored":1704919035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1709309952,"data_stored":1709309952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2748579840,"data_stored":2748579840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2146762752,"data_stored":2146762752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2522480640,"data_stored":2522480640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1342242816,"data_stored":1342242816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1364787200,"data_stored":1364787200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2821324800,"data_stored":2821324800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2056716288,"data_stored":2056716288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:19:00.083 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T13:19:00.083 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_4 pg_num 26 2024-09-13T13:19:00.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:00.116 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:00.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:00.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:00.251 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:00.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:00.353 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:00.407 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:00.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:00.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:00 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:19:00.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:00 smithi052 ceph-mon[29471]: from='client.36740 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:19:00.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:00 smithi052 ceph-mon[29471]: osdmap e448: 8 total, 8 up, 8 in 2024-09-13T13:19:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:00 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:19:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:00 smithi052 ceph-mon[36682]: from='client.36740 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:19:00.438 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:00 smithi052 ceph-mon[36682]: osdmap e448: 8 total, 8 up, 8 in 2024-09-13T13:19:00.508 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:00.555 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:00.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:00.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:00 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:19:00.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:00 smithi167 ceph-mon[33178]: from='client.36740 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:19:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:00 smithi167 ceph-mon[33178]: osdmap e448: 8 total, 8 up, 8 in 2024-09-13T13:19:00.657 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:00.694 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:00.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:00.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:00.796 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:00.827 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:00.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:00.887 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:19:00.887316+0000 min lat: 0.00125067 max lat: 0.321739 avg lat: 0.0105528 2024-09-13T13:19:00.887 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:19:00.887 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 60 16 90958 90942 94.7065 70.8125 0.0208409 0.0105528 2024-09-13T13:19:00.929 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:00.939 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:00.944 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:01.017 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:01.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:01.119 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:01.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:01.149 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:01.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:01.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:01.275 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:01.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:01.280 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:01.377 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:01.385 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:01 smithi052 ceph-mon[36682]: pgmap v903: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-13T13:19:01.385 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:01 smithi052 ceph-mon[29471]: pgmap v903: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-13T13:19:01.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:01.413 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:01.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:01.514 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:01.562 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:01.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:01.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:01.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:01 smithi167 ceph-mon[33178]: pgmap v903: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-13T13:19:01.665 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:01.707 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:01.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:01.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:01.715 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:19:01.716 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:19:01.765 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:19:01.765 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:19:01.809 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:01.810 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:19:01.811 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:19:01.854 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:19:01.854 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:19:01.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:01.855 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:01.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:01.886 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:19:01.886 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:19:01.887 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 61 16 91978 91962 94.1989 63.75 0.0205682 0.0106111 2024-09-13T13:19:01.922 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:19:01.922 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:19:01.956 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:01.958 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:19:01.958 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:19:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:01.998 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:01.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:01.999 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:19:01.999 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:19:02.100 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:02.139 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:02.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:02.241 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:02.277 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:02.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:02.379 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:02.386 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:02 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1325181323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:02.386 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:02 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1325181323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:02.405 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:02.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:02.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:02.507 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:02.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:02.536 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:02.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:02.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:02 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1325181323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:02.637 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:02.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:02.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:02.767 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:02.813 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:02.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:02.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:02.887 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 62 15 93093 93078 93.8046 69.75 0.0185155 0.0106557 2024-09-13T13:19:02.916 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:02.943 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:02.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:02.945 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:19:03.045 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:03.109 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:03.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:03.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:03.150 INFO:teuthology.orchestra.run.smithi052.stderr:set pool 11 pg_num to 26 2024-09-13T13:19:03.211 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:03.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:03 smithi052 ceph-mon[36682]: pgmap v904: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 964 op/s 2024-09-13T13:19:03.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:03 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3870513045' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T13:19:03.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:03 smithi052 ceph-mon[36682]: from='client.36758 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T13:19:03.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:03 smithi052 ceph-mon[29471]: pgmap v904: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 964 op/s 2024-09-13T13:19:03.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:03 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3870513045' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T13:19:03.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:03 smithi052 ceph-mon[29471]: from='client.36758 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T13:19:03.272 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:03.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.36635.0:93483 11.1 e449/423 11:84c66e3b:::benchmark_data_smithi052_182575_object93482:head v 449'5757, mlcod=449'5752)", 2024-09-13T13:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:03.304617+0000", 2024-09-13T13:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0099695259999999994, 2024-09-13T13:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0099972359999999996, 2024-09-13T13:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.304617+0000", 2024-09-13T13:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:03.322 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.304617+0000", 2024-09-13T13:19:03.322 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:03.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.322 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.322 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:03.322 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.304619+0000", 2024-09-13T13:19:03.322 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4980000000000001e-06 2024-09-13T13:19:03.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.322 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.322 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:03.322 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.304659+0000", 2024-09-13T13:19:03.322 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0649999999999999e-05 2024-09-13T13:19:03.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.323 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:03.323 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.304671+0000", 2024-09-13T13:19:03.323 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1873e-05 2024-09-13T13:19:03.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.323 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:03.323 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.304679+0000", 2024-09-13T13:19:03.323 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3769999999999998e-06 2024-09-13T13:19:03.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.323 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:03.323 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.306755+0000", 2024-09-13T13:19:03.324 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020760349999999999 2024-09-13T13:19:03.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.324 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:03.324 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.306774+0000", 2024-09-13T13:19:03.324 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8740999999999999e-05 2024-09-13T13:19:03.324 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:03.324 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:03.324 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:03.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.325 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.325 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.36635.0:93491 11.1 e449/423 11:816105b0:::benchmark_data_smithi052_182575_object93490:head v 449'5758, mlcod=449'5755)", 2024-09-13T13:19:03.325 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:03.309912+0000", 2024-09-13T13:19:03.325 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0046744839999999996, 2024-09-13T13:19:03.325 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047651439999999998, 2024-09-13T13:19:03.325 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:03.325 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:03.325 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:19:03.325 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:03.325 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.325 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:03.326 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.309912+0000", 2024-09-13T13:19:03.326 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:03.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.326 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:03.326 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.309912+0000", 2024-09-13T13:19:03.326 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:03.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.326 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:03.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.309914+0000", 2024-09-13T13:19:03.327 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.482e-06 2024-09-13T13:19:03.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.327 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:03.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.309954+0000", 2024-09-13T13:19:03.327 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0599999999999998e-05 2024-09-13T13:19:03.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.327 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:03.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.309970+0000", 2024-09-13T13:19:03.328 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5098e-05 2024-09-13T13:19:03.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.328 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:03.328 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.309977+0000", 2024-09-13T13:19:03.328 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3610000000000003e-06 2024-09-13T13:19:03.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.328 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:03.328 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.312442+0000", 2024-09-13T13:19:03.329 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0024648000000000001 2024-09-13T13:19:03.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.329 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:03.329 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.312461+0000", 2024-09-13T13:19:03.329 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9001999999999999e-05 2024-09-13T13:19:03.329 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:03.329 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:03.329 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:03.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.330 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.36635.0:93493 11.1 e449/423)", 2024-09-13T13:19:03.330 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:03.313989+0000", 2024-09-13T13:19:03.330 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.00059820200000000004, 2024-09-13T13:19:03.330 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00073871199999999998, 2024-09-13T13:19:03.330 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:03.330 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:03.330 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:19:03.330 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.313989+0000", 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.313989+0000", 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.313990+0000", 2024-09-13T13:19:03.331 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5600000000000001e-06 2024-09-13T13:19:03.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.332 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.332 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:03.332 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.314034+0000", 2024-09-13T13:19:03.332 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4147999999999998e-05 2024-09-13T13:19:03.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.332 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.332 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:03.332 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.314050+0000", 2024-09-13T13:19:03.332 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5234e-05 2024-09-13T13:19:03.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:03.332 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:03.332 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:03.333 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:03.314058+0000", 2024-09-13T13:19:03.333 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4169999999999992e-06 2024-09-13T13:19:03.333 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:03.333 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:03.333 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:03.333 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:03.333 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:19:03.333 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 3 2024-09-13T13:19:03.333 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:19:03.375 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:03.417 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:03.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:03.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:03.546 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:03.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:03.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:03 smithi167 ceph-mon[33178]: pgmap v904: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 964 op/s 2024-09-13T13:19:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:03 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3870513045' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T13:19:03.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:03 smithi167 ceph-mon[33178]: from='client.36758 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T13:19:03.648 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:03.688 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:03.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:03.790 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:03.831 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:03.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:03.887 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 63 16 94180 94164 93.393 67.875 0.00820313 0.0107023 2024-09-13T13:19:03.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:03.963 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:03.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:04.065 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:04.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:04.096 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:04.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:04.197 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:04.246 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:04.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:04.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:04.390 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:04.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:04 smithi052 ceph-mon[36682]: from='client.36758 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-13T13:19:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:04 smithi052 ceph-mon[36682]: osdmap e449: 8 total, 8 up, 8 in 2024-09-13T13:19:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:04 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T13:19:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:04 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T13:19:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:04 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:19:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:04 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:19:04.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:04 smithi052 ceph-mon[29471]: from='client.36758 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-13T13:19:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:04 smithi052 ceph-mon[29471]: osdmap e449: 8 total, 8 up, 8 in 2024-09-13T13:19:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:04 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T13:19:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:04 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T13:19:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:04 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:19:04.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:04 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:19:04.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:04.512 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:04.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:04.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:04.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:04 smithi167 ceph-mon[33178]: from='client.36758 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-13T13:19:04.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:04 smithi167 ceph-mon[33178]: osdmap e449: 8 total, 8 up, 8 in 2024-09-13T13:19:04.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:04 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T13:19:04.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:04 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T13:19:04.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:04 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:19:04.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:04 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:19:04.614 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:04.637 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:04.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:04.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:04.739 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:04.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:04.762 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:04.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:04.863 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:04.885 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:04.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:04.887 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 64 16 94547 94531 92.2922 22.9375 0.214448 0.0107149 2024-09-13T13:19:04.986 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:05.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:05.008 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:05.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:05.057 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:19:05.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:19:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:19:05.110 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:05.135 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:05.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:05.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:05.237 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:05.285 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:05.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:05.387 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:05.427 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:05.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:05.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:05 smithi052 ceph-mon[36682]: pgmap v906: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-13T13:19:05.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:05 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T13:19:05.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:05 smithi052 ceph-mon[36682]: osdmap e450: 8 total, 8 up, 8 in 2024-09-13T13:19:05.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:05 smithi052 ceph-mon[29471]: pgmap v906: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-13T13:19:05.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:05 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T13:19:05.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:05 smithi052 ceph-mon[29471]: osdmap e450: 8 total, 8 up, 8 in 2024-09-13T13:19:05.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:05.529 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:05.558 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:05.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:05.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:05 smithi167 ceph-mon[33178]: pgmap v906: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-13T13:19:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:05 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T13:19:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:05 smithi167 ceph-mon[33178]: osdmap e450: 8 total, 8 up, 8 in 2024-09-13T13:19:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:05.659 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:05.681 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:05.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:05.783 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:05.820 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:05.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:05.888 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 65 15 95193 95178 91.4945 40.4375 0.00887756 0.0109254 2024-09-13T13:19:05.922 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:05.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:05.963 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:05.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:06.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:06.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:06.088 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:06.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:06.189 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:06.220 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:06.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:06.321 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:06.342 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:06.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:06.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:06.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:06.448 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:06 smithi167 ceph-mon[33178]: osdmap e451: 8 total, 8 up, 8 in 2024-09-13T13:19:06.448 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:06 smithi167 ceph-mon[33178]: 11.6 scrub starts 2024-09-13T13:19:06.448 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:06 smithi167 ceph-mon[33178]: 11.1 scrub starts 2024-09-13T13:19:06.448 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:06 smithi167 ceph-mon[33178]: 11.7 scrub starts 2024-09-13T13:19:06.449 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:06 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T13:19:06.449 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:06 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T13:19:06.466 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:06.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:06.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:06.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:19:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:19:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:19:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:06 smithi052 ceph-mon[36682]: osdmap e451: 8 total, 8 up, 8 in 2024-09-13T13:19:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:06 smithi052 ceph-mon[36682]: 11.6 scrub starts 2024-09-13T13:19:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:06 smithi052 ceph-mon[36682]: 11.1 scrub starts 2024-09-13T13:19:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:06 smithi052 ceph-mon[36682]: 11.7 scrub starts 2024-09-13T13:19:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:06 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T13:19:06.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:06 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T13:19:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:06 smithi052 ceph-mon[29471]: osdmap e451: 8 total, 8 up, 8 in 2024-09-13T13:19:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:06 smithi052 ceph-mon[29471]: 11.6 scrub starts 2024-09-13T13:19:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:06 smithi052 ceph-mon[29471]: 11.1 scrub starts 2024-09-13T13:19:06.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:06 smithi052 ceph-mon[29471]: 11.7 scrub starts 2024-09-13T13:19:06.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:06 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T13:19:06.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:06 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T13:19:06.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:06.591 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:06.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:06.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:06.692 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:06.715 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:06.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:06.818 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:06.839 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:06.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:06.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:06.888 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 66 16 95629 95613 90.5202 27.1875 0.00264101 0.010928 2024-09-13T13:19:06.941 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:06.962 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:06.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:06.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:07.064 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:07.091 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:07.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:07.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:07.193 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:07.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:07.215 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:07.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:07.316 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:07.337 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:07.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:07.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:07.439 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:07.466 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:07.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[36682]: 11.2 scrub starts 2024-09-13T13:19:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[36682]: pgmap v909: 168 pgs: 2 peering, 10 unknown, 156 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 24 MiB/s wr, 384 op/s 2024-09-13T13:19:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[36682]: 11.9 scrub starts 2024-09-13T13:19:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[36682]: 11.0 scrub starts 2024-09-13T13:19:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T13:19:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[36682]: osdmap e452: 8 total, 8 up, 8 in 2024-09-13T13:19:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[36682]: 11.1 scrub starts 2024-09-13T13:19:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[36682]: 11.6 scrub starts 2024-09-13T13:19:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[36682]: 11.5 scrub starts 2024-09-13T13:19:07.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[36682]: osdmap e453: 8 total, 8 up, 8 in 2024-09-13T13:19:07.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[29471]: 11.2 scrub starts 2024-09-13T13:19:07.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[29471]: pgmap v909: 168 pgs: 2 peering, 10 unknown, 156 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 24 MiB/s wr, 384 op/s 2024-09-13T13:19:07.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[29471]: 11.9 scrub starts 2024-09-13T13:19:07.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[29471]: 11.0 scrub starts 2024-09-13T13:19:07.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T13:19:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[29471]: osdmap e452: 8 total, 8 up, 8 in 2024-09-13T13:19:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[29471]: 11.1 scrub starts 2024-09-13T13:19:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[29471]: 11.6 scrub starts 2024-09-13T13:19:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[29471]: 11.5 scrub starts 2024-09-13T13:19:07.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:07 smithi052 ceph-mon[29471]: osdmap e453: 8 total, 8 up, 8 in 2024-09-13T13:19:07.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:07.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:07 smithi167 ceph-mon[33178]: 11.2 scrub starts 2024-09-13T13:19:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:07 smithi167 ceph-mon[33178]: pgmap v909: 168 pgs: 2 peering, 10 unknown, 156 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 24 MiB/s wr, 384 op/s 2024-09-13T13:19:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:07 smithi167 ceph-mon[33178]: 11.9 scrub starts 2024-09-13T13:19:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:07 smithi167 ceph-mon[33178]: 11.0 scrub starts 2024-09-13T13:19:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:07 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T13:19:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:07 smithi167 ceph-mon[33178]: osdmap e452: 8 total, 8 up, 8 in 2024-09-13T13:19:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:07 smithi167 ceph-mon[33178]: 11.1 scrub starts 2024-09-13T13:19:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:07 smithi167 ceph-mon[33178]: 11.6 scrub starts 2024-09-13T13:19:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:07 smithi167 ceph-mon[33178]: 11.5 scrub starts 2024-09-13T13:19:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:07 smithi167 ceph-mon[33178]: osdmap e453: 8 total, 8 up, 8 in 2024-09-13T13:19:07.593 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:07.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:07.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:07.695 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:07.722 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:07.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:07.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:07.846 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:07.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:07.888 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 67 16 95629 95613 89.1693 0 - 0.010928 2024-09-13T13:19:07.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:07.969 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:07.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:08.071 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:08.098 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:08.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:08.200 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:08.270 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:08.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:08.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:08.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:08.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:08 smithi052 ceph-mon[36682]: 11.2 scrub starts 2024-09-13T13:19:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:08 smithi052 ceph-mon[36682]: 11.8 scrub starts 2024-09-13T13:19:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:08 smithi052 ceph-mon[36682]: 11.0 scrub starts 2024-09-13T13:19:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:08 smithi052 ceph-mon[36682]: 11.4 scrub starts 2024-09-13T13:19:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:08 smithi052 ceph-mon[36682]: 11.7 scrub starts 2024-09-13T13:19:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:08 smithi052 ceph-mon[36682]: osdmap e454: 8 total, 8 up, 8 in 2024-09-13T13:19:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:08 smithi052 ceph-mon[29471]: 11.2 scrub starts 2024-09-13T13:19:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:08 smithi052 ceph-mon[29471]: 11.8 scrub starts 2024-09-13T13:19:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:08 smithi052 ceph-mon[29471]: 11.0 scrub starts 2024-09-13T13:19:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:08 smithi052 ceph-mon[29471]: 11.4 scrub starts 2024-09-13T13:19:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:08 smithi052 ceph-mon[29471]: 11.7 scrub starts 2024-09-13T13:19:08.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:08 smithi052 ceph-mon[29471]: osdmap e454: 8 total, 8 up, 8 in 2024-09-13T13:19:08.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:08.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:08.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:08 smithi167 ceph-mon[33178]: 11.2 scrub starts 2024-09-13T13:19:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:08 smithi167 ceph-mon[33178]: 11.8 scrub starts 2024-09-13T13:19:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:08 smithi167 ceph-mon[33178]: 11.0 scrub starts 2024-09-13T13:19:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:08 smithi167 ceph-mon[33178]: 11.4 scrub starts 2024-09-13T13:19:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:08 smithi167 ceph-mon[33178]: 11.7 scrub starts 2024-09-13T13:19:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:08 smithi167 ceph-mon[33178]: osdmap e454: 8 total, 8 up, 8 in 2024-09-13T13:19:08.633 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:08.667 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:08.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:08.769 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:08.800 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:08.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:08.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:08.888 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 68 15 96346 96331 88.5179 22.4375 0.0161069 0.0112918 2024-09-13T13:19:08.902 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:08.937 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:08.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:08.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:09.039 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:09.080 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:09.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:09.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:09.209 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:09.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:09.235 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:19:09.235 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:19:09.235 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T13:19:09.235 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.599680 2024-09-13T13:19:09.236 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 0.5996798312787667 2024-09-13T13:19:09.311 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:09.339 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:09.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:09.359 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T13:19:09.439 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:09.464 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:09.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:09.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:09.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.9 scrub starts 2024-09-13T13:19:09.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: pgmap v912: 168 pgs: 1 active+clean+scrubbing, 2 peering, 2 remapped+peering, 8 unknown, 155 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 16 KiB/s, 0 objects/s recovering 2024-09-13T13:19:09.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 451'6153 2024-09-13T13:19:09.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 451'5996 2024-09-13T13:19:09.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 451'6059 2024-09-13T13:19:09.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 451'5844 2024-09-13T13:19:09.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 451'5922 2024-09-13T13:19:09.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.14 starting backfill to osd.1 from (0'0,0'0] MAX to 451'5847 2024-09-13T13:19:09.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 451'5827 2024-09-13T13:19:09.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 451'6029 2024-09-13T13:19:09.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 451'6153 2024-09-13T13:19:09.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 451'5996 2024-09-13T13:19:09.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 451'5920 2024-09-13T13:19:09.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.14 starting backfill to osd.4 from (0'0,0'0] MAX to 451'5847 2024-09-13T13:19:09.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.17 starting backfill to osd.7 from (0'0,0'0] MAX to 451'5827 2024-09-13T13:19:09.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.15 starting backfill to osd.4 from (0'0,0'0] MAX to 451'5996 2024-09-13T13:19:09.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.16 starting backfill to osd.5 from (0'0,0'0] MAX to 451'5920 2024-09-13T13:19:09.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.13 starting backfill to osd.4 from (0'0,0'0] MAX to 451'6059 2024-09-13T13:19:09.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.10 starting backfill to osd.6 from (0'0,0'0] MAX to 451'6153 2024-09-13T13:19:09.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.18 starting backfill to osd.4 from (0'0,0'0] MAX to 451'6029 2024-09-13T13:19:09.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 451'5922 2024-09-13T13:19:09.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 451'5920 2024-09-13T13:19:09.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 451'5844 2024-09-13T13:19:09.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[29471]: 11.3 scrub starts 2024-09-13T13:19:09.495 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.9 scrub starts 2024-09-13T13:19:09.495 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: pgmap v912: 168 pgs: 1 active+clean+scrubbing, 2 peering, 2 remapped+peering, 8 unknown, 155 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 16 KiB/s, 0 objects/s recovering 2024-09-13T13:19:09.495 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 451'6153 2024-09-13T13:19:09.495 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 451'5996 2024-09-13T13:19:09.495 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 451'6059 2024-09-13T13:19:09.495 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 451'5844 2024-09-13T13:19:09.495 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 451'5922 2024-09-13T13:19:09.495 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.14 starting backfill to osd.1 from (0'0,0'0] MAX to 451'5847 2024-09-13T13:19:09.495 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 451'5827 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 451'6029 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 451'6153 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 451'5996 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 451'5920 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.14 starting backfill to osd.4 from (0'0,0'0] MAX to 451'5847 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.17 starting backfill to osd.7 from (0'0,0'0] MAX to 451'5827 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.15 starting backfill to osd.4 from (0'0,0'0] MAX to 451'5996 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.16 starting backfill to osd.5 from (0'0,0'0] MAX to 451'5920 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.13 starting backfill to osd.4 from (0'0,0'0] MAX to 451'6059 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.10 starting backfill to osd.6 from (0'0,0'0] MAX to 451'6153 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.18 starting backfill to osd.4 from (0'0,0'0] MAX to 451'6029 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 451'5922 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 451'5920 2024-09-13T13:19:09.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 451'5844 2024-09-13T13:19:09.497 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:09 smithi052 ceph-mon[36682]: 11.3 scrub starts 2024-09-13T13:19:09.497 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:09.546 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:09.566 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:09.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.9 scrub starts 2024-09-13T13:19:09.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: pgmap v912: 168 pgs: 1 active+clean+scrubbing, 2 peering, 2 remapped+peering, 8 unknown, 155 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 16 KiB/s, 0 objects/s recovering 2024-09-13T13:19:09.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 451'6153 2024-09-13T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 451'5996 2024-09-13T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 451'6059 2024-09-13T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 451'5844 2024-09-13T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 451'5922 2024-09-13T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.14 starting backfill to osd.1 from (0'0,0'0] MAX to 451'5847 2024-09-13T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 451'5827 2024-09-13T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 451'6029 2024-09-13T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 451'6153 2024-09-13T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 451'5996 2024-09-13T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 451'5920 2024-09-13T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.14 starting backfill to osd.4 from (0'0,0'0] MAX to 451'5847 2024-09-13T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.17 starting backfill to osd.7 from (0'0,0'0] MAX to 451'5827 2024-09-13T13:19:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.15 starting backfill to osd.4 from (0'0,0'0] MAX to 451'5996 2024-09-13T13:19:09.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.16 starting backfill to osd.5 from (0'0,0'0] MAX to 451'5920 2024-09-13T13:19:09.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.13 starting backfill to osd.4 from (0'0,0'0] MAX to 451'6059 2024-09-13T13:19:09.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.10 starting backfill to osd.6 from (0'0,0'0] MAX to 451'6153 2024-09-13T13:19:09.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.18 starting backfill to osd.4 from (0'0,0'0] MAX to 451'6029 2024-09-13T13:19:09.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 451'5922 2024-09-13T13:19:09.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 451'5920 2024-09-13T13:19:09.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 451'5844 2024-09-13T13:19:09.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:09 smithi167 ceph-mon[33178]: 11.3 scrub starts 2024-09-13T13:19:09.644 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:09.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:09.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:09.746 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:09.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:09.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:09.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:09.880 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:09.888 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 69 16 97283 97267 88.0828 58.5 0.00780006 0.011348 2024-09-13T13:19:09.907 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:09.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:09.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:10.009 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:10.061 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:10.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:10.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:10.163 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:10.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:10.244 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:10.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:10.345 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:10.388 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:10.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:10.490 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:10.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:10 smithi052 ceph-mon[36682]: 11.13 starting backfill to osd.5 from (0'0,0'0] MAX to 451'6059 2024-09-13T13:19:10.496 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:10 smithi052 ceph-mon[29471]: 11.13 starting backfill to osd.5 from (0'0,0'0] MAX to 451'6059 2024-09-13T13:19:10.512 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:10.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:10.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:10 smithi167 ceph-mon[33178]: 11.13 starting backfill to osd.5 from (0'0,0'0] MAX to 451'6059 2024-09-13T13:19:10.614 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:10.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:10.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:10.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:10.731 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:19:10.731 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:19:10.731 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:19:10.731 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:19:10.731 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:19:10.731 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:19:10.732 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:19:10.732 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:19:10.732 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:19:10.732 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:19:10.732 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:19:10.732 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:19:10.746 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:10.752 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:19:10.752 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:19:10.805 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:19:10.805 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:19:10.823 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:10.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:10.846 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:10.868 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:10.888 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 70 16 98138 98122 87.5877 53.4375 0.0183354 0.0114103 2024-09-13T13:19:10.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:10.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:10.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:11.054 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:11.084 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:11.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:11.186 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:11.221 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:11.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:11.245 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:19:11.321 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:11.357 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:11.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:11.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:11.460 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:11 smithi052 ceph-mon[36682]: pgmap v914: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 6 active+remapped+backfill_toofull, 157 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 54495/289137 objects misplaced (18.847%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T13:19:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:11 smithi052 ceph-mon[36682]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/905227379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3404553294' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:19:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:11 smithi052 ceph-mon[29471]: pgmap v914: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 6 active+remapped+backfill_toofull, 157 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 54495/289137 objects misplaced (18.847%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T13:19:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:11 smithi052 ceph-mon[29471]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/905227379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:11.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3404553294' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:19:11.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:11.499 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:11.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:11 smithi167 ceph-mon[33178]: pgmap v914: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 6 active+remapped+backfill_toofull, 157 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 54495/289137 objects misplaced (18.847%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T13:19:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:11 smithi167 ceph-mon[33178]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/905227379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3404553294' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:19:11.600 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:11.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:11.652 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:11.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:11.753 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:11.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:11.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:11.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:11.881 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:11.888 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 71 16 99300 99284 87.3769 72.625 0.0270241 0.0114386 2024-09-13T13:19:11.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:11.930 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:11.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:12.031 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:12.073 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:12.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:12.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:12.175 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:12.204 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:12.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:12.306 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:12.335 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:12.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:12.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:12.437 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:12.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:12.481 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:12.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:12.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:12 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:19:12.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:12 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3404553294' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:19:12.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:12 smithi052 ceph-mon[36682]: osdmap e455: 8 total, 8 up, 8 in 2024-09-13T13:19:12.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:12 smithi052 ceph-mon[36682]: 5.19 scrub starts 2024-09-13T13:19:12.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:12 smithi052 ceph-mon[36682]: 5.19 scrub ok 2024-09-13T13:19:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:12 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:19:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:12 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3404553294' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:19:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:12 smithi052 ceph-mon[29471]: osdmap e455: 8 total, 8 up, 8 in 2024-09-13T13:19:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:12 smithi052 ceph-mon[29471]: 5.19 scrub starts 2024-09-13T13:19:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:12 smithi052 ceph-mon[29471]: 5.19 scrub ok 2024-09-13T13:19:12.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:12 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:19:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:12 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3404553294' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:19:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:12 smithi167 ceph-mon[33178]: osdmap e455: 8 total, 8 up, 8 in 2024-09-13T13:19:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:12 smithi167 ceph-mon[33178]: 5.19 scrub starts 2024-09-13T13:19:12.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:12 smithi167 ceph-mon[33178]: 5.19 scrub ok 2024-09-13T13:19:12.581 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:12.624 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:12.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:12.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:12.726 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:12.750 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:12.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:12.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:12.852 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:12.888 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 72 16 100364 100348 87.0869 66.5 0.00558374 0.0114781 2024-09-13T13:19:12.894 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:12.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:12.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:13.031 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:13.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:13.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:13.160 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:13.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:13.262 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:13.294 INFO:teuthology.orchestra.run.smithi052.stderr:set osd.2 primary-affinity to 0.59968 (8393002) 2024-09-13T13:19:13.314 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:13.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:13.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:13.416 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:13.449 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:19:13.449 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:19:13.450 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:13 smithi052 ceph-mon[29471]: pgmap v916: 168 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 7 active+remapped+backfill_toofull, 158 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 878 B/s rd, 68 MiB/s wr, 1.08k op/s; 60551/290232 objects misplaced (20.863%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T13:19:13.450 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2721161410' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.5996798312787667} : dispatch 2024-09-13T13:19:13.450 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:13 smithi052 ceph-mon[29471]: from='client.26668 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.5996798312787667} : dispatch 2024-09-13T13:19:13.450 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:13 smithi052 ceph-mon[36682]: pgmap v916: 168 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 7 active+remapped+backfill_toofull, 158 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 878 B/s rd, 68 MiB/s wr, 1.08k op/s; 60551/290232 objects misplaced (20.863%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T13:19:13.450 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2721161410' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.5996798312787667} : dispatch 2024-09-13T13:19:13.451 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:13 smithi052 ceph-mon[36682]: from='client.26668 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.5996798312787667} : dispatch 2024-09-13T13:19:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:13.458 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:13.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:13.559 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:13 smithi167 ceph-mon[33178]: pgmap v916: 168 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 7 active+remapped+backfill_toofull, 158 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 878 B/s rd, 68 MiB/s wr, 1.08k op/s; 60551/290232 objects misplaced (20.863%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T13:19:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2721161410' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.5996798312787667} : dispatch 2024-09-13T13:19:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:13 smithi167 ceph-mon[33178]: from='client.26668 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.5996798312787667} : dispatch 2024-09-13T13:19:13.594 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:13.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:13.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:13.696 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:13.728 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:13.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:13.813 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:19:13.830 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:13.852 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:13.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:13.888 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 73 16 101130 101114 86.5498 47.875 0.0035299 0.0114829 2024-09-13T13:19:13.954 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:13.976 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:13.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:14.078 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:14.101 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:14.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:14.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:14.202 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:14.235 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:14.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:14.336 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:14.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:14.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:14.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:14.469 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[36682]: from='client.26668 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.5996798312787667}]': finished 2024-09-13T13:19:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[36682]: osdmap e456: 8 total, 8 up, 8 in 2024-09-13T13:19:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[36682]: 4.5 scrub starts 2024-09-13T13:19:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[36682]: 4.5 scrub ok 2024-09-13T13:19:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[36682]: 3.5 scrub starts 2024-09-13T13:19:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[36682]: 3.5 scrub ok 2024-09-13T13:19:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[36682]: osdmap e457: 8 total, 8 up, 8 in 2024-09-13T13:19:14.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[29471]: from='client.26668 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.5996798312787667}]': finished 2024-09-13T13:19:14.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[29471]: osdmap e456: 8 total, 8 up, 8 in 2024-09-13T13:19:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[29471]: 4.5 scrub starts 2024-09-13T13:19:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[29471]: 4.5 scrub ok 2024-09-13T13:19:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[29471]: 3.5 scrub starts 2024-09-13T13:19:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[29471]: 3.5 scrub ok 2024-09-13T13:19:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:14 smithi052 ceph-mon[29471]: osdmap e457: 8 total, 8 up, 8 in 2024-09-13T13:19:14.505 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:14.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:14.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:14.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:14 smithi167 ceph-mon[33178]: from='client.26668 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.5996798312787667}]': finished 2024-09-13T13:19:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:14 smithi167 ceph-mon[33178]: osdmap e456: 8 total, 8 up, 8 in 2024-09-13T13:19:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:14 smithi167 ceph-mon[33178]: 4.5 scrub starts 2024-09-13T13:19:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:14 smithi167 ceph-mon[33178]: 4.5 scrub ok 2024-09-13T13:19:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:14 smithi167 ceph-mon[33178]: 3.5 scrub starts 2024-09-13T13:19:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:14 smithi167 ceph-mon[33178]: 3.5 scrub ok 2024-09-13T13:19:14.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:14 smithi167 ceph-mon[33178]: osdmap e457: 8 total, 8 up, 8 in 2024-09-13T13:19:14.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:14.649 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:14.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:14.751 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:14.797 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:14.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:14.889 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T13:19:14.892 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 74 16 101804 101788 85.9494 42.125 0.00998243 0.0116294 2024-09-13T13:19:14.898 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:14.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:14.925 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:14.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:15.027 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:15.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:15.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:15.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:15.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:19:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:19:15.159 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:15.192 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:19:15.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:15.240 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:15.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:15.342 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:15.370 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:15.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:15.472 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[36682]: pgmap v918: 168 pgs: 1 remapped+peering, 2 peering, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 156 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 167 MiB/s wr, 2.68k op/s; 59904/297798 objects misplaced (20.116%); 8.4 MiB/s, 134 objects/s recovering 2024-09-13T13:19:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[36682]: 11.15 continuing backfill to osd.1 from (427'600,455'6151] MIN to 455'6151 2024-09-13T13:19:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[36682]: 11.15 continuing backfill to osd.2 from (427'600,455'6151] MIN to 455'6151 2024-09-13T13:19:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[36682]: 11.15 continuing backfill to osd.4 from (427'600,455'6151] MIN to 455'6151 2024-09-13T13:19:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[36682]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T13:19:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[36682]: 3.a scrub starts 2024-09-13T13:19:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[36682]: 3.a scrub ok 2024-09-13T13:19:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[36682]: 5.c scrub starts 2024-09-13T13:19:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[36682]: 5.c scrub ok 2024-09-13T13:19:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[29471]: pgmap v918: 168 pgs: 1 remapped+peering, 2 peering, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 156 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 167 MiB/s wr, 2.68k op/s; 59904/297798 objects misplaced (20.116%); 8.4 MiB/s, 134 objects/s recovering 2024-09-13T13:19:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[29471]: 11.15 continuing backfill to osd.1 from (427'600,455'6151] MIN to 455'6151 2024-09-13T13:19:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[29471]: 11.15 continuing backfill to osd.2 from (427'600,455'6151] MIN to 455'6151 2024-09-13T13:19:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[29471]: 11.15 continuing backfill to osd.4 from (427'600,455'6151] MIN to 455'6151 2024-09-13T13:19:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[29471]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T13:19:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[29471]: 3.a scrub starts 2024-09-13T13:19:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[29471]: 3.a scrub ok 2024-09-13T13:19:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[29471]: 5.c scrub starts 2024-09-13T13:19:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:15 smithi052 ceph-mon[29471]: 5.c scrub ok 2024-09-13T13:19:15.524 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:15.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:15.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:15.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:15 smithi167 ceph-mon[33178]: pgmap v918: 168 pgs: 1 remapped+peering, 2 peering, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 156 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 167 MiB/s wr, 2.68k op/s; 59904/297798 objects misplaced (20.116%); 8.4 MiB/s, 134 objects/s recovering 2024-09-13T13:19:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:15 smithi167 ceph-mon[33178]: 11.15 continuing backfill to osd.1 from (427'600,455'6151] MIN to 455'6151 2024-09-13T13:19:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:15 smithi167 ceph-mon[33178]: 11.15 continuing backfill to osd.2 from (427'600,455'6151] MIN to 455'6151 2024-09-13T13:19:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:15 smithi167 ceph-mon[33178]: 11.15 continuing backfill to osd.4 from (427'600,455'6151] MIN to 455'6151 2024-09-13T13:19:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:15 smithi167 ceph-mon[33178]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T13:19:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:15 smithi167 ceph-mon[33178]: 3.a scrub starts 2024-09-13T13:19:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:15 smithi167 ceph-mon[33178]: 3.a scrub ok 2024-09-13T13:19:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:15 smithi167 ceph-mon[33178]: 5.c scrub starts 2024-09-13T13:19:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:15 smithi167 ceph-mon[33178]: 5.c scrub ok 2024-09-13T13:19:15.627 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:15.652 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:15.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:15.739 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:19:15.739 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:19:15.746 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:15.754 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:15.785 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:19:15.785 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:19:15.807 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:15.819 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:15.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:15.835 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:19:15.836 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:19:15.876 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:19:15.876 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:19:15.889 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 75 15 102667 102652 85.5228 54 0.0180544 0.0116871 2024-09-13T13:19:15.911 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:19:15.911 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:19:15.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:15.949 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:19:15.950 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:19:15.950 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:15.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:15.986 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:19:15.987 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:19:16.022 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:19:16.022 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:19:16.052 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:16.099 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:16.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:16.201 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:16.245 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:16.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:16.338 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:19:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:19:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:19:16.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:16.375 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:16.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:16.476 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:16.522 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:16.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:16.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:16.623 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:16.659 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:16.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:16.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:16 smithi052 ceph-mon[36682]: 11.2 scrub starts 2024-09-13T13:19:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:16 smithi052 ceph-mon[36682]: 4.1f scrub starts 2024-09-13T13:19:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:16 smithi052 ceph-mon[36682]: 4.1f scrub ok 2024-09-13T13:19:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:16 smithi052 ceph-mon[36682]: 5.1b scrub starts 2024-09-13T13:19:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:16 smithi052 ceph-mon[36682]: 5.1b scrub ok 2024-09-13T13:19:16.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:16 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3407089245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:16 smithi052 ceph-mon[29471]: 11.2 scrub starts 2024-09-13T13:19:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:16 smithi052 ceph-mon[29471]: 4.1f scrub starts 2024-09-13T13:19:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:16 smithi052 ceph-mon[29471]: 4.1f scrub ok 2024-09-13T13:19:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:16 smithi052 ceph-mon[29471]: 5.1b scrub starts 2024-09-13T13:19:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:16 smithi052 ceph-mon[29471]: 5.1b scrub ok 2024-09-13T13:19:16.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:16 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3407089245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:16.761 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:16.789 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:16.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:16.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:16 smithi167 ceph-mon[33178]: 11.2 scrub starts 2024-09-13T13:19:16.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:16 smithi167 ceph-mon[33178]: 4.1f scrub starts 2024-09-13T13:19:16.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:16 smithi167 ceph-mon[33178]: 4.1f scrub ok 2024-09-13T13:19:16.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:16 smithi167 ceph-mon[33178]: 5.1b scrub starts 2024-09-13T13:19:16.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:16 smithi167 ceph-mon[33178]: 5.1b scrub ok 2024-09-13T13:19:16.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:16 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3407089245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:16.889 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 76 16 103653 103637 85.2075 61.5625 0.0162273 0.0117312 2024-09-13T13:19:16.891 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:16.930 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:16.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:17.032 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:17.085 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:17.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:17.187 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:17.229 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:17.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:17.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:17.331 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:17.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:17.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:17.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:17.469 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:17.500 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:17.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:17.602 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:17.644 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:17.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:17.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:17 smithi052 ceph-mon[36682]: pgmap v920: 168 pgs: 1 remapped+peering, 2 peering, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 156 active+clean; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 47 MiB/s wr, 749 op/s; 60065/302619 objects misplaced (19.848%); 16 MiB/s, 255 objects/s recovering 2024-09-13T13:19:17.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:17 smithi052 ceph-mon[36682]: 11.2 scrub ok 2024-09-13T13:19:17.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:17 smithi052 ceph-mon[36682]: 5.15 scrub starts 2024-09-13T13:19:17.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:17 smithi052 ceph-mon[36682]: 5.15 scrub ok 2024-09-13T13:19:17.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:17 smithi052 ceph-mon[29471]: pgmap v920: 168 pgs: 1 remapped+peering, 2 peering, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 156 active+clean; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 47 MiB/s wr, 749 op/s; 60065/302619 objects misplaced (19.848%); 16 MiB/s, 255 objects/s recovering 2024-09-13T13:19:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:17 smithi052 ceph-mon[29471]: 11.2 scrub ok 2024-09-13T13:19:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:17 smithi052 ceph-mon[29471]: 5.15 scrub starts 2024-09-13T13:19:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:17 smithi052 ceph-mon[29471]: 5.15 scrub ok 2024-09-13T13:19:17.745 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:17.776 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:17.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:17.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:17.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:17 smithi167 ceph-mon[33178]: pgmap v920: 168 pgs: 1 remapped+peering, 2 peering, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 156 active+clean; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 47 MiB/s wr, 749 op/s; 60065/302619 objects misplaced (19.848%); 16 MiB/s, 255 objects/s recovering 2024-09-13T13:19:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:17 smithi167 ceph-mon[33178]: 11.2 scrub ok 2024-09-13T13:19:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:17 smithi167 ceph-mon[33178]: 5.15 scrub starts 2024-09-13T13:19:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:17 smithi167 ceph-mon[33178]: 5.15 scrub ok 2024-09-13T13:19:17.878 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:17.889 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 77 16 104738 104722 84.9815 67.8125 0.00519414 0.0117599 2024-09-13T13:19:17.901 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:17.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:18.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:18.043 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:18.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:18.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:18.145 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:18.176 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:18.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:18.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:18.278 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:18.298 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:18.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:18.400 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:18.432 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:18.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:18.534 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:18.573 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:18.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:18.675 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:18.715 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:18.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:18.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:18 smithi052 ceph-mon[36682]: 5.11 scrub starts 2024-09-13T13:19:18.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:18 smithi052 ceph-mon[36682]: 5.11 scrub ok 2024-09-13T13:19:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:18 smithi052 ceph-mon[29471]: 5.11 scrub starts 2024-09-13T13:19:18.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:18 smithi052 ceph-mon[29471]: 5.11 scrub ok 2024-09-13T13:19:18.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:18.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:18 smithi167 ceph-mon[33178]: 5.11 scrub starts 2024-09-13T13:19:18.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:18 smithi167 ceph-mon[33178]: 5.11 scrub ok 2024-09-13T13:19:18.849 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:18.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:18.890 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 78 16 105814 105798 84.7542 67.25 0.00782661 0.0117939 2024-09-13T13:19:18.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:18.978 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:18.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:19.080 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:19.116 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:19.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:19.189 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:19:19.190 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:19:19.190 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 3, 5, 6, 7, 4, 2] 2024-09-13T13:19:19.190 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd out 7 2024-09-13T13:19:19.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:19.242 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:19.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:19.344 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:19.370 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:19.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:19.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:19.472 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:19.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:19 smithi052 ceph-mon[29471]: pgmap v921: 168 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 peering, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 156 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 756 B/s rd, 30 MiB/s wr, 487 op/s; 58698/308481 objects misplaced (19.028%); 24 MiB/s, 377 objects/s recovering 2024-09-13T13:19:19.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:19 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:19:19.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:19 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:19:19.482 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:19 smithi052 ceph-mon[36682]: pgmap v921: 168 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 peering, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 156 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 756 B/s rd, 30 MiB/s wr, 487 op/s; 58698/308481 objects misplaced (19.028%); 24 MiB/s, 377 objects/s recovering 2024-09-13T13:19:19.482 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:19 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:19:19.482 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:19 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:19:19.501 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:19.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:19.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:19.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:19.637 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:19.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:19.721 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:19 smithi167 ceph-mon[33178]: pgmap v921: 168 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 peering, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 156 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 756 B/s rd, 30 MiB/s wr, 487 op/s; 58698/308481 objects misplaced (19.028%); 24 MiB/s, 377 objects/s recovering 2024-09-13T13:19:19.721 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:19 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:19:19.721 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:19 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:19:19.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:19.780 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:19.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:19.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:19.882 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:19.890 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 79 16 106886 106870 84.5294 67 0.0100863 0.0118252 2024-09-13T13:19:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:19.914 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:19.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:20.015 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:20.076 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:20.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:20.177 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:20.220 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:20.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:20.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:20.323 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:20.355 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:20.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:20.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:20.457 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:20.486 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:20.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:20.588 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:20.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:20 smithi052 ceph-mon[36682]: 5.1d scrub starts 2024-09-13T13:19:20.602 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:20 smithi052 ceph-mon[36682]: 5.1d scrub ok 2024-09-13T13:19:20.602 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:20 smithi052 ceph-mon[29471]: 5.1d scrub starts 2024-09-13T13:19:20.602 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:20 smithi052 ceph-mon[29471]: 5.1d scrub ok 2024-09-13T13:19:20.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:20.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:20.718 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:20.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:20.750 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:20.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:20.750 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:20 smithi167 ceph-mon[33178]: 5.1d scrub starts 2024-09-13T13:19:20.751 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:20 smithi167 ceph-mon[33178]: 5.1d scrub ok 2024-09-13T13:19:20.785 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:19:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:19:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:19:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:19:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133006 10.8 10:1945bf5a:::benchmark_data_smithi052_153703_object5626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:19:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.287772+0000", 2024-09-13T13:19:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.49357380000001, 2024-09-13T13:19:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.372937727, 2024-09-13T13:19:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:19:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:19:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133006 2024-09-13T13:19:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.287772+0000", 2024-09-13T13:19:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.287772+0000", 2024-09-13T13:19:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.287775+0000", 2024-09-13T13:19:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2960000000000001e-06 2024-09-13T13:19:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.287782+0000", 2024-09-13T13:19:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8700000000000003e-06 2024-09-13T13:19:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.287797+0000", 2024-09-13T13:19:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5159e-05 2024-09-13T13:19:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.287804+0000", 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8059999999999995e-06 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.288282+0000", 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00047746099999999999 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.288400+0000", 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118169 2024-09-13T13:19:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.288514+0000", 2024-09-13T13:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114286 2024-09-13T13:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291531+0000", 2024-09-13T13:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0030161659999999998 2024-09-13T13:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302056+0000", 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010525097000000001 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.660629+0000", 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.35857315699999998 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.660674+0000", 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4702e-05 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.660710+0000", 2024-09-13T13:19:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6557999999999998e-05 2024-09-13T13:19:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133009 10.8 10:1c6c448b:::benchmark_data_smithi052_153703_object5629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:19:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.291955+0000", 2024-09-13T13:19:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.48939082999999, 2024-09-13T13:19:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.38164577399999999, 2024-09-13T13:19:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133009 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291955+0000", 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291955+0000", 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291958+0000", 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2089999999999999e-06 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291964+0000", 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0650000000000004e-06 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291975+0000", 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1916000000000001e-05 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.291983+0000", 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0840000000000003e-06 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.292037+0000", 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4836999999999997e-05 2024-09-13T13:19:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.292157+0000", 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000119206 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.292287+0000", 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013012500000000001 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.296788+0000", 2024-09-13T13:19:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045008069999999999 2024-09-13T13:19:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302358+0000", 2024-09-13T13:19:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055702039999999996 2024-09-13T13:19:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.673484+0000", 2024-09-13T13:19:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.371126542 2024-09-13T13:19:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.673549+0000", 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4498000000000005e-05 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.673601+0000", 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2281000000000003e-05 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133021 10.8 10:11f4c5d2:::benchmark_data_smithi052_153703_object5641:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:19:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.302866+0000", 2024-09-13T13:19:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.478479822, 2024-09-13T13:19:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.37089920399999998, 2024-09-13T13:19:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:19:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:19:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133021 2024-09-13T13:19:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302866+0000", 2024-09-13T13:19:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302866+0000", 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302867+0000", 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3200000000000004e-07 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302871+0000", 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4570000000000001e-06 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302876+0000", 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9349999999999999e-06 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302882+0000", 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2669999999999997e-06 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.302927+0000", 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5624000000000002e-05 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.303052+0000", 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012477699999999999 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.303152+0000", 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000100256 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.310146+0000", 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0069940749999999998 2024-09-13T13:19:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.314157+0000", 2024-09-13T13:19:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0040102250000000001 2024-09-13T13:19:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.673676+0000", 2024-09-13T13:19:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.35951962999999998 2024-09-13T13:19:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.673728+0000", 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1481999999999999e-05 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.673766+0000", 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7744000000000003e-05 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133022 10.8 10:1fafbb00:::benchmark_data_smithi052_153703_object5642:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.305318+0000", 2024-09-13T13:19:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.47602786600001, 2024-09-13T13:19:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.37413038399999998, 2024-09-13T13:19:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:19:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:19:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133022 2024-09-13T13:19:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.305318+0000", 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.305318+0000", 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.305321+0000", 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4669999999999998e-06 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.305326+0000", 2024-09-13T13:19:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.22e-06 2024-09-13T13:19:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.305338+0000", 2024-09-13T13:19:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1765e-05 2024-09-13T13:19:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.305345+0000", 2024-09-13T13:19:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5290000000000003e-06 2024-09-13T13:19:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.306501+0000", 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011562359999999999 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.306616+0000", 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114815 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.306690+0000", 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3603000000000007e-05 2024-09-13T13:19:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.314145+0000", 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0074549120000000002 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.314161+0000", 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6424999999999999e-05 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.679280+0000", 2024-09-13T13:19:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.36511832700000002 2024-09-13T13:19:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.679402+0000", 2024-09-13T13:19:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000122399 2024-09-13T13:19:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.679449+0000", 2024-09-13T13:19:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6686e-05 2024-09-13T13:19:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133025 10.8 10:1f5a4099:::benchmark_data_smithi052_153703_object5645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:19:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.308577+0000", 2024-09-13T13:19:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.47276956499999, 2024-09-13T13:19:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.37100242, 2024-09-13T13:19:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:19:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:19:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133025 2024-09-13T13:19:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.308577+0000", 2024-09-13T13:19:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.308577+0000", 2024-09-13T13:19:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.308579+0000", 2024-09-13T13:19:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0439999999999998e-06 2024-09-13T13:19:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.308584+0000", 2024-09-13T13:19:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3850000000000001e-06 2024-09-13T13:19:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.308595+0000", 2024-09-13T13:19:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1073999999999999e-05 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.308673+0000", 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8386000000000005e-05 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.310177+0000", 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0015036279999999999 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.310276+0000", 2024-09-13T13:19:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9358000000000007e-05 2024-09-13T13:19:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:19:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.310380+0000", 2024-09-13T13:19:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103349 2024-09-13T13:19:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.316371+0000", 2024-09-13T13:19:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0059909020000000002 2024-09-13T13:19:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.321955+0000", 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055840250000000003 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.679487+0000", 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.35753211000000001 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.679545+0000", 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8440999999999998e-05 2024-09-13T13:19:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.679579+0000", 2024-09-13T13:19:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3717999999999998e-05 2024-09-13T13:19:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133039 10.8 10:1d3cc177:::benchmark_data_smithi052_153703_object5659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:19:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.317059+0000", 2024-09-13T13:19:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.464287429, 2024-09-13T13:19:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.36894714499999998, 2024-09-13T13:19:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:19:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:19:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133039 2024-09-13T13:19:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317059+0000", 2024-09-13T13:19:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317059+0000", 2024-09-13T13:19:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317081+0000", 2024-09-13T13:19:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2044000000000001e-05 2024-09-13T13:19:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317087+0000", 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2550000000000003e-06 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317102+0000", 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5398000000000001e-05 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317110+0000", 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1339999999999998e-06 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317163+0000", 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3170000000000001e-05 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317278+0000", 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000115739 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.317487+0000", 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020869700000000001 2024-09-13T13:19:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.320968+0000", 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0034804110000000001 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.321999+0000", 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010309749999999999 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.685909+0000", 2024-09-13T13:19:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.36391061000000002 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.685965+0000", 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5831999999999997e-05 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686006+0000", 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0880000000000002e-05 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133060 10.8 10:1fe5c61e:::benchmark_data_smithi052_153703_object5680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.336351+0000", 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.444994772, 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.349824423, 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133060 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336351+0000", 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336351+0000", 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336353+0000", 2024-09-13T13:19:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.796e-06 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336358+0000", 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.147e-06 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336371+0000", 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2359e-05 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336378+0000", 2024-09-13T13:19:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8510000000000001e-06 2024-09-13T13:19:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336444+0000", 2024-09-13T13:19:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6919000000000001e-05 2024-09-13T13:19:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336557+0000", 2024-09-13T13:19:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011230699999999999 2024-09-13T13:19:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:19:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.336668+0000", 2024-09-13T13:19:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000111459 2024-09-13T13:19:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.339885+0000", 2024-09-13T13:19:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0032169550000000001 2024-09-13T13:19:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.339996+0000", 2024-09-13T13:19:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110497 2024-09-13T13:19:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686049+0000", 2024-09-13T13:19:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34605333399999999 2024-09-13T13:19:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686137+0000", 2024-09-13T13:19:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8381999999999995e-05 2024-09-13T13:19:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686176+0000", 2024-09-13T13:19:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8417000000000002e-05 2024-09-13T13:19:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133065 10.8 10:12b65a36:::benchmark_data_smithi052_153703_object5685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:19:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.341093+0000", 2024-09-13T13:19:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.44025347300001, 2024-09-13T13:19:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34520810800000001, 2024-09-13T13:19:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:19:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:19:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133065 2024-09-13T13:19:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341093+0000", 2024-09-13T13:19:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341093+0000", 2024-09-13T13:19:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341094+0000", 2024-09-13T13:19:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.457e-06 2024-09-13T13:19:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341099+0000", 2024-09-13T13:19:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7690000000000004e-06 2024-09-13T13:19:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341110+0000", 2024-09-13T13:19:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1153e-05 2024-09-13T13:19:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341115+0000", 2024-09-13T13:19:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1909999999999999e-06 2024-09-13T13:19:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341158+0000", 2024-09-13T13:19:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2528999999999999e-05 2024-09-13T13:19:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341238+0000", 2024-09-13T13:19:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0149999999999997e-05 2024-09-13T13:19:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.341308+0000", 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9954000000000001e-05 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.343428+0000", 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0021204190000000001 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.343759+0000", 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033045600000000002 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686214+0000", 2024-09-13T13:19:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34245503799999999 2024-09-13T13:19:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686267+0000", 2024-09-13T13:19:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3248000000000002e-05 2024-09-13T13:19:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.686301+0000", 2024-09-13T13:19:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3744000000000001e-05 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:133068 10.8 10:107081bf:::benchmark_data_smithi052_153703_object5688:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:15:39.346485+0000", 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 221.43486075300001, 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34413897900000001, 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 133068 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346485+0000", 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346485+0000", 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346487+0000", 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.767e-06 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346492+0000", 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8539999999999996e-06 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346502+0000", 2024-09-13T13:19:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.533e-06 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346507+0000", 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1109999999999999e-06 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346558+0000", 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1841000000000001e-05 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346641+0000", 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2157000000000006e-05 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.346718+0000", 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7080000000000001e-05 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.348847+0000", 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0021295789999999999 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.348979+0000", 2024-09-13T13:19:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013179000000000001 2024-09-13T13:19:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.690469+0000", 2024-09-13T13:19:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34149033299999998 2024-09-13T13:19:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.690568+0000", 2024-09-13T13:19:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8403e-05 2024-09-13T13:19:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:15:39.690624+0000", 2024-09-13T13:19:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6530999999999998e-05 2024-09-13T13:19:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:178645 10.5 10:a90a155a:::benchmark_data_smithi052_153703_object51265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:16:22.255202+0000", 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 178.52614404900001, 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.31956353900000001, 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 178645 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.255202+0000", 2024-09-13T13:19:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.255202+0000", 2024-09-13T13:19:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.255203+0000", 2024-09-13T13:19:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2759999999999999e-06 2024-09-13T13:19:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.255208+0000", 2024-09-13T13:19:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4190000000000004e-06 2024-09-13T13:19:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.255220+0000", 2024-09-13T13:19:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1976e-05 2024-09-13T13:19:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.255227+0000", 2024-09-13T13:19:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1230000000000002e-06 2024-09-13T13:19:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.255287+0000", 2024-09-13T13:19:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0439999999999997e-05 2024-09-13T13:19:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.255422+0000", 2024-09-13T13:19:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013499700000000001 2024-09-13T13:19:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T13:19:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.255579+0000", 2024-09-13T13:19:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015703900000000001 2024-09-13T13:19:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.259057+0000", 2024-09-13T13:19:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0034776999999999998 2024-09-13T13:19:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.259650+0000", 2024-09-13T13:19:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00059286600000000005 2024-09-13T13:19:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.259764+0000", 2024-09-13T13:19:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114437 2024-09-13T13:19:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.574499+0000", 2024-09-13T13:19:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.31473457700000002 2024-09-13T13:19:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.574565+0000", 2024-09-13T13:19:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6556000000000004e-05 2024-09-13T13:19:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.574766+0000", 2024-09-13T13:19:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020013299999999999 2024-09-13T13:19:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:178653 10.5 10:a597e333:::benchmark_data_smithi052_153703_object51273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-13T13:19:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:16:22.274775+0000", 2024-09-13T13:19:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 178.50657105100001, 2024-09-13T13:19:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.300142354, 2024-09-13T13:19:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:19:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:19:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 178653 2024-09-13T13:19:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.274775+0000", 2024-09-13T13:19:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.274775+0000", 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.274777+0000", 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6789999999999999e-06 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.274782+0000", 2024-09-13T13:19:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7069999999999997e-06 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.274794+0000", 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1742999999999999e-05 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.274800+0000", 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.874e-06 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.274848+0000", 2024-09-13T13:19:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8025999999999997e-05 2024-09-13T13:19:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.274950+0000", 2024-09-13T13:19:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102308 2024-09-13T13:19:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,3,7", 2024-09-13T13:19:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.275031+0000", 2024-09-13T13:19:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0308000000000004e-05 2024-09-13T13:19:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.276941+0000", 2024-09-13T13:19:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001910337 2024-09-13T13:19:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.277010+0000", 2024-09-13T13:19:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9330000000000002e-05 2024-09-13T13:19:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.277183+0000", 2024-09-13T13:19:20.859 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017285799999999999 2024-09-13T13:19:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.574815+0000", 2024-09-13T13:19:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.29763214599999999 2024-09-13T13:19:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.574852+0000", 2024-09-13T13:19:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6443000000000003e-05 2024-09-13T13:19:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.574917+0000", 2024-09-13T13:19:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5594999999999998e-05 2024-09-13T13:19:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:242835 10.8 10:12074b7c:::benchmark_data_smithi052_153703_object115455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:19:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:17:30.151285+0000", 2024-09-13T13:19:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 110.630060704, 2024-09-13T13:19:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.93503249799999999, 2024-09-13T13:19:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:19:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:19:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 242835 2024-09-13T13:19:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151285+0000", 2024-09-13T13:19:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151285+0000", 2024-09-13T13:19:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151288+0000", 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2560000000000001e-06 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151294+0000", 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1199999999999999e-06 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151308+0000", 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4329e-05 2024-09-13T13:19:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151316+0000", 2024-09-13T13:19:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5610000000000001e-06 2024-09-13T13:19:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151379+0000", 2024-09-13T13:19:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3212000000000004e-05 2024-09-13T13:19:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151388+0000", 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2129999999999997e-06 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.081972+0000", 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.93058433200000001 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.082112+0000", 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000139121 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:19:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.082242+0000", 2024-09-13T13:19:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013008599999999999 2024-09-13T13:19:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.085361+0000", 2024-09-13T13:19:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031196280000000002 2024-09-13T13:19:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.086121+0000", 2024-09-13T13:19:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00076016099999999997 2024-09-13T13:19:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.086161+0000", 2024-09-13T13:19:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9515999999999999e-05 2024-09-13T13:19:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.086270+0000", 2024-09-13T13:19:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010909 2024-09-13T13:19:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.086318+0000", 2024-09-13T13:19:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7873000000000001e-05 2024-09-13T13:19:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94535 11.6 11:67ee0d1d:::benchmark_data_smithi052_182575_object94534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:19:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.197231+0000", 2024-09-13T13:19:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 16.584115521000001, 2024-09-13T13:19:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99585655699999998, 2024-09-13T13:19:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:19:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94535 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197231+0000", 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197231+0000", 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197233+0000", 2024-09-13T13:19:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6860000000000002e-06 2024-09-13T13:19:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197299+0000", 2024-09-13T13:19:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5452999999999998e-05 2024-09-13T13:19:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197312+0000", 2024-09-13T13:19:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3123e-05 2024-09-13T13:19:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197321+0000", 2024-09-13T13:19:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6689999999999995e-06 2024-09-13T13:19:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.391516+0000", 2024-09-13T13:19:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19419592099999999 2024-09-13T13:19:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.391528+0000", 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1972e-05 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.193003+0000", 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.80147465100000004 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.193087+0000", 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4081999999999999e-05 2024-09-13T13:19:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94545 11.16 11:6f6df5ad:::benchmark_data_smithi052_182575_object94544:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:19:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.197343+0000", 2024-09-13T13:19:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 16.584002891000001, 2024-09-13T13:19:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.039569797, 2024-09-13T13:19:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94545 2024-09-13T13:19:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197343+0000", 2024-09-13T13:19:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197343+0000", 2024-09-13T13:19:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197344+0000", 2024-09-13T13:19:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:19:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197395+0000", 2024-09-13T13:19:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1091999999999998e-05 2024-09-13T13:19:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197402+0000", 2024-09-13T13:19:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0489999999999998e-06 2024-09-13T13:19:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197409+0000", 2024-09-13T13:19:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.46e-06 2024-09-13T13:19:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.392329+0000", 2024-09-13T13:19:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.194920121 2024-09-13T13:19:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.392335+0000", 2024-09-13T13:19:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8070000000000002e-06 2024-09-13T13:19:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.218051+0000", 2024-09-13T13:19:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.82571548400000006 2024-09-13T13:19:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.218191+0000", 2024-09-13T13:19:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000139796 2024-09-13T13:19:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:19:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.218510+0000", 2024-09-13T13:19:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00031968200000000001 2024-09-13T13:19:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.231664+0000", 2024-09-13T13:19:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.013153855000000001 2024-09-13T13:19:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.233621+0000", 2024-09-13T13:19:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019570260000000002 2024-09-13T13:19:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.234815+0000", 2024-09-13T13:19:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011941300000000001 2024-09-13T13:19:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.236861+0000", 2024-09-13T13:19:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020457180000000002 2024-09-13T13:19:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.236913+0000", 2024-09-13T13:19:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1867000000000003e-05 2024-09-13T13:19:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94546 11.6 11:6495996a:::benchmark_data_smithi052_182575_object94545:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:19:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.381884+0000", 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 16.399462410000002, 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.82819121900000003, 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94546 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.381884+0000", 2024-09-13T13:19:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.381884+0000", 2024-09-13T13:19:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.381887+0000", 2024-09-13T13:19:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9220000000000001e-06 2024-09-13T13:19:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.381942+0000", 2024-09-13T13:19:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4935999999999997e-05 2024-09-13T13:19:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.381965+0000", 2024-09-13T13:19:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3189000000000001e-05 2024-09-13T13:19:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.381974+0000", 2024-09-13T13:19:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6770000000000007e-06 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.391569+0000", 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0095950419999999998 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.391575+0000", 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7359999999999998e-06 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.193113+0000", 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.80153745200000004 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.193251+0000", 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013861900000000001 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.193630+0000", 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00037908499999999998 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.202549+0000", 2024-09-13T13:19:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0089188300000000009 2024-09-13T13:19:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.204191+0000", 2024-09-13T13:19:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016421210000000001 2024-09-13T13:19:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.209977+0000", 2024-09-13T13:19:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057851980000000001 2024-09-13T13:19:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.210033+0000", 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6082999999999998e-05 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.210075+0000", 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2329000000000001e-05 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.891 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95596 11.10 11.3871e090 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:19:20.891 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:06.189941+0000", 2024-09-13T13:19:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 14.591404868, 2024-09-13T13:19:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99412121499999995, 2024-09-13T13:19:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:19:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95596 2024-09-13T13:19:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189941+0000", 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189941+0000", 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189944+0000", 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3209999999999999e-06 2024-09-13T13:19:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189998+0000", 2024-09-13T13:19:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4691999999999997e-05 2024-09-13T13:19:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190017+0000", 2024-09-13T13:19:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8853999999999999e-05 2024-09-13T13:19:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190026+0000", 2024-09-13T13:19:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9840000000000007e-06 2024-09-13T13:19:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190300+0000", 2024-09-13T13:19:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027355400000000002 2024-09-13T13:19:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190308+0000", 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1629999999999996e-06 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184050+0000", 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99374201200000001 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184062+0000", 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2635e-05 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95599 11.10 11.86c1c970 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:06.190054+0000", 2024-09-13T13:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 14.591292407999999, 2024-09-13T13:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99507878000000005, 2024-09-13T13:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95599 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190054+0000", 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190054+0000", 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190055+0000", 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9299999999999996e-07 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190103+0000", 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7883999999999998e-05 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190111+0000", 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9439999999999994e-06 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190118+0000", 2024-09-13T13:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6209999999999998e-06 2024-09-13T13:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190330+0000", 2024-09-13T13:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000212062 2024-09-13T13:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190335+0000", 2024-09-13T13:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.904e-06 2024-09-13T13:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185123+0000", 2024-09-13T13:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99478760200000005 2024-09-13T13:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185133+0000", 2024-09-13T13:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8700000000000004e-06 2024-09-13T13:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95604 11.19 11:9cb7ac0c:::benchmark_data_smithi052_182575_object95603:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:06.190129+0000", 2024-09-13T13:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 14.591217181999999, 2024-09-13T13:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1591429629999999, 2024-09-13T13:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95604 2024-09-13T13:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190129+0000", 2024-09-13T13:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190129+0000", 2024-09-13T13:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190130+0000", 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0500000000000002e-07 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190186+0000", 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6026000000000002e-05 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190195+0000", 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7199999999999995e-06 2024-09-13T13:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190201+0000", 2024-09-13T13:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5930000000000002e-06 2024-09-13T13:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.192356+0000", 2024-09-13T13:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002155213 2024-09-13T13:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.192366+0000", 2024-09-13T13:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3109999999999995e-06 2024-09-13T13:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.189740+0000", 2024-09-13T13:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99737389399999998 2024-09-13T13:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.189750+0000", 2024-09-13T13:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0695e-05 2024-09-13T13:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.336476+0000", 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.146725397 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.336929+0000", 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00045369799999999999 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,3,6,7", 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.337090+0000", 2024-09-13T13:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016081400000000001 2024-09-13T13:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.342581+0000", 2024-09-13T13:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0054909169999999997 2024-09-13T13:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.342596+0000", 2024-09-13T13:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4824000000000001e-05 2024-09-13T13:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.343084+0000", 2024-09-13T13:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00048794499999999999 2024-09-13T13:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.348967+0000", 2024-09-13T13:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058832540000000001 2024-09-13T13:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.349081+0000", 2024-09-13T13:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011430999999999999 2024-09-13T13:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.349146+0000", 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5075999999999998e-05 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.349272+0000", 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012547099999999999 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95598 11.16 11:6bc1a0ca:::benchmark_data_smithi052_182575_object95597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:19:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:07.182439+0000", 2024-09-13T13:19:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 13.598907381, 2024-09-13T13:19:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1598586870000001, 2024-09-13T13:19:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:19:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95598 2024-09-13T13:19:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182439+0000", 2024-09-13T13:19:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182439+0000", 2024-09-13T13:19:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182442+0000", 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9950000000000001e-06 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182567+0000", 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012562599999999999 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182584+0000", 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6447e-05 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182600+0000", 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6181999999999999e-05 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.186622+0000", 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0040218770000000001 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.186632+0000", 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0027000000000001e-05 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.338172+0000", 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1515402800000001 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for degraded object", 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.338941+0000", 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00076854799999999997 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.342181+0000", 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00323989 2024-09-13T13:19:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.342297+0000", 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116815 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95629 11.11 11:89554284:::benchmark_data_smithi052_182575_object95628:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:07.182622+0000", 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 13.598723718, 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1767014709999999, 2024-09-13T13:19:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95629 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182622+0000", 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182622+0000", 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182624+0000", 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.246e-06 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182722+0000", 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8034000000000004e-05 2024-09-13T13:19:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182733+0000", 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1208e-05 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182741+0000", 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4589999999999998e-06 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.189984+0000", 2024-09-13T13:19:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0072423599999999998 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.189994+0000", 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0319999999999999e-05 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.346998+0000", 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1570035679999999 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.347142+0000", 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014469099999999999 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,3,6,7", 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.347473+0000", 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033115599999999999 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.349325+0000", 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001851927 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.356482+0000", 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0071570720000000004 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.356521+0000", 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8787e-05 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.356707+0000", 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018582000000000001 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.359074+0000", 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023674260000000002 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.359189+0000", 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114036 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.359324+0000", 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013536099999999999 2024-09-13T13:19:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:20.924 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:19:20.924 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:2024-09-13T13:19:20.890095+0000 min lat: 0.00125067 max lat: 2.16136 avg lat: 0.0118674 2024-09-13T13:19:20.924 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T13:19:20.924 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 80 16 107846 107830 84.2227 60 0.0396062 0.0118674 2024-09-13T13:19:20.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:20.925 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:20.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:21.026 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:21.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:21.071 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:21.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:21.172 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:21.215 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:21.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:21.317 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:21.350 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:21.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:21.371 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:19:21.451 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:21.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:21 smithi052 ceph-mon[36682]: pgmap v922: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 686 op/s; 70050/316143 objects misplaced (22.158%); 23 MiB/s, 372 objects/s recovering 2024-09-13T13:19:21.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:21 smithi052 ceph-mon[36682]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:21.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:21 smithi052 ceph-mon[36682]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T13:19:21.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:21 smithi052 ceph-mon[36682]: 5.2 scrub starts 2024-09-13T13:19:21.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:21 smithi052 ceph-mon[36682]: 5.2 scrub ok 2024-09-13T13:19:21.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:21 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/571828889' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:19:21.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:21 smithi052 ceph-mon[29471]: pgmap v922: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 686 op/s; 70050/316143 objects misplaced (22.158%); 23 MiB/s, 372 objects/s recovering 2024-09-13T13:19:21.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:21 smithi052 ceph-mon[29471]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:21.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:21 smithi052 ceph-mon[29471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T13:19:21.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:21 smithi052 ceph-mon[29471]: 5.2 scrub starts 2024-09-13T13:19:21.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:21 smithi052 ceph-mon[29471]: 5.2 scrub ok 2024-09-13T13:19:21.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:21 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/571828889' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:19:21.490 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:21.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:21.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:21.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:21.623 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:21.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:21.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:21.754 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:21.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:21.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:21 smithi167 ceph-mon[33178]: pgmap v922: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 686 op/s; 70050/316143 objects misplaced (22.158%); 23 MiB/s, 372 objects/s recovering 2024-09-13T13:19:21.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:21 smithi167 ceph-mon[33178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:21.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:21 smithi167 ceph-mon[33178]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T13:19:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:21 smithi167 ceph-mon[33178]: 5.2 scrub starts 2024-09-13T13:19:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:21 smithi167 ceph-mon[33178]: 5.2 scrub ok 2024-09-13T13:19:21.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:21 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/571828889' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:19:21.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:21.890 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 81 16 108744 108728 83.8758 56.125 0.0126415 0.0119177 2024-09-13T13:19:21.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:21.891 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:21.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:21.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:22.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:22.014 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:22.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:22.050 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:19:22.115 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:22.141 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:22.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:22.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:22.243 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:22.264 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:22.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:22.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:22.365 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:22.396 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:22 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:19:22.397 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:22 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/571828889' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:19:22.397 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:22 smithi052 ceph-mon[29471]: osdmap e458: 8 total, 8 up, 8 in 2024-09-13T13:19:22.397 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:22 smithi052 ceph-mon[29471]: 4.7 scrub starts 2024-09-13T13:19:22.429 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:22.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:22.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:22.552 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:22.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:22.654 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:22.701 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:22.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:22.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:22 smithi052 ceph-mon[29471]: 4.7 scrub ok 2024-09-13T13:19:22.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:22 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:19:22.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:22 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/571828889' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:19:22.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:22 smithi052 ceph-mon[36682]: osdmap e458: 8 total, 8 up, 8 in 2024-09-13T13:19:22.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:22 smithi052 ceph-mon[36682]: 4.7 scrub starts 2024-09-13T13:19:22.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:22 smithi052 ceph-mon[36682]: 4.7 scrub ok 2024-09-13T13:19:22.802 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:22.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:22 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:19:22.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:22 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/571828889' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:19:22.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:22 smithi167 ceph-mon[33178]: osdmap e458: 8 total, 8 up, 8 in 2024-09-13T13:19:22.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:22 smithi167 ceph-mon[33178]: 4.7 scrub starts 2024-09-13T13:19:22.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:22 smithi167 ceph-mon[33178]: 4.7 scrub ok 2024-09-13T13:19:22.833 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:22.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:22.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:22.890 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 82 16 109683 109667 83.5686 58.6875 0.0077011 0.0119614 2024-09-13T13:19:22.934 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:22.968 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:22.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:22.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:23.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:23.098 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:23.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:23.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:23.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:23.232 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:23.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:23.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:23.334 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:23.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:23.365 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:23.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:23.466 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:23.489 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:23.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:23.590 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:23.621 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:23.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:23.723 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:23 smithi052 ceph-mon[36682]: pgmap v924: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 453 op/s; 70378/318117 objects misplaced (22.123%); 18 MiB/s, 288 objects/s recovering 2024-09-13T13:19:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:23 smithi052 ceph-mon[36682]: 3.10 scrub starts 2024-09-13T13:19:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:23 smithi052 ceph-mon[36682]: 3.10 scrub ok 2024-09-13T13:19:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:23 smithi052 ceph-mon[36682]: 6.1 scrub starts 2024-09-13T13:19:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:23 smithi052 ceph-mon[36682]: 6.1 scrub ok 2024-09-13T13:19:23.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:23 smithi052 ceph-mon[29471]: pgmap v924: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 453 op/s; 70378/318117 objects misplaced (22.123%); 18 MiB/s, 288 objects/s recovering 2024-09-13T13:19:23.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:23 smithi052 ceph-mon[29471]: 3.10 scrub starts 2024-09-13T13:19:23.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:23 smithi052 ceph-mon[29471]: 3.10 scrub ok 2024-09-13T13:19:23.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:23 smithi052 ceph-mon[29471]: 6.1 scrub starts 2024-09-13T13:19:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:23 smithi052 ceph-mon[29471]: 6.1 scrub ok 2024-09-13T13:19:23.752 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:23.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:23 smithi167 ceph-mon[33178]: pgmap v924: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 453 op/s; 70378/318117 objects misplaced (22.123%); 18 MiB/s, 288 objects/s recovering 2024-09-13T13:19:23.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:23 smithi167 ceph-mon[33178]: 3.10 scrub starts 2024-09-13T13:19:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:23 smithi167 ceph-mon[33178]: 3.10 scrub ok 2024-09-13T13:19:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:23 smithi167 ceph-mon[33178]: 6.1 scrub starts 2024-09-13T13:19:23.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:23 smithi167 ceph-mon[33178]: 6.1 scrub ok 2024-09-13T13:19:23.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:23.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:23.876 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:23.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:23.890 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 83 16 110514 110498 83.1875 51.9375 0.0139529 0.0120079 2024-09-13T13:19:23.976 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:24.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:24.009 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:24.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:24.110 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:24.136 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:24.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:24.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:24.238 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:24.265 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:24.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:24.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:24.367 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:24.410 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:24.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:24.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:24.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:24.545 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:24.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:24.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:24.678 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:24.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:24.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:24 smithi052 ceph-mon[29471]: 1.4 scrub starts 2024-09-13T13:19:24.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:24 smithi052 ceph-mon[29471]: 1.4 scrub ok 2024-09-13T13:19:24.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:24 smithi052 ceph-mon[29471]: 11.f scrub starts 2024-09-13T13:19:24.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:24 smithi052 ceph-mon[36682]: 1.4 scrub starts 2024-09-13T13:19:24.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:24 smithi052 ceph-mon[36682]: 1.4 scrub ok 2024-09-13T13:19:24.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:24 smithi052 ceph-mon[36682]: 11.f scrub starts 2024-09-13T13:19:24.780 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:24.809 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:24.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:24.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:24.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:24 smithi167 ceph-mon[33178]: 1.4 scrub starts 2024-09-13T13:19:24.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:24 smithi167 ceph-mon[33178]: 1.4 scrub ok 2024-09-13T13:19:24.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:24 smithi167 ceph-mon[33178]: 11.f scrub starts 2024-09-13T13:19:24.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:19:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:19:24.890 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 84 16 111468 111452 82.9069 59.625 0.0142305 0.0120565 2024-09-13T13:19:24.912 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:24.941 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:24.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:24.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:25.043 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:25.066 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:25.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:25.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:25.168 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:25.199 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:25.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:25.301 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:25.334 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:25.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:25.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:25.436 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:25.459 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:25.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:25.560 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:25.604 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:25.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:25.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:25.706 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:25 smithi052 ceph-mon[29471]: pgmap v925: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 41 MiB/s wr, 649 op/s; 70298/326169 objects misplaced (21.553%); 19 MiB/s, 305 objects/s recovering 2024-09-13T13:19:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:25 smithi052 ceph-mon[29471]: 3.c scrub starts 2024-09-13T13:19:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:25 smithi052 ceph-mon[29471]: 3.c scrub ok 2024-09-13T13:19:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:25 smithi052 ceph-mon[29471]: 3.19 scrub starts 2024-09-13T13:19:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:25 smithi052 ceph-mon[29471]: 3.19 scrub ok 2024-09-13T13:19:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:25 smithi052 ceph-mon[36682]: pgmap v925: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 41 MiB/s wr, 649 op/s; 70298/326169 objects misplaced (21.553%); 19 MiB/s, 305 objects/s recovering 2024-09-13T13:19:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:25 smithi052 ceph-mon[36682]: 3.c scrub starts 2024-09-13T13:19:25.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:25 smithi052 ceph-mon[36682]: 3.c scrub ok 2024-09-13T13:19:25.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:25 smithi052 ceph-mon[36682]: 3.19 scrub starts 2024-09-13T13:19:25.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:25 smithi052 ceph-mon[36682]: 3.19 scrub ok 2024-09-13T13:19:25.748 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:25.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:25.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:25.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:25 smithi167 ceph-mon[33178]: pgmap v925: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 41 MiB/s wr, 649 op/s; 70298/326169 objects misplaced (21.553%); 19 MiB/s, 305 objects/s recovering 2024-09-13T13:19:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:25 smithi167 ceph-mon[33178]: 3.c scrub starts 2024-09-13T13:19:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:25 smithi167 ceph-mon[33178]: 3.c scrub ok 2024-09-13T13:19:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:25 smithi167 ceph-mon[33178]: 3.19 scrub starts 2024-09-13T13:19:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:25 smithi167 ceph-mon[33178]: 3.19 scrub ok 2024-09-13T13:19:25.850 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:25.873 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:25.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:25.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:25.890 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 85 16 112406 112390 82.6212 58.625 0.00261071 0.0120924 2024-09-13T13:19:25.976 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:26.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:26.007 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:26.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:26.108 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:26.135 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:26.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:26.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:26.238 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:26.277 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:26.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:26.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:26.380 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:26.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:26.402 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:26.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:26.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:19:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:19:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:19:26.502 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:26.529 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:26.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:26.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:26.631 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:26.670 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:26.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:26.772 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:26.801 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:26.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:26.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:26.890 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 86 16 113222 113206 82.2536 51 0.0099767 0.0121523 2024-09-13T13:19:26.903 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:26.950 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:26.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:26.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:27.052 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:27.084 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:27.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:27.185 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:27.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:27.224 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:27.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:27.325 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:27.366 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:27.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:27.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:27.468 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:27.512 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:27.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:27.614 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:27.644 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:27.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:27.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:27.714 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:19:27.715 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:19:27.715 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[36682]: pgmap v926: 168 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 157 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 39 MiB/s wr, 630 op/s; 71031/330963 objects misplaced (21.462%); 16 MiB/s, 258 objects/s recovering 2024-09-13T13:19:27.715 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[36682]: 6.1d scrub starts 2024-09-13T13:19:27.715 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[36682]: 6.1d scrub ok 2024-09-13T13:19:27.715 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[36682]: osdmap e459: 8 total, 8 up, 8 in 2024-09-13T13:19:27.715 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[36682]: 6.5 scrub starts 2024-09-13T13:19:27.715 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[36682]: 6.5 scrub ok 2024-09-13T13:19:27.715 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:19:27.715 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:19:27.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:19:27.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:19:27.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[29471]: pgmap v926: 168 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 157 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 39 MiB/s wr, 630 op/s; 71031/330963 objects misplaced (21.462%); 16 MiB/s, 258 objects/s recovering 2024-09-13T13:19:27.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[29471]: 6.1d scrub starts 2024-09-13T13:19:27.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[29471]: 6.1d scrub ok 2024-09-13T13:19:27.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[29471]: osdmap e459: 8 total, 8 up, 8 in 2024-09-13T13:19:27.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[29471]: 6.5 scrub starts 2024-09-13T13:19:27.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[29471]: 6.5 scrub ok 2024-09-13T13:19:27.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:19:27.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:27 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:19:27.746 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:27.768 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:27.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:27.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:27.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:27 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:19:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:27 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:19:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:27 smithi167 ceph-mon[33178]: pgmap v926: 168 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 157 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 39 MiB/s wr, 630 op/s; 71031/330963 objects misplaced (21.462%); 16 MiB/s, 258 objects/s recovering 2024-09-13T13:19:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:27 smithi167 ceph-mon[33178]: 6.1d scrub starts 2024-09-13T13:19:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:27 smithi167 ceph-mon[33178]: 6.1d scrub ok 2024-09-13T13:19:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:27 smithi167 ceph-mon[33178]: osdmap e459: 8 total, 8 up, 8 in 2024-09-13T13:19:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:27 smithi167 ceph-mon[33178]: 6.5 scrub starts 2024-09-13T13:19:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:27 smithi167 ceph-mon[33178]: 6.5 scrub ok 2024-09-13T13:19:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:27 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:19:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:27 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:19:27.870 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:27.890 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 87 16 114081 114065 81.9252 53.6875 0.0184144 0.0122018 2024-09-13T13:19:27.905 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:27.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:27.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:28.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:28.038 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:28.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:28.139 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:28.167 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:28.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:28.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:28.269 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:28.312 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:28.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:28.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:28.414 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:28.460 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:28.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:28.562 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:28.596 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:28.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:28.698 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:28.738 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:28.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:28 smithi052 ceph-mon[29471]: osdmap e460: 8 total, 8 up, 8 in 2024-09-13T13:19:28.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:28 smithi052 ceph-mon[29471]: 11.f scrub ok 2024-09-13T13:19:28.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:28 smithi052 ceph-mon[29471]: 4.8 scrub starts 2024-09-13T13:19:28.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:28 smithi052 ceph-mon[29471]: 4.8 scrub ok 2024-09-13T13:19:28.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:28 smithi052 ceph-mon[36682]: osdmap e460: 8 total, 8 up, 8 in 2024-09-13T13:19:28.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:28 smithi052 ceph-mon[36682]: 11.f scrub ok 2024-09-13T13:19:28.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:28 smithi052 ceph-mon[36682]: 4.8 scrub starts 2024-09-13T13:19:28.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:28 smithi052 ceph-mon[36682]: 4.8 scrub ok 2024-09-13T13:19:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:28.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:28.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:28 smithi167 ceph-mon[33178]: osdmap e460: 8 total, 8 up, 8 in 2024-09-13T13:19:28.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:28 smithi167 ceph-mon[33178]: 11.f scrub ok 2024-09-13T13:19:28.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:28 smithi167 ceph-mon[33178]: 4.8 scrub starts 2024-09-13T13:19:28.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:28 smithi167 ceph-mon[33178]: 4.8 scrub ok 2024-09-13T13:19:28.841 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:28.868 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:28.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:28.890 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 88 16 115017 115001 81.6589 58.5 0.0321686 0.0122353 2024-09-13T13:19:28.970 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:29.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:29.005 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:29.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:29.106 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:29.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:29.150 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:29.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:29.251 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:29.289 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:29.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:29.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:29.391 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:29.431 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:29.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:29.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:29.563 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:29.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:29.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:29.665 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:29.697 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:29.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:29.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:29 smithi052 ceph-mon[29471]: pgmap v929: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 157 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 70321/337257 objects misplaced (20.851%); 17 MiB/s, 268 objects/s recovering 2024-09-13T13:19:29.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:29 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:19:29.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:29 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:19:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:29 smithi052 ceph-mon[36682]: pgmap v929: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 157 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 70321/337257 objects misplaced (20.851%); 17 MiB/s, 268 objects/s recovering 2024-09-13T13:19:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:29 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:19:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:29 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:19:29.799 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:29.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:29 smithi167 ceph-mon[33178]: pgmap v929: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 157 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 70321/337257 objects misplaced (20.851%); 17 MiB/s, 268 objects/s recovering 2024-09-13T13:19:29.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:29 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:19:29.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:29 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:19:29.838 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:29.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:29.891 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 89 16 116095 116079 81.4984 67.375 0.020501 0.0122624 2024-09-13T13:19:29.939 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:29.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:29.977 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:29.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:30.078 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:30.123 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:30.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:30.225 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:30.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:30.254 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:30.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:30.355 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:30.393 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:30.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:30.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:30.496 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:30.523 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:30.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:30.625 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:30.658 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:30.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:30.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:30.792 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:30.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:30.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:30 smithi167 ceph-mon[33178]: 4.a scrub starts 2024-09-13T13:19:30.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:30 smithi167 ceph-mon[33178]: 4.a scrub ok 2024-09-13T13:19:30.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:30 smithi167 ceph-mon[33178]: osdmap e461: 8 total, 8 up, 8 in 2024-09-13T13:19:30.891 INFO:tasks.radosbench.radosbench.0.smithi052.stdout: 90 16 117006 116990 81.2254 56.9375 0.00863272 0.0123002 2024-09-13T13:19:30.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:30.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:30.916 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:30.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:30.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:30 smithi052 ceph-mon[36682]: 4.a scrub starts 2024-09-13T13:19:30.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:30 smithi052 ceph-mon[36682]: 4.a scrub ok 2024-09-13T13:19:30.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:30 smithi052 ceph-mon[36682]: osdmap e461: 8 total, 8 up, 8 in 2024-09-13T13:19:30.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:30 smithi052 ceph-mon[29471]: 4.a scrub starts 2024-09-13T13:19:30.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:30 smithi052 ceph-mon[29471]: 4.a scrub ok 2024-09-13T13:19:30.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:30 smithi052 ceph-mon[29471]: osdmap e461: 8 total, 8 up, 8 in 2024-09-13T13:19:31.018 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:31.049 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:31.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:31.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:31.151 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:31.182 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:31.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:31.285 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:31.307 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:31.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:31.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:31.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:31.432 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:31.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:31.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:31.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:31.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:31.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:31.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:31.679 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:31.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:31.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:31.781 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:31.805 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:31.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:31.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:31 smithi167 ceph-mon[33178]: pgmap v931: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 71950/344556 objects misplaced (20.882%); 16 MiB/s, 258 objects/s recovering 2024-09-13T13:19:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:31 smithi167 ceph-mon[33178]: 5.f scrub starts 2024-09-13T13:19:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:31 smithi167 ceph-mon[33178]: 5.f scrub ok 2024-09-13T13:19:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:31 smithi167 ceph-mon[33178]: 3.4 scrub starts 2024-09-13T13:19:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:31 smithi167 ceph-mon[33178]: 3.4 scrub ok 2024-09-13T13:19:31.891 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Total time run: 90.0489 2024-09-13T13:19:31.891 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Total writes made: 117006 2024-09-13T13:19:31.891 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Write size: 65536 2024-09-13T13:19:31.891 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Object size: 65536 2024-09-13T13:19:31.891 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Bandwidth (MB/sec): 81.21 2024-09-13T13:19:31.891 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev Bandwidth: 35.6537 2024-09-13T13:19:31.891 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max bandwidth (MB/sec): 209.25 2024-09-13T13:19:31.892 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min bandwidth (MB/sec): 0 2024-09-13T13:19:31.892 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Average IOPS: 1299 2024-09-13T13:19:31.892 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev IOPS: 570.459 2024-09-13T13:19:31.892 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max IOPS: 3348 2024-09-13T13:19:31.892 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min IOPS: 0 2024-09-13T13:19:31.892 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Average Latency(s): 0.0123093 2024-09-13T13:19:31.892 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Stddev Latency(s): 0.0311227 2024-09-13T13:19:31.892 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Max latency(s): 2.16136 2024-09-13T13:19:31.892 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Min latency(s): 0.00125067 2024-09-13T13:19:31.896 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T13:19:31.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:31.938 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:31.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:31 smithi052 ceph-mon[36682]: pgmap v931: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 71950/344556 objects misplaced (20.882%); 16 MiB/s, 258 objects/s recovering 2024-09-13T13:19:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:31 smithi052 ceph-mon[36682]: 5.f scrub starts 2024-09-13T13:19:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:31 smithi052 ceph-mon[36682]: 5.f scrub ok 2024-09-13T13:19:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:31 smithi052 ceph-mon[36682]: 3.4 scrub starts 2024-09-13T13:19:31.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:31 smithi052 ceph-mon[36682]: 3.4 scrub ok 2024-09-13T13:19:31.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:31 smithi052 ceph-mon[29471]: pgmap v931: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 71950/344556 objects misplaced (20.882%); 16 MiB/s, 258 objects/s recovering 2024-09-13T13:19:31.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:31 smithi052 ceph-mon[29471]: 5.f scrub starts 2024-09-13T13:19:31.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:31 smithi052 ceph-mon[29471]: 5.f scrub ok 2024-09-13T13:19:31.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:31 smithi052 ceph-mon[29471]: 3.4 scrub starts 2024-09-13T13:19:31.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:31 smithi052 ceph-mon[29471]: 3.4 scrub ok 2024-09-13T13:19:32.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:32.071 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:32.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:32.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:32.173 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:32.213 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:32.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:32.314 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:32.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:32.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:32.446 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:32.487 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:32.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:32.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:32.590 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:32.626 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:32.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:32.728 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:32.767 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:32.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:32 smithi167 ceph-mon[33178]: 3.1c scrub starts 2024-09-13T13:19:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:32 smithi167 ceph-mon[33178]: 3.1c scrub ok 2024-09-13T13:19:32.868 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:32.900 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:32.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:32 smithi052 ceph-mon[36682]: 3.1c scrub starts 2024-09-13T13:19:32.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:32 smithi052 ceph-mon[36682]: 3.1c scrub ok 2024-09-13T13:19:32.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:32 smithi052 ceph-mon[29471]: 3.1c scrub starts 2024-09-13T13:19:32.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:32 smithi052 ceph-mon[29471]: 3.1c scrub ok 2024-09-13T13:19:33.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:33.037 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:33.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:33.138 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:33.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:33.165 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:33.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:33.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:33.301 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:33.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:33.403 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:33.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:33.434 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:33.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:33.536 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:33.571 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:33.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:33.672 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:33.702 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:33.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:33.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:33.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:33 smithi167 ceph-mon[33178]: pgmap v932: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 52 MiB/s wr, 837 op/s; 72202/346080 objects misplaced (20.863%); 11 MiB/s, 180 objects/s recovering 2024-09-13T13:19:33.826 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:33.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:33.928 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:33.974 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:33.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:33.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:33 smithi052 ceph-mon[29471]: pgmap v932: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 52 MiB/s wr, 837 op/s; 72202/346080 objects misplaced (20.863%); 11 MiB/s, 180 objects/s recovering 2024-09-13T13:19:33.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:33 smithi052 ceph-mon[36682]: pgmap v932: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 52 MiB/s wr, 837 op/s; 72202/346080 objects misplaced (20.863%); 11 MiB/s, 180 objects/s recovering 2024-09-13T13:19:34.075 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:34.113 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:34.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:34.215 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:34.245 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:34.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:34.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:34.383 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:34.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:34.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:34.485 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:34.526 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:34.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:34.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:34.628 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:34.657 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:34.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:34.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:34.759 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:34.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:34.805 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:34.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:34.906 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:34.945 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:34.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:34.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:35.047 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:35.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:19:34 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:19:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:35.077 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:35.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:35.178 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:35.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:35.285 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T13:19:35.326 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:35.350 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:35.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:35.452 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:35.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[36682]: 11.c scrub starts 2024-09-13T13:19:35.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:19:35.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:19:35.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[36682]: pgmap v933: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 48 MiB/s wr, 860 op/s; 70545/348039 objects misplaced (20.269%); 17 MiB/s, 278 objects/s recovering 2024-09-13T13:19:35.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:19:35.492 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:19:35.493 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[36682]: osdmap e462: 8 total, 8 up, 8 in 2024-09-13T13:19:35.493 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:35.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[29471]: 11.c scrub starts 2024-09-13T13:19:35.493 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:19:35.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:19:35.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[29471]: pgmap v933: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 48 MiB/s wr, 860 op/s; 70545/348039 objects misplaced (20.269%); 17 MiB/s, 278 objects/s recovering 2024-09-13T13:19:35.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:19:35.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:19:35.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[29471]: osdmap e462: 8 total, 8 up, 8 in 2024-09-13T13:19:35.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:35.506 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:35.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:35.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:35 smithi167 ceph-mon[33178]: 11.c scrub starts 2024-09-13T13:19:35.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:35 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:19:35.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:35 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:19:35.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:35 smithi167 ceph-mon[33178]: pgmap v933: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 48 MiB/s wr, 860 op/s; 70545/348039 objects misplaced (20.269%); 17 MiB/s, 278 objects/s recovering 2024-09-13T13:19:35.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:35 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:19:35.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:35 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:19:35.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:35 smithi167 ceph-mon[33178]: osdmap e462: 8 total, 8 up, 8 in 2024-09-13T13:19:35.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:35.609 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:35.664 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:35.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:35.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:35.766 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:35.809 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:35.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:35.911 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:35.942 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:35.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:35.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:36.044 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:36.066 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:36.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:36.167 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:36.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:36.215 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:36.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:36.316 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:36.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:36.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:36.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:36.446 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:36.468 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:36.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:36.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:36.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:36 smithi052 ceph-mon[36682]: 4.f scrub starts 2024-09-13T13:19:36.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:36 smithi052 ceph-mon[36682]: 4.f scrub ok 2024-09-13T13:19:36.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:36 smithi052 ceph-mon[36682]: 6.1c scrub starts 2024-09-13T13:19:36.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:36 smithi052 ceph-mon[36682]: 6.1c scrub ok 2024-09-13T13:19:36.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:36 smithi052 ceph-mon[29471]: 4.f scrub starts 2024-09-13T13:19:36.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:36 smithi052 ceph-mon[29471]: 4.f scrub ok 2024-09-13T13:19:36.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:36 smithi052 ceph-mon[29471]: 6.1c scrub starts 2024-09-13T13:19:36.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:36 smithi052 ceph-mon[29471]: 6.1c scrub ok 2024-09-13T13:19:36.491 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:19:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:19:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:19:36.570 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:36 smithi167 ceph-mon[33178]: 4.f scrub starts 2024-09-13T13:19:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:36 smithi167 ceph-mon[33178]: 4.f scrub ok 2024-09-13T13:19:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:36 smithi167 ceph-mon[33178]: 6.1c scrub starts 2024-09-13T13:19:36.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:36 smithi167 ceph-mon[33178]: 6.1c scrub ok 2024-09-13T13:19:36.593 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:36.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:36.695 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:36.782 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:36.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:36.883 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:36.908 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:36.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:36.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:36.990 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:36.990 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:37.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:37.043 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:37.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:37.146 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:37.147 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:37 smithi052 ceph-mon[29471]: pgmap v935: 168 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 157 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 37 MiB/s wr, 800 op/s; 68571/346554 objects misplaced (19.787%); 17 MiB/s, 274 objects/s recovering 2024-09-13T13:19:37.147 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:37 smithi052 ceph-mon[29471]: 11.6 scrub starts 2024-09-13T13:19:37.202 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:37.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:37.304 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:37.344 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:37.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:37.446 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:37.483 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:37.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:37.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:37 smithi052 ceph-mon[36682]: pgmap v935: 168 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 157 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 37 MiB/s wr, 800 op/s; 68571/346554 objects misplaced (19.787%); 17 MiB/s, 274 objects/s recovering 2024-09-13T13:19:37.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:37 smithi052 ceph-mon[36682]: 11.6 scrub starts 2024-09-13T13:19:37.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:37 smithi052 ceph-mon[36682]: 11.5 scrub starts 2024-09-13T13:19:37.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:37 smithi052 ceph-mon[29471]: 11.5 scrub starts 2024-09-13T13:19:37.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:37 smithi167 ceph-mon[33178]: pgmap v935: 168 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 157 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 37 MiB/s wr, 800 op/s; 68571/346554 objects misplaced (19.787%); 17 MiB/s, 274 objects/s recovering 2024-09-13T13:19:37.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:37 smithi167 ceph-mon[33178]: 11.6 scrub starts 2024-09-13T13:19:37.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:37 smithi167 ceph-mon[33178]: 11.5 scrub starts 2024-09-13T13:19:37.585 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:37.607 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:37.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:37.709 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:37.752 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:37.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:37.854 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:37.892 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:37.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:37.995 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:38.029 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:38.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:38.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:38.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:38.155 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:38.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:38.257 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:38.297 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:38.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:38.398 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:38.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:38.428 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:38.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[36682]: 11.c scrub ok 2024-09-13T13:19:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[36682]: 11.1 scrub starts 2024-09-13T13:19:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[36682]: 11.8 scrub starts 2024-09-13T13:19:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[36682]: osdmap e463: 8 total, 8 up, 8 in 2024-09-13T13:19:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[36682]: 11.0 scrub starts 2024-09-13T13:19:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[36682]: 11.5 scrub ok 2024-09-13T13:19:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[36682]: 11.7 scrub starts 2024-09-13T13:19:38.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[36682]: 11.6 scrub ok 2024-09-13T13:19:38.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[36682]: osdmap e464: 8 total, 8 up, 8 in 2024-09-13T13:19:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[29471]: 11.c scrub ok 2024-09-13T13:19:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[29471]: 11.1 scrub starts 2024-09-13T13:19:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[29471]: 11.8 scrub starts 2024-09-13T13:19:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[29471]: osdmap e463: 8 total, 8 up, 8 in 2024-09-13T13:19:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[29471]: 11.0 scrub starts 2024-09-13T13:19:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[29471]: 11.5 scrub ok 2024-09-13T13:19:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[29471]: 11.7 scrub starts 2024-09-13T13:19:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[29471]: 11.6 scrub ok 2024-09-13T13:19:38.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:38 smithi052 ceph-mon[29471]: osdmap e464: 8 total, 8 up, 8 in 2024-09-13T13:19:38.529 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:38.569 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:38.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:38.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:38 smithi167 ceph-mon[33178]: 11.c scrub ok 2024-09-13T13:19:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:38 smithi167 ceph-mon[33178]: 11.1 scrub starts 2024-09-13T13:19:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:38 smithi167 ceph-mon[33178]: 11.8 scrub starts 2024-09-13T13:19:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:38 smithi167 ceph-mon[33178]: osdmap e463: 8 total, 8 up, 8 in 2024-09-13T13:19:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:38 smithi167 ceph-mon[33178]: 11.0 scrub starts 2024-09-13T13:19:38.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:38 smithi167 ceph-mon[33178]: 11.5 scrub ok 2024-09-13T13:19:38.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:38 smithi167 ceph-mon[33178]: 11.7 scrub starts 2024-09-13T13:19:38.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:38 smithi167 ceph-mon[33178]: 11.6 scrub ok 2024-09-13T13:19:38.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:38 smithi167 ceph-mon[33178]: osdmap e464: 8 total, 8 up, 8 in 2024-09-13T13:19:38.672 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:38.688 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:19:38.700 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:38.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:38.802 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:38.823 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:38.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:38.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:38.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:38.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:38.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:38.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:39.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:39.078 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:39.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:39.180 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:39.226 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:39.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:39.327 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:39.349 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:39.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:39.450 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:39.484 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:39.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:39.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[36682]: 11.1 scrub ok 2024-09-13T13:19:39.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[36682]: 11.4 scrub starts 2024-09-13T13:19:39.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[36682]: 11.8 scrub ok 2024-09-13T13:19:39.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[36682]: pgmap v937: 168 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 157 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 192 op/s; 68311/342252 objects misplaced (19.959%); 14 MiB/s, 220 objects/s recovering 2024-09-13T13:19:39.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[36682]: 11.9 scrub starts 2024-09-13T13:19:39.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[36682]: 11.4 scrub ok 2024-09-13T13:19:39.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[36682]: 11.7 scrub ok 2024-09-13T13:19:39.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[36682]: osdmap e465: 8 total, 8 up, 8 in 2024-09-13T13:19:39.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[29471]: 11.1 scrub ok 2024-09-13T13:19:39.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[29471]: 11.4 scrub starts 2024-09-13T13:19:39.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[29471]: 11.8 scrub ok 2024-09-13T13:19:39.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[29471]: pgmap v937: 168 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 157 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 192 op/s; 68311/342252 objects misplaced (19.959%); 14 MiB/s, 220 objects/s recovering 2024-09-13T13:19:39.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[29471]: 11.9 scrub starts 2024-09-13T13:19:39.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[29471]: 11.4 scrub ok 2024-09-13T13:19:39.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[29471]: 11.7 scrub ok 2024-09-13T13:19:39.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:39 smithi052 ceph-mon[29471]: osdmap e465: 8 total, 8 up, 8 in 2024-09-13T13:19:39.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:39 smithi167 ceph-mon[33178]: 11.1 scrub ok 2024-09-13T13:19:39.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:39 smithi167 ceph-mon[33178]: 11.4 scrub starts 2024-09-13T13:19:39.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:39 smithi167 ceph-mon[33178]: 11.8 scrub ok 2024-09-13T13:19:39.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:39 smithi167 ceph-mon[33178]: pgmap v937: 168 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 157 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 192 op/s; 68311/342252 objects misplaced (19.959%); 14 MiB/s, 220 objects/s recovering 2024-09-13T13:19:39.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:39 smithi167 ceph-mon[33178]: 11.9 scrub starts 2024-09-13T13:19:39.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:39 smithi167 ceph-mon[33178]: 11.4 scrub ok 2024-09-13T13:19:39.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:39 smithi167 ceph-mon[33178]: 11.7 scrub ok 2024-09-13T13:19:39.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:39 smithi167 ceph-mon[33178]: osdmap e465: 8 total, 8 up, 8 in 2024-09-13T13:19:39.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:39.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:39.620 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:39.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:39.721 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:39.744 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:39.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:39.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:39.845 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:39.896 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:39.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:39.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:39.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:40.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:40.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:40.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:40.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:40.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:40.171 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:40.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:40.272 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:40.304 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:40.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:40.406 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:40.437 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:40.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:40.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[36682]: 11.0 scrub ok 2024-09-13T13:19:40.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[36682]: 11.18 scrub starts 2024-09-13T13:19:40.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[36682]: 11.3 scrub starts 2024-09-13T13:19:40.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[36682]: 5.16 scrub starts 2024-09-13T13:19:40.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[36682]: 5.16 scrub ok 2024-09-13T13:19:40.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[36682]: 11.9 scrub ok 2024-09-13T13:19:40.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[36682]: 11.19 scrub starts 2024-09-13T13:19:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[29471]: 11.0 scrub ok 2024-09-13T13:19:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[29471]: 11.18 scrub starts 2024-09-13T13:19:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[29471]: 11.3 scrub starts 2024-09-13T13:19:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[29471]: 5.16 scrub starts 2024-09-13T13:19:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[29471]: 5.16 scrub ok 2024-09-13T13:19:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[29471]: 11.9 scrub ok 2024-09-13T13:19:40.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:40 smithi052 ceph-mon[29471]: 11.19 scrub starts 2024-09-13T13:19:40.540 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:40.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:40 smithi167 ceph-mon[33178]: 11.0 scrub ok 2024-09-13T13:19:40.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:40 smithi167 ceph-mon[33178]: 11.18 scrub starts 2024-09-13T13:19:40.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:40 smithi167 ceph-mon[33178]: 11.3 scrub starts 2024-09-13T13:19:40.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:40 smithi167 ceph-mon[33178]: 5.16 scrub starts 2024-09-13T13:19:40.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:40 smithi167 ceph-mon[33178]: 5.16 scrub ok 2024-09-13T13:19:40.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:40 smithi167 ceph-mon[33178]: 11.9 scrub ok 2024-09-13T13:19:40.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:40 smithi167 ceph-mon[33178]: 11.19 scrub starts 2024-09-13T13:19:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:40.590 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:40.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:40.691 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:40.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:40.730 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:40.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:40.831 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:40.871 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:40.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:40.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:40.990 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:41.027 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:41.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:41.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:41.129 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:41.169 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:41.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:41.271 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:41.312 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:41.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:41.414 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:41.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:41.460 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:41.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:41.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[36682]: 11.18 scrub ok 2024-09-13T13:19:41.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[36682]: 11.3 scrub ok 2024-09-13T13:19:41.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[36682]: 6.7 scrub starts 2024-09-13T13:19:41.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[36682]: 6.7 scrub ok 2024-09-13T13:19:41.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[36682]: pgmap v940: 168 pgs: 1 active+clean+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s; 64561/335079 objects misplaced (19.267%); 21 MiB/s, 334 objects/s recovering 2024-09-13T13:19:41.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:19:41.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[36682]: 11.19 scrub ok 2024-09-13T13:19:41.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[29471]: 11.18 scrub ok 2024-09-13T13:19:41.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[29471]: 11.3 scrub ok 2024-09-13T13:19:41.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[29471]: 6.7 scrub starts 2024-09-13T13:19:41.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[29471]: 6.7 scrub ok 2024-09-13T13:19:41.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[29471]: pgmap v940: 168 pgs: 1 active+clean+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s; 64561/335079 objects misplaced (19.267%); 21 MiB/s, 334 objects/s recovering 2024-09-13T13:19:41.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:19:41.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-mon[29471]: 11.19 scrub ok 2024-09-13T13:19:41.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:41 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a[29467]: 2024-09-13T13:19:41.177+0000 7effa8af8640 -1 mon.a@0(leader).osd e465 definitely_dead 0 2024-09-13T13:19:41.562 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:41.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:41 smithi167 ceph-mon[33178]: 11.18 scrub ok 2024-09-13T13:19:41.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:41 smithi167 ceph-mon[33178]: 11.3 scrub ok 2024-09-13T13:19:41.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:41 smithi167 ceph-mon[33178]: 6.7 scrub starts 2024-09-13T13:19:41.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:41 smithi167 ceph-mon[33178]: 6.7 scrub ok 2024-09-13T13:19:41.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:41 smithi167 ceph-mon[33178]: pgmap v940: 168 pgs: 1 active+clean+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s; 64561/335079 objects misplaced (19.267%); 21 MiB/s, 334 objects/s recovering 2024-09-13T13:19:41.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:41 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:19:41.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:41 smithi167 ceph-mon[33178]: 11.19 scrub ok 2024-09-13T13:19:41.616 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:41.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:41.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:41.718 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:41.752 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:41.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:41.853 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:41.902 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:41.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:41.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:42.004 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:42.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:42.032 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:42.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:42.133 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:42.182 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:42.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:42.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:42.187 INFO:teuthology.orchestra.run.smithi052.stderr:marked out osd.7. 2024-09-13T13:19:42.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:42.300 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:19:42.300 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:19:42.307 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:42.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:42.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:42.319 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:19:42.320 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:19:42.337 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:19:42.338 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:19:42.355 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:19:42.356 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:19:42.374 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:19:42.374 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:19:42.392 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:19:42.393 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:19:42.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:42.410 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:19:42.411 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:19:42.411 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2555521475' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T13:19:42.411 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[36682]: from='client.36791 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T13:19:42.411 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[36682]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-13T13:19:42.411 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:19:42.411 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:19:42.411 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:19:42.411 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:19:42.412 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:19:42.412 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2555521475' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T13:19:42.412 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[29471]: from='client.36791 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T13:19:42.412 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[29471]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-13T13:19:42.412 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:19:42.412 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:19:42.412 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:19:42.412 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:19:42.412 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:42 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:19:42.428 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:19:42.428 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:19:42.432 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:42.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:42.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:42.557 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:42.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:42.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:42 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2555521475' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T13:19:42.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:42 smithi167 ceph-mon[33178]: from='client.36791 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T13:19:42.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:42 smithi167 ceph-mon[33178]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-13T13:19:42.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:42 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:19:42.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:42 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:19:42.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:42 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:19:42.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:42 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:19:42.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:42 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:19:42.659 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:42.685 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:42.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:42.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:42.787 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:42.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:42.814 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:42.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:42.915 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:42.939 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:42.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:42.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:43.041 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:43.065 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:43.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:43.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:43.195 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:43.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:43.297 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:43.320 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:43.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:43.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:43.421 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:43.444 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:43.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:43.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:43.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[36682]: 1.8 scrub starts 2024-09-13T13:19:43.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[36682]: 1.8 scrub ok 2024-09-13T13:19:43.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[36682]: pgmap v941: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 63663/333411 objects misplaced (19.094%); 15 MiB/s, 238 objects/s recovering 2024-09-13T13:19:43.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[36682]: 3.b scrub starts 2024-09-13T13:19:43.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[36682]: 3.b scrub ok 2024-09-13T13:19:43.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[36682]: from='client.36791 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-13T13:19:43.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[36682]: osdmap e466: 8 total, 8 up, 7 in 2024-09-13T13:19:43.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1334096594' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:43.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:19:43.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:19:43.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[36682]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:43.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[36682]: osdmap e467: 8 total, 8 up, 7 in 2024-09-13T13:19:43.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[29471]: 1.8 scrub starts 2024-09-13T13:19:43.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[29471]: 1.8 scrub ok 2024-09-13T13:19:43.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[29471]: pgmap v941: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 63663/333411 objects misplaced (19.094%); 15 MiB/s, 238 objects/s recovering 2024-09-13T13:19:43.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[29471]: 3.b scrub starts 2024-09-13T13:19:43.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[29471]: 3.b scrub ok 2024-09-13T13:19:43.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[29471]: from='client.36791 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-13T13:19:43.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[29471]: osdmap e466: 8 total, 8 up, 7 in 2024-09-13T13:19:43.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1334096594' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:43.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:19:43.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:19:43.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[29471]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:43.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:43 smithi052 ceph-mon[29471]: osdmap e467: 8 total, 8 up, 7 in 2024-09-13T13:19:43.546 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:43.568 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:43.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:43.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:43 smithi167 ceph-mon[33178]: 1.8 scrub starts 2024-09-13T13:19:43.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:43 smithi167 ceph-mon[33178]: 1.8 scrub ok 2024-09-13T13:19:43.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:43 smithi167 ceph-mon[33178]: pgmap v941: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 63663/333411 objects misplaced (19.094%); 15 MiB/s, 238 objects/s recovering 2024-09-13T13:19:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:43 smithi167 ceph-mon[33178]: 3.b scrub starts 2024-09-13T13:19:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:43 smithi167 ceph-mon[33178]: 3.b scrub ok 2024-09-13T13:19:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:43 smithi167 ceph-mon[33178]: from='client.36791 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-13T13:19:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:43 smithi167 ceph-mon[33178]: osdmap e466: 8 total, 8 up, 7 in 2024-09-13T13:19:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:43 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1334096594' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:43 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:19:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:43 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:19:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:43 smithi167 ceph-mon[33178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:43.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:43 smithi167 ceph-mon[33178]: osdmap e467: 8 total, 8 up, 7 in 2024-09-13T13:19:43.671 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:43.693 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:43.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:43.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:43.795 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:43.819 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:43.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:43.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:43.921 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:43.943 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:43.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:43.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:44.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:44.069 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:44.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:44.171 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:44.197 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:44.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:44.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:44.299 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:44.341 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:44.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:44.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:44.443 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:44.474 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:44.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:44.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:44.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:44 smithi052 ceph-mon[36682]: 11.d scrub starts 2024-09-13T13:19:44.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:44 smithi052 ceph-mon[36682]: 11.11 continuing backfill to osd.2 from (427'400,465'6821] MIN to 465'6821 2024-09-13T13:19:44.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:44 smithi052 ceph-mon[36682]: 11.17 continuing backfill to osd.6 from (427'500,465'6784] 11:e804acc4:::benchmark_data_smithi052_182575_object80902:head to 465'6784 2024-09-13T13:19:44.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:44 smithi052 ceph-mon[36682]: 11.d scrub ok 2024-09-13T13:19:44.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:44 smithi052 ceph-mon[36682]: osdmap e468: 8 total, 8 up, 7 in 2024-09-13T13:19:44.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:44 smithi052 ceph-mon[29471]: 11.d scrub starts 2024-09-13T13:19:44.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:44 smithi052 ceph-mon[29471]: 11.11 continuing backfill to osd.2 from (427'400,465'6821] MIN to 465'6821 2024-09-13T13:19:44.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:44 smithi052 ceph-mon[29471]: 11.17 continuing backfill to osd.6 from (427'500,465'6784] 11:e804acc4:::benchmark_data_smithi052_182575_object80902:head to 465'6784 2024-09-13T13:19:44.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:44 smithi052 ceph-mon[29471]: 11.d scrub ok 2024-09-13T13:19:44.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:44 smithi052 ceph-mon[29471]: osdmap e468: 8 total, 8 up, 7 in 2024-09-13T13:19:44.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:44 smithi167 ceph-mon[33178]: 11.d scrub starts 2024-09-13T13:19:44.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:44 smithi167 ceph-mon[33178]: 11.11 continuing backfill to osd.2 from (427'400,465'6821] MIN to 465'6821 2024-09-13T13:19:44.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:44 smithi167 ceph-mon[33178]: 11.17 continuing backfill to osd.6 from (427'500,465'6784] 11:e804acc4:::benchmark_data_smithi052_182575_object80902:head to 465'6784 2024-09-13T13:19:44.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:44 smithi167 ceph-mon[33178]: 11.d scrub ok 2024-09-13T13:19:44.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:44 smithi167 ceph-mon[33178]: osdmap e468: 8 total, 8 up, 7 in 2024-09-13T13:19:44.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:44.605 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:44.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:44.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:44.707 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:44.738 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:44.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:44.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:44.873 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:44.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:44.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:44.975 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:45.021 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:45.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:45.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:45.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:19:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:19:45.124 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:45.153 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:45.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:45.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:45.255 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:45.297 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:45.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:45.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:45.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:45.431 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:45.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:45.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[36682]: pgmap v944: 168 pgs: 3 remapped+peering, 2 active+remapped+backfilling, 2 active+clean+scrubbing, 18 peering, 5 active+remapped+backfill_toofull, 138 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 861 B/s rd, 0 op/s; 57844/327939 objects misplaced (17.639%) 2024-09-13T13:19:45.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[36682]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 465'6945 2024-09-13T13:19:45.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[36682]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 465'7164 2024-09-13T13:19:45.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[36682]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 465'6810 2024-09-13T13:19:45.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[36682]: 11.19 starting backfill to osd.1 from (0'0,0'0] MAX to 465'6867 2024-09-13T13:19:45.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[36682]: 11.16 continuing backfill to osd.3 from (427'400,465'6832] MIN to 465'6832 2024-09-13T13:19:45.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[36682]: 11.16 continuing backfill to osd.5 from (427'400,465'6832] MIN to 465'6832 2024-09-13T13:19:45.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[36682]: 11.3 starting backfill to osd.5 from (0'0,0'0] MAX to 465'6989 2024-09-13T13:19:45.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[36682]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 465'7894 2024-09-13T13:19:45.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[36682]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 465'7933 2024-09-13T13:19:45.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:45.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[29471]: pgmap v944: 168 pgs: 3 remapped+peering, 2 active+remapped+backfilling, 2 active+clean+scrubbing, 18 peering, 5 active+remapped+backfill_toofull, 138 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 861 B/s rd, 0 op/s; 57844/327939 objects misplaced (17.639%) 2024-09-13T13:19:45.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[29471]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 465'6945 2024-09-13T13:19:45.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[29471]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 465'7164 2024-09-13T13:19:45.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[29471]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 465'6810 2024-09-13T13:19:45.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[29471]: 11.19 starting backfill to osd.1 from (0'0,0'0] MAX to 465'6867 2024-09-13T13:19:45.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[29471]: 11.16 continuing backfill to osd.3 from (427'400,465'6832] MIN to 465'6832 2024-09-13T13:19:45.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[29471]: 11.16 continuing backfill to osd.5 from (427'400,465'6832] MIN to 465'6832 2024-09-13T13:19:45.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[29471]: 11.3 starting backfill to osd.5 from (0'0,0'0] MAX to 465'6989 2024-09-13T13:19:45.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[29471]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 465'7894 2024-09-13T13:19:45.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[29471]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 465'7933 2024-09-13T13:19:45.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:45 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:45.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:45.569 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:45.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:45 smithi167 ceph-mon[33178]: pgmap v944: 168 pgs: 3 remapped+peering, 2 active+remapped+backfilling, 2 active+clean+scrubbing, 18 peering, 5 active+remapped+backfill_toofull, 138 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 861 B/s rd, 0 op/s; 57844/327939 objects misplaced (17.639%) 2024-09-13T13:19:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:45 smithi167 ceph-mon[33178]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 465'6945 2024-09-13T13:19:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:45 smithi167 ceph-mon[33178]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 465'7164 2024-09-13T13:19:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:45 smithi167 ceph-mon[33178]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 465'6810 2024-09-13T13:19:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:45 smithi167 ceph-mon[33178]: 11.19 starting backfill to osd.1 from (0'0,0'0] MAX to 465'6867 2024-09-13T13:19:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:45 smithi167 ceph-mon[33178]: 11.16 continuing backfill to osd.3 from (427'400,465'6832] MIN to 465'6832 2024-09-13T13:19:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:45 smithi167 ceph-mon[33178]: 11.16 continuing backfill to osd.5 from (427'400,465'6832] MIN to 465'6832 2024-09-13T13:19:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:45 smithi167 ceph-mon[33178]: 11.3 starting backfill to osd.5 from (0'0,0'0] MAX to 465'6989 2024-09-13T13:19:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:45 smithi167 ceph-mon[33178]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 465'7894 2024-09-13T13:19:45.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:45 smithi167 ceph-mon[33178]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 465'7933 2024-09-13T13:19:45.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:45 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:45.672 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:45.720 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:45.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:45.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:45.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:45.855 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:45.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:45.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:45.957 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:45.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:45.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:45.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:46.091 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:46.094 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:46.118 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:19:46.118 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:19:46.120 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:46.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:46.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:46.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:46.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:46.253 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:46.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:46.354 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:46.382 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:19:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:19:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:19:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:46.418 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:46.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:46.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:46.552 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:46.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:46.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:46.684 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:46.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:46.786 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:46.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:46.834 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:46.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:46.935 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:46.975 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:46.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:46.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:47.043 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T13:19:47.077 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:47.111 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:47.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:47.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:47.213 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:47.239 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:47.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:47.340 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:47.383 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:47.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:47.485 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:47.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:47 smithi052 ceph-mon[36682]: pgmap v946: 168 pgs: 3 remapped+peering, 5 active+remapped+backfilling, 1 active+clean+scrubbing, 18 peering, 5 active+remapped+backfill_toofull, 136 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 170 B/s rd, 0 op/s; 63609/324564 objects misplaced (19.598%) 2024-09-13T13:19:47.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:47 smithi052 ceph-mon[29471]: pgmap v946: 168 pgs: 3 remapped+peering, 5 active+remapped+backfilling, 1 active+clean+scrubbing, 18 peering, 5 active+remapped+backfill_toofull, 136 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 170 B/s rd, 0 op/s; 63609/324564 objects misplaced (19.598%) 2024-09-13T13:19:47.531 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:47.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:47.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:47 smithi167 ceph-mon[33178]: pgmap v946: 168 pgs: 3 remapped+peering, 5 active+remapped+backfilling, 1 active+clean+scrubbing, 18 peering, 5 active+remapped+backfill_toofull, 136 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 170 B/s rd, 0 op/s; 63609/324564 objects misplaced (19.598%) 2024-09-13T13:19:47.633 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:47.673 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:47.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:47.760 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:19:47.775 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:47.804 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:47.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:47.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:47.906 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:47.984 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:47.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:47.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:48.086 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:48.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:48.148 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:48.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:48.249 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:48.297 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:48.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:48.399 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:48.443 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:48.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:48.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:48.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:48 smithi052 ceph-mon[36682]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:48.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:48 smithi052 ceph-mon[29471]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:48.545 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:48.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:48 smithi167 ceph-mon[33178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:48.605 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:48.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:48.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:48.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:48.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:48.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:48.839 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:48.891 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:48.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:48.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:48.993 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:49.036 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:49.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:49.138 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:49.178 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:49.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:49.279 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:49.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:49.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:49.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:49.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:49.444 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:49.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:49.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:49.471 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:49 smithi167 ceph-mon[33178]: pgmap v947: 168 pgs: 3 remapped+peering, 5 active+remapped+backfilling, 1 active+clean+scrubbing, 18 peering, 7 active+remapped+backfill_toofull, 134 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 170 B/s rd, 0 op/s; 68728/323424 objects misplaced (21.250%) 2024-09-13T13:19:49.471 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:49 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:19:49.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:49 smithi052 ceph-mon[36682]: pgmap v947: 168 pgs: 3 remapped+peering, 5 active+remapped+backfilling, 1 active+clean+scrubbing, 18 peering, 7 active+remapped+backfill_toofull, 134 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 170 B/s rd, 0 op/s; 68728/323424 objects misplaced (21.250%) 2024-09-13T13:19:49.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:49 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:19:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:49 smithi052 ceph-mon[29471]: pgmap v947: 168 pgs: 3 remapped+peering, 5 active+remapped+backfilling, 1 active+clean+scrubbing, 18 peering, 7 active+remapped+backfill_toofull, 134 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 170 B/s rd, 0 op/s; 68728/323424 objects misplaced (21.250%) 2024-09-13T13:19:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:49 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:19:49.547 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:49.592 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:49.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:49.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:49.730 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:49.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:49.832 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:49.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:49.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:49.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:49.993 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:49.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:49.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:50.095 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:50.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:50.119 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:50.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:50.183 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 4, 2] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:19:50.183 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T13:19:50.221 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:50.279 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:50.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:50.382 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:50.414 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:50.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:50.486 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:19:50.486 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:50 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2677266111' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:19:50.487 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:50 smithi052 ceph-mon[36682]: from='client.36800 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:19:50.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:50 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2677266111' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:19:50.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:50 smithi052 ceph-mon[29471]: from='client.36800 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:19:50.516 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:50.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:50 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2677266111' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:19:50.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:50 smithi167 ceph-mon[33178]: from='client.36800 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:19:50.572 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:50.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:50.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:50.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:50.706 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:50.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:50.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:50.808 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:50.849 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:50.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:50.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:50.951 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:50.993 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:50.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:51.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:51.126 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:51.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:51.228 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:51.256 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:51.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:51.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:51.357 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:51.392 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:51.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:51.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:51 smithi052 ceph-mon[36682]: pgmap v948: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 153 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 782 B/s rd, 0 op/s; 2/312153 objects degraded (0.001%); 74571/312153 objects misplaced (23.889%) 2024-09-13T13:19:51.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:51 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:19:51.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:51 smithi052 ceph-mon[36682]: Health check failed: Degraded data redundancy: 2/312153 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:19:51.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:51 smithi052 ceph-mon[36682]: from='client.36800 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:19:51.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:51 smithi052 ceph-mon[36682]: osdmap e469: 8 total, 8 up, 7 in 2024-09-13T13:19:51.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:51 smithi052 ceph-mon[29471]: pgmap v948: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 153 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 782 B/s rd, 0 op/s; 2/312153 objects degraded (0.001%); 74571/312153 objects misplaced (23.889%) 2024-09-13T13:19:51.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:51 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:19:51.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:51 smithi052 ceph-mon[29471]: Health check failed: Degraded data redundancy: 2/312153 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:19:51.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:51 smithi052 ceph-mon[29471]: from='client.36800 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:19:51.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:51 smithi052 ceph-mon[29471]: osdmap e469: 8 total, 8 up, 7 in 2024-09-13T13:19:51.493 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:51.529 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:51.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:51.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:51.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:51 smithi167 ceph-mon[33178]: pgmap v948: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 153 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 782 B/s rd, 0 op/s; 2/312153 objects degraded (0.001%); 74571/312153 objects misplaced (23.889%) 2024-09-13T13:19:51.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:51 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:19:51.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:51 smithi167 ceph-mon[33178]: Health check failed: Degraded data redundancy: 2/312153 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:19:51.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:51 smithi167 ceph-mon[33178]: from='client.36800 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:19:51.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:51 smithi167 ceph-mon[33178]: osdmap e469: 8 total, 8 up, 7 in 2024-09-13T13:19:51.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:51.668 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:51.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:51.709 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:51.769 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:51.796 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:51.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:51.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:51.898 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:51.931 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:51.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:52.033 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:52.075 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:52.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:52.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:52.162 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:52.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:52.202 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:52.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:52.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:52.332 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:52.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:52.434 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:52.442 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:52.466 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:19:52.466 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:19:52.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:52.487 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:52.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:52.588 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:52.640 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:52.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:52.741 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:52.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:52.786 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:52.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:52.887 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:52.924 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:52.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:53.025 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:53.054 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:53.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:53.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:53.156 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:53.194 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:53.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:53.295 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:53.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:53.337 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:53.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:53.365 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T13:19:53.439 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:53.516 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:53.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:53.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:53.524 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:53 smithi052 ceph-mon[29471]: pgmap v950: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 153 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 243 op/s; 2/310506 objects degraded (0.001%); 73651/310506 objects misplaced (23.720%); 21 MiB/s, 336 objects/s recovering 2024-09-13T13:19:53.525 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:53 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:19:53.525 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:53 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:19:53.525 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:53 smithi052 ceph-mon[29471]: osdmap e470: 8 total, 8 up, 7 in 2024-09-13T13:19:53.525 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:53 smithi052 ceph-mon[29471]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:53.525 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:53 smithi052 ceph-mon[36682]: pgmap v950: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 153 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 243 op/s; 2/310506 objects degraded (0.001%); 73651/310506 objects misplaced (23.720%); 21 MiB/s, 336 objects/s recovering 2024-09-13T13:19:53.525 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:53 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:19:53.525 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:53 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:19:53.525 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:53 smithi052 ceph-mon[36682]: osdmap e470: 8 total, 8 up, 7 in 2024-09-13T13:19:53.525 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:53 smithi052 ceph-mon[36682]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:53 smithi167 ceph-mon[33178]: pgmap v950: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 153 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 243 op/s; 2/310506 objects degraded (0.001%); 73651/310506 objects misplaced (23.720%); 21 MiB/s, 336 objects/s recovering 2024-09-13T13:19:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:53 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:19:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:53 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:19:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:53 smithi167 ceph-mon[33178]: osdmap e470: 8 total, 8 up, 7 in 2024-09-13T13:19:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:53 smithi167 ceph-mon[33178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:19:53.618 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:53.665 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:53.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:53.758 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:19:53.766 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:53.896 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:53.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:53.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:54.022 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:54.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:54.123 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:54.186 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:54.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:54.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:54.229 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:19:54.229 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:19:54.229 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:19:54.229 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:19:54.229 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:19:54.229 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:19:54.277 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:19:54.277 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:19:54.288 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:54.317 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:19:54.317 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:19:54.338 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:19:54.367 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:19:54.367 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:19:54.389 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:54.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:54.424 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:54 smithi052 ceph-mon[36682]: osdmap e471: 8 total, 8 up, 7 in 2024-09-13T13:19:54.424 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:54 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2031350953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:54.424 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:19:54.424 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:19:54.427 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:54 smithi052 ceph-mon[29471]: osdmap e471: 8 total, 8 up, 7 in 2024-09-13T13:19:54.427 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:54 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2031350953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:54.467 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:19:54.467 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:19:54.494 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:54.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:54.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:54.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:54 smithi167 ceph-mon[33178]: osdmap e471: 8 total, 8 up, 7 in 2024-09-13T13:19:54.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:54 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2031350953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:19:54.622 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:54.651 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:54.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:54.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:54.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:54.798 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:54.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:54.899 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:54.944 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:54.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:55.046 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:55.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:19:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:19:55.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:55.085 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:55.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:55.186 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:55.224 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:55.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:55.288 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T13:19:55.326 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:55.368 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:55.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:55.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:55 smithi052 ceph-mon[29471]: 11.17 scrub starts 2024-09-13T13:19:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:55 smithi052 ceph-mon[29471]: pgmap v953: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 153 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s; 2/304230 objects degraded (0.001%); 71424/304230 objects misplaced (23.477%); 69 MiB/s, 1.10k objects/s recovering 2024-09-13T13:19:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:55 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2299917867' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:19:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:55 smithi052 ceph-mon[29471]: from='client.36806 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:19:55.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:55.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:55 smithi052 ceph-mon[36682]: 11.17 scrub starts 2024-09-13T13:19:55.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:55 smithi052 ceph-mon[36682]: pgmap v953: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 153 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s; 2/304230 objects degraded (0.001%); 71424/304230 objects misplaced (23.477%); 69 MiB/s, 1.10k objects/s recovering 2024-09-13T13:19:55.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:55 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2299917867' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:19:55.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:55 smithi052 ceph-mon[36682]: from='client.36806 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:19:55.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:55.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:55.502 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:55.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:55.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:55 smithi167 ceph-mon[33178]: 11.17 scrub starts 2024-09-13T13:19:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:55 smithi167 ceph-mon[33178]: pgmap v953: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 153 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s; 2/304230 objects degraded (0.001%); 71424/304230 objects misplaced (23.477%); 69 MiB/s, 1.10k objects/s recovering 2024-09-13T13:19:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:55 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2299917867' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:19:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:55 smithi167 ceph-mon[33178]: from='client.36806 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:19:55.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:19:55.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:55.630 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:55.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:55.732 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:55.773 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:55.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:55.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:55.875 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:55.924 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:55.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:56.025 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:56.051 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:56.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:56.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:56.153 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:56.179 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:56.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:56.281 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:56.313 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:56.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:56.415 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:56.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:56 smithi052 ceph-mon[29471]: from='client.36806 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:19:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:56 smithi052 ceph-mon[29471]: osdmap e472: 8 total, 8 up, 7 in 2024-09-13T13:19:56.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:56 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 2/300318 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:19:56.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:56 smithi052 ceph-mon[36682]: from='client.36806 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:19:56.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:56 smithi052 ceph-mon[36682]: osdmap e472: 8 total, 8 up, 7 in 2024-09-13T13:19:56.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:56 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 2/300318 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:19:56.490 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:19:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:19:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:19:56.548 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:56.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:56 smithi167 ceph-mon[33178]: from='client.36806 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:19:56.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:56 smithi167 ceph-mon[33178]: osdmap e472: 8 total, 8 up, 7 in 2024-09-13T13:19:56.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:56 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 2/300318 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:19:56.590 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:56.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:56.692 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:56.724 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:56.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:56.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:56.859 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:56.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:56.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:56.961 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:57.005 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:57.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:57.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:57.130 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:57.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:57.231 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:57.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:57.273 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:57.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:57.374 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:57.407 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:57.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:57.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:57.509 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:57.548 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:57.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:57.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:57.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:57 smithi167 ceph-mon[33178]: pgmap v955: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 153 active+clean; 6.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 882 B/s rd, 0 B/s wr, 680 op/s; 2/300318 objects degraded (0.001%); 67635/300318 objects misplaced (22.521%); 41 MiB/s, 662 objects/s recovering 2024-09-13T13:19:57.650 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:19:57.671 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:57 smithi052 ceph-mon[29471]: pgmap v955: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 153 active+clean; 6.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 882 B/s rd, 0 B/s wr, 680 op/s; 2/300318 objects degraded (0.001%); 67635/300318 objects misplaced (22.521%); 41 MiB/s, 662 objects/s recovering 2024-09-13T13:19:57.671 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:57 smithi052 ceph-mon[36682]: pgmap v955: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 153 active+clean; 6.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 882 B/s rd, 0 B/s wr, 680 op/s; 2/300318 objects degraded (0.001%); 67635/300318 objects misplaced (22.521%); 41 MiB/s, 662 objects/s recovering 2024-09-13T13:19:57.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:57.703 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:57.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:19:57.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:57.826 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:57.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:57.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:57.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:57.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:57.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:58.062 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:58.092 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:58.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:58.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:58.194 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:19:58.241 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:58.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:19:58.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:58.373 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:58.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:58.475 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:58.517 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:58.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:58.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:58.668 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:58.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:58.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:58.729 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:19:58.729 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:19:58.729 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:19:58.729 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:19:58.729 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.729 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189885 9.4 9:25913ce2:::benchmark_data_smithi052_128427_object59216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-09-13T13:19:58.729 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:19:58.729 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 434.75874559900001, 2024-09-13T13:19:58.730 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.048349118, 2024-09-13T13:19:58.730 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.730 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.730 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:19:58.730 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.730 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:19:58.730 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:19:58.730 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189885 2024-09-13T13:19:58.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.730 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.730 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.730 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.730 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909609+0000", 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0860000000000001e-06 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.731 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909614+0000", 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5199999999999999e-06 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909627+0000", 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3623e-05 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909634+0000", 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0210000000000002e-06 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.732 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909697+0000", 2024-09-13T13:19:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2769999999999997e-05 2024-09-13T13:19:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909840+0000", 2024-09-13T13:19:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014305399999999999 2024-09-13T13:19:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3,4", 2024-09-13T13:19:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909967+0000", 2024-09-13T13:19:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012707300000000001 2024-09-13T13:19:58.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.734 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.734 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.913962+0000", 2024-09-13T13:19:58.734 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039947009999999998 2024-09-13T13:19:58.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.734 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.734 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.913977+0000", 2024-09-13T13:19:58.734 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.506e-05 2024-09-13T13:19:58.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.734 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.735 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.916641+0000", 2024-09-13T13:19:58.735 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026644839999999999 2024-09-13T13:19:58.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.735 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.735 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.937520+0000", 2024-09-13T13:19:58.735 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.020878821999999998 2024-09-13T13:19:58.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.735 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.735 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.937533+0000", 2024-09-13T13:19:58.735 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2896e-05 2024-09-13T13:19:58.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.957873+0000", 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0203400570000001 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.957957+0000", 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3950999999999996e-05 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.736 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.737 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189915 9.4 9:2fa1035b:::benchmark_data_smithi052_128427_object59246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-09-13T13:19:58.737 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:19:58.737 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 434.725411775, 2024-09-13T13:19:58.737 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0238765249999999, 2024-09-13T13:19:58.737 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.737 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.737 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.737 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.737 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:19:58.737 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:19:58.737 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189915 2024-09-13T13:19:58.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.737 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942944+0000", 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.655e-06 2024-09-13T13:19:58.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942948+0000", 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3220000000000003e-06 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942957+0000", 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2609999999999999e-06 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942964+0000", 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6379999999999998e-06 2024-09-13T13:19:58.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.944838+0000", 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018746419999999999 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.944845+0000", 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958435+0000", 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0135895370000001 2024-09-13T13:19:58.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958549+0000", 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114144 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958677+0000", 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012798 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966592+0000", 2024-09-13T13:19:58.741 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0079149440000000001 2024-09-13T13:19:58.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.742 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.742 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966687+0000", 2024-09-13T13:19:58.742 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4741999999999995e-05 2024-09-13T13:19:58.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.742 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.742 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966725+0000", 2024-09-13T13:19:58.742 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7768000000000002e-05 2024-09-13T13:19:58.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.742 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966777+0000", 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2605999999999997e-05 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966818+0000", 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1279999999999998e-05 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189933 9.4 9:2eec938f:::benchmark_data_smithi052_128427_object59264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:19:58.743 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 434.71028317700001, 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.008927224, 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189933 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.744 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:19:58.745 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.745 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.745 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:19:58.745 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.745 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.745 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958072+0000", 2024-09-13T13:19:58.745 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7e-06 2024-09-13T13:19:58.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958078+0000", 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4990000000000002e-06 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958089+0000", 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0865e-05 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958095+0000", 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8279999999999997e-06 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.746 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958183+0000", 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7655e-05 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958191+0000", 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9529999999999999e-06 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961131+0000", 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0029402620000001 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.747 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961259+0000", 2024-09-13T13:19:58.748 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012816999999999999 2024-09-13T13:19:58.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.748 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:19:58.748 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961395+0000", 2024-09-13T13:19:58.748 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013549800000000001 2024-09-13T13:19:58.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.748 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.748 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966691+0000", 2024-09-13T13:19:58.748 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0052965979999999996 2024-09-13T13:19:58.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966864+0000", 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017277899999999999 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966899+0000", 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4377e-05 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966955+0000", 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6495e-05 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.749 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966998+0000", 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2545e-05 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189960 9.4 9:2cd09825:::benchmark_data_smithi052_128427_object59291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 434.68623770300002, 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99033151900000005, 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.750 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189960 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.751 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982118+0000", 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982123+0000", 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8389999999999997e-06 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.752 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.753 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982137+0000", 2024-09-13T13:19:58.753 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4433000000000001e-05 2024-09-13T13:19:58.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.753 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.753 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.753 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982145+0000", 2024-09-13T13:19:58.753 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8290000000000008e-06 2024-09-13T13:19:58.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.753 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.753 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.753 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982197+0000", 2024-09-13T13:19:58.753 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2431e-05 2024-09-13T13:19:58.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982205+0000", 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8630000000000008e-06 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964055+0000", 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98184965300000004 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964156+0000", 2024-09-13T13:19:58.754 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101055 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964261+0000", 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104961 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972104+0000", 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0078433300000000008 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972147+0000", 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3022000000000002e-05 2024-09-13T13:19:58.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972189+0000", 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1727000000000002e-05 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972384+0000", 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019545200000000001 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972447+0000", 2024-09-13T13:19:58.756 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3144000000000004e-05 2024-09-13T13:19:58.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.757 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.757 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.757 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189972 9.4 9:28374290:::benchmark_data_smithi052_128427_object59303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:19:58.757 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:19:58.757 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 434.677665537, 2024-09-13T13:19:58.757 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99084523700000005, 2024-09-13T13:19:58.757 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.757 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189972 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.758 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990690+0000", 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990694+0000", 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6029999999999998e-06 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.759 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990704+0000", 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5060000000000001e-06 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990710+0000", 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990764+0000", 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4252000000000002e-05 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.760 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990773+0000", 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2379999999999997e-06 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967034+0000", 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97626137000000002 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967153+0000", 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118855 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967267+0000", 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113855 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972462+0000", 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051953219999999996 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.976022+0000", 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003559563 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.762 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981419+0000", 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053977260000000003 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981488+0000", 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8950999999999998e-05 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981533+0000", 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4981999999999997e-05 2024-09-13T13:19:58.763 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202468 9.4 9:2e7e6063:::benchmark_data_smithi052_128427_object71799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.70244521900003, 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.030148343, 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:19:58.764 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202468 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.765 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965910+0000", 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2810000000000001e-06 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965915+0000", 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0749999999999997e-06 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965926+0000", 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1056999999999999e-05 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.766 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965943+0000", 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7178000000000001e-05 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.966053+0000", 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109768 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.966062+0000", 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4029999999999996e-06 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.767 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.971764+0000", 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0057021909999999 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.971894+0000", 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000129987 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.972041+0000", 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000146398 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.768 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.769 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976575+0000", 2024-09-13T13:19:58.769 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045341729999999998 2024-09-13T13:19:58.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.769 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.769 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976634+0000", 2024-09-13T13:19:58.769 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8996999999999998e-05 2024-09-13T13:19:58.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.769 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.769 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.995927+0000", 2024-09-13T13:19:58.769 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.019293457999999999 2024-09-13T13:19:58.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.996003+0000", 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5221000000000004e-05 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.996057+0000", 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4156000000000001e-05 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.770 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202476 9.4 9:205ce74f:::benchmark_data_smithi052_128427_object71807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.69547250599999, 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0253464960000001, 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202476 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:19:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972883+0000", 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.792e-06 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972888+0000", 2024-09-13T13:19:58.772 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7380000000000002e-06 2024-09-13T13:19:58.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.773 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.773 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.773 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972896+0000", 2024-09-13T13:19:58.773 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9230000000000002e-06 2024-09-13T13:19:58.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.777 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.777 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.777 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972903+0000", 2024-09-13T13:19:58.777 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5310000000000004e-06 2024-09-13T13:19:58.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.777 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.777 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.777 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.973002+0000", 2024-09-13T13:19:58.777 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8484000000000001e-05 2024-09-13T13:19:58.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.777 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.777 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.973012+0000", 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0016e-05 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974404+0000", 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0013926820000001 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974530+0000", 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012599 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.778 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974651+0000", 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012022400000000001 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976695+0000", 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020448139999999998 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981646+0000", 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049504229999999998 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.779 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998145+0000", 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.016499092 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998183+0000", 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8457999999999997e-05 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998228+0000", 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4329000000000002e-05 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202503 9.4 9:257a0b3a:::benchmark_data_smithi052_128427_object71834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.67642417100001, 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.009335557, 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202503 2024-09-13T13:19:58.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991936+0000", 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1149999999999999e-06 2024-09-13T13:19:58.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991941+0000", 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.463e-06 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991954+0000", 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2533e-05 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991961+0000", 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1690000000000003e-06 2024-09-13T13:19:58.783 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.992017+0000", 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5856999999999997e-05 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.992026+0000", 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976789+0000", 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98476255999999995 2024-09-13T13:19:58.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976898+0000", 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109548 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.977048+0000", 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015011899999999999 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981632+0000", 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045837020000000003 2024-09-13T13:19:58.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981701+0000", 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8511999999999997e-05 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001159+0000", 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.019457953 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001221+0000", 2024-09-13T13:19:58.786 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2229000000000003e-05 2024-09-13T13:19:58.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.787 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.787 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.787 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001265+0000", 2024-09-13T13:19:58.787 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4267999999999997e-05 2024-09-13T13:19:58.787 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.787 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.787 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.787 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.787 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202507 9.4 9:20f0d2e1:::benchmark_data_smithi052_128427_object71838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:19:58.787 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:19:58.787 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.67433902800002, 2024-09-13T13:19:58.788 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.007409886, 2024-09-13T13:19:58.788 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.788 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.788 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.788 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.788 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:19:58.788 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:19:58.788 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202507 2024-09-13T13:19:58.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.788 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.788 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.788 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.788 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994016+0000", 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.001e-06 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994020+0000", 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2320000000000003e-06 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994031+0000", 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1259e-05 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994036+0000", 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3580000000000002e-06 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.790 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994089+0000", 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2120000000000002e-05 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994097+0000", 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8000000000000004e-06 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979607+0000", 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98551001400000005 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.791 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979708+0000", 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000100108 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979813+0000", 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000105238 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981689+0000", 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018764299999999999 2024-09-13T13:19:58.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987043+0000", 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053537139999999999 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001312+0000", 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014268729000000001 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001378+0000", 2024-09-13T13:19:58.793 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5980000000000002e-05 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001424+0000", 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6903000000000001e-05 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202513 9.4 9:293cc3ac:::benchmark_data_smithi052_128427_object71844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 419.66888308599999, 2024-09-13T13:19:58.794 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0035790550000001, 2024-09-13T13:19:58.795 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.795 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.795 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.795 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.795 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:19:58.795 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:19:58.795 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202513 2024-09-13T13:19:58.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.795 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.795 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.795 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.795 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:19:58.795 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999472+0000", 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.536e-06 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.796 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999477+0000", 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7260000000000002e-06 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999492+0000", 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5495999999999999e-05 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999500+0000", 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3100000000000003e-06 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999557+0000", 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7491999999999998e-05 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999565+0000", 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9359999999999999e-06 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981724+0000", 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98215919600000001 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.798 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981838+0000", 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114092 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981966+0000", 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012748300000000001 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987109+0000", 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.005142771 2024-09-13T13:19:58.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987186+0000", 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7822000000000001e-05 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.002955+0000", 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015768880999999998 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003010+0000", 2024-09-13T13:19:58.800 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4959e-05 2024-09-13T13:19:58.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.801 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.801 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.801 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003050+0000", 2024-09-13T13:19:58.801 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9354999999999998e-05 2024-09-13T13:19:58.801 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.801 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.801 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.801 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.801 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:60365 11.0 11:088590ea:::benchmark_data_smithi052_182575_object60364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-13T13:19:58.801 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:18:33.473604+0000", 2024-09-13T13:19:58.801 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 84.194749454999993, 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.321431674, 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 60365 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.473604+0000", 2024-09-13T13:19:58.802 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.803 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.803 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.803 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.803 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.473604+0000", 2024-09-13T13:19:58.803 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.803 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.803 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.803 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.803 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.473606+0000", 2024-09-13T13:19:58.803 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1100000000000001e-06 2024-09-13T13:19:58.803 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.803 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.803 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.473631+0000", 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4805999999999999e-05 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.473646+0000", 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5223e-05 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.473654+0000", 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6190000000000002e-06 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.804 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.473777+0000", 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012313800000000001 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.474777+0000", 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010000250000000001 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.474978+0000", 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020041500000000001 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.805 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.480249+0000", 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0052712360000000003 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.482973+0000", 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0027247040000000001 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.794961+0000", 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.311987599 2024-09-13T13:19:58.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.795008+0000", 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6830999999999999e-05 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.795036+0000", 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7968000000000001e-05 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.808 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:60369 11.0 11:00a94002:::benchmark_data_smithi052_182575_object60368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-13T13:19:58.808 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:18:33.475442+0000", 2024-09-13T13:19:58.808 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 84.192911359999997, 2024-09-13T13:19:58.808 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.31968944999999999, 2024-09-13T13:19:58.808 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.808 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.808 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.808 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.808 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:58.808 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:58.808 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 60369 2024-09-13T13:19:58.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.808 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.809 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.809 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.475442+0000", 2024-09-13T13:19:58.809 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.809 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.809 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.475442+0000", 2024-09-13T13:19:58.809 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.809 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.809 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.475444+0000", 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0509999999999999e-06 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.475466+0000", 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.156e-05 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.475479+0000", 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3389e-05 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.810 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.475487+0000", 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5279999999999998e-06 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.477845+0000", 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023585749999999999 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.478065+0000", 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021975200000000001 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:19:58.811 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.478209+0000", 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000143938 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.482996+0000", 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047872799999999997 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.484345+0000", 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0013482839999999999 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.795070+0000", 2024-09-13T13:19:58.812 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.31072561300000001 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.795105+0000", 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4598999999999999e-05 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.795132+0000", 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6880999999999999e-05 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:60372 11.0 11:055cf520:::benchmark_data_smithi052_182575_object60371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:18:33.477960+0000", 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 84.190393287999996, 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.32125207300000003, 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 60372 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.814 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.477960+0000", 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.477960+0000", 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.477962+0000", 2024-09-13T13:19:58.815 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8899999999999999e-06 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.477988+0000", 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5605e-05 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.477997+0000", 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5699999999999999e-06 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.816 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.478004+0000", 2024-09-13T13:19:58.817 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7259999999999996e-06 2024-09-13T13:19:58.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.817 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.817 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.480399+0000", 2024-09-13T13:19:58.817 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023953540000000001 2024-09-13T13:19:58.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.817 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.817 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.480540+0000", 2024-09-13T13:19:58.817 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000140459 2024-09-13T13:19:58.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.480669+0000", 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012934599999999999 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.483943+0000", 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0032739800000000001 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.488117+0000", 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041734190000000003 2024-09-13T13:19:58.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.819 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.819 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.799153+0000", 2024-09-13T13:19:58.819 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.31103627099999998 2024-09-13T13:19:58.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.819 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.819 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.799188+0000", 2024-09-13T13:19:58.819 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5169999999999997e-05 2024-09-13T13:19:58.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.819 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.819 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.799212+0000", 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4283000000000001e-05 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:60399 11.f 11:f7529dfa:::benchmark_data_smithi052_182575_object60398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:18:33.501917+0000", 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 84.166436924999999, 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.29755051599999999, 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.820 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.821 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:58.821 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:58.821 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 60399 2024-09-13T13:19:58.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.821 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.821 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.821 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.501917+0000", 2024-09-13T13:19:58.821 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.821 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.821 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.501917+0000", 2024-09-13T13:19:58.822 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.822 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.822 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.822 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.501919+0000", 2024-09-13T13:19:58.822 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0820000000000001e-06 2024-09-13T13:19:58.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.822 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.822 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.822 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.501954+0000", 2024-09-13T13:19:58.822 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5108999999999999e-05 2024-09-13T13:19:58.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.822 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.501971+0000", 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6636000000000001e-05 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.501978+0000", 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.943e-06 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.502034+0000", 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5788999999999997e-05 2024-09-13T13:19:58.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.502160+0000", 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012569299999999999 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.502294+0000", 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000133814 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.504685+0000", 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00239157 2024-09-13T13:19:58.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.505120+0000", 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00043417 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.799377+0000", 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.29425727699999998 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.799428+0000", 2024-09-13T13:19:58.825 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1523000000000001e-05 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.799467+0000", 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8909999999999998e-05 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:60404 11.f 11:fa73bafd:::benchmark_data_smithi052_182575_object60403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:18:33.509755+0000", 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 84.158598764999994, 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.28991548499999997, 2024-09-13T13:19:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 60404 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.509755+0000", 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.827 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.828 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.509755+0000", 2024-09-13T13:19:58.828 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.828 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.828 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.828 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.509757+0000", 2024-09-13T13:19:58.828 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8160000000000001e-06 2024-09-13T13:19:58.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.828 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.828 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.828 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.509780+0000", 2024-09-13T13:19:58.828 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2997999999999999e-05 2024-09-13T13:19:58.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.829 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.829 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.509793+0000", 2024-09-13T13:19:58.829 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3455000000000001e-05 2024-09-13T13:19:58.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.829 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.829 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.509801+0000", 2024-09-13T13:19:58.829 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7659999999999998e-06 2024-09-13T13:19:58.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.829 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.509852+0000", 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1399e-05 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.509976+0000", 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000123535 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.510108+0000", 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000131922 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.830 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.512069+0000", 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019616059999999999 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.512644+0000", 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00057498499999999997 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.799581+0000", 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.28693624400000001 2024-09-13T13:19:58.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.832 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.832 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.832 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.799640+0000", 2024-09-13T13:19:58.832 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9715e-05 2024-09-13T13:19:58.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.832 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.832 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.832 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:18:33.799670+0000", 2024-09-13T13:19:58.832 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0043999999999999e-05 2024-09-13T13:19:58.832 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.832 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.832 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.833 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.833 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94527 11.0 11:04477f51:::benchmark_data_smithi052_182575_object94526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:19:58.833 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.196615+0000", 2024-09-13T13:19:58.833 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 53.471738203999998, 2024-09-13T13:19:58.833 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0158819889999999, 2024-09-13T13:19:58.833 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.833 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.833 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:19:58.833 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.833 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:58.833 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:58.833 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94527 2024-09-13T13:19:58.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196615+0000", 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196615+0000", 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.834 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196618+0000", 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6960000000000001e-06 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196651+0000", 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2749000000000001e-05 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196675+0000", 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3983000000000001e-05 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.835 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.836 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196687+0000", 2024-09-13T13:19:58.836 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1652e-05 2024-09-13T13:19:58.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.836 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.836 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.836 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.434487+0000", 2024-09-13T13:19:58.836 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.237800123 2024-09-13T13:19:58.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.836 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.836 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.836 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.434495+0000", 2024-09-13T13:19:58.836 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7439999999999996e-06 2024-09-13T13:19:58.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.212419+0000", 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.77792370300000002 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.212497+0000", 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8338999999999997e-05 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.837 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94541 11.10 11:0f9c248a:::benchmark_data_smithi052_182575_object94540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.196705+0000", 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 53.471648438999999, 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0441548679999999, 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94541 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.838 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196705+0000", 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196705+0000", 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196706+0000", 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3e-07 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.839 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.840 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196728+0000", 2024-09-13T13:19:58.840 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2404000000000001e-05 2024-09-13T13:19:58.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.840 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.840 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.840 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196736+0000", 2024-09-13T13:19:58.840 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.187e-06 2024-09-13T13:19:58.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.840 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.840 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:58.841 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196741+0000", 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.434669+0000", 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.23792806899999999 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.434676+0000", 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.842 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.220430+0000", 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.78575403499999996 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.220564+0000", 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013392499999999999 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.220848+0000", 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00028404 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.843 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.224043+0000", 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031949190000000001 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.231463+0000", 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0074202490000000003 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.240771+0000", 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0093081469999999993 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.844 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.240822+0000", 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0217999999999999e-05 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.240860+0000", 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8559999999999997e-05 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94542 11.10 11:09397811:::benchmark_data_smithi052_182575_object94541:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.196747+0000", 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 53.471606285999997, 2024-09-13T13:19:58.845 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0442601499999999, 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94542 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196747+0000", 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.846 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196747+0000", 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196748+0000", 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4799999999999998e-07 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196776+0000", 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7542000000000002e-05 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196782+0000", 2024-09-13T13:19:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.088e-06 2024-09-13T13:19:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196786+0000", 2024-09-13T13:19:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8389999999999997e-06 2024-09-13T13:19:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.434704+0000", 2024-09-13T13:19:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.23791717000000001 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.434712+0000", 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0290000000000005e-06 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.222512+0000", 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.78780074899999997 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:19:58.849 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.222642+0000", 2024-09-13T13:19:58.850 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000129429 2024-09-13T13:19:58.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.850 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:19:58.850 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.222915+0000", 2024-09-13T13:19:58.850 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027360500000000001 2024-09-13T13:19:58.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.850 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.850 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.226523+0000", 2024-09-13T13:19:58.850 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003608024 2024-09-13T13:19:58.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.231470+0000", 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049461469999999997 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.240916+0000", 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0094464400000000004 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.240975+0000", 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9135999999999998e-05 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.851 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.241008+0000", 2024-09-13T13:19:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2304000000000001e-05 2024-09-13T13:19:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95596 11.10 11:09078e1c:::benchmark_data_smithi052_182575_object95595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:19:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:07.182481+0000", 2024-09-13T13:19:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 50.485872143999998, 2024-09-13T13:19:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.131915893, 2024-09-13T13:19:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95596 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182481+0000", 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.853 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182481+0000", 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182484+0000", 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.931e-06 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182533+0000", 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8446000000000003e-05 2024-09-13T13:19:58.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.855 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.855 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182546+0000", 2024-09-13T13:19:58.855 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3392e-05 2024-09-13T13:19:58.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.855 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.855 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182558+0000", 2024-09-13T13:19:58.855 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1542000000000001e-05 2024-09-13T13:19:58.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.855 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.855 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.188517+0000", 2024-09-13T13:19:58.856 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0059587579999999998 2024-09-13T13:19:58.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.856 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.188527+0000", 2024-09-13T13:19:58.856 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0458e-05 2024-09-13T13:19:58.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.856 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.856 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.304374+0000", 2024-09-13T13:19:58.856 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.115846943 2024-09-13T13:19:58.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for degraded object", 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.304766+0000", 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00039178999999999999 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.314229+0000", 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0094630389999999995 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.314397+0000", 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016859399999999999 2024-09-13T13:19:58.857 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95599 11.10 11:0e938361:::benchmark_data_smithi052_182575_object95598:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:07.184322+0000", 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 50.484031995000002, 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.131418136, 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95599 2024-09-13T13:19:58.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184322+0000", 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184322+0000", 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184323+0000", 2024-09-13T13:19:58.859 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.243e-06 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184360+0000", 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.68e-05 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184372+0000", 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2177999999999999e-05 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184380+0000", 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6079999999999996e-06 2024-09-13T13:19:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.188555+0000", 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041741979999999996 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.188562+0000", 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2470000000000002e-06 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.304806+0000", 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.116244088 2024-09-13T13:19:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for degraded object", 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.305311+0000", 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00050533299999999995 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.315517+0000", 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010206125 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.315740+0000", 2024-09-13T13:19:58.862 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00022231599999999999 2024-09-13T13:19:58.863 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.863 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.863 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.863 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:19:58.863 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:19:58.863 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:19:58.870 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:58.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:58.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:58.972 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:58.997 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:58.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:58.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:19:59.099 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:19:59.135 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:59.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:19:59.237 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:19:59.268 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:59.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:59.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:19:59.369 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:59.397 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:59.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:59.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:59.499 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:19:59.529 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:59.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:59.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:19:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:19:59 smithi167 ceph-mon[33178]: pgmap v956: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 6.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 672 op/s; 2/298410 objects degraded (0.001%); 65587/298410 objects misplaced (21.979%); 53 MiB/s, 845 objects/s recovering 2024-09-13T13:19:59.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:19:59.658 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:19:59.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:59.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:19:59.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:19:59 smithi052 ceph-mon[36682]: pgmap v956: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 6.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 672 op/s; 2/298410 objects degraded (0.001%); 65587/298410 objects misplaced (21.979%); 53 MiB/s, 845 objects/s recovering 2024-09-13T13:19:59.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:19:59 smithi052 ceph-mon[29471]: pgmap v956: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 6.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 672 op/s; 2/298410 objects degraded (0.001%); 65587/298410 objects misplaced (21.979%); 53 MiB/s, 845 objects/s recovering 2024-09-13T13:19:59.759 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:19:59.804 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:59.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:59.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:19:59.906 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:19:59.943 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:19:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:19:59.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:00.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:00.078 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:00.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:00.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:00.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:00.206 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:00.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:00.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:00.308 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:00.341 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:00.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:00.442 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:00.486 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:00.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:00.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:00 smithi167 ceph-mon[33178]: Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull; Degraded data redundancy: 2/298410 objects degraded (0.001%), 1 pg degraded 2024-09-13T13:20:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:00 smithi167 ceph-mon[33178]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-13T13:20:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:00 smithi167 ceph-mon[33178]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull 2024-09-13T13:20:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:00 smithi167 ceph-mon[33178]: pg 11.3 is active+remapped+backfill_toofull, acting [6,1,7] 2024-09-13T13:20:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:00 smithi167 ceph-mon[33178]: pg 11.f is active+remapped+backfill_toofull, acting [1,2,7] 2024-09-13T13:20:00.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:00 smithi167 ceph-mon[33178]: pg 11.10 is active+remapped+backfill_toofull, acting [1,7,3] 2024-09-13T13:20:00.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:00 smithi167 ceph-mon[33178]: pg 11.11 is active+remapped+backfill_toofull, acting [0,3,6] 2024-09-13T13:20:00.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:00 smithi167 ceph-mon[33178]: pg 11.13 is active+remapped+backfill_toofull, acting [1,6,7] 2024-09-13T13:20:00.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:00 smithi167 ceph-mon[33178]: pg 11.14 is active+remapped+backfill_toofull, acting [6,2,7] 2024-09-13T13:20:00.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:00 smithi167 ceph-mon[33178]: pg 11.15 is active+remapped+backfill_toofull, acting [5,6,7] 2024-09-13T13:20:00.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:00 smithi167 ceph-mon[33178]: [WRN] PG_DEGRADED: Degraded data redundancy: 2/298410 objects degraded (0.001%), 1 pg degraded 2024-09-13T13:20:00.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:00 smithi167 ceph-mon[33178]: pg 11.12 is active+recovering+undersized+degraded+remapped, acting [2,0] 2024-09-13T13:20:00.588 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:00.628 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:00.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:00.730 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[29471]: Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull; Degraded data redundancy: 2/298410 objects degraded (0.001%), 1 pg degraded 2024-09-13T13:20:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[29471]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-13T13:20:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[29471]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull 2024-09-13T13:20:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[29471]: pg 11.3 is active+remapped+backfill_toofull, acting [6,1,7] 2024-09-13T13:20:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[29471]: pg 11.f is active+remapped+backfill_toofull, acting [1,2,7] 2024-09-13T13:20:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[29471]: pg 11.10 is active+remapped+backfill_toofull, acting [1,7,3] 2024-09-13T13:20:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[29471]: pg 11.11 is active+remapped+backfill_toofull, acting [0,3,6] 2024-09-13T13:20:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[29471]: pg 11.13 is active+remapped+backfill_toofull, acting [1,6,7] 2024-09-13T13:20:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[29471]: pg 11.14 is active+remapped+backfill_toofull, acting [6,2,7] 2024-09-13T13:20:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[29471]: pg 11.15 is active+remapped+backfill_toofull, acting [5,6,7] 2024-09-13T13:20:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[29471]: [WRN] PG_DEGRADED: Degraded data redundancy: 2/298410 objects degraded (0.001%), 1 pg degraded 2024-09-13T13:20:00.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[29471]: pg 11.12 is active+recovering+undersized+degraded+remapped, acting [2,0] 2024-09-13T13:20:00.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[36682]: Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull; Degraded data redundancy: 2/298410 objects degraded (0.001%), 1 pg degraded 2024-09-13T13:20:00.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[36682]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-13T13:20:00.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[36682]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull 2024-09-13T13:20:00.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[36682]: pg 11.3 is active+remapped+backfill_toofull, acting [6,1,7] 2024-09-13T13:20:00.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[36682]: pg 11.f is active+remapped+backfill_toofull, acting [1,2,7] 2024-09-13T13:20:00.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[36682]: pg 11.10 is active+remapped+backfill_toofull, acting [1,7,3] 2024-09-13T13:20:00.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[36682]: pg 11.11 is active+remapped+backfill_toofull, acting [0,3,6] 2024-09-13T13:20:00.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[36682]: pg 11.13 is active+remapped+backfill_toofull, acting [1,6,7] 2024-09-13T13:20:00.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[36682]: pg 11.14 is active+remapped+backfill_toofull, acting [6,2,7] 2024-09-13T13:20:00.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[36682]: pg 11.15 is active+remapped+backfill_toofull, acting [5,6,7] 2024-09-13T13:20:00.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[36682]: [WRN] PG_DEGRADED: Degraded data redundancy: 2/298410 objects degraded (0.001%), 1 pg degraded 2024-09-13T13:20:00.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:00 smithi052 ceph-mon[36682]: pg 11.12 is active+recovering+undersized+degraded+remapped, acting [2,0] 2024-09-13T13:20:00.768 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:00.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:00.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:00.870 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:00.912 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:00.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:00.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:01.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:01.042 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:01.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:01.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:01.144 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:01.175 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:01.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:01.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:01.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:01.308 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:01.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:01.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:01.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:01.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:01.436 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:01.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:01.537 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:01.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:01 smithi167 ceph-mon[33178]: pgmap v957: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 953 op/s; 2/288300 objects degraded (0.001%); 60555/288300 objects misplaced (21.004%); 62 MiB/s, 986 objects/s recovering 2024-09-13T13:20:01.579 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:01.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:01.680 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:01.723 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:01.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:01.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:01.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:01 smithi052 ceph-mon[29471]: pgmap v957: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 953 op/s; 2/288300 objects degraded (0.001%); 60555/288300 objects misplaced (21.004%); 62 MiB/s, 986 objects/s recovering 2024-09-13T13:20:01.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:01 smithi052 ceph-mon[36682]: pgmap v957: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 953 op/s; 2/288300 objects degraded (0.001%); 60555/288300 objects misplaced (21.004%); 62 MiB/s, 986 objects/s recovering 2024-09-13T13:20:01.825 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:01.866 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:01.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:01.968 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:02.016 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:02.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:02.118 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:02.163 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:02.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:02.264 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:02.294 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:02.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:02.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:02.396 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:02.441 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:02.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:02.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:02.543 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:02.583 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:02.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:02.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:02.685 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:02.719 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:02.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:02.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:02.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:02 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 2/286404 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:02.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:02 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 2/286404 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:02.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:02 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 2/286404 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:02.822 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:02.851 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:02.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:02.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:02.954 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:02.976 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:02.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:02.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:03.078 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:03.128 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:03.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:03.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:03.231 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:03.273 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:03.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:03.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:03.375 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:03.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:03.509 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:03.556 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:03.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:03.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:03.658 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:03.707 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:03.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:03.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:03 smithi052 ceph-mon[36682]: pgmap v958: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 749 op/s; 2/286404 objects degraded (0.001%); 59786/286404 objects misplaced (20.875%); 56 MiB/s, 897 objects/s recovering 2024-09-13T13:20:03.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:03 smithi052 ceph-mon[29471]: pgmap v958: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 749 op/s; 2/286404 objects degraded (0.001%); 59786/286404 objects misplaced (20.875%); 56 MiB/s, 897 objects/s recovering 2024-09-13T13:20:03.808 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:03.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:03 smithi167 ceph-mon[33178]: pgmap v958: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 749 op/s; 2/286404 objects degraded (0.001%); 59786/286404 objects misplaced (20.875%); 56 MiB/s, 897 objects/s recovering 2024-09-13T13:20:03.837 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:03.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:03.939 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:03.973 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:03.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:04.075 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:04.109 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:04.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:04.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:04.211 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:04.237 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:04.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:04.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:04.339 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:04.379 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:04.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:04.481 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:04.523 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:04.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:04.625 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:04.669 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:04.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:04.771 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:04.801 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:04.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:04.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:04.902 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:04.932 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:04.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:05.034 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:05.070 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:05.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:20:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:20:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:05.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:05.172 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:05.208 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:05.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:05.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:05.310 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:05.355 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:05.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:05.457 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:05.483 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:05.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:05.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:05.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:05 smithi052 ceph-mon[36682]: pgmap v959: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 810 op/s; 2/279948 objects degraded (0.001%); 57847/279948 objects misplaced (20.663%); 49 MiB/s, 784 objects/s recovering 2024-09-13T13:20:05.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:05 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:20:05.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:05 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:20:05.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:05 smithi052 ceph-mon[29471]: pgmap v959: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 810 op/s; 2/279948 objects degraded (0.001%); 57847/279948 objects misplaced (20.663%); 49 MiB/s, 784 objects/s recovering 2024-09-13T13:20:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:05 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:20:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:05 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:20:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:05 smithi167 ceph-mon[33178]: pgmap v959: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 810 op/s; 2/279948 objects degraded (0.001%); 57847/279948 objects misplaced (20.663%); 49 MiB/s, 784 objects/s recovering 2024-09-13T13:20:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:05 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:20:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:05 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:20:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:05.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:05.620 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:05.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:05.721 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:05.754 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:05.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:05.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:05.856 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:05.889 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:05.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:05.992 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:06.034 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:06.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:06.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:06.136 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:06.167 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:06.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:06.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:06.270 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:06.310 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:06.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:06.413 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:06.444 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:06.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:06.488 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:20:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:20:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:20:06.546 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:06.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:06.588 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:06.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:06.690 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:06.712 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:06.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:06.813 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:06.838 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:06.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:06.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:06.939 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:06.967 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:06.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:07.068 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:07.104 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:07.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:07.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:07.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:07.248 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:07.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:07.349 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:07.382 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:07.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:07.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:07.484 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:07.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:07 smithi052 ceph-mon[29471]: pgmap v960: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 952 B/s rd, 0 B/s wr, 777 op/s; 2/275265 objects degraded (0.001%); 54490/275265 objects misplaced (19.795%); 43 MiB/s, 685 objects/s recovering 2024-09-13T13:20:07.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:07 smithi052 ceph-mon[29471]: osdmap e473: 8 total, 8 up, 7 in 2024-09-13T13:20:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:07 smithi052 ceph-mon[36682]: pgmap v960: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 952 B/s rd, 0 B/s wr, 777 op/s; 2/275265 objects degraded (0.001%); 54490/275265 objects misplaced (19.795%); 43 MiB/s, 685 objects/s recovering 2024-09-13T13:20:07.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:07 smithi052 ceph-mon[36682]: osdmap e473: 8 total, 8 up, 7 in 2024-09-13T13:20:07.525 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:07.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:07.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:07 smithi167 ceph-mon[33178]: pgmap v960: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 952 B/s rd, 0 B/s wr, 777 op/s; 2/275265 objects degraded (0.001%); 54490/275265 objects misplaced (19.795%); 43 MiB/s, 685 objects/s recovering 2024-09-13T13:20:07.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:07 smithi167 ceph-mon[33178]: osdmap e473: 8 total, 8 up, 7 in 2024-09-13T13:20:07.627 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:07.672 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:07.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:07.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:07.821 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:07.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:07.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:07.923 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:07.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:07.958 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:07.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:08.059 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:08.088 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:08.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:08.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:08.190 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:08.233 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:08.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:08.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:08.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:08.382 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:08.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:08.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:08.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:08 smithi052 ceph-mon[36682]: osdmap e474: 8 total, 8 up, 7 in 2024-09-13T13:20:08.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:08 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 2/275265 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:08.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:08 smithi052 ceph-mon[29471]: osdmap e474: 8 total, 8 up, 7 in 2024-09-13T13:20:08.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:08 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 2/275265 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:08.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:08.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:08.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:08.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:08 smithi167 ceph-mon[33178]: osdmap e474: 8 total, 8 up, 7 in 2024-09-13T13:20:08.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:08 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 2/275265 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:08.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:08.661 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:08.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:08.762 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:08.803 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:08.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:08.905 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:08.936 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:08.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:09.038 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:09.065 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:09.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:09.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:09.201 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:09.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:09.302 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:09.325 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:09.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:09.427 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:09.467 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:09.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:09.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:09 smithi052 ceph-mon[36682]: pgmap v963: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 620 op/s; 2/273408 objects degraded (0.001%); 53651/273408 objects misplaced (19.623%); 28 MiB/s, 442 objects/s recovering 2024-09-13T13:20:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:09 smithi052 ceph-mon[29471]: pgmap v963: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 620 op/s; 2/273408 objects degraded (0.001%); 53651/273408 objects misplaced (19.623%); 28 MiB/s, 442 objects/s recovering 2024-09-13T13:20:09.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:09.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:09 smithi167 ceph-mon[33178]: pgmap v963: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 154 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 620 op/s; 2/273408 objects degraded (0.001%); 53651/273408 objects misplaced (19.623%); 28 MiB/s, 442 objects/s recovering 2024-09-13T13:20:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:09.610 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:09.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:09.711 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:09.748 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:09.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:09.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:09.850 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:09.877 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:09.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:09.979 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:10.004 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:10.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:10.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:10.106 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:10.130 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:10.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:10.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:10.232 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:10.258 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:10.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:10.359 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:10.404 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:10.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:10 smithi052 ceph-mon[36682]: osdmap e475: 8 total, 8 up, 7 in 2024-09-13T13:20:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:10 smithi052 ceph-mon[29471]: osdmap e475: 8 total, 8 up, 7 in 2024-09-13T13:20:10.506 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:10.545 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:10.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:10.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:10 smithi167 ceph-mon[33178]: osdmap e475: 8 total, 8 up, 7 in 2024-09-13T13:20:10.647 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:10.695 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:10.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:10.796 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:10.838 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:10.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:10.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:10.940 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:10.983 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:10.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:10.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:11.067 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:11.085 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:11.129 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T13:20:11.155 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:11.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:11.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:11.258 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:11.312 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:11 smithi052 ceph-mon[29471]: pgmap v965: 168 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 155 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 920 op/s; 2/263406 objects degraded (0.001%); 48530/263406 objects misplaced (18.424%); 48 MiB/s, 769 objects/s recovering 2024-09-13T13:20:11.312 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:11 smithi052 ceph-mon[29471]: osdmap e476: 8 total, 8 up, 7 in 2024-09-13T13:20:11.312 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:11 smithi052 ceph-mon[36682]: pgmap v965: 168 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 155 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 920 op/s; 2/263406 objects degraded (0.001%); 48530/263406 objects misplaced (18.424%); 48 MiB/s, 769 objects/s recovering 2024-09-13T13:20:11.312 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:11 smithi052 ceph-mon[36682]: osdmap e476: 8 total, 8 up, 7 in 2024-09-13T13:20:11.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:11.339 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:11.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:11.440 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:11.497 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:11.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:11 smithi167 ceph-mon[33178]: pgmap v965: 168 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 155 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 920 op/s; 2/263406 objects degraded (0.001%); 48530/263406 objects misplaced (18.424%); 48 MiB/s, 769 objects/s recovering 2024-09-13T13:20:11.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:11 smithi167 ceph-mon[33178]: osdmap e476: 8 total, 8 up, 7 in 2024-09-13T13:20:11.601 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:11.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:11.645 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:11.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:11.746 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:11.772 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:11.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:11.874 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:11.920 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:11.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:12.022 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:12.064 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:12.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:12.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:12.101 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T13:20:12.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:12.191 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:12.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:12.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:12.293 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:12.321 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:12.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:12.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:12.424 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:12.443 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:12.496 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:12.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:12.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:12.599 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:12.658 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:12.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:12.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:12.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:12.786 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:12.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:12.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:12.887 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:12.924 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:12.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:12.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:13.026 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:13.055 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:13.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:13.109 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:20:13.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:13.186 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:13.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:13.288 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:13.313 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:20:13.318 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:13.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:13.420 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:13.445 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:13.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[36682]: pgmap v967: 168 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 155 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 874 B/s rd, 0 B/s wr, 776 op/s; 2/261636 objects degraded (0.001%); 47514/261636 objects misplaced (18.160%); 42 MiB/s, 673 objects/s recovering 2024-09-13T13:20:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1826656806' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:20:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[36682]: from='client.36812 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:20:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 2/261636 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:20:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[36682]: from='client.36812 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:20:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[36682]: osdmap e477: 8 total, 8 up, 7 in 2024-09-13T13:20:13.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[29471]: pgmap v967: 168 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 155 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 874 B/s rd, 0 B/s wr, 776 op/s; 2/261636 objects degraded (0.001%); 47514/261636 objects misplaced (18.160%); 42 MiB/s, 673 objects/s recovering 2024-09-13T13:20:13.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1826656806' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:20:13.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[29471]: from='client.36812 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:20:13.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 2/261636 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:13.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:20:13.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[29471]: from='client.36812 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:20:13.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:13 smithi052 ceph-mon[29471]: osdmap e477: 8 total, 8 up, 7 in 2024-09-13T13:20:13.547 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:13 smithi167 ceph-mon[33178]: pgmap v967: 168 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 155 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 874 B/s rd, 0 B/s wr, 776 op/s; 2/261636 objects degraded (0.001%); 47514/261636 objects misplaced (18.160%); 42 MiB/s, 673 objects/s recovering 2024-09-13T13:20:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1826656806' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:20:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:13 smithi167 ceph-mon[33178]: from='client.36812 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:20:13.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:13 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 2/261636 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:13 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:20:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:13 smithi167 ceph-mon[33178]: from='client.36812 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:20:13.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:13 smithi167 ceph-mon[33178]: osdmap e477: 8 total, 8 up, 7 in 2024-09-13T13:20:13.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:13.573 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:13.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:13.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:13.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:13.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:13.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:13.834 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:13.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:13.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:13.936 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:13.961 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:13.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:13.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:14.063 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:14.088 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:14.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:14.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:14.190 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:14.246 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:14.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:14.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:14.378 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:14.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:14.419 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:14 smithi052 ceph-mon[36682]: 11.16 scrub starts 2024-09-13T13:20:14.419 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:14 smithi052 ceph-mon[36682]: osdmap e478: 8 total, 8 up, 7 in 2024-09-13T13:20:14.419 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:14 smithi052 ceph-mon[29471]: 11.16 scrub starts 2024-09-13T13:20:14.419 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:14 smithi052 ceph-mon[29471]: osdmap e478: 8 total, 8 up, 7 in 2024-09-13T13:20:14.479 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:14.528 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:14.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:14.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:14 smithi167 ceph-mon[33178]: 11.16 scrub starts 2024-09-13T13:20:14.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:14 smithi167 ceph-mon[33178]: osdmap e478: 8 total, 8 up, 7 in 2024-09-13T13:20:14.630 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:14.669 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:14.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:14.771 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:14.800 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:14.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:14.902 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:14.935 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:14.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:14.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:14.971 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:15.007 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:15.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:15.065 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:15.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:15.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:20:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:20:15.109 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T13:20:15.167 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:15.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:15.209 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:15.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:15.310 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:15.340 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:15.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:15.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:15.442 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:15.489 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:15.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:15.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:15.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:15 smithi167 ceph-mon[33178]: pgmap v969: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 155 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1k op/s; 2/255396 objects degraded (0.001%); 46105/255396 objects misplaced (18.052%); 45 MiB/s, 726 objects/s recovering 2024-09-13T13:20:15.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:15 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:20:15.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:15 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:20:15.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:15 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1930962115' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:20:15.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:15 smithi167 ceph-mon[33178]: from='client.26716 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:20:15.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:15.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:15 smithi167 ceph-mon[33178]: from='client.26716 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:20:15.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:15 smithi167 ceph-mon[33178]: osdmap e479: 8 total, 8 up, 7 in 2024-09-13T13:20:15.591 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:15.619 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[36682]: pgmap v969: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 155 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1k op/s; 2/255396 objects degraded (0.001%); 46105/255396 objects misplaced (18.052%); 45 MiB/s, 726 objects/s recovering 2024-09-13T13:20:15.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:20:15.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:20:15.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1930962115' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:20:15.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[36682]: from='client.26716 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:20:15.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:15.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[36682]: from='client.26716 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:20:15.620 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[36682]: osdmap e479: 8 total, 8 up, 7 in 2024-09-13T13:20:15.621 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[29471]: pgmap v969: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 155 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1k op/s; 2/255396 objects degraded (0.001%); 46105/255396 objects misplaced (18.052%); 45 MiB/s, 726 objects/s recovering 2024-09-13T13:20:15.622 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:20:15.622 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:20:15.622 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1930962115' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:20:15.622 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[29471]: from='client.26716 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:20:15.622 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:15.622 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[29471]: from='client.26716 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:20:15.622 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:15 smithi052 ceph-mon[29471]: osdmap e479: 8 total, 8 up, 7 in 2024-09-13T13:20:15.633 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:15.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:15.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:15.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:15.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:15.769 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:15.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:15.870 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:15.871 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:20:15.871 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:20:15.871 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.871 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "MOSDPGPushReply(11.12 479/466 [PushReplyOp(11:4c9ec2de:::benchmark_data_smithi052_182575_object55920:head)])", 2024-09-13T13:20:15.872 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:15.860241+0000", 2024-09-13T13:20:15.872 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0034418169999999998, 2024-09-13T13:20:15.872 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0034651270000000001, 2024-09-13T13:20:15.872 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:20:15.872 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:20:15.872 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:20:15.872 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:20:15.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.872 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:20:15.872 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.860241+0000", 2024-09-13T13:20:15.872 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:15.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.873 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:20:15.873 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.860241+0000", 2024-09-13T13:20:15.873 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:15.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.873 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:20:15.873 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.860243+0000", 2024-09-13T13:20:15.873 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1390000000000002e-06 2024-09-13T13:20:15.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.873 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:20:15.873 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.860250+0000", 2024-09-13T13:20:15.873 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4159999999999998e-06 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.860262+0000", 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2004000000000001e-05 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.860272+0000", 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0022e-05 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:20:15.874 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862350+0000", 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020776750000000002 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.36635.0:151958 11.19 e479/477 11:9e5d1455:::benchmark_data_smithi052_182575_object34949:head v 479'7699, mlcod=479'7697)", 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:15.862611+0000", 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0010711519999999999, 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011466060000000001, 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "started", 2024-09-13T13:20:15.875 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:20:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:20:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862611+0000", 2024-09-13T13:20:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:20:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862611+0000", 2024-09-13T13:20:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:20:15.876 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862613+0000", 2024-09-13T13:20:15.877 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7910000000000001e-06 2024-09-13T13:20:15.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.877 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.877 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:20:15.877 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862620+0000", 2024-09-13T13:20:15.877 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1799999999999999e-06 2024-09-13T13:20:15.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.877 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.877 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:20:15.877 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862631+0000", 2024-09-13T13:20:15.877 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0659000000000001e-05 2024-09-13T13:20:15.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.878 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.878 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:20:15.878 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862639+0000", 2024-09-13T13:20:15.878 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5440000000000001e-06 2024-09-13T13:20:15.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.878 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.878 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:20:15.878 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862911+0000", 2024-09-13T13:20:15.878 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027203600000000002 2024-09-13T13:20:15.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.878 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.878 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862936+0000", 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4964999999999999e-05 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "MOSDPGPush(11.11 479/466 [PushOp(11:8a1ce90c:::benchmark_data_smithi052_182575_object82149:head, version: 446'5048, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:8a1ce90c:::benchmark_data_smithi052_182575_object82149:head@446'5048, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:15.862658+0000", 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0010247399999999999, 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001187303, 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:20:15.879 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862658+0000", 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862658+0000", 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862659+0000", 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0900000000000001e-07 2024-09-13T13:20:15.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862703+0000", 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4687000000000001e-05 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862720+0000", 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6782999999999998e-05 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862736+0000", 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5928000000000001e-05 2024-09-13T13:20:15.881 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "pg_backfill(progress 11.11 e 479/479 lb 11:8a1c0946:::benchmark_data_smithi052_182575_object88610:head)", 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:15.862754+0000", 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.000928633, 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001134344, 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:20:15.882 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862754+0000", 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862754+0000", 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862754+0000", 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:20:15.883 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862780+0000", 2024-09-13T13:20:15.884 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5989000000000001e-05 2024-09-13T13:20:15.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.884 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:20:15.884 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862789+0000", 2024-09-13T13:20:15.884 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6240000000000008e-06 2024-09-13T13:20:15.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.884 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:20:15.884 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862795+0000", 2024-09-13T13:20:15.884 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.156e-06 2024-09-13T13:20:15.884 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:15.885 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:20:15.885 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:15.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.36635.0:151960 11.19 e479/477)", 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:15.862805+0000", 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.00087801299999999997, 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011250380000000001, 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "queued for pg", 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862805+0000", 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862805+0000", 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862805+0000", 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4000000000000002e-07 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862810+0000", 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0769999999999998e-06 2024-09-13T13:20:15.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862816+0000", 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4110000000000004e-06 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862820+0000", 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5600000000000004e-06 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.888 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:151956 11.18 11:1aa92582:::benchmark_data_smithi052_182575_object34947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e479)", 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:15.856926+0000", 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0067561720000000004, 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0070579650000000002, 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 151956 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.889 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.856926+0000", 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.856926+0000", 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.856929+0000", 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2019999999999998e-06 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.856943+0000", 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4419e-05 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.856956+0000", 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2918999999999999e-05 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.857055+0000", 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8812000000000002e-05 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:20:15.892 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.857118+0000", 2024-09-13T13:20:15.892 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3499999999999999e-05 2024-09-13T13:20:15.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.892 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.892 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:20:15.892 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.857362+0000", 2024-09-13T13:20:15.892 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024369300000000001 2024-09-13T13:20:15.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.892 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.892 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,6", 2024-09-13T13:20:15.892 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.857789+0000", 2024-09-13T13:20:15.892 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00042732600000000001 2024-09-13T13:20:15.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:15.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:15.893 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:20:15.893 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:15.862831+0000", 2024-09-13T13:20:15.893 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050417200000000004 2024-09-13T13:20:15.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:15.893 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:20:15.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:15.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:15.893 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:20:15.893 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 6 2024-09-13T13:20:15.893 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:20:15.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:15.908 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:15.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:16.009 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:16.040 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:16.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:16.142 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:16.145 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:20:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:20:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:20:16.171 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:16.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:16.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:16.273 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:16.316 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:16.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:16.418 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:16.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:16.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:16.455 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:16 smithi052 ceph-mon[36682]: 11.16 scrub starts 2024-09-13T13:20:16.455 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:16 smithi052 ceph-mon[29471]: 11.16 scrub starts 2024-09-13T13:20:16.551 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:16.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:16 smithi167 ceph-mon[33178]: 11.16 scrub starts 2024-09-13T13:20:16.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:16.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:16.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:16.684 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:16.718 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:16.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:16.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:16.844 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:16.894 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:16.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:16.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:16.910 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T13:20:16.992 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:20:16.997 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:17.062 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:17.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:17.164 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:17.193 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:17.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:17.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:17.327 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:17.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:17.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:17.369 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:17 smithi052 ceph-mon[29471]: pgmap v972: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 880 B/s rd, 0 B/s wr, 663 op/s; 2/251850 objects degraded (0.001%); 44067/251850 objects misplaced (17.497%); 27 MiB/s, 439 objects/s recovering 2024-09-13T13:20:17.369 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:17 smithi052 ceph-mon[29471]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:20:17.370 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:17 smithi052 ceph-mon[36682]: pgmap v972: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 880 B/s rd, 0 B/s wr, 663 op/s; 2/251850 objects degraded (0.001%); 44067/251850 objects misplaced (17.497%); 27 MiB/s, 439 objects/s recovering 2024-09-13T13:20:17.370 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:17 smithi052 ceph-mon[36682]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:20:17.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:17.448 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T13:20:17.459 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:17.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:17.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:17.561 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:17 smithi167 ceph-mon[33178]: pgmap v972: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 880 B/s rd, 0 B/s wr, 663 op/s; 2/251850 objects degraded (0.001%); 44067/251850 objects misplaced (17.497%); 27 MiB/s, 439 objects/s recovering 2024-09-13T13:20:17.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:17 smithi167 ceph-mon[33178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:20:17.626 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:17.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:17.728 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:17.754 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:17.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:17.793 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:20:17.793 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:20:17.808 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:17.829 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:20:17.829 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:20:17.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:17.880 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:17.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:17.883 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:17.890 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:20:17.890 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:20:17.937 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:20:17.937 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:20:17.965 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:20:17.965 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:20:17.982 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:18.004 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:20:18.005 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:20:18.038 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:18.038 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:20:18.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:18.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:18.039 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:20:18.075 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:20:18.075 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:20:18.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:18.166 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:18.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:18.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:18.268 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:18.299 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:18.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:18.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:18.401 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:18.429 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:18.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:18.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:18.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:18 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1703192059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:20:18.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:18 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 2/251850 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:18.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:18 smithi052 ceph-mon[36682]: osdmap e480: 8 total, 8 up, 7 in 2024-09-13T13:20:18.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:18 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1703192059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:20:18.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:18 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 2/251850 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:18.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:18 smithi052 ceph-mon[29471]: osdmap e480: 8 total, 8 up, 7 in 2024-09-13T13:20:18.532 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:18 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1703192059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:20:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:18 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 2/251850 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:18.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:18 smithi167 ceph-mon[33178]: osdmap e480: 8 total, 8 up, 7 in 2024-09-13T13:20:18.576 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:18.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:18.678 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:18.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:18.716 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:18.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:18.817 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:18.859 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:18.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:18.961 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:18.997 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:18.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:19.099 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:19.143 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:19.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:19.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:19.245 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:19.286 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:19.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:19.388 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:19.414 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:19.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:19.516 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:19.560 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:19.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:19.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:19 smithi167 ceph-mon[33178]: pgmap v973: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 158 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 641 op/s; 2/250101 objects degraded (0.001%); 41522/250101 objects misplaced (16.602%); 36 MiB/s, 578 objects/s recovering 2024-09-13T13:20:19.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:19 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:20:19.634 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:19 smithi052 ceph-mon[36682]: pgmap v973: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 158 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 641 op/s; 2/250101 objects degraded (0.001%); 41522/250101 objects misplaced (16.602%); 36 MiB/s, 578 objects/s recovering 2024-09-13T13:20:19.634 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:19 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:20:19.634 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:19 smithi052 ceph-mon[29471]: pgmap v973: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 158 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 641 op/s; 2/250101 objects degraded (0.001%); 41522/250101 objects misplaced (16.602%); 36 MiB/s, 578 objects/s recovering 2024-09-13T13:20:19.634 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:19 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:20:19.662 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:19.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:19.693 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:19.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:19.793 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:19.816 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:19.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:19.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:19.918 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:19.959 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:19.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:19.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:20.061 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:20.103 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:20.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:20.204 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:20.235 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:20.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:20.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:20.337 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:20.359 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:20.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:20.461 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:20.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:20.502 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:20.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:20.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:20.636 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:20.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:20.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:20.737 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:20.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:20.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:20.872 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:20.910 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:20.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:21.012 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:21.049 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:21.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:21.151 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:21.183 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:21.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:21.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:21.285 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:21.329 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:21.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:21.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:21.431 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:21.453 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:21.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:21.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:21 smithi052 ceph-mon[29471]: pgmap v975: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 159 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 898 op/s; 2/239241 objects degraded (0.001%); 36305/239241 objects misplaced (15.175%); 56 MiB/s, 901 objects/s recovering 2024-09-13T13:20:21.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:21 smithi052 ceph-mon[36682]: pgmap v975: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 159 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 898 op/s; 2/239241 objects degraded (0.001%); 36305/239241 objects misplaced (15.175%); 56 MiB/s, 901 objects/s recovering 2024-09-13T13:20:21.555 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:21.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:21 smithi167 ceph-mon[33178]: pgmap v975: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 159 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 898 op/s; 2/239241 objects degraded (0.001%); 36305/239241 objects misplaced (15.175%); 56 MiB/s, 901 objects/s recovering 2024-09-13T13:20:21.596 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:21.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:21.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:21.698 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:21.727 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:21.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:21.829 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:21.859 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:21.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:21.961 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:22.005 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:22.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:22.104 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:20:22.105 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:20:22.106 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:22.141 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:22.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:22.242 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:22.285 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:22.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:22.387 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:22.410 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:22.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:22.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:22.512 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:22.561 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:22.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:22.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:22.663 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:22.686 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:22.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:22.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:22.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:22.816 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:22.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:22.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:22.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:22.944 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:22.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:23.046 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:23.067 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:23.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:23.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:23.169 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:23.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:23.209 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:23.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:23.310 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:23.348 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:23.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:23.449 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:23.497 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:23.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:23.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:23 smithi167 ceph-mon[33178]: pgmap v976: 168 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 159 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 769 op/s; 2/237102 objects degraded (0.001%); 33546/237102 objects misplaced (14.148%); 60 MiB/s, 957 objects/s recovering 2024-09-13T13:20:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:23 smithi167 ceph-mon[33178]: osdmap e481: 8 total, 8 up, 7 in 2024-09-13T13:20:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:23 smithi167 ceph-mon[33178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:20:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:23 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 2/237102 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:23 smithi167 ceph-mon[33178]: osdmap e482: 8 total, 8 up, 7 in 2024-09-13T13:20:23.598 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:23.621 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:23.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:23.722 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:23 smithi052 ceph-mon[36682]: pgmap v976: 168 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 159 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 769 op/s; 2/237102 objects degraded (0.001%); 33546/237102 objects misplaced (14.148%); 60 MiB/s, 957 objects/s recovering 2024-09-13T13:20:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:23 smithi052 ceph-mon[36682]: osdmap e481: 8 total, 8 up, 7 in 2024-09-13T13:20:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:23 smithi052 ceph-mon[36682]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:20:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:23 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 2/237102 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:23 smithi052 ceph-mon[36682]: osdmap e482: 8 total, 8 up, 7 in 2024-09-13T13:20:23.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:23 smithi052 ceph-mon[29471]: pgmap v976: 168 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 159 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 769 op/s; 2/237102 objects degraded (0.001%); 33546/237102 objects misplaced (14.148%); 60 MiB/s, 957 objects/s recovering 2024-09-13T13:20:23.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:23 smithi052 ceph-mon[29471]: osdmap e481: 8 total, 8 up, 7 in 2024-09-13T13:20:23.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:23 smithi052 ceph-mon[29471]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:20:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:23 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 2/237102 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:23 smithi052 ceph-mon[29471]: osdmap e482: 8 total, 8 up, 7 in 2024-09-13T13:20:23.750 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:23.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:23.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:23.852 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:23.873 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:23.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:23.975 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:24.007 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:24.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:24.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:24.109 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:24.150 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:24.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:24.252 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:24.282 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:24.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:24.384 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:24.416 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:24.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:24.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:24.546 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:24.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:24.648 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:24.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:24.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:24.771 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:24.797 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:24.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:24.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:24.934 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:24.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:25.035 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:25.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:20:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:20:25.077 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:25.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:25.179 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:25.210 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:25.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:25.312 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:25.345 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:25.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:25.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:25.447 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:25.474 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:25.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:25.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:25 smithi167 ceph-mon[33178]: pgmap v979: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 159 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s; 2/230283 objects degraded (0.001%); 32193/230283 objects misplaced (13.980%); 61 MiB/s, 974 objects/s recovering 2024-09-13T13:20:25.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:25.575 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:25.620 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:25.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:25.722 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:25.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:25 smithi052 ceph-mon[36682]: pgmap v979: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 159 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s; 2/230283 objects degraded (0.001%); 32193/230283 objects misplaced (13.980%); 61 MiB/s, 974 objects/s recovering 2024-09-13T13:20:25.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:25 smithi052 ceph-mon[29471]: pgmap v979: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 159 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s; 2/230283 objects degraded (0.001%); 32193/230283 objects misplaced (13.980%); 61 MiB/s, 974 objects/s recovering 2024-09-13T13:20:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:25.749 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:25.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:25.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:25.851 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:25.873 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:25.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:25.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:25.974 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:26.003 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:26.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:26.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:26.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:26.135 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:26.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:26.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:26.237 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:26.258 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:26.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:26.360 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:26.389 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:26.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:26.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:26 smithi052 ceph-mon[36682]: 11.17 scrub starts 2024-09-13T13:20:26.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:26 smithi052 ceph-mon[29471]: 11.17 scrub starts 2024-09-13T13:20:26.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:20:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:20:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:20:26.490 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:26.522 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:26.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:26.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:26 smithi167 ceph-mon[33178]: 11.17 scrub starts 2024-09-13T13:20:26.624 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:26.646 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:26.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:26.748 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:26.774 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:26.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:26.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:26.875 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:26.898 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:26.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:26.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:26.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:27.032 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:27.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:27.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:27.134 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:27.178 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:27.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:27.279 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:27.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:27.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:27.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:27.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:27.465 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:27.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:27.567 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:27.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:27 smithi167 ceph-mon[33178]: pgmap v980: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 159 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 898 op/s; 2/226305 objects degraded (0.001%); 30160/226305 objects misplaced (13.327%); 54 MiB/s, 856 objects/s recovering 2024-09-13T13:20:27.594 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:27.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:27.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:27.696 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:27 smithi052 ceph-mon[29471]: pgmap v980: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 159 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 898 op/s; 2/226305 objects degraded (0.001%); 30160/226305 objects misplaced (13.327%); 54 MiB/s, 856 objects/s recovering 2024-09-13T13:20:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:27 smithi052 ceph-mon[36682]: pgmap v980: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 159 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 898 op/s; 2/226305 objects degraded (0.001%); 30160/226305 objects misplaced (13.327%); 54 MiB/s, 856 objects/s recovering 2024-09-13T13:20:27.740 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:27.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:27.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:27.841 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:27.868 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:27.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:27.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:27.970 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:28.011 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:28.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:28.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:28.112 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:28.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:28.149 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:28.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:28.250 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:28.296 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:28.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:28.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:28.398 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:28.448 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:28.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:28.549 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:28.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:28 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 2/226305 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:28.593 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:28.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:28.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:28.729 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:28.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:28.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:28 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 2/226305 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:28.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:28 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 2/226305 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:28.830 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:28.865 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:28.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:28.967 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:28.989 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:28.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:29.090 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:29.112 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:29.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:29.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:29.214 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:29.237 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:29.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:29.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:29.339 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:29.389 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:29.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:29.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:29.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:29 smithi052 ceph-mon[29471]: pgmap v981: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 616 op/s; 2/224442 objects degraded (0.001%); 29224/224442 objects misplaced (13.021%); 35 MiB/s, 552 objects/s recovering 2024-09-13T13:20:29.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:29 smithi052 ceph-mon[36682]: pgmap v981: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 616 op/s; 2/224442 objects degraded (0.001%); 29224/224442 objects misplaced (13.021%); 35 MiB/s, 552 objects/s recovering 2024-09-13T13:20:29.490 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:29.529 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:29.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:29.631 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:29.673 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:29.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:29.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:29.775 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:29.799 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:29.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:29.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:29.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:29 smithi167 ceph-mon[33178]: pgmap v981: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 616 op/s; 2/224442 objects degraded (0.001%); 29224/224442 objects misplaced (13.021%); 35 MiB/s, 552 objects/s recovering 2024-09-13T13:20:29.901 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:29.923 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:29.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:30.025 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:30.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:30.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:30.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:30.152 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:30.187 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:30.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:30.289 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:30.312 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:30.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:30.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:30.362 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:20:30.413 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:30.458 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:30.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:30.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:30 smithi052 ceph-mon[36682]: osdmap e483: 8 total, 8 up, 7 in 2024-09-13T13:20:30.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:30 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/671824590' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:20:30.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:30 smithi052 ceph-mon[36682]: from='client.36827 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:20:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:30 smithi052 ceph-mon[29471]: osdmap e483: 8 total, 8 up, 7 in 2024-09-13T13:20:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:30 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/671824590' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:20:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:30 smithi052 ceph-mon[29471]: from='client.36827 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:20:30.560 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:30.581 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:30.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:30.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:30.684 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:30.715 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:30.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:30.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:30.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:30 smithi167 ceph-mon[33178]: osdmap e483: 8 total, 8 up, 7 in 2024-09-13T13:20:30.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:30 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/671824590' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:20:30.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:30 smithi167 ceph-mon[33178]: from='client.36827 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:20:30.838 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:30.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:30.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:30.940 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:30.968 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:30.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:30.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:31.070 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:31.110 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:31.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:31.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:31.213 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:31.256 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:31.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:31.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:31.358 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:31.380 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:31.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:31.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:31.482 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:31.516 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:31.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:31.618 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:31.643 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:31.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:31.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:31.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:31 smithi052 ceph-mon[36682]: pgmap v983: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.01k op/s; 2/213696 objects degraded (0.001%); 25723/213696 objects misplaced (12.037%); 34 MiB/s, 545 objects/s recovering 2024-09-13T13:20:31.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:31 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:20:31.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:31 smithi052 ceph-mon[36682]: from='client.36827 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:20:31.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:31 smithi052 ceph-mon[36682]: osdmap e484: 8 total, 8 up, 7 in 2024-09-13T13:20:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:31 smithi052 ceph-mon[29471]: pgmap v983: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.01k op/s; 2/213696 objects degraded (0.001%); 25723/213696 objects misplaced (12.037%); 34 MiB/s, 545 objects/s recovering 2024-09-13T13:20:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:31 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:20:31.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:31 smithi052 ceph-mon[29471]: from='client.36827 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:20:31.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:31 smithi052 ceph-mon[29471]: osdmap e484: 8 total, 8 up, 7 in 2024-09-13T13:20:31.746 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:31.767 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:31.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:31.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:31.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:31 smithi167 ceph-mon[33178]: pgmap v983: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 160 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.01k op/s; 2/213696 objects degraded (0.001%); 25723/213696 objects misplaced (12.037%); 34 MiB/s, 545 objects/s recovering 2024-09-13T13:20:31.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:31 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:20:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:31 smithi167 ceph-mon[33178]: from='client.36827 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:20:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:31 smithi167 ceph-mon[33178]: osdmap e484: 8 total, 8 up, 7 in 2024-09-13T13:20:31.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:31.891 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:31.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:31.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:32.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:32.014 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:32.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:32.115 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:32.136 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:32.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:32.238 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:32.260 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:32.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:32.362 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:32.383 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:32.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:32.485 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:32.511 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:32.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:32.612 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:32.638 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:32.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:32.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:32.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:32 smithi052 ceph-mon[36682]: 11.14 scrub starts 2024-09-13T13:20:32.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:32 smithi052 ceph-mon[36682]: 5.18 scrub starts 2024-09-13T13:20:32.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:32 smithi052 ceph-mon[36682]: 5.18 scrub ok 2024-09-13T13:20:32.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:32 smithi052 ceph-mon[36682]: 11.11 scrub starts 2024-09-13T13:20:32.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:32 smithi052 ceph-mon[36682]: osdmap e485: 8 total, 8 up, 7 in 2024-09-13T13:20:32.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:32 smithi052 ceph-mon[36682]: 11.14 scrub ok 2024-09-13T13:20:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:32 smithi052 ceph-mon[29471]: 11.14 scrub starts 2024-09-13T13:20:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:32 smithi052 ceph-mon[29471]: 5.18 scrub starts 2024-09-13T13:20:32.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:32 smithi052 ceph-mon[29471]: 5.18 scrub ok 2024-09-13T13:20:32.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:32 smithi052 ceph-mon[29471]: 11.11 scrub starts 2024-09-13T13:20:32.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:32 smithi052 ceph-mon[29471]: osdmap e485: 8 total, 8 up, 7 in 2024-09-13T13:20:32.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:32 smithi052 ceph-mon[29471]: 11.14 scrub ok 2024-09-13T13:20:32.740 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:32.763 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:32.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:32.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:32.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:32 smithi167 ceph-mon[33178]: 11.14 scrub starts 2024-09-13T13:20:32.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:32 smithi167 ceph-mon[33178]: 5.18 scrub starts 2024-09-13T13:20:32.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:32 smithi167 ceph-mon[33178]: 5.18 scrub ok 2024-09-13T13:20:32.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:32 smithi167 ceph-mon[33178]: 11.11 scrub starts 2024-09-13T13:20:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:32 smithi167 ceph-mon[33178]: osdmap e485: 8 total, 8 up, 7 in 2024-09-13T13:20:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:32 smithi167 ceph-mon[33178]: 11.14 scrub ok 2024-09-13T13:20:32.864 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:32.886 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:32.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:32.988 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:33.035 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:33.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:33.137 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:33.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:33.181 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:33.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:33.282 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:33.309 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:33.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:33.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:33.411 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:33.437 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:33.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:33.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:33.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:33.569 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:33.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:33.670 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:33.714 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:33.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:33.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:33.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:33 smithi052 ceph-mon[36682]: pgmap v986: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 2/211899 objects degraded (0.001%); 25263/211899 objects misplaced (11.922%); 107 KiB/s, 1 objects/s recovering 2024-09-13T13:20:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:33 smithi052 ceph-mon[36682]: osdmap e486: 8 total, 8 up, 7 in 2024-09-13T13:20:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:33 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 2/211899 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:33 smithi052 ceph-mon[29471]: pgmap v986: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 2/211899 objects degraded (0.001%); 25263/211899 objects misplaced (11.922%); 107 KiB/s, 1 objects/s recovering 2024-09-13T13:20:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:33 smithi052 ceph-mon[29471]: osdmap e486: 8 total, 8 up, 7 in 2024-09-13T13:20:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:33 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 2/211899 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:33.816 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:33.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:33 smithi167 ceph-mon[33178]: pgmap v986: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 161 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 2/211899 objects degraded (0.001%); 25263/211899 objects misplaced (11.922%); 107 KiB/s, 1 objects/s recovering 2024-09-13T13:20:33.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:33 smithi167 ceph-mon[33178]: osdmap e486: 8 total, 8 up, 7 in 2024-09-13T13:20:33.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:33 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 2/211899 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:33.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:33.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:33.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:33.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:33.975 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:33.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:33.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:34.077 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:34.120 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:34.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:34.222 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:34.249 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:34.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:34.351 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:34.378 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:34.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:34.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:34.480 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:34.521 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:34.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:34.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:34.623 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:34.654 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:34.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:34.718 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:34 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:20:34.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:34 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:20:34.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:34 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:20:34.757 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:34.784 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:34.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:34.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:34.907 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:34.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:35.008 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:35.036 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:35.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:35.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:20:34 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:20:35.139 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:35.176 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:35.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:35.277 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:35.313 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:35.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:35.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:35.415 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:35.460 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:35.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:35.561 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:35.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:35.606 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:35.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:35.707 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:35.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:35 smithi052 ceph-mon[36682]: 11.11 scrub starts 2024-09-13T13:20:35.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:35 smithi052 ceph-mon[36682]: pgmap v988: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 2/207597 objects degraded (0.001%); 23677/207597 objects misplaced (11.405%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T13:20:35.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:35 smithi052 ceph-mon[36682]: 3.8 scrub starts 2024-09-13T13:20:35.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:35 smithi052 ceph-mon[36682]: 3.8 scrub ok 2024-09-13T13:20:35.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:35 smithi052 ceph-mon[36682]: 11.11 scrub ok 2024-09-13T13:20:35.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:35.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:35 smithi052 ceph-mon[29471]: 11.11 scrub starts 2024-09-13T13:20:35.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:35 smithi052 ceph-mon[29471]: pgmap v988: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 2/207597 objects degraded (0.001%); 23677/207597 objects misplaced (11.405%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T13:20:35.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:35 smithi052 ceph-mon[29471]: 3.8 scrub starts 2024-09-13T13:20:35.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:35 smithi052 ceph-mon[29471]: 3.8 scrub ok 2024-09-13T13:20:35.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:35 smithi052 ceph-mon[29471]: 11.11 scrub ok 2024-09-13T13:20:35.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:35.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:35.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:35.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:35 smithi167 ceph-mon[33178]: 11.11 scrub starts 2024-09-13T13:20:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:35 smithi167 ceph-mon[33178]: pgmap v988: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 2/207597 objects degraded (0.001%); 23677/207597 objects misplaced (11.405%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T13:20:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:35 smithi167 ceph-mon[33178]: 3.8 scrub starts 2024-09-13T13:20:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:35 smithi167 ceph-mon[33178]: 3.8 scrub ok 2024-09-13T13:20:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:35 smithi167 ceph-mon[33178]: 11.11 scrub ok 2024-09-13T13:20:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:35.855 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:35.894 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:35.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:35.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:35.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:36.024 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:36.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:36.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:36.126 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:36.153 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:36.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:36.254 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:36.282 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:36.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:36.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:36.383 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:36.414 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:36 smithi052 ceph-mon[29471]: 5.1f scrub starts 2024-09-13T13:20:36.415 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:20:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:20:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:20:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:36.422 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:36.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:36.524 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:36.570 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:36.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:36.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:36.673 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:36.707 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:36.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:36.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:36.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:36 smithi052 ceph-mon[29471]: 5.1f scrub ok 2024-09-13T13:20:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:36 smithi052 ceph-mon[36682]: 5.1f scrub starts 2024-09-13T13:20:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:36 smithi052 ceph-mon[36682]: 5.1f scrub ok 2024-09-13T13:20:36.753 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:36.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T13:20:36.809 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:36.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:36 smithi167 ceph-mon[33178]: 5.1f scrub starts 2024-09-13T13:20:36.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:36 smithi167 ceph-mon[33178]: 5.1f scrub ok 2024-09-13T13:20:36.885 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:36.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:36.987 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:37.013 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:37.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:37.115 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:37.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:37.163 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:37.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:37.263 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:37.317 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:37.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:37.418 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:37.461 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:37.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:37.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:37.563 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:37.594 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:37.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:37.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:37.600 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:37 smithi052 ceph-mon[36682]: pgmap v989: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 163 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 2/204753 objects degraded (0.001%); 21795/204753 objects misplaced (10.645%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T13:20:37.600 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:37 smithi052 ceph-mon[36682]: 6.b scrub starts 2024-09-13T13:20:37.600 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:37 smithi052 ceph-mon[36682]: 6.b scrub ok 2024-09-13T13:20:37.600 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:37 smithi052 ceph-mon[36682]: 3.f scrub starts 2024-09-13T13:20:37.601 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:37 smithi052 ceph-mon[36682]: 3.f scrub ok 2024-09-13T13:20:37.601 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:37 smithi052 ceph-mon[29471]: pgmap v989: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 163 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 2/204753 objects degraded (0.001%); 21795/204753 objects misplaced (10.645%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T13:20:37.601 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:37 smithi052 ceph-mon[29471]: 6.b scrub starts 2024-09-13T13:20:37.601 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:37 smithi052 ceph-mon[29471]: 6.b scrub ok 2024-09-13T13:20:37.601 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:37 smithi052 ceph-mon[29471]: 3.f scrub starts 2024-09-13T13:20:37.601 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:37 smithi052 ceph-mon[29471]: 3.f scrub ok 2024-09-13T13:20:37.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:37.723 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:37.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:37.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:37.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:37 smithi167 ceph-mon[33178]: pgmap v989: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 163 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 2/204753 objects degraded (0.001%); 21795/204753 objects misplaced (10.645%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T13:20:37.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:37 smithi167 ceph-mon[33178]: 6.b scrub starts 2024-09-13T13:20:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:37 smithi167 ceph-mon[33178]: 6.b scrub ok 2024-09-13T13:20:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:37 smithi167 ceph-mon[33178]: 3.f scrub starts 2024-09-13T13:20:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:37 smithi167 ceph-mon[33178]: 3.f scrub ok 2024-09-13T13:20:37.825 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:37.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:37.869 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:37.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:37.970 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:38.006 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:38.037 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:38.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:38.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:38.139 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:38.199 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:38.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:38.278 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:20:38.301 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:38.344 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:38.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:38.446 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:38.473 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:38.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:38.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:38.608 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:38.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:38.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:38.709 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:38.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:38 smithi052 ceph-mon[36682]: 6.15 scrub starts 2024-09-13T13:20:38.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:38 smithi052 ceph-mon[36682]: 6.15 scrub ok 2024-09-13T13:20:38.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:38 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 2/204753 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:38.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:38 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2164432749' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:20:38.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:38 smithi052 ceph-mon[36682]: from='client.26734 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:20:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:38 smithi052 ceph-mon[29471]: 6.15 scrub starts 2024-09-13T13:20:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:38 smithi052 ceph-mon[29471]: 6.15 scrub ok 2024-09-13T13:20:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:38 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 2/204753 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2164432749' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:20:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:38 smithi052 ceph-mon[29471]: from='client.26734 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:20:38.748 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:38.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:38.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:38 smithi167 ceph-mon[33178]: 6.15 scrub starts 2024-09-13T13:20:38.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:38 smithi167 ceph-mon[33178]: 6.15 scrub ok 2024-09-13T13:20:38.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:38 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 2/204753 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:38.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:38 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2164432749' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:20:38.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:38 smithi167 ceph-mon[33178]: from='client.26734 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:20:38.850 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:38.878 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:38.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:38.980 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:39.009 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:39.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:39.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:39.111 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:39.125 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T13:20:39.137 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:39.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:39.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:39.239 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:39.267 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:39.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:39.369 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:39.404 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:20:39.411 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:39.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:39.513 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:39.558 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:39.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:39.660 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:39.689 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:39.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:39.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:39.695 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:39 smithi052 ceph-mon[29471]: pgmap v990: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 163 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 665 B/s rd, 0 op/s; 2/203112 objects degraded (0.001%); 21795/203112 objects misplaced (10.731%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T13:20:39.695 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:39 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:20:39.695 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:39 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:20:39.695 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:39 smithi052 ceph-mon[29471]: from='client.26734 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:20:39.695 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:39 smithi052 ceph-mon[29471]: osdmap e487: 8 total, 8 up, 7 in 2024-09-13T13:20:39.696 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:39 smithi052 ceph-mon[36682]: pgmap v990: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 163 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 665 B/s rd, 0 op/s; 2/203112 objects degraded (0.001%); 21795/203112 objects misplaced (10.731%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T13:20:39.696 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:39 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:20:39.696 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:39 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:20:39.696 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:39 smithi052 ceph-mon[36682]: from='client.26734 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:20:39.696 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:39 smithi052 ceph-mon[36682]: osdmap e487: 8 total, 8 up, 7 in 2024-09-13T13:20:39.791 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:39.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:39 smithi167 ceph-mon[33178]: pgmap v990: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 163 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 665 B/s rd, 0 op/s; 2/203112 objects degraded (0.001%); 21795/203112 objects misplaced (10.731%); 1.3 MiB/s, 20 objects/s recovering 2024-09-13T13:20:39.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:39 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:20:39.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:39 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:20:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:39 smithi167 ceph-mon[33178]: from='client.26734 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:20:39.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:39 smithi167 ceph-mon[33178]: osdmap e487: 8 total, 8 up, 7 in 2024-09-13T13:20:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:39.842 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:39.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:39.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:39.971 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:39.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:40.073 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:40.103 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:40.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:40.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:40.204 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:40.241 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:40.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:40.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:40.342 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:40.374 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:40.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:40.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:40.475 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:40.519 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:40.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:40.621 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:40.649 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:40.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:40.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:40.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:40 smithi052 ceph-mon[36682]: 6.3 scrub starts 2024-09-13T13:20:40.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:40 smithi052 ceph-mon[36682]: 6.3 scrub ok 2024-09-13T13:20:40.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:40 smithi052 ceph-mon[36682]: 3.11 scrub starts 2024-09-13T13:20:40.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:40 smithi052 ceph-mon[36682]: 3.11 scrub ok 2024-09-13T13:20:40.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:40 smithi052 ceph-mon[29471]: 6.3 scrub starts 2024-09-13T13:20:40.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:40 smithi052 ceph-mon[29471]: 6.3 scrub ok 2024-09-13T13:20:40.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:40 smithi052 ceph-mon[29471]: 3.11 scrub starts 2024-09-13T13:20:40.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:40 smithi052 ceph-mon[29471]: 3.11 scrub ok 2024-09-13T13:20:40.751 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:40.776 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:40.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:40.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:40 smithi167 ceph-mon[33178]: 6.3 scrub starts 2024-09-13T13:20:40.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:40 smithi167 ceph-mon[33178]: 6.3 scrub ok 2024-09-13T13:20:40.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:40 smithi167 ceph-mon[33178]: 3.11 scrub starts 2024-09-13T13:20:40.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:40 smithi167 ceph-mon[33178]: 3.11 scrub ok 2024-09-13T13:20:40.878 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:40.904 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:40.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:41.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:41.035 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:41.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:41.137 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:41.182 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:41.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:41.283 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:41.325 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:41.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:41.373 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T13:20:41.427 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:41.497 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:41.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:41.599 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:41.641 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:41 smithi052 ceph-mon[29471]: pgmap v992: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 163 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 310 op/s; 2/189153 objects degraded (0.001%); 19201/189153 objects misplaced (10.151%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T13:20:41.641 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:41 smithi052 ceph-mon[36682]: pgmap v992: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 163 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 310 op/s; 2/189153 objects degraded (0.001%); 19201/189153 objects misplaced (10.151%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T13:20:41.642 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:20:41.642 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:20:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:41.688 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:41.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:41.789 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:41.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:41 smithi167 ceph-mon[33178]: pgmap v992: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 163 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 310 op/s; 2/189153 objects degraded (0.001%); 19201/189153 objects misplaced (10.151%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T13:20:41.821 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:41.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:41.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:41.913 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:41.914 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:41.923 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:41.997 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:41.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:42.097 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:42.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:42.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:42.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:42.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:42.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:42.268 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:42.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:42.368 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:42.409 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:42.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:42.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:42.507 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T13:20:42.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:42.525 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:42.537 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:42.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:42.638 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:42.707 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:42.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:42.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:42 smithi052 ceph-mon[36682]: 4.1 scrub starts 2024-09-13T13:20:42.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:42 smithi052 ceph-mon[36682]: 4.1 scrub ok 2024-09-13T13:20:42.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:42 smithi052 ceph-mon[29471]: 4.1 scrub starts 2024-09-13T13:20:42.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:42 smithi052 ceph-mon[29471]: 4.1 scrub ok 2024-09-13T13:20:42.808 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:42.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:42 smithi167 ceph-mon[33178]: 4.1 scrub starts 2024-09-13T13:20:42.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:42 smithi167 ceph-mon[33178]: 4.1 scrub ok 2024-09-13T13:20:42.834 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:42.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:42.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:42.936 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:42.961 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:42.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:42.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:43.063 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:43.126 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:43.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:43.226 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:43.263 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:43.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:43.365 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:43.413 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:43.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:43.514 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:43.541 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:43.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:43.642 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:43.666 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:43.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:43.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:43.722 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[36682]: pgmap v993: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 163 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 350 op/s; 2/186447 objects degraded (0.001%); 19201/186447 objects misplaced (10.298%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T13:20:43.722 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:20:43.722 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 2/186447 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:43.722 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[36682]: osdmap e488: 8 total, 8 up, 7 in 2024-09-13T13:20:43.722 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:20:43.722 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:20:43.722 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:20:43.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[29471]: pgmap v993: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 163 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 350 op/s; 2/186447 objects degraded (0.001%); 19201/186447 objects misplaced (10.298%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T13:20:43.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:20:43.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 2/186447 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:43.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[29471]: osdmap e488: 8 total, 8 up, 7 in 2024-09-13T13:20:43.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:20:43.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:20:43.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:43 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:20:43.768 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:43.792 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:43.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:43.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:43.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:43 smithi167 ceph-mon[33178]: pgmap v993: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 163 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 350 op/s; 2/186447 objects degraded (0.001%); 19201/186447 objects misplaced (10.298%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T13:20:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:43 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:20:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:43 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 2/186447 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-13T13:20:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:43 smithi167 ceph-mon[33178]: osdmap e488: 8 total, 8 up, 7 in 2024-09-13T13:20:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:43 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:20:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:43 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:20:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:43 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:20:43.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:43.915 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:43.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:43.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:44.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:44.045 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:44.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:44.147 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:44.179 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:44.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:44.281 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:44.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:44.312 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:44.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:44.413 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:44.439 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:44.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:44.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:44.541 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:44.577 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:44.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:44.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:44.679 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:44.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:44.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:44.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:44.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:44 smithi167 ceph-mon[33178]: 4.14 scrub starts 2024-09-13T13:20:44.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:44 smithi167 ceph-mon[33178]: 4.14 scrub ok 2024-09-13T13:20:44.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:44 smithi167 ceph-mon[33178]: 3.1d scrub starts 2024-09-13T13:20:44.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:44 smithi167 ceph-mon[33178]: 3.1d scrub ok 2024-09-13T13:20:44.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2203546434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:20:44.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1522820894' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:20:44.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1522820894' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:20:44.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:44 smithi167 ceph-mon[33178]: osdmap e489: 8 total, 8 up, 7 in 2024-09-13T13:20:44.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[29471]: 4.14 scrub starts 2024-09-13T13:20:44.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[29471]: 4.14 scrub ok 2024-09-13T13:20:44.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[29471]: 3.1d scrub starts 2024-09-13T13:20:44.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[29471]: 3.1d scrub ok 2024-09-13T13:20:44.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2203546434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:20:44.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1522820894' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:20:44.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1522820894' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:20:44.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[29471]: osdmap e489: 8 total, 8 up, 7 in 2024-09-13T13:20:44.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[36682]: 4.14 scrub starts 2024-09-13T13:20:44.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[36682]: 4.14 scrub ok 2024-09-13T13:20:44.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[36682]: 3.1d scrub starts 2024-09-13T13:20:44.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[36682]: 3.1d scrub ok 2024-09-13T13:20:44.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2203546434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:20:44.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1522820894' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:20:44.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1522820894' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:20:44.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:44 smithi052 ceph-mon[36682]: osdmap e489: 8 total, 8 up, 7 in 2024-09-13T13:20:44.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:44.832 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:44.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:44.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:44.934 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:44.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:44.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:44.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:45.054 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T13:20:45.057 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:45.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:20:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:20:45.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:45.087 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:45.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:45.188 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:45.223 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:45.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:45.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:45.325 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:45.353 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:45.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:45.456 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:45.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:45.496 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:45.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:45.597 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:45.627 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:45.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:45.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:45.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:45.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[29471]: pgmap v995: 168 pgs: 1 peering, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 489 op/s; 18470/177699 objects misplaced (10.394%) 2024-09-13T13:20:45.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[29471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/186447 objects degraded (0.001%), 1 pg degraded) 2024-09-13T13:20:45.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:20:45.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:20:45.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[29471]: 5.9 scrub starts 2024-09-13T13:20:45.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[29471]: 5.9 scrub ok 2024-09-13T13:20:45.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:45.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[29471]: 5.1a scrub starts 2024-09-13T13:20:45.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[29471]: 5.1a scrub ok 2024-09-13T13:20:45.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[36682]: pgmap v995: 168 pgs: 1 peering, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 489 op/s; 18470/177699 objects misplaced (10.394%) 2024-09-13T13:20:45.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[36682]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/186447 objects degraded (0.001%), 1 pg degraded) 2024-09-13T13:20:45.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:20:45.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:20:45.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[36682]: 5.9 scrub starts 2024-09-13T13:20:45.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[36682]: 5.9 scrub ok 2024-09-13T13:20:45.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:45.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[36682]: 5.1a scrub starts 2024-09-13T13:20:45.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:45 smithi052 ceph-mon[36682]: 5.1a scrub ok 2024-09-13T13:20:45.759 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:45.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:45.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:45 smithi167 ceph-mon[33178]: pgmap v995: 168 pgs: 1 peering, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 489 op/s; 18470/177699 objects misplaced (10.394%) 2024-09-13T13:20:45.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:45 smithi167 ceph-mon[33178]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/186447 objects degraded (0.001%), 1 pg degraded) 2024-09-13T13:20:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:45 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:20:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:45 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:20:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:45 smithi167 ceph-mon[33178]: 5.9 scrub starts 2024-09-13T13:20:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:45 smithi167 ceph-mon[33178]: 5.9 scrub ok 2024-09-13T13:20:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:45 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:45 smithi167 ceph-mon[33178]: 5.1a scrub starts 2024-09-13T13:20:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:45 smithi167 ceph-mon[33178]: 5.1a scrub ok 2024-09-13T13:20:45.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:45.887 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:45.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:45.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:45.989 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:46.011 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:46.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:46.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:46.113 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:46.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:46.155 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:46.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:46.256 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:46.277 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:46.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:46.379 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:46.426 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:46.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:46.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:46.477 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:20:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:20:46.528 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:46.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:46.576 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:46.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:46.677 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:46.715 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:46.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:46.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:46.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[36682]: 11.12 scrub starts 2024-09-13T13:20:46.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[36682]: 11.12 scrub ok 2024-09-13T13:20:46.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:20:46.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:20:46.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[36682]: 3.1e scrub starts 2024-09-13T13:20:46.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[36682]: 3.1e scrub ok 2024-09-13T13:20:46.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[36682]: osdmap e490: 8 total, 8 up, 7 in 2024-09-13T13:20:46.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[36682]: 5.10 scrub starts 2024-09-13T13:20:46.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[36682]: 5.10 scrub ok 2024-09-13T13:20:46.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[29471]: 11.12 scrub starts 2024-09-13T13:20:46.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[29471]: 11.12 scrub ok 2024-09-13T13:20:46.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:20:46.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:20:46.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[29471]: 3.1e scrub starts 2024-09-13T13:20:46.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[29471]: 3.1e scrub ok 2024-09-13T13:20:46.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[29471]: osdmap e490: 8 total, 8 up, 7 in 2024-09-13T13:20:46.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[29471]: 5.10 scrub starts 2024-09-13T13:20:46.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:46 smithi052 ceph-mon[29471]: 5.10 scrub ok 2024-09-13T13:20:46.818 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:46 smithi167 ceph-mon[33178]: 11.12 scrub starts 2024-09-13T13:20:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:46 smithi167 ceph-mon[33178]: 11.12 scrub ok 2024-09-13T13:20:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:46 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:20:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:46 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:20:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:46 smithi167 ceph-mon[33178]: 3.1e scrub starts 2024-09-13T13:20:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:46 smithi167 ceph-mon[33178]: 3.1e scrub ok 2024-09-13T13:20:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:46 smithi167 ceph-mon[33178]: osdmap e490: 8 total, 8 up, 7 in 2024-09-13T13:20:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:46 smithi167 ceph-mon[33178]: 5.10 scrub starts 2024-09-13T13:20:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:46 smithi167 ceph-mon[33178]: 5.10 scrub ok 2024-09-13T13:20:46.854 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:46.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:46.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:46.956 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:46.979 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:46.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:46.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:47.080 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:47.119 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:47.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:47.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:47.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:47.244 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:47.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:47.346 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:47.368 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:47.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:47.470 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:47.511 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:47.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:47.612 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:47.647 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:47.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:47.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:47.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:47 smithi052 ceph-mon[36682]: pgmap v998: 168 pgs: 1 active+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 163 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 874 op/s; 16916/173412 objects misplaced (9.755%); 9.1 MiB/s, 193 objects/s recovering 2024-09-13T13:20:47.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:47 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:20:47.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:47 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:20:47.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:47 smithi052 ceph-mon[36682]: 11.16 scrub starts 2024-09-13T13:20:47.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:47 smithi052 ceph-mon[29471]: pgmap v998: 168 pgs: 1 active+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 163 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 874 op/s; 16916/173412 objects misplaced (9.755%); 9.1 MiB/s, 193 objects/s recovering 2024-09-13T13:20:47.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:47 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:20:47.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:47 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:20:47.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:47 smithi052 ceph-mon[29471]: 11.16 scrub starts 2024-09-13T13:20:47.749 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:47.777 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:47.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:47.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:47 smithi167 ceph-mon[33178]: pgmap v998: 168 pgs: 1 active+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 163 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 874 op/s; 16916/173412 objects misplaced (9.755%); 9.1 MiB/s, 193 objects/s recovering 2024-09-13T13:20:47.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:47 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:20:47.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:47 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:20:47.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:47 smithi167 ceph-mon[33178]: 11.16 scrub starts 2024-09-13T13:20:47.878 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:47.900 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:47.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:47.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:48.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:48.025 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:48.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:48.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:48.127 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:48.153 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:48.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:48.213 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:20:48.213 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:20:48.213 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:20:48.213 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:20:48.213 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:20:48.213 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:20:48.213 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:20:48.213 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:20:48.214 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:20:48.214 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:20:48.214 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:20:48.214 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:20:48.214 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:20:48.214 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:20:48.214 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:20:48.214 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:20:48.256 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:48.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:48.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:48.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:48.415 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:48.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:48.517 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:48.544 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:48.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:48.646 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:48.675 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:48.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[36682]: 11.16 scrub ok 2024-09-13T13:20:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:20:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:20:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[36682]: osdmap e491: 8 total, 8 up, 7 in 2024-09-13T13:20:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[36682]: 6.d scrub starts 2024-09-13T13:20:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[36682]: 6.d scrub ok 2024-09-13T13:20:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[36682]: 4.1b scrub starts 2024-09-13T13:20:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[36682]: 4.1b scrub ok 2024-09-13T13:20:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[36682]: osdmap e492: 8 total, 8 up, 7 in 2024-09-13T13:20:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2229091098' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:20:48.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[36682]: from='client.36851 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:20:48.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[36682]: 6.6 scrub starts 2024-09-13T13:20:48.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[36682]: 6.6 scrub ok 2024-09-13T13:20:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[29471]: 11.16 scrub ok 2024-09-13T13:20:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:20:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:20:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[29471]: osdmap e491: 8 total, 8 up, 7 in 2024-09-13T13:20:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[29471]: 6.d scrub starts 2024-09-13T13:20:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[29471]: 6.d scrub ok 2024-09-13T13:20:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[29471]: 4.1b scrub starts 2024-09-13T13:20:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[29471]: 4.1b scrub ok 2024-09-13T13:20:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[29471]: osdmap e492: 8 total, 8 up, 7 in 2024-09-13T13:20:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2229091098' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:20:48.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[29471]: from='client.36851 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:20:48.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[29471]: 6.6 scrub starts 2024-09-13T13:20:48.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:48 smithi052 ceph-mon[29471]: 6.6 scrub ok 2024-09-13T13:20:48.777 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:48.805 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:48.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:48.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:48.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:48 smithi167 ceph-mon[33178]: 11.16 scrub ok 2024-09-13T13:20:48.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:48 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:20:48.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:48 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:20:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:48 smithi167 ceph-mon[33178]: osdmap e491: 8 total, 8 up, 7 in 2024-09-13T13:20:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:48 smithi167 ceph-mon[33178]: 6.d scrub starts 2024-09-13T13:20:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:48 smithi167 ceph-mon[33178]: 6.d scrub ok 2024-09-13T13:20:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:48 smithi167 ceph-mon[33178]: 4.1b scrub starts 2024-09-13T13:20:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:48 smithi167 ceph-mon[33178]: 4.1b scrub ok 2024-09-13T13:20:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:48 smithi167 ceph-mon[33178]: osdmap e492: 8 total, 8 up, 7 in 2024-09-13T13:20:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:48 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2229091098' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:20:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:48 smithi167 ceph-mon[33178]: from='client.36851 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:20:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:48 smithi167 ceph-mon[33178]: 6.6 scrub starts 2024-09-13T13:20:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:48 smithi167 ceph-mon[33178]: 6.6 scrub ok 2024-09-13T13:20:48.907 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:48.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:48.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:49.047 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:49.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:49.090 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:49.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:49.135 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:20:49.191 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:49.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:49.241 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:49.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:49.342 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:49.392 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:49.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:49.494 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:49.521 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:49.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:49.623 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:49.660 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:49.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:49.762 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:49.802 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:49.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:49.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:49.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: pgmap v1000: 168 pgs: 1 active+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 163 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 790 op/s; 16916/171366 objects misplaced (9.871%); 11 MiB/s, 235 objects/s recovering 2024-09-13T13:20:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: 1.7 scrub starts 2024-09-13T13:20:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: 4.2 scrub starts 2024-09-13T13:20:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: 1.7 scrub ok 2024-09-13T13:20:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: 4.2 scrub ok 2024-09-13T13:20:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: 5.d scrub starts 2024-09-13T13:20:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: 5.d scrub ok 2024-09-13T13:20:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: 3.e scrub starts 2024-09-13T13:20:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: 3.e scrub ok 2024-09-13T13:20:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:20:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: from='client.36851 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:20:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: osdmap e493: 8 total, 8 up, 7 in 2024-09-13T13:20:49.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:20:49.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:49 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:20:49.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:49.934 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:49.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:49.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:49.936 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: pgmap v1000: 168 pgs: 1 active+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 163 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 790 op/s; 16916/171366 objects misplaced (9.871%); 11 MiB/s, 235 objects/s recovering 2024-09-13T13:20:49.936 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: 1.7 scrub starts 2024-09-13T13:20:49.936 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: 4.2 scrub starts 2024-09-13T13:20:49.937 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: 1.7 scrub ok 2024-09-13T13:20:49.937 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: 4.2 scrub ok 2024-09-13T13:20:49.937 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: 5.d scrub starts 2024-09-13T13:20:49.937 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: 5.d scrub ok 2024-09-13T13:20:49.937 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: 3.e scrub starts 2024-09-13T13:20:49.937 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: 3.e scrub ok 2024-09-13T13:20:49.937 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:20:49.937 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: from='client.36851 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:20:49.937 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: osdmap e493: 8 total, 8 up, 7 in 2024-09-13T13:20:49.937 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:20:49.937 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:20:49.937 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: pgmap v1000: 168 pgs: 1 active+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 163 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 790 op/s; 16916/171366 objects misplaced (9.871%); 11 MiB/s, 235 objects/s recovering 2024-09-13T13:20:49.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: 1.7 scrub starts 2024-09-13T13:20:49.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: 4.2 scrub starts 2024-09-13T13:20:49.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: 1.7 scrub ok 2024-09-13T13:20:49.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: 4.2 scrub ok 2024-09-13T13:20:49.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: 5.d scrub starts 2024-09-13T13:20:49.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: 5.d scrub ok 2024-09-13T13:20:49.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: 3.e scrub starts 2024-09-13T13:20:49.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: 3.e scrub ok 2024-09-13T13:20:49.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:20:49.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: from='client.36851 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:20:49.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: osdmap e493: 8 total, 8 up, 7 in 2024-09-13T13:20:49.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:20:49.938 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:49 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:20:50.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:50.064 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:50.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:50.166 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:50.204 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:50.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:50.306 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:50.343 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:50.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:50.444 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:50.472 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:50.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:50.574 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:50.602 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:50.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:50.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:50.696 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T13:20:50.704 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:50.728 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:50.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:50.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:50.742 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:50.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:50 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:20:50.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:50 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:20:50.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:50 smithi167 ceph-mon[33178]: 4.b scrub starts 2024-09-13T13:20:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:50 smithi167 ceph-mon[33178]: 4.b scrub ok 2024-09-13T13:20:50.830 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:50.886 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:50.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:50.886 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:50 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:20:50.886 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:50 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:20:50.887 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:50 smithi052 ceph-mon[36682]: 4.b scrub starts 2024-09-13T13:20:50.887 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:50 smithi052 ceph-mon[36682]: 4.b scrub ok 2024-09-13T13:20:50.887 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:50 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:20:50.887 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:50 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:20:50.887 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:50 smithi052 ceph-mon[29471]: 4.b scrub starts 2024-09-13T13:20:50.887 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:50 smithi052 ceph-mon[29471]: 4.b scrub ok 2024-09-13T13:20:50.987 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:51.053 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:51.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:51.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:51.154 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:51.199 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:51.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:51.284 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:20:51.302 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:51.328 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:51.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:51.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:51.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:51.456 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:51.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:51.461 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:51.531 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:20:51.531 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:20:51.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.531 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:184055 11.9 11:92748a07:::benchmark_data_smithi052_182575_object67046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e493)", 2024-09-13T13:20:51.531 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:51.512842+0000", 2024-09-13T13:20:51.531 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.013268162999999999, 2024-09-13T13:20:51.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.013297443000000001, 2024-09-13T13:20:51.531 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:20:51.531 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:20:51.532 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:20:51.532 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:20:51.532 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:20:51.532 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:20:51.532 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 184055 2024-09-13T13:20:51.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.532 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:20:51.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.532 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:20:51.532 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.512842+0000", 2024-09-13T13:20:51.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:51.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:20:51.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.512842+0000", 2024-09-13T13:20:51.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:51.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:20:51.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.512845+0000", 2024-09-13T13:20:51.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9110000000000002e-06 2024-09-13T13:20:51.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:20:51.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.512854+0000", 2024-09-13T13:20:51.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5580000000000001e-06 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.512870+0000", 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6050999999999999e-05 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.512881+0000", 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1117e-05 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:20:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.512982+0000", 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010071100000000001 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.513115+0000", 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013290700000000001 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,6", 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.513543+0000", 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00042802500000000001 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:20:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.516062+0000", 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0025189079999999998 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.523914+0000", 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0078522880000000007 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:184063 11.8 11:17ae0ca0:::benchmark_data_smithi052_182575_object67054:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e493)", 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:51.523769+0000", 2024-09-13T13:20:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0023411130000000001, 2024-09-13T13:20:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00245195, 2024-09-13T13:20:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:20:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:20:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:20:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:20:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:20:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:20:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 184063 2024-09-13T13:20:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:20:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:20:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.523769+0000", 2024-09-13T13:20:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:20:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.523769+0000", 2024-09-13T13:20:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:20:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:20:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.523772+0000", 2024-09-13T13:20:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8700000000000001e-06 2024-09-13T13:20:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.523781+0000", 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9940000000000006e-06 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.523802+0000", 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0673999999999999e-05 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.523814+0000", 2024-09-13T13:20:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1695999999999999e-05 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.524116+0000", 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00030225100000000002 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.524236+0000", 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120282 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 5,6", 2024-09-13T13:20:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:51.524639+0000", 2024-09-13T13:20:51.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000402877 2024-09-13T13:20:51.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:51.541 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:20:51.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:51.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:20:51.541 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-09-13T13:20:51.541 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 2 2024-09-13T13:20:51.541 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:20:51.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:51.583 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:51.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:51.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:51.719 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:51.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:51.786 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[29471]: pgmap v1003: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 15306/160683 objects misplaced (9.526%) 2024-09-13T13:20:51.787 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[29471]: 5.5 scrub starts 2024-09-13T13:20:51.787 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[29471]: 5.5 scrub ok 2024-09-13T13:20:51.787 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[29471]: 1.6 scrub starts 2024-09-13T13:20:51.787 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[29471]: 1.6 scrub ok 2024-09-13T13:20:51.787 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[29471]: 3.16 scrub starts 2024-09-13T13:20:51.787 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[29471]: 3.16 scrub ok 2024-09-13T13:20:51.787 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[29471]: 5.17 scrub starts 2024-09-13T13:20:51.787 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[29471]: 5.17 scrub ok 2024-09-13T13:20:51.787 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[36682]: pgmap v1003: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 15306/160683 objects misplaced (9.526%) 2024-09-13T13:20:51.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[36682]: 5.5 scrub starts 2024-09-13T13:20:51.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[36682]: 5.5 scrub ok 2024-09-13T13:20:51.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[36682]: 1.6 scrub starts 2024-09-13T13:20:51.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[36682]: 1.6 scrub ok 2024-09-13T13:20:51.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[36682]: 3.16 scrub starts 2024-09-13T13:20:51.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[36682]: 3.16 scrub ok 2024-09-13T13:20:51.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[36682]: 5.17 scrub starts 2024-09-13T13:20:51.788 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:51 smithi052 ceph-mon[36682]: 5.17 scrub ok 2024-09-13T13:20:51.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:51 smithi167 ceph-mon[33178]: pgmap v1003: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 15306/160683 objects misplaced (9.526%) 2024-09-13T13:20:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:51 smithi167 ceph-mon[33178]: 5.5 scrub starts 2024-09-13T13:20:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:51 smithi167 ceph-mon[33178]: 5.5 scrub ok 2024-09-13T13:20:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:51 smithi167 ceph-mon[33178]: 1.6 scrub starts 2024-09-13T13:20:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:51 smithi167 ceph-mon[33178]: 1.6 scrub ok 2024-09-13T13:20:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:51 smithi167 ceph-mon[33178]: 3.16 scrub starts 2024-09-13T13:20:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:51 smithi167 ceph-mon[33178]: 3.16 scrub ok 2024-09-13T13:20:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:51 smithi167 ceph-mon[33178]: 5.17 scrub starts 2024-09-13T13:20:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:51 smithi167 ceph-mon[33178]: 5.17 scrub ok 2024-09-13T13:20:51.821 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:51.868 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:51.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:51.970 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:52.012 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:52.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:52.111 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:20:52.113 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:52.151 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:52.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:52.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:52.286 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:52.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:52.388 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:52.421 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:52.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:52.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:52.522 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:52.551 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:52.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:52.610 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T13:20:52.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:52.677 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:52.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:52.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:52.782 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:52 smithi052 ceph-mon[36682]: 4.1f scrub starts 2024-09-13T13:20:52.782 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:52 smithi052 ceph-mon[36682]: 4.1f scrub ok 2024-09-13T13:20:52.783 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:52 smithi052 ceph-mon[36682]: 5.e scrub starts 2024-09-13T13:20:52.783 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:52 smithi052 ceph-mon[36682]: 5.e scrub ok 2024-09-13T13:20:52.783 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:52 smithi052 ceph-mon[36682]: 4.19 scrub starts 2024-09-13T13:20:52.783 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:52 smithi052 ceph-mon[36682]: 4.19 scrub ok 2024-09-13T13:20:52.783 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:52 smithi052 ceph-mon[29471]: 4.1f scrub starts 2024-09-13T13:20:52.783 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:52 smithi052 ceph-mon[29471]: 4.1f scrub ok 2024-09-13T13:20:52.783 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:52 smithi052 ceph-mon[29471]: 5.e scrub starts 2024-09-13T13:20:52.784 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:52 smithi052 ceph-mon[29471]: 5.e scrub ok 2024-09-13T13:20:52.784 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:52 smithi052 ceph-mon[29471]: 4.19 scrub starts 2024-09-13T13:20:52.784 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:52 smithi052 ceph-mon[29471]: 4.19 scrub ok 2024-09-13T13:20:52.803 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:52.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:52.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:52.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:52 smithi167 ceph-mon[33178]: 4.1f scrub starts 2024-09-13T13:20:52.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:52 smithi167 ceph-mon[33178]: 4.1f scrub ok 2024-09-13T13:20:52.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:52 smithi167 ceph-mon[33178]: 5.e scrub starts 2024-09-13T13:20:52.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:52 smithi167 ceph-mon[33178]: 5.e scrub ok 2024-09-13T13:20:52.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:52 smithi167 ceph-mon[33178]: 4.19 scrub starts 2024-09-13T13:20:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:52 smithi167 ceph-mon[33178]: 4.19 scrub ok 2024-09-13T13:20:52.880 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:52.903 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:52.905 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:52.986 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:52.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:52.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:53.088 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:53.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:53.140 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:53.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:53.241 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:53.267 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:53.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:53.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:53.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:53.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:53.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:53.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:53.498 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:53.521 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:53.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:53.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:53.596 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:20:53.596 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:20:53.623 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:53.631 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:20:53.631 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:20:53.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[36682]: pgmap v1004: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 15306/158343 objects misplaced (9.666%) 2024-09-13T13:20:53.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[36682]: 5.6 scrub starts 2024-09-13T13:20:53.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[36682]: 5.6 scrub ok 2024-09-13T13:20:53.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[36682]: 4.9 scrub starts 2024-09-13T13:20:53.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[36682]: 4.9 scrub ok 2024-09-13T13:20:53.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[36682]: osdmap e494: 8 total, 8 up, 7 in 2024-09-13T13:20:53.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[36682]: 6.11 scrub starts 2024-09-13T13:20:53.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[36682]: 6.11 scrub ok 2024-09-13T13:20:53.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/105996374' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:20:53.648 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[36682]: from='client.36857 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:20:53.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[29471]: pgmap v1004: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 15306/158343 objects misplaced (9.666%) 2024-09-13T13:20:53.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[29471]: 5.6 scrub starts 2024-09-13T13:20:53.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[29471]: 5.6 scrub ok 2024-09-13T13:20:53.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[29471]: 4.9 scrub starts 2024-09-13T13:20:53.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[29471]: 4.9 scrub ok 2024-09-13T13:20:53.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[29471]: osdmap e494: 8 total, 8 up, 7 in 2024-09-13T13:20:53.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[29471]: 6.11 scrub starts 2024-09-13T13:20:53.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[29471]: 6.11 scrub ok 2024-09-13T13:20:53.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/105996374' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:20:53.649 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:53 smithi052 ceph-mon[29471]: from='client.36857 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:20:53.651 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:53.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:53.669 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:20:53.669 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:20:53.701 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:20:53.702 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:20:53.733 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:20:53.733 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:20:53.753 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:53.769 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:20:53.769 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:20:53.779 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:53.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:53.805 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:20:53.805 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:20:53.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:53 smithi167 ceph-mon[33178]: pgmap v1004: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 15306/158343 objects misplaced (9.666%) 2024-09-13T13:20:53.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:53 smithi167 ceph-mon[33178]: 5.6 scrub starts 2024-09-13T13:20:53.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:53 smithi167 ceph-mon[33178]: 5.6 scrub ok 2024-09-13T13:20:53.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:53 smithi167 ceph-mon[33178]: 4.9 scrub starts 2024-09-13T13:20:53.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:53 smithi167 ceph-mon[33178]: 4.9 scrub ok 2024-09-13T13:20:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:53 smithi167 ceph-mon[33178]: osdmap e494: 8 total, 8 up, 7 in 2024-09-13T13:20:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:53 smithi167 ceph-mon[33178]: 6.11 scrub starts 2024-09-13T13:20:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:53 smithi167 ceph-mon[33178]: 6.11 scrub ok 2024-09-13T13:20:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:53 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/105996374' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:20:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:53 smithi167 ceph-mon[33178]: from='client.36857 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:20:53.840 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:20:53.840 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:20:53.881 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:53.929 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:53.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:54.031 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:54.067 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:54.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:54.150 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T13:20:54.169 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:54.194 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:54.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:54.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:54.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:54.322 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:54.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:54.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:54.417 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:54.424 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:54.449 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:54.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:54.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:54.551 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:54.595 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:54.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:54.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:54.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:54.728 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:54.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:54.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:20:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:20:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:54 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3236252133' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:20:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:54 smithi167 ceph-mon[33178]: 3.14 scrub starts 2024-09-13T13:20:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:54 smithi167 ceph-mon[33178]: 3.14 scrub ok 2024-09-13T13:20:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:54 smithi167 ceph-mon[33178]: from='client.36857 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:20:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:54 smithi167 ceph-mon[33178]: osdmap e495: 8 total, 8 up, 7 in 2024-09-13T13:20:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:54 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/812146688' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:20:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:54 smithi167 ceph-mon[33178]: from='client.26767 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:20:54.830 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:54.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:54.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:54.943 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3236252133' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:20:54.944 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[29471]: 3.14 scrub starts 2024-09-13T13:20:54.944 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[29471]: 3.14 scrub ok 2024-09-13T13:20:54.944 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[29471]: from='client.36857 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:20:54.944 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[29471]: osdmap e495: 8 total, 8 up, 7 in 2024-09-13T13:20:54.944 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/812146688' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:20:54.944 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[29471]: from='client.26767 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:20:54.944 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3236252133' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:20:54.944 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[36682]: 3.14 scrub starts 2024-09-13T13:20:54.944 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[36682]: 3.14 scrub ok 2024-09-13T13:20:54.944 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[36682]: from='client.36857 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:20:54.945 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[36682]: osdmap e495: 8 total, 8 up, 7 in 2024-09-13T13:20:54.945 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/812146688' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:20:54.945 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:54 smithi052 ceph-mon[36682]: from='client.26767 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:20:54.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:20:54.985 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:54.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:54.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:20:55.087 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:55.113 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:55.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:55.151 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:20:55.215 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:55.247 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:55.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:55.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:55.372 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:55.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:55.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:55.474 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:55.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:55.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:55.614 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:55.645 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:55.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:55.747 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:55.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:55.786 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:55.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:55.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:55 smithi167 ceph-mon[33178]: pgmap v1006: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 3.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 14802/148785 objects misplaced (9.949%) 2024-09-13T13:20:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:55 smithi167 ceph-mon[33178]: 3.6 scrub starts 2024-09-13T13:20:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:55 smithi167 ceph-mon[33178]: 3.6 scrub ok 2024-09-13T13:20:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:55 smithi167 ceph-mon[33178]: 5.4 scrub starts 2024-09-13T13:20:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:55 smithi167 ceph-mon[33178]: 5.4 scrub ok 2024-09-13T13:20:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:55 smithi167 ceph-mon[33178]: 3.d scrub starts 2024-09-13T13:20:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:55 smithi167 ceph-mon[33178]: 3.d scrub ok 2024-09-13T13:20:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:55 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:20:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:55 smithi167 ceph-mon[33178]: from='client.26767 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:20:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:55 smithi167 ceph-mon[33178]: osdmap e496: 8 total, 8 up, 7 in 2024-09-13T13:20:55.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[29471]: pgmap v1006: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 3.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 14802/148785 objects misplaced (9.949%) 2024-09-13T13:20:55.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[29471]: 3.6 scrub starts 2024-09-13T13:20:55.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[29471]: 3.6 scrub ok 2024-09-13T13:20:55.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[29471]: 5.4 scrub starts 2024-09-13T13:20:55.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[29471]: 5.4 scrub ok 2024-09-13T13:20:55.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[29471]: 3.d scrub starts 2024-09-13T13:20:55.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[29471]: 3.d scrub ok 2024-09-13T13:20:55.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:55.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:20:55.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[29471]: from='client.26767 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:20:55.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[29471]: osdmap e496: 8 total, 8 up, 7 in 2024-09-13T13:20:55.834 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[36682]: pgmap v1006: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 3.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 14802/148785 objects misplaced (9.949%) 2024-09-13T13:20:55.834 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[36682]: 3.6 scrub starts 2024-09-13T13:20:55.834 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[36682]: 3.6 scrub ok 2024-09-13T13:20:55.834 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[36682]: 5.4 scrub starts 2024-09-13T13:20:55.834 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[36682]: 5.4 scrub ok 2024-09-13T13:20:55.834 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[36682]: 3.d scrub starts 2024-09-13T13:20:55.834 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[36682]: 3.d scrub ok 2024-09-13T13:20:55.834 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:20:55.834 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:20:55.834 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[36682]: from='client.26767 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:20:55.834 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:55 smithi052 ceph-mon[36682]: osdmap e496: 8 total, 8 up, 7 in 2024-09-13T13:20:55.888 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:55.891 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T13:20:55.930 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:55.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:56.033 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:56.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:56.084 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:56.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:56.185 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:56.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:56.216 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:56.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:56.318 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:56.350 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:56.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:56.404 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:20:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:20:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:20:56.452 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:56.477 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:56.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:56.579 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:56.628 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:56.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:56.690 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:56 smithi052 ceph-mon[36682]: 11.17 scrub starts 2024-09-13T13:20:56.691 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:56 smithi052 ceph-mon[29471]: 11.17 scrub starts 2024-09-13T13:20:56.730 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:56.767 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:56.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:56.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:56.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:56 smithi167 ceph-mon[33178]: 11.17 scrub starts 2024-09-13T13:20:56.867 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:56.868 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:20:56.869 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:20:56.870 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:56.930 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:56.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:56.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:56.993 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:20:57.032 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:57.092 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:57.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:57.193 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:57.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:57.221 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:57.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:57.322 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:57.347 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:57.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:57.449 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:57.474 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:57.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:57.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:57.600 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:57.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:57.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:57.665 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:57 smithi052 ceph-mon[29471]: pgmap v1009: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 921 op/s; 13740/144102 objects misplaced (9.535%) 2024-09-13T13:20:57.665 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:57 smithi052 ceph-mon[36682]: pgmap v1009: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 921 op/s; 13740/144102 objects misplaced (9.535%) 2024-09-13T13:20:57.702 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:20:57.724 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T13:20:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:57.763 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:57.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:20:57.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:57 smithi167 ceph-mon[33178]: pgmap v1009: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 921 op/s; 13740/144102 objects misplaced (9.535%) 2024-09-13T13:20:57.864 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:57.911 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:57.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:58.013 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:20:58.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:58.082 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:58.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:20:58.183 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:58.231 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:58.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:20:58.319 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:20:58.332 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:58.343 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:58.354 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:58.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:58.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:58.456 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:58.484 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:58.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:58.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:58.611 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:58.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:58.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:58.713 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:58.745 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:58.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:58.847 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:58.883 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:58.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:58.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:58.985 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:59.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:59.022 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:59.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:59.124 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:59.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:59.163 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:59.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:59.265 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:20:59.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:59.291 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:59.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:20:59.392 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:59.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:59.441 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:59.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:59.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:20:59.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:59.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:20:59.573 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T13:20:59.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:20:59.677 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:59 smithi052 ceph-mon[29471]: pgmap v1010: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 938 op/s; 13740/141456 objects misplaced (9.713%) 2024-09-13T13:20:59.677 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:20:59 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2738400112' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:20:59.677 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:59 smithi052 ceph-mon[36682]: pgmap v1010: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 938 op/s; 13740/141456 objects misplaced (9.713%) 2024-09-13T13:20:59.677 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:20:59 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2738400112' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:20:59.696 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:20:59.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:59.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:20:59.775 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:20:59.776 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:20:59.798 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:20:59.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:59 smithi167 ceph-mon[33178]: pgmap v1010: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 938 op/s; 13740/141456 objects misplaced (9.713%) 2024-09-13T13:20:59.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:20:59 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2738400112' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:20:59.825 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:20:59.825 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:20:59.845 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:59.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:59.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:20:59.873 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:20:59.873 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:20:59.908 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:20:59.909 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:20:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:20:59.946 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:20:59.957 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:20:59.958 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:20:59.997 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:20:59.997 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:20:59.997 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:20:59.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:20:59.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:00.031 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:21:00.031 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:21:00.064 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:21:00.064 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:21:00.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:00.135 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:00.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:00.236 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:00.276 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:00.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:00.378 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:00.406 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:00.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:00.508 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:00.546 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T13:21:00.558 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:00.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:00.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:00.662 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:00.690 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:00.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:00 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2738400112' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:21:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:00 smithi052 ceph-mon[29471]: osdmap e497: 8 total, 8 up, 7 in 2024-09-13T13:21:00.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:00 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2145865014' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:00.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:00 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2738400112' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:21:00.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:00 smithi052 ceph-mon[36682]: osdmap e497: 8 total, 8 up, 7 in 2024-09-13T13:21:00.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:00 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2145865014' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:00.791 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:00.816 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:00.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:00.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:00.918 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:00.970 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:00.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:00.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:00 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2738400112' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:21:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:00 smithi167 ceph-mon[33178]: osdmap e497: 8 total, 8 up, 7 in 2024-09-13T13:21:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:00 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2145865014' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:01.072 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:01.117 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:01.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:01.219 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:01.260 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:01.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:01.362 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:01.389 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:01.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:01.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:01.517 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:01.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:01.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:01.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:01.648 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:01.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:01.749 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:01.789 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:01.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:01.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:01.891 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:01.917 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:01.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:01.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:01 smithi052 ceph-mon[36682]: pgmap v1012: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 12141/126591 objects misplaced (9.591%) 2024-09-13T13:21:01.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:01 smithi052 ceph-mon[29471]: pgmap v1012: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 12141/126591 objects misplaced (9.591%) 2024-09-13T13:21:02.019 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:02.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:02.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:02.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:02.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:01 smithi167 ceph-mon[33178]: pgmap v1012: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 12141/126591 objects misplaced (9.591%) 2024-09-13T13:21:02.152 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:02.183 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:02.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:02.285 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:02.313 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:02.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:02.414 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:02.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:02.456 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:02.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:02.557 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:02.590 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:02.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:02.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:02.692 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:02.719 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:02.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:02.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:02.821 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:02.861 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:02.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:02.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:02.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:02.966 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:02 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:21:02.966 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:02 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:21:02.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:02 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:21:02.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:02 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:21:02.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:02 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:21:02.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:02 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:21:02.991 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:02.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:02.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:03.094 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:03.133 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:03.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:03.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:03.235 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:03.277 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:03.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:03.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:03.379 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:03.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:03.504 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:03.535 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:03.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:03.637 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:03.660 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:03.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:03.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:03.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:03.791 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:03.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:03.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:03.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:03.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:03.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:03 smithi052 ceph-mon[36682]: pgmap v1013: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.05k op/s; 12141/123804 objects misplaced (9.807%) 2024-09-13T13:21:03.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:03 smithi052 ceph-mon[36682]: osdmap e498: 8 total, 8 up, 7 in 2024-09-13T13:21:03.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:03 smithi052 ceph-mon[29471]: pgmap v1013: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.05k op/s; 12141/123804 objects misplaced (9.807%) 2024-09-13T13:21:03.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:03 smithi052 ceph-mon[29471]: osdmap e498: 8 total, 8 up, 7 in 2024-09-13T13:21:04.030 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:04.052 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:04.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:04.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:03 smithi167 ceph-mon[33178]: pgmap v1013: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.05k op/s; 12141/123804 objects misplaced (9.807%) 2024-09-13T13:21:04.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:03 smithi167 ceph-mon[33178]: osdmap e498: 8 total, 8 up, 7 in 2024-09-13T13:21:04.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:04.182 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:04.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:04.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:04.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:04.311 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:04.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:04.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:04.369 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:04.413 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:04.466 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:04.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:04.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:04.618 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:04.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:04.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:04.721 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:04.760 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:04.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:04.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:04.889 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:04.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:04.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:04.991 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:05.039 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:05.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:05.058 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:21:05.058 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:21:05.058 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:21:05.058 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:21:05.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.058 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190217 9.c 9:3da1d190:::benchmark_data_smithi052_128427_object59548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:21:05.059 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:43.982184+0000", 2024-09-13T13:21:05.059 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 497.48725365500002, 2024-09-13T13:21:05.059 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99036358000000002, 2024-09-13T13:21:05.059 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.059 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.059 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.059 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.059 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.059 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.059 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190217 2024-09-13T13:21:05.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.059 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982184+0000", 2024-09-13T13:21:05.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982184+0000", 2024-09-13T13:21:05.060 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.060 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.060 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982185+0000", 2024-09-13T13:21:05.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5859999999999999e-06 2024-09-13T13:21:05.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982190+0000", 2024-09-13T13:21:05.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9749999999999998e-06 2024-09-13T13:21:05.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.061 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.061 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982200+0000", 2024-09-13T13:21:05.061 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2310000000000002e-06 2024-09-13T13:21:05.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982209+0000", 2024-09-13T13:21:05.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9220000000000003e-06 2024-09-13T13:21:05.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.062 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982540+0000", 2024-09-13T13:21:05.062 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033174499999999999 2024-09-13T13:21:05.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.062 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982550+0000", 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5629999999999998e-06 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.966848+0000", 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.9842978 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.966978+0000", 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013044200000000001 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:21:05.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.967121+0000", 2024-09-13T13:21:05.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000143096 2024-09-13T13:21:05.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972248+0000", 2024-09-13T13:21:05.064 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051265260000000002 2024-09-13T13:21:05.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.064 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.064 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972262+0000", 2024-09-13T13:21:05.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4037e-05 2024-09-13T13:21:05.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972452+0000", 2024-09-13T13:21:05.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019067699999999999 2024-09-13T13:21:05.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.065 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.065 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972517+0000", 2024-09-13T13:21:05.065 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4935000000000001e-05 2024-09-13T13:21:05.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.066 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.066 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972547+0000", 2024-09-13T13:21:05.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0045000000000001e-05 2024-09-13T13:21:05.066 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.066 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.066 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.066 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190219 9.c 9:30476813:::benchmark_data_smithi052_128427_object59550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:21:05.066 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:43.982580+0000", 2024-09-13T13:21:05.066 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 497.48685751300002, 2024-09-13T13:21:05.066 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.990151738, 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190219 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982580+0000", 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.068 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982580+0000", 2024-09-13T13:21:05.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.068 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982581+0000", 2024-09-13T13:21:05.068 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0249999999999999e-06 2024-09-13T13:21:05.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.068 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.068 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982585+0000", 2024-09-13T13:21:05.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1189999999999999e-06 2024-09-13T13:21:05.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.069 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.069 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982595+0000", 2024-09-13T13:21:05.069 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0074999999999999e-05 2024-09-13T13:21:05.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.069 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.069 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982601+0000", 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2149999999999998e-06 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982868+0000", 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00026633000000000001 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982876+0000", 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8900000000000007e-06 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.070 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.969658+0000", 2024-09-13T13:21:05.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98678257999999996 2024-09-13T13:21:05.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.969792+0000", 2024-09-13T13:21:05.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013425500000000001 2024-09-13T13:21:05.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.071 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:21:05.071 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.969926+0000", 2024-09-13T13:21:05.071 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013390200000000001 2024-09-13T13:21:05.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972556+0000", 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026292170000000001 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972600+0000", 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4697999999999998e-05 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972657+0000", 2024-09-13T13:21:05.072 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6453000000000003e-05 2024-09-13T13:21:05.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972711+0000", 2024-09-13T13:21:05.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.452e-05 2024-09-13T13:21:05.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.073 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.073 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972732+0000", 2024-09-13T13:21:05.073 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0458999999999998e-05 2024-09-13T13:21:05.073 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.073 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190252 9.c 9:3916637a:::benchmark_data_smithi052_128427_object59583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.013719+0000", 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 497.45571878700002, 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96423604299999999, 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.074 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190252 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013719+0000", 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013719+0000", 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.075 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013720+0000", 2024-09-13T13:21:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7039999999999999e-06 2024-09-13T13:21:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013726+0000", 2024-09-13T13:21:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3299999999999998e-06 2024-09-13T13:21:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013745+0000", 2024-09-13T13:21:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9623000000000002e-05 2024-09-13T13:21:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013753+0000", 2024-09-13T13:21:05.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.498e-06 2024-09-13T13:21:05.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.014204+0000", 2024-09-13T13:21:05.077 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00045132700000000001 2024-09-13T13:21:05.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.077 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.077 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.014212+0000", 2024-09-13T13:21:05.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1510000000000004e-06 2024-09-13T13:21:05.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972762+0000", 2024-09-13T13:21:05.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.958549706 2024-09-13T13:21:05.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.078 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.078 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972843+0000", 2024-09-13T13:21:05.078 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1415000000000006e-05 2024-09-13T13:21:05.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:21:05.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972915+0000", 2024-09-13T13:21:05.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:21:05.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976630+0000", 2024-09-13T13:21:05.079 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037151879999999999 2024-09-13T13:21:05.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.079 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.079 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976739+0000", 2024-09-13T13:21:05.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109011 2024-09-13T13:21:05.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.977890+0000", 2024-09-13T13:21:05.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011513350000000001 2024-09-13T13:21:05.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.080 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.080 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.977916+0000", 2024-09-13T13:21:05.080 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5948e-05 2024-09-13T13:21:05.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.977955+0000", 2024-09-13T13:21:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8451999999999997e-05 2024-09-13T13:21:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190263 9.c 9:307a9428:::benchmark_data_smithi052_128427_object59594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:21:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.022090+0000", 2024-09-13T13:21:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 497.447347858, 2024-09-13T13:21:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95977093599999996, 2024-09-13T13:21:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.082 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.082 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.082 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190263 2024-09-13T13:21:05.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.082 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.082 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.082 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022090+0000", 2024-09-13T13:21:05.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022090+0000", 2024-09-13T13:21:05.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022092+0000", 2024-09-13T13:21:05.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8870000000000001e-06 2024-09-13T13:21:05.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022097+0000", 2024-09-13T13:21:05.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.31e-06 2024-09-13T13:21:05.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022108+0000", 2024-09-13T13:21:05.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0941000000000001e-05 2024-09-13T13:21:05.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022115+0000", 2024-09-13T13:21:05.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3050000000000004e-06 2024-09-13T13:21:05.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022210+0000", 2024-09-13T13:21:05.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5169999999999999e-05 2024-09-13T13:21:05.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022219+0000", 2024-09-13T13:21:05.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1500000000000005e-06 2024-09-13T13:21:05.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.974786+0000", 2024-09-13T13:21:05.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95256630799999997 2024-09-13T13:21:05.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.974884+0000", 2024-09-13T13:21:05.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8256000000000003e-05 2024-09-13T13:21:05.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:21:05.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.975003+0000", 2024-09-13T13:21:05.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011911100000000001 2024-09-13T13:21:05.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976679+0000", 2024-09-13T13:21:05.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016762019999999999 2024-09-13T13:21:05.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.978004+0000", 2024-09-13T13:21:05.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001324605 2024-09-13T13:21:05.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981764+0000", 2024-09-13T13:21:05.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003760556 2024-09-13T13:21:05.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981816+0000", 2024-09-13T13:21:05.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1412999999999997e-05 2024-09-13T13:21:05.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981861+0000", 2024-09-13T13:21:05.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4722000000000003e-05 2024-09-13T13:21:05.088 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.088 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.089 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190265 9.c 9:3c2318c0:::benchmark_data_smithi052_128427_object59596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:21:05.089 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.022643+0000", 2024-09-13T13:21:05.089 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 497.44679479899997, 2024-09-13T13:21:05.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95946595400000001, 2024-09-13T13:21:05.089 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.089 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.089 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.089 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.089 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.089 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190265 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022643+0000", 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022643+0000", 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022644+0000", 2024-09-13T13:21:05.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.282e-06 2024-09-13T13:21:05.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022648+0000", 2024-09-13T13:21:05.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3189999999999996e-06 2024-09-13T13:21:05.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022657+0000", 2024-09-13T13:21:05.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1710000000000008e-06 2024-09-13T13:21:05.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022664+0000", 2024-09-13T13:21:05.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8600000000000004e-06 2024-09-13T13:21:05.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022843+0000", 2024-09-13T13:21:05.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017918 2024-09-13T13:21:05.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022851+0000", 2024-09-13T13:21:05.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5970000000000003e-06 2024-09-13T13:21:05.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976779+0000", 2024-09-13T13:21:05.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95392760200000004 2024-09-13T13:21:05.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976846+0000", 2024-09-13T13:21:05.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7539000000000005e-05 2024-09-13T13:21:05.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:21:05.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976913+0000", 2024-09-13T13:21:05.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6785999999999994e-05 2024-09-13T13:21:05.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981913+0000", 2024-09-13T13:21:05.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050001179999999996 2024-09-13T13:21:05.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981978+0000", 2024-09-13T13:21:05.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4917000000000002e-05 2024-09-13T13:21:05.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982009+0000", 2024-09-13T13:21:05.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0499999999999999e-05 2024-09-13T13:21:05.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982067+0000", 2024-09-13T13:21:05.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8573000000000003e-05 2024-09-13T13:21:05.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982109+0000", 2024-09-13T13:21:05.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1510000000000001e-05 2024-09-13T13:21:05.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.096 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.096 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190292 9.c 9:3187c4a7:::benchmark_data_smithi052_128427_object59623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:21:05.096 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.039937+0000", 2024-09-13T13:21:05.097 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 497.42950084699999, 2024-09-13T13:21:05.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95233577000000003, 2024-09-13T13:21:05.097 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.097 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.097 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.097 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.097 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.097 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.097 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190292 2024-09-13T13:21:05.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.097 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039937+0000", 2024-09-13T13:21:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039937+0000", 2024-09-13T13:21:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039938+0000", 2024-09-13T13:21:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.776e-06 2024-09-13T13:21:05.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039944+0000", 2024-09-13T13:21:05.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5949999999999998e-06 2024-09-13T13:21:05.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039959+0000", 2024-09-13T13:21:05.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4854e-05 2024-09-13T13:21:05.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039966+0000", 2024-09-13T13:21:05.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2239999999999998e-06 2024-09-13T13:21:05.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.040018+0000", 2024-09-13T13:21:05.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1578e-05 2024-09-13T13:21:05.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.040025+0000", 2024-09-13T13:21:05.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6690000000000006e-06 2024-09-13T13:21:05.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982498+0000", 2024-09-13T13:21:05.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.94247312100000002 2024-09-13T13:21:05.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982590+0000", 2024-09-13T13:21:05.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1283e-05 2024-09-13T13:21:05.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:21:05.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982692+0000", 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102144 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.987375+0000", 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0046834329999999999 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.987419+0000", 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3282999999999999e-05 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992172+0000", 2024-09-13T13:21:05.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047538839999999999 2024-09-13T13:21:05.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992231+0000", 2024-09-13T13:21:05.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8329999999999999e-05 2024-09-13T13:21:05.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992272+0000", 2024-09-13T13:21:05.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1596e-05 2024-09-13T13:21:05.104 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.104 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.104 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.104 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190295 9.c 9:3895a0c0:::benchmark_data_smithi052_128427_object59626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:21:05.104 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.041745+0000", 2024-09-13T13:21:05.104 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 497.42769234899998, 2024-09-13T13:21:05.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96494661000000004, 2024-09-13T13:21:05.105 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.105 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.105 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.105 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.105 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.105 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.105 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190295 2024-09-13T13:21:05.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.105 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041745+0000", 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041745+0000", 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041746+0000", 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2619999999999999e-06 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041751+0000", 2024-09-13T13:21:05.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.216e-06 2024-09-13T13:21:05.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041759+0000", 2024-09-13T13:21:05.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7129999999999994e-06 2024-09-13T13:21:05.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041805+0000", 2024-09-13T13:21:05.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5739999999999999e-05 2024-09-13T13:21:05.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041899+0000", 2024-09-13T13:21:05.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3598999999999996e-05 2024-09-13T13:21:05.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041907+0000", 2024-09-13T13:21:05.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4169999999999992e-06 2024-09-13T13:21:05.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.984980+0000", 2024-09-13T13:21:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.94307334399999998 2024-09-13T13:21:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.985078+0000", 2024-09-13T13:21:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7633000000000005e-05 2024-09-13T13:21:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:21:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.985188+0000", 2024-09-13T13:21:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110245 2024-09-13T13:21:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989611+0000", 2024-09-13T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0044224889999999999 2024-09-13T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.994760+0000", 2024-09-13T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051493700000000003 2024-09-13T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.006626+0000", 2024-09-13T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011865504000000001 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.006651+0000", 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5629000000000001e-05 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.006692+0000", 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0448999999999998e-05 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.112 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190317 9.c 9:34b3085e:::benchmark_data_smithi052_128427_object59648:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:21:05.112 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.058087+0000", 2024-09-13T13:21:05.112 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 497.41135075800003, 2024-09-13T13:21:05.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95460188400000001, 2024-09-13T13:21:05.112 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.112 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.112 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.112 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.112 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.112 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.112 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190317 2024-09-13T13:21:05.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058087+0000", 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058087+0000", 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058088+0000", 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.192e-06 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058093+0000", 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7380000000000002e-06 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058102+0000", 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6590000000000002e-06 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058109+0000", 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9770000000000003e-06 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058175+0000", 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5343000000000001e-05 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058182+0000", 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8059999999999995e-06 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989661+0000", 2024-09-13T13:21:05.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.931478994 2024-09-13T13:21:05.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989768+0000", 2024-09-13T13:21:05.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010650900000000001 2024-09-13T13:21:05.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:21:05.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989877+0000", 2024-09-13T13:21:05.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109343 2024-09-13T13:21:05.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.994834+0000", 2024-09-13T13:21:05.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049566610000000002 2024-09-13T13:21:05.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.994870+0000", 2024-09-13T13:21:05.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6399999999999997e-05 2024-09-13T13:21:05.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.012452+0000", 2024-09-13T13:21:05.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.017581513999999999 2024-09-13T13:21:05.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.012635+0000", 2024-09-13T13:21:05.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018336900000000001 2024-09-13T13:21:05.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.012689+0000", 2024-09-13T13:21:05.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3378999999999998e-05 2024-09-13T13:21:05.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.119 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.119 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190320 9.c 9:35ad6b23:::benchmark_data_smithi052_128427_object59651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:21:05.119 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.059653+0000", 2024-09-13T13:21:05.119 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 497.40978458699999, 2024-09-13T13:21:05.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95393949300000003, 2024-09-13T13:21:05.119 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.119 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.119 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.119 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.120 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.120 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.120 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190320 2024-09-13T13:21:05.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.120 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.120 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059653+0000", 2024-09-13T13:21:05.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.120 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059653+0000", 2024-09-13T13:21:05.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059654+0000", 2024-09-13T13:21:05.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1370000000000001e-06 2024-09-13T13:21:05.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059658+0000", 2024-09-13T13:21:05.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.755e-06 2024-09-13T13:21:05.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059668+0000", 2024-09-13T13:21:05.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8709999999999992e-06 2024-09-13T13:21:05.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059675+0000", 2024-09-13T13:21:05.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5870000000000004e-06 2024-09-13T13:21:05.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059730+0000", 2024-09-13T13:21:05.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5180999999999999e-05 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059739+0000", 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0569999999999999e-06 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992325+0000", 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.93258522899999996 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992426+0000", 2024-09-13T13:21:05.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101094 2024-09-13T13:21:05.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:21:05.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992530+0000", 2024-09-13T13:21:05.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104051 2024-09-13T13:21:05.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.999704+0000", 2024-09-13T13:21:05.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0071743629999999996 2024-09-13T13:21:05.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.004507+0000", 2024-09-13T13:21:05.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048030340000000003 2024-09-13T13:21:05.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.013507+0000", 2024-09-13T13:21:05.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0089995439999999999 2024-09-13T13:21:05.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.013555+0000", 2024-09-13T13:21:05.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8572000000000002e-05 2024-09-13T13:21:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.013592+0000", 2024-09-13T13:21:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7017999999999997e-05 2024-09-13T13:21:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202437 9.c 9:31d379dd:::benchmark_data_smithi052_128427_object71768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-13T13:21:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.936285+0000", 2024-09-13T13:21:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 483.53315270000002, 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0668128299999999, 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202437 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936285+0000", 2024-09-13T13:21:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936285+0000", 2024-09-13T13:21:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936286+0000", 2024-09-13T13:21:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5540000000000001e-06 2024-09-13T13:21:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936291+0000", 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0100000000000003e-06 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936305+0000", 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3562e-05 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936312+0000", 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3780000000000003e-06 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936374+0000", 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1909999999999995e-05 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936528+0000", 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015417 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4,7", 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936642+0000", 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011353000000000001 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.941575+0000", 2024-09-13T13:21:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049326969999999998 2024-09-13T13:21:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.947466+0000", 2024-09-13T13:21:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058910960000000002 2024-09-13T13:21:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.947475+0000", 2024-09-13T13:21:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6350000000000001e-06 2024-09-13T13:21:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.990524+0000", 2024-09-13T13:21:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0430484790000001 2024-09-13T13:21:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for missing object", 2024-09-13T13:21:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.990633+0000", 2024-09-13T13:21:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108872 2024-09-13T13:21:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003024+0000", 2024-09-13T13:21:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.012390988 2024-09-13T13:21:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003098+0000", 2024-09-13T13:21:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3949000000000006e-05 2024-09-13T13:21:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202453 9.c 9:3d8e06d1:::benchmark_data_smithi052_128427_object71784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.956183+0000", 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 483.51325447900001, 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.056296729, 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202453 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956183+0000", 2024-09-13T13:21:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956183+0000", 2024-09-13T13:21:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956185+0000", 2024-09-13T13:21:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.784e-06 2024-09-13T13:21:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956190+0000", 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956200+0000", 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2900000000000008e-06 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956206+0000", 2024-09-13T13:21:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1489999999999999e-06 2024-09-13T13:21:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957496+0000", 2024-09-13T13:21:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012905519999999999 2024-09-13T13:21:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957505+0000", 2024-09-13T13:21:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9930000000000001e-06 2024-09-13T13:21:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003310+0000", 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.045804368 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003406+0000", 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6335000000000005e-05 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003506+0000", 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000100295 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012143+0000", 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0086370389999999991 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012233+0000", 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9973000000000001e-05 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012279+0000", 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5966e-05 2024-09-13T13:21:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012376+0000", 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6220999999999999e-05 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012480+0000", 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010412900000000001 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202455 9.c 9:304f4878:::benchmark_data_smithi052_128427_object71786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 483.51321957200003, 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.071265975, 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202455 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5000000000000003e-07 2024-09-13T13:21:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956221+0000", 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.441e-06 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956226+0000", 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0089999999999998e-06 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956230+0000", 2024-09-13T13:21:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5410000000000002e-06 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957529+0000", 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012988469999999999 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957535+0000", 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8109999999999999e-06 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.005874+0000", 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0483387770000001 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.005991+0000", 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116669 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.006107+0000", 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011618 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012522+0000", 2024-09-13T13:21:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064157060000000002 2024-09-13T13:21:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012562+0000", 2024-09-13T13:21:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9756999999999999e-05 2024-09-13T13:21:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.026940+0000", 2024-09-13T13:21:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014377463 2024-09-13T13:21:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027300+0000", 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00036073000000000002 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027484+0000", 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000183494 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202464 9.c 9:3602984d:::benchmark_data_smithi052_128427_object71795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 483.50815367199999, 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.066828723, 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202464 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4099999999999999e-07 2024-09-13T13:21:05.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961286+0000", 2024-09-13T13:21:05.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2010000000000002e-06 2024-09-13T13:21:05.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961291+0000", 2024-09-13T13:21:05.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5909999999999998e-06 2024-09-13T13:21:05.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961295+0000", 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1110000000000001e-06 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.962375+0000", 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00107939 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.962382+0000", 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5900000000000002e-06 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.008652+0000", 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0462698880000001 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.008749+0000", 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7125000000000003e-05 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.008858+0000", 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010874699999999999 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012711+0000", 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0038533619999999999 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012755+0000", 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3773000000000003e-05 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027802+0000", 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015047115 2024-09-13T13:21:05.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027905+0000", 2024-09-13T13:21:05.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103278 2024-09-13T13:21:05.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028113+0000", 2024-09-13T13:21:05.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000207111 2024-09-13T13:21:05.154 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.154 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.154 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202480 9.c 9:3e2ac5d3:::benchmark_data_smithi052_128427_object71811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.976129+0000", 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 483.49330867200001, 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.052223253, 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202480 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.155 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976129+0000", 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976129+0000", 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976130+0000", 2024-09-13T13:21:05.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3799999999999999e-06 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976135+0000", 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2809999999999999e-06 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976147+0000", 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1182000000000001e-05 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976154+0000", 2024-09-13T13:21:05.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4440000000000002e-06 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976219+0000", 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4414999999999999e-05 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976227+0000", 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012784+0000", 2024-09-13T13:21:05.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.036556815 2024-09-13T13:21:05.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012892+0000", 2024-09-13T13:21:05.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010768999999999999 2024-09-13T13:21:05.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:21:05.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.013010+0000", 2024-09-13T13:21:05.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118244 2024-09-13T13:21:05.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.018728+0000", 2024-09-13T13:21:05.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057179079999999998 2024-09-13T13:21:05.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.018768+0000", 2024-09-13T13:21:05.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.985e-05 2024-09-13T13:21:05.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028119+0000", 2024-09-13T13:21:05.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0093518460000000005 2024-09-13T13:21:05.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028244+0000", 2024-09-13T13:21:05.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012501799999999999 2024-09-13T13:21:05.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028352+0000", 2024-09-13T13:21:05.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107557 2024-09-13T13:21:05.163 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.163 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.163 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.164 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202488 9.c 9:3231b30e:::benchmark_data_smithi052_128427_object71819:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:21:05.164 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.980401+0000", 2024-09-13T13:21:05.164 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 483.489036199, 2024-09-13T13:21:05.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0481403709999999, 2024-09-13T13:21:05.165 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.165 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.165 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.165 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.165 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.165 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.166 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202488 2024-09-13T13:21:05.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.166 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980401+0000", 2024-09-13T13:21:05.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980401+0000", 2024-09-13T13:21:05.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980403+0000", 2024-09-13T13:21:05.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.339e-06 2024-09-13T13:21:05.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980408+0000", 2024-09-13T13:21:05.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1909999999999999e-06 2024-09-13T13:21:05.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980421+0000", 2024-09-13T13:21:05.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3267e-05 2024-09-13T13:21:05.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980428+0000", 2024-09-13T13:21:05.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0310000000000001e-06 2024-09-13T13:21:05.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980485+0000", 2024-09-13T13:21:05.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6527999999999998e-05 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980493+0000", 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2460000000000003e-06 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.015559+0000", 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0350661029999999 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.015857+0000", 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000297726 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.016037+0000", 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018051499999999999 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.021797+0000", 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057596000000000001 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.171 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.021836+0000", 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9180000000000001e-05 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028359+0000", 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0065231589999999997 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028449+0000", 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9874e-05 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028542+0000", 2024-09-13T13:21:05.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2612e-05 2024-09-13T13:21:05.173 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.173 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.173 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.173 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202524 9.c 9:322e12cc:::benchmark_data_smithi052_128427_object71855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:21:05.176 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:58.008117+0000", 2024-09-13T13:21:05.176 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 483.46132092599998, 2024-09-13T13:21:05.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0206121399999999, 2024-09-13T13:21:05.176 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.176 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202524 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008117+0000", 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008117+0000", 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008118+0000", 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3200000000000001e-06 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008123+0000", 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8189999999999999e-06 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008132+0000", 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6979999999999993e-06 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008141+0000", 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7620000000000001e-06 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008199+0000", 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7942999999999997e-05 2024-09-13T13:21:05.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008208+0000", 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1069999999999994e-06 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.018947+0000", 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0107389179999999 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.019046+0000", 2024-09-13T13:21:05.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8896999999999999e-05 2024-09-13T13:21:05.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:21:05.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.019147+0000", 2024-09-13T13:21:05.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101001 2024-09-13T13:21:05.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.021878+0000", 2024-09-13T13:21:05.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002730985 2024-09-13T13:21:05.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.026889+0000", 2024-09-13T13:21:05.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050107529999999997 2024-09-13T13:21:05.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028548+0000", 2024-09-13T13:21:05.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016596950000000001 2024-09-13T13:21:05.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028637+0000", 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8687e-05 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028729+0000", 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1555e-05 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:178620 10.f e380/377 10:f7cb9e30:::benchmark_data_smithi052_153703_object51240:head v 380'11131, mlcod=380'11128)", 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:16:22.237173+0000", 2024-09-13T13:21:05.183 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 279.23226496500001, 2024-09-13T13:21:05.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34096315700000002, 2024-09-13T13:21:05.184 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.184 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.184 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.184 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.184 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237173+0000", 2024-09-13T13:21:05.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.184 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237173+0000", 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237174+0000", 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.373e-06 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237182+0000", 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9310000000000008e-06 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.185 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237233+0000", 2024-09-13T13:21:05.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0778000000000001e-05 2024-09-13T13:21:05.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237240+0000", 2024-09-13T13:21:05.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8120000000000008e-06 2024-09-13T13:21:05.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237756+0000", 2024-09-13T13:21:05.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000516009 2024-09-13T13:21:05.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237780+0000", 2024-09-13T13:21:05.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4009e-05 2024-09-13T13:21:05.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.578044+0000", 2024-09-13T13:21:05.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34026351900000001 2024-09-13T13:21:05.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.578136+0000", 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1725999999999996e-05 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94540 11.8 11:13a8cb6c:::benchmark_data_smithi052_182575_object94539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.195993+0000", 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 117.273444942, 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.020347348, 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:05.188 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.189 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:05.189 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:05.189 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94540 2024-09-13T13:21:05.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.189 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.195993+0000", 2024-09-13T13:21:05.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.195993+0000", 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.195997+0000", 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2390000000000004e-06 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196038+0000", 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1275e-05 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.203061+0000", 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.007023151 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.203080+0000", 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8929000000000001e-05 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.470011+0000", 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.26693045999999998 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.192 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.470019+0000", 2024-09-13T13:21:05.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8470000000000002e-06 2024-09-13T13:21:05.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.192 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.192 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.198605+0000", 2024-09-13T13:21:05.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.72858517899999997 2024-09-13T13:21:05.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.192 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for degraded object", 2024-09-13T13:21:05.192 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.202890+0000", 2024-09-13T13:21:05.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0042850730000000004 2024-09-13T13:21:05.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.216266+0000", 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.013376702000000001 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.216340+0000", 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3492999999999996e-05 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95597 11.18 11:198e97c8:::benchmark_data_smithi052_182575_object95596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:07.184969+0000", 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 114.284468593, 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1262475439999999, 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95597 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184969+0000", 2024-09-13T13:21:05.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184969+0000", 2024-09-13T13:21:05.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184972+0000", 2024-09-13T13:21:05.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7769999999999998e-06 2024-09-13T13:21:05.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185010+0000", 2024-09-13T13:21:05.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8263999999999998e-05 2024-09-13T13:21:05.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185023+0000", 2024-09-13T13:21:05.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.271e-05 2024-09-13T13:21:05.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185032+0000", 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7499999999999998e-06 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.190626+0000", 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055940989999999999 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.190637+0000", 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0614999999999999e-05 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.302635+0000", 2024-09-13T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.111997723 2024-09-13T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.302817+0000", 2024-09-13T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018175 2024-09-13T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4,5,6", 2024-09-13T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.303156+0000", 2024-09-13T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033972700000000001 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.305064+0000", 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019073569999999999 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.306588+0000", 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0015246109999999999 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.306637+0000", 2024-09-13T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8702999999999998e-05 2024-09-13T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.310746+0000", 2024-09-13T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041093400000000004 2024-09-13T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.311036+0000", 2024-09-13T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00028986300000000001 2024-09-13T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.311104+0000", 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7554999999999999e-05 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.311216+0000", 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0001127 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95626 11.18 11:1f52487a:::benchmark_data_smithi052_182575_object95625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:07.185050+0000", 2024-09-13T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 114.284387553, 2024-09-13T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1376889400000001, 2024-09-13T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95626 2024-09-13T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185050+0000", 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185050+0000", 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185051+0000", 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.023e-06 2024-09-13T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.204 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:05.204 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185086+0000", 2024-09-13T13:21:05.204 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4774000000000003e-05 2024-09-13T13:21:05.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.204 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:05.204 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185096+0000", 2024-09-13T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0681e-05 2024-09-13T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185105+0000", 2024-09-13T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1040000000000007e-06 2024-09-13T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.190690+0000", 2024-09-13T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055853229999999997 2024-09-13T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.190701+0000", 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1103e-05 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.306669+0000", 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.11596831 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.306827+0000", 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015811199999999999 2024-09-13T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4,5,6", 2024-09-13T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.307460+0000", 2024-09-13T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00063293599999999996 2024-09-13T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.313317+0000", 2024-09-13T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058565539999999999 2024-09-13T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.313333+0000", 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6140000000000001e-05 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.313435+0000", 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102132 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.313634+0000", 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019934400000000001 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.322535+0000", 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0089002040000000001 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.322605+0000", 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0468000000000004e-05 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.322739+0000", 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013373200000000001 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.210 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.210 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:05.210 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:05.210 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:21:05.210 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:05.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:21:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:21:05.251 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:05.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:05.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:05.354 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:05.403 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:05.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:05.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:05.505 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:05.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:05.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:05.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:05 smithi167 ceph-mon[33178]: pgmap v1015: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.26k op/s; 11550/113835 objects misplaced (10.146%) 2024-09-13T13:21:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:05 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:21:05.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:05 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:21:05.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:05.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:05.640 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:05 smithi052 ceph-mon[29471]: pgmap v1015: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.26k op/s; 11550/113835 objects misplaced (10.146%) 2024-09-13T13:21:05.640 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:05 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:21:05.640 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:05 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:21:05.640 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:05.641 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:05 smithi052 ceph-mon[36682]: pgmap v1015: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.26k op/s; 11550/113835 objects misplaced (10.146%) 2024-09-13T13:21:05.641 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:05 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:21:05.641 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:05 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:21:05.641 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:05.661 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:05.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:05.762 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:05.804 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:05.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:05.905 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:05.921 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:05.930 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:05.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:05.949 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:21:06.032 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:06.064 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:06.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:06.166 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:06.212 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:21:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:21:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:21:06.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:06.218 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:06.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:06.307 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:21:06.319 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:06.341 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:21:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:06.360 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:06.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:06.412 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:21:06.461 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:06.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:06 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3026887253' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:06.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:06 smithi052 ceph-mon[29471]: from='client.36881 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:06.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:06 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3026887253' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:06.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:06 smithi052 ceph-mon[36682]: from='client.36881 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:06.504 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:06.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:06.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:06 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3026887253' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:06.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:06 smithi167 ceph-mon[33178]: from='client.36881 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:06.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:06.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:06.679 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:06.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:06.780 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:06.810 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:06.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:06.912 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:06.952 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:06.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:07.054 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:07.086 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:07.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:07.188 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:07.211 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:07.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:07.255 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:21:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:21:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:202449 11.e 11:7bdd4775:::benchmark_data_smithi052_182575_object85440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-13T13:21:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:21:07.246771+0000", 2024-09-13T13:21:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0049378970000000001, 2024-09-13T13:21:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004965199, 2024-09-13T13:21:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:21:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:21:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:21:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 202449 2024-09-13T13:21:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.246771+0000", 2024-09-13T13:21:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.246771+0000", 2024-09-13T13:21:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.246773+0000", 2024-09-13T13:21:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1399999999999998e-06 2024-09-13T13:21:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.246779+0000", 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8610000000000003e-06 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.246811+0000", 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2274000000000003e-05 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.246819+0000", 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9179999999999994e-06 2024-09-13T13:21:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.246877+0000", 2024-09-13T13:21:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.787e-05 2024-09-13T13:21:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.247008+0000", 2024-09-13T13:21:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013127399999999999 2024-09-13T13:21:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,1", 2024-09-13T13:21:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.247134+0000", 2024-09-13T13:21:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012540699999999999 2024-09-13T13:21:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:21:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.249793+0000", 2024-09-13T13:21:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026598619999999998 2024-09-13T13:21:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:202456 11.c e499/423 11:3e643222:::benchmark_data_smithi052_182575_object85447:head v 499'12729, mlcod=499'12726)", 2024-09-13T13:21:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:21:07.250618+0000", 2024-09-13T13:21:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.001090646, 2024-09-13T13:21:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001202467, 2024-09-13T13:21:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:21:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250618+0000", 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250618+0000", 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250620+0000", 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4479999999999999e-06 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250630+0000", 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0026e-05 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250645+0000", 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5511000000000001e-05 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250652+0000", 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2869999999999999e-06 2024-09-13T13:21:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250720+0000", 2024-09-13T13:21:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7156000000000006e-05 2024-09-13T13:21:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250740+0000", 2024-09-13T13:21:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.067e-05 2024-09-13T13:21:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:202455 11.d e499/423 11:be60706c:::benchmark_data_smithi052_182575_object85446:head v 499'12382, mlcod=499'12379)", 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:21:07.250852+0000", 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00085725500000000002, 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010236780000000001, 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250852+0000", 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250852+0000", 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250853+0000", 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2279999999999999e-06 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250860+0000", 2024-09-13T13:21:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8800000000000002e-06 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250869+0000", 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0960000000000007e-06 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250875+0000", 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6989999999999997e-06 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250941+0000", 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5581999999999997e-05 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250960+0000", 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9372e-05 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:202458 11.15 e499/456 11:aba7d680:::benchmark_data_smithi052_182575_object85449:head v 499'9404, mlcod=499'9402)", 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:21:07.250975+0000", 2024-09-13T13:21:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00073331000000000004, 2024-09-13T13:21:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00094740199999999997, 2024-09-13T13:21:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:21:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250975+0000", 2024-09-13T13:21:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250975+0000", 2024-09-13T13:21:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250977+0000", 2024-09-13T13:21:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4330000000000001e-06 2024-09-13T13:21:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250983+0000", 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.964e-06 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250992+0000", 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2410000000000001e-06 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.250999+0000", 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4949999999999996e-06 2024-09-13T13:21:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.251287+0000", 2024-09-13T13:21:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028877599999999997 2024-09-13T13:21:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.251306+0000", 2024-09-13T13:21:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8847000000000001e-05 2024-09-13T13:21:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:202461 11.18 e499/463 11:18470624:::benchmark_data_smithi052_182575_object85452:head v 499'9427, mlcod=499'9425)", 2024-09-13T13:21:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:21:07.251724+0000", 2024-09-13T13:21:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4294967295.9999852, 2024-09-13T13:21:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024684199999999999, 2024-09-13T13:21:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:21:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.251724+0000", 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.251724+0000", 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.251725+0000", 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4099999999999997e-07 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.251732+0000", 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6680000000000001e-06 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.251740+0000", 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7659999999999998e-06 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.251746+0000", 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8610000000000003e-06 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:07.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.251818+0000", 2024-09-13T13:21:07.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2484000000000006e-05 2024-09-13T13:21:07.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:07.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:07.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:07.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:07.251836+0000", 2024-09-13T13:21:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7376000000000001e-05 2024-09-13T13:21:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-09-13T13:21:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 5 2024-09-13T13:21:07.278 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:21:07.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:07.314 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T13:21:07.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:07.347 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:07.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:07.448 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:07.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:07.478 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:07.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:07.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:07 smithi167 ceph-mon[33178]: pgmap v1016: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 10392/108621 objects misplaced (9.567%) 2024-09-13T13:21:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:07 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:07 smithi167 ceph-mon[33178]: from='client.36881 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:21:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:07 smithi167 ceph-mon[33178]: osdmap e499: 8 total, 8 up, 7 in 2024-09-13T13:21:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:07 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1446314872' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:21:07.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:07 smithi167 ceph-mon[33178]: from='client.26779 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:21:07.579 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:07.616 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:07.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:07.718 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:07.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:07 smithi052 ceph-mon[36682]: pgmap v1016: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 10392/108621 objects misplaced (9.567%) 2024-09-13T13:21:07.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:07 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:07.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:07 smithi052 ceph-mon[36682]: from='client.36881 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:21:07.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:07 smithi052 ceph-mon[36682]: osdmap e499: 8 total, 8 up, 7 in 2024-09-13T13:21:07.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:07 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1446314872' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:21:07.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:07 smithi052 ceph-mon[36682]: from='client.26779 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:21:07.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:07 smithi052 ceph-mon[29471]: pgmap v1016: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 10392/108621 objects misplaced (9.567%) 2024-09-13T13:21:07.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:07 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:07.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:07 smithi052 ceph-mon[29471]: from='client.36881 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:21:07.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:07 smithi052 ceph-mon[29471]: osdmap e499: 8 total, 8 up, 7 in 2024-09-13T13:21:07.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:07 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1446314872' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:21:07.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:07 smithi052 ceph-mon[29471]: from='client.26779 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:21:07.751 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:07.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:07.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:07.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:07.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:07.984 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:08.015 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:08.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:08.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:08.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:08.158 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:08.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:08.259 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:08.286 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T13:21:08.297 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:08.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:08.399 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:08.438 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:08.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:08.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:08.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:08 smithi167 ceph-mon[33178]: from='client.26779 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:21:08.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:08 smithi167 ceph-mon[33178]: osdmap e500: 8 total, 8 up, 7 in 2024-09-13T13:21:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:08.579 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:08.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:08.634 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:08 smithi052 ceph-mon[36682]: from='client.26779 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:21:08.634 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:08 smithi052 ceph-mon[36682]: osdmap e500: 8 total, 8 up, 7 in 2024-09-13T13:21:08.634 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:08 smithi052 ceph-mon[29471]: from='client.26779 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:21:08.634 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:08 smithi052 ceph-mon[29471]: osdmap e500: 8 total, 8 up, 7 in 2024-09-13T13:21:08.650 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:21:08.679 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:08.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:08.710 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:08.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:08.811 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:08.853 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:08.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:08.955 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:08.983 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:08.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:08.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:09.085 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:09.123 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:09.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:09.225 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:09.265 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:09.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:09.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:09.364 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:09 smithi167 ceph-mon[33178]: pgmap v1019: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.00k op/s; 10392/105750 objects misplaced (9.827%) 2024-09-13T13:21:09.367 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:09.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:09.414 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:09.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:09.420 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:21:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:21:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:21:09.420 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:21:09.420 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:21:09.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:09.548 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:09.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:09.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:09.650 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:09.690 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:09.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:09.691 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:09 smithi052 ceph-mon[36682]: pgmap v1019: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.00k op/s; 10392/105750 objects misplaced (9.827%) 2024-09-13T13:21:09.693 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:09 smithi052 ceph-mon[29471]: pgmap v1019: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.00k op/s; 10392/105750 objects misplaced (9.827%) 2024-09-13T13:21:09.792 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:09.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:09.838 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:09.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:09.939 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:09.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:09.985 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:09.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:10.087 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:10.108 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:10.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:10.211 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:10.247 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:10.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:10.349 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:10.376 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:10.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:10.478 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:10.519 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:10.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:10.620 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:10.654 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:10.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:10.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:10.756 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:10.785 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:10.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:10.887 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:10.915 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:10.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:11.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:11.050 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:11.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:11.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:11.152 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:11.183 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:11.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:11.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:11.285 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:11.327 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:11.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:11.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:11.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:11.451 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:11.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:11.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:11.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:11.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:11 smithi167 ceph-mon[33178]: pgmap v1020: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 1.47k op/s; 8685/90966 objects misplaced (9.548%) 2024-09-13T13:21:11.581 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:11.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:11.683 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:11.712 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:11.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:11.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:11.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:11 smithi052 ceph-mon[29471]: pgmap v1020: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 1.47k op/s; 8685/90966 objects misplaced (9.548%) 2024-09-13T13:21:11.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:11 smithi052 ceph-mon[36682]: pgmap v1020: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 1.47k op/s; 8685/90966 objects misplaced (9.548%) 2024-09-13T13:21:11.748 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:21:11.814 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:11.836 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:11.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:11.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:11.938 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:11.993 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:11.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:11.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:12.095 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:12.119 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:12.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:12.220 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:12.250 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:12.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:12.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:12.352 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:12.387 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:12.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:12.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:12.489 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:12.529 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:12.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:12.630 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:12.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:12.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:12.770 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:12.811 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:12.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:12.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:12.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:12.939 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:12.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:13.041 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:13.085 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:13.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:13.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T13:21:13.169 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T13:21:13.169 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:21:13.188 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:13.196 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:13.224 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:13.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:13.325 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:13.382 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:13.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:13.392 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:13 smithi052 ceph-mon[36682]: pgmap v1021: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 6.9 KiB/s rd, 0 B/s wr, 1.07k op/s; 8685/88329 objects misplaced (9.833%) 2024-09-13T13:21:13.393 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:13 smithi052 ceph-mon[29471]: pgmap v1021: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 6.9 KiB/s rd, 0 B/s wr, 1.07k op/s; 8685/88329 objects misplaced (9.833%) 2024-09-13T13:21:13.458 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:21:13.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:13.516 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:13.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:13.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:13.591 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:13 smithi167 ceph-mon[33178]: pgmap v1021: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 6.9 KiB/s rd, 0 B/s wr, 1.07k op/s; 8685/88329 objects misplaced (9.833%) 2024-09-13T13:21:13.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:13.642 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:13.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:13.745 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:13.794 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:13.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:13.877 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:13.895 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:13.922 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:13.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:13.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:13.924 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:21:13.924 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:21:13.958 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:21:13.958 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:21:13.969 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:13.989 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:21:13.989 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:21:14.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:14.025 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:21:14.026 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:21:14.051 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:14.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:14.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:14.063 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:21:14.063 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:21:14.097 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:21:14.098 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:21:14.125 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:21:14.125 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:21:14.137 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:21:14.137 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:21:14.153 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:14.171 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:21:14.171 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:21:14.185 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:14.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:14.287 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:14.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:14.316 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:14.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:14.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:14.447 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:14.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:14.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:14.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:14.564 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:14 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/649869919' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:14.564 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:14 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/649869919' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:14.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:14 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/649869919' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:14.579 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:14.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:14.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:14.681 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:14.720 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:14.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:14.822 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:14.835 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T13:21:14.852 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:14.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:14.954 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:15.002 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:15.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:15.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:15.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:21:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:21:15.104 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:15.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:15.144 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:15.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:15.216 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:21:15.245 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:15.283 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:15.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:15.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:15.385 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:15.412 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:15.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:15 smithi052 ceph-mon[36682]: pgmap v1022: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 38 KiB/s rd, 0 B/s wr, 1.29k op/s; 8151/79134 objects misplaced (10.300%) 2024-09-13T13:21:15.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:15.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:15 smithi052 ceph-mon[29471]: pgmap v1022: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 38 KiB/s rd, 0 B/s wr, 1.29k op/s; 8151/79134 objects misplaced (10.300%) 2024-09-13T13:21:15.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:15.514 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:15.542 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:15.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:15.603 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:15 smithi167 ceph-mon[33178]: pgmap v1022: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 38 KiB/s rd, 0 B/s wr, 1.29k op/s; 8151/79134 objects misplaced (10.300%) 2024-09-13T13:21:15.603 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:15.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:15.672 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:15.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:15.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:15.774 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:15.800 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:15.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:15.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:15.872 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:21:15.873 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:21:15.902 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:15.938 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:15.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:16.040 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:16.076 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:16.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:16.179 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:16.218 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:16.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:16.256 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:21:16.261 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:21:16.319 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:16.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:16.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:16.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:16.354 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:21:16.398 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:16 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3530857766' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:21:16.398 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:21:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:21:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:21:16.399 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:16 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3530857766' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:21:16.446 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:16.485 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:16.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:16.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:16.587 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:16.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:16.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:16.717 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:16.746 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:16.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:16.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:16.785 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:16 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3530857766' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:21:16.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T13:21:16.848 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:16.891 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:16.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:16.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:16.993 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:16.993 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:21:17.024 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:17.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:17.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:17.126 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:17.153 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:17.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:17.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:17.211 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:21:17.255 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:17.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:17.299 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:17.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:17.401 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:17.426 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:17.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:17.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:17.479 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":1023,"stamp":"2024-09-13T13:21:16.064879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1613503027,"num_objects":24812,"num_object_clones":0,"num_object_copies":74436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7065,"num_objects_unfound":0,"num_objects_dirty":24812,"num_whiteouts":0,"num_read":2047,"num_read_kb":1799,"num_write":143761,"num_write_kb":3776942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28016,"num_bytes_recovered":1815094946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121236,"ondisk_log_size":121236,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":513,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":10450764,"kb_used_data":4849180,"kb_used_omap":13,"kb_used_meta":5601330,"kb_avail":645765300,"statfs":{"total":671965249536,"available":661263667200,"internally_reserved":0,"allocated":4965560320,"data_stored":4962900878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14001,"internal_metadata":5735762255},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-746782720,"num_objects":-11395,"num_object_clones":0,"num_object_copies":-34185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3327,"num_objects_unfound":0,"num_objects_dirty":-11395,"num_whiteouts":0,"num_read":412,"num_read_kb":412,"num_write":11663,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.761756"},"pg_stats":[{"pgid":"11.17","version":"500'9285","reported_seq":15957,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.451195+0000","last_change":"2024-09-13T13:20:55.540544+0000","last_active":"2024-09-13T13:21:11.451195+0000","last_peered":"2024-09-13T13:21:11.451195+0000","last_clean":"2024-09-13T13:21:11.451195+0000","last_became_active":"2024-09-13T13:19:53.269607+0000","last_became_peered":"2024-09-13T13:19:53.269607+0000","last_unstale":"2024-09-13T13:21:11.451195+0000","last_undegraded":"2024-09-13T13:21:11.451195+0000","last_fullsized":"2024-09-13T13:21:11.451195+0000","mapping_epoch":470,"log_start":"443'4214","ondisk_log_start":"443'4214","created":450,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":5071,"log_dups_size":0,"ondisk_log_size":5071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:21:25.516783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53706752,"num_objects":820,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6390,"num_write_kb":230688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3223,"num_bytes_recovered":211222528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":953,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.314327+0000","last_change":"2024-09-13T13:18:32.440533+0000","last_active":"2024-09-13T13:21:07.314327+0000","last_peered":"2024-09-13T13:21:07.314327+0000","last_clean":"2024-09-13T13:21:07.314327+0000","last_became_active":"2024-09-13T13:17:25.335900+0000","last_became_peered":"2024-09-13T13:17:25.335900+0000","last_unstale":"2024-09-13T13:21:07.314327+0000","last_undegraded":"2024-09-13T13:21:07.314327+0000","last_fullsized":"2024-09-13T13:21:07.314327+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:32.440448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:32.440448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"500'21","reported_seq":438,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:10.997142+0000","last_change":"2024-09-13T13:18:08.548626+0000","last_active":"2024-09-13T13:21:10.997142+0000","last_peered":"2024-09-13T13:21:10.997142+0000","last_clean":"2024-09-13T13:21:10.997142+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:21:10.997142+0000","last_undegraded":"2024-09-13T13:21:10.997142+0000","last_fullsized":"2024-09-13T13:21:10.997142+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:18:08.548556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:08.548556+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":955,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.328787+0000","last_change":"2024-09-13T13:19:43.119757+0000","last_active":"2024-09-13T13:21:07.328787+0000","last_peered":"2024-09-13T13:21:07.328787+0000","last_clean":"2024-09-13T13:21:07.328787+0000","last_became_active":"2024-09-13T13:19:43.119056+0000","last_became_peered":"2024-09-13T13:19:43.119056+0000","last_unstale":"2024-09-13T13:21:07.328787+0000","last_undegraded":"2024-09-13T13:21:07.328787+0000","last_fullsized":"2024-09-13T13:21:07.328787+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:12.060429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:12.060429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":427,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.318968+0000","last_change":"2024-09-13T13:18:49.194573+0000","last_active":"2024-09-13T13:21:07.318968+0000","last_peered":"2024-09-13T13:21:07.318968+0000","last_clean":"2024-09-13T13:21:07.318968+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:21:07.318968+0000","last_undegraded":"2024-09-13T13:21:07.318968+0000","last_fullsized":"2024-09-13T13:21:07.318968+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:49.194510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:49.194510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"500'9445","reported_seq":18297,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:13.725508+0000","last_change":"2024-09-13T13:20:47.197380+0000","last_active":"2024-09-13T13:21:13.725508+0000","last_peered":"2024-09-13T13:21:13.725508+0000","last_clean":"2024-09-13T13:21:13.725508+0000","last_became_active":"2024-09-13T13:20:15.116368+0000","last_became_peered":"2024-09-13T13:20:15.116368+0000","last_unstale":"2024-09-13T13:21:13.725508+0000","last_undegraded":"2024-09-13T13:21:13.725508+0000","last_fullsized":"2024-09-13T13:21:13.725508+0000","mapping_epoch":478,"log_start":"443'4276","ondisk_log_start":"443'4276","created":450,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"490'8518","last_scrub_stamp":"2024-09-13T13:20:47.197308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:20:47.197308+0000","objects_scrubbed":1663,"log_size":5169,"log_dups_size":0,"ondisk_log_size":5169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:13.351498+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47906816,"num_objects":731,"num_object_clones":0,"num_object_copies":2193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":932,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":937,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.318923+0000","last_change":"2024-09-13T13:19:43.132324+0000","last_active":"2024-09-13T13:21:07.318923+0000","last_peered":"2024-09-13T13:21:07.318923+0000","last_clean":"2024-09-13T13:21:07.318923+0000","last_became_active":"2024-09-13T13:19:43.131732+0000","last_became_peered":"2024-09-13T13:19:43.131732+0000","last_unstale":"2024-09-13T13:21:07.318923+0000","last_undegraded":"2024-09-13T13:21:07.318923+0000","last_fullsized":"2024-09-13T13:21:07.318923+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:17.137902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":982,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.321564+0000","last_change":"2024-09-13T13:20:45.377028+0000","last_active":"2024-09-13T13:21:07.321564+0000","last_peered":"2024-09-13T13:21:07.321564+0000","last_clean":"2024-09-13T13:21:07.321564+0000","last_became_active":"2024-09-13T13:17:25.243371+0000","last_became_peered":"2024-09-13T13:17:25.243371+0000","last_unstale":"2024-09-13T13:21:07.321564+0000","last_undegraded":"2024-09-13T13:21:07.321564+0000","last_fullsized":"2024-09-13T13:21:07.321564+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:45.376945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:45.376945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:58.142506+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"500'39","reported_seq":1074,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.838720+0000","last_change":"2024-09-13T13:20:52.246795+0000","last_active":"2024-09-13T13:21:12.838720+0000","last_peered":"2024-09-13T13:21:12.838720+0000","last_clean":"2024-09-13T13:21:12.838720+0000","last_became_active":"2024-09-13T13:17:25.245131+0000","last_became_peered":"2024-09-13T13:17:25.245131+0000","last_unstale":"2024-09-13T13:21:12.838720+0000","last_undegraded":"2024-09-13T13:21:12.838720+0000","last_fullsized":"2024-09-13T13:21:12.838720+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:20:52.246624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:52.246624+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:02.856958+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":411,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319829+0000","last_change":"2024-09-13T13:20:31.015443+0000","last_active":"2024-09-13T13:21:07.319829+0000","last_peered":"2024-09-13T13:21:07.319829+0000","last_clean":"2024-09-13T13:21:07.319829+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:21:07.319829+0000","last_undegraded":"2024-09-13T13:21:07.319829+0000","last_fullsized":"2024-09-13T13:21:07.319829+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:31.015382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:31.015382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:35.021147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"500'9634","reported_seq":9838,"reported_epoch":500,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T13:21:13.736078+0000","last_change":"2024-09-13T13:20:44.322698+0000","last_active":"2024-09-13T13:21:13.736078+0000","last_peered":"2024-09-13T13:21:13.736078+0000","last_clean":"2024-09-13T13:19:06.181780+0000","last_became_active":"2024-09-13T13:19:14.296332+0000","last_became_peered":"2024-09-13T13:19:14.296332+0000","last_unstale":"2024-09-13T13:21:13.736078+0000","last_undegraded":"2024-09-13T13:21:13.736078+0000","last_fullsized":"2024-09-13T13:21:13.736078+0000","mapping_epoch":456,"log_start":"444'4655","ondisk_log_start":"444'4655","created":450,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4979,"log_dups_size":0,"ondisk_log_size":4979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56164352,"num_objects":857,"num_object_clones":0,"num_object_copies":2571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2571,"num_objects_unfound":0,"num_objects_dirty":857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6653,"num_write_kb":240320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[5,6,7],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":857}],"blocked_by":[],"up_primary":1,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":1002,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.325339+0000","last_change":"2024-09-13T13:20:43.694870+0000","last_active":"2024-09-13T13:21:07.325339+0000","last_peered":"2024-09-13T13:21:07.325339+0000","last_clean":"2024-09-13T13:21:07.325339+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:21:07.325339+0000","last_undegraded":"2024-09-13T13:21:07.325339+0000","last_fullsized":"2024-09-13T13:21:07.325339+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:20:43.694821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:43.694821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:12.759248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"500'21","reported_seq":430,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.944860+0000","last_change":"2024-09-13T13:17:06.406253+0000","last_active":"2024-09-13T13:21:11.944860+0000","last_peered":"2024-09-13T13:21:11.944860+0000","last_clean":"2024-09-13T13:21:11.944860+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:21:11.944860+0000","last_undegraded":"2024-09-13T13:21:11.944860+0000","last_fullsized":"2024-09-13T13:21:11.944860+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:06.406179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:06.406179+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:55.937461+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":954,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.323854+0000","last_change":"2024-09-13T13:19:43.103191+0000","last_active":"2024-09-13T13:21:07.323854+0000","last_peered":"2024-09-13T13:21:07.323854+0000","last_clean":"2024-09-13T13:21:07.323854+0000","last_became_active":"2024-09-13T13:19:43.102694+0000","last_became_peered":"2024-09-13T13:19:43.102694+0000","last_unstale":"2024-09-13T13:21:07.323854+0000","last_undegraded":"2024-09-13T13:21:07.323854+0000","last_fullsized":"2024-09-13T13:21:07.323854+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:15.686363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:15.686363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":949,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.314162+0000","last_change":"2024-09-13T13:19:43.116648+0000","last_active":"2024-09-13T13:21:07.314162+0000","last_peered":"2024-09-13T13:21:07.314162+0000","last_clean":"2024-09-13T13:21:07.314162+0000","last_became_active":"2024-09-13T13:19:43.116357+0000","last_became_peered":"2024-09-13T13:19:43.116357+0000","last_unstale":"2024-09-13T13:21:07.314162+0000","last_undegraded":"2024-09-13T13:21:07.314162+0000","last_fullsized":"2024-09-13T13:21:07.314162+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:32.559884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:32.559884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:02.033055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"500'9278","reported_seq":17539,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.384159+0000","last_change":"2024-09-13T13:20:31.432554+0000","last_active":"2024-09-13T13:21:11.384159+0000","last_peered":"2024-09-13T13:21:11.384159+0000","last_clean":"2024-09-13T13:21:11.384159+0000","last_became_active":"2024-09-13T13:20:30.358134+0000","last_became_peered":"2024-09-13T13:20:30.358134+0000","last_unstale":"2024-09-13T13:21:11.384159+0000","last_undegraded":"2024-09-13T13:21:11.384159+0000","last_fullsized":"2024-09-13T13:21:11.384159+0000","mapping_epoch":483,"log_start":"443'4174","ondisk_log_start":"443'4174","created":450,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"485'7948","last_scrub_stamp":"2024-09-13T13:20:31.432498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:20:31.432498+0000","objects_scrubbed":2164,"log_size":5104,"log_dups_size":0,"ondisk_log_size":5104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:51.136811+0000","scrub_duration":750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54198272,"num_objects":827,"num_object_clones":0,"num_object_copies":2481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1337,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":977,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.324233+0000","last_change":"2024-09-13T13:19:31.753911+0000","last_active":"2024-09-13T13:21:07.324233+0000","last_peered":"2024-09-13T13:21:07.324233+0000","last_clean":"2024-09-13T13:21:07.324233+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:21:07.324233+0000","last_undegraded":"2024-09-13T13:21:07.324233+0000","last_fullsized":"2024-09-13T13:21:07.324233+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:31.753864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:31.753864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"500'9","reported_seq":428,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.619355+0000","last_change":"2024-09-13T13:20:47.925194+0000","last_active":"2024-09-13T13:21:11.619355+0000","last_peered":"2024-09-13T13:21:11.619355+0000","last_clean":"2024-09-13T13:21:11.619355+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:21:11.619355+0000","last_undegraded":"2024-09-13T13:21:11.619355+0000","last_fullsized":"2024-09-13T13:21:11.619355+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:20:47.925142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:47.925142+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:48.646990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":629,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.318567+0000","last_change":"2024-09-13T13:20:44.928714+0000","last_active":"2024-09-13T13:21:07.318567+0000","last_peered":"2024-09-13T13:21:07.318567+0000","last_clean":"2024-09-13T13:21:07.318567+0000","last_became_active":"2024-09-13T13:19:43.130534+0000","last_became_peered":"2024-09-13T13:19:43.130534+0000","last_unstale":"2024-09-13T13:21:07.318567+0000","last_undegraded":"2024-09-13T13:21:07.318567+0000","last_fullsized":"2024-09-13T13:21:07.318567+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:44.928667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:44.928667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:56.039898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":919,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.323822+0000","last_change":"2024-09-13T13:18:34.846273+0000","last_active":"2024-09-13T13:21:07.323822+0000","last_peered":"2024-09-13T13:21:07.323822+0000","last_clean":"2024-09-13T13:21:07.323822+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:21:07.323822+0000","last_undegraded":"2024-09-13T13:21:07.323822+0000","last_fullsized":"2024-09-13T13:21:07.323822+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:34.846222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:34.846222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"500'9762","reported_seq":9957,"reported_epoch":500,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T13:21:14.812622+0000","last_change":"2024-09-13T13:21:08.387402+0000","last_active":"2024-09-13T13:21:14.812622+0000","last_peered":"2024-09-13T13:21:14.812622+0000","last_clean":"2024-09-13T13:19:06.161926+0000","last_became_active":"2024-09-13T13:19:08.330177+0000","last_became_peered":"2024-09-13T13:19:08.330177+0000","last_unstale":"2024-09-13T13:21:14.812622+0000","last_undegraded":"2024-09-13T13:21:14.812622+0000","last_fullsized":"2024-09-13T13:21:14.812622+0000","mapping_epoch":453,"log_start":"454'6093","ondisk_log_start":"454'6093","created":450,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49217536,"num_objects":751,"num_object_clones":0,"num_object_copies":2253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2253,"num_objects_unfound":0,"num_objects_dirty":751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6753,"num_write_kb":240128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[1,6,7],"avail_no_missing":["1","6","7"],"object_location_counts":[{"shards":"1,6,7","objects":751}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":416,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.320620+0000","last_change":"2024-09-13T13:17:07.444148+0000","last_active":"2024-09-13T13:21:07.320620+0000","last_peered":"2024-09-13T13:21:07.320620+0000","last_clean":"2024-09-13T13:21:07.320620+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:21:07.320620+0000","last_undegraded":"2024-09-13T13:21:07.320620+0000","last_fullsized":"2024-09-13T13:21:07.320620+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:07.443951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:07.443951+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:02.940965+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":975,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.317923+0000","last_change":"2024-09-13T13:19:43.118595+0000","last_active":"2024-09-13T13:21:07.317923+0000","last_peered":"2024-09-13T13:21:07.317923+0000","last_clean":"2024-09-13T13:21:07.317923+0000","last_became_active":"2024-09-13T13:19:43.118289+0000","last_became_peered":"2024-09-13T13:19:43.118289+0000","last_unstale":"2024-09-13T13:21:07.317923+0000","last_undegraded":"2024-09-13T13:21:07.317923+0000","last_fullsized":"2024-09-13T13:21:07.317923+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:45.470790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.470790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"500'39","reported_seq":237,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.277337+0000","last_change":"2024-09-13T13:18:37.714594+0000","last_active":"2024-09-13T13:21:12.277337+0000","last_peered":"2024-09-13T13:21:12.277337+0000","last_clean":"2024-09-13T13:21:12.277337+0000","last_became_active":"2024-09-13T13:17:25.245658+0000","last_became_peered":"2024-09-13T13:17:25.245658+0000","last_unstale":"2024-09-13T13:21:12.277337+0000","last_undegraded":"2024-09-13T13:21:12.277337+0000","last_fullsized":"2024-09-13T13:21:12.277337+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:18:37.714503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:37.714503+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:57.581192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":970,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.329880+0000","last_change":"2024-09-13T13:19:20.177040+0000","last_active":"2024-09-13T13:21:07.329880+0000","last_peered":"2024-09-13T13:21:07.329880+0000","last_clean":"2024-09-13T13:21:07.329880+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:21:07.329880+0000","last_undegraded":"2024-09-13T13:21:07.329880+0000","last_fullsized":"2024-09-13T13:21:07.329880+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:20.176963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:20.176963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"500'9551","reported_seq":20375,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:13.317467+0000","last_change":"2024-09-13T13:20:44.972218+0000","last_active":"2024-09-13T13:21:13.317467+0000","last_peered":"2024-09-13T13:21:13.317467+0000","last_clean":"2024-09-13T13:21:13.317467+0000","last_became_active":"2024-09-13T13:20:44.158986+0000","last_became_peered":"2024-09-13T13:20:44.158986+0000","last_unstale":"2024-09-13T13:21:13.317467+0000","last_undegraded":"2024-09-13T13:21:13.317467+0000","last_fullsized":"2024-09-13T13:21:13.317467+0000","mapping_epoch":488,"log_start":"442'3982","ondisk_log_start":"442'3982","created":450,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"489'8553","last_scrub_stamp":"2024-09-13T13:20:44.972097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:20:44.972097+0000","objects_scrubbed":1779,"log_size":5569,"log_dups_size":0,"ondisk_log_size":5569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:09.658625+0000","scrub_duration":351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51118080,"num_objects":780,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":999,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":939,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.320776+0000","last_change":"2024-09-13T13:18:41.949104+0000","last_active":"2024-09-13T13:21:07.320776+0000","last_peered":"2024-09-13T13:21:07.320776+0000","last_clean":"2024-09-13T13:21:07.320776+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:21:07.320776+0000","last_undegraded":"2024-09-13T13:21:07.320776+0000","last_fullsized":"2024-09-13T13:21:07.320776+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:41.949029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.949029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":441,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.318806+0000","last_change":"2024-09-13T13:18:46.181826+0000","last_active":"2024-09-13T13:21:07.318806+0000","last_peered":"2024-09-13T13:21:07.318806+0000","last_clean":"2024-09-13T13:21:07.318806+0000","last_became_active":"2024-09-13T13:17:25.244839+0000","last_became_peered":"2024-09-13T13:17:25.244839+0000","last_unstale":"2024-09-13T13:21:07.318806+0000","last_undegraded":"2024-09-13T13:21:07.318806+0000","last_fullsized":"2024-09-13T13:21:07.318806+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:46.181741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.181741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"500'24","reported_seq":1033,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.862087+0000","last_change":"2024-09-13T13:18:35.281750+0000","last_active":"2024-09-13T13:21:12.862087+0000","last_peered":"2024-09-13T13:21:12.862087+0000","last_clean":"2024-09-13T13:21:12.862087+0000","last_became_active":"2024-09-13T13:18:35.239702+0000","last_became_peered":"2024-09-13T13:18:35.239702+0000","last_unstale":"2024-09-13T13:21:12.862087+0000","last_undegraded":"2024-09-13T13:21:12.862087+0000","last_fullsized":"2024-09-13T13:21:12.862087+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:17:56.001763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:56.001763+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:01.267454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":435,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.318547+0000","last_change":"2024-09-13T13:18:32.362014+0000","last_active":"2024-09-13T13:21:07.318547+0000","last_peered":"2024-09-13T13:21:07.318547+0000","last_clean":"2024-09-13T13:21:07.318547+0000","last_became_active":"2024-09-13T13:17:25.245239+0000","last_became_peered":"2024-09-13T13:17:25.245239+0000","last_unstale":"2024-09-13T13:21:07.318547+0000","last_undegraded":"2024-09-13T13:21:07.318547+0000","last_fullsized":"2024-09-13T13:21:07.318547+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:32.361971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:32.361971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.11","version":"500'9401","reported_seq":14384,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:13.275306+0000","last_change":"2024-09-13T13:20:34.418954+0000","last_active":"2024-09-13T13:21:13.275306+0000","last_peered":"2024-09-13T13:21:13.275306+0000","last_clean":"2024-09-13T13:21:13.275306+0000","last_became_active":"2024-09-13T13:20:32.683660+0000","last_became_peered":"2024-09-13T13:20:32.683660+0000","last_unstale":"2024-09-13T13:21:13.275306+0000","last_undegraded":"2024-09-13T13:21:13.275306+0000","last_fullsized":"2024-09-13T13:21:13.275306+0000","mapping_epoch":485,"log_start":"441'3784","ondisk_log_start":"441'3784","created":450,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":5,"last_scrub":"486'8086","last_scrub_stamp":"2024-09-13T13:20:34.418689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:20:34.418689+0000","objects_scrubbed":2089,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:49.790205+0000","scrub_duration":766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50003968,"num_objects":763,"num_object_clones":0,"num_object_copies":2289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1326,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":487,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.323984+0000","last_change":"2024-09-13T13:19:43.128188+0000","last_active":"2024-09-13T13:21:07.323984+0000","last_peered":"2024-09-13T13:21:07.323984+0000","last_clean":"2024-09-13T13:21:07.323984+0000","last_became_active":"2024-09-13T13:19:43.116203+0000","last_became_peered":"2024-09-13T13:19:43.116203+0000","last_unstale":"2024-09-13T13:21:07.323984+0000","last_undegraded":"2024-09-13T13:21:07.323984+0000","last_fullsized":"2024-09-13T13:21:07.323984+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:19:35.778718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:19:35.778718+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":971,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.328925+0000","last_change":"2024-09-13T13:19:25.256499+0000","last_active":"2024-09-13T13:21:07.328925+0000","last_peered":"2024-09-13T13:21:07.328925+0000","last_clean":"2024-09-13T13:21:07.328925+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:21:07.328925+0000","last_undegraded":"2024-09-13T13:21:07.328925+0000","last_fullsized":"2024-09-13T13:21:07.328925+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:19:25.256420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:25.256420+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"500'30","reported_seq":1024,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.775460+0000","last_change":"2024-09-13T13:19:43.167125+0000","last_active":"2024-09-13T13:21:12.775460+0000","last_peered":"2024-09-13T13:21:12.775460+0000","last_clean":"2024-09-13T13:21:12.775460+0000","last_became_active":"2024-09-13T13:19:43.128174+0000","last_became_peered":"2024-09-13T13:19:43.128174+0000","last_unstale":"2024-09-13T13:21:12.775460+0000","last_undegraded":"2024-09-13T13:21:12.775460+0000","last_fullsized":"2024-09-13T13:21:12.775460+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:18:41.472008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.472008+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"467'13","reported_seq":1181,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.323834+0000","last_change":"2024-09-13T13:20:35.599418+0000","last_active":"2024-09-13T13:21:07.323834+0000","last_peered":"2024-09-13T13:21:07.323834+0000","last_clean":"2024-09-13T13:21:07.323834+0000","last_became_active":"2024-09-13T13:19:43.110544+0000","last_became_peered":"2024-09-13T13:19:43.110544+0000","last_unstale":"2024-09-13T13:21:07.323834+0000","last_undegraded":"2024-09-13T13:21:07.323834+0000","last_fullsized":"2024-09-13T13:21:07.323834+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'13","last_scrub_stamp":"2024-09-13T13:20:35.599368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:35.599368+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:47.345844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"500'9935","reported_seq":16564,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:14.818148+0000","last_change":"2024-09-13T13:20:10.241834+0000","last_active":"2024-09-13T13:21:14.818148+0000","last_peered":"2024-09-13T13:21:14.818148+0000","last_clean":"2024-09-13T13:21:14.818148+0000","last_became_active":"2024-09-13T13:20:10.229726+0000","last_became_peered":"2024-09-13T13:20:10.229726+0000","last_unstale":"2024-09-13T13:21:14.818148+0000","last_undegraded":"2024-09-13T13:21:14.818148+0000","last_fullsized":"2024-09-13T13:21:14.818148+0000","mapping_epoch":475,"log_start":"455'6346","ondisk_log_start":"455'6346","created":423,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'7038","last_scrub_stamp":"2024-09-13T13:19:38.280658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:38.280658+0000","objects_scrubbed":3671,"log_size":3589,"log_dups_size":0,"ondisk_log_size":3589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1040,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50200576,"num_objects":766,"num_object_clones":0,"num_object_copies":2298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2905,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3194,"num_bytes_recovered":209321984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":185,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.324711+0000","last_change":"2024-09-13T13:18:49.869928+0000","last_active":"2024-09-13T13:21:07.324711+0000","last_peered":"2024-09-13T13:21:07.324711+0000","last_clean":"2024-09-13T13:21:07.324711+0000","last_became_active":"2024-09-13T13:17:49.052966+0000","last_became_peered":"2024-09-13T13:17:49.052966+0000","last_unstale":"2024-09-13T13:21:07.324711+0000","last_undegraded":"2024-09-13T13:21:07.324711+0000","last_fullsized":"2024-09-13T13:21:07.324711+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:49.869851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:49.869851+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"500'39","reported_seq":1067,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.910606+0000","last_change":"2024-09-13T13:19:34.288799+0000","last_active":"2024-09-13T13:21:12.910606+0000","last_peered":"2024-09-13T13:21:12.910606+0000","last_clean":"2024-09-13T13:21:12.910606+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:21:12.910606+0000","last_undegraded":"2024-09-13T13:21:12.910606+0000","last_fullsized":"2024-09-13T13:21:12.910606+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:19:34.288614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:19:34.288614+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":979,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.321350+0000","last_change":"2024-09-13T13:20:34.332345+0000","last_active":"2024-09-13T13:21:07.321350+0000","last_peered":"2024-09-13T13:21:07.321350+0000","last_clean":"2024-09-13T13:21:07.321350+0000","last_became_active":"2024-09-13T13:19:43.137877+0000","last_became_peered":"2024-09-13T13:19:43.137877+0000","last_unstale":"2024-09-13T13:21:07.321350+0000","last_undegraded":"2024-09-13T13:21:07.321350+0000","last_fullsized":"2024-09-13T13:21:07.321350+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:34.332278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:34.332278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:58.646130+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":524,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.486621+0000","last_change":"2024-09-13T13:20:51.899838+0000","last_active":"2024-09-13T13:21:07.486621+0000","last_peered":"2024-09-13T13:21:07.486621+0000","last_clean":"2024-09-13T13:21:07.486621+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:21:07.486621+0000","last_undegraded":"2024-09-13T13:21:07.486621+0000","last_fullsized":"2024-09-13T13:21:07.486621+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:20:51.899778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:51.899778+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:10.314133+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":943,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.323491+0000","last_change":"2024-09-13T13:20:47.624758+0000","last_active":"2024-09-13T13:21:07.323491+0000","last_peered":"2024-09-13T13:21:07.323491+0000","last_clean":"2024-09-13T13:21:07.323491+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:21:07.323491+0000","last_undegraded":"2024-09-13T13:21:07.323491+0000","last_fullsized":"2024-09-13T13:21:07.323491+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:47.624694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:20:47.624694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:10.975469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"500'13317","reported_seq":22112,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:14.853616+0000","last_change":"2024-09-13T13:20:48.223870+0000","last_active":"2024-09-13T13:21:14.853616+0000","last_peered":"2024-09-13T13:21:14.853616+0000","last_clean":"2024-09-13T13:21:14.853616+0000","last_became_active":"2024-09-13T13:20:48.144620+0000","last_became_peered":"2024-09-13T13:20:48.144620+0000","last_unstale":"2024-09-13T13:21:14.853616+0000","last_undegraded":"2024-09-13T13:21:14.853616+0000","last_fullsized":"2024-09-13T13:21:14.853616+0000","mapping_epoch":491,"log_start":"479'9662","ondisk_log_start":"479'9662","created":423,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'7156","last_scrub_stamp":"2024-09-13T13:19:27.505480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:27.505480+0000","objects_scrubbed":7086,"log_size":3655,"log_dups_size":0,"ondisk_log_size":3655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92995584,"num_objects":1419,"num_object_clones":0,"num_object_copies":4257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13317,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4261,"num_bytes_recovered":279117824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":185,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.320186+0000","last_change":"2024-09-13T13:18:51.159546+0000","last_active":"2024-09-13T13:21:07.320186+0000","last_peered":"2024-09-13T13:21:07.320186+0000","last_clean":"2024-09-13T13:21:07.320186+0000","last_became_active":"2024-09-13T13:17:49.050655+0000","last_became_peered":"2024-09-13T13:17:49.050655+0000","last_unstale":"2024-09-13T13:21:07.320186+0000","last_undegraded":"2024-09-13T13:21:07.320186+0000","last_fullsized":"2024-09-13T13:21:07.320186+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.159496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.159496+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"500'42","reported_seq":1022,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.553673+0000","last_change":"2024-09-13T13:19:43.214168+0000","last_active":"2024-09-13T13:21:12.553673+0000","last_peered":"2024-09-13T13:21:12.553673+0000","last_clean":"2024-09-13T13:21:12.553673+0000","last_became_active":"2024-09-13T13:19:43.133611+0000","last_became_peered":"2024-09-13T13:19:43.133611+0000","last_unstale":"2024-09-13T13:21:12.553673+0000","last_undegraded":"2024-09-13T13:21:12.553673+0000","last_fullsized":"2024-09-13T13:21:12.553673+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:18:36.921964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:36.921964+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":949,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.321174+0000","last_change":"2024-09-13T13:19:43.112484+0000","last_active":"2024-09-13T13:21:07.321174+0000","last_peered":"2024-09-13T13:21:07.321174+0000","last_clean":"2024-09-13T13:21:07.321174+0000","last_became_active":"2024-09-13T13:19:43.102458+0000","last_became_peered":"2024-09-13T13:19:43.102458+0000","last_unstale":"2024-09-13T13:21:07.321174+0000","last_undegraded":"2024-09-13T13:21:07.321174+0000","last_fullsized":"2024-09-13T13:21:07.321174+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:16:26.098365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:26.098365+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:22.128239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":427,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319237+0000","last_change":"2024-09-13T13:19:43.104198+0000","last_active":"2024-09-13T13:21:07.319237+0000","last_peered":"2024-09-13T13:21:07.319237+0000","last_clean":"2024-09-13T13:21:07.319237+0000","last_became_active":"2024-09-13T13:19:43.103878+0000","last_became_peered":"2024-09-13T13:19:43.103878+0000","last_unstale":"2024-09-13T13:21:07.319237+0000","last_undegraded":"2024-09-13T13:21:07.319237+0000","last_fullsized":"2024-09-13T13:21:07.319237+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:58.389763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:58.389763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:09.938779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":442,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319184+0000","last_change":"2024-09-13T13:17:59.373240+0000","last_active":"2024-09-13T13:21:07.319184+0000","last_peered":"2024-09-13T13:21:07.319184+0000","last_clean":"2024-09-13T13:21:07.319184+0000","last_became_active":"2024-09-13T13:17:25.244171+0000","last_became_peered":"2024-09-13T13:17:25.244171+0000","last_unstale":"2024-09-13T13:21:07.319184+0000","last_undegraded":"2024-09-13T13:21:07.319184+0000","last_fullsized":"2024-09-13T13:21:07.319184+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.373194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.373194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"500'13181","reported_seq":24208,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:13.323674+0000","last_change":"2024-09-13T13:20:32.417405+0000","last_active":"2024-09-13T13:21:13.323674+0000","last_peered":"2024-09-13T13:21:13.323674+0000","last_clean":"2024-09-13T13:21:13.323674+0000","last_became_active":"2024-09-13T13:20:32.403062+0000","last_became_peered":"2024-09-13T13:20:32.403062+0000","last_unstale":"2024-09-13T13:21:13.323674+0000","last_undegraded":"2024-09-13T13:21:13.323674+0000","last_fullsized":"2024-09-13T13:21:13.323674+0000","mapping_epoch":485,"log_start":"473'9094","ondisk_log_start":"473'9094","created":423,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101515264,"num_objects":1549,"num_object_clones":0,"num_object_copies":4647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13181,"num_write_kb":471360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5421,"num_bytes_recovered":355205120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":204,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.318717+0000","last_change":"2024-09-13T13:19:43.120655+0000","last_active":"2024-09-13T13:21:07.318717+0000","last_peered":"2024-09-13T13:21:07.318717+0000","last_clean":"2024-09-13T13:21:07.318717+0000","last_became_active":"2024-09-13T13:19:43.106076+0000","last_became_peered":"2024-09-13T13:19:43.106076+0000","last_unstale":"2024-09-13T13:21:07.318717+0000","last_undegraded":"2024-09-13T13:21:07.318717+0000","last_fullsized":"2024-09-13T13:21:07.318717+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:19:23.064923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:19:23.064923+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"500'42","reported_seq":457,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.783287+0000","last_change":"2024-09-13T13:20:41.949473+0000","last_active":"2024-09-13T13:21:12.783287+0000","last_peered":"2024-09-13T13:21:12.783287+0000","last_clean":"2024-09-13T13:21:12.783287+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:21:12.783287+0000","last_undegraded":"2024-09-13T13:21:12.783287+0000","last_fullsized":"2024-09-13T13:21:12.783287+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:20:41.949422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:41.949422+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:56.729205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":954,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.316920+0000","last_change":"2024-09-13T13:20:54.179763+0000","last_active":"2024-09-13T13:21:07.316920+0000","last_peered":"2024-09-13T13:21:07.316920+0000","last_clean":"2024-09-13T13:21:07.316920+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:21:07.316920+0000","last_undegraded":"2024-09-13T13:21:07.316920+0000","last_fullsized":"2024-09-13T13:21:07.316920+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:20:54.179075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:54.179075+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:16.137252+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":1214,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.486149+0000","last_change":"2024-09-13T13:18:48.032272+0000","last_active":"2024-09-13T13:21:07.486149+0000","last_peered":"2024-09-13T13:21:07.486149+0000","last_clean":"2024-09-13T13:21:07.486149+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:21:07.486149+0000","last_undegraded":"2024-09-13T13:21:07.486149+0000","last_fullsized":"2024-09-13T13:21:07.486149+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:18:48.032199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:48.032199+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":599,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.324556+0000","last_change":"2024-09-13T13:20:39.526912+0000","last_active":"2024-09-13T13:21:07.324556+0000","last_peered":"2024-09-13T13:21:07.324556+0000","last_clean":"2024-09-13T13:21:07.324556+0000","last_became_active":"2024-09-13T13:19:43.130374+0000","last_became_peered":"2024-09-13T13:19:43.130374+0000","last_unstale":"2024-09-13T13:21:07.324556+0000","last_undegraded":"2024-09-13T13:21:07.324556+0000","last_fullsized":"2024-09-13T13:21:07.324556+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:39.526852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:20:39.526852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:04.402168+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"500'12806","reported_seq":13050,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:13.358237+0000","last_change":"2024-09-13T13:19:43.288579+0000","last_active":"2024-09-13T13:21:13.358237+0000","last_peered":"2024-09-13T13:21:13.358237+0000","last_clean":"2024-09-13T13:21:13.358237+0000","last_became_active":"2024-09-13T13:17:55.131230+0000","last_became_peered":"2024-09-13T13:17:55.131230+0000","last_unstale":"2024-09-13T13:21:13.358237+0000","last_undegraded":"2024-09-13T13:21:13.358237+0000","last_fullsized":"2024-09-13T13:21:13.358237+0000","mapping_epoch":423,"log_start":"469'8141","ondisk_log_start":"469'8141","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'7729","last_scrub_stamp":"2024-09-13T13:19:43.288518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:43.288518+0000","objects_scrubbed":6606,"log_size":4665,"log_dups_size":0,"ondisk_log_size":4665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:08.647064+0000","scrub_duration":765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100073496,"num_objects":1528,"num_object_clones":0,"num_object_copies":4584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12806,"num_write_kb":458625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":276,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.323400+0000","last_change":"2024-09-13T13:20:48.542448+0000","last_active":"2024-09-13T13:21:07.323400+0000","last_peered":"2024-09-13T13:21:07.323400+0000","last_clean":"2024-09-13T13:21:07.323400+0000","last_became_active":"2024-09-13T13:19:43.106463+0000","last_became_peered":"2024-09-13T13:19:43.106463+0000","last_unstale":"2024-09-13T13:21:07.323400+0000","last_undegraded":"2024-09-13T13:21:07.323400+0000","last_fullsized":"2024-09-13T13:21:07.323400+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:20:48.542384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:20:48.542384+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:52.910447+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"500'30","reported_seq":1020,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.706102+0000","last_change":"2024-09-13T13:20:48.549680+0000","last_active":"2024-09-13T13:21:12.706102+0000","last_peered":"2024-09-13T13:21:12.706102+0000","last_clean":"2024-09-13T13:21:12.706102+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:21:12.706102+0000","last_undegraded":"2024-09-13T13:21:12.706102+0000","last_fullsized":"2024-09-13T13:21:12.706102+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:20:48.549598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:48.549598+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:59.549477+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":986,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.325455+0000","last_change":"2024-09-13T13:19:13.702236+0000","last_active":"2024-09-13T13:21:07.325455+0000","last_peered":"2024-09-13T13:21:07.325455+0000","last_clean":"2024-09-13T13:21:07.325455+0000","last_became_active":"2024-09-13T13:17:25.331166+0000","last_became_peered":"2024-09-13T13:17:25.331166+0000","last_unstale":"2024-09-13T13:21:07.325455+0000","last_undegraded":"2024-09-13T13:21:07.325455+0000","last_fullsized":"2024-09-13T13:21:07.325455+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:13.702185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:13.702185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":1196,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.484786+0000","last_change":"2024-09-13T13:18:40.464456+0000","last_active":"2024-09-13T13:21:07.484786+0000","last_peered":"2024-09-13T13:21:07.484786+0000","last_clean":"2024-09-13T13:21:07.484786+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:21:07.484786+0000","last_undegraded":"2024-09-13T13:21:07.484786+0000","last_fullsized":"2024-09-13T13:21:07.484786+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:18:40.464398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:40.464398+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":974,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.314434+0000","last_change":"2024-09-13T13:18:36.466657+0000","last_active":"2024-09-13T13:21:07.314434+0000","last_peered":"2024-09-13T13:21:07.314434+0000","last_clean":"2024-09-13T13:21:07.314434+0000","last_became_active":"2024-09-13T13:17:25.252712+0000","last_became_peered":"2024-09-13T13:17:25.252712+0000","last_unstale":"2024-09-13T13:21:07.314434+0000","last_undegraded":"2024-09-13T13:21:07.314434+0000","last_fullsized":"2024-09-13T13:21:07.314434+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:36.466575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:36.466575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"500'13234","reported_seq":13484,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:14.850654+0000","last_change":"2024-09-13T13:19:36.549311+0000","last_active":"2024-09-13T13:21:14.850654+0000","last_peered":"2024-09-13T13:21:14.850654+0000","last_clean":"2024-09-13T13:21:14.850654+0000","last_became_active":"2024-09-13T13:17:55.140134+0000","last_became_peered":"2024-09-13T13:17:55.140134+0000","last_unstale":"2024-09-13T13:21:14.850654+0000","last_undegraded":"2024-09-13T13:21:14.850654+0000","last_fullsized":"2024-09-13T13:21:14.850654+0000","mapping_epoch":423,"log_start":"479'9579","ondisk_log_start":"479'9579","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'7653","last_scrub_stamp":"2024-09-13T13:19:36.549087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:36.549087+0000","objects_scrubbed":7193,"log_size":3655,"log_dups_size":0,"ondisk_log_size":3655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3276,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99221504,"num_objects":1514,"num_object_clones":0,"num_object_copies":4542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13234,"num_write_kb":471936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":269,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.323260+0000","last_change":"2024-09-13T13:20:50.537234+0000","last_active":"2024-09-13T13:21:07.323260+0000","last_peered":"2024-09-13T13:21:07.323260+0000","last_clean":"2024-09-13T13:21:07.323260+0000","last_became_active":"2024-09-13T13:17:49.055888+0000","last_became_peered":"2024-09-13T13:17:49.055888+0000","last_unstale":"2024-09-13T13:21:07.323260+0000","last_undegraded":"2024-09-13T13:21:07.323260+0000","last_fullsized":"2024-09-13T13:21:07.323260+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:20:50.537158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:20:50.537158+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:54.570017+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"500'39","reported_seq":1050,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.655610+0000","last_change":"2024-09-13T13:19:43.203331+0000","last_active":"2024-09-13T13:21:12.655610+0000","last_peered":"2024-09-13T13:21:12.655610+0000","last_clean":"2024-09-13T13:21:12.655610+0000","last_became_active":"2024-09-13T13:19:43.119487+0000","last_became_peered":"2024-09-13T13:19:43.119487+0000","last_unstale":"2024-09-13T13:21:12.655610+0000","last_undegraded":"2024-09-13T13:21:12.655610+0000","last_fullsized":"2024-09-13T13:21:12.655610+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:18:33.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.796644+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":963,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.331501+0000","last_change":"2024-09-13T13:19:31.224359+0000","last_active":"2024-09-13T13:21:07.331501+0000","last_peered":"2024-09-13T13:21:07.331501+0000","last_clean":"2024-09-13T13:21:07.331501+0000","last_became_active":"2024-09-13T13:17:25.332589+0000","last_became_peered":"2024-09-13T13:17:25.332589+0000","last_unstale":"2024-09-13T13:21:07.331501+0000","last_undegraded":"2024-09-13T13:21:07.331501+0000","last_fullsized":"2024-09-13T13:21:07.331501+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:31.224272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:31.224272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":925,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.325712+0000","last_change":"2024-09-13T13:19:20.502538+0000","last_active":"2024-09-13T13:21:07.325712+0000","last_peered":"2024-09-13T13:21:07.325712+0000","last_clean":"2024-09-13T13:21:07.325712+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:21:07.325712+0000","last_undegraded":"2024-09-13T13:21:07.325712+0000","last_fullsized":"2024-09-13T13:21:07.325712+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:20.502415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:20.502415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:00.660401+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":970,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.328690+0000","last_change":"2024-09-13T13:19:23.209075+0000","last_active":"2024-09-13T13:21:07.328690+0000","last_peered":"2024-09-13T13:21:07.328690+0000","last_clean":"2024-09-13T13:21:07.328690+0000","last_became_active":"2024-09-13T13:17:25.244517+0000","last_became_peered":"2024-09-13T13:17:25.244517+0000","last_unstale":"2024-09-13T13:21:07.328690+0000","last_undegraded":"2024-09-13T13:21:07.328690+0000","last_fullsized":"2024-09-13T13:21:07.328690+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:23.208994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:19:23.208994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"500'12673","reported_seq":12842,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:13.063345+0000","last_change":"2024-09-13T13:17:55.146287+0000","last_active":"2024-09-13T13:21:13.063345+0000","last_peered":"2024-09-13T13:21:13.063345+0000","last_clean":"2024-09-13T13:21:13.063345+0000","last_became_active":"2024-09-13T13:17:55.145616+0000","last_became_peered":"2024-09-13T13:17:55.145616+0000","last_unstale":"2024-09-13T13:21:13.063345+0000","last_undegraded":"2024-09-13T13:21:13.063345+0000","last_fullsized":"2024-09-13T13:21:13.063345+0000","mapping_epoch":423,"log_start":"472'8738","ondisk_log_start":"472'8738","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":3935,"log_dups_size":0,"ondisk_log_size":3935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:07.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100859904,"num_objects":1539,"num_object_clones":0,"num_object_copies":4617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12673,"num_write_kb":454784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":185,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.323628+0000","last_change":"2024-09-13T13:18:50.856493+0000","last_active":"2024-09-13T13:21:07.323628+0000","last_peered":"2024-09-13T13:21:07.323628+0000","last_clean":"2024-09-13T13:21:07.323628+0000","last_became_active":"2024-09-13T13:17:49.054449+0000","last_became_peered":"2024-09-13T13:17:49.054449+0000","last_unstale":"2024-09-13T13:21:07.323628+0000","last_undegraded":"2024-09-13T13:21:07.323628+0000","last_fullsized":"2024-09-13T13:21:07.323628+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:50.856437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:50.856437+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"500'55","reported_seq":201,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.718739+0000","last_change":"2024-09-13T13:18:35.332086+0000","last_active":"2024-09-13T13:21:12.718739+0000","last_peered":"2024-09-13T13:21:12.718739+0000","last_clean":"2024-09-13T13:21:12.718739+0000","last_became_active":"2024-09-13T13:18:35.244311+0000","last_became_peered":"2024-09-13T13:18:35.244311+0000","last_unstale":"2024-09-13T13:21:12.718739+0000","last_undegraded":"2024-09-13T13:21:12.718739+0000","last_fullsized":"2024-09-13T13:21:12.718739+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'42","last_scrub_stamp":"2024-09-13T13:18:01.158345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:01.158345+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":428,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319009+0000","last_change":"2024-09-13T13:17:52.422920+0000","last_active":"2024-09-13T13:21:07.319009+0000","last_peered":"2024-09-13T13:21:07.319009+0000","last_clean":"2024-09-13T13:21:07.319009+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:21:07.319009+0000","last_undegraded":"2024-09-13T13:21:07.319009+0000","last_fullsized":"2024-09-13T13:21:07.319009+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:52.422872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.422872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":959,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.325287+0000","last_change":"2024-09-13T13:18:07.306441+0000","last_active":"2024-09-13T13:21:07.325287+0000","last_peered":"2024-09-13T13:21:07.325287+0000","last_clean":"2024-09-13T13:21:07.325287+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:21:07.325287+0000","last_undegraded":"2024-09-13T13:21:07.325287+0000","last_fullsized":"2024-09-13T13:21:07.325287+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:18:07.306366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:07.306366+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:06.626914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":961,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.313670+0000","last_change":"2024-09-13T13:20:50.196625+0000","last_active":"2024-09-13T13:21:07.313670+0000","last_peered":"2024-09-13T13:21:07.313670+0000","last_clean":"2024-09-13T13:21:07.313670+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:21:07.313670+0000","last_undegraded":"2024-09-13T13:21:07.313670+0000","last_fullsized":"2024-09-13T13:21:07.313670+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:50.196539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:50.196539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:56.679929+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":965,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.320140+0000","last_change":"2024-09-13T13:20:48.336180+0000","last_active":"2024-09-13T13:21:07.320140+0000","last_peered":"2024-09-13T13:21:07.320140+0000","last_clean":"2024-09-13T13:21:07.320140+0000","last_became_active":"2024-09-13T13:19:43.105507+0000","last_became_peered":"2024-09-13T13:19:43.105507+0000","last_unstale":"2024-09-13T13:21:07.320140+0000","last_undegraded":"2024-09-13T13:21:07.320140+0000","last_fullsized":"2024-09-13T13:21:07.320140+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:48.336110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:20:48.336110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:53.933105+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"500'9551","reported_seq":9776,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.920821+0000","last_change":"2024-09-13T13:19:38.017114+0000","last_active":"2024-09-13T13:21:12.920821+0000","last_peered":"2024-09-13T13:21:12.920821+0000","last_clean":"2024-09-13T13:21:12.920821+0000","last_became_active":"2024-09-13T13:19:05.196340+0000","last_became_peered":"2024-09-13T13:19:05.196340+0000","last_unstale":"2024-09-13T13:21:12.920821+0000","last_undegraded":"2024-09-13T13:21:12.920821+0000","last_fullsized":"2024-09-13T13:21:12.920821+0000","mapping_epoch":423,"log_start":"447'5300","ondisk_log_start":"447'5300","created":423,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'6830","last_scrub_stamp":"2024-09-13T13:19:38.016951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:38.016951+0000","objects_scrubbed":3491,"log_size":4251,"log_dups_size":0,"ondisk_log_size":4251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50266112,"num_objects":767,"num_object_clones":0,"num_object_copies":2301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2724,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":554,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.313227+0000","last_change":"2024-09-13T13:19:43.165270+0000","last_active":"2024-09-13T13:21:07.313227+0000","last_peered":"2024-09-13T13:21:07.313227+0000","last_clean":"2024-09-13T13:21:07.313227+0000","last_became_active":"2024-09-13T13:19:43.143230+0000","last_became_peered":"2024-09-13T13:19:43.143230+0000","last_unstale":"2024-09-13T13:21:07.313227+0000","last_undegraded":"2024-09-13T13:21:07.313227+0000","last_fullsized":"2024-09-13T13:21:07.313227+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.529089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.529089+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"500'21","reported_seq":984,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.152301+0000","last_change":"2024-09-13T13:19:22.172479+0000","last_active":"2024-09-13T13:21:12.152301+0000","last_peered":"2024-09-13T13:21:12.152301+0000","last_clean":"2024-09-13T13:21:12.152301+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:21:12.152301+0000","last_undegraded":"2024-09-13T13:21:12.152301+0000","last_fullsized":"2024-09-13T13:21:12.152301+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'17","last_scrub_stamp":"2024-09-13T13:19:22.172183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:19:22.172183+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":983,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.328573+0000","last_change":"2024-09-13T13:18:33.892926+0000","last_active":"2024-09-13T13:21:07.328573+0000","last_peered":"2024-09-13T13:21:07.328573+0000","last_clean":"2024-09-13T13:21:07.328573+0000","last_became_active":"2024-09-13T13:17:25.246400+0000","last_became_peered":"2024-09-13T13:17:25.246400+0000","last_unstale":"2024-09-13T13:21:07.328573+0000","last_undegraded":"2024-09-13T13:21:07.328573+0000","last_fullsized":"2024-09-13T13:21:07.328573+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:33.892597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.892597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:04.671259+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":207,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.327051+0000","last_change":"2024-09-13T13:18:44.719355+0000","last_active":"2024-09-13T13:21:07.327051+0000","last_peered":"2024-09-13T13:21:07.327051+0000","last_clean":"2024-09-13T13:21:07.327051+0000","last_became_active":"2024-09-13T13:17:25.252664+0000","last_became_peered":"2024-09-13T13:17:25.252664+0000","last_unstale":"2024-09-13T13:21:07.327051+0000","last_undegraded":"2024-09-13T13:21:07.327051+0000","last_fullsized":"2024-09-13T13:21:07.327051+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:18:44.719269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:44.719269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":218,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.323237+0000","last_change":"2024-09-13T13:20:52.590805+0000","last_active":"2024-09-13T13:21:07.323237+0000","last_peered":"2024-09-13T13:21:07.323237+0000","last_clean":"2024-09-13T13:21:07.323237+0000","last_became_active":"2024-09-13T13:19:43.121063+0000","last_became_peered":"2024-09-13T13:19:43.121063+0000","last_unstale":"2024-09-13T13:21:07.323237+0000","last_undegraded":"2024-09-13T13:21:07.323237+0000","last_fullsized":"2024-09-13T13:21:07.323237+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:52.590733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:52.590733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:55.206604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":569,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.324152+0000","last_change":"2024-09-13T13:19:43.111710+0000","last_active":"2024-09-13T13:21:07.324152+0000","last_peered":"2024-09-13T13:21:07.324152+0000","last_clean":"2024-09-13T13:21:07.324152+0000","last_became_active":"2024-09-13T13:19:43.110460+0000","last_became_peered":"2024-09-13T13:19:43.110460+0000","last_unstale":"2024-09-13T13:21:07.324152+0000","last_undegraded":"2024-09-13T13:21:07.324152+0000","last_fullsized":"2024-09-13T13:21:07.324152+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:26.607881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:19:26.607881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"500'9478","reported_seq":9703,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:13.057830+0000","last_change":"2024-09-13T13:19:39.176073+0000","last_active":"2024-09-13T13:21:13.057830+0000","last_peered":"2024-09-13T13:21:13.057830+0000","last_clean":"2024-09-13T13:21:13.057830+0000","last_became_active":"2024-09-13T13:19:05.190236+0000","last_became_peered":"2024-09-13T13:19:05.190236+0000","last_unstale":"2024-09-13T13:21:13.057830+0000","last_undegraded":"2024-09-13T13:21:13.057830+0000","last_fullsized":"2024-09-13T13:21:13.057830+0000","mapping_epoch":423,"log_start":"447'5273","ondisk_log_start":"447'5273","created":423,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'6785","last_scrub_stamp":"2024-09-13T13:19:39.175880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:39.175880+0000","objects_scrubbed":3498,"log_size":4205,"log_dups_size":0,"ondisk_log_size":4205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52363264,"num_objects":799,"num_object_clones":0,"num_object_copies":2397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2699,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":185,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.331111+0000","last_change":"2024-09-13T13:18:51.048420+0000","last_active":"2024-09-13T13:21:07.331111+0000","last_peered":"2024-09-13T13:21:07.331111+0000","last_clean":"2024-09-13T13:21:07.331111+0000","last_became_active":"2024-09-13T13:17:49.049531+0000","last_became_peered":"2024-09-13T13:17:49.049531+0000","last_unstale":"2024-09-13T13:21:07.331111+0000","last_undegraded":"2024-09-13T13:21:07.331111+0000","last_fullsized":"2024-09-13T13:21:07.331111+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.048347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.048347+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"500'36","reported_seq":1039,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.244667+0000","last_change":"2024-09-13T13:18:04.569761+0000","last_active":"2024-09-13T13:21:12.244667+0000","last_peered":"2024-09-13T13:21:12.244667+0000","last_clean":"2024-09-13T13:21:12.244667+0000","last_became_active":"2024-09-13T13:17:25.253120+0000","last_became_peered":"2024-09-13T13:17:25.253120+0000","last_unstale":"2024-09-13T13:21:12.244667+0000","last_undegraded":"2024-09-13T13:21:12.244667+0000","last_fullsized":"2024-09-13T13:21:12.244667+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:18:04.569670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:04.569670+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:12.008022+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":961,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319425+0000","last_change":"2024-09-13T13:18:37.465516+0000","last_active":"2024-09-13T13:21:07.319425+0000","last_peered":"2024-09-13T13:21:07.319425+0000","last_clean":"2024-09-13T13:21:07.319425+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:21:07.319425+0000","last_undegraded":"2024-09-13T13:21:07.319425+0000","last_fullsized":"2024-09-13T13:21:07.319425+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:37.465371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:37.465371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1008,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:13.763841+0000","last_change":"2024-09-13T13:19:43.169544+0000","last_active":"2024-09-13T13:21:13.763841+0000","last_peered":"2024-09-13T13:21:13.763841+0000","last_clean":"2024-09-13T13:21:13.763841+0000","last_became_active":"2024-09-13T13:19:43.137630+0000","last_became_peered":"2024-09-13T13:19:43.137630+0000","last_unstale":"2024-09-13T13:21:13.763841+0000","last_undegraded":"2024-09-13T13:21:13.763841+0000","last_fullsized":"2024-09-13T13:21:13.763841+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:17:25.325275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:25.325275+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:55.618401+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":516,"num_read_kb":511,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":932,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.327333+0000","last_change":"2024-09-13T13:17:55.293423+0000","last_active":"2024-09-13T13:21:07.327333+0000","last_peered":"2024-09-13T13:21:07.327333+0000","last_clean":"2024-09-13T13:21:07.327333+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:21:07.327333+0000","last_undegraded":"2024-09-13T13:21:07.327333+0000","last_fullsized":"2024-09-13T13:21:07.327333+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:55.293375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:55.293375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":937,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.331009+0000","last_change":"2024-09-13T13:18:46.991251+0000","last_active":"2024-09-13T13:21:07.331009+0000","last_peered":"2024-09-13T13:21:07.331009+0000","last_clean":"2024-09-13T13:21:07.331009+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:21:07.331009+0000","last_undegraded":"2024-09-13T13:21:07.331009+0000","last_fullsized":"2024-09-13T13:21:07.331009+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:46.991076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.991076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"500'13113","reported_seq":13282,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:13.148749+0000","last_change":"2024-09-13T13:17:55.142415+0000","last_active":"2024-09-13T13:21:13.148749+0000","last_peered":"2024-09-13T13:21:13.148749+0000","last_clean":"2024-09-13T13:21:13.148749+0000","last_became_active":"2024-09-13T13:17:55.142112+0000","last_became_peered":"2024-09-13T13:17:55.142112+0000","last_unstale":"2024-09-13T13:21:13.148749+0000","last_undegraded":"2024-09-13T13:21:13.148749+0000","last_fullsized":"2024-09-13T13:21:13.148749+0000","mapping_epoch":423,"log_start":"474'9188","ondisk_log_start":"474'9188","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:02.224353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98107392,"num_objects":1497,"num_object_clones":0,"num_object_copies":4491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13113,"num_write_kb":467520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"500'41","reported_seq":1165,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.325982+0000","last_change":"2024-09-13T13:19:43.166438+0000","last_active":"2024-09-13T13:21:11.325982+0000","last_peered":"2024-09-13T13:21:11.325982+0000","last_clean":"2024-09-13T13:21:11.325982+0000","last_became_active":"2024-09-13T13:19:43.107259+0000","last_became_peered":"2024-09-13T13:19:43.107259+0000","last_unstale":"2024-09-13T13:21:11.325982+0000","last_undegraded":"2024-09-13T13:21:11.325982+0000","last_fullsized":"2024-09-13T13:21:11.325982+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:19:13.332009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:19:13.332009+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:04.660674+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":33,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":951,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.320440+0000","last_change":"2024-09-13T13:16:53.951208+0000","last_active":"2024-09-13T13:21:07.320440+0000","last_peered":"2024-09-13T13:21:07.320440+0000","last_clean":"2024-09-13T13:21:07.320440+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:21:07.320440+0000","last_undegraded":"2024-09-13T13:21:07.320440+0000","last_fullsized":"2024-09-13T13:21:07.320440+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:53.951135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:53.951135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:28.122637+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":552,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.314670+0000","last_change":"2024-09-13T13:19:43.117386+0000","last_active":"2024-09-13T13:21:07.314670+0000","last_peered":"2024-09-13T13:21:07.314670+0000","last_clean":"2024-09-13T13:21:07.314670+0000","last_became_active":"2024-09-13T13:19:43.117052+0000","last_became_peered":"2024-09-13T13:19:43.117052+0000","last_unstale":"2024-09-13T13:21:07.314670+0000","last_undegraded":"2024-09-13T13:21:07.314670+0000","last_fullsized":"2024-09-13T13:21:07.314670+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:50.485924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:50.485924+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":218,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.324365+0000","last_change":"2024-09-13T13:20:54.560490+0000","last_active":"2024-09-13T13:21:07.324365+0000","last_peered":"2024-09-13T13:21:07.324365+0000","last_clean":"2024-09-13T13:21:07.324365+0000","last_became_active":"2024-09-13T13:19:43.120724+0000","last_became_peered":"2024-09-13T13:19:43.120724+0000","last_unstale":"2024-09-13T13:21:07.324365+0000","last_undegraded":"2024-09-13T13:21:07.324365+0000","last_fullsized":"2024-09-13T13:21:07.324365+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:54.560428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:54.560428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:08.160905+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":982,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.329729+0000","last_change":"2024-09-13T13:19:39.865452+0000","last_active":"2024-09-13T13:21:07.329729+0000","last_peered":"2024-09-13T13:21:07.329729+0000","last_clean":"2024-09-13T13:21:07.329729+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:21:07.329729+0000","last_undegraded":"2024-09-13T13:21:07.329729+0000","last_fullsized":"2024-09-13T13:21:07.329729+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:39.865400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:19:39.865400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"500'9276","reported_seq":9500,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.374521+0000","last_change":"2024-09-13T13:19:37.669705+0000","last_active":"2024-09-13T13:21:11.374521+0000","last_peered":"2024-09-13T13:21:11.374521+0000","last_clean":"2024-09-13T13:21:11.374521+0000","last_became_active":"2024-09-13T13:19:05.190417+0000","last_became_peered":"2024-09-13T13:19:05.190417+0000","last_unstale":"2024-09-13T13:21:11.374521+0000","last_undegraded":"2024-09-13T13:21:11.374521+0000","last_fullsized":"2024-09-13T13:21:11.374521+0000","mapping_epoch":423,"log_start":"443'4133","ondisk_log_start":"443'4133","created":423,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'6678","last_scrub_stamp":"2024-09-13T13:19:37.669545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:37.669545+0000","objects_scrubbed":3453,"log_size":5143,"log_dups_size":0,"ondisk_log_size":5143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55377920,"num_objects":845,"num_object_clones":0,"num_object_copies":2535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2608,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"500'27","reported_seq":451,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.760224+0000","last_change":"2024-09-13T13:18:09.522990+0000","last_active":"2024-09-13T13:21:12.760224+0000","last_peered":"2024-09-13T13:21:12.760224+0000","last_clean":"2024-09-13T13:21:12.760224+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:21:12.760224+0000","last_undegraded":"2024-09-13T13:21:12.760224+0000","last_fullsized":"2024-09-13T13:21:12.760224+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:09.522908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:09.522908+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":440,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.318010+0000","last_change":"2024-09-13T13:19:43.131658+0000","last_active":"2024-09-13T13:21:07.318010+0000","last_peered":"2024-09-13T13:21:07.318010+0000","last_clean":"2024-09-13T13:21:07.318010+0000","last_became_active":"2024-09-13T13:19:43.131319+0000","last_became_peered":"2024-09-13T13:19:43.131319+0000","last_unstale":"2024-09-13T13:21:07.318010+0000","last_undegraded":"2024-09-13T13:21:07.318010+0000","last_fullsized":"2024-09-13T13:21:07.318010+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.465363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.465363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":988,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.329778+0000","last_change":"2024-09-13T13:19:30.758268+0000","last_active":"2024-09-13T13:21:07.329778+0000","last_peered":"2024-09-13T13:21:07.329778+0000","last_clean":"2024-09-13T13:21:07.329778+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:21:07.329778+0000","last_undegraded":"2024-09-13T13:21:07.329778+0000","last_fullsized":"2024-09-13T13:21:07.329778+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:30.758170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:30.758170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":938,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.318874+0000","last_change":"2024-09-13T13:18:49.072017+0000","last_active":"2024-09-13T13:21:07.318874+0000","last_peered":"2024-09-13T13:21:07.318874+0000","last_clean":"2024-09-13T13:21:07.318874+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:21:07.318874+0000","last_undegraded":"2024-09-13T13:21:07.318874+0000","last_fullsized":"2024-09-13T13:21:07.318874+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:49.071942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:49.071942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"500'9456","reported_seq":9681,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:13.081197+0000","last_change":"2024-09-13T13:19:16.555177+0000","last_active":"2024-09-13T13:21:13.081197+0000","last_peered":"2024-09-13T13:21:13.081197+0000","last_clean":"2024-09-13T13:21:13.081197+0000","last_became_active":"2024-09-13T13:19:14.300133+0000","last_became_peered":"2024-09-13T13:19:14.300133+0000","last_unstale":"2024-09-13T13:21:13.081197+0000","last_undegraded":"2024-09-13T13:21:13.081197+0000","last_fullsized":"2024-09-13T13:21:13.081197+0000","mapping_epoch":456,"log_start":"447'5177","ondisk_log_start":"447'5177","created":423,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'6169","last_scrub_stamp":"2024-09-13T13:19:16.554998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:16.554998+0000","objects_scrubbed":3151,"log_size":4279,"log_dups_size":0,"ondisk_log_size":4279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50200576,"num_objects":766,"num_object_clones":0,"num_object_copies":2298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3309,"num_write_kb":29568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":194,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.323341+0000","last_change":"2024-09-13T13:19:41.866832+0000","last_active":"2024-09-13T13:21:07.323341+0000","last_peered":"2024-09-13T13:21:07.323341+0000","last_clean":"2024-09-13T13:21:07.323341+0000","last_became_active":"2024-09-13T13:17:49.053269+0000","last_became_peered":"2024-09-13T13:17:49.053269+0000","last_unstale":"2024-09-13T13:21:07.323341+0000","last_undegraded":"2024-09-13T13:21:07.323341+0000","last_fullsized":"2024-09-13T13:21:07.323341+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:19:41.866752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:19:41.866752+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"500'45","reported_seq":516,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.747450+0000","last_change":"2024-09-13T13:18:48.229223+0000","last_active":"2024-09-13T13:21:12.747450+0000","last_peered":"2024-09-13T13:21:12.747450+0000","last_clean":"2024-09-13T13:21:12.747450+0000","last_became_active":"2024-09-13T13:17:25.244686+0000","last_became_peered":"2024-09-13T13:17:25.244686+0000","last_unstale":"2024-09-13T13:21:12.747450+0000","last_undegraded":"2024-09-13T13:21:12.747450+0000","last_fullsized":"2024-09-13T13:21:12.747450+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:18:48.229160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:48.229160+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":978,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.326038+0000","last_change":"2024-09-13T13:19:14.352608+0000","last_active":"2024-09-13T13:21:07.326038+0000","last_peered":"2024-09-13T13:21:07.326038+0000","last_clean":"2024-09-13T13:21:07.326038+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:21:07.326038+0000","last_undegraded":"2024-09-13T13:21:07.326038+0000","last_fullsized":"2024-09-13T13:21:07.326038+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:14.352340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:14.352340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:01.678090+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":962,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.331201+0000","last_change":"2024-09-13T13:19:15.064401+0000","last_active":"2024-09-13T13:21:07.331201+0000","last_peered":"2024-09-13T13:21:07.331201+0000","last_clean":"2024-09-13T13:21:07.331201+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:21:07.331201+0000","last_undegraded":"2024-09-13T13:21:07.331201+0000","last_fullsized":"2024-09-13T13:21:07.331201+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:15.064195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:15.064195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":704,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.320615+0000","last_change":"2024-09-13T13:19:14.297300+0000","last_active":"2024-09-13T13:21:07.320615+0000","last_peered":"2024-09-13T13:21:07.320615+0000","last_clean":"2024-09-13T13:21:07.320615+0000","last_became_active":"2024-09-13T13:19:14.296958+0000","last_became_peered":"2024-09-13T13:19:14.296958+0000","last_unstale":"2024-09-13T13:21:07.320615+0000","last_undegraded":"2024-09-13T13:21:07.320615+0000","last_fullsized":"2024-09-13T13:21:07.320615+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:35.064188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:35.064188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:19.119240+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"500'9554","reported_seq":14888,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.323745+0000","last_change":"2024-09-13T13:20:23.134100+0000","last_active":"2024-09-13T13:21:11.323745+0000","last_peered":"2024-09-13T13:21:11.323745+0000","last_clean":"2024-09-13T13:21:11.323745+0000","last_became_active":"2024-09-13T13:20:23.124197+0000","last_became_peered":"2024-09-13T13:20:23.124197+0000","last_unstale":"2024-09-13T13:21:11.323745+0000","last_undegraded":"2024-09-13T13:21:11.323745+0000","last_fullsized":"2024-09-13T13:21:11.323745+0000","mapping_epoch":481,"log_start":"443'4521","ondisk_log_start":"443'4521","created":423,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'6911","last_scrub_stamp":"2024-09-13T13:19:39.721961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:39.721961+0000","objects_scrubbed":3475,"log_size":5033,"log_dups_size":0,"ondisk_log_size":5033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:08.621085+0000","scrub_duration":1014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53280768,"num_objects":813,"num_object_clones":0,"num_object_copies":2439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2662,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2548,"num_bytes_recovered":166985728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":192,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319841+0000","last_change":"2024-09-13T13:19:43.110394+0000","last_active":"2024-09-13T13:21:07.319841+0000","last_peered":"2024-09-13T13:21:07.319841+0000","last_clean":"2024-09-13T13:21:07.319841+0000","last_became_active":"2024-09-13T13:19:43.109856+0000","last_became_peered":"2024-09-13T13:19:43.109856+0000","last_unstale":"2024-09-13T13:21:07.319841+0000","last_undegraded":"2024-09-13T13:21:07.319841+0000","last_fullsized":"2024-09-13T13:21:07.319841+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.111449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.111449+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"500'30","reported_seq":438,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.360472+0000","last_change":"2024-09-13T13:17:02.421924+0000","last_active":"2024-09-13T13:21:12.360472+0000","last_peered":"2024-09-13T13:21:12.360472+0000","last_clean":"2024-09-13T13:21:12.360472+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:21:12.360472+0000","last_undegraded":"2024-09-13T13:21:12.360472+0000","last_fullsized":"2024-09-13T13:21:12.360472+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.421825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.421825+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:56.954562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":945,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319324+0000","last_change":"2024-09-13T13:19:42.134101+0000","last_active":"2024-09-13T13:21:07.319324+0000","last_peered":"2024-09-13T13:21:07.319324+0000","last_clean":"2024-09-13T13:21:07.319324+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:21:07.319324+0000","last_undegraded":"2024-09-13T13:21:07.319324+0000","last_fullsized":"2024-09-13T13:21:07.319324+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:42.134039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:42.134039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"467'11","reported_seq":272,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.486160+0000","last_change":"2024-09-13T13:20:48.601614+0000","last_active":"2024-09-13T13:21:07.486160+0000","last_peered":"2024-09-13T13:21:07.486160+0000","last_clean":"2024-09-13T13:21:07.486160+0000","last_became_active":"2024-09-13T13:19:43.122949+0000","last_became_peered":"2024-09-13T13:19:43.122949+0000","last_unstale":"2024-09-13T13:21:07.486160+0000","last_undegraded":"2024-09-13T13:21:07.486160+0000","last_fullsized":"2024-09-13T13:21:07.486160+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-09-13T13:20:48.601418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:48.601418+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:15.765338+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":433,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.320497+0000","last_change":"2024-09-13T13:18:42.196223+0000","last_active":"2024-09-13T13:21:07.320497+0000","last_peered":"2024-09-13T13:21:07.320497+0000","last_clean":"2024-09-13T13:21:07.320497+0000","last_became_active":"2024-09-13T13:17:25.252583+0000","last_became_peered":"2024-09-13T13:17:25.252583+0000","last_unstale":"2024-09-13T13:21:07.320497+0000","last_undegraded":"2024-09-13T13:21:07.320497+0000","last_fullsized":"2024-09-13T13:21:07.320497+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:42.196120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:42.196120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"500'9293","reported_seq":15355,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.370012+0000","last_change":"2024-09-13T13:20:14.126474+0000","last_active":"2024-09-13T13:21:11.370012+0000","last_peered":"2024-09-13T13:21:11.370012+0000","last_clean":"2024-09-13T13:21:11.370012+0000","last_became_active":"2024-09-13T13:20:14.117394+0000","last_became_peered":"2024-09-13T13:20:14.117394+0000","last_unstale":"2024-09-13T13:21:11.370012+0000","last_undegraded":"2024-09-13T13:21:11.370012+0000","last_fullsized":"2024-09-13T13:21:11.370012+0000","mapping_epoch":477,"log_start":"443'4204","ondisk_log_start":"443'4204","created":423,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'6710","last_scrub_stamp":"2024-09-13T13:19:38.419629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:38.419629+0000","objects_scrubbed":3441,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55836672,"num_objects":852,"num_object_clones":0,"num_object_copies":2556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2589,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2912,"num_bytes_recovered":190840832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"500'21","reported_seq":986,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.614255+0000","last_change":"2024-09-13T13:20:49.223989+0000","last_active":"2024-09-13T13:21:12.614255+0000","last_peered":"2024-09-13T13:21:12.614255+0000","last_clean":"2024-09-13T13:21:12.614255+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:21:12.614255+0000","last_undegraded":"2024-09-13T13:21:12.614255+0000","last_fullsized":"2024-09-13T13:21:12.614255+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:20:49.223848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:49.223848+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:18.199856+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":966,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.324361+0000","last_change":"2024-09-13T13:19:24.741190+0000","last_active":"2024-09-13T13:21:07.324361+0000","last_peered":"2024-09-13T13:21:07.324361+0000","last_clean":"2024-09-13T13:21:07.324361+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:21:07.324361+0000","last_undegraded":"2024-09-13T13:21:07.324361+0000","last_fullsized":"2024-09-13T13:21:07.324361+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:24.741107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:24.741107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":716,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.318889+0000","last_change":"2024-09-13T13:19:14.300232+0000","last_active":"2024-09-13T13:21:07.318889+0000","last_peered":"2024-09-13T13:21:07.318889+0000","last_clean":"2024-09-13T13:21:07.318889+0000","last_became_active":"2024-09-13T13:19:14.299528+0000","last_became_peered":"2024-09-13T13:19:14.299528+0000","last_unstale":"2024-09-13T13:21:07.318889+0000","last_undegraded":"2024-09-13T13:21:07.318889+0000","last_fullsized":"2024-09-13T13:21:07.318889+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.620524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.620524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":948,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319335+0000","last_change":"2024-09-13T13:19:43.113417+0000","last_active":"2024-09-13T13:21:07.319335+0000","last_peered":"2024-09-13T13:21:07.319335+0000","last_clean":"2024-09-13T13:21:07.319335+0000","last_became_active":"2024-09-13T13:19:43.112175+0000","last_became_peered":"2024-09-13T13:19:43.112175+0000","last_unstale":"2024-09-13T13:21:07.319335+0000","last_undegraded":"2024-09-13T13:21:07.319335+0000","last_fullsized":"2024-09-13T13:21:07.319335+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:30.598736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:30.598736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:06.959216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"500'9610","reported_seq":16086,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:13.650372+0000","last_change":"2024-09-13T13:20:07.198855+0000","last_active":"2024-09-13T13:21:13.650372+0000","last_peered":"2024-09-13T13:21:13.650372+0000","last_clean":"2024-09-13T13:21:13.650372+0000","last_became_active":"2024-09-13T13:20:07.188879+0000","last_became_peered":"2024-09-13T13:20:07.188879+0000","last_unstale":"2024-09-13T13:21:13.650372+0000","last_undegraded":"2024-09-13T13:21:13.650372+0000","last_fullsized":"2024-09-13T13:21:13.650372+0000","mapping_epoch":473,"log_start":"443'4518","ondisk_log_start":"443'4518","created":423,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'6823","last_scrub_stamp":"2024-09-13T13:19:37.423410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:37.423410+0000","objects_scrubbed":3564,"log_size":5092,"log_dups_size":0,"ondisk_log_size":5092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":652,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50462720,"num_objects":770,"num_object_clones":0,"num_object_copies":2310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2794,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3118,"num_bytes_recovered":204341248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"500'41","reported_seq":987,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.457027+0000","last_change":"2024-09-13T13:19:43.242980+0000","last_active":"2024-09-13T13:21:11.457027+0000","last_peered":"2024-09-13T13:21:11.457027+0000","last_clean":"2024-09-13T13:21:11.457027+0000","last_became_active":"2024-09-13T13:19:43.138916+0000","last_became_peered":"2024-09-13T13:19:43.138916+0000","last_unstale":"2024-09-13T13:21:11.457027+0000","last_undegraded":"2024-09-13T13:21:11.457027+0000","last_fullsized":"2024-09-13T13:21:11.457027+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:19:28.677161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:19:28.677161+0000","objects_scrubbed":9,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:05.624895+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":588,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.324946+0000","last_change":"2024-09-13T13:20:54.627605+0000","last_active":"2024-09-13T13:21:07.324946+0000","last_peered":"2024-09-13T13:21:07.324946+0000","last_clean":"2024-09-13T13:21:07.324946+0000","last_became_active":"2024-09-13T13:19:43.105487+0000","last_became_peered":"2024-09-13T13:19:43.105487+0000","last_unstale":"2024-09-13T13:21:07.324946+0000","last_undegraded":"2024-09-13T13:21:07.324946+0000","last_fullsized":"2024-09-13T13:21:07.324946+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:54.627528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:54.627528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:18.687600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":203,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.322563+0000","last_change":"2024-09-13T13:18:39.692048+0000","last_active":"2024-09-13T13:21:07.322563+0000","last_peered":"2024-09-13T13:21:07.322563+0000","last_clean":"2024-09-13T13:21:07.322563+0000","last_became_active":"2024-09-13T13:17:25.332964+0000","last_became_peered":"2024-09-13T13:17:25.332964+0000","last_unstale":"2024-09-13T13:21:07.322563+0000","last_undegraded":"2024-09-13T13:21:07.322563+0000","last_fullsized":"2024-09-13T13:21:07.322563+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:39.691978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:39.691978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:06.640480+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":180,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.321123+0000","last_change":"2024-09-13T13:19:43.121945+0000","last_active":"2024-09-13T13:21:07.321123+0000","last_peered":"2024-09-13T13:21:07.321123+0000","last_clean":"2024-09-13T13:21:07.321123+0000","last_became_active":"2024-09-13T13:19:43.115173+0000","last_became_peered":"2024-09-13T13:19:43.115173+0000","last_unstale":"2024-09-13T13:21:07.321123+0000","last_undegraded":"2024-09-13T13:21:07.321123+0000","last_fullsized":"2024-09-13T13:21:07.321123+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:48.717456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:48.717456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"500'9508","reported_seq":9731,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:14.932461+0000","last_change":"2024-09-13T13:19:37.887846+0000","last_active":"2024-09-13T13:21:14.932461+0000","last_peered":"2024-09-13T13:21:14.932461+0000","last_clean":"2024-09-13T13:21:14.932461+0000","last_became_active":"2024-09-13T13:19:05.189425+0000","last_became_peered":"2024-09-13T13:19:05.189425+0000","last_unstale":"2024-09-13T13:21:14.932461+0000","last_undegraded":"2024-09-13T13:21:14.932461+0000","last_fullsized":"2024-09-13T13:21:14.932461+0000","mapping_epoch":423,"log_start":"445'4950","ondisk_log_start":"445'4950","created":423,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'6719","last_scrub_stamp":"2024-09-13T13:19:37.887431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:37.887431+0000","objects_scrubbed":3488,"log_size":4558,"log_dups_size":0,"ondisk_log_size":4558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45023232,"num_objects":687,"num_object_clones":0,"num_object_copies":2061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"500'24","reported_seq":448,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.062533+0000","last_change":"2024-09-13T13:20:52.936125+0000","last_active":"2024-09-13T13:21:11.062533+0000","last_peered":"2024-09-13T13:21:11.062533+0000","last_clean":"2024-09-13T13:21:11.062533+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:21:11.062533+0000","last_undegraded":"2024-09-13T13:21:11.062533+0000","last_fullsized":"2024-09-13T13:21:11.062533+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:20:52.936042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:52.936042+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:55.203142+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":434,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.320423+0000","last_change":"2024-09-13T13:20:48.892166+0000","last_active":"2024-09-13T13:21:07.320423+0000","last_peered":"2024-09-13T13:21:07.320423+0000","last_clean":"2024-09-13T13:21:07.320423+0000","last_became_active":"2024-09-13T13:19:43.130573+0000","last_became_peered":"2024-09-13T13:19:43.130573+0000","last_unstale":"2024-09-13T13:21:07.320423+0000","last_undegraded":"2024-09-13T13:21:07.320423+0000","last_fullsized":"2024-09-13T13:21:07.320423+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:48.892105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:48.892105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:13.024586+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":172,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.314054+0000","last_change":"2024-09-13T13:19:14.297431+0000","last_active":"2024-09-13T13:21:07.314054+0000","last_peered":"2024-09-13T13:21:07.314054+0000","last_clean":"2024-09-13T13:21:07.314054+0000","last_became_active":"2024-09-13T13:19:14.297087+0000","last_became_peered":"2024-09-13T13:19:14.297087+0000","last_unstale":"2024-09-13T13:21:07.314054+0000","last_undegraded":"2024-09-13T13:21:07.314054+0000","last_fullsized":"2024-09-13T13:21:07.314054+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:52.607404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.607404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:03.002320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":934,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319019+0000","last_change":"2024-09-13T13:20:36.368297+0000","last_active":"2024-09-13T13:21:07.319019+0000","last_peered":"2024-09-13T13:21:07.319019+0000","last_clean":"2024-09-13T13:21:07.319019+0000","last_became_active":"2024-09-13T13:19:43.130698+0000","last_became_peered":"2024-09-13T13:19:43.130698+0000","last_unstale":"2024-09-13T13:21:07.319019+0000","last_undegraded":"2024-09-13T13:21:07.319019+0000","last_fullsized":"2024-09-13T13:21:07.319019+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:36.368094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:20:36.368094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:45.881027+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"500'9304","reported_seq":9528,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:13.727550+0000","last_change":"2024-09-13T13:19:38.545875+0000","last_active":"2024-09-13T13:21:13.727550+0000","last_peered":"2024-09-13T13:21:13.727550+0000","last_clean":"2024-09-13T13:21:13.727550+0000","last_became_active":"2024-09-13T13:19:05.192183+0000","last_became_peered":"2024-09-13T13:19:05.192183+0000","last_unstale":"2024-09-13T13:21:13.727550+0000","last_undegraded":"2024-09-13T13:21:13.727550+0000","last_fullsized":"2024-09-13T13:21:13.727550+0000","mapping_epoch":423,"log_start":"443'4124","ondisk_log_start":"443'4124","created":423,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'6618","last_scrub_stamp":"2024-09-13T13:19:38.545676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:38.545676+0000","objects_scrubbed":3416,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":740,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47185920,"num_objects":720,"num_object_clones":0,"num_object_copies":2160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2696,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"500'39","reported_seq":1055,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.348254+0000","last_change":"2024-09-13T13:19:43.126593+0000","last_active":"2024-09-13T13:21:12.348254+0000","last_peered":"2024-09-13T13:21:12.348254+0000","last_clean":"2024-09-13T13:21:12.348254+0000","last_became_active":"2024-09-13T13:19:43.102750+0000","last_became_peered":"2024-09-13T13:19:43.102750+0000","last_unstale":"2024-09-13T13:21:12.348254+0000","last_undegraded":"2024-09-13T13:21:12.348254+0000","last_fullsized":"2024-09-13T13:21:12.348254+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:19:27.727761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:19:27.727761+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":438,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.320378+0000","last_change":"2024-09-13T13:20:36.925279+0000","last_active":"2024-09-13T13:21:07.320378+0000","last_peered":"2024-09-13T13:21:07.320378+0000","last_clean":"2024-09-13T13:21:07.320378+0000","last_became_active":"2024-09-13T13:19:43.130688+0000","last_became_peered":"2024-09-13T13:19:43.130688+0000","last_unstale":"2024-09-13T13:21:07.320378+0000","last_undegraded":"2024-09-13T13:21:07.320378+0000","last_fullsized":"2024-09-13T13:21:07.320378+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:20:36.925209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:36.925209+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:01.494744+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"467'14","reported_seq":748,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.325001+0000","last_change":"2024-09-13T13:20:44.533905+0000","last_active":"2024-09-13T13:21:07.325001+0000","last_peered":"2024-09-13T13:21:07.325001+0000","last_clean":"2024-09-13T13:21:07.325001+0000","last_became_active":"2024-09-13T13:19:43.130138+0000","last_became_peered":"2024-09-13T13:19:43.130138+0000","last_unstale":"2024-09-13T13:21:07.325001+0000","last_undegraded":"2024-09-13T13:21:07.325001+0000","last_fullsized":"2024-09-13T13:21:07.325001+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'14","last_scrub_stamp":"2024-09-13T13:20:44.533856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:44.533856+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:47.644688+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":939,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.325041+0000","last_change":"2024-09-13T13:18:43.850216+0000","last_active":"2024-09-13T13:21:07.325041+0000","last_peered":"2024-09-13T13:21:07.325041+0000","last_clean":"2024-09-13T13:21:07.325041+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:21:07.325041+0000","last_undegraded":"2024-09-13T13:21:07.325041+0000","last_fullsized":"2024-09-13T13:21:07.325041+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:43.850163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:43.850163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.18","version":"500'9631","reported_seq":20157,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:13.306145+0000","last_change":"2024-09-13T13:19:39.707330+0000","last_active":"2024-09-13T13:21:13.306145+0000","last_peered":"2024-09-13T13:21:13.306145+0000","last_clean":"2024-09-13T13:21:13.306145+0000","last_became_active":"2024-09-13T13:19:38.087942+0000","last_became_peered":"2024-09-13T13:19:38.087942+0000","last_unstale":"2024-09-13T13:21:13.306145+0000","last_undegraded":"2024-09-13T13:21:13.306145+0000","last_fullsized":"2024-09-13T13:21:13.306145+0000","mapping_epoch":463,"log_start":"442'4043","ondisk_log_start":"442'4043","created":450,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":5,"last_scrub":"465'6879","last_scrub_stamp":"2024-09-13T13:19:39.707182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:39.707182+0000","objects_scrubbed":3533,"log_size":5588,"log_dups_size":0,"ondisk_log_size":5588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:11.618138+0000","scrub_duration":1134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50135040,"num_objects":765,"num_object_clones":0,"num_object_copies":2295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":916,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.324317+0000","last_change":"2024-09-13T13:19:22.559601+0000","last_active":"2024-09-13T13:21:07.324317+0000","last_peered":"2024-09-13T13:21:07.324317+0000","last_clean":"2024-09-13T13:21:07.324317+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:21:07.324317+0000","last_undegraded":"2024-09-13T13:21:07.324317+0000","last_fullsized":"2024-09-13T13:21:07.324317+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:22.559551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:22.559551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:02.640001+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"500'18","reported_seq":485,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.478524+0000","last_change":"2024-09-13T13:19:43.170962+0000","last_active":"2024-09-13T13:21:12.478524+0000","last_peered":"2024-09-13T13:21:12.478524+0000","last_clean":"2024-09-13T13:21:12.478524+0000","last_became_active":"2024-09-13T13:19:43.132163+0000","last_became_peered":"2024-09-13T13:19:43.132163+0000","last_unstale":"2024-09-13T13:21:12.478524+0000","last_undegraded":"2024-09-13T13:21:12.478524+0000","last_fullsized":"2024-09-13T13:21:12.478524+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:17:59.217309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.217309+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:03.663552+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":971,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.325074+0000","last_change":"2024-09-13T13:19:38.925198+0000","last_active":"2024-09-13T13:21:07.325074+0000","last_peered":"2024-09-13T13:21:07.325074+0000","last_clean":"2024-09-13T13:21:07.325074+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:21:07.325074+0000","last_undegraded":"2024-09-13T13:21:07.325074+0000","last_fullsized":"2024-09-13T13:21:07.325074+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:38.925136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:38.925136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":637,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.324104+0000","last_change":"2024-09-13T13:20:37.601177+0000","last_active":"2024-09-13T13:21:07.324104+0000","last_peered":"2024-09-13T13:21:07.324104+0000","last_clean":"2024-09-13T13:21:07.324104+0000","last_became_active":"2024-09-13T13:19:43.104791+0000","last_became_peered":"2024-09-13T13:19:43.104791+0000","last_unstale":"2024-09-13T13:21:07.324104+0000","last_undegraded":"2024-09-13T13:21:07.324104+0000","last_fullsized":"2024-09-13T13:21:07.324104+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:37.601130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:20:37.601130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:54.167807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"500'9544","reported_seq":26027,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:14.895929+0000","last_change":"2024-09-13T13:20:14.168863+0000","last_active":"2024-09-13T13:21:14.895929+0000","last_peered":"2024-09-13T13:21:14.895929+0000","last_clean":"2024-09-13T13:21:14.895929+0000","last_became_active":"2024-09-13T13:20:14.121004+0000","last_became_peered":"2024-09-13T13:20:14.121004+0000","last_unstale":"2024-09-13T13:21:14.895929+0000","last_undegraded":"2024-09-13T13:21:14.895929+0000","last_fullsized":"2024-09-13T13:21:14.895929+0000","mapping_epoch":477,"log_start":"445'4981","ondisk_log_start":"445'4981","created":450,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"465'6824","last_scrub_stamp":"2024-09-13T13:19:40.848436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:40.848436+0000","objects_scrubbed":3476,"log_size":4563,"log_dups_size":0,"ondisk_log_size":4563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48562176,"num_objects":741,"num_object_clones":0,"num_object_copies":2223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2735,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3016,"num_bytes_recovered":197591040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"500'21","reported_seq":985,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.181737+0000","last_change":"2024-09-13T13:19:43.142025+0000","last_active":"2024-09-13T13:21:11.181737+0000","last_peered":"2024-09-13T13:21:11.181737+0000","last_clean":"2024-09-13T13:21:11.181737+0000","last_became_active":"2024-09-13T13:19:43.124275+0000","last_became_peered":"2024-09-13T13:19:43.124275+0000","last_unstale":"2024-09-13T13:21:11.181737+0000","last_undegraded":"2024-09-13T13:21:11.181737+0000","last_fullsized":"2024-09-13T13:21:11.181737+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:52.518502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.518502+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:09.990996+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":426,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319278+0000","last_change":"2024-09-13T13:20:39.917442+0000","last_active":"2024-09-13T13:21:07.319278+0000","last_peered":"2024-09-13T13:21:07.319278+0000","last_clean":"2024-09-13T13:21:07.319278+0000","last_became_active":"2024-09-13T13:19:43.107386+0000","last_became_peered":"2024-09-13T13:19:43.107386+0000","last_unstale":"2024-09-13T13:21:07.319278+0000","last_undegraded":"2024-09-13T13:21:07.319278+0000","last_fullsized":"2024-09-13T13:21:07.319278+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:39.917361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:39.917361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:50.531949+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":954,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.320983+0000","last_change":"2024-09-13T13:20:51.277038+0000","last_active":"2024-09-13T13:21:07.320983+0000","last_peered":"2024-09-13T13:21:07.320983+0000","last_clean":"2024-09-13T13:21:07.320983+0000","last_became_active":"2024-09-13T13:19:43.131436+0000","last_became_peered":"2024-09-13T13:19:43.131436+0000","last_unstale":"2024-09-13T13:21:07.320983+0000","last_undegraded":"2024-09-13T13:21:07.320983+0000","last_fullsized":"2024-09-13T13:21:07.320983+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:51.276953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:51.276953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:03.121291+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":210,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.324676+0000","last_change":"2024-09-13T13:19:43.114011+0000","last_active":"2024-09-13T13:21:07.324676+0000","last_peered":"2024-09-13T13:21:07.324676+0000","last_clean":"2024-09-13T13:21:07.324676+0000","last_became_active":"2024-09-13T13:19:43.110373+0000","last_became_peered":"2024-09-13T13:19:43.110373+0000","last_unstale":"2024-09-13T13:21:07.324676+0000","last_undegraded":"2024-09-13T13:21:07.324676+0000","last_fullsized":"2024-09-13T13:21:07.324676+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:45.736080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.736080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:04.594732+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"500'21","reported_seq":1007,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.692777+0000","last_change":"2024-09-13T13:19:43.127624+0000","last_active":"2024-09-13T13:21:12.692777+0000","last_peered":"2024-09-13T13:21:12.692777+0000","last_clean":"2024-09-13T13:21:12.692777+0000","last_became_active":"2024-09-13T13:19:43.108087+0000","last_became_peered":"2024-09-13T13:19:43.108087+0000","last_unstale":"2024-09-13T13:21:12.692777+0000","last_undegraded":"2024-09-13T13:21:12.692777+0000","last_fullsized":"2024-09-13T13:21:12.692777+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:04.222760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:04.222760+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:57.607945+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":937,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.313588+0000","last_change":"2024-09-13T13:19:43.115328+0000","last_active":"2024-09-13T13:21:07.313588+0000","last_peered":"2024-09-13T13:21:07.313588+0000","last_clean":"2024-09-13T13:21:07.313588+0000","last_became_active":"2024-09-13T13:19:43.115022+0000","last_became_peered":"2024-09-13T13:19:43.115022+0000","last_unstale":"2024-09-13T13:21:07.313588+0000","last_undegraded":"2024-09-13T13:21:07.313588+0000","last_fullsized":"2024-09-13T13:21:07.313588+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.634027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.634027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:59.119121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"467'11","reported_seq":1207,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.484517+0000","last_change":"2024-09-13T13:19:43.204263+0000","last_active":"2024-09-13T13:21:07.484517+0000","last_peered":"2024-09-13T13:21:07.484517+0000","last_clean":"2024-09-13T13:21:07.484517+0000","last_became_active":"2024-09-13T13:19:43.130547+0000","last_became_peered":"2024-09-13T13:19:43.130547+0000","last_unstale":"2024-09-13T13:21:07.484517+0000","last_undegraded":"2024-09-13T13:21:07.484517+0000","last_fullsized":"2024-09-13T13:21:07.484517+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'9","last_scrub_stamp":"2024-09-13T13:17:27.165657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:27.165657+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:57.287216+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":433,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.320438+0000","last_change":"2024-09-13T13:18:41.148239+0000","last_active":"2024-09-13T13:21:07.320438+0000","last_peered":"2024-09-13T13:21:07.320438+0000","last_clean":"2024-09-13T13:21:07.320438+0000","last_became_active":"2024-09-13T13:17:25.333587+0000","last_became_peered":"2024-09-13T13:17:25.333587+0000","last_unstale":"2024-09-13T13:21:07.320438+0000","last_undegraded":"2024-09-13T13:21:07.320438+0000","last_fullsized":"2024-09-13T13:21:07.320438+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:41.147987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.147987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"500'30","reported_seq":1023,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.682503+0000","last_change":"2024-09-13T13:20:43.428468+0000","last_active":"2024-09-13T13:21:12.682503+0000","last_peered":"2024-09-13T13:21:12.682503+0000","last_clean":"2024-09-13T13:21:12.682503+0000","last_became_active":"2024-09-13T13:19:43.128700+0000","last_became_peered":"2024-09-13T13:19:43.128700+0000","last_unstale":"2024-09-13T13:21:12.682503+0000","last_undegraded":"2024-09-13T13:21:12.682503+0000","last_fullsized":"2024-09-13T13:21:12.682503+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:20:43.428398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:43.428398+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:09.431737+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":837,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.318812+0000","last_change":"2024-09-13T13:19:43.117077+0000","last_active":"2024-09-13T13:21:07.318812+0000","last_peered":"2024-09-13T13:21:07.318812+0000","last_clean":"2024-09-13T13:21:07.318812+0000","last_became_active":"2024-09-13T13:19:43.116734+0000","last_became_peered":"2024-09-13T13:19:43.116734+0000","last_unstale":"2024-09-13T13:21:07.318812+0000","last_undegraded":"2024-09-13T13:21:07.318812+0000","last_fullsized":"2024-09-13T13:21:07.318812+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:28.963152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:28.963152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:05.956876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":1206,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.489331+0000","last_change":"2024-09-13T13:19:16.725417+0000","last_active":"2024-09-13T13:21:07.489331+0000","last_peered":"2024-09-13T13:21:07.489331+0000","last_clean":"2024-09-13T13:21:07.489331+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:21:07.489331+0000","last_undegraded":"2024-09-13T13:21:07.489331+0000","last_fullsized":"2024-09-13T13:21:07.489331+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:19:16.725342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:16.725342+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":949,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.313232+0000","last_change":"2024-09-13T13:19:43.129405+0000","last_active":"2024-09-13T13:21:07.313232+0000","last_peered":"2024-09-13T13:21:07.313232+0000","last_clean":"2024-09-13T13:21:07.313232+0000","last_became_active":"2024-09-13T13:19:43.125093+0000","last_became_peered":"2024-09-13T13:19:43.125093+0000","last_unstale":"2024-09-13T13:21:07.313232+0000","last_undegraded":"2024-09-13T13:21:07.313232+0000","last_fullsized":"2024-09-13T13:21:07.313232+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:33.554870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:33.554870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:00.095054+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"500'27","reported_seq":443,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.593933+0000","last_change":"2024-09-13T13:18:45.145089+0000","last_active":"2024-09-13T13:21:12.593933+0000","last_peered":"2024-09-13T13:21:12.593933+0000","last_clean":"2024-09-13T13:21:12.593933+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:21:12.593933+0000","last_undegraded":"2024-09-13T13:21:12.593933+0000","last_fullsized":"2024-09-13T13:21:12.593933+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:45.145010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.145010+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":426,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319835+0000","last_change":"2024-09-13T13:20:53.986006+0000","last_active":"2024-09-13T13:21:07.319835+0000","last_peered":"2024-09-13T13:21:07.319835+0000","last_clean":"2024-09-13T13:21:07.319835+0000","last_became_active":"2024-09-13T13:19:43.116578+0000","last_became_peered":"2024-09-13T13:19:43.116578+0000","last_unstale":"2024-09-13T13:21:07.319835+0000","last_undegraded":"2024-09-13T13:21:07.319835+0000","last_fullsized":"2024-09-13T13:21:07.319835+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:53.985950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:53.985950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:04.500372+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":948,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.329074+0000","last_change":"2024-09-13T13:17:33.279005+0000","last_active":"2024-09-13T13:21:07.329074+0000","last_peered":"2024-09-13T13:21:07.329074+0000","last_clean":"2024-09-13T13:21:07.329074+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:21:07.329074+0000","last_undegraded":"2024-09-13T13:21:07.329074+0000","last_fullsized":"2024-09-13T13:21:07.329074+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:33.278909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:33.278909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:57.662698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":951,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319990+0000","last_change":"2024-09-13T13:20:53.246026+0000","last_active":"2024-09-13T13:21:07.319990+0000","last_peered":"2024-09-13T13:21:07.319990+0000","last_clean":"2024-09-13T13:21:07.319990+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:21:07.319990+0000","last_undegraded":"2024-09-13T13:21:07.319990+0000","last_fullsized":"2024-09-13T13:21:07.319990+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:53.245964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:20:53.245964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:14.628738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"500'21","reported_seq":947,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.747362+0000","last_change":"2024-09-13T13:17:26.125163+0000","last_active":"2024-09-13T13:21:11.747362+0000","last_peered":"2024-09-13T13:21:11.747362+0000","last_clean":"2024-09-13T13:21:11.747362+0000","last_became_active":"2024-09-13T13:17:25.331530+0000","last_became_peered":"2024-09-13T13:17:25.331530+0000","last_unstale":"2024-09-13T13:21:11.747362+0000","last_undegraded":"2024-09-13T13:21:11.747362+0000","last_fullsized":"2024-09-13T13:21:11.747362+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:26.125088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:26.125088+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:59.603582+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":938,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319867+0000","last_change":"2024-09-13T13:19:43.106069+0000","last_active":"2024-09-13T13:21:07.319867+0000","last_peered":"2024-09-13T13:21:07.319867+0000","last_clean":"2024-09-13T13:21:07.319867+0000","last_became_active":"2024-09-13T13:19:43.103538+0000","last_became_peered":"2024-09-13T13:19:43.103538+0000","last_unstale":"2024-09-13T13:21:07.319867+0000","last_undegraded":"2024-09-13T13:21:07.319867+0000","last_fullsized":"2024-09-13T13:21:07.319867+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:56.965870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:56.965870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":939,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319104+0000","last_change":"2024-09-13T13:18:03.037546+0000","last_active":"2024-09-13T13:21:07.319104+0000","last_peered":"2024-09-13T13:21:07.319104+0000","last_clean":"2024-09-13T13:21:07.319104+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:21:07.319104+0000","last_undegraded":"2024-09-13T13:21:07.319104+0000","last_fullsized":"2024-09-13T13:21:07.319104+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:03.037472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:03.037472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:06.218060+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":926,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.313963+0000","last_change":"2024-09-13T13:18:44.505493+0000","last_active":"2024-09-13T13:21:07.313963+0000","last_peered":"2024-09-13T13:21:07.313963+0000","last_clean":"2024-09-13T13:21:07.313963+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:21:07.313963+0000","last_undegraded":"2024-09-13T13:21:07.313963+0000","last_fullsized":"2024-09-13T13:21:07.313963+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:44.505427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:44.505427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"500'27","reported_seq":1000,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:12.788967+0000","last_change":"2024-09-13T13:19:43.128341+0000","last_active":"2024-09-13T13:21:12.788967+0000","last_peered":"2024-09-13T13:21:12.788967+0000","last_clean":"2024-09-13T13:21:12.788967+0000","last_became_active":"2024-09-13T13:19:43.103262+0000","last_became_peered":"2024-09-13T13:19:43.103262+0000","last_unstale":"2024-09-13T13:21:12.788967+0000","last_undegraded":"2024-09-13T13:21:12.788967+0000","last_fullsized":"2024-09-13T13:21:12.788967+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:33.848726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.848726+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":962,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.325164+0000","last_change":"2024-09-13T13:20:50.661034+0000","last_active":"2024-09-13T13:21:07.325164+0000","last_peered":"2024-09-13T13:21:07.325164+0000","last_clean":"2024-09-13T13:21:07.325164+0000","last_became_active":"2024-09-13T13:19:43.121975+0000","last_became_peered":"2024-09-13T13:19:43.121975+0000","last_unstale":"2024-09-13T13:21:07.325164+0000","last_undegraded":"2024-09-13T13:21:07.325164+0000","last_fullsized":"2024-09-13T13:21:07.325164+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:50.660981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:50.660981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:13.399315+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":634,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319679+0000","last_change":"2024-09-13T13:20:45.965033+0000","last_active":"2024-09-13T13:21:07.319679+0000","last_peered":"2024-09-13T13:21:07.319679+0000","last_clean":"2024-09-13T13:21:07.319679+0000","last_became_active":"2024-09-13T13:19:43.108262+0000","last_became_peered":"2024-09-13T13:19:43.108262+0000","last_unstale":"2024-09-13T13:21:07.319679+0000","last_undegraded":"2024-09-13T13:21:07.319679+0000","last_fullsized":"2024-09-13T13:21:07.319679+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:45.964988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:45.964988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:05.860268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":914,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.319240+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:21:07.319240+0000","last_peered":"2024-09-13T13:21:07.319240+0000","last_clean":"2024-09-13T13:21:07.319240+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:21:07.319240+0000","last_undegraded":"2024-09-13T13:21:07.319240+0000","last_fullsized":"2024-09-13T13:21:07.319240+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:18.090301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"500'12","reported_seq":990,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.084390+0000","last_change":"2024-09-13T13:18:46.033316+0000","last_active":"2024-09-13T13:21:11.084390+0000","last_peered":"2024-09-13T13:21:11.084390+0000","last_clean":"2024-09-13T13:21:11.084390+0000","last_became_active":"2024-09-13T13:16:12.073702+0000","last_became_peered":"2024-09-13T13:16:12.073702+0000","last_unstale":"2024-09-13T13:21:11.084390+0000","last_undegraded":"2024-09-13T13:21:11.084390+0000","last_fullsized":"2024-09-13T13:21:11.084390+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:18:46.033238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.033238+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":962,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.314355+0000","last_change":"2024-09-13T13:17:57.579827+0000","last_active":"2024-09-13T13:21:07.314355+0000","last_peered":"2024-09-13T13:21:07.314355+0000","last_clean":"2024-09-13T13:21:07.314355+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:21:07.314355+0000","last_undegraded":"2024-09-13T13:21:07.314355+0000","last_fullsized":"2024-09-13T13:21:07.314355+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:57.579778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:57.579778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":885,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.325049+0000","last_change":"2024-09-13T13:19:43.104851+0000","last_active":"2024-09-13T13:21:07.325049+0000","last_peered":"2024-09-13T13:21:07.325049+0000","last_clean":"2024-09-13T13:21:07.325049+0000","last_became_active":"2024-09-13T13:19:43.104438+0000","last_became_peered":"2024-09-13T13:19:43.104438+0000","last_unstale":"2024-09-13T13:21:07.325049+0000","last_undegraded":"2024-09-13T13:21:07.325049+0000","last_fullsized":"2024-09-13T13:21:07.325049+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:17.404669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:17.404669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:06.627251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":688,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.318703+0000","last_change":"2024-09-13T13:19:43.142300+0000","last_active":"2024-09-13T13:21:07.318703+0000","last_peered":"2024-09-13T13:21:07.318703+0000","last_clean":"2024-09-13T13:21:07.318703+0000","last_became_active":"2024-09-13T13:19:43.141978+0000","last_became_peered":"2024-09-13T13:19:43.141978+0000","last_unstale":"2024-09-13T13:21:07.318703+0000","last_undegraded":"2024-09-13T13:21:07.318703+0000","last_fullsized":"2024-09-13T13:21:07.318703+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:07.682648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:07.682648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"500'9827","reported_seq":10034,"reported_epoch":500,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T13:21:14.838992+0000","last_change":"2024-09-13T13:21:08.434795+0000","last_active":"2024-09-13T13:21:14.838992+0000","last_peered":"2024-09-13T13:21:14.838992+0000","last_clean":"2024-09-13T13:19:06.170841+0000","last_became_active":"2024-09-13T13:19:08.301760+0000","last_became_peered":"2024-09-13T13:19:08.301760+0000","last_unstale":"2024-09-13T13:21:14.838992+0000","last_undegraded":"2024-09-13T13:21:14.838992+0000","last_fullsized":"2024-09-13T13:21:14.838992+0000","mapping_epoch":453,"log_start":"454'6188","ondisk_log_start":"454'6188","created":450,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":3639,"log_dups_size":0,"ondisk_log_size":3639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48922624,"num_objects":747,"num_object_clones":0,"num_object_copies":2241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2241,"num_objects_unfound":0,"num_objects_dirty":747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6767,"num_write_kb":240416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[1,7,3],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3,7","objects":747}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":958,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.330138+0000","last_change":"2024-09-13T13:19:26.266256+0000","last_active":"2024-09-13T13:21:07.330138+0000","last_peered":"2024-09-13T13:21:07.330138+0000","last_clean":"2024-09-13T13:21:07.330138+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:21:07.330138+0000","last_undegraded":"2024-09-13T13:21:07.330138+0000","last_fullsized":"2024-09-13T13:21:07.330138+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:26.266186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:19:26.266186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":931,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.321208+0000","last_change":"2024-09-13T13:16:36.980668+0000","last_active":"2024-09-13T13:21:07.321208+0000","last_peered":"2024-09-13T13:21:07.321208+0000","last_clean":"2024-09-13T13:21:07.321208+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:21:07.321208+0000","last_undegraded":"2024-09-13T13:21:07.321208+0000","last_fullsized":"2024-09-13T13:21:07.321208+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:36.980594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:36.980594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:24.137197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"500'23","reported_seq":972,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:11.113730+0000","last_change":"2024-09-13T13:20:51.567559+0000","last_active":"2024-09-13T13:21:11.113730+0000","last_peered":"2024-09-13T13:21:11.113730+0000","last_clean":"2024-09-13T13:21:11.113730+0000","last_became_active":"2024-09-13T13:16:12.070277+0000","last_became_peered":"2024-09-13T13:16:12.070277+0000","last_unstale":"2024-09-13T13:21:11.113730+0000","last_undegraded":"2024-09-13T13:21:11.113730+0000","last_fullsized":"2024-09-13T13:21:11.113730+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:20:51.567506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:51.567506+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:06.487621+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":24,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":977,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T13:21:07.313482+0000","last_change":"2024-09-13T13:19:43.129850+0000","last_active":"2024-09-13T13:21:07.313482+0000","last_peered":"2024-09-13T13:21:07.313482+0000","last_clean":"2024-09-13T13:21:07.313482+0000","last_became_active":"2024-09-13T13:19:43.129130+0000","last_became_peered":"2024-09-13T13:19:43.129130+0000","last_unstale":"2024-09-13T13:21:07.313482+0000","last_undegraded":"2024-09-13T13:21:07.313482+0000","last_fullsized":"2024-09-13T13:21:07.313482+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:23.547758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:23.547758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:58.162800+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":26,"stat_sum":{"num_bytes":1612906520,"num_objects":24613,"num_object_clones":0,"num_object_copies":73839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7065,"num_objects_unfound":0,"num_objects_dirty":24613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142771,"num_write_kb":3776897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27695,"num_bytes_recovered":1814757376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5776814080,"data_stored":5776801864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119309,"ondisk_log_size":119309,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1516,"num_read_kb":1273,"num_write":971,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":6046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":955,"ondisk_log_size":955,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2233,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":516,"num_read_kb":511,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":236223201525,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141135,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1713396,"kb_used_data":945700,"kb_used_omap":4,"kb_used_meta":767675,"kb_avail":92031756,"statfs":{"total":95995035648,"available":94240518144,"internally_reserved":0,"allocated":968396800,"data_stored":968039047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":786099885},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692097,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1280836,"kb_used_data":487476,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92464316,"statfs":{"total":95995035648,"available":94683459584,"internally_reserved":0,"allocated":499175424,"data_stored":498803379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921031,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1217616,"kb_used_data":496448,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92527536,"statfs":{"total":95995035648,"available":94748196864,"internally_reserved":0,"allocated":508362752,"data_stored":507976932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":738458054},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117262,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1550872,"kb_used_data":675592,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92194280,"statfs":{"total":95995035648,"available":94406942720,"internally_reserved":0,"allocated":691806208,"data_stored":691414228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":896268743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411601,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1483444,"kb_used_data":784484,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":92261708,"statfs":{"total":95995035648,"available":94475988992,"internally_reserved":0,"allocated":803311616,"data_stored":802954326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":715717062},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575128,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1688484,"kb_used_data":881236,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92056668,"statfs":{"total":95995035648,"available":94266028032,"internally_reserved":0,"allocated":902385664,"data_stored":901979915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738655,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1516116,"kb_used_data":578244,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92229036,"statfs":{"total":95995035648,"available":94442532864,"internally_reserved":0,"allocated":592121856,"data_stored":591733051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":560594944,"data_stored":560594944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":898433024,"data_stored":898433024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":776736768,"data_stored":776732696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":660078592,"data_stored":660078592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":500305920,"data_stored":500301848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":493748224,"data_stored":493748224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":936448000,"data_stored":936443928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":950468608,"data_stored":950468608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:21:17.487 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:21:17.528 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:17.555 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:17.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:17.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:17.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:17.684 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:17.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:17.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[29471]: pgmap v1023: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.24k op/s; 7065/74436 objects misplaced (9.491%) 2024-09-13T13:21:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[29471]: from='client.17016 v1:172.21.15.52:0/15780075' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:21:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3530857766' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:21:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[29471]: osdmap e501: 8 total, 8 up, 7 in 2024-09-13T13:21:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[29471]: 6.19 scrub starts 2024-09-13T13:21:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[29471]: 6.19 scrub ok 2024-09-13T13:21:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[29471]: 6.5 scrub starts 2024-09-13T13:21:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[29471]: 6.5 scrub ok 2024-09-13T13:21:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[29471]: 4.4 scrub starts 2024-09-13T13:21:17.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[29471]: 4.4 scrub ok 2024-09-13T13:21:17.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[36682]: pgmap v1023: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.24k op/s; 7065/74436 objects misplaced (9.491%) 2024-09-13T13:21:17.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[36682]: from='client.17016 v1:172.21.15.52:0/15780075' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:21:17.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:17.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3530857766' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:21:17.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[36682]: osdmap e501: 8 total, 8 up, 7 in 2024-09-13T13:21:17.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[36682]: 6.19 scrub starts 2024-09-13T13:21:17.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[36682]: 6.19 scrub ok 2024-09-13T13:21:17.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[36682]: 6.5 scrub starts 2024-09-13T13:21:17.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[36682]: 6.5 scrub ok 2024-09-13T13:21:17.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[36682]: 4.4 scrub starts 2024-09-13T13:21:17.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:17 smithi052 ceph-mon[36682]: 4.4 scrub ok 2024-09-13T13:21:17.786 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:17.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:17 smithi167 ceph-mon[33178]: pgmap v1023: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.24k op/s; 7065/74436 objects misplaced (9.491%) 2024-09-13T13:21:17.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:17 smithi167 ceph-mon[33178]: from='client.17016 v1:172.21.15.52:0/15780075' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:21:17.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:17 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:17 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3530857766' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:21:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:17 smithi167 ceph-mon[33178]: osdmap e501: 8 total, 8 up, 7 in 2024-09-13T13:21:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:17 smithi167 ceph-mon[33178]: 6.19 scrub starts 2024-09-13T13:21:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:17 smithi167 ceph-mon[33178]: 6.19 scrub ok 2024-09-13T13:21:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:17 smithi167 ceph-mon[33178]: 6.5 scrub starts 2024-09-13T13:21:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:17 smithi167 ceph-mon[33178]: 6.5 scrub ok 2024-09-13T13:21:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:17 smithi167 ceph-mon[33178]: 4.4 scrub starts 2024-09-13T13:21:17.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:17 smithi167 ceph-mon[33178]: 4.4 scrub ok 2024-09-13T13:21:17.842 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:17.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:17.944 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:17.990 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:17.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:18.092 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:18.150 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:18.155 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:18.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:18.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:18.198 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:18.257 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:18.283 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:18.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:18.385 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:18.417 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:18.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:18.518 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:18.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:18.557 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:18.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:18.573 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:18 smithi052 ceph-mon[36682]: 5.1d scrub starts 2024-09-13T13:21:18.573 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:18 smithi052 ceph-mon[36682]: 5.1d scrub ok 2024-09-13T13:21:18.573 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:18 smithi052 ceph-mon[36682]: 3.1 scrub starts 2024-09-13T13:21:18.573 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:18 smithi052 ceph-mon[36682]: 3.1 scrub ok 2024-09-13T13:21:18.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:18 smithi052 ceph-mon[29471]: 5.1d scrub starts 2024-09-13T13:21:18.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:18 smithi052 ceph-mon[29471]: 5.1d scrub ok 2024-09-13T13:21:18.574 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:18 smithi052 ceph-mon[29471]: 3.1 scrub starts 2024-09-13T13:21:18.574 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:18 smithi052 ceph-mon[29471]: 3.1 scrub ok 2024-09-13T13:21:18.658 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:18.664 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:18 smithi167 ceph-mon[33178]: 5.1d scrub starts 2024-09-13T13:21:18.665 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:18 smithi167 ceph-mon[33178]: 5.1d scrub ok 2024-09-13T13:21:18.665 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:18 smithi167 ceph-mon[33178]: 3.1 scrub starts 2024-09-13T13:21:18.665 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:18 smithi167 ceph-mon[33178]: 3.1 scrub ok 2024-09-13T13:21:18.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:18.690 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:18.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:18.791 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:18.798 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:21:18.798 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:21:18.827 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:18.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:18.930 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:18.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:18.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:18.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:19.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:19.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:19.094 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:19.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:19.195 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:19.234 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:19.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:19.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:19.336 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:19.378 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:19.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:19.480 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:19.504 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:19.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:19.532 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T13:21:19.605 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:19.636 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:19.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:19.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:19.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:19.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:19 smithi052 ceph-mon[36682]: pgmap v1025: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 1.20k op/s; 7065/71859 objects misplaced (9.832%) 2024-09-13T13:21:19.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:19 smithi052 ceph-mon[36682]: 6.1b scrub starts 2024-09-13T13:21:19.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:19 smithi052 ceph-mon[36682]: 6.1b scrub ok 2024-09-13T13:21:19.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:19 smithi052 ceph-mon[36682]: 1.a scrub starts 2024-09-13T13:21:19.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:19 smithi052 ceph-mon[36682]: 1.a scrub ok 2024-09-13T13:21:19.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:19 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:21:19.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:19 smithi052 ceph-mon[29471]: pgmap v1025: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 1.20k op/s; 7065/71859 objects misplaced (9.832%) 2024-09-13T13:21:19.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:19 smithi052 ceph-mon[29471]: 6.1b scrub starts 2024-09-13T13:21:19.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:19 smithi052 ceph-mon[29471]: 6.1b scrub ok 2024-09-13T13:21:19.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:19 smithi052 ceph-mon[29471]: 1.a scrub starts 2024-09-13T13:21:19.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:19 smithi052 ceph-mon[29471]: 1.a scrub ok 2024-09-13T13:21:19.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:19 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:21:19.783 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:19.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:19.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:19 smithi167 ceph-mon[33178]: pgmap v1025: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 1.20k op/s; 7065/71859 objects misplaced (9.832%) 2024-09-13T13:21:19.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:19 smithi167 ceph-mon[33178]: 6.1b scrub starts 2024-09-13T13:21:19.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:19 smithi167 ceph-mon[33178]: 6.1b scrub ok 2024-09-13T13:21:19.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:19 smithi167 ceph-mon[33178]: 1.a scrub starts 2024-09-13T13:21:19.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:19 smithi167 ceph-mon[33178]: 1.a scrub ok 2024-09-13T13:21:19.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:19 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:21:19.856 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:21:19.885 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:19.920 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:19.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:19.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:20.021 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:20.063 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:20.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:20.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:20.076 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:21:20.165 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:20.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:20.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:20.301 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:20.338 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:20.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:20.440 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:20.458 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-09-13T13:21:20.458 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:21:20.468 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:20.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:20.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:20.569 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:20 smithi167 ceph-mon[33178]: 6.10 scrub starts 2024-09-13T13:21:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:20 smithi167 ceph-mon[33178]: 6.10 scrub ok 2024-09-13T13:21:20.591 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:20.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:20.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:20.693 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:20.715 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:20.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:20.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:20 smithi052 ceph-mon[36682]: 6.10 scrub starts 2024-09-13T13:21:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:20 smithi052 ceph-mon[36682]: 6.10 scrub ok 2024-09-13T13:21:20.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:20 smithi052 ceph-mon[29471]: 6.10 scrub starts 2024-09-13T13:21:20.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:20 smithi052 ceph-mon[29471]: 6.10 scrub ok 2024-09-13T13:21:20.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:20.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:20.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:20.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:20.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:20.975 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:20.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:20.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:21.077 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:21.104 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:21.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:21.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:21.235 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:21:21.235 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:21:21.238 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T13:21:21.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:21.252 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:21.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:21.274 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:21:21.275 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:21:21.309 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:21:21.310 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:21:21.349 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:21:21.349 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:21:21.353 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:21.382 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4: {} 2024-09-13T13:21:21.383 INFO:teuthology.orchestra.run.smithi052.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:21:21.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:21.406 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:21.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:21.417 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:21:21.417 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:21:21.453 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:21:21.453 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:21:21.487 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:21:21.488 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:21:21.507 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:21.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:21.550 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:21.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:21.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:21 smithi167 ceph-mon[33178]: 11.4 scrub starts 2024-09-13T13:21:21.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:21 smithi167 ceph-mon[33178]: 11.4 scrub ok 2024-09-13T13:21:21.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:21 smithi167 ceph-mon[33178]: pgmap v1026: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.19k op/s; 5403/57270 objects misplaced (9.434%) 2024-09-13T13:21:21.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:21 smithi167 ceph-mon[33178]: 6.13 scrub starts 2024-09-13T13:21:21.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:21 smithi167 ceph-mon[33178]: 6.13 scrub ok 2024-09-13T13:21:21.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:21 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/492887348' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:21.597 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:21:21.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:21.676 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:21.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:21.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:21.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:21 smithi052 ceph-mon[29471]: 11.4 scrub starts 2024-09-13T13:21:21.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:21 smithi052 ceph-mon[29471]: 11.4 scrub ok 2024-09-13T13:21:21.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:21 smithi052 ceph-mon[29471]: pgmap v1026: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.19k op/s; 5403/57270 objects misplaced (9.434%) 2024-09-13T13:21:21.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:21 smithi052 ceph-mon[29471]: 6.13 scrub starts 2024-09-13T13:21:21.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:21 smithi052 ceph-mon[29471]: 6.13 scrub ok 2024-09-13T13:21:21.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:21 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/492887348' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:21.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:21 smithi052 ceph-mon[36682]: 11.4 scrub starts 2024-09-13T13:21:21.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:21 smithi052 ceph-mon[36682]: 11.4 scrub ok 2024-09-13T13:21:21.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:21 smithi052 ceph-mon[36682]: pgmap v1026: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.19k op/s; 5403/57270 objects misplaced (9.434%) 2024-09-13T13:21:21.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:21 smithi052 ceph-mon[36682]: 6.13 scrub starts 2024-09-13T13:21:21.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:21 smithi052 ceph-mon[36682]: 6.13 scrub ok 2024-09-13T13:21:21.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:21 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/492887348' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:21.778 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4 2024-09-13T13:21:21.804 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:21.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.4' 2024-09-13T13:21:21.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:21.932 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:21.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:21.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:22.034 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:22.071 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:21:22.073 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:21:22.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:22.075 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:22.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:22.176 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:22.219 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:22.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:22.290 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:21:22.290 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:21:22.290 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-09-13T13:21:22.290 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:21:22.321 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:22.357 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:22.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:22.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:22.461 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:22.489 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:22.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:22.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:22.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:22 smithi167 ceph-mon[33178]: 5.19 scrub starts 2024-09-13T13:21:22.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:22 smithi167 ceph-mon[33178]: 5.19 scrub ok 2024-09-13T13:21:22.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:22.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:22.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:22.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:22.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:22 smithi052 ceph-mon[36682]: 5.19 scrub starts 2024-09-13T13:21:22.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:22 smithi052 ceph-mon[36682]: 5.19 scrub ok 2024-09-13T13:21:22.677 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:22 smithi052 ceph-mon[29471]: 5.19 scrub starts 2024-09-13T13:21:22.677 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:22 smithi052 ceph-mon[29471]: 5.19 scrub ok 2024-09-13T13:21:22.718 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:22.757 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:22.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:22.859 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:22.896 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:22.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:22.998 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:23.034 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:23.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:23.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:23.136 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:23.177 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:23.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:23.207 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":1026,"stamp":"2024-09-13T13:21:20.068182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1238506035,"num_objects":19090,"num_object_clones":0,"num_object_copies":57270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5403,"num_objects_unfound":0,"num_objects_dirty":19090,"num_whiteouts":0,"num_read":2079,"num_read_kb":1831,"num_write":149501,"num_write_kb":3776942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28016,"num_bytes_recovered":1815094946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118218,"ondisk_log_size":118218,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":513,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":9359252,"kb_used_data":3739620,"kb_used_omap":13,"kb_used_meta":5619378,"kb_avail":646856812,"statfs":{"total":671965249536,"available":662381375488,"internally_reserved":0,"allocated":3829370880,"data_stored":3826679614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14001,"internal_metadata":5754243407},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-736100352,"num_objects":-11232,"num_object_clones":0,"num_object_copies":-33696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3282,"num_objects_unfound":0,"num_objects_dirty":-11232,"num_whiteouts":0,"num_read":421,"num_read_kb":421,"num_write":11506,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006863"},"pg_stats":[{"pgid":"11.17","version":"501'9450","reported_seq":16124,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.426493+0000","last_change":"2024-09-13T13:20:55.540544+0000","last_active":"2024-09-13T13:21:16.426493+0000","last_peered":"2024-09-13T13:21:16.426493+0000","last_clean":"2024-09-13T13:21:16.426493+0000","last_became_active":"2024-09-13T13:19:53.269607+0000","last_became_peered":"2024-09-13T13:19:53.269607+0000","last_unstale":"2024-09-13T13:21:16.426493+0000","last_undegraded":"2024-09-13T13:21:16.426493+0000","last_fullsized":"2024-09-13T13:21:16.426493+0000","mapping_epoch":470,"log_start":"444'4549","ondisk_log_start":"444'4549","created":450,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4901,"log_dups_size":0,"ondisk_log_size":4901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T13:21:25.516783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42893312,"num_objects":655,"num_object_clones":0,"num_object_copies":1965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6555,"num_write_kb":230688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3223,"num_bytes_recovered":211222528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":955,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.386108+0000","last_change":"2024-09-13T13:18:32.440533+0000","last_active":"2024-09-13T13:21:16.386108+0000","last_peered":"2024-09-13T13:21:16.386108+0000","last_clean":"2024-09-13T13:21:16.386108+0000","last_became_active":"2024-09-13T13:17:25.335900+0000","last_became_peered":"2024-09-13T13:17:25.335900+0000","last_unstale":"2024-09-13T13:21:16.386108+0000","last_undegraded":"2024-09-13T13:21:16.386108+0000","last_fullsized":"2024-09-13T13:21:16.386108+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:32.440448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:32.440448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"500'21","reported_seq":440,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.369634+0000","last_change":"2024-09-13T13:18:08.548626+0000","last_active":"2024-09-13T13:21:16.369634+0000","last_peered":"2024-09-13T13:21:16.369634+0000","last_clean":"2024-09-13T13:21:16.369634+0000","last_became_active":"2024-09-13T13:12:52.997300+0000","last_became_peered":"2024-09-13T13:12:52.997300+0000","last_unstale":"2024-09-13T13:21:16.369634+0000","last_undegraded":"2024-09-13T13:21:16.369634+0000","last_fullsized":"2024-09-13T13:21:16.369634+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15","last_scrub_stamp":"2024-09-13T13:18:08.548556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:08.548556+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":957,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.366008+0000","last_change":"2024-09-13T13:19:43.119757+0000","last_active":"2024-09-13T13:21:16.366008+0000","last_peered":"2024-09-13T13:21:16.366008+0000","last_clean":"2024-09-13T13:21:16.366008+0000","last_became_active":"2024-09-13T13:19:43.119056+0000","last_became_peered":"2024-09-13T13:19:43.119056+0000","last_unstale":"2024-09-13T13:21:16.366008+0000","last_undegraded":"2024-09-13T13:21:16.366008+0000","last_fullsized":"2024-09-13T13:21:16.366008+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:12.060429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:12.060429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":429,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370212+0000","last_change":"2024-09-13T13:18:49.194573+0000","last_active":"2024-09-13T13:21:16.370212+0000","last_peered":"2024-09-13T13:21:16.370212+0000","last_clean":"2024-09-13T13:21:16.370212+0000","last_became_active":"2024-09-13T13:12:52.996167+0000","last_became_peered":"2024-09-13T13:12:52.996167+0000","last_unstale":"2024-09-13T13:21:16.370212+0000","last_undegraded":"2024-09-13T13:21:16.370212+0000","last_fullsized":"2024-09-13T13:21:16.370212+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:49.194510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:49.194510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"501'9613","reported_seq":18468,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.577059+0000","last_change":"2024-09-13T13:20:47.197380+0000","last_active":"2024-09-13T13:21:18.577059+0000","last_peered":"2024-09-13T13:21:18.577059+0000","last_clean":"2024-09-13T13:21:18.577059+0000","last_became_active":"2024-09-13T13:20:15.116368+0000","last_became_peered":"2024-09-13T13:20:15.116368+0000","last_unstale":"2024-09-13T13:21:18.577059+0000","last_undegraded":"2024-09-13T13:21:18.577059+0000","last_fullsized":"2024-09-13T13:21:18.577059+0000","mapping_epoch":478,"log_start":"444'4602","ondisk_log_start":"444'4602","created":450,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":5,"last_scrub":"490'8518","last_scrub_stamp":"2024-09-13T13:20:47.197308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:20:47.197308+0000","objects_scrubbed":1663,"log_size":5011,"log_dups_size":0,"ondisk_log_size":5011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:13.351498+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36896768,"num_objects":563,"num_object_clones":0,"num_object_copies":1689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1100,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":939,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370170+0000","last_change":"2024-09-13T13:19:43.132324+0000","last_active":"2024-09-13T13:21:16.370170+0000","last_peered":"2024-09-13T13:21:16.370170+0000","last_clean":"2024-09-13T13:21:16.370170+0000","last_became_active":"2024-09-13T13:19:43.131732+0000","last_became_peered":"2024-09-13T13:19:43.131732+0000","last_unstale":"2024-09-13T13:21:16.370170+0000","last_undegraded":"2024-09-13T13:21:16.370170+0000","last_fullsized":"2024-09-13T13:21:16.370170+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:47.852697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:47.852697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:17.137902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":984,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.367393+0000","last_change":"2024-09-13T13:20:45.377028+0000","last_active":"2024-09-13T13:21:16.367393+0000","last_peered":"2024-09-13T13:21:16.367393+0000","last_clean":"2024-09-13T13:21:16.367393+0000","last_became_active":"2024-09-13T13:17:25.243371+0000","last_became_peered":"2024-09-13T13:17:25.243371+0000","last_unstale":"2024-09-13T13:21:16.367393+0000","last_undegraded":"2024-09-13T13:21:16.367393+0000","last_fullsized":"2024-09-13T13:21:16.367393+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:45.376945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:45.376945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:58.142506+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"500'39","reported_seq":1076,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.365915+0000","last_change":"2024-09-13T13:20:52.246795+0000","last_active":"2024-09-13T13:21:16.365915+0000","last_peered":"2024-09-13T13:21:16.365915+0000","last_clean":"2024-09-13T13:21:16.365915+0000","last_became_active":"2024-09-13T13:17:25.245131+0000","last_became_peered":"2024-09-13T13:17:25.245131+0000","last_unstale":"2024-09-13T13:21:16.365915+0000","last_undegraded":"2024-09-13T13:21:16.365915+0000","last_fullsized":"2024-09-13T13:21:16.365915+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:20:52.246624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:52.246624+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:02.856958+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":413,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.369605+0000","last_change":"2024-09-13T13:20:31.015443+0000","last_active":"2024-09-13T13:21:16.369605+0000","last_peered":"2024-09-13T13:21:16.369605+0000","last_clean":"2024-09-13T13:21:16.369605+0000","last_became_active":"2024-09-13T13:12:52.996641+0000","last_became_peered":"2024-09-13T13:12:52.996641+0000","last_unstale":"2024-09-13T13:21:16.369605+0000","last_undegraded":"2024-09-13T13:21:16.369605+0000","last_fullsized":"2024-09-13T13:21:16.369605+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:31.015382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:31.015382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:35.021147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"501'9819","reported_seq":10029,"reported_epoch":501,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T13:21:18.730976+0000","last_change":"2024-09-13T13:21:14.324168+0000","last_active":"2024-09-13T13:21:18.730976+0000","last_peered":"2024-09-13T13:21:18.730976+0000","last_clean":"2024-09-13T13:19:06.181780+0000","last_became_active":"2024-09-13T13:19:14.296332+0000","last_became_peered":"2024-09-13T13:19:14.296332+0000","last_unstale":"2024-09-13T13:21:18.730976+0000","last_undegraded":"2024-09-13T13:21:18.730976+0000","last_fullsized":"2024-09-13T13:21:18.730976+0000","mapping_epoch":456,"log_start":"445'5025","ondisk_log_start":"445'5025","created":450,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4794,"log_dups_size":0,"ondisk_log_size":4794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44040192,"num_objects":672,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2016,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":240320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[5,6,7],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":672}],"blocked_by":[],"up_primary":1,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":1004,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360714+0000","last_change":"2024-09-13T13:20:43.694870+0000","last_active":"2024-09-13T13:21:16.360714+0000","last_peered":"2024-09-13T13:21:16.360714+0000","last_clean":"2024-09-13T13:21:16.360714+0000","last_became_active":"2024-09-13T13:12:52.995833+0000","last_became_peered":"2024-09-13T13:12:52.995833+0000","last_unstale":"2024-09-13T13:21:16.360714+0000","last_undegraded":"2024-09-13T13:21:16.360714+0000","last_fullsized":"2024-09-13T13:21:16.360714+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:20:43.694821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:43.694821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:12.759248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"500'21","reported_seq":432,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370263+0000","last_change":"2024-09-13T13:17:06.406253+0000","last_active":"2024-09-13T13:21:16.370263+0000","last_peered":"2024-09-13T13:21:16.370263+0000","last_clean":"2024-09-13T13:21:16.370263+0000","last_became_active":"2024-09-13T13:12:52.999080+0000","last_became_peered":"2024-09-13T13:12:52.999080+0000","last_unstale":"2024-09-13T13:21:16.370263+0000","last_undegraded":"2024-09-13T13:21:16.370263+0000","last_fullsized":"2024-09-13T13:21:16.370263+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:06.406179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:06.406179+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:55.937461+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":956,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.359982+0000","last_change":"2024-09-13T13:19:43.103191+0000","last_active":"2024-09-13T13:21:16.359982+0000","last_peered":"2024-09-13T13:21:16.359982+0000","last_clean":"2024-09-13T13:21:16.359982+0000","last_became_active":"2024-09-13T13:19:43.102694+0000","last_became_peered":"2024-09-13T13:19:43.102694+0000","last_unstale":"2024-09-13T13:21:16.359982+0000","last_undegraded":"2024-09-13T13:21:16.359982+0000","last_fullsized":"2024-09-13T13:21:16.359982+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:15.686363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:15.686363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":951,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.371568+0000","last_change":"2024-09-13T13:19:43.116648+0000","last_active":"2024-09-13T13:21:16.371568+0000","last_peered":"2024-09-13T13:21:16.371568+0000","last_clean":"2024-09-13T13:21:16.371568+0000","last_became_active":"2024-09-13T13:19:43.116357+0000","last_became_peered":"2024-09-13T13:19:43.116357+0000","last_unstale":"2024-09-13T13:21:16.371568+0000","last_undegraded":"2024-09-13T13:21:16.371568+0000","last_fullsized":"2024-09-13T13:21:16.371568+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:32.559884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:32.559884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:02.033055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"501'9467","reported_seq":17730,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.447503+0000","last_change":"2024-09-13T13:20:31.432554+0000","last_active":"2024-09-13T13:21:16.447503+0000","last_peered":"2024-09-13T13:21:16.447503+0000","last_clean":"2024-09-13T13:21:16.447503+0000","last_became_active":"2024-09-13T13:20:30.358134+0000","last_became_peered":"2024-09-13T13:20:30.358134+0000","last_unstale":"2024-09-13T13:21:16.447503+0000","last_undegraded":"2024-09-13T13:21:16.447503+0000","last_fullsized":"2024-09-13T13:21:16.447503+0000","mapping_epoch":483,"log_start":"444'4561","ondisk_log_start":"444'4561","created":450,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"485'7948","last_scrub_stamp":"2024-09-13T13:20:31.432498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:20:31.432498+0000","objects_scrubbed":2164,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:51.136811+0000","scrub_duration":750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41811968,"num_objects":638,"num_object_clones":0,"num_object_copies":1914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1526,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":979,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360227+0000","last_change":"2024-09-13T13:19:31.753911+0000","last_active":"2024-09-13T13:21:16.360227+0000","last_peered":"2024-09-13T13:21:16.360227+0000","last_clean":"2024-09-13T13:21:16.360227+0000","last_became_active":"2024-09-13T13:12:52.951318+0000","last_became_peered":"2024-09-13T13:12:52.951318+0000","last_unstale":"2024-09-13T13:21:16.360227+0000","last_undegraded":"2024-09-13T13:21:16.360227+0000","last_fullsized":"2024-09-13T13:21:16.360227+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:31.753864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:31.753864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"500'9","reported_seq":430,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370692+0000","last_change":"2024-09-13T13:20:47.925194+0000","last_active":"2024-09-13T13:21:16.370692+0000","last_peered":"2024-09-13T13:21:16.370692+0000","last_clean":"2024-09-13T13:21:16.370692+0000","last_became_active":"2024-09-13T13:12:52.997537+0000","last_became_peered":"2024-09-13T13:12:52.997537+0000","last_unstale":"2024-09-13T13:21:16.370692+0000","last_undegraded":"2024-09-13T13:21:16.370692+0000","last_fullsized":"2024-09-13T13:21:16.370692+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'7","last_scrub_stamp":"2024-09-13T13:20:47.925142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:47.925142+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:48.646990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":631,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.369901+0000","last_change":"2024-09-13T13:20:44.928714+0000","last_active":"2024-09-13T13:21:16.369901+0000","last_peered":"2024-09-13T13:21:16.369901+0000","last_clean":"2024-09-13T13:21:16.369901+0000","last_became_active":"2024-09-13T13:19:43.130534+0000","last_became_peered":"2024-09-13T13:19:43.130534+0000","last_unstale":"2024-09-13T13:21:16.369901+0000","last_undegraded":"2024-09-13T13:21:16.369901+0000","last_fullsized":"2024-09-13T13:21:16.369901+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:44.928667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:44.928667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:56.039898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":929,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.610520+0000","last_change":"2024-09-13T13:21:16.610520+0000","last_active":"2024-09-13T13:21:16.610520+0000","last_peered":"2024-09-13T13:21:16.610520+0000","last_clean":"2024-09-13T13:21:16.610520+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:21:16.610520+0000","last_undegraded":"2024-09-13T13:21:16.610520+0000","last_fullsized":"2024-09-13T13:21:16.610520+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:21:16.610456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:21:16.610456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:25.683776+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"501'9955","reported_seq":10152,"reported_epoch":501,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T13:21:19.774583+0000","last_change":"2024-09-13T13:21:08.387402+0000","last_active":"2024-09-13T13:21:19.774583+0000","last_peered":"2024-09-13T13:21:19.774583+0000","last_clean":"2024-09-13T13:19:06.161926+0000","last_became_active":"2024-09-13T13:19:08.330177+0000","last_became_peered":"2024-09-13T13:19:08.330177+0000","last_unstale":"2024-09-13T13:21:19.774583+0000","last_undegraded":"2024-09-13T13:21:19.774583+0000","last_fullsized":"2024-09-13T13:21:19.774583+0000","mapping_epoch":453,"log_start":"457'6293","ondisk_log_start":"457'6293","created":450,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":3662,"log_dups_size":0,"ondisk_log_size":3662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36569088,"num_objects":558,"num_object_clones":0,"num_object_copies":1674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1674,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6946,"num_write_kb":240128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[1,6,7],"avail_no_missing":["1","6","7"],"object_location_counts":[{"shards":"1,6,7","objects":558}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":418,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.371036+0000","last_change":"2024-09-13T13:17:07.444148+0000","last_active":"2024-09-13T13:21:16.371036+0000","last_peered":"2024-09-13T13:21:16.371036+0000","last_clean":"2024-09-13T13:21:16.371036+0000","last_became_active":"2024-09-13T13:12:52.945958+0000","last_became_peered":"2024-09-13T13:12:52.945958+0000","last_unstale":"2024-09-13T13:21:16.371036+0000","last_undegraded":"2024-09-13T13:21:16.371036+0000","last_fullsized":"2024-09-13T13:21:16.371036+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:17:07.443951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:07.443951+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:02.940965+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":977,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.374271+0000","last_change":"2024-09-13T13:19:43.118595+0000","last_active":"2024-09-13T13:21:16.374271+0000","last_peered":"2024-09-13T13:21:16.374271+0000","last_clean":"2024-09-13T13:21:16.374271+0000","last_became_active":"2024-09-13T13:19:43.118289+0000","last_became_peered":"2024-09-13T13:19:43.118289+0000","last_unstale":"2024-09-13T13:21:16.374271+0000","last_undegraded":"2024-09-13T13:21:16.374271+0000","last_fullsized":"2024-09-13T13:21:16.374271+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:45.470790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.470790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"500'39","reported_seq":239,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360437+0000","last_change":"2024-09-13T13:18:37.714594+0000","last_active":"2024-09-13T13:21:16.360437+0000","last_peered":"2024-09-13T13:21:16.360437+0000","last_clean":"2024-09-13T13:21:16.360437+0000","last_became_active":"2024-09-13T13:17:25.245658+0000","last_became_peered":"2024-09-13T13:17:25.245658+0000","last_unstale":"2024-09-13T13:21:16.360437+0000","last_undegraded":"2024-09-13T13:21:16.360437+0000","last_fullsized":"2024-09-13T13:21:16.360437+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:18:37.714503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:37.714503+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:57.581192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":980,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.885837+0000","last_change":"2024-09-13T13:21:16.885837+0000","last_active":"2024-09-13T13:21:16.885837+0000","last_peered":"2024-09-13T13:21:16.885837+0000","last_clean":"2024-09-13T13:21:16.885837+0000","last_became_active":"2024-09-13T13:12:52.999972+0000","last_became_peered":"2024-09-13T13:12:52.999972+0000","last_unstale":"2024-09-13T13:21:16.885837+0000","last_undegraded":"2024-09-13T13:21:16.885837+0000","last_fullsized":"2024-09-13T13:21:16.885837+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:21:16.885766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:21:16.885766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:46.106661+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"501'9731","reported_seq":20556,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.369438+0000","last_change":"2024-09-13T13:20:44.972218+0000","last_active":"2024-09-13T13:21:18.369438+0000","last_peered":"2024-09-13T13:21:18.369438+0000","last_clean":"2024-09-13T13:21:18.369438+0000","last_became_active":"2024-09-13T13:20:44.158986+0000","last_became_peered":"2024-09-13T13:20:44.158986+0000","last_unstale":"2024-09-13T13:21:18.369438+0000","last_undegraded":"2024-09-13T13:21:18.369438+0000","last_fullsized":"2024-09-13T13:21:18.369438+0000","mapping_epoch":488,"log_start":"443'4344","ondisk_log_start":"443'4344","created":450,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"489'8553","last_scrub_stamp":"2024-09-13T13:20:44.972097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:20:44.972097+0000","objects_scrubbed":1779,"log_size":5387,"log_dups_size":0,"ondisk_log_size":5387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:09.658625+0000","scrub_duration":351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39321600,"num_objects":600,"num_object_clones":0,"num_object_copies":1800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1179,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":941,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.367679+0000","last_change":"2024-09-13T13:18:41.949104+0000","last_active":"2024-09-13T13:21:16.367679+0000","last_peered":"2024-09-13T13:21:16.367679+0000","last_clean":"2024-09-13T13:21:16.367679+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:21:16.367679+0000","last_undegraded":"2024-09-13T13:21:16.367679+0000","last_fullsized":"2024-09-13T13:21:16.367679+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:41.949029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.949029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":443,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370125+0000","last_change":"2024-09-13T13:18:46.181826+0000","last_active":"2024-09-13T13:21:16.370125+0000","last_peered":"2024-09-13T13:21:16.370125+0000","last_clean":"2024-09-13T13:21:16.370125+0000","last_became_active":"2024-09-13T13:17:25.244839+0000","last_became_peered":"2024-09-13T13:17:25.244839+0000","last_unstale":"2024-09-13T13:21:16.370125+0000","last_undegraded":"2024-09-13T13:21:16.370125+0000","last_fullsized":"2024-09-13T13:21:16.370125+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:46.181741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.181741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"500'24","reported_seq":1035,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.368394+0000","last_change":"2024-09-13T13:18:35.281750+0000","last_active":"2024-09-13T13:21:16.368394+0000","last_peered":"2024-09-13T13:21:16.368394+0000","last_clean":"2024-09-13T13:21:16.368394+0000","last_became_active":"2024-09-13T13:18:35.239702+0000","last_became_peered":"2024-09-13T13:18:35.239702+0000","last_unstale":"2024-09-13T13:21:16.368394+0000","last_undegraded":"2024-09-13T13:21:16.368394+0000","last_fullsized":"2024-09-13T13:21:16.368394+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'18","last_scrub_stamp":"2024-09-13T13:17:56.001763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:56.001763+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:01.267454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":437,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370209+0000","last_change":"2024-09-13T13:18:32.362014+0000","last_active":"2024-09-13T13:21:16.370209+0000","last_peered":"2024-09-13T13:21:16.370209+0000","last_clean":"2024-09-13T13:21:16.370209+0000","last_became_active":"2024-09-13T13:17:25.245239+0000","last_became_peered":"2024-09-13T13:17:25.245239+0000","last_unstale":"2024-09-13T13:21:16.370209+0000","last_undegraded":"2024-09-13T13:21:16.370209+0000","last_fullsized":"2024-09-13T13:21:16.370209+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:32.361971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:32.361971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.11","version":"501'9581","reported_seq":14566,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.381536+0000","last_change":"2024-09-13T13:20:34.418954+0000","last_active":"2024-09-13T13:21:18.381536+0000","last_peered":"2024-09-13T13:21:18.381536+0000","last_clean":"2024-09-13T13:21:18.381536+0000","last_became_active":"2024-09-13T13:20:32.683660+0000","last_became_peered":"2024-09-13T13:20:32.683660+0000","last_unstale":"2024-09-13T13:21:18.381536+0000","last_undegraded":"2024-09-13T13:21:18.381536+0000","last_fullsized":"2024-09-13T13:21:18.381536+0000","mapping_epoch":485,"log_start":"443'4134","ondisk_log_start":"443'4134","created":450,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":5,"last_scrub":"486'8086","last_scrub_stamp":"2024-09-13T13:20:34.418689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:20:34.418689+0000","objects_scrubbed":2089,"log_size":5447,"log_dups_size":0,"ondisk_log_size":5447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:49.790205+0000","scrub_duration":766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38207488,"num_objects":583,"num_object_clones":0,"num_object_copies":1749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1506,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":489,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.359937+0000","last_change":"2024-09-13T13:19:43.128188+0000","last_active":"2024-09-13T13:21:16.359937+0000","last_peered":"2024-09-13T13:21:16.359937+0000","last_clean":"2024-09-13T13:21:16.359937+0000","last_became_active":"2024-09-13T13:19:43.116203+0000","last_became_peered":"2024-09-13T13:19:43.116203+0000","last_unstale":"2024-09-13T13:21:16.359937+0000","last_undegraded":"2024-09-13T13:21:16.359937+0000","last_fullsized":"2024-09-13T13:21:16.359937+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:19:35.778718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:19:35.778718+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":973,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.366160+0000","last_change":"2024-09-13T13:19:25.256499+0000","last_active":"2024-09-13T13:21:16.366160+0000","last_peered":"2024-09-13T13:21:16.366160+0000","last_clean":"2024-09-13T13:21:16.366160+0000","last_became_active":"2024-09-13T13:12:52.997446+0000","last_became_peered":"2024-09-13T13:12:52.997446+0000","last_unstale":"2024-09-13T13:21:16.366160+0000","last_undegraded":"2024-09-13T13:21:16.366160+0000","last_fullsized":"2024-09-13T13:21:16.366160+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:19:25.256420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:25.256420+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"500'30","reported_seq":1026,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.375949+0000","last_change":"2024-09-13T13:19:43.167125+0000","last_active":"2024-09-13T13:21:16.375949+0000","last_peered":"2024-09-13T13:21:16.375949+0000","last_clean":"2024-09-13T13:21:16.375949+0000","last_became_active":"2024-09-13T13:19:43.128174+0000","last_became_peered":"2024-09-13T13:19:43.128174+0000","last_unstale":"2024-09-13T13:21:16.375949+0000","last_undegraded":"2024-09-13T13:21:16.375949+0000","last_fullsized":"2024-09-13T13:21:16.375949+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:18:41.472008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.472008+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"467'13","reported_seq":1184,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.362866+0000","last_change":"2024-09-13T13:20:35.599418+0000","last_active":"2024-09-13T13:21:16.362866+0000","last_peered":"2024-09-13T13:21:16.362866+0000","last_clean":"2024-09-13T13:21:16.362866+0000","last_became_active":"2024-09-13T13:19:43.110544+0000","last_became_peered":"2024-09-13T13:19:43.110544+0000","last_unstale":"2024-09-13T13:21:16.362866+0000","last_undegraded":"2024-09-13T13:21:16.362866+0000","last_fullsized":"2024-09-13T13:21:16.362866+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'13","last_scrub_stamp":"2024-09-13T13:20:35.599368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:35.599368+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:47.345844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"501'10135","reported_seq":16767,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:19.808561+0000","last_change":"2024-09-13T13:20:10.241834+0000","last_active":"2024-09-13T13:21:19.808561+0000","last_peered":"2024-09-13T13:21:19.808561+0000","last_clean":"2024-09-13T13:21:19.808561+0000","last_became_active":"2024-09-13T13:20:10.229726+0000","last_became_peered":"2024-09-13T13:20:10.229726+0000","last_unstale":"2024-09-13T13:21:19.808561+0000","last_undegraded":"2024-09-13T13:21:19.808561+0000","last_fullsized":"2024-09-13T13:21:19.808561+0000","mapping_epoch":475,"log_start":"457'6546","ondisk_log_start":"457'6546","created":423,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'7038","last_scrub_stamp":"2024-09-13T13:19:38.280658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:38.280658+0000","objects_scrubbed":3671,"log_size":3589,"log_dups_size":0,"ondisk_log_size":3589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1040,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37093376,"num_objects":566,"num_object_clones":0,"num_object_copies":1698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3105,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3194,"num_bytes_recovered":209321984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":195,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.572280+0000","last_change":"2024-09-13T13:21:18.572280+0000","last_active":"2024-09-13T13:21:18.572280+0000","last_peered":"2024-09-13T13:21:18.572280+0000","last_clean":"2024-09-13T13:21:18.572280+0000","last_became_active":"2024-09-13T13:17:49.052966+0000","last_became_peered":"2024-09-13T13:17:49.052966+0000","last_unstale":"2024-09-13T13:21:18.572280+0000","last_undegraded":"2024-09-13T13:21:18.572280+0000","last_fullsized":"2024-09-13T13:21:18.572280+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:21:18.572231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:21:18.572231+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:22.069660+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"500'39","reported_seq":1069,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370070+0000","last_change":"2024-09-13T13:19:34.288799+0000","last_active":"2024-09-13T13:21:16.370070+0000","last_peered":"2024-09-13T13:21:16.370070+0000","last_clean":"2024-09-13T13:21:16.370070+0000","last_became_active":"2024-09-13T13:12:52.997799+0000","last_became_peered":"2024-09-13T13:12:52.997799+0000","last_unstale":"2024-09-13T13:21:16.370070+0000","last_undegraded":"2024-09-13T13:21:16.370070+0000","last_fullsized":"2024-09-13T13:21:16.370070+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'27","last_scrub_stamp":"2024-09-13T13:19:34.288614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:19:34.288614+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":981,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.375856+0000","last_change":"2024-09-13T13:20:34.332345+0000","last_active":"2024-09-13T13:21:16.375856+0000","last_peered":"2024-09-13T13:21:16.375856+0000","last_clean":"2024-09-13T13:21:16.375856+0000","last_became_active":"2024-09-13T13:19:43.137877+0000","last_became_peered":"2024-09-13T13:19:43.137877+0000","last_unstale":"2024-09-13T13:21:16.375856+0000","last_undegraded":"2024-09-13T13:21:16.375856+0000","last_fullsized":"2024-09-13T13:21:16.375856+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:34.332278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:34.332278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:58.646130+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"306'10","reported_seq":527,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.371137+0000","last_change":"2024-09-13T13:20:51.899838+0000","last_active":"2024-09-13T13:21:16.371137+0000","last_peered":"2024-09-13T13:21:16.371137+0000","last_clean":"2024-09-13T13:21:16.371137+0000","last_became_active":"2024-09-13T13:12:53.004096+0000","last_became_peered":"2024-09-13T13:12:53.004096+0000","last_unstale":"2024-09-13T13:21:16.371137+0000","last_undegraded":"2024-09-13T13:21:16.371137+0000","last_fullsized":"2024-09-13T13:21:16.371137+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:20:51.899778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:51.899778+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:10.314133+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":945,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360118+0000","last_change":"2024-09-13T13:20:47.624758+0000","last_active":"2024-09-13T13:21:16.360118+0000","last_peered":"2024-09-13T13:21:16.360118+0000","last_clean":"2024-09-13T13:21:16.360118+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:21:16.360118+0000","last_undegraded":"2024-09-13T13:21:16.360118+0000","last_fullsized":"2024-09-13T13:21:16.360118+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:47.624694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:20:47.624694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:10.975469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"501'13676","reported_seq":22473,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:19.820113+0000","last_change":"2024-09-13T13:20:48.223870+0000","last_active":"2024-09-13T13:21:19.820113+0000","last_peered":"2024-09-13T13:21:19.820113+0000","last_clean":"2024-09-13T13:21:19.820113+0000","last_became_active":"2024-09-13T13:20:48.144620+0000","last_became_peered":"2024-09-13T13:20:48.144620+0000","last_unstale":"2024-09-13T13:21:19.820113+0000","last_undegraded":"2024-09-13T13:21:19.820113+0000","last_fullsized":"2024-09-13T13:21:19.820113+0000","mapping_epoch":491,"log_start":"482'10062","ondisk_log_start":"482'10062","created":423,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'7156","last_scrub_stamp":"2024-09-13T13:19:27.505480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:27.505480+0000","objects_scrubbed":7086,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69468160,"num_objects":1060,"num_object_clones":0,"num_object_copies":3180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13676,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4261,"num_bytes_recovered":279117824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":187,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370536+0000","last_change":"2024-09-13T13:18:51.159546+0000","last_active":"2024-09-13T13:21:16.370536+0000","last_peered":"2024-09-13T13:21:16.370536+0000","last_clean":"2024-09-13T13:21:16.370536+0000","last_became_active":"2024-09-13T13:17:49.050655+0000","last_became_peered":"2024-09-13T13:17:49.050655+0000","last_unstale":"2024-09-13T13:21:16.370536+0000","last_undegraded":"2024-09-13T13:21:16.370536+0000","last_fullsized":"2024-09-13T13:21:16.370536+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.159496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.159496+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"500'42","reported_seq":1024,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.375704+0000","last_change":"2024-09-13T13:19:43.214168+0000","last_active":"2024-09-13T13:21:16.375704+0000","last_peered":"2024-09-13T13:21:16.375704+0000","last_clean":"2024-09-13T13:21:16.375704+0000","last_became_active":"2024-09-13T13:19:43.133611+0000","last_became_peered":"2024-09-13T13:19:43.133611+0000","last_unstale":"2024-09-13T13:21:16.375704+0000","last_undegraded":"2024-09-13T13:21:16.375704+0000","last_fullsized":"2024-09-13T13:21:16.375704+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-13T13:18:36.921964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:36.921964+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":951,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.367937+0000","last_change":"2024-09-13T13:19:43.112484+0000","last_active":"2024-09-13T13:21:16.367937+0000","last_peered":"2024-09-13T13:21:16.367937+0000","last_clean":"2024-09-13T13:21:16.367937+0000","last_became_active":"2024-09-13T13:19:43.102458+0000","last_became_peered":"2024-09-13T13:19:43.102458+0000","last_unstale":"2024-09-13T13:21:16.367937+0000","last_undegraded":"2024-09-13T13:21:16.367937+0000","last_fullsized":"2024-09-13T13:21:16.367937+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:16:26.098365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:26.098365+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:22.128239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":429,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.369672+0000","last_change":"2024-09-13T13:19:43.104198+0000","last_active":"2024-09-13T13:21:16.369672+0000","last_peered":"2024-09-13T13:21:16.369672+0000","last_clean":"2024-09-13T13:21:16.369672+0000","last_became_active":"2024-09-13T13:19:43.103878+0000","last_became_peered":"2024-09-13T13:19:43.103878+0000","last_unstale":"2024-09-13T13:21:16.369672+0000","last_undegraded":"2024-09-13T13:21:16.369672+0000","last_fullsized":"2024-09-13T13:21:16.369672+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:58.389763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:58.389763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:09.938779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":444,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370349+0000","last_change":"2024-09-13T13:17:59.373240+0000","last_active":"2024-09-13T13:21:16.370349+0000","last_peered":"2024-09-13T13:21:16.370349+0000","last_clean":"2024-09-13T13:21:16.370349+0000","last_became_active":"2024-09-13T13:17:25.244171+0000","last_became_peered":"2024-09-13T13:17:25.244171+0000","last_unstale":"2024-09-13T13:21:16.370349+0000","last_undegraded":"2024-09-13T13:21:16.370349+0000","last_fullsized":"2024-09-13T13:21:16.370349+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:59.373194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.373194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"501'13551","reported_seq":24580,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.352186+0000","last_change":"2024-09-13T13:20:32.417405+0000","last_active":"2024-09-13T13:21:18.352186+0000","last_peered":"2024-09-13T13:21:18.352186+0000","last_clean":"2024-09-13T13:21:18.352186+0000","last_became_active":"2024-09-13T13:20:32.403062+0000","last_became_peered":"2024-09-13T13:20:32.403062+0000","last_unstale":"2024-09-13T13:21:18.352186+0000","last_undegraded":"2024-09-13T13:21:18.352186+0000","last_fullsized":"2024-09-13T13:21:18.352186+0000","mapping_epoch":485,"log_start":"478'9494","ondisk_log_start":"478'9494","created":423,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":4057,"log_dups_size":0,"ondisk_log_size":4057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77266944,"num_objects":1179,"num_object_clones":0,"num_object_copies":3537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13551,"num_write_kb":471360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5421,"num_bytes_recovered":355205120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":206,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.366031+0000","last_change":"2024-09-13T13:19:43.120655+0000","last_active":"2024-09-13T13:21:16.366031+0000","last_peered":"2024-09-13T13:21:16.366031+0000","last_clean":"2024-09-13T13:21:16.366031+0000","last_became_active":"2024-09-13T13:19:43.106076+0000","last_became_peered":"2024-09-13T13:19:43.106076+0000","last_unstale":"2024-09-13T13:21:16.366031+0000","last_undegraded":"2024-09-13T13:21:16.366031+0000","last_fullsized":"2024-09-13T13:21:16.366031+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:19:23.064923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:19:23.064923+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"500'42","reported_seq":459,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.369870+0000","last_change":"2024-09-13T13:20:41.949473+0000","last_active":"2024-09-13T13:21:16.369870+0000","last_peered":"2024-09-13T13:21:16.369870+0000","last_clean":"2024-09-13T13:21:16.369870+0000","last_became_active":"2024-09-13T13:12:52.944297+0000","last_became_peered":"2024-09-13T13:12:52.944297+0000","last_unstale":"2024-09-13T13:21:16.369870+0000","last_undegraded":"2024-09-13T13:21:16.369870+0000","last_fullsized":"2024-09-13T13:21:16.369870+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'28","last_scrub_stamp":"2024-09-13T13:20:41.949422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:41.949422+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:56.729205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":956,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.376264+0000","last_change":"2024-09-13T13:20:54.179763+0000","last_active":"2024-09-13T13:21:16.376264+0000","last_peered":"2024-09-13T13:21:16.376264+0000","last_clean":"2024-09-13T13:21:16.376264+0000","last_became_active":"2024-09-13T13:12:53.008663+0000","last_became_peered":"2024-09-13T13:12:53.008663+0000","last_unstale":"2024-09-13T13:21:16.376264+0000","last_undegraded":"2024-09-13T13:21:16.376264+0000","last_fullsized":"2024-09-13T13:21:16.376264+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:20:54.179075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:54.179075+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:16.137252+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"306'10","reported_seq":1218,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:17.487229+0000","last_change":"2024-09-13T13:18:48.032272+0000","last_active":"2024-09-13T13:21:17.487229+0000","last_peered":"2024-09-13T13:21:17.487229+0000","last_clean":"2024-09-13T13:21:17.487229+0000","last_became_active":"2024-09-13T13:12:52.953399+0000","last_became_peered":"2024-09-13T13:12:52.953399+0000","last_unstale":"2024-09-13T13:21:17.487229+0000","last_undegraded":"2024-09-13T13:21:17.487229+0000","last_fullsized":"2024-09-13T13:21:17.487229+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:18:48.032199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:48.032199+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":601,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.362616+0000","last_change":"2024-09-13T13:20:39.526912+0000","last_active":"2024-09-13T13:21:16.362616+0000","last_peered":"2024-09-13T13:21:16.362616+0000","last_clean":"2024-09-13T13:21:16.362616+0000","last_became_active":"2024-09-13T13:19:43.130374+0000","last_became_peered":"2024-09-13T13:19:43.130374+0000","last_unstale":"2024-09-13T13:21:16.362616+0000","last_undegraded":"2024-09-13T13:21:16.362616+0000","last_fullsized":"2024-09-13T13:21:16.362616+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:39.526852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:20:39.526852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:04.402168+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"501'13157","reported_seq":13403,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.381776+0000","last_change":"2024-09-13T13:19:43.288579+0000","last_active":"2024-09-13T13:21:18.381776+0000","last_peered":"2024-09-13T13:21:18.381776+0000","last_clean":"2024-09-13T13:21:18.381776+0000","last_became_active":"2024-09-13T13:17:55.131230+0000","last_became_peered":"2024-09-13T13:17:55.131230+0000","last_unstale":"2024-09-13T13:21:18.381776+0000","last_undegraded":"2024-09-13T13:21:18.381776+0000","last_fullsized":"2024-09-13T13:21:18.381776+0000","mapping_epoch":423,"log_start":"472'8541","ondisk_log_start":"472'8541","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'7729","last_scrub_stamp":"2024-09-13T13:19:43.288518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:43.288518+0000","objects_scrubbed":6606,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:08.647064+0000","scrub_duration":765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77070360,"num_objects":1177,"num_object_clones":0,"num_object_copies":3531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1177,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13157,"num_write_kb":458625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":278,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.362398+0000","last_change":"2024-09-13T13:20:48.542448+0000","last_active":"2024-09-13T13:21:16.362398+0000","last_peered":"2024-09-13T13:21:16.362398+0000","last_clean":"2024-09-13T13:21:16.362398+0000","last_became_active":"2024-09-13T13:19:43.106463+0000","last_became_peered":"2024-09-13T13:19:43.106463+0000","last_unstale":"2024-09-13T13:21:16.362398+0000","last_undegraded":"2024-09-13T13:21:16.362398+0000","last_fullsized":"2024-09-13T13:21:16.362398+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:20:48.542384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:20:48.542384+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:52.910447+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"500'30","reported_seq":1022,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.375507+0000","last_change":"2024-09-13T13:20:48.549680+0000","last_active":"2024-09-13T13:21:16.375507+0000","last_peered":"2024-09-13T13:21:16.375507+0000","last_clean":"2024-09-13T13:21:16.375507+0000","last_became_active":"2024-09-13T13:12:52.999133+0000","last_became_peered":"2024-09-13T13:12:52.999133+0000","last_unstale":"2024-09-13T13:21:16.375507+0000","last_undegraded":"2024-09-13T13:21:16.375507+0000","last_fullsized":"2024-09-13T13:21:16.375507+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:20:48.549598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:48.549598+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:59.549477+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":988,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360882+0000","last_change":"2024-09-13T13:19:13.702236+0000","last_active":"2024-09-13T13:21:16.360882+0000","last_peered":"2024-09-13T13:21:16.360882+0000","last_clean":"2024-09-13T13:21:16.360882+0000","last_became_active":"2024-09-13T13:17:25.331166+0000","last_became_peered":"2024-09-13T13:17:25.331166+0000","last_unstale":"2024-09-13T13:21:16.360882+0000","last_undegraded":"2024-09-13T13:21:16.360882+0000","last_fullsized":"2024-09-13T13:21:16.360882+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:13.702185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:13.702185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":1199,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.373839+0000","last_change":"2024-09-13T13:18:40.464456+0000","last_active":"2024-09-13T13:21:16.373839+0000","last_peered":"2024-09-13T13:21:16.373839+0000","last_clean":"2024-09-13T13:21:16.373839+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:21:16.373839+0000","last_undegraded":"2024-09-13T13:21:16.373839+0000","last_fullsized":"2024-09-13T13:21:16.373839+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:18:40.464398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:40.464398+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":976,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.371696+0000","last_change":"2024-09-13T13:18:36.466657+0000","last_active":"2024-09-13T13:21:16.371696+0000","last_peered":"2024-09-13T13:21:16.371696+0000","last_clean":"2024-09-13T13:21:16.371696+0000","last_became_active":"2024-09-13T13:17:25.252712+0000","last_became_peered":"2024-09-13T13:17:25.252712+0000","last_unstale":"2024-09-13T13:21:16.371696+0000","last_undegraded":"2024-09-13T13:21:16.371696+0000","last_fullsized":"2024-09-13T13:21:16.371696+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:36.466575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:36.466575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"501'13614","reported_seq":13866,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:19.859876+0000","last_change":"2024-09-13T13:19:36.549311+0000","last_active":"2024-09-13T13:21:19.859876+0000","last_peered":"2024-09-13T13:21:19.859876+0000","last_clean":"2024-09-13T13:21:19.859876+0000","last_became_active":"2024-09-13T13:17:55.140134+0000","last_became_peered":"2024-09-13T13:17:55.140134+0000","last_unstale":"2024-09-13T13:21:19.859876+0000","last_undegraded":"2024-09-13T13:21:19.859876+0000","last_fullsized":"2024-09-13T13:21:19.859876+0000","mapping_epoch":423,"log_start":"481'9979","ondisk_log_start":"481'9979","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'7653","last_scrub_stamp":"2024-09-13T13:19:36.549087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:36.549087+0000","objects_scrubbed":7193,"log_size":3635,"log_dups_size":0,"ondisk_log_size":3635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3276,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74317824,"num_objects":1134,"num_object_clones":0,"num_object_copies":3402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13614,"num_write_kb":471936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":271,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.362185+0000","last_change":"2024-09-13T13:20:50.537234+0000","last_active":"2024-09-13T13:21:16.362185+0000","last_peered":"2024-09-13T13:21:16.362185+0000","last_clean":"2024-09-13T13:21:16.362185+0000","last_became_active":"2024-09-13T13:17:49.055888+0000","last_became_peered":"2024-09-13T13:17:49.055888+0000","last_unstale":"2024-09-13T13:21:16.362185+0000","last_undegraded":"2024-09-13T13:21:16.362185+0000","last_fullsized":"2024-09-13T13:21:16.362185+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:20:50.537158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:20:50.537158+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:54.570017+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"500'39","reported_seq":1052,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.374001+0000","last_change":"2024-09-13T13:19:43.203331+0000","last_active":"2024-09-13T13:21:16.374001+0000","last_peered":"2024-09-13T13:21:16.374001+0000","last_clean":"2024-09-13T13:21:16.374001+0000","last_became_active":"2024-09-13T13:19:43.119487+0000","last_became_peered":"2024-09-13T13:19:43.119487+0000","last_unstale":"2024-09-13T13:21:16.374001+0000","last_undegraded":"2024-09-13T13:21:16.374001+0000","last_fullsized":"2024-09-13T13:21:16.374001+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'29","last_scrub_stamp":"2024-09-13T13:18:33.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.796644+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":965,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.369231+0000","last_change":"2024-09-13T13:19:31.224359+0000","last_active":"2024-09-13T13:21:16.369231+0000","last_peered":"2024-09-13T13:21:16.369231+0000","last_clean":"2024-09-13T13:21:16.369231+0000","last_became_active":"2024-09-13T13:17:25.332589+0000","last_became_peered":"2024-09-13T13:17:25.332589+0000","last_unstale":"2024-09-13T13:21:16.369231+0000","last_undegraded":"2024-09-13T13:21:16.369231+0000","last_fullsized":"2024-09-13T13:21:16.369231+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:31.224272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:31.224272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":927,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.361916+0000","last_change":"2024-09-13T13:19:20.502538+0000","last_active":"2024-09-13T13:21:16.361916+0000","last_peered":"2024-09-13T13:21:16.361916+0000","last_clean":"2024-09-13T13:21:16.361916+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:21:16.361916+0000","last_undegraded":"2024-09-13T13:21:16.361916+0000","last_fullsized":"2024-09-13T13:21:16.361916+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:20.502415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:20.502415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:00.660401+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":972,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.365895+0000","last_change":"2024-09-13T13:19:23.209075+0000","last_active":"2024-09-13T13:21:16.365895+0000","last_peered":"2024-09-13T13:21:16.365895+0000","last_clean":"2024-09-13T13:21:16.365895+0000","last_became_active":"2024-09-13T13:17:25.244517+0000","last_became_peered":"2024-09-13T13:17:25.244517+0000","last_unstale":"2024-09-13T13:21:16.365895+0000","last_undegraded":"2024-09-13T13:21:16.365895+0000","last_fullsized":"2024-09-13T13:21:16.365895+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:23.208994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:19:23.208994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"501'13007","reported_seq":13178,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.131768+0000","last_change":"2024-09-13T13:17:55.146287+0000","last_active":"2024-09-13T13:21:18.131768+0000","last_peered":"2024-09-13T13:21:18.131768+0000","last_clean":"2024-09-13T13:21:18.131768+0000","last_became_active":"2024-09-13T13:17:55.145616+0000","last_became_peered":"2024-09-13T13:17:55.145616+0000","last_unstale":"2024-09-13T13:21:18.131768+0000","last_undegraded":"2024-09-13T13:21:18.131768+0000","last_fullsized":"2024-09-13T13:21:18.131768+0000","mapping_epoch":423,"log_start":"476'9038","ondisk_log_start":"476'9038","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":3969,"log_dups_size":0,"ondisk_log_size":3969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:07.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78970880,"num_objects":1205,"num_object_clones":0,"num_object_copies":3615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13007,"num_write_kb":454784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":187,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.359638+0000","last_change":"2024-09-13T13:18:50.856493+0000","last_active":"2024-09-13T13:21:16.359638+0000","last_peered":"2024-09-13T13:21:16.359638+0000","last_clean":"2024-09-13T13:21:16.359638+0000","last_became_active":"2024-09-13T13:17:49.054449+0000","last_became_peered":"2024-09-13T13:17:49.054449+0000","last_unstale":"2024-09-13T13:21:16.359638+0000","last_undegraded":"2024-09-13T13:21:16.359638+0000","last_fullsized":"2024-09-13T13:21:16.359638+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:50.856437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:50.856437+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"500'55","reported_seq":212,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:17.133643+0000","last_change":"2024-09-13T13:21:17.133567+0000","last_active":"2024-09-13T13:21:17.133643+0000","last_peered":"2024-09-13T13:21:17.133643+0000","last_clean":"2024-09-13T13:21:17.133643+0000","last_became_active":"2024-09-13T13:18:35.244311+0000","last_became_peered":"2024-09-13T13:18:35.244311+0000","last_unstale":"2024-09-13T13:21:17.133643+0000","last_undegraded":"2024-09-13T13:21:17.133643+0000","last_fullsized":"2024-09-13T13:21:17.133643+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'55","last_scrub_stamp":"2024-09-13T13:21:17.133484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:21:17.133484+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:20.942317+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":430,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370794+0000","last_change":"2024-09-13T13:17:52.422920+0000","last_active":"2024-09-13T13:21:16.370794+0000","last_peered":"2024-09-13T13:21:16.370794+0000","last_clean":"2024-09-13T13:21:16.370794+0000","last_became_active":"2024-09-13T13:12:53.001726+0000","last_became_peered":"2024-09-13T13:12:53.001726+0000","last_unstale":"2024-09-13T13:21:16.370794+0000","last_undegraded":"2024-09-13T13:21:16.370794+0000","last_fullsized":"2024-09-13T13:21:16.370794+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:52.422872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.422872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":961,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360162+0000","last_change":"2024-09-13T13:18:07.306441+0000","last_active":"2024-09-13T13:21:16.360162+0000","last_peered":"2024-09-13T13:21:16.360162+0000","last_clean":"2024-09-13T13:21:16.360162+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:21:16.360162+0000","last_undegraded":"2024-09-13T13:21:16.360162+0000","last_fullsized":"2024-09-13T13:21:16.360162+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:18:07.306366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:07.306366+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:06.626914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":963,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.374895+0000","last_change":"2024-09-13T13:20:50.196625+0000","last_active":"2024-09-13T13:21:16.374895+0000","last_peered":"2024-09-13T13:21:16.374895+0000","last_clean":"2024-09-13T13:21:16.374895+0000","last_became_active":"2024-09-13T13:12:53.003482+0000","last_became_peered":"2024-09-13T13:12:53.003482+0000","last_unstale":"2024-09-13T13:21:16.374895+0000","last_undegraded":"2024-09-13T13:21:16.374895+0000","last_fullsized":"2024-09-13T13:21:16.374895+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:50.196539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:50.196539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:56.679929+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":967,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.367052+0000","last_change":"2024-09-13T13:20:48.336180+0000","last_active":"2024-09-13T13:21:16.367052+0000","last_peered":"2024-09-13T13:21:16.367052+0000","last_clean":"2024-09-13T13:21:16.367052+0000","last_became_active":"2024-09-13T13:19:43.105507+0000","last_became_peered":"2024-09-13T13:19:43.105507+0000","last_unstale":"2024-09-13T13:21:16.367052+0000","last_undegraded":"2024-09-13T13:21:16.367052+0000","last_fullsized":"2024-09-13T13:21:16.367052+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:48.336110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:20:48.336110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:53.933105+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"501'9723","reported_seq":9950,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.127144+0000","last_change":"2024-09-13T13:19:38.017114+0000","last_active":"2024-09-13T13:21:18.127144+0000","last_peered":"2024-09-13T13:21:18.127144+0000","last_clean":"2024-09-13T13:21:18.127144+0000","last_became_active":"2024-09-13T13:19:05.196340+0000","last_became_peered":"2024-09-13T13:19:05.196340+0000","last_unstale":"2024-09-13T13:21:18.127144+0000","last_undegraded":"2024-09-13T13:21:18.127144+0000","last_fullsized":"2024-09-13T13:21:18.127144+0000","mapping_epoch":423,"log_start":"448'5667","ondisk_log_start":"448'5667","created":423,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'6830","last_scrub_stamp":"2024-09-13T13:19:38.016951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:38.016951+0000","objects_scrubbed":3491,"log_size":4056,"log_dups_size":0,"ondisk_log_size":4056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38993920,"num_objects":595,"num_object_clones":0,"num_object_copies":1785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":556,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.376572+0000","last_change":"2024-09-13T13:19:43.165270+0000","last_active":"2024-09-13T13:21:16.376572+0000","last_peered":"2024-09-13T13:21:16.376572+0000","last_clean":"2024-09-13T13:21:16.376572+0000","last_became_active":"2024-09-13T13:19:43.143230+0000","last_became_peered":"2024-09-13T13:19:43.143230+0000","last_unstale":"2024-09-13T13:21:16.376572+0000","last_undegraded":"2024-09-13T13:21:16.376572+0000","last_fullsized":"2024-09-13T13:21:16.376572+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.529089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.529089+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"500'21","reported_seq":986,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370163+0000","last_change":"2024-09-13T13:19:22.172479+0000","last_active":"2024-09-13T13:21:16.370163+0000","last_peered":"2024-09-13T13:21:16.370163+0000","last_clean":"2024-09-13T13:21:16.370163+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:21:16.370163+0000","last_undegraded":"2024-09-13T13:21:16.370163+0000","last_fullsized":"2024-09-13T13:21:16.370163+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'17","last_scrub_stamp":"2024-09-13T13:19:22.172183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:19:22.172183+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":985,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.366345+0000","last_change":"2024-09-13T13:18:33.892926+0000","last_active":"2024-09-13T13:21:16.366345+0000","last_peered":"2024-09-13T13:21:16.366345+0000","last_clean":"2024-09-13T13:21:16.366345+0000","last_became_active":"2024-09-13T13:17:25.246400+0000","last_became_peered":"2024-09-13T13:17:25.246400+0000","last_unstale":"2024-09-13T13:21:16.366345+0000","last_undegraded":"2024-09-13T13:21:16.366345+0000","last_fullsized":"2024-09-13T13:21:16.366345+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:33.892597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.892597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:04.671259+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":209,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.357705+0000","last_change":"2024-09-13T13:18:44.719355+0000","last_active":"2024-09-13T13:21:16.357705+0000","last_peered":"2024-09-13T13:21:16.357705+0000","last_clean":"2024-09-13T13:21:16.357705+0000","last_became_active":"2024-09-13T13:17:25.252664+0000","last_became_peered":"2024-09-13T13:17:25.252664+0000","last_unstale":"2024-09-13T13:21:16.357705+0000","last_undegraded":"2024-09-13T13:21:16.357705+0000","last_fullsized":"2024-09-13T13:21:16.357705+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:18:44.719269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:18:44.719269+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:07.690140+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":220,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.357871+0000","last_change":"2024-09-13T13:20:52.590805+0000","last_active":"2024-09-13T13:21:16.357871+0000","last_peered":"2024-09-13T13:21:16.357871+0000","last_clean":"2024-09-13T13:21:16.357871+0000","last_became_active":"2024-09-13T13:19:43.121063+0000","last_became_peered":"2024-09-13T13:19:43.121063+0000","last_unstale":"2024-09-13T13:21:16.357871+0000","last_undegraded":"2024-09-13T13:21:16.357871+0000","last_fullsized":"2024-09-13T13:21:16.357871+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:52.590733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:52.590733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:55.206604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":571,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.362627+0000","last_change":"2024-09-13T13:19:43.111710+0000","last_active":"2024-09-13T13:21:16.362627+0000","last_peered":"2024-09-13T13:21:16.362627+0000","last_clean":"2024-09-13T13:21:16.362627+0000","last_became_active":"2024-09-13T13:19:43.110460+0000","last_became_peered":"2024-09-13T13:19:43.110460+0000","last_unstale":"2024-09-13T13:21:16.362627+0000","last_undegraded":"2024-09-13T13:21:16.362627+0000","last_fullsized":"2024-09-13T13:21:16.362627+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:26.607881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:19:26.607881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"501'9664","reported_seq":9891,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.118338+0000","last_change":"2024-09-13T13:19:39.176073+0000","last_active":"2024-09-13T13:21:18.118338+0000","last_peered":"2024-09-13T13:21:18.118338+0000","last_clean":"2024-09-13T13:21:18.118338+0000","last_became_active":"2024-09-13T13:19:05.190236+0000","last_became_peered":"2024-09-13T13:19:05.190236+0000","last_unstale":"2024-09-13T13:21:18.118338+0000","last_undegraded":"2024-09-13T13:21:18.118338+0000","last_fullsized":"2024-09-13T13:21:18.118338+0000","mapping_epoch":423,"log_start":"448'5638","ondisk_log_start":"448'5638","created":423,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'6785","last_scrub_stamp":"2024-09-13T13:19:39.175880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:39.175880+0000","objects_scrubbed":3498,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40173568,"num_objects":613,"num_object_clones":0,"num_object_copies":1839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2885,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":187,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370484+0000","last_change":"2024-09-13T13:18:51.048420+0000","last_active":"2024-09-13T13:21:16.370484+0000","last_peered":"2024-09-13T13:21:16.370484+0000","last_clean":"2024-09-13T13:21:16.370484+0000","last_became_active":"2024-09-13T13:17:49.049531+0000","last_became_peered":"2024-09-13T13:17:49.049531+0000","last_unstale":"2024-09-13T13:21:16.370484+0000","last_undegraded":"2024-09-13T13:21:16.370484+0000","last_fullsized":"2024-09-13T13:21:16.370484+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.048347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.048347+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:09.781187+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"500'36","reported_seq":1041,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.376058+0000","last_change":"2024-09-13T13:18:04.569761+0000","last_active":"2024-09-13T13:21:16.376058+0000","last_peered":"2024-09-13T13:21:16.376058+0000","last_clean":"2024-09-13T13:21:16.376058+0000","last_became_active":"2024-09-13T13:17:25.253120+0000","last_became_peered":"2024-09-13T13:17:25.253120+0000","last_unstale":"2024-09-13T13:21:16.376058+0000","last_undegraded":"2024-09-13T13:21:16.376058+0000","last_fullsized":"2024-09-13T13:21:16.376058+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'24","last_scrub_stamp":"2024-09-13T13:18:04.569670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:04.569670+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:12.008022+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":971,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.115370+0000","last_change":"2024-09-13T13:21:18.115370+0000","last_active":"2024-09-13T13:21:18.115370+0000","last_peered":"2024-09-13T13:21:18.115370+0000","last_clean":"2024-09-13T13:21:18.115370+0000","last_became_active":"2024-09-13T13:12:53.004652+0000","last_became_peered":"2024-09-13T13:12:53.004652+0000","last_unstale":"2024-09-13T13:21:18.115370+0000","last_undegraded":"2024-09-13T13:21:18.115370+0000","last_fullsized":"2024-09-13T13:21:18.115370+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:21:18.115176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:21:18.115176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:18.798981+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1015,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.792373+0000","last_change":"2024-09-13T13:19:43.169544+0000","last_active":"2024-09-13T13:21:18.792373+0000","last_peered":"2024-09-13T13:21:18.792373+0000","last_clean":"2024-09-13T13:21:18.792373+0000","last_became_active":"2024-09-13T13:19:43.137630+0000","last_became_peered":"2024-09-13T13:19:43.137630+0000","last_unstale":"2024-09-13T13:21:18.792373+0000","last_undegraded":"2024-09-13T13:21:18.792373+0000","last_fullsized":"2024-09-13T13:21:18.792373+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:17:25.325275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:25.325275+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:55.618401+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":521,"num_read_kb":516,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":934,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360859+0000","last_change":"2024-09-13T13:17:55.293423+0000","last_active":"2024-09-13T13:21:16.360859+0000","last_peered":"2024-09-13T13:21:16.360859+0000","last_clean":"2024-09-13T13:21:16.360859+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:21:16.360859+0000","last_undegraded":"2024-09-13T13:21:16.360859+0000","last_fullsized":"2024-09-13T13:21:16.360859+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:55.293375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:55.293375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:09.643979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":939,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.368714+0000","last_change":"2024-09-13T13:18:46.991251+0000","last_active":"2024-09-13T13:21:16.368714+0000","last_peered":"2024-09-13T13:21:16.368714+0000","last_clean":"2024-09-13T13:21:16.368714+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:21:16.368714+0000","last_undegraded":"2024-09-13T13:21:16.368714+0000","last_fullsized":"2024-09-13T13:21:16.368714+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:46.991076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.991076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:10.771653+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"501'13484","reported_seq":13655,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.150904+0000","last_change":"2024-09-13T13:17:55.142415+0000","last_active":"2024-09-13T13:21:18.150904+0000","last_peered":"2024-09-13T13:21:18.150904+0000","last_clean":"2024-09-13T13:21:18.150904+0000","last_became_active":"2024-09-13T13:17:55.142112+0000","last_became_peered":"2024-09-13T13:17:55.142112+0000","last_unstale":"2024-09-13T13:21:18.150904+0000","last_undegraded":"2024-09-13T13:21:18.150904+0000","last_fullsized":"2024-09-13T13:21:18.150904+0000","mapping_epoch":423,"log_start":"479'9488","ondisk_log_start":"479'9488","created":423,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":3996,"log_dups_size":0,"ondisk_log_size":3996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:02.224353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73793536,"num_objects":1126,"num_object_clones":0,"num_object_copies":3378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13484,"num_write_kb":467520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"500'45","reported_seq":1174,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.362165+0000","last_change":"2024-09-13T13:19:43.166438+0000","last_active":"2024-09-13T13:21:16.362165+0000","last_peered":"2024-09-13T13:21:16.362165+0000","last_clean":"2024-09-13T13:21:16.362165+0000","last_became_active":"2024-09-13T13:19:43.107259+0000","last_became_peered":"2024-09-13T13:19:43.107259+0000","last_unstale":"2024-09-13T13:21:16.362165+0000","last_undegraded":"2024-09-13T13:21:16.362165+0000","last_fullsized":"2024-09-13T13:21:16.362165+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'32","last_scrub_stamp":"2024-09-13T13:19:13.332009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:19:13.332009+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:04.660674+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":953,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.367284+0000","last_change":"2024-09-13T13:16:53.951208+0000","last_active":"2024-09-13T13:21:16.367284+0000","last_peered":"2024-09-13T13:21:16.367284+0000","last_clean":"2024-09-13T13:21:16.367284+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:21:16.367284+0000","last_undegraded":"2024-09-13T13:21:16.367284+0000","last_fullsized":"2024-09-13T13:21:16.367284+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:53.951135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:53.951135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:28.122637+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":554,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.371878+0000","last_change":"2024-09-13T13:19:43.117386+0000","last_active":"2024-09-13T13:21:16.371878+0000","last_peered":"2024-09-13T13:21:16.371878+0000","last_clean":"2024-09-13T13:21:16.371878+0000","last_became_active":"2024-09-13T13:19:43.117052+0000","last_became_peered":"2024-09-13T13:19:43.117052+0000","last_unstale":"2024-09-13T13:21:16.371878+0000","last_undegraded":"2024-09-13T13:21:16.371878+0000","last_fullsized":"2024-09-13T13:21:16.371878+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:50.485924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:50.485924+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":220,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360350+0000","last_change":"2024-09-13T13:20:54.560490+0000","last_active":"2024-09-13T13:21:16.360350+0000","last_peered":"2024-09-13T13:21:16.360350+0000","last_clean":"2024-09-13T13:21:16.360350+0000","last_became_active":"2024-09-13T13:19:43.120724+0000","last_became_peered":"2024-09-13T13:19:43.120724+0000","last_unstale":"2024-09-13T13:21:16.360350+0000","last_undegraded":"2024-09-13T13:21:16.360350+0000","last_fullsized":"2024-09-13T13:21:16.360350+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:54.560428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:54.560428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:08.160905+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":984,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360988+0000","last_change":"2024-09-13T13:19:39.865452+0000","last_active":"2024-09-13T13:21:16.360988+0000","last_peered":"2024-09-13T13:21:16.360988+0000","last_clean":"2024-09-13T13:21:16.360988+0000","last_became_active":"2024-09-13T13:12:52.959568+0000","last_became_peered":"2024-09-13T13:12:52.959568+0000","last_unstale":"2024-09-13T13:21:16.360988+0000","last_undegraded":"2024-09-13T13:21:16.360988+0000","last_fullsized":"2024-09-13T13:21:16.360988+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:39.865400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:19:39.865400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"501'9430","reported_seq":9656,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.431960+0000","last_change":"2024-09-13T13:19:37.669705+0000","last_active":"2024-09-13T13:21:16.431960+0000","last_peered":"2024-09-13T13:21:16.431960+0000","last_clean":"2024-09-13T13:21:16.431960+0000","last_became_active":"2024-09-13T13:19:05.190417+0000","last_became_peered":"2024-09-13T13:19:05.190417+0000","last_unstale":"2024-09-13T13:21:16.431960+0000","last_undegraded":"2024-09-13T13:21:16.431960+0000","last_fullsized":"2024-09-13T13:21:16.431960+0000","mapping_epoch":423,"log_start":"444'4446","ondisk_log_start":"444'4446","created":423,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'6678","last_scrub_stamp":"2024-09-13T13:19:37.669545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:37.669545+0000","objects_scrubbed":3453,"log_size":4984,"log_dups_size":0,"ondisk_log_size":4984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:14.665625+0000","scrub_duration":984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45285376,"num_objects":691,"num_object_clones":0,"num_object_copies":2073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"500'27","reported_seq":453,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.371012+0000","last_change":"2024-09-13T13:18:09.522990+0000","last_active":"2024-09-13T13:21:16.371012+0000","last_peered":"2024-09-13T13:21:16.371012+0000","last_clean":"2024-09-13T13:21:16.371012+0000","last_became_active":"2024-09-13T13:12:52.998508+0000","last_became_peered":"2024-09-13T13:12:52.998508+0000","last_unstale":"2024-09-13T13:21:16.371012+0000","last_undegraded":"2024-09-13T13:21:16.371012+0000","last_fullsized":"2024-09-13T13:21:16.371012+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:09.522908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:09.522908+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":442,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.369651+0000","last_change":"2024-09-13T13:19:43.131658+0000","last_active":"2024-09-13T13:21:16.369651+0000","last_peered":"2024-09-13T13:21:16.369651+0000","last_clean":"2024-09-13T13:21:16.369651+0000","last_became_active":"2024-09-13T13:19:43.131319+0000","last_became_peered":"2024-09-13T13:19:43.131319+0000","last_unstale":"2024-09-13T13:21:16.369651+0000","last_undegraded":"2024-09-13T13:21:16.369651+0000","last_fullsized":"2024-09-13T13:21:16.369651+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.465363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.465363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":990,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.361092+0000","last_change":"2024-09-13T13:19:30.758268+0000","last_active":"2024-09-13T13:21:16.361092+0000","last_peered":"2024-09-13T13:21:16.361092+0000","last_clean":"2024-09-13T13:21:16.361092+0000","last_became_active":"2024-09-13T13:12:52.995515+0000","last_became_peered":"2024-09-13T13:12:52.995515+0000","last_unstale":"2024-09-13T13:21:16.361092+0000","last_undegraded":"2024-09-13T13:21:16.361092+0000","last_fullsized":"2024-09-13T13:21:16.361092+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:30.758170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:30.758170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":940,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.366181+0000","last_change":"2024-09-13T13:18:49.072017+0000","last_active":"2024-09-13T13:21:16.366181+0000","last_peered":"2024-09-13T13:21:16.366181+0000","last_clean":"2024-09-13T13:21:16.366181+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:21:16.366181+0000","last_undegraded":"2024-09-13T13:21:16.366181+0000","last_fullsized":"2024-09-13T13:21:16.366181+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:49.071942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:49.071942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"501'9625","reported_seq":9852,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.101529+0000","last_change":"2024-09-13T13:19:16.555177+0000","last_active":"2024-09-13T13:21:18.101529+0000","last_peered":"2024-09-13T13:21:18.101529+0000","last_clean":"2024-09-13T13:21:18.101529+0000","last_became_active":"2024-09-13T13:19:14.300133+0000","last_became_peered":"2024-09-13T13:19:14.300133+0000","last_unstale":"2024-09-13T13:21:18.101529+0000","last_undegraded":"2024-09-13T13:21:18.101529+0000","last_fullsized":"2024-09-13T13:21:18.101529+0000","mapping_epoch":456,"log_start":"447'5529","ondisk_log_start":"447'5529","created":423,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'6169","last_scrub_stamp":"2024-09-13T13:19:16.554998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:16.554998+0000","objects_scrubbed":3151,"log_size":4096,"log_dups_size":0,"ondisk_log_size":4096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39124992,"num_objects":597,"num_object_clones":0,"num_object_copies":1791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3478,"num_write_kb":29568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":196,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.359468+0000","last_change":"2024-09-13T13:19:41.866832+0000","last_active":"2024-09-13T13:21:16.359468+0000","last_peered":"2024-09-13T13:21:16.359468+0000","last_clean":"2024-09-13T13:21:16.359468+0000","last_became_active":"2024-09-13T13:17:49.053269+0000","last_became_peered":"2024-09-13T13:17:49.053269+0000","last_unstale":"2024-09-13T13:21:16.359468+0000","last_undegraded":"2024-09-13T13:21:16.359468+0000","last_fullsized":"2024-09-13T13:21:16.359468+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:19:41.866752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:19:41.866752+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"500'45","reported_seq":518,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370988+0000","last_change":"2024-09-13T13:18:48.229223+0000","last_active":"2024-09-13T13:21:16.370988+0000","last_peered":"2024-09-13T13:21:16.370988+0000","last_clean":"2024-09-13T13:21:16.370988+0000","last_became_active":"2024-09-13T13:17:25.244686+0000","last_became_peered":"2024-09-13T13:17:25.244686+0000","last_unstale":"2024-09-13T13:21:16.370988+0000","last_undegraded":"2024-09-13T13:21:16.370988+0000","last_fullsized":"2024-09-13T13:21:16.370988+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'31","last_scrub_stamp":"2024-09-13T13:18:48.229160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:48.229160+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":980,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.362190+0000","last_change":"2024-09-13T13:19:14.352608+0000","last_active":"2024-09-13T13:21:16.362190+0000","last_peered":"2024-09-13T13:21:16.362190+0000","last_clean":"2024-09-13T13:21:16.362190+0000","last_became_active":"2024-09-13T13:12:52.953926+0000","last_became_peered":"2024-09-13T13:12:52.953926+0000","last_unstale":"2024-09-13T13:21:16.362190+0000","last_undegraded":"2024-09-13T13:21:16.362190+0000","last_fullsized":"2024-09-13T13:21:16.362190+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:14.352340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:14.352340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:01.678090+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":964,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.368821+0000","last_change":"2024-09-13T13:19:15.064401+0000","last_active":"2024-09-13T13:21:16.368821+0000","last_peered":"2024-09-13T13:21:16.368821+0000","last_clean":"2024-09-13T13:21:16.368821+0000","last_became_active":"2024-09-13T13:12:52.997322+0000","last_became_peered":"2024-09-13T13:12:52.997322+0000","last_unstale":"2024-09-13T13:21:16.368821+0000","last_undegraded":"2024-09-13T13:21:16.368821+0000","last_fullsized":"2024-09-13T13:21:16.368821+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:15.064195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:15.064195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":706,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.367601+0000","last_change":"2024-09-13T13:19:14.297300+0000","last_active":"2024-09-13T13:21:16.367601+0000","last_peered":"2024-09-13T13:21:16.367601+0000","last_clean":"2024-09-13T13:21:16.367601+0000","last_became_active":"2024-09-13T13:19:14.296958+0000","last_became_peered":"2024-09-13T13:19:14.296958+0000","last_unstale":"2024-09-13T13:21:16.367601+0000","last_undegraded":"2024-09-13T13:21:16.367601+0000","last_fullsized":"2024-09-13T13:21:16.367601+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:35.064188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:16:35.064188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:19.119240+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"501'9735","reported_seq":15071,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.418578+0000","last_change":"2024-09-13T13:20:23.134100+0000","last_active":"2024-09-13T13:21:16.418578+0000","last_peered":"2024-09-13T13:21:16.418578+0000","last_clean":"2024-09-13T13:21:16.418578+0000","last_became_active":"2024-09-13T13:20:23.124197+0000","last_became_peered":"2024-09-13T13:20:23.124197+0000","last_unstale":"2024-09-13T13:21:16.418578+0000","last_undegraded":"2024-09-13T13:21:16.418578+0000","last_fullsized":"2024-09-13T13:21:16.418578+0000","mapping_epoch":481,"log_start":"444'4866","ondisk_log_start":"444'4866","created":423,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'6911","last_scrub_stamp":"2024-09-13T13:19:39.721961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:39.721961+0000","objects_scrubbed":3475,"log_size":4869,"log_dups_size":0,"ondisk_log_size":4869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:08.621085+0000","scrub_duration":1014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41418752,"num_objects":632,"num_object_clones":0,"num_object_copies":1896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2843,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2548,"num_bytes_recovered":166985728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":194,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.366144+0000","last_change":"2024-09-13T13:19:43.110394+0000","last_active":"2024-09-13T13:21:16.366144+0000","last_peered":"2024-09-13T13:21:16.366144+0000","last_clean":"2024-09-13T13:21:16.366144+0000","last_became_active":"2024-09-13T13:19:43.109856+0000","last_became_peered":"2024-09-13T13:19:43.109856+0000","last_unstale":"2024-09-13T13:21:16.366144+0000","last_undegraded":"2024-09-13T13:21:16.366144+0000","last_fullsized":"2024-09-13T13:21:16.366144+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:18:51.111449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:18:51.111449+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"500'30","reported_seq":440,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370490+0000","last_change":"2024-09-13T13:17:02.421924+0000","last_active":"2024-09-13T13:21:16.370490+0000","last_peered":"2024-09-13T13:21:16.370490+0000","last_clean":"2024-09-13T13:21:16.370490+0000","last_became_active":"2024-09-13T13:12:52.996617+0000","last_became_peered":"2024-09-13T13:12:52.996617+0000","last_unstale":"2024-09-13T13:21:16.370490+0000","last_undegraded":"2024-09-13T13:21:16.370490+0000","last_fullsized":"2024-09-13T13:21:16.370490+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:17:02.421825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:02.421825+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:56.954562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":948,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370770+0000","last_change":"2024-09-13T13:19:42.134101+0000","last_active":"2024-09-13T13:21:16.370770+0000","last_peered":"2024-09-13T13:21:16.370770+0000","last_clean":"2024-09-13T13:21:16.370770+0000","last_became_active":"2024-09-13T13:12:53.003182+0000","last_became_peered":"2024-09-13T13:12:53.003182+0000","last_unstale":"2024-09-13T13:21:16.370770+0000","last_undegraded":"2024-09-13T13:21:16.370770+0000","last_fullsized":"2024-09-13T13:21:16.370770+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:42.134039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:42.134039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"467'11","reported_seq":275,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.361180+0000","last_change":"2024-09-13T13:20:48.601614+0000","last_active":"2024-09-13T13:21:16.361180+0000","last_peered":"2024-09-13T13:21:16.361180+0000","last_clean":"2024-09-13T13:21:16.361180+0000","last_became_active":"2024-09-13T13:19:43.122949+0000","last_became_peered":"2024-09-13T13:19:43.122949+0000","last_unstale":"2024-09-13T13:21:16.361180+0000","last_undegraded":"2024-09-13T13:21:16.361180+0000","last_fullsized":"2024-09-13T13:21:16.361180+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-09-13T13:20:48.601418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:48.601418+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:15.765338+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":435,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370925+0000","last_change":"2024-09-13T13:18:42.196223+0000","last_active":"2024-09-13T13:21:16.370925+0000","last_peered":"2024-09-13T13:21:16.370925+0000","last_clean":"2024-09-13T13:21:16.370925+0000","last_became_active":"2024-09-13T13:17:25.252583+0000","last_became_peered":"2024-09-13T13:17:25.252583+0000","last_unstale":"2024-09-13T13:21:16.370925+0000","last_undegraded":"2024-09-13T13:21:16.370925+0000","last_fullsized":"2024-09-13T13:21:16.370925+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:42.196120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:42.196120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"501'9463","reported_seq":15527,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.430381+0000","last_change":"2024-09-13T13:20:14.126474+0000","last_active":"2024-09-13T13:21:16.430381+0000","last_peered":"2024-09-13T13:21:16.430381+0000","last_clean":"2024-09-13T13:21:16.430381+0000","last_became_active":"2024-09-13T13:20:14.117394+0000","last_became_peered":"2024-09-13T13:20:14.117394+0000","last_unstale":"2024-09-13T13:21:16.430381+0000","last_undegraded":"2024-09-13T13:21:16.430381+0000","last_fullsized":"2024-09-13T13:21:16.430381+0000","mapping_epoch":477,"log_start":"444'4533","ondisk_log_start":"444'4533","created":423,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'6710","last_scrub_stamp":"2024-09-13T13:19:38.419629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:38.419629+0000","objects_scrubbed":3441,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44695552,"num_objects":682,"num_object_clones":0,"num_object_copies":2046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2759,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2912,"num_bytes_recovered":190840832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"500'21","reported_seq":988,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.373891+0000","last_change":"2024-09-13T13:20:49.223989+0000","last_active":"2024-09-13T13:21:16.373891+0000","last_peered":"2024-09-13T13:21:16.373891+0000","last_clean":"2024-09-13T13:21:16.373891+0000","last_became_active":"2024-09-13T13:12:53.004575+0000","last_became_peered":"2024-09-13T13:12:53.004575+0000","last_unstale":"2024-09-13T13:21:16.373891+0000","last_undegraded":"2024-09-13T13:21:16.373891+0000","last_fullsized":"2024-09-13T13:21:16.373891+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:20:49.223848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:49.223848+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:18.199856+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":969,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360443+0000","last_change":"2024-09-13T13:19:24.741190+0000","last_active":"2024-09-13T13:21:16.360443+0000","last_peered":"2024-09-13T13:21:16.360443+0000","last_clean":"2024-09-13T13:21:16.360443+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:21:16.360443+0000","last_undegraded":"2024-09-13T13:21:16.360443+0000","last_fullsized":"2024-09-13T13:21:16.360443+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:24.741107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:24.741107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":718,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370145+0000","last_change":"2024-09-13T13:19:14.300232+0000","last_active":"2024-09-13T13:21:16.370145+0000","last_peered":"2024-09-13T13:21:16.370145+0000","last_clean":"2024-09-13T13:21:16.370145+0000","last_became_active":"2024-09-13T13:19:14.299528+0000","last_became_peered":"2024-09-13T13:19:14.299528+0000","last_unstale":"2024-09-13T13:21:16.370145+0000","last_undegraded":"2024-09-13T13:21:16.370145+0000","last_fullsized":"2024-09-13T13:21:16.370145+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.620524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.620524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:07.944019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":950,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.372418+0000","last_change":"2024-09-13T13:19:43.113417+0000","last_active":"2024-09-13T13:21:16.372418+0000","last_peered":"2024-09-13T13:21:16.372418+0000","last_clean":"2024-09-13T13:21:16.372418+0000","last_became_active":"2024-09-13T13:19:43.112175+0000","last_became_peered":"2024-09-13T13:19:43.112175+0000","last_unstale":"2024-09-13T13:21:16.372418+0000","last_undegraded":"2024-09-13T13:21:16.372418+0000","last_fullsized":"2024-09-13T13:21:16.372418+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:30.598736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:30.598736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:06.959216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"501'9796","reported_seq":16274,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.695467+0000","last_change":"2024-09-13T13:20:07.198855+0000","last_active":"2024-09-13T13:21:18.695467+0000","last_peered":"2024-09-13T13:21:18.695467+0000","last_clean":"2024-09-13T13:21:18.695467+0000","last_became_active":"2024-09-13T13:20:07.188879+0000","last_became_peered":"2024-09-13T13:20:07.188879+0000","last_unstale":"2024-09-13T13:21:18.695467+0000","last_undegraded":"2024-09-13T13:21:18.695467+0000","last_fullsized":"2024-09-13T13:21:18.695467+0000","mapping_epoch":473,"log_start":"445'4882","ondisk_log_start":"445'4882","created":423,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'6823","last_scrub_stamp":"2024-09-13T13:19:37.423410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:37.423410+0000","objects_scrubbed":3564,"log_size":4914,"log_dups_size":0,"ondisk_log_size":4914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":652,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38273024,"num_objects":584,"num_object_clones":0,"num_object_copies":1752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2980,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3118,"num_bytes_recovered":204341248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"500'51","reported_seq":1004,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.362022+0000","last_change":"2024-09-13T13:19:43.242980+0000","last_active":"2024-09-13T13:21:16.362022+0000","last_peered":"2024-09-13T13:21:16.362022+0000","last_clean":"2024-09-13T13:21:16.362022+0000","last_became_active":"2024-09-13T13:19:43.138916+0000","last_became_peered":"2024-09-13T13:19:43.138916+0000","last_unstale":"2024-09-13T13:21:16.362022+0000","last_undegraded":"2024-09-13T13:21:16.362022+0000","last_fullsized":"2024-09-13T13:21:16.362022+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'35","last_scrub_stamp":"2024-09-13T13:19:28.677161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:19:28.677161+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:05.624895+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":590,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360453+0000","last_change":"2024-09-13T13:20:54.627605+0000","last_active":"2024-09-13T13:21:16.360453+0000","last_peered":"2024-09-13T13:21:16.360453+0000","last_clean":"2024-09-13T13:21:16.360453+0000","last_became_active":"2024-09-13T13:19:43.105487+0000","last_became_peered":"2024-09-13T13:19:43.105487+0000","last_unstale":"2024-09-13T13:21:16.360453+0000","last_undegraded":"2024-09-13T13:21:16.360453+0000","last_fullsized":"2024-09-13T13:21:16.360453+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:54.627528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:54.627528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:18.687600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":205,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.357451+0000","last_change":"2024-09-13T13:18:39.692048+0000","last_active":"2024-09-13T13:21:16.357451+0000","last_peered":"2024-09-13T13:21:16.357451+0000","last_clean":"2024-09-13T13:21:16.357451+0000","last_became_active":"2024-09-13T13:17:25.332964+0000","last_became_peered":"2024-09-13T13:17:25.332964+0000","last_unstale":"2024-09-13T13:21:16.357451+0000","last_undegraded":"2024-09-13T13:21:16.357451+0000","last_fullsized":"2024-09-13T13:21:16.357451+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:39.691978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:39.691978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:06.640480+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":182,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.369157+0000","last_change":"2024-09-13T13:19:43.121945+0000","last_active":"2024-09-13T13:21:16.369157+0000","last_peered":"2024-09-13T13:21:16.369157+0000","last_clean":"2024-09-13T13:21:16.369157+0000","last_became_active":"2024-09-13T13:19:43.115173+0000","last_became_peered":"2024-09-13T13:19:43.115173+0000","last_unstale":"2024-09-13T13:21:16.369157+0000","last_undegraded":"2024-09-13T13:21:16.369157+0000","last_fullsized":"2024-09-13T13:21:16.369157+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:48.717456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:48.717456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"501'9672","reported_seq":9897,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:19.940291+0000","last_change":"2024-09-13T13:19:37.887846+0000","last_active":"2024-09-13T13:21:19.940291+0000","last_peered":"2024-09-13T13:21:19.940291+0000","last_clean":"2024-09-13T13:21:19.940291+0000","last_became_active":"2024-09-13T13:19:05.189425+0000","last_became_peered":"2024-09-13T13:19:05.189425+0000","last_unstale":"2024-09-13T13:21:19.940291+0000","last_undegraded":"2024-09-13T13:21:19.940291+0000","last_fullsized":"2024-09-13T13:21:19.940291+0000","mapping_epoch":423,"log_start":"447'5270","ondisk_log_start":"447'5270","created":423,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'6719","last_scrub_stamp":"2024-09-13T13:19:37.887431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:37.887431+0000","objects_scrubbed":3488,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34275328,"num_objects":523,"num_object_clones":0,"num_object_copies":1569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2965,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"500'24","reported_seq":450,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.369959+0000","last_change":"2024-09-13T13:20:52.936125+0000","last_active":"2024-09-13T13:21:16.369959+0000","last_peered":"2024-09-13T13:21:16.369959+0000","last_clean":"2024-09-13T13:21:16.369959+0000","last_became_active":"2024-09-13T13:12:52.997548+0000","last_became_peered":"2024-09-13T13:12:52.997548+0000","last_unstale":"2024-09-13T13:21:16.369959+0000","last_undegraded":"2024-09-13T13:21:16.369959+0000","last_fullsized":"2024-09-13T13:21:16.369959+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:20:52.936042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:52.936042+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:55.203142+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":436,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370751+0000","last_change":"2024-09-13T13:20:48.892166+0000","last_active":"2024-09-13T13:21:16.370751+0000","last_peered":"2024-09-13T13:21:16.370751+0000","last_clean":"2024-09-13T13:21:16.370751+0000","last_became_active":"2024-09-13T13:19:43.130573+0000","last_became_peered":"2024-09-13T13:19:43.130573+0000","last_unstale":"2024-09-13T13:21:16.370751+0000","last_undegraded":"2024-09-13T13:21:16.370751+0000","last_fullsized":"2024-09-13T13:21:16.370751+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:48.892105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:48.892105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:13.024586+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":174,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.371464+0000","last_change":"2024-09-13T13:19:14.297431+0000","last_active":"2024-09-13T13:21:16.371464+0000","last_peered":"2024-09-13T13:21:16.371464+0000","last_clean":"2024-09-13T13:21:16.371464+0000","last_became_active":"2024-09-13T13:19:14.297087+0000","last_became_peered":"2024-09-13T13:19:14.297087+0000","last_unstale":"2024-09-13T13:21:16.371464+0000","last_undegraded":"2024-09-13T13:21:16.371464+0000","last_fullsized":"2024-09-13T13:21:16.371464+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:52.607404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.607404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:03.002320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":936,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370385+0000","last_change":"2024-09-13T13:20:36.368297+0000","last_active":"2024-09-13T13:21:16.370385+0000","last_peered":"2024-09-13T13:21:16.370385+0000","last_clean":"2024-09-13T13:21:16.370385+0000","last_became_active":"2024-09-13T13:19:43.130698+0000","last_became_peered":"2024-09-13T13:19:43.130698+0000","last_unstale":"2024-09-13T13:21:16.370385+0000","last_undegraded":"2024-09-13T13:21:16.370385+0000","last_fullsized":"2024-09-13T13:21:16.370385+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:36.368094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:20:36.368094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:45.881027+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"501'9477","reported_seq":9701,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.747450+0000","last_change":"2024-09-13T13:19:38.545875+0000","last_active":"2024-09-13T13:21:18.747450+0000","last_peered":"2024-09-13T13:21:18.747450+0000","last_clean":"2024-09-13T13:21:18.747450+0000","last_became_active":"2024-09-13T13:19:05.192183+0000","last_became_peered":"2024-09-13T13:19:05.192183+0000","last_unstale":"2024-09-13T13:21:18.747450+0000","last_undegraded":"2024-09-13T13:21:18.747450+0000","last_fullsized":"2024-09-13T13:21:18.747450+0000","mapping_epoch":423,"log_start":"444'4469","ondisk_log_start":"444'4469","created":423,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'6618","last_scrub_stamp":"2024-09-13T13:19:38.545676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:38.545676+0000","objects_scrubbed":3416,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":740,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35848192,"num_objects":547,"num_object_clones":0,"num_object_copies":1641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2869,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"500'39","reported_seq":1057,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.359574+0000","last_change":"2024-09-13T13:19:43.126593+0000","last_active":"2024-09-13T13:21:16.359574+0000","last_peered":"2024-09-13T13:21:16.359574+0000","last_clean":"2024-09-13T13:21:16.359574+0000","last_became_active":"2024-09-13T13:19:43.102750+0000","last_became_peered":"2024-09-13T13:19:43.102750+0000","last_unstale":"2024-09-13T13:21:16.359574+0000","last_undegraded":"2024-09-13T13:21:16.359574+0000","last_fullsized":"2024-09-13T13:21:16.359574+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'27","last_scrub_stamp":"2024-09-13T13:19:27.727761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:19:27.727761+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":440,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.371152+0000","last_change":"2024-09-13T13:20:36.925279+0000","last_active":"2024-09-13T13:21:16.371152+0000","last_peered":"2024-09-13T13:21:16.371152+0000","last_clean":"2024-09-13T13:21:16.371152+0000","last_became_active":"2024-09-13T13:19:43.130688+0000","last_became_peered":"2024-09-13T13:19:43.130688+0000","last_unstale":"2024-09-13T13:21:16.371152+0000","last_undegraded":"2024-09-13T13:21:16.371152+0000","last_fullsized":"2024-09-13T13:21:16.371152+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:20:36.925209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:36.925209+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:01.494744+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"467'14","reported_seq":751,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.362962+0000","last_change":"2024-09-13T13:20:44.533905+0000","last_active":"2024-09-13T13:21:16.362962+0000","last_peered":"2024-09-13T13:21:16.362962+0000","last_clean":"2024-09-13T13:21:16.362962+0000","last_became_active":"2024-09-13T13:19:43.130138+0000","last_became_peered":"2024-09-13T13:19:43.130138+0000","last_unstale":"2024-09-13T13:21:16.362962+0000","last_undegraded":"2024-09-13T13:21:16.362962+0000","last_fullsized":"2024-09-13T13:21:16.362962+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'14","last_scrub_stamp":"2024-09-13T13:20:44.533856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:44.533856+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:47.644688+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":941,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.359979+0000","last_change":"2024-09-13T13:18:43.850216+0000","last_active":"2024-09-13T13:21:16.359979+0000","last_peered":"2024-09-13T13:21:16.359979+0000","last_clean":"2024-09-13T13:21:16.359979+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:21:16.359979+0000","last_undegraded":"2024-09-13T13:21:16.359979+0000","last_fullsized":"2024-09-13T13:21:16.359979+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:43.850163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:43.850163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.18","version":"501'9800","reported_seq":20328,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:18.205343+0000","last_change":"2024-09-13T13:19:39.707330+0000","last_active":"2024-09-13T13:21:18.205343+0000","last_peered":"2024-09-13T13:21:18.205343+0000","last_clean":"2024-09-13T13:21:18.205343+0000","last_became_active":"2024-09-13T13:19:38.087942+0000","last_became_peered":"2024-09-13T13:19:38.087942+0000","last_unstale":"2024-09-13T13:21:18.205343+0000","last_undegraded":"2024-09-13T13:21:18.205343+0000","last_fullsized":"2024-09-13T13:21:18.205343+0000","mapping_epoch":463,"log_start":"443'4398","ondisk_log_start":"443'4398","created":450,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":5,"last_scrub":"465'6879","last_scrub_stamp":"2024-09-13T13:19:39.707182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:39.707182+0000","objects_scrubbed":3533,"log_size":5402,"log_dups_size":0,"ondisk_log_size":5402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:11.618138+0000","scrub_duration":1134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39059456,"num_objects":596,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2937,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":918,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.361988+0000","last_change":"2024-09-13T13:19:22.559601+0000","last_active":"2024-09-13T13:21:16.361988+0000","last_peered":"2024-09-13T13:21:16.361988+0000","last_clean":"2024-09-13T13:21:16.361988+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:21:16.361988+0000","last_undegraded":"2024-09-13T13:21:16.361988+0000","last_fullsized":"2024-09-13T13:21:16.361988+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:22.559551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:19:22.559551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:02.640001+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"500'18","reported_seq":487,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.361264+0000","last_change":"2024-09-13T13:19:43.170962+0000","last_active":"2024-09-13T13:21:16.361264+0000","last_peered":"2024-09-13T13:21:16.361264+0000","last_clean":"2024-09-13T13:21:16.361264+0000","last_became_active":"2024-09-13T13:19:43.132163+0000","last_became_peered":"2024-09-13T13:19:43.132163+0000","last_unstale":"2024-09-13T13:21:16.361264+0000","last_undegraded":"2024-09-13T13:21:16.361264+0000","last_fullsized":"2024-09-13T13:21:16.361264+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'12","last_scrub_stamp":"2024-09-13T13:17:59.217309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:59.217309+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:03.663552+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":973,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360772+0000","last_change":"2024-09-13T13:19:38.925198+0000","last_active":"2024-09-13T13:21:16.360772+0000","last_peered":"2024-09-13T13:21:16.360772+0000","last_clean":"2024-09-13T13:21:16.360772+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:21:16.360772+0000","last_undegraded":"2024-09-13T13:21:16.360772+0000","last_fullsized":"2024-09-13T13:21:16.360772+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:38.925136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:38.925136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":639,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.362624+0000","last_change":"2024-09-13T13:20:37.601177+0000","last_active":"2024-09-13T13:21:16.362624+0000","last_peered":"2024-09-13T13:21:16.362624+0000","last_clean":"2024-09-13T13:21:16.362624+0000","last_became_active":"2024-09-13T13:19:43.104791+0000","last_became_peered":"2024-09-13T13:19:43.104791+0000","last_unstale":"2024-09-13T13:21:16.362624+0000","last_undegraded":"2024-09-13T13:21:16.362624+0000","last_fullsized":"2024-09-13T13:21:16.362624+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:37.601130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:20:37.601130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:54.167807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"501'9741","reported_seq":26226,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:19.996997+0000","last_change":"2024-09-13T13:20:14.168863+0000","last_active":"2024-09-13T13:21:19.996997+0000","last_peered":"2024-09-13T13:21:19.996997+0000","last_clean":"2024-09-13T13:21:19.996997+0000","last_became_active":"2024-09-13T13:20:14.121004+0000","last_became_peered":"2024-09-13T13:20:14.121004+0000","last_unstale":"2024-09-13T13:21:19.996997+0000","last_undegraded":"2024-09-13T13:21:19.996997+0000","last_fullsized":"2024-09-13T13:21:19.996997+0000","mapping_epoch":477,"log_start":"447'5354","ondisk_log_start":"447'5354","created":450,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"465'6824","last_scrub_stamp":"2024-09-13T13:19:40.848436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:19:40.848436+0000","objects_scrubbed":3476,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35651584,"num_objects":544,"num_object_clones":0,"num_object_copies":1632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2932,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3016,"num_bytes_recovered":197591040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"500'21","reported_seq":987,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.376230+0000","last_change":"2024-09-13T13:19:43.142025+0000","last_active":"2024-09-13T13:21:16.376230+0000","last_peered":"2024-09-13T13:21:16.376230+0000","last_clean":"2024-09-13T13:21:16.376230+0000","last_became_active":"2024-09-13T13:19:43.124275+0000","last_became_peered":"2024-09-13T13:19:43.124275+0000","last_unstale":"2024-09-13T13:21:16.376230+0000","last_undegraded":"2024-09-13T13:21:16.376230+0000","last_fullsized":"2024-09-13T13:21:16.376230+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:52.518502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:52.518502+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:09.990996+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":428,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.369764+0000","last_change":"2024-09-13T13:20:39.917442+0000","last_active":"2024-09-13T13:21:16.369764+0000","last_peered":"2024-09-13T13:21:16.369764+0000","last_clean":"2024-09-13T13:21:16.369764+0000","last_became_active":"2024-09-13T13:19:43.107386+0000","last_became_peered":"2024-09-13T13:19:43.107386+0000","last_unstale":"2024-09-13T13:21:16.369764+0000","last_undegraded":"2024-09-13T13:21:16.369764+0000","last_fullsized":"2024-09-13T13:21:16.369764+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:39.917361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:39.917361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:50.531949+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":956,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.367757+0000","last_change":"2024-09-13T13:20:51.277038+0000","last_active":"2024-09-13T13:21:16.367757+0000","last_peered":"2024-09-13T13:21:16.367757+0000","last_clean":"2024-09-13T13:21:16.367757+0000","last_became_active":"2024-09-13T13:19:43.131436+0000","last_became_peered":"2024-09-13T13:19:43.131436+0000","last_unstale":"2024-09-13T13:21:16.367757+0000","last_undegraded":"2024-09-13T13:21:16.367757+0000","last_fullsized":"2024-09-13T13:21:16.367757+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:51.276953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:51.276953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:03.121291+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":212,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360591+0000","last_change":"2024-09-13T13:19:43.114011+0000","last_active":"2024-09-13T13:21:16.360591+0000","last_peered":"2024-09-13T13:21:16.360591+0000","last_clean":"2024-09-13T13:21:16.360591+0000","last_became_active":"2024-09-13T13:19:43.110373+0000","last_became_peered":"2024-09-13T13:19:43.110373+0000","last_unstale":"2024-09-13T13:21:16.360591+0000","last_undegraded":"2024-09-13T13:21:16.360591+0000","last_fullsized":"2024-09-13T13:21:16.360591+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:45.736080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.736080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:04.594732+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"500'21","reported_seq":1009,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360604+0000","last_change":"2024-09-13T13:19:43.127624+0000","last_active":"2024-09-13T13:21:16.360604+0000","last_peered":"2024-09-13T13:21:16.360604+0000","last_clean":"2024-09-13T13:21:16.360604+0000","last_became_active":"2024-09-13T13:19:43.108087+0000","last_became_peered":"2024-09-13T13:19:43.108087+0000","last_unstale":"2024-09-13T13:21:16.360604+0000","last_undegraded":"2024-09-13T13:21:16.360604+0000","last_fullsized":"2024-09-13T13:21:16.360604+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:04.222760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:04.222760+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:57.607945+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":939,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.376898+0000","last_change":"2024-09-13T13:19:43.115328+0000","last_active":"2024-09-13T13:21:16.376898+0000","last_peered":"2024-09-13T13:21:16.376898+0000","last_clean":"2024-09-13T13:21:16.376898+0000","last_became_active":"2024-09-13T13:19:43.115022+0000","last_became_peered":"2024-09-13T13:19:43.115022+0000","last_unstale":"2024-09-13T13:21:16.376898+0000","last_undegraded":"2024-09-13T13:21:16.376898+0000","last_fullsized":"2024-09-13T13:21:16.376898+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:09.634027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:09.634027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:59.119121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"467'11","reported_seq":1210,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.366328+0000","last_change":"2024-09-13T13:19:43.204263+0000","last_active":"2024-09-13T13:21:16.366328+0000","last_peered":"2024-09-13T13:21:16.366328+0000","last_clean":"2024-09-13T13:21:16.366328+0000","last_became_active":"2024-09-13T13:19:43.130547+0000","last_became_peered":"2024-09-13T13:19:43.130547+0000","last_unstale":"2024-09-13T13:21:16.366328+0000","last_undegraded":"2024-09-13T13:21:16.366328+0000","last_fullsized":"2024-09-13T13:21:16.366328+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'9","last_scrub_stamp":"2024-09-13T13:17:27.165657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:27.165657+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:57.287216+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":435,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.371199+0000","last_change":"2024-09-13T13:18:41.148239+0000","last_active":"2024-09-13T13:21:16.371199+0000","last_peered":"2024-09-13T13:21:16.371199+0000","last_clean":"2024-09-13T13:21:16.371199+0000","last_became_active":"2024-09-13T13:17:25.333587+0000","last_became_peered":"2024-09-13T13:17:25.333587+0000","last_unstale":"2024-09-13T13:21:16.371199+0000","last_undegraded":"2024-09-13T13:21:16.371199+0000","last_fullsized":"2024-09-13T13:21:16.371199+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:41.147987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:41.147987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"500'30","reported_seq":1025,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.366655+0000","last_change":"2024-09-13T13:20:43.428468+0000","last_active":"2024-09-13T13:21:16.366655+0000","last_peered":"2024-09-13T13:21:16.366655+0000","last_clean":"2024-09-13T13:21:16.366655+0000","last_became_active":"2024-09-13T13:19:43.128700+0000","last_became_peered":"2024-09-13T13:19:43.128700+0000","last_unstale":"2024-09-13T13:21:16.366655+0000","last_undegraded":"2024-09-13T13:21:16.366655+0000","last_fullsized":"2024-09-13T13:21:16.366655+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'20","last_scrub_stamp":"2024-09-13T13:20:43.428398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:43.428398+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:09.431737+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":839,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370595+0000","last_change":"2024-09-13T13:19:43.117077+0000","last_active":"2024-09-13T13:21:16.370595+0000","last_peered":"2024-09-13T13:21:16.370595+0000","last_clean":"2024-09-13T13:21:16.370595+0000","last_became_active":"2024-09-13T13:19:43.116734+0000","last_became_peered":"2024-09-13T13:19:43.116734+0000","last_unstale":"2024-09-13T13:21:16.370595+0000","last_undegraded":"2024-09-13T13:21:16.370595+0000","last_fullsized":"2024-09-13T13:21:16.370595+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:28.963152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:28.963152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:05.956876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":1209,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360376+0000","last_change":"2024-09-13T13:19:16.725417+0000","last_active":"2024-09-13T13:21:16.360376+0000","last_peered":"2024-09-13T13:21:16.360376+0000","last_clean":"2024-09-13T13:21:16.360376+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:21:16.360376+0000","last_undegraded":"2024-09-13T13:21:16.360376+0000","last_fullsized":"2024-09-13T13:21:16.360376+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:19:16.725342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:16.725342+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":951,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.374835+0000","last_change":"2024-09-13T13:19:43.129405+0000","last_active":"2024-09-13T13:21:16.374835+0000","last_peered":"2024-09-13T13:21:16.374835+0000","last_clean":"2024-09-13T13:21:16.374835+0000","last_became_active":"2024-09-13T13:19:43.125093+0000","last_became_peered":"2024-09-13T13:19:43.125093+0000","last_unstale":"2024-09-13T13:21:16.374835+0000","last_undegraded":"2024-09-13T13:21:16.374835+0000","last_fullsized":"2024-09-13T13:21:16.374835+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:33.554870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:17:33.554870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:00.095054+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"500'27","reported_seq":445,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370884+0000","last_change":"2024-09-13T13:18:45.145089+0000","last_active":"2024-09-13T13:21:16.370884+0000","last_peered":"2024-09-13T13:21:16.370884+0000","last_clean":"2024-09-13T13:21:16.370884+0000","last_became_active":"2024-09-13T13:12:52.997417+0000","last_became_peered":"2024-09-13T13:12:52.997417+0000","last_unstale":"2024-09-13T13:21:16.370884+0000","last_undegraded":"2024-09-13T13:21:16.370884+0000","last_fullsized":"2024-09-13T13:21:16.370884+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:45.145010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:45.145010+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":428,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370583+0000","last_change":"2024-09-13T13:20:53.986006+0000","last_active":"2024-09-13T13:21:16.370583+0000","last_peered":"2024-09-13T13:21:16.370583+0000","last_clean":"2024-09-13T13:21:16.370583+0000","last_became_active":"2024-09-13T13:19:43.116578+0000","last_became_peered":"2024-09-13T13:19:43.116578+0000","last_unstale":"2024-09-13T13:21:16.370583+0000","last_undegraded":"2024-09-13T13:21:16.370583+0000","last_fullsized":"2024-09-13T13:21:16.370583+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:53.985950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:53.985950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:04.500372+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":950,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.375338+0000","last_change":"2024-09-13T13:17:33.279005+0000","last_active":"2024-09-13T13:21:16.375338+0000","last_peered":"2024-09-13T13:21:16.375338+0000","last_clean":"2024-09-13T13:21:16.375338+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:21:16.375338+0000","last_undegraded":"2024-09-13T13:21:16.375338+0000","last_fullsized":"2024-09-13T13:21:16.375338+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:33.278909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:33.278909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:57.662698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":953,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.366306+0000","last_change":"2024-09-13T13:20:53.246026+0000","last_active":"2024-09-13T13:21:16.366306+0000","last_peered":"2024-09-13T13:21:16.366306+0000","last_clean":"2024-09-13T13:21:16.366306+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:21:16.366306+0000","last_undegraded":"2024-09-13T13:21:16.366306+0000","last_fullsized":"2024-09-13T13:21:16.366306+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:53.245964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:20:53.245964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:14.628738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"500'21","reported_seq":949,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.359798+0000","last_change":"2024-09-13T13:17:26.125163+0000","last_active":"2024-09-13T13:21:16.359798+0000","last_peered":"2024-09-13T13:21:16.359798+0000","last_clean":"2024-09-13T13:21:16.359798+0000","last_became_active":"2024-09-13T13:17:25.331530+0000","last_became_peered":"2024-09-13T13:17:25.331530+0000","last_unstale":"2024-09-13T13:21:16.359798+0000","last_undegraded":"2024-09-13T13:21:16.359798+0000","last_fullsized":"2024-09-13T13:21:16.359798+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15","last_scrub_stamp":"2024-09-13T13:17:26.125088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:17:26.125088+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:59.603582+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":940,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.368537+0000","last_change":"2024-09-13T13:19:43.106069+0000","last_active":"2024-09-13T13:21:16.368537+0000","last_peered":"2024-09-13T13:21:16.368537+0000","last_clean":"2024-09-13T13:21:16.368537+0000","last_became_active":"2024-09-13T13:19:43.103538+0000","last_became_peered":"2024-09-13T13:19:43.103538+0000","last_unstale":"2024-09-13T13:21:16.368537+0000","last_undegraded":"2024-09-13T13:21:16.368537+0000","last_fullsized":"2024-09-13T13:21:16.368537+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:56.965870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:56.965870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:15.279520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":941,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370482+0000","last_change":"2024-09-13T13:18:03.037546+0000","last_active":"2024-09-13T13:21:16.370482+0000","last_peered":"2024-09-13T13:21:16.370482+0000","last_clean":"2024-09-13T13:21:16.370482+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:21:16.370482+0000","last_undegraded":"2024-09-13T13:21:16.370482+0000","last_fullsized":"2024-09-13T13:21:16.370482+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:03.037472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:18:03.037472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:06.218060+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":928,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.371505+0000","last_change":"2024-09-13T13:18:44.505493+0000","last_active":"2024-09-13T13:21:16.371505+0000","last_peered":"2024-09-13T13:21:16.371505+0000","last_clean":"2024-09-13T13:21:16.371505+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:21:16.371505+0000","last_undegraded":"2024-09-13T13:21:16.371505+0000","last_fullsized":"2024-09-13T13:21:16.371505+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:44.505427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:44.505427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"500'27","reported_seq":1002,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.366040+0000","last_change":"2024-09-13T13:19:43.128341+0000","last_active":"2024-09-13T13:21:16.366040+0000","last_peered":"2024-09-13T13:21:16.366040+0000","last_clean":"2024-09-13T13:21:16.366040+0000","last_became_active":"2024-09-13T13:19:43.103262+0000","last_became_peered":"2024-09-13T13:19:43.103262+0000","last_unstale":"2024-09-13T13:21:16.366040+0000","last_undegraded":"2024-09-13T13:21:16.366040+0000","last_fullsized":"2024-09-13T13:21:16.366040+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:18:33.848726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:33.848726+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":964,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.360774+0000","last_change":"2024-09-13T13:20:50.661034+0000","last_active":"2024-09-13T13:21:16.360774+0000","last_peered":"2024-09-13T13:21:16.360774+0000","last_clean":"2024-09-13T13:21:16.360774+0000","last_became_active":"2024-09-13T13:19:43.121975+0000","last_became_peered":"2024-09-13T13:19:43.121975+0000","last_unstale":"2024-09-13T13:21:16.360774+0000","last_undegraded":"2024-09-13T13:21:16.360774+0000","last_fullsized":"2024-09-13T13:21:16.360774+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:50.660981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:50.660981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:13.399315+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":636,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.369536+0000","last_change":"2024-09-13T13:20:45.965033+0000","last_active":"2024-09-13T13:21:16.369536+0000","last_peered":"2024-09-13T13:21:16.369536+0000","last_clean":"2024-09-13T13:21:16.369536+0000","last_became_active":"2024-09-13T13:19:43.108262+0000","last_became_peered":"2024-09-13T13:19:43.108262+0000","last_unstale":"2024-09-13T13:21:16.369536+0000","last_undegraded":"2024-09-13T13:21:16.369536+0000","last_fullsized":"2024-09-13T13:21:16.369536+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:45.964988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:45.964988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:05.860268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":916,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370636+0000","last_change":"2024-09-13T13:11:55.038230+0000","last_active":"2024-09-13T13:21:16.370636+0000","last_peered":"2024-09-13T13:21:16.370636+0000","last_clean":"2024-09-13T13:21:16.370636+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:21:16.370636+0000","last_undegraded":"2024-09-13T13:21:16.370636+0000","last_fullsized":"2024-09-13T13:21:16.370636+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:11:55.038185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:11:55.038185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:18.090301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"500'12","reported_seq":992,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.368849+0000","last_change":"2024-09-13T13:18:46.033316+0000","last_active":"2024-09-13T13:21:16.368849+0000","last_peered":"2024-09-13T13:21:16.368849+0000","last_clean":"2024-09-13T13:21:16.368849+0000","last_became_active":"2024-09-13T13:16:12.073702+0000","last_became_peered":"2024-09-13T13:16:12.073702+0000","last_unstale":"2024-09-13T13:21:16.368849+0000","last_undegraded":"2024-09-13T13:21:16.368849+0000","last_fullsized":"2024-09-13T13:21:16.368849+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8","last_scrub_stamp":"2024-09-13T13:18:46.033238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:18:46.033238+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":964,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.386195+0000","last_change":"2024-09-13T13:17:57.579827+0000","last_active":"2024-09-13T13:21:16.386195+0000","last_peered":"2024-09-13T13:21:16.386195+0000","last_clean":"2024-09-13T13:21:16.386195+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:21:16.386195+0000","last_undegraded":"2024-09-13T13:21:16.386195+0000","last_fullsized":"2024-09-13T13:21:16.386195+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:57.579778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:17:57.579778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:11.006204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":887,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.363025+0000","last_change":"2024-09-13T13:19:43.104851+0000","last_active":"2024-09-13T13:21:16.363025+0000","last_peered":"2024-09-13T13:21:16.363025+0000","last_clean":"2024-09-13T13:21:16.363025+0000","last_became_active":"2024-09-13T13:19:43.104438+0000","last_became_peered":"2024-09-13T13:19:43.104438+0000","last_unstale":"2024-09-13T13:21:16.363025+0000","last_undegraded":"2024-09-13T13:21:16.363025+0000","last_fullsized":"2024-09-13T13:21:16.363025+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:17.404669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:19:17.404669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:06.627251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":690,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.370011+0000","last_change":"2024-09-13T13:19:43.142300+0000","last_active":"2024-09-13T13:21:16.370011+0000","last_peered":"2024-09-13T13:21:16.370011+0000","last_clean":"2024-09-13T13:21:16.370011+0000","last_became_active":"2024-09-13T13:19:43.141978+0000","last_became_peered":"2024-09-13T13:19:43.141978+0000","last_unstale":"2024-09-13T13:21:16.370011+0000","last_undegraded":"2024-09-13T13:21:16.370011+0000","last_fullsized":"2024-09-13T13:21:16.370011+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:18:07.682648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:18:07.682648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"501'10003","reported_seq":10212,"reported_epoch":501,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-13T13:21:19.798945+0000","last_change":"2024-09-13T13:21:08.434795+0000","last_active":"2024-09-13T13:21:19.798945+0000","last_peered":"2024-09-13T13:21:19.798945+0000","last_clean":"2024-09-13T13:19:06.170841+0000","last_became_active":"2024-09-13T13:19:08.301760+0000","last_became_peered":"2024-09-13T13:19:08.301760+0000","last_unstale":"2024-09-13T13:21:19.798945+0000","last_undegraded":"2024-09-13T13:21:19.798945+0000","last_fullsized":"2024-09-13T13:21:19.798945+0000","mapping_epoch":453,"log_start":"457'6388","ondisk_log_start":"457'6388","created":450,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:17:54.116068+0000","last_clean_scrub_stamp":"2024-09-13T13:17:54.116068+0000","objects_scrubbed":0,"log_size":3615,"log_dups_size":0,"ondisk_log_size":3615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37388288,"num_objects":571,"num_object_clones":0,"num_object_copies":1713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1713,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6943,"num_write_kb":240416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[1,7,3],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3,7","objects":571}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":960,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.368704+0000","last_change":"2024-09-13T13:19:26.266256+0000","last_active":"2024-09-13T13:21:16.368704+0000","last_peered":"2024-09-13T13:21:16.368704+0000","last_clean":"2024-09-13T13:21:16.368704+0000","last_became_active":"2024-09-13T13:12:52.997455+0000","last_became_peered":"2024-09-13T13:12:52.997455+0000","last_unstale":"2024-09-13T13:21:16.368704+0000","last_undegraded":"2024-09-13T13:21:16.368704+0000","last_fullsized":"2024-09-13T13:21:16.368704+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:19:26.266186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:19:26.266186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":933,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.375775+0000","last_change":"2024-09-13T13:16:36.980668+0000","last_active":"2024-09-13T13:21:16.375775+0000","last_peered":"2024-09-13T13:21:16.375775+0000","last_clean":"2024-09-13T13:21:16.375775+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:21:16.375775+0000","last_undegraded":"2024-09-13T13:21:16.375775+0000","last_fullsized":"2024-09-13T13:21:16.375775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:16:36.980594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:16:36.980594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:24.137197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"500'27","reported_seq":980,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.362501+0000","last_change":"2024-09-13T13:20:51.567559+0000","last_active":"2024-09-13T13:21:16.362501+0000","last_peered":"2024-09-13T13:21:16.362501+0000","last_clean":"2024-09-13T13:21:16.362501+0000","last_became_active":"2024-09-13T13:16:12.070277+0000","last_became_peered":"2024-09-13T13:16:12.070277+0000","last_unstale":"2024-09-13T13:21:16.362501+0000","last_undegraded":"2024-09-13T13:21:16.362501+0000","last_fullsized":"2024-09-13T13:21:16.362501+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-13T13:20:51.567506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:51.567506+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:22:06.487621+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":979,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-09-13T13:21:16.374997+0000","last_change":"2024-09-13T13:19:43.129850+0000","last_active":"2024-09-13T13:21:16.374997+0000","last_peered":"2024-09-13T13:21:16.374997+0000","last_clean":"2024-09-13T13:21:16.374997+0000","last_became_active":"2024-09-13T13:19:43.129130+0000","last_became_peered":"2024-09-13T13:19:43.129130+0000","last_unstale":"2024-09-13T13:21:16.374997+0000","last_undegraded":"2024-09-13T13:21:16.374997+0000","last_fullsized":"2024-09-13T13:21:16.374997+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:17:23.547758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:17:23.547758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:21:58.162800+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":26,"stat_sum":{"num_bytes":1237909528,"num_objects":18891,"num_object_clones":0,"num_object_copies":56673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5403,"num_objects_unfound":0,"num_objects_dirty":18891,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148493,"num_write_kb":3776897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27695,"num_bytes_recovered":1814757376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4533137408,"data_stored":4533125192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116273,"ondisk_log_size":116273,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1543,"num_read_kb":1300,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":6046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":973,"ondisk_log_size":973,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2233,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":521,"num_read_kb":516,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":236223201526,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141136,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1518132,"kb_used_data":754596,"kb_used_omap":4,"kb_used_meta":763515,"kb_avail":92227020,"statfs":{"total":95995035648,"available":94440468480,"internally_reserved":0,"allocated":772706304,"data_stored":772348551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":781840045},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692098,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1157072,"kb_used_data":372032,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92588080,"statfs":{"total":95995035648,"available":94810193920,"internally_reserved":0,"allocated":380960768,"data_stored":380583419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921032,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1093724,"kb_used_data":380876,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":92651428,"statfs":{"total":95995035648,"available":94875062272,"internally_reserved":0,"allocated":390017024,"data_stored":389625900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117263,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1409508,"kb_used_data":521748,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92335644,"statfs":{"total":95995035648,"available":94551699456,"internally_reserved":0,"allocated":534269952,"data_stored":533872668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":909048263},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411602,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1336960,"kb_used_data":611632,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92408192,"statfs":{"total":95995035648,"available":94625988608,"internally_reserved":0,"allocated":626311168,"data_stored":625948574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":742717894},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575129,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1476784,"kb_used_data":661216,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92268368,"statfs":{"total":95995035648,"available":94482808832,"internally_reserved":0,"allocated":677085184,"data_stored":676674131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":835123653},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738656,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1367072,"kb_used_data":437520,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92378080,"statfs":{"total":95995035648,"available":94595153920,"internally_reserved":0,"allocated":448020480,"data_stored":447626371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":951712197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":421986304,"data_stored":421986304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":675151872,"data_stored":675151872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":602738688,"data_stored":602734616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":508166144,"data_stored":508166144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":384372736,"data_stored":384368664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":375652352,"data_stored":375652352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":739446784,"data_stored":739442712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":825622528,"data_stored":825622528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:21:23.214 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T13:21:23.214 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:21:23.214 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T13:21:23.214 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-13T13:21:23.214 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.4 2024-09-13T13:21:23.234 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:23.246 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T13:21:23.279 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:23.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:23.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:23.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:23.419 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:23 smithi052 ceph-mon[36682]: 5.c scrub starts 2024-09-13T13:21:23.420 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:23 smithi052 ceph-mon[36682]: 5.c scrub ok 2024-09-13T13:21:23.420 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:23 smithi052 ceph-mon[36682]: from='client.36908 v1:172.21.15.52:0/1798572283' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:21:23.420 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:23 smithi052 ceph-mon[36682]: pgmap v1027: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.1 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 39 KiB/s rd, 0 B/s wr, 1.18k op/s; 5403/54870 objects misplaced (9.847%) 2024-09-13T13:21:23.420 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:23 smithi052 ceph-mon[36682]: 4.3 scrub starts 2024-09-13T13:21:23.420 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:23 smithi052 ceph-mon[36682]: 4.3 scrub ok 2024-09-13T13:21:23.421 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:23 smithi052 ceph-mon[29471]: 5.c scrub starts 2024-09-13T13:21:23.421 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:23 smithi052 ceph-mon[29471]: 5.c scrub ok 2024-09-13T13:21:23.421 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:23 smithi052 ceph-mon[29471]: from='client.36908 v1:172.21.15.52:0/1798572283' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:21:23.421 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:23 smithi052 ceph-mon[29471]: pgmap v1027: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.1 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 39 KiB/s rd, 0 B/s wr, 1.18k op/s; 5403/54870 objects misplaced (9.847%) 2024-09-13T13:21:23.421 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:23 smithi052 ceph-mon[29471]: 4.3 scrub starts 2024-09-13T13:21:23.421 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:23 smithi052 ceph-mon[29471]: 4.3 scrub ok 2024-09-13T13:21:23.430 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:23.476 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:23.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:23.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:23.570 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:23 smithi167 systemd[1]: Stopping Ceph osd.4 for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:21:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:23 smithi167 ceph-mon[33178]: 5.c scrub starts 2024-09-13T13:21:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:23 smithi167 ceph-mon[33178]: 5.c scrub ok 2024-09-13T13:21:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:23 smithi167 ceph-mon[33178]: from='client.36908 v1:172.21.15.52:0/1798572283' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:21:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:23 smithi167 ceph-mon[33178]: pgmap v1027: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.1 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 39 KiB/s rd, 0 B/s wr, 1.18k op/s; 5403/54870 objects misplaced (9.847%) 2024-09-13T13:21:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:23 smithi167 ceph-mon[33178]: 4.3 scrub starts 2024-09-13T13:21:23.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:23 smithi167 ceph-mon[33178]: 4.3 scrub ok 2024-09-13T13:21:23.578 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:23.619 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:23.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:23.721 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:23.752 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:23.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:23.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:23.880 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:23.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:23.982 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:24.008 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:24.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:24.110 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:24.154 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:24.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:24.255 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:24.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:24.286 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:24.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:24.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:24.402 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:21:24.415 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:24.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:24.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:24.466 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:21:24.517 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:24.530 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-mon[33178]: 3.9 scrub starts 2024-09-13T13:21:24.530 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-mon[33178]: 3.9 scrub ok 2024-09-13T13:21:24.530 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-mon[33178]: 6.f scrub starts 2024-09-13T13:21:24.530 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-mon[33178]: 6.f scrub ok 2024-09-13T13:21:24.530 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:21:24.530 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:21:24.530 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-mon[33178]: 1.1 scrub starts 2024-09-13T13:21:24.531 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-mon[33178]: 1.1 scrub ok 2024-09-13T13:21:24.531 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-mon[33178]: osdmap e502: 8 total, 8 up, 7 in 2024-09-13T13:21:24.531 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-mon[33178]: osd.4 marked itself down and dead 2024-09-13T13:21:24.531 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1551502649' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:24.531 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-mon[33178]: from='client.26809 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:24.531 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[37339]: 2024-09-13T13:21:24.226+0000 7f9454243640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:21:24.531 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[37339]: 2024-09-13T13:21:24.226+0000 7f9454243640 -1 osd.4 502 *** Got signal Terminated *** 2024-09-13T13:21:24.531 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[37339]: 2024-09-13T13:21:24.226+0000 7f9454243640 -1 osd.4 502 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T13:21:24.561 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:24.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:24.663 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:24.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:24.698 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:24.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:24.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[36682]: 3.9 scrub starts 2024-09-13T13:21:24.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[36682]: 3.9 scrub ok 2024-09-13T13:21:24.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[36682]: 6.f scrub starts 2024-09-13T13:21:24.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[36682]: 6.f scrub ok 2024-09-13T13:21:24.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:21:24.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:21:24.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[36682]: 1.1 scrub starts 2024-09-13T13:21:24.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[36682]: 1.1 scrub ok 2024-09-13T13:21:24.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[36682]: osdmap e502: 8 total, 8 up, 7 in 2024-09-13T13:21:24.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[36682]: osd.4 marked itself down and dead 2024-09-13T13:21:24.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1551502649' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:24.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[36682]: from='client.26809 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:24.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[29471]: 3.9 scrub starts 2024-09-13T13:21:24.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[29471]: 3.9 scrub ok 2024-09-13T13:21:24.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[29471]: 6.f scrub starts 2024-09-13T13:21:24.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[29471]: 6.f scrub ok 2024-09-13T13:21:24.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:21:24.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:21:24.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[29471]: 1.1 scrub starts 2024-09-13T13:21:24.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[29471]: 1.1 scrub ok 2024-09-13T13:21:24.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[29471]: osdmap e502: 8 total, 8 up, 7 in 2024-09-13T13:21:24.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[29471]: osd.4 marked itself down and dead 2024-09-13T13:21:24.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1551502649' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:24.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:24 smithi052 ceph-mon[29471]: from='client.26809 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:24.799 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:24.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:21:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:21:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:24.858 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:24.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:24.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:24.959 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:25.003 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:25.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:25.104 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:25.126 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:25.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:25.151 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:21:25.229 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:25.252 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:25.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:25.353 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:25.375 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:25.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:25.394 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:21:25.394 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:21:25.394 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:21:25.394 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:21:25.394 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:21:25.476 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:25.497 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:25.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:25.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:25.621 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:25.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:25.722 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[36682]: pgmap v1028: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 950 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 14 KiB/s rd, 0 B/s wr, 1.12k op/s; 4926/46173 objects misplaced (10.669%) 2024-09-13T13:21:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[36682]: 3.3 scrub starts 2024-09-13T13:21:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[36682]: 3.3 scrub ok 2024-09-13T13:21:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[36682]: 6.a scrub starts 2024-09-13T13:21:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[36682]: 6.a scrub ok 2024-09-13T13:21:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[36682]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T13:21:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:21:25.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[36682]: from='client.26809 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:21:25.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[36682]: osdmap e503: 8 total, 7 up, 7 in 2024-09-13T13:21:25.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[36682]: 3.7 scrub starts 2024-09-13T13:21:25.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[36682]: 3.7 scrub ok 2024-09-13T13:21:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[29471]: pgmap v1028: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 950 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 14 KiB/s rd, 0 B/s wr, 1.12k op/s; 4926/46173 objects misplaced (10.669%) 2024-09-13T13:21:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[29471]: 3.3 scrub starts 2024-09-13T13:21:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[29471]: 3.3 scrub ok 2024-09-13T13:21:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[29471]: 6.a scrub starts 2024-09-13T13:21:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[29471]: 6.a scrub ok 2024-09-13T13:21:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[29471]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T13:21:25.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:21:25.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[29471]: from='client.26809 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:21:25.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[29471]: osdmap e503: 8 total, 7 up, 7 in 2024-09-13T13:21:25.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[29471]: 3.7 scrub starts 2024-09-13T13:21:25.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:25 smithi052 ceph-mon[29471]: 3.7 scrub ok 2024-09-13T13:21:25.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:25.746 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:25.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:25 smithi167 ceph-mon[33178]: pgmap v1028: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 950 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 14 KiB/s rd, 0 B/s wr, 1.12k op/s; 4926/46173 objects misplaced (10.669%) 2024-09-13T13:21:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:25 smithi167 ceph-mon[33178]: 3.3 scrub starts 2024-09-13T13:21:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:25 smithi167 ceph-mon[33178]: 3.3 scrub ok 2024-09-13T13:21:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:25 smithi167 ceph-mon[33178]: 6.a scrub starts 2024-09-13T13:21:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:25 smithi167 ceph-mon[33178]: 6.a scrub ok 2024-09-13T13:21:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:25 smithi167 ceph-mon[33178]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T13:21:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:25 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:21:25.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:25 smithi167 ceph-mon[33178]: from='client.26809 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:21:25.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:25 smithi167 ceph-mon[33178]: osdmap e503: 8 total, 7 up, 7 in 2024-09-13T13:21:25.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:25 smithi167 ceph-mon[33178]: 3.7 scrub starts 2024-09-13T13:21:25.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:25 smithi167 ceph-mon[33178]: 3.7 scrub ok 2024-09-13T13:21:25.847 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:25.870 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:25.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:25.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:25.972 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:25.995 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:25.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:26.097 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:26.120 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:26.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:26.222 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:26.245 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:26.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:26.320 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:26 smithi167 podman[150878]: 2024-09-13 13:21:26.176950341 +0000 UTC m=+2.664706250 container died a301911ddd7ade05985d64ced1589b4aac4554227de84b8f3568b05b72269575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-13T13:21:26.347 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:26.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T13:21:26.389 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:26.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:26.402 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:21:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:21:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:21:26.491 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:26.529 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:26.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:26.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:26.630 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:26.667 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:26.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:26.682 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:26 smithi167 ceph-mon[33178]: 11.17 scrub starts 2024-09-13T13:21:26.683 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:26 smithi167 ceph-mon[33178]: osdmap e504: 8 total, 7 up, 7 in 2024-09-13T13:21:26.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:26 smithi052 ceph-mon[36682]: 11.17 scrub starts 2024-09-13T13:21:26.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:26 smithi052 ceph-mon[36682]: osdmap e504: 8 total, 7 up, 7 in 2024-09-13T13:21:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:26 smithi052 ceph-mon[29471]: 11.17 scrub starts 2024-09-13T13:21:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:26 smithi052 ceph-mon[29471]: osdmap e504: 8 total, 7 up, 7 in 2024-09-13T13:21:26.769 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:26.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:26.811 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:26.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:26.912 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:26.943 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:26.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:27.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:27.069 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:27.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:27.070 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:26 smithi167 podman[150878]: 2024-09-13 13:21:26.802917955 +0000 UTC m=+3.290673863 container cleanup a301911ddd7ade05985d64ced1589b4aac4554227de84b8f3568b05b72269575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T13:21:27.070 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:26 smithi167 bash[150878]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4 2024-09-13T13:21:27.171 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:27.184 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:21:27.193 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:27.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:27.295 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:27.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:27.328 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:27.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:27.340 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:27 smithi167 podman[151131]: 2024-09-13 13:21:27.071953958 +0000 UTC m=+0.964341402 container remove a301911ddd7ade05985d64ced1589b4aac4554227de84b8f3568b05b72269575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-13T13:21:27.429 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:27.467 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:27.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:27.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:27.569 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:27.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:27.607 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:27.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:27.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:27 smithi167 ceph-mon[33178]: 11.17 scrub ok 2024-09-13T13:21:27.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:27 smithi167 ceph-mon[33178]: pgmap v1031: 168 pgs: 32 stale+active+clean, 3 active+remapped+backfill_toofull, 133 active+clean; 856 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 3927/41643 objects misplaced (9.430%) 2024-09-13T13:21:27.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:27 smithi167 ceph-mon[33178]: 11.15 continuing backfill to osd.1 from (447'5462,502'10045] MIN to 502'10045 2024-09-13T13:21:27.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:27 smithi167 ceph-mon[33178]: 11.15 continuing backfill to osd.2 from (447'5462,502'10045] MIN to 502'10045 2024-09-13T13:21:27.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:27 smithi167 ceph-mon[33178]: 11.13 continuing backfill to osd.0 from (458'6493,502'10121] MIN to 502'10121 2024-09-13T13:21:27.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:27 smithi167 ceph-mon[33178]: 11.13 continuing backfill to osd.5 from (458'6493,502'10121] MIN to 502'10121 2024-09-13T13:21:27.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:27 smithi167 ceph-mon[33178]: 3.18 scrub starts 2024-09-13T13:21:27.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:27 smithi167 ceph-mon[33178]: 3.18 scrub ok 2024-09-13T13:21:27.708 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:27.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[36682]: 11.17 scrub ok 2024-09-13T13:21:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[36682]: pgmap v1031: 168 pgs: 32 stale+active+clean, 3 active+remapped+backfill_toofull, 133 active+clean; 856 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 3927/41643 objects misplaced (9.430%) 2024-09-13T13:21:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[36682]: 11.15 continuing backfill to osd.1 from (447'5462,502'10045] MIN to 502'10045 2024-09-13T13:21:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[36682]: 11.15 continuing backfill to osd.2 from (447'5462,502'10045] MIN to 502'10045 2024-09-13T13:21:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[36682]: 11.13 continuing backfill to osd.0 from (458'6493,502'10121] MIN to 502'10121 2024-09-13T13:21:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[36682]: 11.13 continuing backfill to osd.5 from (458'6493,502'10121] MIN to 502'10121 2024-09-13T13:21:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[36682]: 3.18 scrub starts 2024-09-13T13:21:27.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[36682]: 3.18 scrub ok 2024-09-13T13:21:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[29471]: 11.17 scrub ok 2024-09-13T13:21:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[29471]: pgmap v1031: 168 pgs: 32 stale+active+clean, 3 active+remapped+backfill_toofull, 133 active+clean; 856 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 3927/41643 objects misplaced (9.430%) 2024-09-13T13:21:27.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[29471]: 11.15 continuing backfill to osd.1 from (447'5462,502'10045] MIN to 502'10045 2024-09-13T13:21:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[29471]: 11.15 continuing backfill to osd.2 from (447'5462,502'10045] MIN to 502'10045 2024-09-13T13:21:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[29471]: 11.13 continuing backfill to osd.0 from (458'6493,502'10121] MIN to 502'10121 2024-09-13T13:21:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[29471]: 11.13 continuing backfill to osd.5 from (458'6493,502'10121] MIN to 502'10121 2024-09-13T13:21:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[29471]: 3.18 scrub starts 2024-09-13T13:21:27.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:27 smithi052 ceph-mon[29471]: 3.18 scrub ok 2024-09-13T13:21:27.744 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:27.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:27.768 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:21:27.768 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:21:27.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:21:27.769 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:21:27.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.769 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:133006 10.8 e366/334 10:1945bf5a:::benchmark_data_smithi052_153703_object5626:head v 366'8411, mlcod=366'8409)", 2024-09-13T13:21:27.769 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.291495+0000", 2024-09-13T13:21:27.769 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 348.47472691299998, 2024-09-13T13:21:27.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.36882864700000001, 2024-09-13T13:21:27.769 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.769 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.769 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.769 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291495+0000", 2024-09-13T13:21:27.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291495+0000", 2024-09-13T13:21:27.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291497+0000", 2024-09-13T13:21:27.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4109999999999999e-06 2024-09-13T13:21:27.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291503+0000", 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2190000000000001e-06 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291513+0000", 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0451999999999999e-05 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291520+0000", 2024-09-13T13:21:27.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.759e-06 2024-09-13T13:21:27.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291570+0000", 2024-09-13T13:21:27.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9979999999999999e-05 2024-09-13T13:21:27.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291585+0000", 2024-09-13T13:21:27.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4688999999999999e-05 2024-09-13T13:21:27.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660271+0000", 2024-09-13T13:21:27.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.36868583799999999 2024-09-13T13:21:27.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660324+0000", 2024-09-13T13:21:27.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3298999999999999e-05 2024-09-13T13:21:27.773 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.773 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.773 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.774 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:133009 10.8 e366/334 10:1c6c448b:::benchmark_data_smithi052_153703_object5629:head v 366'8412, mlcod=366'8409)", 2024-09-13T13:21:27.774 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.292418+0000", 2024-09-13T13:21:27.774 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 348.473803938, 2024-09-13T13:21:27.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.380843235, 2024-09-13T13:21:27.774 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.774 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.774 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.774 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292418+0000", 2024-09-13T13:21:27.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292418+0000", 2024-09-13T13:21:27.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292419+0000", 2024-09-13T13:21:27.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.569999999999999e-07 2024-09-13T13:21:27.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292425+0000", 2024-09-13T13:21:27.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2050000000000001e-06 2024-09-13T13:21:27.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660256+0000", 2024-09-13T13:21:27.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.36783123499999998 2024-09-13T13:21:27.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660266+0000", 2024-09-13T13:21:27.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0210000000000001e-05 2024-09-13T13:21:27.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660362+0000", 2024-09-13T13:21:27.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5999e-05 2024-09-13T13:21:27.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660382+0000", 2024-09-13T13:21:27.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0035999999999999e-05 2024-09-13T13:21:27.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.673210+0000", 2024-09-13T13:21:27.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012828381999999999 2024-09-13T13:21:27.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.673262+0000", 2024-09-13T13:21:27.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1211000000000001e-05 2024-09-13T13:21:27.778 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.778 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.778 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.778 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:133029 10.9 e366/334 10:97ab46ca:::benchmark_data_smithi052_153703_object5649:head v 366'8225, mlcod=366'8223)", 2024-09-13T13:21:27.778 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.309291+0000", 2024-09-13T13:21:27.779 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 348.45693155100003, 2024-09-13T13:21:27.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.356083341, 2024-09-13T13:21:27.779 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.779 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.779 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.779 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309291+0000", 2024-09-13T13:21:27.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309291+0000", 2024-09-13T13:21:27.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309293+0000", 2024-09-13T13:21:27.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9889999999999999e-06 2024-09-13T13:21:27.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309303+0000", 2024-09-13T13:21:27.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0162e-05 2024-09-13T13:21:27.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660206+0000", 2024-09-13T13:21:27.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.35090332600000002 2024-09-13T13:21:27.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660227+0000", 2024-09-13T13:21:27.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0778999999999999e-05 2024-09-13T13:21:27.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660332+0000", 2024-09-13T13:21:27.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104756 2024-09-13T13:21:27.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660360+0000", 2024-09-13T13:21:27.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8252999999999999e-05 2024-09-13T13:21:27.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665339+0000", 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049790090000000004 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665374+0000", 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5067000000000002e-05 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:191396 10.9 e382/377 10:9fe2a3d5:::benchmark_data_smithi052_153703_object64016:head v 382'11888, mlcod=382'11886)", 2024-09-13T13:21:27.783 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:34.022668+0000", 2024-09-13T13:21:27.784 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 293.74355453200002, 2024-09-13T13:21:27.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.227577587, 2024-09-13T13:21:27.784 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.785 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.785 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.785 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022668+0000", 2024-09-13T13:21:27.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022668+0000", 2024-09-13T13:21:27.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022669+0000", 2024-09-13T13:21:27.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.519e-06 2024-09-13T13:21:27.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022677+0000", 2024-09-13T13:21:27.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3889999999999999e-06 2024-09-13T13:21:27.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022687+0000", 2024-09-13T13:21:27.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0190999999999999e-05 2024-09-13T13:21:27.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022693+0000", 2024-09-13T13:21:27.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5080000000000002e-06 2024-09-13T13:21:27.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022747+0000", 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3820000000000003e-05 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022769+0000", 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1829000000000001e-05 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.026913+0000", 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041442939999999998 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250245+0000", 2024-09-13T13:21:27.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.22333203700000001 2024-09-13T13:21:27.789 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.789 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.789 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.789 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:191388 10.c e382/334 10:3518f5d7:::benchmark_data_smithi052_153703_object64008:head v 382'11871, mlcod=382'11868)", 2024-09-13T13:21:27.789 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:34.028033+0000", 2024-09-13T13:21:27.789 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 293.73818906899999, 2024-09-13T13:21:27.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.237613568, 2024-09-13T13:21:27.789 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.790 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.790 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.790 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028033+0000", 2024-09-13T13:21:27.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028033+0000", 2024-09-13T13:21:27.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028035+0000", 2024-09-13T13:21:27.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8649999999999999e-06 2024-09-13T13:21:27.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028044+0000", 2024-09-13T13:21:27.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7630000000000006e-06 2024-09-13T13:21:27.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250215+0000", 2024-09-13T13:21:27.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.222171601 2024-09-13T13:21:27.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250226+0000", 2024-09-13T13:21:27.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0395e-05 2024-09-13T13:21:27.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250281+0000", 2024-09-13T13:21:27.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4833999999999997e-05 2024-09-13T13:21:27.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250301+0000", 2024-09-13T13:21:27.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0452999999999999e-05 2024-09-13T13:21:27.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.265605+0000", 2024-09-13T13:21:27.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01530372 2024-09-13T13:21:27.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.265647+0000", 2024-09-13T13:21:27.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1937000000000002e-05 2024-09-13T13:21:27.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.794 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.794 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:191418 10.9 e382/377 10:975b1348:::benchmark_data_smithi052_153703_object64038:head v 382'11889, mlcod=382'11886)", 2024-09-13T13:21:27.794 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:34.041411+0000", 2024-09-13T13:21:27.794 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 293.72481181400002, 2024-09-13T13:21:27.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.214851654, 2024-09-13T13:21:27.794 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.794 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041411+0000", 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041411+0000", 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041413+0000", 2024-09-13T13:21:27.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3819999999999998e-06 2024-09-13T13:21:27.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041424+0000", 2024-09-13T13:21:27.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1031e-05 2024-09-13T13:21:27.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250212+0000", 2024-09-13T13:21:27.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.208787785 2024-09-13T13:21:27.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250227+0000", 2024-09-13T13:21:27.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4887e-05 2024-09-13T13:21:27.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250280+0000", 2024-09-13T13:21:27.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3733999999999997e-05 2024-09-13T13:21:27.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250340+0000", 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9212000000000002e-05 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.256042+0000", 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057022210000000004 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.256262+0000", 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022040199999999999 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.798 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203978 10.9 e385/377 10:9bb10788:::benchmark_data_smithi052_153703_object76598:head v 385'12669, mlcod=385'12664)", 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.815708+0000", 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 281.95051397200001, 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27159179500000002, 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815708+0000", 2024-09-13T13:21:27.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815708+0000", 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815710+0000", 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8840000000000001e-06 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815718+0000", 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815731+0000", 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2853e-05 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815739+0000", 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6550000000000004e-06 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816390+0000", 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00065173800000000001 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816411+0000", 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0727e-05 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087219+0000", 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27080745899999997 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087300+0000", 2024-09-13T13:21:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1477999999999995e-05 2024-09-13T13:21:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203975 10.1 e385/334 10:87de9ad5:::benchmark_data_smithi052_153703_object76595:head v 385'12786, mlcod=385'12784)", 2024-09-13T13:21:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:21:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 281.94944056499997, 2024-09-13T13:21:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27055213299999997, 2024-09-13T13:21:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.804 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.804 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:21:27.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:21:27.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4199999999999995e-07 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816796+0000", 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3298e-05 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816803+0000", 2024-09-13T13:21:27.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8379999999999996e-06 2024-09-13T13:21:27.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816808+0000", 2024-09-13T13:21:27.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2020000000000003e-06 2024-09-13T13:21:27.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079290+0000", 2024-09-13T13:21:27.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.26248217499999998 2024-09-13T13:21:27.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079309+0000", 2024-09-13T13:21:27.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8927e-05 2024-09-13T13:21:27.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087304+0000", 2024-09-13T13:21:27.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079953999999999997 2024-09-13T13:21:27.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087334+0000", 2024-09-13T13:21:27.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9550999999999999e-05 2024-09-13T13:21:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203977 10.c e385/334 10:35b89a8a:::benchmark_data_smithi052_153703_object76597:head v 385'12628, mlcod=385'12626)", 2024-09-13T13:21:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.816846+0000", 2024-09-13T13:21:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 281.94937629899999, 2024-09-13T13:21:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27022401899999998, 2024-09-13T13:21:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816846+0000", 2024-09-13T13:21:27.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816846+0000", 2024-09-13T13:21:27.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816847+0000", 2024-09-13T13:21:27.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9200000000000001e-07 2024-09-13T13:21:27.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816851+0000", 2024-09-13T13:21:27.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1989999999999999e-06 2024-09-13T13:21:27.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816857+0000", 2024-09-13T13:21:27.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9610000000000002e-06 2024-09-13T13:21:27.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816862+0000", 2024-09-13T13:21:27.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8489999999999997e-06 2024-09-13T13:21:27.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816923+0000", 2024-09-13T13:21:27.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1781999999999999e-05 2024-09-13T13:21:27.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816940+0000", 2024-09-13T13:21:27.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6844999999999999e-05 2024-09-13T13:21:27.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087036+0000", 2024-09-13T13:21:27.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27009603100000001 2024-09-13T13:21:27.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087070+0000", 2024-09-13T13:21:27.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3760000000000002e-05 2024-09-13T13:21:27.812 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.812 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.813 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.813 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203979 10.c e385/334 10:3e03f71a:::benchmark_data_smithi052_153703_object76599:head v 385'12629, mlcod=385'12626)", 2024-09-13T13:21:27.813 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.820677+0000", 2024-09-13T13:21:27.813 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 281.94554530099998, 2024-09-13T13:21:27.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.26856709400000001, 2024-09-13T13:21:27.813 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.813 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.813 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.813 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820677+0000", 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820677+0000", 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820678+0000", 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4699999999999999e-06 2024-09-13T13:21:27.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820696+0000", 2024-09-13T13:21:27.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7331999999999999e-05 2024-09-13T13:21:27.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079076+0000", 2024-09-13T13:21:27.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.25837983199999998 2024-09-13T13:21:27.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079091+0000", 2024-09-13T13:21:27.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5832999999999999e-05 2024-09-13T13:21:27.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.081590+0000", 2024-09-13T13:21:27.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002498545 2024-09-13T13:21:27.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.081623+0000", 2024-09-13T13:21:27.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3442000000000003e-05 2024-09-13T13:21:27.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.089192+0000", 2024-09-13T13:21:27.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075682340000000001 2024-09-13T13:21:27.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.089244+0000", 2024-09-13T13:21:27.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2405999999999999e-05 2024-09-13T13:21:27.817 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.817 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.817 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203982 10.9 e385/377 10:90b0e29e:::benchmark_data_smithi052_153703_object76602:head v 385'12670, mlcod=385'12666)", 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.825124+0000", 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 281.94109823100001, 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.26226309199999998, 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825124+0000", 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825124+0000", 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825126+0000", 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.894e-06 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825134+0000", 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1960000000000008e-06 2024-09-13T13:21:27.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079730+0000", 2024-09-13T13:21:27.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.25459629499999997 2024-09-13T13:21:27.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079741+0000", 2024-09-13T13:21:27.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-09-13T13:21:27.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079895+0000", 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154881 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079919+0000", 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3139e-05 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087337+0000", 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.007417964 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087387+0000", 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0612999999999998e-05 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:60365 11.0 e440/423 11:088590ea:::benchmark_data_smithi052_182575_object60364:head v 440'3937, mlcod=440'3934)", 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.475832+0000", 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 174.29038990800001, 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31868556799999997, 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.822 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.823 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475832+0000", 2024-09-13T13:21:27.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475832+0000", 2024-09-13T13:21:27.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475834+0000", 2024-09-13T13:21:27.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.643e-06 2024-09-13T13:21:27.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475869+0000", 2024-09-13T13:21:27.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4582000000000002e-05 2024-09-13T13:21:27.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475879+0000", 2024-09-13T13:21:27.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0244000000000001e-05 2024-09-13T13:21:27.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475886+0000", 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8759999999999999e-06 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475941+0000", 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4975999999999997e-05 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475960+0000", 2024-09-13T13:21:27.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9381999999999998e-05 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480677+0000", 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047167650000000004 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794518+0000", 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31384109999999998 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:60369 11.0 e440/423 11:00a94002:::benchmark_data_smithi052_182575_object60368:head v 440'3938, mlcod=440'3935)", 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.478361+0000", 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 174.287860892, 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31619510000000001, 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478361+0000", 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478361+0000", 2024-09-13T13:21:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478363+0000", 2024-09-13T13:21:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3540000000000001e-06 2024-09-13T13:21:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478408+0000", 2024-09-13T13:21:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5260999999999999e-05 2024-09-13T13:21:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478417+0000", 2024-09-13T13:21:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7690000000000002e-06 2024-09-13T13:21:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478423+0000", 2024-09-13T13:21:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.331e-06 2024-09-13T13:21:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478857+0000", 2024-09-13T13:21:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043396400000000001 2024-09-13T13:21:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478875+0000", 2024-09-13T13:21:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7611000000000001e-05 2024-09-13T13:21:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794529+0000", 2024-09-13T13:21:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31565414899999999 2024-09-13T13:21:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794557+0000", 2024-09-13T13:21:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7661000000000002e-05 2024-09-13T13:21:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:60375 11.5 e440/423 11:ab976586:::benchmark_data_smithi052_182575_object60374:head v 440'3806, mlcod=440'3804)", 2024-09-13T13:21:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.480358+0000", 2024-09-13T13:21:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 174.285863947, 2024-09-13T13:21:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31425154100000002, 2024-09-13T13:21:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480358+0000", 2024-09-13T13:21:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480358+0000", 2024-09-13T13:21:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480360+0000", 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.139e-06 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480381+0000", 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1444e-05 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480390+0000", 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480396+0000", 2024-09-13T13:21:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2389999999999999e-06 2024-09-13T13:21:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480450+0000", 2024-09-13T13:21:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3616000000000003e-05 2024-09-13T13:21:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480467+0000", 2024-09-13T13:21:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7686e-05 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794579+0000", 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31411161500000001 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794610+0000", 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1027e-05 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:60372 11.0 e440/423 11:055cf520:::benchmark_data_smithi052_182575_object60371:head v 440'3939, mlcod=440'3935)", 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.480901+0000", 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 174.28532084099999, 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31806362700000002, 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480901+0000", 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480901+0000", 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480903+0000", 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1969999999999999e-06 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480964+0000", 2024-09-13T13:21:27.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1058999999999999e-05 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794586+0000", 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31362191099999998 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794593+0000", 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7139999999999993e-06 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794668+0000", 2024-09-13T13:21:27.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4833000000000007e-05 2024-09-13T13:21:27.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794690+0000", 2024-09-13T13:21:27.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2263e-05 2024-09-13T13:21:27.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:21:27.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.798931+0000", 2024-09-13T13:21:27.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042401610000000001 2024-09-13T13:21:27.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.798965+0000", 2024-09-13T13:21:27.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4489000000000001e-05 2024-09-13T13:21:27.841 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.841 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.841 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.841 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "replica_scrub(pg: 4.3,from:0'0,to:268'29,epoch:440/61,start:4:c0000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T13:21:27.841 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.481428+0000", 2024-09-13T13:21:27.841 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 174.284794293, 2024-09-13T13:21:27.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.313186768, 2024-09-13T13:21:27.841 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.841 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.481428+0000", 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.481428+0000", 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.481429+0000", 2024-09-13T13:21:27.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.178e-06 2024-09-13T13:21:27.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.481435+0000", 2024-09-13T13:21:27.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6860000000000003e-06 2024-09-13T13:21:27.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794453+0000", 2024-09-13T13:21:27.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31301767800000002 2024-09-13T13:21:27.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794474+0000", 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1761999999999999e-05 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794570+0000", 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6155000000000003e-05 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794584+0000", 2024-09-13T13:21:27.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3613e-05 2024-09-13T13:21:27.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794615+0000", 2024-09-13T13:21:27.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0695999999999999e-05 2024-09-13T13:21:27.845 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.845 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.845 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.845 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95600 11.17 11.4e8f0b17 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:21:27.845 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:06.191050+0000", 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.5751727, 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99082065699999999, 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95600 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191050+0000", 2024-09-13T13:21:27.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191050+0000", 2024-09-13T13:21:27.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191056+0000", 2024-09-13T13:21:27.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6000000000000003e-06 2024-09-13T13:21:27.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191161+0000", 2024-09-13T13:21:27.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104625 2024-09-13T13:21:27.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191171+0000", 2024-09-13T13:21:27.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0596e-05 2024-09-13T13:21:27.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191179+0000", 2024-09-13T13:21:27.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5889999999999997e-06 2024-09-13T13:21:27.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191231+0000", 2024-09-13T13:21:27.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2178000000000001e-05 2024-09-13T13:21:27.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:21:27.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191239+0000", 2024-09-13T13:21:27.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8350000000000004e-06 2024-09-13T13:21:27.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.181852+0000", 2024-09-13T13:21:27.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99061336 2024-09-13T13:21:27.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.181870+0000", 2024-09-13T13:21:27.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873999999999999e-05 2024-09-13T13:21:27.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.850 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.850 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95607 11.17 11.5b4199b7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:21:27.851 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:06.194159+0000", 2024-09-13T13:21:27.851 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.572063719, 2024-09-13T13:21:27.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.988161768, 2024-09-13T13:21:27.851 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.851 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.851 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:21:27.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:21:27.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:21:27.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:27.851 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95607 2024-09-13T13:21:27.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.851 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194159+0000", 2024-09-13T13:21:27.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194159+0000", 2024-09-13T13:21:27.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194160+0000", 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.736e-06 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194226+0000", 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6062999999999994e-05 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194236+0000", 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194242+0000", 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7969999999999997e-06 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194290+0000", 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7939999999999998e-05 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194299+0000", 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.833e-06 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182312+0000", 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98801272299999998 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182320+0000", 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6969999999999999e-06 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95614 11.17 11.ed911577 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:21:27.855 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:06.201993+0000", 2024-09-13T13:21:27.856 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.56422911999999, 2024-09-13T13:21:27.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98036120999999998, 2024-09-13T13:21:27.856 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.856 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.856 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:21:27.856 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:21:27.856 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:21:27.856 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:27.856 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95614 2024-09-13T13:21:27.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.856 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.201993+0000", 2024-09-13T13:21:27.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.201993+0000", 2024-09-13T13:21:27.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.201995+0000", 2024-09-13T13:21:27.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6360000000000001e-06 2024-09-13T13:21:27.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.202054+0000", 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8721999999999998e-05 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.202064+0000", 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9979999999999998e-06 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.202069+0000", 2024-09-13T13:21:27.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1819999999999997e-06 2024-09-13T13:21:27.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.202119+0000", 2024-09-13T13:21:27.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9948999999999999e-05 2024-09-13T13:21:27.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:21:27.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.202127+0000", 2024-09-13T13:21:27.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.028e-06 2024-09-13T13:21:27.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182345+0000", 2024-09-13T13:21:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98021872399999999 2024-09-13T13:21:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182354+0000", 2024-09-13T13:21:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9709999999999993e-06 2024-09-13T13:21:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95625 11.17 11.fe23b37 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:21:27.861 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:06.211655+0000", 2024-09-13T13:21:27.861 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.55456738300001, 2024-09-13T13:21:27.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97072697900000005, 2024-09-13T13:21:27.861 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:21:27.861 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:21:27.861 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:21:27.861 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:21:27.861 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:21:27.861 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:27.861 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95625 2024-09-13T13:21:27.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.861 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:21:27.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211655+0000", 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211655+0000", 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211657+0000", 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0470000000000001e-06 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:21:27.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211700+0000", 2024-09-13T13:21:27.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3359999999999998e-05 2024-09-13T13:21:27.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:21:27.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211713+0000", 2024-09-13T13:21:27.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2322e-05 2024-09-13T13:21:27.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:21:27.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211720+0000", 2024-09-13T13:21:27.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.43e-06 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211780+0000", 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9879e-05 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211803+0000", 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3051999999999999e-05 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:21:27.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182375+0000", 2024-09-13T13:21:27.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97057152000000002 2024-09-13T13:21:27.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:21:27.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:21:27.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:21:27.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182382+0000", 2024-09-13T13:21:27.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3690000000000001e-06 2024-09-13T13:21:27.865 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.865 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.865 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.865 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:21:27.865 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:21:27.865 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:21:27.866 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:27.896 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:27.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:27.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:27.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:28.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:28.034 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:28.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:28.135 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:28.160 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:28.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:28.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:28.262 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:28.288 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:28.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:28.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:28.390 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:28.428 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:21:28.428 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:21:28.435 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:28.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:28.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:28.469 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:21:28.470 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:21:28.507 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:21:28.507 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:21:28.537 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:28.541 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:21:28.541 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:21:28.545 INFO:teuthology.orchestra.run.smithi052.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:21:28.562 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:28.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:28.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:28 smithi167 ceph-mon[33178]: 6.0 scrub starts 2024-09-13T13:21:28.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:28 smithi167 ceph-mon[33178]: 6.0 scrub ok 2024-09-13T13:21:28.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:28 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2496532463' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:28.578 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:21:28.578 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:21:28.612 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:21:28.612 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:21:28.613 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:28 smithi052 ceph-mon[36682]: 6.0 scrub starts 2024-09-13T13:21:28.613 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:28 smithi052 ceph-mon[36682]: 6.0 scrub ok 2024-09-13T13:21:28.613 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:28 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2496532463' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:28.613 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:28 smithi052 ceph-mon[29471]: 6.0 scrub starts 2024-09-13T13:21:28.613 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:28 smithi052 ceph-mon[29471]: 6.0 scrub ok 2024-09-13T13:21:28.613 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:28 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2496532463' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:28.644 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:21:28.644 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:21:28.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:28.704 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:28.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:28.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:28.806 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:28.841 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T13:21:28.844 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:28.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:28.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:28.945 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:28.969 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:28.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:28.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:29.070 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:28 smithi167 podman[151461]: 2024-09-13 13:21:28.880844795 +0000 UTC m=+0.021130215 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:21:29.071 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:29 smithi167 podman[151461]: 2024-09-13 13:21:29.004119584 +0000 UTC m=+0.144405001 container create cc4586c43076d4b842a32502fbe1c75d7df65822c5bbffe5b53315e97a733684 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True) 2024-09-13T13:21:29.071 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:29.114 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:29.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:29.216 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:29.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:29.261 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:29.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:29.362 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:29.400 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:21:29.401 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:21:29.407 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:29 smithi167 podman[151461]: 2024-09-13 13:21:29.290902417 +0000 UTC m=+0.431187837 container init cc4586c43076d4b842a32502fbe1c75d7df65822c5bbffe5b53315e97a733684 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-13T13:21:29.407 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:29 smithi167 podman[151461]: 2024-09-13 13:21:29.295973011 +0000 UTC m=+0.436258427 container start cc4586c43076d4b842a32502fbe1c75d7df65822c5bbffe5b53315e97a733684 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-13T13:21:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:29.425 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:29.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:29.509 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:29.526 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:29.577 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:29.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:29.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:29.652 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:29 smithi052 ceph-mon[29471]: pgmap v1033: 168 pgs: 7 active+undersized, 24 stale+active+clean, 8 active+undersized+degraded, 3 active+remapped+backfill_toofull, 126 active+clean; 815 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 341 B/s rd, 0 op/s; 381/39681 objects degraded (0.960%); 3927/39681 objects misplaced (9.896%) 2024-09-13T13:21:29.652 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:29 smithi052 ceph-mon[29471]: Health check failed: Degraded data redundancy: 381/39681 objects degraded (0.960%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T13:21:29.652 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:29 smithi052 ceph-mon[29471]: 3.1f scrub starts 2024-09-13T13:21:29.652 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:29 smithi052 ceph-mon[29471]: 3.1f scrub ok 2024-09-13T13:21:29.653 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:29 smithi052 ceph-mon[36682]: pgmap v1033: 168 pgs: 7 active+undersized, 24 stale+active+clean, 8 active+undersized+degraded, 3 active+remapped+backfill_toofull, 126 active+clean; 815 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 341 B/s rd, 0 op/s; 381/39681 objects degraded (0.960%); 3927/39681 objects misplaced (9.896%) 2024-09-13T13:21:29.653 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:29 smithi052 ceph-mon[36682]: Health check failed: Degraded data redundancy: 381/39681 objects degraded (0.960%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T13:21:29.653 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:29 smithi052 ceph-mon[36682]: 3.1f scrub starts 2024-09-13T13:21:29.653 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:29 smithi052 ceph-mon[36682]: 3.1f scrub ok 2024-09-13T13:21:29.678 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:29.726 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:29.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:29.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:29.821 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:29 smithi167 podman[151461]: 2024-09-13 13:21:29.407494861 +0000 UTC m=+0.547780283 container attach cc4586c43076d4b842a32502fbe1c75d7df65822c5bbffe5b53315e97a733684 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T13:21:29.821 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:21:29 smithi167 podman[151461]: 2024-09-13 13:21:29.610153593 +0000 UTC m=+0.750439020 container died cc4586c43076d4b842a32502fbe1c75d7df65822c5bbffe5b53315e97a733684 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-13T13:21:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:29 smithi167 ceph-mon[33178]: pgmap v1033: 168 pgs: 7 active+undersized, 24 stale+active+clean, 8 active+undersized+degraded, 3 active+remapped+backfill_toofull, 126 active+clean; 815 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 341 B/s rd, 0 op/s; 381/39681 objects degraded (0.960%); 3927/39681 objects misplaced (9.896%) 2024-09-13T13:21:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:29 smithi167 ceph-mon[33178]: Health check failed: Degraded data redundancy: 381/39681 objects degraded (0.960%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T13:21:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:29 smithi167 ceph-mon[33178]: 3.1f scrub starts 2024-09-13T13:21:29.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:29 smithi167 ceph-mon[33178]: 3.1f scrub ok 2024-09-13T13:21:29.828 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:29.854 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:29.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:29.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:29.868 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T13:21:29.955 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:29.993 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:29.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:29.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:30.095 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:30.120 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:30.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:30.191 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:21:30.191 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:21:30.221 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:30.270 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:30.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:30.372 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:30.374 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.4.service' 2024-09-13T13:21:30.411 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:30.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:30.429 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:21:30.429 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-13T13:21:30.429 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-09-13T13:21:30.429 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi167.front.sepia.ceph.com 2024-09-13T13:21:30.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -n osd.4 --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-09-13T13:21:30.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:30 smithi052 ceph-mon[36682]: 3.2 scrub starts 2024-09-13T13:21:30.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:30 smithi052 ceph-mon[36682]: 3.2 scrub ok 2024-09-13T13:21:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:30 smithi052 ceph-mon[29471]: 3.2 scrub starts 2024-09-13T13:21:30.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:30 smithi052 ceph-mon[29471]: 3.2 scrub ok 2024-09-13T13:21:30.513 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:30.543 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:30.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:30.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:30.645 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:30.671 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:30.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:30.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:30.673 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/osd.4/config 2024-09-13T13:21:30.773 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:30.799 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:30.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:30.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:30.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:30 smithi167 ceph-mon[33178]: 3.2 scrub starts 2024-09-13T13:21:30.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:30 smithi167 ceph-mon[33178]: 3.2 scrub ok 2024-09-13T13:21:30.901 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:30.926 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:30.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:30.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:30.941 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T13:21:30.971 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:31.028 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:31.051 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:31.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:31.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:31.153 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:31.175 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:31.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:31.277 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:31.299 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:31.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:31.401 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:31.402 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:21:31.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:31.425 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:31.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:31.512 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:31 smithi052 ceph-mon[29471]: pgmap v1034: 168 pgs: 1 active+remapped+backfilling, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 576 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 3.5 KiB/s rd, 1 op/s; 2535/28227 objects degraded (8.981%); 2168/28227 objects misplaced (7.681%) 2024-09-13T13:21:31.512 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:31 smithi052 ceph-mon[29471]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:21:31.512 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:31 smithi052 ceph-mon[29471]: osdmap e505: 8 total, 7 up, 7 in 2024-09-13T13:21:31.512 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:31 smithi052 ceph-mon[29471]: 4.1e scrub starts 2024-09-13T13:21:31.512 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:31 smithi052 ceph-mon[29471]: 4.1e scrub ok 2024-09-13T13:21:31.512 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:31 smithi052 ceph-mon[36682]: pgmap v1034: 168 pgs: 1 active+remapped+backfilling, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 576 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 3.5 KiB/s rd, 1 op/s; 2535/28227 objects degraded (8.981%); 2168/28227 objects misplaced (7.681%) 2024-09-13T13:21:31.512 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:31 smithi052 ceph-mon[36682]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:21:31.512 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:31 smithi052 ceph-mon[36682]: osdmap e505: 8 total, 7 up, 7 in 2024-09-13T13:21:31.513 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:31 smithi052 ceph-mon[36682]: 4.1e scrub starts 2024-09-13T13:21:31.513 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:31 smithi052 ceph-mon[36682]: 4.1e scrub ok 2024-09-13T13:21:31.526 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:31.597 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:31.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:31.617 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:31.694 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:31 smithi167 ceph-mon[33178]: pgmap v1034: 168 pgs: 1 active+remapped+backfilling, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 576 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 3.5 KiB/s rd, 1 op/s; 2535/28227 objects degraded (8.981%); 2168/28227 objects misplaced (7.681%) 2024-09-13T13:21:31.694 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:31 smithi167 ceph-mon[33178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:21:31.694 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:31 smithi167 ceph-mon[33178]: osdmap e505: 8 total, 7 up, 7 in 2024-09-13T13:21:31.694 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:31 smithi167 ceph-mon[33178]: 4.1e scrub starts 2024-09-13T13:21:31.694 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:31 smithi167 ceph-mon[33178]: 4.1e scrub ok 2024-09-13T13:21:31.699 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:31.746 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:31.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:31.848 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:31.890 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:31.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:31.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:32.018 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:32.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:32.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:32.120 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:32.143 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:32.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:32.229 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:32.234 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:21:32.235 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:21:32.245 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:32.316 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:32.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:32.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:32.463 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is set 2024-09-13T13:21:32.464 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:32.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:32.566 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:32.587 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:32.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:32.689 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:32.715 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:32.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:32.725 INFO:teuthology.orchestra.run.smithi167.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-13T13:21:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:32 smithi052 ceph-mon[29471]: osdmap e506: 8 total, 7 up, 7 in 2024-09-13T13:21:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:32 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/147826317' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:21:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:32 smithi052 ceph-mon[29471]: from='client.26818 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:21:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:32 smithi052 ceph-mon[29471]: 5.3 scrub starts 2024-09-13T13:21:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:32 smithi052 ceph-mon[29471]: 5.3 scrub ok 2024-09-13T13:21:32.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:32 smithi052 ceph-mon[36682]: osdmap e506: 8 total, 7 up, 7 in 2024-09-13T13:21:32.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:32 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/147826317' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:21:32.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:32 smithi052 ceph-mon[36682]: from='client.26818 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:21:32.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:32 smithi052 ceph-mon[36682]: 5.3 scrub starts 2024-09-13T13:21:32.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:32 smithi052 ceph-mon[36682]: 5.3 scrub ok 2024-09-13T13:21:32.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:32.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:32 smithi167 ceph-mon[33178]: osdmap e506: 8 total, 7 up, 7 in 2024-09-13T13:21:32.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:32 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/147826317' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:21:32.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:32 smithi167 ceph-mon[33178]: from='client.26818 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T13:21:32.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:32 smithi167 ceph-mon[33178]: 5.3 scrub starts 2024-09-13T13:21:32.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:32 smithi167 ceph-mon[33178]: 5.3 scrub ok 2024-09-13T13:21:32.841 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:32.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:32.943 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:32.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:32.987 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:32.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:33.088 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:33.110 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:33.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:33.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:33.212 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:33.239 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:33.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:33.341 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:33.379 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:33.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:33.481 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:33.524 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:33.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:33.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:33.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:33.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:33.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:33.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:33 smithi052 ceph-mon[36682]: 11.15 scrub starts 2024-09-13T13:21:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:33 smithi052 ceph-mon[36682]: pgmap v1037: 168 pgs: 1 active+remapped+backfilling, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 526 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 2.7 KiB/s rd, 0 op/s; 2384/25842 objects degraded (9.225%); 2168/25842 objects misplaced (8.389%) 2024-09-13T13:21:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:33 smithi052 ceph-mon[36682]: 11.15 scrub ok 2024-09-13T13:21:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:33 smithi052 ceph-mon[36682]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:33 smithi052 ceph-mon[36682]: from='client.26818 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:21:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:33 smithi052 ceph-mon[36682]: osdmap e507: 8 total, 7 up, 7 in 2024-09-13T13:21:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:33 smithi052 ceph-mon[29471]: 11.15 scrub starts 2024-09-13T13:21:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:33 smithi052 ceph-mon[29471]: pgmap v1037: 168 pgs: 1 active+remapped+backfilling, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 526 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 2.7 KiB/s rd, 0 op/s; 2384/25842 objects degraded (9.225%); 2168/25842 objects misplaced (8.389%) 2024-09-13T13:21:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:33 smithi052 ceph-mon[29471]: 11.15 scrub ok 2024-09-13T13:21:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:33 smithi052 ceph-mon[29471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:33 smithi052 ceph-mon[29471]: from='client.26818 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:21:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:33 smithi052 ceph-mon[29471]: osdmap e507: 8 total, 7 up, 7 in 2024-09-13T13:21:33.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:33.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:33 smithi167 ceph-mon[33178]: 11.15 scrub starts 2024-09-13T13:21:33.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:33 smithi167 ceph-mon[33178]: pgmap v1037: 168 pgs: 1 active+remapped+backfilling, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 526 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 2.7 KiB/s rd, 0 op/s; 2384/25842 objects degraded (9.225%); 2168/25842 objects misplaced (8.389%) 2024-09-13T13:21:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:33 smithi167 ceph-mon[33178]: 11.15 scrub ok 2024-09-13T13:21:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:33 smithi167 ceph-mon[33178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:33 smithi167 ceph-mon[33178]: from='client.26818 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T13:21:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:33 smithi167 ceph-mon[33178]: osdmap e507: 8 total, 7 up, 7 in 2024-09-13T13:21:33.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:33.847 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:33.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:33.948 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:33.985 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:33.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:34.087 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:34.111 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:34.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:34.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:34.213 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:34.253 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:34.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:34.355 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:34.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:34.400 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:34.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:34.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:34.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:34.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:34.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:34.631 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:34.673 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:34.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:34.775 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:34.816 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:34.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:34.918 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:34.959 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:34.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:35.060 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:35.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:21:34 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:21:35.103 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:35.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:35.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:35.205 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:35.248 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:35.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:35.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:35.350 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:35.395 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:35.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:35.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:35.397 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T13:21:35.475 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:35 smithi052 ceph-mon[36682]: pgmap v1039: 168 pgs: 1 active+remapped+backfilling, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 392 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 3.2 KiB/s rd, 25 KiB/s wr, 1.13k op/s; 1805/19386 objects degraded (9.311%); 2168/19386 objects misplaced (11.183%); 2.3 MiB/s, 35 objects/s recovering 2024-09-13T13:21:35.476 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:35 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:21:35.476 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:35 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:21:35.476 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:35 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 1805/19386 objects degraded (9.311%), 30 pgs degraded (PG_DEGRADED) 2024-09-13T13:21:35.476 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:35.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:35 smithi052 ceph-mon[29471]: pgmap v1039: 168 pgs: 1 active+remapped+backfilling, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 392 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 3.2 KiB/s rd, 25 KiB/s wr, 1.13k op/s; 1805/19386 objects degraded (9.311%); 2168/19386 objects misplaced (11.183%); 2.3 MiB/s, 35 objects/s recovering 2024-09-13T13:21:35.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:35 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:21:35.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:35 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:21:35.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:35 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 1805/19386 objects degraded (9.311%), 30 pgs degraded (PG_DEGRADED) 2024-09-13T13:21:35.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:35.497 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:35.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:35.548 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:35.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:35.649 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:35.720 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:21:35.720 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:21:35.726 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:35.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:35.760 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:21:35.760 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:21:35.803 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:21:35.803 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:21:35.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:35 smithi167 ceph-mon[33178]: pgmap v1039: 168 pgs: 1 active+remapped+backfilling, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 392 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 3.2 KiB/s rd, 25 KiB/s wr, 1.13k op/s; 1805/19386 objects degraded (9.311%); 2168/19386 objects misplaced (11.183%); 2.3 MiB/s, 35 objects/s recovering 2024-09-13T13:21:35.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:35 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:21:35.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:35 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:21:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:35 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 1805/19386 objects degraded (9.311%), 30 pgs degraded (PG_DEGRADED) 2024-09-13T13:21:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:35.828 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:35.839 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:21:35.839 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:21:35.841 INFO:teuthology.orchestra.run.smithi052.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:21:35.874 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:21:35.875 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:21:35.878 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:35.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:35.914 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:35.914 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:21:35.914 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:21:35.945 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:21:35.945 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:21:35.981 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:36.026 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:36.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:36.127 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:36.170 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:36.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:36.272 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:36.316 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:36.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:36.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:36.346 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:21:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:21:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:21:36.418 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:36.443 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:36.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:36.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:36.544 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:36.582 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:36.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:36.621 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:36 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/104183448' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:36.622 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:36 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/104183448' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:21:36.674 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:21:36.683 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:36.763 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:36.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:36.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:36.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:36 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/104183448' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:36.865 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:36.887 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T13:21:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:36.891 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:36.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:36.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:37.017 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:37.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:37.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:37.118 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:37.143 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:37.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:37.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:37.245 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:37.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:37.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:37.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:37.297 INFO:teuthology.orchestra.run.smithi167.stdout:6.f 2024-09-13T13:21:37.297 INFO:teuthology.orchestra.run.smithi167.stdout:6.e 2024-09-13T13:21:37.298 INFO:teuthology.orchestra.run.smithi167.stdout:6.6 2024-09-13T13:21:37.298 INFO:teuthology.orchestra.run.smithi167.stdout:6.2 2024-09-13T13:21:37.298 INFO:teuthology.orchestra.run.smithi167.stdout:6.1e 2024-09-13T13:21:37.298 INFO:teuthology.orchestra.run.smithi167.stdout:6.1a 2024-09-13T13:21:37.298 INFO:teuthology.orchestra.run.smithi167.stdout:6.16 2024-09-13T13:21:37.298 INFO:teuthology.orchestra.run.smithi167.stdout:6.14 2024-09-13T13:21:37.298 INFO:teuthology.orchestra.run.smithi167.stdout:5.f 2024-09-13T13:21:37.298 INFO:teuthology.orchestra.run.smithi167.stdout:5.e 2024-09-13T13:21:37.298 INFO:teuthology.orchestra.run.smithi167.stdout:5.c 2024-09-13T13:21:37.298 INFO:teuthology.orchestra.run.smithi167.stdout:3.b 2024-09-13T13:21:37.298 INFO:teuthology.orchestra.run.smithi167.stdout:5.6 2024-09-13T13:21:37.298 INFO:teuthology.orchestra.run.smithi167.stdout:3.9 2024-09-13T13:21:37.298 INFO:teuthology.orchestra.run.smithi167.stdout:3.8 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:3.1a 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:4.2 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:3.19 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:3.1d 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:3.1b 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:3.14 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:3.13 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:5.18 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:1.3 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:11.14 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:11.13 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:5.0 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:1.5 2024-09-13T13:21:37.299 INFO:teuthology.orchestra.run.smithi167.stdout:3.f 2024-09-13T13:21:37.300 INFO:teuthology.orchestra.run.smithi167.stdout:3.e 2024-09-13T13:21:37.300 INFO:teuthology.orchestra.run.smithi167.stdout:11.17 2024-09-13T13:21:37.300 INFO:teuthology.orchestra.run.smithi167.stdout:6.17 2024-09-13T13:21:37.300 INFO:teuthology.orchestra.run.smithi167.stdout:11.15 2024-09-13T13:21:37.300 INFO:teuthology.orchestra.run.smithi167.stdout:1.2 2024-09-13T13:21:37.300 INFO:teuthology.orchestra.run.smithi167.stdout:4.18 2024-09-13T13:21:37.300 INFO:teuthology.orchestra.run.smithi167.stdout:4.5 2024-09-13T13:21:37.300 INFO:teuthology.orchestra.run.smithi167.stdout:11.18 2024-09-13T13:21:37.300 INFO:teuthology.orchestra.run.smithi167.stdout:11.7 2024-09-13T13:21:37.300 INFO:teuthology.orchestra.run.smithi167.stdout:3.3 2024-09-13T13:21:37.300 INFO:teuthology.orchestra.run.smithi167.stdout:3.11 2024-09-13T13:21:37.300 INFO:teuthology.orchestra.run.smithi167.stdout:11.e 2024-09-13T13:21:37.300 INFO:teuthology.orchestra.run.smithi167.stdout:4.f 2024-09-13T13:21:37.301 INFO:teuthology.orchestra.run.smithi167.stdout:3.6 2024-09-13T13:21:37.301 INFO:teuthology.orchestra.run.smithi167.stdout:3.15 2024-09-13T13:21:37.301 INFO:teuthology.orchestra.run.smithi167.stdout:4.1 2024-09-13T13:21:37.301 INFO:teuthology.orchestra.run.smithi167.stdout:11.d 2024-09-13T13:21:37.301 INFO:teuthology.orchestra.run.smithi167.stdout:4.13 2024-09-13T13:21:37.301 INFO:teuthology.orchestra.run.smithi167.stdout:6.7 2024-09-13T13:21:37.301 INFO:teuthology.orchestra.run.smithi167.stdout:4.1a 2024-09-13T13:21:37.301 INFO:teuthology.orchestra.run.smithi167.stdout:3.a 2024-09-13T13:21:37.301 INFO:teuthology.orchestra.run.smithi167.stdout:4.1b 2024-09-13T13:21:37.301 INFO:teuthology.orchestra.run.smithi167.stdout:6.15 2024-09-13T13:21:37.301 INFO:teuthology.orchestra.run.smithi167.stdout:4.8 2024-09-13T13:21:37.301 INFO:teuthology.orchestra.run.smithi167.stdout:3.1 2024-09-13T13:21:37.301 INFO:teuthology.orchestra.run.smithi167.stdout:5.a 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:5.1b 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:4.9 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:4.c 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:4.b 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:11.c 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:5.1 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:4.1d 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:4.d 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:4.e 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:5.10 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:5.11 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:5.1a 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:6.1d 2024-09-13T13:21:37.302 INFO:teuthology.orchestra.run.smithi167.stdout:5.1c 2024-09-13T13:21:37.303 INFO:teuthology.orchestra.run.smithi167.stdout:6.12 2024-09-13T13:21:37.303 INFO:teuthology.orchestra.run.smithi167.stdout:5.1d 2024-09-13T13:21:37.303 INFO:teuthology.orchestra.run.smithi167.stdout:5.1f 2024-09-13T13:21:37.303 INFO:teuthology.orchestra.run.smithi167.stdout:5.5 2024-09-13T13:21:37.303 INFO:teuthology.orchestra.run.smithi167.stdout:3.1c 2024-09-13T13:21:37.303 INFO:teuthology.orchestra.run.smithi167.stdout:5.9 2024-09-13T13:21:37.313 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Removed 117006 objects 2024-09-13T13:21:37.314 INFO:tasks.radosbench.radosbench.0.smithi052.stdout:Clean up completed and total clean up time :125.418 2024-09-13T13:21:37.323 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:21:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:21:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:21:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:21:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:242835 10.8 10:12074b7c:::benchmark_data_smithi052_153703_object115455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:21:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:17:30.151285+0000", 2024-09-13T13:21:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 247.16966513, 2024-09-13T13:21:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.93503249799999999, 2024-09-13T13:21:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:21:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:21:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 242835 2024-09-13T13:21:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151285+0000", 2024-09-13T13:21:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151285+0000", 2024-09-13T13:21:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151288+0000", 2024-09-13T13:21:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2560000000000001e-06 2024-09-13T13:21:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151294+0000", 2024-09-13T13:21:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1199999999999999e-06 2024-09-13T13:21:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151308+0000", 2024-09-13T13:21:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4329e-05 2024-09-13T13:21:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151316+0000", 2024-09-13T13:21:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5610000000000001e-06 2024-09-13T13:21:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151379+0000", 2024-09-13T13:21:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3212000000000004e-05 2024-09-13T13:21:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151388+0000", 2024-09-13T13:21:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2129999999999997e-06 2024-09-13T13:21:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.081972+0000", 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.93058433200000001 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.082112+0000", 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000139121 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.082242+0000", 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013008599999999999 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.085361+0000", 2024-09-13T13:21:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031196280000000002 2024-09-13T13:21:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.086121+0000", 2024-09-13T13:21:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00076016099999999997 2024-09-13T13:21:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.086161+0000", 2024-09-13T13:21:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9515999999999999e-05 2024-09-13T13:21:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.086270+0000", 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010909 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.086318+0000", 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7873000000000001e-05 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94535 11.6 11:67ee0d1d:::benchmark_data_smithi052_182575_object94534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:21:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.197231+0000", 2024-09-13T13:21:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 153.12371994700001, 2024-09-13T13:21:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99585655699999998, 2024-09-13T13:21:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94535 2024-09-13T13:21:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197231+0000", 2024-09-13T13:21:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197231+0000", 2024-09-13T13:21:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197233+0000", 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6860000000000002e-06 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197299+0000", 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5452999999999998e-05 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197312+0000", 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3123e-05 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197321+0000", 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6689999999999995e-06 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.391516+0000", 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.19419592099999999 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.391528+0000", 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1972e-05 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.193003+0000", 2024-09-13T13:21:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.80147465100000004 2024-09-13T13:21:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.193087+0000", 2024-09-13T13:21:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4081999999999999e-05 2024-09-13T13:21:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94545 11.16 11:6f6df5ad:::benchmark_data_smithi052_182575_object94544:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:21:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.197343+0000", 2024-09-13T13:21:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 153.12360731699999, 2024-09-13T13:21:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.039569797, 2024-09-13T13:21:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94545 2024-09-13T13:21:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.337 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.337 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.337 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197343+0000", 2024-09-13T13:21:37.337 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.337 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.337 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.337 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197343+0000", 2024-09-13T13:21:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197344+0000", 2024-09-13T13:21:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:21:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197395+0000", 2024-09-13T13:21:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1091999999999998e-05 2024-09-13T13:21:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197402+0000", 2024-09-13T13:21:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0489999999999998e-06 2024-09-13T13:21:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.197409+0000", 2024-09-13T13:21:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.46e-06 2024-09-13T13:21:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.392329+0000", 2024-09-13T13:21:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.194920121 2024-09-13T13:21:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.392335+0000", 2024-09-13T13:21:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8070000000000002e-06 2024-09-13T13:21:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.218051+0000", 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.82571548400000006 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.218191+0000", 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000139796 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.218510+0000", 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00031968200000000001 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.231664+0000", 2024-09-13T13:21:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.013153855000000001 2024-09-13T13:21:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.233621+0000", 2024-09-13T13:21:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019570260000000002 2024-09-13T13:21:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.234815+0000", 2024-09-13T13:21:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011941300000000001 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.236861+0000", 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020457180000000002 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.236913+0000", 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1867000000000003e-05 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95596 11.10 11.3871e090 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:21:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:06.189941+0000", 2024-09-13T13:21:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 151.13100929399999, 2024-09-13T13:21:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99412121499999995, 2024-09-13T13:21:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95596 2024-09-13T13:21:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189941+0000", 2024-09-13T13:21:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189941+0000", 2024-09-13T13:21:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189944+0000", 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3209999999999999e-06 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189998+0000", 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4691999999999997e-05 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190017+0000", 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8853999999999999e-05 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190026+0000", 2024-09-13T13:21:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9840000000000007e-06 2024-09-13T13:21:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190300+0000", 2024-09-13T13:21:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027355400000000002 2024-09-13T13:21:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190308+0000", 2024-09-13T13:21:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1629999999999996e-06 2024-09-13T13:21:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184050+0000", 2024-09-13T13:21:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99374201200000001 2024-09-13T13:21:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184062+0000", 2024-09-13T13:21:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2635e-05 2024-09-13T13:21:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95599 11.10 11.86c1c970 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:06.190054+0000", 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 151.130896834, 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99507878000000005, 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95599 2024-09-13T13:21:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190054+0000", 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190054+0000", 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190055+0000", 2024-09-13T13:21:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9299999999999996e-07 2024-09-13T13:21:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190103+0000", 2024-09-13T13:21:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7883999999999998e-05 2024-09-13T13:21:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190111+0000", 2024-09-13T13:21:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9439999999999994e-06 2024-09-13T13:21:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190118+0000", 2024-09-13T13:21:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6209999999999998e-06 2024-09-13T13:21:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190330+0000", 2024-09-13T13:21:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000212062 2024-09-13T13:21:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190335+0000", 2024-09-13T13:21:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.904e-06 2024-09-13T13:21:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185123+0000", 2024-09-13T13:21:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99478760200000005 2024-09-13T13:21:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185133+0000", 2024-09-13T13:21:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8700000000000004e-06 2024-09-13T13:21:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95604 11.19 11:9cb7ac0c:::benchmark_data_smithi052_182575_object95603:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:06.190129+0000", 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 151.13082160799999, 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1591429629999999, 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95604 2024-09-13T13:21:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190129+0000", 2024-09-13T13:21:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190129+0000", 2024-09-13T13:21:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190130+0000", 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0500000000000002e-07 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190186+0000", 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6026000000000002e-05 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190195+0000", 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7199999999999995e-06 2024-09-13T13:21:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.190201+0000", 2024-09-13T13:21:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5930000000000002e-06 2024-09-13T13:21:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.192356+0000", 2024-09-13T13:21:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002155213 2024-09-13T13:21:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.192366+0000", 2024-09-13T13:21:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3109999999999995e-06 2024-09-13T13:21:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.189740+0000", 2024-09-13T13:21:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99737389399999998 2024-09-13T13:21:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.189750+0000", 2024-09-13T13:21:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0695e-05 2024-09-13T13:21:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.336476+0000", 2024-09-13T13:21:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.146725397 2024-09-13T13:21:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.336929+0000", 2024-09-13T13:21:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00045369799999999999 2024-09-13T13:21:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,3,6,7", 2024-09-13T13:21:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.337090+0000", 2024-09-13T13:21:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016081400000000001 2024-09-13T13:21:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.342581+0000", 2024-09-13T13:21:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0054909169999999997 2024-09-13T13:21:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.342596+0000", 2024-09-13T13:21:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4824000000000001e-05 2024-09-13T13:21:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.343084+0000", 2024-09-13T13:21:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00048794499999999999 2024-09-13T13:21:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.348967+0000", 2024-09-13T13:21:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058832540000000001 2024-09-13T13:21:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.349081+0000", 2024-09-13T13:21:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011430999999999999 2024-09-13T13:21:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.349146+0000", 2024-09-13T13:21:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5075999999999998e-05 2024-09-13T13:21:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.349272+0000", 2024-09-13T13:21:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012547099999999999 2024-09-13T13:21:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95598 11.16 11:6bc1a0ca:::benchmark_data_smithi052_182575_object95597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:07.182439+0000", 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 150.13851180699999, 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1598586870000001, 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95598 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182439+0000", 2024-09-13T13:21:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182439+0000", 2024-09-13T13:21:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182442+0000", 2024-09-13T13:21:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9950000000000001e-06 2024-09-13T13:21:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182567+0000", 2024-09-13T13:21:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012562599999999999 2024-09-13T13:21:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182584+0000", 2024-09-13T13:21:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6447e-05 2024-09-13T13:21:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182600+0000", 2024-09-13T13:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6181999999999999e-05 2024-09-13T13:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.186622+0000", 2024-09-13T13:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0040218770000000001 2024-09-13T13:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.186632+0000", 2024-09-13T13:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0027000000000001e-05 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.338172+0000", 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1515402800000001 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for degraded object", 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.338941+0000", 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00076854799999999997 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.342181+0000", 2024-09-13T13:21:37.368 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00323989 2024-09-13T13:21:37.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.368 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.368 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.368 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.342297+0000", 2024-09-13T13:21:37.368 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116815 2024-09-13T13:21:37.368 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.368 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.368 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.368 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.368 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95629 11.11 11:89554284:::benchmark_data_smithi052_182575_object95628:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:21:37.368 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:07.182622+0000", 2024-09-13T13:21:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 150.13832814400001, 2024-09-13T13:21:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1767014709999999, 2024-09-13T13:21:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95629 2024-09-13T13:21:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182622+0000", 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182622+0000", 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182624+0000", 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.246e-06 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182722+0000", 2024-09-13T13:21:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8034000000000004e-05 2024-09-13T13:21:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182733+0000", 2024-09-13T13:21:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1208e-05 2024-09-13T13:21:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182741+0000", 2024-09-13T13:21:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4589999999999998e-06 2024-09-13T13:21:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.189984+0000", 2024-09-13T13:21:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0072423599999999998 2024-09-13T13:21:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.189994+0000", 2024-09-13T13:21:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0319999999999999e-05 2024-09-13T13:21:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.346998+0000", 2024-09-13T13:21:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1570035679999999 2024-09-13T13:21:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.347142+0000", 2024-09-13T13:21:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014469099999999999 2024-09-13T13:21:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,3,6,7", 2024-09-13T13:21:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.347473+0000", 2024-09-13T13:21:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033115599999999999 2024-09-13T13:21:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.349325+0000", 2024-09-13T13:21:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001851927 2024-09-13T13:21:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.356482+0000", 2024-09-13T13:21:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0071570720000000004 2024-09-13T13:21:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.356521+0000", 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8787e-05 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.356707+0000", 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018582000000000001 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.359074+0000", 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023674260000000002 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.375 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.376 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.359189+0000", 2024-09-13T13:21:37.376 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114036 2024-09-13T13:21:37.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.376 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.376 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.376 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.359324+0000", 2024-09-13T13:21:37.376 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013536099999999999 2024-09-13T13:21:37.376 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.376 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.376 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.376 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.376 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:122327 11.19 11:9ce6f140:::benchmark_data_smithi052_182575_object5318:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-13T13:21:37.377 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:38.207433+0000", 2024-09-13T13:21:37.377 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 119.11351770500001, 2024-09-13T13:21:37.377 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.93009936299999996, 2024-09-13T13:21:37.377 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.377 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.377 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.377 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.377 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.377 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.377 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 122327 2024-09-13T13:21:37.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.377 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.377 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.207433+0000", 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.207433+0000", 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.207436+0000", 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.8849999999999999e-06 2024-09-13T13:21:37.378 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.379 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.379 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.379 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.207444+0000", 2024-09-13T13:21:37.379 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0290000000000005e-06 2024-09-13T13:21:37.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.379 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.379 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.379 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.207466+0000", 2024-09-13T13:21:37.379 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.181e-05 2024-09-13T13:21:37.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.379 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.379 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.379 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.207475+0000", 2024-09-13T13:21:37.380 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3619999999999995e-06 2024-09-13T13:21:37.380 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.380 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.380 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.380 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.207543+0000", 2024-09-13T13:21:37.380 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7695999999999997e-05 2024-09-13T13:21:37.380 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.380 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.380 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.380 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.207553+0000", 2024-09-13T13:21:37.380 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0032e-05 2024-09-13T13:21:37.380 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.380 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.381 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.381 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.130530+0000", 2024-09-13T13:21:37.381 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.92297724699999995 2024-09-13T13:21:37.381 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:37.382 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.382 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.382 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.382 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.130635+0000", 2024-09-13T13:21:37.382 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010493999999999999 2024-09-13T13:21:37.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.383 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.383 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:21:37.383 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.130991+0000", 2024-09-13T13:21:37.383 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035577900000000002 2024-09-13T13:21:37.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.383 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.383 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.383 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.134837+0000", 2024-09-13T13:21:37.383 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0038467890000000002 2024-09-13T13:21:37.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.383 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.383 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.384 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.135188+0000", 2024-09-13T13:21:37.384 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00035089600000000001 2024-09-13T13:21:37.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.384 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.384 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.384 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.137147+0000", 2024-09-13T13:21:37.384 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019584630000000001 2024-09-13T13:21:37.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.384 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.384 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.384 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.137486+0000", 2024-09-13T13:21:37.384 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033941400000000001 2024-09-13T13:21:37.385 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.385 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.385 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.385 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.137532+0000", 2024-09-13T13:21:37.385 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6020999999999999e-05 2024-09-13T13:21:37.385 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.385 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.385 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.385 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.385 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.385 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:122335 11.19 11:99606b18:::benchmark_data_smithi052_182575_object5326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-13T13:21:37.385 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:38.214781+0000", 2024-09-13T13:21:37.386 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 119.106170003, 2024-09-13T13:21:37.386 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.92609652399999998, 2024-09-13T13:21:37.386 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.386 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.386 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.386 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.386 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.386 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.386 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 122335 2024-09-13T13:21:37.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.386 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.386 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.386 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.214781+0000", 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.214781+0000", 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.214782+0000", 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.35e-06 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.387 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.388 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.388 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.214786+0000", 2024-09-13T13:21:37.388 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4839999999999998e-06 2024-09-13T13:21:37.388 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.388 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.388 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.388 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.214796+0000", 2024-09-13T13:21:37.388 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1500000000000005e-06 2024-09-13T13:21:37.388 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.388 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.388 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.389 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.214803+0000", 2024-09-13T13:21:37.389 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1910000000000003e-06 2024-09-13T13:21:37.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.389 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.389 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.389 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.221713+0000", 2024-09-13T13:21:37.389 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0069102249999999999 2024-09-13T13:21:37.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.389 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.389 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.389 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.221721+0000", 2024-09-13T13:21:37.389 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3049999999999993e-06 2024-09-13T13:21:37.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.132705+0000", 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.910983765 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.133015+0000", 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00031022800000000001 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.133331+0000", 2024-09-13T13:21:37.390 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00031620800000000001 2024-09-13T13:21:37.391 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.391 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.391 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.391 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.137106+0000", 2024-09-13T13:21:37.391 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037748479999999999 2024-09-13T13:21:37.391 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.391 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.391 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.391 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.139214+0000", 2024-09-13T13:21:37.391 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0021077600000000002 2024-09-13T13:21:37.391 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.391 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.391 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.392 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.140800+0000", 2024-09-13T13:21:37.392 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0015857709999999999 2024-09-13T13:21:37.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.392 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.392 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.392 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.140847+0000", 2024-09-13T13:21:37.392 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7070000000000002e-05 2024-09-13T13:21:37.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.392 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.392 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.392 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.140877+0000", 2024-09-13T13:21:37.392 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0168999999999999e-05 2024-09-13T13:21:37.393 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.393 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.393 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.393 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.393 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.393 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:122433 11.19 11:9f8d80da:::benchmark_data_smithi052_182575_object5424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-13T13:21:37.393 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:38.308223+0000", 2024-09-13T13:21:37.393 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 119.012727531, 2024-09-13T13:21:37.393 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.83791811599999999, 2024-09-13T13:21:37.393 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.393 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.393 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.393 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 122433 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.308223+0000", 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.308223+0000", 2024-09-13T13:21:37.394 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.395 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.395 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.395 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.395 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.308224+0000", 2024-09-13T13:21:37.395 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5600000000000004e-07 2024-09-13T13:21:37.395 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.395 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.395 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.395 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.308228+0000", 2024-09-13T13:21:37.395 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1180000000000002e-06 2024-09-13T13:21:37.395 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.395 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.395 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.308235+0000", 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4599999999999997e-06 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.308242+0000", 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1290000000000001e-06 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.308325+0000", 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3021000000000004e-05 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.396 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.397 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.397 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.308333+0000", 2024-09-13T13:21:37.397 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8200000000000003e-06 2024-09-13T13:21:37.397 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.397 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.397 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.397 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.137563+0000", 2024-09-13T13:21:37.397 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.82922945000000003 2024-09-13T13:21:37.397 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.397 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.397 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.397 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.137665+0000", 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.137883+0000", 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000218402 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.141010+0000", 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031268020000000001 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.398 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.399 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.141159+0000", 2024-09-13T13:21:37.399 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000149383 2024-09-13T13:21:37.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.399 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.399 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.399 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.146013+0000", 2024-09-13T13:21:37.399 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048533450000000002 2024-09-13T13:21:37.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.399 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.399 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.399 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.146090+0000", 2024-09-13T13:21:37.399 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7496000000000005e-05 2024-09-13T13:21:37.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.400 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.400 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.400 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.146141+0000", 2024-09-13T13:21:37.400 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0834000000000002e-05 2024-09-13T13:21:37.400 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.400 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.400 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.400 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.400 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.400 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:122434 11.19 11:9beea776:::benchmark_data_smithi052_182575_object5425:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-13T13:21:37.400 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:38.309380+0000", 2024-09-13T13:21:37.400 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 119.011570564, 2024-09-13T13:21:37.400 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.83694504700000005, 2024-09-13T13:21:37.401 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.401 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.401 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.401 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.401 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.401 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.401 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 122434 2024-09-13T13:21:37.401 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.401 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.401 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.401 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.401 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.309380+0000", 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.309380+0000", 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.309381+0000", 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4330000000000001e-06 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.402 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.403 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.309386+0000", 2024-09-13T13:21:37.403 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.972e-06 2024-09-13T13:21:37.403 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.403 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.403 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.403 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.309398+0000", 2024-09-13T13:21:37.403 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1454000000000001e-05 2024-09-13T13:21:37.403 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.403 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.403 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.403 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.309405+0000", 2024-09-13T13:21:37.403 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:21:37.403 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.404 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.404 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.404 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.309462+0000", 2024-09-13T13:21:37.404 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7188000000000002e-05 2024-09-13T13:21:37.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.404 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.404 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.404 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.309470+0000", 2024-09-13T13:21:37.404 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8730000000000007e-06 2024-09-13T13:21:37.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.404 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.404 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.404 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.139245+0000", 2024-09-13T13:21:37.405 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.829775233 2024-09-13T13:21:37.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.405 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.405 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.405 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.139350+0000", 2024-09-13T13:21:37.405 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104821 2024-09-13T13:21:37.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.405 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.405 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,7", 2024-09-13T13:21:37.405 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.139575+0000", 2024-09-13T13:21:37.405 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000224811 2024-09-13T13:21:37.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.144476+0000", 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.004901697 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.144496+0000", 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9162999999999999e-05 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.146222+0000", 2024-09-13T13:21:37.406 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0017262149999999999 2024-09-13T13:21:37.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.407 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.407 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.407 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.146279+0000", 2024-09-13T13:21:37.407 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6962000000000001e-05 2024-09-13T13:21:37.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.407 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.407 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.407 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:39.146325+0000", 2024-09-13T13:21:37.407 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6218999999999999e-05 2024-09-13T13:21:37.407 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.407 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.407 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.408 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.408 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:150237 11.19 11:9e97d7b7:::benchmark_data_smithi052_182575_object33228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-13T13:21:37.408 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:13.106475+0000", 2024-09-13T13:21:37.408 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 84.214476008000005, 2024-09-13T13:21:37.408 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.035596205, 2024-09-13T13:21:37.408 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.408 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.408 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:37.408 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.408 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.408 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.408 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 150237 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.106475+0000", 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.106475+0000", 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.409 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.106477+0000", 2024-09-13T13:21:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2110000000000001e-06 2024-09-13T13:21:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.106483+0000", 2024-09-13T13:21:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1689999999999997e-06 2024-09-13T13:21:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.106495+0000", 2024-09-13T13:21:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2238e-05 2024-09-13T13:21:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.106503+0000", 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6890000000000004e-06 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.106617+0000", 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114066 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.106747+0000", 2024-09-13T13:21:37.411 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013031300000000001 2024-09-13T13:21:37.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.412 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.412 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2,7", 2024-09-13T13:21:37.412 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.107165+0000", 2024-09-13T13:21:37.412 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00041814700000000001 2024-09-13T13:21:37.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.412 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.412 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.412 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.111525+0000", 2024-09-13T13:21:37.412 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0043591669999999997 2024-09-13T13:21:37.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.412 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.412 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.413 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.122702+0000", 2024-09-13T13:21:37.413 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011177817 2024-09-13T13:21:37.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.413 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.413 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.413 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.122713+0000", 2024-09-13T13:21:37.413 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0502e-05 2024-09-13T13:21:37.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.413 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.413 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.413 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.131372+0000", 2024-09-13T13:21:37.413 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.008659467 2024-09-13T13:21:37.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.414 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.414 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for missing object", 2024-09-13T13:21:37.414 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.131583+0000", 2024-09-13T13:21:37.414 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021027700000000001 2024-09-13T13:21:37.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.414 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.414 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.414 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.141866+0000", 2024-09-13T13:21:37.414 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010283044 2024-09-13T13:21:37.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.414 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.414 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.414 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.142071+0000", 2024-09-13T13:21:37.415 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00020509799999999999 2024-09-13T13:21:37.415 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.415 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.415 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.415 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.415 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:150266 11.19 11:9982d1e6:::benchmark_data_smithi052_182575_object33257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-13T13:21:37.415 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:13.139629+0000", 2024-09-13T13:21:37.415 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 84.181321522000005, 2024-09-13T13:21:37.415 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0236999449999999, 2024-09-13T13:21:37.415 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.415 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.416 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.416 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.416 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.416 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.416 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 150266 2024-09-13T13:21:37.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.416 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.416 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.416 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.416 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.139629+0000", 2024-09-13T13:21:37.416 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.416 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.417 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.417 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.139629+0000", 2024-09-13T13:21:37.417 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.417 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.417 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.417 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.139632+0000", 2024-09-13T13:21:37.417 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7949999999999999e-06 2024-09-13T13:21:37.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.417 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.417 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.417 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.139640+0000", 2024-09-13T13:21:37.417 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6660000000000008e-06 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.139658+0000", 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8416000000000001e-05 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.139667+0000", 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5179999999999993e-06 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.418 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.139946+0000", 2024-09-13T13:21:37.419 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027899100000000003 2024-09-13T13:21:37.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.419 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.419 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.419 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.139956+0000", 2024-09-13T13:21:37.419 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5799999999999998e-06 2024-09-13T13:21:37.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.419 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.419 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.419 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.133086+0000", 2024-09-13T13:21:37.419 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99312970300000003 2024-09-13T13:21:37.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.419 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.420 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.420 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.133211+0000", 2024-09-13T13:21:37.420 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012532099999999999 2024-09-13T13:21:37.420 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.420 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.420 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:21:37.420 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.133600+0000", 2024-09-13T13:21:37.420 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00038862499999999997 2024-09-13T13:21:37.420 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.420 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.420 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.420 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.141315+0000", 2024-09-13T13:21:37.420 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0077154420000000003 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.142107+0000", 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00079142600000000002 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163150+0000", 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.021043646999999999 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.421 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163209+0000", 2024-09-13T13:21:37.422 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8570999999999999e-05 2024-09-13T13:21:37.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.422 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.422 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.422 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163329+0000", 2024-09-13T13:21:37.422 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000120244 2024-09-13T13:21:37.422 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.422 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.422 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.422 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.422 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:150313 11.19 11:9ba35f01:::benchmark_data_smithi052_182575_object33304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-13T13:21:37.422 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:13.186201+0000", 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 84.134749601999999, 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97728062800000004, 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 150313 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.423 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.186201+0000", 2024-09-13T13:21:37.424 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.424 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.424 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.424 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.186201+0000", 2024-09-13T13:21:37.424 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.424 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.424 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.424 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.186203+0000", 2024-09-13T13:21:37.424 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0789999999999999e-06 2024-09-13T13:21:37.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.424 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.186209+0000", 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6219999999999997e-06 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.186222+0000", 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3509000000000001e-05 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.186230+0000", 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.588e-06 2024-09-13T13:21:37.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.426 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.426 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.426 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.186290+0000", 2024-09-13T13:21:37.426 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9895000000000002e-05 2024-09-13T13:21:37.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.426 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.426 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.426 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.186298+0000", 2024-09-13T13:21:37.426 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8839999999999994e-06 2024-09-13T13:21:37.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.426 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.426 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.426 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.135735+0000", 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.949437737 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.135854+0000", 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118874 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.136189+0000", 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033446600000000001 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.427 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.428 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.142144+0000", 2024-09-13T13:21:37.428 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0059553970000000003 2024-09-13T13:21:37.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.428 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.428 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.428 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.142181+0000", 2024-09-13T13:21:37.428 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6656999999999998e-05 2024-09-13T13:21:37.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.428 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.428 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.428 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163344+0000", 2024-09-13T13:21:37.428 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.021163641 2024-09-13T13:21:37.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.429 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.429 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.429 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163389+0000", 2024-09-13T13:21:37.429 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4926999999999999e-05 2024-09-13T13:21:37.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.429 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.429 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.429 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163482+0000", 2024-09-13T13:21:37.429 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2351999999999999e-05 2024-09-13T13:21:37.429 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.429 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.429 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.430 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.430 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:150329 11.19 11:9e6169b3:::benchmark_data_smithi052_182575_object33320:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-13T13:21:37.430 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:13.200154+0000", 2024-09-13T13:21:37.430 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 84.120796139999996, 2024-09-13T13:21:37.430 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96347059000000002, 2024-09-13T13:21:37.430 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.430 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.430 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.430 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.430 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.430 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.430 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 150329 2024-09-13T13:21:37.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.200154+0000", 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.200154+0000", 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.431 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.200156+0000", 2024-09-13T13:21:37.432 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9929999999999998e-06 2024-09-13T13:21:37.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.432 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.432 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.432 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.200162+0000", 2024-09-13T13:21:37.432 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8579999999999996e-06 2024-09-13T13:21:37.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.432 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.432 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.432 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.200235+0000", 2024-09-13T13:21:37.432 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3138000000000003e-05 2024-09-13T13:21:37.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.432 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.433 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.433 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.200245+0000", 2024-09-13T13:21:37.433 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9599999999999995e-06 2024-09-13T13:21:37.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.433 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.433 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.433 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.200306+0000", 2024-09-13T13:21:37.433 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0399999999999998e-05 2024-09-13T13:21:37.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.433 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.433 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.433 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.200314+0000", 2024-09-13T13:21:37.433 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6119999999999999e-06 2024-09-13T13:21:37.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.434 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.434 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.434 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.138578+0000", 2024-09-13T13:21:37.434 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.93826384699999998 2024-09-13T13:21:37.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.434 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.434 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.434 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.138703+0000", 2024-09-13T13:21:37.434 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000124437 2024-09-13T13:21:37.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.434 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.434 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:21:37.435 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.139154+0000", 2024-09-13T13:21:37.435 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000451574 2024-09-13T13:21:37.435 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.435 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.435 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.435 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.146682+0000", 2024-09-13T13:21:37.435 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0075273689999999999 2024-09-13T13:21:37.435 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.435 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.435 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.435 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.146718+0000", 2024-09-13T13:21:37.435 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.5984e-05 2024-09-13T13:21:37.435 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163493+0000", 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.016775314 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163536+0000", 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3367e-05 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163625+0000", 2024-09-13T13:21:37.436 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8737000000000001e-05 2024-09-13T13:21:37.437 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.437 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.437 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.437 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.437 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.437 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:150384 11.19 11:99d9a282:::benchmark_data_smithi052_182575_object33375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-13T13:21:37.437 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:13.256205+0000", 2024-09-13T13:21:37.437 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 84.064745621, 2024-09-13T13:21:37.437 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.907557909, 2024-09-13T13:21:37.437 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.437 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.437 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 150384 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.256205+0000", 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.438 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.256205+0000", 2024-09-13T13:21:37.439 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.439 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.439 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.439 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.256207+0000", 2024-09-13T13:21:37.439 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0770000000000002e-06 2024-09-13T13:21:37.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.439 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.439 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.439 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.256213+0000", 2024-09-13T13:21:37.439 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6219999999999997e-06 2024-09-13T13:21:37.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.439 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.440 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.440 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.256224+0000", 2024-09-13T13:21:37.440 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1199000000000001e-05 2024-09-13T13:21:37.440 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.440 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.440 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.440 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.256231+0000", 2024-09-13T13:21:37.440 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:37.441 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1679999999999998e-06 2024-09-13T13:21:37.441 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.441 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.441 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.441 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.258194+0000", 2024-09-13T13:21:37.441 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019626499999999998 2024-09-13T13:21:37.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.442 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.442 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.442 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.258201+0000", 2024-09-13T13:21:37.442 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2439999999999996e-06 2024-09-13T13:21:37.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.442 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.442 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.442 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.142209+0000", 2024-09-13T13:21:37.442 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.88400794999999999 2024-09-13T13:21:37.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.442 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.442 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.443 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.142310+0000", 2024-09-13T13:21:37.443 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010079800000000001 2024-09-13T13:21:37.443 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.443 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.443 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:21:37.443 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.142676+0000", 2024-09-13T13:21:37.443 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00036680399999999998 2024-09-13T13:21:37.443 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.443 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.443 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.443 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.156008+0000", 2024-09-13T13:21:37.443 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.013331185000000001 2024-09-13T13:21:37.443 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.444 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.444 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.444 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.161653+0000", 2024-09-13T13:21:37.444 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0056455769999999997 2024-09-13T13:21:37.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.444 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.444 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.444 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163632+0000", 2024-09-13T13:21:37.444 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019784020000000002 2024-09-13T13:21:37.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.444 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.444 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.444 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163674+0000", 2024-09-13T13:21:37.445 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2794999999999999e-05 2024-09-13T13:21:37.445 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.445 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.445 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.445 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163763+0000", 2024-09-13T13:21:37.445 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8437999999999996e-05 2024-09-13T13:21:37.445 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.445 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.445 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.445 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.445 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.445 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:150401 11.19 11:9e3e0fa4:::benchmark_data_smithi052_182575_object33392:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-13T13:21:37.445 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:13.276657+0000", 2024-09-13T13:21:37.446 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 84.044293585000005, 2024-09-13T13:21:37.446 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.88724545300000002, 2024-09-13T13:21:37.446 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.446 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.446 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.446 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.446 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.446 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.446 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 150401 2024-09-13T13:21:37.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.446 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.446 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.446 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.447 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.276657+0000", 2024-09-13T13:21:37.447 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.447 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.447 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.447 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.276657+0000", 2024-09-13T13:21:37.447 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.447 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.447 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.447 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.276659+0000", 2024-09-13T13:21:37.447 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:21:37.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.448 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.448 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.448 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.276665+0000", 2024-09-13T13:21:37.448 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8669999999999999e-06 2024-09-13T13:21:37.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.448 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.448 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.448 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.276679+0000", 2024-09-13T13:21:37.448 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4079e-05 2024-09-13T13:21:37.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.448 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.448 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.448 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.276686+0000", 2024-09-13T13:21:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1430000000000001e-06 2024-09-13T13:21:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.277706+0000", 2024-09-13T13:21:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0010203300000000001 2024-09-13T13:21:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.277717+0000", 2024-09-13T13:21:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0365e-05 2024-09-13T13:21:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.449 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.144509+0000", 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.86679229599999996 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.144628+0000", 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118767 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.144914+0000", 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00028597200000000001 2024-09-13T13:21:37.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.451 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.451 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.451 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.156048+0000", 2024-09-13T13:21:37.451 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011134732 2024-09-13T13:21:37.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.451 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.451 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.451 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.161690+0000", 2024-09-13T13:21:37.451 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0056418229999999998 2024-09-13T13:21:37.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.451 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.451 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.451 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163769+0000", 2024-09-13T13:21:37.452 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020791579999999998 2024-09-13T13:21:37.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.452 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.452 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.452 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163812+0000", 2024-09-13T13:21:37.452 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2460999999999999e-05 2024-09-13T13:21:37.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.452 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.452 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.452 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163902+0000", 2024-09-13T13:21:37.452 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0629000000000003e-05 2024-09-13T13:21:37.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.452 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.453 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.453 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.453 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:150421 11.19 11:9c67065e:::benchmark_data_smithi052_182575_object33412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-13T13:21:37.453 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:13.298266+0000", 2024-09-13T13:21:37.453 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 84.022684373000004, 2024-09-13T13:21:37.453 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.86580449199999998, 2024-09-13T13:21:37.453 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.453 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.453 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.453 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.453 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.453 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.454 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 150421 2024-09-13T13:21:37.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.454 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.454 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.454 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.454 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.298266+0000", 2024-09-13T13:21:37.454 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.454 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.454 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.454 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.298266+0000", 2024-09-13T13:21:37.454 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.298268+0000", 2024-09-13T13:21:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.187e-06 2024-09-13T13:21:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.298275+0000", 2024-09-13T13:21:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9450000000000004e-06 2024-09-13T13:21:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.298291+0000", 2024-09-13T13:21:37.456 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5668e-05 2024-09-13T13:21:37.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.456 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.456 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.298301+0000", 2024-09-13T13:21:37.456 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5789999999999993e-06 2024-09-13T13:21:37.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.456 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.456 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.298365+0000", 2024-09-13T13:21:37.456 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4247999999999999e-05 2024-09-13T13:21:37.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.298375+0000", 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.025e-05 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.146858+0000", 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.84848339500000003 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.146971+0000", 2024-09-13T13:21:37.457 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011215899999999999 2024-09-13T13:21:37.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.458 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.458 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:21:37.458 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.147311+0000", 2024-09-13T13:21:37.458 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033991400000000003 2024-09-13T13:21:37.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.458 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.458 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.458 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.156092+0000", 2024-09-13T13:21:37.458 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0087811050000000009 2024-09-13T13:21:37.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.458 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.458 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.459 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.161723+0000", 2024-09-13T13:21:37.459 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0056316259999999998 2024-09-13T13:21:37.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.459 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.459 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163911+0000", 2024-09-13T13:21:37.459 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002187853 2024-09-13T13:21:37.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.459 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.459 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.163971+0000", 2024-09-13T13:21:37.459 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0093999999999998e-05 2024-09-13T13:21:37.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.460 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.460 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.460 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.164071+0000", 2024-09-13T13:21:37.460 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9469000000000006e-05 2024-09-13T13:21:37.460 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.460 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.460 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.460 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.460 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:150440 11.19 11:9e143dd5:::benchmark_data_smithi052_182575_object33431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-13T13:21:37.460 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:13.321420+0000", 2024-09-13T13:21:37.460 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 83.999530226000005, 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.84280733100000005, 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 150440 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.321420+0000", 2024-09-13T13:21:37.461 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.462 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.462 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:37.462 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.321420+0000", 2024-09-13T13:21:37.462 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:37.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.462 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.462 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:37.462 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.321422+0000", 2024-09-13T13:21:37.462 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5770000000000001e-06 2024-09-13T13:21:37.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.462 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.462 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.321427+0000", 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3990000000000003e-06 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.321437+0000", 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0149999999999999e-05 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.321445+0000", 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3080000000000002e-06 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.324355+0000", 2024-09-13T13:21:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0029097329999999999 2024-09-13T13:21:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:13.324364+0000", 2024-09-13T13:21:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6840000000000008e-06 2024-09-13T13:21:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.149657+0000", 2024-09-13T13:21:37.464 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.82529247699999997 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.149773+0000", 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116601 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.150294+0000", 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00052083399999999997 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.465 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.156748+0000", 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064540939999999996 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.161756+0000", 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.005008223 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.164081+0000", 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0023248460000000002 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:37.467 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.164133+0000", 2024-09-13T13:21:37.467 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1266e-05 2024-09-13T13:21:37.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:37.467 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:37.467 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:37.467 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:14.164228+0000", 2024-09-13T13:21:37.467 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5138999999999998e-05 2024-09-13T13:21:37.467 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.467 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.467 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.467 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:37.467 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:37.467 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:21:37.490 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:37.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:37.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:37.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:37 smithi167 ceph-mon[33178]: pgmap v1040: 168 pgs: 1 active+clean+remapped, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 274 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 1204/13722 objects degraded (8.774%); 721/13722 objects misplaced (5.254%) 2024-09-13T13:21:37.592 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:37.613 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:37.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:37.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:37.715 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:37.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:37.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:37.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:37.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:37 smithi052 ceph-mon[36682]: pgmap v1040: 168 pgs: 1 active+clean+remapped, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 274 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 1204/13722 objects degraded (8.774%); 721/13722 objects misplaced (5.254%) 2024-09-13T13:21:37.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:37 smithi052 ceph-mon[29471]: pgmap v1040: 168 pgs: 1 active+clean+remapped, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 274 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 1204/13722 objects degraded (8.774%); 721/13722 objects misplaced (5.254%) 2024-09-13T13:21:37.838 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:37.860 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:37.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:37.962 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:37.988 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:37.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:38.090 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:38.112 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:38.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:38.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:38.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:38.239 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:38.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:38.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:38.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -n osd.4 --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op export --pgid 6.16 --file /var/log/ceph/exp.6.16.4 2024-09-13T13:21:38.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:38.361 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:38.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:38.463 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:38.485 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:38.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:38.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:38.489 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/osd.4/config 2024-09-13T13:21:38.587 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:38.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:38.609 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:38.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:38.675 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:21:38.711 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:38.743 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:38.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:38.845 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:38.867 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:38.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:38.968 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:38.989 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:38.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:38.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:39.091 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:39.115 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:39.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:39.216 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:39.239 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:39.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:39.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:39.366 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:39.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:39.467 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:39.492 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:39.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:39.593 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:39.615 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:39.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:39.651 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:39 smithi167 ceph-mon[33178]: pgmap v1041: 168 pgs: 1 active+clean+remapped, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 218 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 669 B/s rd, 0 op/s; 1025/11046 objects degraded (9.279%); 721/11046 objects misplaced (6.527%) 2024-09-13T13:21:39.690 INFO:teuthology.orchestra.run.smithi167.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-13T13:21:39.717 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:39.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:39 smithi052 ceph-mon[36682]: pgmap v1041: 168 pgs: 1 active+clean+remapped, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 218 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 669 B/s rd, 0 op/s; 1025/11046 objects degraded (9.279%); 721/11046 objects misplaced (6.527%) 2024-09-13T13:21:39.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:39 smithi052 ceph-mon[29471]: pgmap v1041: 168 pgs: 1 active+clean+remapped, 42 active+undersized, 30 active+undersized+degraded, 2 active+remapped+backfill_toofull, 93 active+clean; 218 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 669 B/s rd, 0 op/s; 1025/11046 objects degraded (9.279%); 721/11046 objects misplaced (6.527%) 2024-09-13T13:21:39.744 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:39.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:39.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:39.846 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:39.872 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:39.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:39.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:39.974 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:39.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:39.997 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:39.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:40.098 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:40.125 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:40.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:40.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:40.227 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:40.253 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:40.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:40.330 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T13:21:40.355 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:40.379 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:40.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:40.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:40.481 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:40.503 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:40.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:40.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:40.546 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-13T13:21:40.546 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-13T13:21:40.569 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:40 smithi052 ceph-mon[36682]: 11.10 scrub starts 2024-09-13T13:21:40.569 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:40 smithi052 ceph-mon[36682]: osdmap e508: 8 total, 7 up, 7 in 2024-09-13T13:21:40.570 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:40 smithi052 ceph-mon[29471]: 11.10 scrub starts 2024-09-13T13:21:40.570 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:40 smithi052 ceph-mon[29471]: osdmap e508: 8 total, 7 up, 7 in 2024-09-13T13:21:40.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:40 smithi167 ceph-mon[33178]: 11.10 scrub starts 2024-09-13T13:21:40.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:40 smithi167 ceph-mon[33178]: osdmap e508: 8 total, 7 up, 7 in 2024-09-13T13:21:40.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:40.629 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:40.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:40.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:40.653 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:40.699 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:40.732 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:40.754 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:40.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:40.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:40.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:40.879 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:40.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:40.980 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:41.001 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:41.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:41.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:41.103 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:41.125 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:41.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:41.226 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:41.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:41.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:41.350 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:41.368 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is set 2024-09-13T13:21:41.377 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:41.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:41.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:41.420 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:41 smithi052 ceph-mon[36682]: pgmap v1043: 168 pgs: 1 active+clean+remapped, 1 peering, 45 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 93 active+clean; 13 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 117 op/s; 117/1182 objects degraded (9.898%) 2024-09-13T13:21:41.420 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:41 smithi052 ceph-mon[36682]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:21:41.420 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:41 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 117/1182 objects degraded (9.898%), 27 pgs degraded (PG_DEGRADED) 2024-09-13T13:21:41.420 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:41 smithi052 ceph-mon[36682]: osdmap e509: 8 total, 7 up, 7 in 2024-09-13T13:21:41.420 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:41 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2007725751' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:41.420 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:41 smithi052 ceph-mon[36682]: 11.10 scrub starts 2024-09-13T13:21:41.420 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:41 smithi052 ceph-mon[29471]: pgmap v1043: 168 pgs: 1 active+clean+remapped, 1 peering, 45 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 93 active+clean; 13 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 117 op/s; 117/1182 objects degraded (9.898%) 2024-09-13T13:21:41.421 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:41 smithi052 ceph-mon[29471]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:21:41.421 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:41 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 117/1182 objects degraded (9.898%), 27 pgs degraded (PG_DEGRADED) 2024-09-13T13:21:41.421 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:41 smithi052 ceph-mon[29471]: osdmap e509: 8 total, 7 up, 7 in 2024-09-13T13:21:41.421 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:41 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2007725751' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:41.421 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:41 smithi052 ceph-mon[29471]: 11.10 scrub starts 2024-09-13T13:21:41.435 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:21:41.436 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:21:41.452 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:21:41.452 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:21:41.469 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:21:41.469 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:21:41.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:41.485 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:21:41.486 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:21:41.486 INFO:teuthology.orchestra.run.smithi052.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:21:41.499 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:41.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:41.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:41.504 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:21:41.505 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:21:41.522 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:21:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:21:41.541 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:21:41.541 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:21:41.602 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:41.624 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:41.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:41.679 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:41 smithi167 ceph-mon[33178]: pgmap v1043: 168 pgs: 1 active+clean+remapped, 1 peering, 45 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 93 active+clean; 13 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 117 op/s; 117/1182 objects degraded (9.898%) 2024-09-13T13:21:41.679 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:41 smithi167 ceph-mon[33178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-13T13:21:41.679 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:41 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 117/1182 objects degraded (9.898%), 27 pgs degraded (PG_DEGRADED) 2024-09-13T13:21:41.679 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:41 smithi167 ceph-mon[33178]: osdmap e509: 8 total, 7 up, 7 in 2024-09-13T13:21:41.679 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:41 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2007725751' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:41.679 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:41 smithi167 ceph-mon[33178]: 11.10 scrub starts 2024-09-13T13:21:41.726 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:41.750 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:41.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:41.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:41.851 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:41.873 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:41.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:41.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:41.975 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:41.997 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:41.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:41.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:42.099 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:42.121 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:42.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:42.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:42.223 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:42.245 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:42.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:42.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:42.347 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:42.368 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:42.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:42.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:42.492 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:42.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:42.594 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:42.617 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:42.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:42.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:42.721 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:42.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:42 smithi052 ceph-mon[36682]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:42.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:42 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2007725751' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:21:42.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:42 smithi052 ceph-mon[36682]: osdmap e510: 8 total, 7 up, 7 in 2024-09-13T13:21:42.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:42 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2696064886' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:42.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:42 smithi052 ceph-mon[29471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:42.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2007725751' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:21:42.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:42 smithi052 ceph-mon[29471]: osdmap e510: 8 total, 7 up, 7 in 2024-09-13T13:21:42.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2696064886' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:42.744 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:42.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:42.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:21:42.790 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:21:42.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:42 smithi167 ceph-mon[33178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:42.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:42 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2007725751' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T13:21:42.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:42 smithi167 ceph-mon[33178]: osdmap e510: 8 total, 7 up, 7 in 2024-09-13T13:21:42.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:42 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2696064886' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:42.846 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:42.868 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:42.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:42.969 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:42.992 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:42.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:43.033 INFO:teuthology.orchestra.run.smithi167.stderr:Exporting 6.16 info 6.16( empty local-lis/les=466/467 n=0 ec=65/65 lis/c=466/466 les/c/f=467/467/0 sis=466) 2024-09-13T13:21:43.033 INFO:teuthology.orchestra.run.smithi167.stderr:Export successful 2024-09-13T13:21:43.050 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:21:43.050 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:21:43.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:43.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:43.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:43.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:43.217 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:43.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:43.241 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:43.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:43.342 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:43.364 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:43.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:43.466 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:43.489 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:43.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:43.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:43.613 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:43.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:43.652 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:43 smithi167 ceph-mon[33178]: pgmap v1046: 168 pgs: 1 active+clean+remapped, 1 peering, 46 active+undersized, 1 active+remapped+backfill_toofull, 26 active+undersized+degraded, 93 active+clean; 3.0 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 726 op/s; 91/711 objects degraded (12.799%) 2024-09-13T13:21:43.668 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:43 smithi052 ceph-mon[36682]: pgmap v1046: 168 pgs: 1 active+clean+remapped, 1 peering, 46 active+undersized, 1 active+remapped+backfill_toofull, 26 active+undersized+degraded, 93 active+clean; 3.0 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 726 op/s; 91/711 objects degraded (12.799%) 2024-09-13T13:21:43.668 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:43 smithi052 ceph-mon[29471]: pgmap v1046: 168 pgs: 1 active+clean+remapped, 1 peering, 46 active+undersized, 1 active+remapped+backfill_toofull, 26 active+undersized+degraded, 93 active+clean; 3.0 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 726 op/s; 91/711 objects degraded (12.799%) 2024-09-13T13:21:43.697 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T13:21:43.715 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:43.717 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:43.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:43.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:43.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:43.837 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:43.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:43.861 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:43.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:43.962 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:43.984 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:43.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:43.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:43.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -n osd.4 --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --force --op remove --pgid 6.16 2024-09-13T13:21:44.086 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:44.094 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:21:44.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:44.110 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:44.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:44.184 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/osd.4/config 2024-09-13T13:21:44.211 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:44.234 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:44.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:44.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:44.310 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:44.336 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:44.359 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:44.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:44.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:44.461 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:44.487 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:44.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:44.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:44 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:21:44.584 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:44 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:21:44.584 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:44 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:21:44.589 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:44.610 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:44.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:44.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:44.712 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:44.735 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:44.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:44.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:44.791 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:21:44.824 INFO:teuthology.orchestra.run.smithi052.stdout:ERROR: (22) Invalid argument 2024-09-13T13:21:44.825 INFO:teuthology.orchestra.run.smithi052.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:21:44.837 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:44.862 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:44.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:44.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:44.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:44.987 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:44.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:44.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:45.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:21:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:21:45.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:45.110 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:45.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:45.212 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:45.235 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:45.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:45.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:45.337 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3 2024-09-13T13:21:45.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:45.360 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:45.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.3' 2024-09-13T13:21:45.383 INFO:teuthology.orchestra.run.smithi167.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-13T13:21:45.461 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1 2024-09-13T13:21:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:45.489 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:45.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.1' 2024-09-13T13:21:45.493 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'unique_pool_4' removed 2024-09-13T13:21:45.590 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:45.592 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T13:21:45.611 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:45.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:45.655 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[36682]: pgmap v1047: 168 pgs: 1 active+clean+remapped, 1 peering, 48 active+undersized, 1 active+remapped+backfill_toofull, 24 active+undersized+degraded, 93 active+clean; 3.0 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 577 op/s; 91/711 objects degraded (12.799%) 2024-09-13T13:21:45.656 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:21:45.656 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:21:45.656 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:21:45.656 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1845569780' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:21:45.656 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[36682]: from='client.26842 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:21:45.656 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:45.656 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[29471]: pgmap v1047: 168 pgs: 1 active+clean+remapped, 1 peering, 48 active+undersized, 1 active+remapped+backfill_toofull, 24 active+undersized+degraded, 93 active+clean; 3.0 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 577 op/s; 91/711 objects degraded (12.799%) 2024-09-13T13:21:45.656 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:21:45.657 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:21:45.657 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:21:45.657 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1845569780' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:21:45.657 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[29471]: from='client.26842 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:21:45.657 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:45 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:45.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:45 smithi167 ceph-mon[33178]: pgmap v1047: 168 pgs: 1 active+clean+remapped, 1 peering, 48 active+undersized, 1 active+remapped+backfill_toofull, 24 active+undersized+degraded, 93 active+clean; 3.0 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 577 op/s; 91/711 objects degraded (12.799%) 2024-09-13T13:21:45.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:45 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:21:45.658 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:45 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:21:45.658 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:45 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:21:45.658 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:45 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1845569780' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:21:45.658 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:45 smithi167 ceph-mon[33178]: from='client.26842 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T13:21:45.658 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:45 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:45.713 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:45.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:45.735 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:45.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:45.785 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:45.836 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7 2024-09-13T13:21:45.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:45.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.7' 2024-09-13T13:21:45.959 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0 2024-09-13T13:21:45.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:45.983 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:45.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.0' 2024-09-13T13:21:46.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6 2024-09-13T13:21:46.106 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:46.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.6' 2024-09-13T13:21:46.208 DEBUG:teuthology.orchestra.run.smithi052:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2 2024-09-13T13:21:46.231 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: docker: command not found 2024-09-13T13:21:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:46.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi052 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.2' 2024-09-13T13:21:46.238 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:21:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:21:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:21:46.333 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5 2024-09-13T13:21:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:21:46.355 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-09-13T13:21:46.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd.5' 2024-09-13T13:21:46.357 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-13T13:21:46.366 INFO:tasks.thrashosds:joining thrashosds 2024-09-13T13:21:46.601 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:46 smithi052 ceph-mon[36682]: from='client.26842 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:21:46.601 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:46 smithi052 ceph-mon[36682]: osdmap e511: 8 total, 7 up, 7 in 2024-09-13T13:21:46.601 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:46 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:21:46.601 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:46 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:21:46.602 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:46 smithi052 ceph-mon[29471]: from='client.26842 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:21:46.602 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:46 smithi052 ceph-mon[29471]: osdmap e511: 8 total, 7 up, 7 in 2024-09-13T13:21:46.602 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:46 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:21:46.602 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:46 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:21:46.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:46 smithi167 ceph-mon[33178]: from='client.26842 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T13:21:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:46 smithi167 ceph-mon[33178]: osdmap e511: 8 total, 7 up, 7 in 2024-09-13T13:21:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:46 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:21:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:46 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:21:46.868 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:47.426 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:47.429 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0: {} 2024-09-13T13:21:47.429 INFO:teuthology.orchestra.run.smithi052.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:21:47.452 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1: {} 2024-09-13T13:21:47.452 INFO:teuthology.orchestra.run.smithi052.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:21:47.469 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2: {} 2024-09-13T13:21:47.470 INFO:teuthology.orchestra.run.smithi052.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:21:47.486 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3: {} 2024-09-13T13:21:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:21:47.489 INFO:teuthology.orchestra.run.smithi052.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:21:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:21:47.510 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5: {} 2024-09-13T13:21:47.510 INFO:teuthology.orchestra.run.smithi052.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:21:47.531 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6: {} 2024-09-13T13:21:47.531 INFO:teuthology.orchestra.run.smithi052.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:21:47.548 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7: {} 2024-09-13T13:21:47.548 INFO:teuthology.orchestra.run.smithi052.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:21:47.578 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:47 smithi052 ceph-mon[36682]: pgmap v1049: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:47.578 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:47 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 91/597 objects degraded (15.243%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:21:47.578 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:47 smithi052 ceph-mon[36682]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T13:21:47.578 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:47 smithi052 ceph-mon[36682]: osdmap e512: 8 total, 7 up, 7 in 2024-09-13T13:21:47.578 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:47 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2400478260' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:21:47.579 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:47 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3502454381' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:47.579 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:47 smithi052 ceph-mon[29471]: pgmap v1049: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:47.579 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:47 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 91/597 objects degraded (15.243%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:21:47.579 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:47 smithi052 ceph-mon[29471]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T13:21:47.579 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:47 smithi052 ceph-mon[29471]: osdmap e512: 8 total, 7 up, 7 in 2024-09-13T13:21:47.579 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:47 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2400478260' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:21:47.579 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:47 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3502454381' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:47.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:47 smithi167 ceph-mon[33178]: pgmap v1049: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:47.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:47 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 91/597 objects degraded (15.243%), 24 pgs degraded (PG_DEGRADED) 2024-09-13T13:21:47.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:47 smithi167 ceph-mon[33178]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-13T13:21:47.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:47 smithi167 ceph-mon[33178]: osdmap e512: 8 total, 7 up, 7 in 2024-09-13T13:21:47.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:47 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2400478260' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:21:47.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:47 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3502454381' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T13:21:48.294 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:21:48.294 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:21:48.294 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:21:48.294 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:21:48.294 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.294 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189885 9.4 9:25913ce2:::benchmark_data_smithi052_128427_object59216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-09-13T13:21:48.294 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:21:48.295 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 545.37643195500004, 2024-09-13T13:21:48.295 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.048349118, 2024-09-13T13:21:48.295 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.295 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.295 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:48.295 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.295 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:48.295 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:48.295 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189885 2024-09-13T13:21:48.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.295 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.295 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.295 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.296 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:21:48.296 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.296 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.296 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.296 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909608+0000", 2024-09-13T13:21:48.296 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.296 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.296 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.296 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909609+0000", 2024-09-13T13:21:48.296 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0860000000000001e-06 2024-09-13T13:21:48.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.297 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.297 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.297 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909614+0000", 2024-09-13T13:21:48.297 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5199999999999999e-06 2024-09-13T13:21:48.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.297 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.297 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.297 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909627+0000", 2024-09-13T13:21:48.297 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3623e-05 2024-09-13T13:21:48.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.297 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.298 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.298 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909634+0000", 2024-09-13T13:21:48.298 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0210000000000002e-06 2024-09-13T13:21:48.298 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.298 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.298 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.298 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909697+0000", 2024-09-13T13:21:48.298 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2769999999999997e-05 2024-09-13T13:21:48.298 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.298 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.298 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.299 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909840+0000", 2024-09-13T13:21:48.299 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014305399999999999 2024-09-13T13:21:48.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.299 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3,4", 2024-09-13T13:21:48.299 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.909967+0000", 2024-09-13T13:21:48.299 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012707300000000001 2024-09-13T13:21:48.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.299 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.299 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.913962+0000", 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0039947009999999998 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.913977+0000", 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.506e-05 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.916641+0000", 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026644839999999999 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.300 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.301 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.937520+0000", 2024-09-13T13:21:48.301 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.020878821999999998 2024-09-13T13:21:48.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.301 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.301 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.301 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.937533+0000", 2024-09-13T13:21:48.301 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2896e-05 2024-09-13T13:21:48.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.301 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.301 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.301 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.957873+0000", 2024-09-13T13:21:48.301 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0203400570000001 2024-09-13T13:21:48.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.957957+0000", 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3950999999999996e-05 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189915 9.4 9:2fa1035b:::benchmark_data_smithi052_128427_object59246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 545.34309813100003, 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0238765249999999, 2024-09-13T13:21:48.302 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189915 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.303 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.304 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.304 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942942+0000", 2024-09-13T13:21:48.304 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.304 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.304 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.304 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942944+0000", 2024-09-13T13:21:48.304 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.655e-06 2024-09-13T13:21:48.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.304 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.304 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.304 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942948+0000", 2024-09-13T13:21:48.304 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3220000000000003e-06 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942957+0000", 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2609999999999999e-06 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.942964+0000", 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6379999999999998e-06 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.305 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.944838+0000", 2024-09-13T13:21:48.306 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018746419999999999 2024-09-13T13:21:48.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.306 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.306 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.944845+0000", 2024-09-13T13:21:48.306 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:21:48.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.306 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.306 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958435+0000", 2024-09-13T13:21:48.306 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0135895370000001 2024-09-13T13:21:48.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958549+0000", 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114144 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.958677+0000", 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012798 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966592+0000", 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0079149440000000001 2024-09-13T13:21:48.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.308 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.308 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966687+0000", 2024-09-13T13:21:48.308 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4741999999999995e-05 2024-09-13T13:21:48.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.308 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.308 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966725+0000", 2024-09-13T13:21:48.308 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7768000000000002e-05 2024-09-13T13:21:48.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.308 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.308 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966777+0000", 2024-09-13T13:21:48.309 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2605999999999997e-05 2024-09-13T13:21:48.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.309 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.309 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966818+0000", 2024-09-13T13:21:48.309 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1279999999999998e-05 2024-09-13T13:21:48.309 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.309 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.309 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.309 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189933 9.4 9:2eec938f:::benchmark_data_smithi052_128427_object59264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:21:48.310 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:21:48.310 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 545.32796953299999, 2024-09-13T13:21:48.310 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.008927224, 2024-09-13T13:21:48.310 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.310 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.310 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.310 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.310 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:48.310 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:48.310 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189933 2024-09-13T13:21:48.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.310 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.310 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.311 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.311 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:21:48.311 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.311 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.311 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958070+0000", 2024-09-13T13:21:48.311 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.311 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.311 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958072+0000", 2024-09-13T13:21:48.312 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7e-06 2024-09-13T13:21:48.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.312 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.312 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.312 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958078+0000", 2024-09-13T13:21:48.312 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4990000000000002e-06 2024-09-13T13:21:48.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.312 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.312 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.312 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958089+0000", 2024-09-13T13:21:48.312 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0865e-05 2024-09-13T13:21:48.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.312 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958095+0000", 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8279999999999997e-06 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958183+0000", 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7655e-05 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.958191+0000", 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9529999999999999e-06 2024-09-13T13:21:48.313 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.314 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.314 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.314 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961131+0000", 2024-09-13T13:21:48.314 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0029402620000001 2024-09-13T13:21:48.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.314 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.314 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.314 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961259+0000", 2024-09-13T13:21:48.314 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012816999999999999 2024-09-13T13:21:48.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.314 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.315 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:21:48.315 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.961395+0000", 2024-09-13T13:21:48.315 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013549800000000001 2024-09-13T13:21:48.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.315 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.315 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.315 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966691+0000", 2024-09-13T13:21:48.315 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0052965979999999996 2024-09-13T13:21:48.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.315 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.315 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.315 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966864+0000", 2024-09-13T13:21:48.315 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017277899999999999 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966899+0000", 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4377e-05 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966955+0000", 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6495e-05 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.316 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.966998+0000", 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2545e-05 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189960 9.4 9:2cd09825:::benchmark_data_smithi052_128427_object59291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 545.303924059, 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99033151900000005, 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189960 2024-09-13T13:21:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:21:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982116+0000", 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982118+0000", 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982123+0000", 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8389999999999997e-06 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.320 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982137+0000", 2024-09-13T13:21:48.320 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4433000000000001e-05 2024-09-13T13:21:48.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.320 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.320 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982145+0000", 2024-09-13T13:21:48.320 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8290000000000008e-06 2024-09-13T13:21:48.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.320 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.320 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982197+0000", 2024-09-13T13:21:48.320 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2431e-05 2024-09-13T13:21:48.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.321 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.321 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.321 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.982205+0000", 2024-09-13T13:21:48.321 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8630000000000008e-06 2024-09-13T13:21:48.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.321 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.321 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.321 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964055+0000", 2024-09-13T13:21:48.321 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98184965300000004 2024-09-13T13:21:48.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.321 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.321 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.321 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964156+0000", 2024-09-13T13:21:48.322 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101055 2024-09-13T13:21:48.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.322 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.322 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:21:48.322 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.964261+0000", 2024-09-13T13:21:48.322 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104961 2024-09-13T13:21:48.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.322 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.322 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.322 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972104+0000", 2024-09-13T13:21:48.322 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0078433300000000008 2024-09-13T13:21:48.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.322 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.323 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.323 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972147+0000", 2024-09-13T13:21:48.323 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3022000000000002e-05 2024-09-13T13:21:48.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.323 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.323 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972189+0000", 2024-09-13T13:21:48.323 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1727000000000002e-05 2024-09-13T13:21:48.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.323 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.323 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972384+0000", 2024-09-13T13:21:48.323 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019545200000000001 2024-09-13T13:21:48.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.324 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.324 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972447+0000", 2024-09-13T13:21:48.324 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3144000000000004e-05 2024-09-13T13:21:48.324 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.324 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.324 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.324 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:189972 9.4 9:28374290:::benchmark_data_smithi052_128427_object59303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-13T13:21:48.324 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:21:48.324 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 545.29535189299997, 2024-09-13T13:21:48.325 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99084523700000005, 2024-09-13T13:21:48.325 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.325 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.325 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.325 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.325 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:48.325 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:48.325 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 189972 2024-09-13T13:21:48.325 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.325 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.325 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.325 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.325 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990688+0000", 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990690+0000", 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.326 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990694+0000", 2024-09-13T13:21:48.327 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6029999999999998e-06 2024-09-13T13:21:48.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.327 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990704+0000", 2024-09-13T13:21:48.327 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5060000000000001e-06 2024-09-13T13:21:48.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.327 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.327 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990710+0000", 2024-09-13T13:21:48.327 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:21:48.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.328 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.328 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990764+0000", 2024-09-13T13:21:48.328 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4252000000000002e-05 2024-09-13T13:21:48.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.328 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.328 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:42.990773+0000", 2024-09-13T13:21:48.328 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2379999999999997e-06 2024-09-13T13:21:48.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.328 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967034+0000", 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97626137000000002 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967153+0000", 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118855 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.967267+0000", 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113855 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.330 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.330 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.972462+0000", 2024-09-13T13:21:48.330 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051953219999999996 2024-09-13T13:21:48.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.330 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.330 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.976022+0000", 2024-09-13T13:21:48.330 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003559563 2024-09-13T13:21:48.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.330 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.330 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981419+0000", 2024-09-13T13:21:48.330 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053977260000000003 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981488+0000", 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8950999999999998e-05 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.981533+0000", 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4981999999999997e-05 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202468 9.4 9:2e7e6063:::benchmark_data_smithi052_128427_object71799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 530.320131575, 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.030148343, 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202468 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.332 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.333 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.333 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.333 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:21:48.333 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.333 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.333 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.333 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.333 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965908+0000", 2024-09-13T13:21:48.333 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.333 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.333 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.333 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.333 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965910+0000", 2024-09-13T13:21:48.334 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2810000000000001e-06 2024-09-13T13:21:48.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.334 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.334 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.334 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965915+0000", 2024-09-13T13:21:48.334 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0749999999999997e-06 2024-09-13T13:21:48.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.334 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.334 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.334 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965926+0000", 2024-09-13T13:21:48.334 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1056999999999999e-05 2024-09-13T13:21:48.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.334 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.965943+0000", 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7178000000000001e-05 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.966053+0000", 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109768 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.966062+0000", 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4029999999999996e-06 2024-09-13T13:21:48.335 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.971764+0000", 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0057021909999999 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.971894+0000", 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000129987 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.972041+0000", 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000146398 2024-09-13T13:21:48.336 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976575+0000", 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045341729999999998 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976634+0000", 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8996999999999998e-05 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.995927+0000", 2024-09-13T13:21:48.337 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.019293457999999999 2024-09-13T13:21:48.339 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.339 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.339 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.339 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.996003+0000", 2024-09-13T13:21:48.339 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5221000000000004e-05 2024-09-13T13:21:48.339 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.339 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.339 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.339 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.996057+0000", 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4156000000000001e-05 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202476 9.4 9:205ce74f:::benchmark_data_smithi052_128427_object71807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 530.31315886200002, 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0253464960000001, 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.340 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202476 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972881+0000", 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.342 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.342 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.342 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972883+0000", 2024-09-13T13:21:48.342 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.792e-06 2024-09-13T13:21:48.342 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.342 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.342 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.342 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972888+0000", 2024-09-13T13:21:48.342 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7380000000000002e-06 2024-09-13T13:21:48.342 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.342 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.342 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.342 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972896+0000", 2024-09-13T13:21:48.343 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9230000000000002e-06 2024-09-13T13:21:48.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.343 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.343 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.343 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.972903+0000", 2024-09-13T13:21:48.343 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5310000000000004e-06 2024-09-13T13:21:48.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.343 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.343 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.343 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.973002+0000", 2024-09-13T13:21:48.343 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8484000000000001e-05 2024-09-13T13:21:48.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.343 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.344 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.344 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.973012+0000", 2024-09-13T13:21:48.344 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0016e-05 2024-09-13T13:21:48.344 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.344 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.344 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.344 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974404+0000", 2024-09-13T13:21:48.344 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0013926820000001 2024-09-13T13:21:48.344 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.344 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.344 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.344 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974530+0000", 2024-09-13T13:21:48.344 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012599 2024-09-13T13:21:48.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.345 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.345 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:21:48.345 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.974651+0000", 2024-09-13T13:21:48.345 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012022400000000001 2024-09-13T13:21:48.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.345 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.345 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.345 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976695+0000", 2024-09-13T13:21:48.345 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0020448139999999998 2024-09-13T13:21:48.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.345 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.346 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.346 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981646+0000", 2024-09-13T13:21:48.346 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049504229999999998 2024-09-13T13:21:48.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.346 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.346 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.346 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998145+0000", 2024-09-13T13:21:48.346 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.016499092 2024-09-13T13:21:48.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.346 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.346 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.346 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998183+0000", 2024-09-13T13:21:48.346 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8457999999999997e-05 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.998228+0000", 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4329000000000002e-05 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202503 9.4 9:257a0b3a:::benchmark_data_smithi052_128427_object71834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 530.29411052700004, 2024-09-13T13:21:48.347 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.009335557, 2024-09-13T13:21:48.348 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.348 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.348 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.348 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.348 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:48.348 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:48.348 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202503 2024-09-13T13:21:48.348 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.348 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.348 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.348 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.348 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991929+0000", 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991936+0000", 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1149999999999999e-06 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.349 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991941+0000", 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.463e-06 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991954+0000", 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2533e-05 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.991961+0000", 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1690000000000003e-06 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.350 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.992017+0000", 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5856999999999997e-05 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.992026+0000", 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976789+0000", 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98476255999999995 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.976898+0000", 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109548 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.977048+0000", 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015011899999999999 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981632+0000", 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0045837020000000003 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.352 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.353 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.353 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981701+0000", 2024-09-13T13:21:48.353 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8511999999999997e-05 2024-09-13T13:21:48.353 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.353 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.353 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.353 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001159+0000", 2024-09-13T13:21:48.353 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.019457953 2024-09-13T13:21:48.353 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.353 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.353 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.353 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001221+0000", 2024-09-13T13:21:48.353 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2229000000000003e-05 2024-09-13T13:21:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001265+0000", 2024-09-13T13:21:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4267999999999997e-05 2024-09-13T13:21:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202507 9.4 9:20f0d2e1:::benchmark_data_smithi052_128427_object71838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:21:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:21:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 530.292025384, 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.007409886, 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202507 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:21:48.355 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.356 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.356 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.356 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994015+0000", 2024-09-13T13:21:48.356 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.356 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.356 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.356 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994016+0000", 2024-09-13T13:21:48.356 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.001e-06 2024-09-13T13:21:48.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.356 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.356 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994020+0000", 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2320000000000003e-06 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994031+0000", 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1259e-05 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994036+0000", 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3580000000000002e-06 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.357 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994089+0000", 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2120000000000002e-05 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.994097+0000", 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8000000000000004e-06 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979607+0000", 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98551001400000005 2024-09-13T13:21:48.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979708+0000", 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000100108 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.979813+0000", 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000105238 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981689+0000", 2024-09-13T13:21:48.359 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0018764299999999999 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987043+0000", 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0053537139999999999 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001312+0000", 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014268729000000001 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.360 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001378+0000", 2024-09-13T13:21:48.361 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5980000000000002e-05 2024-09-13T13:21:48.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.361 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.361 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.361 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.001424+0000", 2024-09-13T13:21:48.361 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6903000000000001e-05 2024-09-13T13:21:48.361 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.361 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.361 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202513 9.4 9:293cc3ac:::benchmark_data_smithi052_128427_object71844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 530.28656944199997, 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0035790550000001, 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202513 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.362 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999471+0000", 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999472+0000", 2024-09-13T13:21:48.363 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.536e-06 2024-09-13T13:21:48.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.364 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.364 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999477+0000", 2024-09-13T13:21:48.364 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7260000000000002e-06 2024-09-13T13:21:48.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.364 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.364 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999492+0000", 2024-09-13T13:21:48.364 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5495999999999999e-05 2024-09-13T13:21:48.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.364 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999500+0000", 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3100000000000003e-06 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999557+0000", 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7491999999999998e-05 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.999565+0000", 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9359999999999999e-06 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981724+0000", 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98215919600000001 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981838+0000", 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000114092 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,4", 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.981966+0000", 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012748300000000001 2024-09-13T13:21:48.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.367 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.367 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.367 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987109+0000", 2024-09-13T13:21:48.367 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.005142771 2024-09-13T13:21:48.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.367 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.367 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.367 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.987186+0000", 2024-09-13T13:21:48.367 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7822000000000001e-05 2024-09-13T13:21:48.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.367 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.367 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.367 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.002955+0000", 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015768880999999998 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003010+0000", 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4959e-05 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003050+0000", 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9354999999999998e-05 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.368 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.369 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.369 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94527 11.0 11:04477f51:::benchmark_data_smithi052_182575_object94526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:21:48.369 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.196615+0000", 2024-09-13T13:21:48.369 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 164.08942456, 2024-09-13T13:21:48.369 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0158819889999999, 2024-09-13T13:21:48.369 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.369 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.369 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:48.369 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.369 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:48.369 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:48.369 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94527 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196615+0000", 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196615+0000", 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.370 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196618+0000", 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6960000000000001e-06 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196651+0000", 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2749000000000001e-05 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196675+0000", 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3983000000000001e-05 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.372 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.372 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196687+0000", 2024-09-13T13:21:48.372 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1652e-05 2024-09-13T13:21:48.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.372 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.372 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.372 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.434487+0000", 2024-09-13T13:21:48.372 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.237800123 2024-09-13T13:21:48.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.372 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.372 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.372 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.434495+0000", 2024-09-13T13:21:48.372 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7439999999999996e-06 2024-09-13T13:21:48.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.373 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.373 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.373 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.212419+0000", 2024-09-13T13:21:48.373 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.77792370300000002 2024-09-13T13:21:48.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.373 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.373 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.373 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.212497+0000", 2024-09-13T13:21:48.373 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8338999999999997e-05 2024-09-13T13:21:48.373 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.373 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.373 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94541 11.10 11:0f9c248a:::benchmark_data_smithi052_182575_object94540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.196705+0000", 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 164.08933479500001, 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0441548679999999, 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94541 2024-09-13T13:21:48.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.375 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.375 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.375 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.375 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196705+0000", 2024-09-13T13:21:48.375 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.375 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.375 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.375 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196705+0000", 2024-09-13T13:21:48.375 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.375 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.375 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196706+0000", 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3e-07 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196728+0000", 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2404000000000001e-05 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196736+0000", 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.187e-06 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.376 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196741+0000", 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.434669+0000", 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.23792806899999999 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.434676+0000", 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:21:48.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.378 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.378 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.378 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.220430+0000", 2024-09-13T13:21:48.378 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.78575403499999996 2024-09-13T13:21:48.378 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.378 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.378 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.378 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.220564+0000", 2024-09-13T13:21:48.378 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013392499999999999 2024-09-13T13:21:48.378 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.378 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.378 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:21:48.378 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.220848+0000", 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00028404 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.224043+0000", 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0031949190000000001 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.231463+0000", 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0074202490000000003 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.379 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.240771+0000", 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0093081469999999993 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.240822+0000", 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0217999999999999e-05 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.240860+0000", 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8559999999999997e-05 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.380 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.381 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.381 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.381 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.381 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94542 11.10 11:09397811:::benchmark_data_smithi052_182575_object94541:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:21:48.381 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.196747+0000", 2024-09-13T13:21:48.381 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 164.089292642, 2024-09-13T13:21:48.381 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0442601499999999, 2024-09-13T13:21:48.381 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.381 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.381 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.381 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.381 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:48.381 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94542 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196747+0000", 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196747+0000", 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.382 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196748+0000", 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4799999999999998e-07 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196776+0000", 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7542000000000002e-05 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196782+0000", 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.088e-06 2024-09-13T13:21:48.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.384 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.384 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.384 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196786+0000", 2024-09-13T13:21:48.384 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8389999999999997e-06 2024-09-13T13:21:48.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.384 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.384 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.384 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.434704+0000", 2024-09-13T13:21:48.384 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.23791717000000001 2024-09-13T13:21:48.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.384 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.384 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.384 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.434712+0000", 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0290000000000005e-06 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.222512+0000", 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.78780074899999997 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.222642+0000", 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000129429 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.385 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,7", 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.222915+0000", 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00027360500000000001 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.226523+0000", 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003608024 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.231470+0000", 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049461469999999997 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.386 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.240916+0000", 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0094464400000000004 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.240975+0000", 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9135999999999998e-05 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.241008+0000", 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2304000000000001e-05 2024-09-13T13:21:48.387 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.388 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.388 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.388 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.388 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.388 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95596 11.10 11:09078e1c:::benchmark_data_smithi052_182575_object95595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:21:48.388 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:07.182481+0000", 2024-09-13T13:21:48.388 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 161.10355849999999, 2024-09-13T13:21:48.388 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.131915893, 2024-09-13T13:21:48.388 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.388 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.388 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:48.388 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.388 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95596 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182481+0000", 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182481+0000", 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.390 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.390 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.390 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182484+0000", 2024-09-13T13:21:48.390 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.931e-06 2024-09-13T13:21:48.390 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.390 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.390 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.390 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182533+0000", 2024-09-13T13:21:48.390 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8446000000000003e-05 2024-09-13T13:21:48.390 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.390 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.390 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.390 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182546+0000", 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3392e-05 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.182558+0000", 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1542000000000001e-05 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.188517+0000", 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0059587579999999998 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.391 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.392 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.188527+0000", 2024-09-13T13:21:48.392 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0458e-05 2024-09-13T13:21:48.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.392 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.392 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.392 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.304374+0000", 2024-09-13T13:21:48.392 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.115846943 2024-09-13T13:21:48.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.392 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.392 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for degraded object", 2024-09-13T13:21:48.392 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.304766+0000", 2024-09-13T13:21:48.392 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00039178999999999999 2024-09-13T13:21:48.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.314229+0000", 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0094630389999999995 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.314397+0000", 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016859399999999999 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.393 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95599 11.10 11:0e938361:::benchmark_data_smithi052_182575_object95598:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:07.184322+0000", 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 161.10171835099999, 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.131418136, 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95599 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.394 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.395 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.395 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184322+0000", 2024-09-13T13:21:48.395 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.395 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.395 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.395 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.395 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184322+0000", 2024-09-13T13:21:48.395 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.395 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.395 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.395 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.395 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184323+0000", 2024-09-13T13:21:48.395 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.243e-06 2024-09-13T13:21:48.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.396 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.396 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.396 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184360+0000", 2024-09-13T13:21:48.396 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.68e-05 2024-09-13T13:21:48.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.396 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.396 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.396 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184372+0000", 2024-09-13T13:21:48.396 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2177999999999999e-05 2024-09-13T13:21:48.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.396 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.396 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184380+0000", 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6079999999999996e-06 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.188555+0000", 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041741979999999996 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.188562+0000", 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2470000000000002e-06 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.397 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.398 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.398 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.304806+0000", 2024-09-13T13:21:48.398 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.116244088 2024-09-13T13:21:48.398 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.398 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.398 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for degraded object", 2024-09-13T13:21:48.398 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.305311+0000", 2024-09-13T13:21:48.398 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00050533299999999995 2024-09-13T13:21:48.398 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.398 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.398 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.398 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.315517+0000", 2024-09-13T13:21:48.398 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010206125 2024-09-13T13:21:48.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.399 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.399 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.399 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.315740+0000", 2024-09-13T13:21:48.399 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00022231599999999999 2024-09-13T13:21:48.399 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.399 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.399 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.399 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.399 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:147037 11.0 11:049b2c22:::benchmark_data_smithi052_182575_object30028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-13T13:21:48.399 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:09.230975+0000", 2024-09-13T13:21:48.399 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 99.055064569999999, 2024-09-13T13:21:48.400 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0155612060000001, 2024-09-13T13:21:48.400 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.400 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.400 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.400 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.400 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:48.400 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:48.400 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 147037 2024-09-13T13:21:48.400 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.400 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.400 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.400 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.400 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.230975+0000", 2024-09-13T13:21:48.401 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.401 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.401 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.401 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.401 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.230975+0000", 2024-09-13T13:21:48.401 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.401 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.401 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.401 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.401 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.230978+0000", 2024-09-13T13:21:48.401 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1490000000000001e-06 2024-09-13T13:21:48.401 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.401 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.230983+0000", 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.230996+0000", 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2773e-05 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.231003+0000", 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0419999999999997e-06 2024-09-13T13:21:48.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.231057+0000", 2024-09-13T13:21:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4239e-05 2024-09-13T13:21:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.231065+0000", 2024-09-13T13:21:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1359999999999997e-06 2024-09-13T13:21:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.403 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.239373+0000", 2024-09-13T13:21:48.404 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0083080769999999 2024-09-13T13:21:48.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.404 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.404 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.404 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.239496+0000", 2024-09-13T13:21:48.404 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000122982 2024-09-13T13:21:48.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.404 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.404 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:21:48.404 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.239918+0000", 2024-09-13T13:21:48.404 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00042127199999999999 2024-09-13T13:21:48.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.404 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.244250+0000", 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0043323809999999997 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.246356+0000", 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0021056590000000002 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.246422+0000", 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5871999999999995e-05 2024-09-13T13:21:48.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.406 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.406 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.406 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.246491+0000", 2024-09-13T13:21:48.406 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9717999999999999e-05 2024-09-13T13:21:48.406 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.406 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.406 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.406 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.246537+0000", 2024-09-13T13:21:48.406 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5271000000000001e-05 2024-09-13T13:21:48.406 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.406 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.406 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.406 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:147052 11.0 11:01c46763:::benchmark_data_smithi052_182575_object30043:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:20:09.241633+0000", 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 99.044407070000005, 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0090214710000001, 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 147052 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.407 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.241633+0000", 2024-09-13T13:21:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.241633+0000", 2024-09-13T13:21:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.241635+0000", 2024-09-13T13:21:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9800000000000001e-06 2024-09-13T13:21:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.241641+0000", 2024-09-13T13:21:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8209999999999998e-06 2024-09-13T13:21:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.241652+0000", 2024-09-13T13:21:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1251e-05 2024-09-13T13:21:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.241659+0000", 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1300000000000003e-06 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.241726+0000", 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6438000000000003e-05 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:09.241733+0000", 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9000000000000006e-06 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.411 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.411 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.242122+0000", 2024-09-13T13:21:48.411 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.000388909 2024-09-13T13:21:48.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.411 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.411 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.411 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.242238+0000", 2024-09-13T13:21:48.411 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000115186 2024-09-13T13:21:48.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.411 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.411 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,6", 2024-09-13T13:21:48.411 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.242674+0000", 2024-09-13T13:21:48.411 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00043663000000000003 2024-09-13T13:21:48.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.412 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.412 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.412 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.246368+0000", 2024-09-13T13:21:48.412 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0036936099999999999 2024-09-13T13:21:48.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.412 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.412 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.412 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.246583+0000", 2024-09-13T13:21:48.412 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00021521100000000001 2024-09-13T13:21:48.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.412 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.250476+0000", 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0038929239999999999 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.250608+0000", 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013210300000000001 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:20:10.250654+0000", 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6378000000000002e-05 2024-09-13T13:21:48.413 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.414 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.414 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.414 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.414 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:222105 11.e 11:7cbff4d1:::benchmark_data_smithi052_182575_object105096:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e503)", 2024-09-13T13:21:48.414 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:21:25.151656+0000", 2024-09-13T13:21:48.414 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 23.134384234999999, 2024-09-13T13:21:48.414 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0953254569999999, 2024-09-13T13:21:48.414 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.414 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.414 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:48.414 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.414 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:48.415 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:48.415 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 222105 2024-09-13T13:21:48.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.415 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.415 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.415 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.415 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.151656+0000", 2024-09-13T13:21:48.415 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.415 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.415 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.415 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.151656+0000", 2024-09-13T13:21:48.415 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.151658+0000", 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3949999999999999e-06 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.151666+0000", 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3560000000000004e-06 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.416 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.151683+0000", 2024-09-13T13:21:48.417 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7872000000000001e-05 2024-09-13T13:21:48.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.417 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.417 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.417 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.151695+0000", 2024-09-13T13:21:48.417 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1671000000000001e-05 2024-09-13T13:21:48.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.417 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.417 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.417 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.151802+0000", 2024-09-13T13:21:48.417 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107105 2024-09-13T13:21:48.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op must wait for map", 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.151810+0000", 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6559999999999992e-06 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.164753+0000", 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.012943197 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.164762+0000", 2024-09-13T13:21:48.418 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4099999999999997e-06 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.245424+0000", 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.080661812 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.246981+0000", 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0015569830000000001 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:222121 11.13 11:cb9ad86c:::benchmark_data_smithi052_182575_object105112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:21:25.342798+0000", 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 22.943242081000001, 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.90090187200000005, 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 222121 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.420 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.342798+0000", 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.342798+0000", 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.342800+0000", 2024-09-13T13:21:48.421 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9599999999999999e-06 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.342805+0000", 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0340000000000004e-06 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.342813+0000", 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4460000000000001e-06 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.422 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.342818+0000", 2024-09-13T13:21:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8609999999999997e-06 2024-09-13T13:21:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.342865+0000", 2024-09-13T13:21:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6751e-05 2024-09-13T13:21:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.342873+0000", 2024-09-13T13:21:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3629999999999994e-06 2024-09-13T13:21:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.423 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.239566+0000", 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.89669239000000001 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.239678+0000", 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112543 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,5,6,7", 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.239752+0000", 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3831000000000005e-05 2024-09-13T13:21:48.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.242023+0000", 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0022705529999999998 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.242466+0000", 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00044327 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.242553+0000", 2024-09-13T13:21:48.425 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7114000000000007e-05 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.242928+0000", 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00037532400000000001 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.242998+0000", 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9516000000000003e-05 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.426 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.243583+0000", 2024-09-13T13:21:48.427 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00058514699999999995 2024-09-13T13:21:48.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.427 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.427 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.427 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.243700+0000", 2024-09-13T13:21:48.427 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011676900000000001 2024-09-13T13:21:48.427 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.427 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.427 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.427 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.427 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:222135 11.c 11:3cd382da:::benchmark_data_smithi052_182575_object105126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-13T13:21:48.427 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:21:25.378234+0000", 2024-09-13T13:21:48.428 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 22.907806442000002, 2024-09-13T13:21:48.428 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.86396626200000004, 2024-09-13T13:21:48.428 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:48.428 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:48.428 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:48.428 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:48.428 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:48.428 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:48.428 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 222135 2024-09-13T13:21:48.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.428 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:48.428 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.428 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.378234+0000", 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.378234+0000", 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.378235+0000", 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9319999999999999e-06 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.429 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.378240+0000", 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.933e-06 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.378251+0000", 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0910000000000001e-05 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.378259+0000", 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7780000000000007e-06 2024-09-13T13:21:48.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.378307+0000", 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7945999999999998e-05 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.378312+0000", 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4160000000000003e-06 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.236522+0000", 2024-09-13T13:21:48.431 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.85820933799999999 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.236672+0000", 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015057499999999999 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2", 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.236812+0000", 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000139219 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:48.432 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.241813+0000", 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050015800000000003 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.242087+0000", 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000273446 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.242139+0000", 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1900000000000001e-05 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:48.433 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:48.434 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.242200+0000", 2024-09-13T13:21:48.434 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1289000000000002e-05 2024-09-13T13:21:48.434 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.434 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.434 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.434 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:48.434 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:48.434 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:21:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:48 smithi052 ceph-mon[36682]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:48 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2400478260' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:21:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:48 smithi052 ceph-mon[36682]: osdmap e513: 8 total, 7 up, 7 in 2024-09-13T13:21:48.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:48 smithi052 ceph-mon[36682]: osdmap e514: 8 total, 7 up, 7 in 2024-09-13T13:21:48.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:48 smithi052 ceph-mon[29471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:48.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:48 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2400478260' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:21:48.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:48 smithi052 ceph-mon[29471]: osdmap e513: 8 total, 7 up, 7 in 2024-09-13T13:21:48.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:48 smithi052 ceph-mon[29471]: osdmap e514: 8 total, 7 up, 7 in 2024-09-13T13:21:48.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:48 smithi167 ceph-mon[33178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T13:21:48.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:48 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2400478260' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:21:48.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:48 smithi167 ceph-mon[33178]: osdmap e513: 8 total, 7 up, 7 in 2024-09-13T13:21:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:48 smithi167 ceph-mon[33178]: osdmap e514: 8 total, 7 up, 7 in 2024-09-13T13:21:48.850 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:21:48.850 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:21:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:49 smithi052 ceph-mon[36682]: pgmap v1052: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:49 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:21:49.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:49 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:21:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:49 smithi052 ceph-mon[29471]: pgmap v1052: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:49 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:21:49.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:49 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:21:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:49 smithi167 ceph-mon[33178]: pgmap v1052: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:49 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:21:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:49 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:21:50.389 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T13:21:50.658 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:50.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:50 smithi052 ceph-mon[36682]: 5.15 scrub starts 2024-09-13T13:21:50.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:50 smithi052 ceph-mon[36682]: 5.15 scrub ok 2024-09-13T13:21:50.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:50 smithi052 ceph-mon[29471]: 5.15 scrub starts 2024-09-13T13:21:50.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:50 smithi052 ceph-mon[29471]: 5.15 scrub ok 2024-09-13T13:21:50.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:50 smithi167 ceph-mon[33178]: 5.15 scrub starts 2024-09-13T13:21:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:50 smithi167 ceph-mon[33178]: 5.15 scrub ok 2024-09-13T13:21:51.147 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:21:51.147 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:21:51.147 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 0 2024-09-13T13:21:51.147 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:21:51.522 INFO:teuthology.orchestra.run.smithi167.stdout: marking collection for removal 2024-09-13T13:21:51.522 INFO:teuthology.orchestra.run.smithi167.stdout:setting '_remove' omap key 2024-09-13T13:21:51.522 INFO:teuthology.orchestra.run.smithi167.stdout:finish_remove_pgs 6.16_head removing 6.16 2024-09-13T13:21:51.523 INFO:teuthology.orchestra.run.smithi167.stdout:Remove successful 2024-09-13T13:21:51.800 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:51 smithi052 ceph-mon[36682]: pgmap v1054: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:51.800 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:51 smithi052 ceph-mon[29471]: pgmap v1054: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:51.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:51 smithi167 ceph-mon[33178]: pgmap v1054: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:51.853 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T13:21:51.880 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T13:21:52.125 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:52.164 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:52.515 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:52 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:21:52.515 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:52 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:21:52.515 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:52 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:21:52.515 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:52 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:21:52.627 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:52 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:21:52.627 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:52 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:21:52.658 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell -n osd.4 --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.6.16.4 2024-09-13T13:21:52.673 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:21:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:21:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:21:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:21:52.674 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:21:52.852 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/osd.4/config 2024-09-13T13:21:53.372 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T13:21:53.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:53 smithi167 ceph-mon[33178]: pgmap v1055: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:53 smithi167 ceph-mon[33178]: osdmap e515: 8 total, 7 up, 7 in 2024-09-13T13:21:53.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:53 smithi167 ceph-mon[33178]: osdmap e516: 8 total, 7 up, 7 in 2024-09-13T13:21:53.608 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:53 smithi052 ceph-mon[36682]: pgmap v1055: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:53.608 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:53 smithi052 ceph-mon[36682]: osdmap e515: 8 total, 7 up, 7 in 2024-09-13T13:21:53.608 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:53 smithi052 ceph-mon[36682]: osdmap e516: 8 total, 7 up, 7 in 2024-09-13T13:21:53.609 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:53 smithi052 ceph-mon[29471]: pgmap v1055: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:53.609 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:53 smithi052 ceph-mon[29471]: osdmap e515: 8 total, 7 up, 7 in 2024-09-13T13:21:53.609 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:53 smithi052 ceph-mon[29471]: osdmap e516: 8 total, 7 up, 7 in 2024-09-13T13:21:53.901 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:54.126 INFO:teuthology.orchestra.run.smithi167.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-13T13:21:54.446 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:21:54.446 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:21:54.446 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:21:54.446 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:21:54.446 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.446 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:242631 10.5 10:a0e14904:::benchmark_data_smithi052_153703_object115251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T13:21:54.446 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:17:29.082673+0000", 2024-09-13T13:21:54.446 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 265.36113481899997, 2024-09-13T13:21:54.447 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99442527400000003, 2024-09-13T13:21:54.447 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.447 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.447 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:54.447 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.447 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:21:54.447 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:21:54.447 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 242631 2024-09-13T13:21:54.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.447 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.447 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.447 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.448 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.082673+0000", 2024-09-13T13:21:54.448 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.448 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.448 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.448 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.082673+0000", 2024-09-13T13:21:54.448 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.448 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.448 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.448 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.082674+0000", 2024-09-13T13:21:54.448 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.584e-06 2024-09-13T13:21:54.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.449 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.449 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.449 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.082679+0000", 2024-09-13T13:21:54.449 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2859999999999999e-06 2024-09-13T13:21:54.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.449 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.449 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.449 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.082690+0000", 2024-09-13T13:21:54.449 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0446e-05 2024-09-13T13:21:54.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.449 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.450 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.450 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.082695+0000", 2024-09-13T13:21:54.450 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4519999999999996e-06 2024-09-13T13:21:54.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.450 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.450 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.450 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.082747+0000", 2024-09-13T13:21:54.450 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1526000000000001e-05 2024-09-13T13:21:54.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.450 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.450 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.450 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.082756+0000", 2024-09-13T13:21:54.450 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8890000000000008e-06 2024-09-13T13:21:54.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.451 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.451 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.451 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.072695+0000", 2024-09-13T13:21:54.451 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98993964000000001 2024-09-13T13:21:54.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.451 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.451 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.451 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.072778+0000", 2024-09-13T13:21:54.451 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2515000000000005e-05 2024-09-13T13:21:54.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.451 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.452 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:21:54.452 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.072852+0000", 2024-09-13T13:21:54.452 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4474000000000006e-05 2024-09-13T13:21:54.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.452 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.452 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.452 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.076996+0000", 2024-09-13T13:21:54.452 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041439620000000002 2024-09-13T13:21:54.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.452 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.452 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.452 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.077010+0000", 2024-09-13T13:21:54.453 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3806e-05 2024-09-13T13:21:54.453 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.453 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.453 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.077039+0000", 2024-09-13T13:21:54.453 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9255e-05 2024-09-13T13:21:54.453 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.453 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:54.453 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.077071+0000", 2024-09-13T13:21:54.453 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1965000000000003e-05 2024-09-13T13:21:54.453 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.454 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.454 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.454 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.077098+0000", 2024-09-13T13:21:54.454 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6474000000000001e-05 2024-09-13T13:21:54.454 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.454 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.454 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.454 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.454 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:242648 10.5 10:a36483a1:::benchmark_data_smithi052_153703_object115268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T13:21:54.454 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:17:29.099112+0000", 2024-09-13T13:21:54.454 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 265.34469534999999, 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98211603599999997, 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 242648 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.099112+0000", 2024-09-13T13:21:54.455 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.099112+0000", 2024-09-13T13:21:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.099114+0000", 2024-09-13T13:21:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0279999999999999e-06 2024-09-13T13:21:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.457 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.099120+0000", 2024-09-13T13:21:54.457 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8680000000000004e-06 2024-09-13T13:21:54.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.457 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.457 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.099133+0000", 2024-09-13T13:21:54.457 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3332e-05 2024-09-13T13:21:54.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.457 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.457 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.099140+0000", 2024-09-13T13:21:54.457 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1489999999999997e-06 2024-09-13T13:21:54.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.458 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.458 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.458 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.099209+0000", 2024-09-13T13:21:54.458 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8400999999999998e-05 2024-09-13T13:21:54.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.458 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.458 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.458 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.099220+0000", 2024-09-13T13:21:54.458 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.082e-05 2024-09-13T13:21:54.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.458 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.458 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.458 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.075410+0000", 2024-09-13T13:21:54.459 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97619012800000005 2024-09-13T13:21:54.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.459 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.459 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.075509+0000", 2024-09-13T13:21:54.459 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9345000000000005e-05 2024-09-13T13:21:54.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.459 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:21:54.459 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.075577+0000", 2024-09-13T13:21:54.459 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7428999999999994e-05 2024-09-13T13:21:54.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.460 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.460 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.081061+0000", 2024-09-13T13:21:54.460 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0054843909999999999 2024-09-13T13:21:54.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.460 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.460 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.460 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.081075+0000", 2024-09-13T13:21:54.460 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3789e-05 2024-09-13T13:21:54.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.460 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.460 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.460 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.081167+0000", 2024-09-13T13:21:54.460 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2612e-05 2024-09-13T13:21:54.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.461 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:54.461 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.081201+0000", 2024-09-13T13:21:54.461 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3779999999999998e-05 2024-09-13T13:21:54.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.462 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.462 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.081228+0000", 2024-09-13T13:21:54.462 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6964000000000001e-05 2024-09-13T13:21:54.462 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.462 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.462 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.463 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:242656 10.5 10:acb1fe91:::benchmark_data_smithi052_153703_object115276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T13:21:54.463 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:17:29.105228+0000", 2024-09-13T13:21:54.463 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 265.33857934600002, 2024-09-13T13:21:54.463 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97614441500000004, 2024-09-13T13:21:54.463 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.463 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.463 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:54.463 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.463 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:21:54.463 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:21:54.463 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 242656 2024-09-13T13:21:54.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.464 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.464 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.464 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.105228+0000", 2024-09-13T13:21:54.464 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.464 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.464 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.105228+0000", 2024-09-13T13:21:54.464 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.465 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.465 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.105230+0000", 2024-09-13T13:21:54.465 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5370000000000001e-06 2024-09-13T13:21:54.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.465 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.465 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.105234+0000", 2024-09-13T13:21:54.465 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6779999999999999e-06 2024-09-13T13:21:54.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.465 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.465 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.105244+0000", 2024-09-13T13:21:54.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3500000000000003e-06 2024-09-13T13:21:54.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.466 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.466 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.466 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.105250+0000", 2024-09-13T13:21:54.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1850000000000001e-06 2024-09-13T13:21:54.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.466 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.466 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.466 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.105324+0000", 2024-09-13T13:21:54.466 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4289999999999995e-05 2024-09-13T13:21:54.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.466 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:29.105334+0000", 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7089999999999997e-06 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.077134+0000", 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97180073300000003 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.077248+0000", 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113539 2024-09-13T13:21:54.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.468 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:21:54.468 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.077357+0000", 2024-09-13T13:21:54.468 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108744 2024-09-13T13:21:54.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.468 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.468 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.081128+0000", 2024-09-13T13:21:54.468 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037716440000000002 2024-09-13T13:21:54.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.468 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.468 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.081235+0000", 2024-09-13T13:21:54.469 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107039 2024-09-13T13:21:54.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.469 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.469 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.081267+0000", 2024-09-13T13:21:54.469 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.1399000000000002e-05 2024-09-13T13:21:54.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.469 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:54.469 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.081338+0000", 2024-09-13T13:21:54.469 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0915999999999997e-05 2024-09-13T13:21:54.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.470 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.470 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.470 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.081372+0000", 2024-09-13T13:21:54.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4652e-05 2024-09-13T13:21:54.470 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.470 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.470 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.470 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.470 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:242784 10.3 10:c505fa13:::benchmark_data_smithi052_153703_object115404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:21:54.470 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:17:30.098998+0000", 2024-09-13T13:21:54.470 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 264.34480963700003, 2024-09-13T13:21:54.470 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0042657589999999, 2024-09-13T13:21:54.471 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.471 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.471 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:54.471 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.471 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:21:54.471 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:21:54.471 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 242784 2024-09-13T13:21:54.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.471 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.471 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.471 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.098998+0000", 2024-09-13T13:21:54.471 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.472 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.472 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.098998+0000", 2024-09-13T13:21:54.472 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.472 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.472 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.099000+0000", 2024-09-13T13:21:54.472 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0729999999999999e-06 2024-09-13T13:21:54.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.472 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.473 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.099006+0000", 2024-09-13T13:21:54.473 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.122e-06 2024-09-13T13:21:54.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.473 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.473 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.473 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.099020+0000", 2024-09-13T13:21:54.473 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3798000000000001e-05 2024-09-13T13:21:54.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.473 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.473 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.473 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.099028+0000", 2024-09-13T13:21:54.473 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9589999999999995e-06 2024-09-13T13:21:54.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.474 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.474 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.474 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.099090+0000", 2024-09-13T13:21:54.474 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1971999999999996e-05 2024-09-13T13:21:54.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.474 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.474 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.474 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.099098+0000", 2024-09-13T13:21:54.474 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5359999999999992e-06 2024-09-13T13:21:54.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.474 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.474 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.474 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.092324+0000", 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99322554799999996 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.092469+0000", 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014552900000000001 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.092590+0000", 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012040999999999999 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.102779+0000", 2024-09-13T13:21:54.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010189617 2024-09-13T13:21:54.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.476 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.103020+0000", 2024-09-13T13:21:54.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00024082699999999999 2024-09-13T13:21:54.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.476 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.476 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.103167+0000", 2024-09-13T13:21:54.476 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00014694099999999999 2024-09-13T13:21:54.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.477 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:54.477 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.103223+0000", 2024-09-13T13:21:54.477 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5957e-05 2024-09-13T13:21:54.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.477 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.477 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.103263+0000", 2024-09-13T13:21:54.477 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0469999999999997e-05 2024-09-13T13:21:54.477 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.477 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.477 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.478 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:242786 10.3 10:ca78af6a:::benchmark_data_smithi052_153703_object115406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:21:54.478 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:17:30.101372+0000", 2024-09-13T13:21:54.478 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 264.34243513000001, 2024-09-13T13:21:54.478 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0128771510000001, 2024-09-13T13:21:54.478 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.478 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.478 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:54.478 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.478 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:21:54.478 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:21:54.478 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 242786 2024-09-13T13:21:54.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.479 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.479 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.479 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.101372+0000", 2024-09-13T13:21:54.479 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.479 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.479 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.101372+0000", 2024-09-13T13:21:54.479 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.479 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.480 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.101374+0000", 2024-09-13T13:21:54.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9860000000000001e-06 2024-09-13T13:21:54.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.480 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.480 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.480 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.101380+0000", 2024-09-13T13:21:54.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2110000000000003e-06 2024-09-13T13:21:54.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.480 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.480 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.480 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.101392+0000", 2024-09-13T13:21:54.480 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1806e-05 2024-09-13T13:21:54.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.481 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.481 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.101401+0000", 2024-09-13T13:21:54.481 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4060000000000005e-06 2024-09-13T13:21:54.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.481 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.481 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.101458+0000", 2024-09-13T13:21:54.481 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6971000000000001e-05 2024-09-13T13:21:54.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.481 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.101466+0000", 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9489999999999996e-06 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.096302+0000", 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99483624999999998 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.096402+0000", 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9828999999999996e-05 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:21:54.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.096501+0000", 2024-09-13T13:21:54.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9444999999999994e-05 2024-09-13T13:21:54.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.103033+0000", 2024-09-13T13:21:54.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0065316439999999996 2024-09-13T13:21:54.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.483 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.483 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114072+0000", 2024-09-13T13:21:54.483 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011039531 2024-09-13T13:21:54.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114150+0000", 2024-09-13T13:21:54.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8150000000000002e-05 2024-09-13T13:21:54.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:54.484 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114203+0000", 2024-09-13T13:21:54.484 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2713999999999997e-05 2024-09-13T13:21:54.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.484 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.485 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114249+0000", 2024-09-13T13:21:54.485 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.6258999999999999e-05 2024-09-13T13:21:54.485 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.485 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.485 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.485 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:242788 10.3 10:c5161d6b:::benchmark_data_smithi052_153703_object115408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:21:54.485 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:17:30.106680+0000", 2024-09-13T13:21:54.485 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 264.33712727699998, 2024-09-13T13:21:54.485 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.007684552, 2024-09-13T13:21:54.485 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.485 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.486 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:54.486 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.486 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:21:54.486 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:21:54.486 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 242788 2024-09-13T13:21:54.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.486 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.486 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.486 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.106680+0000", 2024-09-13T13:21:54.486 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.487 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.487 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.106680+0000", 2024-09-13T13:21:54.487 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.487 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.487 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.106682+0000", 2024-09-13T13:21:54.487 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3350000000000001e-06 2024-09-13T13:21:54.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.487 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.487 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.106689+0000", 2024-09-13T13:21:54.487 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5810000000000002e-06 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.106705+0000", 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.596e-05 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.106713+0000", 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4309999999999994e-06 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.488 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.106772+0000", 2024-09-13T13:21:54.489 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8131999999999998e-05 2024-09-13T13:21:54.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.489 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.489 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.106779+0000", 2024-09-13T13:21:54.489 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.289e-06 2024-09-13T13:21:54.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.489 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.489 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.099199+0000", 2024-09-13T13:21:54.489 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99241995599999999 2024-09-13T13:21:54.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.099329+0000", 2024-09-13T13:21:54.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013038899999999999 2024-09-13T13:21:54.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:21:54.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.099453+0000", 2024-09-13T13:21:54.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000124308 2024-09-13T13:21:54.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.490 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.490 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.103038+0000", 2024-09-13T13:21:54.490 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0035842209999999998 2024-09-13T13:21:54.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114113+0000", 2024-09-13T13:21:54.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011075368 2024-09-13T13:21:54.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.491 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114290+0000", 2024-09-13T13:21:54.491 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017679999999999999 2024-09-13T13:21:54.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.491 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:54.492 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114332+0000", 2024-09-13T13:21:54.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1922999999999998e-05 2024-09-13T13:21:54.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.492 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.492 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114365+0000", 2024-09-13T13:21:54.492 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2858999999999998e-05 2024-09-13T13:21:54.492 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.492 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.492 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.492 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:242797 10.3 10:c8e790f4:::benchmark_data_smithi052_153703_object115417:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:21:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:17:30.119607+0000", 2024-09-13T13:21:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 264.32420034, 2024-09-13T13:21:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99977440399999995, 2024-09-13T13:21:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:21:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:21:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 242797 2024-09-13T13:21:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.119607+0000", 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.119607+0000", 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.119610+0000", 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7410000000000001e-06 2024-09-13T13:21:54.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.495 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.495 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.119616+0000", 2024-09-13T13:21:54.495 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3500000000000002e-06 2024-09-13T13:21:54.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.495 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.495 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.119631+0000", 2024-09-13T13:21:54.495 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4579999999999999e-05 2024-09-13T13:21:54.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.495 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.495 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.119640+0000", 2024-09-13T13:21:54.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9479999999999997e-06 2024-09-13T13:21:54.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.496 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.496 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.119701+0000", 2024-09-13T13:21:54.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1870000000000002e-05 2024-09-13T13:21:54.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.496 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.496 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.119710+0000", 2024-09-13T13:21:54.496 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3380000000000005e-06 2024-09-13T13:21:54.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.497 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.497 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.103298+0000", 2024-09-13T13:21:54.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98358866199999995 2024-09-13T13:21:54.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.498 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.103412+0000", 2024-09-13T13:21:54.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000113513 2024-09-13T13:21:54.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.498 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:21:54.498 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.103526+0000", 2024-09-13T13:21:54.498 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011367899999999999 2024-09-13T13:21:54.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.499 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.499 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.109399+0000", 2024-09-13T13:21:54.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058731679999999998 2024-09-13T13:21:54.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.499 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.499 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114402+0000", 2024-09-13T13:21:54.499 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050030200000000004 2024-09-13T13:21:54.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.499 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.499 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.119275+0000", 2024-09-13T13:21:54.500 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048731590000000002 2024-09-13T13:21:54.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.500 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:54.500 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.119334+0000", 2024-09-13T13:21:54.500 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8576999999999998e-05 2024-09-13T13:21:54.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.500 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.500 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.119381+0000", 2024-09-13T13:21:54.500 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7799000000000001e-05 2024-09-13T13:21:54.500 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.500 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.501 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.501 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:242816 10.3 10:c64362d0:::benchmark_data_smithi052_153703_object115436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:21:54.501 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:17:30.134661+0000", 2024-09-13T13:21:54.501 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 264.309145951, 2024-09-13T13:21:54.501 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98484013400000003, 2024-09-13T13:21:54.504 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.504 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.504 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:54.504 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.504 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:21:54.504 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:21:54.504 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 242816 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.134661+0000", 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.134661+0000", 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.134664+0000", 2024-09-13T13:21:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6280000000000001e-06 2024-09-13T13:21:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.134671+0000", 2024-09-13T13:21:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0169999999999999e-06 2024-09-13T13:21:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.134689+0000", 2024-09-13T13:21:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8380999999999999e-05 2024-09-13T13:21:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.134698+0000", 2024-09-13T13:21:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.3899999999999993e-06 2024-09-13T13:21:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.134756+0000", 2024-09-13T13:21:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8443000000000003e-05 2024-09-13T13:21:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.134765+0000", 2024-09-13T13:21:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1889999999999996e-06 2024-09-13T13:21:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.106376+0000", 2024-09-13T13:21:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97161034300000004 2024-09-13T13:21:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.106482+0000", 2024-09-13T13:21:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010617599999999999 2024-09-13T13:21:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:21:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.106588+0000", 2024-09-13T13:21:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010636700000000001 2024-09-13T13:21:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114026+0000", 2024-09-13T13:21:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00743774 2024-09-13T13:21:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114440+0000", 2024-09-13T13:21:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00041421699999999998 2024-09-13T13:21:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.119425+0000", 2024-09-13T13:21:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049850750000000003 2024-09-13T13:21:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.119469+0000", 2024-09-13T13:21:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3705000000000003e-05 2024-09-13T13:21:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.119502+0000", 2024-09-13T13:21:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:21:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:242824 10.3 10:c7eb1c12:::benchmark_data_smithi052_153703_object115444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:21:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:17:30.143087+0000", 2024-09-13T13:21:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 264.30071998800003, 2024-09-13T13:21:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97658049300000005, 2024-09-13T13:21:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:21:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:21:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 242824 2024-09-13T13:21:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.143087+0000", 2024-09-13T13:21:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.143087+0000", 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.143091+0000", 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.3019999999999999e-06 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.143100+0000", 2024-09-13T13:21:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.4900000000000006e-06 2024-09-13T13:21:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.143122+0000", 2024-09-13T13:21:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.1880000000000001e-05 2024-09-13T13:21:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.143132+0000", 2024-09-13T13:21:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8789999999999998e-06 2024-09-13T13:21:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.143204+0000", 2024-09-13T13:21:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2429999999999996e-05 2024-09-13T13:21:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.143211+0000", 2024-09-13T13:21:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8809999999999998e-06 2024-09-13T13:21:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.109429+0000", 2024-09-13T13:21:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96621766899999995 2024-09-13T13:21:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.109528+0000", 2024-09-13T13:21:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.9011999999999993e-05 2024-09-13T13:21:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:21:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.109624+0000", 2024-09-13T13:21:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5696e-05 2024-09-13T13:21:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114412+0000", 2024-09-13T13:21:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047883700000000001 2024-09-13T13:21:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114476+0000", 2024-09-13T13:21:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4406999999999995e-05 2024-09-13T13:21:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.119587+0000", 2024-09-13T13:21:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051110790000000001 2024-09-13T13:21:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.119631+0000", 2024-09-13T13:21:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3853999999999997e-05 2024-09-13T13:21:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.119668+0000", 2024-09-13T13:21:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6544000000000001e-05 2024-09-13T13:21:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.26071.0:242834 10.3 10:c62f1a54:::benchmark_data_smithi052_153703_object115454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:17:30.151172+0000", 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 264.29263522299999, 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98000325300000002, 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.26071", 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 242834 2024-09-13T13:21:54.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.520 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.520 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151172+0000", 2024-09-13T13:21:54.520 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.520 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151172+0000", 2024-09-13T13:21:54.520 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.520 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151174+0000", 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5600000000000001e-06 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151179+0000", 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3700000000000003e-06 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151196+0000", 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6532e-05 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.521 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151203+0000", 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6279999999999996e-06 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151269+0000", 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5650999999999999e-05 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:30.151279+0000", 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0609e-05 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114502+0000", 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96322301200000005 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114610+0000", 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107604 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 3,5", 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.114714+0000", 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103453 2024-09-13T13:21:54.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.524 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.524 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.119545+0000", 2024-09-13T13:21:54.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048309700000000004 2024-09-13T13:21:54.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.125383+0000", 2024-09-13T13:21:54.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058389139999999997 2024-09-13T13:21:54.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.525 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.525 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.131063+0000", 2024-09-13T13:21:54.525 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0056791669999999997 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.131122+0000", 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.9030999999999998e-05 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:17:31.131175+0000", 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3752000000000003e-05 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94521 11.2 11:42169fbc:::benchmark_data_smithi052_182575_object94520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.192249+0000", 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 170.25155807300001, 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.017257995, 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94521 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.527 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192249+0000", 2024-09-13T13:21:54.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.528 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192249+0000", 2024-09-13T13:21:54.528 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.528 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192252+0000", 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5339999999999998e-06 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192286+0000", 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4251999999999997e-05 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192304+0000", 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7737e-05 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192315+0000", 2024-09-13T13:21:54.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0900999999999999e-05 2024-09-13T13:21:54.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.411081+0000", 2024-09-13T13:21:54.530 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.21876661999999999 2024-09-13T13:21:54.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.530 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.530 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.411091+0000", 2024-09-13T13:21:54.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2760000000000006e-06 2024-09-13T13:21:54.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.209073+0000", 2024-09-13T13:21:54.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.79798205899999997 2024-09-13T13:21:54.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.531 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.531 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.209507+0000", 2024-09-13T13:21:54.531 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00043461599999999997 2024-09-13T13:21:54.531 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.532 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.532 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.532 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94529 11.2 11:47cc386d:::benchmark_data_smithi052_182575_object94528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:21:54.532 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.192333+0000", 2024-09-13T13:21:54.532 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 170.25147441799999, 2024-09-13T13:21:54.532 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0189333380000001, 2024-09-13T13:21:54.532 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.532 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.532 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:54.532 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.532 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:54.533 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:54.533 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94529 2024-09-13T13:21:54.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.533 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192333+0000", 2024-09-13T13:21:54.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.533 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.533 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192333+0000", 2024-09-13T13:21:54.533 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192334+0000", 2024-09-13T13:21:54.534 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0400000000000005e-07 2024-09-13T13:21:54.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.534 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.534 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192363+0000", 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9689e-05 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192372+0000", 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2649999999999996e-06 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192378+0000", 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0639999999999999e-06 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.535 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.411111+0000", 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.218732964 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.411117+0000", 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2920000000000001e-06 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.211189+0000", 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.80007196700000005 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.537 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.537 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.211266+0000", 2024-09-13T13:21:54.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.7292999999999993e-05 2024-09-13T13:21:54.537 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.537 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.537 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.537 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94530 11.12 11:4c3e42a0:::benchmark_data_smithi052_182575_object94529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:21:54.537 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.192387+0000", 2024-09-13T13:21:54.537 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 170.251419989, 2024-09-13T13:21:54.537 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.031252603, 2024-09-13T13:21:54.537 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94530 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192387+0000", 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.539 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192387+0000", 2024-09-13T13:21:54.539 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.539 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.539 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192388+0000", 2024-09-13T13:21:54.539 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:21:54.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192414+0000", 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5656e-05 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192422+0000", 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5210000000000002e-06 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192429+0000", 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5640000000000002e-06 2024-09-13T13:21:54.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.411455+0000", 2024-09-13T13:21:54.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.21902586700000001 2024-09-13T13:21:54.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.411462+0000", 2024-09-13T13:21:54.541 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1130000000000003e-06 2024-09-13T13:21:54.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.541 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.541 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.223564+0000", 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.81210195100000004 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.223640+0000", 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6209000000000002e-05 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94538 11.2 11:47ab5aa8:::benchmark_data_smithi052_182575_object94537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.192437+0000", 2024-09-13T13:21:54.542 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 170.25137061800001, 2024-09-13T13:21:54.543 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0478259969999999, 2024-09-13T13:21:54.543 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.543 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.543 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:54.543 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.543 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:54.543 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:54.543 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94538 2024-09-13T13:21:54.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.543 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.543 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.543 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192437+0000", 2024-09-13T13:21:54.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192437+0000", 2024-09-13T13:21:54.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.544 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.544 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192437+0000", 2024-09-13T13:21:54.544 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:21:54.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192467+0000", 2024-09-13T13:21:54.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.9519e-05 2024-09-13T13:21:54.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192474+0000", 2024-09-13T13:21:54.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6590000000000001e-06 2024-09-13T13:21:54.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.545 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.545 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.192481+0000", 2024-09-13T13:21:54.545 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3649999999999998e-06 2024-09-13T13:21:54.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.411151+0000", 2024-09-13T13:21:54.546 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.21867025000000001 2024-09-13T13:21:54.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.546 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.546 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.411158+0000", 2024-09-13T13:21:54.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4910000000000002e-06 2024-09-13T13:21:54.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.211293+0000", 2024-09-13T13:21:54.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.80013580399999995 2024-09-13T13:21:54.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.547 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.547 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.211421+0000", 2024-09-13T13:21:54.547 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012778299999999999 2024-09-13T13:21:54.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:21:54.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.211590+0000", 2024-09-13T13:21:54.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00016899300000000001 2024-09-13T13:21:54.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.548 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.219197+0000", 2024-09-13T13:21:54.548 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0076069420000000002 2024-09-13T13:21:54.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.548 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.220077+0000", 2024-09-13T13:21:54.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00087949199999999995 2024-09-13T13:21:54.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.240164+0000", 2024-09-13T13:21:54.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.020087197000000001 2024-09-13T13:21:54.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.549 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:54.549 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.240218+0000", 2024-09-13T13:21:54.549 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4574000000000003e-05 2024-09-13T13:21:54.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.550 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.550 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.240263+0000", 2024-09-13T13:21:54.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4289000000000002e-05 2024-09-13T13:21:54.550 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.550 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.550 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.550 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95597 11.18 11.13e97198 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:21:54.550 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:06.189478+0000", 2024-09-13T13:21:54.550 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 168.254329525, 2024-09-13T13:21:54.550 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98908341600000005, 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95597 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189478+0000", 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189478+0000", 2024-09-13T13:21:54.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.552 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189480+0000", 2024-09-13T13:21:54.552 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2840000000000001e-06 2024-09-13T13:21:54.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.552 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189508+0000", 2024-09-13T13:21:54.553 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7526e-05 2024-09-13T13:21:54.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.553 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.553 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189525+0000", 2024-09-13T13:21:54.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7379000000000001e-05 2024-09-13T13:21:54.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.189535+0000", 2024-09-13T13:21:54.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6190000000000006e-06 2024-09-13T13:21:54.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.554 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.554 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.191924+0000", 2024-09-13T13:21:54.554 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002388874 2024-09-13T13:21:54.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.191942+0000", 2024-09-13T13:21:54.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8144000000000001e-05 2024-09-13T13:21:54.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.178548+0000", 2024-09-13T13:21:54.555 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98660607899999997 2024-09-13T13:21:54.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.555 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.555 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.178561+0000", 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3511e-05 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95611 11.15 11.6634ccf5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:06.197352+0000", 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 168.24645545999999, 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98378791399999999, 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:54.556 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.557 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:54.557 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:54.557 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95611 2024-09-13T13:21:54.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.557 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.557 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.197352+0000", 2024-09-13T13:21:54.557 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.557 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.197352+0000", 2024-09-13T13:21:54.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.558 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.197355+0000", 2024-09-13T13:21:54.558 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7259999999999998e-06 2024-09-13T13:21:54.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.558 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.197380+0000", 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5873000000000001e-05 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.197395+0000", 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4724e-05 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.197404+0000", 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0450000000000006e-06 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.559 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.197469+0000", 2024-09-13T13:21:54.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5081999999999998e-05 2024-09-13T13:21:54.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.197477+0000", 2024-09-13T13:21:54.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9750000000000007e-06 2024-09-13T13:21:54.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.560 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.560 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.181120+0000", 2024-09-13T13:21:54.560 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98364273700000004 2024-09-13T13:21:54.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.561 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.561 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.181140+0000", 2024-09-13T13:21:54.561 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9752e-05 2024-09-13T13:21:54.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.561 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.561 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95616 11.15 11.fb407195 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:21:54.561 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:06.203442+0000", 2024-09-13T13:21:54.561 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 168.24036572599999, 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.977740787, 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95616 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.203442+0000", 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.203442+0000", 2024-09-13T13:21:54.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.563 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.203444+0000", 2024-09-13T13:21:54.563 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.446e-06 2024-09-13T13:21:54.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.563 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.203471+0000", 2024-09-13T13:21:54.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.6913000000000001e-05 2024-09-13T13:21:54.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.564 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.203486+0000", 2024-09-13T13:21:54.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5079000000000001e-05 2024-09-13T13:21:54.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.564 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.564 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.564 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.203494+0000", 2024-09-13T13:21:54.564 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2519999999999999e-06 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.203563+0000", 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8454999999999994e-05 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:06.203571+0000", 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2859999999999999e-06 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.565 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.181173+0000", 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.97760195800000005 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.181182+0000", 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3980000000000005e-06 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:121775 11.18 11:1e8dbdef:::benchmark_data_smithi052_182575_object4766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:37.169077+0000", 2024-09-13T13:21:54.566 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 137.27473008999999, 2024-09-13T13:21:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.007192788, 2024-09-13T13:21:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 121775 2024-09-13T13:21:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.169077+0000", 2024-09-13T13:21:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.169077+0000", 2024-09-13T13:21:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.169079+0000", 2024-09-13T13:21:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.655e-06 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.169084+0000", 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8849999999999998e-06 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.169099+0000", 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5503999999999999e-05 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.569 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.169107+0000", 2024-09-13T13:21:54.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.0390000000000005e-06 2024-09-13T13:21:54.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.570 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.570 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.169168+0000", 2024-09-13T13:21:54.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.0404e-05 2024-09-13T13:21:54.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.570 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.570 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.169177+0000", 2024-09-13T13:21:54.570 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3610000000000007e-06 2024-09-13T13:21:54.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.097472+0000", 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.928295333 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.097595+0000", 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000122232 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,6", 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.097746+0000", 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000150953 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.101514+0000", 2024-09-13T13:21:54.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037683769999999998 2024-09-13T13:21:54.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.112403+0000", 2024-09-13T13:21:54.572 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010889352 2024-09-13T13:21:54.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.572 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.572 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.176125+0000", 2024-09-13T13:21:54.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.063721611999999997 2024-09-13T13:21:54.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:54.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.176180+0000", 2024-09-13T13:21:54.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4842000000000001e-05 2024-09-13T13:21:54.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.573 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.573 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.176270+0000", 2024-09-13T13:21:54.573 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0239000000000001e-05 2024-09-13T13:21:54.573 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.573 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.574 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.574 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:121788 11.18 11:1fdb6ed1:::benchmark_data_smithi052_182575_object4779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-09-13T13:21:54.574 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:37.188174+0000", 2024-09-13T13:21:54.574 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 137.25563373, 2024-09-13T13:21:54.574 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98825778200000003, 2024-09-13T13:21:54.574 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.574 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.574 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:21:54.574 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.574 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:54.574 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:54.575 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 121788 2024-09-13T13:21:54.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.575 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.188174+0000", 2024-09-13T13:21:54.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.575 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.575 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.188174+0000", 2024-09-13T13:21:54.575 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.188176+0000", 2024-09-13T13:21:54.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.3240000000000001e-06 2024-09-13T13:21:54.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.576 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.576 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.188181+0000", 2024-09-13T13:21:54.576 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4190000000000001e-06 2024-09-13T13:21:54.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.188192+0000", 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0934000000000001e-05 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.188199+0000", 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9929999999999998e-06 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.188254+0000", 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.4827999999999998e-05 2024-09-13T13:21:54.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:37.188263+0000", 2024-09-13T13:21:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.5990000000000002e-06 2024-09-13T13:21:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.099497+0000", 2024-09-13T13:21:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.91123421299999996 2024-09-13T13:21:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.578 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.099609+0000", 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000112025 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,6", 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.099735+0000", 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012583399999999999 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.103222+0000", 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0034874179999999999 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.579 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.113485+0000", 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.010262485999999999 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.176358+0000", 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.062873374999999995 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.176399+0000", 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0877999999999997e-05 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.581 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.581 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:38.176431+0000", 2024-09-13T13:21:54.581 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.2456000000000002e-05 2024-09-13T13:21:54.581 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.581 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.581 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.581 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:222100 11.d 11:bb3b6da2:::benchmark_data_smithi052_182575_object105091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-13T13:21:54.581 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:21:25.044736+0000", 2024-09-13T13:21:54.581 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 29.399071116000002, 2024-09-13T13:21:54.582 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1912501849999999, 2024-09-13T13:21:54.582 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:21:54.582 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:21:54.582 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:21:54.582 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:21:54.582 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:21:54.582 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:21:54.582 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 222100 2024-09-13T13:21:54.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.582 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:21:54.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.582 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:21:54.582 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.044736+0000", 2024-09-13T13:21:54.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:21:54.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.044736+0000", 2024-09-13T13:21:54.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:21:54.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.583 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:21:54.583 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.044738+0000", 2024-09-13T13:21:54.583 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2029999999999999e-06 2024-09-13T13:21:54.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:21:54.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.044744+0000", 2024-09-13T13:21:54.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6280000000000002e-06 2024-09-13T13:21:54.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:21:54.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.044760+0000", 2024-09-13T13:21:54.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5775999999999999e-05 2024-09-13T13:21:54.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.584 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:21:54.584 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.044769+0000", 2024-09-13T13:21:54.584 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6959999999999994e-06 2024-09-13T13:21:54.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.044843+0000", 2024-09-13T13:21:54.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4546000000000001e-05 2024-09-13T13:21:54.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:21:54.585 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.044965+0000", 2024-09-13T13:21:54.585 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012219999999999999 2024-09-13T13:21:54.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.585 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,6", 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.045076+0000", 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011113899999999999 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.048475+0000", 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0033981380000000002 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.050467+0000", 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019921909999999999 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.163208+0000", 2024-09-13T13:21:54.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.112741574 2024-09-13T13:21:54.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:21:54.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:25.163222+0000", 2024-09-13T13:21:54.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.345e-05 2024-09-13T13:21:54.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.587 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:21:54.587 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.235875+0000", 2024-09-13T13:21:54.587 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0726534809999999 2024-09-13T13:21:54.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:21:54.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:21:54.588 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:21:54.588 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:21:26.235986+0000", 2024-09-13T13:21:54.588 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011116300000000001 2024-09-13T13:21:54.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.588 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:21:54.588 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:21:54.588 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:21:54.589 INFO:teuthology.orchestra.run.smithi052.stderr:noscrub is unset 2024-09-13T13:21:54.615 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:54 smithi167 ceph-mon[33178]: 3.c scrub starts 2024-09-13T13:21:54.616 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:54 smithi167 ceph-mon[33178]: 3.c scrub ok 2024-09-13T13:21:54.616 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:54 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/92349039' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:21:54.616 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:54 smithi167 ceph-mon[33178]: 4.0 scrub starts 2024-09-13T13:21:54.616 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:54 smithi167 ceph-mon[33178]: 4.0 scrub ok 2024-09-13T13:21:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:54 smithi052 ceph-mon[29471]: 3.c scrub starts 2024-09-13T13:21:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:54 smithi052 ceph-mon[29471]: 3.c scrub ok 2024-09-13T13:21:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:54 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/92349039' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:21:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:54 smithi052 ceph-mon[29471]: 4.0 scrub starts 2024-09-13T13:21:54.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:54 smithi052 ceph-mon[29471]: 4.0 scrub ok 2024-09-13T13:21:54.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:54 smithi052 ceph-mon[36682]: 3.c scrub starts 2024-09-13T13:21:54.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:54 smithi052 ceph-mon[36682]: 3.c scrub ok 2024-09-13T13:21:54.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:54 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/92349039' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T13:21:54.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:54 smithi052 ceph-mon[36682]: 4.0 scrub starts 2024-09-13T13:21:54.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:54 smithi052 ceph-mon[36682]: 4.0 scrub ok 2024-09-13T13:21:55.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:21:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:21:55.138 INFO:teuthology.orchestra.run.smithi167.stdout:get_pg_num_history pg_num_history pg_num_history(e502 pg_nums {1={418=11},2={59=3},3={59=32},4={61=32},5={63=32},6={65=32},11={423=16,450=26}} deleted_pools ) 2024-09-13T13:21:55.138 INFO:teuthology.orchestra.run.smithi167.stdout:Importing pgid 6.16 2024-09-13T13:21:55.138 INFO:teuthology.orchestra.run.smithi167.stdout:write_pg epoch 467 info 6.16( empty local-lis/les=466/467 n=0 ec=65/65 lis/c=466/466 les/c/f=467/467/0 sis=466) 2024-09-13T13:21:55.139 INFO:teuthology.orchestra.run.smithi167.stdout:Import successful 2024-09-13T13:21:55.317 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T13:21:55.508 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:55 smithi167 ceph-mon[33178]: pgmap v1058: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:55.509 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:55 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/92349039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:21:55.509 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:55 smithi167 ceph-mon[33178]: osdmap e517: 8 total, 7 up, 7 in 2024-09-13T13:21:55.509 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:55 smithi167 ceph-mon[33178]: 1.7 scrub starts 2024-09-13T13:21:55.509 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:55 smithi167 ceph-mon[33178]: 1.7 scrub ok 2024-09-13T13:21:55.509 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:55 smithi167 ceph-mon[33178]: 6.1c scrub starts 2024-09-13T13:21:55.509 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:55 smithi167 ceph-mon[33178]: 6.1c scrub ok 2024-09-13T13:21:55.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[29471]: pgmap v1058: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:55.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/92349039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:21:55.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[29471]: osdmap e517: 8 total, 7 up, 7 in 2024-09-13T13:21:55.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[29471]: 1.7 scrub starts 2024-09-13T13:21:55.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[29471]: 1.7 scrub ok 2024-09-13T13:21:55.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[29471]: 6.1c scrub starts 2024-09-13T13:21:55.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[29471]: 6.1c scrub ok 2024-09-13T13:21:55.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:55.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[29471]: 6.8 scrub starts 2024-09-13T13:21:55.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[29471]: 6.8 scrub ok 2024-09-13T13:21:55.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[36682]: pgmap v1058: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:55.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/92349039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T13:21:55.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[36682]: osdmap e517: 8 total, 7 up, 7 in 2024-09-13T13:21:55.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[36682]: 1.7 scrub starts 2024-09-13T13:21:55.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[36682]: 1.7 scrub ok 2024-09-13T13:21:55.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[36682]: 6.1c scrub starts 2024-09-13T13:21:55.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[36682]: 6.1c scrub ok 2024-09-13T13:21:55.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:55.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[36682]: 6.8 scrub starts 2024-09-13T13:21:55.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:55 smithi052 ceph-mon[36682]: 6.8 scrub ok 2024-09-13T13:21:55.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:21:55.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:55 smithi167 ceph-mon[33178]: 6.8 scrub starts 2024-09-13T13:21:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:55 smithi167 ceph-mon[33178]: 6.8 scrub ok 2024-09-13T13:21:56.167 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T13:21:56.235 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:56.395 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:21:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:21:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:21:56.436 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/exp.6.16.4 2024-09-13T13:21:56.664 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:56 smithi052 ceph-mon[36682]: 4.7 scrub starts 2024-09-13T13:21:56.665 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:56 smithi052 ceph-mon[36682]: 4.7 scrub ok 2024-09-13T13:21:56.665 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:56 smithi052 ceph-mon[29471]: 4.7 scrub starts 2024-09-13T13:21:56.665 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:56 smithi052 ceph-mon[29471]: 4.7 scrub ok 2024-09-13T13:21:56.771 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:21:56.771 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:21:56.771 INFO:teuthology.orchestra.run.smithi052.stdout: "num_ops": 0 2024-09-13T13:21:56.771 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:21:56.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:56 smithi167 ceph-mon[33178]: 4.7 scrub starts 2024-09-13T13:21:56.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:56 smithi167 ceph-mon[33178]: 4.7 scrub ok 2024-09-13T13:21:56.978 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:57.476 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T13:21:57.670 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:57 smithi052 ceph-mon[36682]: pgmap v1060: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:57.670 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:57 smithi052 ceph-mon[29471]: pgmap v1060: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:57.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:57 smithi167 ceph-mon[33178]: pgmap v1060: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:58.054 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:21:58.528 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:21:58.529 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [], 2024-09-13T13:21:58.529 INFO:teuthology.orchestra.run.smithi052.stdout: "complaint_time": 30, 2024-09-13T13:21:58.529 INFO:teuthology.orchestra.run.smithi052.stdout: "num_blocked_ops": 0 2024-09-13T13:21:58.529 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:21:59.164 INFO:teuthology.orchestra.run.smithi052.stderr:nodeep-scrub is unset 2024-09-13T13:21:59.178 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T13:21:59.445 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[29471]: 4.1c scrub starts 2024-09-13T13:21:59.445 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[29471]: 4.1c scrub ok 2024-09-13T13:21:59.445 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[29471]: pgmap v1061: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:59.445 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/444789215' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:59.445 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[29471]: from='client.36977 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:59.445 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[29471]: 5.14 scrub starts 2024-09-13T13:21:59.445 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[29471]: 5.14 scrub ok 2024-09-13T13:21:59.445 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[36682]: 4.1c scrub starts 2024-09-13T13:21:59.446 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[36682]: 4.1c scrub ok 2024-09-13T13:21:59.446 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[36682]: pgmap v1061: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:59.446 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/444789215' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:59.446 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[36682]: from='client.36977 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:59.446 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[36682]: 5.14 scrub starts 2024-09-13T13:21:59.446 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:21:59 smithi052 ceph-mon[36682]: 5.14 scrub ok 2024-09-13T13:21:59.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:59 smithi167 ceph-mon[33178]: 4.1c scrub starts 2024-09-13T13:21:59.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:59 smithi167 ceph-mon[33178]: 4.1c scrub ok 2024-09-13T13:21:59.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:59 smithi167 ceph-mon[33178]: pgmap v1061: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:21:59.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:59 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/444789215' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:59.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:59 smithi167 ceph-mon[33178]: from='client.36977 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T13:21:59.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:59 smithi167 ceph-mon[33178]: 5.14 scrub starts 2024-09-13T13:21:59.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:21:59 smithi167 ceph-mon[33178]: 5.14 scrub ok 2024-09-13T13:22:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:00 smithi052 ceph-mon[36682]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:22:00.437 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:00 smithi052 ceph-mon[36682]: from='client.36977 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:22:00.438 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:00 smithi052 ceph-mon[36682]: osdmap e518: 8 total, 7 up, 7 in 2024-09-13T13:22:00.438 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:00 smithi052 ceph-mon[36682]: 1.9 scrub starts 2024-09-13T13:22:00.438 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:00 smithi052 ceph-mon[36682]: 1.9 scrub ok 2024-09-13T13:22:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:00 smithi052 ceph-mon[29471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:22:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:00 smithi052 ceph-mon[29471]: from='client.36977 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:22:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:00 smithi052 ceph-mon[29471]: osdmap e518: 8 total, 7 up, 7 in 2024-09-13T13:22:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:00 smithi052 ceph-mon[29471]: 1.9 scrub starts 2024-09-13T13:22:00.438 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:00 smithi052 ceph-mon[29471]: 1.9 scrub ok 2024-09-13T13:22:00.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:00 smithi167 ceph-mon[33178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T13:22:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:00 smithi167 ceph-mon[33178]: from='client.36977 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T13:22:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:00 smithi167 ceph-mon[33178]: osdmap e518: 8 total, 7 up, 7 in 2024-09-13T13:22:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:00 smithi167 ceph-mon[33178]: 1.9 scrub starts 2024-09-13T13:22:00.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:00 smithi167 ceph-mon[33178]: 1.9 scrub ok 2024-09-13T13:22:00.576 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:01.081 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-09-13T13:22:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "size": 20, 2024-09-13T13:22:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 600, 2024-09-13T13:22:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "ops": [ 2024-09-13T13:22:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190217 9.c 9:3da1d190:::benchmark_data_smithi052_128427_object59548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:22:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:43.982184+0000", 2024-09-13T13:22:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 557.09667955299994, 2024-09-13T13:22:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.99036358000000002, 2024-09-13T13:22:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.082 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190217 2024-09-13T13:22:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982184+0000", 2024-09-13T13:22:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982184+0000", 2024-09-13T13:22:01.083 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982185+0000", 2024-09-13T13:22:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5859999999999999e-06 2024-09-13T13:22:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982190+0000", 2024-09-13T13:22:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.9749999999999998e-06 2024-09-13T13:22:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.084 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982200+0000", 2024-09-13T13:22:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2310000000000002e-06 2024-09-13T13:22:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982209+0000", 2024-09-13T13:22:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9220000000000003e-06 2024-09-13T13:22:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982540+0000", 2024-09-13T13:22:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033174499999999999 2024-09-13T13:22:01.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982550+0000", 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5629999999999998e-06 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.966848+0000", 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.9842978 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.966978+0000", 2024-09-13T13:22:01.086 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013044200000000001 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.967121+0000", 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000143096 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972248+0000", 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051265260000000002 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.087 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972262+0000", 2024-09-13T13:22:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4037e-05 2024-09-13T13:22:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972452+0000", 2024-09-13T13:22:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019067699999999999 2024-09-13T13:22:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972517+0000", 2024-09-13T13:22:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4935000000000001e-05 2024-09-13T13:22:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972547+0000", 2024-09-13T13:22:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0045000000000001e-05 2024-09-13T13:22:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190219 9.c 9:30476813:::benchmark_data_smithi052_128427_object59550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:22:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:43.982580+0000", 2024-09-13T13:22:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 557.09628341099994, 2024-09-13T13:22:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.990151738, 2024-09-13T13:22:01.089 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190219 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982580+0000", 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982580+0000", 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982581+0000", 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0249999999999999e-06 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982585+0000", 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1189999999999999e-06 2024-09-13T13:22:01.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982595+0000", 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0074999999999999e-05 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982601+0000", 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.2149999999999998e-06 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982868+0000", 2024-09-13T13:22:01.092 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00026633000000000001 2024-09-13T13:22:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:43.982876+0000", 2024-09-13T13:22:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8900000000000007e-06 2024-09-13T13:22:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.969658+0000", 2024-09-13T13:22:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.98678257999999996 2024-09-13T13:22:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.093 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.969792+0000", 2024-09-13T13:22:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013425500000000001 2024-09-13T13:22:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:22:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.969926+0000", 2024-09-13T13:22:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013390200000000001 2024-09-13T13:22:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972556+0000", 2024-09-13T13:22:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0026292170000000001 2024-09-13T13:22:01.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972600+0000", 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4697999999999998e-05 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972657+0000", 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6453000000000003e-05 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972711+0000", 2024-09-13T13:22:01.095 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.452e-05 2024-09-13T13:22:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972732+0000", 2024-09-13T13:22:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.0458999999999998e-05 2024-09-13T13:22:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190252 9.c 9:3916637a:::benchmark_data_smithi052_128427_object59583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:22:01.096 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.013719+0000", 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 557.06514468499995, 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96423604299999999, 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190252 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.097 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013719+0000", 2024-09-13T13:22:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013719+0000", 2024-09-13T13:22:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013720+0000", 2024-09-13T13:22:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.7039999999999999e-06 2024-09-13T13:22:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013726+0000", 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3299999999999998e-06 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013745+0000", 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.9623000000000002e-05 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.013753+0000", 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.498e-06 2024-09-13T13:22:01.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.014204+0000", 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00045132700000000001 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.014212+0000", 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1510000000000004e-06 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972762+0000", 2024-09-13T13:22:01.100 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.958549706 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972843+0000", 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1415000000000006e-05 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.972915+0000", 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.101 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976630+0000", 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0037151879999999999 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976739+0000", 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109011 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.977890+0000", 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0011513350000000001 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.977916+0000", 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5948e-05 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.977955+0000", 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8451999999999997e-05 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190263 9.c 9:307a9428:::benchmark_data_smithi052_128427_object59594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:22:01.103 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.022090+0000", 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 557.05677375599998, 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95977093599999996, 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190263 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.104 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022090+0000", 2024-09-13T13:22:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022090+0000", 2024-09-13T13:22:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022092+0000", 2024-09-13T13:22:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8870000000000001e-06 2024-09-13T13:22:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022097+0000", 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.31e-06 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022108+0000", 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0941000000000001e-05 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022115+0000", 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3050000000000004e-06 2024-09-13T13:22:01.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022210+0000", 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.5169999999999999e-05 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022219+0000", 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1500000000000005e-06 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.974786+0000", 2024-09-13T13:22:01.107 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95256630799999997 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.974884+0000", 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8256000000000003e-05 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.975003+0000", 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011911100000000001 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.108 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976679+0000", 2024-09-13T13:22:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016762019999999999 2024-09-13T13:22:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.978004+0000", 2024-09-13T13:22:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.001324605 2024-09-13T13:22:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981764+0000", 2024-09-13T13:22:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.003760556 2024-09-13T13:22:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981816+0000", 2024-09-13T13:22:01.110 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1412999999999997e-05 2024-09-13T13:22:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981861+0000", 2024-09-13T13:22:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4722000000000003e-05 2024-09-13T13:22:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190265 9.c 9:3c2318c0:::benchmark_data_smithi052_128427_object59596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:22:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.022643+0000", 2024-09-13T13:22:01.111 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 557.05622069699996, 2024-09-13T13:22:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95946595400000001, 2024-09-13T13:22:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190265 2024-09-13T13:22:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.112 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022643+0000", 2024-09-13T13:22:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022643+0000", 2024-09-13T13:22:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022644+0000", 2024-09-13T13:22:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.282e-06 2024-09-13T13:22:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022648+0000", 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3189999999999996e-06 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022657+0000", 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1710000000000008e-06 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022664+0000", 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.8600000000000004e-06 2024-09-13T13:22:01.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022843+0000", 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00017918 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.022851+0000", 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5970000000000003e-06 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976779+0000", 2024-09-13T13:22:01.115 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95392760200000004 2024-09-13T13:22:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976846+0000", 2024-09-13T13:22:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7539000000000005e-05 2024-09-13T13:22:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:22:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.976913+0000", 2024-09-13T13:22:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.6785999999999994e-05 2024-09-13T13:22:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.116 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981913+0000", 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050001179999999996 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.981978+0000", 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4917000000000002e-05 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982009+0000", 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.0499999999999999e-05 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982067+0000", 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8573000000000003e-05 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982109+0000", 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1510000000000001e-05 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.118 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190292 9.c 9:3187c4a7:::benchmark_data_smithi052_128427_object59623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.039937+0000", 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 557.03892674500003, 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95233577000000003, 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190292 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.119 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039937+0000", 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039937+0000", 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039938+0000", 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.776e-06 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039944+0000", 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5949999999999998e-06 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039959+0000", 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.4854e-05 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.039966+0000", 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.2239999999999998e-06 2024-09-13T13:22:01.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.040018+0000", 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1578e-05 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.040025+0000", 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.6690000000000006e-06 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982498+0000", 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.94247312100000002 2024-09-13T13:22:01.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982590+0000", 2024-09-13T13:22:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1283e-05 2024-09-13T13:22:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:22:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.982692+0000", 2024-09-13T13:22:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102144 2024-09-13T13:22:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.124 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.987375+0000", 2024-09-13T13:22:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0046834329999999999 2024-09-13T13:22:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.987419+0000", 2024-09-13T13:22:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3282999999999999e-05 2024-09-13T13:22:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992172+0000", 2024-09-13T13:22:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0047538839999999999 2024-09-13T13:22:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992231+0000", 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8329999999999999e-05 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992272+0000", 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1596e-05 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.126 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190295 9.c 9:3895a0c0:::benchmark_data_smithi052_128427_object59626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.041745+0000", 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 557.03711824699997, 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.96494661000000004, 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190295 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041745+0000", 2024-09-13T13:22:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041745+0000", 2024-09-13T13:22:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041746+0000", 2024-09-13T13:22:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.2619999999999999e-06 2024-09-13T13:22:01.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041751+0000", 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.216e-06 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041759+0000", 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7129999999999994e-06 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041805+0000", 2024-09-13T13:22:01.129 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5739999999999999e-05 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041899+0000", 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.3598999999999996e-05 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.041907+0000", 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.4169999999999992e-06 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.130 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.984980+0000", 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.94307334399999998 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.985078+0000", 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7633000000000005e-05 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.985188+0000", 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000110245 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.131 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989611+0000", 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0044224889999999999 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.994760+0000", 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0051493700000000003 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.006626+0000", 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.011865504000000001 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.006651+0000", 2024-09-13T13:22:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.5629000000000001e-05 2024-09-13T13:22:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.006692+0000", 2024-09-13T13:22:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.0448999999999998e-05 2024-09-13T13:22:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190317 9.c 9:34b3085e:::benchmark_data_smithi052_128427_object59648:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.058087+0000", 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 557.02077665599995, 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95460188400000001, 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190317 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058087+0000", 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058087+0000", 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058088+0000", 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.192e-06 2024-09-13T13:22:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058093+0000", 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.7380000000000002e-06 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058102+0000", 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6590000000000002e-06 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058109+0000", 2024-09-13T13:22:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.9770000000000003e-06 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058175+0000", 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.5343000000000001e-05 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.058182+0000", 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8059999999999995e-06 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989661+0000", 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.931478994 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989768+0000", 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010650900000000001 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.989877+0000", 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000109343 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.994834+0000", 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049566610000000002 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.994870+0000", 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.6399999999999997e-05 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.012452+0000", 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.017581513999999999 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.012635+0000", 2024-09-13T13:22:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018336900000000001 2024-09-13T13:22:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.012689+0000", 2024-09-13T13:22:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.3378999999999998e-05 2024-09-13T13:22:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:190320 9.c 9:35ad6b23:::benchmark_data_smithi052_128427_object59651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:44.059653+0000", 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 557.01921048500003, 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.95393949300000003, 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 190320 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059653+0000", 2024-09-13T13:22:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059653+0000", 2024-09-13T13:22:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059654+0000", 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1370000000000001e-06 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059658+0000", 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.755e-06 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059668+0000", 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8709999999999992e-06 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059675+0000", 2024-09-13T13:22:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5870000000000004e-06 2024-09-13T13:22:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059730+0000", 2024-09-13T13:22:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5180999999999999e-05 2024-09-13T13:22:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.059739+0000", 2024-09-13T13:22:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.0569999999999999e-06 2024-09-13T13:22:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992325+0000", 2024-09-13T13:22:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.93258522899999996 2024-09-13T13:22:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992426+0000", 2024-09-13T13:22:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101094 2024-09-13T13:22:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,7", 2024-09-13T13:22:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.992530+0000", 2024-09-13T13:22:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000104051 2024-09-13T13:22:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:44.999704+0000", 2024-09-13T13:22:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0071743629999999996 2024-09-13T13:22:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.004507+0000", 2024-09-13T13:22:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0048030340000000003 2024-09-13T13:22:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.013507+0000", 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0089995439999999999 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.013555+0000", 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8572000000000002e-05 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:45.013592+0000", 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.7017999999999997e-05 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.147 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202437 9.c 9:31d379dd:::benchmark_data_smithi052_128427_object71768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-13T13:22:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.936285+0000", 2024-09-13T13:22:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 543.142578598, 2024-09-13T13:22:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0668128299999999, 2024-09-13T13:22:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:22:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.148 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202437 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936285+0000", 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936285+0000", 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936286+0000", 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.5540000000000001e-06 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936291+0000", 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0100000000000003e-06 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936305+0000", 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3562e-05 2024-09-13T13:22:01.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936312+0000", 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3780000000000003e-06 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936374+0000", 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1909999999999995e-05 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936528+0000", 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015417 2024-09-13T13:22:01.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 0,4,7", 2024-09-13T13:22:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.936642+0000", 2024-09-13T13:22:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011353000000000001 2024-09-13T13:22:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.941575+0000", 2024-09-13T13:22:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0049326969999999998 2024-09-13T13:22:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.152 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.947466+0000", 2024-09-13T13:22:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058910960000000002 2024-09-13T13:22:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.947475+0000", 2024-09-13T13:22:01.153 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6350000000000001e-06 2024-09-13T13:22:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.990524+0000", 2024-09-13T13:22:01.154 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0430484790000001 2024-09-13T13:22:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for missing object", 2024-09-13T13:22:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.990633+0000", 2024-09-13T13:22:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000108872 2024-09-13T13:22:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003024+0000", 2024-09-13T13:22:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.012390988 2024-09-13T13:22:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.155 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003098+0000", 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3949000000000006e-05 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202453 9.c 9:3d8e06d1:::benchmark_data_smithi052_128427_object71784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.956183+0000", 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 543.12268037700005, 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.056296729, 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.156 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202453 2024-09-13T13:22:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956183+0000", 2024-09-13T13:22:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.157 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956183+0000", 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956185+0000", 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.784e-06 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956190+0000", 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956200+0000", 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2900000000000008e-06 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956206+0000", 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.1489999999999999e-06 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957496+0000", 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012905519999999999 2024-09-13T13:22:01.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957505+0000", 2024-09-13T13:22:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9930000000000001e-06 2024-09-13T13:22:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003310+0000", 2024-09-13T13:22:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.045804368 2024-09-13T13:22:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.160 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003406+0000", 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6335000000000005e-05 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.003506+0000", 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000100295 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012143+0000", 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0086370389999999991 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.161 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012233+0000", 2024-09-13T13:22:01.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9973000000000001e-05 2024-09-13T13:22:01.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.162 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012279+0000", 2024-09-13T13:22:01.162 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5966e-05 2024-09-13T13:22:01.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.162 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012376+0000", 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6220999999999999e-05 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012480+0000", 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010412900000000001 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202455 9.c 9:304f4878:::benchmark_data_smithi052_128427_object71786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:22:01.163 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:22:01.164 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 543.12264546999995, 2024-09-13T13:22:01.164 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.071265975, 2024-09-13T13:22:01.164 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.164 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.164 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.164 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.164 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.164 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.164 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202455 2024-09-13T13:22:01.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.164 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.164 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956218+0000", 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.5000000000000003e-07 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.165 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956221+0000", 2024-09-13T13:22:01.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.441e-06 2024-09-13T13:22:01.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956226+0000", 2024-09-13T13:22:01.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0089999999999998e-06 2024-09-13T13:22:01.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.166 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.166 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.956230+0000", 2024-09-13T13:22:01.166 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5410000000000002e-06 2024-09-13T13:22:01.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957529+0000", 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0012988469999999999 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.957535+0000", 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.8109999999999999e-06 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.005874+0000", 2024-09-13T13:22:01.167 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0483387770000001 2024-09-13T13:22:01.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.005991+0000", 2024-09-13T13:22:01.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000116669 2024-09-13T13:22:01.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:22:01.168 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.006107+0000", 2024-09-13T13:22:01.168 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00011618 2024-09-13T13:22:01.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.168 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012522+0000", 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0064157060000000002 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012562+0000", 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9756999999999999e-05 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.026940+0000", 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.014377463 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027300+0000", 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00036073000000000002 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027484+0000", 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000183494 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.170 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202464 9.c 9:3602984d:::benchmark_data_smithi052_128427_object71795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:22:01.171 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:22:01.171 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 543.11757956999998, 2024-09-13T13:22:01.171 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.066828723, 2024-09-13T13:22:01.171 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.171 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.171 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.171 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.171 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.171 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.171 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202464 2024-09-13T13:22:01.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.171 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:22:01.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:22:01.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.172 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.172 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961284+0000", 2024-09-13T13:22:01.172 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.4099999999999999e-07 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961286+0000", 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.2010000000000002e-06 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961291+0000", 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.5909999999999998e-06 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.173 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.961295+0000", 2024-09-13T13:22:01.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1110000000000001e-06 2024-09-13T13:22:01.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.962375+0000", 2024-09-13T13:22:01.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00107939 2024-09-13T13:22:01.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.174 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.174 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.962382+0000", 2024-09-13T13:22:01.174 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.5900000000000002e-06 2024-09-13T13:22:01.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.008652+0000", 2024-09-13T13:22:01.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0462698880000001 2024-09-13T13:22:01.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.175 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.175 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.008749+0000", 2024-09-13T13:22:01.175 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7125000000000003e-05 2024-09-13T13:22:01.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.176 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:22:01.176 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.008858+0000", 2024-09-13T13:22:01.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010874699999999999 2024-09-13T13:22:01.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.176 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.176 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012711+0000", 2024-09-13T13:22:01.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0038533619999999999 2024-09-13T13:22:01.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.176 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.176 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012755+0000", 2024-09-13T13:22:01.176 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.3773000000000003e-05 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027802+0000", 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.015047115 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.027905+0000", 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000103278 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.177 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028113+0000", 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000207111 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202480 9.c 9:3e2ac5d3:::benchmark_data_smithi052_128427_object71811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.976129+0000", 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 543.10273457000005, 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.052223253, 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.178 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202480 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976129+0000", 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976129+0000", 2024-09-13T13:22:01.179 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976130+0000", 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3799999999999999e-06 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976135+0000", 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.2809999999999999e-06 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.180 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976147+0000", 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1182000000000001e-05 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976154+0000", 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.4440000000000002e-06 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976219+0000", 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.4414999999999999e-05 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.181 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.976227+0000", 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012784+0000", 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.036556815 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.012892+0000", 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00010768999999999999 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.013010+0000", 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000118244 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.018728+0000", 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057179079999999998 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.018768+0000", 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.985e-05 2024-09-13T13:22:01.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.184 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028119+0000", 2024-09-13T13:22:01.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0093518460000000005 2024-09-13T13:22:01.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.184 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028244+0000", 2024-09-13T13:22:01.184 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00012501799999999999 2024-09-13T13:22:01.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.184 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.184 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028352+0000", 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000107557 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202488 9.c 9:3231b30e:::benchmark_data_smithi052_128427_object71819:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:57.980401+0000", 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 543.09846209700004, 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0481403709999999, 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.185 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202488 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980401+0000", 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980401+0000", 2024-09-13T13:22:01.186 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980403+0000", 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.339e-06 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980408+0000", 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.1909999999999999e-06 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.187 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980421+0000", 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3267e-05 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980428+0000", 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0310000000000001e-06 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980485+0000", 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.6527999999999998e-05 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.188 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:57.980493+0000", 2024-09-13T13:22:01.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.2460000000000003e-06 2024-09-13T13:22:01.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.015559+0000", 2024-09-13T13:22:01.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0350661029999999 2024-09-13T13:22:01.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.189 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.189 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.015857+0000", 2024-09-13T13:22:01.189 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000297726 2024-09-13T13:22:01.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.016037+0000", 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018051499999999999 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.021797+0000", 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0057596000000000001 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.021836+0000", 2024-09-13T13:22:01.190 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.9180000000000001e-05 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028359+0000", 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0065231589999999997 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028449+0000", 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.9874e-05 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.191 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028542+0000", 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.2612e-05 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.15789.0:202524 9.c 9:322e12cc:::benchmark_data_smithi052_128427_object71855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:12:58.008117+0000", 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 543.07074682400003, 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0206121399999999, 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.192 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.193 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.15789", 2024-09-13T13:22:01.193 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/287136491", 2024-09-13T13:22:01.193 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 202524 2024-09-13T13:22:01.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.193 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.193 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008117+0000", 2024-09-13T13:22:01.193 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.193 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.193 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.193 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008117+0000", 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008118+0000", 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.3200000000000001e-06 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008123+0000", 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8189999999999999e-06 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.194 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008132+0000", 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.6979999999999993e-06 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008141+0000", 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.7620000000000001e-06 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008199+0000", 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.7942999999999997e-05 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.195 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:58.008208+0000", 2024-09-13T13:22:01.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1069999999999994e-06 2024-09-13T13:22:01.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.018947+0000", 2024-09-13T13:22:01.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0107389179999999 2024-09-13T13:22:01.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.196 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.196 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.019046+0000", 2024-09-13T13:22:01.196 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.8896999999999999e-05 2024-09-13T13:22:01.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:22:01.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.019147+0000", 2024-09-13T13:22:01.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000101001 2024-09-13T13:22:01.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.021878+0000", 2024-09-13T13:22:01.197 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.002730985 2024-09-13T13:22:01.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.197 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.197 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.026889+0000", 2024-09-13T13:22:01.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0050107529999999997 2024-09-13T13:22:01.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028548+0000", 2024-09-13T13:22:01.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0016596950000000001 2024-09-13T13:22:01.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.198 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.198 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028637+0000", 2024-09-13T13:22:01.198 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8687e-05 2024-09-13T13:22:01.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:12:59.028729+0000", 2024-09-13T13:22:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1555e-05 2024-09-13T13:22:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_repop(client.26071.0:178620 10.f e380/377 10:f7cb9e30:::benchmark_data_smithi052_153703_object51240:head v 380'11131, mlcod=380'11128)", 2024-09-13T13:22:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:16:22.237173+0000", 2024-09-13T13:22:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 338.841690863, 2024-09-13T13:22:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34096315700000002, 2024-09-13T13:22:01.199 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237173+0000", 2024-09-13T13:22:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237173+0000", 2024-09-13T13:22:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.201 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.201 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237174+0000", 2024-09-13T13:22:01.201 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.373e-06 2024-09-13T13:22:01.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.201 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.201 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237182+0000", 2024-09-13T13:22:01.201 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.9310000000000008e-06 2024-09-13T13:22:01.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.201 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.201 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237233+0000", 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 5.0778000000000001e-05 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237240+0000", 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.8120000000000008e-06 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237756+0000", 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000516009 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.202 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.237780+0000", 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.4009e-05 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.578044+0000", 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.34026351900000001 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:16:22.578136+0000", 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.1725999999999996e-05 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.203 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.204 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.204 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:94540 11.8 11:13a8cb6c:::benchmark_data_smithi052_182575_object94539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:22:01.204 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:04.195993+0000", 2024-09-13T13:22:01.204 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 176.88287084000001, 2024-09-13T13:22:01.204 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.020347348, 2024-09-13T13:22:01.204 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.204 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.204 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "reached pg", 2024-09-13T13:22:01.204 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.204 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:22:01.204 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 94540 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.195993+0000", 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.195993+0000", 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.195997+0000", 2024-09-13T13:22:01.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.2390000000000004e-06 2024-09-13T13:22:01.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.196038+0000", 2024-09-13T13:22:01.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.1275e-05 2024-09-13T13:22:01.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.206 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.206 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.203061+0000", 2024-09-13T13:22:01.206 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.007023151 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.203080+0000", 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.8929000000000001e-05 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.470011+0000", 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.26693045999999998 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.207 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:04.470019+0000", 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.8470000000000002e-06 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.198605+0000", 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.72858517899999997 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for degraded object", 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.202890+0000", 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0042850730000000004 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.208 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.216266+0000", 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.013376702000000001 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:05.216340+0000", 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.3492999999999996e-05 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95597 11.18 11:198e97c8:::benchmark_data_smithi052_182575_object95596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:22:01.209 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:07.184969+0000", 2024-09-13T13:22:01.210 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 173.893894491, 2024-09-13T13:22:01.210 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1262475439999999, 2024-09-13T13:22:01.210 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.210 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.210 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.210 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.210 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:22:01.210 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:01.210 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95597 2024-09-13T13:22:01.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.210 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.210 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184969+0000", 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184969+0000", 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.184972+0000", 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 2.7769999999999998e-06 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185010+0000", 2024-09-13T13:22:01.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.8263999999999998e-05 2024-09-13T13:22:01.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185023+0000", 2024-09-13T13:22:01.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.271e-05 2024-09-13T13:22:01.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.212 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.212 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185032+0000", 2024-09-13T13:22:01.212 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 9.7499999999999998e-06 2024-09-13T13:22:01.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.213 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.213 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.190626+0000", 2024-09-13T13:22:01.213 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055940989999999999 2024-09-13T13:22:01.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.213 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.213 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.190637+0000", 2024-09-13T13:22:01.213 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0614999999999999e-05 2024-09-13T13:22:01.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.213 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.302635+0000", 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.111997723 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.302817+0000", 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00018175 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4,5,6", 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.303156+0000", 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00033972700000000001 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.214 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.305064+0000", 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0019073569999999999 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.306588+0000", 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0015246109999999999 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.306637+0000", 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 4.8702999999999998e-05 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.215 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.310746+0000", 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0041093400000000004 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.311036+0000", 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00028986300000000001 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.311104+0000", 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 6.7554999999999999e-05 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.311216+0000", 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0001127 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: "description": "osd_op(client.36635.0:95626 11.18 11:1f52487a:::benchmark_data_smithi052_182575_object95625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: "initiated_at": "2024-09-13T13:19:07.185050+0000", 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 173.893813451, 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1376889400000001, 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: "continuous": false, 2024-09-13T13:22:01.217 INFO:teuthology.orchestra.run.smithi052.stdout: "type_data": { 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: "client_info": { 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: "client": "client.36635", 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: "tid": 95626 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: "events": [ 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "initiated", 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185050+0000", 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.218 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "header_read", 2024-09-13T13:22:01.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185050+0000", 2024-09-13T13:22:01.219 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0 2024-09-13T13:22:01.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.219 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "throttled", 2024-09-13T13:22:01.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185051+0000", 2024-09-13T13:22:01.219 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.023e-06 2024-09-13T13:22:01.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.219 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "all_read", 2024-09-13T13:22:01.219 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185086+0000", 2024-09-13T13:22:01.219 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 3.4774000000000003e-05 2024-09-13T13:22:01.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "dispatched", 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185096+0000", 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.0681e-05 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "queued_for_pg", 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.185105+0000", 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 8.1040000000000007e-06 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.190690+0000", 2024-09-13T13:22:01.220 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0055853229999999997 2024-09-13T13:22:01.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.221 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for peered", 2024-09-13T13:22:01.221 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:07.190701+0000", 2024-09-13T13:22:01.221 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.1103e-05 2024-09-13T13:22:01.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.221 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "reached_pg", 2024-09-13T13:22:01.221 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.306669+0000", 2024-09-13T13:22:01.221 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.11596831 2024-09-13T13:22:01.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.221 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "started", 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.306827+0000", 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00015811199999999999 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "waiting for subops from 2,4,5,6", 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.307460+0000", 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00063293599999999996 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.313317+0000", 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0058565539999999999 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "op_commit", 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.313333+0000", 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 1.6140000000000001e-05 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.313435+0000", 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.000102132 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.313634+0000", 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00019934400000000001 2024-09-13T13:22:01.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.322535+0000", 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.0089002040000000001 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "commit_sent", 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.322605+0000", 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 7.0468000000000004e-05 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: "event": "done", 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: "time": "2024-09-13T13:19:08.322739+0000", 2024-09-13T13:22:01.224 INFO:teuthology.orchestra.run.smithi052.stdout: "duration": 0.00013373200000000001 2024-09-13T13:22:01.225 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.225 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.225 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.225 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-09-13T13:22:01.225 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-09-13T13:22:01.225 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-09-13T13:22:01.238 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[36682]: 2.0 scrub starts 2024-09-13T13:22:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[36682]: 2.0 scrub ok 2024-09-13T13:22:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:22:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:22:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[36682]: pgmap v1063: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[36682]: osdmap e519: 8 total, 7 up, 7 in 2024-09-13T13:22:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[36682]: 4.15 scrub starts 2024-09-13T13:22:01.239 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[36682]: 4.15 scrub ok 2024-09-13T13:22:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[29471]: 2.0 scrub starts 2024-09-13T13:22:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[29471]: 2.0 scrub ok 2024-09-13T13:22:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:22:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:22:01.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[29471]: pgmap v1063: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[29471]: osdmap e519: 8 total, 7 up, 7 in 2024-09-13T13:22:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[29471]: 4.15 scrub starts 2024-09-13T13:22:01.240 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:01 smithi052 ceph-mon[29471]: 4.15 scrub ok 2024-09-13T13:22:01.466 INFO:tasks.thrashosds.thrasher:reviving osd 2024-09-13T13:22:01.466 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-09-13T13:22:01.466 INFO:tasks.cephadm.osd.4:Restarting osd.4 (starting--it wasn't running)... 2024-09-13T13:22:01.467 DEBUG:teuthology.orchestra.run.smithi167:osd.4> sudo journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.4.service 2024-09-13T13:22:01.469 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl start ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.4 2024-09-13T13:22:01.494 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:01 smithi167 ceph-mon[33178]: 2.0 scrub starts 2024-09-13T13:22:01.494 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:01 smithi167 ceph-mon[33178]: 2.0 scrub ok 2024-09-13T13:22:01.494 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:01 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:22:01.494 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:01 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:22:01.494 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:01 smithi167 ceph-mon[33178]: pgmap v1063: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:01.494 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:01 smithi167 ceph-mon[33178]: osdmap e519: 8 total, 7 up, 7 in 2024-09-13T13:22:01.494 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:01 smithi167 ceph-mon[33178]: 4.15 scrub starts 2024-09-13T13:22:01.494 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:01 smithi167 ceph-mon[33178]: 4.15 scrub ok 2024-09-13T13:22:01.772 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:01 smithi167 systemd[1]: Starting Ceph osd.4 for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:22:02.067 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:01 smithi167 podman[154157]: 2024-09-13 13:22:01.674730822 +0000 UTC m=+0.016137147 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:22:02.067 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:01 smithi167 podman[154157]: 2024-09-13 13:22:01.796547719 +0000 UTC m=+0.137954038 container create 81354cfadd1090fe01ea8d76b1dce4c7fc9d8f4cb9ba7c6bdaf8011ef35431f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-09-13T13:22:02.067 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:02 smithi167 podman[154157]: 2024-09-13 13:22:02.066329221 +0000 UTC m=+0.407735541 container init 81354cfadd1090fe01ea8d76b1dce4c7fc9d8f4cb9ba7c6bdaf8011ef35431f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, maintainer=Guillaume Abrioux ) 2024-09-13T13:22:02.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T13:22:02.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:02 smithi167 ceph-mon[33178]: 5.12 scrub starts 2024-09-13T13:22:02.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:02 smithi167 ceph-mon[33178]: 5.12 scrub ok 2024-09-13T13:22:02.321 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:02 smithi167 podman[154157]: 2024-09-13 13:22:02.070083317 +0000 UTC m=+0.411489634 container start 81354cfadd1090fe01ea8d76b1dce4c7fc9d8f4cb9ba7c6bdaf8011ef35431f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-13T13:22:02.321 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:02 smithi167 podman[154157]: 2024-09-13 13:22:02.157599035 +0000 UTC m=+0.499005368 container attach 81354cfadd1090fe01ea8d76b1dce4c7fc9d8f4cb9ba7c6bdaf8011ef35431f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-13T13:22:02.321 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:02 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate[154172]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T13:22:02.321 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:02 smithi167 bash[154157]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T13:22:02.321 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:02 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate[154172]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T13:22:02.321 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:02 smithi167 bash[154157]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T13:22:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:02 smithi052 ceph-mon[36682]: 5.12 scrub starts 2024-09-13T13:22:02.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:02 smithi052 ceph-mon[36682]: 5.12 scrub ok 2024-09-13T13:22:02.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:02 smithi052 ceph-mon[29471]: 5.12 scrub starts 2024-09-13T13:22:02.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:02 smithi052 ceph-mon[29471]: 5.12 scrub ok 2024-09-13T13:22:02.506 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:03.133 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:22:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:22:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-09-13T13:22:03.133 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:22:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:03 smithi052 ceph-mon[36682]: pgmap v1065: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:03 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:22:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:03 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:22:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:03 smithi052 ceph-mon[36682]: osdmap e520: 8 total, 7 up, 7 in 2024-09-13T13:22:03.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:03 smithi052 ceph-mon[36682]: osdmap e521: 8 total, 7 up, 7 in 2024-09-13T13:22:03.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:03 smithi052 ceph-mon[29471]: pgmap v1065: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:03 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:22:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:03 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:22:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:03 smithi052 ceph-mon[29471]: osdmap e520: 8 total, 7 up, 7 in 2024-09-13T13:22:03.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:03 smithi052 ceph-mon[29471]: osdmap e521: 8 total, 7 up, 7 in 2024-09-13T13:22:03.509 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:03 smithi167 ceph-mon[33178]: pgmap v1065: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:03.510 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:03 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:22:03.510 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:03 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:22:03.510 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:03 smithi167 ceph-mon[33178]: osdmap e520: 8 total, 7 up, 7 in 2024-09-13T13:22:03.510 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:03 smithi167 ceph-mon[33178]: osdmap e521: 8 total, 7 up, 7 in 2024-09-13T13:22:03.930 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T13:22:04.185 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:04.443 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:04 smithi167 ceph-mon[33178]: 5.2 scrub starts 2024-09-13T13:22:04.443 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:04 smithi167 ceph-mon[33178]: 5.2 scrub ok 2024-09-13T13:22:04.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:04 smithi052 ceph-mon[29471]: 5.2 scrub starts 2024-09-13T13:22:04.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:04 smithi052 ceph-mon[29471]: 5.2 scrub ok 2024-09-13T13:22:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:04 smithi052 ceph-mon[36682]: 5.2 scrub starts 2024-09-13T13:22:04.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:04 smithi052 ceph-mon[36682]: 5.2 scrub ok 2024-09-13T13:22:04.652 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:22:04.653 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:22:04.653 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:22:04.653 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:22:04.653 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:22:04.810 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate[154172]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T13:22:04.810 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 bash[154157]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T13:22:04.810 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate[154172]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-13T13:22:04.811 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 bash[154157]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-13T13:22:05.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:22:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:22:05.071 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate[154172]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-13T13:22:05.071 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 bash[154157]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-13T13:22:05.071 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate[154172]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-13T13:22:05.071 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 bash[154157]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-13T13:22:05.071 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate[154172]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-13T13:22:05.071 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 bash[154157]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-13T13:22:05.071 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate[154172]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T13:22:05.071 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 bash[154157]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T13:22:05.071 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate[154172]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-13T13:22:05.072 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:04 smithi167 bash[154157]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-13T13:22:05.468 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:05 smithi167 ceph-mon[33178]: 5.1e scrub starts 2024-09-13T13:22:05.468 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:05 smithi167 ceph-mon[33178]: 5.1e scrub ok 2024-09-13T13:22:05.468 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:05 smithi167 ceph-mon[33178]: 6.1 scrub starts 2024-09-13T13:22:05.468 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:05 smithi167 ceph-mon[33178]: 6.1 scrub ok 2024-09-13T13:22:05.468 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:05 smithi167 ceph-mon[33178]: pgmap v1068: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:05.468 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:05 smithi167 ceph-mon[33178]: 4.11 scrub starts 2024-09-13T13:22:05.469 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:05 smithi167 ceph-mon[33178]: 4.11 scrub ok 2024-09-13T13:22:05.469 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:05 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:22:05.469 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:05 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:22:05.469 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:05 smithi167 ceph-mon[33178]: 3.10 scrub starts 2024-09-13T13:22:05.469 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:05 smithi167 ceph-mon[33178]: 3.10 scrub ok 2024-09-13T13:22:05.469 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[29471]: 5.1e scrub starts 2024-09-13T13:22:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[29471]: 5.1e scrub ok 2024-09-13T13:22:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[29471]: 6.1 scrub starts 2024-09-13T13:22:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[29471]: 6.1 scrub ok 2024-09-13T13:22:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[29471]: pgmap v1068: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[29471]: 4.11 scrub starts 2024-09-13T13:22:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[29471]: 4.11 scrub ok 2024-09-13T13:22:05.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:22:05.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:22:05.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[29471]: 3.10 scrub starts 2024-09-13T13:22:05.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[29471]: 3.10 scrub ok 2024-09-13T13:22:05.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:05.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[36682]: 5.1e scrub starts 2024-09-13T13:22:05.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[36682]: 5.1e scrub ok 2024-09-13T13:22:05.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[36682]: 6.1 scrub starts 2024-09-13T13:22:05.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[36682]: 6.1 scrub ok 2024-09-13T13:22:05.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[36682]: pgmap v1068: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:05.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[36682]: 4.11 scrub starts 2024-09-13T13:22:05.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[36682]: 4.11 scrub ok 2024-09-13T13:22:05.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:22:05.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:22:05.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[36682]: 3.10 scrub starts 2024-09-13T13:22:05.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[36682]: 3.10 scrub ok 2024-09-13T13:22:05.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:05.496 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T13:22:05.820 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:05 smithi167 podman[154157]: 2024-09-13 13:22:05.562134664 +0000 UTC m=+3.903540997 container died 81354cfadd1090fe01ea8d76b1dce4c7fc9d8f4cb9ba7c6bdaf8011ef35431f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-13T13:22:06.265 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:06.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:22:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:22:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:22:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:06 smithi052 ceph-mon[29471]: 3.0 scrub starts 2024-09-13T13:22:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:06 smithi052 ceph-mon[29471]: 3.0 scrub ok 2024-09-13T13:22:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:06 smithi052 ceph-mon[29471]: 4.a scrub starts 2024-09-13T13:22:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:06 smithi052 ceph-mon[29471]: 4.a scrub ok 2024-09-13T13:22:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:06 smithi052 ceph-mon[29471]: 4.12 scrub starts 2024-09-13T13:22:06.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:06 smithi052 ceph-mon[29471]: 4.12 scrub ok 2024-09-13T13:22:06.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:06 smithi052 ceph-mon[36682]: 3.0 scrub starts 2024-09-13T13:22:06.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:06 smithi052 ceph-mon[36682]: 3.0 scrub ok 2024-09-13T13:22:06.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:06 smithi052 ceph-mon[36682]: 4.a scrub starts 2024-09-13T13:22:06.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:06 smithi052 ceph-mon[36682]: 4.a scrub ok 2024-09-13T13:22:06.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:06 smithi052 ceph-mon[36682]: 4.12 scrub starts 2024-09-13T13:22:06.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:06 smithi052 ceph-mon[36682]: 4.12 scrub ok 2024-09-13T13:22:06.510 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:06 smithi167 ceph-mon[33178]: 3.0 scrub starts 2024-09-13T13:22:06.510 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:06 smithi167 ceph-mon[33178]: 3.0 scrub ok 2024-09-13T13:22:06.510 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:06 smithi167 ceph-mon[33178]: 4.a scrub starts 2024-09-13T13:22:06.510 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:06 smithi167 ceph-mon[33178]: 4.a scrub ok 2024-09-13T13:22:06.510 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:06 smithi167 ceph-mon[33178]: 4.12 scrub starts 2024-09-13T13:22:06.511 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:06 smithi167 ceph-mon[33178]: 4.12 scrub ok 2024-09-13T13:22:06.511 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:06 smithi167 podman[154520]: 2024-09-13 13:22:06.197020262 +0000 UTC m=+1.253500437 container remove 81354cfadd1090fe01ea8d76b1dce4c7fc9d8f4cb9ba7c6bdaf8011ef35431f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-13T13:22:06.762 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:06 smithi167 podman[154648]: 2024-09-13 13:22:06.500757881 +0000 UTC m=+0.196858962 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:22:06.762 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:06 smithi167 podman[154648]: 2024-09-13 13:22:06.650563979 +0000 UTC m=+0.346665041 container create afb0d0c00411815038146794dc90f42f55bb5b3ec965fe0ecb87b5882d703db9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T13:22:07.328 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:22:07.439 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:07 smithi167 ceph-mon[33178]: 3.12 scrub starts 2024-09-13T13:22:07.439 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:07 smithi167 ceph-mon[33178]: 3.12 scrub ok 2024-09-13T13:22:07.439 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:07 smithi167 ceph-mon[33178]: pgmap v1069: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:07.440 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:07 smithi167 ceph-mon[33178]: 5.13 scrub starts 2024-09-13T13:22:07.440 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:07 smithi167 ceph-mon[33178]: 5.13 scrub ok 2024-09-13T13:22:07.440 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:22:07.440 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:22:07.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:22:07.441 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:22:07.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.441 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205092 10.9 10:9bc37bd5:::benchmark_data_smithi052_153703_object77712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-13T13:22:07.441 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.052783+0000", 2024-09-13T13:22:07.441 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 320.15546574799998, 2024-09-13T13:22:07.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2303267840000001, 2024-09-13T13:22:07.441 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.441 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.441 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:07.441 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.441 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:07.441 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205092 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.052783+0000", 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.052783+0000", 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.052783+0000", 2024-09-13T13:22:07.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7299999999999995e-07 2024-09-13T13:22:07.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.052791+0000", 2024-09-13T13:22:07.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6490000000000008e-06 2024-09-13T13:22:07.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.052797+0000", 2024-09-13T13:22:07.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0240000000000003e-06 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.052802+0000", 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3700000000000003e-06 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.053393+0000", 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00059056899999999999 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.053495+0000", 2024-09-13T13:22:07.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101637 2024-09-13T13:22:07.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4,7", 2024-09-13T13:22:07.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.053601+0000", 2024-09-13T13:22:07.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106696 2024-09-13T13:22:07.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.059189+0000", 2024-09-13T13:22:07.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055881150000000003 2024-09-13T13:22:07.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.059357+0000", 2024-09-13T13:22:07.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000167887 2024-09-13T13:22:07.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.059729+0000", 2024-09-13T13:22:07.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000371553 2024-09-13T13:22:07.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.192738+0000", 2024-09-13T13:22:07.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.133009234 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.192748+0000", 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5829999999999996e-06 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.283042+0000", 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0902940800000001 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.283109+0000", 2024-09-13T13:22:07.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7713999999999996e-05 2024-09-13T13:22:07.448 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.448 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.448 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.448 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205124 10.9 10:97a36986:::benchmark_data_smithi052_153703_object77744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:22:07.448 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.194292+0000", 2024-09-13T13:22:07.448 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 320.01395685400001, 2024-09-13T13:22:07.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1059571340000001, 2024-09-13T13:22:07.448 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.448 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.449 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:07.449 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.449 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:07.449 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:07.449 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205124 2024-09-13T13:22:07.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.449 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194292+0000", 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194292+0000", 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194292+0000", 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4200000000000001e-07 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194295+0000", 2024-09-13T13:22:07.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2310000000000001e-06 2024-09-13T13:22:07.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194300+0000", 2024-09-13T13:22:07.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6489999999999999e-06 2024-09-13T13:22:07.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194305+0000", 2024-09-13T13:22:07.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8119999999999998e-06 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194346+0000", 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1486000000000002e-05 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.194355+0000", 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1819999999999995e-06 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.283537+0000", 2024-09-13T13:22:07.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0891813960000001 2024-09-13T13:22:07.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.283632+0000", 2024-09-13T13:22:07.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4914000000000006e-05 2024-09-13T13:22:07.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:22:07.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.283740+0000", 2024-09-13T13:22:07.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108137 2024-09-13T13:22:07.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.292289+0000", 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085489859999999997 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.292329+0000", 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9690000000000001e-05 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300162+0000", 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078332079999999995 2024-09-13T13:22:07.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:07.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300209+0000", 2024-09-13T13:22:07.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6978000000000003e-05 2024-09-13T13:22:07.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300249+0000", 2024-09-13T13:22:07.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0022999999999999e-05 2024-09-13T13:22:07.455 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.455 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.455 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.456 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205128 10.9 10:90438080:::benchmark_data_smithi052_153703_object77748:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:22:07.456 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.204172+0000", 2024-09-13T13:22:07.456 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 320.004076011, 2024-09-13T13:22:07.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0962660289999999, 2024-09-13T13:22:07.456 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.456 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.456 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:07.456 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.456 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:07.456 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:07.456 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205128 2024-09-13T13:22:07.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.457 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204172+0000", 2024-09-13T13:22:07.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204172+0000", 2024-09-13T13:22:07.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204174+0000", 2024-09-13T13:22:07.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.747e-06 2024-09-13T13:22:07.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204180+0000", 2024-09-13T13:22:07.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1550000000000004e-06 2024-09-13T13:22:07.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204195+0000", 2024-09-13T13:22:07.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4477999999999999e-05 2024-09-13T13:22:07.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204203+0000", 2024-09-13T13:22:07.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8849999999999999e-06 2024-09-13T13:22:07.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204271+0000", 2024-09-13T13:22:07.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8397000000000003e-05 2024-09-13T13:22:07.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.204281+0000", 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0349e-05 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.286598+0000", 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.082317011 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.286697+0000", 2024-09-13T13:22:07.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8559000000000003e-05 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.286816+0000", 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119269 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300064+0000", 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013247765 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300261+0000", 2024-09-13T13:22:07.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019662099999999999 2024-09-13T13:22:07.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300310+0000", 2024-09-13T13:22:07.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9104999999999998e-05 2024-09-13T13:22:07.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:07.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300397+0000", 2024-09-13T13:22:07.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7416000000000005e-05 2024-09-13T13:22:07.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300438+0000", 2024-09-13T13:22:07.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1272000000000001e-05 2024-09-13T13:22:07.463 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.463 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.463 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.463 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205146 10.9 10:93785db0:::benchmark_data_smithi052_153703_object77766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:22:07.463 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.218025+0000", 2024-09-13T13:22:07.463 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 319.99022374999998, 2024-09-13T13:22:07.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.082663522, 2024-09-13T13:22:07.464 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.464 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.464 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:07.464 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.464 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:07.464 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:07.464 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205146 2024-09-13T13:22:07.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.464 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218025+0000", 2024-09-13T13:22:07.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218025+0000", 2024-09-13T13:22:07.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218027+0000", 2024-09-13T13:22:07.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.193e-06 2024-09-13T13:22:07.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218033+0000", 2024-09-13T13:22:07.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0549999999999996e-06 2024-09-13T13:22:07.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218045+0000", 2024-09-13T13:22:07.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1770000000000001e-05 2024-09-13T13:22:07.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218052+0000", 2024-09-13T13:22:07.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4580000000000004e-06 2024-09-13T13:22:07.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218182+0000", 2024-09-13T13:22:07.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012962600000000001 2024-09-13T13:22:07.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.218190+0000", 2024-09-13T13:22:07.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7800000000000001e-06 2024-09-13T13:22:07.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.289337+0000", 2024-09-13T13:22:07.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.071147294 2024-09-13T13:22:07.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.289501+0000", 2024-09-13T13:22:07.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016407999999999999 2024-09-13T13:22:07.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:22:07.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.289617+0000", 2024-09-13T13:22:07.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011618 2024-09-13T13:22:07.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300266+0000", 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010648502000000001 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300541+0000", 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027551399999999998 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300600+0000", 2024-09-13T13:22:07.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8829999999999997e-05 2024-09-13T13:22:07.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:07.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300652+0000", 2024-09-13T13:22:07.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2049999999999998e-05 2024-09-13T13:22:07.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300688+0000", 2024-09-13T13:22:07.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6189999999999997e-05 2024-09-13T13:22:07.470 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.470 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.471 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.471 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205179 10.9 10:98229baf:::benchmark_data_smithi052_153703_object77799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:22:07.471 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.251737+0000", 2024-09-13T13:22:07.471 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 319.95651122700002, 2024-09-13T13:22:07.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0491762529999999, 2024-09-13T13:22:07.471 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.471 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.471 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:07.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:07.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:07.472 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205179 2024-09-13T13:22:07.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.472 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251737+0000", 2024-09-13T13:22:07.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251737+0000", 2024-09-13T13:22:07.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251739+0000", 2024-09-13T13:22:07.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.272e-06 2024-09-13T13:22:07.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251748+0000", 2024-09-13T13:22:07.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2479999999999996e-06 2024-09-13T13:22:07.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251763+0000", 2024-09-13T13:22:07.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5529e-05 2024-09-13T13:22:07.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251771+0000", 2024-09-13T13:22:07.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6580000000000002e-06 2024-09-13T13:22:07.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251881+0000", 2024-09-13T13:22:07.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110259 2024-09-13T13:22:07.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.251889+0000", 2024-09-13T13:22:07.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7929999999999998e-06 2024-09-13T13:22:07.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.292357+0000", 2024-09-13T13:22:07.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.04046767 2024-09-13T13:22:07.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.292529+0000", 2024-09-13T13:22:07.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017212400000000001 2024-09-13T13:22:07.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:22:07.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.292646+0000", 2024-09-13T13:22:07.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011692999999999999 2024-09-13T13:22:07.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300270+0000", 2024-09-13T13:22:07.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.007624713 2024-09-13T13:22:07.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300763+0000", 2024-09-13T13:22:07.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00049291299999999999 2024-09-13T13:22:07.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300836+0000", 2024-09-13T13:22:07.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2694000000000006e-05 2024-09-13T13:22:07.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:07.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300879+0000", 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3108999999999997e-05 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300913+0000", 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4341000000000002e-05 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:205184 10.9 10:98702a24:::benchmark_data_smithi052_153703_object77804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:47.256046+0000", 2024-09-13T13:22:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 319.952202134, 2024-09-13T13:22:07.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0452002440000001, 2024-09-13T13:22:07.479 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.479 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.479 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:07.479 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.479 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:07.479 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:07.479 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 205184 2024-09-13T13:22:07.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.479 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256046+0000", 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256046+0000", 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256048+0000", 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.218e-06 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256054+0000", 2024-09-13T13:22:07.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3210000000000001e-06 2024-09-13T13:22:07.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256063+0000", 2024-09-13T13:22:07.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3530000000000001e-06 2024-09-13T13:22:07.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256070+0000", 2024-09-13T13:22:07.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3779999999999998e-06 2024-09-13T13:22:07.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256126+0000", 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6461e-05 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:47.256134+0000", 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0679999999999997e-06 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.295757+0000", 2024-09-13T13:22:07.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0396233669999999 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.295880+0000", 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012273699999999999 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.296241+0000", 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036082900000000001 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300453+0000", 2024-09-13T13:22:07.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042117359999999998 2024-09-13T13:22:07.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.300971+0000", 2024-09-13T13:22:07.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051871199999999995 2024-09-13T13:22:07.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.301033+0000", 2024-09-13T13:22:07.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1320000000000002e-05 2024-09-13T13:22:07.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:07.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.301208+0000", 2024-09-13T13:22:07.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017559800000000001 2024-09-13T13:22:07.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:48.301247+0000", 2024-09-13T13:22:07.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8145999999999997e-05 2024-09-13T13:22:07.485 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.485 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.485 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.486 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95600 11.17 11:e8d0f172:::benchmark_data_smithi052_182575_object95599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:22:07.486 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:07.181925+0000", 2024-09-13T13:22:07.486 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 180.026323752, 2024-09-13T13:22:07.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1157274699999999, 2024-09-13T13:22:07.486 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.486 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.486 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:07.486 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.486 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.486 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.486 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95600 2024-09-13T13:22:07.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.486 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.181925+0000", 2024-09-13T13:22:07.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.181925+0000", 2024-09-13T13:22:07.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.181927+0000", 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6469999999999998e-06 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182156+0000", 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022827399999999999 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182173+0000", 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182185+0000", 2024-09-13T13:22:07.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1039000000000001e-05 2024-09-13T13:22:07.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182405+0000", 2024-09-13T13:22:07.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022038400000000001 2024-09-13T13:22:07.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182419+0000", 2024-09-13T13:22:07.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4085e-05 2024-09-13T13:22:07.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.264403+0000", 2024-09-13T13:22:07.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.081984166 2024-09-13T13:22:07.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for degraded object", 2024-09-13T13:22:07.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.265376+0000", 2024-09-13T13:22:07.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097267800000000004 2024-09-13T13:22:07.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.297581+0000", 2024-09-13T13:22:07.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032205342999999997 2024-09-13T13:22:07.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.297652+0000", 2024-09-13T13:22:07.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0981000000000004e-05 2024-09-13T13:22:07.491 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.491 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.491 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.492 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95607 11.17 11:ed9982da:::benchmark_data_smithi052_182575_object95606:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:22:07.492 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:07.182240+0000", 2024-09-13T13:22:07.492 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 180.02600799000001, 2024-09-13T13:22:07.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1148083479999999, 2024-09-13T13:22:07.492 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.492 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.492 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:07.492 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.492 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.492 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.492 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95607 2024-09-13T13:22:07.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.492 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182240+0000", 2024-09-13T13:22:07.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182240+0000", 2024-09-13T13:22:07.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182242+0000", 2024-09-13T13:22:07.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.049e-06 2024-09-13T13:22:07.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182301+0000", 2024-09-13T13:22:07.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9796999999999997e-05 2024-09-13T13:22:07.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182315+0000", 2024-09-13T13:22:07.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4055e-05 2024-09-13T13:22:07.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182323+0000", 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6369999999999999e-06 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182447+0000", 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123769 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182455+0000", 2024-09-13T13:22:07.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7999999999999999e-06 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.265689+0000", 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0832345109999999 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.265872+0000", 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018342 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4,6,7", 2024-09-13T13:22:07.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.266251+0000", 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000378191 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.269810+0000", 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035597419999999999 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.272256+0000", 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024457329999999998 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.273037+0000", 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00078064599999999997 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.274833+0000", 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017961640000000001 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.296863+0000", 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022030315000000002 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:07.499 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:07 smithi167 podman[154648]: 2024-09-13 13:22:07.220974519 +0000 UTC m=+0.917075616 container init afb0d0c00411815038146794dc90f42f55bb5b3ec965fe0ecb87b5882d703db9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-13T13:22:07.499 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:07 smithi167 podman[154648]: 2024-09-13 13:22:07.224663538 +0000 UTC m=+0.920764600 container start afb0d0c00411815038146794dc90f42f55bb5b3ec965fe0ecb87b5882d703db9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-09-13T13:22:07.499 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:07 smithi167 bash[154648]: afb0d0c00411815038146794dc90f42f55bb5b3ec965fe0ecb87b5882d703db9 2024-09-13T13:22:07.499 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:07 smithi167 systemd[1]: Started Ceph osd.4 for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:22:07.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.296929+0000", 2024-09-13T13:22:07.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5969999999999993e-05 2024-09-13T13:22:07.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.297049+0000", 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119549 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95611 11.15 11:af332c66:::benchmark_data_smithi052_182575_object95610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:07.182392+0000", 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 180.02585622000001, 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1247146619999999, 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.501 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.502 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:07.502 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.502 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.502 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.502 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95611 2024-09-13T13:22:07.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.502 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182392+0000", 2024-09-13T13:22:07.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182392+0000", 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182393+0000", 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.176e-06 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182445+0000", 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1221999999999998e-05 2024-09-13T13:22:07.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182454+0000", 2024-09-13T13:22:07.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4499999999999993e-06 2024-09-13T13:22:07.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182461+0000", 2024-09-13T13:22:07.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6170000000000003e-06 2024-09-13T13:22:07.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182532+0000", 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1600999999999994e-05 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182539+0000", 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0110000000000003e-06 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.294072+0000", 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1115328870000001 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.294250+0000", 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000177639 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.294587+0000", 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033675399999999998 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.304389+0000", 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0098028639999999997 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.304402+0000", 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2520999999999999e-05 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.304448+0000", 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6421000000000002e-05 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.304730+0000", 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028191199999999999 2024-09-13T13:22:07.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.304766+0000", 2024-09-13T13:22:07.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5679999999999997e-05 2024-09-13T13:22:07.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.306847+0000", 2024-09-13T13:22:07.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020809079999999998 2024-09-13T13:22:07.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:07.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.306954+0000", 2024-09-13T13:22:07.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106662 2024-09-13T13:22:07.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.307107+0000", 2024-09-13T13:22:07.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153337 2024-09-13T13:22:07.509 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.509 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.509 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.509 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95614 11.17 11:eea889b7:::benchmark_data_smithi052_182575_object95613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:22:07.509 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:07.182482+0000", 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 180.02576629399999, 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.114794748, 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95614 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182482+0000", 2024-09-13T13:22:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182482+0000", 2024-09-13T13:22:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182483+0000", 2024-09-13T13:22:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0999999999999997e-07 2024-09-13T13:22:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182521+0000", 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7855000000000003e-05 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182528+0000", 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7229999999999998e-06 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182534+0000", 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8640000000000001e-06 2024-09-13T13:22:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182823+0000", 2024-09-13T13:22:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028997199999999999 2024-09-13T13:22:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182831+0000", 2024-09-13T13:22:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6429999999999995e-06 2024-09-13T13:22:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.267717+0000", 2024-09-13T13:22:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.084886239 2024-09-13T13:22:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.267912+0000", 2024-09-13T13:22:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000194828 2024-09-13T13:22:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4,6,7", 2024-09-13T13:22:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.268242+0000", 2024-09-13T13:22:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033029400000000002 2024-09-13T13:22:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.273078+0000", 2024-09-13T13:22:07.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048357690000000002 2024-09-13T13:22:07.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.273125+0000", 2024-09-13T13:22:07.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6848000000000003e-05 2024-09-13T13:22:07.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.273422+0000", 2024-09-13T13:22:07.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029665900000000001 2024-09-13T13:22:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.274905+0000", 2024-09-13T13:22:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001482964 2024-09-13T13:22:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.297097+0000", 2024-09-13T13:22:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022192492000000001 2024-09-13T13:22:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:07.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.297158+0000", 2024-09-13T13:22:07.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0800000000000001e-05 2024-09-13T13:22:07.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.297277+0000", 2024-09-13T13:22:07.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118888 2024-09-13T13:22:07.517 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.517 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.517 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.517 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95616 11.15 11:a98e02df:::benchmark_data_smithi052_182575_object95615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:22:07.518 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:07.182559+0000", 2024-09-13T13:22:07.518 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 180.02568902300001, 2024-09-13T13:22:07.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1249057250000001, 2024-09-13T13:22:07.518 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.518 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.518 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:07.518 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.518 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.518 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.518 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95616 2024-09-13T13:22:07.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.518 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182559+0000", 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182559+0000", 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182560+0000", 2024-09-13T13:22:07.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:22:07.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182604+0000", 2024-09-13T13:22:07.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3219000000000001e-05 2024-09-13T13:22:07.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182612+0000", 2024-09-13T13:22:07.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6910000000000003e-06 2024-09-13T13:22:07.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182618+0000", 2024-09-13T13:22:07.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0000000000000002e-06 2024-09-13T13:22:07.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.183166+0000", 2024-09-13T13:22:07.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00054816700000000001 2024-09-13T13:22:07.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.183175+0000", 2024-09-13T13:22:07.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6980000000000004e-06 2024-09-13T13:22:07.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.296619+0000", 2024-09-13T13:22:07.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.113443698 2024-09-13T13:22:07.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.296751+0000", 2024-09-13T13:22:07.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013257899999999999 2024-09-13T13:22:07.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-09-13T13:22:07.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.297950+0000", 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011986729999999999 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.304407+0000", 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064569559999999998 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.304485+0000", 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7683999999999997e-05 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.304693+0000", 2024-09-13T13:22:07.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020863300000000001 2024-09-13T13:22:07.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.305287+0000", 2024-09-13T13:22:07.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00059384399999999995 2024-09-13T13:22:07.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.305326+0000", 2024-09-13T13:22:07.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8455999999999999e-05 2024-09-13T13:22:07.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.307234+0000", 2024-09-13T13:22:07.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019087349999999999 2024-09-13T13:22:07.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:07.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.307336+0000", 2024-09-13T13:22:07.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101464 2024-09-13T13:22:07.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.307465+0000", 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129338 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95625 11.17 11:ecdc47f0:::benchmark_data_smithi052_182575_object95624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:07.182637+0000", 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 180.02561140700001, 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1148995070000001, 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:07.526 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95625 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182637+0000", 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182637+0000", 2024-09-13T13:22:07.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182638+0000", 2024-09-13T13:22:07.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:22:07.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182705+0000", 2024-09-13T13:22:07.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7164999999999998e-05 2024-09-13T13:22:07.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182717+0000", 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1983e-05 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182724+0000", 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0129999999999996e-06 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182855+0000", 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013093600000000001 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182860+0000", 2024-09-13T13:22:07.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5010000000000004e-06 2024-09-13T13:22:07.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.270147+0000", 2024-09-13T13:22:07.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0872863640000001 2024-09-13T13:22:07.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.270299+0000", 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015238799999999999 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4,6,7", 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.270641+0000", 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034196600000000003 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.273163+0000", 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002521368 2024-09-13T13:22:07.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.273208+0000", 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5729000000000002e-05 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.274955+0000", 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001746608 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.275844+0000", 2024-09-13T13:22:07.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088918500000000002 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.297378+0000", 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02153422 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.297432+0000", 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3891999999999998e-05 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.297537+0000", 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104388 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95627 11.15 11:ab569ea1:::benchmark_data_smithi052_182575_object95626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e453)", 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:07.182733+0000", 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 180.02551538899999, 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1295366849999999, 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:07.534 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.535 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.535 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.535 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95627 2024-09-13T13:22:07.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.535 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182733+0000", 2024-09-13T13:22:07.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182733+0000", 2024-09-13T13:22:07.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182734+0000", 2024-09-13T13:22:07.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9899999999999999e-07 2024-09-13T13:22:07.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182796+0000", 2024-09-13T13:22:07.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2624000000000002e-05 2024-09-13T13:22:07.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182804+0000", 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.498e-06 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182810+0000", 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.392e-06 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.183237+0000", 2024-09-13T13:22:07.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00042699300000000002 2024-09-13T13:22:07.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.183244+0000", 2024-09-13T13:22:07.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5139999999999998e-06 2024-09-13T13:22:07.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.300688+0000", 2024-09-13T13:22:07.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.117443889 2024-09-13T13:22:07.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.302565+0000", 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018769629999999999 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.302913+0000", 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000348551 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.305393+0000", 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002479654 2024-09-13T13:22:07.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.305471+0000", 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7608999999999995e-05 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.305528+0000", 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7785000000000003e-05 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.306807+0000", 2024-09-13T13:22:07.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001278775 2024-09-13T13:22:07.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.311073+0000", 2024-09-13T13:22:07.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042659769999999998 2024-09-13T13:22:07.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.311954+0000", 2024-09-13T13:22:07.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088105900000000001 2024-09-13T13:22:07.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:07.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.312095+0000", 2024-09-13T13:22:07.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140858 2024-09-13T13:22:07.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.312270+0000", 2024-09-13T13:22:07.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017474499999999999 2024-09-13T13:22:07.542 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.542 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.542 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.543 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:222092 11.15 11:a861d5db:::benchmark_data_smithi052_182575_object105083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-13T13:22:07.543 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:21:25.039399+0000", 2024-09-13T13:22:07.543 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.168849160999997, 2024-09-13T13:22:07.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1531876030000001, 2024-09-13T13:22:07.543 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.543 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.543 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:07.543 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.543 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.543 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.543 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 222092 2024-09-13T13:22:07.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.544 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039399+0000", 2024-09-13T13:22:07.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039399+0000", 2024-09-13T13:22:07.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039401+0000", 2024-09-13T13:22:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.702e-06 2024-09-13T13:22:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039407+0000", 2024-09-13T13:22:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5280000000000003e-06 2024-09-13T13:22:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039417+0000", 2024-09-13T13:22:07.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0427e-05 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039422+0000", 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3940000000000004e-06 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039480+0000", 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8007999999999998e-05 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039581+0000", 2024-09-13T13:22:07.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101097 2024-09-13T13:22:07.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-09-13T13:22:07.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.040036+0000", 2024-09-13T13:22:07.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045423399999999997 2024-09-13T13:22:07.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.045292+0000", 2024-09-13T13:22:07.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0052562490000000002 2024-09-13T13:22:07.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.045329+0000", 2024-09-13T13:22:07.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7468000000000001e-05 2024-09-13T13:22:07.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.049547+0000", 2024-09-13T13:22:07.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042172939999999999 2024-09-13T13:22:07.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.049654+0000", 2024-09-13T13:22:07.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107748 2024-09-13T13:22:07.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.049712+0000", 2024-09-13T13:22:07.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7309000000000003e-05 2024-09-13T13:22:07.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151110+0000", 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10139854800000001 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151126+0000", 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6013999999999999e-05 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.192536+0000", 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0414099349999999 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.192587+0000", 2024-09-13T13:22:07.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0648e-05 2024-09-13T13:22:07.551 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.551 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.551 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.551 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:222093 11.15 11:ab48f5a1:::benchmark_data_smithi052_182575_object105084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-13T13:22:07.551 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:21:25.039470+0000", 2024-09-13T13:22:07.551 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.168778641999999, 2024-09-13T13:22:07.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1533860220000001, 2024-09-13T13:22:07.551 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.551 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 222093 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039470+0000", 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039470+0000", 2024-09-13T13:22:07.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039471+0000", 2024-09-13T13:22:07.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0079999999999999e-06 2024-09-13T13:22:07.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039474+0000", 2024-09-13T13:22:07.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6219999999999998e-06 2024-09-13T13:22:07.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039482+0000", 2024-09-13T13:22:07.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2799999999999998e-06 2024-09-13T13:22:07.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.039489+0000", 2024-09-13T13:22:07.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8820000000000003e-06 2024-09-13T13:22:07.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.042317+0000", 2024-09-13T13:22:07.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028287609999999999 2024-09-13T13:22:07.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.042462+0000", 2024-09-13T13:22:07.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144573 2024-09-13T13:22:07.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-09-13T13:22:07.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.042877+0000", 2024-09-13T13:22:07.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00041524200000000002 2024-09-13T13:22:07.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.049606+0000", 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0067290320000000002 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.052873+0000", 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032664619999999999 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.052931+0000", 2024-09-13T13:22:07.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8391999999999999e-05 2024-09-13T13:22:07.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.052989+0000", 2024-09-13T13:22:07.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7756000000000001e-05 2024-09-13T13:22:07.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.053083+0000", 2024-09-13T13:22:07.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4440999999999999e-05 2024-09-13T13:22:07.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151186+0000", 2024-09-13T13:22:07.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098103214999999994 2024-09-13T13:22:07.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151196+0000", 2024-09-13T13:22:07.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7329999999999998e-06 2024-09-13T13:22:07.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.192819+0000", 2024-09-13T13:22:07.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0416228169999999 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.192856+0000", 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6806e-05 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:222097 11.15 11:acd25fdb:::benchmark_data_smithi052_182575_object105088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:21:25.044729+0000", 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.163519583000003, 2024-09-13T13:22:07.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.148205921, 2024-09-13T13:22:07.560 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.560 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.560 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:07.560 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.560 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.560 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.560 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 222097 2024-09-13T13:22:07.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.560 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.044729+0000", 2024-09-13T13:22:07.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.044729+0000", 2024-09-13T13:22:07.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.044730+0000", 2024-09-13T13:22:07.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3859999999999999e-06 2024-09-13T13:22:07.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.044735+0000", 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7929999999999997e-06 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.044745+0000", 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8719999999999997e-06 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.044753+0000", 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8089999999999993e-06 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.045356+0000", 2024-09-13T13:22:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00060352099999999998 2024-09-13T13:22:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.045468+0000", 2024-09-13T13:22:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011179999999999999 2024-09-13T13:22:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-09-13T13:22:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.045864+0000", 2024-09-13T13:22:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039633300000000002 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.053036+0000", 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0071718449999999996 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.053107+0000", 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0742999999999997e-05 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.053169+0000", 2024-09-13T13:22:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2168000000000005e-05 2024-09-13T13:22:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.053231+0000", 2024-09-13T13:22:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2277e-05 2024-09-13T13:22:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.058406+0000", 2024-09-13T13:22:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0051749459999999997 2024-09-13T13:22:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151243+0000", 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.092836703000000007 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151249+0000", 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3749999999999999e-06 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.192893+0000", 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0416431369999999 2024-09-13T13:22:07.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.192935+0000", 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2212999999999997e-05 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:222101 11.15 11:a9e8bcdf:::benchmark_data_smithi052_182575_object105092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:21:25.044768+0000", 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.163480757000002, 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.148221103, 2024-09-13T13:22:07.567 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 222101 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.044768+0000", 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.044768+0000", 2024-09-13T13:22:07.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.044768+0000", 2024-09-13T13:22:07.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2300000000000001e-07 2024-09-13T13:22:07.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.044771+0000", 2024-09-13T13:22:07.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6709999999999999e-06 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.044776+0000", 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1510000000000003e-06 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.044781+0000", 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7249999999999997e-06 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.049868+0000", 2024-09-13T13:22:07.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050872000000000001 2024-09-13T13:22:07.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.050013+0000", 2024-09-13T13:22:07.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014466199999999999 2024-09-13T13:22:07.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-09-13T13:22:07.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.050461+0000", 2024-09-13T13:22:07.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00044845900000000001 2024-09-13T13:22:07.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.053116+0000", 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026543439999999999 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.053287+0000", 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017153700000000001 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.053346+0000", 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8943000000000001e-05 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.058362+0000", 2024-09-13T13:22:07.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050164820000000001 2024-09-13T13:22:07.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.058540+0000", 2024-09-13T13:22:07.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000177629 2024-09-13T13:22:07.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151286+0000", 2024-09-13T13:22:07.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.092746040000000002 2024-09-13T13:22:07.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151295+0000", 2024-09-13T13:22:07.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9120000000000004e-06 2024-09-13T13:22:07.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.192957+0000", 2024-09-13T13:22:07.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0416623679999999 2024-09-13T13:22:07.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.192989+0000", 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1356999999999998e-05 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:222106 11.15 11:ac9a7f55:::benchmark_data_smithi052_182575_object105097:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:21:25.050887+0000", 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.157360971999999, 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.142256436, 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.575 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:07.576 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.576 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.576 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.576 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 222106 2024-09-13T13:22:07.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.576 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.050887+0000", 2024-09-13T13:22:07.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.050887+0000", 2024-09-13T13:22:07.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.050889+0000", 2024-09-13T13:22:07.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.905e-06 2024-09-13T13:22:07.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.050897+0000", 2024-09-13T13:22:07.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0839999999999992e-06 2024-09-13T13:22:07.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.050910+0000", 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2525e-05 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.050984+0000", 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3876999999999998e-05 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.053388+0000", 2024-09-13T13:22:07.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024042820000000002 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.053547+0000", 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015839700000000001 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.053992+0000", 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00044514700000000001 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.058423+0000", 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044313089999999996 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.058454+0000", 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0716999999999997e-05 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.058500+0000", 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6587e-05 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.058644+0000", 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014406999999999999 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151329+0000", 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.092684947000000004 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151334+0000", 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1139999999999997e-06 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.193112+0000", 2024-09-13T13:22:07.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0417779709999999 2024-09-13T13:22:07.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.193144+0000", 2024-09-13T13:22:07.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1504000000000002e-05 2024-09-13T13:22:07.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.582 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.583 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:222109 11.15 11:a970a2c2:::benchmark_data_smithi052_182575_object105100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-13T13:22:07.583 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:21:25.052807+0000", 2024-09-13T13:22:07.583 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.155441668000002, 2024-09-13T13:22:07.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.140392981, 2024-09-13T13:22:07.583 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.583 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.583 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:07.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.583 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 222109 2024-09-13T13:22:07.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.583 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.052807+0000", 2024-09-13T13:22:07.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.052807+0000", 2024-09-13T13:22:07.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.052809+0000", 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7549999999999999e-06 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.052814+0000", 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4999999999999999e-06 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.052825+0000", 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0998999999999999e-05 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.052833+0000", 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7999999999999999e-06 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.056019+0000", 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031863830000000001 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.056160+0000", 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014102900000000001 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.056435+0000", 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027488600000000001 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.058584+0000", 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021492019999999998 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.058601+0000", 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6786000000000002e-05 2024-09-13T13:22:07.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.058698+0000", 2024-09-13T13:22:07.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7133000000000007e-05 2024-09-13T13:22:07.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.060854+0000", 2024-09-13T13:22:07.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021553129999999998 2024-09-13T13:22:07.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151378+0000", 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090524257999999996 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151387+0000", 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1630000000000002e-06 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.193166+0000", 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0417791620000001 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.193200+0000", 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3612000000000003e-05 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:222112 11.15 11:ae7c9e42:::benchmark_data_smithi052_182575_object105103:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:21:25.055905+0000", 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.152343397999999, 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1373564949999999, 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:07.590 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 222112 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.055905+0000", 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.055905+0000", 2024-09-13T13:22:07.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:07.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:07.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.055907+0000", 2024-09-13T13:22:07.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.838e-06 2024-09-13T13:22:07.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:07.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.055913+0000", 2024-09-13T13:22:07.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9089999999999997e-06 2024-09-13T13:22:07.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.055926+0000", 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2829e-05 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.055933+0000", 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6539999999999999e-06 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.058741+0000", 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002807768 2024-09-13T13:22:07.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.058893+0000", 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015244400000000001 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.059283+0000", 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000389078 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.060895+0000", 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016122549999999999 2024-09-13T13:22:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.060935+0000", 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0275000000000003e-05 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.060952+0000", 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6949e-05 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.061304+0000", 2024-09-13T13:22:07.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000351584 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151410+0000", 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090106449000000005 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.151415+0000", 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1449999999999999e-06 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:07.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.193221+0000", 2024-09-13T13:22:07.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0418058219999999 2024-09-13T13:22:07.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:07.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:07.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:07.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.193262+0000", 2024-09-13T13:22:07.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0496e-05 2024-09-13T13:22:07.597 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.597 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.597 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.597 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:07.597 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:07.597 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:22:07.598 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:07 smithi052 ceph-mon[29471]: 3.12 scrub starts 2024-09-13T13:22:07.598 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:07 smithi052 ceph-mon[29471]: 3.12 scrub ok 2024-09-13T13:22:07.598 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:07 smithi052 ceph-mon[29471]: pgmap v1069: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:07.598 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:07 smithi052 ceph-mon[29471]: 5.13 scrub starts 2024-09-13T13:22:07.598 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:07 smithi052 ceph-mon[29471]: 5.13 scrub ok 2024-09-13T13:22:07.598 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:07 smithi052 ceph-mon[36682]: 3.12 scrub starts 2024-09-13T13:22:07.599 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:07 smithi052 ceph-mon[36682]: 3.12 scrub ok 2024-09-13T13:22:07.599 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:07 smithi052 ceph-mon[36682]: pgmap v1069: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:07.599 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:07 smithi052 ceph-mon[36682]: 5.13 scrub starts 2024-09-13T13:22:07.599 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:07 smithi052 ceph-mon[36682]: 5.13 scrub ok 2024-09-13T13:22:08.268 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:08 smithi167 ceph-mon[33178]: 4.17 scrub starts 2024-09-13T13:22:08.268 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:08 smithi167 ceph-mon[33178]: 4.17 scrub ok 2024-09-13T13:22:08.269 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:22:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=cleanup t=2024-09-13T13:22:08.125379413Z level=info msg="Completed cleanup jobs" duration=15.822541ms 2024-09-13T13:22:08.269 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:22:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=plugins.update.checker t=2024-09-13T13:22:08.267823554Z level=info msg="Update check succeeded" duration=67.525241ms 2024-09-13T13:22:08.488 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:08 smithi052 ceph-mon[36682]: 4.17 scrub starts 2024-09-13T13:22:08.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:08 smithi052 ceph-mon[36682]: 4.17 scrub ok 2024-09-13T13:22:08.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:08 smithi052 ceph-mon[29471]: 4.17 scrub starts 2024-09-13T13:22:08.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:08 smithi052 ceph-mon[29471]: 4.17 scrub ok 2024-09-13T13:22:08.570 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:22:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=grafana.update.checker t=2024-09-13T13:22:08.26812969Z level=info msg="Update check succeeded" duration=71.698392ms 2024-09-13T13:22:08.605 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T13:22:08.718 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:09.211 INFO:teuthology.orchestra.run.smithi167.stderr:no valid command found; 10 closest matches: 2024-09-13T13:22:09.211 INFO:teuthology.orchestra.run.smithi167.stderr:0 2024-09-13T13:22:09.212 INFO:teuthology.orchestra.run.smithi167.stderr:1 2024-09-13T13:22:09.212 INFO:teuthology.orchestra.run.smithi167.stderr:2 2024-09-13T13:22:09.212 INFO:teuthology.orchestra.run.smithi167.stderr:abort 2024-09-13T13:22:09.212 INFO:teuthology.orchestra.run.smithi167.stderr:assert 2024-09-13T13:22:09.212 INFO:teuthology.orchestra.run.smithi167.stderr:config diff 2024-09-13T13:22:09.212 INFO:teuthology.orchestra.run.smithi167.stderr:config diff get 2024-09-13T13:22:09.212 INFO:teuthology.orchestra.run.smithi167.stderr:config get 2024-09-13T13:22:09.212 INFO:teuthology.orchestra.run.smithi167.stderr:config help [] 2024-09-13T13:22:09.212 INFO:teuthology.orchestra.run.smithi167.stderr:config set ... 2024-09-13T13:22:09.212 INFO:teuthology.orchestra.run.smithi167.stderr:admin_socket: invalid command 2024-09-13T13:22:09.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:09 smithi052 ceph-mon[36682]: 5.b scrub starts 2024-09-13T13:22:09.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:09 smithi052 ceph-mon[36682]: 5.b scrub ok 2024-09-13T13:22:09.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:09 smithi052 ceph-mon[36682]: pgmap v1070: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 646 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:09.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:09 smithi052 ceph-mon[36682]: 1.6 scrub starts 2024-09-13T13:22:09.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:09 smithi052 ceph-mon[36682]: 1.6 scrub ok 2024-09-13T13:22:09.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:09 smithi052 ceph-mon[29471]: 5.b scrub starts 2024-09-13T13:22:09.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:09 smithi052 ceph-mon[29471]: 5.b scrub ok 2024-09-13T13:22:09.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:09 smithi052 ceph-mon[29471]: pgmap v1070: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 646 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:09.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:09 smithi052 ceph-mon[29471]: 1.6 scrub starts 2024-09-13T13:22:09.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:09 smithi052 ceph-mon[29471]: 1.6 scrub ok 2024-09-13T13:22:09.506 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:09 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:22:09.107+0000 7f4770e7b640 -1 Falling back to public interface 2024-09-13T13:22:09.506 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:09 smithi167 ceph-mon[33178]: 5.b scrub starts 2024-09-13T13:22:09.506 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:09 smithi167 ceph-mon[33178]: 5.b scrub ok 2024-09-13T13:22:09.506 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:09 smithi167 ceph-mon[33178]: pgmap v1070: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 646 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:09.506 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:09 smithi167 ceph-mon[33178]: 1.6 scrub starts 2024-09-13T13:22:09.506 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:09 smithi167 ceph-mon[33178]: 1.6 scrub ok 2024-09-13T13:22:09.836 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T13:22:09.836 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-09-13T13:22:09.893 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:10.436 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:22:10.436 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:22:10.436 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-09-13T13:22:10.436 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:22:10.436 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:10 smithi167 ceph-mon[33178]: 6.18 scrub starts 2024-09-13T13:22:10.436 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:10 smithi167 ceph-mon[33178]: 6.18 scrub ok 2024-09-13T13:22:10.436 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:10 smithi167 ceph-mon[33178]: 3.4 scrub starts 2024-09-13T13:22:10.437 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:10 smithi167 ceph-mon[33178]: 3.4 scrub ok 2024-09-13T13:22:10.437 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:10 smithi167 ceph-mon[33178]: 4.10 scrub starts 2024-09-13T13:22:10.437 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:10 smithi167 ceph-mon[33178]: 4.10 scrub ok 2024-09-13T13:22:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:10 smithi052 ceph-mon[29471]: 6.18 scrub starts 2024-09-13T13:22:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:10 smithi052 ceph-mon[29471]: 6.18 scrub ok 2024-09-13T13:22:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:10 smithi052 ceph-mon[29471]: 3.4 scrub starts 2024-09-13T13:22:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:10 smithi052 ceph-mon[29471]: 3.4 scrub ok 2024-09-13T13:22:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:10 smithi052 ceph-mon[29471]: 4.10 scrub starts 2024-09-13T13:22:10.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:10 smithi052 ceph-mon[29471]: 4.10 scrub ok 2024-09-13T13:22:10.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:10 smithi052 ceph-mon[36682]: 6.18 scrub starts 2024-09-13T13:22:10.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:10 smithi052 ceph-mon[36682]: 6.18 scrub ok 2024-09-13T13:22:10.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:10 smithi052 ceph-mon[36682]: 3.4 scrub starts 2024-09-13T13:22:10.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:10 smithi052 ceph-mon[36682]: 3.4 scrub ok 2024-09-13T13:22:10.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:10 smithi052 ceph-mon[36682]: 4.10 scrub starts 2024-09-13T13:22:10.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:10 smithi052 ceph-mon[36682]: 4.10 scrub ok 2024-09-13T13:22:11.196 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T13:22:11.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:11 smithi167 ceph-mon[33178]: 5.8 scrub starts 2024-09-13T13:22:11.320 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:11 smithi167 ceph-mon[33178]: 5.8 scrub ok 2024-09-13T13:22:11.321 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:11 smithi167 ceph-mon[33178]: pgmap v1071: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:11.456 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:11 smithi052 ceph-mon[36682]: 5.8 scrub starts 2024-09-13T13:22:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:11 smithi052 ceph-mon[36682]: 5.8 scrub ok 2024-09-13T13:22:11.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:11 smithi052 ceph-mon[36682]: pgmap v1071: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:11 smithi052 ceph-mon[29471]: 5.8 scrub starts 2024-09-13T13:22:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:11 smithi052 ceph-mon[29471]: 5.8 scrub ok 2024-09-13T13:22:11.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:11 smithi052 ceph-mon[29471]: pgmap v1071: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:11.953 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:22:11.953 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:22:11.953 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:22:11.953 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:22:11.953 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:22:12.470 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:12 smithi167 ceph-mon[33178]: 6.9 scrub starts 2024-09-13T13:22:12.471 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:12 smithi167 ceph-mon[33178]: 6.9 scrub ok 2024-09-13T13:22:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:12 smithi052 ceph-mon[29471]: 6.9 scrub starts 2024-09-13T13:22:12.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:12 smithi052 ceph-mon[29471]: 6.9 scrub ok 2024-09-13T13:22:12.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:12 smithi052 ceph-mon[36682]: 6.9 scrub starts 2024-09-13T13:22:12.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:12 smithi052 ceph-mon[36682]: 6.9 scrub ok 2024-09-13T13:22:12.908 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T13:22:13.175 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:13.459 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:13 smithi167 ceph-mon[33178]: pgmap v1072: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:13.460 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:22:13.200+0000 7f4770e7b640 -1 osd.4 502 log_to_monitors true 2024-09-13T13:22:13.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:13 smithi052 ceph-mon[29471]: pgmap v1072: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:13.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:13 smithi052 ceph-mon[36682]: pgmap v1072: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:13.663 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:22:13.663 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:22:13.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:22:13.663 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:22:13.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.663 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:133006 10.8 e366/334 10:1945bf5a:::benchmark_data_smithi052_153703_object5626:head v 366'8411, mlcod=366'8409)", 2024-09-13T13:22:13.663 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.291495+0000", 2024-09-13T13:22:13.663 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 394.36902113399998, 2024-09-13T13:22:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.36882864700000001, 2024-09-13T13:22:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291495+0000", 2024-09-13T13:22:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291495+0000", 2024-09-13T13:22:13.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291497+0000", 2024-09-13T13:22:13.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4109999999999999e-06 2024-09-13T13:22:13.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291503+0000", 2024-09-13T13:22:13.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2190000000000001e-06 2024-09-13T13:22:13.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291513+0000", 2024-09-13T13:22:13.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0451999999999999e-05 2024-09-13T13:22:13.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291520+0000", 2024-09-13T13:22:13.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.759e-06 2024-09-13T13:22:13.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291570+0000", 2024-09-13T13:22:13.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9979999999999999e-05 2024-09-13T13:22:13.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.291585+0000", 2024-09-13T13:22:13.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4688999999999999e-05 2024-09-13T13:22:13.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660271+0000", 2024-09-13T13:22:13.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.36868583799999999 2024-09-13T13:22:13.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660324+0000", 2024-09-13T13:22:13.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3298999999999999e-05 2024-09-13T13:22:13.668 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.668 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.668 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.668 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:133009 10.8 e366/334 10:1c6c448b:::benchmark_data_smithi052_153703_object5629:head v 366'8412, mlcod=366'8409)", 2024-09-13T13:22:13.668 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.292418+0000", 2024-09-13T13:22:13.668 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 394.368098159, 2024-09-13T13:22:13.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.380843235, 2024-09-13T13:22:13.668 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.669 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.669 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.669 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292418+0000", 2024-09-13T13:22:13.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292418+0000", 2024-09-13T13:22:13.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292419+0000", 2024-09-13T13:22:13.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.569999999999999e-07 2024-09-13T13:22:13.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.292425+0000", 2024-09-13T13:22:13.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2050000000000001e-06 2024-09-13T13:22:13.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660256+0000", 2024-09-13T13:22:13.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.36783123499999998 2024-09-13T13:22:13.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660266+0000", 2024-09-13T13:22:13.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0210000000000001e-05 2024-09-13T13:22:13.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660362+0000", 2024-09-13T13:22:13.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5999e-05 2024-09-13T13:22:13.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660382+0000", 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0035999999999999e-05 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.673210+0000", 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012828381999999999 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.673262+0000", 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1211000000000001e-05 2024-09-13T13:22:13.672 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:133029 10.9 e366/334 10:97ab46ca:::benchmark_data_smithi052_153703_object5649:head v 366'8225, mlcod=366'8223)", 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:15:39.309291+0000", 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 394.35122577200002, 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.356083341, 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309291+0000", 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309291+0000", 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309293+0000", 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9889999999999999e-06 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.309303+0000", 2024-09-13T13:22:13.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0162e-05 2024-09-13T13:22:13.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660206+0000", 2024-09-13T13:22:13.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.35090332600000002 2024-09-13T13:22:13.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660227+0000", 2024-09-13T13:22:13.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0778999999999999e-05 2024-09-13T13:22:13.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660332+0000", 2024-09-13T13:22:13.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104756 2024-09-13T13:22:13.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.660360+0000", 2024-09-13T13:22:13.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8252999999999999e-05 2024-09-13T13:22:13.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665339+0000", 2024-09-13T13:22:13.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049790090000000004 2024-09-13T13:22:13.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:15:39.665374+0000", 2024-09-13T13:22:13.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5067000000000002e-05 2024-09-13T13:22:13.677 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.677 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.677 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.678 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:191396 10.9 e382/377 10:9fe2a3d5:::benchmark_data_smithi052_153703_object64016:head v 382'11888, mlcod=382'11886)", 2024-09-13T13:22:13.678 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:34.022668+0000", 2024-09-13T13:22:13.678 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 339.63784875300001, 2024-09-13T13:22:13.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.227577587, 2024-09-13T13:22:13.678 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.678 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.678 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.678 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022668+0000", 2024-09-13T13:22:13.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022668+0000", 2024-09-13T13:22:13.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022669+0000", 2024-09-13T13:22:13.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.519e-06 2024-09-13T13:22:13.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022677+0000", 2024-09-13T13:22:13.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3889999999999999e-06 2024-09-13T13:22:13.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022687+0000", 2024-09-13T13:22:13.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0190999999999999e-05 2024-09-13T13:22:13.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022693+0000", 2024-09-13T13:22:13.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5080000000000002e-06 2024-09-13T13:22:13.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022747+0000", 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3820000000000003e-05 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.022769+0000", 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1829000000000001e-05 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.026913+0000", 2024-09-13T13:22:13.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041442939999999998 2024-09-13T13:22:13.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250245+0000", 2024-09-13T13:22:13.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.22333203700000001 2024-09-13T13:22:13.682 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.682 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.682 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.682 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:191388 10.c e382/334 10:3518f5d7:::benchmark_data_smithi052_153703_object64008:head v 382'11871, mlcod=382'11868)", 2024-09-13T13:22:13.682 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:34.028033+0000", 2024-09-13T13:22:13.682 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 339.63248328999998, 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.237613568, 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028033+0000", 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028033+0000", 2024-09-13T13:22:13.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028035+0000", 2024-09-13T13:22:13.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8649999999999999e-06 2024-09-13T13:22:13.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.028044+0000", 2024-09-13T13:22:13.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7630000000000006e-06 2024-09-13T13:22:13.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250215+0000", 2024-09-13T13:22:13.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.222171601 2024-09-13T13:22:13.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250226+0000", 2024-09-13T13:22:13.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0395e-05 2024-09-13T13:22:13.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250281+0000", 2024-09-13T13:22:13.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4833999999999997e-05 2024-09-13T13:22:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250301+0000", 2024-09-13T13:22:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0452999999999999e-05 2024-09-13T13:22:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.265605+0000", 2024-09-13T13:22:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01530372 2024-09-13T13:22:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.265647+0000", 2024-09-13T13:22:13.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1937000000000002e-05 2024-09-13T13:22:13.687 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.687 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.687 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.687 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:191418 10.9 e382/377 10:975b1348:::benchmark_data_smithi052_153703_object64038:head v 382'11889, mlcod=382'11886)", 2024-09-13T13:22:13.687 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:34.041411+0000", 2024-09-13T13:22:13.687 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 339.61910603500002, 2024-09-13T13:22:13.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.214851654, 2024-09-13T13:22:13.687 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.687 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.688 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.688 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041411+0000", 2024-09-13T13:22:13.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041411+0000", 2024-09-13T13:22:13.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041413+0000", 2024-09-13T13:22:13.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3819999999999998e-06 2024-09-13T13:22:13.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.041424+0000", 2024-09-13T13:22:13.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1031e-05 2024-09-13T13:22:13.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250212+0000", 2024-09-13T13:22:13.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.208787785 2024-09-13T13:22:13.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250227+0000", 2024-09-13T13:22:13.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4887e-05 2024-09-13T13:22:13.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250280+0000", 2024-09-13T13:22:13.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3733999999999997e-05 2024-09-13T13:22:13.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.250340+0000", 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9212000000000002e-05 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.256042+0000", 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057022210000000004 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:34.256262+0000", 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022040199999999999 2024-09-13T13:22:13.691 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203978 10.9 e385/377 10:9bb10788:::benchmark_data_smithi052_153703_object76598:head v 385'12669, mlcod=385'12664)", 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.815708+0000", 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 327.84480819300001, 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27159179500000002, 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815708+0000", 2024-09-13T13:22:13.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815708+0000", 2024-09-13T13:22:13.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815710+0000", 2024-09-13T13:22:13.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8840000000000001e-06 2024-09-13T13:22:13.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815718+0000", 2024-09-13T13:22:13.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:22:13.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815731+0000", 2024-09-13T13:22:13.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2853e-05 2024-09-13T13:22:13.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.815739+0000", 2024-09-13T13:22:13.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6550000000000004e-06 2024-09-13T13:22:13.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816390+0000", 2024-09-13T13:22:13.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00065173800000000001 2024-09-13T13:22:13.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816411+0000", 2024-09-13T13:22:13.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0727e-05 2024-09-13T13:22:13.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087219+0000", 2024-09-13T13:22:13.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27080745899999997 2024-09-13T13:22:13.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087300+0000", 2024-09-13T13:22:13.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1477999999999995e-05 2024-09-13T13:22:13.697 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203975 10.1 e385/334 10:87de9ad5:::benchmark_data_smithi052_153703_object76595:head v 385'12786, mlcod=385'12784)", 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 327.84373478600003, 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27055213299999997, 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:22:13.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:22:13.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816782+0000", 2024-09-13T13:22:13.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4199999999999995e-07 2024-09-13T13:22:13.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816796+0000", 2024-09-13T13:22:13.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3298e-05 2024-09-13T13:22:13.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816803+0000", 2024-09-13T13:22:13.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8379999999999996e-06 2024-09-13T13:22:13.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816808+0000", 2024-09-13T13:22:13.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2020000000000003e-06 2024-09-13T13:22:13.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079290+0000", 2024-09-13T13:22:13.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.26248217499999998 2024-09-13T13:22:13.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079309+0000", 2024-09-13T13:22:13.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8927e-05 2024-09-13T13:22:13.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087304+0000", 2024-09-13T13:22:13.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079953999999999997 2024-09-13T13:22:13.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087334+0000", 2024-09-13T13:22:13.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9550999999999999e-05 2024-09-13T13:22:13.702 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.702 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.703 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.703 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203977 10.c e385/334 10:35b89a8a:::benchmark_data_smithi052_153703_object76597:head v 385'12628, mlcod=385'12626)", 2024-09-13T13:22:13.703 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.816846+0000", 2024-09-13T13:22:13.703 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 327.84367051999999, 2024-09-13T13:22:13.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27022401899999998, 2024-09-13T13:22:13.703 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.703 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.703 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.703 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816846+0000", 2024-09-13T13:22:13.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816846+0000", 2024-09-13T13:22:13.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816847+0000", 2024-09-13T13:22:13.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9200000000000001e-07 2024-09-13T13:22:13.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816851+0000", 2024-09-13T13:22:13.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1989999999999999e-06 2024-09-13T13:22:13.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816857+0000", 2024-09-13T13:22:13.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9610000000000002e-06 2024-09-13T13:22:13.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816862+0000", 2024-09-13T13:22:13.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8489999999999997e-06 2024-09-13T13:22:13.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816923+0000", 2024-09-13T13:22:13.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1781999999999999e-05 2024-09-13T13:22:13.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.816940+0000", 2024-09-13T13:22:13.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6844999999999999e-05 2024-09-13T13:22:13.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087036+0000", 2024-09-13T13:22:13.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27009603100000001 2024-09-13T13:22:13.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087070+0000", 2024-09-13T13:22:13.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3760000000000002e-05 2024-09-13T13:22:13.708 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.708 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.708 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.708 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203979 10.c e385/334 10:3e03f71a:::benchmark_data_smithi052_153703_object76599:head v 385'12629, mlcod=385'12626)", 2024-09-13T13:22:13.708 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.820677+0000", 2024-09-13T13:22:13.708 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 327.83983952199998, 2024-09-13T13:22:13.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.26856709400000001, 2024-09-13T13:22:13.708 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.708 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.708 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820677+0000", 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820677+0000", 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820678+0000", 2024-09-13T13:22:13.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4699999999999999e-06 2024-09-13T13:22:13.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.820696+0000", 2024-09-13T13:22:13.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7331999999999999e-05 2024-09-13T13:22:13.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079076+0000", 2024-09-13T13:22:13.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.25837983199999998 2024-09-13T13:22:13.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079091+0000", 2024-09-13T13:22:13.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5832999999999999e-05 2024-09-13T13:22:13.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.081590+0000", 2024-09-13T13:22:13.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002498545 2024-09-13T13:22:13.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.081623+0000", 2024-09-13T13:22:13.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3442000000000003e-05 2024-09-13T13:22:13.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.089192+0000", 2024-09-13T13:22:13.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075682340000000001 2024-09-13T13:22:13.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.089244+0000", 2024-09-13T13:22:13.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2405999999999999e-05 2024-09-13T13:22:13.712 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.713 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.713 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.713 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.26071.0:203982 10.9 e385/377 10:90b0e29e:::benchmark_data_smithi052_153703_object76602:head v 385'12670, mlcod=385'12666)", 2024-09-13T13:22:13.713 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:16:45.825124+0000", 2024-09-13T13:22:13.713 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 327.83539245200001, 2024-09-13T13:22:13.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.26226309199999998, 2024-09-13T13:22:13.713 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.713 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.713 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.713 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825124+0000", 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825124+0000", 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825126+0000", 2024-09-13T13:22:13.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.894e-06 2024-09-13T13:22:13.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:45.825134+0000", 2024-09-13T13:22:13.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1960000000000008e-06 2024-09-13T13:22:13.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079730+0000", 2024-09-13T13:22:13.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.25459629499999997 2024-09-13T13:22:13.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079741+0000", 2024-09-13T13:22:13.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-09-13T13:22:13.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079895+0000", 2024-09-13T13:22:13.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154881 2024-09-13T13:22:13.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.079919+0000", 2024-09-13T13:22:13.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3139e-05 2024-09-13T13:22:13.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087337+0000", 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.007417964 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:16:46.087387+0000", 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0612999999999998e-05 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:60365 11.0 e440/423 11:088590ea:::benchmark_data_smithi052_182575_object60364:head v 440'3937, mlcod=440'3934)", 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.475832+0000", 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 220.184684129, 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31868556799999997, 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475832+0000", 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475832+0000", 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475834+0000", 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.643e-06 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475869+0000", 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4582000000000002e-05 2024-09-13T13:22:13.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475879+0000", 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0244000000000001e-05 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475886+0000", 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8759999999999999e-06 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475941+0000", 2024-09-13T13:22:13.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4975999999999997e-05 2024-09-13T13:22:13.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.475960+0000", 2024-09-13T13:22:13.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9381999999999998e-05 2024-09-13T13:22:13.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480677+0000", 2024-09-13T13:22:13.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047167650000000004 2024-09-13T13:22:13.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794518+0000", 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31384109999999998 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:60369 11.0 e440/423 11:00a94002:::benchmark_data_smithi052_182575_object60368:head v 440'3938, mlcod=440'3935)", 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.478361+0000", 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 220.18215511299999, 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31619510000000001, 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.722 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478361+0000", 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478361+0000", 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478363+0000", 2024-09-13T13:22:13.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3540000000000001e-06 2024-09-13T13:22:13.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478408+0000", 2024-09-13T13:22:13.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5260999999999999e-05 2024-09-13T13:22:13.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478417+0000", 2024-09-13T13:22:13.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7690000000000002e-06 2024-09-13T13:22:13.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478423+0000", 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.331e-06 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478857+0000", 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043396400000000001 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.478875+0000", 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7611000000000001e-05 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794529+0000", 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31565414899999999 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794557+0000", 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7661000000000002e-05 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.726 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:60375 11.5 e440/423 11:ab976586:::benchmark_data_smithi052_182575_object60374:head v 440'3806, mlcod=440'3804)", 2024-09-13T13:22:13.727 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.480358+0000", 2024-09-13T13:22:13.727 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 220.18015816799999, 2024-09-13T13:22:13.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31425154100000002, 2024-09-13T13:22:13.727 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.727 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.727 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.727 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480358+0000", 2024-09-13T13:22:13.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480358+0000", 2024-09-13T13:22:13.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480360+0000", 2024-09-13T13:22:13.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.139e-06 2024-09-13T13:22:13.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480381+0000", 2024-09-13T13:22:13.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1444e-05 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480390+0000", 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480396+0000", 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2389999999999999e-06 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480450+0000", 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3616000000000003e-05 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480467+0000", 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7686e-05 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794579+0000", 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31411161500000001 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794610+0000", 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1027e-05 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.36635.0:60372 11.0 e440/423 11:055cf520:::benchmark_data_smithi052_182575_object60371:head v 440'3939, mlcod=440'3935)", 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.480901+0000", 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 220.17961506200001, 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31806362700000002, 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.731 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480901+0000", 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480901+0000", 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480903+0000", 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1969999999999999e-06 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.480964+0000", 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1058999999999999e-05 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794586+0000", 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31362191099999998 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794593+0000", 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7139999999999993e-06 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794668+0000", 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4833000000000007e-05 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794690+0000", 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2263e-05 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.798931+0000", 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042401610000000001 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.798965+0000", 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4489000000000001e-05 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.735 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "replica_scrub(pg: 4.3,from:0'0,to:268'29,epoch:440/61,start:4:c0000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:18:33.481428+0000", 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 220.179088514, 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.313186768, 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.481428+0000", 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.481428+0000", 2024-09-13T13:22:13.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.481429+0000", 2024-09-13T13:22:13.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.178e-06 2024-09-13T13:22:13.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.481435+0000", 2024-09-13T13:22:13.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6860000000000003e-06 2024-09-13T13:22:13.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794453+0000", 2024-09-13T13:22:13.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31301767800000002 2024-09-13T13:22:13.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794474+0000", 2024-09-13T13:22:13.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1761999999999999e-05 2024-09-13T13:22:13.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794570+0000", 2024-09-13T13:22:13.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6155000000000003e-05 2024-09-13T13:22:13.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:13.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794584+0000", 2024-09-13T13:22:13.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3613e-05 2024-09-13T13:22:13.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:18:33.794615+0000", 2024-09-13T13:22:13.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0695999999999999e-05 2024-09-13T13:22:13.739 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.739 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95600 11.17 11.4e8f0b17 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:06.191050+0000", 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.46946692099999, 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99082065699999999, 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:13.740 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95600 2024-09-13T13:22:13.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.741 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191050+0000", 2024-09-13T13:22:13.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191050+0000", 2024-09-13T13:22:13.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191056+0000", 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6000000000000003e-06 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191161+0000", 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104625 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191171+0000", 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0596e-05 2024-09-13T13:22:13.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191179+0000", 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5889999999999997e-06 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191231+0000", 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2178000000000001e-05 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.191239+0000", 2024-09-13T13:22:13.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8350000000000004e-06 2024-09-13T13:22:13.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.181852+0000", 2024-09-13T13:22:13.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99061336 2024-09-13T13:22:13.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.181870+0000", 2024-09-13T13:22:13.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873999999999999e-05 2024-09-13T13:22:13.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.744 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95607 11.17 11.5b4199b7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:06.194159+0000", 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.46635793999999, 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.988161768, 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95607 2024-09-13T13:22:13.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194159+0000", 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194159+0000", 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194160+0000", 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.736e-06 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194226+0000", 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6062999999999994e-05 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194236+0000", 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194242+0000", 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7969999999999997e-06 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194290+0000", 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7939999999999998e-05 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.194299+0000", 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.833e-06 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182312+0000", 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98801272299999998 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182320+0000", 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6969999999999999e-06 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.749 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95614 11.17 11.ed911577 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:06.201993+0000", 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.45852334099999, 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98036120999999998, 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95614 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.201993+0000", 2024-09-13T13:22:13.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.201993+0000", 2024-09-13T13:22:13.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.201995+0000", 2024-09-13T13:22:13.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6360000000000001e-06 2024-09-13T13:22:13.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.202054+0000", 2024-09-13T13:22:13.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8721999999999998e-05 2024-09-13T13:22:13.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.202064+0000", 2024-09-13T13:22:13.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9979999999999998e-06 2024-09-13T13:22:13.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.202069+0000", 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1819999999999997e-06 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.202119+0000", 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9948999999999999e-05 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.202127+0000", 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.028e-06 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182345+0000", 2024-09-13T13:22:13.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98021872399999999 2024-09-13T13:22:13.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182354+0000", 2024-09-13T13:22:13.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9709999999999993e-06 2024-09-13T13:22:13.754 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.754 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.754 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.754 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95625 11.17 11.fe23b37 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:06.211655+0000", 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.448861604, 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97072697900000005, 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95625 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211655+0000", 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211655+0000", 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211657+0000", 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0470000000000001e-06 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:13.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211700+0000", 2024-09-13T13:22:13.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3359999999999998e-05 2024-09-13T13:22:13.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:13.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211713+0000", 2024-09-13T13:22:13.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2322e-05 2024-09-13T13:22:13.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:13.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211720+0000", 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.43e-06 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211780+0000", 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9879e-05 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.211803+0000", 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3051999999999999e-05 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:13.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182375+0000", 2024-09-13T13:22:13.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97057152000000002 2024-09-13T13:22:13.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:13.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:13.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:13.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.182382+0000", 2024-09-13T13:22:13.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3690000000000001e-06 2024-09-13T13:22:13.759 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.759 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.759 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.759 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:13.759 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:13.759 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:22:14.442 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:14 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:22:14.443 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:14 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:22:14.443 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:14 smithi167 ceph-mon[33178]: from='osd.4 v1:172.21.15.167:6800/545860898' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-13T13:22:14.443 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:14 smithi167 ceph-mon[33178]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-13T13:22:14.443 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:22:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:22:14.251+0000 7f4768c25640 -1 osd.4 522 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:22:14.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T13:22:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:14 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:22:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:14 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:22:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:14 smithi052 ceph-mon[36682]: from='osd.4 v1:172.21.15.167:6800/545860898' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-13T13:22:14.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:14 smithi052 ceph-mon[36682]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-13T13:22:14.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:14 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:22:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:14 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:22:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:14 smithi052 ceph-mon[29471]: from='osd.4 v1:172.21.15.167:6800/545860898' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-13T13:22:14.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:14 smithi052 ceph-mon[29471]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-13T13:22:14.718 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:14.837 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:22:15.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:22:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:22:15.247 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:22:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:22:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-09-13T13:22:15.247 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:22:15.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:22:15.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:22:15.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[29471]: 2.1 scrub starts 2024-09-13T13:22:15.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[29471]: 2.1 scrub ok 2024-09-13T13:22:15.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[29471]: pgmap v1073: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 936 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:15.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[29471]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T13:22:15.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[29471]: from='osd.4 v1:172.21.15.167:6800/545860898' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:22:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[29471]: osdmap e522: 8 total, 7 up, 7 in 2024-09-13T13:22:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[29471]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:22:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[29471]: 6.3 scrub starts 2024-09-13T13:22:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[29471]: 6.3 scrub ok 2024-09-13T13:22:15.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:15.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:22:15.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:22:15.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[36682]: 2.1 scrub starts 2024-09-13T13:22:15.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[36682]: 2.1 scrub ok 2024-09-13T13:22:15.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[36682]: pgmap v1073: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 936 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:15.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[36682]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T13:22:15.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[36682]: from='osd.4 v1:172.21.15.167:6800/545860898' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:22:15.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[36682]: osdmap e522: 8 total, 7 up, 7 in 2024-09-13T13:22:15.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[36682]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:22:15.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[36682]: 6.3 scrub starts 2024-09-13T13:22:15.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[36682]: 6.3 scrub ok 2024-09-13T13:22:15.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:15 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:22:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:15 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:22:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:15 smithi167 ceph-mon[33178]: 2.1 scrub starts 2024-09-13T13:22:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:15 smithi167 ceph-mon[33178]: 2.1 scrub ok 2024-09-13T13:22:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:15 smithi167 ceph-mon[33178]: pgmap v1073: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 936 B/s rd, 0 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:15 smithi167 ceph-mon[33178]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T13:22:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:15 smithi167 ceph-mon[33178]: from='osd.4 v1:172.21.15.167:6800/545860898' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:22:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:15 smithi167 ceph-mon[33178]: osdmap e522: 8 total, 7 up, 7 in 2024-09-13T13:22:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:15 smithi167 ceph-mon[33178]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]} : dispatch 2024-09-13T13:22:15.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:15 smithi167 ceph-mon[33178]: 6.3 scrub starts 2024-09-13T13:22:15.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:15 smithi167 ceph-mon[33178]: 6.3 scrub ok 2024-09-13T13:22:15.572 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:15.953 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T13:22:16.032 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:16.489 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:16 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:22:16.489 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:16 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:22:16.489 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:16 smithi167 ceph-mon[33178]: 5.4 scrub starts 2024-09-13T13:22:16.489 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:16 smithi167 ceph-mon[33178]: 5.4 scrub ok 2024-09-13T13:22:16.489 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:16 smithi167 ceph-mon[33178]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T13:22:16.489 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:16 smithi167 ceph-mon[33178]: osd.4 v1:172.21.15.167:6800/545860898 boot 2024-09-13T13:22:16.489 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:16 smithi167 ceph-mon[33178]: osdmap e523: 8 total, 8 up, 7 in 2024-09-13T13:22:16.489 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:16 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T13:22:16.489 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:16 smithi167 ceph-mon[33178]: osdmap e524: 8 total, 8 up, 7 in 2024-09-13T13:22:16.490 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:22:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:22:16.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:22:16.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:22:16.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[36682]: 5.4 scrub starts 2024-09-13T13:22:16.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[36682]: 5.4 scrub ok 2024-09-13T13:22:16.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[36682]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T13:22:16.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[36682]: osd.4 v1:172.21.15.167:6800/545860898 boot 2024-09-13T13:22:16.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[36682]: osdmap e523: 8 total, 8 up, 7 in 2024-09-13T13:22:16.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T13:22:16.491 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[36682]: osdmap e524: 8 total, 8 up, 7 in 2024-09-13T13:22:16.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:22:16.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:22:16.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[29471]: 5.4 scrub starts 2024-09-13T13:22:16.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[29471]: 5.4 scrub ok 2024-09-13T13:22:16.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[29471]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T13:22:16.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[29471]: osd.4 v1:172.21.15.167:6800/545860898 boot 2024-09-13T13:22:16.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[29471]: osdmap e523: 8 total, 8 up, 7 in 2024-09-13T13:22:16.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-13T13:22:16.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:16 smithi052 ceph-mon[29471]: osdmap e524: 8 total, 8 up, 7 in 2024-09-13T13:22:16.499 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:22:16.499 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:22:16.499 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-09-13T13:22:16.499 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:22:17.148 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-13T13:22:17.148 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-13T13:22:17.148 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd in 7 2024-09-13T13:22:17.153 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:17.420 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:17 smithi167 ceph-mon[33178]: 4.16 scrub starts 2024-09-13T13:22:17.420 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:17 smithi167 ceph-mon[33178]: 4.16 scrub ok 2024-09-13T13:22:17.420 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:17 smithi167 ceph-mon[33178]: 6.4 scrub starts 2024-09-13T13:22:17.420 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:17 smithi167 ceph-mon[33178]: 6.4 scrub ok 2024-09-13T13:22:17.420 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:17 smithi167 ceph-mon[33178]: pgmap v1076: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:17.421 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:17 smithi167 ceph-mon[33178]: 3.15 scrub starts 2024-09-13T13:22:17.421 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:17 smithi167 ceph-mon[33178]: 3.15 scrub ok 2024-09-13T13:22:17.421 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:17 smithi167 ceph-mon[33178]: 6.16 scrub starts 2024-09-13T13:22:17.421 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:17 smithi167 ceph-mon[33178]: 6.16 scrub ok 2024-09-13T13:22:17.421 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:17 smithi167 ceph-mon[33178]: 1.5 scrub starts 2024-09-13T13:22:17.421 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:17 smithi167 ceph-mon[33178]: 1.5 scrub ok 2024-09-13T13:22:17.426 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[36682]: 4.16 scrub starts 2024-09-13T13:22:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[36682]: 4.16 scrub ok 2024-09-13T13:22:17.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[36682]: 6.4 scrub starts 2024-09-13T13:22:17.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[36682]: 6.4 scrub ok 2024-09-13T13:22:17.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[36682]: pgmap v1076: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:17.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[36682]: 3.15 scrub starts 2024-09-13T13:22:17.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[36682]: 3.15 scrub ok 2024-09-13T13:22:17.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[36682]: 6.16 scrub starts 2024-09-13T13:22:17.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[36682]: 6.16 scrub ok 2024-09-13T13:22:17.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[36682]: 1.5 scrub starts 2024-09-13T13:22:17.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[36682]: 1.5 scrub ok 2024-09-13T13:22:17.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[29471]: 4.16 scrub starts 2024-09-13T13:22:17.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[29471]: 4.16 scrub ok 2024-09-13T13:22:17.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[29471]: 6.4 scrub starts 2024-09-13T13:22:17.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[29471]: 6.4 scrub ok 2024-09-13T13:22:17.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[29471]: pgmap v1076: 142 pgs: 42 active+undersized, 24 active+undersized+degraded, 76 active+clean; 635 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 91/597 objects degraded (15.243%) 2024-09-13T13:22:17.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[29471]: 3.15 scrub starts 2024-09-13T13:22:17.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[29471]: 3.15 scrub ok 2024-09-13T13:22:17.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[29471]: 6.16 scrub starts 2024-09-13T13:22:17.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[29471]: 6.16 scrub ok 2024-09-13T13:22:17.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[29471]: 1.5 scrub starts 2024-09-13T13:22:17.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:17 smithi052 ceph-mon[29471]: 1.5 scrub ok 2024-09-13T13:22:17.680 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:22:17.680 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:22:17.680 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:22:17.680 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:22:17.680 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:22:18.155 INFO:teuthology.orchestra.run.smithi052.stderr:marked in osd.7. 2024-09-13T13:22:18.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[36682]: 1.3 scrub starts 2024-09-13T13:22:18.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[36682]: 1.3 scrub ok 2024-09-13T13:22:18.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[36682]: 6.14 scrub starts 2024-09-13T13:22:18.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[36682]: 6.14 scrub ok 2024-09-13T13:22:18.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[36682]: 2.2 scrub starts 2024-09-13T13:22:18.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[36682]: 2.2 scrub ok 2024-09-13T13:22:18.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1469630414' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T13:22:18.489 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1469630414' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-13T13:22:18.490 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[36682]: osdmap e525: 8 total, 8 up, 8 in 2024-09-13T13:22:18.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a[29467]: 2024-09-13T13:22:18.122+0000 7effa8af8640 -1 mon.a@0(leader).osd e524 definitely_dead 0 2024-09-13T13:22:18.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[29471]: 1.3 scrub starts 2024-09-13T13:22:18.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[29471]: 1.3 scrub ok 2024-09-13T13:22:18.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[29471]: 6.14 scrub starts 2024-09-13T13:22:18.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[29471]: 6.14 scrub ok 2024-09-13T13:22:18.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[29471]: 2.2 scrub starts 2024-09-13T13:22:18.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[29471]: 2.2 scrub ok 2024-09-13T13:22:18.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1469630414' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T13:22:18.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1469630414' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-13T13:22:18.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:18 smithi052 ceph-mon[29471]: osdmap e525: 8 total, 8 up, 8 in 2024-09-13T13:22:18.502 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:18 smithi167 ceph-mon[33178]: 1.3 scrub starts 2024-09-13T13:22:18.502 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:18 smithi167 ceph-mon[33178]: 1.3 scrub ok 2024-09-13T13:22:18.502 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:18 smithi167 ceph-mon[33178]: 6.14 scrub starts 2024-09-13T13:22:18.502 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:18 smithi167 ceph-mon[33178]: 6.14 scrub ok 2024-09-13T13:22:18.502 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:18 smithi167 ceph-mon[33178]: 2.2 scrub starts 2024-09-13T13:22:18.502 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:18 smithi167 ceph-mon[33178]: 2.2 scrub ok 2024-09-13T13:22:18.502 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:18 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1469630414' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T13:22:18.503 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:18 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1469630414' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-13T13:22:18.503 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:18 smithi167 ceph-mon[33178]: osdmap e525: 8 total, 8 up, 8 in 2024-09-13T13:22:18.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T13:22:18.820 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:18.966 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-13T13:22:18.966 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-13T13:22:18.966 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-13T13:22:19.261 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:19.344 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:22:19.344 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:22:19.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:22:19.345 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:22:19.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.345 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242614 10.f 10:f901b0d3:::benchmark_data_smithi052_153703_object115234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T13:22:19.345 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:29.053950+0000", 2024-09-13T13:22:19.345 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 290.28798743300001, 2024-09-13T13:22:19.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0472787020000001, 2024-09-13T13:22:19.345 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.345 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.345 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:19.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:19.345 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242614 2024-09-13T13:22:19.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.346 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.053950+0000", 2024-09-13T13:22:19.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.053950+0000", 2024-09-13T13:22:19.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.053951+0000", 2024-09-13T13:22:19.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3769999999999999e-06 2024-09-13T13:22:19.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.053956+0000", 2024-09-13T13:22:19.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5390000000000001e-06 2024-09-13T13:22:19.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.053968+0000", 2024-09-13T13:22:19.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943000000000001e-05 2024-09-13T13:22:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.053975+0000", 2024-09-13T13:22:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5059999999999998e-06 2024-09-13T13:22:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.054035+0000", 2024-09-13T13:22:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9787999999999997e-05 2024-09-13T13:22:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.054045+0000", 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0015e-05 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.088035+0000", 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0339899770000001 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.088163+0000", 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012799400000000001 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3", 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.088262+0000", 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9026999999999999e-05 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.094917+0000", 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066546410000000002 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.094956+0000", 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9546999999999999e-05 2024-09-13T13:22:19.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.101141+0000", 2024-09-13T13:22:19.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061843849999999997 2024-09-13T13:22:19.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.101186+0000", 2024-09-13T13:22:19.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5116e-05 2024-09-13T13:22:19.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.101229+0000", 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2846999999999998e-05 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242654 10.f 10:f22ae86f:::benchmark_data_smithi052_153703_object115274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:29.104470+0000", 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 290.23746713100002, 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0039240330000001, 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.352 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.353 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:19.353 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:19.353 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242654 2024-09-13T13:22:19.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.353 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.104470+0000", 2024-09-13T13:22:19.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.104470+0000", 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.104472+0000", 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7209999999999999e-06 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.104478+0000", 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8760000000000002e-06 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.104494+0000", 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5752000000000001e-05 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.104502+0000", 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2309999999999996e-06 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.104608+0000", 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010657100000000001 2024-09-13T13:22:19.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.104616+0000", 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9189999999999999e-06 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.094980+0000", 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99036415300000002 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.095074+0000", 2024-09-13T13:22:19.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4021e-05 2024-09-13T13:22:19.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3", 2024-09-13T13:22:19.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.095177+0000", 2024-09-13T13:22:19.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102224 2024-09-13T13:22:19.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.103679+0000", 2024-09-13T13:22:19.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085021379999999994 2024-09-13T13:22:19.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.103725+0000", 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6350000000000002e-05 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.108305+0000", 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045795899999999997 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.108351+0000", 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6786000000000002e-05 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.108394+0000", 2024-09-13T13:22:19.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2701000000000003e-05 2024-09-13T13:22:19.359 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.359 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.359 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.359 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242660 10.f 10:fbeb4f62:::benchmark_data_smithi052_153703_object115280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T13:22:19.359 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:29.107779+0000", 2024-09-13T13:22:19.359 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 290.23415785499998, 2024-09-13T13:22:19.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.000710473, 2024-09-13T13:22:19.359 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242660 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.107779+0000", 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.107779+0000", 2024-09-13T13:22:19.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.107781+0000", 2024-09-13T13:22:19.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4720000000000001e-06 2024-09-13T13:22:19.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.107793+0000", 2024-09-13T13:22:19.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1789e-05 2024-09-13T13:22:19.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.107807+0000", 2024-09-13T13:22:19.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4771e-05 2024-09-13T13:22:19.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.107814+0000", 2024-09-13T13:22:19.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8059999999999997e-06 2024-09-13T13:22:19.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.107884+0000", 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9680999999999999e-05 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.107891+0000", 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.503e-06 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.098159+0000", 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99026766300000002 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.098288+0000", 2024-09-13T13:22:19.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000128801 2024-09-13T13:22:19.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3", 2024-09-13T13:22:19.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.098415+0000", 2024-09-13T13:22:19.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012738800000000001 2024-09-13T13:22:19.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.103765+0000", 2024-09-13T13:22:19.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0053501460000000001 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.103817+0000", 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1833000000000003e-05 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.108405+0000", 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045872609999999996 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.108445+0000", 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0812999999999997e-05 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.108490+0000", 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4545999999999997e-05 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242671 10.f 10:f8c32854:::benchmark_data_smithi052_153703_object115291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:29.119843+0000", 2024-09-13T13:22:19.366 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 290.22209412000001, 2024-09-13T13:22:19.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99181222899999999, 2024-09-13T13:22:19.367 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.367 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.367 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:19.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:19.367 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242671 2024-09-13T13:22:19.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.367 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.119843+0000", 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.119843+0000", 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.119845+0000", 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.361e-06 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.119849+0000", 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5789999999999997e-06 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.119859+0000", 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7370000000000001e-06 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.119866+0000", 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6630000000000004e-06 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.119918+0000", 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2098999999999997e-05 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.119925+0000", 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7009999999999996e-06 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.101252+0000", 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98132635999999995 2024-09-13T13:22:19.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.101326+0000", 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3829000000000001e-05 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3", 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.101393+0000", 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7914000000000001e-05 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.108543+0000", 2024-09-13T13:22:19.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0071496880000000004 2024-09-13T13:22:19.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.108599+0000", 2024-09-13T13:22:19.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6138999999999999e-05 2024-09-13T13:22:19.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.111530+0000", 2024-09-13T13:22:19.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002930729 2024-09-13T13:22:19.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.111607+0000", 2024-09-13T13:22:19.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6911999999999998e-05 2024-09-13T13:22:19.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.111655+0000", 2024-09-13T13:22:19.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8517999999999999e-05 2024-09-13T13:22:19.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.374 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.375 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.375 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242707 10.f 10:f7b988b9:::benchmark_data_smithi052_153703_object115327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T13:22:19.375 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:29.233438+0000", 2024-09-13T13:22:19.375 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 290.10849898700002, 2024-09-13T13:22:19.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.878368861, 2024-09-13T13:22:19.375 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.375 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.375 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:19.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:19.376 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242707 2024-09-13T13:22:19.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.376 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.233438+0000", 2024-09-13T13:22:19.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.233438+0000", 2024-09-13T13:22:19.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.233440+0000", 2024-09-13T13:22:19.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.793e-06 2024-09-13T13:22:19.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.233445+0000", 2024-09-13T13:22:19.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2850000000000002e-06 2024-09-13T13:22:19.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.233458+0000", 2024-09-13T13:22:19.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2944e-05 2024-09-13T13:22:19.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.233469+0000", 2024-09-13T13:22:19.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0983e-05 2024-09-13T13:22:19.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.233546+0000", 2024-09-13T13:22:19.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7083999999999996e-05 2024-09-13T13:22:19.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.233554+0000", 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0970000000000006e-06 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.103851+0000", 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87029699500000002 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.103947+0000", 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5241999999999994e-05 2024-09-13T13:22:19.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3", 2024-09-13T13:22:19.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.104039+0000", 2024-09-13T13:22:19.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1953000000000005e-05 2024-09-13T13:22:19.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.108590+0000", 2024-09-13T13:22:19.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045514300000000004 2024-09-13T13:22:19.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.108603+0000", 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.256e-05 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.111708+0000", 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031054569999999998 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.111748+0000", 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9993000000000001e-05 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.111807+0000", 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9045000000000001e-05 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242743 10.f 10:ffadbd3f:::benchmark_data_smithi052_153703_object115363:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:29.274376+0000", 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 290.067561684, 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.83755999599999997, 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.382 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242743 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.274376+0000", 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.274376+0000", 2024-09-13T13:22:19.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.274377+0000", 2024-09-13T13:22:19.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.455e-06 2024-09-13T13:22:19.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.274382+0000", 2024-09-13T13:22:19.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.51e-06 2024-09-13T13:22:19.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.274394+0000", 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2051e-05 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.274399+0000", 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4589999999999997e-06 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.278334+0000", 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039345559999999996 2024-09-13T13:22:19.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.278343+0000", 2024-09-13T13:22:19.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1079999999999999e-06 2024-09-13T13:22:19.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.106257+0000", 2024-09-13T13:22:19.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.82791389199999998 2024-09-13T13:22:19.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.106349+0000", 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2138000000000005e-05 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3", 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.106413+0000", 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3903000000000001e-05 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.111452+0000", 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050395259999999999 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.111476+0000", 2024-09-13T13:22:19.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3784000000000001e-05 2024-09-13T13:22:19.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.111851+0000", 2024-09-13T13:22:19.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037528799999999999 2024-09-13T13:22:19.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.111907+0000", 2024-09-13T13:22:19.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5652000000000002e-05 2024-09-13T13:22:19.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.111936+0000", 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8674e-05 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242749 10.f 10:f3ab5042:::benchmark_data_smithi052_153703_object115369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:29.280632+0000", 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 290.06130537000001, 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.83147585899999998, 2024-09-13T13:22:19.389 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.390 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.390 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.390 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.390 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:19.390 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:19.390 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242749 2024-09-13T13:22:19.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.390 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.280632+0000", 2024-09-13T13:22:19.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.280632+0000", 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.280633+0000", 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4640000000000001e-06 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.280639+0000", 2024-09-13T13:22:19.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.113e-06 2024-09-13T13:22:19.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.280649+0000", 2024-09-13T13:22:19.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0861e-05 2024-09-13T13:22:19.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.280656+0000", 2024-09-13T13:22:19.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9990000000000002e-06 2024-09-13T13:22:19.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.280712+0000", 2024-09-13T13:22:19.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5841999999999998e-05 2024-09-13T13:22:19.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.280721+0000", 2024-09-13T13:22:19.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6889999999999993e-06 2024-09-13T13:22:19.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.108634+0000", 2024-09-13T13:22:19.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.82791335499999996 2024-09-13T13:22:19.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.108719+0000", 2024-09-13T13:22:19.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4289999999999994e-05 2024-09-13T13:22:19.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3", 2024-09-13T13:22:19.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.108781+0000", 2024-09-13T13:22:19.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2860999999999993e-05 2024-09-13T13:22:19.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.111947+0000", 2024-09-13T13:22:19.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031651230000000002 2024-09-13T13:22:19.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.111996+0000", 2024-09-13T13:22:19.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9030000000000003e-05 2024-09-13T13:22:19.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.112035+0000", 2024-09-13T13:22:19.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9412999999999997e-05 2024-09-13T13:22:19.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.112066+0000", 2024-09-13T13:22:19.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0670000000000003e-05 2024-09-13T13:22:19.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.112108+0000", 2024-09-13T13:22:19.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2148999999999999e-05 2024-09-13T13:22:19.396 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.396 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.396 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.397 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.26071.0:242752 10.f 10:fe622cc0:::benchmark_data_smithi052_153703_object115372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T13:22:19.397 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:17:29.284042+0000", 2024-09-13T13:22:19.397 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 290.05789562299998, 2024-09-13T13:22:19.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.83707015299999998, 2024-09-13T13:22:19.397 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.397 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.397 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.397 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.397 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.26071", 2024-09-13T13:22:19.397 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/949264571", 2024-09-13T13:22:19.397 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 242752 2024-09-13T13:22:19.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.284042+0000", 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.284042+0000", 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.284043+0000", 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3090000000000001e-06 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.284048+0000", 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0969999999999996e-06 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.284058+0000", 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0335e-05 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.284064+0000", 2024-09-13T13:22:19.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3290000000000001e-06 2024-09-13T13:22:19.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.284113+0000", 2024-09-13T13:22:19.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8999999999999998e-05 2024-09-13T13:22:19.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:29.284123+0000", 2024-09-13T13:22:19.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7890000000000006e-06 2024-09-13T13:22:19.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.112135+0000", 2024-09-13T13:22:19.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.82801250699999995 2024-09-13T13:22:19.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.112225+0000", 2024-09-13T13:22:19.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0067000000000004e-05 2024-09-13T13:22:19.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,3", 2024-09-13T13:22:19.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.112300+0000", 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5275e-05 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.120883+0000", 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085823070000000008 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.120922+0000", 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9159999999999998e-05 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.121025+0000", 2024-09-13T13:22:19.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103433 2024-09-13T13:22:19.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.121074+0000", 2024-09-13T13:22:19.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8943999999999998e-05 2024-09-13T13:22:19.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:17:30.121112+0000", 2024-09-13T13:22:19.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7601000000000001e-05 2024-09-13T13:22:19.403 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.404 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.404 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.404 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:94531 11.3 11:c5212371:::benchmark_data_smithi052_182575_object94530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:22:19.404 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:04.196846+0000", 2024-09-13T13:22:19.404 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 195.145091447, 2024-09-13T13:22:19.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99911598599999996, 2024-09-13T13:22:19.404 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.404 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.404 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:19.404 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.404 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:19.405 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:19.405 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94531 2024-09-13T13:22:19.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.405 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.196846+0000", 2024-09-13T13:22:19.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.196846+0000", 2024-09-13T13:22:19.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.196858+0000", 2024-09-13T13:22:19.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2455e-05 2024-09-13T13:22:19.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.196972+0000", 2024-09-13T13:22:19.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113578 2024-09-13T13:22:19.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.196991+0000", 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8842e-05 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197004+0000", 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230999999999999e-05 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.386715+0000", 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.18971091300000001 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.386724+0000", 2024-09-13T13:22:19.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8030000000000003e-06 2024-09-13T13:22:19.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.195881+0000", 2024-09-13T13:22:19.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.80915748099999996 2024-09-13T13:22:19.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.195962+0000", 2024-09-13T13:22:19.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0683e-05 2024-09-13T13:22:19.409 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.409 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.409 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.409 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:94537 11.3 11:c22e03e6:::benchmark_data_smithi052_182575_object94536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:22:19.409 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:04.197034+0000", 2024-09-13T13:22:19.409 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 195.14490296400001, 2024-09-13T13:22:19.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99942792400000002, 2024-09-13T13:22:19.409 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.409 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.409 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:19.409 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94537 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197034+0000", 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197034+0000", 2024-09-13T13:22:19.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197035+0000", 2024-09-13T13:22:19.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2399999999999997e-07 2024-09-13T13:22:19.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197082+0000", 2024-09-13T13:22:19.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6406000000000003e-05 2024-09-13T13:22:19.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197088+0000", 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7279999999999998e-06 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197094+0000", 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.386745+0000", 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.18965116100000001 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.386752+0000", 2024-09-13T13:22:19.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8600000000000004e-06 2024-09-13T13:22:19.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.196405+0000", 2024-09-13T13:22:19.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.80965324299999997 2024-09-13T13:22:19.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.196462+0000", 2024-09-13T13:22:19.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6827999999999999e-05 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:94539 11.1 11:8032e11e:::benchmark_data_smithi052_182575_object94538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:04.197102+0000", 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 195.14483560400001, 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0112109359999999, 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.414 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94539 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197102+0000", 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197102+0000", 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197102+0000", 2024-09-13T13:22:19.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0299999999999999e-07 2024-09-13T13:22:19.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197147+0000", 2024-09-13T13:22:19.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4872e-05 2024-09-13T13:22:19.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197153+0000", 2024-09-13T13:22:19.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2439999999999998e-06 2024-09-13T13:22:19.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197159+0000", 2024-09-13T13:22:19.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4890000000000003e-06 2024-09-13T13:22:19.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.389673+0000", 2024-09-13T13:22:19.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19251448199999999 2024-09-13T13:22:19.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.389690+0000", 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6923e-05 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.193378+0000", 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.80368818500000005 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.193580+0000", 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020200700000000001 2024-09-13T13:22:19.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-09-13T13:22:19.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.193956+0000", 2024-09-13T13:22:19.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037574 2024-09-13T13:22:19.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.196203+0000", 2024-09-13T13:22:19.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022467020000000002 2024-09-13T13:22:19.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.207781+0000", 2024-09-13T13:22:19.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011577728000000001 2024-09-13T13:22:19.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.208230+0000", 2024-09-13T13:22:19.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00044883199999999997 2024-09-13T13:22:19.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.208271+0000", 2024-09-13T13:22:19.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1517999999999998e-05 2024-09-13T13:22:19.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.208313+0000", 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1610999999999999e-05 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:94543 11.3 11:c3fe3cba:::benchmark_data_smithi052_182575_object94542:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:04.197167+0000", 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 195.144770674, 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0123877489999999, 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.421 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.422 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:19.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:19.422 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94543 2024-09-13T13:22:19.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.422 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197167+0000", 2024-09-13T13:22:19.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197167+0000", 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197167+0000", 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9100000000000003e-07 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197207+0000", 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0006000000000003e-05 2024-09-13T13:22:19.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197213+0000", 2024-09-13T13:22:19.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.998e-06 2024-09-13T13:22:19.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.197219+0000", 2024-09-13T13:22:19.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2340000000000002e-06 2024-09-13T13:22:19.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.386773+0000", 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.18955466000000001 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:04.386780+0000", 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7689999999999999e-06 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.196482+0000", 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.809702225 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.196605+0000", 2024-09-13T13:22:19.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012294199999999999 2024-09-13T13:22:19.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,7", 2024-09-13T13:22:19.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.196916+0000", 2024-09-13T13:22:19.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031042199999999998 2024-09-13T13:22:19.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.199444+0000", 2024-09-13T13:22:19.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025283889999999998 2024-09-13T13:22:19.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.202052+0000", 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002608478 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.209487+0000", 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0074344820000000001 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.209525+0000", 2024-09-13T13:22:19.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8175e-05 2024-09-13T13:22:19.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:05.209554+0000", 2024-09-13T13:22:19.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9278e-05 2024-09-13T13:22:19.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.428 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.428 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95615 11.14 11:288bf634:::benchmark_data_smithi052_182575_object95614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:22:19.428 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:06.203222+0000", 2024-09-13T13:22:19.428 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 193.13871494399999, 2024-09-13T13:22:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1021186639999998, 2024-09-13T13:22:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95615 2024-09-13T13:22:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.203222+0000", 2024-09-13T13:22:19.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.203222+0000", 2024-09-13T13:22:19.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.203224+0000", 2024-09-13T13:22:19.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1999999999999999e-06 2024-09-13T13:22:19.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.203284+0000", 2024-09-13T13:22:19.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0850000000000002e-05 2024-09-13T13:22:19.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.203295+0000", 2024-09-13T13:22:19.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0358e-05 2024-09-13T13:22:19.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.203301+0000", 2024-09-13T13:22:19.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8510000000000004e-06 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.203355+0000", 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4758e-05 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.203366+0000", 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1096e-05 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.181646+0000", 2024-09-13T13:22:19.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97827977799999999 2024-09-13T13:22:19.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.181658+0000", 2024-09-13T13:22:19.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2174999999999999e-05 2024-09-13T13:22:19.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.294205+0000", 2024-09-13T13:22:19.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1125468110000001 2024-09-13T13:22:19.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.294339+0000", 2024-09-13T13:22:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013360800000000001 2024-09-13T13:22:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2,4,7", 2024-09-13T13:22:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.294589+0000", 2024-09-13T13:22:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024978199999999999 2024-09-13T13:22:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.299968+0000", 2024-09-13T13:22:19.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0053795179999999998 2024-09-13T13:22:19.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.301371+0000", 2024-09-13T13:22:19.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014031740000000001 2024-09-13T13:22:19.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.301445+0000", 2024-09-13T13:22:19.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3743000000000002e-05 2024-09-13T13:22:19.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.303042+0000", 2024-09-13T13:22:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015969319999999999 2024-09-13T13:22:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.305174+0000", 2024-09-13T13:22:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021324099999999999 2024-09-13T13:22:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.305231+0000", 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7077000000000003e-05 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.305341+0000", 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109543 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:95618 11.14 11:29e6beaf:::benchmark_data_smithi052_182575_object95617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:19:06.204708+0000", 2024-09-13T13:22:19.437 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 193.137229332, 2024-09-13T13:22:19.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1030405029999999, 2024-09-13T13:22:19.438 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.438 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.438 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.438 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.438 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:19.438 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:19.438 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 95618 2024-09-13T13:22:19.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.438 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.204708+0000", 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.204708+0000", 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.204709+0000", 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1999999999999998e-07 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.204783+0000", 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4388000000000007e-05 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.204797+0000", 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3716999999999999e-05 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.204802+0000", 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5190000000000003e-06 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.204850+0000", 2024-09-13T13:22:19.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8511999999999999e-05 2024-09-13T13:22:19.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:06.204859+0000", 2024-09-13T13:22:19.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2410000000000001e-06 2024-09-13T13:22:19.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.181685+0000", 2024-09-13T13:22:19.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97682536499999995 2024-09-13T13:22:19.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:07.181693+0000", 2024-09-13T13:22:19.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1869999999999997e-06 2024-09-13T13:22:19.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.296498+0000", 2024-09-13T13:22:19.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.114805375 2024-09-13T13:22:19.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.296664+0000", 2024-09-13T13:22:19.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016615 2024-09-13T13:22:19.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2,4,7", 2024-09-13T13:22:19.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.296974+0000", 2024-09-13T13:22:19.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030919500000000001 2024-09-13T13:22:19.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.300719+0000", 2024-09-13T13:22:19.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003745445 2024-09-13T13:22:19.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.301410+0000", 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00069057400000000001 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.303082+0000", 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016725690000000001 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.305379+0000", 2024-09-13T13:22:19.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022964940000000001 2024-09-13T13:22:19.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.307594+0000", 2024-09-13T13:22:19.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002215322 2024-09-13T13:22:19.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.307648+0000", 2024-09-13T13:22:19.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4419999999999997e-05 2024-09-13T13:22:19.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:19:08.307748+0000", 2024-09-13T13:22:19.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010011 2024-09-13T13:22:19.446 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.446 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.446 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.446 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:150295 11.4 11:20a3ad74:::benchmark_data_smithi052_182575_object33286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-13T13:22:19.446 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:20:13.169517+0000", 2024-09-13T13:22:19.446 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 126.172420769, 2024-09-13T13:22:19.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96086420400000006, 2024-09-13T13:22:19.447 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.447 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.447 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.447 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.447 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:19.447 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:19.447 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 150295 2024-09-13T13:22:19.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.447 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.169517+0000", 2024-09-13T13:22:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.169517+0000", 2024-09-13T13:22:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.169519+0000", 2024-09-13T13:22:19.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5940000000000001e-06 2024-09-13T13:22:19.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.169525+0000", 2024-09-13T13:22:19.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0220000000000001e-06 2024-09-13T13:22:19.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.169550+0000", 2024-09-13T13:22:19.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4927999999999999e-05 2024-09-13T13:22:19.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.169558+0000", 2024-09-13T13:22:19.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1589999999999993e-06 2024-09-13T13:22:19.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.170446+0000", 2024-09-13T13:22:19.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088734199999999995 2024-09-13T13:22:19.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.170455+0000", 2024-09-13T13:22:19.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1460000000000002e-06 2024-09-13T13:22:19.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.124396+0000", 2024-09-13T13:22:19.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.953941708 2024-09-13T13:22:19.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.124487+0000", 2024-09-13T13:22:19.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0822000000000006e-05 2024-09-13T13:22:19.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:22:19.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.124791+0000", 2024-09-13T13:22:19.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000303403 2024-09-13T13:22:19.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.130174+0000", 2024-09-13T13:22:19.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005382876 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.130201+0000", 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7640999999999998e-05 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.130296+0000", 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5266999999999994e-05 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.130338+0000", 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1746999999999999e-05 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.130381+0000", 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2549000000000002e-05 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:150296 11.4 11:23980bb9:::benchmark_data_smithi052_182575_object33287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:20:13.169595+0000", 2024-09-13T13:22:19.454 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 126.172341876, 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96969239600000001, 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 150296 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.169595+0000", 2024-09-13T13:22:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.169595+0000", 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.169596+0000", 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0279999999999999e-06 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.169601+0000", 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0729999999999998e-06 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.169610+0000", 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5910000000000002e-06 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.169616+0000", 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6400000000000002e-06 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.170478+0000", 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00086178999999999997 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.170484+0000", 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0850000000000002e-06 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.126620+0000", 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95613602200000003 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.126730+0000", 2024-09-13T13:22:19.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110838 2024-09-13T13:22:19.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:22:19.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.126894+0000", 2024-09-13T13:22:19.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000163072 2024-09-13T13:22:19.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.132202+0000", 2024-09-13T13:22:19.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0053088079999999999 2024-09-13T13:22:19.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.135095+0000", 2024-09-13T13:22:19.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028922520000000001 2024-09-13T13:22:19.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.139216+0000", 2024-09-13T13:22:19.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041216869999999997 2024-09-13T13:22:19.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.139249+0000", 2024-09-13T13:22:19.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2388000000000002e-05 2024-09-13T13:22:19.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.139288+0000", 2024-09-13T13:22:19.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9122000000000003e-05 2024-09-13T13:22:19.462 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.462 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.462 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.462 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:150317 11.4 11:21ac3e53:::benchmark_data_smithi052_182575_object33308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-13T13:22:19.462 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:20:13.192698+0000", 2024-09-13T13:22:19.462 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 126.149239356, 2024-09-13T13:22:19.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94668405200000005, 2024-09-13T13:22:19.463 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.463 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.463 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.463 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.463 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:19.463 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:19.463 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 150317 2024-09-13T13:22:19.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.463 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.192698+0000", 2024-09-13T13:22:19.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.192698+0000", 2024-09-13T13:22:19.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.192700+0000", 2024-09-13T13:22:19.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3420000000000002e-06 2024-09-13T13:22:19.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.192706+0000", 2024-09-13T13:22:19.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-09-13T13:22:19.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.192720+0000", 2024-09-13T13:22:19.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3677e-05 2024-09-13T13:22:19.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.192727+0000", 2024-09-13T13:22:19.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8809999999999998e-06 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.192780+0000", 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2957999999999997e-05 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.192828+0000", 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7889999999999997e-05 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.128001+0000", 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93517313099999999 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.128061+0000", 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0177999999999999e-05 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.128209+0000", 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147403 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.132210+0000", 2024-09-13T13:22:19.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040012509999999999 2024-09-13T13:22:19.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.137140+0000", 2024-09-13T13:22:19.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049305160000000002 2024-09-13T13:22:19.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.139321+0000", 2024-09-13T13:22:19.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021810470000000002 2024-09-13T13:22:19.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.139356+0000", 2024-09-13T13:22:19.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4669999999999998e-05 2024-09-13T13:22:19.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.139382+0000", 2024-09-13T13:22:19.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5987999999999999e-05 2024-09-13T13:22:19.469 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.469 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.469 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:150377 11.4 11:25bdfa86:::benchmark_data_smithi052_182575_object33368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:20:13.249020+0000", 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 126.092917576, 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.89044860000000003, 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 150377 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.470 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.249020+0000", 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.249020+0000", 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.249021+0000", 2024-09-13T13:22:19.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.674e-06 2024-09-13T13:22:19.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.249026+0000", 2024-09-13T13:22:19.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0409999999999997e-06 2024-09-13T13:22:19.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.249037+0000", 2024-09-13T13:22:19.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0142e-05 2024-09-13T13:22:19.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.249044+0000", 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9990000000000002e-06 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.249130+0000", 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6669999999999995e-05 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.249138+0000", 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5490000000000001e-06 2024-09-13T13:22:19.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.129103+0000", 2024-09-13T13:22:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.879965094 2024-09-13T13:22:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.129164+0000", 2024-09-13T13:22:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0843e-05 2024-09-13T13:22:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:22:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.129312+0000", 2024-09-13T13:22:19.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014812700000000001 2024-09-13T13:22:19.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.135056+0000", 2024-09-13T13:22:19.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057437779999999997 2024-09-13T13:22:19.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.137200+0000", 2024-09-13T13:22:19.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00214452 2024-09-13T13:22:19.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.139410+0000", 2024-09-13T13:22:19.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022095940000000001 2024-09-13T13:22:19.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.139438+0000", 2024-09-13T13:22:19.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7928000000000001e-05 2024-09-13T13:22:19.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.139468+0000", 2024-09-13T13:22:19.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0641e-05 2024-09-13T13:22:19.477 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.477 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.477 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.477 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:150426 11.4 11:20abb328:::benchmark_data_smithi052_182575_object33417:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-13T13:22:19.477 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:20:13.303382+0000", 2024-09-13T13:22:19.477 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 126.03855575999999, 2024-09-13T13:22:19.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.83619816199999997, 2024-09-13T13:22:19.477 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.477 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.477 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:22:19.477 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.478 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:19.478 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:19.478 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 150426 2024-09-13T13:22:19.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.478 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.303382+0000", 2024-09-13T13:22:19.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.303382+0000", 2024-09-13T13:22:19.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.303383+0000", 2024-09-13T13:22:19.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.88e-06 2024-09-13T13:22:19.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.303389+0000", 2024-09-13T13:22:19.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7899999999999996e-06 2024-09-13T13:22:19.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.303418+0000", 2024-09-13T13:22:19.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.881e-05 2024-09-13T13:22:19.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.303427+0000", 2024-09-13T13:22:19.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.816e-06 2024-09-13T13:22:19.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.303514+0000", 2024-09-13T13:22:19.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6890000000000003e-05 2024-09-13T13:22:19.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:13.303522+0000", 2024-09-13T13:22:19.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2779999999999993e-06 2024-09-13T13:22:19.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.130434+0000", 2024-09-13T13:22:19.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.82691157900000001 2024-09-13T13:22:19.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.130546+0000", 2024-09-13T13:22:19.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112219 2024-09-13T13:22:19.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-09-13T13:22:19.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.130792+0000", 2024-09-13T13:22:19.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024603899999999998 2024-09-13T13:22:19.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-09-13T13:22:19.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.135107+0000", 2024-09-13T13:22:19.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0043148300000000004 2024-09-13T13:22:19.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.137247+0000", 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021404710000000001 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.139524+0000", 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022765429999999998 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.139550+0000", 2024-09-13T13:22:19.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6080000000000001e-05 2024-09-13T13:22:19.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:20:14.139580+0000", 2024-09-13T13:22:19.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9937000000000002e-05 2024-09-13T13:22:19.484 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.484 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.484 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.484 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.36635.0:222107 11.14 11:29998208:::benchmark_data_smithi052_182575_object105098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-13T13:22:19.484 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:21:25.332956+0000", 2024-09-13T13:22:19.484 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 54.008981036999998, 2024-09-13T13:22:19.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.939874243, 2024-09-13T13:22:19.485 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:19.485 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:19.485 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:19.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:19.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.36635", 2024-09-13T13:22:19.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/3853305648", 2024-09-13T13:22:19.485 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 222107 2024-09-13T13:22:19.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.488 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:19.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:19.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.332956+0000", 2024-09-13T13:22:19.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:19.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.332956+0000", 2024-09-13T13:22:19.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:19.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:19.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.332957+0000", 2024-09-13T13:22:19.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7999999999999995e-07 2024-09-13T13:22:19.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:19.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.332959+0000", 2024-09-13T13:22:19.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.599e-06 2024-09-13T13:22:19.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:19.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.332964+0000", 2024-09-13T13:22:19.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.006e-06 2024-09-13T13:22:19.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.332969+0000", 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7489999999999998e-06 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.333629+0000", 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066001799999999995 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.333738+0000", 2024-09-13T13:22:19.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108967 2024-09-13T13:22:19.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-09-13T13:22:19.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.334122+0000", 2024-09-13T13:22:19.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000383383 2024-09-13T13:22:19.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.341971+0000", 2024-09-13T13:22:19.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078491780000000001 2024-09-13T13:22:19.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-09-13T13:22:19.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:25.341980+0000", 2024-09-13T13:22:19.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8829999999999995e-06 2024-09-13T13:22:19.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.263567+0000", 2024-09-13T13:22:19.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92158704199999997 2024-09-13T13:22:19.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-09-13T13:22:19.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.263668+0000", 2024-09-13T13:22:19.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100991 2024-09-13T13:22:19.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.272753+0000", 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090848669999999999 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:21:26.272831+0000", 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7979999999999995e-05 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:19.493 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:19.494 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:22:19.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[29471]: 5.6 scrub starts 2024-09-13T13:22:19.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[29471]: 5.6 scrub ok 2024-09-13T13:22:19.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[29471]: pgmap v1078: 142 pgs: 39 active+undersized, 21 active+undersized+degraded, 82 active+clean; 635 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 81/597 objects degraded (13.568%) 2024-09-13T13:22:19.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[29471]: Health check update: Degraded data redundancy: 81/597 objects degraded (13.568%), 21 pgs degraded (PG_DEGRADED) 2024-09-13T13:22:19.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[29471]: 3.17 scrub starts 2024-09-13T13:22:19.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[29471]: 3.17 scrub ok 2024-09-13T13:22:19.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[29471]: 3.a scrub starts 2024-09-13T13:22:19.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[29471]: 3.a scrub ok 2024-09-13T13:22:19.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[29471]: 5.7 scrub starts 2024-09-13T13:22:19.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[29471]: 5.7 scrub ok 2024-09-13T13:22:19.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[29471]: 4.c scrub starts 2024-09-13T13:22:19.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[29471]: 4.c scrub ok 2024-09-13T13:22:19.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[29471]: osdmap e526: 8 total, 8 up, 8 in 2024-09-13T13:22:19.495 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[36682]: 5.6 scrub starts 2024-09-13T13:22:19.495 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[36682]: 5.6 scrub ok 2024-09-13T13:22:19.495 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[36682]: pgmap v1078: 142 pgs: 39 active+undersized, 21 active+undersized+degraded, 82 active+clean; 635 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 81/597 objects degraded (13.568%) 2024-09-13T13:22:19.495 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[36682]: Health check update: Degraded data redundancy: 81/597 objects degraded (13.568%), 21 pgs degraded (PG_DEGRADED) 2024-09-13T13:22:19.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[36682]: 3.17 scrub starts 2024-09-13T13:22:19.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[36682]: 3.17 scrub ok 2024-09-13T13:22:19.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[36682]: 3.a scrub starts 2024-09-13T13:22:19.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[36682]: 3.a scrub ok 2024-09-13T13:22:19.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[36682]: 5.7 scrub starts 2024-09-13T13:22:19.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[36682]: 5.7 scrub ok 2024-09-13T13:22:19.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[36682]: 4.c scrub starts 2024-09-13T13:22:19.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[36682]: 4.c scrub ok 2024-09-13T13:22:19.496 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:19 smithi052 ceph-mon[36682]: osdmap e526: 8 total, 8 up, 8 in 2024-09-13T13:22:19.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:19 smithi167 ceph-mon[33178]: 5.6 scrub starts 2024-09-13T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:19 smithi167 ceph-mon[33178]: 5.6 scrub ok 2024-09-13T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:19 smithi167 ceph-mon[33178]: pgmap v1078: 142 pgs: 39 active+undersized, 21 active+undersized+degraded, 82 active+clean; 635 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 81/597 objects degraded (13.568%) 2024-09-13T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:19 smithi167 ceph-mon[33178]: Health check update: Degraded data redundancy: 81/597 objects degraded (13.568%), 21 pgs degraded (PG_DEGRADED) 2024-09-13T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:19 smithi167 ceph-mon[33178]: 3.17 scrub starts 2024-09-13T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:19 smithi167 ceph-mon[33178]: 3.17 scrub ok 2024-09-13T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:19 smithi167 ceph-mon[33178]: 3.a scrub starts 2024-09-13T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:19 smithi167 ceph-mon[33178]: 3.a scrub ok 2024-09-13T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:19 smithi167 ceph-mon[33178]: 5.7 scrub starts 2024-09-13T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:19 smithi167 ceph-mon[33178]: 5.7 scrub ok 2024-09-13T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:19 smithi167 ceph-mon[33178]: 4.c scrub starts 2024-09-13T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:19 smithi167 ceph-mon[33178]: 4.c scrub ok 2024-09-13T13:22:19.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:19 smithi167 ceph-mon[33178]: osdmap e526: 8 total, 8 up, 8 in 2024-09-13T13:22:20.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T13:22:20.404 INFO:teuthology.orchestra.run.smithi052.stderr:set pool 1 pgp_num to 11 2024-09-13T13:22:20.524 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:20.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:20 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:22:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:20 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:22:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:20 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1310084372' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-13T13:22:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:20 smithi167 ceph-mon[33178]: from='client.36995 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-13T13:22:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:20 smithi167 ceph-mon[33178]: 4.1a scrub starts 2024-09-13T13:22:20.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:20 smithi167 ceph-mon[33178]: 4.1a scrub ok 2024-09-13T13:22:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:20 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:22:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:20 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:22:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:20 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1310084372' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-13T13:22:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:20 smithi052 ceph-mon[36682]: from='client.36995 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-13T13:22:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:20 smithi052 ceph-mon[36682]: 4.1a scrub starts 2024-09-13T13:22:20.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:20 smithi052 ceph-mon[36682]: 4.1a scrub ok 2024-09-13T13:22:20.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:20 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:22:20.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:20 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:22:20.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:20 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1310084372' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-13T13:22:20.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:20 smithi052 ceph-mon[29471]: from='client.36995 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-13T13:22:20.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:20 smithi052 ceph-mon[29471]: 4.1a scrub starts 2024-09-13T13:22:20.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:20 smithi052 ceph-mon[29471]: 4.1a scrub ok 2024-09-13T13:22:21.060 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:22:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:22:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-09-13T13:22:21.060 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:22:21.153 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-13T13:22:21.419 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:21.674 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[36682]: pgmap v1081: 142 pgs: 29 peering, 113 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2024-09-13T13:22:21.674 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[36682]: 3.13 scrub starts 2024-09-13T13:22:21.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[36682]: 3.13 scrub ok 2024-09-13T13:22:21.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[36682]: 4.1d scrub starts 2024-09-13T13:22:21.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[36682]: 4.1d scrub ok 2024-09-13T13:22:21.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[36682]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T13:22:21.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[36682]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/597 objects degraded (13.568%), 21 pgs degraded) 2024-09-13T13:22:21.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[36682]: from='client.36995 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-13T13:22:21.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[36682]: osdmap e527: 8 total, 8 up, 8 in 2024-09-13T13:22:21.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[36682]: 4.6 scrub starts 2024-09-13T13:22:21.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[36682]: 4.6 scrub ok 2024-09-13T13:22:21.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[36682]: 6.1e scrub starts 2024-09-13T13:22:21.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[36682]: 6.1e scrub ok 2024-09-13T13:22:21.675 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[29471]: pgmap v1081: 142 pgs: 29 peering, 113 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2024-09-13T13:22:21.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[29471]: 3.13 scrub starts 2024-09-13T13:22:21.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[29471]: 3.13 scrub ok 2024-09-13T13:22:21.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[29471]: 4.1d scrub starts 2024-09-13T13:22:21.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[29471]: 4.1d scrub ok 2024-09-13T13:22:21.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[29471]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T13:22:21.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[29471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/597 objects degraded (13.568%), 21 pgs degraded) 2024-09-13T13:22:21.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[29471]: from='client.36995 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-13T13:22:21.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[29471]: osdmap e527: 8 total, 8 up, 8 in 2024-09-13T13:22:21.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[29471]: 4.6 scrub starts 2024-09-13T13:22:21.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[29471]: 4.6 scrub ok 2024-09-13T13:22:21.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[29471]: 6.1e scrub starts 2024-09-13T13:22:21.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:21 smithi052 ceph-mon[29471]: 6.1e scrub ok 2024-09-13T13:22:21.790 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:21 smithi167 ceph-mon[33178]: pgmap v1081: 142 pgs: 29 peering, 113 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2024-09-13T13:22:21.790 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:21 smithi167 ceph-mon[33178]: 3.13 scrub starts 2024-09-13T13:22:21.790 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:21 smithi167 ceph-mon[33178]: 3.13 scrub ok 2024-09-13T13:22:21.791 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:21 smithi167 ceph-mon[33178]: 4.1d scrub starts 2024-09-13T13:22:21.791 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:21 smithi167 ceph-mon[33178]: 4.1d scrub ok 2024-09-13T13:22:21.791 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:21 smithi167 ceph-mon[33178]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T13:22:21.791 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:21 smithi167 ceph-mon[33178]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/597 objects degraded (13.568%), 21 pgs degraded) 2024-09-13T13:22:21.791 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:21 smithi167 ceph-mon[33178]: from='client.36995 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-13T13:22:21.791 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:21 smithi167 ceph-mon[33178]: osdmap e527: 8 total, 8 up, 8 in 2024-09-13T13:22:21.791 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:21 smithi167 ceph-mon[33178]: 4.6 scrub starts 2024-09-13T13:22:21.791 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:21 smithi167 ceph-mon[33178]: 4.6 scrub ok 2024-09-13T13:22:21.791 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:21 smithi167 ceph-mon[33178]: 6.1e scrub starts 2024-09-13T13:22:21.791 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:21 smithi167 ceph-mon[33178]: 6.1e scrub ok 2024-09-13T13:22:21.819 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T13:22:21.882 INFO:teuthology.orchestra.run.smithi052.stdout:mon.a: {} 2024-09-13T13:22:21.882 INFO:teuthology.orchestra.run.smithi052.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T13:22:21.891 INFO:teuthology.orchestra.run.smithi052.stdout:mon.b: {} 2024-09-13T13:22:21.891 INFO:teuthology.orchestra.run.smithi052.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T13:22:21.900 INFO:teuthology.orchestra.run.smithi052.stdout:mon.c: {} 2024-09-13T13:22:21.900 INFO:teuthology.orchestra.run.smithi052.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T13:22:22.066 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:22.494 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:22 smithi167 ceph-mon[33178]: 6.1d scrub starts 2024-09-13T13:22:22.495 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:22 smithi167 ceph-mon[33178]: 6.1d scrub ok 2024-09-13T13:22:22.495 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:22 smithi167 ceph-mon[33178]: 5.d scrub starts 2024-09-13T13:22:22.495 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:22 smithi167 ceph-mon[33178]: 5.d scrub ok 2024-09-13T13:22:22.495 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:22 smithi167 ceph-mon[33178]: 6.15 scrub starts 2024-09-13T13:22:22.495 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:22 smithi167 ceph-mon[33178]: 6.15 scrub ok 2024-09-13T13:22:22.495 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:22 smithi167 ceph-mon[33178]: 5.0 scrub starts 2024-09-13T13:22:22.495 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:22 smithi167 ceph-mon[33178]: 5.0 scrub ok 2024-09-13T13:22:22.495 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:22 smithi167 ceph-mon[33178]: 1.2 scrub starts 2024-09-13T13:22:22.495 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:22 smithi167 ceph-mon[33178]: 1.2 scrub ok 2024-09-13T13:22:22.495 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:22 smithi167 ceph-mon[33178]: 5.f scrub starts 2024-09-13T13:22:22.496 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:22 smithi167 ceph-mon[33178]: 5.f scrub ok 2024-09-13T13:22:22.496 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:22 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1577685651' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T13:22:22.595 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:22:22.595 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-09-13T13:22:22.595 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-09-13T13:22:22.595 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-09-13T13:22:22.595 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:22:22.695 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[36682]: 6.1d scrub starts 2024-09-13T13:22:22.695 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[36682]: 6.1d scrub ok 2024-09-13T13:22:22.695 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[36682]: 5.d scrub starts 2024-09-13T13:22:22.695 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[36682]: 5.d scrub ok 2024-09-13T13:22:22.695 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[36682]: 6.15 scrub starts 2024-09-13T13:22:22.695 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[36682]: 6.15 scrub ok 2024-09-13T13:22:22.695 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[36682]: 5.0 scrub starts 2024-09-13T13:22:22.695 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[36682]: 5.0 scrub ok 2024-09-13T13:22:22.695 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[36682]: 1.2 scrub starts 2024-09-13T13:22:22.695 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[36682]: 1.2 scrub ok 2024-09-13T13:22:22.695 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[36682]: 5.f scrub starts 2024-09-13T13:22:22.695 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[36682]: 5.f scrub ok 2024-09-13T13:22:22.696 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1577685651' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T13:22:22.696 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[29471]: 6.1d scrub starts 2024-09-13T13:22:22.696 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[29471]: 6.1d scrub ok 2024-09-13T13:22:22.696 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[29471]: 5.d scrub starts 2024-09-13T13:22:22.696 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[29471]: 5.d scrub ok 2024-09-13T13:22:22.696 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[29471]: 6.15 scrub starts 2024-09-13T13:22:22.696 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[29471]: 6.15 scrub ok 2024-09-13T13:22:22.696 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[29471]: 5.0 scrub starts 2024-09-13T13:22:22.696 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[29471]: 5.0 scrub ok 2024-09-13T13:22:22.696 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[29471]: 1.2 scrub starts 2024-09-13T13:22:22.697 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[29471]: 1.2 scrub ok 2024-09-13T13:22:22.697 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[29471]: 5.f scrub starts 2024-09-13T13:22:22.697 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[29471]: 5.f scrub ok 2024-09-13T13:22:22.697 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:22 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1577685651' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T13:22:22.727 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T13:22:22.988 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:23.692 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:23 smithi167 ceph-mon[33178]: 4.f scrub starts 2024-09-13T13:22:23.692 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:23 smithi167 ceph-mon[33178]: 4.f scrub ok 2024-09-13T13:22:23.692 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:23 smithi167 ceph-mon[33178]: pgmap v1083: 142 pgs: 29 peering, 113 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2024-09-13T13:22:23.692 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:23 smithi167 ceph-mon[33178]: 5.10 scrub starts 2024-09-13T13:22:23.692 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:23 smithi167 ceph-mon[33178]: 5.10 scrub ok 2024-09-13T13:22:23.693 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:23 smithi167 ceph-mon[33178]: 6.c scrub starts 2024-09-13T13:22:23.693 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:23 smithi167 ceph-mon[33178]: 6.c scrub ok 2024-09-13T13:22:23.693 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:23 smithi167 ceph-mon[33178]: 3.1b scrub starts 2024-09-13T13:22:23.693 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:23 smithi167 ceph-mon[33178]: 3.1b scrub ok 2024-09-13T13:22:23.693 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:23 smithi167 ceph-mon[33178]: 1.4 scrub starts 2024-09-13T13:22:23.693 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:23 smithi167 ceph-mon[33178]: 1.4 scrub ok 2024-09-13T13:22:23.724 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T13:22:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[36682]: 4.f scrub starts 2024-09-13T13:22:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[36682]: 4.f scrub ok 2024-09-13T13:22:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[36682]: pgmap v1083: 142 pgs: 29 peering, 113 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2024-09-13T13:22:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[36682]: 5.10 scrub starts 2024-09-13T13:22:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[36682]: 5.10 scrub ok 2024-09-13T13:22:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[36682]: 6.c scrub starts 2024-09-13T13:22:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[36682]: 6.c scrub ok 2024-09-13T13:22:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[36682]: 3.1b scrub starts 2024-09-13T13:22:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[36682]: 3.1b scrub ok 2024-09-13T13:22:23.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[36682]: 1.4 scrub starts 2024-09-13T13:22:23.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[36682]: 1.4 scrub ok 2024-09-13T13:22:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[29471]: 4.f scrub starts 2024-09-13T13:22:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[29471]: 4.f scrub ok 2024-09-13T13:22:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[29471]: pgmap v1083: 142 pgs: 29 peering, 113 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2024-09-13T13:22:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[29471]: 5.10 scrub starts 2024-09-13T13:22:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[29471]: 5.10 scrub ok 2024-09-13T13:22:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[29471]: 6.c scrub starts 2024-09-13T13:22:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[29471]: 6.c scrub ok 2024-09-13T13:22:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[29471]: 3.1b scrub starts 2024-09-13T13:22:23.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[29471]: 3.1b scrub ok 2024-09-13T13:22:23.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[29471]: 1.4 scrub starts 2024-09-13T13:22:23.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:23 smithi052 ceph-mon[29471]: 1.4 scrub ok 2024-09-13T13:22:23.973 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T13:22:24.443 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T13:22:24.481 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-09-13T13:22:24.482 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-09-13T13:22:24.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-09-13T13:22:24.482 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-09-13T13:22:24.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.482 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDScrubReserve(6.16,QREQUEST,e:524,from:0,reservation_nonce:3)", 2024-09-13T13:22:24.482 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:16.487260+0000", 2024-09-13T13:22:24.482 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 7.9923204219999997, 2024-09-13T13:22:24.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018842899999999999, 2024-09-13T13:22:24.482 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.482 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.483 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:22:24.483 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.487260+0000", 2024-09-13T13:22:24.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.487260+0000", 2024-09-13T13:22:24.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.487262+0000", 2024-09-13T13:22:24.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2680000000000001e-06 2024-09-13T13:22:24.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.487269+0000", 2024-09-13T13:22:24.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5299999999999999e-06 2024-09-13T13:22:24.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.487285+0000", 2024-09-13T13:22:24.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.522e-05 2024-09-13T13:22:24.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.487309+0000", 2024-09-13T13:22:24.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4570999999999999e-05 2024-09-13T13:22:24.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.487381+0000", 2024-09-13T13:22:24.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2231999999999996e-05 2024-09-13T13:22:24.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:24.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.487404+0000", 2024-09-13T13:22:24.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2495000000000001e-05 2024-09-13T13:22:24.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.487448+0000", 2024-09-13T13:22:24.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4113000000000003e-05 2024-09-13T13:22:24.487 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.487 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.487 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.487 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDScrubReserve(6.16,RELEASE,e:524,from:0,reservation_nonce:0)", 2024-09-13T13:22:24.487 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:16.489077+0000", 2024-09-13T13:22:24.487 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 7.9905025089999997, 2024-09-13T13:22:24.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000171199, 2024-09-13T13:22:24.487 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.488 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.488 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:22:24.488 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.489077+0000", 2024-09-13T13:22:24.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.489077+0000", 2024-09-13T13:22:24.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.489078+0000", 2024-09-13T13:22:24.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7899999999999998e-07 2024-09-13T13:22:24.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.489081+0000", 2024-09-13T13:22:24.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1190000000000001e-06 2024-09-13T13:22:24.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.489097+0000", 2024-09-13T13:22:24.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6123000000000001e-05 2024-09-13T13:22:24.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.489103+0000", 2024-09-13T13:22:24.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5990000000000001e-06 2024-09-13T13:22:24.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.489219+0000", 2024-09-13T13:22:24.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011616999999999999 2024-09-13T13:22:24.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:24.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.489234+0000", 2024-09-13T13:22:24.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4576000000000001e-05 2024-09-13T13:22:24.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.489249+0000", 2024-09-13T13:22:24.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5033e-05 2024-09-13T13:22:24.492 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.492 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.492 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.493 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDScrubReserve(1.3,QREQUEST,e:524,from:1,reservation_nonce:2)", 2024-09-13T13:22:24.493 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:16.981228+0000", 2024-09-13T13:22:24.493 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 7.4983517720000004, 2024-09-13T13:22:24.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030082600000000002, 2024-09-13T13:22:24.493 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.493 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.493 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:22:24.493 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.981228+0000", 2024-09-13T13:22:24.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.981228+0000", 2024-09-13T13:22:24.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.981231+0000", 2024-09-13T13:22:24.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0199999999999999e-06 2024-09-13T13:22:24.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.981241+0000", 2024-09-13T13:22:24.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0117e-05 2024-09-13T13:22:24.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.981267+0000", 2024-09-13T13:22:24.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6149999999999999e-05 2024-09-13T13:22:24.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.981282+0000", 2024-09-13T13:22:24.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4929000000000001e-05 2024-09-13T13:22:24.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.981429+0000", 2024-09-13T13:22:24.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000146911 2024-09-13T13:22:24.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:24.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.981464+0000", 2024-09-13T13:22:24.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5040000000000003e-05 2024-09-13T13:22:24.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.981529+0000", 2024-09-13T13:22:24.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4659000000000006e-05 2024-09-13T13:22:24.498 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.498 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.498 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.498 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDScrubReserve(1.3,RELEASE,e:524,from:1,reservation_nonce:0)", 2024-09-13T13:22:24.499 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:16.984431+0000", 2024-09-13T13:22:24.499 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 7.4951494289999996, 2024-09-13T13:22:24.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022982499999999999, 2024-09-13T13:22:24.499 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.499 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.499 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:22:24.499 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.984431+0000", 2024-09-13T13:22:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.984431+0000", 2024-09-13T13:22:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.984431+0000", 2024-09-13T13:22:24.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5000000000000001e-07 2024-09-13T13:22:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.984436+0000", 2024-09-13T13:22:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7740000000000004e-06 2024-09-13T13:22:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.984445+0000", 2024-09-13T13:22:24.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1060000000000006e-06 2024-09-13T13:22:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.984462+0000", 2024-09-13T13:22:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6317e-05 2024-09-13T13:22:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.984621+0000", 2024-09-13T13:22:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000159409 2024-09-13T13:22:24.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.984641+0000", 2024-09-13T13:22:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.003e-05 2024-09-13T13:22:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:16.984660+0000", 2024-09-13T13:22:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9338999999999999e-05 2024-09-13T13:22:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDScrubReserve(6.14,QREQUEST,e:524,from:2,reservation_nonce:3)", 2024-09-13T13:22:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:17.026560+0000", 2024-09-13T13:22:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 7.4530199450000003, 2024-09-13T13:22:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025680199999999999, 2024-09-13T13:22:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:22:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.026560+0000", 2024-09-13T13:22:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.026560+0000", 2024-09-13T13:22:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.026563+0000", 2024-09-13T13:22:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.41e-06 2024-09-13T13:22:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.026574+0000", 2024-09-13T13:22:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0662000000000001e-05 2024-09-13T13:22:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.026600+0000", 2024-09-13T13:22:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6279999999999999e-05 2024-09-13T13:22:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.026620+0000", 2024-09-13T13:22:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9502999999999999e-05 2024-09-13T13:22:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.026770+0000", 2024-09-13T13:22:24.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000150261 2024-09-13T13:22:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.026791+0000", 2024-09-13T13:22:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0843e-05 2024-09-13T13:22:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.026817+0000", 2024-09-13T13:22:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5843e-05 2024-09-13T13:22:24.508 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14493.0:4441 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94064716860672 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-13T13:22:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:17.497987+0000", 2024-09-13T13:22:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.9815932309999997, 2024-09-13T13:22:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010738029999999999, 2024-09-13T13:22:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.509 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:22:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14493", 2024-09-13T13:22:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:22:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4441 2024-09-13T13:22:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.497987+0000", 2024-09-13T13:22:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.497987+0000", 2024-09-13T13:22:24.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.497989+0000", 2024-09-13T13:22:24.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3410000000000001e-06 2024-09-13T13:22:24.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.497997+0000", 2024-09-13T13:22:24.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4430000000000003e-06 2024-09-13T13:22:24.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498009+0000", 2024-09-13T13:22:24.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1209999999999999e-05 2024-09-13T13:22:24.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498017+0000", 2024-09-13T13:22:24.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3669999999999997e-06 2024-09-13T13:22:24.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498094+0000", 2024-09-13T13:22:24.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7294999999999998e-05 2024-09-13T13:22:24.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:24.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498213+0000", 2024-09-13T13:22:24.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011887000000000001 2024-09-13T13:22:24.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.499061+0000", 2024-09-13T13:22:24.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00084727700000000001 2024-09-13T13:22:24.515 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.515 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.515 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.515 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "pg_update_log_missing(4.5 epoch 524/523 rep_tid 175504 entries 524'47 (0'0) error 4:a60f9269:lc::lua_package_allowlist:head by client.14493.0:4444 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-13T13:22:24.515 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:17.498126+0000", 2024-09-13T13:22:24.515 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.9814537090000002, 2024-09-13T13:22:24.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00047055300000000002, 2024-09-13T13:22:24.515 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.516 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.516 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:24.516 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498126+0000", 2024-09-13T13:22:24.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498126+0000", 2024-09-13T13:22:24.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498127+0000", 2024-09-13T13:22:24.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2559999999999999e-06 2024-09-13T13:22:24.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498136+0000", 2024-09-13T13:22:24.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5909999999999996e-06 2024-09-13T13:22:24.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498149+0000", 2024-09-13T13:22:24.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3151e-05 2024-09-13T13:22:24.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498158+0000", 2024-09-13T13:22:24.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5010000000000004e-06 2024-09-13T13:22:24.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498231+0000", 2024-09-13T13:22:24.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3749000000000002e-05 2024-09-13T13:22:24.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498597+0000", 2024-09-13T13:22:24.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036530500000000001 2024-09-13T13:22:24.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.520 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.521 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "pg_update_log_missing(5.1f epoch 524/523 rep_tid 175505 entries 524'15 (0'0) error 5:f95f44c2:::notify.0:head by client.14493.0:4442 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-13T13:22:24.521 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:17.498172+0000", 2024-09-13T13:22:24.521 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.9814075869999996, 2024-09-13T13:22:24.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040096700000000001, 2024-09-13T13:22:24.521 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.521 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.521 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:24.521 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498172+0000", 2024-09-13T13:22:24.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498172+0000", 2024-09-13T13:22:24.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498173+0000", 2024-09-13T13:22:24.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3200000000000001e-07 2024-09-13T13:22:24.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498176+0000", 2024-09-13T13:22:24.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.489e-06 2024-09-13T13:22:24.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498181+0000", 2024-09-13T13:22:24.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1429999999999997e-06 2024-09-13T13:22:24.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498196+0000", 2024-09-13T13:22:24.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4952e-05 2024-09-13T13:22:24.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498255+0000", 2024-09-13T13:22:24.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8817999999999998e-05 2024-09-13T13:22:24.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498573+0000", 2024-09-13T13:22:24.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031823300000000001 2024-09-13T13:22:24.525 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.525 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.525 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.525 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "pg_update_log_missing(5.9 epoch 524/523 rep_tid 175506 entries 524'16 (0'0) error 5:93e5b521:::notify.7:head by client.14493.0:4443 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-13T13:22:24.526 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:17.498207+0000", 2024-09-13T13:22:24.526 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.9813728340000001, 2024-09-13T13:22:24.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051166999999999999, 2024-09-13T13:22:24.526 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.526 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.526 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:24.526 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498207+0000", 2024-09-13T13:22:24.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498207+0000", 2024-09-13T13:22:24.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498208+0000", 2024-09-13T13:22:24.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6100000000000001e-07 2024-09-13T13:22:24.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498212+0000", 2024-09-13T13:22:24.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5020000000000003e-06 2024-09-13T13:22:24.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498220+0000", 2024-09-13T13:22:24.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:22:24.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498226+0000", 2024-09-13T13:22:24.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9120000000000003e-06 2024-09-13T13:22:24.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498296+0000", 2024-09-13T13:22:24.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9905000000000002e-05 2024-09-13T13:22:24.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498719+0000", 2024-09-13T13:22:24.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00042272300000000001 2024-09-13T13:22:24.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.529 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.530 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.530 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "pg_update_log_missing(5.0 epoch 524/523 rep_tid 222164 entries 524'12 (0'0) error 5:05bf5b68:::notify.1:head by client.14493.0:4440 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-13T13:22:24.530 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:17.498251+0000", 2024-09-13T13:22:24.530 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.9813286860000003, 2024-09-13T13:22:24.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00046977199999999997, 2024-09-13T13:22:24.530 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.530 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.530 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:24.530 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498251+0000", 2024-09-13T13:22:24.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498251+0000", 2024-09-13T13:22:24.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498252+0000", 2024-09-13T13:22:24.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.92e-07 2024-09-13T13:22:24.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498257+0000", 2024-09-13T13:22:24.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6290000000000001e-06 2024-09-13T13:22:24.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498264+0000", 2024-09-13T13:22:24.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2989999999999999e-06 2024-09-13T13:22:24.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498273+0000", 2024-09-13T13:22:24.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6600000000000001e-06 2024-09-13T13:22:24.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498335+0000", 2024-09-13T13:22:24.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2242999999999993e-05 2024-09-13T13:22:24.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:17.498721+0000", 2024-09-13T13:22:24.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038624900000000002 2024-09-13T13:22:24.534 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.534 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.534 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.534 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDScrubReserve(4.c,GRANT,e:525,from:6,reservation_nonce:0)", 2024-09-13T13:22:24.535 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:19.129379+0000", 2024-09-13T13:22:24.535 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3502011439999997, 2024-09-13T13:22:24.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018640099999999999, 2024-09-13T13:22:24.535 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.535 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.535 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:22:24.535 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.129379+0000", 2024-09-13T13:22:24.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.129379+0000", 2024-09-13T13:22:24.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.129380+0000", 2024-09-13T13:22:24.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:22:24.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.129382+0000", 2024-09-13T13:22:24.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3769999999999999e-06 2024-09-13T13:22:24.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.129389+0000", 2024-09-13T13:22:24.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.883e-06 2024-09-13T13:22:24.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.129394+0000", 2024-09-13T13:22:24.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7709999999999997e-06 2024-09-13T13:22:24.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.129430+0000", 2024-09-13T13:22:24.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6440999999999999e-05 2024-09-13T13:22:24.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:24.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.129443+0000", 2024-09-13T13:22:24.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3142999999999999e-05 2024-09-13T13:22:24.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.129565+0000", 2024-09-13T13:22:24.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121975 2024-09-13T13:22:24.539 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.539 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.539 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.540 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPushReply(3.f 526/525 [PushReplyOp(3:f43b6ece:::zone_names.default:head)])", 2024-09-13T13:22:24.540 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:19.214085+0000", 2024-09-13T13:22:24.540 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.265495252, 2024-09-13T13:22:24.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028303600000000002, 2024-09-13T13:22:24.540 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.540 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.540 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:24.540 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.214085+0000", 2024-09-13T13:22:24.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.214085+0000", 2024-09-13T13:22:24.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.214086+0000", 2024-09-13T13:22:24.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7999999999999993e-07 2024-09-13T13:22:24.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.214091+0000", 2024-09-13T13:22:24.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9239999999999998e-06 2024-09-13T13:22:24.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.214103+0000", 2024-09-13T13:22:24.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2301e-05 2024-09-13T13:22:24.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.214113+0000", 2024-09-13T13:22:24.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0275000000000001e-05 2024-09-13T13:22:24.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.214166+0000", 2024-09-13T13:22:24.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2973999999999998e-05 2024-09-13T13:22:24.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.214368+0000", 2024-09-13T13:22:24.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020158200000000001 2024-09-13T13:22:24.544 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.544 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.544 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.544 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPushReply(3.f 526/525 [PushReplyOp(3:f4c53578:::zonegroups_names.default:head)])", 2024-09-13T13:22:24.544 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:19.215308+0000", 2024-09-13T13:22:24.544 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.2642718300000002, 2024-09-13T13:22:24.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025379100000000001, 2024-09-13T13:22:24.545 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.545 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.545 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:24.545 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.215308+0000", 2024-09-13T13:22:24.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.215308+0000", 2024-09-13T13:22:24.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.215309+0000", 2024-09-13T13:22:24.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2300000000000002e-07 2024-09-13T13:22:24.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.215312+0000", 2024-09-13T13:22:24.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.315e-06 2024-09-13T13:22:24.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.215319+0000", 2024-09-13T13:22:24.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8499999999999996e-06 2024-09-13T13:22:24.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.215326+0000", 2024-09-13T13:22:24.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9600000000000003e-06 2024-09-13T13:22:24.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.215378+0000", 2024-09-13T13:22:24.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2404999999999997e-05 2024-09-13T13:22:24.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:19.215562+0000", 2024-09-13T13:22:24.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000183838 2024-09-13T13:22:24.548 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.548 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.549 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.549 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDScrubReserve(6.1e,GRANT,e:527,from:6,reservation_nonce:0)", 2024-09-13T13:22:24.549 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:21.134328+0000", 2024-09-13T13:22:24.549 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.345252092, 2024-09-13T13:22:24.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017133400000000001, 2024-09-13T13:22:24.549 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.549 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.549 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:22:24.549 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.134328+0000", 2024-09-13T13:22:24.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.134328+0000", 2024-09-13T13:22:24.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.134329+0000", 2024-09-13T13:22:24.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5199999999999995e-07 2024-09-13T13:22:24.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.134332+0000", 2024-09-13T13:22:24.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1980000000000001e-06 2024-09-13T13:22:24.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.134341+0000", 2024-09-13T13:22:24.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7970000000000006e-06 2024-09-13T13:22:24.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.134348+0000", 2024-09-13T13:22:24.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2200000000000003e-06 2024-09-13T13:22:24.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.134387+0000", 2024-09-13T13:22:24.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8753e-05 2024-09-13T13:22:24.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:24.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.134402+0000", 2024-09-13T13:22:24.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5659e-05 2024-09-13T13:22:24.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.134499+0000", 2024-09-13T13:22:24.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6755000000000005e-05 2024-09-13T13:22:24.554 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.554 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.554 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.554 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "replica_scrub(pg: 5.0,from:0'0,to:524'12,epoch:527/523,start:5:00000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T13:22:24.554 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:21.324620+0000", 2024-09-13T13:22:24.554 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.154959598, 2024-09-13T13:22:24.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029732100000000002, 2024-09-13T13:22:24.554 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.554 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.555 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:22:24.555 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.324620+0000", 2024-09-13T13:22:24.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.324620+0000", 2024-09-13T13:22:24.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.324621+0000", 2024-09-13T13:22:24.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0869999999999999e-06 2024-09-13T13:22:24.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.324631+0000", 2024-09-13T13:22:24.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6129999999999993e-06 2024-09-13T13:22:24.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.324650+0000", 2024-09-13T13:22:24.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9449999999999998e-05 2024-09-13T13:22:24.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.324662+0000", 2024-09-13T13:22:24.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2004000000000001e-05 2024-09-13T13:22:24.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.324812+0000", 2024-09-13T13:22:24.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149903 2024-09-13T13:22:24.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:24.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.324852+0000", 2024-09-13T13:22:24.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9594000000000001e-05 2024-09-13T13:22:24.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.324918+0000", 2024-09-13T13:22:24.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.567e-05 2024-09-13T13:22:24.559 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.559 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.559 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.559 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDScrubReserve(4.f,QREQUEST,e:527,from:1,reservation_nonce:3)", 2024-09-13T13:22:24.560 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:21.988747+0000", 2024-09-13T13:22:24.560 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.4908324610000001, 2024-09-13T13:22:24.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021495600000000001, 2024-09-13T13:22:24.560 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.560 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.560 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:22:24.560 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.988747+0000", 2024-09-13T13:22:24.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.988747+0000", 2024-09-13T13:22:24.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.988752+0000", 2024-09-13T13:22:24.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1049999999999997e-06 2024-09-13T13:22:24.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.988763+0000", 2024-09-13T13:22:24.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1647999999999999e-05 2024-09-13T13:22:24.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.988817+0000", 2024-09-13T13:22:24.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3999999999999998e-05 2024-09-13T13:22:24.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.988833+0000", 2024-09-13T13:22:24.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5311999999999998e-05 2024-09-13T13:22:24.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.988909+0000", 2024-09-13T13:22:24.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6298999999999996e-05 2024-09-13T13:22:24.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:24.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.988927+0000", 2024-09-13T13:22:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7742999999999999e-05 2024-09-13T13:22:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:21.988962+0000", 2024-09-13T13:22:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5849000000000002e-05 2024-09-13T13:22:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.565 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "pg_update_log_missing(5.1f epoch 527/525 rep_tid 175507 entries 527'16 (0'0) error 5:f95f44c2:::notify.0:head by client.14493.0:4448 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-13T13:22:24.565 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:22.283256+0000", 2024-09-13T13:22:24.565 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.1963239859999999, 2024-09-13T13:22:24.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00062083600000000004, 2024-09-13T13:22:24.565 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.565 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.565 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:24.565 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283256+0000", 2024-09-13T13:22:24.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283256+0000", 2024-09-13T13:22:24.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283257+0000", 2024-09-13T13:22:24.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5120000000000001e-06 2024-09-13T13:22:24.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283268+0000", 2024-09-13T13:22:24.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0933e-05 2024-09-13T13:22:24.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283292+0000", 2024-09-13T13:22:24.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3467000000000001e-05 2024-09-13T13:22:24.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283308+0000", 2024-09-13T13:22:24.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5916000000000001e-05 2024-09-13T13:22:24.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283410+0000", 2024-09-13T13:22:24.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102293 2024-09-13T13:22:24.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283877+0000", 2024-09-13T13:22:24.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00046671499999999998 2024-09-13T13:22:24.569 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.569 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.569 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.569 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "pg_update_log_missing(5.9 epoch 527/525 rep_tid 175508 entries 527'17 (0'0) error 5:93e5b521:::notify.7:head by client.14493.0:4449 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-13T13:22:24.569 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:22.283344+0000", 2024-09-13T13:22:24.569 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.196236023, 2024-09-13T13:22:24.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00065471799999999999, 2024-09-13T13:22:24.570 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.570 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.570 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:24.570 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283344+0000", 2024-09-13T13:22:24.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283344+0000", 2024-09-13T13:22:24.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283345+0000", 2024-09-13T13:22:24.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6899999999999997e-07 2024-09-13T13:22:24.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283351+0000", 2024-09-13T13:22:24.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7460000000000003e-06 2024-09-13T13:22:24.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283366+0000", 2024-09-13T13:22:24.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5163e-05 2024-09-13T13:22:24.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283377+0000", 2024-09-13T13:22:24.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0417999999999999e-05 2024-09-13T13:22:24.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283701+0000", 2024-09-13T13:22:24.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032362299999999999 2024-09-13T13:22:24.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283999+0000", 2024-09-13T13:22:24.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029809900000000002 2024-09-13T13:22:24.573 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.574 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.574 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.574 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "pg_update_log_missing(4.5 epoch 527/525 rep_tid 175509 entries 527'48 (0'0) error 4:a60f9269:lc::lua_package_allowlist:head by client.14493.0:4450 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-13T13:22:24.574 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:22.283398+0000", 2024-09-13T13:22:24.574 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.1961818040000001, 2024-09-13T13:22:24.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077713700000000001, 2024-09-13T13:22:24.574 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.574 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.574 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-09-13T13:22:24.575 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283398+0000", 2024-09-13T13:22:24.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283398+0000", 2024-09-13T13:22:24.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283399+0000", 2024-09-13T13:22:24.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:22:24.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283404+0000", 2024-09-13T13:22:24.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3480000000000003e-06 2024-09-13T13:22:24.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283749+0000", 2024-09-13T13:22:24.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034542399999999999 2024-09-13T13:22:24.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283764+0000", 2024-09-13T13:22:24.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4508999999999999e-05 2024-09-13T13:22:24.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.283849+0000", 2024-09-13T13:22:24.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4609000000000003e-05 2024-09-13T13:22:24.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.284175+0000", 2024-09-13T13:22:24.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032669799999999999 2024-09-13T13:22:24.578 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.578 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.578 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.579 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14493.0:4456 5.e 5:7759931f:::notify.3:head [watch ping cookie 94064716860672 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-09-13T13:22:24.579 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-09-13T13:22:22.498108+0000", 2024-09-13T13:22:24.579 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.981472326, 2024-09-13T13:22:24.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031789600000000002, 2024-09-13T13:22:24.579 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-09-13T13:22:24.579 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-09-13T13:22:24.579 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-09-13T13:22:24.579 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-09-13T13:22:24.579 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14493", 2024-09-13T13:22:24.579 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.52:0/2628990817", 2024-09-13T13:22:24.579 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4456 2024-09-13T13:22:24.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.580 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-09-13T13:22:24.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-09-13T13:22:24.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.498108+0000", 2024-09-13T13:22:24.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-09-13T13:22:24.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.498108+0000", 2024-09-13T13:22:24.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-09-13T13:22:24.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-09-13T13:22:24.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.498109+0000", 2024-09-13T13:22:24.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5300000000000002e-07 2024-09-13T13:22:24.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-09-13T13:22:24.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.498116+0000", 2024-09-13T13:22:24.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8480000000000001e-06 2024-09-13T13:22:24.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-09-13T13:22:24.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.498125+0000", 2024-09-13T13:22:24.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2320000000000001e-06 2024-09-13T13:22:24.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-09-13T13:22:24.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.498135+0000", 2024-09-13T13:22:24.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0597e-05 2024-09-13T13:22:24.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-09-13T13:22:24.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.498216+0000", 2024-09-13T13:22:24.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0316999999999997e-05 2024-09-13T13:22:24.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-09-13T13:22:24.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.498306+0000", 2024-09-13T13:22:24.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0921999999999994e-05 2024-09-13T13:22:24.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-09-13T13:22:24.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-09-13T13:22:24.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-09-13T13:22:24.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-09-13T13:22:22.498426+0000", 2024-09-13T13:22:24.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119027 2024-09-13T13:22:24.584 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.584 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.584 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.584 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-09-13T13:22:24.584 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-09-13T13:22:24.584 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-09-13T13:22:24.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:24 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/155187078' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:22:24.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:24 smithi052 ceph-mon[36682]: from='client.37013 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:22:24.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:24 smithi052 ceph-mon[36682]: 5.1b scrub starts 2024-09-13T13:22:24.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:24 smithi052 ceph-mon[36682]: 5.1b scrub ok 2024-09-13T13:22:24.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:24 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/155187078' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:22:24.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:24 smithi052 ceph-mon[29471]: from='client.37013 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:22:24.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:24 smithi052 ceph-mon[29471]: 5.1b scrub starts 2024-09-13T13:22:24.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:24 smithi052 ceph-mon[29471]: 5.1b scrub ok 2024-09-13T13:22:24.814 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:24 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/155187078' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:22:24.815 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:24 smithi167 ceph-mon[33178]: from='client.37013 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:22:24.815 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:24 smithi167 ceph-mon[33178]: 5.1b scrub starts 2024-09-13T13:22:24.815 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:24 smithi167 ceph-mon[33178]: 5.1b scrub ok 2024-09-13T13:22:25.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:22:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:22:25.214 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-13T13:22:25.435 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[29471]: 3.19 scrub starts 2024-09-13T13:22:25.484 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[29471]: 3.19 scrub ok 2024-09-13T13:22:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[29471]: pgmap v1084: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 45 KiB/s, 12 objects/s recovering 2024-09-13T13:22:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[29471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T13:22:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[29471]: Cluster is now healthy 2024-09-13T13:22:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[29471]: from='client.37013 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:22:25.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[29471]: osdmap e528: 8 total, 8 up, 8 in 2024-09-13T13:22:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[29471]: 4.5 scrub starts 2024-09-13T13:22:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[29471]: 4.5 scrub ok 2024-09-13T13:22:25.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:25.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[36682]: 3.19 scrub starts 2024-09-13T13:22:25.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[36682]: 3.19 scrub ok 2024-09-13T13:22:25.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[36682]: pgmap v1084: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 45 KiB/s, 12 objects/s recovering 2024-09-13T13:22:25.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[36682]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T13:22:25.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[36682]: Cluster is now healthy 2024-09-13T13:22:25.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[36682]: from='client.37013 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:22:25.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[36682]: osdmap e528: 8 total, 8 up, 8 in 2024-09-13T13:22:25.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[36682]: 4.5 scrub starts 2024-09-13T13:22:25.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[36682]: 4.5 scrub ok 2024-09-13T13:22:25.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:25 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:25 smithi167 ceph-mon[33178]: 3.19 scrub starts 2024-09-13T13:22:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:25 smithi167 ceph-mon[33178]: 3.19 scrub ok 2024-09-13T13:22:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:25 smithi167 ceph-mon[33178]: pgmap v1084: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 45 KiB/s, 12 objects/s recovering 2024-09-13T13:22:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:25 smithi167 ceph-mon[33178]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T13:22:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:25 smithi167 ceph-mon[33178]: Cluster is now healthy 2024-09-13T13:22:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:25 smithi167 ceph-mon[33178]: from='client.37013 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:22:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:25 smithi167 ceph-mon[33178]: osdmap e528: 8 total, 8 up, 8 in 2024-09-13T13:22:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:25 smithi167 ceph-mon[33178]: 4.5 scrub starts 2024-09-13T13:22:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:25 smithi167 ceph-mon[33178]: 4.5 scrub ok 2024-09-13T13:22:25.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:25 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:26.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:22:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:22:26.436 INFO:teuthology.orchestra.run.smithi052.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-13T13:22:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[29471]: 4.2 scrub starts 2024-09-13T13:22:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[29471]: 4.2 scrub ok 2024-09-13T13:22:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[29471]: 5.1a scrub starts 2024-09-13T13:22:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[29471]: 5.1a scrub ok 2024-09-13T13:22:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[29471]: 1.0 scrub starts 2024-09-13T13:22:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[29471]: 1.0 scrub ok 2024-09-13T13:22:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[29471]: 6.7 scrub starts 2024-09-13T13:22:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[29471]: 6.7 scrub ok 2024-09-13T13:22:26.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[29471]: 5.a scrub starts 2024-09-13T13:22:26.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[29471]: 5.a scrub ok 2024-09-13T13:22:26.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4187602243' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:22:26.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[36682]: 4.2 scrub starts 2024-09-13T13:22:26.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[36682]: 4.2 scrub ok 2024-09-13T13:22:26.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[36682]: 5.1a scrub starts 2024-09-13T13:22:26.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[36682]: 5.1a scrub ok 2024-09-13T13:22:26.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[36682]: 1.0 scrub starts 2024-09-13T13:22:26.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[36682]: 1.0 scrub ok 2024-09-13T13:22:26.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[36682]: 6.7 scrub starts 2024-09-13T13:22:26.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[36682]: 6.7 scrub ok 2024-09-13T13:22:26.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[36682]: 5.a scrub starts 2024-09-13T13:22:26.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[36682]: 5.a scrub ok 2024-09-13T13:22:26.741 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:26 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4187602243' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:22:26.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:26 smithi167 ceph-mon[33178]: 4.2 scrub starts 2024-09-13T13:22:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:26 smithi167 ceph-mon[33178]: 4.2 scrub ok 2024-09-13T13:22:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:26 smithi167 ceph-mon[33178]: 5.1a scrub starts 2024-09-13T13:22:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:26 smithi167 ceph-mon[33178]: 5.1a scrub ok 2024-09-13T13:22:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:26 smithi167 ceph-mon[33178]: 1.0 scrub starts 2024-09-13T13:22:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:26 smithi167 ceph-mon[33178]: 1.0 scrub ok 2024-09-13T13:22:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:26 smithi167 ceph-mon[33178]: 6.7 scrub starts 2024-09-13T13:22:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:26 smithi167 ceph-mon[33178]: 6.7 scrub ok 2024-09-13T13:22:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:26 smithi167 ceph-mon[33178]: 5.a scrub starts 2024-09-13T13:22:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:26 smithi167 ceph-mon[33178]: 5.a scrub ok 2024-09-13T13:22:26.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:26 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4187602243' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T13:22:27.072 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T13:22:27.340 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:27.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[29471]: pgmap v1086: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 39 KiB/s, 10 objects/s recovering 2024-09-13T13:22:27.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4187602243' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:22:27.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[29471]: osdmap e529: 8 total, 8 up, 8 in 2024-09-13T13:22:27.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[29471]: 1.8 scrub starts 2024-09-13T13:22:27.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[29471]: 1.8 scrub ok 2024-09-13T13:22:27.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[29471]: 6.2 scrub starts 2024-09-13T13:22:27.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[29471]: 6.2 scrub ok 2024-09-13T13:22:27.734 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[36682]: pgmap v1086: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 39 KiB/s, 10 objects/s recovering 2024-09-13T13:22:27.734 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4187602243' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:22:27.734 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[36682]: osdmap e529: 8 total, 8 up, 8 in 2024-09-13T13:22:27.734 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[36682]: 1.8 scrub starts 2024-09-13T13:22:27.734 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[36682]: 1.8 scrub ok 2024-09-13T13:22:27.734 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[36682]: 6.2 scrub starts 2024-09-13T13:22:27.734 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:27 smithi052 ceph-mon[36682]: 6.2 scrub ok 2024-09-13T13:22:27.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:27 smithi167 ceph-mon[33178]: pgmap v1086: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 39 KiB/s, 10 objects/s recovering 2024-09-13T13:22:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:27 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4187602243' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T13:22:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:27 smithi167 ceph-mon[33178]: osdmap e529: 8 total, 8 up, 8 in 2024-09-13T13:22:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:27 smithi167 ceph-mon[33178]: 1.8 scrub starts 2024-09-13T13:22:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:27 smithi167 ceph-mon[33178]: 1.8 scrub ok 2024-09-13T13:22:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:27 smithi167 ceph-mon[33178]: 6.2 scrub starts 2024-09-13T13:22:27.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:27 smithi167 ceph-mon[33178]: 6.2 scrub ok 2024-09-13T13:22:28.444 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T13:22:28.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[36682]: 5.5 scrub starts 2024-09-13T13:22:28.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[36682]: 5.5 scrub ok 2024-09-13T13:22:28.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[36682]: 5.11 scrub starts 2024-09-13T13:22:28.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[36682]: 5.11 scrub ok 2024-09-13T13:22:28.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2803309943' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:22:28.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[36682]: 6.12 scrub starts 2024-09-13T13:22:28.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[36682]: 6.12 scrub ok 2024-09-13T13:22:28.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[29471]: 5.5 scrub starts 2024-09-13T13:22:28.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[29471]: 5.5 scrub ok 2024-09-13T13:22:28.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[29471]: 5.11 scrub starts 2024-09-13T13:22:28.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[29471]: 5.11 scrub ok 2024-09-13T13:22:28.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2803309943' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:22:28.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[29471]: 6.12 scrub starts 2024-09-13T13:22:28.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:28 smithi052 ceph-mon[29471]: 6.12 scrub ok 2024-09-13T13:22:28.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:28 smithi167 ceph-mon[33178]: 5.5 scrub starts 2024-09-13T13:22:28.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:28 smithi167 ceph-mon[33178]: 5.5 scrub ok 2024-09-13T13:22:28.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:28 smithi167 ceph-mon[33178]: 5.11 scrub starts 2024-09-13T13:22:28.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:28 smithi167 ceph-mon[33178]: 5.11 scrub ok 2024-09-13T13:22:28.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:28 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2803309943' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:22:28.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:28 smithi167 ceph-mon[33178]: 6.12 scrub starts 2024-09-13T13:22:28.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:28 smithi167 ceph-mon[33178]: 6.12 scrub ok 2024-09-13T13:22:29.107 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-13T13:22:29.376 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:29.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:29 smithi167 ceph-mon[33178]: 5.19 scrub starts 2024-09-13T13:22:29.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:29 smithi167 ceph-mon[33178]: 5.19 scrub ok 2024-09-13T13:22:29.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:29 smithi167 ceph-mon[33178]: pgmap v1088: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 665 B/s rd, 0 op/s; 42 KiB/s, 7 objects/s recovering 2024-09-13T13:22:29.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:29 smithi167 ceph-mon[33178]: 3.b scrub starts 2024-09-13T13:22:29.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:29 smithi167 ceph-mon[33178]: 3.b scrub ok 2024-09-13T13:22:29.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:29 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2803309943' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:22:29.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:29 smithi167 ceph-mon[33178]: osdmap e530: 8 total, 8 up, 8 in 2024-09-13T13:22:29.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:29 smithi167 ceph-mon[33178]: 4.8 scrub starts 2024-09-13T13:22:29.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:29 smithi167 ceph-mon[33178]: 4.8 scrub ok 2024-09-13T13:22:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[36682]: 5.19 scrub starts 2024-09-13T13:22:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[36682]: 5.19 scrub ok 2024-09-13T13:22:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[36682]: pgmap v1088: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 665 B/s rd, 0 op/s; 42 KiB/s, 7 objects/s recovering 2024-09-13T13:22:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[36682]: 3.b scrub starts 2024-09-13T13:22:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[36682]: 3.b scrub ok 2024-09-13T13:22:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2803309943' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:22:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[36682]: osdmap e530: 8 total, 8 up, 8 in 2024-09-13T13:22:29.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[36682]: 4.8 scrub starts 2024-09-13T13:22:29.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[36682]: 4.8 scrub ok 2024-09-13T13:22:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[29471]: 5.19 scrub starts 2024-09-13T13:22:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[29471]: 5.19 scrub ok 2024-09-13T13:22:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[29471]: pgmap v1088: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 665 B/s rd, 0 op/s; 42 KiB/s, 7 objects/s recovering 2024-09-13T13:22:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[29471]: 3.b scrub starts 2024-09-13T13:22:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[29471]: 3.b scrub ok 2024-09-13T13:22:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2803309943' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:22:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[29471]: osdmap e530: 8 total, 8 up, 8 in 2024-09-13T13:22:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[29471]: 4.8 scrub starts 2024-09-13T13:22:29.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:29 smithi052 ceph-mon[29471]: 4.8 scrub ok 2024-09-13T13:22:30.460 INFO:teuthology.orchestra.run.smithi052.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-13T13:22:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[36682]: 5.1f scrub starts 2024-09-13T13:22:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[36682]: 5.1f scrub ok 2024-09-13T13:22:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1152027209' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:22:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[36682]: from='client.37031 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:22:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[36682]: 5.1 scrub starts 2024-09-13T13:22:30.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[36682]: 5.1 scrub ok 2024-09-13T13:22:30.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[36682]: 3.8 scrub starts 2024-09-13T13:22:30.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[36682]: 3.8 scrub ok 2024-09-13T13:22:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[29471]: 5.1f scrub starts 2024-09-13T13:22:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[29471]: 5.1f scrub ok 2024-09-13T13:22:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1152027209' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:22:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[29471]: from='client.37031 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:22:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[29471]: 5.1 scrub starts 2024-09-13T13:22:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[29471]: 5.1 scrub ok 2024-09-13T13:22:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[29471]: 3.8 scrub starts 2024-09-13T13:22:30.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:30 smithi052 ceph-mon[29471]: 3.8 scrub ok 2024-09-13T13:22:30.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:30 smithi167 ceph-mon[33178]: 5.1f scrub starts 2024-09-13T13:22:30.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:30 smithi167 ceph-mon[33178]: 5.1f scrub ok 2024-09-13T13:22:30.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:30 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1152027209' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:22:30.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:30 smithi167 ceph-mon[33178]: from='client.37031 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T13:22:30.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:30 smithi167 ceph-mon[33178]: 5.1 scrub starts 2024-09-13T13:22:30.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:30 smithi167 ceph-mon[33178]: 5.1 scrub ok 2024-09-13T13:22:30.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:30 smithi167 ceph-mon[33178]: 3.8 scrub starts 2024-09-13T13:22:30.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:30 smithi167 ceph-mon[33178]: 3.8 scrub ok 2024-09-13T13:22:31.232 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T13:22:31.499 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:31.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:31 smithi052 ceph-mon[29471]: pgmap v1090: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T13:22:31.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:31 smithi052 ceph-mon[29471]: from='client.37031 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:22:31.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:31 smithi052 ceph-mon[29471]: osdmap e531: 8 total, 8 up, 8 in 2024-09-13T13:22:31.728 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:31 smithi052 ceph-mon[36682]: pgmap v1090: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T13:22:31.728 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:31 smithi052 ceph-mon[36682]: from='client.37031 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:22:31.728 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:31 smithi052 ceph-mon[36682]: osdmap e531: 8 total, 8 up, 8 in 2024-09-13T13:22:31.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:31 smithi167 ceph-mon[33178]: pgmap v1090: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-13T13:22:31.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:31 smithi167 ceph-mon[33178]: from='client.37031 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T13:22:31.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:31 smithi167 ceph-mon[33178]: osdmap e531: 8 total, 8 up, 8 in 2024-09-13T13:22:32.462 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T13:22:32.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:32 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1367546465' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:22:32.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:32 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1367546465' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:22:32.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:32 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1367546465' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:22:33.155 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-13T13:22:33.421 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:33 smithi052 ceph-mon[36682]: pgmap v1092: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:22:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:33 smithi052 ceph-mon[36682]: 4.b scrub starts 2024-09-13T13:22:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:33 smithi052 ceph-mon[36682]: 4.b scrub ok 2024-09-13T13:22:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:33 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1367546465' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:22:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:33 smithi052 ceph-mon[36682]: osdmap e532: 8 total, 8 up, 8 in 2024-09-13T13:22:33.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:33 smithi052 ceph-mon[36682]: osdmap e533: 8 total, 8 up, 8 in 2024-09-13T13:22:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:33 smithi052 ceph-mon[29471]: pgmap v1092: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:22:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:33 smithi052 ceph-mon[29471]: 4.b scrub starts 2024-09-13T13:22:33.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:33 smithi052 ceph-mon[29471]: 4.b scrub ok 2024-09-13T13:22:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:33 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1367546465' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:22:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:33 smithi052 ceph-mon[29471]: osdmap e532: 8 total, 8 up, 8 in 2024-09-13T13:22:33.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:33 smithi052 ceph-mon[29471]: osdmap e533: 8 total, 8 up, 8 in 2024-09-13T13:22:33.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:33 smithi167 ceph-mon[33178]: pgmap v1092: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:22:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:33 smithi167 ceph-mon[33178]: 4.b scrub starts 2024-09-13T13:22:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:33 smithi167 ceph-mon[33178]: 4.b scrub ok 2024-09-13T13:22:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:33 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1367546465' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:22:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:33 smithi167 ceph-mon[33178]: osdmap e532: 8 total, 8 up, 8 in 2024-09-13T13:22:33.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:33 smithi167 ceph-mon[33178]: osdmap e533: 8 total, 8 up, 8 in 2024-09-13T13:22:34.468 INFO:teuthology.orchestra.run.smithi052.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-13T13:22:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:34 smithi052 ceph-mon[36682]: 4.1f scrub starts 2024-09-13T13:22:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:34 smithi052 ceph-mon[36682]: 4.1f scrub ok 2024-09-13T13:22:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:34 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/619511799' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:22:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:34 smithi052 ceph-mon[36682]: from='client.37043 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:22:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:34 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:22:34.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:34 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:22:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:34 smithi052 ceph-mon[29471]: 4.1f scrub starts 2024-09-13T13:22:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:34 smithi052 ceph-mon[29471]: 4.1f scrub ok 2024-09-13T13:22:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:34 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/619511799' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:22:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:34 smithi052 ceph-mon[29471]: from='client.37043 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:22:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:34 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:22:34.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:34 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:22:34.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:34 smithi167 ceph-mon[33178]: 4.1f scrub starts 2024-09-13T13:22:34.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:34 smithi167 ceph-mon[33178]: 4.1f scrub ok 2024-09-13T13:22:34.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:34 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/619511799' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:22:34.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:34 smithi167 ceph-mon[33178]: from='client.37043 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T13:22:34.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:34 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:22:34.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:34 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:22:35.175 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T13:22:35.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:22:34 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:22:35.441 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:35.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[29471]: pgmap v1095: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-09-13T13:22:35.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[29471]: 3.6 scrub starts 2024-09-13T13:22:35.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[29471]: 3.6 scrub ok 2024-09-13T13:22:35.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[29471]: 4.14 scrub starts 2024-09-13T13:22:35.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[29471]: 4.14 scrub ok 2024-09-13T13:22:35.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[29471]: from='client.37043 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:22:35.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[29471]: osdmap e534: 8 total, 8 up, 8 in 2024-09-13T13:22:35.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:35.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[29471]: 3.1c scrub starts 2024-09-13T13:22:35.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[29471]: 3.1c scrub ok 2024-09-13T13:22:35.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[29471]: 4.18 scrub starts 2024-09-13T13:22:35.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[29471]: 4.18 scrub ok 2024-09-13T13:22:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[36682]: pgmap v1095: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-09-13T13:22:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[36682]: 3.6 scrub starts 2024-09-13T13:22:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[36682]: 3.6 scrub ok 2024-09-13T13:22:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[36682]: 4.14 scrub starts 2024-09-13T13:22:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[36682]: 4.14 scrub ok 2024-09-13T13:22:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[36682]: from='client.37043 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:22:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[36682]: osdmap e534: 8 total, 8 up, 8 in 2024-09-13T13:22:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[36682]: 3.1c scrub starts 2024-09-13T13:22:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[36682]: 3.1c scrub ok 2024-09-13T13:22:35.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[36682]: 4.18 scrub starts 2024-09-13T13:22:35.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:35 smithi052 ceph-mon[36682]: 4.18 scrub ok 2024-09-13T13:22:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:35 smithi167 ceph-mon[33178]: pgmap v1095: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-09-13T13:22:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:35 smithi167 ceph-mon[33178]: 3.6 scrub starts 2024-09-13T13:22:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:35 smithi167 ceph-mon[33178]: 3.6 scrub ok 2024-09-13T13:22:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:35 smithi167 ceph-mon[33178]: 4.14 scrub starts 2024-09-13T13:22:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:35 smithi167 ceph-mon[33178]: 4.14 scrub ok 2024-09-13T13:22:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:35 smithi167 ceph-mon[33178]: from='client.37043 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T13:22:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:35 smithi167 ceph-mon[33178]: osdmap e534: 8 total, 8 up, 8 in 2024-09-13T13:22:35.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:35 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:35.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:35 smithi167 ceph-mon[33178]: 3.1c scrub starts 2024-09-13T13:22:35.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:35 smithi167 ceph-mon[33178]: 3.1c scrub ok 2024-09-13T13:22:35.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:35 smithi167 ceph-mon[33178]: 4.18 scrub starts 2024-09-13T13:22:35.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:35 smithi167 ceph-mon[33178]: 4.18 scrub ok 2024-09-13T13:22:36.481 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T13:22:36.481 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:22:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:22:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[36682]: 5.1d scrub starts 2024-09-13T13:22:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[36682]: 5.1d scrub ok 2024-09-13T13:22:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[36682]: 5.9 scrub starts 2024-09-13T13:22:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[36682]: 5.9 scrub ok 2024-09-13T13:22:36.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[36682]: osdmap e535: 8 total, 8 up, 8 in 2024-09-13T13:22:36.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[36682]: 4.e scrub starts 2024-09-13T13:22:36.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[36682]: 4.e scrub ok 2024-09-13T13:22:36.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3795341780' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:22:36.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[36682]: from='client.37049 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:22:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[29471]: 5.1d scrub starts 2024-09-13T13:22:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[29471]: 5.1d scrub ok 2024-09-13T13:22:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[29471]: 5.9 scrub starts 2024-09-13T13:22:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[29471]: 5.9 scrub ok 2024-09-13T13:22:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[29471]: osdmap e535: 8 total, 8 up, 8 in 2024-09-13T13:22:36.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[29471]: 4.e scrub starts 2024-09-13T13:22:36.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[29471]: 4.e scrub ok 2024-09-13T13:22:36.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3795341780' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:22:36.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:36 smithi052 ceph-mon[29471]: from='client.37049 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:22:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:36 smithi167 ceph-mon[33178]: 5.1d scrub starts 2024-09-13T13:22:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:36 smithi167 ceph-mon[33178]: 5.1d scrub ok 2024-09-13T13:22:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:36 smithi167 ceph-mon[33178]: 5.9 scrub starts 2024-09-13T13:22:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:36 smithi167 ceph-mon[33178]: 5.9 scrub ok 2024-09-13T13:22:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:36 smithi167 ceph-mon[33178]: osdmap e535: 8 total, 8 up, 8 in 2024-09-13T13:22:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:36 smithi167 ceph-mon[33178]: 4.e scrub starts 2024-09-13T13:22:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:36 smithi167 ceph-mon[33178]: 4.e scrub ok 2024-09-13T13:22:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:36 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3795341780' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:22:36.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:36 smithi167 ceph-mon[33178]: from='client.37049 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:22:37.034 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-13T13:22:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:37.675 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[36682]: 5.16 scrub starts 2024-09-13T13:22:37.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[36682]: 5.16 scrub ok 2024-09-13T13:22:37.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[36682]: pgmap v1098: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:22:37.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[36682]: 4.4 scrub starts 2024-09-13T13:22:37.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[36682]: 4.4 scrub ok 2024-09-13T13:22:37.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[36682]: from='client.37049 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:22:37.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[36682]: osdmap e536: 8 total, 8 up, 8 in 2024-09-13T13:22:37.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[36682]: 3.5 scrub starts 2024-09-13T13:22:37.676 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[36682]: 3.5 scrub ok 2024-09-13T13:22:37.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[29471]: 5.16 scrub starts 2024-09-13T13:22:37.676 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[29471]: 5.16 scrub ok 2024-09-13T13:22:37.677 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[29471]: pgmap v1098: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:22:37.677 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[29471]: 4.4 scrub starts 2024-09-13T13:22:37.677 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[29471]: 4.4 scrub ok 2024-09-13T13:22:37.677 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[29471]: from='client.37049 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:22:37.677 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[29471]: osdmap e536: 8 total, 8 up, 8 in 2024-09-13T13:22:37.677 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[29471]: 3.5 scrub starts 2024-09-13T13:22:37.677 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:37 smithi052 ceph-mon[29471]: 3.5 scrub ok 2024-09-13T13:22:37.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:37 smithi167 ceph-mon[33178]: 5.16 scrub starts 2024-09-13T13:22:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:37 smithi167 ceph-mon[33178]: 5.16 scrub ok 2024-09-13T13:22:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:37 smithi167 ceph-mon[33178]: pgmap v1098: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:22:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:37 smithi167 ceph-mon[33178]: 4.4 scrub starts 2024-09-13T13:22:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:37 smithi167 ceph-mon[33178]: 4.4 scrub ok 2024-09-13T13:22:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:37 smithi167 ceph-mon[33178]: from='client.37049 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:22:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:37 smithi167 ceph-mon[33178]: osdmap e536: 8 total, 8 up, 8 in 2024-09-13T13:22:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:37 smithi167 ceph-mon[33178]: 3.5 scrub starts 2024-09-13T13:22:37.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:37 smithi167 ceph-mon[33178]: 3.5 scrub ok 2024-09-13T13:22:38.162 INFO:teuthology.orchestra.run.smithi052.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-13T13:22:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[29471]: 6.b scrub starts 2024-09-13T13:22:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[29471]: 6.b scrub ok 2024-09-13T13:22:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[29471]: 6.5 scrub starts 2024-09-13T13:22:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[29471]: 6.5 scrub ok 2024-09-13T13:22:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3952348343' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:22:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3952348343' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:22:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[29471]: 5.1c scrub starts 2024-09-13T13:22:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[29471]: 5.1c scrub ok 2024-09-13T13:22:38.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[29471]: osdmap e537: 8 total, 8 up, 8 in 2024-09-13T13:22:38.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[36682]: 6.b scrub starts 2024-09-13T13:22:38.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[36682]: 6.b scrub ok 2024-09-13T13:22:38.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[36682]: 6.5 scrub starts 2024-09-13T13:22:38.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[36682]: 6.5 scrub ok 2024-09-13T13:22:38.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3952348343' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:22:38.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3952348343' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:22:38.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[36682]: 5.1c scrub starts 2024-09-13T13:22:38.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[36682]: 5.1c scrub ok 2024-09-13T13:22:38.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:38 smithi052 ceph-mon[36682]: osdmap e537: 8 total, 8 up, 8 in 2024-09-13T13:22:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:38 smithi167 ceph-mon[33178]: 6.b scrub starts 2024-09-13T13:22:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:38 smithi167 ceph-mon[33178]: 6.b scrub ok 2024-09-13T13:22:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:38 smithi167 ceph-mon[33178]: 6.5 scrub starts 2024-09-13T13:22:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:38 smithi167 ceph-mon[33178]: 6.5 scrub ok 2024-09-13T13:22:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:38 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3952348343' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T13:22:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:38 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3952348343' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T13:22:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:38 smithi167 ceph-mon[33178]: 5.1c scrub starts 2024-09-13T13:22:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:38 smithi167 ceph-mon[33178]: 5.1c scrub ok 2024-09-13T13:22:38.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:38 smithi167 ceph-mon[33178]: osdmap e537: 8 total, 8 up, 8 in 2024-09-13T13:22:38.900 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T13:22:39.162 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:39.577 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:39 smithi052 ceph-mon[29471]: pgmap v1100: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T13:22:39.577 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:39 smithi052 ceph-mon[36682]: pgmap v1100: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T13:22:39.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:39 smithi167 ceph-mon[33178]: pgmap v1100: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-13T13:22:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T13:22:40.789 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:40 smithi052 ceph-mon[36682]: 3.16 scrub starts 2024-09-13T13:22:40.789 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:40 smithi052 ceph-mon[36682]: 3.16 scrub ok 2024-09-13T13:22:40.789 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:40 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/475300121' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:22:40.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:40 smithi052 ceph-mon[29471]: 3.16 scrub starts 2024-09-13T13:22:40.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:40 smithi052 ceph-mon[29471]: 3.16 scrub ok 2024-09-13T13:22:40.790 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:40 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/475300121' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:22:40.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:40 smithi167 ceph-mon[33178]: 3.16 scrub starts 2024-09-13T13:22:40.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:40 smithi167 ceph-mon[33178]: 3.16 scrub ok 2024-09-13T13:22:40.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:40 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/475300121' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:22:41.234 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-13T13:22:41.502 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:41.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:41 smithi167 ceph-mon[33178]: pgmap v1102: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-09-13T13:22:41.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:41 smithi167 ceph-mon[33178]: 6.f scrub starts 2024-09-13T13:22:41.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:41 smithi167 ceph-mon[33178]: 6.f scrub ok 2024-09-13T13:22:41.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:41 smithi167 ceph-mon[33178]: 3.1 scrub starts 2024-09-13T13:22:41.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:41 smithi167 ceph-mon[33178]: 3.1 scrub ok 2024-09-13T13:22:41.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:41 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/475300121' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:22:41.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:41 smithi167 ceph-mon[33178]: osdmap e538: 8 total, 8 up, 8 in 2024-09-13T13:22:41.842 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[36682]: pgmap v1102: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-09-13T13:22:41.842 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[36682]: 6.f scrub starts 2024-09-13T13:22:41.842 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[36682]: 6.f scrub ok 2024-09-13T13:22:41.842 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[36682]: 3.1 scrub starts 2024-09-13T13:22:41.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[36682]: 3.1 scrub ok 2024-09-13T13:22:41.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/475300121' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:22:41.843 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[36682]: osdmap e538: 8 total, 8 up, 8 in 2024-09-13T13:22:41.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[29471]: pgmap v1102: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-09-13T13:22:41.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[29471]: 6.f scrub starts 2024-09-13T13:22:41.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[29471]: 6.f scrub ok 2024-09-13T13:22:41.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[29471]: 3.1 scrub starts 2024-09-13T13:22:41.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[29471]: 3.1 scrub ok 2024-09-13T13:22:41.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/475300121' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:22:41.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:41 smithi052 ceph-mon[29471]: osdmap e538: 8 total, 8 up, 8 in 2024-09-13T13:22:42.510 INFO:teuthology.orchestra.run.smithi052.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T13:22:42.795 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:42 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/119693935' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:22:42.796 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:42 smithi052 ceph-mon[36682]: from='client.37067 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:22:42.796 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:42 smithi052 ceph-mon[36682]: 3.1e scrub starts 2024-09-13T13:22:42.796 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:42 smithi052 ceph-mon[36682]: 3.1e scrub ok 2024-09-13T13:22:42.796 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:42 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/119693935' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:22:42.796 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:42 smithi052 ceph-mon[29471]: from='client.37067 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:22:42.796 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:42 smithi052 ceph-mon[29471]: 3.1e scrub starts 2024-09-13T13:22:42.796 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:42 smithi052 ceph-mon[29471]: 3.1e scrub ok 2024-09-13T13:22:42.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:42 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/119693935' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:22:42.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:42 smithi167 ceph-mon[33178]: from='client.37067 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T13:22:42.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:42 smithi167 ceph-mon[33178]: 3.1e scrub starts 2024-09-13T13:22:42.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:42 smithi167 ceph-mon[33178]: 3.1e scrub ok 2024-09-13T13:22:43.236 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T13:22:43.496 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:43.759 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:43 smithi052 ceph-mon[36682]: pgmap v1104: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:22:43.760 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:43 smithi052 ceph-mon[36682]: from='client.37067 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:22:43.760 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:43 smithi052 ceph-mon[36682]: osdmap e539: 8 total, 8 up, 8 in 2024-09-13T13:22:43.760 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:43 smithi052 ceph-mon[36682]: 5.17 scrub starts 2024-09-13T13:22:43.760 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:43 smithi052 ceph-mon[36682]: 5.17 scrub ok 2024-09-13T13:22:43.760 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:43 smithi052 ceph-mon[29471]: pgmap v1104: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:22:43.760 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:43 smithi052 ceph-mon[29471]: from='client.37067 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:22:43.760 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:43 smithi052 ceph-mon[29471]: osdmap e539: 8 total, 8 up, 8 in 2024-09-13T13:22:43.760 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:43 smithi052 ceph-mon[29471]: 5.17 scrub starts 2024-09-13T13:22:43.760 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:43 smithi052 ceph-mon[29471]: 5.17 scrub ok 2024-09-13T13:22:43.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:43 smithi167 ceph-mon[33178]: pgmap v1104: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:22:43.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:43 smithi167 ceph-mon[33178]: from='client.37067 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T13:22:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:43 smithi167 ceph-mon[33178]: osdmap e539: 8 total, 8 up, 8 in 2024-09-13T13:22:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:43 smithi167 ceph-mon[33178]: 5.17 scrub starts 2024-09-13T13:22:43.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:43 smithi167 ceph-mon[33178]: 5.17 scrub ok 2024-09-13T13:22:44.518 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T13:22:44.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:44 smithi167 ceph-mon[33178]: 5.3 scrub starts 2024-09-13T13:22:44.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:44 smithi167 ceph-mon[33178]: 5.3 scrub ok 2024-09-13T13:22:44.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:44 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:22:44.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:44 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:22:44.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:44 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3105383446' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:22:44.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:44 smithi167 ceph-mon[33178]: from='client.37073 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:22:44.830 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:44 smithi052 ceph-mon[36682]: 5.3 scrub starts 2024-09-13T13:22:44.830 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:44 smithi052 ceph-mon[36682]: 5.3 scrub ok 2024-09-13T13:22:44.830 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:44 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:22:44.830 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:44 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:22:44.830 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:44 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3105383446' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:22:44.830 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:44 smithi052 ceph-mon[36682]: from='client.37073 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:22:44.830 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:44 smithi052 ceph-mon[29471]: 5.3 scrub starts 2024-09-13T13:22:44.830 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:44 smithi052 ceph-mon[29471]: 5.3 scrub ok 2024-09-13T13:22:44.831 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:44 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:22:44.831 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:44 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:22:44.831 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:44 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3105383446' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:22:44.831 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:44 smithi052 ceph-mon[29471]: from='client.37073 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:22:45.302 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-13T13:22:45.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:22:44 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:22:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:45.704 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[36682]: pgmap v1106: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:22:45.705 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[36682]: 6.17 scrub starts 2024-09-13T13:22:45.705 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[36682]: 6.17 scrub ok 2024-09-13T13:22:45.705 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[36682]: from='client.37073 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:22:45.705 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[36682]: osdmap e540: 8 total, 8 up, 8 in 2024-09-13T13:22:45.705 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:45.705 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:22:45.705 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:22:45.705 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:22:45.705 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:22:45.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[29471]: pgmap v1106: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:22:45.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[29471]: 6.17 scrub starts 2024-09-13T13:22:45.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[29471]: 6.17 scrub ok 2024-09-13T13:22:45.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[29471]: from='client.37073 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:22:45.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[29471]: osdmap e540: 8 total, 8 up, 8 in 2024-09-13T13:22:45.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:45.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:22:45.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:22:45.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:22:45.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:45 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:22:45.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:45 smithi167 ceph-mon[33178]: pgmap v1106: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:22:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:45 smithi167 ceph-mon[33178]: 6.17 scrub starts 2024-09-13T13:22:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:45 smithi167 ceph-mon[33178]: 6.17 scrub ok 2024-09-13T13:22:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:45 smithi167 ceph-mon[33178]: from='client.37073 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:22:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:45 smithi167 ceph-mon[33178]: osdmap e540: 8 total, 8 up, 8 in 2024-09-13T13:22:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:45 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:45 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:22:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:45 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:22:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:45 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:22:45.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:45 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:22:46.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:22:46 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:22:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:22:46.526 INFO:teuthology.orchestra.run.smithi052.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-13T13:22:46.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:46 smithi167 ceph-mon[33178]: 3.1d scrub starts 2024-09-13T13:22:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:46 smithi167 ceph-mon[33178]: 3.1d scrub ok 2024-09-13T13:22:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:46 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2661854673' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:22:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:46 smithi167 ceph-mon[33178]: from='client.37079 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:22:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:46 smithi167 ceph-mon[33178]: 6.6 scrub starts 2024-09-13T13:22:46.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:46 smithi167 ceph-mon[33178]: 6.6 scrub ok 2024-09-13T13:22:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:46 smithi052 ceph-mon[36682]: 3.1d scrub starts 2024-09-13T13:22:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:46 smithi052 ceph-mon[36682]: 3.1d scrub ok 2024-09-13T13:22:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:46 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2661854673' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:22:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:46 smithi052 ceph-mon[36682]: from='client.37079 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:22:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:46 smithi052 ceph-mon[36682]: 6.6 scrub starts 2024-09-13T13:22:46.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:46 smithi052 ceph-mon[36682]: 6.6 scrub ok 2024-09-13T13:22:46.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:46 smithi052 ceph-mon[29471]: 3.1d scrub starts 2024-09-13T13:22:46.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:46 smithi052 ceph-mon[29471]: 3.1d scrub ok 2024-09-13T13:22:46.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:46 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2661854673' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:22:46.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:46 smithi052 ceph-mon[29471]: from='client.37079 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T13:22:46.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:46 smithi052 ceph-mon[29471]: 6.6 scrub starts 2024-09-13T13:22:46.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:46 smithi052 ceph-mon[29471]: 6.6 scrub ok 2024-09-13T13:22:47.445 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T13:22:47.713 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:47.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[36682]: pgmap v1108: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:22:47.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[36682]: 4.3 scrub starts 2024-09-13T13:22:47.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[36682]: 4.3 scrub ok 2024-09-13T13:22:47.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[36682]: 4.13 scrub starts 2024-09-13T13:22:47.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[36682]: 4.13 scrub ok 2024-09-13T13:22:47.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[36682]: from='client.37079 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:22:47.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[36682]: osdmap e541: 8 total, 8 up, 8 in 2024-09-13T13:22:47.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[29471]: pgmap v1108: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:22:47.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[29471]: 4.3 scrub starts 2024-09-13T13:22:47.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[29471]: 4.3 scrub ok 2024-09-13T13:22:47.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[29471]: 4.13 scrub starts 2024-09-13T13:22:47.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[29471]: 4.13 scrub ok 2024-09-13T13:22:47.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[29471]: from='client.37079 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:22:47.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:47 smithi052 ceph-mon[29471]: osdmap e541: 8 total, 8 up, 8 in 2024-09-13T13:22:47.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:47 smithi167 ceph-mon[33178]: pgmap v1108: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-13T13:22:47.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:47 smithi167 ceph-mon[33178]: 4.3 scrub starts 2024-09-13T13:22:47.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:47 smithi167 ceph-mon[33178]: 4.3 scrub ok 2024-09-13T13:22:47.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:47 smithi167 ceph-mon[33178]: 4.13 scrub starts 2024-09-13T13:22:47.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:47 smithi167 ceph-mon[33178]: 4.13 scrub ok 2024-09-13T13:22:47.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:47 smithi167 ceph-mon[33178]: from='client.37079 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T13:22:47.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:47 smithi167 ceph-mon[33178]: osdmap e541: 8 total, 8 up, 8 in 2024-09-13T13:22:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:48 smithi167 ceph-mon[33178]: 6.d scrub starts 2024-09-13T13:22:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:48 smithi167 ceph-mon[33178]: 6.d scrub ok 2024-09-13T13:22:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:48 smithi167 ceph-mon[33178]: 6.10 scrub starts 2024-09-13T13:22:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:48 smithi167 ceph-mon[33178]: 6.10 scrub ok 2024-09-13T13:22:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:48 smithi167 ceph-mon[33178]: 3.1a scrub starts 2024-09-13T13:22:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:48 smithi167 ceph-mon[33178]: 3.1a scrub ok 2024-09-13T13:22:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:48 smithi167 ceph-mon[33178]: osdmap e542: 8 total, 8 up, 8 in 2024-09-13T13:22:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:48 smithi167 ceph-mon[33178]: osdmap e543: 8 total, 8 up, 8 in 2024-09-13T13:22:48.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:48 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3643137799' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:22:48.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[36682]: 6.d scrub starts 2024-09-13T13:22:48.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[36682]: 6.d scrub ok 2024-09-13T13:22:48.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[36682]: 6.10 scrub starts 2024-09-13T13:22:48.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[36682]: 6.10 scrub ok 2024-09-13T13:22:48.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[36682]: 3.1a scrub starts 2024-09-13T13:22:48.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[36682]: 3.1a scrub ok 2024-09-13T13:22:48.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[36682]: osdmap e542: 8 total, 8 up, 8 in 2024-09-13T13:22:48.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[36682]: osdmap e543: 8 total, 8 up, 8 in 2024-09-13T13:22:48.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3643137799' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:22:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[29471]: 6.d scrub starts 2024-09-13T13:22:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[29471]: 6.d scrub ok 2024-09-13T13:22:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[29471]: 6.10 scrub starts 2024-09-13T13:22:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[29471]: 6.10 scrub ok 2024-09-13T13:22:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[29471]: 3.1a scrub starts 2024-09-13T13:22:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[29471]: 3.1a scrub ok 2024-09-13T13:22:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[29471]: osdmap e542: 8 total, 8 up, 8 in 2024-09-13T13:22:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[29471]: osdmap e543: 8 total, 8 up, 8 in 2024-09-13T13:22:48.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:48 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3643137799' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:22:49.166 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T13:22:49.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:49 smithi167 ceph-mon[33178]: pgmap v1111: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-09-13T13:22:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:49 smithi167 ceph-mon[33178]: 3.f scrub starts 2024-09-13T13:22:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:49 smithi167 ceph-mon[33178]: 3.f scrub ok 2024-09-13T13:22:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:49 smithi167 ceph-mon[33178]: 6.e scrub starts 2024-09-13T13:22:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:49 smithi167 ceph-mon[33178]: 6.e scrub ok 2024-09-13T13:22:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:49 smithi167 ceph-mon[33178]: 6.1f scrub starts 2024-09-13T13:22:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:49 smithi167 ceph-mon[33178]: 6.1f scrub ok 2024-09-13T13:22:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:49 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3643137799' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:22:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:49 smithi167 ceph-mon[33178]: osdmap e544: 8 total, 8 up, 8 in 2024-09-13T13:22:49.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:49 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:22:49.865 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[36682]: pgmap v1111: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-09-13T13:22:49.865 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[36682]: 3.f scrub starts 2024-09-13T13:22:49.865 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[36682]: 3.f scrub ok 2024-09-13T13:22:49.865 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[36682]: 6.e scrub starts 2024-09-13T13:22:49.865 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[36682]: 6.e scrub ok 2024-09-13T13:22:49.865 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[36682]: 6.1f scrub starts 2024-09-13T13:22:49.866 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[36682]: 6.1f scrub ok 2024-09-13T13:22:49.866 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3643137799' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:22:49.866 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[36682]: osdmap e544: 8 total, 8 up, 8 in 2024-09-13T13:22:49.866 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:22:49.866 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[29471]: pgmap v1111: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-09-13T13:22:49.866 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[29471]: 3.f scrub starts 2024-09-13T13:22:49.866 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[29471]: 3.f scrub ok 2024-09-13T13:22:49.866 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[29471]: 6.e scrub starts 2024-09-13T13:22:49.866 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[29471]: 6.e scrub ok 2024-09-13T13:22:49.866 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[29471]: 6.1f scrub starts 2024-09-13T13:22:49.867 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[29471]: 6.1f scrub ok 2024-09-13T13:22:49.867 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3643137799' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:22:49.867 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[29471]: osdmap e544: 8 total, 8 up, 8 in 2024-09-13T13:22:49.867 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:49 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:22:49.902 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-13T13:22:50.167 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:50.586 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:50 smithi052 ceph-mon[36682]: 6.19 scrub starts 2024-09-13T13:22:50.586 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:50 smithi052 ceph-mon[36682]: 6.19 scrub ok 2024-09-13T13:22:50.586 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:50 smithi052 ceph-mon[36682]: 3.11 scrub starts 2024-09-13T13:22:50.586 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:50 smithi052 ceph-mon[36682]: 3.11 scrub ok 2024-09-13T13:22:50.586 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:50 smithi052 ceph-mon[29471]: 6.19 scrub starts 2024-09-13T13:22:50.587 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:50 smithi052 ceph-mon[29471]: 6.19 scrub ok 2024-09-13T13:22:50.587 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:50 smithi052 ceph-mon[29471]: 3.11 scrub starts 2024-09-13T13:22:50.587 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:50 smithi052 ceph-mon[29471]: 3.11 scrub ok 2024-09-13T13:22:50.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:50 smithi167 ceph-mon[33178]: 6.19 scrub starts 2024-09-13T13:22:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:50 smithi167 ceph-mon[33178]: 6.19 scrub ok 2024-09-13T13:22:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:50 smithi167 ceph-mon[33178]: 3.11 scrub starts 2024-09-13T13:22:50.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:50 smithi167 ceph-mon[33178]: 3.11 scrub ok 2024-09-13T13:22:51.543 INFO:teuthology.orchestra.run.smithi052.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-13T13:22:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:51 smithi167 ceph-mon[33178]: pgmap v1114: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:22:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:51 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2035068622' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:22:51.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:51 smithi167 ceph-mon[33178]: 6.0 scrub starts 2024-09-13T13:22:51.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:51 smithi167 ceph-mon[33178]: 6.0 scrub ok 2024-09-13T13:22:51.865 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:51 smithi052 ceph-mon[36682]: pgmap v1114: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:22:51.865 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:51 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2035068622' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:22:51.865 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:51 smithi052 ceph-mon[36682]: 6.0 scrub starts 2024-09-13T13:22:51.865 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:51 smithi052 ceph-mon[36682]: 6.0 scrub ok 2024-09-13T13:22:51.865 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:51 smithi052 ceph-mon[29471]: pgmap v1114: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:22:51.865 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:51 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2035068622' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T13:22:51.866 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:51 smithi052 ceph-mon[29471]: 6.0 scrub starts 2024-09-13T13:22:51.866 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:51 smithi052 ceph-mon[29471]: 6.0 scrub ok 2024-09-13T13:22:52.362 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T13:22:52.632 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:52 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2035068622' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:22:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:52 smithi052 ceph-mon[36682]: osdmap e545: 8 total, 8 up, 8 in 2024-09-13T13:22:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:52 smithi052 ceph-mon[36682]: 4.1e scrub starts 2024-09-13T13:22:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:52 smithi052 ceph-mon[36682]: 4.1e scrub ok 2024-09-13T13:22:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:52 smithi052 ceph-mon[36682]: 4.19 scrub starts 2024-09-13T13:22:52.739 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:52 smithi052 ceph-mon[36682]: 4.19 scrub ok 2024-09-13T13:22:52.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:52 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2035068622' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:22:52.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:52 smithi052 ceph-mon[29471]: osdmap e545: 8 total, 8 up, 8 in 2024-09-13T13:22:52.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:52 smithi052 ceph-mon[29471]: 4.1e scrub starts 2024-09-13T13:22:52.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:52 smithi052 ceph-mon[29471]: 4.1e scrub ok 2024-09-13T13:22:52.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:52 smithi052 ceph-mon[29471]: 4.19 scrub starts 2024-09-13T13:22:52.740 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:52 smithi052 ceph-mon[29471]: 4.19 scrub ok 2024-09-13T13:22:52.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:52 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2035068622' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T13:22:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:52 smithi167 ceph-mon[33178]: osdmap e545: 8 total, 8 up, 8 in 2024-09-13T13:22:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:52 smithi167 ceph-mon[33178]: 4.1e scrub starts 2024-09-13T13:22:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:52 smithi167 ceph-mon[33178]: 4.1e scrub ok 2024-09-13T13:22:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:52 smithi167 ceph-mon[33178]: 4.19 scrub starts 2024-09-13T13:22:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:52 smithi167 ceph-mon[33178]: 4.19 scrub ok 2024-09-13T13:22:53.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:53 smithi167 ceph-mon[33178]: pgmap v1116: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:22:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:53 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3005230284' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:22:53.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:53 smithi167 ceph-mon[33178]: from='client.37097 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:22:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:53 smithi052 ceph-mon[36682]: pgmap v1116: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:22:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:53 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3005230284' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:22:53.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:53 smithi052 ceph-mon[36682]: from='client.37097 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:22:53.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:53 smithi052 ceph-mon[29471]: pgmap v1116: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:22:53.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:53 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3005230284' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:22:53.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:53 smithi052 ceph-mon[29471]: from='client.37097 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:22:54.171 INFO:teuthology.orchestra.run.smithi052.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T13:22:54.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:54 smithi167 ceph-mon[33178]: 5.c scrub starts 2024-09-13T13:22:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:54 smithi167 ceph-mon[33178]: 5.c scrub ok 2024-09-13T13:22:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:54 smithi167 ceph-mon[33178]: 3.e scrub starts 2024-09-13T13:22:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:54 smithi167 ceph-mon[33178]: 3.e scrub ok 2024-09-13T13:22:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:54 smithi167 ceph-mon[33178]: from='client.37097 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:22:54.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:54 smithi167 ceph-mon[33178]: osdmap e546: 8 total, 8 up, 8 in 2024-09-13T13:22:54.858 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:54 smithi052 ceph-mon[36682]: 5.c scrub starts 2024-09-13T13:22:54.858 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:54 smithi052 ceph-mon[36682]: 5.c scrub ok 2024-09-13T13:22:54.858 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:54 smithi052 ceph-mon[36682]: 3.e scrub starts 2024-09-13T13:22:54.858 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:54 smithi052 ceph-mon[36682]: 3.e scrub ok 2024-09-13T13:22:54.859 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:54 smithi052 ceph-mon[36682]: from='client.37097 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:22:54.859 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:54 smithi052 ceph-mon[36682]: osdmap e546: 8 total, 8 up, 8 in 2024-09-13T13:22:54.859 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:54 smithi052 ceph-mon[29471]: 5.c scrub starts 2024-09-13T13:22:54.859 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:54 smithi052 ceph-mon[29471]: 5.c scrub ok 2024-09-13T13:22:54.859 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:54 smithi052 ceph-mon[29471]: 3.e scrub starts 2024-09-13T13:22:54.859 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:54 smithi052 ceph-mon[29471]: 3.e scrub ok 2024-09-13T13:22:54.859 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:54 smithi052 ceph-mon[29471]: from='client.37097 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:22:54.859 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:54 smithi052 ceph-mon[29471]: osdmap e546: 8 total, 8 up, 8 in 2024-09-13T13:22:54.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-13T13:22:55.160 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:55.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:22:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:22:55.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:55 smithi167 ceph-mon[33178]: pgmap v1117: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:22:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:55 smithi167 ceph-mon[33178]: 4.d scrub starts 2024-09-13T13:22:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:55 smithi167 ceph-mon[33178]: 4.d scrub ok 2024-09-13T13:22:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:55 smithi167 ceph-mon[33178]: 3.d scrub starts 2024-09-13T13:22:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:55 smithi167 ceph-mon[33178]: 3.d scrub ok 2024-09-13T13:22:55.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:55 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:55.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:55 smithi052 ceph-mon[29471]: pgmap v1117: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:22:55.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:55 smithi052 ceph-mon[29471]: 4.d scrub starts 2024-09-13T13:22:55.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:55 smithi052 ceph-mon[29471]: 4.d scrub ok 2024-09-13T13:22:55.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:55 smithi052 ceph-mon[29471]: 3.d scrub starts 2024-09-13T13:22:55.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:55 smithi052 ceph-mon[29471]: 3.d scrub ok 2024-09-13T13:22:55.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:55 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:55.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:55 smithi052 ceph-mon[36682]: pgmap v1117: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail 2024-09-13T13:22:55.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:55 smithi052 ceph-mon[36682]: 4.d scrub starts 2024-09-13T13:22:55.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:55 smithi052 ceph-mon[36682]: 4.d scrub ok 2024-09-13T13:22:55.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:55 smithi052 ceph-mon[36682]: 3.d scrub starts 2024-09-13T13:22:55.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:55 smithi052 ceph-mon[36682]: 3.d scrub ok 2024-09-13T13:22:55.991 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:55 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:22:56.489 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:22:56 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:22:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:22:56.561 INFO:teuthology.orchestra.run.smithi052.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-13T13:22:56.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:56 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/809956745' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:22:56.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:56 smithi167 ceph-mon[33178]: from='client.37103 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:22:56.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:56 smithi167 ceph-mon[33178]: 6.11 scrub starts 2024-09-13T13:22:56.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:56 smithi167 ceph-mon[33178]: 6.11 scrub ok 2024-09-13T13:22:56.900 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:56 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/809956745' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:22:56.900 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:56 smithi052 ceph-mon[36682]: from='client.37103 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:22:56.901 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:56 smithi052 ceph-mon[36682]: 6.11 scrub starts 2024-09-13T13:22:56.901 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:56 smithi052 ceph-mon[36682]: 6.11 scrub ok 2024-09-13T13:22:56.901 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:56 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/809956745' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:22:56.901 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:56 smithi052 ceph-mon[29471]: from='client.37103 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T13:22:56.901 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:56 smithi052 ceph-mon[29471]: 6.11 scrub starts 2024-09-13T13:22:56.901 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:56 smithi052 ceph-mon[29471]: 6.11 scrub ok 2024-09-13T13:22:57.335 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-13T13:22:57.335 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-13T13:22:57.335 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-13T13:22:57.335 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-13T13:22:57.335 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T13:22:57.335 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T13:22:57.599 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:57.817 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:57 smithi052 ceph-mon[36682]: pgmap v1119: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-09-13T13:22:57.817 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:57 smithi052 ceph-mon[36682]: from='client.37103 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:22:57.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:57 smithi052 ceph-mon[36682]: osdmap e547: 8 total, 8 up, 8 in 2024-09-13T13:22:57.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:57 smithi052 ceph-mon[29471]: pgmap v1119: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-09-13T13:22:57.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:57 smithi052 ceph-mon[29471]: from='client.37103 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:22:57.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:57 smithi052 ceph-mon[29471]: osdmap e547: 8 total, 8 up, 8 in 2024-09-13T13:22:57.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:57 smithi167 ceph-mon[33178]: pgmap v1119: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-09-13T13:22:57.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:57 smithi167 ceph-mon[33178]: from='client.37103 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T13:22:57.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:57 smithi167 ceph-mon[33178]: osdmap e547: 8 total, 8 up, 8 in 2024-09-13T13:22:58.220 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:22:58.220 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":547,"fsid":"2b415b38-71cf-11ef-bceb-c7b262605968","created":"2024-09-13T12:55:27.893882+0000","modified":"2024-09-13T13:22:56.554184+0000","last_up_change":"2024-09-13T13:22:15.208527+0000","last_in_change":"2024-09-13T13:22:18.153442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T12:58:31.271893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"527","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"418","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-13T13:01:00.306747+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-13T13:01:01.103523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-13T13:01:02.249986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-13T13:01:04.203764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-13T13:01:06.293676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d52dca1a-94c1-4e9e-93f5-687c038cd7c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":525,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6801","nonce":542298937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6802","nonce":542298937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6804","nonce":542298937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6803","nonce":542298937}]},"public_addr":"172.21.15.52:6801/542298937","cluster_addr":"172.21.15.52:6802/542298937","heartbeat_back_addr":"172.21.15.52:6804/542298937","heartbeat_front_addr":"172.21.15.52:6803/542298937","state":["exists","up"]},{"osd":1,"uuid":"b40a3574-3c6f-4810-a651-8d64aee6801b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":525,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6805","nonce":26289431}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6806","nonce":26289431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6808","nonce":26289431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6807","nonce":26289431}]},"public_addr":"172.21.15.52:6805/26289431","cluster_addr":"172.21.15.52:6806/26289431","heartbeat_back_addr":"172.21.15.52:6808/26289431","heartbeat_front_addr":"172.21.15.52:6807/26289431","state":["exists","up"]},{"osd":2,"uuid":"1e8e49a0-ccf8-4b7f-839b-104b64601408","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":534,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6809","nonce":3960240393}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6810","nonce":3960240393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6812","nonce":3960240393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6811","nonce":3960240393}]},"public_addr":"172.21.15.52:6809/3960240393","cluster_addr":"172.21.15.52:6810/3960240393","heartbeat_back_addr":"172.21.15.52:6812/3960240393","heartbeat_front_addr":"172.21.15.52:6811/3960240393","state":["exists","up"]},{"osd":3,"uuid":"9f6f1d22-be68-4d86-9d4b-86fd9e87d141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":525,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6813","nonce":2025522885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6814","nonce":2025522885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6816","nonce":2025522885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6815","nonce":2025522885}]},"public_addr":"172.21.15.52:6813/2025522885","cluster_addr":"172.21.15.52:6814/2025522885","heartbeat_back_addr":"172.21.15.52:6816/2025522885","heartbeat_front_addr":"172.21.15.52:6815/2025522885","state":["exists","up"]},{"osd":4,"uuid":"1326d850-caf4-484e-90a1-052f54f9f306","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":33,"last_clean_end":502,"up_from":523,"up_thru":525,"down_at":503,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6800","nonce":545860898}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6801","nonce":545860898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6803","nonce":545860898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6802","nonce":545860898}]},"public_addr":"172.21.15.167:6800/545860898","cluster_addr":"172.21.15.167:6801/545860898","heartbeat_back_addr":"172.21.15.167:6803/545860898","heartbeat_front_addr":"172.21.15.167:6802/545860898","state":["exists","up"]},{"osd":5,"uuid":"ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":525,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6804","nonce":2906884337}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6805","nonce":2906884337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6807","nonce":2906884337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6806","nonce":2906884337}]},"public_addr":"172.21.15.167:6804/2906884337","cluster_addr":"172.21.15.167:6805/2906884337","heartbeat_back_addr":"172.21.15.167:6807/2906884337","heartbeat_front_addr":"172.21.15.167:6806/2906884337","state":["exists","up"]},{"osd":6,"uuid":"2471907d-fcbd-44e0-9d29-54151ada9df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":48,"last_clean_end":102,"up_from":118,"up_thru":525,"down_at":103,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6808","nonce":143093129}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6809","nonce":143093129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6811","nonce":143093129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6810","nonce":143093129}]},"public_addr":"172.21.15.167:6808/143093129","cluster_addr":"172.21.15.167:6809/143093129","heartbeat_back_addr":"172.21.15.167:6811/143093129","heartbeat_front_addr":"172.21.15.167:6810/143093129","state":["exists","up"]},{"osd":7,"uuid":"97d2c6d7-7989-4c11-91e0-32ee87e76c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":541,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6812","nonce":403155155}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6813","nonce":403155155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6815","nonce":403155155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6814","nonce":403155155}]},"public_addr":"172.21.15.167:6812/403155155","cluster_addr":"172.21.15.167:6813/403155155","heartbeat_back_addr":"172.21.15.167:6815/403155155","heartbeat_front_addr":"172.21.15.167:6814/403155155","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:22:43.896490+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:21:51.681502+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:22:13.205595+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:22:13.561249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:22:14.229657+0000","dead_epoch":502},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:21:59.915877+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:22:02.109733+0000","dead_epoch":102},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:21:45.552573+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":0}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":3,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/2104786675":"2024-09-14T13:01:31.849627+0000","172.21.15.52:6800/910649698":"2024-09-14T12:55:49.232470+0000","172.21.15.52:0/1471436716":"2024-09-14T12:55:49.232470+0000","172.21.15.52:0/4151877300":"2024-09-14T12:55:49.232470+0000","172.21.15.52:0/3080924817":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/4248579079":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/455730280":"2024-09-14T13:01:31.849627+0000","172.21.15.52:0/1887735845":"2024-09-14T12:55:49.232470+0000","172.21.15.52:6800/1437783219":"2024-09-14T12:56:14.802531+0000","172.21.15.52:6800/326637643":"2024-09-14T13:01:31.849627+0000","172.21.15.52:0/991228238":"2024-09-14T13:01:31.849627+0000","172.21.15.52:0/3415403690":"2024-09-14T13:01:31.849627+0000","172.21.15.52:0/1933060029":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/1009267179":"2024-09-14T13:01:31.849627+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:22:58.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:58 smithi167 ceph-mon[33178]: 1.a scrub starts 2024-09-13T13:22:58.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:58 smithi167 ceph-mon[33178]: 1.a scrub ok 2024-09-13T13:22:58.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:58 smithi167 ceph-mon[33178]: 3.1f scrub starts 2024-09-13T13:22:58.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:58 smithi167 ceph-mon[33178]: 3.1f scrub ok 2024-09-13T13:22:58.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:58 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2507176184' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:22:58.824 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:58 smithi052 ceph-mon[36682]: 1.a scrub starts 2024-09-13T13:22:58.824 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:58 smithi052 ceph-mon[36682]: 1.a scrub ok 2024-09-13T13:22:58.824 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:58 smithi052 ceph-mon[36682]: 3.1f scrub starts 2024-09-13T13:22:58.825 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:58 smithi052 ceph-mon[36682]: 3.1f scrub ok 2024-09-13T13:22:58.825 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:58 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2507176184' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:22:58.825 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:58 smithi052 ceph-mon[29471]: 1.a scrub starts 2024-09-13T13:22:58.825 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:58 smithi052 ceph-mon[29471]: 1.a scrub ok 2024-09-13T13:22:58.825 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:58 smithi052 ceph-mon[29471]: 3.1f scrub starts 2024-09-13T13:22:58.825 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:58 smithi052 ceph-mon[29471]: 3.1f scrub ok 2024-09-13T13:22:58.826 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:58 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2507176184' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:22:58.858 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T13:22:58.858 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T13:22:59.121 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:22:59.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:59 smithi167 ceph-mon[33178]: pgmap v1121: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-09-13T13:22:59.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:59 smithi167 ceph-mon[33178]: 6.1a scrub starts 2024-09-13T13:22:59.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:22:59 smithi167 ceph-mon[33178]: 6.1a scrub ok 2024-09-13T13:22:59.836 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:22:59.836 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":547,"fsid":"2b415b38-71cf-11ef-bceb-c7b262605968","created":"2024-09-13T12:55:27.893882+0000","modified":"2024-09-13T13:22:56.554184+0000","last_up_change":"2024-09-13T13:22:15.208527+0000","last_in_change":"2024-09-13T13:22:18.153442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T12:58:31.271893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"527","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"418","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-13T13:01:00.306747+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-13T13:01:01.103523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-13T13:01:02.249986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-13T13:01:04.203764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-13T13:01:06.293676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d52dca1a-94c1-4e9e-93f5-687c038cd7c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":525,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6801","nonce":542298937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6802","nonce":542298937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6804","nonce":542298937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6803","nonce":542298937}]},"public_addr":"172.21.15.52:6801/542298937","cluster_addr":"172.21.15.52:6802/542298937","heartbeat_back_addr":"172.21.15.52:6804/542298937","heartbeat_front_addr":"172.21.15.52:6803/542298937","state":["exists","up"]},{"osd":1,"uuid":"b40a3574-3c6f-4810-a651-8d64aee6801b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":525,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6805","nonce":26289431}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6806","nonce":26289431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6808","nonce":26289431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6807","nonce":26289431}]},"public_addr":"172.21.15.52:6805/26289431","cluster_addr":"172.21.15.52:6806/26289431","heartbeat_back_addr":"172.21.15.52:6808/26289431","heartbeat_front_addr":"172.21.15.52:6807/26289431","state":["exists","up"]},{"osd":2,"uuid":"1e8e49a0-ccf8-4b7f-839b-104b64601408","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":534,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6809","nonce":3960240393}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6810","nonce":3960240393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6812","nonce":3960240393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6811","nonce":3960240393}]},"public_addr":"172.21.15.52:6809/3960240393","cluster_addr":"172.21.15.52:6810/3960240393","heartbeat_back_addr":"172.21.15.52:6812/3960240393","heartbeat_front_addr":"172.21.15.52:6811/3960240393","state":["exists","up"]},{"osd":3,"uuid":"9f6f1d22-be68-4d86-9d4b-86fd9e87d141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":525,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6813","nonce":2025522885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6814","nonce":2025522885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6816","nonce":2025522885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.52:6815","nonce":2025522885}]},"public_addr":"172.21.15.52:6813/2025522885","cluster_addr":"172.21.15.52:6814/2025522885","heartbeat_back_addr":"172.21.15.52:6816/2025522885","heartbeat_front_addr":"172.21.15.52:6815/2025522885","state":["exists","up"]},{"osd":4,"uuid":"1326d850-caf4-484e-90a1-052f54f9f306","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":33,"last_clean_end":502,"up_from":523,"up_thru":525,"down_at":503,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6800","nonce":545860898}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6801","nonce":545860898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6803","nonce":545860898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6802","nonce":545860898}]},"public_addr":"172.21.15.167:6800/545860898","cluster_addr":"172.21.15.167:6801/545860898","heartbeat_back_addr":"172.21.15.167:6803/545860898","heartbeat_front_addr":"172.21.15.167:6802/545860898","state":["exists","up"]},{"osd":5,"uuid":"ad6f1a07-6615-4dc9-9ed0-fb9f7e23c8f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":525,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6804","nonce":2906884337}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6805","nonce":2906884337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6807","nonce":2906884337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6806","nonce":2906884337}]},"public_addr":"172.21.15.167:6804/2906884337","cluster_addr":"172.21.15.167:6805/2906884337","heartbeat_back_addr":"172.21.15.167:6807/2906884337","heartbeat_front_addr":"172.21.15.167:6806/2906884337","state":["exists","up"]},{"osd":6,"uuid":"2471907d-fcbd-44e0-9d29-54151ada9df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":48,"last_clean_end":102,"up_from":118,"up_thru":525,"down_at":103,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6808","nonce":143093129}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6809","nonce":143093129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6811","nonce":143093129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6810","nonce":143093129}]},"public_addr":"172.21.15.167:6808/143093129","cluster_addr":"172.21.15.167:6809/143093129","heartbeat_back_addr":"172.21.15.167:6811/143093129","heartbeat_front_addr":"172.21.15.167:6810/143093129","state":["exists","up"]},{"osd":7,"uuid":"97d2c6d7-7989-4c11-91e0-32ee87e76c6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":541,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6812","nonce":403155155}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6813","nonce":403155155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6815","nonce":403155155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6814","nonce":403155155}]},"public_addr":"172.21.15.167:6812/403155155","cluster_addr":"172.21.15.167:6813/403155155","heartbeat_back_addr":"172.21.15.167:6815/403155155","heartbeat_front_addr":"172.21.15.167:6814/403155155","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:22:43.896490+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:21:51.681502+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:22:13.205595+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:22:13.561249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:22:14.229657+0000","dead_epoch":502},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:21:59.915877+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:22:02.109733+0000","dead_epoch":102},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:21:45.552573+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":0}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":3,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/2104786675":"2024-09-14T13:01:31.849627+0000","172.21.15.52:6800/910649698":"2024-09-14T12:55:49.232470+0000","172.21.15.52:0/1471436716":"2024-09-14T12:55:49.232470+0000","172.21.15.52:0/4151877300":"2024-09-14T12:55:49.232470+0000","172.21.15.52:0/3080924817":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/4248579079":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/455730280":"2024-09-14T13:01:31.849627+0000","172.21.15.52:0/1887735845":"2024-09-14T12:55:49.232470+0000","172.21.15.52:6800/1437783219":"2024-09-14T12:56:14.802531+0000","172.21.15.52:6800/326637643":"2024-09-14T13:01:31.849627+0000","172.21.15.52:0/991228238":"2024-09-14T13:01:31.849627+0000","172.21.15.52:0/3415403690":"2024-09-14T13:01:31.849627+0000","172.21.15.52:0/1933060029":"2024-09-14T12:56:14.802531+0000","172.21.15.52:0/1009267179":"2024-09-14T13:01:31.849627+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:22:59.837 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:59 smithi052 ceph-mon[36682]: pgmap v1121: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-09-13T13:22:59.837 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:59 smithi052 ceph-mon[36682]: 6.1a scrub starts 2024-09-13T13:22:59.837 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:22:59 smithi052 ceph-mon[36682]: 6.1a scrub ok 2024-09-13T13:22:59.837 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:59 smithi052 ceph-mon[29471]: pgmap v1121: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-09-13T13:22:59.837 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:59 smithi052 ceph-mon[29471]: 6.1a scrub starts 2024-09-13T13:22:59.837 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:22:59 smithi052 ceph-mon[29471]: 6.1a scrub ok 2024-09-13T13:23:00.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T13:23:00.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T13:23:00.606 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T13:23:00.606 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T13:23:00.606 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T13:23:00.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T13:23:00.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T13:23:00.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T13:23:00.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:00 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4162158281' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:23:00.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:00 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4162158281' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:23:00.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:00 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4162158281' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T13:23:01.145 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:01.147 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:01.148 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:01.150 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:01.150 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:01.151 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:01.154 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:01.155 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:01.886 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:01 smithi052 ceph-mon[29471]: pgmap v1122: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:23:01.887 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:01 smithi052 ceph-mon[36682]: pgmap v1122: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:23:02.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:01 smithi167 ceph-mon[33178]: pgmap v1122: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:23:03.609 INFO:teuthology.orchestra.run.smithi052.stdout:2246267895819 2024-09-13T13:23:03.609 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T13:23:03.633 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:03 smithi052 ceph-mon[36682]: pgmap v1123: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:23:03.633 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:03 smithi052 ceph-mon[36682]: 5.18 scrub starts 2024-09-13T13:23:03.633 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:03 smithi052 ceph-mon[36682]: 5.18 scrub ok 2024-09-13T13:23:03.634 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:03 smithi052 ceph-mon[29471]: pgmap v1123: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:23:03.634 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:03 smithi052 ceph-mon[29471]: 5.18 scrub starts 2024-09-13T13:23:03.634 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:03 smithi052 ceph-mon[29471]: 5.18 scrub ok 2024-09-13T13:23:04.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:03 smithi167 ceph-mon[33178]: pgmap v1123: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:23:04.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:03 smithi167 ceph-mon[33178]: 5.18 scrub starts 2024-09-13T13:23:04.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:03 smithi167 ceph-mon[33178]: 5.18 scrub ok 2024-09-13T13:23:04.465 INFO:teuthology.orchestra.run.smithi052.stdout:236223201548 2024-09-13T13:23:04.466 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T13:23:04.637 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:04 smithi052 ceph-mon[36682]: 1.1 scrub starts 2024-09-13T13:23:04.637 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:04 smithi052 ceph-mon[36682]: 1.1 scrub ok 2024-09-13T13:23:04.637 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:04 smithi052 ceph-mon[36682]: 5.12 scrub starts 2024-09-13T13:23:04.637 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:04 smithi052 ceph-mon[36682]: 5.12 scrub ok 2024-09-13T13:23:04.637 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:04 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:23:04.637 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:04 smithi052 ceph-mon[29471]: 1.1 scrub starts 2024-09-13T13:23:04.637 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:04 smithi052 ceph-mon[29471]: 1.1 scrub ok 2024-09-13T13:23:04.637 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:04 smithi052 ceph-mon[29471]: 5.12 scrub starts 2024-09-13T13:23:04.638 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:04 smithi052 ceph-mon[29471]: 5.12 scrub ok 2024-09-13T13:23:04.638 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:04 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:23:05.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:04 smithi167 ceph-mon[33178]: 1.1 scrub starts 2024-09-13T13:23:05.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:04 smithi167 ceph-mon[33178]: 1.1 scrub ok 2024-09-13T13:23:05.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:04 smithi167 ceph-mon[33178]: 5.12 scrub starts 2024-09-13T13:23:05.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:04 smithi167 ceph-mon[33178]: 5.12 scrub ok 2024-09-13T13:23:05.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:04 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:23:05.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:23:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:23:05.872 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:05 smithi052 ceph-mon[36682]: pgmap v1124: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:23:05.872 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:05 smithi052 ceph-mon[36682]: 4.1b scrub starts 2024-09-13T13:23:05.872 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:05 smithi052 ceph-mon[36682]: 4.1b scrub ok 2024-09-13T13:23:05.872 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:05 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:23:05.872 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:05 smithi052 ceph-mon[29471]: pgmap v1124: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:23:05.873 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:05 smithi052 ceph-mon[29471]: 4.1b scrub starts 2024-09-13T13:23:05.873 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:05 smithi052 ceph-mon[29471]: 4.1b scrub ok 2024-09-13T13:23:05.873 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:05 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:23:05.905 INFO:teuthology.orchestra.run.smithi052.stdout:171798692119 2024-09-13T13:23:05.905 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T13:23:06.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:05 smithi167 ceph-mon[33178]: pgmap v1124: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:23:06.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:05 smithi167 ceph-mon[33178]: 4.1b scrub starts 2024-09-13T13:23:06.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:05 smithi167 ceph-mon[33178]: 4.1b scrub ok 2024-09-13T13:23:06.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:05 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:23:06.434 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:23:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:23:06.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[29471]: 6.a scrub starts 2024-09-13T13:23:06.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[29471]: 6.a scrub ok 2024-09-13T13:23:06.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[29471]: 6.13 scrub starts 2024-09-13T13:23:06.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[29471]: 6.13 scrub ok 2024-09-13T13:23:06.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:23:06.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:23:06.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[29471]: osdmap e548: 8 total, 8 up, 8 in 2024-09-13T13:23:06.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[36682]: 6.a scrub starts 2024-09-13T13:23:06.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[36682]: 6.a scrub ok 2024-09-13T13:23:06.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[36682]: 6.13 scrub starts 2024-09-13T13:23:06.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[36682]: 6.13 scrub ok 2024-09-13T13:23:06.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:23:06.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:23:06.740 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:06 smithi052 ceph-mon[36682]: osdmap e548: 8 total, 8 up, 8 in 2024-09-13T13:23:07.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:06 smithi167 ceph-mon[33178]: 6.a scrub starts 2024-09-13T13:23:07.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:06 smithi167 ceph-mon[33178]: 6.a scrub ok 2024-09-13T13:23:07.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:06 smithi167 ceph-mon[33178]: 6.13 scrub starts 2024-09-13T13:23:07.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:06 smithi167 ceph-mon[33178]: 6.13 scrub ok 2024-09-13T13:23:07.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:06 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:23:07.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:06 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:23:07.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:06 smithi167 ceph-mon[33178]: osdmap e548: 8 total, 8 up, 8 in 2024-09-13T13:23:07.720 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:07 smithi052 ceph-mon[36682]: pgmap v1126: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:23:07.720 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:07 smithi052 ceph-mon[36682]: 6.8 scrub starts 2024-09-13T13:23:07.720 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:07 smithi052 ceph-mon[36682]: 6.8 scrub ok 2024-09-13T13:23:07.720 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:07 smithi052 ceph-mon[36682]: 5.15 scrub starts 2024-09-13T13:23:07.720 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:07 smithi052 ceph-mon[36682]: 5.15 scrub ok 2024-09-13T13:23:07.721 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:07 smithi052 ceph-mon[29471]: pgmap v1126: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:23:07.721 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:07 smithi052 ceph-mon[29471]: 6.8 scrub starts 2024-09-13T13:23:07.721 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:07 smithi052 ceph-mon[29471]: 6.8 scrub ok 2024-09-13T13:23:07.721 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:07 smithi052 ceph-mon[29471]: 5.15 scrub starts 2024-09-13T13:23:07.721 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:07 smithi052 ceph-mon[29471]: 5.15 scrub ok 2024-09-13T13:23:08.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:07 smithi167 ceph-mon[33178]: pgmap v1126: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-13T13:23:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:07 smithi167 ceph-mon[33178]: 6.8 scrub starts 2024-09-13T13:23:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:07 smithi167 ceph-mon[33178]: 6.8 scrub ok 2024-09-13T13:23:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:07 smithi167 ceph-mon[33178]: 5.15 scrub starts 2024-09-13T13:23:08.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:07 smithi167 ceph-mon[33178]: 5.15 scrub ok 2024-09-13T13:23:08.415 INFO:teuthology.orchestra.run.smithi052.stdout:34359738677 2024-09-13T13:23:08.416 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T13:23:08.750 INFO:teuthology.orchestra.run.smithi052.stdout:77309411624 2024-09-13T13:23:08.751 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T13:23:09.163 INFO:teuthology.orchestra.run.smithi052.stdout:55834575151 2024-09-13T13:23:09.164 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T13:23:09.483 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:09 smithi052 ceph-mon[36682]: pgmap v1127: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:23:09.483 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:09 smithi052 ceph-mon[36682]: osdmap e549: 8 total, 8 up, 8 in 2024-09-13T13:23:09.483 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:09 smithi052 ceph-mon[36682]: 1.7 scrub starts 2024-09-13T13:23:09.484 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:09 smithi052 ceph-mon[36682]: 1.7 scrub ok 2024-09-13T13:23:09.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:09 smithi052 ceph-mon[29471]: pgmap v1127: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:23:09.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:09 smithi052 ceph-mon[29471]: osdmap e549: 8 total, 8 up, 8 in 2024-09-13T13:23:09.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:09 smithi052 ceph-mon[29471]: 1.7 scrub starts 2024-09-13T13:23:09.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:09 smithi052 ceph-mon[29471]: 1.7 scrub ok 2024-09-13T13:23:09.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:09 smithi167 ceph-mon[33178]: pgmap v1127: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:23:09.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:09 smithi167 ceph-mon[33178]: osdmap e549: 8 total, 8 up, 8 in 2024-09-13T13:23:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:09 smithi167 ceph-mon[33178]: 1.7 scrub starts 2024-09-13T13:23:09.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:09 smithi167 ceph-mon[33178]: 1.7 scrub ok 2024-09-13T13:23:09.739 INFO:teuthology.orchestra.run.smithi052.stdout:506806141159 2024-09-13T13:23:09.739 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T13:23:10.145 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:10.145 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:10.185 INFO:teuthology.orchestra.run.smithi052.stdout:115964117285 2024-09-13T13:23:10.185 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T13:23:10.187 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:10.206 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:10.709 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:23:10.709 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:23:10.709 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[36682]: 1.9 scrub starts 2024-09-13T13:23:10.709 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[36682]: 1.9 scrub ok 2024-09-13T13:23:10.710 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[36682]: osdmap e550: 8 total, 8 up, 8 in 2024-09-13T13:23:10.710 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[36682]: 4.1 scrub starts 2024-09-13T13:23:10.710 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[36682]: 4.1 scrub ok 2024-09-13T13:23:10.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:23:10.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:23:10.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[29471]: 1.9 scrub starts 2024-09-13T13:23:10.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[29471]: 1.9 scrub ok 2024-09-13T13:23:10.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[29471]: osdmap e550: 8 total, 8 up, 8 in 2024-09-13T13:23:10.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[29471]: 4.1 scrub starts 2024-09-13T13:23:10.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:10 smithi052 ceph-mon[29471]: 4.1 scrub ok 2024-09-13T13:23:11.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:10 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:23:11.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:10 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:23:11.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:10 smithi167 ceph-mon[33178]: 1.9 scrub starts 2024-09-13T13:23:11.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:10 smithi167 ceph-mon[33178]: 1.9 scrub ok 2024-09-13T13:23:11.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:10 smithi167 ceph-mon[33178]: osdmap e550: 8 total, 8 up, 8 in 2024-09-13T13:23:11.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:10 smithi167 ceph-mon[33178]: 4.1 scrub starts 2024-09-13T13:23:11.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:10 smithi167 ceph-mon[33178]: 4.1 scrub ok 2024-09-13T13:23:11.117 INFO:teuthology.orchestra.run.smithi052.stdout:2246267895821 2024-09-13T13:23:11.310 INFO:teuthology.orchestra.run.smithi052.stdout:236223201549 2024-09-13T13:23:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:11 smithi052 ceph-mon[36682]: pgmap v1130: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-13T13:23:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:11 smithi052 ceph-mon[36682]: 5.2 scrub starts 2024-09-13T13:23:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:11 smithi052 ceph-mon[36682]: 5.2 scrub ok 2024-09-13T13:23:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/3268102605' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T13:23:11.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:11 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/593259637' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T13:23:11.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:11 smithi052 ceph-mon[29471]: pgmap v1130: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-13T13:23:11.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:11 smithi052 ceph-mon[29471]: 5.2 scrub starts 2024-09-13T13:23:11.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:11 smithi052 ceph-mon[29471]: 5.2 scrub ok 2024-09-13T13:23:11.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/3268102605' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T13:23:11.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:11 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/593259637' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T13:23:12.068 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201548 got 236223201549 for osd.7 2024-09-13T13:23:12.069 DEBUG:teuthology.parallel:result is None 2024-09-13T13:23:12.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:11 smithi167 ceph-mon[33178]: pgmap v1130: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-13T13:23:12.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:11 smithi167 ceph-mon[33178]: 5.2 scrub starts 2024-09-13T13:23:12.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:11 smithi167 ceph-mon[33178]: 5.2 scrub ok 2024-09-13T13:23:12.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/3268102605' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T13:23:12.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:11 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/593259637' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T13:23:12.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:12 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:23:12.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:12 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:23:12.697 INFO:teuthology.orchestra.run.smithi052.stdout:34359738679 2024-09-13T13:23:12.802 INFO:teuthology.orchestra.run.smithi052.stdout:77309411625 2024-09-13T13:23:12.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:12 smithi052 ceph-mon[29471]: osdmap e551: 8 total, 8 up, 8 in 2024-09-13T13:23:12.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:12 smithi052 ceph-mon[29471]: 4.1c scrub starts 2024-09-13T13:23:12.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:12 smithi052 ceph-mon[29471]: 4.1c scrub ok 2024-09-13T13:23:12.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:12 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/315407293' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T13:23:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:12 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:23:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:12 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:23:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:12 smithi052 ceph-mon[36682]: osdmap e551: 8 total, 8 up, 8 in 2024-09-13T13:23:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:12 smithi052 ceph-mon[36682]: 4.1c scrub starts 2024-09-13T13:23:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:12 smithi052 ceph-mon[36682]: 4.1c scrub ok 2024-09-13T13:23:12.990 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:12 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/315407293' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T13:23:13.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:12 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:23:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:12 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:23:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:12 smithi167 ceph-mon[33178]: osdmap e551: 8 total, 8 up, 8 in 2024-09-13T13:23:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:12 smithi167 ceph-mon[33178]: 4.1c scrub starts 2024-09-13T13:23:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:12 smithi167 ceph-mon[33178]: 4.1c scrub ok 2024-09-13T13:23:13.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:12 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/315407293' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T13:23:13.585 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411624 got 77309411625 for osd.2 2024-09-13T13:23:13.586 DEBUG:teuthology.parallel:result is None 2024-09-13T13:23:13.964 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:13 smithi052 ceph-mon[36682]: pgmap v1132: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-13T13:23:13.965 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:13 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2952513650' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T13:23:13.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:13 smithi052 ceph-mon[29471]: pgmap v1132: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-13T13:23:13.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:13 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2952513650' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T13:23:13.974 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:14.005 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2246267895819 got 2246267895821 for osd.4 2024-09-13T13:23:14.005 DEBUG:teuthology.parallel:result is None 2024-09-13T13:23:14.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:13 smithi167 ceph-mon[33178]: pgmap v1132: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-13T13:23:14.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:13 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2952513650' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T13:23:14.232 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:14.232 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:14.232 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:14.676 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738677 got 34359738679 for osd.0 2024-09-13T13:23:14.676 DEBUG:teuthology.parallel:result is None 2024-09-13T13:23:14.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:14 smithi052 ceph-mon[36682]: 6.9 scrub starts 2024-09-13T13:23:14.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:14 smithi052 ceph-mon[36682]: 6.9 scrub ok 2024-09-13T13:23:14.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:14 smithi052 ceph-mon[29471]: 6.9 scrub starts 2024-09-13T13:23:14.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:14 smithi052 ceph-mon[29471]: 6.9 scrub ok 2024-09-13T13:23:15.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:14 smithi167 ceph-mon[33178]: 6.9 scrub starts 2024-09-13T13:23:15.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:14 smithi167 ceph-mon[33178]: 6.9 scrub ok 2024-09-13T13:23:15.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:23:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:23:15.081 INFO:teuthology.orchestra.run.smithi052.stdout:171798692122 2024-09-13T13:23:15.818 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798692119 got 171798692122 for osd.5 2024-09-13T13:23:15.818 DEBUG:teuthology.parallel:result is None 2024-09-13T13:23:15.914 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:15 smithi052 ceph-mon[29471]: pgmap v1133: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T13:23:15.914 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:15 smithi052 ceph-mon[29471]: 2.0 scrub starts 2024-09-13T13:23:15.914 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:15 smithi052 ceph-mon[29471]: 2.0 scrub ok 2024-09-13T13:23:15.914 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:15 smithi052 ceph-mon[29471]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:23:15.914 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:15 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/4263138616' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T13:23:15.914 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:15 smithi052 ceph-mon[36682]: pgmap v1133: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T13:23:15.914 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:15 smithi052 ceph-mon[36682]: 2.0 scrub starts 2024-09-13T13:23:15.915 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:15 smithi052 ceph-mon[36682]: 2.0 scrub ok 2024-09-13T13:23:15.915 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:15 smithi052 ceph-mon[36682]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:23:15.915 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:15 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/4263138616' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T13:23:16.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:15 smithi167 ceph-mon[33178]: pgmap v1133: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-13T13:23:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:15 smithi167 ceph-mon[33178]: 2.0 scrub starts 2024-09-13T13:23:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:15 smithi167 ceph-mon[33178]: 2.0 scrub ok 2024-09-13T13:23:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:15 smithi167 ceph-mon[33178]: from='client.24487 v1:172.21.15.167:0/4127319679' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-13T13:23:16.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:15 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/4263138616' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T13:23:16.229 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:23:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:23:16.695 INFO:teuthology.orchestra.run.smithi052.stdout:506806141160 2024-09-13T13:23:16.754 INFO:teuthology.orchestra.run.smithi052.stdout:55834575153 2024-09-13T13:23:16.895 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[36682]: 1.6 scrub starts 2024-09-13T13:23:16.895 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[36682]: 1.6 scrub ok 2024-09-13T13:23:16.895 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1841983642' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T13:23:16.895 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[36682]: 4.9 scrub starts 2024-09-13T13:23:16.896 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[36682]: 4.9 scrub ok 2024-09-13T13:23:16.896 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[36682]: 3.2 scrub starts 2024-09-13T13:23:16.896 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[36682]: 3.2 scrub ok 2024-09-13T13:23:16.896 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1646002028' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T13:23:16.897 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[29471]: 1.6 scrub starts 2024-09-13T13:23:16.897 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[29471]: 1.6 scrub ok 2024-09-13T13:23:16.897 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1841983642' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T13:23:16.897 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[29471]: 4.9 scrub starts 2024-09-13T13:23:16.897 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[29471]: 4.9 scrub ok 2024-09-13T13:23:16.897 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[29471]: 3.2 scrub starts 2024-09-13T13:23:16.898 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[29471]: 3.2 scrub ok 2024-09-13T13:23:16.898 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:16 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1646002028' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T13:23:17.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:16 smithi167 ceph-mon[33178]: 1.6 scrub starts 2024-09-13T13:23:17.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:16 smithi167 ceph-mon[33178]: 1.6 scrub ok 2024-09-13T13:23:17.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:16 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1841983642' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T13:23:17.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:16 smithi167 ceph-mon[33178]: 4.9 scrub starts 2024-09-13T13:23:17.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:16 smithi167 ceph-mon[33178]: 4.9 scrub ok 2024-09-13T13:23:17.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:16 smithi167 ceph-mon[33178]: 3.2 scrub starts 2024-09-13T13:23:17.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:16 smithi167 ceph-mon[33178]: 3.2 scrub ok 2024-09-13T13:23:17.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:16 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1646002028' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T13:23:17.747 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575151 got 55834575153 for osd.1 2024-09-13T13:23:17.747 DEBUG:teuthology.parallel:result is None 2024-09-13T13:23:17.827 INFO:teuthology.orchestra.run.smithi052.stdout:115964117287 2024-09-13T13:23:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:17 smithi052 ceph-mon[36682]: pgmap v1134: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:23:17.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:17 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/1794199841' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T13:23:17.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:17 smithi052 ceph-mon[29471]: pgmap v1134: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:23:17.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:17 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/1794199841' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T13:23:18.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:17 smithi167 ceph-mon[33178]: pgmap v1134: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-13T13:23:18.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:17 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/1794199841' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T13:23:18.441 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117285 got 115964117287 for osd.3 2024-09-13T13:23:18.441 DEBUG:teuthology.parallel:result is None 2024-09-13T13:23:18.829 INFO:tasks.cephadm.ceph_manager.ceph:need seq 506806141159 got 506806141160 for osd.6 2024-09-13T13:23:18.829 DEBUG:teuthology.parallel:result is None 2024-09-13T13:23:18.829 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T13:23:18.829 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:23:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:18 smithi052 ceph-mon[36682]: 4.17 scrub starts 2024-09-13T13:23:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:18 smithi052 ceph-mon[36682]: 4.17 scrub ok 2024-09-13T13:23:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:18 smithi052 ceph-mon[36682]: 6.1c scrub starts 2024-09-13T13:23:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:18 smithi052 ceph-mon[36682]: 6.1c scrub ok 2024-09-13T13:23:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:18 smithi052 ceph-mon[36682]: 3.18 scrub starts 2024-09-13T13:23:18.989 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:18 smithi052 ceph-mon[36682]: 3.18 scrub ok 2024-09-13T13:23:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:18 smithi052 ceph-mon[29471]: 4.17 scrub starts 2024-09-13T13:23:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:18 smithi052 ceph-mon[29471]: 4.17 scrub ok 2024-09-13T13:23:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:18 smithi052 ceph-mon[29471]: 6.1c scrub starts 2024-09-13T13:23:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:18 smithi052 ceph-mon[29471]: 6.1c scrub ok 2024-09-13T13:23:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:18 smithi052 ceph-mon[29471]: 3.18 scrub starts 2024-09-13T13:23:18.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:18 smithi052 ceph-mon[29471]: 3.18 scrub ok 2024-09-13T13:23:19.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:18 smithi167 ceph-mon[33178]: 4.17 scrub starts 2024-09-13T13:23:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:18 smithi167 ceph-mon[33178]: 4.17 scrub ok 2024-09-13T13:23:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:18 smithi167 ceph-mon[33178]: 6.1c scrub starts 2024-09-13T13:23:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:18 smithi167 ceph-mon[33178]: 6.1c scrub ok 2024-09-13T13:23:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:18 smithi167 ceph-mon[33178]: 3.18 scrub starts 2024-09-13T13:23:19.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:18 smithi167 ceph-mon[33178]: 3.18 scrub ok 2024-09-13T13:23:19.096 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:19.816 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:23:19.817 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:19 smithi052 ceph-mon[36682]: 3.c scrub starts 2024-09-13T13:23:19.817 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:19 smithi052 ceph-mon[36682]: 3.c scrub ok 2024-09-13T13:23:19.817 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:19 smithi052 ceph-mon[36682]: pgmap v1135: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-09-13T13:23:19.817 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:19 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:23:19.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:19 smithi052 ceph-mon[29471]: 3.c scrub starts 2024-09-13T13:23:19.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:19 smithi052 ceph-mon[29471]: 3.c scrub ok 2024-09-13T13:23:19.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:19 smithi052 ceph-mon[29471]: pgmap v1135: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-09-13T13:23:19.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:19 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:23:19.823 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:23:19.972 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:19 smithi167 ceph-mon[33178]: 3.c scrub starts 2024-09-13T13:23:19.972 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:19 smithi167 ceph-mon[33178]: 3.c scrub ok 2024-09-13T13:23:19.972 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:19 smithi167 ceph-mon[33178]: pgmap v1135: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-09-13T13:23:19.972 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:19 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T13:23:20.575 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":1135,"stamp":"2024-09-13T13:23:18.107842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2151,"num_read_kb":1903,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":386,"num_bytes_recovered":667300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1961,"ondisk_log_size":1961,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":427,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5763168,"kb_used_data":26812,"kb_used_omap":18,"kb_used_meta":5736173,"kb_avail":744198048,"statfs":{"total":767960285184,"available":762058801152,"internally_reserved":0,"allocated":27455488,"data_stored":22213857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18440,"internal_metadata":5873842168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.477433"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":1072,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910956+0000","last_change":"2024-09-13T13:22:58.056700+0000","last_active":"2024-09-13T13:23:11.910956+0000","last_peered":"2024-09-13T13:23:11.910956+0000","last_clean":"2024-09-13T13:23:11.910956+0000","last_became_active":"2024-09-13T13:17:25.335900+0000","last_became_peered":"2024-09-13T13:17:25.335900+0000","last_unstale":"2024-09-13T13:23:11.910956+0000","last_undegraded":"2024-09-13T13:23:11.910956+0000","last_fullsized":"2024-09-13T13:23:11.910956+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:58.056653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:58.056653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:24.248375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"500'21","reported_seq":514,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910020+0000","last_change":"2024-09-13T13:22:35.109312+0000","last_active":"2024-09-13T13:23:11.910020+0000","last_peered":"2024-09-13T13:23:11.910020+0000","last_clean":"2024-09-13T13:23:11.910020+0000","last_became_active":"2024-09-13T13:22:16.231034+0000","last_became_peered":"2024-09-13T13:22:16.231034+0000","last_unstale":"2024-09-13T13:23:11.910020+0000","last_undegraded":"2024-09-13T13:23:11.910020+0000","last_fullsized":"2024-09-13T13:23:11.910020+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:22:35.109229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:35.109229+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:00.178488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1080,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909822+0000","last_change":"2024-09-13T13:22:28.036382+0000","last_active":"2024-09-13T13:23:11.909822+0000","last_peered":"2024-09-13T13:23:11.909822+0000","last_clean":"2024-09-13T13:23:11.909822+0000","last_became_active":"2024-09-13T13:22:19.167858+0000","last_became_peered":"2024-09-13T13:22:19.167858+0000","last_unstale":"2024-09-13T13:23:11.909822+0000","last_undegraded":"2024-09-13T13:23:11.909822+0000","last_fullsized":"2024-09-13T13:23:11.909822+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:28.036335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:28.036335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:56.564561+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":475,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911351+0000","last_change":"2024-09-13T13:22:58.283257+0000","last_active":"2024-09-13T13:23:11.911351+0000","last_peered":"2024-09-13T13:23:11.911351+0000","last_clean":"2024-09-13T13:23:11.911351+0000","last_became_active":"2024-09-13T13:22:16.242543+0000","last_became_peered":"2024-09-13T13:22:16.242543+0000","last_unstale":"2024-09-13T13:23:11.911351+0000","last_undegraded":"2024-09-13T13:23:11.911351+0000","last_fullsized":"2024-09-13T13:23:11.911351+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:58.283191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:58.283191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:59.591998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1054,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642580+0000","last_change":"2024-09-13T13:22:19.171791+0000","last_active":"2024-09-13T13:23:11.642580+0000","last_peered":"2024-09-13T13:23:11.642580+0000","last_clean":"2024-09-13T13:23:11.642580+0000","last_became_active":"2024-09-13T13:22:19.171472+0000","last_became_peered":"2024-09-13T13:22:19.171472+0000","last_unstale":"2024-09-13T13:23:11.642580+0000","last_undegraded":"2024-09-13T13:23:11.642580+0000","last_fullsized":"2024-09-13T13:23:11.642580+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:21:18.313800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:21:18.313800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1093,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643776+0000","last_change":"2024-09-13T13:22:42.380173+0000","last_active":"2024-09-13T13:23:11.643776+0000","last_peered":"2024-09-13T13:23:11.643776+0000","last_clean":"2024-09-13T13:23:11.643776+0000","last_became_active":"2024-09-13T13:17:25.243371+0000","last_became_peered":"2024-09-13T13:17:25.243371+0000","last_unstale":"2024-09-13T13:23:11.643776+0000","last_undegraded":"2024-09-13T13:23:11.643776+0000","last_fullsized":"2024-09-13T13:23:11.643776+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:42.380129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:42.380129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:59.325163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"500'39","reported_seq":1185,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643524+0000","last_change":"2024-09-13T13:22:52.349355+0000","last_active":"2024-09-13T13:23:11.643524+0000","last_peered":"2024-09-13T13:23:11.643524+0000","last_clean":"2024-09-13T13:23:11.643524+0000","last_became_active":"2024-09-13T13:17:25.245131+0000","last_became_peered":"2024-09-13T13:17:25.245131+0000","last_unstale":"2024-09-13T13:23:11.643524+0000","last_undegraded":"2024-09-13T13:23:11.643524+0000","last_fullsized":"2024-09-13T13:23:11.643524+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'39","last_scrub_stamp":"2024-09-13T13:22:52.349278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:52.349278+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:59.651974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":454,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910201+0000","last_change":"2024-09-13T13:23:02.305469+0000","last_active":"2024-09-13T13:23:11.910201+0000","last_peered":"2024-09-13T13:23:11.910201+0000","last_clean":"2024-09-13T13:23:11.910201+0000","last_became_active":"2024-09-13T13:22:16.236142+0000","last_became_peered":"2024-09-13T13:22:16.236142+0000","last_unstale":"2024-09-13T13:23:11.910201+0000","last_undegraded":"2024-09-13T13:23:11.910201+0000","last_fullsized":"2024-09-13T13:23:11.910201+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:02.305405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:23:02.305405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:02.773497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":1122,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911573+0000","last_change":"2024-09-13T13:22:44.954301+0000","last_active":"2024-09-13T13:23:11.911573+0000","last_peered":"2024-09-13T13:23:11.911573+0000","last_clean":"2024-09-13T13:23:11.911573+0000","last_became_active":"2024-09-13T13:22:16.235036+0000","last_became_peered":"2024-09-13T13:22:16.235036+0000","last_unstale":"2024-09-13T13:23:11.911573+0000","last_undegraded":"2024-09-13T13:23:11.911573+0000","last_fullsized":"2024-09-13T13:23:11.911573+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:22:44.954077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:44.954077+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:58.901786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"500'21","reported_seq":505,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911449+0000","last_change":"2024-09-13T13:22:20.130618+0000","last_active":"2024-09-13T13:23:11.911449+0000","last_peered":"2024-09-13T13:23:11.911449+0000","last_clean":"2024-09-13T13:23:11.911449+0000","last_became_active":"2024-09-13T13:22:16.241297+0000","last_became_peered":"2024-09-13T13:22:16.241297+0000","last_unstale":"2024-09-13T13:23:11.911449+0000","last_undegraded":"2024-09-13T13:23:11.911449+0000","last_fullsized":"2024-09-13T13:23:11.911449+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:22:20.130562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:20.130562+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:46.912600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1081,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910984+0000","last_change":"2024-09-13T13:22:23.733890+0000","last_active":"2024-09-13T13:23:11.910984+0000","last_peered":"2024-09-13T13:23:11.910984+0000","last_clean":"2024-09-13T13:23:11.910984+0000","last_became_active":"2024-09-13T13:22:19.169471+0000","last_became_peered":"2024-09-13T13:22:19.169471+0000","last_unstale":"2024-09-13T13:23:11.910984+0000","last_undegraded":"2024-09-13T13:23:11.910984+0000","last_fullsized":"2024-09-13T13:23:11.910984+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:23.733828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:23.733828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:28.354839+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1067,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909632+0000","last_change":"2024-09-13T13:22:19.181827+0000","last_active":"2024-09-13T13:23:11.909632+0000","last_peered":"2024-09-13T13:23:11.909632+0000","last_clean":"2024-09-13T13:23:11.909632+0000","last_became_active":"2024-09-13T13:22:19.181508+0000","last_became_peered":"2024-09-13T13:22:19.181508+0000","last_unstale":"2024-09-13T13:23:11.909632+0000","last_undegraded":"2024-09-13T13:23:11.909632+0000","last_fullsized":"2024-09-13T13:23:11.909632+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:08.489422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:08.489422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:18.900312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1097,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910726+0000","last_change":"2024-09-13T13:22:34.822934+0000","last_active":"2024-09-13T13:23:11.910726+0000","last_peered":"2024-09-13T13:23:11.910726+0000","last_clean":"2024-09-13T13:23:11.910726+0000","last_became_active":"2024-09-13T13:22:16.234121+0000","last_became_peered":"2024-09-13T13:22:16.234121+0000","last_unstale":"2024-09-13T13:23:11.910726+0000","last_undegraded":"2024-09-13T13:23:11.910726+0000","last_fullsized":"2024-09-13T13:23:11.910726+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:34.822864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:34.822864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:00.744920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"500'9","reported_seq":504,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910131+0000","last_change":"2024-09-13T13:23:04.254691+0000","last_active":"2024-09-13T13:23:11.910131+0000","last_peered":"2024-09-13T13:23:11.910131+0000","last_clean":"2024-09-13T13:23:11.910131+0000","last_became_active":"2024-09-13T13:22:16.235422+0000","last_became_peered":"2024-09-13T13:22:16.235422+0000","last_unstale":"2024-09-13T13:23:11.910131+0000","last_undegraded":"2024-09-13T13:23:11.910131+0000","last_fullsized":"2024-09-13T13:23:11.910131+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'9","last_scrub_stamp":"2024-09-13T13:23:04.254619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:23:04.254619+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:10.814239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":70,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910512+0000","last_change":"2024-09-13T13:22:25.321028+0000","last_active":"2024-09-13T13:23:11.910512+0000","last_peered":"2024-09-13T13:23:11.910512+0000","last_clean":"2024-09-13T13:23:11.910512+0000","last_became_active":"2024-09-13T13:22:19.179612+0000","last_became_peered":"2024-09-13T13:22:19.179612+0000","last_unstale":"2024-09-13T13:23:11.910512+0000","last_undegraded":"2024-09-13T13:23:11.910512+0000","last_fullsized":"2024-09-13T13:23:11.910512+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:25.320980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:25.320980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:35.114273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1037,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910112+0000","last_change":"2024-09-13T13:22:48.926914+0000","last_active":"2024-09-13T13:23:11.910112+0000","last_peered":"2024-09-13T13:23:11.910112+0000","last_clean":"2024-09-13T13:23:11.910112+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:23:11.910112+0000","last_undegraded":"2024-09-13T13:23:11.910112+0000","last_fullsized":"2024-09-13T13:23:11.910112+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:48.926852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:48.926852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:56.884810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":450,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910602+0000","last_change":"2024-09-13T13:22:21.135547+0000","last_active":"2024-09-13T13:23:11.910602+0000","last_peered":"2024-09-13T13:23:11.910602+0000","last_clean":"2024-09-13T13:23:11.910602+0000","last_became_active":"2024-09-13T13:22:16.229533+0000","last_became_peered":"2024-09-13T13:22:16.229533+0000","last_unstale":"2024-09-13T13:23:11.910602+0000","last_undegraded":"2024-09-13T13:23:11.910602+0000","last_fullsized":"2024-09-13T13:23:11.910602+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:22:21.135494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.135494+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:31.996151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1103,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911103+0000","last_change":"2024-09-13T13:22:22.393258+0000","last_active":"2024-09-13T13:23:11.911103+0000","last_peered":"2024-09-13T13:23:11.911103+0000","last_clean":"2024-09-13T13:23:11.911103+0000","last_became_active":"2024-09-13T13:22:19.182768+0000","last_became_peered":"2024-09-13T13:22:19.182768+0000","last_unstale":"2024-09-13T13:23:11.911103+0000","last_undegraded":"2024-09-13T13:23:11.911103+0000","last_fullsized":"2024-09-13T13:23:11.911103+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:22.393212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:22.393212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:29.582027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"500'39","reported_seq":354,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:12.085460+0000","last_change":"2024-09-13T13:23:12.085460+0000","last_active":"2024-09-13T13:23:12.085460+0000","last_peered":"2024-09-13T13:23:12.085460+0000","last_clean":"2024-09-13T13:23:12.085460+0000","last_became_active":"2024-09-13T13:17:25.245658+0000","last_became_peered":"2024-09-13T13:17:25.245658+0000","last_unstale":"2024-09-13T13:23:12.085460+0000","last_undegraded":"2024-09-13T13:23:12.085460+0000","last_fullsized":"2024-09-13T13:23:12.085460+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'39","last_scrub_stamp":"2024-09-13T13:23:12.085392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:23:12.085392+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:39.522471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1098,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910952+0000","last_change":"2024-09-13T13:22:35.161005+0000","last_active":"2024-09-13T13:23:11.910952+0000","last_peered":"2024-09-13T13:23:11.910952+0000","last_clean":"2024-09-13T13:23:11.910952+0000","last_became_active":"2024-09-13T13:22:16.236884+0000","last_became_peered":"2024-09-13T13:22:16.236884+0000","last_unstale":"2024-09-13T13:23:11.910952+0000","last_undegraded":"2024-09-13T13:23:11.910952+0000","last_fullsized":"2024-09-13T13:23:11.910952+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:35.160926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:35.160926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:57.173319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1050,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643593+0000","last_change":"2024-09-13T13:22:48.411884+0000","last_active":"2024-09-13T13:23:11.643593+0000","last_peered":"2024-09-13T13:23:11.643593+0000","last_clean":"2024-09-13T13:23:11.643593+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:23:11.643593+0000","last_undegraded":"2024-09-13T13:23:11.643593+0000","last_fullsized":"2024-09-13T13:23:11.643593+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:48.411837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:48.411837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:17.222365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":488,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911561+0000","last_change":"2024-09-13T13:22:47.320545+0000","last_active":"2024-09-13T13:23:11.911561+0000","last_peered":"2024-09-13T13:23:11.911561+0000","last_clean":"2024-09-13T13:23:11.911561+0000","last_became_active":"2024-09-13T13:22:16.231331+0000","last_became_peered":"2024-09-13T13:22:16.231331+0000","last_unstale":"2024-09-13T13:23:11.911561+0000","last_undegraded":"2024-09-13T13:23:11.911561+0000","last_fullsized":"2024-09-13T13:23:11.911561+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:47.320485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:47.320485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:07.855412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"500'24","reported_seq":1154,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643334+0000","last_change":"2024-09-13T13:22:20.325637+0000","last_active":"2024-09-13T13:23:11.643334+0000","last_peered":"2024-09-13T13:23:11.643334+0000","last_clean":"2024-09-13T13:23:11.643334+0000","last_became_active":"2024-09-13T13:22:16.230459+0000","last_became_peered":"2024-09-13T13:22:16.230459+0000","last_unstale":"2024-09-13T13:23:11.643334+0000","last_undegraded":"2024-09-13T13:23:11.643334+0000","last_fullsized":"2024-09-13T13:23:11.643334+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'24","last_scrub_stamp":"2024-09-13T13:22:20.325574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:20.325574+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:46.191688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":471,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911641+0000","last_change":"2024-09-13T13:22:38.163671+0000","last_active":"2024-09-13T13:23:11.911641+0000","last_peered":"2024-09-13T13:23:11.911641+0000","last_clean":"2024-09-13T13:23:11.911641+0000","last_became_active":"2024-09-13T13:22:16.233955+0000","last_became_peered":"2024-09-13T13:22:16.233955+0000","last_unstale":"2024-09-13T13:23:11.911641+0000","last_undegraded":"2024-09-13T13:23:11.911641+0000","last_fullsized":"2024-09-13T13:23:11.911641+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:38.163586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:38.163586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:54.313246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":574,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911509+0000","last_change":"2024-09-13T13:22:47.527948+0000","last_active":"2024-09-13T13:23:11.911509+0000","last_peered":"2024-09-13T13:23:11.911509+0000","last_clean":"2024-09-13T13:23:11.911509+0000","last_became_active":"2024-09-13T13:22:47.527459+0000","last_became_peered":"2024-09-13T13:22:47.527459+0000","last_unstale":"2024-09-13T13:23:11.911509+0000","last_undegraded":"2024-09-13T13:23:11.911509+0000","last_fullsized":"2024-09-13T13:23:11.911509+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:21:54.619079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:21:54.619079+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:15.107939+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":1092,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909973+0000","last_change":"2024-09-13T13:22:23.999443+0000","last_active":"2024-09-13T13:23:11.909973+0000","last_peered":"2024-09-13T13:23:11.909973+0000","last_clean":"2024-09-13T13:23:11.909973+0000","last_became_active":"2024-09-13T13:22:16.234943+0000","last_became_peered":"2024-09-13T13:22:16.234943+0000","last_unstale":"2024-09-13T13:23:11.909973+0000","last_undegraded":"2024-09-13T13:23:11.909973+0000","last_fullsized":"2024-09-13T13:23:11.909973+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:22:23.999388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:23.999388+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:49.113211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"500'30","reported_seq":1162,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910941+0000","last_change":"2024-09-13T13:22:52.124180+0000","last_active":"2024-09-13T13:23:11.910941+0000","last_peered":"2024-09-13T13:23:11.910941+0000","last_clean":"2024-09-13T13:23:11.910941+0000","last_became_active":"2024-09-13T13:22:19.186550+0000","last_became_peered":"2024-09-13T13:22:19.186550+0000","last_unstale":"2024-09-13T13:23:11.910941+0000","last_undegraded":"2024-09-13T13:23:11.910941+0000","last_fullsized":"2024-09-13T13:23:11.910941+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'30","last_scrub_stamp":"2024-09-13T13:22:52.124061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:52.124061+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:11.147858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"527'16","reported_seq":1342,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.648653+0000","last_change":"2024-09-13T13:22:29.517702+0000","last_active":"2024-09-13T13:23:11.648653+0000","last_peered":"2024-09-13T13:23:11.648653+0000","last_clean":"2024-09-13T13:23:11.648653+0000","last_became_active":"2024-09-13T13:22:19.182105+0000","last_became_peered":"2024-09-13T13:22:19.182105+0000","last_unstale":"2024-09-13T13:23:11.648653+0000","last_undegraded":"2024-09-13T13:23:11.648653+0000","last_fullsized":"2024-09-13T13:23:11.648653+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'16","last_scrub_stamp":"2024-09-13T13:22:29.517651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:29.517651+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:44.363666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":304,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910293+0000","last_change":"2024-09-13T13:22:56.902357+0000","last_active":"2024-09-13T13:23:11.910293+0000","last_peered":"2024-09-13T13:23:11.910293+0000","last_clean":"2024-09-13T13:23:11.910293+0000","last_became_active":"2024-09-13T13:17:49.052966+0000","last_became_peered":"2024-09-13T13:17:49.052966+0000","last_unstale":"2024-09-13T13:23:11.910293+0000","last_undegraded":"2024-09-13T13:23:11.910293+0000","last_fullsized":"2024-09-13T13:23:11.910293+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:56.902076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:56.902076+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:09.481046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"500'39","reported_seq":1188,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911313+0000","last_change":"2024-09-13T13:22:21.990924+0000","last_active":"2024-09-13T13:23:11.911313+0000","last_peered":"2024-09-13T13:23:11.911313+0000","last_clean":"2024-09-13T13:23:11.911313+0000","last_became_active":"2024-09-13T13:22:16.238592+0000","last_became_peered":"2024-09-13T13:22:16.238592+0000","last_unstale":"2024-09-13T13:23:11.911313+0000","last_undegraded":"2024-09-13T13:23:11.911313+0000","last_fullsized":"2024-09-13T13:23:11.911313+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'39","last_scrub_stamp":"2024-09-13T13:22:21.990869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.990869+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:45.280437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1108,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643203+0000","last_change":"2024-09-13T13:22:30.391264+0000","last_active":"2024-09-13T13:23:11.643203+0000","last_peered":"2024-09-13T13:23:11.643203+0000","last_clean":"2024-09-13T13:23:11.643203+0000","last_became_active":"2024-09-13T13:22:19.178783+0000","last_became_peered":"2024-09-13T13:22:19.178783+0000","last_unstale":"2024-09-13T13:23:11.643203+0000","last_undegraded":"2024-09-13T13:23:11.643203+0000","last_fullsized":"2024-09-13T13:23:11.643203+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:30.391215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:30.391215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:50.744920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"524'12","reported_seq":584,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910541+0000","last_change":"2024-09-13T13:22:16.240555+0000","last_active":"2024-09-13T13:23:11.910541+0000","last_peered":"2024-09-13T13:23:11.910541+0000","last_clean":"2024-09-13T13:23:11.910541+0000","last_became_active":"2024-09-13T13:22:16.239504+0000","last_became_peered":"2024-09-13T13:22:16.239504+0000","last_unstale":"2024-09-13T13:23:11.910541+0000","last_undegraded":"2024-09-13T13:23:11.910541+0000","last_fullsized":"2024-09-13T13:23:11.910541+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:20:51.899778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:51.899778+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1053,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911092+0000","last_change":"2024-09-13T13:22:46.942292+0000","last_active":"2024-09-13T13:23:11.911092+0000","last_peered":"2024-09-13T13:23:11.911092+0000","last_clean":"2024-09-13T13:23:11.911092+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:23:11.911092+0000","last_undegraded":"2024-09-13T13:23:11.911092+0000","last_fullsized":"2024-09-13T13:23:11.911092+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:46.942232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:46.942232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:50.825260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":235,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911840+0000","last_change":"2024-09-13T13:22:17.195980+0000","last_active":"2024-09-13T13:23:11.911840+0000","last_peered":"2024-09-13T13:23:11.911840+0000","last_clean":"2024-09-13T13:23:11.911840+0000","last_became_active":"2024-09-13T13:22:16.240045+0000","last_became_peered":"2024-09-13T13:22:16.240045+0000","last_unstale":"2024-09-13T13:23:11.911840+0000","last_undegraded":"2024-09-13T13:23:11.911840+0000","last_fullsized":"2024-09-13T13:23:11.911840+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:17.195931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:17.195931+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:43.916437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"500'42","reported_seq":1158,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643154+0000","last_change":"2024-09-13T13:22:19.212922+0000","last_active":"2024-09-13T13:23:11.643154+0000","last_peered":"2024-09-13T13:23:11.643154+0000","last_clean":"2024-09-13T13:23:11.643154+0000","last_became_active":"2024-09-13T13:22:19.180137+0000","last_became_peered":"2024-09-13T13:22:19.180137+0000","last_unstale":"2024-09-13T13:23:11.643154+0000","last_undegraded":"2024-09-13T13:23:11.643154+0000","last_fullsized":"2024-09-13T13:23:11.643154+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'42","last_scrub_stamp":"2024-09-13T13:21:54.255372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:21:54.255372+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":1071,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643798+0000","last_change":"2024-09-13T13:22:19.193027+0000","last_active":"2024-09-13T13:23:11.643798+0000","last_peered":"2024-09-13T13:23:11.643798+0000","last_clean":"2024-09-13T13:23:11.643798+0000","last_became_active":"2024-09-13T13:22:19.176881+0000","last_became_peered":"2024-09-13T13:22:19.176881+0000","last_unstale":"2024-09-13T13:23:11.643798+0000","last_undegraded":"2024-09-13T13:23:11.643798+0000","last_fullsized":"2024-09-13T13:23:11.643798+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:21:25.329252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:21:25.329252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":443,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910860+0000","last_change":"2024-09-13T13:22:30.125943+0000","last_active":"2024-09-13T13:23:11.910860+0000","last_peered":"2024-09-13T13:23:11.910860+0000","last_clean":"2024-09-13T13:23:11.910860+0000","last_became_active":"2024-09-13T13:22:19.182770+0000","last_became_peered":"2024-09-13T13:22:19.182770+0000","last_unstale":"2024-09-13T13:23:11.910860+0000","last_undegraded":"2024-09-13T13:23:11.910860+0000","last_fullsized":"2024-09-13T13:23:11.910860+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:30.125880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:30.125880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:42.523997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":451,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911584+0000","last_change":"2024-09-13T13:22:27.175549+0000","last_active":"2024-09-13T13:23:11.911584+0000","last_peered":"2024-09-13T13:23:11.911584+0000","last_clean":"2024-09-13T13:23:11.911584+0000","last_became_active":"2024-09-13T13:22:16.237053+0000","last_became_peered":"2024-09-13T13:22:16.237053+0000","last_unstale":"2024-09-13T13:23:11.911584+0000","last_undegraded":"2024-09-13T13:23:11.911584+0000","last_fullsized":"2024-09-13T13:23:11.911584+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:27.175477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:27.175477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:50.938997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":327,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642206+0000","last_change":"2024-09-13T13:22:23.390156+0000","last_active":"2024-09-13T13:23:11.642206+0000","last_peered":"2024-09-13T13:23:11.642206+0000","last_clean":"2024-09-13T13:23:11.642206+0000","last_became_active":"2024-09-13T13:22:19.184359+0000","last_became_peered":"2024-09-13T13:22:19.184359+0000","last_unstale":"2024-09-13T13:23:11.642206+0000","last_undegraded":"2024-09-13T13:23:11.642206+0000","last_fullsized":"2024-09-13T13:23:11.642206+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:23.390102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:23.390102+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:31.322915+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"500'42","reported_seq":532,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910956+0000","last_change":"2024-09-13T13:23:10.137666+0000","last_active":"2024-09-13T13:23:11.910956+0000","last_peered":"2024-09-13T13:23:11.910956+0000","last_clean":"2024-09-13T13:23:11.910956+0000","last_became_active":"2024-09-13T13:22:16.227649+0000","last_became_peered":"2024-09-13T13:22:16.227649+0000","last_unstale":"2024-09-13T13:23:11.910956+0000","last_undegraded":"2024-09-13T13:23:11.910956+0000","last_fullsized":"2024-09-13T13:23:11.910956+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'42","last_scrub_stamp":"2024-09-13T13:23:10.137582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:23:10.137582+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:18.073050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":1076,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911103+0000","last_change":"2024-09-13T13:22:34.380999+0000","last_active":"2024-09-13T13:23:11.911103+0000","last_peered":"2024-09-13T13:23:11.911103+0000","last_clean":"2024-09-13T13:23:11.911103+0000","last_became_active":"2024-09-13T13:22:16.242546+0000","last_became_peered":"2024-09-13T13:22:16.242546+0000","last_unstale":"2024-09-13T13:23:11.911103+0000","last_undegraded":"2024-09-13T13:23:11.911103+0000","last_fullsized":"2024-09-13T13:23:11.911103+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:22:34.380943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:34.380943+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:01.372265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"524'12","reported_seq":1362,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643048+0000","last_change":"2024-09-13T13:22:21.325588+0000","last_active":"2024-09-13T13:23:11.643048+0000","last_peered":"2024-09-13T13:23:11.643048+0000","last_clean":"2024-09-13T13:23:11.643048+0000","last_became_active":"2024-09-13T13:22:16.230421+0000","last_became_peered":"2024-09-13T13:22:16.230421+0000","last_unstale":"2024-09-13T13:23:11.643048+0000","last_undegraded":"2024-09-13T13:23:11.643048+0000","last_fullsized":"2024-09-13T13:23:11.643048+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"524'12","last_scrub_stamp":"2024-09-13T13:22:21.325542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.325542+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:21.878400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":687,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910770+0000","last_change":"2024-09-13T13:22:47.532169+0000","last_active":"2024-09-13T13:23:11.910770+0000","last_peered":"2024-09-13T13:23:11.910770+0000","last_clean":"2024-09-13T13:23:11.910770+0000","last_became_active":"2024-09-13T13:22:47.531836+0000","last_became_peered":"2024-09-13T13:22:47.531836+0000","last_unstale":"2024-09-13T13:23:11.910770+0000","last_undegraded":"2024-09-13T13:23:11.910770+0000","last_fullsized":"2024-09-13T13:23:11.910770+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:14.659140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:14.659140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:43.244378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":403,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.648461+0000","last_change":"2024-09-13T13:23:08.273753+0000","last_active":"2024-09-13T13:23:11.648461+0000","last_peered":"2024-09-13T13:23:11.648461+0000","last_clean":"2024-09-13T13:23:11.648461+0000","last_became_active":"2024-09-13T13:22:19.177690+0000","last_became_peered":"2024-09-13T13:22:19.177690+0000","last_unstale":"2024-09-13T13:23:11.648461+0000","last_undegraded":"2024-09-13T13:23:11.648461+0000","last_fullsized":"2024-09-13T13:23:11.648461+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:23:08.273687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:23:08.273687+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:19.600171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"500'30","reported_seq":1141,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911059+0000","last_change":"2024-09-13T13:22:25.004419+0000","last_active":"2024-09-13T13:23:11.911059+0000","last_peered":"2024-09-13T13:23:11.911059+0000","last_clean":"2024-09-13T13:23:11.911059+0000","last_became_active":"2024-09-13T13:22:16.238614+0000","last_became_peered":"2024-09-13T13:22:16.238614+0000","last_unstale":"2024-09-13T13:23:11.911059+0000","last_undegraded":"2024-09-13T13:23:11.911059+0000","last_fullsized":"2024-09-13T13:23:11.911059+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'30","last_scrub_stamp":"2024-09-13T13:22:25.004346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:25.004346+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:42.170769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1096,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911672+0000","last_change":"2024-09-13T13:22:36.832637+0000","last_active":"2024-09-13T13:23:11.911672+0000","last_peered":"2024-09-13T13:23:11.911672+0000","last_clean":"2024-09-13T13:23:11.911672+0000","last_became_active":"2024-09-13T13:17:25.331166+0000","last_became_peered":"2024-09-13T13:17:25.331166+0000","last_unstale":"2024-09-13T13:23:11.911672+0000","last_undegraded":"2024-09-13T13:23:11.911672+0000","last_fullsized":"2024-09-13T13:23:11.911672+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:36.832583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:36.832583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:38.268212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":1339,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910663+0000","last_change":"2024-09-13T13:22:43.264022+0000","last_active":"2024-09-13T13:23:11.910663+0000","last_peered":"2024-09-13T13:23:11.910663+0000","last_clean":"2024-09-13T13:23:11.910663+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:23:11.910663+0000","last_undegraded":"2024-09-13T13:23:11.910663+0000","last_fullsized":"2024-09-13T13:23:11.910663+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:22:43.263941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:43.263941+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:43.990589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1093,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909791+0000","last_change":"2024-09-13T13:22:51.158255+0000","last_active":"2024-09-13T13:23:11.909791+0000","last_peered":"2024-09-13T13:23:11.909791+0000","last_clean":"2024-09-13T13:23:11.909791+0000","last_became_active":"2024-09-13T13:17:25.252712+0000","last_became_peered":"2024-09-13T13:17:25.252712+0000","last_unstale":"2024-09-13T13:23:11.909791+0000","last_undegraded":"2024-09-13T13:23:11.909791+0000","last_fullsized":"2024-09-13T13:23:11.909791+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:51.158206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:51.158206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:14.705773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":389,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:15.303949+0000","last_change":"2024-09-13T13:23:15.303949+0000","last_active":"2024-09-13T13:23:15.303949+0000","last_peered":"2024-09-13T13:23:15.303949+0000","last_clean":"2024-09-13T13:23:15.303949+0000","last_became_active":"2024-09-13T13:17:49.055888+0000","last_became_peered":"2024-09-13T13:17:49.055888+0000","last_unstale":"2024-09-13T13:23:15.303949+0000","last_undegraded":"2024-09-13T13:23:15.303949+0000","last_fullsized":"2024-09-13T13:23:15.303949+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:23:15.303887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:23:15.303887+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:37.788183+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"500'39","reported_seq":1194,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910861+0000","last_change":"2024-09-13T13:22:46.261774+0000","last_active":"2024-09-13T13:23:11.910861+0000","last_peered":"2024-09-13T13:23:11.910861+0000","last_clean":"2024-09-13T13:23:11.910861+0000","last_became_active":"2024-09-13T13:22:19.182252+0000","last_became_peered":"2024-09-13T13:22:19.182252+0000","last_unstale":"2024-09-13T13:23:11.910861+0000","last_undegraded":"2024-09-13T13:23:11.910861+0000","last_fullsized":"2024-09-13T13:23:11.910861+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'39","last_scrub_stamp":"2024-09-13T13:22:46.261717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:46.261717+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:51.517823+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1073,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910523+0000","last_change":"2024-09-13T13:22:08.998496+0000","last_active":"2024-09-13T13:23:11.910523+0000","last_peered":"2024-09-13T13:23:11.910523+0000","last_clean":"2024-09-13T13:23:11.910523+0000","last_became_active":"2024-09-13T13:17:25.332589+0000","last_became_peered":"2024-09-13T13:17:25.332589+0000","last_unstale":"2024-09-13T13:23:11.910523+0000","last_undegraded":"2024-09-13T13:23:11.910523+0000","last_fullsized":"2024-09-13T13:23:11.910523+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:08.998430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:08.998430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:27.122610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1044,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.647962+0000","last_change":"2024-09-13T13:23:10.224215+0000","last_active":"2024-09-13T13:23:11.647962+0000","last_peered":"2024-09-13T13:23:11.647962+0000","last_clean":"2024-09-13T13:23:11.647962+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:23:11.647962+0000","last_undegraded":"2024-09-13T13:23:11.647962+0000","last_fullsized":"2024-09-13T13:23:11.647962+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:10.224158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:23:10.224158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:15.429296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1080,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909675+0000","last_change":"2024-09-13T13:22:03.982427+0000","last_active":"2024-09-13T13:23:11.909675+0000","last_peered":"2024-09-13T13:23:11.909675+0000","last_clean":"2024-09-13T13:23:11.909675+0000","last_became_active":"2024-09-13T13:17:25.244517+0000","last_became_peered":"2024-09-13T13:17:25.244517+0000","last_unstale":"2024-09-13T13:23:11.909675+0000","last_undegraded":"2024-09-13T13:23:11.909675+0000","last_fullsized":"2024-09-13T13:23:11.909675+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:03.982366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:03.982366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:23.535566+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":304,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909888+0000","last_change":"2024-09-13T13:23:02.862296+0000","last_active":"2024-09-13T13:23:11.909888+0000","last_peered":"2024-09-13T13:23:11.909888+0000","last_clean":"2024-09-13T13:23:11.909888+0000","last_became_active":"2024-09-13T13:17:49.054449+0000","last_became_peered":"2024-09-13T13:17:49.054449+0000","last_unstale":"2024-09-13T13:23:11.909888+0000","last_undegraded":"2024-09-13T13:23:11.909888+0000","last_fullsized":"2024-09-13T13:23:11.909888+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:23:02.862230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:23:02.862230+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:12.888603+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"500'55","reported_seq":344,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910419+0000","last_change":"2024-09-13T13:22:36.396551+0000","last_active":"2024-09-13T13:23:11.910419+0000","last_peered":"2024-09-13T13:23:11.910419+0000","last_clean":"2024-09-13T13:23:11.910419+0000","last_became_active":"2024-09-13T13:18:35.244311+0000","last_became_peered":"2024-09-13T13:18:35.244311+0000","last_unstale":"2024-09-13T13:23:11.910419+0000","last_undegraded":"2024-09-13T13:23:11.910419+0000","last_fullsized":"2024-09-13T13:23:11.910419+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'55","last_scrub_stamp":"2024-09-13T13:22:36.396487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:36.396487+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:59.662802+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":505,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910363+0000","last_change":"2024-09-13T13:22:16.239128+0000","last_active":"2024-09-13T13:23:11.910363+0000","last_peered":"2024-09-13T13:23:11.910363+0000","last_clean":"2024-09-13T13:23:11.910363+0000","last_became_active":"2024-09-13T13:22:16.238858+0000","last_became_peered":"2024-09-13T13:22:16.238858+0000","last_unstale":"2024-09-13T13:23:11.910363+0000","last_undegraded":"2024-09-13T13:23:11.910363+0000","last_fullsized":"2024-09-13T13:23:11.910363+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:21:24.103840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:21:24.103840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":1069,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910809+0000","last_change":"2024-09-13T13:22:17.722161+0000","last_active":"2024-09-13T13:23:11.910809+0000","last_peered":"2024-09-13T13:23:11.910809+0000","last_clean":"2024-09-13T13:23:11.910809+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:23:11.910809+0000","last_undegraded":"2024-09-13T13:23:11.910809+0000","last_fullsized":"2024-09-13T13:23:11.910809+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:22:17.722104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:17.722104+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:28.690048+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1083,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910200+0000","last_change":"2024-09-13T13:22:27.400902+0000","last_active":"2024-09-13T13:23:11.910200+0000","last_peered":"2024-09-13T13:23:11.910200+0000","last_clean":"2024-09-13T13:23:11.910200+0000","last_became_active":"2024-09-13T13:22:16.244949+0000","last_became_peered":"2024-09-13T13:22:16.244949+0000","last_unstale":"2024-09-13T13:23:11.910200+0000","last_undegraded":"2024-09-13T13:23:11.910200+0000","last_fullsized":"2024-09-13T13:23:11.910200+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:27.400861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:27.400861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:45.218687+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1094,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643500+0000","last_change":"2024-09-13T13:22:46.399851+0000","last_active":"2024-09-13T13:23:11.643500+0000","last_peered":"2024-09-13T13:23:11.643500+0000","last_clean":"2024-09-13T13:23:11.643500+0000","last_became_active":"2024-09-13T13:22:19.183707+0000","last_became_peered":"2024-09-13T13:22:19.183707+0000","last_unstale":"2024-09-13T13:23:11.643500+0000","last_undegraded":"2024-09-13T13:23:11.643500+0000","last_fullsized":"2024-09-13T13:23:11.643500+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:46.399808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:46.399808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:08.382448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":646,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910345+0000","last_change":"2024-09-13T13:22:47.534825+0000","last_active":"2024-09-13T13:23:11.910345+0000","last_peered":"2024-09-13T13:23:11.910345+0000","last_clean":"2024-09-13T13:23:11.910345+0000","last_became_active":"2024-09-13T13:22:47.534458+0000","last_became_peered":"2024-09-13T13:22:47.534458+0000","last_unstale":"2024-09-13T13:23:11.910345+0000","last_undegraded":"2024-09-13T13:23:11.910345+0000","last_fullsized":"2024-09-13T13:23:11.910345+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:21.394085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.394085+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:24.520621+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"500'21","reported_seq":1094,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911372+0000","last_change":"2024-09-13T13:21:54.845235+0000","last_active":"2024-09-13T13:23:11.911372+0000","last_peered":"2024-09-13T13:23:11.911372+0000","last_clean":"2024-09-13T13:23:11.911372+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:23:11.911372+0000","last_undegraded":"2024-09-13T13:23:11.911372+0000","last_fullsized":"2024-09-13T13:23:11.911372+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:21:54.845177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:21:54.845177+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:13.333386+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1093,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909555+0000","last_change":"2024-09-13T13:22:04.995555+0000","last_active":"2024-09-13T13:23:11.909555+0000","last_peered":"2024-09-13T13:23:11.909555+0000","last_clean":"2024-09-13T13:23:11.909555+0000","last_became_active":"2024-09-13T13:17:25.246400+0000","last_became_peered":"2024-09-13T13:17:25.246400+0000","last_unstale":"2024-09-13T13:23:11.909555+0000","last_undegraded":"2024-09-13T13:23:11.909555+0000","last_fullsized":"2024-09-13T13:23:11.909555+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:04.995492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:04.995492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:14.088299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":316,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642384+0000","last_change":"2024-09-13T13:22:13.952594+0000","last_active":"2024-09-13T13:23:11.642384+0000","last_peered":"2024-09-13T13:23:11.642384+0000","last_clean":"2024-09-13T13:23:11.642384+0000","last_became_active":"2024-09-13T13:17:25.252664+0000","last_became_peered":"2024-09-13T13:17:25.252664+0000","last_unstale":"2024-09-13T13:23:11.642384+0000","last_undegraded":"2024-09-13T13:23:11.642384+0000","last_fullsized":"2024-09-13T13:23:11.642384+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:22:13.952523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:13.952523+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:42.420034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":345,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642322+0000","last_change":"2024-09-13T13:22:19.182229+0000","last_active":"2024-09-13T13:23:11.642322+0000","last_peered":"2024-09-13T13:23:11.642322+0000","last_clean":"2024-09-13T13:23:11.642322+0000","last_became_active":"2024-09-13T13:22:19.181898+0000","last_became_peered":"2024-09-13T13:22:19.181898+0000","last_unstale":"2024-09-13T13:23:11.642322+0000","last_undegraded":"2024-09-13T13:23:11.642322+0000","last_fullsized":"2024-09-13T13:23:11.642322+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:18.014751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:18.014751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:24.442038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":657,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911237+0000","last_change":"2024-09-13T13:22:47.532518+0000","last_active":"2024-09-13T13:23:11.911237+0000","last_peered":"2024-09-13T13:23:11.911237+0000","last_clean":"2024-09-13T13:23:11.911237+0000","last_became_active":"2024-09-13T13:22:47.532197+0000","last_became_peered":"2024-09-13T13:22:47.532197+0000","last_unstale":"2024-09-13T13:23:11.911237+0000","last_undegraded":"2024-09-13T13:23:11.911237+0000","last_fullsized":"2024-09-13T13:23:11.911237+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:37.494471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:37.494471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:48.751049+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":307,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910484+0000","last_change":"2024-09-13T13:22:16.984074+0000","last_active":"2024-09-13T13:23:11.910484+0000","last_peered":"2024-09-13T13:23:11.910484+0000","last_clean":"2024-09-13T13:23:11.910484+0000","last_became_active":"2024-09-13T13:22:16.237260+0000","last_became_peered":"2024-09-13T13:22:16.237260+0000","last_unstale":"2024-09-13T13:23:11.910484+0000","last_undegraded":"2024-09-13T13:23:11.910484+0000","last_fullsized":"2024-09-13T13:23:11.910484+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:16.984009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:16.984009+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:34.249874+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"500'36","reported_seq":1150,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910894+0000","last_change":"2024-09-13T13:22:20.418575+0000","last_active":"2024-09-13T13:23:11.910894+0000","last_peered":"2024-09-13T13:23:11.910894+0000","last_clean":"2024-09-13T13:23:11.910894+0000","last_became_active":"2024-09-13T13:17:25.253120+0000","last_became_peered":"2024-09-13T13:17:25.253120+0000","last_unstale":"2024-09-13T13:23:11.910894+0000","last_undegraded":"2024-09-13T13:23:11.910894+0000","last_fullsized":"2024-09-13T13:23:11.910894+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'36","last_scrub_stamp":"2024-09-13T13:22:20.418475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:20.418475+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:42.537200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1091,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910198+0000","last_change":"2024-09-13T13:22:40.338901+0000","last_active":"2024-09-13T13:23:11.910198+0000","last_peered":"2024-09-13T13:23:11.910198+0000","last_clean":"2024-09-13T13:23:11.910198+0000","last_became_active":"2024-09-13T13:22:16.243256+0000","last_became_peered":"2024-09-13T13:22:16.243256+0000","last_unstale":"2024-09-13T13:23:11.910198+0000","last_undegraded":"2024-09-13T13:23:11.910198+0000","last_fullsized":"2024-09-13T13:23:11.910198+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:40.338852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:40.338852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:49.459516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1184,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912214+0000","last_change":"2024-09-13T13:22:47.534049+0000","last_active":"2024-09-13T13:23:11.912214+0000","last_peered":"2024-09-13T13:23:11.912214+0000","last_clean":"2024-09-13T13:23:11.912214+0000","last_became_active":"2024-09-13T13:22:47.533679+0000","last_became_peered":"2024-09-13T13:22:47.533679+0000","last_unstale":"2024-09-13T13:23:11.912214+0000","last_undegraded":"2024-09-13T13:23:11.912214+0000","last_fullsized":"2024-09-13T13:23:11.912214+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:21:59.909697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:21:59.909697+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":603,"num_read_kb":598,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1042,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911318+0000","last_change":"2024-09-13T13:22:18.751296+0000","last_active":"2024-09-13T13:23:11.911318+0000","last_peered":"2024-09-13T13:23:11.911318+0000","last_clean":"2024-09-13T13:23:11.911318+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:23:11.911318+0000","last_undegraded":"2024-09-13T13:23:11.911318+0000","last_fullsized":"2024-09-13T13:23:11.911318+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:18.751227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:18.751227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:32.619966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1047,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910115+0000","last_change":"2024-09-13T13:22:16.014224+0000","last_active":"2024-09-13T13:23:11.910115+0000","last_peered":"2024-09-13T13:23:11.910115+0000","last_clean":"2024-09-13T13:23:11.910115+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:23:11.910115+0000","last_undegraded":"2024-09-13T13:23:11.910115+0000","last_fullsized":"2024-09-13T13:23:11.910115+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:16.014163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:16.014163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:38.568946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"527'48","reported_seq":1346,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.648350+0000","last_change":"2024-09-13T13:22:24.618283+0000","last_active":"2024-09-13T13:23:11.648350+0000","last_peered":"2024-09-13T13:23:11.648350+0000","last_clean":"2024-09-13T13:23:11.648350+0000","last_became_active":"2024-09-13T13:22:19.182848+0000","last_became_peered":"2024-09-13T13:22:19.182848+0000","last_unstale":"2024-09-13T13:23:11.648350+0000","last_undegraded":"2024-09-13T13:23:11.648350+0000","last_fullsized":"2024-09-13T13:23:11.648350+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'48","last_scrub_stamp":"2024-09-13T13:22:24.618210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:24.618210+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:52.030178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1062,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642479+0000","last_change":"2024-09-13T13:21:30.281525+0000","last_active":"2024-09-13T13:23:11.642479+0000","last_peered":"2024-09-13T13:23:11.642479+0000","last_clean":"2024-09-13T13:23:11.642479+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:23:11.642479+0000","last_undegraded":"2024-09-13T13:23:11.642479+0000","last_fullsized":"2024-09-13T13:23:11.642479+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:21:30.281357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:21:30.281357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":632,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912016+0000","last_change":"2024-09-13T13:22:47.533933+0000","last_active":"2024-09-13T13:23:11.912016+0000","last_peered":"2024-09-13T13:23:11.912016+0000","last_clean":"2024-09-13T13:23:11.912016+0000","last_became_active":"2024-09-13T13:22:47.533570+0000","last_became_peered":"2024-09-13T13:22:47.533570+0000","last_unstale":"2024-09-13T13:23:11.912016+0000","last_undegraded":"2024-09-13T13:23:11.912016+0000","last_fullsized":"2024-09-13T13:23:11.912016+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:25.340401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:25.340401+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:30.901220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":305,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911416+0000","last_change":"2024-09-13T13:22:47.532580+0000","last_active":"2024-09-13T13:23:11.911416+0000","last_peered":"2024-09-13T13:23:11.911416+0000","last_clean":"2024-09-13T13:23:11.911416+0000","last_became_active":"2024-09-13T13:22:47.531816+0000","last_became_peered":"2024-09-13T13:22:47.531816+0000","last_unstale":"2024-09-13T13:23:11.911416+0000","last_undegraded":"2024-09-13T13:23:11.911416+0000","last_fullsized":"2024-09-13T13:23:11.911416+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:14.978063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:14.978063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:17.499392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1103,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911535+0000","last_change":"2024-09-13T13:22:25.669469+0000","last_active":"2024-09-13T13:23:11.911535+0000","last_peered":"2024-09-13T13:23:11.911535+0000","last_clean":"2024-09-13T13:23:11.911535+0000","last_became_active":"2024-09-13T13:22:16.234810+0000","last_became_peered":"2024-09-13T13:22:16.234810+0000","last_unstale":"2024-09-13T13:23:11.911535+0000","last_undegraded":"2024-09-13T13:23:11.911535+0000","last_fullsized":"2024-09-13T13:23:11.911535+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:25.669424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:25.669424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:54.213701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"500'27","reported_seq":527,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910318+0000","last_change":"2024-09-13T13:22:36.131450+0000","last_active":"2024-09-13T13:23:11.910318+0000","last_peered":"2024-09-13T13:23:11.910318+0000","last_clean":"2024-09-13T13:23:11.910318+0000","last_became_active":"2024-09-13T13:22:16.230526+0000","last_became_peered":"2024-09-13T13:22:16.230526+0000","last_unstale":"2024-09-13T13:23:11.910318+0000","last_undegraded":"2024-09-13T13:23:11.910318+0000","last_fullsized":"2024-09-13T13:23:11.910318+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'27","last_scrub_stamp":"2024-09-13T13:22:36.131368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:36.131368+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:54.686004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":524,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911127+0000","last_change":"2024-09-13T13:22:19.181749+0000","last_active":"2024-09-13T13:23:11.911127+0000","last_peered":"2024-09-13T13:23:11.911127+0000","last_clean":"2024-09-13T13:23:11.911127+0000","last_became_active":"2024-09-13T13:22:19.181452+0000","last_became_peered":"2024-09-13T13:22:19.181452+0000","last_unstale":"2024-09-13T13:23:11.911127+0000","last_undegraded":"2024-09-13T13:23:11.911127+0000","last_fullsized":"2024-09-13T13:23:11.911127+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:21:23.132552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:21:23.132552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1109,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911660+0000","last_change":"2024-09-13T13:22:21.702328+0000","last_active":"2024-09-13T13:23:11.911660+0000","last_peered":"2024-09-13T13:23:11.911660+0000","last_clean":"2024-09-13T13:23:11.911660+0000","last_became_active":"2024-09-13T13:22:16.234609+0000","last_became_peered":"2024-09-13T13:22:16.234609+0000","last_unstale":"2024-09-13T13:23:11.911660+0000","last_undegraded":"2024-09-13T13:23:11.911660+0000","last_fullsized":"2024-09-13T13:23:11.911660+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:21.702274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.702274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:49.240755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1049,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643166+0000","last_change":"2024-09-13T13:22:22.360530+0000","last_active":"2024-09-13T13:23:11.643166+0000","last_peered":"2024-09-13T13:23:11.643166+0000","last_clean":"2024-09-13T13:23:11.643166+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:23:11.643166+0000","last_undegraded":"2024-09-13T13:23:11.643166+0000","last_fullsized":"2024-09-13T13:23:11.643166+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:22.360486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:22.360486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:30.852654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":305,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909661+0000","last_change":"2024-09-13T13:22:26.695747+0000","last_active":"2024-09-13T13:23:11.909661+0000","last_peered":"2024-09-13T13:23:11.909661+0000","last_clean":"2024-09-13T13:23:11.909661+0000","last_became_active":"2024-09-13T13:17:49.053269+0000","last_became_peered":"2024-09-13T13:17:49.053269+0000","last_unstale":"2024-09-13T13:23:11.909661+0000","last_undegraded":"2024-09-13T13:23:11.909661+0000","last_fullsized":"2024-09-13T13:23:11.909661+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:26.695675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:26.695675+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:51.057379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"500'45","reported_seq":592,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911558+0000","last_change":"2024-09-13T13:22:54.285628+0000","last_active":"2024-09-13T13:23:11.911558+0000","last_peered":"2024-09-13T13:23:11.911558+0000","last_clean":"2024-09-13T13:23:11.911558+0000","last_became_active":"2024-09-13T13:22:16.233365+0000","last_became_peered":"2024-09-13T13:22:16.233365+0000","last_unstale":"2024-09-13T13:23:11.911558+0000","last_undegraded":"2024-09-13T13:23:11.911558+0000","last_fullsized":"2024-09-13T13:23:11.911558+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'45","last_scrub_stamp":"2024-09-13T13:22:54.285538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:54.285538+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:16.096853+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1100,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.648146+0000","last_change":"2024-09-13T13:22:18.642070+0000","last_active":"2024-09-13T13:23:11.648146+0000","last_peered":"2024-09-13T13:23:11.648146+0000","last_clean":"2024-09-13T13:23:11.648146+0000","last_became_active":"2024-09-13T13:22:16.237377+0000","last_became_peered":"2024-09-13T13:22:16.237377+0000","last_unstale":"2024-09-13T13:23:11.648146+0000","last_undegraded":"2024-09-13T13:23:11.648146+0000","last_fullsized":"2024-09-13T13:23:11.648146+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:18.642020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:18.642020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:38.310216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1091,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910235+0000","last_change":"2024-09-13T13:22:53.248219+0000","last_active":"2024-09-13T13:23:11.910235+0000","last_peered":"2024-09-13T13:23:11.910235+0000","last_clean":"2024-09-13T13:23:11.910235+0000","last_became_active":"2024-09-13T13:22:16.235993+0000","last_became_peered":"2024-09-13T13:22:16.235993+0000","last_unstale":"2024-09-13T13:23:11.910235+0000","last_undegraded":"2024-09-13T13:23:11.910235+0000","last_fullsized":"2024-09-13T13:23:11.910235+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:53.248174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:53.248174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:03.348537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":815,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642476+0000","last_change":"2024-09-13T13:22:40.208908+0000","last_active":"2024-09-13T13:23:11.642476+0000","last_peered":"2024-09-13T13:23:11.642476+0000","last_clean":"2024-09-13T13:23:11.642476+0000","last_became_active":"2024-09-13T13:22:35.478558+0000","last_became_peered":"2024-09-13T13:22:35.478558+0000","last_unstale":"2024-09-13T13:23:11.642476+0000","last_undegraded":"2024-09-13T13:23:11.642476+0000","last_fullsized":"2024-09-13T13:23:11.642476+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:40.208856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:40.208856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:00.142193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":289,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912089+0000","last_change":"2024-09-13T13:23:09.248748+0000","last_active":"2024-09-13T13:23:11.912089+0000","last_peered":"2024-09-13T13:23:11.912089+0000","last_clean":"2024-09-13T13:23:11.912089+0000","last_became_active":"2024-09-13T13:22:47.528411+0000","last_became_peered":"2024-09-13T13:22:47.528411+0000","last_unstale":"2024-09-13T13:23:11.912089+0000","last_undegraded":"2024-09-13T13:23:11.912089+0000","last_fullsized":"2024-09-13T13:23:11.912089+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:23:09.248673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:23:09.248673+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:13.574733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"500'30","reported_seq":514,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911741+0000","last_change":"2024-09-13T13:22:19.131026+0000","last_active":"2024-09-13T13:23:11.911741+0000","last_peered":"2024-09-13T13:23:11.911741+0000","last_clean":"2024-09-13T13:23:11.911741+0000","last_became_active":"2024-09-13T13:22:16.235373+0000","last_became_peered":"2024-09-13T13:22:16.235373+0000","last_unstale":"2024-09-13T13:23:11.911741+0000","last_undegraded":"2024-09-13T13:23:11.911741+0000","last_fullsized":"2024-09-13T13:23:11.911741+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'30","last_scrub_stamp":"2024-09-13T13:22:19.130964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:19.130964+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:32.369797+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1067,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643020+0000","last_change":"2024-09-13T13:22:28.411364+0000","last_active":"2024-09-13T13:23:11.643020+0000","last_peered":"2024-09-13T13:23:11.643020+0000","last_clean":"2024-09-13T13:23:11.643020+0000","last_became_active":"2024-09-13T13:22:16.232309+0000","last_became_peered":"2024-09-13T13:22:16.232309+0000","last_unstale":"2024-09-13T13:23:11.643020+0000","last_undegraded":"2024-09-13T13:23:11.643020+0000","last_fullsized":"2024-09-13T13:23:11.643020+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:28.411316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:28.411316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:46.788056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"527'12","reported_seq":417,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642288+0000","last_change":"2024-09-13T13:22:21.092049+0000","last_active":"2024-09-13T13:23:11.642288+0000","last_peered":"2024-09-13T13:23:11.642288+0000","last_clean":"2024-09-13T13:23:11.642288+0000","last_became_active":"2024-09-13T13:22:19.177775+0000","last_became_peered":"2024-09-13T13:22:19.177775+0000","last_unstale":"2024-09-13T13:23:11.642288+0000","last_undegraded":"2024-09-13T13:23:11.642288+0000","last_fullsized":"2024-09-13T13:23:11.642288+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-09-13T13:22:21.091979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.091979+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:25.002440+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":478,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910853+0000","last_change":"2024-09-13T13:22:48.353671+0000","last_active":"2024-09-13T13:23:11.910853+0000","last_peered":"2024-09-13T13:23:11.910853+0000","last_clean":"2024-09-13T13:23:11.910853+0000","last_became_active":"2024-09-13T13:22:16.235741+0000","last_became_peered":"2024-09-13T13:22:16.235741+0000","last_unstale":"2024-09-13T13:23:11.910853+0000","last_undegraded":"2024-09-13T13:23:11.910853+0000","last_fullsized":"2024-09-13T13:23:11.910853+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:48.353583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:48.353583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:05.112215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"500'21","reported_seq":1108,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910723+0000","last_change":"2024-09-13T13:22:32.427139+0000","last_active":"2024-09-13T13:23:11.910723+0000","last_peered":"2024-09-13T13:23:11.910723+0000","last_clean":"2024-09-13T13:23:11.910723+0000","last_became_active":"2024-09-13T13:22:16.240384+0000","last_became_peered":"2024-09-13T13:22:16.240384+0000","last_unstale":"2024-09-13T13:23:11.910723+0000","last_undegraded":"2024-09-13T13:23:11.910723+0000","last_fullsized":"2024-09-13T13:23:11.910723+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:22:32.427082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:32.427082+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:58.305683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1076,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910885+0000","last_change":"2024-09-13T13:21:53.632799+0000","last_active":"2024-09-13T13:23:11.910885+0000","last_peered":"2024-09-13T13:23:11.910885+0000","last_clean":"2024-09-13T13:23:11.910885+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:23:11.910885+0000","last_undegraded":"2024-09-13T13:23:11.910885+0000","last_fullsized":"2024-09-13T13:23:11.910885+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:21:53.632739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:21:53.632739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:13.578212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":760,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642779+0000","last_change":"2024-09-13T13:22:35.478889+0000","last_active":"2024-09-13T13:23:11.642779+0000","last_peered":"2024-09-13T13:23:11.642779+0000","last_clean":"2024-09-13T13:23:11.642779+0000","last_became_active":"2024-09-13T13:22:35.478590+0000","last_became_peered":"2024-09-13T13:22:35.478590+0000","last_unstale":"2024-09-13T13:23:11.642779+0000","last_undegraded":"2024-09-13T13:23:11.642779+0000","last_fullsized":"2024-09-13T13:23:11.642779+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:26.187534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:26.187534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:31.475512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1074,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:14.154185+0000","last_change":"2024-09-13T13:23:14.154185+0000","last_active":"2024-09-13T13:23:14.154185+0000","last_peered":"2024-09-13T13:23:14.154185+0000","last_clean":"2024-09-13T13:23:14.154185+0000","last_became_active":"2024-09-13T13:22:19.181437+0000","last_became_peered":"2024-09-13T13:22:19.181437+0000","last_unstale":"2024-09-13T13:23:14.154185+0000","last_undegraded":"2024-09-13T13:23:14.154185+0000","last_fullsized":"2024-09-13T13:23:14.154185+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:14.154130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:23:14.154130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:23.655302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"500'51","reported_seq":1140,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.647963+0000","last_change":"2024-09-13T13:22:19.204413+0000","last_active":"2024-09-13T13:23:11.647963+0000","last_peered":"2024-09-13T13:23:11.647963+0000","last_clean":"2024-09-13T13:23:11.647963+0000","last_became_active":"2024-09-13T13:22:19.181886+0000","last_became_peered":"2024-09-13T13:22:19.181886+0000","last_unstale":"2024-09-13T13:23:11.647963+0000","last_undegraded":"2024-09-13T13:23:11.647963+0000","last_fullsized":"2024-09-13T13:23:11.647963+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'51","last_scrub_stamp":"2024-09-13T13:22:05.631182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:05.631182+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:22.090957+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":676,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912454+0000","last_change":"2024-09-13T13:22:54.301383+0000","last_active":"2024-09-13T13:23:11.912454+0000","last_peered":"2024-09-13T13:23:11.912454+0000","last_clean":"2024-09-13T13:23:11.912454+0000","last_became_active":"2024-09-13T13:22:47.532468+0000","last_became_peered":"2024-09-13T13:22:47.532468+0000","last_unstale":"2024-09-13T13:23:11.912454+0000","last_undegraded":"2024-09-13T13:23:11.912454+0000","last_fullsized":"2024-09-13T13:23:11.912454+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:54.301324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:54.301324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:03.666147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":312,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642711+0000","last_change":"2024-09-13T13:22:07.927188+0000","last_active":"2024-09-13T13:23:11.642711+0000","last_peered":"2024-09-13T13:23:11.642711+0000","last_clean":"2024-09-13T13:23:11.642711+0000","last_became_active":"2024-09-13T13:17:25.332964+0000","last_became_peered":"2024-09-13T13:17:25.332964+0000","last_unstale":"2024-09-13T13:23:11.642711+0000","last_undegraded":"2024-09-13T13:23:11.642711+0000","last_fullsized":"2024-09-13T13:23:11.642711+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:07.927129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:07.927129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:29.295379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":279,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912059+0000","last_change":"2024-09-13T13:23:06.338891+0000","last_active":"2024-09-13T13:23:11.912059+0000","last_peered":"2024-09-13T13:23:11.912059+0000","last_clean":"2024-09-13T13:23:11.912059+0000","last_became_active":"2024-09-13T13:22:47.527491+0000","last_became_peered":"2024-09-13T13:22:47.527491+0000","last_unstale":"2024-09-13T13:23:11.912059+0000","last_undegraded":"2024-09-13T13:23:11.912059+0000","last_fullsized":"2024-09-13T13:23:11.912059+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:06.338822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:23:06.338822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:35.121728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"500'24","reported_seq":516,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911019+0000","last_change":"2024-09-13T13:22:16.234267+0000","last_active":"2024-09-13T13:23:11.911019+0000","last_peered":"2024-09-13T13:23:11.911019+0000","last_clean":"2024-09-13T13:23:11.911019+0000","last_became_active":"2024-09-13T13:22:16.233929+0000","last_became_peered":"2024-09-13T13:22:16.233929+0000","last_unstale":"2024-09-13T13:23:11.911019+0000","last_undegraded":"2024-09-13T13:23:11.911019+0000","last_fullsized":"2024-09-13T13:23:11.911019+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'18","last_scrub_stamp":"2024-09-13T13:20:52.936042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:20:52.936042+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":470,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912500+0000","last_change":"2024-09-13T13:22:53.286147+0000","last_active":"2024-09-13T13:23:11.912500+0000","last_peered":"2024-09-13T13:23:11.912500+0000","last_clean":"2024-09-13T13:23:11.912500+0000","last_became_active":"2024-09-13T13:22:47.535502+0000","last_became_peered":"2024-09-13T13:22:47.535502+0000","last_unstale":"2024-09-13T13:23:11.912500+0000","last_undegraded":"2024-09-13T13:23:11.912500+0000","last_fullsized":"2024-09-13T13:23:11.912500+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:53.286090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:53.286090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:17.427121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":156,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.641892+0000","last_change":"2024-09-13T13:22:35.853140+0000","last_active":"2024-09-13T13:23:11.641892+0000","last_peered":"2024-09-13T13:23:11.641892+0000","last_clean":"2024-09-13T13:23:11.641892+0000","last_became_active":"2024-09-13T13:22:35.852785+0000","last_became_peered":"2024-09-13T13:22:35.852785+0000","last_unstale":"2024-09-13T13:23:11.641892+0000","last_undegraded":"2024-09-13T13:23:11.641892+0000","last_fullsized":"2024-09-13T13:23:11.641892+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:09.512475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:09.512475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:20.652174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1052,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642677+0000","last_change":"2024-09-13T13:22:37.448390+0000","last_active":"2024-09-13T13:23:11.642677+0000","last_peered":"2024-09-13T13:23:11.642677+0000","last_clean":"2024-09-13T13:23:11.642677+0000","last_became_active":"2024-09-13T13:22:19.171361+0000","last_became_peered":"2024-09-13T13:22:19.171361+0000","last_unstale":"2024-09-13T13:23:11.642677+0000","last_undegraded":"2024-09-13T13:23:11.642677+0000","last_fullsized":"2024-09-13T13:23:11.642677+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:37.448338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:37.448338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:46.152282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"500'39","reported_seq":1199,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909761+0000","last_change":"2024-09-13T13:22:28.725925+0000","last_active":"2024-09-13T13:23:11.909761+0000","last_peered":"2024-09-13T13:23:11.909761+0000","last_clean":"2024-09-13T13:23:11.909761+0000","last_became_active":"2024-09-13T13:22:19.170732+0000","last_became_peered":"2024-09-13T13:22:19.170732+0000","last_unstale":"2024-09-13T13:23:11.909761+0000","last_undegraded":"2024-09-13T13:23:11.909761+0000","last_fullsized":"2024-09-13T13:23:11.909761+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'39","last_scrub_stamp":"2024-09-13T13:22:28.725850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:28.725850+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:28.974062+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":462,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912523+0000","last_change":"2024-09-13T13:22:48.307594+0000","last_active":"2024-09-13T13:23:11.912523+0000","last_peered":"2024-09-13T13:23:11.912523+0000","last_clean":"2024-09-13T13:23:11.912523+0000","last_became_active":"2024-09-13T13:22:47.535093+0000","last_became_peered":"2024-09-13T13:22:47.535093+0000","last_unstale":"2024-09-13T13:23:11.912523+0000","last_undegraded":"2024-09-13T13:23:11.912523+0000","last_fullsized":"2024-09-13T13:23:11.912523+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:22:48.307530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:48.307530+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:58.238165+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"542'18","reported_seq":862,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911958+0000","last_change":"2024-09-13T13:22:47.537118+0000","last_active":"2024-09-13T13:23:11.911958+0000","last_peered":"2024-09-13T13:23:11.911958+0000","last_clean":"2024-09-13T13:23:11.911958+0000","last_became_active":"2024-09-13T13:22:47.536735+0000","last_became_peered":"2024-09-13T13:22:47.536735+0000","last_unstale":"2024-09-13T13:23:11.911958+0000","last_undegraded":"2024-09-13T13:23:11.911958+0000","last_fullsized":"2024-09-13T13:23:11.911958+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'17","last_scrub_stamp":"2024-09-13T13:22:35.467482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:35.467482+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:54.275482+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1057,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910529+0000","last_change":"2024-09-13T13:23:04.914299+0000","last_active":"2024-09-13T13:23:11.910529+0000","last_peered":"2024-09-13T13:23:11.910529+0000","last_clean":"2024-09-13T13:23:11.910529+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:23:11.910529+0000","last_undegraded":"2024-09-13T13:23:11.910529+0000","last_fullsized":"2024-09-13T13:23:11.910529+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:04.914219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:23:04.914219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:28.450483+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1027,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.647776+0000","last_change":"2024-09-13T13:22:04.595087+0000","last_active":"2024-09-13T13:23:11.647776+0000","last_peered":"2024-09-13T13:23:11.647776+0000","last_clean":"2024-09-13T13:23:11.647776+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:23:11.647776+0000","last_undegraded":"2024-09-13T13:23:11.647776+0000","last_fullsized":"2024-09-13T13:23:11.647776+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:04.595027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:04.595027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:24.252511+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"500'18","reported_seq":610,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911894+0000","last_change":"2024-09-13T13:22:19.190902+0000","last_active":"2024-09-13T13:23:11.911894+0000","last_peered":"2024-09-13T13:23:11.911894+0000","last_clean":"2024-09-13T13:23:11.911894+0000","last_became_active":"2024-09-13T13:22:19.170662+0000","last_became_peered":"2024-09-13T13:22:19.170662+0000","last_unstale":"2024-09-13T13:23:11.911894+0000","last_undegraded":"2024-09-13T13:23:11.911894+0000","last_fullsized":"2024-09-13T13:23:11.911894+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'18","last_scrub_stamp":"2024-09-13T13:22:07.608120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:07.608120+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1081,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910291+0000","last_change":"2024-09-13T13:22:35.850436+0000","last_active":"2024-09-13T13:23:11.910291+0000","last_peered":"2024-09-13T13:23:11.910291+0000","last_clean":"2024-09-13T13:23:11.910291+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:23:11.910291+0000","last_undegraded":"2024-09-13T13:23:11.910291+0000","last_fullsized":"2024-09-13T13:23:11.910291+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:35.850384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:35.850384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:54.841106+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":70,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912288+0000","last_change":"2024-09-13T13:22:21.182286+0000","last_active":"2024-09-13T13:23:11.912288+0000","last_peered":"2024-09-13T13:23:11.912288+0000","last_clean":"2024-09-13T13:23:11.912288+0000","last_became_active":"2024-09-13T13:22:19.182619+0000","last_became_peered":"2024-09-13T13:22:19.182619+0000","last_unstale":"2024-09-13T13:23:11.912288+0000","last_undegraded":"2024-09-13T13:23:11.912288+0000","last_fullsized":"2024-09-13T13:23:11.912288+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:21.182244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.182244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:28.883635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"500'21","reported_seq":1113,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910997+0000","last_change":"2024-09-13T13:22:19.213600+0000","last_active":"2024-09-13T13:23:11.910997+0000","last_peered":"2024-09-13T13:23:11.910997+0000","last_clean":"2024-09-13T13:23:11.910997+0000","last_became_active":"2024-09-13T13:22:19.186228+0000","last_became_peered":"2024-09-13T13:22:19.186228+0000","last_unstale":"2024-09-13T13:23:11.910997+0000","last_undegraded":"2024-09-13T13:23:11.910997+0000","last_fullsized":"2024-09-13T13:23:11.910997+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:22:15.517537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:15.517537+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:37.123521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":452,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911643+0000","last_change":"2024-09-13T13:22:49.261274+0000","last_active":"2024-09-13T13:23:11.911643+0000","last_peered":"2024-09-13T13:23:11.911643+0000","last_clean":"2024-09-13T13:23:11.911643+0000","last_became_active":"2024-09-13T13:22:47.535128+0000","last_became_peered":"2024-09-13T13:22:47.535128+0000","last_unstale":"2024-09-13T13:23:11.911643+0000","last_undegraded":"2024-09-13T13:23:11.911643+0000","last_fullsized":"2024-09-13T13:23:11.911643+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:49.261197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:49.261197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:14.816989+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1072,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643670+0000","last_change":"2024-09-13T13:22:43.367744+0000","last_active":"2024-09-13T13:23:11.643670+0000","last_peered":"2024-09-13T13:23:11.643670+0000","last_clean":"2024-09-13T13:23:11.643670+0000","last_became_active":"2024-09-13T13:22:19.170501+0000","last_became_peered":"2024-09-13T13:22:19.170501+0000","last_unstale":"2024-09-13T13:23:11.643670+0000","last_undegraded":"2024-09-13T13:23:11.643670+0000","last_fullsized":"2024-09-13T13:23:11.643670+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:43.367697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:43.367697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:45.151015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":337,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642854+0000","last_change":"2024-09-13T13:22:19.181069+0000","last_active":"2024-09-13T13:23:11.642854+0000","last_peered":"2024-09-13T13:23:11.642854+0000","last_clean":"2024-09-13T13:23:11.642854+0000","last_became_active":"2024-09-13T13:22:19.180372+0000","last_became_peered":"2024-09-13T13:22:19.180372+0000","last_unstale":"2024-09-13T13:23:11.642854+0000","last_undegraded":"2024-09-13T13:23:11.642854+0000","last_fullsized":"2024-09-13T13:23:11.642854+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:17.028121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:17.028121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:45.610983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"500'21","reported_seq":1134,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911479+0000","last_change":"2024-09-13T13:22:19.208997+0000","last_active":"2024-09-13T13:23:11.911479+0000","last_peered":"2024-09-13T13:23:11.911479+0000","last_clean":"2024-09-13T13:23:11.911479+0000","last_became_active":"2024-09-13T13:22:19.174582+0000","last_became_peered":"2024-09-13T13:22:19.174582+0000","last_unstale":"2024-09-13T13:23:11.911479+0000","last_undegraded":"2024-09-13T13:23:11.911479+0000","last_fullsized":"2024-09-13T13:23:11.911479+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:22:00.644695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:00.644695+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:21.578445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1055,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910325+0000","last_change":"2024-09-13T13:22:19.181892+0000","last_active":"2024-09-13T13:23:11.910325+0000","last_peered":"2024-09-13T13:23:11.910325+0000","last_clean":"2024-09-13T13:23:11.910325+0000","last_became_active":"2024-09-13T13:22:19.181582+0000","last_became_peered":"2024-09-13T13:22:19.181582+0000","last_unstale":"2024-09-13T13:23:11.910325+0000","last_undegraded":"2024-09-13T13:23:11.910325+0000","last_fullsized":"2024-09-13T13:23:11.910325+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:05.491514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:05.491514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:35.470624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"527'12","reported_seq":1354,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643327+0000","last_change":"2024-09-13T13:22:19.217268+0000","last_active":"2024-09-13T13:23:11.643327+0000","last_peered":"2024-09-13T13:23:11.643327+0000","last_clean":"2024-09-13T13:23:11.643327+0000","last_became_active":"2024-09-13T13:22:19.184087+0000","last_became_peered":"2024-09-13T13:22:19.184087+0000","last_unstale":"2024-09-13T13:23:11.643327+0000","last_undegraded":"2024-09-13T13:23:11.643327+0000","last_fullsized":"2024-09-13T13:23:11.643327+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-09-13T13:21:58.322548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:21:58.322548+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:25.836363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":478,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911336+0000","last_change":"2024-09-13T13:22:44.323857+0000","last_active":"2024-09-13T13:23:11.911336+0000","last_peered":"2024-09-13T13:23:11.911336+0000","last_clean":"2024-09-13T13:23:11.911336+0000","last_became_active":"2024-09-13T13:22:16.227359+0000","last_became_peered":"2024-09-13T13:22:16.227359+0000","last_unstale":"2024-09-13T13:23:11.911336+0000","last_undegraded":"2024-09-13T13:23:11.911336+0000","last_fullsized":"2024-09-13T13:23:11.911336+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:44.323776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:44.323776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:50.755419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"500'30","reported_seq":1153,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643941+0000","last_change":"2024-09-13T13:22:34.456861+0000","last_active":"2024-09-13T13:23:11.643941+0000","last_peered":"2024-09-13T13:23:11.643941+0000","last_clean":"2024-09-13T13:23:11.643941+0000","last_became_active":"2024-09-13T13:22:19.183816+0000","last_became_peered":"2024-09-13T13:22:19.183816+0000","last_unstale":"2024-09-13T13:23:11.643941+0000","last_undegraded":"2024-09-13T13:23:11.643941+0000","last_fullsized":"2024-09-13T13:23:11.643941+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'30","last_scrub_stamp":"2024-09-13T13:22:34.456804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:34.456804+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:45.262815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":70,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911042+0000","last_change":"2024-09-13T13:22:20.138632+0000","last_active":"2024-09-13T13:23:11.911042+0000","last_peered":"2024-09-13T13:23:11.911042+0000","last_clean":"2024-09-13T13:23:11.911042+0000","last_became_active":"2024-09-13T13:22:19.179741+0000","last_became_peered":"2024-09-13T13:22:19.179741+0000","last_unstale":"2024-09-13T13:23:11.911042+0000","last_undegraded":"2024-09-13T13:23:11.911042+0000","last_fullsized":"2024-09-13T13:23:11.911042+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:20.138585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:20.138585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:25.482073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":1348,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911288+0000","last_change":"2024-09-13T13:23:06.836450+0000","last_active":"2024-09-13T13:23:11.911288+0000","last_peered":"2024-09-13T13:23:11.911288+0000","last_clean":"2024-09-13T13:23:11.911288+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:23:11.911288+0000","last_undegraded":"2024-09-13T13:23:11.911288+0000","last_fullsized":"2024-09-13T13:23:11.911288+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:23:06.836372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:23:06.836372+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:08.930779+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1078,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910725+0000","last_change":"2024-09-13T13:22:19.187614+0000","last_active":"2024-09-13T13:23:11.910725+0000","last_peered":"2024-09-13T13:23:11.910725+0000","last_clean":"2024-09-13T13:23:11.910725+0000","last_became_active":"2024-09-13T13:22:19.187294+0000","last_became_peered":"2024-09-13T13:22:19.187294+0000","last_unstale":"2024-09-13T13:23:11.910725+0000","last_undegraded":"2024-09-13T13:23:11.910725+0000","last_fullsized":"2024-09-13T13:23:11.910725+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:16.488651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:16.488651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:42.763643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"500'27","reported_seq":519,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910257+0000","last_change":"2024-09-13T13:22:46.352617+0000","last_active":"2024-09-13T13:23:11.910257+0000","last_peered":"2024-09-13T13:23:11.910257+0000","last_clean":"2024-09-13T13:23:11.910257+0000","last_became_active":"2024-09-13T13:22:16.230929+0000","last_became_peered":"2024-09-13T13:22:16.230929+0000","last_unstale":"2024-09-13T13:23:11.910257+0000","last_undegraded":"2024-09-13T13:23:11.910257+0000","last_fullsized":"2024-09-13T13:23:11.910257+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'27","last_scrub_stamp":"2024-09-13T13:22:46.352558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:46.352558+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:05.088199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":488,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911859+0000","last_change":"2024-09-13T13:22:19.187717+0000","last_active":"2024-09-13T13:23:11.911859+0000","last_peered":"2024-09-13T13:23:11.911859+0000","last_clean":"2024-09-13T13:23:11.911859+0000","last_became_active":"2024-09-13T13:22:19.187348+0000","last_became_peered":"2024-09-13T13:22:19.187348+0000","last_unstale":"2024-09-13T13:23:11.911859+0000","last_undegraded":"2024-09-13T13:23:11.911859+0000","last_fullsized":"2024-09-13T13:23:11.911859+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:53.985950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:53.985950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1066,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910145+0000","last_change":"2024-09-13T13:23:03.302340+0000","last_active":"2024-09-13T13:23:11.910145+0000","last_peered":"2024-09-13T13:23:11.910145+0000","last_clean":"2024-09-13T13:23:11.910145+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:23:11.910145+0000","last_undegraded":"2024-09-13T13:23:11.910145+0000","last_fullsized":"2024-09-13T13:23:11.910145+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:03.302284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:23:03.302284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:07.736447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1062,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643804+0000","last_change":"2024-09-13T13:22:56.330924+0000","last_active":"2024-09-13T13:23:11.643804+0000","last_peered":"2024-09-13T13:23:11.643804+0000","last_clean":"2024-09-13T13:23:11.643804+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:23:11.643804+0000","last_undegraded":"2024-09-13T13:23:11.643804+0000","last_fullsized":"2024-09-13T13:23:11.643804+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:56.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:56.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:04.534532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"500'21","reported_seq":1056,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910365+0000","last_change":"2024-09-13T13:22:05.680067+0000","last_active":"2024-09-13T13:23:11.910365+0000","last_peered":"2024-09-13T13:23:11.910365+0000","last_clean":"2024-09-13T13:23:11.910365+0000","last_became_active":"2024-09-13T13:17:25.331530+0000","last_became_peered":"2024-09-13T13:17:25.331530+0000","last_unstale":"2024-09-13T13:23:11.910365+0000","last_undegraded":"2024-09-13T13:23:11.910365+0000","last_fullsized":"2024-09-13T13:23:11.910365+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:22:05.680001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:05.680001+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:13.710980+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1038,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912250+0000","last_change":"2024-09-13T13:22:47.535815+0000","last_active":"2024-09-13T13:23:11.912250+0000","last_peered":"2024-09-13T13:23:11.912250+0000","last_clean":"2024-09-13T13:23:11.912250+0000","last_became_active":"2024-09-13T13:22:47.534993+0000","last_became_peered":"2024-09-13T13:22:47.534993+0000","last_unstale":"2024-09-13T13:23:11.912250+0000","last_undegraded":"2024-09-13T13:23:11.912250+0000","last_fullsized":"2024-09-13T13:23:11.912250+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:16.339803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:16.339803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:28.262817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1050,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642766+0000","last_change":"2024-09-13T13:22:06.261846+0000","last_active":"2024-09-13T13:23:11.642766+0000","last_peered":"2024-09-13T13:23:11.642766+0000","last_clean":"2024-09-13T13:23:11.642766+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:23:11.642766+0000","last_undegraded":"2024-09-13T13:23:11.642766+0000","last_fullsized":"2024-09-13T13:23:11.642766+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:06.261789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:06.261789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:18.852246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1045,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909545+0000","last_change":"2024-09-13T13:22:47.237155+0000","last_active":"2024-09-13T13:23:11.909545+0000","last_peered":"2024-09-13T13:23:11.909545+0000","last_clean":"2024-09-13T13:23:11.909545+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:23:11.909545+0000","last_undegraded":"2024-09-13T13:23:11.909545+0000","last_fullsized":"2024-09-13T13:23:11.909545+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:47.237103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:47.237103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:56.255001+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"500'27","reported_seq":1130,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643631+0000","last_change":"2024-09-13T13:22:19.207124+0000","last_active":"2024-09-13T13:23:11.643631+0000","last_peered":"2024-09-13T13:23:11.643631+0000","last_clean":"2024-09-13T13:23:11.643631+0000","last_became_active":"2024-09-13T13:22:19.180944+0000","last_became_peered":"2024-09-13T13:22:19.180944+0000","last_unstale":"2024-09-13T13:23:11.643631+0000","last_undegraded":"2024-09-13T13:23:11.643631+0000","last_fullsized":"2024-09-13T13:23:11.643631+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'27","last_scrub_stamp":"2024-09-13T13:22:04.286906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:04.286906+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:21.875230+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1079,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910392+0000","last_change":"2024-09-13T13:22:38.892146+0000","last_active":"2024-09-13T13:23:11.910392+0000","last_peered":"2024-09-13T13:23:11.910392+0000","last_clean":"2024-09-13T13:23:11.910392+0000","last_became_active":"2024-09-13T13:22:19.167085+0000","last_became_peered":"2024-09-13T13:22:19.167085+0000","last_unstale":"2024-09-13T13:23:11.910392+0000","last_undegraded":"2024-09-13T13:23:11.910392+0000","last_fullsized":"2024-09-13T13:23:11.910392+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:38.892088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:38.892088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:57.793545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":70,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912310+0000","last_change":"2024-09-13T13:22:22.229134+0000","last_active":"2024-09-13T13:23:11.912310+0000","last_peered":"2024-09-13T13:23:11.912310+0000","last_clean":"2024-09-13T13:23:11.912310+0000","last_became_active":"2024-09-13T13:22:19.183449+0000","last_became_peered":"2024-09-13T13:22:19.183449+0000","last_unstale":"2024-09-13T13:23:11.912310+0000","last_undegraded":"2024-09-13T13:23:11.912310+0000","last_fullsized":"2024-09-13T13:23:11.912310+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:22.229084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:22.229084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:48.688848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1033,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642913+0000","last_change":"2024-09-13T13:23:05.419607+0000","last_active":"2024-09-13T13:23:11.642913+0000","last_peered":"2024-09-13T13:23:11.642913+0000","last_clean":"2024-09-13T13:23:11.642913+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:23:11.642913+0000","last_undegraded":"2024-09-13T13:23:11.642913+0000","last_fullsized":"2024-09-13T13:23:11.642913+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:05.419553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:23:05.419553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:17.518689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"500'12","reported_seq":1101,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643024+0000","last_change":"2024-09-13T13:22:09.293452+0000","last_active":"2024-09-13T13:23:11.643024+0000","last_peered":"2024-09-13T13:23:11.643024+0000","last_clean":"2024-09-13T13:23:11.643024+0000","last_became_active":"2024-09-13T13:16:12.073702+0000","last_became_peered":"2024-09-13T13:16:12.073702+0000","last_unstale":"2024-09-13T13:23:11.643024+0000","last_undegraded":"2024-09-13T13:23:11.643024+0000","last_fullsized":"2024-09-13T13:23:11.643024+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'12","last_scrub_stamp":"2024-09-13T13:22:09.293379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:09.293379+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:12.740712+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1073,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910980+0000","last_change":"2024-09-13T13:22:18.459760+0000","last_active":"2024-09-13T13:23:11.910980+0000","last_peered":"2024-09-13T13:23:11.910980+0000","last_clean":"2024-09-13T13:23:11.910980+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:23:11.910980+0000","last_undegraded":"2024-09-13T13:23:11.910980+0000","last_fullsized":"2024-09-13T13:23:11.910980+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:18.458898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:18.458898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:21.316464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1014,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.647834+0000","last_change":"2024-09-13T13:22:27.508192+0000","last_active":"2024-09-13T13:23:11.647834+0000","last_peered":"2024-09-13T13:23:11.647834+0000","last_clean":"2024-09-13T13:23:11.647834+0000","last_became_active":"2024-09-13T13:22:19.182103+0000","last_became_peered":"2024-09-13T13:22:19.182103+0000","last_unstale":"2024-09-13T13:23:11.647834+0000","last_undegraded":"2024-09-13T13:23:11.647834+0000","last_fullsized":"2024-09-13T13:23:11.647834+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:27.508129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:27.508129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:52.685949+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":669,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910161+0000","last_change":"2024-09-13T13:22:47.534836+0000","last_active":"2024-09-13T13:23:11.910161+0000","last_peered":"2024-09-13T13:23:11.910161+0000","last_clean":"2024-09-13T13:23:11.910161+0000","last_became_active":"2024-09-13T13:22:47.534502+0000","last_became_peered":"2024-09-13T13:22:47.534502+0000","last_unstale":"2024-09-13T13:23:11.910161+0000","last_undegraded":"2024-09-13T13:23:11.910161+0000","last_fullsized":"2024-09-13T13:23:11.910161+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:28.174939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:28.174939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:41.419485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1078,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910674+0000","last_change":"2024-09-13T13:22:21.004605+0000","last_active":"2024-09-13T13:23:11.910674+0000","last_peered":"2024-09-13T13:23:11.910674+0000","last_clean":"2024-09-13T13:23:11.910674+0000","last_became_active":"2024-09-13T13:22:16.235979+0000","last_became_peered":"2024-09-13T13:22:16.235979+0000","last_unstale":"2024-09-13T13:23:11.910674+0000","last_undegraded":"2024-09-13T13:23:11.910674+0000","last_fullsized":"2024-09-13T13:23:11.910674+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:21.004542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.004542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:26.169929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1042,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643184+0000","last_change":"2024-09-13T13:21:27.318950+0000","last_active":"2024-09-13T13:23:11.643184+0000","last_peered":"2024-09-13T13:23:11.643184+0000","last_clean":"2024-09-13T13:23:11.643184+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:23:11.643184+0000","last_undegraded":"2024-09-13T13:23:11.643184+0000","last_fullsized":"2024-09-13T13:23:11.643184+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:21:27.318866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:21:27.318866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"500'27","reported_seq":1089,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.648461+0000","last_change":"2024-09-13T13:22:33.460384+0000","last_active":"2024-09-13T13:23:11.648461+0000","last_peered":"2024-09-13T13:23:11.648461+0000","last_clean":"2024-09-13T13:23:11.648461+0000","last_became_active":"2024-09-13T13:16:12.070277+0000","last_became_peered":"2024-09-13T13:16:12.070277+0000","last_unstale":"2024-09-13T13:23:11.648461+0000","last_undegraded":"2024-09-13T13:23:11.648461+0000","last_fullsized":"2024-09-13T13:23:11.648461+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'27","last_scrub_stamp":"2024-09-13T13:22:33.460303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:33.460303+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:48.717656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1095,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910839+0000","last_change":"2024-09-13T13:22:19.186586+0000","last_active":"2024-09-13T13:23:11.910839+0000","last_peered":"2024-09-13T13:23:11.910839+0000","last_clean":"2024-09-13T13:23:11.910839+0000","last_became_active":"2024-09-13T13:22:19.185732+0000","last_became_peered":"2024-09-13T13:22:19.185732+0000","last_unstale":"2024-09-13T13:23:11.910839+0000","last_undegraded":"2024-09-13T13:23:11.910839+0000","last_fullsized":"2024-09-13T13:23:11.910839+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:03.489521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:03.489521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:23.546162+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1534,"num_read_kb":1291,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":356,"num_bytes_recovered":7300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":976,"ondisk_log_size":976,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3042,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":603,"num_read_kb":598,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201550,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":710632,"kb_used_data":15828,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93034520,"statfs":{"total":95995035648,"available":95267348480,"internally_reserved":0,"allocated":16207872,"data_stored":15556877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":711457226},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141161,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":782932,"kb_used_data":1348,"kb_used_omap":4,"kb_used_meta":781563,"kb_avail":92962220,"statfs":{"total":95995035648,"available":95193313280,"internally_reserved":0,"allocated":1380352,"data_stored":744112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":800321197},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692122,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":813352,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92931800,"statfs":{"total":95995035648,"available":95162163200,"internally_reserved":0,"allocated":1982464,"data_stored":1334949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":523,"seq":2246267895822,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72376,"kb_used_data":1376,"kb_used_omap":4,"kb_used_meta":70971,"kb_avail":93672776,"statfs":{"total":95995035648,"available":95920922624,"internally_reserved":0,"allocated":1409024,"data_stored":744717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":72674981},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117287,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":885564,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92859588,"statfs":{"total":95995035648,"available":95088218112,"internally_reserved":0,"allocated":2007040,"data_stored":1335915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":904788423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411626,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":714456,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93030696,"statfs":{"total":95995035648,"available":95263432704,"internally_reserved":0,"allocated":1642496,"data_stored":1006719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575153,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822516,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92922636,"statfs":{"total":95995035648,"available":95152779264,"internally_reserved":0,"allocated":1409024,"data_stored":744760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738680,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":961340,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92783812,"statfs":{"total":95995035648,"available":95010623488,"internally_reserved":0,"allocated":1417216,"data_stored":745808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":982972869},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:23:20.583 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T13:23:20.849 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:20.970 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:20 smithi052 ceph-mon[36682]: from='client.37217 v1:172.21.15.52:0/882087491' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:23:20.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:20 smithi052 ceph-mon[29471]: from='client.37217 v1:172.21.15.52:0/882087491' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:23:21.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:20 smithi167 ceph-mon[33178]: from='client.37217 v1:172.21.15.52:0/882087491' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:23:21.514 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-13T13:23:21.516 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-13T13:23:21.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: pgmap v1136: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T13:23:21.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: 5.e scrub starts 2024-09-13T13:23:21.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: 5.e scrub ok 2024-09-13T13:23:21.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: 4.7 scrub starts 2024-09-13T13:23:21.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: 4.7 scrub ok 2024-09-13T13:23:21.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:23:21.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:23:21.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: osdmap e552: 8 total, 8 up, 8 in 2024-09-13T13:23:21.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:23:21.818 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:23:21.819 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: 6.18 scrub starts 2024-09-13T13:23:21.819 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: 6.18 scrub ok 2024-09-13T13:23:21.819 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: 3.7 scrub starts 2024-09-13T13:23:21.819 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[36682]: 3.7 scrub ok 2024-09-13T13:23:21.819 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: pgmap v1136: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T13:23:21.819 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: 5.e scrub starts 2024-09-13T13:23:21.819 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: 5.e scrub ok 2024-09-13T13:23:21.819 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: 4.7 scrub starts 2024-09-13T13:23:21.819 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: 4.7 scrub ok 2024-09-13T13:23:21.819 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:23:21.819 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:23:21.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: osdmap e552: 8 total, 8 up, 8 in 2024-09-13T13:23:21.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:23:21.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:23:21.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: 6.18 scrub starts 2024-09-13T13:23:21.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: 6.18 scrub ok 2024-09-13T13:23:21.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: 3.7 scrub starts 2024-09-13T13:23:21.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:21 smithi052 ceph-mon[29471]: 3.7 scrub ok 2024-09-13T13:23:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: pgmap v1136: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-13T13:23:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: 5.e scrub starts 2024-09-13T13:23:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: 5.e scrub ok 2024-09-13T13:23:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: 4.7 scrub starts 2024-09-13T13:23:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: 4.7 scrub ok 2024-09-13T13:23:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:23:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:23:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: osdmap e552: 8 total, 8 up, 8 in 2024-09-13T13:23:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:23:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:23:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: 6.18 scrub starts 2024-09-13T13:23:22.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: 6.18 scrub ok 2024-09-13T13:23:22.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: 3.7 scrub starts 2024-09-13T13:23:22.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:21 smithi167 ceph-mon[33178]: 3.7 scrub ok 2024-09-13T13:23:22.243 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":1137,"stamp":"2024-09-13T13:23:20.671283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2156,"num_read_kb":1908,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":386,"num_bytes_recovered":667300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1961,"ondisk_log_size":1961,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":427,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5753536,"kb_used_data":17212,"kb_used_omap":18,"kb_used_meta":5736173,"kb_avail":744207680,"statfs":{"total":767960285184,"available":762068664320,"internally_reserved":0,"allocated":17625088,"data_stored":12383457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18440,"internal_metadata":5873842168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.030180"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":1072,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910956+0000","last_change":"2024-09-13T13:22:58.056700+0000","last_active":"2024-09-13T13:23:11.910956+0000","last_peered":"2024-09-13T13:23:11.910956+0000","last_clean":"2024-09-13T13:23:11.910956+0000","last_became_active":"2024-09-13T13:17:25.335900+0000","last_became_peered":"2024-09-13T13:17:25.335900+0000","last_unstale":"2024-09-13T13:23:11.910956+0000","last_undegraded":"2024-09-13T13:23:11.910956+0000","last_fullsized":"2024-09-13T13:23:11.910956+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:58.056653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:58.056653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:24.248375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"500'21","reported_seq":514,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910020+0000","last_change":"2024-09-13T13:22:35.109312+0000","last_active":"2024-09-13T13:23:11.910020+0000","last_peered":"2024-09-13T13:23:11.910020+0000","last_clean":"2024-09-13T13:23:11.910020+0000","last_became_active":"2024-09-13T13:22:16.231034+0000","last_became_peered":"2024-09-13T13:22:16.231034+0000","last_unstale":"2024-09-13T13:23:11.910020+0000","last_undegraded":"2024-09-13T13:23:11.910020+0000","last_fullsized":"2024-09-13T13:23:11.910020+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:22:35.109229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:35.109229+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:00.178488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1080,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909822+0000","last_change":"2024-09-13T13:22:28.036382+0000","last_active":"2024-09-13T13:23:11.909822+0000","last_peered":"2024-09-13T13:23:11.909822+0000","last_clean":"2024-09-13T13:23:11.909822+0000","last_became_active":"2024-09-13T13:22:19.167858+0000","last_became_peered":"2024-09-13T13:22:19.167858+0000","last_unstale":"2024-09-13T13:23:11.909822+0000","last_undegraded":"2024-09-13T13:23:11.909822+0000","last_fullsized":"2024-09-13T13:23:11.909822+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:28.036335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:28.036335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:56.564561+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":475,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911351+0000","last_change":"2024-09-13T13:22:58.283257+0000","last_active":"2024-09-13T13:23:11.911351+0000","last_peered":"2024-09-13T13:23:11.911351+0000","last_clean":"2024-09-13T13:23:11.911351+0000","last_became_active":"2024-09-13T13:22:16.242543+0000","last_became_peered":"2024-09-13T13:22:16.242543+0000","last_unstale":"2024-09-13T13:23:11.911351+0000","last_undegraded":"2024-09-13T13:23:11.911351+0000","last_fullsized":"2024-09-13T13:23:11.911351+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:58.283191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:58.283191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:59.591998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1054,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642580+0000","last_change":"2024-09-13T13:22:19.171791+0000","last_active":"2024-09-13T13:23:11.642580+0000","last_peered":"2024-09-13T13:23:11.642580+0000","last_clean":"2024-09-13T13:23:11.642580+0000","last_became_active":"2024-09-13T13:22:19.171472+0000","last_became_peered":"2024-09-13T13:22:19.171472+0000","last_unstale":"2024-09-13T13:23:11.642580+0000","last_undegraded":"2024-09-13T13:23:11.642580+0000","last_fullsized":"2024-09-13T13:23:11.642580+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:21:18.313800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:21:18.313800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1093,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643776+0000","last_change":"2024-09-13T13:22:42.380173+0000","last_active":"2024-09-13T13:23:11.643776+0000","last_peered":"2024-09-13T13:23:11.643776+0000","last_clean":"2024-09-13T13:23:11.643776+0000","last_became_active":"2024-09-13T13:17:25.243371+0000","last_became_peered":"2024-09-13T13:17:25.243371+0000","last_unstale":"2024-09-13T13:23:11.643776+0000","last_undegraded":"2024-09-13T13:23:11.643776+0000","last_fullsized":"2024-09-13T13:23:11.643776+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:42.380129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:42.380129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:59.325163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"500'39","reported_seq":1185,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643524+0000","last_change":"2024-09-13T13:22:52.349355+0000","last_active":"2024-09-13T13:23:11.643524+0000","last_peered":"2024-09-13T13:23:11.643524+0000","last_clean":"2024-09-13T13:23:11.643524+0000","last_became_active":"2024-09-13T13:17:25.245131+0000","last_became_peered":"2024-09-13T13:17:25.245131+0000","last_unstale":"2024-09-13T13:23:11.643524+0000","last_undegraded":"2024-09-13T13:23:11.643524+0000","last_fullsized":"2024-09-13T13:23:11.643524+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'39","last_scrub_stamp":"2024-09-13T13:22:52.349278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:52.349278+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:59.651974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":454,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910201+0000","last_change":"2024-09-13T13:23:02.305469+0000","last_active":"2024-09-13T13:23:11.910201+0000","last_peered":"2024-09-13T13:23:11.910201+0000","last_clean":"2024-09-13T13:23:11.910201+0000","last_became_active":"2024-09-13T13:22:16.236142+0000","last_became_peered":"2024-09-13T13:22:16.236142+0000","last_unstale":"2024-09-13T13:23:11.910201+0000","last_undegraded":"2024-09-13T13:23:11.910201+0000","last_fullsized":"2024-09-13T13:23:11.910201+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:02.305405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:23:02.305405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:02.773497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":1122,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911573+0000","last_change":"2024-09-13T13:22:44.954301+0000","last_active":"2024-09-13T13:23:11.911573+0000","last_peered":"2024-09-13T13:23:11.911573+0000","last_clean":"2024-09-13T13:23:11.911573+0000","last_became_active":"2024-09-13T13:22:16.235036+0000","last_became_peered":"2024-09-13T13:22:16.235036+0000","last_unstale":"2024-09-13T13:23:11.911573+0000","last_undegraded":"2024-09-13T13:23:11.911573+0000","last_fullsized":"2024-09-13T13:23:11.911573+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:22:44.954077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:44.954077+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:58.901786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"500'21","reported_seq":505,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911449+0000","last_change":"2024-09-13T13:22:20.130618+0000","last_active":"2024-09-13T13:23:11.911449+0000","last_peered":"2024-09-13T13:23:11.911449+0000","last_clean":"2024-09-13T13:23:11.911449+0000","last_became_active":"2024-09-13T13:22:16.241297+0000","last_became_peered":"2024-09-13T13:22:16.241297+0000","last_unstale":"2024-09-13T13:23:11.911449+0000","last_undegraded":"2024-09-13T13:23:11.911449+0000","last_fullsized":"2024-09-13T13:23:11.911449+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:22:20.130562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:20.130562+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:46.912600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1081,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910984+0000","last_change":"2024-09-13T13:22:23.733890+0000","last_active":"2024-09-13T13:23:11.910984+0000","last_peered":"2024-09-13T13:23:11.910984+0000","last_clean":"2024-09-13T13:23:11.910984+0000","last_became_active":"2024-09-13T13:22:19.169471+0000","last_became_peered":"2024-09-13T13:22:19.169471+0000","last_unstale":"2024-09-13T13:23:11.910984+0000","last_undegraded":"2024-09-13T13:23:11.910984+0000","last_fullsized":"2024-09-13T13:23:11.910984+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:23.733828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:23.733828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:28.354839+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1067,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909632+0000","last_change":"2024-09-13T13:22:19.181827+0000","last_active":"2024-09-13T13:23:11.909632+0000","last_peered":"2024-09-13T13:23:11.909632+0000","last_clean":"2024-09-13T13:23:11.909632+0000","last_became_active":"2024-09-13T13:22:19.181508+0000","last_became_peered":"2024-09-13T13:22:19.181508+0000","last_unstale":"2024-09-13T13:23:11.909632+0000","last_undegraded":"2024-09-13T13:23:11.909632+0000","last_fullsized":"2024-09-13T13:23:11.909632+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:08.489422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:08.489422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:18.900312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1097,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910726+0000","last_change":"2024-09-13T13:22:34.822934+0000","last_active":"2024-09-13T13:23:11.910726+0000","last_peered":"2024-09-13T13:23:11.910726+0000","last_clean":"2024-09-13T13:23:11.910726+0000","last_became_active":"2024-09-13T13:22:16.234121+0000","last_became_peered":"2024-09-13T13:22:16.234121+0000","last_unstale":"2024-09-13T13:23:11.910726+0000","last_undegraded":"2024-09-13T13:23:11.910726+0000","last_fullsized":"2024-09-13T13:23:11.910726+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:34.822864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:34.822864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:00.744920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"500'9","reported_seq":504,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910131+0000","last_change":"2024-09-13T13:23:04.254691+0000","last_active":"2024-09-13T13:23:11.910131+0000","last_peered":"2024-09-13T13:23:11.910131+0000","last_clean":"2024-09-13T13:23:11.910131+0000","last_became_active":"2024-09-13T13:22:16.235422+0000","last_became_peered":"2024-09-13T13:22:16.235422+0000","last_unstale":"2024-09-13T13:23:11.910131+0000","last_undegraded":"2024-09-13T13:23:11.910131+0000","last_fullsized":"2024-09-13T13:23:11.910131+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'9","last_scrub_stamp":"2024-09-13T13:23:04.254619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:23:04.254619+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:10.814239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":70,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910512+0000","last_change":"2024-09-13T13:22:25.321028+0000","last_active":"2024-09-13T13:23:11.910512+0000","last_peered":"2024-09-13T13:23:11.910512+0000","last_clean":"2024-09-13T13:23:11.910512+0000","last_became_active":"2024-09-13T13:22:19.179612+0000","last_became_peered":"2024-09-13T13:22:19.179612+0000","last_unstale":"2024-09-13T13:23:11.910512+0000","last_undegraded":"2024-09-13T13:23:11.910512+0000","last_fullsized":"2024-09-13T13:23:11.910512+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:25.320980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:25.320980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:35.114273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1037,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910112+0000","last_change":"2024-09-13T13:22:48.926914+0000","last_active":"2024-09-13T13:23:11.910112+0000","last_peered":"2024-09-13T13:23:11.910112+0000","last_clean":"2024-09-13T13:23:11.910112+0000","last_became_active":"2024-09-13T13:01:08.213325+0000","last_became_peered":"2024-09-13T13:01:08.213325+0000","last_unstale":"2024-09-13T13:23:11.910112+0000","last_undegraded":"2024-09-13T13:23:11.910112+0000","last_fullsized":"2024-09-13T13:23:11.910112+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:48.926852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:48.926852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:56.884810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"67'1","reported_seq":450,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910602+0000","last_change":"2024-09-13T13:22:21.135547+0000","last_active":"2024-09-13T13:23:11.910602+0000","last_peered":"2024-09-13T13:23:11.910602+0000","last_clean":"2024-09-13T13:23:11.910602+0000","last_became_active":"2024-09-13T13:22:16.229533+0000","last_became_peered":"2024-09-13T13:22:16.229533+0000","last_unstale":"2024-09-13T13:23:11.910602+0000","last_undegraded":"2024-09-13T13:23:11.910602+0000","last_fullsized":"2024-09-13T13:23:11.910602+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:22:21.135494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.135494+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:31.996151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1103,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911103+0000","last_change":"2024-09-13T13:22:22.393258+0000","last_active":"2024-09-13T13:23:11.911103+0000","last_peered":"2024-09-13T13:23:11.911103+0000","last_clean":"2024-09-13T13:23:11.911103+0000","last_became_active":"2024-09-13T13:22:19.182768+0000","last_became_peered":"2024-09-13T13:22:19.182768+0000","last_unstale":"2024-09-13T13:23:11.911103+0000","last_undegraded":"2024-09-13T13:23:11.911103+0000","last_fullsized":"2024-09-13T13:23:11.911103+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:22.393212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:22.393212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:29.582027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"500'39","reported_seq":354,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:12.085460+0000","last_change":"2024-09-13T13:23:12.085460+0000","last_active":"2024-09-13T13:23:12.085460+0000","last_peered":"2024-09-13T13:23:12.085460+0000","last_clean":"2024-09-13T13:23:12.085460+0000","last_became_active":"2024-09-13T13:17:25.245658+0000","last_became_peered":"2024-09-13T13:17:25.245658+0000","last_unstale":"2024-09-13T13:23:12.085460+0000","last_undegraded":"2024-09-13T13:23:12.085460+0000","last_fullsized":"2024-09-13T13:23:12.085460+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'39","last_scrub_stamp":"2024-09-13T13:23:12.085392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:23:12.085392+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:39.522471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1098,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910952+0000","last_change":"2024-09-13T13:22:35.161005+0000","last_active":"2024-09-13T13:23:11.910952+0000","last_peered":"2024-09-13T13:23:11.910952+0000","last_clean":"2024-09-13T13:23:11.910952+0000","last_became_active":"2024-09-13T13:22:16.236884+0000","last_became_peered":"2024-09-13T13:22:16.236884+0000","last_unstale":"2024-09-13T13:23:11.910952+0000","last_undegraded":"2024-09-13T13:23:11.910952+0000","last_fullsized":"2024-09-13T13:23:11.910952+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:35.160926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:35.160926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:57.173319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1050,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643593+0000","last_change":"2024-09-13T13:22:48.411884+0000","last_active":"2024-09-13T13:23:11.643593+0000","last_peered":"2024-09-13T13:23:11.643593+0000","last_clean":"2024-09-13T13:23:11.643593+0000","last_became_active":"2024-09-13T13:04:02.794992+0000","last_became_peered":"2024-09-13T13:04:02.794992+0000","last_unstale":"2024-09-13T13:23:11.643593+0000","last_undegraded":"2024-09-13T13:23:11.643593+0000","last_fullsized":"2024-09-13T13:23:11.643593+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:48.411837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:48.411837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:17.222365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":488,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911561+0000","last_change":"2024-09-13T13:22:47.320545+0000","last_active":"2024-09-13T13:23:11.911561+0000","last_peered":"2024-09-13T13:23:11.911561+0000","last_clean":"2024-09-13T13:23:11.911561+0000","last_became_active":"2024-09-13T13:22:16.231331+0000","last_became_peered":"2024-09-13T13:22:16.231331+0000","last_unstale":"2024-09-13T13:23:11.911561+0000","last_undegraded":"2024-09-13T13:23:11.911561+0000","last_fullsized":"2024-09-13T13:23:11.911561+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:47.320485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:47.320485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:07.855412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"500'24","reported_seq":1154,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643334+0000","last_change":"2024-09-13T13:22:20.325637+0000","last_active":"2024-09-13T13:23:11.643334+0000","last_peered":"2024-09-13T13:23:11.643334+0000","last_clean":"2024-09-13T13:23:11.643334+0000","last_became_active":"2024-09-13T13:22:16.230459+0000","last_became_peered":"2024-09-13T13:22:16.230459+0000","last_unstale":"2024-09-13T13:23:11.643334+0000","last_undegraded":"2024-09-13T13:23:11.643334+0000","last_fullsized":"2024-09-13T13:23:11.643334+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'24","last_scrub_stamp":"2024-09-13T13:22:20.325574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:20.325574+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:46.191688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":471,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911641+0000","last_change":"2024-09-13T13:22:38.163671+0000","last_active":"2024-09-13T13:23:11.911641+0000","last_peered":"2024-09-13T13:23:11.911641+0000","last_clean":"2024-09-13T13:23:11.911641+0000","last_became_active":"2024-09-13T13:22:16.233955+0000","last_became_peered":"2024-09-13T13:22:16.233955+0000","last_unstale":"2024-09-13T13:23:11.911641+0000","last_undegraded":"2024-09-13T13:23:11.911641+0000","last_fullsized":"2024-09-13T13:23:11.911641+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:38.163586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:38.163586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:54.313246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":582,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:17.186682+0000","last_change":"2024-09-13T13:23:17.186682+0000","last_active":"2024-09-13T13:23:17.186682+0000","last_peered":"2024-09-13T13:23:17.186682+0000","last_clean":"2024-09-13T13:23:17.186682+0000","last_became_active":"2024-09-13T13:22:47.527459+0000","last_became_peered":"2024-09-13T13:22:47.527459+0000","last_unstale":"2024-09-13T13:23:17.186682+0000","last_undegraded":"2024-09-13T13:23:17.186682+0000","last_fullsized":"2024-09-13T13:23:17.186682+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-13T13:23:17.186620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:23:17.186620+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:46.402514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":1092,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909973+0000","last_change":"2024-09-13T13:22:23.999443+0000","last_active":"2024-09-13T13:23:11.909973+0000","last_peered":"2024-09-13T13:23:11.909973+0000","last_clean":"2024-09-13T13:23:11.909973+0000","last_became_active":"2024-09-13T13:22:16.234943+0000","last_became_peered":"2024-09-13T13:22:16.234943+0000","last_unstale":"2024-09-13T13:23:11.909973+0000","last_undegraded":"2024-09-13T13:23:11.909973+0000","last_fullsized":"2024-09-13T13:23:11.909973+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:22:23.999388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:23.999388+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:49.113211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"500'30","reported_seq":1162,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910941+0000","last_change":"2024-09-13T13:22:52.124180+0000","last_active":"2024-09-13T13:23:11.910941+0000","last_peered":"2024-09-13T13:23:11.910941+0000","last_clean":"2024-09-13T13:23:11.910941+0000","last_became_active":"2024-09-13T13:22:19.186550+0000","last_became_peered":"2024-09-13T13:22:19.186550+0000","last_unstale":"2024-09-13T13:23:11.910941+0000","last_undegraded":"2024-09-13T13:23:11.910941+0000","last_fullsized":"2024-09-13T13:23:11.910941+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'30","last_scrub_stamp":"2024-09-13T13:22:52.124061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:52.124061+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:11.147858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"527'16","reported_seq":1342,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.648653+0000","last_change":"2024-09-13T13:22:29.517702+0000","last_active":"2024-09-13T13:23:11.648653+0000","last_peered":"2024-09-13T13:23:11.648653+0000","last_clean":"2024-09-13T13:23:11.648653+0000","last_became_active":"2024-09-13T13:22:19.182105+0000","last_became_peered":"2024-09-13T13:22:19.182105+0000","last_unstale":"2024-09-13T13:23:11.648653+0000","last_undegraded":"2024-09-13T13:23:11.648653+0000","last_fullsized":"2024-09-13T13:23:11.648653+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'16","last_scrub_stamp":"2024-09-13T13:22:29.517651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:29.517651+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:44.363666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":304,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910293+0000","last_change":"2024-09-13T13:22:56.902357+0000","last_active":"2024-09-13T13:23:11.910293+0000","last_peered":"2024-09-13T13:23:11.910293+0000","last_clean":"2024-09-13T13:23:11.910293+0000","last_became_active":"2024-09-13T13:17:49.052966+0000","last_became_peered":"2024-09-13T13:17:49.052966+0000","last_unstale":"2024-09-13T13:23:11.910293+0000","last_undegraded":"2024-09-13T13:23:11.910293+0000","last_fullsized":"2024-09-13T13:23:11.910293+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:56.902076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:56.902076+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:09.481046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"500'39","reported_seq":1188,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911313+0000","last_change":"2024-09-13T13:22:21.990924+0000","last_active":"2024-09-13T13:23:11.911313+0000","last_peered":"2024-09-13T13:23:11.911313+0000","last_clean":"2024-09-13T13:23:11.911313+0000","last_became_active":"2024-09-13T13:22:16.238592+0000","last_became_peered":"2024-09-13T13:22:16.238592+0000","last_unstale":"2024-09-13T13:23:11.911313+0000","last_undegraded":"2024-09-13T13:23:11.911313+0000","last_fullsized":"2024-09-13T13:23:11.911313+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'39","last_scrub_stamp":"2024-09-13T13:22:21.990869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.990869+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:45.280437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1108,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643203+0000","last_change":"2024-09-13T13:22:30.391264+0000","last_active":"2024-09-13T13:23:11.643203+0000","last_peered":"2024-09-13T13:23:11.643203+0000","last_clean":"2024-09-13T13:23:11.643203+0000","last_became_active":"2024-09-13T13:22:19.178783+0000","last_became_peered":"2024-09-13T13:22:19.178783+0000","last_unstale":"2024-09-13T13:23:11.643203+0000","last_undegraded":"2024-09-13T13:23:11.643203+0000","last_fullsized":"2024-09-13T13:23:11.643203+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:30.391215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:30.391215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:50.744920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"524'12","reported_seq":585,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:12.507964+0000","last_change":"2024-09-13T13:22:16.240555+0000","last_active":"2024-09-13T13:23:12.507964+0000","last_peered":"2024-09-13T13:23:12.507964+0000","last_clean":"2024-09-13T13:23:12.507964+0000","last_became_active":"2024-09-13T13:22:16.239504+0000","last_became_peered":"2024-09-13T13:22:16.239504+0000","last_unstale":"2024-09-13T13:23:12.507964+0000","last_undegraded":"2024-09-13T13:23:12.507964+0000","last_fullsized":"2024-09-13T13:23:12.507964+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'10","last_scrub_stamp":"2024-09-13T13:20:51.899778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:20:51.899778+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1053,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911092+0000","last_change":"2024-09-13T13:22:46.942292+0000","last_active":"2024-09-13T13:23:11.911092+0000","last_peered":"2024-09-13T13:23:11.911092+0000","last_clean":"2024-09-13T13:23:11.911092+0000","last_became_active":"2024-09-13T13:01:08.215811+0000","last_became_peered":"2024-09-13T13:01:08.215811+0000","last_unstale":"2024-09-13T13:23:11.911092+0000","last_undegraded":"2024-09-13T13:23:11.911092+0000","last_fullsized":"2024-09-13T13:23:11.911092+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:46.942232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:46.942232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:50.825260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":235,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911840+0000","last_change":"2024-09-13T13:22:17.195980+0000","last_active":"2024-09-13T13:23:11.911840+0000","last_peered":"2024-09-13T13:23:11.911840+0000","last_clean":"2024-09-13T13:23:11.911840+0000","last_became_active":"2024-09-13T13:22:16.240045+0000","last_became_peered":"2024-09-13T13:22:16.240045+0000","last_unstale":"2024-09-13T13:23:11.911840+0000","last_undegraded":"2024-09-13T13:23:11.911840+0000","last_fullsized":"2024-09-13T13:23:11.911840+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:17.195931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:17.195931+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:43.916437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"500'42","reported_seq":1158,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643154+0000","last_change":"2024-09-13T13:22:19.212922+0000","last_active":"2024-09-13T13:23:11.643154+0000","last_peered":"2024-09-13T13:23:11.643154+0000","last_clean":"2024-09-13T13:23:11.643154+0000","last_became_active":"2024-09-13T13:22:19.180137+0000","last_became_peered":"2024-09-13T13:22:19.180137+0000","last_unstale":"2024-09-13T13:23:11.643154+0000","last_undegraded":"2024-09-13T13:23:11.643154+0000","last_fullsized":"2024-09-13T13:23:11.643154+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'42","last_scrub_stamp":"2024-09-13T13:21:54.255372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:21:54.255372+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":1071,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643798+0000","last_change":"2024-09-13T13:22:19.193027+0000","last_active":"2024-09-13T13:23:11.643798+0000","last_peered":"2024-09-13T13:23:11.643798+0000","last_clean":"2024-09-13T13:23:11.643798+0000","last_became_active":"2024-09-13T13:22:19.176881+0000","last_became_peered":"2024-09-13T13:22:19.176881+0000","last_unstale":"2024-09-13T13:23:11.643798+0000","last_undegraded":"2024-09-13T13:23:11.643798+0000","last_fullsized":"2024-09-13T13:23:11.643798+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:21:25.329252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:21:25.329252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":443,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910860+0000","last_change":"2024-09-13T13:22:30.125943+0000","last_active":"2024-09-13T13:23:11.910860+0000","last_peered":"2024-09-13T13:23:11.910860+0000","last_clean":"2024-09-13T13:23:11.910860+0000","last_became_active":"2024-09-13T13:22:19.182770+0000","last_became_peered":"2024-09-13T13:22:19.182770+0000","last_unstale":"2024-09-13T13:23:11.910860+0000","last_undegraded":"2024-09-13T13:23:11.910860+0000","last_fullsized":"2024-09-13T13:23:11.910860+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:30.125880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:30.125880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:42.523997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":451,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911584+0000","last_change":"2024-09-13T13:22:27.175549+0000","last_active":"2024-09-13T13:23:11.911584+0000","last_peered":"2024-09-13T13:23:11.911584+0000","last_clean":"2024-09-13T13:23:11.911584+0000","last_became_active":"2024-09-13T13:22:16.237053+0000","last_became_peered":"2024-09-13T13:22:16.237053+0000","last_unstale":"2024-09-13T13:23:11.911584+0000","last_undegraded":"2024-09-13T13:23:11.911584+0000","last_fullsized":"2024-09-13T13:23:11.911584+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:27.175477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:27.175477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:50.938997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":327,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642206+0000","last_change":"2024-09-13T13:22:23.390156+0000","last_active":"2024-09-13T13:23:11.642206+0000","last_peered":"2024-09-13T13:23:11.642206+0000","last_clean":"2024-09-13T13:23:11.642206+0000","last_became_active":"2024-09-13T13:22:19.184359+0000","last_became_peered":"2024-09-13T13:22:19.184359+0000","last_unstale":"2024-09-13T13:23:11.642206+0000","last_undegraded":"2024-09-13T13:23:11.642206+0000","last_fullsized":"2024-09-13T13:23:11.642206+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:23.390102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:23.390102+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:31.322915+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"500'42","reported_seq":532,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910956+0000","last_change":"2024-09-13T13:23:10.137666+0000","last_active":"2024-09-13T13:23:11.910956+0000","last_peered":"2024-09-13T13:23:11.910956+0000","last_clean":"2024-09-13T13:23:11.910956+0000","last_became_active":"2024-09-13T13:22:16.227649+0000","last_became_peered":"2024-09-13T13:22:16.227649+0000","last_unstale":"2024-09-13T13:23:11.910956+0000","last_undegraded":"2024-09-13T13:23:11.910956+0000","last_fullsized":"2024-09-13T13:23:11.910956+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'42","last_scrub_stamp":"2024-09-13T13:23:10.137582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:23:10.137582+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:18.073050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":1076,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911103+0000","last_change":"2024-09-13T13:22:34.380999+0000","last_active":"2024-09-13T13:23:11.911103+0000","last_peered":"2024-09-13T13:23:11.911103+0000","last_clean":"2024-09-13T13:23:11.911103+0000","last_became_active":"2024-09-13T13:22:16.242546+0000","last_became_peered":"2024-09-13T13:22:16.242546+0000","last_unstale":"2024-09-13T13:23:11.911103+0000","last_undegraded":"2024-09-13T13:23:11.911103+0000","last_fullsized":"2024-09-13T13:23:11.911103+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-13T13:22:34.380943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:34.380943+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:01.372265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"524'12","reported_seq":1364,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:17.508552+0000","last_change":"2024-09-13T13:22:21.325588+0000","last_active":"2024-09-13T13:23:17.508552+0000","last_peered":"2024-09-13T13:23:17.508552+0000","last_clean":"2024-09-13T13:23:17.508552+0000","last_became_active":"2024-09-13T13:22:16.230421+0000","last_became_peered":"2024-09-13T13:22:16.230421+0000","last_unstale":"2024-09-13T13:23:17.508552+0000","last_undegraded":"2024-09-13T13:23:17.508552+0000","last_fullsized":"2024-09-13T13:23:17.508552+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"524'12","last_scrub_stamp":"2024-09-13T13:22:21.325542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.325542+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:21.878400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":687,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910770+0000","last_change":"2024-09-13T13:22:47.532169+0000","last_active":"2024-09-13T13:23:11.910770+0000","last_peered":"2024-09-13T13:23:11.910770+0000","last_clean":"2024-09-13T13:23:11.910770+0000","last_became_active":"2024-09-13T13:22:47.531836+0000","last_became_peered":"2024-09-13T13:22:47.531836+0000","last_unstale":"2024-09-13T13:23:11.910770+0000","last_undegraded":"2024-09-13T13:23:11.910770+0000","last_fullsized":"2024-09-13T13:23:11.910770+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:14.659140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:14.659140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:43.244378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":403,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.648461+0000","last_change":"2024-09-13T13:23:08.273753+0000","last_active":"2024-09-13T13:23:11.648461+0000","last_peered":"2024-09-13T13:23:11.648461+0000","last_clean":"2024-09-13T13:23:11.648461+0000","last_became_active":"2024-09-13T13:22:19.177690+0000","last_became_peered":"2024-09-13T13:22:19.177690+0000","last_unstale":"2024-09-13T13:23:11.648461+0000","last_undegraded":"2024-09-13T13:23:11.648461+0000","last_fullsized":"2024-09-13T13:23:11.648461+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:23:08.273687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:23:08.273687+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:19.600171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"500'30","reported_seq":1141,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911059+0000","last_change":"2024-09-13T13:22:25.004419+0000","last_active":"2024-09-13T13:23:11.911059+0000","last_peered":"2024-09-13T13:23:11.911059+0000","last_clean":"2024-09-13T13:23:11.911059+0000","last_became_active":"2024-09-13T13:22:16.238614+0000","last_became_peered":"2024-09-13T13:22:16.238614+0000","last_unstale":"2024-09-13T13:23:11.911059+0000","last_undegraded":"2024-09-13T13:23:11.911059+0000","last_fullsized":"2024-09-13T13:23:11.911059+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'30","last_scrub_stamp":"2024-09-13T13:22:25.004346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:25.004346+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:42.170769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1096,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911672+0000","last_change":"2024-09-13T13:22:36.832637+0000","last_active":"2024-09-13T13:23:11.911672+0000","last_peered":"2024-09-13T13:23:11.911672+0000","last_clean":"2024-09-13T13:23:11.911672+0000","last_became_active":"2024-09-13T13:17:25.331166+0000","last_became_peered":"2024-09-13T13:17:25.331166+0000","last_unstale":"2024-09-13T13:23:11.911672+0000","last_undegraded":"2024-09-13T13:23:11.911672+0000","last_fullsized":"2024-09-13T13:23:11.911672+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:36.832583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:36.832583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:38.268212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"119'10","reported_seq":1340,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:12.507595+0000","last_change":"2024-09-13T13:22:43.264022+0000","last_active":"2024-09-13T13:23:12.507595+0000","last_peered":"2024-09-13T13:23:12.507595+0000","last_clean":"2024-09-13T13:23:12.507595+0000","last_became_active":"2024-09-13T13:04:02.763551+0000","last_became_peered":"2024-09-13T13:04:02.763551+0000","last_unstale":"2024-09-13T13:23:12.507595+0000","last_undegraded":"2024-09-13T13:23:12.507595+0000","last_fullsized":"2024-09-13T13:23:12.507595+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-09-13T13:22:43.263941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:43.263941+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:43.990589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1093,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909791+0000","last_change":"2024-09-13T13:22:51.158255+0000","last_active":"2024-09-13T13:23:11.909791+0000","last_peered":"2024-09-13T13:23:11.909791+0000","last_clean":"2024-09-13T13:23:11.909791+0000","last_became_active":"2024-09-13T13:17:25.252712+0000","last_became_peered":"2024-09-13T13:17:25.252712+0000","last_unstale":"2024-09-13T13:23:11.909791+0000","last_undegraded":"2024-09-13T13:23:11.909791+0000","last_fullsized":"2024-09-13T13:23:11.909791+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:51.158206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:51.158206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:14.705773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":389,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:15.303949+0000","last_change":"2024-09-13T13:23:15.303949+0000","last_active":"2024-09-13T13:23:15.303949+0000","last_peered":"2024-09-13T13:23:15.303949+0000","last_clean":"2024-09-13T13:23:15.303949+0000","last_became_active":"2024-09-13T13:17:49.055888+0000","last_became_peered":"2024-09-13T13:17:49.055888+0000","last_unstale":"2024-09-13T13:23:15.303949+0000","last_undegraded":"2024-09-13T13:23:15.303949+0000","last_fullsized":"2024-09-13T13:23:15.303949+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:23:15.303887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:23:15.303887+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:37.788183+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"500'39","reported_seq":1194,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910861+0000","last_change":"2024-09-13T13:22:46.261774+0000","last_active":"2024-09-13T13:23:11.910861+0000","last_peered":"2024-09-13T13:23:11.910861+0000","last_clean":"2024-09-13T13:23:11.910861+0000","last_became_active":"2024-09-13T13:22:19.182252+0000","last_became_peered":"2024-09-13T13:22:19.182252+0000","last_unstale":"2024-09-13T13:23:11.910861+0000","last_undegraded":"2024-09-13T13:23:11.910861+0000","last_fullsized":"2024-09-13T13:23:11.910861+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'39","last_scrub_stamp":"2024-09-13T13:22:46.261717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:46.261717+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:51.517823+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1073,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910523+0000","last_change":"2024-09-13T13:22:08.998496+0000","last_active":"2024-09-13T13:23:11.910523+0000","last_peered":"2024-09-13T13:23:11.910523+0000","last_clean":"2024-09-13T13:23:11.910523+0000","last_became_active":"2024-09-13T13:17:25.332589+0000","last_became_peered":"2024-09-13T13:17:25.332589+0000","last_unstale":"2024-09-13T13:23:11.910523+0000","last_undegraded":"2024-09-13T13:23:11.910523+0000","last_fullsized":"2024-09-13T13:23:11.910523+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:08.998430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:08.998430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:27.122610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1044,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.647962+0000","last_change":"2024-09-13T13:23:10.224215+0000","last_active":"2024-09-13T13:23:11.647962+0000","last_peered":"2024-09-13T13:23:11.647962+0000","last_clean":"2024-09-13T13:23:11.647962+0000","last_became_active":"2024-09-13T13:04:02.758663+0000","last_became_peered":"2024-09-13T13:04:02.758663+0000","last_unstale":"2024-09-13T13:23:11.647962+0000","last_undegraded":"2024-09-13T13:23:11.647962+0000","last_fullsized":"2024-09-13T13:23:11.647962+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:10.224158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:23:10.224158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:15.429296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1080,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909675+0000","last_change":"2024-09-13T13:22:03.982427+0000","last_active":"2024-09-13T13:23:11.909675+0000","last_peered":"2024-09-13T13:23:11.909675+0000","last_clean":"2024-09-13T13:23:11.909675+0000","last_became_active":"2024-09-13T13:17:25.244517+0000","last_became_peered":"2024-09-13T13:17:25.244517+0000","last_unstale":"2024-09-13T13:23:11.909675+0000","last_undegraded":"2024-09-13T13:23:11.909675+0000","last_fullsized":"2024-09-13T13:23:11.909675+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:03.982366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:03.982366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:23.535566+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":304,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909888+0000","last_change":"2024-09-13T13:23:02.862296+0000","last_active":"2024-09-13T13:23:11.909888+0000","last_peered":"2024-09-13T13:23:11.909888+0000","last_clean":"2024-09-13T13:23:11.909888+0000","last_became_active":"2024-09-13T13:17:49.054449+0000","last_became_peered":"2024-09-13T13:17:49.054449+0000","last_unstale":"2024-09-13T13:23:11.909888+0000","last_undegraded":"2024-09-13T13:23:11.909888+0000","last_fullsized":"2024-09-13T13:23:11.909888+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:23:02.862230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:23:02.862230+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:12.888603+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"500'55","reported_seq":345,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:12.507459+0000","last_change":"2024-09-13T13:22:36.396551+0000","last_active":"2024-09-13T13:23:12.507459+0000","last_peered":"2024-09-13T13:23:12.507459+0000","last_clean":"2024-09-13T13:23:12.507459+0000","last_became_active":"2024-09-13T13:18:35.244311+0000","last_became_peered":"2024-09-13T13:18:35.244311+0000","last_unstale":"2024-09-13T13:23:12.507459+0000","last_undegraded":"2024-09-13T13:23:12.507459+0000","last_fullsized":"2024-09-13T13:23:12.507459+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'55","last_scrub_stamp":"2024-09-13T13:22:36.396487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:36.396487+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:59.662802+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":505,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910363+0000","last_change":"2024-09-13T13:22:16.239128+0000","last_active":"2024-09-13T13:23:11.910363+0000","last_peered":"2024-09-13T13:23:11.910363+0000","last_clean":"2024-09-13T13:23:11.910363+0000","last_became_active":"2024-09-13T13:22:16.238858+0000","last_became_peered":"2024-09-13T13:22:16.238858+0000","last_unstale":"2024-09-13T13:23:11.910363+0000","last_undegraded":"2024-09-13T13:23:11.910363+0000","last_fullsized":"2024-09-13T13:23:11.910363+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:21:24.103840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:21:24.103840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":1069,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910809+0000","last_change":"2024-09-13T13:22:17.722161+0000","last_active":"2024-09-13T13:23:11.910809+0000","last_peered":"2024-09-13T13:23:11.910809+0000","last_clean":"2024-09-13T13:23:11.910809+0000","last_became_active":"2024-09-13T13:04:02.757435+0000","last_became_peered":"2024-09-13T13:04:02.757435+0000","last_unstale":"2024-09-13T13:23:11.910809+0000","last_undegraded":"2024-09-13T13:23:11.910809+0000","last_fullsized":"2024-09-13T13:23:11.910809+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-13T13:22:17.722104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:17.722104+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:28.690048+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063230700000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1083,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910200+0000","last_change":"2024-09-13T13:22:27.400902+0000","last_active":"2024-09-13T13:23:11.910200+0000","last_peered":"2024-09-13T13:23:11.910200+0000","last_clean":"2024-09-13T13:23:11.910200+0000","last_became_active":"2024-09-13T13:22:16.244949+0000","last_became_peered":"2024-09-13T13:22:16.244949+0000","last_unstale":"2024-09-13T13:23:11.910200+0000","last_undegraded":"2024-09-13T13:23:11.910200+0000","last_fullsized":"2024-09-13T13:23:11.910200+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:27.400861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:27.400861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:45.218687+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1094,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643500+0000","last_change":"2024-09-13T13:22:46.399851+0000","last_active":"2024-09-13T13:23:11.643500+0000","last_peered":"2024-09-13T13:23:11.643500+0000","last_clean":"2024-09-13T13:23:11.643500+0000","last_became_active":"2024-09-13T13:22:19.183707+0000","last_became_peered":"2024-09-13T13:22:19.183707+0000","last_unstale":"2024-09-13T13:23:11.643500+0000","last_undegraded":"2024-09-13T13:23:11.643500+0000","last_fullsized":"2024-09-13T13:23:11.643500+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:46.399808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:46.399808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:08.382448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":646,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910345+0000","last_change":"2024-09-13T13:22:47.534825+0000","last_active":"2024-09-13T13:23:11.910345+0000","last_peered":"2024-09-13T13:23:11.910345+0000","last_clean":"2024-09-13T13:23:11.910345+0000","last_became_active":"2024-09-13T13:22:47.534458+0000","last_became_peered":"2024-09-13T13:22:47.534458+0000","last_unstale":"2024-09-13T13:23:11.910345+0000","last_undegraded":"2024-09-13T13:23:11.910345+0000","last_fullsized":"2024-09-13T13:23:11.910345+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:21.394085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.394085+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:24.520621+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"500'21","reported_seq":1094,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911372+0000","last_change":"2024-09-13T13:21:54.845235+0000","last_active":"2024-09-13T13:23:11.911372+0000","last_peered":"2024-09-13T13:23:11.911372+0000","last_clean":"2024-09-13T13:23:11.911372+0000","last_became_active":"2024-09-13T13:01:04.192599+0000","last_became_peered":"2024-09-13T13:01:04.192599+0000","last_unstale":"2024-09-13T13:23:11.911372+0000","last_undegraded":"2024-09-13T13:23:11.911372+0000","last_fullsized":"2024-09-13T13:23:11.911372+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:21:54.845177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:21:54.845177+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:13.333386+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1093,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909555+0000","last_change":"2024-09-13T13:22:04.995555+0000","last_active":"2024-09-13T13:23:11.909555+0000","last_peered":"2024-09-13T13:23:11.909555+0000","last_clean":"2024-09-13T13:23:11.909555+0000","last_became_active":"2024-09-13T13:17:25.246400+0000","last_became_peered":"2024-09-13T13:17:25.246400+0000","last_unstale":"2024-09-13T13:23:11.909555+0000","last_undegraded":"2024-09-13T13:23:11.909555+0000","last_fullsized":"2024-09-13T13:23:11.909555+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:04.995492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:04.995492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:14.088299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":316,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642384+0000","last_change":"2024-09-13T13:22:13.952594+0000","last_active":"2024-09-13T13:23:11.642384+0000","last_peered":"2024-09-13T13:23:11.642384+0000","last_clean":"2024-09-13T13:23:11.642384+0000","last_became_active":"2024-09-13T13:17:25.252664+0000","last_became_peered":"2024-09-13T13:17:25.252664+0000","last_unstale":"2024-09-13T13:23:11.642384+0000","last_undegraded":"2024-09-13T13:23:11.642384+0000","last_fullsized":"2024-09-13T13:23:11.642384+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-13T13:22:13.952523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:13.952523+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:42.420034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":345,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642322+0000","last_change":"2024-09-13T13:22:19.182229+0000","last_active":"2024-09-13T13:23:11.642322+0000","last_peered":"2024-09-13T13:23:11.642322+0000","last_clean":"2024-09-13T13:23:11.642322+0000","last_became_active":"2024-09-13T13:22:19.181898+0000","last_became_peered":"2024-09-13T13:22:19.181898+0000","last_unstale":"2024-09-13T13:23:11.642322+0000","last_undegraded":"2024-09-13T13:23:11.642322+0000","last_fullsized":"2024-09-13T13:23:11.642322+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:18.014751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:18.014751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:24.442038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":657,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911237+0000","last_change":"2024-09-13T13:22:47.532518+0000","last_active":"2024-09-13T13:23:11.911237+0000","last_peered":"2024-09-13T13:23:11.911237+0000","last_clean":"2024-09-13T13:23:11.911237+0000","last_became_active":"2024-09-13T13:22:47.532197+0000","last_became_peered":"2024-09-13T13:22:47.532197+0000","last_unstale":"2024-09-13T13:23:11.911237+0000","last_undegraded":"2024-09-13T13:23:11.911237+0000","last_fullsized":"2024-09-13T13:23:11.911237+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:37.494471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:37.494471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:48.751049+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":307,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910484+0000","last_change":"2024-09-13T13:22:16.984074+0000","last_active":"2024-09-13T13:23:11.910484+0000","last_peered":"2024-09-13T13:23:11.910484+0000","last_clean":"2024-09-13T13:23:11.910484+0000","last_became_active":"2024-09-13T13:22:16.237260+0000","last_became_peered":"2024-09-13T13:22:16.237260+0000","last_unstale":"2024-09-13T13:23:11.910484+0000","last_undegraded":"2024-09-13T13:23:11.910484+0000","last_fullsized":"2024-09-13T13:23:11.910484+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:16.984009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:16.984009+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:34.249874+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"500'36","reported_seq":1150,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910894+0000","last_change":"2024-09-13T13:22:20.418575+0000","last_active":"2024-09-13T13:23:11.910894+0000","last_peered":"2024-09-13T13:23:11.910894+0000","last_clean":"2024-09-13T13:23:11.910894+0000","last_became_active":"2024-09-13T13:17:25.253120+0000","last_became_peered":"2024-09-13T13:17:25.253120+0000","last_unstale":"2024-09-13T13:23:11.910894+0000","last_undegraded":"2024-09-13T13:23:11.910894+0000","last_fullsized":"2024-09-13T13:23:11.910894+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'36","last_scrub_stamp":"2024-09-13T13:22:20.418475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:20.418475+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:42.537200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1091,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910198+0000","last_change":"2024-09-13T13:22:40.338901+0000","last_active":"2024-09-13T13:23:11.910198+0000","last_peered":"2024-09-13T13:23:11.910198+0000","last_clean":"2024-09-13T13:23:11.910198+0000","last_became_active":"2024-09-13T13:22:16.243256+0000","last_became_peered":"2024-09-13T13:22:16.243256+0000","last_unstale":"2024-09-13T13:23:11.910198+0000","last_undegraded":"2024-09-13T13:23:11.910198+0000","last_fullsized":"2024-09-13T13:23:11.910198+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:40.338852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:40.338852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:49.459516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1197,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:16.577010+0000","last_change":"2024-09-13T13:23:14.112429+0000","last_active":"2024-09-13T13:23:16.577010+0000","last_peered":"2024-09-13T13:23:16.577010+0000","last_clean":"2024-09-13T13:23:16.577010+0000","last_became_active":"2024-09-13T13:22:47.533679+0000","last_became_peered":"2024-09-13T13:22:47.533679+0000","last_unstale":"2024-09-13T13:23:16.577010+0000","last_undegraded":"2024-09-13T13:23:16.577010+0000","last_fullsized":"2024-09-13T13:23:16.577010+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-13T13:23:14.112335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:23:14.112335+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:27.805334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00071015599999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":608,"num_read_kb":603,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1042,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911318+0000","last_change":"2024-09-13T13:22:18.751296+0000","last_active":"2024-09-13T13:23:11.911318+0000","last_peered":"2024-09-13T13:23:11.911318+0000","last_clean":"2024-09-13T13:23:11.911318+0000","last_became_active":"2024-09-13T13:01:06.208144+0000","last_became_peered":"2024-09-13T13:01:06.208144+0000","last_unstale":"2024-09-13T13:23:11.911318+0000","last_undegraded":"2024-09-13T13:23:11.911318+0000","last_fullsized":"2024-09-13T13:23:11.911318+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:18.751227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:18.751227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:32.619966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1047,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910115+0000","last_change":"2024-09-13T13:22:16.014224+0000","last_active":"2024-09-13T13:23:11.910115+0000","last_peered":"2024-09-13T13:23:11.910115+0000","last_clean":"2024-09-13T13:23:11.910115+0000","last_became_active":"2024-09-13T13:01:08.213170+0000","last_became_peered":"2024-09-13T13:01:08.213170+0000","last_unstale":"2024-09-13T13:23:11.910115+0000","last_undegraded":"2024-09-13T13:23:11.910115+0000","last_fullsized":"2024-09-13T13:23:11.910115+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:16.014163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:16.014163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:38.568946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"527'48","reported_seq":1346,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.648350+0000","last_change":"2024-09-13T13:22:24.618283+0000","last_active":"2024-09-13T13:23:11.648350+0000","last_peered":"2024-09-13T13:23:11.648350+0000","last_clean":"2024-09-13T13:23:11.648350+0000","last_became_active":"2024-09-13T13:22:19.182848+0000","last_became_peered":"2024-09-13T13:22:19.182848+0000","last_unstale":"2024-09-13T13:23:11.648350+0000","last_undegraded":"2024-09-13T13:23:11.648350+0000","last_fullsized":"2024-09-13T13:23:11.648350+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'48","last_scrub_stamp":"2024-09-13T13:22:24.618210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:24.618210+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:52.030178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1070,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:16.379867+0000","last_change":"2024-09-13T13:23:16.379867+0000","last_active":"2024-09-13T13:23:16.379867+0000","last_peered":"2024-09-13T13:23:16.379867+0000","last_clean":"2024-09-13T13:23:16.379867+0000","last_became_active":"2024-09-13T13:04:02.764962+0000","last_became_peered":"2024-09-13T13:04:02.764962+0000","last_unstale":"2024-09-13T13:23:16.379867+0000","last_undegraded":"2024-09-13T13:23:16.379867+0000","last_fullsized":"2024-09-13T13:23:16.379867+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:16.379819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:23:16.379819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:23.065831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":632,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912016+0000","last_change":"2024-09-13T13:22:47.533933+0000","last_active":"2024-09-13T13:23:11.912016+0000","last_peered":"2024-09-13T13:23:11.912016+0000","last_clean":"2024-09-13T13:23:11.912016+0000","last_became_active":"2024-09-13T13:22:47.533570+0000","last_became_peered":"2024-09-13T13:22:47.533570+0000","last_unstale":"2024-09-13T13:23:11.912016+0000","last_undegraded":"2024-09-13T13:23:11.912016+0000","last_fullsized":"2024-09-13T13:23:11.912016+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:25.340401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:25.340401+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:30.901220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":305,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911416+0000","last_change":"2024-09-13T13:22:47.532580+0000","last_active":"2024-09-13T13:23:11.911416+0000","last_peered":"2024-09-13T13:23:11.911416+0000","last_clean":"2024-09-13T13:23:11.911416+0000","last_became_active":"2024-09-13T13:22:47.531816+0000","last_became_peered":"2024-09-13T13:22:47.531816+0000","last_unstale":"2024-09-13T13:23:11.911416+0000","last_undegraded":"2024-09-13T13:23:11.911416+0000","last_fullsized":"2024-09-13T13:23:11.911416+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:14.978063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:14.978063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:17.499392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1103,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911535+0000","last_change":"2024-09-13T13:22:25.669469+0000","last_active":"2024-09-13T13:23:11.911535+0000","last_peered":"2024-09-13T13:23:11.911535+0000","last_clean":"2024-09-13T13:23:11.911535+0000","last_became_active":"2024-09-13T13:22:16.234810+0000","last_became_peered":"2024-09-13T13:22:16.234810+0000","last_unstale":"2024-09-13T13:23:11.911535+0000","last_undegraded":"2024-09-13T13:23:11.911535+0000","last_fullsized":"2024-09-13T13:23:11.911535+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:25.669424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:25.669424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:54.213701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"500'27","reported_seq":527,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910318+0000","last_change":"2024-09-13T13:22:36.131450+0000","last_active":"2024-09-13T13:23:11.910318+0000","last_peered":"2024-09-13T13:23:11.910318+0000","last_clean":"2024-09-13T13:23:11.910318+0000","last_became_active":"2024-09-13T13:22:16.230526+0000","last_became_peered":"2024-09-13T13:22:16.230526+0000","last_unstale":"2024-09-13T13:23:11.910318+0000","last_undegraded":"2024-09-13T13:23:11.910318+0000","last_fullsized":"2024-09-13T13:23:11.910318+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'27","last_scrub_stamp":"2024-09-13T13:22:36.131368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:36.131368+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:54.686004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":524,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911127+0000","last_change":"2024-09-13T13:22:19.181749+0000","last_active":"2024-09-13T13:23:11.911127+0000","last_peered":"2024-09-13T13:23:11.911127+0000","last_clean":"2024-09-13T13:23:11.911127+0000","last_became_active":"2024-09-13T13:22:19.181452+0000","last_became_peered":"2024-09-13T13:22:19.181452+0000","last_unstale":"2024-09-13T13:23:11.911127+0000","last_undegraded":"2024-09-13T13:23:11.911127+0000","last_fullsized":"2024-09-13T13:23:11.911127+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:21:23.132552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:21:23.132552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1109,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911660+0000","last_change":"2024-09-13T13:22:21.702328+0000","last_active":"2024-09-13T13:23:11.911660+0000","last_peered":"2024-09-13T13:23:11.911660+0000","last_clean":"2024-09-13T13:23:11.911660+0000","last_became_active":"2024-09-13T13:22:16.234609+0000","last_became_peered":"2024-09-13T13:22:16.234609+0000","last_unstale":"2024-09-13T13:23:11.911660+0000","last_undegraded":"2024-09-13T13:23:11.911660+0000","last_fullsized":"2024-09-13T13:23:11.911660+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:21.702274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.702274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:49.240755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1049,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643166+0000","last_change":"2024-09-13T13:22:22.360530+0000","last_active":"2024-09-13T13:23:11.643166+0000","last_peered":"2024-09-13T13:23:11.643166+0000","last_clean":"2024-09-13T13:23:11.643166+0000","last_became_active":"2024-09-13T13:04:02.756946+0000","last_became_peered":"2024-09-13T13:04:02.756946+0000","last_unstale":"2024-09-13T13:23:11.643166+0000","last_undegraded":"2024-09-13T13:23:11.643166+0000","last_fullsized":"2024-09-13T13:23:11.643166+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:22.360486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:22.360486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:30.852654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":305,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909661+0000","last_change":"2024-09-13T13:22:26.695747+0000","last_active":"2024-09-13T13:23:11.909661+0000","last_peered":"2024-09-13T13:23:11.909661+0000","last_clean":"2024-09-13T13:23:11.909661+0000","last_became_active":"2024-09-13T13:17:49.053269+0000","last_became_peered":"2024-09-13T13:17:49.053269+0000","last_unstale":"2024-09-13T13:23:11.909661+0000","last_undegraded":"2024-09-13T13:23:11.909661+0000","last_fullsized":"2024-09-13T13:23:11.909661+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:22:26.695675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:22:26.695675+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:51.057379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"500'45","reported_seq":592,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911558+0000","last_change":"2024-09-13T13:22:54.285628+0000","last_active":"2024-09-13T13:23:11.911558+0000","last_peered":"2024-09-13T13:23:11.911558+0000","last_clean":"2024-09-13T13:23:11.911558+0000","last_became_active":"2024-09-13T13:22:16.233365+0000","last_became_peered":"2024-09-13T13:22:16.233365+0000","last_unstale":"2024-09-13T13:23:11.911558+0000","last_undegraded":"2024-09-13T13:23:11.911558+0000","last_fullsized":"2024-09-13T13:23:11.911558+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'45","last_scrub_stamp":"2024-09-13T13:22:54.285538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:54.285538+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:16.096853+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1100,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.648146+0000","last_change":"2024-09-13T13:22:18.642070+0000","last_active":"2024-09-13T13:23:11.648146+0000","last_peered":"2024-09-13T13:23:11.648146+0000","last_clean":"2024-09-13T13:23:11.648146+0000","last_became_active":"2024-09-13T13:22:16.237377+0000","last_became_peered":"2024-09-13T13:22:16.237377+0000","last_unstale":"2024-09-13T13:23:11.648146+0000","last_undegraded":"2024-09-13T13:23:11.648146+0000","last_fullsized":"2024-09-13T13:23:11.648146+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:18.642020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:18.642020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:38.310216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1091,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910235+0000","last_change":"2024-09-13T13:22:53.248219+0000","last_active":"2024-09-13T13:23:11.910235+0000","last_peered":"2024-09-13T13:23:11.910235+0000","last_clean":"2024-09-13T13:23:11.910235+0000","last_became_active":"2024-09-13T13:22:16.235993+0000","last_became_peered":"2024-09-13T13:22:16.235993+0000","last_unstale":"2024-09-13T13:23:11.910235+0000","last_undegraded":"2024-09-13T13:23:11.910235+0000","last_fullsized":"2024-09-13T13:23:11.910235+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:53.248174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:53.248174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:03.348537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":815,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642476+0000","last_change":"2024-09-13T13:22:40.208908+0000","last_active":"2024-09-13T13:23:11.642476+0000","last_peered":"2024-09-13T13:23:11.642476+0000","last_clean":"2024-09-13T13:23:11.642476+0000","last_became_active":"2024-09-13T13:22:35.478558+0000","last_became_peered":"2024-09-13T13:22:35.478558+0000","last_unstale":"2024-09-13T13:23:11.642476+0000","last_undegraded":"2024-09-13T13:23:11.642476+0000","last_fullsized":"2024-09-13T13:23:11.642476+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:40.208856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:40.208856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:00.142193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":289,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912089+0000","last_change":"2024-09-13T13:23:09.248748+0000","last_active":"2024-09-13T13:23:11.912089+0000","last_peered":"2024-09-13T13:23:11.912089+0000","last_clean":"2024-09-13T13:23:11.912089+0000","last_became_active":"2024-09-13T13:22:47.528411+0000","last_became_peered":"2024-09-13T13:22:47.528411+0000","last_unstale":"2024-09-13T13:23:11.912089+0000","last_undegraded":"2024-09-13T13:23:11.912089+0000","last_fullsized":"2024-09-13T13:23:11.912089+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-13T13:23:09.248673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:58:31.642092+0000","last_clean_scrub_stamp":"2024-09-13T13:23:09.248673+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:13.574733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"500'30","reported_seq":514,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911741+0000","last_change":"2024-09-13T13:22:19.131026+0000","last_active":"2024-09-13T13:23:11.911741+0000","last_peered":"2024-09-13T13:23:11.911741+0000","last_clean":"2024-09-13T13:23:11.911741+0000","last_became_active":"2024-09-13T13:22:16.235373+0000","last_became_peered":"2024-09-13T13:22:16.235373+0000","last_unstale":"2024-09-13T13:23:11.911741+0000","last_undegraded":"2024-09-13T13:23:11.911741+0000","last_fullsized":"2024-09-13T13:23:11.911741+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'30","last_scrub_stamp":"2024-09-13T13:22:19.130964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:19.130964+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:32.369797+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1067,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643020+0000","last_change":"2024-09-13T13:22:28.411364+0000","last_active":"2024-09-13T13:23:11.643020+0000","last_peered":"2024-09-13T13:23:11.643020+0000","last_clean":"2024-09-13T13:23:11.643020+0000","last_became_active":"2024-09-13T13:22:16.232309+0000","last_became_peered":"2024-09-13T13:22:16.232309+0000","last_unstale":"2024-09-13T13:23:11.643020+0000","last_undegraded":"2024-09-13T13:23:11.643020+0000","last_fullsized":"2024-09-13T13:23:11.643020+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:28.411316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:28.411316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:46.788056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"527'12","reported_seq":418,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:12.507592+0000","last_change":"2024-09-13T13:22:21.092049+0000","last_active":"2024-09-13T13:23:12.507592+0000","last_peered":"2024-09-13T13:23:12.507592+0000","last_clean":"2024-09-13T13:23:12.507592+0000","last_became_active":"2024-09-13T13:22:19.177775+0000","last_became_peered":"2024-09-13T13:22:19.177775+0000","last_unstale":"2024-09-13T13:23:12.507592+0000","last_undegraded":"2024-09-13T13:23:12.507592+0000","last_fullsized":"2024-09-13T13:23:12.507592+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-09-13T13:22:21.091979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.091979+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:25.002440+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":478,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910853+0000","last_change":"2024-09-13T13:22:48.353671+0000","last_active":"2024-09-13T13:23:11.910853+0000","last_peered":"2024-09-13T13:23:11.910853+0000","last_clean":"2024-09-13T13:23:11.910853+0000","last_became_active":"2024-09-13T13:22:16.235741+0000","last_became_peered":"2024-09-13T13:22:16.235741+0000","last_unstale":"2024-09-13T13:23:11.910853+0000","last_undegraded":"2024-09-13T13:23:11.910853+0000","last_fullsized":"2024-09-13T13:23:11.910853+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:48.353583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:48.353583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:05.112215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"500'21","reported_seq":1108,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910723+0000","last_change":"2024-09-13T13:22:32.427139+0000","last_active":"2024-09-13T13:23:11.910723+0000","last_peered":"2024-09-13T13:23:11.910723+0000","last_clean":"2024-09-13T13:23:11.910723+0000","last_became_active":"2024-09-13T13:22:16.240384+0000","last_became_peered":"2024-09-13T13:22:16.240384+0000","last_unstale":"2024-09-13T13:23:11.910723+0000","last_undegraded":"2024-09-13T13:23:11.910723+0000","last_fullsized":"2024-09-13T13:23:11.910723+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:22:32.427082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:32.427082+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:58.305683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1084,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:17.965428+0000","last_change":"2024-09-13T13:23:17.965428+0000","last_active":"2024-09-13T13:23:17.965428+0000","last_peered":"2024-09-13T13:23:17.965428+0000","last_clean":"2024-09-13T13:23:17.965428+0000","last_became_active":"2024-09-13T13:04:02.748621+0000","last_became_peered":"2024-09-13T13:04:02.748621+0000","last_unstale":"2024-09-13T13:23:17.965428+0000","last_undegraded":"2024-09-13T13:23:17.965428+0000","last_fullsized":"2024-09-13T13:23:17.965428+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:17.965363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:23:17.965363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:46.933254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":760,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642779+0000","last_change":"2024-09-13T13:22:35.478889+0000","last_active":"2024-09-13T13:23:11.642779+0000","last_peered":"2024-09-13T13:23:11.642779+0000","last_clean":"2024-09-13T13:23:11.642779+0000","last_became_active":"2024-09-13T13:22:35.478590+0000","last_became_peered":"2024-09-13T13:22:35.478590+0000","last_unstale":"2024-09-13T13:23:11.642779+0000","last_undegraded":"2024-09-13T13:23:11.642779+0000","last_fullsized":"2024-09-13T13:23:11.642779+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:26.187534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:26.187534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:31.475512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1074,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:14.154185+0000","last_change":"2024-09-13T13:23:14.154185+0000","last_active":"2024-09-13T13:23:14.154185+0000","last_peered":"2024-09-13T13:23:14.154185+0000","last_clean":"2024-09-13T13:23:14.154185+0000","last_became_active":"2024-09-13T13:22:19.181437+0000","last_became_peered":"2024-09-13T13:22:19.181437+0000","last_unstale":"2024-09-13T13:23:14.154185+0000","last_undegraded":"2024-09-13T13:23:14.154185+0000","last_fullsized":"2024-09-13T13:23:14.154185+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:14.154130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:23:14.154130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:23.655302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"500'51","reported_seq":1140,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.647963+0000","last_change":"2024-09-13T13:22:19.204413+0000","last_active":"2024-09-13T13:23:11.647963+0000","last_peered":"2024-09-13T13:23:11.647963+0000","last_clean":"2024-09-13T13:23:11.647963+0000","last_became_active":"2024-09-13T13:22:19.181886+0000","last_became_peered":"2024-09-13T13:22:19.181886+0000","last_unstale":"2024-09-13T13:23:11.647963+0000","last_undegraded":"2024-09-13T13:23:11.647963+0000","last_fullsized":"2024-09-13T13:23:11.647963+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'51","last_scrub_stamp":"2024-09-13T13:22:05.631182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:05.631182+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:22.090957+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":676,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912454+0000","last_change":"2024-09-13T13:22:54.301383+0000","last_active":"2024-09-13T13:23:11.912454+0000","last_peered":"2024-09-13T13:23:11.912454+0000","last_clean":"2024-09-13T13:23:11.912454+0000","last_became_active":"2024-09-13T13:22:47.532468+0000","last_became_peered":"2024-09-13T13:22:47.532468+0000","last_unstale":"2024-09-13T13:23:11.912454+0000","last_undegraded":"2024-09-13T13:23:11.912454+0000","last_fullsized":"2024-09-13T13:23:11.912454+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:54.301324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:54.301324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:03.666147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":312,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642711+0000","last_change":"2024-09-13T13:22:07.927188+0000","last_active":"2024-09-13T13:23:11.642711+0000","last_peered":"2024-09-13T13:23:11.642711+0000","last_clean":"2024-09-13T13:23:11.642711+0000","last_became_active":"2024-09-13T13:17:25.332964+0000","last_became_peered":"2024-09-13T13:17:25.332964+0000","last_unstale":"2024-09-13T13:23:11.642711+0000","last_undegraded":"2024-09-13T13:23:11.642711+0000","last_fullsized":"2024-09-13T13:23:11.642711+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:07.927129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:07.927129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:29.295379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":279,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912059+0000","last_change":"2024-09-13T13:23:06.338891+0000","last_active":"2024-09-13T13:23:11.912059+0000","last_peered":"2024-09-13T13:23:11.912059+0000","last_clean":"2024-09-13T13:23:11.912059+0000","last_became_active":"2024-09-13T13:22:47.527491+0000","last_became_peered":"2024-09-13T13:22:47.527491+0000","last_unstale":"2024-09-13T13:23:11.912059+0000","last_undegraded":"2024-09-13T13:23:11.912059+0000","last_fullsized":"2024-09-13T13:23:11.912059+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:06.338822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:23:06.338822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:35.121728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"500'24","reported_seq":524,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:16.141395+0000","last_change":"2024-09-13T13:23:16.141395+0000","last_active":"2024-09-13T13:23:16.141395+0000","last_peered":"2024-09-13T13:23:16.141395+0000","last_clean":"2024-09-13T13:23:16.141395+0000","last_became_active":"2024-09-13T13:22:16.233929+0000","last_became_peered":"2024-09-13T13:22:16.233929+0000","last_unstale":"2024-09-13T13:23:16.141395+0000","last_undegraded":"2024-09-13T13:23:16.141395+0000","last_fullsized":"2024-09-13T13:23:16.141395+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'24","last_scrub_stamp":"2024-09-13T13:23:16.141323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:23:16.141323+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:41.907732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":470,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912500+0000","last_change":"2024-09-13T13:22:53.286147+0000","last_active":"2024-09-13T13:23:11.912500+0000","last_peered":"2024-09-13T13:23:11.912500+0000","last_clean":"2024-09-13T13:23:11.912500+0000","last_became_active":"2024-09-13T13:22:47.535502+0000","last_became_peered":"2024-09-13T13:22:47.535502+0000","last_unstale":"2024-09-13T13:23:11.912500+0000","last_undegraded":"2024-09-13T13:23:11.912500+0000","last_fullsized":"2024-09-13T13:23:11.912500+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:53.286090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:53.286090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:17.427121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":156,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.641892+0000","last_change":"2024-09-13T13:22:35.853140+0000","last_active":"2024-09-13T13:23:11.641892+0000","last_peered":"2024-09-13T13:23:11.641892+0000","last_clean":"2024-09-13T13:23:11.641892+0000","last_became_active":"2024-09-13T13:22:35.852785+0000","last_became_peered":"2024-09-13T13:22:35.852785+0000","last_unstale":"2024-09-13T13:23:11.641892+0000","last_undegraded":"2024-09-13T13:23:11.641892+0000","last_fullsized":"2024-09-13T13:23:11.641892+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:09.512475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:09.512475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:20.652174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1052,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642677+0000","last_change":"2024-09-13T13:22:37.448390+0000","last_active":"2024-09-13T13:23:11.642677+0000","last_peered":"2024-09-13T13:23:11.642677+0000","last_clean":"2024-09-13T13:23:11.642677+0000","last_became_active":"2024-09-13T13:22:19.171361+0000","last_became_peered":"2024-09-13T13:22:19.171361+0000","last_unstale":"2024-09-13T13:23:11.642677+0000","last_undegraded":"2024-09-13T13:23:11.642677+0000","last_fullsized":"2024-09-13T13:23:11.642677+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:37.448338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:37.448338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:46.152282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"500'39","reported_seq":1199,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909761+0000","last_change":"2024-09-13T13:22:28.725925+0000","last_active":"2024-09-13T13:23:11.909761+0000","last_peered":"2024-09-13T13:23:11.909761+0000","last_clean":"2024-09-13T13:23:11.909761+0000","last_became_active":"2024-09-13T13:22:19.170732+0000","last_became_peered":"2024-09-13T13:22:19.170732+0000","last_unstale":"2024-09-13T13:23:11.909761+0000","last_undegraded":"2024-09-13T13:23:11.909761+0000","last_fullsized":"2024-09-13T13:23:11.909761+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'39","last_scrub_stamp":"2024-09-13T13:22:28.725850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:28.725850+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:28.974062+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":462,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912523+0000","last_change":"2024-09-13T13:22:48.307594+0000","last_active":"2024-09-13T13:23:11.912523+0000","last_peered":"2024-09-13T13:23:11.912523+0000","last_clean":"2024-09-13T13:23:11.912523+0000","last_became_active":"2024-09-13T13:22:47.535093+0000","last_became_peered":"2024-09-13T13:22:47.535093+0000","last_unstale":"2024-09-13T13:23:11.912523+0000","last_undegraded":"2024-09-13T13:23:11.912523+0000","last_fullsized":"2024-09-13T13:23:11.912523+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-13T13:22:48.307530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:48.307530+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:58.238165+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"542'18","reported_seq":863,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:12.508102+0000","last_change":"2024-09-13T13:22:47.537118+0000","last_active":"2024-09-13T13:23:12.508102+0000","last_peered":"2024-09-13T13:23:12.508102+0000","last_clean":"2024-09-13T13:23:12.508102+0000","last_became_active":"2024-09-13T13:22:47.536735+0000","last_became_peered":"2024-09-13T13:22:47.536735+0000","last_unstale":"2024-09-13T13:23:12.508102+0000","last_undegraded":"2024-09-13T13:23:12.508102+0000","last_fullsized":"2024-09-13T13:23:12.508102+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'17","last_scrub_stamp":"2024-09-13T13:22:35.467482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:35.467482+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:54.275482+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1057,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910529+0000","last_change":"2024-09-13T13:23:04.914299+0000","last_active":"2024-09-13T13:23:11.910529+0000","last_peered":"2024-09-13T13:23:11.910529+0000","last_clean":"2024-09-13T13:23:11.910529+0000","last_became_active":"2024-09-13T13:04:02.757205+0000","last_became_peered":"2024-09-13T13:04:02.757205+0000","last_unstale":"2024-09-13T13:23:11.910529+0000","last_undegraded":"2024-09-13T13:23:11.910529+0000","last_fullsized":"2024-09-13T13:23:11.910529+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:04.914219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:23:04.914219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:28.450483+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1027,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.647776+0000","last_change":"2024-09-13T13:22:04.595087+0000","last_active":"2024-09-13T13:23:11.647776+0000","last_peered":"2024-09-13T13:23:11.647776+0000","last_clean":"2024-09-13T13:23:11.647776+0000","last_became_active":"2024-09-13T13:04:02.756254+0000","last_became_peered":"2024-09-13T13:04:02.756254+0000","last_unstale":"2024-09-13T13:23:11.647776+0000","last_undegraded":"2024-09-13T13:23:11.647776+0000","last_fullsized":"2024-09-13T13:23:11.647776+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:04.595027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:04.595027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:24.252511+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"500'18","reported_seq":618,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:16.981535+0000","last_change":"2024-09-13T13:23:16.981535+0000","last_active":"2024-09-13T13:23:16.981535+0000","last_peered":"2024-09-13T13:23:16.981535+0000","last_clean":"2024-09-13T13:23:16.981535+0000","last_became_active":"2024-09-13T13:22:19.170662+0000","last_became_peered":"2024-09-13T13:22:19.170662+0000","last_unstale":"2024-09-13T13:23:16.981535+0000","last_undegraded":"2024-09-13T13:23:16.981535+0000","last_fullsized":"2024-09-13T13:23:16.981535+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'18","last_scrub_stamp":"2024-09-13T13:23:16.981428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:23:16.981428+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:38.777943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1081,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910291+0000","last_change":"2024-09-13T13:22:35.850436+0000","last_active":"2024-09-13T13:23:11.910291+0000","last_peered":"2024-09-13T13:23:11.910291+0000","last_clean":"2024-09-13T13:23:11.910291+0000","last_became_active":"2024-09-13T13:01:06.207839+0000","last_became_peered":"2024-09-13T13:01:06.207839+0000","last_unstale":"2024-09-13T13:23:11.910291+0000","last_undegraded":"2024-09-13T13:23:11.910291+0000","last_fullsized":"2024-09-13T13:23:11.910291+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:35.850384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:35.850384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:54.841106+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":70,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912288+0000","last_change":"2024-09-13T13:22:21.182286+0000","last_active":"2024-09-13T13:23:11.912288+0000","last_peered":"2024-09-13T13:23:11.912288+0000","last_clean":"2024-09-13T13:23:11.912288+0000","last_became_active":"2024-09-13T13:22:19.182619+0000","last_became_peered":"2024-09-13T13:22:19.182619+0000","last_unstale":"2024-09-13T13:23:11.912288+0000","last_undegraded":"2024-09-13T13:23:11.912288+0000","last_fullsized":"2024-09-13T13:23:11.912288+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:21.182244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.182244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:28.883635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"500'21","reported_seq":1113,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910997+0000","last_change":"2024-09-13T13:22:19.213600+0000","last_active":"2024-09-13T13:23:11.910997+0000","last_peered":"2024-09-13T13:23:11.910997+0000","last_clean":"2024-09-13T13:23:11.910997+0000","last_became_active":"2024-09-13T13:22:19.186228+0000","last_became_peered":"2024-09-13T13:22:19.186228+0000","last_unstale":"2024-09-13T13:23:11.910997+0000","last_undegraded":"2024-09-13T13:23:11.910997+0000","last_fullsized":"2024-09-13T13:23:11.910997+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:22:15.517537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:15.517537+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:37.123521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":452,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911643+0000","last_change":"2024-09-13T13:22:49.261274+0000","last_active":"2024-09-13T13:23:11.911643+0000","last_peered":"2024-09-13T13:23:11.911643+0000","last_clean":"2024-09-13T13:23:11.911643+0000","last_became_active":"2024-09-13T13:22:47.535128+0000","last_became_peered":"2024-09-13T13:22:47.535128+0000","last_unstale":"2024-09-13T13:23:11.911643+0000","last_undegraded":"2024-09-13T13:23:11.911643+0000","last_fullsized":"2024-09-13T13:23:11.911643+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:49.261197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:49.261197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:14.816989+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1072,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643670+0000","last_change":"2024-09-13T13:22:43.367744+0000","last_active":"2024-09-13T13:23:11.643670+0000","last_peered":"2024-09-13T13:23:11.643670+0000","last_clean":"2024-09-13T13:23:11.643670+0000","last_became_active":"2024-09-13T13:22:19.170501+0000","last_became_peered":"2024-09-13T13:22:19.170501+0000","last_unstale":"2024-09-13T13:23:11.643670+0000","last_undegraded":"2024-09-13T13:23:11.643670+0000","last_fullsized":"2024-09-13T13:23:11.643670+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:43.367697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:43.367697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:45.151015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":337,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642854+0000","last_change":"2024-09-13T13:22:19.181069+0000","last_active":"2024-09-13T13:23:11.642854+0000","last_peered":"2024-09-13T13:23:11.642854+0000","last_clean":"2024-09-13T13:23:11.642854+0000","last_became_active":"2024-09-13T13:22:19.180372+0000","last_became_peered":"2024-09-13T13:22:19.180372+0000","last_unstale":"2024-09-13T13:23:11.642854+0000","last_undegraded":"2024-09-13T13:23:11.642854+0000","last_fullsized":"2024-09-13T13:23:11.642854+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:17.028121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:17.028121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:45.610983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"500'21","reported_seq":1134,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911479+0000","last_change":"2024-09-13T13:22:19.208997+0000","last_active":"2024-09-13T13:23:11.911479+0000","last_peered":"2024-09-13T13:23:11.911479+0000","last_clean":"2024-09-13T13:23:11.911479+0000","last_became_active":"2024-09-13T13:22:19.174582+0000","last_became_peered":"2024-09-13T13:22:19.174582+0000","last_unstale":"2024-09-13T13:23:11.911479+0000","last_undegraded":"2024-09-13T13:23:11.911479+0000","last_fullsized":"2024-09-13T13:23:11.911479+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:22:00.644695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:00.644695+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:21.578445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1055,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910325+0000","last_change":"2024-09-13T13:22:19.181892+0000","last_active":"2024-09-13T13:23:11.910325+0000","last_peered":"2024-09-13T13:23:11.910325+0000","last_clean":"2024-09-13T13:23:11.910325+0000","last_became_active":"2024-09-13T13:22:19.181582+0000","last_became_peered":"2024-09-13T13:22:19.181582+0000","last_unstale":"2024-09-13T13:23:11.910325+0000","last_undegraded":"2024-09-13T13:23:11.910325+0000","last_fullsized":"2024-09-13T13:23:11.910325+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:05.491514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:05.491514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:35.470624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"527'12","reported_seq":1355,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:12.507681+0000","last_change":"2024-09-13T13:22:19.217268+0000","last_active":"2024-09-13T13:23:12.507681+0000","last_peered":"2024-09-13T13:23:12.507681+0000","last_clean":"2024-09-13T13:23:12.507681+0000","last_became_active":"2024-09-13T13:22:19.184087+0000","last_became_peered":"2024-09-13T13:22:19.184087+0000","last_unstale":"2024-09-13T13:23:12.507681+0000","last_undegraded":"2024-09-13T13:23:12.507681+0000","last_fullsized":"2024-09-13T13:23:12.507681+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11","last_scrub_stamp":"2024-09-13T13:21:58.322548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:21:58.322548+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:25.836363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":478,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911336+0000","last_change":"2024-09-13T13:22:44.323857+0000","last_active":"2024-09-13T13:23:11.911336+0000","last_peered":"2024-09-13T13:23:11.911336+0000","last_clean":"2024-09-13T13:23:11.911336+0000","last_became_active":"2024-09-13T13:22:16.227359+0000","last_became_peered":"2024-09-13T13:22:16.227359+0000","last_unstale":"2024-09-13T13:23:11.911336+0000","last_undegraded":"2024-09-13T13:23:11.911336+0000","last_fullsized":"2024-09-13T13:23:11.911336+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:44.323776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:44.323776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:50.755419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"500'30","reported_seq":1153,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643941+0000","last_change":"2024-09-13T13:22:34.456861+0000","last_active":"2024-09-13T13:23:11.643941+0000","last_peered":"2024-09-13T13:23:11.643941+0000","last_clean":"2024-09-13T13:23:11.643941+0000","last_became_active":"2024-09-13T13:22:19.183816+0000","last_became_peered":"2024-09-13T13:22:19.183816+0000","last_unstale":"2024-09-13T13:23:11.643941+0000","last_undegraded":"2024-09-13T13:23:11.643941+0000","last_fullsized":"2024-09-13T13:23:11.643941+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'30","last_scrub_stamp":"2024-09-13T13:22:34.456804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:34.456804+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:45.262815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":70,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911042+0000","last_change":"2024-09-13T13:22:20.138632+0000","last_active":"2024-09-13T13:23:11.911042+0000","last_peered":"2024-09-13T13:23:11.911042+0000","last_clean":"2024-09-13T13:23:11.911042+0000","last_became_active":"2024-09-13T13:22:19.179741+0000","last_became_peered":"2024-09-13T13:22:19.179741+0000","last_unstale":"2024-09-13T13:23:11.911042+0000","last_undegraded":"2024-09-13T13:23:11.911042+0000","last_fullsized":"2024-09-13T13:23:11.911042+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:20.138585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:20.138585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:25.482073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":1349,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:12.507893+0000","last_change":"2024-09-13T13:23:06.836450+0000","last_active":"2024-09-13T13:23:12.507893+0000","last_peered":"2024-09-13T13:23:12.507893+0000","last_clean":"2024-09-13T13:23:12.507893+0000","last_became_active":"2024-09-13T13:01:06.206163+0000","last_became_peered":"2024-09-13T13:01:06.206163+0000","last_unstale":"2024-09-13T13:23:12.507893+0000","last_undegraded":"2024-09-13T13:23:12.507893+0000","last_fullsized":"2024-09-13T13:23:12.507893+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'8","last_scrub_stamp":"2024-09-13T13:23:06.836372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:23:06.836372+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:08.930779+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1078,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910725+0000","last_change":"2024-09-13T13:22:19.187614+0000","last_active":"2024-09-13T13:23:11.910725+0000","last_peered":"2024-09-13T13:23:11.910725+0000","last_clean":"2024-09-13T13:23:11.910725+0000","last_became_active":"2024-09-13T13:22:19.187294+0000","last_became_peered":"2024-09-13T13:22:19.187294+0000","last_unstale":"2024-09-13T13:23:11.910725+0000","last_undegraded":"2024-09-13T13:23:11.910725+0000","last_fullsized":"2024-09-13T13:23:11.910725+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:16.488651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:16.488651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:42.763643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"500'27","reported_seq":519,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910257+0000","last_change":"2024-09-13T13:22:46.352617+0000","last_active":"2024-09-13T13:23:11.910257+0000","last_peered":"2024-09-13T13:23:11.910257+0000","last_clean":"2024-09-13T13:23:11.910257+0000","last_became_active":"2024-09-13T13:22:16.230929+0000","last_became_peered":"2024-09-13T13:22:16.230929+0000","last_unstale":"2024-09-13T13:23:11.910257+0000","last_undegraded":"2024-09-13T13:23:11.910257+0000","last_fullsized":"2024-09-13T13:23:11.910257+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'27","last_scrub_stamp":"2024-09-13T13:22:46.352558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:46.352558+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:05.088199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":488,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.911859+0000","last_change":"2024-09-13T13:22:19.187717+0000","last_active":"2024-09-13T13:23:11.911859+0000","last_peered":"2024-09-13T13:23:11.911859+0000","last_clean":"2024-09-13T13:23:11.911859+0000","last_became_active":"2024-09-13T13:22:19.187348+0000","last_became_peered":"2024-09-13T13:22:19.187348+0000","last_unstale":"2024-09-13T13:23:11.911859+0000","last_undegraded":"2024-09-13T13:23:11.911859+0000","last_fullsized":"2024-09-13T13:23:11.911859+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:20:53.985950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:20:53.985950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1066,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910145+0000","last_change":"2024-09-13T13:23:03.302340+0000","last_active":"2024-09-13T13:23:11.910145+0000","last_peered":"2024-09-13T13:23:11.910145+0000","last_clean":"2024-09-13T13:23:11.910145+0000","last_became_active":"2024-09-13T13:01:06.207118+0000","last_became_peered":"2024-09-13T13:01:06.207118+0000","last_unstale":"2024-09-13T13:23:11.910145+0000","last_undegraded":"2024-09-13T13:23:11.910145+0000","last_fullsized":"2024-09-13T13:23:11.910145+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:03.302284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:23:03.302284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:07.736447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1062,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643804+0000","last_change":"2024-09-13T13:22:56.330924+0000","last_active":"2024-09-13T13:23:11.643804+0000","last_peered":"2024-09-13T13:23:11.643804+0000","last_clean":"2024-09-13T13:23:11.643804+0000","last_became_active":"2024-09-13T13:01:08.205059+0000","last_became_peered":"2024-09-13T13:01:08.205059+0000","last_unstale":"2024-09-13T13:23:11.643804+0000","last_undegraded":"2024-09-13T13:23:11.643804+0000","last_fullsized":"2024-09-13T13:23:11.643804+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:56.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:56.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:04.534532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"500'21","reported_seq":1056,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910365+0000","last_change":"2024-09-13T13:22:05.680067+0000","last_active":"2024-09-13T13:23:11.910365+0000","last_peered":"2024-09-13T13:23:11.910365+0000","last_clean":"2024-09-13T13:23:11.910365+0000","last_became_active":"2024-09-13T13:17:25.331530+0000","last_became_peered":"2024-09-13T13:17:25.331530+0000","last_unstale":"2024-09-13T13:23:11.910365+0000","last_undegraded":"2024-09-13T13:23:11.910365+0000","last_fullsized":"2024-09-13T13:23:11.910365+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'21","last_scrub_stamp":"2024-09-13T13:22:05.680001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:05.680001+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:13.710980+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1038,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912250+0000","last_change":"2024-09-13T13:22:47.535815+0000","last_active":"2024-09-13T13:23:11.912250+0000","last_peered":"2024-09-13T13:23:11.912250+0000","last_clean":"2024-09-13T13:23:11.912250+0000","last_became_active":"2024-09-13T13:22:47.534993+0000","last_became_peered":"2024-09-13T13:22:47.534993+0000","last_unstale":"2024-09-13T13:23:11.912250+0000","last_undegraded":"2024-09-13T13:23:11.912250+0000","last_fullsized":"2024-09-13T13:23:11.912250+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:16.339803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:16.339803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:28.262817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1050,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642766+0000","last_change":"2024-09-13T13:22:06.261846+0000","last_active":"2024-09-13T13:23:11.642766+0000","last_peered":"2024-09-13T13:23:11.642766+0000","last_clean":"2024-09-13T13:23:11.642766+0000","last_became_active":"2024-09-13T13:01:06.211671+0000","last_became_peered":"2024-09-13T13:01:06.211671+0000","last_unstale":"2024-09-13T13:23:11.642766+0000","last_undegraded":"2024-09-13T13:23:11.642766+0000","last_fullsized":"2024-09-13T13:23:11.642766+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:06.261789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:06.261789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:18.852246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1045,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.909545+0000","last_change":"2024-09-13T13:22:47.237155+0000","last_active":"2024-09-13T13:23:11.909545+0000","last_peered":"2024-09-13T13:23:11.909545+0000","last_clean":"2024-09-13T13:23:11.909545+0000","last_became_active":"2024-09-13T13:01:08.209760+0000","last_became_peered":"2024-09-13T13:01:08.209760+0000","last_unstale":"2024-09-13T13:23:11.909545+0000","last_undegraded":"2024-09-13T13:23:11.909545+0000","last_fullsized":"2024-09-13T13:23:11.909545+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:47.237103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:47.237103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:56.255001+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"500'27","reported_seq":1130,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643631+0000","last_change":"2024-09-13T13:22:19.207124+0000","last_active":"2024-09-13T13:23:11.643631+0000","last_peered":"2024-09-13T13:23:11.643631+0000","last_clean":"2024-09-13T13:23:11.643631+0000","last_became_active":"2024-09-13T13:22:19.180944+0000","last_became_peered":"2024-09-13T13:22:19.180944+0000","last_unstale":"2024-09-13T13:23:11.643631+0000","last_undegraded":"2024-09-13T13:23:11.643631+0000","last_fullsized":"2024-09-13T13:23:11.643631+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'27","last_scrub_stamp":"2024-09-13T13:22:04.286906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:04.286906+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:21.875230+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1079,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910392+0000","last_change":"2024-09-13T13:22:38.892146+0000","last_active":"2024-09-13T13:23:11.910392+0000","last_peered":"2024-09-13T13:23:11.910392+0000","last_clean":"2024-09-13T13:23:11.910392+0000","last_became_active":"2024-09-13T13:22:19.167085+0000","last_became_peered":"2024-09-13T13:22:19.167085+0000","last_unstale":"2024-09-13T13:23:11.910392+0000","last_undegraded":"2024-09-13T13:23:11.910392+0000","last_fullsized":"2024-09-13T13:23:11.910392+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:38.892088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:38.892088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:57.793545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":70,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.912310+0000","last_change":"2024-09-13T13:22:22.229134+0000","last_active":"2024-09-13T13:23:11.912310+0000","last_peered":"2024-09-13T13:23:11.912310+0000","last_clean":"2024-09-13T13:23:11.912310+0000","last_became_active":"2024-09-13T13:22:19.183449+0000","last_became_peered":"2024-09-13T13:22:19.183449+0000","last_unstale":"2024-09-13T13:23:11.912310+0000","last_undegraded":"2024-09-13T13:23:11.912310+0000","last_fullsized":"2024-09-13T13:23:11.912310+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:22.229084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:22.229084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:48.688848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1033,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.642913+0000","last_change":"2024-09-13T13:23:05.419607+0000","last_active":"2024-09-13T13:23:11.642913+0000","last_peered":"2024-09-13T13:23:11.642913+0000","last_clean":"2024-09-13T13:23:11.642913+0000","last_became_active":"2024-09-13T13:04:02.760349+0000","last_became_peered":"2024-09-13T13:04:02.760349+0000","last_unstale":"2024-09-13T13:23:11.642913+0000","last_undegraded":"2024-09-13T13:23:11.642913+0000","last_fullsized":"2024-09-13T13:23:11.642913+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:23:05.419553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:23:05.419553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:24:17.518689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"500'12","reported_seq":1101,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643024+0000","last_change":"2024-09-13T13:22:09.293452+0000","last_active":"2024-09-13T13:23:11.643024+0000","last_peered":"2024-09-13T13:23:11.643024+0000","last_clean":"2024-09-13T13:23:11.643024+0000","last_became_active":"2024-09-13T13:16:12.073702+0000","last_became_peered":"2024-09-13T13:16:12.073702+0000","last_unstale":"2024-09-13T13:23:11.643024+0000","last_undegraded":"2024-09-13T13:23:11.643024+0000","last_fullsized":"2024-09-13T13:23:11.643024+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'12","last_scrub_stamp":"2024-09-13T13:22:09.293379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:09.293379+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:12.740712+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1073,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910980+0000","last_change":"2024-09-13T13:22:18.459760+0000","last_active":"2024-09-13T13:23:11.910980+0000","last_peered":"2024-09-13T13:23:11.910980+0000","last_clean":"2024-09-13T13:23:11.910980+0000","last_became_active":"2024-09-13T13:01:02.185233+0000","last_became_peered":"2024-09-13T13:01:02.185233+0000","last_unstale":"2024-09-13T13:23:11.910980+0000","last_undegraded":"2024-09-13T13:23:11.910980+0000","last_fullsized":"2024-09-13T13:23:11.910980+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:18.458898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:22:18.458898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:21.316464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1014,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.647834+0000","last_change":"2024-09-13T13:22:27.508192+0000","last_active":"2024-09-13T13:23:11.647834+0000","last_peered":"2024-09-13T13:23:11.647834+0000","last_clean":"2024-09-13T13:23:11.647834+0000","last_became_active":"2024-09-13T13:22:19.182103+0000","last_became_peered":"2024-09-13T13:22:19.182103+0000","last_unstale":"2024-09-13T13:23:11.647834+0000","last_undegraded":"2024-09-13T13:23:11.647834+0000","last_fullsized":"2024-09-13T13:23:11.647834+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:27.508129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:27.508129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:52.685949+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":669,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910161+0000","last_change":"2024-09-13T13:22:47.534836+0000","last_active":"2024-09-13T13:23:11.910161+0000","last_peered":"2024-09-13T13:23:11.910161+0000","last_clean":"2024-09-13T13:23:11.910161+0000","last_became_active":"2024-09-13T13:22:47.534502+0000","last_became_peered":"2024-09-13T13:22:47.534502+0000","last_unstale":"2024-09-13T13:23:11.910161+0000","last_undegraded":"2024-09-13T13:23:11.910161+0000","last_fullsized":"2024-09-13T13:23:11.910161+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:28.174939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:28.174939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:41.419485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1078,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910674+0000","last_change":"2024-09-13T13:22:21.004605+0000","last_active":"2024-09-13T13:23:11.910674+0000","last_peered":"2024-09-13T13:23:11.910674+0000","last_clean":"2024-09-13T13:23:11.910674+0000","last_became_active":"2024-09-13T13:22:16.235979+0000","last_became_peered":"2024-09-13T13:22:16.235979+0000","last_unstale":"2024-09-13T13:23:11.910674+0000","last_undegraded":"2024-09-13T13:23:11.910674+0000","last_fullsized":"2024-09-13T13:23:11.910674+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:21.004542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:07.192321+0000","last_clean_scrub_stamp":"2024-09-13T13:22:21.004542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:26.169929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1042,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.643184+0000","last_change":"2024-09-13T13:21:27.318950+0000","last_active":"2024-09-13T13:23:11.643184+0000","last_peered":"2024-09-13T13:23:11.643184+0000","last_clean":"2024-09-13T13:23:11.643184+0000","last_became_active":"2024-09-13T13:01:02.187587+0000","last_became_peered":"2024-09-13T13:01:02.187587+0000","last_unstale":"2024-09-13T13:23:11.643184+0000","last_undegraded":"2024-09-13T13:23:11.643184+0000","last_fullsized":"2024-09-13T13:23:11.643184+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:21:27.318866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:01.155841+0000","last_clean_scrub_stamp":"2024-09-13T13:21:27.318866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"500'27","reported_seq":1089,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.648461+0000","last_change":"2024-09-13T13:22:33.460384+0000","last_active":"2024-09-13T13:23:11.648461+0000","last_peered":"2024-09-13T13:23:11.648461+0000","last_clean":"2024-09-13T13:23:11.648461+0000","last_became_active":"2024-09-13T13:16:12.070277+0000","last_became_peered":"2024-09-13T13:16:12.070277+0000","last_unstale":"2024-09-13T13:23:11.648461+0000","last_undegraded":"2024-09-13T13:23:11.648461+0000","last_fullsized":"2024-09-13T13:23:11.648461+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'27","last_scrub_stamp":"2024-09-13T13:22:33.460303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:03.168885+0000","last_clean_scrub_stamp":"2024-09-13T13:22:33.460303+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:48.717656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1095,"reported_epoch":551,"state":"active+clean","last_fresh":"2024-09-13T13:23:11.910839+0000","last_change":"2024-09-13T13:22:19.186586+0000","last_active":"2024-09-13T13:23:11.910839+0000","last_peered":"2024-09-13T13:23:11.910839+0000","last_clean":"2024-09-13T13:23:11.910839+0000","last_became_active":"2024-09-13T13:22:19.185732+0000","last_became_peered":"2024-09-13T13:22:19.185732+0000","last_unstale":"2024-09-13T13:23:11.910839+0000","last_undegraded":"2024-09-13T13:23:11.910839+0000","last_fullsized":"2024-09-13T13:23:11.910839+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:22:03.489521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:01:05.184528+0000","last_clean_scrub_stamp":"2024-09-13T13:22:03.489521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:23:23.546162+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1534,"num_read_kb":1291,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":356,"num_bytes_recovered":7300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":976,"ondisk_log_size":976,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3042,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":608,"num_read_kb":603,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201551,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":701028,"kb_used_data":6228,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93044124,"statfs":{"total":95995035648,"available":95277182976,"internally_reserved":0,"allocated":6377472,"data_stored":5726477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":711457226},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":118,"seq":506806141161,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":782932,"kb_used_data":1348,"kb_used_omap":4,"kb_used_meta":781563,"kb_avail":92962220,"statfs":{"total":95995035648,"available":95193313280,"internally_reserved":0,"allocated":1380352,"data_stored":744112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4435,"internal_metadata":800321197},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692123,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":813344,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92931808,"statfs":{"total":95995035648,"available":95162171392,"internally_reserved":0,"allocated":1982464,"data_stored":1334949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":523,"seq":2246267895823,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72368,"kb_used_data":1376,"kb_used_omap":4,"kb_used_meta":70971,"kb_avail":93672784,"statfs":{"total":95995035648,"available":95920930816,"internally_reserved":0,"allocated":1409024,"data_stored":744717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":72674981},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117288,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":885560,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92859592,"statfs":{"total":95995035648,"available":95088222208,"internally_reserved":0,"allocated":2007040,"data_stored":1335915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":904788423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411627,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":714456,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93030696,"statfs":{"total":95995035648,"available":95263432704,"internally_reserved":0,"allocated":1642496,"data_stored":1006719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575154,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822512,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92922640,"statfs":{"total":95995035648,"available":95152783360,"internally_reserved":0,"allocated":1409024,"data_stored":744760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738681,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":961336,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92783816,"statfs":{"total":95995035648,"available":95010627584,"internally_reserved":0,"allocated":1417216,"data_stored":745808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":982972869},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:23:22.249 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T13:23:22.250 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b shell --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-13T13:23:22.517 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T13:23:22.914 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:22 smithi052 ceph-mon[36682]: purged_snaps scrub starts 2024-09-13T13:23:22.914 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:22 smithi052 ceph-mon[36682]: purged_snaps scrub ok 2024-09-13T13:23:22.914 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:22 smithi052 ceph-mon[36682]: from='client.37223 v1:172.21.15.52:0/2983455499' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:23:22.914 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:22 smithi052 ceph-mon[36682]: osdmap e553: 8 total, 8 up, 8 in 2024-09-13T13:23:22.914 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:22 smithi052 ceph-mon[29471]: purged_snaps scrub starts 2024-09-13T13:23:22.914 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:22 smithi052 ceph-mon[29471]: purged_snaps scrub ok 2024-09-13T13:23:22.914 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:22 smithi052 ceph-mon[29471]: from='client.37223 v1:172.21.15.52:0/2983455499' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:23:22.915 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:22 smithi052 ceph-mon[29471]: osdmap e553: 8 total, 8 up, 8 in 2024-09-13T13:23:23.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:22 smithi167 ceph-mon[33178]: purged_snaps scrub starts 2024-09-13T13:23:23.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:22 smithi167 ceph-mon[33178]: purged_snaps scrub ok 2024-09-13T13:23:23.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:22 smithi167 ceph-mon[33178]: from='client.37223 v1:172.21.15.52:0/2983455499' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T13:23:23.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:22 smithi167 ceph-mon[33178]: osdmap e553: 8 total, 8 up, 8 in 2024-09-13T13:23:23.488 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:23:23] ENGINE Bus STOPPING 2024-09-13T13:23:23.827 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:23:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T13:23:23.827 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:23:23] ENGINE Bus STOPPED 2024-09-13T13:23:23.827 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:23:23] ENGINE Bus STARTING 2024-09-13T13:23:23.828 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:23:23] ENGINE Serving on http://:::9283 2024-09-13T13:23:23.828 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:23:23] ENGINE Bus STARTED 2024-09-13T13:23:23.828 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[36682]: pgmap v1139: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:23:23.828 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[36682]: 3.14 scrub starts 2024-09-13T13:23:23.828 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[36682]: 3.14 scrub ok 2024-09-13T13:23:23.828 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[36682]: 4.a scrub starts 2024-09-13T13:23:23.828 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[36682]: 4.a scrub ok 2024-09-13T13:23:23.828 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2224667399' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-13T13:23:23.828 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[36682]: from='client.? v1:172.21.15.52:0/2224667399' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-13T13:23:23.828 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:23:23.829 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:23:23.829 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:23:23.829 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[36682]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:23:23.829 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[36682]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:23:23.829 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[29471]: pgmap v1139: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:23:23.829 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[29471]: 3.14 scrub starts 2024-09-13T13:23:23.829 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[29471]: 3.14 scrub ok 2024-09-13T13:23:23.829 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[29471]: 4.a scrub starts 2024-09-13T13:23:23.830 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[29471]: 4.a scrub ok 2024-09-13T13:23:23.830 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2224667399' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-13T13:23:23.830 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[29471]: from='client.? v1:172.21.15.52:0/2224667399' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-13T13:23:23.830 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:23:23.830 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:23:23.830 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:23:23.831 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[29471]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:23:23.831 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:23 smithi052 ceph-mon[29471]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:23:23.855 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-13T13:23:23.865 INFO:tasks.cephadm:Teardown begin 2024-09-13T13:23:23.866 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T13:23:23.916 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T13:23:23.945 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:23 smithi167 ceph-mon[33178]: pgmap v1139: 142 pgs: 142 active+clean; 583 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-13T13:23:23.945 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:23 smithi167 ceph-mon[33178]: 3.14 scrub starts 2024-09-13T13:23:23.945 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:23 smithi167 ceph-mon[33178]: 3.14 scrub ok 2024-09-13T13:23:23.945 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:23 smithi167 ceph-mon[33178]: 4.a scrub starts 2024-09-13T13:23:23.945 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:23 smithi167 ceph-mon[33178]: 4.a scrub ok 2024-09-13T13:23:23.945 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:23 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2224667399' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-13T13:23:23.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:23 smithi167 ceph-mon[33178]: from='client.? v1:172.21.15.52:0/2224667399' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-13T13:23:23.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:23 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T13:23:23.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:23 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T13:23:23.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:23 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T13:23:23.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:23 smithi167 ceph-mon[33178]: from='mgr.34538 ' entity='mgr.y' 2024-09-13T13:23:23.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:23 smithi167 ceph-mon[33178]: from='mgr.34538 v1:172.21.15.52:0/4160023175' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T13:23:23.971 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-13T13:23:23.971 DEBUG:teuthology.orchestra.run.smithi052:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T13:23:23.994 DEBUG:teuthology.orchestra.run.smithi167:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T13:23:24.037 INFO:tasks.cephadm:Stopping all daemons... 2024-09-13T13:23:24.038 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-13T13:23:24.038 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.a 2024-09-13T13:23:24.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:24 smithi052 systemd[1]: Stopping Ceph mon.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:23:24.639 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:24 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a[29467]: 2024-09-13T13:23:24.372+0000 7effae303640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:23:24.639 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:24 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a[29467]: 2024-09-13T13:23:24.372+0000 7effae303640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-13T13:23:24.903 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:24 smithi052 podman[210968]: 2024-09-13 13:23:24.638332038 +0000 UTC m=+0.389746625 container died 17b694ca3a37bff268f92cc01ddb74df39f87fc959b645e75df9aafba04906b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-13T13:23:25.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:25 smithi052 podman[210968]: 2024-09-13 13:23:25.105675946 +0000 UTC m=+0.857090532 container cleanup 17b694ca3a37bff268f92cc01ddb74df39f87fc959b645e75df9aafba04906b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-13T13:23:25.239 INFO:journalctl@ceph.mon.a.smithi052.stdout:Sep 13 13:23:25 smithi052 bash[210968]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-a 2024-09-13T13:23:25.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:23:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug there is no tcmu-runner data available 2024-09-13T13:23:25.523 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.a.service' 2024-09-13T13:23:25.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:23:25.574 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-13T13:23:25.574 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-13T13:23:25.574 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.c 2024-09-13T13:23:25.738 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:25 smithi052 systemd[1]: Stopping Ceph mon.c for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:23:26.141 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:25 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:23:25] ENGINE Bus STOPPING 2024-09-13T13:23:26.141 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: ::ffff:172.21.15.167 - - [13/Sep/2024:13:23:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-13T13:23:26.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:25 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-c[36678]: 2024-09-13T13:23:25.859+0000 7fd5741eb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:23:26.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:25 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-c[36678]: 2024-09-13T13:23:25.859+0000 7fd5741eb640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-13T13:23:26.141 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:26 smithi052 podman[211089]: 2024-09-13 13:23:26.034698222 +0000 UTC m=+0.270947213 container died cf921afd375aaa9878bfd4307955fb2573c16fad07a9fd31ef0acebb62e60e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T13:23:26.403 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:23:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T13:23:26.403 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:23:26] ENGINE Bus STOPPED 2024-09-13T13:23:26.403 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:23:26] ENGINE Bus STARTING 2024-09-13T13:23:26.403 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:23:26] ENGINE Serving on http://:::9283 2024-09-13T13:23:26.403 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:26 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: [13/Sep/2024:13:23:26] ENGINE Bus STARTED 2024-09-13T13:23:26.662 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:26 smithi052 podman[211089]: 2024-09-13 13:23:26.402617156 +0000 UTC m=+0.638866146 container cleanup cf921afd375aaa9878bfd4307955fb2573c16fad07a9fd31ef0acebb62e60e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-c, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-13T13:23:26.663 INFO:journalctl@ceph.mon.c.smithi052.stdout:Sep 13 13:23:26 smithi052 bash[211089]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-c 2024-09-13T13:23:26.806 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.c.service' 2024-09-13T13:23:26.842 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:23:26.842 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-13T13:23:26.842 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-13T13:23:26.843 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.b 2024-09-13T13:23:27.156 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:26 smithi167 systemd[1]: Stopping Ceph mon.b for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:23:27.570 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:27 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-b[33174]: 2024-09-13T13:23:27.153+0000 7f1bf6fad640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:23:27.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:27 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-b[33174]: 2024-09-13T13:23:27.153+0000 7f1bf6fad640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-13T13:23:27.571 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:27 smithi167 podman[156420]: 2024-09-13 13:23:27.358580044 +0000 UTC m=+0.316653019 container died 54e23c0ffd2c93c7da440fc647bfb5281a83d79c812bdc26be1d523d9c6f4926 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T13:23:28.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:27 smithi167 podman[156420]: 2024-09-13 13:23:27.837354256 +0000 UTC m=+0.795427231 container cleanup 54e23c0ffd2c93c7da440fc647bfb5281a83d79c812bdc26be1d523d9c6f4926 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-13T13:23:28.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Sep 13 13:23:27 smithi167 bash[156420]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mon-b 2024-09-13T13:23:28.255 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mon.b.service' 2024-09-13T13:23:28.294 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:23:28.294 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-13T13:23:28.294 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-13T13:23:28.294 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mgr.y 2024-09-13T13:23:28.607 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:28 smithi052 systemd[1]: Stopping Ceph mgr.y for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:23:28.866 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:28 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: 2024-09-13T13:23:28.606+0000 7f826f733640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:23:28.866 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:28 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y[29697]: 2024-09-13T13:23:28.606+0000 7f826f733640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-13T13:23:28.866 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:28 smithi052 podman[211221]: 2024-09-13 13:23:28.638335352 +0000 UTC m=+0.142886055 container died e79c228297b0de198d0fe51dd2076c97da9e42cff564972fee130c88f6d1d9ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-13T13:23:29.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:29 smithi052 podman[211221]: 2024-09-13 13:23:29.023573849 +0000 UTC m=+0.528124610 container cleanup e79c228297b0de198d0fe51dd2076c97da9e42cff564972fee130c88f6d1d9ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b0201b8c79733293453e7f10a10c7fa43119222b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-13T13:23:29.239 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Sep 13 13:23:29 smithi052 bash[211221]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-y 2024-09-13T13:23:29.396 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-13T13:23:29.433 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:23:29.433 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-13T13:23:29.433 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-13T13:23:29.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mgr.x 2024-09-13T13:23:29.773 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 13:23:29 smithi167 systemd[1]: Stopping Ceph mgr.x for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:23:30.069 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 13:23:29 smithi167 podman[156543]: 2024-09-13 13:23:29.773532977 +0000 UTC m=+0.145538968 container died 91caf7b5c2432b3014080c8f2c589b2df9a925faf933889bd280ae23a4af0a5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T13:23:30.320 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 13:23:30 smithi167 podman[156543]: 2024-09-13 13:23:30.159480099 +0000 UTC m=+0.531486079 container cleanup 91caf7b5c2432b3014080c8f2c589b2df9a925faf933889bd280ae23a4af0a5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T13:23:30.321 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Sep 13 13:23:30 smithi167 bash[156543]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-mgr-x 2024-09-13T13:23:30.587 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-13T13:23:30.622 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:23:30.623 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-13T13:23:30.623 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-13T13:23:30.623 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.0 2024-09-13T13:23:30.937 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:30 smithi052 systemd[1]: Stopping Ceph osd.0 for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:23:31.239 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:30 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0[39971]: 2024-09-13T13:23:30.936+0000 7f1d6ace5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:23:31.239 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:30 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0[39971]: 2024-09-13T13:23:30.936+0000 7f1d6ace5640 -1 osd.0 553 *** Got signal Terminated *** 2024-09-13T13:23:31.239 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:30 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0[39971]: 2024-09-13T13:23:30.936+0000 7f1d6ace5640 -1 osd.0 553 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T13:23:36.954 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:36 smithi052 podman[211342]: 2024-09-13 13:23:36.540194406 +0000 UTC m=+5.715500237 container died 7cb11a8c431dc613260c74854234b1f49d320394d6c29bfd70088f64b4bd5192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-13T13:23:36.954 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:36 smithi052 podman[211342]: 2024-09-13 13:23:36.953291991 +0000 UTC m=+6.128597812 container cleanup 7cb11a8c431dc613260c74854234b1f49d320394d6c29bfd70088f64b4bd5192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T13:23:37.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:36 smithi052 bash[211342]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0 2024-09-13T13:23:37.488 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:37 smithi052 podman[211357]: 2024-09-13 13:23:37.213241135 +0000 UTC m=+0.669970482 container remove 7cb11a8c431dc613260c74854234b1f49d320394d6c29bfd70088f64b4bd5192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-13T13:23:37.989 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:37 smithi052 podman[211445]: 2024-09-13 13:23:37.664061299 +0000 UTC m=+0.112468259 container create 94bb4f34b8918429860575aaa820785e37a5f93bb386799a1e2084793ce95494 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-13T13:23:37.989 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:37 smithi052 podman[211445]: 2024-09-13 13:23:37.568031924 +0000 UTC m=+0.016438836 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:23:38.378 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:38 smithi052 podman[211445]: 2024-09-13 13:23:38.017800076 +0000 UTC m=+0.466206988 container init 94bb4f34b8918429860575aaa820785e37a5f93bb386799a1e2084793ce95494 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-13T13:23:38.378 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:38 smithi052 podman[211445]: 2024-09-13 13:23:38.021585507 +0000 UTC m=+0.469992421 container start 94bb4f34b8918429860575aaa820785e37a5f93bb386799a1e2084793ce95494 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-13T13:23:38.379 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:38 smithi052 podman[211445]: 2024-09-13 13:23:38.108460808 +0000 UTC m=+0.556867781 container attach 94bb4f34b8918429860575aaa820785e37a5f93bb386799a1e2084793ce95494 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-13T13:23:38.671 INFO:journalctl@ceph.osd.0.smithi052.stdout:Sep 13 13:23:38 smithi052 podman[211445]: 2024-09-13 13:23:38.378870172 +0000 UTC m=+0.827277083 container died 94bb4f34b8918429860575aaa820785e37a5f93bb386799a1e2084793ce95494 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-13T13:23:38.947 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.0.service' 2024-09-13T13:23:38.988 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:23:38.988 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-13T13:23:38.988 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-13T13:23:38.988 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.1 2024-09-13T13:23:39.239 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:39 smithi052 systemd[1]: Stopping Ceph osd.1 for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:23:39.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:39 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1[45353]: 2024-09-13T13:23:39.339+0000 7fe37b3d7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:23:39.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:39 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1[45353]: 2024-09-13T13:23:39.339+0000 7fe37b3d7640 -1 osd.1 553 *** Got signal Terminated *** 2024-09-13T13:23:39.739 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:39 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1[45353]: 2024-09-13T13:23:39.339+0000 7fe37b3d7640 -1 osd.1 553 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T13:23:45.083 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:44 smithi052 podman[211550]: 2024-09-13 13:23:44.825278205 +0000 UTC m=+5.605528964 container died 5c5694dd503f618f459e9b40ebcf9c79b1d727b18c601fe59c6cd0921411cb82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-13T13:23:45.489 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:45 smithi052 podman[211550]: 2024-09-13 13:23:45.223253249 +0000 UTC m=+6.003503825 container cleanup 5c5694dd503f618f459e9b40ebcf9c79b1d727b18c601fe59c6cd0921411cb82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T13:23:45.489 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:45 smithi052 bash[211550]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1 2024-09-13T13:23:45.818 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:45 smithi052 podman[211566]: 2024-09-13 13:23:45.491404254 +0000 UTC m=+0.665090835 container remove 5c5694dd503f618f459e9b40ebcf9c79b1d727b18c601fe59c6cd0921411cb82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8) 2024-09-13T13:23:46.104 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:45 smithi052 podman[211651]: 2024-09-13 13:23:45.817617674 +0000 UTC m=+0.106661779 container create 2b501afa55f0194cc9e72188e7c69358d49d95389614c3dbb163e6d14047ee4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8) 2024-09-13T13:23:46.104 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:45 smithi052 podman[211651]: 2024-09-13 13:23:45.727111559 +0000 UTC m=+0.016155632 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:23:46.104 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:46 smithi052 podman[211651]: 2024-09-13 13:23:46.028482346 +0000 UTC m=+0.317526414 container init 2b501afa55f0194cc9e72188e7c69358d49d95389614c3dbb163e6d14047ee4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8) 2024-09-13T13:23:46.104 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:46 smithi052 podman[211651]: 2024-09-13 13:23:46.032123435 +0000 UTC m=+0.321167505 container start 2b501afa55f0194cc9e72188e7c69358d49d95389614c3dbb163e6d14047ee4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-13T13:23:46.104 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:46 smithi052 podman[211651]: 2024-09-13 13:23:46.103208801 +0000 UTC m=+0.392252883 container attach 2b501afa55f0194cc9e72188e7c69358d49d95389614c3dbb163e6d14047ee4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD) 2024-09-13T13:23:46.489 INFO:journalctl@ceph.osd.1.smithi052.stdout:Sep 13 13:23:46 smithi052 podman[211651]: 2024-09-13 13:23:46.223337141 +0000 UTC m=+0.512381205 container died 2b501afa55f0194cc9e72188e7c69358d49d95389614c3dbb163e6d14047ee4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-13T13:23:46.800 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.1.service' 2024-09-13T13:23:46.845 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:23:46.846 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-13T13:23:46.846 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-13T13:23:46.846 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.2 2024-09-13T13:23:47.211 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:46 smithi052 systemd[1]: Stopping Ceph osd.2 for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:23:47.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:47 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2[50706]: 2024-09-13T13:23:47.210+0000 7f75573e4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:23:47.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:47 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2[50706]: 2024-09-13T13:23:47.210+0000 7f75573e4640 -1 osd.2 553 *** Got signal Terminated *** 2024-09-13T13:23:47.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:47 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2[50706]: 2024-09-13T13:23:47.210+0000 7f75573e4640 -1 osd.2 553 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T13:23:53.086 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:52 smithi052 podman[211757]: 2024-09-13 13:23:52.834275593 +0000 UTC m=+5.758445724 container died 395be87781c5fd1188de52434ee35b9b32b1c9f06aff4bdf519d9e9ea0cb176d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-13T13:23:53.478 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:53 smithi052 podman[211757]: 2024-09-13 13:23:53.235039402 +0000 UTC m=+6.159209533 container cleanup 395be87781c5fd1188de52434ee35b9b32b1c9f06aff4bdf519d9e9ea0cb176d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-13T13:23:53.478 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:53 smithi052 bash[211757]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2 2024-09-13T13:23:53.739 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:53 smithi052 podman[211772]: 2024-09-13 13:23:53.478103127 +0000 UTC m=+0.640365223 container remove 395be87781c5fd1188de52434ee35b9b32b1c9f06aff4bdf519d9e9ea0cb176d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-13T13:23:54.082 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:53 smithi052 podman[211859]: 2024-09-13 13:23:53.717123473 +0000 UTC m=+0.016419212 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:23:54.082 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:53 smithi052 podman[211859]: 2024-09-13 13:23:53.862425742 +0000 UTC m=+0.161721467 container create faa2b4fa71446453ba6e4fea742d54ba89af9fd1426cdccc637d9ad4028322c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-13T13:23:54.082 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:54 smithi052 podman[211859]: 2024-09-13 13:23:54.081612219 +0000 UTC m=+0.380907946 container init faa2b4fa71446453ba6e4fea742d54ba89af9fd1426cdccc637d9ad4028322c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T13:23:54.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:54 smithi052 podman[211859]: 2024-09-13 13:23:54.085173239 +0000 UTC m=+0.384468976 container start faa2b4fa71446453ba6e4fea742d54ba89af9fd1426cdccc637d9ad4028322c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-13T13:23:54.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:54 smithi052 podman[211859]: 2024-09-13 13:23:54.164707471 +0000 UTC m=+0.464003205 container attach faa2b4fa71446453ba6e4fea742d54ba89af9fd1426cdccc637d9ad4028322c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T13:23:54.489 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:54 smithi052 podman[211859]: 2024-09-13 13:23:54.281423054 +0000 UTC m=+0.580718791 container died faa2b4fa71446453ba6e4fea742d54ba89af9fd1426cdccc637d9ad4028322c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-13T13:23:54.801 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.2.service' 2024-09-13T13:23:54.826 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:54 smithi052 podman[211883]: 2024-09-13 13:23:54.784818152 +0000 UTC m=+0.493007890 container remove faa2b4fa71446453ba6e4fea742d54ba89af9fd1426cdccc637d9ad4028322c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-13T13:23:54.826 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:54 smithi052 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.2.service: Deactivated successfully. 2024-09-13T13:23:54.826 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:54 smithi052 systemd[1]: Stopped Ceph osd.2 for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:23:54.826 INFO:journalctl@ceph.osd.2.smithi052.stdout:Sep 13 13:23:54 smithi052 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.2.service: Consumed 20min 57.108s CPU time. 2024-09-13T13:23:54.843 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:23:54.843 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-13T13:23:54.844 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-13T13:23:54.844 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.3 2024-09-13T13:23:55.172 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:23:54 smithi052 systemd[1]: Stopping Ceph osd.3 for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:23:55.488 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:23:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3[56695]: 2024-09-13T13:23:55.170+0000 7f0be4e06640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:23:55.489 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:23:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3[56695]: 2024-09-13T13:23:55.170+0000 7f0be4e06640 -1 osd.3 553 *** Got signal Terminated *** 2024-09-13T13:23:55.489 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:23:55 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3[56695]: 2024-09-13T13:23:55.170+0000 7f0be4e06640 -1 osd.3 553 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T13:23:57.571 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:23:57 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:23:57.357+0000 7f476a428640 -1 osd.4 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:32.512717+0000 front 2024-09-13T13:23:32.512950+0000 (oldest deadline 2024-09-13T13:23:56.612758+0000) 2024-09-13T13:23:57.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:23:57 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:23:57.244+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:23:58.570 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:23:58 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:23:58.260+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:23:58.571 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:23:58 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:23:58.398+0000 7f476a428640 -1 osd.4 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:32.512717+0000 front 2024-09-13T13:23:32.512950+0000 (oldest deadline 2024-09-13T13:23:56.612758+0000) 2024-09-13T13:23:58.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:23:58 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:23:58.460+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:23:58.739 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:23:58 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3[56695]: 2024-09-13T13:23:58.309+0000 7f0bdd413640 -1 osd.3 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.030315+0000 front 2024-09-13T13:23:33.030457+0000 (oldest deadline 2024-09-13T13:23:57.730575+0000) 2024-09-13T13:23:59.570 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:23:59 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:23:59.431+0000 7f476a428640 -1 osd.4 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:32.512717+0000 front 2024-09-13T13:23:32.512950+0000 (oldest deadline 2024-09-13T13:23:56.612758+0000) 2024-09-13T13:23:59.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:23:59 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:23:59.281+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:23:59.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:23:59 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:23:59.426+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:23:59.739 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:23:59 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3[56695]: 2024-09-13T13:23:59.292+0000 7f0bdd413640 -1 osd.3 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.030315+0000 front 2024-09-13T13:23:33.030457+0000 (oldest deadline 2024-09-13T13:23:57.730575+0000) 2024-09-13T13:24:00.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:00 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:00.240+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:00.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:00 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:00.378+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:00.572 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:00 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:24:00.414+0000 7f476a428640 -1 osd.4 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:32.512717+0000 front 2024-09-13T13:23:32.512950+0000 (oldest deadline 2024-09-13T13:23:56.612758+0000) 2024-09-13T13:24:00.989 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:24:00 smithi052 podman[211962]: 2024-09-13 13:24:00.725400157 +0000 UTC m=+5.656724286 container died 716a18e76020bd15f424a8af32513b48aa462557253d0bc8f0d107a5ed9ab726 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-13T13:24:01.219 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:00 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:00.926+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:01.239 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:24:01 smithi052 podman[211962]: 2024-09-13 13:24:01.153978159 +0000 UTC m=+6.085302329 container cleanup 716a18e76020bd15f424a8af32513b48aa462557253d0bc8f0d107a5ed9ab726 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-13T13:24:01.239 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:24:01 smithi052 bash[211962]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3 2024-09-13T13:24:01.570 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:01 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:01.366+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:01.571 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:01 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:24:01.464+0000 7f476a428640 -1 osd.4 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:32.512717+0000 front 2024-09-13T13:23:32.512950+0000 (oldest deadline 2024-09-13T13:23:56.612758+0000) 2024-09-13T13:24:01.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:01 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:01.218+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:01.739 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:24:01 smithi052 podman[211979]: 2024-09-13 13:24:01.347059716 +0000 UTC m=+0.618401681 container remove 716a18e76020bd15f424a8af32513b48aa462557253d0bc8f0d107a5ed9ab726 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-13T13:24:02.084 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:24:01 smithi052 podman[212068]: 2024-09-13 13:24:01.68310348 +0000 UTC m=+0.016433319 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:24:02.084 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:24:01 smithi052 podman[212068]: 2024-09-13 13:24:01.797713905 +0000 UTC m=+0.131043776 container create 3fdc4285af6b28cd164ab2575038a7864955307f995816b0cdfc72f5c21c1a72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-09-13T13:24:02.084 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:24:02 smithi052 podman[212068]: 2024-09-13 13:24:02.008521717 +0000 UTC m=+0.341851556 container init 3fdc4285af6b28cd164ab2575038a7864955307f995816b0cdfc72f5c21c1a72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-13T13:24:02.084 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:24:02 smithi052 podman[212068]: 2024-09-13 13:24:02.011876635 +0000 UTC m=+0.345206476 container start 3fdc4285af6b28cd164ab2575038a7864955307f995816b0cdfc72f5c21c1a72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2) 2024-09-13T13:24:02.084 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:24:02 smithi052 podman[212068]: 2024-09-13 13:24:02.083448786 +0000 UTC m=+0.416778623 container attach 3fdc4285af6b28cd164ab2575038a7864955307f995816b0cdfc72f5c21c1a72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-13T13:24:02.193 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:01 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:01.934+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:02.487 INFO:journalctl@ceph.osd.3.smithi052.stdout:Sep 13 13:24:02 smithi052 podman[212068]: 2024-09-13 13:24:02.202062551 +0000 UTC m=+0.535392381 container died 3fdc4285af6b28cd164ab2575038a7864955307f995816b0cdfc72f5c21c1a72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-13T13:24:02.570 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:02 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:24:02.433+0000 7f476a428640 -1 osd.4 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:32.512717+0000 front 2024-09-13T13:23:32.512950+0000 (oldest deadline 2024-09-13T13:23:56.612758+0000) 2024-09-13T13:24:02.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:02 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:02.192+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:02.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:02 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:02.391+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:02.859 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.3.service' 2024-09-13T13:24:02.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:24:02.903 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-13T13:24:02.903 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-13T13:24:02.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.4 2024-09-13T13:24:03.213 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:02 smithi167 systemd[1]: Stopping Ceph osd.4 for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:24:03.213 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:02 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:02.959+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:03.214 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:03.156+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:03.570 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:03.371+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:03.571 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:24:03.212+0000 7f476de13640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:24:03.571 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:24:03.212+0000 7f476de13640 -1 osd.4 553 *** Got signal Terminated *** 2024-09-13T13:24:03.571 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:24:03.212+0000 7f476de13640 -1 osd.4 553 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T13:24:03.571 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:24:03.408+0000 7f476a428640 -1 osd.4 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:32.512717+0000 front 2024-09-13T13:23:32.512950+0000 (oldest deadline 2024-09-13T13:23:56.612758+0000) 2024-09-13T13:24:04.320 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:03 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:03.938+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:04.321 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:04.166+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:04.820 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:04.338+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:04.821 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:24:04.363+0000 7f476a428640 -1 osd.4 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:32.512717+0000 front 2024-09-13T13:23:32.512950+0000 (oldest deadline 2024-09-13T13:23:56.612758+0000) 2024-09-13T13:24:05.320 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:05 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:05.145+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:05.321 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:04 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:04.915+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:05.821 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:05 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:24:05.347+0000 7f476a428640 -1 osd.4 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:32.512717+0000 front 2024-09-13T13:23:32.512950+0000 (oldest deadline 2024-09-13T13:23:56.612758+0000) 2024-09-13T13:24:05.821 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:05 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:05.384+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:06.148 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:05 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:05.868+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:06.570 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:06.342+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:06.571 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:24:06.356+0000 7f476a428640 -1 osd.4 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:32.512717+0000 front 2024-09-13T13:23:32.512950+0000 (oldest deadline 2024-09-13T13:23:56.612758+0000) 2024-09-13T13:24:06.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:06.147+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:06.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:06.147+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:07.163 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:06.869+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:07.570 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:07.380+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:07.571 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4[154667]: 2024-09-13T13:24:07.402+0000 7f476a428640 -1 osd.4 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:32.512717+0000 front 2024-09-13T13:23:32.512950+0000 (oldest deadline 2024-09-13T13:23:56.612758+0000) 2024-09-13T13:24:07.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:07.162+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:07.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:07.162+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:08.178 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:07.823+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:08.178 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:07 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:07.823+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:42.444474+0000 front 2024-09-13T13:23:42.444504+0000 (oldest deadline 2024-09-13T13:24:07.744610+0000) 2024-09-13T13:24:08.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:08.177+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:08.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:08.177+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:08.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:08.348+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:08.947 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:08 smithi167 podman[156666]: 2024-09-13 13:24:08.669652106 +0000 UTC m=+5.568797000 container died afb0d0c00411815038146794dc90f42f55bb5b3ec965fe0ecb87b5882d703db9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.vendor=CentOS) 2024-09-13T13:24:08.947 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:08.836+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:08.947 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:08 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:08.836+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:42.444474+0000 front 2024-09-13T13:23:42.444504+0000 (oldest deadline 2024-09-13T13:24:07.744610+0000) 2024-09-13T13:24:09.220 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:09 smithi167 podman[156666]: 2024-09-13 13:24:09.121161571 +0000 UTC m=+6.020306464 container cleanup afb0d0c00411815038146794dc90f42f55bb5b3ec965fe0ecb87b5882d703db9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8) 2024-09-13T13:24:09.220 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:09 smithi167 bash[156666]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4 2024-09-13T13:24:09.570 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:09 smithi167 podman[156683]: 2024-09-13 13:24:09.431253847 +0000 UTC m=+0.761549753 container remove afb0d0c00411815038146794dc90f42f55bb5b3ec965fe0ecb87b5882d703db9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-13T13:24:09.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:09 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:09.218+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:09.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:09 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:09.218+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:09.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:09 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:09.331+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:09.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:09 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:09.331+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:10.010 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:09 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:09.881+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:10.010 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:09 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:09.881+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:42.444474+0000 front 2024-09-13T13:23:42.444504+0000 (oldest deadline 2024-09-13T13:24:07.744610+0000) 2024-09-13T13:24:10.011 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:09 smithi167 podman[156769]: 2024-09-13 13:24:09.656192328 +0000 UTC m=+0.016232258 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:24:10.011 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:09 smithi167 podman[156769]: 2024-09-13 13:24:09.765435632 +0000 UTC m=+0.125475553 container create 677b6e372ed7fb75b613415738b7ba68705022cb47a4ad4549855c821737cafd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-13T13:24:10.317 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:10 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:10.184+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:10.317 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:10 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:10.184+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:10.317 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:10 smithi167 podman[156769]: 2024-09-13 13:24:10.010084931 +0000 UTC m=+0.370124860 container init 677b6e372ed7fb75b613415738b7ba68705022cb47a4ad4549855c821737cafd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8) 2024-09-13T13:24:10.318 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:10 smithi167 podman[156769]: 2024-09-13 13:24:10.013693339 +0000 UTC m=+0.373733264 container start 677b6e372ed7fb75b613415738b7ba68705022cb47a4ad4549855c821737cafd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T13:24:10.318 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:10 smithi167 podman[156769]: 2024-09-13 13:24:10.118367156 +0000 UTC m=+0.478407105 container attach 677b6e372ed7fb75b613415738b7ba68705022cb47a4ad4549855c821737cafd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-13T13:24:10.318 INFO:journalctl@ceph.osd.4.smithi167.stdout:Sep 13 13:24:10 smithi167 podman[156769]: 2024-09-13 13:24:10.203353439 +0000 UTC m=+0.563393363 container died 677b6e372ed7fb75b613415738b7ba68705022cb47a4ad4549855c821737cafd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-13T13:24:10.570 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:10 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:10.316+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:10.570 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:10 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:10.316+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:10.835 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.4.service' 2024-09-13T13:24:10.875 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:24:10.875 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-13T13:24:10.875 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-13T13:24:10.876 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.5 2024-09-13T13:24:11.070 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:10 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:10.908+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:11.071 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:10 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:10.908+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:42.444474+0000 front 2024-09-13T13:23:42.444504+0000 (oldest deadline 2024-09-13T13:24:07.744610+0000) 2024-09-13T13:24:11.071 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:10 smithi167 systemd[1]: Stopping Ceph osd.5 for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:24:11.570 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:11.210+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:11.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:11.210+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:11.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:11.332+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:11.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:11.332+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:11.571 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:11.259+0000 7f68c63ef640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:24:11.571 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:11.259+0000 7f68c63ef640 -1 osd.5 553 *** Got signal Terminated *** 2024-09-13T13:24:11.571 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:11.259+0000 7f68c63ef640 -1 osd.5 553 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T13:24:12.170 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:11.867+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:12.170 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:11 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:11.867+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:42.444474+0000 front 2024-09-13T13:23:42.444504+0000 (oldest deadline 2024-09-13T13:24:07.744610+0000) 2024-09-13T13:24:12.570 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:12 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:12.169+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:12.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:12 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:12.169+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:12.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:12 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:12.311+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:12.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:12 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:12.311+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:13.131 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:12 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:12.833+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:13.131 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:12 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:12.833+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:42.444474+0000 front 2024-09-13T13:23:42.444504+0000 (oldest deadline 2024-09-13T13:24:07.744610+0000) 2024-09-13T13:24:13.570 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:13.130+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:13.570 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:13.130+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:13.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:13.318+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:13.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:13.318+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:13.571 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:13.275Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.52:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.52:8765: connect: connection refused" 2024-09-13T13:24:13.571 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:13.277Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.52:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.52:8765: connect: connection refused" 2024-09-13T13:24:13.571 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:13.277Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.52:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.52:8765: connect: connection refused" 2024-09-13T13:24:13.571 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:13.280Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.52:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.52:8765: connect: connection refused" 2024-09-13T13:24:13.572 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:13.280Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.52:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.52:8765: connect: connection refused" 2024-09-13T13:24:13.572 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:13.280Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.52:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.52:8765: connect: connection refused" 2024-09-13T13:24:13.572 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:13.285Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.52:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.52:8765: connect: connection refused" 2024-09-13T13:24:14.148 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:13.856+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:14.148 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:13.856+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:42.444474+0000 front 2024-09-13T13:23:42.444504+0000 (oldest deadline 2024-09-13T13:24:07.744610+0000) 2024-09-13T13:24:14.148 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:13 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:13.856+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:47.745217+0000 front 2024-09-13T13:23:47.745269+0000 (oldest deadline 2024-09-13T13:24:13.645082+0000) 2024-09-13T13:24:14.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:14.281+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:14.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:14.281+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:14.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:14.147+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:14.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:14.147+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:14.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:14.147+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:50.122912+0000 front 2024-09-13T13:23:50.122937+0000 (oldest deadline 2024-09-13T13:24:13.622898+0000) 2024-09-13T13:24:15.111 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:14.855+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:15.111 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:14.855+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:42.444474+0000 front 2024-09-13T13:23:42.444504+0000 (oldest deadline 2024-09-13T13:24:07.744610+0000) 2024-09-13T13:24:15.111 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:14 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:14.855+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:47.745217+0000 front 2024-09-13T13:23:47.745269+0000 (oldest deadline 2024-09-13T13:24:13.645082+0000) 2024-09-13T13:24:15.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:15 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:15.110+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:15.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:15 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:15.110+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:15.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:15 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:15.110+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:50.122912+0000 front 2024-09-13T13:23:50.122937+0000 (oldest deadline 2024-09-13T13:24:13.622898+0000) 2024-09-13T13:24:15.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:15 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:15.237+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:15.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:15 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:15.237+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:16.227 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:15 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:15.870+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.443779+0000 front 2024-09-13T13:23:35.443960+0000 (oldest deadline 2024-09-13T13:24:00.743760+0000) 2024-09-13T13:24:16.228 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:15 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:15.870+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:42.444474+0000 front 2024-09-13T13:23:42.444504+0000 (oldest deadline 2024-09-13T13:24:07.744610+0000) 2024-09-13T13:24:16.228 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:15 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5[42595]: 2024-09-13T13:24:15.870+0000 7f68be9fc640 -1 osd.5 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:47.745217+0000 front 2024-09-13T13:23:47.745269+0000 (oldest deadline 2024-09-13T13:24:13.645082+0000) 2024-09-13T13:24:16.228 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:16 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:16.114+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:16.228 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:16 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:16.114+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:16.228 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:16 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:16.114+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:50.122912+0000 front 2024-09-13T13:23:50.122937+0000 (oldest deadline 2024-09-13T13:24:13.622898+0000) 2024-09-13T13:24:16.229 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:16 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:16.226+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:16.229 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:16 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:16.226+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:17.067 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:16 smithi167 podman[156873]: 2024-09-13 13:24:16.813282296 +0000 UTC m=+5.709940624 container died 9a6baf07fa1a233b97c2f82a76f13432c553145fa9cadc8695d56cda6af23e13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS) 2024-09-13T13:24:17.320 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:17 smithi167 podman[156873]: 2024-09-13 13:24:17.184752864 +0000 UTC m=+6.081411184 container cleanup 9a6baf07fa1a233b97c2f82a76f13432c553145fa9cadc8695d56cda6af23e13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-13T13:24:17.321 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:17 smithi167 bash[156873]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5 2024-09-13T13:24:17.321 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:17 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:17.200+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:17.321 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:17 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:17.200+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:17.321 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:17 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:17.200+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:17.321 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:17 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:17.066+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:17.321 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:17 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:17.066+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:17.321 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:17 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:17.066+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:50.122912+0000 front 2024-09-13T13:23:50.122937+0000 (oldest deadline 2024-09-13T13:24:13.622898+0000) 2024-09-13T13:24:17.814 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:17 smithi167 podman[156890]: 2024-09-13 13:24:17.503060875 +0000 UTC m=+0.681311565 container remove 9a6baf07fa1a233b97c2f82a76f13432c553145fa9cadc8695d56cda6af23e13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T13:24:18.065 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:18 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:18.049+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:18.065 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:18 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:18.049+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:18.065 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:18 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:18.049+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:50.122912+0000 front 2024-09-13T13:23:50.122937+0000 (oldest deadline 2024-09-13T13:24:13.622898+0000) 2024-09-13T13:24:18.065 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:17 smithi167 podman[156980]: 2024-09-13 13:24:17.717108545 +0000 UTC m=+0.015960978 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:24:18.065 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:17 smithi167 podman[156980]: 2024-09-13 13:24:17.828972272 +0000 UTC m=+0.127824703 container create 1d493c02650d9cc8168fa685e65d74a8c821045a90818d81cdd2a8e9a8e98d5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-13T13:24:18.320 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:18 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:18.208+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:18.320 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:18 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:18.208+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:18.321 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:18 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:18.208+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:18.321 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:18 smithi167 podman[156980]: 2024-09-13 13:24:18.064740081 +0000 UTC m=+0.363592505 container init 1d493c02650d9cc8168fa685e65d74a8c821045a90818d81cdd2a8e9a8e98d5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, maintainer=Guillaume Abrioux ) 2024-09-13T13:24:18.321 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:18 smithi167 podman[156980]: 2024-09-13 13:24:18.068243695 +0000 UTC m=+0.367096118 container start 1d493c02650d9cc8168fa685e65d74a8c821045a90818d81cdd2a8e9a8e98d5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-13T13:24:18.321 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:18 smithi167 podman[156980]: 2024-09-13 13:24:18.13989384 +0000 UTC m=+0.438746269 container attach 1d493c02650d9cc8168fa685e65d74a8c821045a90818d81cdd2a8e9a8e98d5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-13T13:24:18.321 INFO:journalctl@ceph.osd.5.smithi167.stdout:Sep 13 13:24:18 smithi167 podman[156980]: 2024-09-13 13:24:18.261742054 +0000 UTC m=+0.560594481 container died 1d493c02650d9cc8168fa685e65d74a8c821045a90818d81cdd2a8e9a8e98d5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-13T13:24:18.872 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.5.service' 2024-09-13T13:24:18.911 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:24:18.912 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-13T13:24:18.912 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-13T13:24:18.912 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.6 2024-09-13T13:24:19.214 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:18 smithi167 systemd[1]: Stopping Ceph osd.6 for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:24:19.214 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:19 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:19.037+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:19.215 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:19 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:19.037+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:19.215 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:19 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:19.037+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:50.122912+0000 front 2024-09-13T13:23:50.122937+0000 (oldest deadline 2024-09-13T13:24:13.622898+0000) 2024-09-13T13:24:19.570 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:19 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:19.272+0000 7f9ce144f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:24:19.570 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:19 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:19.272+0000 7f9ce144f640 -1 osd.6 553 *** Got signal Terminated *** 2024-09-13T13:24:19.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:19 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:19.272+0000 7f9ce144f640 -1 osd.6 553 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T13:24:19.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:19 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:19.214+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:19.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:19 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:19.214+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:19.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:19 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:19.214+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:20.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:20 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:20.072+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:20.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:20 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:20.072+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:20.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:20 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:20.072+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:50.122912+0000 front 2024-09-13T13:23:50.122937+0000 (oldest deadline 2024-09-13T13:24:13.622898+0000) 2024-09-13T13:24:20.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:20 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:20.245+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:20.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:20 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:20.245+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:20.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:20 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:20.245+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:21.570 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:21 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:21.120+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:21.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:21 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:21.120+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:21.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:21 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:21.120+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:50.122912+0000 front 2024-09-13T13:23:50.122937+0000 (oldest deadline 2024-09-13T13:24:13.622898+0000) 2024-09-13T13:24:21.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:21 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:21.244+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:21.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:21 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:21.244+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:21.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:21 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:21.244+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:22.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:22 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:22.099+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:22.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:22 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:22.099+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:22.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:22 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:22.099+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:50.122912+0000 front 2024-09-13T13:23:50.122937+0000 (oldest deadline 2024-09-13T13:24:13.622898+0000) 2024-09-13T13:24:22.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:22 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:22.234+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:22.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:22 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:22.234+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:22.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:22 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:22.234+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:22.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:22 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:22.234+0000 7f823c965640 -1 osd.7 553 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24484.0:1353 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e553) 2024-09-13T13:24:23.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:23.085+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:23.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:23.085+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:23.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:23.085+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:50.122912+0000 front 2024-09-13T13:23:50.122937+0000 (oldest deadline 2024-09-13T13:24:13.622898+0000) 2024-09-13T13:24:23.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:23.085+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6815 osd.3 since back 2024-09-13T13:23:58.923876+0000 front 2024-09-13T13:23:58.923827+0000 (oldest deadline 2024-09-13T13:24:22.423899+0000) 2024-09-13T13:24:23.572 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:23.189+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:23.572 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:23.189+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:23.572 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:23.189+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:23.572 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:23.189+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6815 osd.3 since back 2024-09-13T13:23:56.841381+0000 front 2024-09-13T13:23:56.841444+0000 (oldest deadline 2024-09-13T13:24:22.741378+0000) 2024-09-13T13:24:23.572 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:23 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:23.189+0000 7f823c965640 -1 osd.7 553 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24484.0:1353 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e553) 2024-09-13T13:24:24.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:24.096+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:33.920220+0000 front 2024-09-13T13:23:33.920120+0000 (oldest deadline 2024-09-13T13:23:56.819959+0000) 2024-09-13T13:24:24.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:24.096+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:40.821456+0000 front 2024-09-13T13:23:40.821445+0000 (oldest deadline 2024-09-13T13:24:06.121375+0000) 2024-09-13T13:24:24.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:24.096+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:50.122912+0000 front 2024-09-13T13:23:50.122937+0000 (oldest deadline 2024-09-13T13:24:13.622898+0000) 2024-09-13T13:24:24.571 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6[72139]: 2024-09-13T13:24:24.096+0000 7f9cd9a5c640 -1 osd.6 553 heartbeat_check: no reply from 172.21.15.52:6815 osd.3 since back 2024-09-13T13:23:58.923876+0000 front 2024-09-13T13:23:58.923827+0000 (oldest deadline 2024-09-13T13:24:22.423899+0000) 2024-09-13T13:24:24.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:24.233+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:24.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:24.233+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:24.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:24.233+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:24.572 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:24.233+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6815 osd.3 since back 2024-09-13T13:23:56.841381+0000 front 2024-09-13T13:23:56.841444+0000 (oldest deadline 2024-09-13T13:24:22.741378+0000) 2024-09-13T13:24:24.572 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:24 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:24.233+0000 7f823c965640 -1 osd.7 553 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24484.0:1353 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e553) 2024-09-13T13:24:25.024 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:24 smithi167 podman[157084]: 2024-09-13 13:24:24.751399276 +0000 UTC m=+5.619334232 container died c52ff0389c24748910bec7e997ee90cbe349a976842f5a59d8714e44a478dd6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T13:24:25.320 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:25 smithi167 podman[157084]: 2024-09-13 13:24:25.1307727 +0000 UTC m=+5.998707663 container cleanup c52ff0389c24748910bec7e997ee90cbe349a976842f5a59d8714e44a478dd6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T13:24:25.320 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:25 smithi167 bash[157084]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6 2024-09-13T13:24:25.321 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:25.239+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:25.321 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:25.239+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:25.321 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:25.239+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:25.321 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:25.239+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6815 osd.3 since back 2024-09-13T13:23:56.841381+0000 front 2024-09-13T13:23:56.841444+0000 (oldest deadline 2024-09-13T13:24:22.741378+0000) 2024-09-13T13:24:25.321 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:25 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:25.239+0000 7f823c965640 -1 osd.7 553 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24484.0:1353 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e553) 2024-09-13T13:24:25.776 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:25 smithi167 podman[157100]: 2024-09-13 13:24:25.457383999 +0000 UTC m=+0.709550592 container remove c52ff0389c24748910bec7e997ee90cbe349a976842f5a59d8714e44a478dd6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T13:24:26.036 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:25 smithi167 podman[157188]: 2024-09-13 13:24:25.678858206 +0000 UTC m=+0.015769004 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:24:26.036 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:25 smithi167 podman[157188]: 2024-09-13 13:24:25.799972102 +0000 UTC m=+0.136882891 container create f5dde7e4fb176b847f6f7d15c8266262eac8174cbcbbb898cd68d50ac8f39657 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T13:24:26.320 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:26 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:26.245+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:26.321 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:26 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:26.245+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:26.321 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:26 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:26.245+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:26.321 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:26 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:26.245+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6815 osd.3 since back 2024-09-13T13:23:56.841381+0000 front 2024-09-13T13:23:56.841444+0000 (oldest deadline 2024-09-13T13:24:22.741378+0000) 2024-09-13T13:24:26.321 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:26 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:26.245+0000 7f823c965640 -1 osd.7 553 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24484.0:1353 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e553) 2024-09-13T13:24:26.321 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:26 smithi167 podman[157188]: 2024-09-13 13:24:26.035659406 +0000 UTC m=+0.372570195 container init f5dde7e4fb176b847f6f7d15c8266262eac8174cbcbbb898cd68d50ac8f39657 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8) 2024-09-13T13:24:26.321 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:26 smithi167 podman[157188]: 2024-09-13 13:24:26.039279198 +0000 UTC m=+0.376189990 container start f5dde7e4fb176b847f6f7d15c8266262eac8174cbcbbb898cd68d50ac8f39657 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-13T13:24:26.321 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:26 smithi167 podman[157188]: 2024-09-13 13:24:26.118997591 +0000 UTC m=+0.455908390 container attach f5dde7e4fb176b847f6f7d15c8266262eac8174cbcbbb898cd68d50ac8f39657 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-13T13:24:26.321 INFO:journalctl@ceph.osd.6.smithi167.stdout:Sep 13 13:24:26 smithi167 podman[157188]: 2024-09-13 13:24:26.230230784 +0000 UTC m=+0.567141580 container died f5dde7e4fb176b847f6f7d15c8266262eac8174cbcbbb898cd68d50ac8f39657 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T13:24:26.859 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.6.service' 2024-09-13T13:24:26.900 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:24:26.901 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-13T13:24:26.901 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-13T13:24:26.901 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.7 2024-09-13T13:24:27.224 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:26 smithi167 systemd[1]: Stopping Ceph osd.7 for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:24:27.224 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:27 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:27.222+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:27.570 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:27 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:27.222+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:27.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:27 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:27.222+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:27.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:27 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:27.222+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6815 osd.3 since back 2024-09-13T13:23:56.841381+0000 front 2024-09-13T13:23:56.841444+0000 (oldest deadline 2024-09-13T13:24:22.741378+0000) 2024-09-13T13:24:27.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:27 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:27.222+0000 7f823c965640 -1 osd.7 553 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24484.0:1353 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e553) 2024-09-13T13:24:27.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:27 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:27.225+0000 7f8240350640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:24:27.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:27 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:27.225+0000 7f8240350640 -1 osd.7 553 *** Got signal Terminated *** 2024-09-13T13:24:27.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:27 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:27.225+0000 7f8240350640 -1 osd.7 553 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T13:24:28.498 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:28 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:28.261+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:28.499 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:28 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:28.261+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:28.499 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:28 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:28.261+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:28.499 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:28 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:28.261+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6815 osd.3 since back 2024-09-13T13:23:56.841381+0000 front 2024-09-13T13:23:56.841444+0000 (oldest deadline 2024-09-13T13:24:22.741378+0000) 2024-09-13T13:24:29.570 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:29 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:29.289+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:29.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:29 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:29.289+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:29.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:29 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:29.289+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:29.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:29 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:29.289+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6815 osd.3 since back 2024-09-13T13:23:56.841381+0000 front 2024-09-13T13:23:56.841444+0000 (oldest deadline 2024-09-13T13:24:22.741378+0000) 2024-09-13T13:24:29.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:29 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:29.289+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.167:6802 osd.4 since back 2024-09-13T13:24:06.242062+0000 front 2024-09-13T13:24:06.242124+0000 (oldest deadline 2024-09-13T13:24:29.142184+0000) 2024-09-13T13:24:30.570 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:30 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:30.254+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:30.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:30 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:30.254+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:30.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:30 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:30.254+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:30.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:30 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:30.254+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6815 osd.3 since back 2024-09-13T13:23:56.841381+0000 front 2024-09-13T13:23:56.841444+0000 (oldest deadline 2024-09-13T13:24:22.741378+0000) 2024-09-13T13:24:30.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:30 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:30.254+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.167:6802 osd.4 since back 2024-09-13T13:24:06.242062+0000 front 2024-09-13T13:24:06.242124+0000 (oldest deadline 2024-09-13T13:24:29.142184+0000) 2024-09-13T13:24:31.570 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:31 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:31.285+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6803 osd.0 since back 2024-09-13T13:23:35.238531+0000 front 2024-09-13T13:23:35.238800+0000 (oldest deadline 2024-09-13T13:23:57.538717+0000) 2024-09-13T13:24:31.570 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:31 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:31.285+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6807 osd.1 since back 2024-09-13T13:23:43.340278+0000 front 2024-09-13T13:23:43.340306+0000 (oldest deadline 2024-09-13T13:24:08.640165+0000) 2024-09-13T13:24:31.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:31 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:31.285+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6811 osd.2 since back 2024-09-13T13:23:52.141031+0000 front 2024-09-13T13:23:52.141100+0000 (oldest deadline 2024-09-13T13:24:16.840990+0000) 2024-09-13T13:24:31.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:31 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:31.285+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.52:6815 osd.3 since back 2024-09-13T13:23:56.841381+0000 front 2024-09-13T13:23:56.841444+0000 (oldest deadline 2024-09-13T13:24:22.741378+0000) 2024-09-13T13:24:31.571 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:31 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7[53730]: 2024-09-13T13:24:31.285+0000 7f823c965640 -1 osd.7 553 heartbeat_check: no reply from 172.21.15.167:6802 osd.4 since back 2024-09-13T13:24:06.242062+0000 front 2024-09-13T13:24:06.242124+0000 (oldest deadline 2024-09-13T13:24:29.142184+0000) 2024-09-13T13:24:33.135 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:32 smithi167 podman[157289]: 2024-09-13 13:24:32.861073184 +0000 UTC m=+5.733890746 container died e3b046f0771c7f2aeebd3aa6297de0d6309fae4772d951a614b24a4420f5071d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, io.buildah.version=1.37.2) 2024-09-13T13:24:33.502 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:33 smithi167 podman[157289]: 2024-09-13 13:24:33.234205343 +0000 UTC m=+6.107022945 container cleanup e3b046f0771c7f2aeebd3aa6297de0d6309fae4772d951a614b24a4420f5071d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, CEPH_POINT_RELEASE=) 2024-09-13T13:24:33.502 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:33 smithi167 bash[157289]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7 2024-09-13T13:24:33.813 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:33 smithi167 podman[157317]: 2024-09-13 13:24:33.501958313 +0000 UTC m=+0.634875592 container remove e3b046f0771c7f2aeebd3aa6297de0d6309fae4772d951a614b24a4420f5071d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-13T13:24:34.070 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:33 smithi167 podman[157407]: 2024-09-13 13:24:33.715200113 +0000 UTC m=+0.015683116 image pull 1641fe80ce7289a76d112fafe763b98dbdcc395dba398d6702278c8c710a2031 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b 2024-09-13T13:24:34.071 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:33 smithi167 podman[157407]: 2024-09-13 13:24:33.827705044 +0000 UTC m=+0.128188036 container create d0917129db5037e9ebf643521c07ea0bbda72d275e24f3893d065df14b1f0e9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-13T13:24:34.071 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:34 smithi167 podman[157407]: 2024-09-13 13:24:34.022171341 +0000 UTC m=+0.322654338 container init d0917129db5037e9ebf643521c07ea0bbda72d275e24f3893d065df14b1f0e9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-09-13T13:24:34.071 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:34 smithi167 podman[157407]: 2024-09-13 13:24:34.025424463 +0000 UTC m=+0.325907463 container start d0917129db5037e9ebf643521c07ea0bbda72d275e24f3893d065df14b1f0e9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7-deactivate, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T13:24:34.517 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:34 smithi167 podman[157407]: 2024-09-13 13:24:34.114057357 +0000 UTC m=+0.414540350 container attach d0917129db5037e9ebf643521c07ea0bbda72d275e24f3893d065df14b1f0e9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-13T13:24:34.517 INFO:journalctl@ceph.osd.7.smithi167.stdout:Sep 13 13:24:34 smithi167 podman[157407]: 2024-09-13 13:24:34.221569771 +0000 UTC m=+0.522052763 container died d0917129db5037e9ebf643521c07ea0bbda72d275e24f3893d065df14b1f0e9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T13:24:34.820 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@osd.7.service' 2024-09-13T13:24:34.859 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:24:34.859 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-13T13:24:34.859 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-13T13:24:34.860 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@rgw.foo.a 2024-09-13T13:24:35.146 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:24:34 smithi052 systemd[1]: Stopping Ceph rgw.foo.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:24:35.489 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:24:35 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-rgw-foo-a[61006]: 2024-09-13T13:24:35.145+0000 7efdf76fe640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T13:24:35.489 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:24:35 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-rgw-foo-a[61006]: 2024-09-13T13:24:35.145+0000 7efdfaf6a880 -1 shutting down 2024-09-13T13:24:45.489 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:24:45 smithi052 bash[212173]: time="2024-09-13T13:24:45Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-2b415b38-71cf-11ef-bceb-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-13T13:24:45.489 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:24:45 smithi052 podman[212173]: 2024-09-13 13:24:45.165132496 +0000 UTC m=+10.099485069 container died 213c35ecfe3fb5c3a47bb119660d8884fbc00d73801688aaf63cad32a113b7d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-rgw-foo-a, ceph=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T13:24:45.904 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:24:45 smithi052 podman[212173]: 2024-09-13 13:24:45.652664388 +0000 UTC m=+10.587017003 container cleanup 213c35ecfe3fb5c3a47bb119660d8884fbc00d73801688aaf63cad32a113b7d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-09-13T13:24:45.904 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Sep 13 13:24:45 smithi052 bash[212173]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-rgw-foo-a 2024-09-13T13:24:46.032 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@rgw.foo.a.service' 2024-09-13T13:24:46.068 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:24:46.069 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-13T13:24:46.069 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-13T13:24:46.069 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-2b415b38-71cf-11ef-bceb-c7b262605968@prometheus.a 2024-09-13T13:24:46.387 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 systemd[1]: Stopping Ceph prometheus.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:24:46.684 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:46.386Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T13:24:46.685 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:46.387Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-13T13:24:46.685 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:46.387Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-13T13:24:46.685 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:46.387Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T13:24:46.685 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:46.387Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-13T13:24:46.685 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:46.387Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-13T13:24:46.685 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:46.387Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T13:24:46.685 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:46.387Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-13T13:24:46.685 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:46.387Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-13T13:24:46.685 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:46.388Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T13:24:46.685 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:46.388Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-13T13:24:46.685 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a[66415]: ts=2024-09-13T13:24:46.388Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-13T13:24:46.686 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 podman[157510]: 2024-09-13 13:24:46.397022261 +0000 UTC m=+0.113031933 container died dcd09dadbde1fe64b21d69d218c5d29a69c99171a5eeb4c1764fac25b2800599 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T13:24:47.063 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 podman[157510]: 2024-09-13 13:24:46.803017674 +0000 UTC m=+0.519027343 container cleanup dcd09dadbde1fe64b21d69d218c5d29a69c99171a5eeb4c1764fac25b2800599 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T13:24:47.063 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Sep 13 13:24:46 smithi167 bash[157510]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T13:24:47.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b415b38-71cf-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-13T13:24:47.261 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T13:24:47.261 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-13T13:24:47.261 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T13:24:47.476 INFO:teuthology.orchestra.run.smithi052.stdout:Deleting cluster with fsid: 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T13:24:49.392 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:24:49 smithi052 systemd[1]: Stopping Ceph alertmanager.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:24:49.726 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:24:49 smithi052 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a[76921]: ts=2024-09-13T13:24:49.391Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T13:24:49.726 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:24:49 smithi052 podman[212665]: 2024-09-13 13:24:49.432535425 +0000 UTC m=+0.149252464 container died 5dfe3cbc32b00d0b7eea5249089d69a696a2b933add18233e68f9c5fa73d84a6 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T13:24:49.988 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:24:49 smithi052 podman[212665]: 2024-09-13 13:24:49.883298954 +0000 UTC m=+0.600016034 container cleanup 5dfe3cbc32b00d0b7eea5249089d69a696a2b933add18233e68f9c5fa73d84a6 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T13:24:49.989 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:24:49 smithi052 bash[212665]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T13:24:50.416 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:24:50 smithi052 podman[212677]: 2024-09-13 13:24:50.093069739 +0000 UTC m=+0.651205132 container remove 5dfe3cbc32b00d0b7eea5249089d69a696a2b933add18233e68f9c5fa73d84a6 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T13:24:50.417 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:24:50 smithi052 podman[212677]: 2024-09-13 13:24:50.168976261 +0000 UTC m=+0.727111652 volume remove 3b90f463cad43caaec15b3f968c518f912358b1b734fbd8b3ece0993a5080383 2024-09-13T13:24:50.739 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:24:50 smithi052 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T13:24:50.739 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:24:50 smithi052 systemd[1]: Stopped Ceph alertmanager.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:24:50.739 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Sep 13 13:24:50 smithi052 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.721s CPU time. 2024-09-13T13:24:50.739 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:24:50 smithi052 systemd[1]: Stopping Ceph node-exporter.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:24:51.214 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:24:50 smithi052 podman[212789]: 2024-09-13 13:24:50.882322279 +0000 UTC m=+0.093171691 container died 537bfc380255800bd7c5f4abb731e376ab876e2f0966684535ce6f09de19e8ea (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T13:24:51.214 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:24:51 smithi052 podman[212789]: 2024-09-13 13:24:51.213156863 +0000 UTC m=+0.424006269 container cleanup 537bfc380255800bd7c5f4abb731e376ab876e2f0966684535ce6f09de19e8ea (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T13:24:51.489 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:24:51 smithi052 bash[212789]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a 2024-09-13T13:24:51.489 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:24:51 smithi052 podman[212801]: 2024-09-13 13:24:51.389388045 +0000 UTC m=+0.510671584 container remove 537bfc380255800bd7c5f4abb731e376ab876e2f0966684535ce6f09de19e8ea (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T13:24:51.489 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:24:51 smithi052 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T13:24:51.989 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:24:51 smithi052 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-13T13:24:51.989 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:24:51 smithi052 systemd[1]: Stopped Ceph node-exporter.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:24:51.989 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Sep 13 13:24:51 smithi052 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 5.140s CPU time. 2024-09-13T13:24:52.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T13:24:52.503 INFO:teuthology.orchestra.run.smithi167.stdout:Deleting cluster with fsid: 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T13:24:54.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:24:53 smithi167 systemd[1]: Stopping Ceph iscsi.iscsi.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:24:54.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:24:54 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a[58162]: debug Shutdown received 2024-09-13T13:25:04.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:25:04 smithi167 bash[157918]: time="2024-09-13T13:25:04Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-13T13:25:04.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:25:04 smithi167 podman[157918]: 2024-09-13 13:25:04.067913321 +0000 UTC m=+10.122719153 container died f25d33ed71dd3efaf8ebf6f907c6aae0dce2d2e9554f8248319e260b757cc73a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-13T13:25:04.717 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:25:04 smithi167 podman[157918]: 2024-09-13 13:25:04.490548829 +0000 UTC m=+10.545354657 container cleanup f25d33ed71dd3efaf8ebf6f907c6aae0dce2d2e9554f8248319e260b757cc73a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-13T13:25:04.717 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:25:04 smithi167 bash[157918]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a 2024-09-13T13:25:05.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:25:04 smithi167 podman[157931]: 2024-09-13 13:25:04.716783707 +0000 UTC m=+0.647245585 container remove f25d33ed71dd3efaf8ebf6f907c6aae0dce2d2e9554f8248319e260b757cc73a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5ab18e92f956ea4280103e54eef24f76e1a43e77fc4dc78cee3cb12c7a94538b, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-iscsi-iscsi-a, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-0840-b0201b8, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-13T13:25:05.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:25:04 smithi167 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-13T13:25:05.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:25:04 smithi167 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-13T13:25:05.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:25:04 smithi167 systemd[1]: Stopped Ceph iscsi.iscsi.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:25:05.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Sep 13 13:25:04 smithi167 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Consumed 2.694s CPU time. 2024-09-13T13:25:06.518 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:25:06 smithi167 systemd[1]: Stopping Ceph grafana.a for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:25:06.820 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:25:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=server t=2024-09-13T13:25:06.68300113Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-13T13:25:06.820 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:25:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=tracing t=2024-09-13T13:25:06.683263655Z level=info msg="Closing tracing" 2024-09-13T13:25:06.820 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:25:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=ticker t=2024-09-13T13:25:06.683281521Z level=info msg=stopped last_tick=2024-09-13T13:25:00Z 2024-09-13T13:25:06.821 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:25:06 smithi167 ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a[60812]: logger=grafana-apiserver t=2024-09-13T13:25:06.683332863Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-09-13T13:25:06.821 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:25:06 smithi167 podman[158203]: 2024-09-13 13:25:06.703398943 +0000 UTC m=+0.125633184 container died e7993e166a4d71ddd2dc76860cc84404ee0818305461bd91dc884432945a27ee (image=quay.io/ceph/grafana:10.4.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T13:25:07.251 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:25:07 smithi167 podman[158203]: 2024-09-13 13:25:07.024469266 +0000 UTC m=+0.446703515 container cleanup e7993e166a4d71ddd2dc76860cc84404ee0818305461bd91dc884432945a27ee (image=quay.io/ceph/grafana:10.4.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T13:25:07.251 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:25:07 smithi167 bash[158203]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a 2024-09-13T13:25:07.570 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:25:07 smithi167 podman[158214]: 2024-09-13 13:25:07.250548678 +0000 UTC m=+0.540910559 container remove e7993e166a4d71ddd2dc76860cc84404ee0818305461bd91dc884432945a27ee (image=quay.io/ceph/grafana:10.4.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T13:25:07.571 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:25:07 smithi167 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-13T13:25:07.571 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:25:07 smithi167 systemd[1]: Stopped Ceph grafana.a for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:25:07.571 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Sep 13 13:25:07 smithi167 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@grafana.a.service: Consumed 15.307s CPU time. 2024-09-13T13:25:08.070 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:25:07 smithi167 systemd[1]: Stopping Ceph node-exporter.b for 2b415b38-71cf-11ef-bceb-c7b262605968... 2024-09-13T13:25:08.071 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:25:07 smithi167 podman[158333]: 2024-09-13 13:25:07.93992593 +0000 UTC m=+0.105480942 container died 5c4f411e5c51c291675468676bc3d8525e5d9e1e95e6e26c40c440af1e14c610 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T13:25:08.463 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:25:08 smithi167 podman[158333]: 2024-09-13 13:25:08.253869906 +0000 UTC m=+0.419424924 container cleanup 5c4f411e5c51c291675468676bc3d8525e5d9e1e95e6e26c40c440af1e14c610 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T13:25:08.464 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:25:08 smithi167 bash[158333]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b 2024-09-13T13:25:08.812 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:25:08 smithi167 podman[158347]: 2024-09-13 13:25:08.463483694 +0000 UTC m=+0.526812414 container remove 5c4f411e5c51c291675468676bc3d8525e5d9e1e95e6e26c40c440af1e14c610 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b415b38-71cf-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T13:25:08.812 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:25:08 smithi167 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-13T13:25:08.812 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:25:08 smithi167 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-13T13:25:08.812 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:25:08 smithi167 systemd[1]: Stopped Ceph node-exporter.b for 2b415b38-71cf-11ef-bceb-c7b262605968. 2024-09-13T13:25:08.812 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Sep 13 13:25:08 smithi167 systemd[1]: ceph-2b415b38-71cf-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 4.382s CPU time. 2024-09-13T13:25:09.343 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T13:25:09.377 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T13:25:09.406 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-13T13:25:09.409 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-13_12:30:34-orch:cephadm-wip-guits-main-2024-09-13-0840-distro-default-smithi/7903583/remote/smithi052/crash 2024-09-13T13:25:09.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/crash -- . 2024-09-13T13:25:09.461 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T13:25:09.461 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now 2024-09-13T13:25:09.463 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-13_12:30:34-orch:cephadm-wip-guits-main-2024-09-13-0840-distro-default-smithi/7903583/remote/smithi167/crash 2024-09-13T13:25:09.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/crash -- . 2024-09-13T13:25:09.504 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T13:25:09.504 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2024-09-13T13:25:09.506 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-13T13:25:09.506 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-13T13:25:09.615 INFO:tasks.cephadm:Compressing logs... 2024-09-13T13:25:09.616 DEBUG:teuthology.orchestra.run.smithi052:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T13:25:09.618 DEBUG:teuthology.orchestra.run.smithi167:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T13:25:09.660 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-13T13:25:09.661 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-13T13:25:09.661 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-13T13:25:09.662 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-13T13:25:09.662 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-13T13:25:09.662 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-13T13:25:09.662 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-13T13:25:09.663 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-13T13:25:09.664 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-13T13:25:09.664 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-13T13:25:09.664 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.log 2024-09-13T13:25:09.665 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.2.log: /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log 2024-09-13T13:25:09.665 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-13T13:25:09.665 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-13T13:25:09.665 INFO:teuthology.orchestra.run.smithi052.stderr: 59.0% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-09-13T13:25:09.675 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 91.7% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-13T13:25:09.700 INFO:teuthology.orchestra.run.smithi167.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-13T13:25:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/tcmu-runner.log 2024-09-13T13:25:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/objectstore_tool.2.log 2024-09-13T13:25:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-13T13:25:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/tcmu-runner.log: /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-13T13:25:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-13T13:25:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-13T13:25:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-13T13:25:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-13T13:25:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-13T13:25:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.log 2024-09-13T13:25:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.6.log: /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-13T13:25:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-13T13:25:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-13T13:25:09.782 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mon.b.log: 63.3% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/tcmu-runner.log.gz 2024-09-13T13:25:09.795 INFO:teuthology.orchestra.run.smithi052.stderr: 95.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-13T13:25:09.797 INFO:teuthology.orchestra.run.smithi052.stderr: 89.1% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-13T13:25:09.805 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/cephadm.log: 89.7% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-13T13:25:09.831 INFO:teuthology.orchestra.run.smithi167.stderr: 79.6% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-13T13:25:09.875 INFO:teuthology.orchestra.run.smithi167.stderr: 91.5% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-13T13:25:09.876 INFO:teuthology.orchestra.run.smithi052.stderr: 91.7% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-13T13:25:09.929 INFO:teuthology.orchestra.run.smithi167.stderr: 91.7% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-13T13:25:09.943 INFO:teuthology.orchestra.run.smithi167.stderr: 94.6% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/objectstore_tool.2.log.gz 2024-09-13T13:25:10.191 INFO:teuthology.orchestra.run.smithi167.stderr: 85.2% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-13T13:25:10.297 INFO:teuthology.orchestra.run.smithi167.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-13T13:25:10.469 INFO:teuthology.orchestra.run.smithi052.stderr: 93.9% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-13T13:25:16.013 INFO:teuthology.orchestra.run.smithi167.stderr: 90.5% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-13T13:25:17.098 INFO:teuthology.orchestra.run.smithi052.stderr: 89.7% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-13T13:25:24.901 INFO:teuthology.orchestra.run.smithi052.stderr: 90.5% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-13T13:25:47.028 INFO:teuthology.orchestra.run.smithi052.stderr: 90.0% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-13T13:29:49.929 INFO:teuthology.orchestra.run.smithi167.stderr: 93.0% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-13T13:31:53.377 INFO:teuthology.orchestra.run.smithi167.stderr: 93.8% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-13T13:32:22.229 INFO:teuthology.orchestra.run.smithi167.stderr: 93.3% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-13T13:32:27.222 INFO:teuthology.orchestra.run.smithi167.stderr: 93.4% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-13T13:32:27.225 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-09-13T13:32:27.225 INFO:teuthology.orchestra.run.smithi167.stderr:real 7m17.592s 2024-09-13T13:32:27.225 INFO:teuthology.orchestra.run.smithi167.stderr:user 4m52.841s 2024-09-13T13:32:27.225 INFO:teuthology.orchestra.run.smithi167.stderr:sys 0m13.462s 2024-09-13T13:40:59.320 INFO:teuthology.orchestra.run.smithi052.stderr: 93.1% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-13T13:41:32.383 INFO:teuthology.orchestra.run.smithi052.stderr: 93.0% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-13T13:41:46.094 INFO:teuthology.orchestra.run.smithi052.stderr: 93.2% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-13T13:42:41.285 INFO:teuthology.orchestra.run.smithi052.stderr: 92.8% -- replaced with /var/log/ceph/2b415b38-71cf-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-13T13:42:41.286 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-13T13:42:41.287 INFO:teuthology.orchestra.run.smithi052.stderr:real 17m31.653s 2024-09-13T13:42:41.287 INFO:teuthology.orchestra.run.smithi052.stderr:user 7m16.272s 2024-09-13T13:42:41.287 INFO:teuthology.orchestra.run.smithi052.stderr:sys 0m20.637s 2024-09-13T13:42:41.287 INFO:tasks.cephadm:Archiving logs... 2024-09-13T13:42:41.288 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-13_12:30:34-orch:cephadm-wip-guits-main-2024-09-13-0840-distro-default-smithi/7903583/remote/smithi052/log 2024-09-13T13:42:41.289 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T13:43:45.025 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-13_12:30:34-orch:cephadm-wip-guits-main-2024-09-13-0840-distro-default-smithi/7903583/remote/smithi167/log 2024-09-13T13:43:45.026 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T13:44:30.060 INFO:tasks.cephadm:Removing cluster... 2024-09-13T13:44:30.061 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 --force 2024-09-13T13:44:30.308 INFO:teuthology.orchestra.run.smithi052.stdout:Deleting cluster with fsid: 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T13:44:31.015 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b415b38-71cf-11ef-bceb-c7b262605968 --force 2024-09-13T13:44:31.262 INFO:teuthology.orchestra.run.smithi167.stdout:Deleting cluster with fsid: 2b415b38-71cf-11ef-bceb-c7b262605968 2024-09-13T13:44:31.889 INFO:tasks.cephadm:Removing cephadm ... 2024-09-13T13:44:31.890 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T13:44:31.910 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T13:44:31.927 INFO:tasks.cephadm:Teardown complete 2024-09-13T13:44:31.927 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T13:44:31.936 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T13:44:31.936 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T13:44:31.952 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T13:44:32.100 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T13:44:32.132 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T13:44:32.132 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:44:32.136 DEBUG:teuthology.orchestra.run.smithi167:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:44:32.151 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:44:32.151 INFO:teuthology.orchestra.run.smithi167.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:44:32.175 INFO:teuthology.orchestra.run.smithi167.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:44:32.176 INFO:teuthology.orchestra.run.smithi167.stdout:=============================================================================== 2024-09-13T13:44:32.176 INFO:teuthology.orchestra.run.smithi167.stdout:^* hv01.front.sepia.ceph.com 3 6 377 33 -42us[-8004ns] +/- 60ms 2024-09-13T13:44:32.176 INFO:teuthology.orchestra.run.smithi167.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -361us[ -327us] +/- 58ms 2024-09-13T13:44:32.176 INFO:teuthology.orchestra.run.smithi167.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +464us[ +497us] +/- 59ms 2024-09-13T13:44:32.176 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:44:32.184 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:44:32.184 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-09-13T13:44:32.184 INFO:teuthology.orchestra.run.smithi052.stdout:^* hv01.front.sepia.ceph.com 3 6 377 35 -53us[ -22us] +/- 60ms 2024-09-13T13:44:32.184 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -370us[ -370us] +/- 58ms 2024-09-13T13:44:32.184 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +464us[ +495us] +/- 59ms 2024-09-13T13:44:32.184 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:44:32.185 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T13:44:32.225 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T13:44:32.225 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T13:44:32.264 DEBUG:teuthology.orchestra.run.smithi052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T13:44:32.368 DEBUG:teuthology.orchestra.run.smithi167:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T13:44:32.439 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T13:44:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:44:32.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T13:44:32.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:44:32.496 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T13:44:32.507 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T13:44:32.516 INFO:teuthology.task.internal:Duration was 3439.404431 seconds 2024-09-13T13:44:32.516 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T13:44:32.525 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T13:44:32.525 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T13:44:32.528 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T13:44:32.586 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:44:32.596 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:44:32.868 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T13:44:32.868 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi052.front.sepia.ceph.com 2024-09-13T13:44:32.869 DEBUG:teuthology.orchestra.run.smithi052:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T13:44:32.901 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi167.front.sepia.ceph.com 2024-09-13T13:44:32.901 DEBUG:teuthology.orchestra.run.smithi167:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T13:44:32.925 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T13:44:32.925 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T13:44:32.943 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T13:44:32.992 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T13:44:32.992 DEBUG:teuthology.orchestra.run.smithi052:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T13:44:33.013 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T13:44:35.283 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T13:44:35.292 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T13:44:35.292 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T13:44:35.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T13:44:35.368 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T13:44:35.381 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:44:35.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:44:35.418 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = core 2024-09-13T13:44:35.421 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2024-09-13T13:44:35.444 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:44:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:44:35.500 DEBUG:teuthology.orchestra.run.smithi167:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:44:35.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:44:35.517 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T13:44:35.527 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T13:44:35.529 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-13_12:30:34-orch:cephadm-wip-guits-main-2024-09-13-0840-distro-default-smithi/7903583/remote/smithi052 2024-09-13T13:44:35.529 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T13:44:35.622 DEBUG:teuthology.misc:Transferring archived files from smithi167:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-13_12:30:34-orch:cephadm-wip-guits-main-2024-09-13-0840-distro-default-smithi/7903583/remote/smithi167 2024-09-13T13:44:35.623 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T13:44:35.666 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T13:44:35.666 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T13:44:35.670 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T13:44:35.718 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T13:44:35.730 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T13:44:35.730 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T13:44:35.741 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T13:44:35.741 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T13:44:35.744 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T13:44:35.757 INFO:teuthology.orchestra.run.smithi052.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 13:44 /home/ubuntu/cephtest 2024-09-13T13:44:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 13:44 /home/ubuntu/cephtest 2024-09-13T13:44:35.777 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T13:44:35.787 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T13:44:35.812 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} duration: 3439.404430627823 flavor: default owner: scheduled_gabrioux@teuthology success: true 2024-09-13T13:44:35.812 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T13:44:35.926 INFO:teuthology.run:pass